Atozed Forums
Indy HTTP Server in linux daemon shutdown problem - Printable Version

+- Atozed Forums (https://www.atozed.com/forums)
+-- Forum: Indy (https://www.atozed.com/forums/forum-8.html)
+--- Forum: Indy General Discussion (https://www.atozed.com/forums/forum-9.html)
+--- Thread: Indy HTTP Server in linux daemon shutdown problem (/thread-2432.html)

Indy HTTP Server in linux daemon shutdown problem - ZGabrovski@gmail.com - 06-13-2021

I have a small linux/windows daemon written under Lazarus/FPC.
There is some TThread descendant class, which is created at daemon startup event.
The class create Indy HTTP Server in it constructor. The HTTP Server works fine both as linux daemon/windows service.
There is code that set HTTPServer.Active := False and FreeAndNil Httpserver on thread termination.
Under Windows everything working perfect.
But under linux HTTPServer.Active:=false causes HTTPServer to hang up.

Even when I start a daemon with -r and try to press Ctrl+C it will not terminate. Some times after that I received Äccess violation" or "Segmentation fault".

Any ideas? I think it is some-how related with synchronization (TThread.Synchroinize does not work under ??) Can somebody help?

RE: Indy HTTP Server in linux daemon shutdown problem - ZGabrovski@gmail.com - 06-14-2021

I found where the problem comes from.
I see that under debug mode there is no problem and it working fine.
I check what was the difference and found that when the options "check and assertions" in "Compiler options->Debug" are switched on everything is ok.
Now the daemon works fine.

RE: Indy HTTP Server in linux daemon shutdown problem - BartKindt - 09-10-2021

(06-14-2021, 06:17 AM)ZGabrovski@gmail.com Wrote: I found where the problem comes from.
I see that under debug mode there is no problem and it working fine.
I check what was the difference and found that when the options "check and assertions" in "Compiler options->Debug" are switched on everything is ok.
Now the daemon works fine.

I have the same problem, in Delphi 10.4, compiling for Linux.
But what you have done is not the cause of the problem, it seems to be a work-around.
What is happening (and this is a day of intense testing here) if that the system loops in "TerminateAllYarns" and does not come out of it.

This is a BUG in the Indy system, and I cannot fix this; can one of the Indy developers please try to fix this problem?
It isd a _permanent_ bug, and occurs simply every time after the HTTP server is started, at least *one* connection is made to it, and then it is closed.

Code:
I added lots of debug syslogs in the Indy code, and I got this at the end:

IdCustomTCPServer.TerminateAllThreads calls TerminateAllYarns
Sep  1 07:37:47 debian11 sartrackserver[33098]: IdScheduler.TerminateAllYarns loop: Count=1
Sep  1 07:37:47 debian11 sartrackserver[33098]: IdSchedulerOfThread.TerminateYarn
Sep  1 07:37:47 debian11 sartrackserver[33098]: IdSchedulerOfThread.TerminateYarn calls LThread.Stop
Sep  1 07:37:48 debian11 sartrackserver[33098]: IdScheduler.TerminateAllYarns DONE
Sep  1 07:37:48 debian11 sartrackserver[33098]: IdScheduler.TerminateAllYarns loop: Count=1
Sep  1 07:37:48 debian11 sartrackserver[33098]: IdSchedulerOfThread.TerminateYarn
Sep  1 07:37:48 debian11 sartrackserver[33098]: IdSchedulerOfThread.TerminateYarn calls LThread.Stop
Sep  1 07:37:48 debian11 sartrackserver[33098]: IdScheduler.TerminateAllYarns DONE
Sep  1 07:37:48 debian11 sartrackserver[33098]: IdScheduler.TerminateAllYarns loop: Count=1
Etc.

And the code looks like this:

Code:
procedure TIdSchedulerOfThread.TerminateYarn(AYarn: TIdYarn);
var
  LYarn: TIdYarnOfThread;
  LThread: TIdThreadWithTask;
begin
  // Bart
  syslog(LOG_NOTICE, 'IdSchedulerOfThread.TerminateYarn');
  Assert(AYarn<>nil);
  LYarn := TIdYarnOfThread(AYarn);
  LThread := LYarn.Thread;
  if (LThread <> nil) and (not LThread.Suspended) then begin
    // Is still running and will free itself
    syslog(LOG_NOTICE, 'IdSchedulerOfThread.TerminateYarn calls LThread.Stop');
    LThread.Stop;
    // Dont free the yarn. The thread frees it (IdThread.pas)
  end else
  begin
    // If suspended, was created but never started
    // ie waiting on connection accept

    // RLebeau: free the yarn here as well. This allows TIdSchedulerOfThreadPool
    // to put the suspended thread, if present, back in the pool.
    syslog(LOG_NOTICE, 'IdSchedulerOfThread.TerminateYarn calls IdDisposeAndNil(LYarn)');
    IdDisposeAndNil(LYarn);
  end;
end;

procedure TIdThread.Stop;
begin
  syslog(LOG_NOTICE, 'IdThread.Stop: Try enter lock');
  FLock.Enter;
  syslog(LOG_NOTICE, 'IdThread.Stop: locked');
  try
    if not Stopped then
    begin
      case FStopMode of
        smTerminate: begin syslog(LOG_NOTICE, 'IdThread.Stop calls Terminate'); Terminate; end
        // smSuspend: ;{DO not suspend here. Suspend is immediate. See Execute for implementation};
        else syslog(LOG_NOTICE, 'IdThread.Stop does NOT call Terminate');
      end;
      Include(FOptions, itoStopped);
    end else syslog(LOG_NOTICE, 'IdThread.Stop: is Stopped. Not doing anything');
  finally FLock.Leave; end;
end;

procedure TIdThread.Terminate;
begin
  //this assert can only raise if terminate is called on an already-destroyed thread
  Assert(FLock<>nil);

  FLock.Enter;
  try
    syslog(LOG_NOTICE, 'IdThread.Terminate calls Include');
    Include(FOptions, itoStopped);
    syslog(LOG_NOTICE, 'IdThread.Terminate calls inherited Terminate');
    inherited Terminate;
  finally FLock.Leave; end;
end;


RE: Indy HTTP Server in linux daemon shutdown problem - rlebeau - 09-10-2021

(09-10-2021, 07:23 PM)BartKindt Wrote: What is happening (and this is a day of intense testing here) if that the system loops in "TerminateAllYarns" and does not come out of it.

Clearly not intense enough. The hanging shutdown is just a SYMPTOM of ANOTHER PROBLEM. TerminateAllYarns() is simply waiting for all client threads to fully terminate, but there is a client thread that is not terminating properly.

(09-10-2021, 07:23 PM)BartKindt Wrote: This is a BUG in the Indy system, and I cannot fix this

I am not convinced yet that this is a bug in Indy itself. There is obviously a deadlock occurring in the thread that owns the Yarn in question, but I have seen no effort yet to try to diagnose the root cause of THAT issue. You should be able to connect a client to the server and then use a debugger to step through that client thread and find out why it is not responding to the server being shutdown. The client thread is likely blocked on something and thus is not even seeing the server's request for termination.

(09-10-2021, 07:23 PM)BartKindt Wrote: can one of the Indy developers please try to fix this problem?

In any case, this has been recently discussed on StackOverflow: IdHTTPServer hangs on Shutdown in Linux. And is now reported in Indy's issue tracker as #375: HTTPServer loops on shutdown under Linux.