Session Manager in 15.2.69 stops working sometimes

[ebob42]

We have an IntraWeb ISAPI application that runs during the day, with an average of 300 concurrent IW sessions. CPU usage is between 10 and 30% all the time.

We use Delphi 11.3 with IntraWeb 15.2.69 (we also use 15.4.2, but not yet for production work).

However, sometimes, the session manager seems to "die" when terminating a session. We get an Access Violation, and as of that moment, new sessions will be created, but old ones (and new ones) will never be freed and removed again. As a consequence, the session count increases (we've seen 1000 concurrent sessions, with still "only" about 300 really active), and the memory usage increases as well, since these sessions are no longer destroyed. 
When we detect this situation, we often need to restart the application to solve the problem, although it will remain fully functional (until all memory of the server has been used, but this gives us a few hours to respond).

In other situations, even less frequent, the session manager seems to be unable to create new sessions. Connections are made, but no new sessions and no main forms are generated. The session count does not go up.

In both cases, the average CPU usage drops from the minimum of 10% CPU to about 0% CPU. I have a feeling this means that the "overhead" for the session manager is gone, and this helps in my thought that the session manager may be broken at those times.

Question: is there any way to inspect / query / interrogate the IntraWeb session manager to see if it's still alive? Restarting may be a problem as I do not want to kill the sessions that are still in active use (although the second case often leads to an IIS "Ping" reset, as the application pool itself may not respond to a ping within 90 seconds).

I also wonder if anybody else is seeing this behavior, or if we are just unlucky. I may very well be our application that upsets the session manager, but I wonder who and how.

Thanks in advance for any pointers, tips or hints.

Groetjes, Bob Swart

[joelcc]

We also run ISAPI with thousands of active users in a day. I recently moved from 15.2.49 to 15.3.9 and started having issues with running out of memory which in turn causes us to have to restart the apppool and for some reason sometimes the server.

I do not know if it is the same issue you are dealing with. At the same time as that release happened we also had a lot of code changes so maybe it is a memory leak in our code.(?) In looking at the recent logs; However, I do see where it might be related to a user that is uploading files. I am still trying to get to the bottom of the issue. If I see any issues with sessions, I will be sure to comment.

[Alexandre Machado]

Hi Dr. Bob,

According to your report, the scenario that you describe would fit into an hypothetical situation where the cleanup thread just dies. In theory it should never happen.

If your application is active you can use external tools (Sysinternals Process Explorer comes to mind http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx ) and see if you can identify the "IW Session Timeout Thread" running (the thread is named when created). It should be. If it's not running, something is wrong.


All work that the clean up thread does is protected in try..except blocks, meaning that an exception when your session is being destroyed would never affect the normal execution of the cleanup thread (let's say that you have an AV while destroying one of your forms because something unexpected happens, the error will be "swallowed" and the thread will continue cleaning up next thing in the list). But of course, a memory corruption is not impossible, and this can change everything....

When you say you get an AV, where exactly does it happen? Do you have the call stack (the exception logger, if active, should contain all details about it).

Other than that, I'd recommend you to move to IW 15.4.2. There were many important fixes after 15.3 branch has been released...

Cheers,

[Alexandre Machado]

I reviewed the clean up thread code and I couldn't find anything that could break it (unless something really unexpected happens, e.g. a CoInitialize() call fails...) but there are some points for improvement.

I'm adding some more protection and logging to the clean up thread execution code, in such a way that, unless the memory is corrupted and the execution code is really fubared, it should be more resilient to unexpected factors (and the logging should make it more clear about those issues).

[ebob42]

That would be very welcome, thank you very much!

By adding logging, do you mean logging that we can see using the new IntraWeb Server Monitor? Or in another way?

[ebob42]

If your application is active you can use external tools (Sysinternals Process Explorer comes to mind http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx ) and see if you can identify the "IW Session Timeout Thread" running (the thread is named when created). It should be. If it's not running, something is wrong.

Unfortunately, PRocess Explorer does not show the thread names, yet.

[ebob42]

When you say you get an AV, where exactly does it happen? Do you have the call stack (the exception logger, if active, should contain all details about it).

We get an "Invalid Pointer Operation" in the TIWForm.Dispose calling inherited Dispose:

Code:

TIWForm.Dispose inherited dispose
[000000005EDE9A63] System.ErrorAt
[000000005EDE9AB1] System.Error
[000000005EDE97D1] System.@FreeMem
[000000005EE1FFEA] System.SysUtils.StrDispose
[000000005F10BEC9] Vcl.Controls.TControl.Destroy
[00000000607CDA64] IWVCLBaseControl.TIWVCLBaseControl.Destroy (Line 176, "IWVCLBaseControl.pas")
[000000005EDEEED8] System.TObject.Free
[00000000607AACFB] IWContainer.TIWContainer.Dispose (Line 545, "IWContainer.pas")
[00000000607CE8E0] IWFormDsn.TIWModuleDsn.Dispose (Line 139, "IWFormDsn.pas")
[00000000607D12CD] IWBaseForm.TIWBaseForm.Dispose (Line 494, "IWBaseForm.pas")
[000000005FE6F569] IWBaseHTMLForm.TIWBaseHTMLForm.Dispose (Line 90, "IWBaseHTMLForm.pas")
[00000000607BFB5B] IWForm.TIWForm.Dispose (Line 523, "IWForm.pas")
[000000006064A8B3] UnitCaiwBaseForm.TIWCaiwBaseForm.Dispose (Line 1653, "UnitCaiwBaseForm.pas")
[000000005F1E3C66] IWVCLBaseContainer.TIWBaseContainer.Destroy (Line 228, "IWVCLBaseContainer.pas")
[00000000607D0E7D] IWBaseForm.TIWBaseForm.Destroy (Line 394, "IWBaseForm.pas")
[00000000607BF0CF] IWForm.TIWForm.Destroy (Line 361, "IWForm.pas")
[00000000606496D4] UnitCaiwBaseForm.TIWCaiwBaseForm.Destroy (Line 1541, "UnitCaiwBaseForm.pas")
[000000005EDEEED8] System.TObject.Free
[000000005EF25C8F] System.Classes.TComponent.DestroyComponents
[000000005EF2547F] System.Classes.TComponent.Destroy
[00000000607D9C0E] IWApplication.TIWApplication.Destroy (Line 819, "IWApplication.pas")
[000000005EDEEED8] System.TObject.Free
[000000005F1D75C7] IWServerControllerBase.TSessionTimeoutThread.DoExecute (Line 1658, "IWServerControllerBase.pas")
[000000005EFABE2C] IW.Common.Threads.TIWTimedThread.Execute (Line 228, "IW.Common.Threads.pas")
[000000005EF23263] System.Classes.ThreadProc
[000000005EDF224D] System.ThreadWrapper
[00007FFC896C4DE0] BaseThreadInitThunk
[00007FFC8ABBED9B] RtlUserThreadStart
: Invalid pointer operation

[ioan]

I have encountered a similar issue on my servers as well. For instance, I experienced Firedac errors indicating that there were no more connections available in the connection pool. What struck me as odd was that my "POOL_MaximumItems" setting was configured to allow for 1000 connections, yet only about 60 sessions were active at that particular moment.

To investigate further, I ran a query on the Firebird server to list the count of all active connections and what process is using them. Intraweb application was utilizing all 1000 connections. 

For reference, here's a screenshot that a customer sent me when they faced the same problem, and I requested them to run the query on the database:

[font="Proxima Nova Regular", "Helvetica Neue", Helvetica, Arial, sans-serif][/font]

[Alexandre Machado]

If your application is active you can use external tools (Sysinternals Process Explorer comes to mind http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx ) and see if you can identify the "IW Session Timeout Thread" running (the thread is named when created). It should be. If it's not running, something is wrong.

Unfortunately, PRocess Explorer does not show the thread names, yet.

Humm... yes, I believe you are right... I think only in a crash dump file the names of the threads are visible...

[ebob42]

I noticed that it's always the same thread that ends sessions, starting with the PrepareForSessionDestroy right until the IWUserSession itself is destroyed. I can identify that thread number from our logfiles now, so next time when I notice that session destroying is no longer possible, I'll check to see if that particular thread is still up and running.

I'm not sure if this is the "IW Session Timeout Thread", but it is the thread that will actually destroy the IWUserSession.