Sporadically our production environment goes out of Heap, because expired sessions are not scavenged.
To validate this we have performed a heap dump when the server is hitting its maximum heap usage, before crashing the server.
In this .hprof heap dump we used SAP Memory Analyzer to do post-mortem analysis. (screenshot of problem in attachment)
What happens is that somehow the java.util.Timer object (_timer in the HashSessionManager) ends up not having any TimerTask assigned to it (we are absolutely sure about this - in other heap dumps the TimerTask is clearly there - pointing to the anonymous inner class that calls the scavenge() method).
Hence timed-out sessions are not garbage collected as they remain referenced through the SessionManager.
Analyzing the Session objects in the heap dump reveals a very large amount of HashSessionManager$$Session objects (about 10 times more then normal system usage). Ffurthermore the oldest Session objects have lastAccessed times that were more than 24 hours before the system heap problems.
Often the system can run weeks before experiencing this problem.
If more information is needed please contact me at dimitry_dhondt [at] yahoo.com