Issue Details (XML | Word | Printable)

Key: JETTY-331
Type: Improvement Improvement
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: Jan Bartel
Reporter: Aaron Hamid
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
Jetty

Jetty HashSessionIdManager hangs on startup

Created: 01/May/07 04:33 PM   Updated: 16/Sep/08 11:41 AM
Component/s: Security and SSL, Servlet
Affects Version/s: 6.1.1, 6.1.2rc4
Fix Version/s: 7.0.0pre4, 6.1.12.rc2

Time Tracking:
Not Specified

Environment: JDK 1.5, 1.6
tested with Jetty 6.1.1, 6.1.1rc4, 6.1.1rc5
User Mode Linux
Linux XXXXX 2.6.20-XXXXX #1 Mon Feb 19 20:31:43 EST 2007 i686 i686 i386 GNU/Linux


 Description  « Hide
In my Linux environment, the HashSessionIdManager in its default configuration hangs on startup, preventing Jetty from starting up (and binding to a port, etc.). As inspected through JConsole, the SessionIdManager is in the 'starting' state, but never reaches the 'started' state. The stack trace of the 'main' thread is as follows:

#
Stack trace:
#
java.io.FileInputStream.readBytes(Native Method)
#
java.io.FileInputStream.read(FileInputStream.java:199)
#
java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
#
java.io.BufferedInputStream.read(BufferedInputStream.java:317)
#
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
#
java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
#
java.io.BufferedInputStream.read(BufferedInputStream.java:317)
#
sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte(SeedGenerator.java:453)
#
sun.security.provider.SeedGenerator.getSeedBytes(SeedGenerator.java:123)
#
sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:118)
#
sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:114)
#
sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:171)
#
java.security.SecureRandom.nextBytes(SecureRandom.java:433)
#
java.security.SecureRandom.next(SecureRandom.java:455)
#
java.util.Random.nextLong(Random.java:284)
#
org.mortbay.jetty.servlet.HashSessionIdManager.doStart(HashSessionIdManager.java:105)
#
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40)
#
org.mortbay.jetty.servlet.AbstractSessionManager.doStart(AbstractSessionManager.java:166)
#
org.mortbay.jetty.servlet.HashSessionManager.doStart(HashSessionManager.java:53)

Some googling revealed that there was a known issue with SecureRandom on Linux in prior versions of the JDK, which was configured to read from /dev/random, which is blocking on Linux. Indeed /dev/random does block, however under 1.5 and 1.6, the configuration has changed to use /dev/urandom. I also tested explicitly setting the java.security.egd property to file:/dev/urandom. Neither of these solve the problem. Ultimately I found a solution: explicitly passing a java.util.Random (NOT a java.security.SecureRandom) to the HashSessionIdManager:

<Set name="sessionIdManager">
<New class="org.mortbay.jetty.servlet.HashSessionIdManager">
<Arg>
<!-- MUST explicitly set to java.util.Random -->
<New class="java.util.Random"/>
</Arg>
<Set name="workerName">node1</Set>
</New>
</Set>

There are several possible issues here:

1. Maybe User Mode Linux is somehow interfering with the JDK SecureRandom API. Despite all settings appearing to be correct (and a test case using SecureRandom appearing to work), HashSessionIdManager still hangs by default.
2. HashSessionIdManager (or some other jetty code) is doing something nefarious...like trying to read /dev/random and blocking (interestingly, JConsole indicated that the main thread was not blocked, but running...)
3. This is simply "known behavior" that is not documented anywhere. If so, it should be documented because it is very difficult to diagnose.

I can reproduce this on-demand if needed.

Thank you



 All   Comments   Work Log   Change History      Sort Order: Ascending order - Click to sort in descending order
Greg Wilkins added a comment - 01/May/07 05:16 PM
Thanks for the information. I had not seen this before!
I will leave this issue open until we document on the wiki.

Jan Bartel added a comment - 14/Mar/08 05:02 PM
For everyone's future benefit:

http://docs.codehaus.org/display/JETTY/Connectors+slow+to+startup

regards
Jan


Bogdan Calmac added a comment - 23/Jun/08 11:20 AM
I've encountered the same problem on Ubuntu 8.04, so I think we can say this is a general problem with Java + Linux on machines with not enough activity. See also these related Java bugs:

I've investigated this further and discovered that using

new SecureRandom()

instead of

SecureRandom.getInstance("SHA1PRNG")

works fine. See also these two related java bugs:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6202721
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6521844

As mentioned before, this is a really difficult to track bug and I think it would be better if jetty worked properly out of the box instead of relying on an obscure configuration. Would it be acceptable from a security perspective for jetty to use "new SecureRandom()" instead of "SHA1PRNG"?

I know this is not a Jetty problem, but if Sun does not want to fix it ...

Thank you.


Greg Wilkins added a comment - 23/Jun/08 06:21 PM
We will investigate this, but will be cautious before changing.

However, we may bracket the init with Log.info messages so that is it easier to diagnose.


Greg Wilkins added a comment - 04/Sep/08 10:19 PM
Another session clean up

Jan Bartel added a comment - 04/Sep/08 10:24 PM
I cannot see why new SecureRandom() would work any differently to SecureRandom.getInstance("SHA1PRNG"), if the Sun provider is the only one installed on the system. In that case, SHAPRNG1 will be the only algorithm, so the two cases should be identical, at least according to the javadoc for jdk1.5

Which raises another point, that the behaviour may be different if you switch jdk versions if you don't provide the name of the algorithm. Certainly the javadoc for jdk1.6 indicates different behaviour for new SecureRandom(). See
http://java.sun.com/javase/6/docs/api/java/security/SecureRandom.html#SecureRandom().

So, I would rather that we stuck with providing the explicit name of the algorithm, and providing a) better docs (done?) and b) better logging (I'll do that now).

Jan


Jan Bartel added a comment - 04/Sep/08 10:41 PM
Log comments added in trunk rev 3608 and jetty-6 rev 3609.