Jetty

Jetty HashSessionIdManager hangs on startup

Details

  • Type: Improvement Improvement
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 6.1.1, 6.1.2rc4
  • Fix Version/s: 7.0.0pre4, 6.1.12.rc2
  • Component/s: Security and SSL, Servlet
  • Labels:
    None
  • Environment:
    JDK 1.5, 1.6
    tested with Jetty 6.1.1, 6.1.1rc4, 6.1.1rc5
    User Mode Linux
    Linux XXXXX 2.6.20-XXXXX #1 Mon Feb 19 20:31:43 EST 2007 i686 i686 i386 GNU/Linux
  • Number of attachments :
    0

Description

In my Linux environment, the HashSessionIdManager in its default configuration hangs on startup, preventing Jetty from starting up (and binding to a port, etc.). As inspected through JConsole, the SessionIdManager is in the 'starting' state, but never reaches the 'started' state. The stack trace of the 'main' thread is as follows:

#
Stack trace:
#
java.io.FileInputStream.readBytes(Native Method)
#
java.io.FileInputStream.read(FileInputStream.java:199)
#
java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
#
java.io.BufferedInputStream.read(BufferedInputStream.java:317)
#
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
#
java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
#
java.io.BufferedInputStream.read(BufferedInputStream.java:317)
#
sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte(SeedGenerator.java:453)
#
sun.security.provider.SeedGenerator.getSeedBytes(SeedGenerator.java:123)
#
sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:118)
#
sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:114)
#
sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:171)
#
java.security.SecureRandom.nextBytes(SecureRandom.java:433)
#
java.security.SecureRandom.next(SecureRandom.java:455)
#
java.util.Random.nextLong(Random.java:284)
#
org.mortbay.jetty.servlet.HashSessionIdManager.doStart(HashSessionIdManager.java:105)
#
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40)
#
org.mortbay.jetty.servlet.AbstractSessionManager.doStart(AbstractSessionManager.java:166)
#
org.mortbay.jetty.servlet.HashSessionManager.doStart(HashSessionManager.java:53)

Some googling revealed that there was a known issue with SecureRandom on Linux in prior versions of the JDK, which was configured to read from /dev/random, which is blocking on Linux. Indeed /dev/random does block, however under 1.5 and 1.6, the configuration has changed to use /dev/urandom. I also tested explicitly setting the java.security.egd property to file:/dev/urandom. Neither of these solve the problem. Ultimately I found a solution: explicitly passing a java.util.Random (NOT a java.security.SecureRandom) to the HashSessionIdManager:

<Set name="sessionIdManager">
<New class="org.mortbay.jetty.servlet.HashSessionIdManager">
<Arg>
<!-- MUST explicitly set to java.util.Random -->
<New class="java.util.Random"/>
</Arg>
<Set name="workerName">node1</Set>
</New>
</Set>

There are several possible issues here:

1. Maybe User Mode Linux is somehow interfering with the JDK SecureRandom API. Despite all settings appearing to be correct (and a test case using SecureRandom appearing to work), HashSessionIdManager still hangs by default.
2. HashSessionIdManager (or some other jetty code) is doing something nefarious...like trying to read /dev/random and blocking (interestingly, JConsole indicated that the main thread was not blocked, but running...)
3. This is simply "known behavior" that is not documented anywhere. If so, it should be documented because it is very difficult to diagnose.

I can reproduce this on-demand if needed.

Thank you

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Greg Wilkins added a comment -

Thanks for the information. I had not seen this before!
I will leave this issue open until we document on the wiki.

Show
Greg Wilkins added a comment - Thanks for the information. I had not seen this before! I will leave this issue open until we document on the wiki.
Hide
Permalink
Jan Bartel added a comment -

For everyone's future benefit:

http://docs.codehaus.org/display/JETTY/Connectors+slow+to+startup

regards
Jan

Show
Jan Bartel added a comment - For everyone's future benefit: http://docs.codehaus.org/display/JETTY/Connectors+slow+to+startup regards Jan
Hide
Permalink
Bogdan Calmac added a comment -

I've encountered the same problem on Ubuntu 8.04, so I think we can say this is a general problem with Java + Linux on machines with not enough activity. See also these related Java bugs:

I've investigated this further and discovered that using

new SecureRandom()

instead of

SecureRandom.getInstance("SHA1PRNG")

works fine. See also these two related java bugs:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6202721
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6521844

As mentioned before, this is a really difficult to track bug and I think it would be better if jetty worked properly out of the box instead of relying on an obscure configuration. Would it be acceptable from a security perspective for jetty to use "new SecureRandom()" instead of "SHA1PRNG"?

I know this is not a Jetty problem, but if Sun does not want to fix it ...

Thank you.

Show
Bogdan Calmac added a comment - I've encountered the same problem on Ubuntu 8.04, so I think we can say this is a general problem with Java + Linux on machines with not enough activity. See also these related Java bugs: I've investigated this further and discovered that using new SecureRandom() instead of SecureRandom.getInstance("SHA1PRNG") works fine. See also these two related java bugs: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6202721 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6521844 As mentioned before, this is a really difficult to track bug and I think it would be better if jetty worked properly out of the box instead of relying on an obscure configuration. Would it be acceptable from a security perspective for jetty to use "new SecureRandom()" instead of "SHA1PRNG"? I know this is not a Jetty problem, but if Sun does not want to fix it ... Thank you.
Hide
Permalink
Greg Wilkins added a comment -

We will investigate this, but will be cautious before changing.

However, we may bracket the init with Log.info messages so that is it easier to diagnose.

Show
Greg Wilkins added a comment - We will investigate this, but will be cautious before changing. However, we may bracket the init with Log.info messages so that is it easier to diagnose.
Hide
Permalink
Greg Wilkins added a comment -

Another session clean up

Show
Greg Wilkins added a comment - Another session clean up
Hide
Permalink
Jan Bartel added a comment -

I cannot see why new SecureRandom() would work any differently to SecureRandom.getInstance("SHA1PRNG"), if the Sun provider is the only one installed on the system. In that case, SHAPRNG1 will be the only algorithm, so the two cases should be identical, at least according to the javadoc for jdk1.5

Which raises another point, that the behaviour may be different if you switch jdk versions if you don't provide the name of the algorithm. Certainly the javadoc for jdk1.6 indicates different behaviour for new SecureRandom(). See
http://java.sun.com/javase/6/docs/api/java/security/SecureRandom.html#SecureRandom().

So, I would rather that we stuck with providing the explicit name of the algorithm, and providing a) better docs (done?) and b) better logging (I'll do that now).

Jan

Show
Jan Bartel added a comment - I cannot see why new SecureRandom() would work any differently to SecureRandom.getInstance("SHA1PRNG"), if the Sun provider is the only one installed on the system. In that case, SHAPRNG1 will be the only algorithm, so the two cases should be identical, at least according to the javadoc for jdk1.5 Which raises another point, that the behaviour may be different if you switch jdk versions if you don't provide the name of the algorithm. Certainly the javadoc for jdk1.6 indicates different behaviour for new SecureRandom(). See http://java.sun.com/javase/6/docs/api/java/security/SecureRandom.html#SecureRandom(). So, I would rather that we stuck with providing the explicit name of the algorithm, and providing a) better docs (done?) and b) better logging (I'll do that now). Jan
Hide
Permalink
Jan Bartel added a comment -

Log comments added in trunk rev 3608 and jetty-6 rev 3609.

Show
Jan Bartel added a comment - Log comments added in trunk rev 3608 and jetty-6 rev 3609.

People

Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved:
Atlassian JIRA (v4.4.3#663-r165197) | Report a problem
Powered by a free Atlassian JIRA open source license for Codehaus. Try JIRA - bug tracking software for your team.