Jetty
  1. Jetty
  2. JETTY-331

Jetty HashSessionIdManager hangs on startup

    Details

    • Type: Improvement Improvement
    • Status: Resolved Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 6.1.1, 6.1.2rc4
    • Fix Version/s: 7.0.0pre4, 6.1.12.rc2
    • Component/s: Security and SSL, Servlet
    • Labels:
      None
    • Environment:
      JDK 1.5, 1.6
      tested with Jetty 6.1.1, 6.1.1rc4, 6.1.1rc5
      User Mode Linux
      Linux XXXXX 2.6.20-XXXXX #1 Mon Feb 19 20:31:43 EST 2007 i686 i686 i386 GNU/Linux
    • Number of attachments :
      0

      Description

      In my Linux environment, the HashSessionIdManager in its default configuration hangs on startup, preventing Jetty from starting up (and binding to a port, etc.). As inspected through JConsole, the SessionIdManager is in the 'starting' state, but never reaches the 'started' state. The stack trace of the 'main' thread is as follows:

      #
      Stack trace:
      #
      java.io.FileInputStream.readBytes(Native Method)
      #
      java.io.FileInputStream.read(FileInputStream.java:199)
      #
      java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
      #
      java.io.BufferedInputStream.read(BufferedInputStream.java:317)
      #
      java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
      #
      java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
      #
      java.io.BufferedInputStream.read(BufferedInputStream.java:317)
      #
      sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte(SeedGenerator.java:453)
      #
      sun.security.provider.SeedGenerator.getSeedBytes(SeedGenerator.java:123)
      #
      sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:118)
      #
      sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:114)
      #
      sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:171)
      #
      java.security.SecureRandom.nextBytes(SecureRandom.java:433)
      #
      java.security.SecureRandom.next(SecureRandom.java:455)
      #
      java.util.Random.nextLong(Random.java:284)
      #
      org.mortbay.jetty.servlet.HashSessionIdManager.doStart(HashSessionIdManager.java:105)
      #
      org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40)
      #
      org.mortbay.jetty.servlet.AbstractSessionManager.doStart(AbstractSessionManager.java:166)
      #
      org.mortbay.jetty.servlet.HashSessionManager.doStart(HashSessionManager.java:53)

      Some googling revealed that there was a known issue with SecureRandom on Linux in prior versions of the JDK, which was configured to read from /dev/random, which is blocking on Linux. Indeed /dev/random does block, however under 1.5 and 1.6, the configuration has changed to use /dev/urandom. I also tested explicitly setting the java.security.egd property to file:/dev/urandom. Neither of these solve the problem. Ultimately I found a solution: explicitly passing a java.util.Random (NOT a java.security.SecureRandom) to the HashSessionIdManager:

      <Set name="sessionIdManager">
      <New class="org.mortbay.jetty.servlet.HashSessionIdManager">
      <Arg>
      <!-- MUST explicitly set to java.util.Random -->
      <New class="java.util.Random"/>
      </Arg>
      <Set name="workerName">node1</Set>
      </New>
      </Set>

      There are several possible issues here:

      1. Maybe User Mode Linux is somehow interfering with the JDK SecureRandom API. Despite all settings appearing to be correct (and a test case using SecureRandom appearing to work), HashSessionIdManager still hangs by default.
      2. HashSessionIdManager (or some other jetty code) is doing something nefarious...like trying to read /dev/random and blocking (interestingly, JConsole indicated that the main thread was not blocked, but running...)
      3. This is simply "known behavior" that is not documented anywhere. If so, it should be documented because it is very difficult to diagnose.

      I can reproduce this on-demand if needed.

      Thank you

        Activity

        Hide
        Greg Wilkins added a comment -

        Thanks for the information. I had not seen this before!
        I will leave this issue open until we document on the wiki.

        Show
        Greg Wilkins added a comment - Thanks for the information. I had not seen this before! I will leave this issue open until we document on the wiki.
        Hide
        Jan Bartel added a comment -

        For everyone's future benefit:

        http://docs.codehaus.org/display/JETTY/Connectors+slow+to+startup

        regards
        Jan

        Show
        Jan Bartel added a comment - For everyone's future benefit: http://docs.codehaus.org/display/JETTY/Connectors+slow+to+startup regards Jan
        Hide
        Bogdan Calmac added a comment -

        I've encountered the same problem on Ubuntu 8.04, so I think we can say this is a general problem with Java + Linux on machines with not enough activity. See also these related Java bugs:

        I've investigated this further and discovered that using

        new SecureRandom()

        instead of

        SecureRandom.getInstance("SHA1PRNG")

        works fine. See also these two related java bugs:

        http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6202721
        http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6521844

        As mentioned before, this is a really difficult to track bug and I think it would be better if jetty worked properly out of the box instead of relying on an obscure configuration. Would it be acceptable from a security perspective for jetty to use "new SecureRandom()" instead of "SHA1PRNG"?

        I know this is not a Jetty problem, but if Sun does not want to fix it ...

        Thank you.

        Show
        Bogdan Calmac added a comment - I've encountered the same problem on Ubuntu 8.04, so I think we can say this is a general problem with Java + Linux on machines with not enough activity. See also these related Java bugs: I've investigated this further and discovered that using new SecureRandom() instead of SecureRandom.getInstance("SHA1PRNG") works fine. See also these two related java bugs: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6202721 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6521844 As mentioned before, this is a really difficult to track bug and I think it would be better if jetty worked properly out of the box instead of relying on an obscure configuration. Would it be acceptable from a security perspective for jetty to use "new SecureRandom()" instead of "SHA1PRNG"? I know this is not a Jetty problem, but if Sun does not want to fix it ... Thank you.
        Hide
        Greg Wilkins added a comment -

        We will investigate this, but will be cautious before changing.

        However, we may bracket the init with Log.info messages so that is it easier to diagnose.

        Show
        Greg Wilkins added a comment - We will investigate this, but will be cautious before changing. However, we may bracket the init with Log.info messages so that is it easier to diagnose.
        Hide
        Greg Wilkins added a comment -

        Another session clean up

        Show
        Greg Wilkins added a comment - Another session clean up
        Hide
        Jan Bartel added a comment -

        I cannot see why new SecureRandom() would work any differently to SecureRandom.getInstance("SHA1PRNG"), if the Sun provider is the only one installed on the system. In that case, SHAPRNG1 will be the only algorithm, so the two cases should be identical, at least according to the javadoc for jdk1.5

        Which raises another point, that the behaviour may be different if you switch jdk versions if you don't provide the name of the algorithm. Certainly the javadoc for jdk1.6 indicates different behaviour for new SecureRandom(). See
        http://java.sun.com/javase/6/docs/api/java/security/SecureRandom.html#SecureRandom().

        So, I would rather that we stuck with providing the explicit name of the algorithm, and providing a) better docs (done?) and b) better logging (I'll do that now).

        Jan

        Show
        Jan Bartel added a comment - I cannot see why new SecureRandom() would work any differently to SecureRandom.getInstance("SHA1PRNG"), if the Sun provider is the only one installed on the system. In that case, SHAPRNG1 will be the only algorithm, so the two cases should be identical, at least according to the javadoc for jdk1.5 Which raises another point, that the behaviour may be different if you switch jdk versions if you don't provide the name of the algorithm. Certainly the javadoc for jdk1.6 indicates different behaviour for new SecureRandom(). See http://java.sun.com/javase/6/docs/api/java/security/SecureRandom.html#SecureRandom( ). So, I would rather that we stuck with providing the explicit name of the algorithm, and providing a) better docs (done?) and b) better logging (I'll do that now). Jan
        Hide
        Jan Bartel added a comment -

        Log comments added in trunk rev 3608 and jetty-6 rev 3609.

        Show
        Jan Bartel added a comment - Log comments added in trunk rev 3608 and jetty-6 rev 3609.

          People

          • Assignee:
            Jan Bartel
            Reporter:
            Aaron Hamid
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: