Jetty
  1. Jetty
  2. JETTY-80

Jetty interrupts active threads.

    Details

    • Type: Bug Bug
    • Status: Resolved Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.0.0beta18, 6.0.0RC0
    • Component/s: None
    • Labels:
      None
    • Number of attachments :
      0

      Description

      It seems that Jetty tries to get back a thread from the active thread pool (perhaps after MaxIdleTime) by issuing the Thread#interrupt. It closes active NIO channels and breaks Berkley DB JE (see their SR #10463).
      a) Jetty must not intterrupt a thread that is still doing something. If user wants Jetty to do so, he should explicitly ask for it. Issuing Thread#interrupt may break user application running under Jetty in unpredictable ways.
      b) If my assumption about MaxIdleTime is right, it is incorrect to apply idle time counter to the thread which is not, in fact, idle. The time Jetty waits for active server threads to finish must be configured separately from the time Jetty waits for client user-agents. As i've said in JETTY-63 bug report, Jetty must either turn off all timeouts while a server thread is active, or use a separate timeout value for both TCP/IP SO_TIMEOUT and Jetty's internal expiration timeout.

        Issue Links

          Activity

          Hide
          Artem Kozarezov added a comment -

          For the record: this Thread#interrupt behaviour is specific for Jetty-6.

          Show
          Artem Kozarezov added a comment - For the record: this Thread#interrupt behaviour is specific for Jetty-6.
          Hide
          Artem Kozarezov added a comment -

          Evaluation: increasing MaxIdleTime to a much larger value mitigates the problem. It is also obvious, that BDB JE breaks with java.nio.channels.ClosedByInterruptException exactly after MaxIdleTime time.

          Show
          Artem Kozarezov added a comment - Evaluation: increasing MaxIdleTime to a much larger value mitigates the problem. It is also obvious, that BDB JE breaks with java.nio.channels.ClosedByInterruptException exactly after MaxIdleTime time.
          Hide
          Greg Wilkins added a comment -

          We have in fact already fixed this in svn trunk.

          we have simply removed the call to interrupt, but it is still valid for Jetty to close the connection if no bytes have been read or written for the idle period.

          Do you believe this behaviour could have been responsible for the FD leaks reported in JETTY-63?

          Show
          Greg Wilkins added a comment - We have in fact already fixed this in svn trunk. we have simply removed the call to interrupt, but it is still valid for Jetty to close the connection if no bytes have been read or written for the idle period. Do you believe this behaviour could have been responsible for the FD leaks reported in JETTY-63 ?
          Hide
          Greg Wilkins added a comment -

          Note also, that I think that well written code should handle interrupted exception - at least by ignoring it if they do not wish to be interrupted.

          Show
          Greg Wilkins added a comment - Note also, that I think that well written code should handle interrupted exception - at least by ignoring it if they do not wish to be interrupted.
          Hide
          Artem Kozarezov added a comment -

          > we have simply removed the call to interrupt,
          > but it is still valid for Jetty to close the connection
          > if no bytes have been read or written for the idle period.

          Great.
          Still, i do not agree that it is valid for Jetty to drop the connection only becouse the underlying application is "thinking" too long.

          > Do you believe this behaviour could have been responsible for the FD leaks reported in JETTY-63?

          I don't think so, but i will try the new version.

          > Note also, that I think that well written code should handle interrupted exception
          > - at least by ignoring it if they do not wish to be interrupted.

          I do not agree. Thread#interrupt is an action which is invoked by the application itself. It is not an expected behaviour, for the database thread, to receive Thread#interrupt. The only time when Thread#interrupt is received, when not programmed by an application logic, is when the JVM is shutting down. Ignoring Thread#interrupt altogether is not the correct behaviour also. For example, i can use Thread#interrupt when i'm going to close and unload a particular application, so Berkley DB is doing right there.

          Show
          Artem Kozarezov added a comment - > we have simply removed the call to interrupt, > but it is still valid for Jetty to close the connection > if no bytes have been read or written for the idle period. Great. Still, i do not agree that it is valid for Jetty to drop the connection only becouse the underlying application is "thinking" too long. > Do you believe this behaviour could have been responsible for the FD leaks reported in JETTY-63 ? I don't think so, but i will try the new version. > Note also, that I think that well written code should handle interrupted exception > - at least by ignoring it if they do not wish to be interrupted. I do not agree. Thread#interrupt is an action which is invoked by the application itself. It is not an expected behaviour, for the database thread, to receive Thread#interrupt. The only time when Thread#interrupt is received, when not programmed by an application logic, is when the JVM is shutting down. Ignoring Thread#interrupt altogether is not the correct behaviour also. For example, i can use Thread#interrupt when i'm going to close and unload a particular application, so Berkley DB is doing right there.
          Hide
          Greg Wilkins added a comment -

          Not meaning to be argumentative....

          Connections are a primary resource that Jetty manages and it is jetty's responsibility to make sure that these are not exhausted.
          An idle timeout is a standard way of controlling these resources and I think you will find that almost all servers will timeout idle connections.

          With regards to interrupt, it was added to the API as a better way of stopping a thread that stop(). Jetty starts these threads and should be able to stop them when it needs to. Note that if you only were to interrupt code that was your own, then you would never
          be able to call third party software, as you would be unaware if it can handle interrupts or not.

          Show
          Greg Wilkins added a comment - Not meaning to be argumentative.... Connections are a primary resource that Jetty manages and it is jetty's responsibility to make sure that these are not exhausted. An idle timeout is a standard way of controlling these resources and I think you will find that almost all servers will timeout idle connections. With regards to interrupt, it was added to the API as a better way of stopping a thread that stop(). Jetty starts these threads and should be able to stop them when it needs to. Note that if you only were to interrupt code that was your own, then you would never be able to call third party software, as you would be unaware if it can handle interrupts or not.
          Hide
          Artem Kozarezov added a comment -

          > Connections are a primary resource that Jetty manages
          > and it is jetty's responsibility to make sure that these are not exhausted.

          The number of active application threads is limited. Nobody prevents you from using a short timeout after the server thread is no longer active.
          Apache have a hard-coded limit of 250 running processes, that is not so with Jetty+NIO - no point in fighting for connections there, while all other resources (including HttpServletResponse and other Jetty structures) are still held by an application thread still running.

          > An idle timeout is a standard way of controlling these resources
          > and I think you will find that almost all servers will timeout idle connections.

          Please, see the "disableUploadTimeout" flag (http://tomcat.apache.org/tomcat-4.1-doc/config/coyote.html): "This flag allows the servlet container to use a different, longer connection timeout while a servlet is being executed, which in the end allows either the servlet a longer amount of time to complete its execution, or a longer timeout during data upload. If not specified, this attribute is set to "false"."

          Please, see http://ru.php.net/manual/en/features.connection-handling.php

          As you can see, most servers implement what i propose: in Tomcat you can ignore timeouts while the thread is still active; in Apache+mod_php you can manually set the timeouts for the current connection, so you can easily disable timeouts during activity and re-anable when done.

          > Jetty starts these threads and should be able to stop them when it needs to.

          Not when the thread is lent to an application.

          Stopping buggy infinite cycle threads might be a viable option, but it must be an option. JVM thread can't intercept or block the Thread#interrupt, unlike the CGI application, which can block or intercept signals. Stopping user application thread without any option to prevent it may lead to unprecedented problems in the application, like loosing all data or blowing a space station. The difference between short-running application and a long-running one is not as obvious as it seems, for example, a typical operation, like deleting a folder, can run in milliseconds for a small folder and hours for a big folder, so "implement long operations separately" is not a practical answer.

          > Note that if you only were to interrupt code that was your own, then you would never
          > be able to call third party software, as you would be unaware if it can handle interrupts or not.

          BDB JE behaviour considering Thread#interrupt is well docummented and is a part of its contract. I don't see how i "would be unaware if it can handle interrupts".

          Show
          Artem Kozarezov added a comment - > Connections are a primary resource that Jetty manages > and it is jetty's responsibility to make sure that these are not exhausted. The number of active application threads is limited. Nobody prevents you from using a short timeout after the server thread is no longer active. Apache have a hard-coded limit of 250 running processes, that is not so with Jetty+NIO - no point in fighting for connections there, while all other resources (including HttpServletResponse and other Jetty structures) are still held by an application thread still running. > An idle timeout is a standard way of controlling these resources > and I think you will find that almost all servers will timeout idle connections. Please, see the "disableUploadTimeout" flag ( http://tomcat.apache.org/tomcat-4.1-doc/config/coyote.html): "This flag allows the servlet container to use a different, longer connection timeout while a servlet is being executed, which in the end allows either the servlet a longer amount of time to complete its execution, or a longer timeout during data upload. If not specified, this attribute is set to "false"." Please, see http://ru.php.net/manual/en/features.connection-handling.php As you can see, most servers implement what i propose: in Tomcat you can ignore timeouts while the thread is still active; in Apache+mod_php you can manually set the timeouts for the current connection, so you can easily disable timeouts during activity and re-anable when done. > Jetty starts these threads and should be able to stop them when it needs to. Not when the thread is lent to an application. Stopping buggy infinite cycle threads might be a viable option, but it must be an option. JVM thread can't intercept or block the Thread#interrupt, unlike the CGI application, which can block or intercept signals. Stopping user application thread without any option to prevent it may lead to unprecedented problems in the application, like loosing all data or blowing a space station. The difference between short-running application and a long-running one is not as obvious as it seems, for example, a typical operation, like deleting a folder, can run in milliseconds for a small folder and hours for a big folder, so "implement long operations separately" is not a practical answer. > Note that if you only were to interrupt code that was your own, then you would never > be able to call third party software, as you would be unaware if it can handle interrupts or not. BDB JE behaviour considering Thread#interrupt is well docummented and is a part of its contract. I don't see how i "would be unaware if it can handle interrupts".
          Hide
          Greg Wilkins added a comment -

          OK - you have convinced me that we should look at either an alternative timeout or no timeout while request processing is taking place. I have opened JETTY-83 for this.

          But you have not convinced me that it is non a bug for an application to react badly to
          a Thread#interrupt. A good app should handle an interrupt by inspecting it's condition variables and deciding that it should or should not continue. Thread#interrupt replaced Thread#stop because
          it only interrupts a thread at well defined points, making it possible for an application to be
          well behaived.

          But then I am now not calling interrupt for a timeout - only when Server#stop is called.

          cheers

          Show
          Greg Wilkins added a comment - OK - you have convinced me that we should look at either an alternative timeout or no timeout while request processing is taking place. I have opened JETTY-83 for this. But you have not convinced me that it is non a bug for an application to react badly to a Thread#interrupt. A good app should handle an interrupt by inspecting it's condition variables and deciding that it should or should not continue. Thread#interrupt replaced Thread#stop because it only interrupts a thread at well defined points, making it possible for an application to be well behaived. But then I am now not calling interrupt for a timeout - only when Server#stop is called. cheers
          Hide
          Artem Kozarezov added a comment -

          > But you have not convinced me that it is non a bug for an application to react badly to
          > a Thread#interrupt. A good app should handle an interrupt by inspecting it's condition
          > variables and deciding that it should or should not continue. Thread#interrupt replaced
          > Thread#stop because it only interrupts a thread at well defined points,
          > making it possible for an application to be well behaived.

          Agreed. A good application should be configurable in what it does upon receiving Thread#interrupt - ignores it or closes, and preferably this should be configurable at runtime on a thread-local basis.

          Show
          Artem Kozarezov added a comment - > But you have not convinced me that it is non a bug for an application to react badly to > a Thread#interrupt. A good app should handle an interrupt by inspecting it's condition > variables and deciding that it should or should not continue. Thread#interrupt replaced > Thread#stop because it only interrupts a thread at well defined points, > making it possible for an application to be well behaived. Agreed. A good application should be configurable in what it does upon receiving Thread#interrupt - ignores it or closes, and preferably this should be configurable at runtime on a thread-local basis.

            People

            • Assignee:
              Unassigned
              Reporter:
              Artem Kozarezov
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: