BTM
  1. BTM
  2. BTM-76

Multiple threads hang waiting on DiskForceWaitQueue

    Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.3.3
    • Fix Version/s: 2.0.0
    • Labels:
      None
    • Environment:
      Ubuntu 9.10 Linux 2.6.31-21-generic #59-Ubuntu SMP Wed Mar 24 07:28:27 UTC 2010 x86_64 GNU/Linux
    • Number of attachments :
      1

      Description

      I am not sure this is a bug but it does occur intermittently and only a restart allows resuming processing. It seems a bunch of worker threads (e.g. DspThread-3-SectionSyncher) await on DiskForceWaitQueue.waitUntilNotContains() while bitronix-disk-force-batcher awaits on DiskForceWaitQueue.waitUntilNotEmpty. The whole log is attached. Please let me know if this is an expected situation and if there is something I can do to remediate it.

        Activity

        Hide
        Ludovic Orban added a comment -

        Only a race condition can provoke this kind of deadlock.

        How often is that happening to you? How many CPU and/or cores does your machine have? Is that happening on production only or in some test environment?

        Show
        Ludovic Orban added a comment - Only a race condition can provoke this kind of deadlock. How often is that happening to you? How many CPU and/or cores does your machine have? Is that happening on production only or in some test environment?
        Hide
        Ben Daici added a comment -

        This has happened quite frequently since we deployed to the dry run environment, but occurs intermittently. The box has 1 quad-core CPU, Intel Xeon E5410.

        Show
        Ben Daici added a comment - This has happened quite frequently since we deployed to the dry run environment, but occurs intermittently. The box has 1 quad-core CPU, Intel Xeon E5410.
        Hide
        Ludovic Orban added a comment -

        I've identified a potential race condition in the disk force batcher thread, a build containing the fix has been uploaded here: http://snapshots.repository.codehaus.org/org/codehaus/btm/btm/2.0.0-SNAPSHOT/btm-2.0.0-SNAPSHOT.jar

        I'm not sure this will solve your problem as I've been unable to reproduce the problem which is very hard without an exact copy of your environment.

        Show
        Ludovic Orban added a comment - I've identified a potential race condition in the disk force batcher thread, a build containing the fix has been uploaded here: http://snapshots.repository.codehaus.org/org/codehaus/btm/btm/2.0.0-SNAPSHOT/btm-2.0.0-SNAPSHOT.jar I'm not sure this will solve your problem as I've been unable to reproduce the problem which is very hard without an exact copy of your environment.
        Hide
        Ben Daici added a comment -

        Thanks, I will deploy the patch and let you know if we still see the issue.

        Show
        Ben Daici added a comment - Thanks, I will deploy the patch and let you know if we still see the issue.
        Hide
        Ludovic Orban added a comment -

        I haven't heard back, did the patched build help?

        Show
        Ludovic Orban added a comment - I haven't heard back, did the patched build help?
        Hide
        Ben Daici added a comment -

        I was also waiting for word from tech services. I have confirmed with them they haven't seen the issue since the patch was applied. I will certainly let you know if the problem recurs.

        Show
        Ben Daici added a comment - I was also waiting for word from tech services. I have confirmed with them they haven't seen the issue since the patch was applied. I will certainly let you know if the problem recurs.
        Hide
        Ludovic Orban added a comment -

        Marking this issue as resolved. Please reopen it if you see the problem reoccuring.

        Show
        Ludovic Orban added a comment - Marking this issue as resolved. Please reopen it if you see the problem reoccuring.

          People

          • Assignee:
            Ludovic Orban
            Reporter:
            Ben Daici
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: