BTM

Multiple threads hang waiting on DiskForceWaitQueue

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: 1.3.3
  • Fix Version/s: 2.0.0
  • Labels:
    None
  • Environment:
    Ubuntu 9.10 Linux 2.6.31-21-generic #59-Ubuntu SMP Wed Mar 24 07:28:27 UTC 2010 x86_64 GNU/Linux
  • Number of attachments :
    1

Description

I am not sure this is a bug but it does occur intermittently and only a restart allows resuming processing. It seems a bunch of worker threads (e.g. DspThread-3-SectionSyncher) await on DiskForceWaitQueue.waitUntilNotContains() while bitronix-disk-force-batcher awaits on DiskForceWaitQueue.waitUntilNotEmpty. The whole log is attached. Please let me know if this is an expected situation and if there is something I can do to remediate it.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Ludovic Orban added a comment -

Only a race condition can provoke this kind of deadlock.

How often is that happening to you? How many CPU and/or cores does your machine have? Is that happening on production only or in some test environment?

Show
Ludovic Orban added a comment - Only a race condition can provoke this kind of deadlock. How often is that happening to you? How many CPU and/or cores does your machine have? Is that happening on production only or in some test environment?
Hide
Permalink
Ben Daici added a comment -

This has happened quite frequently since we deployed to the dry run environment, but occurs intermittently. The box has 1 quad-core CPU, Intel Xeon E5410.

Show
Ben Daici added a comment - This has happened quite frequently since we deployed to the dry run environment, but occurs intermittently. The box has 1 quad-core CPU, Intel Xeon E5410.
Hide
Permalink
Ludovic Orban added a comment -

I've identified a potential race condition in the disk force batcher thread, a build containing the fix has been uploaded here: http://snapshots.repository.codehaus.org/org/codehaus/btm/btm/2.0.0-SNAPSHOT/btm-2.0.0-SNAPSHOT.jar

I'm not sure this will solve your problem as I've been unable to reproduce the problem which is very hard without an exact copy of your environment.

Show
Ludovic Orban added a comment - I've identified a potential race condition in the disk force batcher thread, a build containing the fix has been uploaded here: http://snapshots.repository.codehaus.org/org/codehaus/btm/btm/2.0.0-SNAPSHOT/btm-2.0.0-SNAPSHOT.jar I'm not sure this will solve your problem as I've been unable to reproduce the problem which is very hard without an exact copy of your environment.
Hide
Permalink
Ben Daici added a comment -

Thanks, I will deploy the patch and let you know if we still see the issue.

Show
Ben Daici added a comment - Thanks, I will deploy the patch and let you know if we still see the issue.
Hide
Permalink
Ludovic Orban added a comment -

I haven't heard back, did the patched build help?

Show
Ludovic Orban added a comment - I haven't heard back, did the patched build help?
Hide
Permalink
Ben Daici added a comment -

I was also waiting for word from tech services. I have confirmed with them they haven't seen the issue since the patch was applied. I will certainly let you know if the problem recurs.

Show
Ben Daici added a comment - I was also waiting for word from tech services. I have confirmed with them they haven't seen the issue since the patch was applied. I will certainly let you know if the problem recurs.
Hide
Permalink
Ludovic Orban added a comment -

Marking this issue as resolved. Please reopen it if you see the problem reoccuring.

Show
Ludovic Orban added a comment - Marking this issue as resolved. Please reopen it if you see the problem reoccuring.

People

  • Assignee:
    Ludovic Orban
    Reporter:
    Ben Daici
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: