BTM
  1. BTM
  2. BTM-86

deadlock in JMS connection pool

    Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.1
    • Labels:
      None
    • Number of attachments :
      0

      Description

      A pool of multiple JMS connections can easily be deadlocked with moderate load.

      Here is the deadlock-related thread dump subset reported by the JVM:

      Found one Java-level deadlock:
      =============================
      "jmsTaskExecutor-2":
      waiting to lock monitor 0x00ac7b34 (object 0x084e4e40, a bitronix.tm.resource.jms.JmsPooledConnection),
      which is held by "jmsTaskExecutor-23"
      "jmsTaskExecutor-23":
      waiting to lock monitor 0x00ac7a74 (object 0x08503b68, a java.util.Collections$SynchronizedRandomAccessList),
      which is held by "jmsTaskExecutor-2"

      Java stack information for the threads listed above:
      ===================================================
      "jmsTaskExecutor-2":
      at bitronix.tm.resource.common.AbstractXAStatefulHolder.getState(AbstractXAStatefulHolder.java:23)

      • waiting to lock <0x084e4e40> (a bitronix.tm.resource.jms.JmsPooledConnection)
        at bitronix.tm.resource.jms.JmsPooledConnection.toString(JmsPooledConnection.java:183)
      • locked <0x08503b68> (a java.util.Collections$SynchronizedRandomAccessList)
        at java.lang.String.valueOf(String.java:2615)
        at java.lang.StringBuffer.append(StringBuffer.java:220)
      • locked <0x2761fa18> (a java.lang.StringBuffer)
        at bitronix.tm.resource.common.XAPool.containsXAResourceHolderMatchingGtrid(XAPool.java:330)
        at bitronix.tm.resource.common.XAPool.getNotAccessible(XAPool.java:318)
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:96)
      • locked <0x0845bd98> (a bitronix.tm.resource.common.XAPool)
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:68)
      • locked <0x0845bd98> (a bitronix.tm.resource.common.XAPool)
        at bitronix.tm.resource.jms.PoolingConnectionFactory.createConnection(PoolingConnectionFactory.java:108)
        at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.access$100(AbstractPollingMessageListenerContainer.java:78)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer$MessageListenerContainerResourceFactory.createConnection(AbstractPollingMessageListenerContainer.java:501)
        at org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:298)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:286)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:241)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:982)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:881)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:595)

      "jmsTaskExecutor-23":
      at bitronix.tm.resource.jms.JmsPooledConnection.toString(JmsPooledConnection.java:182)

      • waiting to lock <0x08503b68> (a java.util.Collections$SynchronizedRandomAccessList)
        at java.lang.String.valueOf(String.java:2615)
        at java.lang.StringBuffer.append(StringBuffer.java:220)
      • locked <0x275eb3f8> (a java.lang.StringBuffer)
        at bitronix.tm.resource.common.AbstractXAStatefulHolder.setState(AbstractXAStatefulHolder.java:35)
      • locked <0x084e4e40> (a bitronix.tm.resource.jms.JmsPooledConnection)
        at bitronix.tm.resource.common.DeferredReleaseSynchronization.afterCompletion(DeferredReleaseSynchronization.java:33)
        at bitronix.tm.BitronixTransaction.fireAfterCompletionEvent(BitronixTransaction.java:449)
        at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:221)
        at bitronix.tm.BitronixTransactionManager.commit(BitronixTransactionManager.java:104)
        at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1028)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:732)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:982)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:881)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:595)

      See http://old.nabble.com/deadlock-with-BTM-td29104439.html for the entire discussion.

        Activity

        Hide
        Ludovic Orban added a comment -

        This seems to be caused by the JmsPolledConnection.toString() method which accesses a synchronized method (sessions.size()) while this should never happen.

        An easy workaround is to lower the logs to INFO and the fix should simply remove the sessions.size() call from JmsPolledConnection.toString().

        Show
        Ludovic Orban added a comment - This seems to be caused by the JmsPolledConnection.toString() method which accesses a synchronized method (sessions.size()) while this should never happen. An easy workaround is to lower the logs to INFO and the fix should simply remove the sessions.size() call from JmsPolledConnection.toString().
        Hide
        Ludovic Orban added a comment -

        Fixed in trunk and user reported the fix is working.

        Show
        Ludovic Orban added a comment - Fixed in trunk and user reported the fix is working.

          People

          • Assignee:
            Ludovic Orban
            Reporter:
            Ludovic Orban
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: