Details
-
Type:
Bug
-
Status:
Closed
-
Priority:
Critical
-
Resolution: Fixed
-
Affects Version/s: 2.0.0
-
Fix Version/s: 2.0.1
-
Labels:None
-
Number of attachments :
Description
A pool of multiple JMS connections can easily be deadlocked with moderate load.
Here is the deadlock-related thread dump subset reported by the JVM:
Found one Java-level deadlock:
=============================
"jmsTaskExecutor-2":
waiting to lock monitor 0x00ac7b34 (object 0x084e4e40, a bitronix.tm.resource.jms.JmsPooledConnection),
which is held by "jmsTaskExecutor-23"
"jmsTaskExecutor-23":
waiting to lock monitor 0x00ac7a74 (object 0x08503b68, a java.util.Collections$SynchronizedRandomAccessList),
which is held by "jmsTaskExecutor-2"
Java stack information for the threads listed above:
===================================================
"jmsTaskExecutor-2":
at bitronix.tm.resource.common.AbstractXAStatefulHolder.getState(AbstractXAStatefulHolder.java:23)
- waiting to lock <0x084e4e40> (a bitronix.tm.resource.jms.JmsPooledConnection)
at bitronix.tm.resource.jms.JmsPooledConnection.toString(JmsPooledConnection.java:183) - locked <0x08503b68> (a java.util.Collections$SynchronizedRandomAccessList)
at java.lang.String.valueOf(String.java:2615)
at java.lang.StringBuffer.append(StringBuffer.java:220) - locked <0x2761fa18> (a java.lang.StringBuffer)
at bitronix.tm.resource.common.XAPool.containsXAResourceHolderMatchingGtrid(XAPool.java:330)
at bitronix.tm.resource.common.XAPool.getNotAccessible(XAPool.java:318)
at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:96) - locked <0x0845bd98> (a bitronix.tm.resource.common.XAPool)
at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:68) - locked <0x0845bd98> (a bitronix.tm.resource.common.XAPool)
at bitronix.tm.resource.jms.PoolingConnectionFactory.createConnection(PoolingConnectionFactory.java:108)
at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.access$100(AbstractPollingMessageListenerContainer.java:78)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer$MessageListenerContainerResourceFactory.createConnection(AbstractPollingMessageListenerContainer.java:501)
at org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:298)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:286)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:241)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:982)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:881)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
"jmsTaskExecutor-23":
at bitronix.tm.resource.jms.JmsPooledConnection.toString(JmsPooledConnection.java:182)
- waiting to lock <0x08503b68> (a java.util.Collections$SynchronizedRandomAccessList)
at java.lang.String.valueOf(String.java:2615)
at java.lang.StringBuffer.append(StringBuffer.java:220) - locked <0x275eb3f8> (a java.lang.StringBuffer)
at bitronix.tm.resource.common.AbstractXAStatefulHolder.setState(AbstractXAStatefulHolder.java:35) - locked <0x084e4e40> (a bitronix.tm.resource.jms.JmsPooledConnection)
at bitronix.tm.resource.common.DeferredReleaseSynchronization.afterCompletion(DeferredReleaseSynchronization.java:33)
at bitronix.tm.BitronixTransaction.fireAfterCompletionEvent(BitronixTransaction.java:449)
at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:221)
at bitronix.tm.BitronixTransactionManager.commit(BitronixTransactionManager.java:104)
at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1028)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:732)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:982)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:881)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
See http://old.nabble.com/deadlock-with-BTM-td29104439.html for the entire discussion.
This seems to be caused by the JmsPolledConnection.toString() method which accesses a synchronized method (sessions.size()) while this should never happen.
An easy workaround is to lower the logs to INFO and the fix should simply remove the sessions.size() call from JmsPolledConnection.toString().