BTM
  1. BTM
  2. BTM-121

Java Deadlock when using BTM with MSSQL 2005

    Details

    • Type: Bug Bug
    • Status: Open Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1.0
    • Fix Version/s: 3.0.0
    • Labels:
      None
    • Environment:
      Red hat Linux, MSSQL server 2005, Java
    • Number of attachments :
      0

      Description

      We experienced a single time Java deadlock with Bitronix involved.
      I'm attaching a stack trace of the threads involved in the deadlock for your kind analysis.

      Found one Java-level deadlock:
      =============================
      "bitronix-recovery-thread":
      waiting to lock monitor 0x00007fd49c4e5678 (object 0x00007fd4e57b70b0, a bitronix.tm.resource.common.XAPool),
      which is held by "http-8080-Processor3990"
      "http-8080-Processor3990":
      waiting to lock monitor 0x00007fd49c76c010 (object 0x00007fd60470b080, a com.microsoft.sqlserver.jdbc.SQLServerConnection),
      which is held by "http-8080-Processor3896"
      "http-8080-Processor3896":
      waiting to lock monitor 0x00007fd49cd141f8 (object 0x00007fd66b776fc0, a com.microsoft.sqlserver.jdbc.SQLServerXAConnection),
      which is held by "http-8080-Processor3990"

      Java stack information for the threads listed above:
      ===================================================
      "bitronix-recovery-thread":
      at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:95)

      • waiting to lock <0x00007fd4e57b70b0> (a bitronix.tm.resource.common.XAPool)
        at bitronix.tm.resource.jdbc.PoolingDataSource.startRecovery(PoolingDataSource.java:223)
        at bitronix.tm.recovery.Recoverer.recover(Recoverer.java:253)
        at bitronix.tm.recovery.Recoverer.recoverAllResources(Recoverer.java:223)
        at bitronix.tm.recovery.Recoverer.run(Recoverer.java:138)
        at java.lang.Thread.run(Thread.java:662)
        "http-8080-Processor3990":
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.DetachFromPool(SQLServerConnection.java:3070)
      • waiting to lock <0x00007fd60470b080> (a com.microsoft.sqlserver.jdbc.SQLServerConnection)
        at com.microsoft.sqlserver.jdbc.SQLServerPooledConnection.close(SQLServerPooledConnection.java:179)
      • locked <0x00007fd66b776fc0> (a com.microsoft.sqlserver.jdbc.SQLServerXAConnection)
        at com.microsoft.sqlserver.jdbc.SQLServerXAConnection.close(SQLServerXAConnection.java:59)
        at bitronix.tm.resource.jdbc.JdbcPooledConnection.close(JdbcPooledConnection.java:159)
        at bitronix.tm.resource.common.XAPool.close(XAPool.java:171)
      • locked <0x00007fd4e57b70b0> (a bitronix.tm.resource.common.XAPool)
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:98)
      • locked <0x00007fd4e57b70b0> (a bitronix.tm.resource.common.XAPool)
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:91)
      • locked <0x00007fd4e57b70b0> (a bitronix.tm.resource.common.XAPool)
        at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:197)
        at org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:92)
        at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
        at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
        at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:116)
        at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:109)
        at org.hibernate.engine.query.NativeSQLQueryPlan.performExecuteUpdate(NativeSQLQueryPlan.java:181)
        at org.hibernate.impl.SessionImpl.executeNativeUpdate(SessionImpl.java:1190)
        at org.hibernate.impl.SQLQueryImpl.executeUpdate(SQLQueryImpl.java:357)
        at com.toluna.as.ext.utils.persistence.hibernate.MultiSchemaHibernateUtil.getCurrentSession(MultiSchemaHibernateUtil.java:178)
        at com.toluna.as.ext.utils.persistence.hibernate.MultiSchemaHibernateUtil.getNamedQuery(MultiSchemaHibernateUtil.java:196)
        at com.toluna.as.ext.utils.persistence.hibernate.MultiSchemaHibernateUtil.getNamedQuery(MultiSchemaHibernateUtil.java:188)
        at com.toluna.as.ext.dao.AbstractGenericHibernateDAO.getNamedQuery(AbstractGenericHibernateDAO.java:223)
        at com.toluna.as.ext.dao.AbstractGenericHibernateDAO.getNamedQuery(AbstractGenericHibernateDAO.java:215)
        at com.toluna.as.ext.dao.admin.hibernate.ASSchemaHibernateDAO.findByPPSchemaName(ASSchemaHibernateDAO.java:43)
        at com.toluna.as.ext.services.admin.AdminServiceImpl.findASSchemaByPPSchemaName(AdminServiceImpl.java:86)
        at sun.reflect.GeneratedMethodAccessor722.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
        at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:50)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy333.findASSchemaByPPSchemaName(Unknown Source)
        at com.toluna.asaccess.service.impl.AsAccessLocalServiceImpl.getASSchemaNameByPPSchemaName(AsAccessLocalServiceImpl.java:38)
        at sun.reflect.GeneratedMethodAccessor721.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy350.getASSchemaNameByPPSchemaName(Unknown Source)
        at com.toluna.asaccess.service.AsAccessLocalServiceUtil.getASSchemaByPPSchemaName(AsAccessLocalServiceUtil.java:18)
        at com.toluna.panelportal.util.liferay.filter.HttpSessionSchemaNameContextIntegrationFilter.generateNewContext(HttpSessionSchemaNameContextIntegrationFilter.java:122)
        at com.toluna.panelportal.util.liferay.filter.HttpSessionSchemaNameContextIntegrationFilter.doFilter(HttpSessionSchemaNameContextIntegrationFilter.java:188)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
        at com.toluna.portal.filter.TolunaSecurityFilter.processFilter(TolunaSecurityFilter.java:107)
        at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.toluna.portal.filter.TolunaCharsetFilter.processFilter(TolunaCharsetFilter.java:17)
        at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
        at org.apache.catalina.connector.RemoteIpValve.invoke(RemoteIpValve.java:609)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:879)
        at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
        at java.lang.Thread.run(Thread.java:662)
        "http-8080-Processor3896":
        at com.microsoft.sqlserver.jdbc.SQLServerPooledConnection.notifyEvent(SQLServerPooledConnection.java:116)
      • waiting to lock <0x00007fd66b776fc0> (a com.microsoft.sqlserver.jdbc.SQLServerXAConnection)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.notifyPooledConnection(SQLServerConnection.java:3059)
      • locked <0x00007fd60470b080> (a com.microsoft.sqlserver.jdbc.SQLServerConnection)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1381)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1355)
        at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1532)
        at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:3274)
      • locked <0x00007fd5530db260> (a com.microsoft.sqlserver.jdbc.TDSReader)
        at com.microsoft.sqlserver.jdbc.TDSReader.nextPacket(IOBuffer.java:3227)
        at com.microsoft.sqlserver.jdbc.TDSReader.ensurePayload(IOBuffer.java:3203)
        at com.microsoft.sqlserver.jdbc.TDSReader.peekTokenType(IOBuffer.java:3420)
        at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:50)
        at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:39)
        at com.microsoft.sqlserver.jdbc.TDSCommand.processResponse(IOBuffer.java:4067)
        at com.microsoft.sqlserver.jdbc.TDSCommand.close(IOBuffer.java:4111)
        at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4041)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1416)
      • locked <0x00007fd5e6066988> (a java.lang.Object)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeDTCCommand(SQLServerConnection.java:2146)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.JTAUnenlistConnection(SQLServerConnection.java:2156)
        at com.microsoft.sqlserver.jdbc.SQLServerXAResource.DTC_XA_Interface(SQLServerXAResource.java:610)
        at com.microsoft.sqlserver.jdbc.SQLServerXAResource.end(SQLServerXAResource.java:687)
        at bitronix.tm.internal.XAResourceHolderState.end(XAResourceHolderState.java:162)
        at bitronix.tm.internal.XAResourceManager.delist(XAResourceManager.java:130)
        at bitronix.tm.BitronixTransaction.delistResource(BitronixTransaction.java:148)
        at bitronix.tm.BitronixTransaction.delistUnclosedResources(BitronixTransaction.java:391)
        at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:209)
        at bitronix.tm.BitronixTransactionManager.commit(BitronixTransactionManager.java:120)
        at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1028)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:732)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy350.getASSchemaNameByPPSchemaName(Unknown Source)
        at com.toluna.asaccess.service.AsAccessLocalServiceUtil.getASSchemaByPPSchemaName(AsAccessLocalServiceUtil.java:18)
        at com.toluna.portal.action.TolunaLoginAction.putSchemaIntoSession(TolunaLoginAction.java:877)
        at com.toluna.portal.action.TolunaLoginAction.execute(TolunaLoginAction.java:423)
        at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431)
        at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236)
        at com.liferay.portal.struts.PortalRequestProcessor.process(PortalRequestProcessor.java:159)
        at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
        at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
        at com.liferay.portal.servlet.MainServlet.callParentService(MainServlet.java:505)
        at com.liferay.portal.servlet.MainServlet.service(MainServlet.java:732)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
        at com.liferay.portal.servlet.filters.strip.StripFilter.processFilter(StripFilter.java:140)
        at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
        at com.liferay.portal.servlet.filters.gzip.GZipFilter.processFilter(GZipFilter.java:140)
        at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
        at com.liferay.portal.servlet.filters.secure.SecureFilter.processFilter(SecureFilter.java:282)
        at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
        at com.liferay.portal.servlet.filters.autologin.AutoLoginFilter.processFilter(AutoLoginFilter.java:244)
        at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
        at com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.doFilter(VirtualHostFilter.java:147)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
        at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:94)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:738)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.toluna.panelportal.util.liferay.filter.HttpSessionSchemaNameContextIntegrationFilter.doFilter(HttpSessionSchemaNameContextIntegrationFilter.java:218)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
        at com.toluna.portal.filter.TolunaSecurityFilter.processFilter(TolunaSecurityFilter.java:107)
        at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.toluna.portal.filter.TolunaCharsetFilter.processFilter(TolunaCharsetFilter.java:17)
        at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
        at org.apache.catalina.connector.RemoteIpValve.invoke(RemoteIpValve.java:609)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:879)
        at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
        at java.lang.Thread.run(Thread.java:662)

      Found 1 deadlock.

      Heap
      par new generation total 76672K, used 3041K [0x00007fd4e0000000, 0x00007fd4e5330000, 0x00007fd4e5330000)
      eden space 68160K, 4% used [0x00007fd4e0000000, 0x00007fd4e02aba80, 0x00007fd4e4290000)
      from space 8512K, 3% used [0x00007fd4e4290000, 0x00007fd4e42dc9e0, 0x00007fd4e4ae0000)
      to space 8512K, 0% used [0x00007fd4e4ae0000, 0x00007fd4e4ae0000, 0x00007fd4e5330000)
      concurrent mark-sweep generation total 8303424K, used 3397629K [0x00007fd4e5330000, 0x00007fd6e0000000, 0x00007fd6e0000000)
      concurrent-mark-sweep perm gen total 387324K, used 232612K [0x00007fd6e0000000, 0x00007fd6f7a3f000, 0x00007fd700000000)

        Activity

        Hide
        Leon Fleysher added a comment -

        Sorry, the stack trace in the description have copy - paste problems. See a better version below. I have no way to edit the original one.
        Found one Java-level deadlock:
        =============================
        "bitronix-recovery-thread":
        waiting to lock monitor 0x00007fd49c4e5678 (object 0x00007fd4e57b70b0, a bitronix.tm.resource.common.XAPool),
        which is held by "http-8080-Processor3990"
        "http-8080-Processor3990":
        waiting to lock monitor 0x00007fd49c76c010 (object 0x00007fd60470b080, a com.microsoft.sqlserver.jdbc.SQLServerConnection),
        which is held by "http-8080-Processor3896"
        "http-8080-Processor3896":
        waiting to lock monitor 0x00007fd49cd141f8 (object 0x00007fd66b776fc0, a com.microsoft.sqlserver.jdbc.SQLServerXAConnection),
        which is held by "http-8080-Processor3990"

        Java stack information for the threads listed above:
        ===================================================
        "bitronix-recovery-thread":
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:95)

        • waiting to lock <0x00007fd4e57b70b0> (a bitronix.tm.resource.common.XAPool)
          at bitronix.tm.resource.jdbc.PoolingDataSource.startRecovery(PoolingDataSource.java:223)
          at bitronix.tm.recovery.Recoverer.recover(Recoverer.java:253)
          at bitronix.tm.recovery.Recoverer.recoverAllResources(Recoverer.java:223)
          at bitronix.tm.recovery.Recoverer.run(Recoverer.java:138)
          at java.lang.Thread.run(Thread.java:662)
          "http-8080-Processor3990":
          at com.microsoft.sqlserver.jdbc.SQLServerConnection.DetachFromPool(SQLServerConnection.java:3070)
        • waiting to lock <0x00007fd60470b080> (a com.microsoft.sqlserver.jdbc.SQLServerConnection)
          at com.microsoft.sqlserver.jdbc.SQLServerPooledConnection.close(SQLServerPooledConnection.java:179)
        • locked <0x00007fd66b776fc0> (a com.microsoft.sqlserver.jdbc.SQLServerXAConnection)
          at com.microsoft.sqlserver.jdbc.SQLServerXAConnection.close(SQLServerXAConnection.java:59)
          at bitronix.tm.resource.jdbc.JdbcPooledConnection.close(JdbcPooledConnection.java:159)
          at bitronix.tm.resource.common.XAPool.close(XAPool.java:171)
        • locked <0x00007fd4e57b70b0> (a bitronix.tm.resource.common.XAPool)
          at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:98)
        • locked <0x00007fd4e57b70b0> (a bitronix.tm.resource.common.XAPool)
          at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:91)
        • locked <0x00007fd4e57b70b0> (a bitronix.tm.resource.common.XAPool)
          at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:197)
          at org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:92)
          at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
          at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
          at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:116)
          at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:109)
          at org.hibernate.engine.query.NativeSQLQueryPlan.performExecuteUpdate(NativeSQLQueryPlan.java:181)
          at org.hibernate.impl.SessionImpl.executeNativeUpdate(SessionImpl.java:1190)
          at org.hibernate.impl.SQLQueryImpl.executeUpdate(SQLQueryImpl.java:357)
          at com.toluna.as.ext.utils.persistence.hibernate.MultiSchemaHibernateUtil.getCurrentSession(MultiSchemaHibernateUtil.java:178)
          at com.toluna.as.ext.utils.persistence.hibernate.MultiSchemaHibernateUtil.getNamedQuery(MultiSchemaHibernateUtil.java:196)
          at com.toluna.as.ext.utils.persistence.hibernate.MultiSchemaHibernateUtil.getNamedQuery(MultiSchemaHibernateUtil.java:188)
          at com.toluna.as.ext.dao.AbstractGenericHibernateDAO.getNamedQuery(AbstractGenericHibernateDAO.java:223)
          at com.toluna.as.ext.dao.AbstractGenericHibernateDAO.getNamedQuery(AbstractGenericHibernateDAO.java:215)
          at com.toluna.as.ext.dao.admin.hibernate.ASSchemaHibernateDAO.findByPPSchemaName(ASSchemaHibernateDAO.java:43)
          at com.toluna.as.ext.services.admin.AdminServiceImpl.findASSchemaByPPSchemaName(AdminServiceImpl.java:86)
          at sun.reflect.GeneratedMethodAccessor722.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
          at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
          at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:50)
          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
          at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
          at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
          at $Proxy333.findASSchemaByPPSchemaName(Unknown Source)
          at com.toluna.asaccess.service.impl.AsAccessLocalServiceImpl.getASSchemaNameByPPSchemaName(AsAccessLocalServiceImpl.java:38)
          at sun.reflect.GeneratedMethodAccessor721.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
          at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
          at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
          at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
          at $Proxy350.getASSchemaNameByPPSchemaName(Unknown Source)
          at com.toluna.asaccess.service.AsAccessLocalServiceUtil.getASSchemaByPPSchemaName(AsAccessLocalServiceUtil.java:18)
          at com.toluna.panelportal.util.liferay.filter.HttpSessionSchemaNameContextIntegrationFilter.generateNewContext(HttpSessionSchemaNameContextIntegrationFilter.java:122)
          at com.toluna.panelportal.util.liferay.filter.HttpSessionSchemaNameContextIntegrationFilter.doFilter(HttpSessionSchemaNameContextIntegrationFilter.java:188)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
          at com.toluna.portal.filter.TolunaSecurityFilter.processFilter(TolunaSecurityFilter.java:107)
          at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at com.toluna.portal.filter.TolunaCharsetFilter.processFilter(TolunaCharsetFilter.java:17)
          at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
          at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
          at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
          at org.apache.catalina.connector.RemoteIpValve.invoke(RemoteIpValve.java:609)
          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
          at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:879)
          at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
          at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
          at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
          at java.lang.Thread.run(Thread.java:662)
          "http-8080-Processor3896":
          at com.microsoft.sqlserver.jdbc.SQLServerPooledConnection.notifyEvent(SQLServerPooledConnection.java:116)
        • waiting to lock <0x00007fd66b776fc0> (a com.microsoft.sqlserver.jdbc.SQLServerXAConnection)
          at com.microsoft.sqlserver.jdbc.SQLServerConnection.notifyPooledConnection(SQLServerConnection.java:3059)
        • locked <0x00007fd60470b080> (a com.microsoft.sqlserver.jdbc.SQLServerConnection)
          at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1381)
          at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1355)
          at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1532)
          at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:3274)
        • locked <0x00007fd5530db260> (a com.microsoft.sqlserver.jdbc.TDSReader)
          at com.microsoft.sqlserver.jdbc.TDSReader.nextPacket(IOBuffer.java:3227)
          at com.microsoft.sqlserver.jdbc.TDSReader.ensurePayload(IOBuffer.java:3203)
          at com.microsoft.sqlserver.jdbc.TDSReader.peekTokenType(IOBuffer.java:3420)
          at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:50)
          at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:39)
          at com.microsoft.sqlserver.jdbc.TDSCommand.processResponse(IOBuffer.java:4067)
          at com.microsoft.sqlserver.jdbc.TDSCommand.close(IOBuffer.java:4111)
          at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4041)
          at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1416)
        • locked <0x00007fd5e6066988> (a java.lang.Object)
          at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeDTCCommand(SQLServerConnection.java:2146)
          at com.microsoft.sqlserver.jdbc.SQLServerConnection.JTAUnenlistConnection(SQLServerConnection.java:2156)
          at com.microsoft.sqlserver.jdbc.SQLServerXAResource.DTC_XA_Interface(SQLServerXAResource.java:610)
          at com.microsoft.sqlserver.jdbc.SQLServerXAResource.end(SQLServerXAResource.java:687)
          at bitronix.tm.internal.XAResourceHolderState.end(XAResourceHolderState.java:162)
          at bitronix.tm.internal.XAResourceManager.delist(XAResourceManager.java:130)
          at bitronix.tm.BitronixTransaction.delistResource(BitronixTransaction.java:148)
          at bitronix.tm.BitronixTransaction.delistUnclosedResources(BitronixTransaction.java:391)
          at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:209)
          at bitronix.tm.BitronixTransactionManager.commit(BitronixTransactionManager.java:120)
          at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1028)
          at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:732)
          at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)
          at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)
          at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
          at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
          at $Proxy350.getASSchemaNameByPPSchemaName(Unknown Source)
          at com.toluna.asaccess.service.AsAccessLocalServiceUtil.getASSchemaByPPSchemaName(AsAccessLocalServiceUtil.java:18)
          at com.toluna.portal.action.TolunaLoginAction.putSchemaIntoSession(TolunaLoginAction.java:877)
          at com.toluna.portal.action.TolunaLoginAction.execute(TolunaLoginAction.java:423)
          at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431)
          at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236)
          at com.liferay.portal.struts.PortalRequestProcessor.process(PortalRequestProcessor.java:159)
          at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
          at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
          at com.liferay.portal.servlet.MainServlet.callParentService(MainServlet.java:505)
          at com.liferay.portal.servlet.MainServlet.service(MainServlet.java:732)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
          at com.liferay.portal.servlet.filters.strip.StripFilter.processFilter(StripFilter.java:140)
          at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
          at com.liferay.portal.servlet.filters.gzip.GZipFilter.processFilter(GZipFilter.java:140)
          at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
          at com.liferay.portal.servlet.filters.secure.SecureFilter.processFilter(SecureFilter.java:282)
          at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
          at com.liferay.portal.servlet.filters.autologin.AutoLoginFilter.processFilter(AutoLoginFilter.java:244)
          at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
          at com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.doFilter(VirtualHostFilter.java:147)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
          at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:94)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:738)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at com.toluna.panelportal.util.liferay.filter.HttpSessionSchemaNameContextIntegrationFilter.doFilter(HttpSessionSchemaNameContextIntegrationFilter.java:218)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
          at com.toluna.portal.filter.TolunaSecurityFilter.processFilter(TolunaSecurityFilter.java:107)
          at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at com.toluna.portal.filter.TolunaCharsetFilter.processFilter(TolunaCharsetFilter.java:17)
          at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
          at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
          at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
          at org.apache.catalina.connector.RemoteIpValve.invoke(RemoteIpValve.java:609)
          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
          at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:879)
          at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
          at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
          at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
          at java.lang.Thread.run(Thread.java:662)

        Found 1 deadlock.

        Heap
        par new generation total 76672K, used 3041K [0x00007fd4e0000000, 0x00007fd4e5330000, 0x00007fd4e5330000)
        eden space 68160K, 4% used [0x00007fd4e0000000, 0x00007fd4e02aba80, 0x00007fd4e4290000)
        from space 8512K, 3% used [0x00007fd4e4290000, 0x00007fd4e42dc9e0, 0x00007fd4e4ae0000)
        to space 8512K, 0% used [0x00007fd4e4ae0000, 0x00007fd4e4ae0000, 0x00007fd4e5330000)
        concurrent mark-sweep generation total 8303424K, used 3397629K [0x00007fd4e5330000, 0x00007fd6e0000000, 0x00007fd6e0000000)
        concurrent-mark-sweep perm gen total 387324K, used 232612K [0x00007fd6e0000000, 0x00007fd6f7a3f000, 0x00007fd700000000)

        Show
        Leon Fleysher added a comment - Sorry, the stack trace in the description have copy - paste problems. See a better version below. I have no way to edit the original one. Found one Java-level deadlock: ============================= "bitronix-recovery-thread": waiting to lock monitor 0x00007fd49c4e5678 (object 0x00007fd4e57b70b0, a bitronix.tm.resource.common.XAPool), which is held by "http-8080-Processor3990" "http-8080-Processor3990": waiting to lock monitor 0x00007fd49c76c010 (object 0x00007fd60470b080, a com.microsoft.sqlserver.jdbc.SQLServerConnection), which is held by "http-8080-Processor3896" "http-8080-Processor3896": waiting to lock monitor 0x00007fd49cd141f8 (object 0x00007fd66b776fc0, a com.microsoft.sqlserver.jdbc.SQLServerXAConnection), which is held by "http-8080-Processor3990" Java stack information for the threads listed above: =================================================== "bitronix-recovery-thread": at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:95) waiting to lock <0x00007fd4e57b70b0> (a bitronix.tm.resource.common.XAPool) at bitronix.tm.resource.jdbc.PoolingDataSource.startRecovery(PoolingDataSource.java:223) at bitronix.tm.recovery.Recoverer.recover(Recoverer.java:253) at bitronix.tm.recovery.Recoverer.recoverAllResources(Recoverer.java:223) at bitronix.tm.recovery.Recoverer.run(Recoverer.java:138) at java.lang.Thread.run(Thread.java:662) "http-8080-Processor3990": at com.microsoft.sqlserver.jdbc.SQLServerConnection.DetachFromPool(SQLServerConnection.java:3070) waiting to lock <0x00007fd60470b080> (a com.microsoft.sqlserver.jdbc.SQLServerConnection) at com.microsoft.sqlserver.jdbc.SQLServerPooledConnection.close(SQLServerPooledConnection.java:179) locked <0x00007fd66b776fc0> (a com.microsoft.sqlserver.jdbc.SQLServerXAConnection) at com.microsoft.sqlserver.jdbc.SQLServerXAConnection.close(SQLServerXAConnection.java:59) at bitronix.tm.resource.jdbc.JdbcPooledConnection.close(JdbcPooledConnection.java:159) at bitronix.tm.resource.common.XAPool.close(XAPool.java:171) locked <0x00007fd4e57b70b0> (a bitronix.tm.resource.common.XAPool) at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:98) locked <0x00007fd4e57b70b0> (a bitronix.tm.resource.common.XAPool) at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:91) locked <0x00007fd4e57b70b0> (a bitronix.tm.resource.common.XAPool) at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:197) at org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:92) at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446) at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167) at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:116) at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:109) at org.hibernate.engine.query.NativeSQLQueryPlan.performExecuteUpdate(NativeSQLQueryPlan.java:181) at org.hibernate.impl.SessionImpl.executeNativeUpdate(SessionImpl.java:1190) at org.hibernate.impl.SQLQueryImpl.executeUpdate(SQLQueryImpl.java:357) at com.toluna.as.ext.utils.persistence.hibernate.MultiSchemaHibernateUtil.getCurrentSession(MultiSchemaHibernateUtil.java:178) at com.toluna.as.ext.utils.persistence.hibernate.MultiSchemaHibernateUtil.getNamedQuery(MultiSchemaHibernateUtil.java:196) at com.toluna.as.ext.utils.persistence.hibernate.MultiSchemaHibernateUtil.getNamedQuery(MultiSchemaHibernateUtil.java:188) at com.toluna.as.ext.dao.AbstractGenericHibernateDAO.getNamedQuery(AbstractGenericHibernateDAO.java:223) at com.toluna.as.ext.dao.AbstractGenericHibernateDAO.getNamedQuery(AbstractGenericHibernateDAO.java:215) at com.toluna.as.ext.dao.admin.hibernate.ASSchemaHibernateDAO.findByPPSchemaName(ASSchemaHibernateDAO.java:43) at com.toluna.as.ext.services.admin.AdminServiceImpl.findASSchemaByPPSchemaName(AdminServiceImpl.java:86) at sun.reflect.GeneratedMethodAccessor722.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:50) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at $Proxy333.findASSchemaByPPSchemaName(Unknown Source) at com.toluna.asaccess.service.impl.AsAccessLocalServiceImpl.getASSchemaNameByPPSchemaName(AsAccessLocalServiceImpl.java:38) at sun.reflect.GeneratedMethodAccessor721.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at $Proxy350.getASSchemaNameByPPSchemaName(Unknown Source) at com.toluna.asaccess.service.AsAccessLocalServiceUtil.getASSchemaByPPSchemaName(AsAccessLocalServiceUtil.java:18) at com.toluna.panelportal.util.liferay.filter.HttpSessionSchemaNameContextIntegrationFilter.generateNewContext(HttpSessionSchemaNameContextIntegrationFilter.java:122) at com.toluna.panelportal.util.liferay.filter.HttpSessionSchemaNameContextIntegrationFilter.doFilter(HttpSessionSchemaNameContextIntegrationFilter.java:188) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154) at com.toluna.portal.filter.TolunaSecurityFilter.processFilter(TolunaSecurityFilter.java:107) at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.toluna.portal.filter.TolunaCharsetFilter.processFilter(TolunaCharsetFilter.java:17) at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581) at org.apache.catalina.connector.RemoteIpValve.invoke(RemoteIpValve.java:609) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:879) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) at java.lang.Thread.run(Thread.java:662) "http-8080-Processor3896": at com.microsoft.sqlserver.jdbc.SQLServerPooledConnection.notifyEvent(SQLServerPooledConnection.java:116) waiting to lock <0x00007fd66b776fc0> (a com.microsoft.sqlserver.jdbc.SQLServerXAConnection) at com.microsoft.sqlserver.jdbc.SQLServerConnection.notifyPooledConnection(SQLServerConnection.java:3059) locked <0x00007fd60470b080> (a com.microsoft.sqlserver.jdbc.SQLServerConnection) at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1381) at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1355) at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1532) at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:3274) locked <0x00007fd5530db260> (a com.microsoft.sqlserver.jdbc.TDSReader) at com.microsoft.sqlserver.jdbc.TDSReader.nextPacket(IOBuffer.java:3227) at com.microsoft.sqlserver.jdbc.TDSReader.ensurePayload(IOBuffer.java:3203) at com.microsoft.sqlserver.jdbc.TDSReader.peekTokenType(IOBuffer.java:3420) at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:50) at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:39) at com.microsoft.sqlserver.jdbc.TDSCommand.processResponse(IOBuffer.java:4067) at com.microsoft.sqlserver.jdbc.TDSCommand.close(IOBuffer.java:4111) at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4041) at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1416) locked <0x00007fd5e6066988> (a java.lang.Object) at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeDTCCommand(SQLServerConnection.java:2146) at com.microsoft.sqlserver.jdbc.SQLServerConnection.JTAUnenlistConnection(SQLServerConnection.java:2156) at com.microsoft.sqlserver.jdbc.SQLServerXAResource.DTC_XA_Interface(SQLServerXAResource.java:610) at com.microsoft.sqlserver.jdbc.SQLServerXAResource.end(SQLServerXAResource.java:687) at bitronix.tm.internal.XAResourceHolderState.end(XAResourceHolderState.java:162) at bitronix.tm.internal.XAResourceManager.delist(XAResourceManager.java:130) at bitronix.tm.BitronixTransaction.delistResource(BitronixTransaction.java:148) at bitronix.tm.BitronixTransaction.delistUnclosedResources(BitronixTransaction.java:391) at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:209) at bitronix.tm.BitronixTransactionManager.commit(BitronixTransactionManager.java:120) at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1028) at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:732) at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701) at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at $Proxy350.getASSchemaNameByPPSchemaName(Unknown Source) at com.toluna.asaccess.service.AsAccessLocalServiceUtil.getASSchemaByPPSchemaName(AsAccessLocalServiceUtil.java:18) at com.toluna.portal.action.TolunaLoginAction.putSchemaIntoSession(TolunaLoginAction.java:877) at com.toluna.portal.action.TolunaLoginAction.execute(TolunaLoginAction.java:423) at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236) at com.liferay.portal.struts.PortalRequestProcessor.process(PortalRequestProcessor.java:159) at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196) at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414) at javax.servlet.http.HttpServlet.service(HttpServlet.java:627) at com.liferay.portal.servlet.MainServlet.callParentService(MainServlet.java:505) at com.liferay.portal.servlet.MainServlet.service(MainServlet.java:732) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154) at com.liferay.portal.servlet.filters.strip.StripFilter.processFilter(StripFilter.java:140) at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154) at com.liferay.portal.servlet.filters.gzip.GZipFilter.processFilter(GZipFilter.java:140) at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154) at com.liferay.portal.servlet.filters.secure.SecureFilter.processFilter(SecureFilter.java:282) at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154) at com.liferay.portal.servlet.filters.autologin.AutoLoginFilter.processFilter(AutoLoginFilter.java:244) at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154) at com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.doFilter(VirtualHostFilter.java:147) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154) at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:94) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:738) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.toluna.panelportal.util.liferay.filter.HttpSessionSchemaNameContextIntegrationFilter.doFilter(HttpSessionSchemaNameContextIntegrationFilter.java:218) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154) at com.toluna.portal.filter.TolunaSecurityFilter.processFilter(TolunaSecurityFilter.java:107) at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.toluna.portal.filter.TolunaCharsetFilter.processFilter(TolunaCharsetFilter.java:17) at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581) at org.apache.catalina.connector.RemoteIpValve.invoke(RemoteIpValve.java:609) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:879) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) at java.lang.Thread.run(Thread.java:662) Found 1 deadlock. Heap par new generation total 76672K, used 3041K [0x00007fd4e0000000, 0x00007fd4e5330000, 0x00007fd4e5330000) eden space 68160K, 4% used [0x00007fd4e0000000, 0x00007fd4e02aba80, 0x00007fd4e4290000) from space 8512K, 3% used [0x00007fd4e4290000, 0x00007fd4e42dc9e0, 0x00007fd4e4ae0000) to space 8512K, 0% used [0x00007fd4e4ae0000, 0x00007fd4e4ae0000, 0x00007fd4e5330000) concurrent mark-sweep generation total 8303424K, used 3397629K [0x00007fd4e5330000, 0x00007fd6e0000000, 0x00007fd6e0000000) concurrent-mark-sweep perm gen total 387324K, used 232612K [0x00007fd6e0000000, 0x00007fd6f7a3f000, 0x00007fd700000000)
        Hide
        Ludovic Orban added a comment -

        It looks like this deadlock happened while your datasource was in failed mode and BTM tried to close connections that were still being using in other threads.

        I believe the XAPool implementation available in the BTM 2.2 experimental branch should be immune to this kind of problem but I'll keep this in mind when integrating that code into the mainstream branch.

        Thanks for the report!

        Show
        Ludovic Orban added a comment - It looks like this deadlock happened while your datasource was in failed mode and BTM tried to close connections that were still being using in other threads. I believe the XAPool implementation available in the BTM 2.2 experimental branch should be immune to this kind of problem but I'll keep this in mind when integrating that code into the mainstream branch. Thanks for the report!
        Hide
        Brett Wooldridge added a comment - - edited

        I believe this [Ludovic's comment] is correct. I would not expect the BTM 3.0 (former 2.2) XAPool to suffer from this particular deadlock. Maybe we can close this if the originating user can verify it when 3.0 is available.

        Show
        Brett Wooldridge added a comment - - edited I believe this [Ludovic's comment] is correct. I would not expect the BTM 3.0 (former 2.2) XAPool to suffer from this particular deadlock. Maybe we can close this if the originating user can verify it when 3.0 is available.
        Hide
        Leon Fleysher added a comment -

        Thanks for your comments, guys.
        Unfortunately the scenario for this bug is not easy to recreate so don't count on me to verify it when 3.0 is available.

        Show
        Leon Fleysher added a comment - Thanks for your comments, guys. Unfortunately the scenario for this bug is not easy to recreate so don't count on me to verify it when 3.0 is available.

          People

          • Assignee:
            Ludovic Orban
            Reporter:
            Leon Fleysher
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated: