SonarQube
  1. SonarQube
  2. SONAR-3306

Use a semaphore to prevent launching several analysis of the same project at the same time

    Details

    • Type: Improvement Improvement
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.4
    • Component/s: Batch Analysis, Database
    • Labels:
      None
    • Number of attachments :
      0

      Issue Links

        Activity

        Hide
        Evgeny Mandrikov added a comment -

        I suppose that it will make sense to provide an API for such semaphores, so it can be reused in plugins.

        Show
        Evgeny Mandrikov added a comment - I suppose that it will make sense to provide an API for such semaphores, so it can be reused in plugins.
        Hide
        Doug Beattie added a comment -

        These comments regarding what this JIRA is about were taken from a series of e-mails on the user list.

        Hi Doug,

        I think this is it: Sonar does not support parallel for a single project
        (or module): http://jira.codehaus.org/browse/SONAR-3306

        Regards

        Olivier

        On Tue, Aug 7, 2012 at 3:22 PM, Doug Beattie <dbb@beatties.us> wrote:

        > Just wondering if anone had more thoughts on the problems asked about in
        > this thread as I have not seen a reply?
        >
        > Thanks,
        > Doug
        >
        > On Wed, Jul 25, 2012 at 10:01:42AM -0600, Doug Beattie wrote:
        > > Freddy,
        > >
        > > I have checked with the engineer responsible for the builds. He said:
        > > "We are not trying to analyze AB and ABB in parallel during a single
        > run. Currently we are running sonar on a multi-module project at the same
        > time as running it on the same multi-module project. Each run is serial
        > though, so no two modules are run at the same time as each other within one
        > of those runs."
        > >
        > > I hope this helps in figuring out what is going on and the best way to
        > resolve the problem without us getting the errors noted and having to do a
        > total re-run of the build job.
        > >
        > > Thanks,
        > > Doug
        > >
        > > On Tue, Jul 24, 2012 at 10:26:41PM +0200, Freddy Mallet wrote:
        > > > Hi Doug,
        > > >
        > > > We have started using a very parallelized build environment for
        > continuous
        > > > > development into a production system. Many of our modules are now
        > building
        > > > > sub-modul
        > > > > es concurrently and we are experiencing random failures during the
        > Sonar
        > > > > analysis cycle.
        > > >
        > > >
        > > > I do confirm that Sonar doesn't support parallel analysis on the same
        > > > project but also on modules having a hierarchical relationship. Let's
        > take
        > > > the following example :
        > > >
        > > > Project A
        > > > - Module AA
        > > > - Module AB
        > > > - Module ABA
        > > > - Module ABB
        > > >
        > > > You can't expect Sonar to run correctly if you analyse Module AB and
        > Module
        > > > ABB simultaneously.
        > > >
        > > > Before answering you other questions could you make sure that you can't
        > > > face such use cases in your context ?
        > > >
        > > > Thanks
        > > > Freddy
        > > >
        > > >
        > > >
        > > > > We want to make Sonar analysis help gate release of the modules but
        > th
        > > > > e random errors we are getting has caused us to consider ignoring
        > certain
        > > > > types of Sonar failures.
        > > > >
        > > > > We are seeing the following 3 failures for example:
        > > > >
        > > > > Failed to execute goal org.codehaus.mojo:sonar-maven-plugin:2.0:sonar
        > > > > (default-cli) on project tree-parent: Can not execute Sonar: Fail to
        > > > > decorate 'org.sonar.api.resources.Project@11ec822c
        > [id=69258,key=org.familysearch.gadget:FavoritesGadget,qualifier=BRC]':
        > > > > Unable to find org.sonar.api.database.model.Snapshot with id
        > 16460103 ->
        > > > > [Help 1]
        > > > >
        > > > > Failed to execute goal org.codehaus.mojo:sonar-maven-plugin:2.0:sonar
        > > > > (default-cli) on project tree-parent: Can not execute Sonar: Fail to
        > > > > decorate 'org.sonar.api.resources.Project@1e7c05f1
        > [id=93862,key=org.familysearch.gadget:FtUtils,qualifier=BRC]':
        > > > > Error while commiting the transaction:
        > org.hibernate.StaleStateException:
        > > > > Batch update returned unexpected row count from update [0]; actual
        > row
        > > > > count: 0; expected: 1 -> [Help 1]
        > > > >
        > > > > org.hibernate.exception.LockAcquisitionException: could not insert:
        > > > > [org.sonar.api.database.model.MeasureModel]: Deadlock found when
        > trying to
        > > > > get lock; try restarting transaction
        > > > >
        > > > > I know there have been problems with Sonar when multiple analysis are
        > > > > attempted to be logged concurrently and that the following query was
        > meant
        > > > > to clean up af
        > > > > ter such problems:
        > > > >
        > > > > mysql -h localhost -u sonar -psonar sonar -e 'update snapshots
        > old_snap,
        > > > > snapshots new_snap set old_snap.islast=0 where old_snap.islast=1 and
        > > > > new_snap.created_at > old_snap.created_at and new_snap.project_id =
        > > > > old_snap.project_id and new_snap.islast=1;'
        > > > >
        > > > > First question, are all three of these problems related?
        > > > >
        > > > > Second question – is something being considered for Sonar to somehow
        > > > > cleanup and retry when a deadlock is encountered?
        > > > > The time required for a new Sonar analysis to be done and reported
        > would
        > > > > seem to be much more costly time wise than having some recovery
        > measures in
        > > > > place in t
        > > > > he client (and possibly the server) to handle these problems.
        > > > >
        > > > > Thanks for any light you can shed on how best to resolve such
        > problems for
        > > > > a continuous delivery environment where there are numerous of build
        > servers
        > > > > handling
        > > > > sub-module builds in parallel.

        Show
        Doug Beattie added a comment - These comments regarding what this JIRA is about were taken from a series of e-mails on the user list. Hi Doug, I think this is it: Sonar does not support parallel for a single project (or module): http://jira.codehaus.org/browse/SONAR-3306 Regards Olivier On Tue, Aug 7, 2012 at 3:22 PM, Doug Beattie <dbb@beatties.us> wrote: > Just wondering if anone had more thoughts on the problems asked about in > this thread as I have not seen a reply? > > Thanks, > Doug > > On Wed, Jul 25, 2012 at 10:01:42AM -0600, Doug Beattie wrote: > > Freddy, > > > > I have checked with the engineer responsible for the builds. He said: > > "We are not trying to analyze AB and ABB in parallel during a single > run. Currently we are running sonar on a multi-module project at the same > time as running it on the same multi-module project. Each run is serial > though, so no two modules are run at the same time as each other within one > of those runs." > > > > I hope this helps in figuring out what is going on and the best way to > resolve the problem without us getting the errors noted and having to do a > total re-run of the build job. > > > > Thanks, > > Doug > > > > On Tue, Jul 24, 2012 at 10:26:41PM +0200, Freddy Mallet wrote: > > > Hi Doug, > > > > > > We have started using a very parallelized build environment for > continuous > > > > development into a production system. Many of our modules are now > building > > > > sub-modul > > > > es concurrently and we are experiencing random failures during the > Sonar > > > > analysis cycle. > > > > > > > > > I do confirm that Sonar doesn't support parallel analysis on the same > > > project but also on modules having a hierarchical relationship. Let's > take > > > the following example : > > > > > > Project A > > > - Module AA > > > - Module AB > > > - Module ABA > > > - Module ABB > > > > > > You can't expect Sonar to run correctly if you analyse Module AB and > Module > > > ABB simultaneously. > > > > > > Before answering you other questions could you make sure that you can't > > > face such use cases in your context ? > > > > > > Thanks > > > Freddy > > > > > > > > > > > > > We want to make Sonar analysis help gate release of the modules but > th > > > > e random errors we are getting has caused us to consider ignoring > certain > > > > types of Sonar failures. > > > > > > > > We are seeing the following 3 failures for example: > > > > > > > > Failed to execute goal org.codehaus.mojo:sonar-maven-plugin:2.0:sonar > > > > (default-cli) on project tree-parent: Can not execute Sonar: Fail to > > > > decorate 'org.sonar.api.resources.Project@11ec822c > [id=69258,key=org.familysearch.gadget:FavoritesGadget,qualifier=BRC] ': > > > > Unable to find org.sonar.api.database.model.Snapshot with id > 16460103 -> > > > > [Help 1] > > > > > > > > Failed to execute goal org.codehaus.mojo:sonar-maven-plugin:2.0:sonar > > > > (default-cli) on project tree-parent: Can not execute Sonar: Fail to > > > > decorate 'org.sonar.api.resources.Project@1e7c05f1 > [id=93862,key=org.familysearch.gadget:FtUtils,qualifier=BRC] ': > > > > Error while commiting the transaction: > org.hibernate.StaleStateException: > > > > Batch update returned unexpected row count from update [0] ; actual > row > > > > count: 0; expected: 1 -> [Help 1] > > > > > > > > org.hibernate.exception.LockAcquisitionException: could not insert: > > > > [org.sonar.api.database.model.MeasureModel] : Deadlock found when > trying to > > > > get lock; try restarting transaction > > > > > > > > I know there have been problems with Sonar when multiple analysis are > > > > attempted to be logged concurrently and that the following query was > meant > > > > to clean up af > > > > ter such problems: > > > > > > > > mysql -h localhost -u sonar -psonar sonar -e 'update snapshots > old_snap, > > > > snapshots new_snap set old_snap.islast=0 where old_snap.islast=1 and > > > > new_snap.created_at > old_snap.created_at and new_snap.project_id = > > > > old_snap.project_id and new_snap.islast=1;' > > > > > > > > First question, are all three of these problems related? > > > > > > > > Second question – is something being considered for Sonar to somehow > > > > cleanup and retry when a deadlock is encountered? > > > > The time required for a new Sonar analysis to be done and reported > would > > > > seem to be much more costly time wise than having some recovery > measures in > > > > place in t > > > > he client (and possibly the server) to handle these problems. > > > > > > > > Thanks for any light you can shed on how best to resolve such > problems for > > > > a continuous delivery environment where there are numerous of build > servers > > > > handling > > > > sub-module builds in parallel.
        Hide
        Freddy Mallet added a comment -

        Manually tested !

        Show
        Freddy Mallet added a comment - Manually tested !
        Hide
        Julien Lancelot added a comment -

        Integration test added

        Show
        Julien Lancelot added a comment - Integration test added

          People

          • Assignee:
            Julien Lancelot
            Reporter:
            Freddy Mallet
          • Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: