SonarQube Plugins
  1. SonarQube Plugins
  2. SONARPLUGINS-1870

Optimisation of initialisation for project with a lot of changesets

    Details

    • Number of attachments :
      0

      Description

      Current algoritm for first run of sonar-scm-activity plugin contains three main steps :
      1.) create full changelog history list ( call maven-scm-plugin method changelog, which return list of all changesets with list of all modified files)
      2.) for each file in working directory call maven-scm-plugin method blame
      3.) put scm changelog statistic for project to sonar DB ( last commit, last date of commit )

      When the project contains a lot of changeset with a lot of renames/deletes of files, there are two kind of problem :
      a.) timeout with communication with Sonar Server DB
      b.) there could be a OutOfMemoryException

      I run into this problems because my Mercurial repository contains 55-thousand of changesets / 10 years development.

      I suggest this way of optimisation :
      A.) new configuration parameter : optimise-first-run ( default false )
      B.) if optimise-first-run == true and lastChangeSet is null then
      B.1) skip changelog generating
      B.2) do only blame phase
      B.3) during blame phase compute number of commits/lastAuthor/lastCommitDate
      B.4) wrote computed measures to SonarDB

      This optimization has one drawback, total number of commits/lastAuthor/lastCommitDate are computed only from existing files, not from changelog history. But I think, that is better to have bad starting numbers than nothing, because for big plugin the sonar scm-activity may not finish.
      User has still option to switch this functionality on/off. Default shoud be off.

        Issue Links

          Activity

          Hide
          David Gageot added a comment -

          With version 1.4 of the plugin, we don't load changesets anymore.
          So part of the problem is fixed.

          However, we still have to request blame information for each file currently in the source repository. This can take some significant time. To reduce this, I plan to load the blame information in parallel.

          Show
          David Gageot added a comment - With version 1.4 of the plugin, we don't load changesets anymore. So part of the problem is fixed. However, we still have to request blame information for each file currently in the source repository. This can take some significant time. To reduce this, I plan to load the blame information in parallel.
          Hide
          Evgeny Mandrikov added a comment -

          Just my 2 cents : I'm not sure that parallel execution of commands will help for CVS, SVN.

          Show
          Evgeny Mandrikov added a comment - Just my 2 cents : I'm not sure that parallel execution of commands will help for CVS, SVN.
          Hide
          David Gageot added a comment -

          I have the same feeling. Let's test!

          Show
          David Gageot added a comment - I have the same feeling. Let's test!
          Hide
          Vladimir Kralik added a comment -

          2 Evgeny : I've feeling, that parallel execution for CVS doesn't help.
          As I know, CVS creates synchronisation lock, during access to repository.
          Look at SONARPLUGINS-1872

          Show
          Vladimir Kralik added a comment - 2 Evgeny : I've feeling, that parallel execution for CVS doesn't help. As I know, CVS creates synchronisation lock, during access to repository. Look at SONARPLUGINS-1872
          Hide
          David Gageot added a comment - - edited

          The problem with changesets is fixed

          Show
          David Gageot added a comment - - edited The problem with changesets is fixed
          Hide
          David Gageot added a comment -

          The problem with initial retrieval of blame information is mitigated by SONARPLUGINS-1872

          Show
          David Gageot added a comment - The problem with initial retrieval of blame information is mitigated by SONARPLUGINS-1872
          Hide
          Freddy Mallet added a comment -

          Manually tested !

          Show
          Freddy Mallet added a comment - Manually tested !
          Hide
          David Gageot added a comment -

          ITs done

          Show
          David Gageot added a comment - ITs done

            People

            • Assignee:
              David Gageot
              Reporter:
              Vladimir Kralik
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: