Continuum
  1. Continuum
  2. CONTINUUM-1871

Continuum does not execute builds when last BUILDRESULT.END_TIME=0

    Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.2
    • Fix Version/s: 1.2
    • Component/s: None
    • Labels:
      None
    • Environment:
      RHEL 5, Continuum 1.2 (SVN 691325), Java 1.6.0_03, MySQL 5.0.45, Maven 2.0.9
    • Complexity:
      Intermediate
    • Number of attachments :
      2

      Description

      After adding a set of projects to a Continuum 1.2 instance, the server stopped performing any builds on any projects. CPU usage shoots up to 100%, projects are queued up and are apparently started, but no build processes are invoked. Even the simplest pom-only job times out.

      Looking at the database I noticed that the last entry for the top-queued project in the BUILDRESULT table had an END_TIME of 0. When I changed that field to a current time and restarted the server Continuum started to perform builds again.

      I was able to duplicate the problem by setting the END_TIME of the last BUILDRESULT of a project to 0, starting the server, and waiting for that project to be queued. The problem was resolved again when I reset END_TIME to its original value.

      I've attached a thread dump as suggested on users@continuum. It seems to show that Continuum is in the SCM module's ChangeSet.toString(), which doesn't seem right, but perhaps it will help point to the root cause.

      1. thread-dump.txt
        53 kB
        Peter Janes
      2. thread-dump.txt
        58 kB
        Peter Janes

        Activity

        Hide
        Peter Janes added a comment -

        Replacement thread dump; the previous one was from a slightly corrupt install (which didn't affect the behaviour, however).

        Show
        Peter Janes added a comment - Replacement thread dump; the previous one was from a slightly corrupt install (which didn't affect the behaviour, however).
        Hide
        Olivier Lamy added a comment -

        The explanation seems to be the following :
        Continuum has certainly been stopped abrutly (killed) when project was building.
        In this case the oldBuildResult of this project is 0.
        During the next build, continuum try to load all scmChange since the last buildResult.
        If the oldBuildResult has an endTime of 0 : all scmResult since the project has been added in Continuum will be loaed.
        This cause a very huge memory (due to a lof of String concat in ChangeSet.toString() method) when the project has a big history in the database.

        Show
        Olivier Lamy added a comment - The explanation seems to be the following : Continuum has certainly been stopped abrutly (killed) when project was building. In this case the oldBuildResult of this project is 0. During the next build, continuum try to load all scmChange since the last buildResult. If the oldBuildResult has an endTime of 0 : all scmResult since the project has been added in Continuum will be loaed. This cause a very huge memory (due to a lof of String concat in ChangeSet.toString() method) when the project has a big history in the database.
        Hide
        Olivier Lamy added a comment -

        The best to fix this should be to use a new method in buildResultDao buildResultDao.getBuildResultsForProject( projectId, buildResultId ).
        This method must return all BuildResult with an id >= buildResultId.
        This will prevent memory use if the lastBuildResult has an endTime of 0.

        Show
        Olivier Lamy added a comment - The best to fix this should be to use a new method in buildResultDao buildResultDao.getBuildResultsForProject( projectId, buildResultId ). This method must return all BuildResult with an id >= buildResultId. This will prevent memory use if the lastBuildResult has an endTime of 0.
        Hide
        Olivier Lamy added a comment -

        fixed in rev 693263

        Show
        Olivier Lamy added a comment - fixed in rev 693263

          People

          • Assignee:
            Olivier Lamy
            Reporter:
            Peter Janes
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: