Continuum

Continuum does not execute builds when last BUILDRESULT.END_TIME=0

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 1.2
  • Fix Version/s: 1.2
  • Component/s: None
  • Labels:
    None
  • Environment:
    RHEL 5, Continuum 1.2 (SVN 691325), Java 1.6.0_03, MySQL 5.0.45, Maven 2.0.9
  • Complexity:
    Intermediate
  • Number of attachments :
    2

Description

After adding a set of projects to a Continuum 1.2 instance, the server stopped performing any builds on any projects. CPU usage shoots up to 100%, projects are queued up and are apparently started, but no build processes are invoked. Even the simplest pom-only job times out.

Looking at the database I noticed that the last entry for the top-queued project in the BUILDRESULT table had an END_TIME of 0. When I changed that field to a current time and restarted the server Continuum started to perform builds again.

I was able to duplicate the problem by setting the END_TIME of the last BUILDRESULT of a project to 0, starting the server, and waiting for that project to be queued. The problem was resolved again when I reset END_TIME to its original value.

I've attached a thread dump as suggested on users@continuum. It seems to show that Continuum is in the SCM module's ChangeSet.toString(), which doesn't seem right, but perhaps it will help point to the root cause.

  1. Text File
    thread-dump.txt
    05/Sep/08 2:26 PM
    53 kB
    Peter Janes
  2. Text File
    thread-dump.txt
    05/Sep/08 10:46 AM
    58 kB
    Peter Janes

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Peter Janes added a comment -

Replacement thread dump; the previous one was from a slightly corrupt install (which didn't affect the behaviour, however).

Show
Peter Janes added a comment - Replacement thread dump; the previous one was from a slightly corrupt install (which didn't affect the behaviour, however).
Hide
Permalink
Olivier Lamy added a comment -

The explanation seems to be the following :
Continuum has certainly been stopped abrutly (killed) when project was building.
In this case the oldBuildResult of this project is 0.
During the next build, continuum try to load all scmChange since the last buildResult.
If the oldBuildResult has an endTime of 0 : all scmResult since the project has been added in Continuum will be loaed.
This cause a very huge memory (due to a lof of String concat in ChangeSet.toString() method) when the project has a big history in the database.

Show
Olivier Lamy added a comment - The explanation seems to be the following : Continuum has certainly been stopped abrutly (killed) when project was building. In this case the oldBuildResult of this project is 0. During the next build, continuum try to load all scmChange since the last buildResult. If the oldBuildResult has an endTime of 0 : all scmResult since the project has been added in Continuum will be loaed. This cause a very huge memory (due to a lof of String concat in ChangeSet.toString() method) when the project has a big history in the database.
Hide
Permalink
Olivier Lamy added a comment -

The best to fix this should be to use a new method in buildResultDao buildResultDao.getBuildResultsForProject( projectId, buildResultId ).
This method must return all BuildResult with an id >= buildResultId.
This will prevent memory use if the lastBuildResult has an endTime of 0.

Show
Olivier Lamy added a comment - The best to fix this should be to use a new method in buildResultDao buildResultDao.getBuildResultsForProject( projectId, buildResultId ). This method must return all BuildResult with an id >= buildResultId. This will prevent memory use if the lastBuildResult has an endTime of 0.
Hide
Permalink
Olivier Lamy added a comment -

fixed in rev 693263

Show
Olivier Lamy added a comment - fixed in rev 693263

People

  • Assignee:
    Olivier Lamy
    Reporter:
    Peter Janes
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: