Concurrent builds

This would be highly useful, especially in situations where you have peers in a dependency chain, but would require a fair bit of dependency management.

Consider:
Project A
BCD depend on A

E depends on C and D
F depends on B and C

G depends on E and F

A would have to be built in isolation, but B, C, and D could be built simultaneously. E and/or F could be built, only with BOTH of their respective dependencies are done. One way would be to fork threads for each build. Each would know their dependencies and poll them until they receive a "yes, I'm done", then they could begin on their merry way.

A key problem with this, is that Maven assumes that all build artifacts go to the central repository, so relying on maven itself won't be a solution. Continuum would have to maintain more build state. This should be fine, since it already scoops up dependencies and displays them, but the point is that it would have to be a Continuum solution, not a maven solution. Also, if it were, and if dependencies for ant/shell build types could be added manually, as long as those dependencies were also in the current Continuum instance, then such a solution could work for any and all project types.

I believe this would extremely useful, even for shell script projects. For example, I have 4 groups of projects setup. Each group represents a platform and I need to build each group of projects concurrently, since each group is built on a separate platform. I am not using maven or ant. I am actually using continuum to build c++ code and it works quite well, except no parallel build ability appears to exist. It would be good to have the ability to define a separate build executor for each project group.

We are managing 100+ projects with a single instance of continuum. I desperately need the ability to scale our CI up to multiple build servers.
Currently we are just building java code and running unit tests. Individual builds take a few minutes each on average. But now we want to add nightly builds with integration tests and these beast will likely take more than one hour each.

Running multiple instances of continuum is not a nice option for us because we like the ability to manage all our CI builds centrally.

The big problem that comes into mind in the repository locking. In fact, what will happen if two separated maven instances download/access/modify the same file concurrently from the local repository.
As Wendy said, either we

• need to handle more than one repository
• wait for this related bug/improvement to be fixed

I'm no maven expert, but I guess there's a risk to see a lot of new problems if the local repository is not unique. What about new maven instances? Which local repository are they going to use? And so on.

I urgently need this also.

There should be configuration to permit N number of concurrent builds, and for each one defined, a different local repository must be specified, or at least a different settings.xml, which then could specify a different repository.

This will avoid any problem with waiting until MNG-2802 is fixed.

Wait - how does Maven handle concurrent access to the local repo. I've often been building several projects in several windows simultaneously, so I wonder how maven handles the conflicts.

Never mind - I looked it up. However, some of this could be resolved by allowing for any projects that use different local repos to be potentially parallelized. Or asserting that each group HAS its own local repo unless overridden, and allowing different groups to be parallel.

I know it's not that simple, because you need to manage thread pools and workers to do the builds and such.

A bigger issue is that it requires a stronger build-order system than a forced linear ranking based on dependency. It would require a build tree where nodes could be pulled off and parallelized and then some logic for finding the synchronization points.

I see in http://continuum.apache.org/docs/1.2/release-notes.html that "now it's possible to configure a local m2 repository per project group". This is excellent!

Now as an interim measure, can each project group run builds in parallel with other project groups?

No, it's not yet possible.

The problem with concurrent builds is dependent projects.

If project A depends on project B depends on project C, it isn't possible/recommended to parallelize builds. But if project A depends on B and C, it should be possible to parallelize B and C builds.

Vulcan is another open-source Java project for CI, it has a build manager which supports multiple threads, and manages complex dependency relationships like you're mentioning here. I don't know if any of the code would be useful for this project, or whether it can be reused. It's been released under the GPLv2.

http://code.google.com/p/vulcan/

How I would tackle it: Every checkout/project gets it's own local repo. Then there should be no problem at all with locking. One could even clean out those repos on every build (if desired) which might be great for making sure all dependencies are really available.

I've posted a draft proposal for this feature here:
http://cwiki.apache.org/confluence/display/CONTINUUM/Draft+-+Continuum+Parallel+or+Concurrent+Builds

Comments and suggestions are much welcome..
Thanks!

Hi. Attached a partial patch for UI requirements.

Hi.. I attach an update patch. updated UI/Configuration #1-4
Not included. #5 queue page should list average wait time for each queue.
Thanks.

Hi... found a problem on my previous patch, the list of build queues in the edit schedule page can't be added in the Schedule. Attached patch fixed the problem.
Thanks.

attached fix for adding build queues in schedule.

schedule-problem.patch applied to continuum-parallel-builds-branch -r726973. Thanks again Gwen!

Patch includes:

• displays the missing checkboxes
• added duplicate checking when adding build queues
• modified checking of the number of build queues allowed
• fixed the action error display

continuum-265-webapp.patch applied to continuum-parallel-builds branch in -r732596. Thanks Jevica!

Attached documentation (continuum-265-docs.patch) and screen shots (parallelBuilds-screenshots.zip).

Hi All,

I've staged another snapshot of the parallel builds branch (-r733009) at:

http://people.apache.org/~oching/continuum-parallel-builds-r733009/org/apache/continuum/continuum-jetty/1.3.1-SNAPSHOT/

To get it working:
1. In the Configuration page, set Number of Allowed Builds in Parallel to a value greater than 1 and save the changes.
2. Create a new Build Queue. You can create a few more if you want but you shouldn't be permitted to create more than the Number of Allowed Builds in Parallel you've set in the configuration.
3. In the Schedules page, edit DEFAULT_SCHEDULE and select the Build Queues you want to attach to it. These build queues would be used to distribute the checkouts/builds to be executed when a build is triggered (either forced or scheduled as long as the build definition used has the DEFAULT_SCHEDULE attached to it).
4. Add a number of projects (different project groups if possible) to Continuum. You should be able to see in the Queues page the current checkout/builds executing and the queued tasks, and to which Build Queue are these tasks executing or queued.

Step 3 is actually optional. If there are no build queues attached to a schedule, the parallel builds manager would use the DEFAULT_BUILD_QUEUE. But in order to demonstrate concurrent builds, we must attach multiple build queues to the schedule.

Comments, suggestions, feedback are welcome

If there are no major bugs found or no further changes in the implementation, I say we are ready to merge the branch to trunk..

Thank you. I am trying this. But I'm confused with respect to this:

"""
3. In the Schedules page, edit DEFAULT_SCHEDULE and select the Build Queues you want to attach to it. These build queues would be used to distribute the checkouts/builds to be executed when a build is triggered (either forced or scheduled as long as the build definition used has the DEFAULT_SCHEDULE attached to it).

Step 3 is actually optional. If there are no build queues attached to a schedule, the parallel builds manager would use the DEFAULT_BUILD_QUEUE. But in order to demonstrate concurrent builds, we must attach multiple build queues to the schedule.
"""

I see my default queue and additional queue when I edit default schedule. But I don't see how to select both to add. I've tried shift click, etc. and then save, but the log does not appear to show that I've selected them.

Also how is this optional if one wants to have parallel builds?

Finally, how does this interact with the local repository? Is it expected that we must set up a repository for each queue to avoid potential corruption? If so, why not make the UI reflect this and simplify setup?

I had trouble getting my selections to "take" in the multi-select box to associate queues with a schedule. I could click and shift-down-arrow to turn them blue (selected) and save, but when I returned, only the default one was selected. I repeated it, (possibly using cmd-click this time?) and then it seemed to save.

I added several projects and saw the checkouts go into the three different queues, though I can't say I ever saw it checking out more than one project at a time. Most of my checkouts went into the default queue though, due to the problem mentioned above.

The Queues page still does not always reflect the actual state of things. I forced builds on everything ... the log shows it happily updating and building away, but the Queues page does not list any projects. (I have a screen shot if you want it...) Some time later, it looks more reasonable, with three projects building, and lots of them listed below waiting their turn.

I think the underlying functionality is working, but the UI isn't quite right.

Hi deckrider: by #3 being optional I meant that if you don't attach any build queues to the schedule the build would technically not be parallel and only the default build queue would be used for building the projects. As for the local repository question, you don't need to setup separate local repositories for each build queue as building of inter-dependent projects is not yet supported (this is for future enhancement) and there's also the limitation of 'A project group cannot be built multiple times simultaneously' as specified in the requirements doc: http://cwiki.apache.org/confluence/display/CONTINUUM/Continuum+Parallel+or+Concurrent+Builds

Wendy, deckrider: Shift + click for selecting build queues seem to work for me.. hmm, could be a browser problem? May I ask what browser you're using? I'm using Firefox 2.0.0.17. I'll also try it in IE to see if I can replicate you problem..

For the queues page, I think the update from scm is executed during the prepare build phase by the prepare build task executor (which IIRC was added for the transient state feature) so the project(s) is/are not in any of the checkout queues. Only after the update from scm does the project(s) get added to the build queue so that's the time when it would show up in the Queues page. On a related note, I think you'll be able to see the checkouts in the Queues page when you initially add projects to Continuum..

I tried selecting build queues in IE and Safari then Save, and both seems to work fine as well. However, when I selected all in the list box including the "-Build Queues-" list heading, I couldn't save the changes. Could this be what happened in your case deckrider? This also occurs in Firefox, btw. I'll just remove the list heading to avoid this..

The multi-select list seems like it's going to be very easy to accidentally change as you're tabbing through the page. Would a different UI element work better here?

It would also be good to add a column on the Schedules page and show the list of queues associated with that schedule.

Yeah, maybe an option transfer list box is better suited instead of just a list box. +1 on adding a column on the Schedules page too..

Parallel builds branch merged to trunk in -r734099.