DamageControl as a hub in a distributed build farm

On one of the big ThoughtWorks projects there are around 10 different continuous integration servers doing different things. Running a DamageControl process on each of them took a lot of resources. Resources that could have otherwised been used for running the build faster. Also, visibility of all of the different builds that were done were limited because nobody had an overview of all of them.

DamageControl could act as a hub in a distributed build farm by doing a special distributed BuildExecutor.

All build requests and all admin is done on the central DamageControl server/hub which runs the ordinary (albeit specially configured) DamageControlServer.

A slave process runs on each slave node. The DistributedBuildExecutor communicates with the slave process. The slave process runs the ordinary BuildExecutor with a special hub that sends all events on to the central server (which would do all the logging, reporting and so on). The DistributedBuildExecutor should probably use drb to communicate and not xmlrpc as it's ruby-to-ruby and needs to be as performant as possible.

The DistributedBuildExecutor should be able to be pinned to a certain server for a project. So a specific server can be dedicated to run a specific project (the build might need special software or hardware installed for example).

If the slave process still takes too much resources (because of running Ruby) it should be possible to implement it in C. This will of course not be entirely trivial.