Continuum gets slower over time and eventually crashes

How many projects in your instance?
How many build results?
build frequency?

We have few instances that run since one month with more than 140 projects and we don't have OOME and it don't get slower over time.

We have about 260 projects built every 15 minutes.

I fully believe you do not run into the problem, but we have reproduced it 3 times already.

Maybe could it be caused by the xmlrpc component?
We are doing a pretty heavy usage of it (called every minute to check that every build is working).

I don't understand why you build all every 15 minutes. 260 projects checked every 15 minutes use lot of resources on your svn server and your continuum server.
If servers can't accept all this charge, all actions are put in the java queue and can create OOME.

The two servers are alongside. The full check takes about 3 to 4 minutes when nothing is built. I agree that's a fairly heavy load. But the two servers seem to be able to cope with it. We do not do fresh builds of course !

Usually during day time, in a 15m interval there are just a few commits. Continuum rebuilds them (and projects which depends on them) immediatly.

You seem to suggest that the queue would fill up till exhaustion ? It is true that sometimes, the "next" schedule is triggered before the first one is completed, but at night (when there is no commit), Continuum seem to be able to process all the queue as it stops building and checking out projects (well, before the next schedule, I mean).

do you have lot of build results by project?

Most projects have one build result (but we also deploy sites), so we deploy one artifact and one site. A few projects have several artifacts.

I have attached a image of the memory heap over time. I will attach an other one in a few days. There seem to be a upward trend, but it may be too early to confirm it.

Today, I removed some unused datas in the db that was loaded with build results. Can you test the latest snapshot to see if it fix your OOME issue?

Do you have modified the maxmemory parameter in wrapper.conf. The default value is 384m. With 260 projects, I think you should increase it to 512m or 1024m.

I have attached an other picture, which is confirming the likelihood of a "memory leak".
Isn't possible that you keep in memory some data for every build result ?

I will increase the maxmemory parameter in the wrapper as you suggested, but if there is indeed a "memory leaf", it will just save me a few days

Otherwise, I'll do my best to try the latest snapshot, but, even with our heavy schedule, I will most likely need at least 4 days or so to test.
I will keep you posted.

I have found some time to profile continuum 1.1.
As seen in the third attachment, it would appear that the memory is still used up (albeit more slowly than previously).
Unfortunately, it is very difficult to figure out what could cause this...

I think the pb is with scm fileset and build output that go in the memory. Need to investigate on this.

Hi Folks

I thought I suffered from this problem. Then we added clean as one of the maven2 targets in our build definition. (Without clean we were seeing deleted classes remain in our build product.) Since adding clean I have seen stable performance from Continuum.

I hope that info helps some people.

Cheers, Robin.

just to encircle the issue: we have deployed continuum on JBoss and we don't see such problems.

Some more probably irrelevant information:

we have some 50 application servers running on SUN Java 1.5.0_07 for months with identical applications on almost identical hardware. The 51st we installed, crashed more or less every week with an OOME, with jconsole graphs similar to yours. We never found out where the memory leak was, but once we upgraded to SUN Java 1.6.0_02, it magically disappeared... FYI, the application was a JBossMQ 4.0.3sp1 with a small additional webapp on top.

Should be fixed by migration to spring and improvement in buid controller (CONTINUUM-1807)