Improve the speed of cat track's graphs

The amount of data we use to generate the graphs probably precludes using the google charts api. One of the main reasons for the slowness is that the web server is effectively single threaded due to the way it is set up but it would be easy enough to fix (see RVM-307).

One option to make the slowness less jarring is to actually specify height-width on all images so the page does not jump about.

Another option is to generate the images when the data is uploaded which would mean that it would be effectively apache serving everything but 3-4 pages and this would be about as fast as we could get it with the setup.

THanks for the pointer, Ian. It looks like a nice API indeed. We (at ANU) have been searching for decent graphing APIs.

I agree with Peter's analysis.

However, I think the real issue is the lack of a summary page, which we've always wanted.

We cannot expect Daniel to do that at the moment---he's at crunch (do-or-die) point for his PhD. Any one of us could have a crack at it though.

The main issue is how to meaningfully aggregate the vast pool of information. I'd love to have a single sparkline summarizing overall performance. This could be embedded in a 24hrly email, for example. Or even if there were 3 sparklines, that would be great. For example, a line which moved up and down according to the overall number of benchmarks which improved or degraded, or perhaps the number of benchmarks which were within 1 std dev of their best. A dip in this line would indicate an overall dip in performance. These are just off the top of my head. I'm sure there are better metrics, but the truth is that providing a meaningful aggregation of 30 or so time series results is not trivial.

Looking again right now, I think there must be something badly wrong with the set-up.

Right now it is taking me in the order of 10 minutes to view a single perf page. Network latency is hardly the issue; it's plugged into the same gigabit switch as my laptop.

When I run top on the machine I see that mongrel_rails is hogging most of virtual memory and plenty of CPU. postgress is also busy. Is there a problem with our database not scaling very well as our volume of data increases?

Peter, if you have the chance, could you please check it out? I know virtually nothing about the setup, but from my naive POV, the patient seems unwell!

It looks like the time is being spent in the image rendering code. There is also significant memory leaks (not in our code it seems?) which is causing gc grinding. I think that we could probably cleanup the rendering code so less inefficient but we could also tune postgres much better. For the moment I have set it to restart mongrel once a day which should reduce some of the performance problems due to mem exhaustion.

However I will try to look at cleaning up the image code within the next week or so.