RVM

We have in the tools directory the perl scripts for computeCompilerDNA and in the build/test-runs we have a compiler-dna properties which states:

1. The compiler-dna test-run is used to measure data on how effective
2. the various levels of optimization in the opt compiler in terms of speedup
3. and how much they cost in terms of compile time.
4. Although this whole process could perhaps be done as an ant task for now
5. I'm just defining an ant task to build/run the images to get the raw
6. data and then will manurally run the perl script in
7. RVM_ROOT/tools/computeCompilerDNA over the output files to get the
8. actual DNA.

When running this properties file the some what surprising message is shown during build-bootimage:

[java] BootImageWriter: compiler arg: O1

so not only is the test measuring the performance of the compiler running at say O1, its also boot image compiling itself with O1. This doesn't make sense to me as we want to know the performance of the O1 compiler when it itself has been compiled as O2. Is the test run wrong?

This is intentional. Neither this or compiling the bootimage at O2 is optimal, but this is the closer approximation to what we want to measure. The issue is that any time the application spends in bootimage code (which includes enough of java.lang.*) needs to be dealt with.

We could do multiple runs where we get compilation time of O1 from an O2 image but still get speedup from an O1 image.

So I think what we want is:

for the speed up rates:

run BaseBase, production_Opt_0, production_Opt_1, production_Opt_2 on 20 iterations of the entire SPEC JVM and measure the total execution time
(production_Opt_X implies the boot image is compiled at level X and we disable adaptive recompilation and use an IRC of Opt X)

we calculate the speed up rates as the total execution time for each run divided by the total execution time for the BaseBase run.

for the compilation rates:

run a production image with no recompilation the IRC as base, O0, O1 and O2 on the entire SPEC JVM but for just 1 iteration as we don't actually care about the execution time

the compilation rate is the bcb/ms reported for base as the baseline rate, and for O0..O2 as the opt compiler bcb/ms in the -X:vm:measureCompilation report.

Given that I think this is what we want, I'm unclear as to:
why in the current compiler-dna we run SPEC JVM benchmarks individually?
why the current compiler-dna doesn't do BaseBase runs?
what we can do to avoid the floating point remainder bug in _200_check (RVM-352) ?
what we can do to avoid the GC crash when running 20 iterations of SPEC JVM (RVM-355) ?

I've scripted extracting the compilation rates to appear in reports. I believe the next steps are to make the runs as I describe them above and possibly write something that will extract the statistics from the report(s?). I think in all cases we need to document somewhere the compiler DNA gathering process.

Failing _200_check means that compiler DNA gathering has failures in the results its trying to use to compute DNA.

Running many iterations of SPEC JVM is a requirement for DNA computation. As this is currently failing we can't compute compiler DNA.

We need to fix 352 and 355. They are regressions in the last 6 months.

For times, we want to pull out the times reported by jvm98 harness (not total time divided by # of iterations).

We run them individually mainly to isolate each benchmark. Doing so isn't critical, but it eliminates some cross-benchmark interactions.

The old scripts also loaded edgeCount & callgraph profile data from a profiling run (approximate what opt compiler would do if it had the runtime profile information).

I believe r13910 fixes most of the out standing issues. Things still left to do:

• fix RVM-352 and RVM-355
• use profile data for compiler DNA
• extract the DNA from the report
• modify potentially troublesome "/" from statistics like "bcb/ms" as it can cause confusion in URLs

Example compiler-dna report generated in my development world with SPEC JVM configured for 10 iterations.

Document added to describe the computation of the compiler DNA from the results of the test run:
http://docs.codehaus.org/display/RVM/Compiler+DNA

Close enough to fully automatic for me. I'm going to open an enhancement for the one remaining piece (the manual steps outlined in the wiki page), but that main work is already done.