Display Statistically significant changes in statistics in nightly report

Sample output

Significant changes in test results:
Caffeinemark/Caffeinemark/Float.run2 +20% (100 => 120)

We have an RFE tracker item (1147584 [1]) to chart performance. Robin has an excellent graph of daily DaCapo results[2]. I would say any speed up was worth reporting/understanding, but as Robin's graphs demonstrate there is a great deal of noise to be filtered out. 20% seems like a high value to filter out noise, but it may be what's necessary.

[1] https://sourceforge.net/tracker/index.php?func=detail&aid=1147584&group_id=128805&atid=712771
[2] http://cs.anu.edu.au/~Robin.Garner/dacapo/regression/

A large portion of the noise that is visible in those graphs is due to to the fact we don't successfully run all the benchmarks each night.

Once we are running those tests without failures we should get a cleaner graph.

Another way to achieve this is assuming values are normally distributed and using standard statistical techniques. A quick test look seems to indicate that this approach is viable and the below SQL demonstrates one such query. The degree of variation we want to detect (below is set to 1.75 std deviations) will need to be evaluated when it is in place with production data. We may also want to adjust the windows size (ie number of test_runs we look at) so that the data produced is more useful and this will likely impact the degree of variation we want to detect.

However the below query takes 4ish seconds to run on production data due to the regex so we need to separate out numerical and non-numerical statistics and it should run in < 0.2 secs.

SELECT * FROM (
SELECT
build_configurations.name AS build_configuration_name,
test_configurations.name AS test_configuration_name,
groups.name AS group_name,
test_cases.name AS test_case_name,
test_case_statistics.key AS test_case_statistic_name,
max(case when build_configurations.test_run_id = 9 then CAST(test_case_statistics.value AS float8) else NULL end) AS current_value,
max(case when build_configurations.test_run_id = 9 then test_cases.id else NULL end) AS test_case_id,
min(CAST(test_case_statistics.value AS float8)) AS min_value,
avg(CAST(test_case_statistics.value AS float8)) AS avg_value,
max(CAST(test_case_statistics.value AS float8)) AS max_value,
stddev(CAST(test_case_statistics.value AS float8)) AS std_deviation,
count AS total_runs
FROM build_configurations
LEFT JOIN test_configurations ON test_configurations.build_configuration_id = build_configurations.id
LEFT JOIN groups ON groups.test_configuration_id = test_configurations.id
LEFT JOIN test_cases ON test_cases.group_id = groups.id
LEFT JOIN test_case_statistics ON test_case_statistics.owner_id = test_cases.id
WHERE
build_configurations.test_run_id IN (9, 10, 8, 7, 11) AND
test_case_statistics.value SIMILAR TO '^\\d+$|\\d+\\.
d+$'
GROUP BY build_configuration_name, test_configuration_name, group_name, test_case_name, test_case_statistic_name
ORDER BY build_configurations.name, test_configurations.name, groups.name, test_cases.name) results
WHERE current_value > (avg_value + 1.75 * std_deviation) OR current_value < (avg_value - 1.75 * std_deviation)