Another way to achieve this is assuming values are normally distributed and using standard statistical techniques. A quick test look seems to indicate that this approach is viable and the below SQL demonstrates one such query. The degree of variation we want to detect (below is set to 1.75 std deviations) will need to be evaluated when it is in place with production data. We may also want to adjust the windows size (ie number of test_runs we look at) so that the data produced is more useful and this will likely impact the degree of variation we want to detect.
However the below query takes 4ish seconds to run on production data due to the regex so we need to separate out numerical and non-numerical statistics and it should run in < 0.2 secs.
SELECT * FROM (
SELECT
build_configurations.name AS build_configuration_name,
test_configurations.name AS test_configuration_name,
groups.name AS group_name,
test_cases.name AS test_case_name,
test_case_statistics.key AS test_case_statistic_name,
max(case when build_configurations.test_run_id = 9 then CAST(test_case_statistics.value AS float8) else NULL end) AS current_value,
max(case when build_configurations.test_run_id = 9 then test_cases.id else NULL end) AS test_case_id,
min(CAST(test_case_statistics.value AS float8)) AS min_value,
avg(CAST(test_case_statistics.value AS float8)) AS avg_value,
max(CAST(test_case_statistics.value AS float8)) AS max_value,
stddev(CAST(test_case_statistics.value AS float8)) AS std_deviation,
count
AS total_runs
FROM build_configurations
LEFT JOIN test_configurations ON test_configurations.build_configuration_id = build_configurations.id
LEFT JOIN groups ON groups.test_configuration_id = test_configurations.id
LEFT JOIN test_cases ON test_cases.group_id = groups.id
LEFT JOIN test_case_statistics ON test_case_statistics.owner_id = test_cases.id
WHERE
build_configurations.test_run_id IN (9, 10, 8, 7, 11) AND
test_case_statistics.value SIMILAR TO '\\d+$|\\d+\\.
d+$'
GROUP BY build_configuration_name, test_configuration_name, group_name, test_case_name, test_case_statistic_name
ORDER BY build_configurations.name, test_configurations.name, groups.name, test_cases.name) results
WHERE current_value > (avg_value + 1.75 * std_deviation) OR current_value < (avg_value - 1.75 * std_deviation)
Sample output
Significant changes in test results:
Caffeinemark/Caffeinemark/Float.run2 +20% (100 => 120)