RVM

Move to a workqueue for performance regressions

Details

  • Type: Improvement Improvement
  • Status: Open Open
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: None
  • Fix Version/s: 1000
  • Component/s: Infrastructure: CatTrack
  • Labels:
    None
  • Number of attachments :
    0

Description

Rather than blindly firing off a test every 6 hours, the performance regressions could use a work queue.

The advantage of this would include:
1. better coverage over revisions
2. ability to slowly work through huge backlog (for example when starting a new machine, gradually filling in historical data would be trivial)
3. better use of machines (when there have been few commits, test untested ones rather than repeatedly test the current revision)

I can imagine the following design:

. two pieces of state:

  • the most recent revision tested on this machine
  • a queue of untested revisions

. use something like the following algorithm

while()
c = get current latest revision in svn
if (c > latest tested)
for each rev in (latest, c]
add to workqueue
t = c
else
t = pop workqueue
fi
run regression against revision t
if (t > latest tested)
latest tested = t
fi
done

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Steve Blackburn added a comment -

Undo automatic assignment

Show
Steve Blackburn added a comment - Undo automatic assignment
Hide
Permalink
Steve Blackburn added a comment -

Further thoughts:

. The workqueue could be centralized, rather than maintained at each regression machine
. The queue could allow for R repeats of each job (for example R=3 for perf runs allowing us to remove some noise, but R=1 for other jobs).
. Perhaps ?? we could use svn both to push work onto the queue at each commit, and also to maintain the queues as plaintext files

Show
Steve Blackburn added a comment - Further thoughts: . The workqueue could be centralized, rather than maintained at each regression machine . The queue could allow for R repeats of each job (for example R=3 for perf runs allowing us to remove some noise, but R=1 for other jobs). . Perhaps ?? we could use svn both to push work onto the queue at each commit, and also to maintain the queues as plaintext files
Hide
Permalink
David Grove added a comment -

Rather than building from scratch, it would probably make sense to look at cruise control (http://cruisecontrol.sourceforge.net/).

Show
David Grove added a comment - Rather than building from scratch, it would probably make sense to look at cruise control ( http://cruisecontrol.sourceforge.net/ ).
Hide
Permalink
Steve Blackburn added a comment -

Dave,

Do you know much about CruiseControl? I looked at the web site and it looks really good. Do you know if it is widely used? Have you heard anything much about it?

After a superficial glance I'm inclined to move that way (widely used stock tools rather than custom project-specific ones).

However, what I was proposing was, I think, an afternoon's work (a pretty trivial increment over what we have now). Moving everything over to CruiseControl looks more like a week's work. But the long term payoff looks promising.

Any thoughts?

--Steve

Show
Steve Blackburn added a comment - Dave, Do you know much about CruiseControl? I looked at the web site and it looks really good. Do you know if it is widely used? Have you heard anything much about it? After a superficial glance I'm inclined to move that way (widely used stock tools rather than custom project-specific ones). However, what I was proposing was, I think, an afternoon's work (a pretty trivial increment over what we have now). Moving everything over to CruiseControl looks more like a week's work. But the long term payoff looks promising. Any thoughts? --Steve
Hide
Permalink
Peter Donald added a comment -

Cruise Control has been around for years and is reasonably well supported. However you may want to have a look at hudson (https://hudson.dev.java.net/) which is newer but just as well supported - however it is far easier to customise and has lots of plugins to do lots of interesting things.

Show
Peter Donald added a comment - Cruise Control has been around for years and is reasonably well supported. However you may want to have a look at hudson ( https://hudson.dev.java.net/ ) which is newer but just as well supported - however it is far easier to customise and has lots of plugins to do lots of interesting things.
Hide
Permalink
David Grove added a comment -

I know of groups within IBM using both Hudson and Cruise Control. Since we have cattrack already, I think we're mainly looking for something to do job scheduling, so Hudson might be a better fit. I've never directly used either of them myself.

Show
David Grove added a comment - I know of groups within IBM using both Hudson and Cruise Control. Since we have cattrack already, I think we're mainly looking for something to do job scheduling, so Hudson might be a better fit. I've never directly used either of them myself.

People

  • Assignee:
    Unassigned
    Reporter:
    Steve Blackburn
Vote (0)
Watch (0)

Dates

  • Created:
    Updated: