I currently use criterion to run benchmarks. Most of the time, what I care about is the change in run time between some prior revision of the code and whatever I just wrote. criterion does support comparison against the previous run and against “baselines”, but:

• I have to explicitly save a baseline for the current state of my upstream branch, and it won’t notice if I forget to update the baseline due to pushing changes. Ideally, every remembered benchmark result would be associated with a revision (and build flags, too, while we’re at it).
• Since I have no dedicated machine for benchmarking, my results are always somewhat unstable, and it’s difficult to tell whether a difference is due to the code or due to the conditions under which I run it. Ideally, the benchmark tool would automatically run both versions interleaved (by building both versions and running both binaries), thus keeping the comparison fairer.
• cargo-criterion is the new way to run criterion, but doesn’t support baselines. Also, criterion development has been slow to nonexistent for a long time.

Is anyone aware of tools or strategies for benchmarking that can better support this kind of workflow? I’m feeling tempted to write my own.

1 post - 1 participant

Read full topic