Chapter 15: Performance and cost

So timing one thing takes more care than it looks, and it is easy to get a number that flatters you.

Benchmark

A repeatable test that times a piece of code many times under the same conditions, so two versions can be compared. One timing isn't a benchmark, because the same code gives a different number on each run.

An agent asked to prove its change helped can write a benchmark that shows what it hopes for: one run, a tiny input, the old code cold and the new code warm. It need not mean any harm by that. So read the benchmark as well as the result, and ask how many runs it took and whether the ranges overlap.

Say it to your agent

“Benchmark both versions with hyperfine --warmup 3 --runs 30, on the real 50 MB fixture and not a sample. Show me both means with their spreads. If the ranges overlap, say you can't tell them apart.”

The workshop

Name a word, or describe what you mean in everyday language.