Measure first

Before changing anything to make it faster, time it, find the slow part and write the number down. Then time it the same way after the change. Without both numbers, nobody can say the change helped.

An agent reads code and spots what looks slow, and what looks slow often isn't what is slow. It reports the improvement in the same confident voice either way. So make the number part of the work you ask for, because it is the one thing it can't produce by reasoning.

Say it to your agent

“Don't change anything yet. Run hyperfine --warmup 3 --runs 20 "bun run scripts/report.ts" and show me the mean and the spread. Make your change, run the same command, and put both results in the pull request. If the two ranges overlap, tell me it made no measurable difference.”

Not to be confused with Reasoning about the code

Saying a loop is now O(n) is an argument about the code. It may be right. It still isn't a measurement of your program on your data, and the part it improved may not have been slow.

Where you see it

Knuth, in 1974: "It is often a mistake to make a priori judgments about what parts of a program are really critical, since the universal experience of programmers who have been using measurement tools has been that their intuitive guesses fail." Peng and colleagues found that agent performance pull requests on GitHub "rely predominantly on static reasoning checks", where those by people more often report a benchmark. That is one preprint, with the limits given at the top of this chapter.

Measure first

Before the agent

The agent

The harness

Everything it can get wrong

The operator

The workshop