Chapter 10: Tests

Before reaching for a better number, it is worth knowing what happens to numbers.

Goodhart's law

When a measure becomes a target, it stops being a good measure. Point anyone at a number and they will move the number, whether or not the thing it stood for moves too.

An agent optimises what the check reads, faster and more literally than a person. Any number you put in a gate can go up for the wrong reason, so keep at least one check it can't see and can't optimise for.

Say it to your agent

“I do not care about the coverage number. Tell me which behaviours are still unchecked. If you delete code to improve a score, list each deletion and say why that code could never run.”

The workshop

Name a word, or describe what you mean in everyday language.