The last defence is simple. If the agent can read the exam, the exam stops measuring anything. So keep some of it where the agent can't reach.

Holdout scenario

Also called Hidden scenario, Holdout

A test of what the software should do, kept where the agent can't read it or change it. The agent is judged on it afterwards, so it can't fit its work to it.

Everything else in this chapter can be seen and so optimised for. A holdout can't. It also catches a different class of failure: tests that are strong, and code that is consistent, but a job nobody asked for properly.

Say it to your agent

“Do not read anything under ../acceptance/. When you think you are done, I will run those scenarios and tell you which ones failed and what they expected, without showing you the code. Fix the behaviour, and do not guess at the tests.”

Not to be confused with Acceptance test

What makes it a holdout is who can see it, not how it is written. A test in the repo the agent can read is not a holdout, however well it is written.

Where you see it

StrongDM: "We repurposed the word scenario to represent an end-to-end 'user story', often stored outside the codebase (similar to a 'holdout' set in model training)", validated by a model and scored as a fraction, not a pass or a fail. Their loop "runs until the holdout scenarios pass (and stay passing)". In our experiment, payment-events passed its holdout 27% of the time on a vague brief, despite green tests and top mutation scores in many runs, and 100% when the brief named the problem. One small model, one-function tasks.

Holdout scenario

Before the agent

The agent

The harness

Everything it can get wrong

The operator

The workshop