Chapter 17: Many agents

Adversarial review

A reviewer whose job is to break the work and not to approve it. It is fair only when each finding must be proved by a run. Without that rule, a reviewer told to find problems will find problems, whether or not they exist.

A pushy prompt finds more real bugs and invents more false ones, and every round costs tokens you could spend on a test or a real run. Asking for a failing run first makes a false alarm cheap to spot, because an opinion cannot produce one. It doesn't make the round free, and the author of the best-known rule for this now doubts it pays.

Say it to your agent

“This change retries payments, so I want one targeted attempt to break it. Name the property you are attacking, write a script that fails, and paste the output. Leave out anything you did not run, including style. No findings is a fine answer. Don't fix anything. If a property test or a real run would answer this for less, do that and tell me.”

The workshop

Name a word, or describe what you mean in everyday language.