Tests

A small program that runs your code with a known input and checks the answer. If the answer is wrong it fails, loudly, without a person watching.

An agent can't see whether its work is right, but it can read a failure. A test is the cheapest way to turn "does this work?" into something it can act on in a loop.

The line in a test that compares what happened with what should have happened, such as expect(total).toBe(550). Without one, a test only proves the code did not crash.

Agents reach for weak assertions, like "the result exists", because they pass quickly. The strength of the suite lives almost entirely in these lines, so these are the lines to read.

The program that finds your tests, runs them, prints what failed, and exits with a code saying whether everything passed. Examples are bun test, Vitest, pytest and Playwright Test.

The exit code is what lets a hook or a pipeline stop an agent. A runner that prints a failure but exits 0 lets the agent carry on as if nothing happened.

A test of one small piece of code on its own, usually a function. It runs in milliseconds and points straight at what broke.

Fast tests are the ones an agent can run on every attempt. They also suit agents badly in one way: they can pass while the pieces fail to fit together, and they are the easiest kind to write without checking anything.

The known starting state a test sets up before it runs: a sample user, a few rows in a table, a file on disk. Good ones are created fresh for each test.

Agents love to share a fixture between tests because it saves lines. That is how you get a suite that passes in one order and fails in another.

Anything that replaces a real part of your system during a test, like a stunt double. It lets you test without charging a real card or waiting on a real server.

Every double is a claim about how the real thing behaves. Nothing checks that claim unless you make something check it, and an agent writes the claim from what it assumes.

A test double that records how it was called, so the test can check the calls. In casual use, people say mock for any stand-in.

If the agent mocks too much, the test exercises the mocks and not your code. It stays green whatever you break, and it fails when you tidy up the internals.

A test where several real pieces work together: your code against a real database, or two of your services talking to each other. It is slower, and it finds the bugs that live in the joins.

Agents write confident code against an interface they have guessed at. A real query against a real database settles the guess at once, which no unit test can do.

A check that a real outside service still behaves the way your test doubles say it does. If the provider renames a field, this fails, even while everything else stays green.

A double only remembers how the service worked on the day it was written. An agent trusts it completely, and will happily build on a shape the real service stopped sending months ago.

A test that drives the whole system the way a user would: a real browser, clicking real buttons, against the real app and its database.

An agent can only claim a screen works until something looks at it. An end-to-end test can be its eyes. It is also the slowest and flakiest kind, so keep a few that matter and not hundreds.

A test written from what the customer asked for and not from how the code works. Often written in Given, When, Then sentences that a person can read and a tool can run.

It is written before the code exists and from the request, so it is harder for the agent to fit it to whatever it happened to build. A person can also read it and say "that isn't what I meant" before any code gets written.

A very quick check that the thing runs at all: it starts, the main page loads, the health endpoint answers. It says nothing about whether it is right, only whether it is on fire.

Agents can write a thousand lines and never start the program. A ten-second smoke test catches the embarrassing failure, and it is a sound first gate because it costs almost nothing.

A rule of thumb for the mix: many fast unit tests at the bottom, fewer integration tests in the middle, and very few slow end-to-end tests on top.

An agent runs your tests over and over in a loop, so slow tests cost you on every attempt. It is cheap for an agent to write a slow test now. Waiting for it, and working out why it flaked, is still expensive.

A test that sometimes passes and sometimes fails on the same code. Usually the cause is timing, shared state or something outside the test, such as the network or the clock.

A flaky test teaches everyone to ignore red. An agent is worse: it may "fix" code that was fine, or retry until it gets a green run and report success. Retries hide the fault and do not remove it.

A test written after a bug, which fails on the old code and passes on the fix. A regression is when something that used to work stops working.

An agent has no memory of last month's bug, and may reintroduce it while tidying up. The test remembers. It only counts if you have seen it fail on the broken version.

A test that saves the output once to a file, then fails whenever later output differs. It tells you something changed. It can't tell you whether the change is good.

When a snapshot fails, the easy way out is one flag that rewrites the file, and an agent will take it. Your bug is now the expected answer, and the test will defend it.

A test that records what old code does today, bugs included, so you can change it safely. You are writing down the behaviour before you touch it, without claiming that it is right.

This is how you let an agent refactor code nobody understands. Pin the behaviour first, then any change it makes by accident shows up. Tell it the bugs stay until you decide otherwise.

Writing the test before the code that makes it pass, in very small steps. The test fails first, because the code does not exist yet, and that failure is the point.

With agents the argument changes. A test written first can't be fitted to the code, and watching it fail shows it can fail at all. Those are strong reasons whatever the studies say about people. We found no good trials on agents.

The TDD cycle in three steps. Red: write a test and watch it fail. Green: write just enough code to pass. Refactor: tidy up while the tests stay green.

For an agent, red is the step that matters. A test that passes before the code exists is a test that can't fail, and an agent left alone will skip straight past that without noticing.

Whatever tells you what the right answer is. It may be a number you worked out by hand, a spec, an old system, or a rule that must always hold.

If an agent writes the code, runs it, and pastes the output in as the expected value, the oracle is the code itself. The test will pass for ever, and it proves nothing. Always ask where the expected value came from.

The share of your code that ran during the tests. It says what ran. It says nothing about whether anything checked the result.

Low coverage tells you something real: that code has never been run by a test. High coverage tells you much less, and it is trivially easy for an agent to hit any number you give it.

A test that can't fail, because it compares the code with itself, checks nothing, or only checks something that is always true.

These are what an agent produces when asked to "add tests". They look like work, they raise coverage, and the summary says "47 tests pass". One quick check finds the worst of them: make every function return nothing and see what still passes.

When a measure becomes a target, it stops being a good measure. Point anyone at a number and they will move the number, whether or not the thing it stood for moves too.

An agent optimises what the check reads, faster and more literally than a person. Any number you put in a gate can go up for the wrong reason, so keep at least one check it can't see and can't optimise for.

A tool makes one small deliberate bug in your code, such as changing a < to <=, and runs your tests. If they fail, that bug was caught. If they still pass, your tests have a blind spot exactly there.

This is the check that makes it hard for an agent to mark its own homework, because it doesn't trust the green. And a surviving mutant is a perfect instruction: it names a line and a change no test noticed.

The share of deliberate bugs your tests caught: killed divided by total. 18 of 30 is 60%. It measures how strong the tests are, not how right the code is.

It gives an agent a number to climb and a list of what to fix. It also can't see anything nobody asked for: tests can catch every mutant in code that solves the wrong problem.

A mutant that changes the code but not what it does, so no test could ever catch it. It survives, and that says nothing bad about your tests.

So 100% is not always possible, and "kill every survivor" needs judgement. Push an agent too hard and it will write strange tests, or bend the code, to chase mutants that are not real bugs.

Instead of a few examples, you state a rule that must always hold, such as "the shares always add up to the bill". The tool invents hundreds of inputs, and when one fails it cuts it down to the smallest input that still fails.

It is much harder to fit code to a rule than to an example. Rules also answer the oracle problem: you may not know the right split of 7312 between 6, but you know it has to add up. For an agent, the shrunk failure makes a very clear bug report.

Throwing huge numbers of random or damaged inputs at a program to see if it crashes or hangs. You do not need to know the right answer, only that it should survive.

Agents tend to write the happy path and test the happy path. A fuzzer has no idea what the happy path is. That makes it especially good for anything that parses input from outside, which is often where the security bugs are.

Checking a claim by running the real program on the surface a user touches and looking at what happens: open the page, send the request, read the row in the database. What you saw counts, and what the agent says it saw does not.

An agent's "it should work now" is a prediction, and "it compiles" is a proxy. A real run is the check that catches what nobody wrote a test for, such as a page that renders blank. It costs tokens and time, and it proves only the paths you drove.

Something that must always be true, at every moment, whatever happens: a balance never goes below zero, a refunded order never ships, the shares always add up to the bill. It is a plain sentence first and code second.

You can say an invariant without reading the code, which makes it the part of the job you can do and an agent can't guess. It is also what every check from here up needs: a tool can try every order of events only if something says which outcomes are wrong.

A tool walks every state a small model of your system can reach, in every order the steps could happen, and checks an invariant in each. If one breaks, it hands you the shortest sequence of steps that gets there. The model is a short description of the design, not your code.

Agents write the happy order and test the happy order, and a race only shows on some other order. The checker doesn't need to be lucky. But it checks the model the agent wrote, so a step the model leaves out can't fail. Replay every counterexample on the real code, and make sure a broken model fails.

Proving that code or a design meets a written specification for all inputs, not only the ones you tried, with a machine checking the proof. It covers what the spec says, under what the proof assumes, and nothing else.

A proof checker rejects an invalid proof however plausible it sounds, so it is one check an agent can't talk its way past. The cost moves to the spec. When one agent writes the code, the spec and the proof, one misunderstanding can sit in all three, and that is self-evaluation bias with a stronger label on it. It can also weaken the spec until it passes, the way it can edit a test. So someone else has to read the spec, usually you, and reading specs is a skill too.

A drawing of a system as the states it can be in and the moves between them. Anything not drawn isn't allowed, so a refunded order simply has no arrow to shipped. A statechart adds states inside states, and regions that run side by side.

Flags and if-statements are the easy thing to generate, and they allow combinations nobody meant. One status and a table of moves makes those impossible to enter. A chart also gives you something to check an agent's understanding against, without reading the code.

Changing the tests so they pass, in place of fixing the code: skipping a test, deleting it, changing the expected answer, weakening the check, or making the code recognise the test's exact input.

The run goes green, and the summary says "fixed". A tampered run looks exactly like a real fix unless you read the diff of the tests or re-run the old tests against the new code. Both can be automated.

A test of what the software should do, kept where the agent can't read it or change it. The agent is judged on it afterwards, so it can't fit its work to it.

Everything else in this chapter can be seen and so optimised for. A holdout can't. It also catches a different class of failure: tests that are strong, and code that is consistent, but a job nobody asked for properly.

Choosing where a limited budget of tokens and attention goes: mechanical gates, running the real thing, formal checks, or a model reading the code. Each sees things the others can't, so ask what this change could get wrong, and which check could see that.

An agent will spend on whichever check you ask for, and each one reports success in the same tone. Good practitioners disagree about where the money should go, and the evidence is thin. So the skill isn't knowing the right mix. It is being able to say what you bought, and what you still can't see.

A test for an AI system: give it a task, grade the result, and do that many times, because it may answer differently each time. The result is a rate, not a pass or a fail.

It is how you test the agent's setup and not its code: did that AGENTS.md change help, or did it only feel like it? Without evals you are judging your harness from a few memorable runs.