Chapter 12 of 18

Review

Reading a diff for what matters, using a second agent well, and owning what you merge.

A green check is a narrow statement. It says that the tests somebody thought of pass on the code that exists. When the same agent wrote both, those tests share the code's blind spots. In this course's own experiment, 18 one-shot solutions were broken, and 13 of them passed their own tests. It was one small model on one-function tasks, so don't stretch the number, but don't expect green to mean right either.

So somebody has to read the work, and whoever merges it owns it. "The agent wrote it" won't get you anywhere at 2am or in an interview. You can get help from a second agent, and this chapter shows what that help was worth when we measured it, including how far a small change of wording moved it. You still decide, and you still answer for the decision.

The hard part is now volume. Chen and Stratton, in a working paper using 445 million work events from 655 firms, found that once teams adopted agents, pull requests opened rose 35.7% and the time from submission to merge rose 56.1%. Their conclusion: "Code review becomes the bottleneck." Writing got cheap and reading did not. Most of the skill in this chapter is spending scarce attention where it counts.

Green doesn't mean right

Also called Verification gap

A passing check only proves what it checked. When one agent writes the code and its tests, all of them can pass while the code does the wrong thing, because the tests came from the same misunderstanding.

Before agents, the person writing the tests had usually spent hours with the problem, so a green run carried that thinking with it. An agent produces code and tests in one breath, from one reading of your brief, in seconds. Green now tells you the two agree with each other.

Say it to your agent

“The tests are green and I don't trust that yet. List three inputs or orderings your tests don't cover. For each one you think is a real bug, write a failing test and show me the red output before you change any code.”

If green can't settle it, someone has to look.

Code review

Someone other than the author reads a change before it joins the main branch and decides whether it should. It is a judgement about the change, made by a person or an agent, on top of whatever the automatic checks said.

Review used to teach a second person the code as a side effect. With an agent as author, that side effect matters more: the author forgets everything when the session ends, so the reviewer may be the only one who ever understands the change.

Say it to your agent

“Before you open the pull request, run git diff main...HEAD, read the whole diff as if someone else wrote it, and tell me what you would ask them to change.”

Reading a diff

A diff shows a change rather than a program: red lines went away, green lines arrived, and plain lines are context. You read it for what changed, and for what should have changed and did not.

Agents reformat, re-indent and move code as they go, and a whole-function re-indent can hide a removed check in plain sight. Learning a few ways to cut the noise pays for itself on the first agent-written pull request you read.

Say it to your agent

“Show me git diff --stat main...HEAD first, then git diff -w main...HEAD for src/auth/ only. Tell me which files changed only mechanically, so I can skip them, and say how you know.”

What to look for, in order

A review has an order: is this the right change, is it designed sensibly, does it work, is it tested, can it be read. Style comes last, because a linter should be handling it.

Agents rarely get syntax wrong. They solve the wrong problem, handle a case nobody asked about and miss the one you meant, add more than you asked for, and weaken a test to get to green. So those go first, and you stop as soon as an early one fails.

Say it to your agent

“Review in this order and stop to tell me if a step fails. 1. Does it do what the brief asked? Quote the brief. 2. What did it change that wasn't asked for? 3. Did any test get weakened or deleted? 4. Which inputs aren't tested? Ignore formatting, because Biome handles that.”

The size of a diff tells you how long it takes to read. It doesn't tell you how much to worry.

Blast radius

How far a change can reach: everything that imports it, calls it or relies on how it behaves, directly or through something else. A one-line change to shared code can have a bigger blast radius than a thousand new lines nobody uses yet.

An agent sees the file it is editing far better than the callers it hasn't opened, and a reviewer reading the diff sees only the lines that changed. Nobody in that exchange is looking at the code that will really break, unless someone goes and looks.

Say it to your agent

“Before you change formatMoney in src/lib/money.ts, run rg "formatMoney" and list every caller. Pick the riskiest one, write a test that pins how it behaves today, and run that test before and after your change.”

The obvious second reader is the agent itself, which is the worst choice.

Self-evaluation bias

An agent marking its own work gives itself too high a mark. It checks what it meant to build rather than what it built, and from the inside those look the same.

"Are you sure?" is a weak question, because you are asking the same mind, with the same assumptions in front of it, to find what those assumptions hid. Separate the checker from the author and the check starts to mean something.

Say it to your agent

“Don't mark this as done yourself. Write what you changed and how to check it into HANDOFF.md. I will give that file and the diff to a new session, and that session decides.”

A fresh agent

A new session with none of the conversation that produced the code, handed the diff and the spec and asked to judge it. It has no history to defend and no memory of why any of this seemed like a good idea.

This has no real equivalent before agents. You could never make a colleague forget a week of context on demand. Now a stranger who has read nothing is one command away and costs a few cents, so there is little excuse for letting an author mark its own work.

Say it to your agent

“Open a new session and run: claude -p "Read docs/spec.md and the output of git diff main...HEAD. You did not write this. For each defect, quote the line and give the input that breaks it. If you find none, say so."”

A fresh agent still has to be told what you want from it. This is the most surprising result in this repo.

The reviewer's prompt

The words you give an AI reviewer. They set how hard it pushes, and they can move the verdict more than the code does. So the prompt is a setting with a cost on each side, and you choose it knowingly.

A sharper prompt catches a little more and blocks far more. Good code that gets blocked costs real time, and it teaches a team to ignore the reviewer. Missed bugs cost something too. Neither prompt is correct. The mistake is not knowing which trade you made.

Say it to your agent

“Review this diff. Report only defects you can demonstrate: quote the line, give the input, and say what it returns and what it should return. If you find none, say so and approve. Approving is a normal outcome here.”

Findings must show evidence

A review finding has to come with what proves it: the line, the input, what happens and what should happen. A finding without that is an opinion, which is worth having, and which should not stop a merge.

A model writes a false finding in exactly the same confident voice as a true one. Asking for a failing input changes the job from persuading you to showing you. It also separates two questions that often get mixed up: is this true, and does it matter.

Say it to your agent

“Run every finding before you report it. For each, give me the command you ran and its output. Anything you could not run goes in a separate list called unverified, and it does not block this merge. Finding nothing is a fine result.”

Evidence deals with false findings. There is a second way a reviewer wastes your time: everything it says is true, tidy, and not about the behaviour.

Review in the abstract

Also called Abstract review

Reading code with nothing to check it against: no run, no stated intent, no question to answer. The reviewer fills the gap with what can be said about any code, which means small style points and warnings about things you may not need.

A reviewer needs a narrower job than "review this". Give it one behaviour to check and a way to check it, and the easy comments stop being the answer. Leave it open and you can spend your tokens, and your afternoon, on comments that were never going to change what the code does.

Say it to your agent

“Review this diff against one question: does POST /refunds still refund at most what was paid, when the same request arrives twice? Run it and paste the output. Leave out style, naming and anything that only matters after a future change. If you have nothing on that question, say so and stop.”

AI code reviewers

A product that reads your pull requests with a model and leaves comments on the lines it doubts. Most of them comment and leave the decision with a person.

They catch things no rule can express, and they cost tokens, take time, and give a different answer on a second run. Their vendors know this: the common default is that the bot's opinion is not an approval. Treat one as a very fast colleague who has never met your users.

Say it to your agent

“Add a REVIEW.md telling the reviewer to report only logic errors and security holes, each with the input that triggers it, and to ignore style because Biome handles that. Then run /code-review on this branch and show me what it says.”

Reviewers can advise, whether they're people or models. The name on the merge is yours.

You own what you merge

Whoever merges a change answers for it, whoever or whatever typed it. That means being able to say what it does, what could go wrong, how you would notice, and how you would undo it.

An agent has nothing at stake. It won't be paged, can't be asked next month why it did this, and won't remember if you ask. This is what is left of the job when typing is cheap, and employers now say so in writing.

Say it to your agent

“I am merging this under my name. Before I do, tell me what could go wrong in production, how we would notice, and the exact command that undoes it. If you can't answer one of those, say so and don't guess.”

Code owners

Also called CODEOWNERS

A file in the repo that maps paths to the people or teams responsible for them, so a pull request touching those paths automatically asks the right reviewer. On GitHub it can also be made a requirement for merging.

An agent is happy to edit anything it can reach, and it has no idea that the payments folder is sensitive at your company. A code owners rule turns that knowledge into something the machine enforces, whoever or whatever opened the pull request.

Say it to your agent

“Add .github/CODEOWNERS with the catch-all first, because the last match wins, then /src/payments/ for @acme/payments. Then tell me the setting to turn on so that review is required and not only requested.”

There is a cost to merging what you don't understand, and two loosely defined terms are competing to name it. Use them, and be ready to say what you mean.

Comprehension debt

The gap between how much code exists and how much of it anyone understands. Like money borrowed, it costs nothing today, and you pay later, in the time it takes to work out the code before you can safely change it.

Agents remove the friction that used to build understanding as a side effect. The code can be clean, tested and passing and still be something nobody on the team could change safely. No tool in your pipeline measures this.

Say it to your agent

“Before you add anything to src/billing/, write a one-page map of how it works today, from reading the code. Mark anything you couldn't explain. Those marked parts are where I need to catch up before we go further.”

Cognitive debt

A close cousin of comprehension debt, and many writers use the two as one. Where a difference is drawn, this one is about people rather than code: a team gradually losing its shared picture of how the system works and why.

One person can understand their corner and the team can still have lost the whole. That explains a team that can't make simple changes without breaking things, even with tidy code. It also shows why one person reading more diffs doesn't fix it.

Say it to your agent

“Write docs/decisions/0007-payment-retries.md explaining why we retry payments this way and what we rejected, from the commit history and our conversation. Then each of us reads it and I will quiz the team on it on Friday.”

Explain it back

Before accepting code, say what it does in your own words, including what it does on awkward inputs, then check yourself by running it. If reality disagrees with you, you have just found what you did not understand.

Reading code gives a strong feeling of understanding that often isn't real, and an agent's tidy explanation makes that feeling stronger. Predicting and then running is a test the feeling can fail. It is also the skill an interviewer is checking when they switch the agent off.

Say it to your agent

“Don't explain src/cart.ts to me yet. I am going to tell you what happens on an empty cart and when the payment call times out. Then write those two cases as tests, run them, and show me where I was wrong.”

Now the third part of the argument. All of this reading has to be done by someone, and there is suddenly much more of it.

The review bottleneck

The point where agent-written work piles up waiting for someone to read it. Writing code got cheaper and reading didn't, so the queue in front of reviewers is now what limits how fast a team ships.

Producing more pull requests stops helping once they wait longer, and the pressure to clear the queue is exactly what produces careless approvals. It is also why individual speed-ups can vanish at team level.

Say it to your agent

“Stop opening new pull requests. Run gh pr list --author "@me", tell me which of mine are waiting and for how long, and which one would unblock the most others if it merged first.”

Small pull requests

Keeping each pull request to one reviewable idea, often a few hundred lines at most. Long changes get split so that each piece makes sense and passes its checks by itself.

An agent can produce 2,000 lines as easily as 20, and it doesn't feel the difference. The reviewer does. Long diffs get skimmed, and a skimmed diff is a diff that wasn't reviewed. Size is the limit you have to set, since nothing about the agent will.

Say it to your agent

“This is 1,400 lines, which is too much to review well. Split it: first a refactor with no change in behaviour, then the schema, then the feature, each under 300 lines and each passing bun test by itself.”

Stacked diffs

Also called Stacked pull requests

A long change split into a chain of pull requests, each built on the one before it, so each can be reviewed alone and you don't have to wait for the first to merge before writing the next.

An agent works faster than any reviewer, so it will always want to carry on before the last piece has been read. A stack lets it, while keeping each piece small. The cost is that a fix low down means updating everything above, and that is the kind of chore you can hand to an agent.

Say it to your agent

“Split this into a stack: payments-1-schema off main, payments-2-api off that, payments-3-ui off that, with one pull request per branch, based on the branch below. After any fix lower down, go to payments-3-ui and run git rebase --update-refs payments-1-schema.”

Making work reviewable

Doing the work so a reviewer can check it cheaply: one idea per pull request, a description saying what changed and why, and pasted evidence that it works, with the output of what you ran. You spend your time to save theirs.

When reading is the bottleneck, the most useful thing an author can do is make reading easier. An agent will happily write a description that claims everything worked. A description that pastes the output has to be true, or at least it can be checked.

Say it to your agent

“Write the pull request description with these headings: Why, What changed, How I checked. Under How I checked, paste the failing test output from before the fix and the passing output from after. Label anything you didn't run as not run.”

The last few words are the small ones reviewers use every day, and they matter more now that a bot may be reading them.

Nit

Also called Nitpick

A review comment marked as small and optional: a matter of polish that the author can take or leave. The label tells them not to hold the change up for it.

To a person, "nit:" means "no rush". An agent told to address the review comments may treat every one as an order, including the ones you meant as a passing thought, so say which are optional. And if the same nit keeps coming up, make it a lint rule and stop saying it.

Say it to your agent

“Fix the two comments labelled blocking. Leave the ones labelled nit alone. Then add a Biome rule for the naming one, so that it doesn't come up again.”

Blocking and non-blocking

A blocking comment must be dealt with before the change can merge. A non-blocking one is advice. Saying which one you mean is half of what makes a review useful.

An AI reviewer can produce a dozen comments in seconds, written in the same serious tone. If all of them block, the queue stops; if none do, the real one gets lost. A good rule for agents: only what can be demonstrated blocks, and the rest goes in the notes.

Say it to your agent

“Label every finding blocking or non-blocking. Blocking means you ran it and can show me the failure. Everything else is non-blocking, however serious it sounds. Put the blocking ones first.”

LGTM

Also called Looks good to me

Short for "looks good to me": the usual way to approve a pull request. It is supposed to be a claim that you read it and would stand behind it.

It costs four letters and carries your name. When agents make the diffs long and frequent, it's very easy to type without the reading behind it. Interviewers now look for exactly this.

Say it to your agent

“I am not approving this on reading alone. Check out the branch, run bun test and the app, try the case from the brief where the cart is empty, and paste me what happened.”

Rubber-stamping

Approving without really reviewing, because the last fifty were fine and there are ten more waiting. The approval still gets recorded, so from outside it looks exactly like a review.

The better the agent gets, the more dangerous this is. If it is right nineteen times in twenty, you learn to stop looking, and the twentieth goes through with your name on it. Human review is not a reliable backstop just because it exists.

Say it to your agent

“I approved your last five without finding anything, so I no longer trust my own attention here. For this one, tell me the single riskiest line and the input that would break it. I will read that first and run it myself.”

Do it for real

Reading gets you the words. This gets you the experience behind them, and something to show.

Find a bug your tests can't

This is the whole argument of the chapter on your own code. You will see green and wrong at the same time, and you will have the commit to prove it.

  1. Pick a function an agent wrote in a real project, with tests that pass. Something that takes outside input is best.
  2. Without showing the agent, write three scenarios from what the function is for: odd input, wrong order, a retry, an empty value.
  3. Run them against the code. If they all pass, write harder ones, because you haven't finished until one fails or you can say why none can.
  4. Commit the failing test on its own, then give it to a fresh agent and have it make the fix in a separate commit.

Done when: Two commits: a failing test that the agent's suite missed, then the fix. You can point at them in an interview and explain the gap.

Measure your own reviewer

Our numbers came from one small model on tiny tasks. Yours is the one you have to live with, and an hour of measuring is what turns an opinion about AI review into a decision you can defend.

  1. Take one correct function from your project, and a copy with a real bug you put in on purpose.
  2. Using a fresh session each time, for example claude -p, have your reviewer judge each one 20 times with your current prompt, and 20 times with a prompt that pushes it to find a defect.
  3. Write the results in a table: approvals on correct code, rejections on buggy code, for each prompt.
  4. Choose a prompt and write one sentence on why you accepted that trade. Commit the table, the prompt and the sentence.

Done when: A committed file showing your reviewer's real approval rates under two prompts, and the one you chose with your reason.

Explain a file back from memory

Interviewers ask you to walk through your own project with the agent switched off. Find out now which parts you can't do, while it's still cheap to fix.

  1. Pick a file an agent wrote for you last week, and close the agent.
  2. Without looking anything up, write down what it does on three inputs, including one awkward one.
  3. Now run those three inputs for real, in a test or a script, and compare.
  4. For each thing you got wrong, read the code until you can say why. Then write one line on what you had assumed.

Done when: A page with your predictions, the real output and what you had wrong. Keep it: it is honest evidence of how you work.

The build, stage 12 of 18

Green and still wrong

Comments on a real group page, two reviews of the same code that found different things, and a crash that 64 green tests missed.

About 120 to 180 minutes, on the same app as every other stage. We ran it first and show what went wrong.

Open stage 12 of the build

Gate 12: Review

0 of 6 right

  1. 1An agent hands you a payment webhook handler. Its 9 tests pass, its mutation score is 100%, and CI is green. What do you do before merging?

  2. 2Your AI reviewer rarely rejects anything, and a bug got through last week. A colleague proposes changing the prompt to "find a real defect, and do not approve out of politeness". Going by this repo's experiment, what should you expect?

  3. 3A reviewer rejects your code: splitBill(100, [1e308, 1e308]) returns NaN. You run it, and it does. Your product splits restaurant bills. What is the right call?

  4. 4An agent-written change caused an incident overnight. You approved it after reading the diff. In the morning a manager asks whose bug this is. What is the honest answer?

  5. 5You added /src/payments/ @acme/payments to CODEOWNERS, with * @acme/everyone at the bottom of the file. A payments change merged with no payments review. Why?

  6. 6Your agent opens a 1,400-line pull request. Reviewers are already days behind. What gets you the most review for the least time?

Get every answer right to install it, and to put a file in your repo. Wrong answers cost nothing.

Next chapterSecurity

The workshop

Name a word, or describe what you mean in everyday language.