The experiment

The course makes a claim: the right words and real gates get you better work from an agent. We tested it, 480 times, on a real model. This page reports what happened, including where the results went against what we expected.

In short: the right words helped, and most of that help came from one task. The harness made code pass more of its own checks, and barely changed whether the code did what was wanted. The gates were weak at spotting broken code, and our own hidden test missed a real security hole.

This is one small measurement. The limits come straight after the result, on purpose.

The result

Runs that passed every hidden scenario

Vague brief, one shot88% 80 to 92

Vague brief, harness on91% 84 to 95

Right words, one shot98% 93 to 99

Right words, harness on100% 97 to 100

Amber is the vague brief and blue is the right words. A stripe means the harness was on. The line under each bar is its 95% interval, and those are the numbers beside the figure. 105 runs per bar without that task, 120 with it.

Tasks counted

88%, 91%, 98%, 100%. The vague brief’s intervals don’t overlap the right words’ at the same harness setting, and each brief’s harness-off and harness-on intervals do overlap. So in this data the brief made a visible difference, and the harness did not make one we can tell from noise.

Then change which tasks are counted. Without one task, payment-events, the figures are 96%, 99%, 97% and 100%, and every interval overlaps every other.

A 95% interval here treats each run as independent. It says how sure we can be about these 8 tasks, on this model. For a task we didn’t try, the number that matters is much closer to 8 than to 480, and no interval on this page covers that.

What it can’t tell you

Read these before the findings, because they decide what the findings are worth.

One small model: Everything ran on Claude Haiku 5.5. A stronger model might need no help on these tasks, and a weaker one might need more.
One-function tasks: Each task is one function in one file. Nothing here tests changing code that already exists, or work that spreads across files, which is where we’d most expect gates to earn their keep.
No repo, no tools, no context: The agent got a brief and replied with files. It couldn’t read a codebase, run anything or look anything up. A real coding agent does all of those.
Small numbers per task: 15 runs per cell. One run going the other way moves a task’s figure by about 7 points.
Tasks chosen by someone with a stake: The tasks and the hidden scenarios were written by the same person who hoped the course would work, after replacing a first set that showed too little. There’s more on that below.

It doesn’t show that gates are useless at scale. We measured whether a gate predicts a hidden-scenario failure on a single function. We didn’t measure whether gates keep a large codebase in order over months.

It doesn’t show that words always win. On 3 of the 8 tasks, every cell scored 100% whatever the brief said, and the biggest effect belongs to a single task.

How it was run

A 2×2 design. The brief is either vague or uses the right words. The harness is either off or on. That makes four cells, with 8 tasks and 15 runs per cell, so 480 runs, all on Claude Haiku 5.5 through claude -p with tools, settings and MCP servers switched off.

Off means one shot: the agent replies once and that reply gets judged. On means what the gates said goes back to the agent, for up to 3 retries.

Every reply went through the same real gates:

Types: tsc with strict plus noUncheckedIndexedAccess.
Lint: Biome’s recommended rules on the implementation.
Tests: the agent’s own, under bun test. At least one has to pass.
Mutation: a mutation tester breaks the code in small ways, up to 30 mutants, and asks whether the agent’s tests notice. The bar was a score of 0.7.

Then the part the agent never saw. A hidden holdout of 91 scenarios across the tasks: 41 core, which is what the vague brief literally asks for, and 50 trap, which is what an experienced engineer would insist on. A run counts as passing only if it passes all of its task’s scenarios. A second agent in a fresh session also reviewed the final code, without being able to run it.

Code the model wrote ran in sandbox-exec with no network, and could only write to its own folder and the temporary folders.

The main run cost $2.51 over 1,097 model calls and about 184 minutes of model time run in parallel, with results stamped 5 November 2026, 09:21 to 09:54 UTC. The cost is what claude -p reported. The reviewer test and the neutral re-review recorded no cost, so they aren’t in that figure.

Task

Split a bill

The vague brief

Write splitBill. It splits a restaurant bill between people according to their weights.

The right words

Write splitBill. It allocates a bill between people in proportion to their weights, using the largest remainder method. Invariant: the shares are integers and always sum to exactly totalCents; no cent is lost or invented. Each person first gets the floor of their exact share. Hand out the leftover cents one at a time to the largest fractional remainders. Break ties deterministically, earliest index first. A person with weight 0 gets 0. Guard the inputs at the boundary: throw RangeError when weights is empty, when any weight is negative, when all weights are zero, or when totalCents is not a non-negative integer.

The signature, sent with both

export function splitBill(totalCents: number, weights: number[]): number[]

Redirect after login

The vague brief

Write safeRedirect. After login we send the user to target. Return target if it is safe to redirect to, otherwise return "/".

The right words

Write safeRedirect. It prevents an open redirect after login. target is untrusted input from a query string. Use an allowlist, never a denylist. Return target unchanged when it is either a same-site relative path (a single leading slash followed by anything other than a slash or backslash), or an absolute http or https URL whose hostname exactly matches an entry in allowedHosts, case-insensitively. Otherwise return "/". Parse with the URL class rather than string matching. Reject protocol-relative URLs (//host), backslash tricks (/\host), control characters and surrounding whitespace, userinfo (https://good.com@evil.com), suffix lookalikes (good.com.evil.com), and every other scheme, including javascript: and data:.

The signature, sent with both

export function safeRedirect(target: string, allowedHosts: string[]): string

Save an uploaded file

The vague brief

Write resolveUpload. It returns the full path to save a user's uploaded file inside baseDir. Use node:path.

The right words

Write resolveUpload using node:path. filename comes from the user, so treat it as untrusted and prevent path traversal. Resolve the full path, then check it is still inside baseDir. Compare with a trailing separator, because a plain prefix check lets /srv/uploads-evil pass for /srv/uploads. Throw an Error for absolute paths, null bytes, an empty name, and anything that escapes or resolves to baseDir itself.

The signature, sent with both

export function resolveUpload(baseDir: string, filename: string): string

Render a comment

The vague brief

Write renderComment. It turns a user's comment into HTML: **text** becomes bold and [label](url) becomes a link.

The right words

Write renderComment. It turns a user's comment into HTML: **text** becomes <strong> and [label](url) becomes a link. The comment is untrusted, so prevent stored XSS. Output-encode the whole input first: & < > " and ' become entities. Only then apply the formatting. Allowlist URL schemes: only http, https and mailto become a link. For any other scheme, including javascript: in any capitalisation, and data:, output the label as plain text.

The signature, sent with both

export function renderComment(text: string): string

Cache an async call

The vague brief

Write memoAsync. It caches the results of an async function by key.

The right words

Write memoAsync. It caches an async function's results by key and prevents a cache stampede: concurrent callers for the same key must share one in-flight promise, so fn runs once. Do not cache failures. When the promise rejects, every waiter sees the rejection, the entry is evicted, and the next call runs fn again. Keys are arbitrary strings, including ones like "constructor", so use a Map rather than a plain object.

The signature, sent with both

export function memoAsync<T>(fn: (key: string) => Promise<T>): (key: string) => Promise<T>

Handle payment webhooks

The vague brief

Write createPaymentProcessor. apply(event) updates an order's status from a payment provider's webhook event. It calls onPaid when an order gets paid, so we can ship it.

The right words

Write createPaymentProcessor. Webhooks are delivered at least once and out of order, so apply must be idempotent. Dedupe by event id: a redelivered event changes nothing. An event older than the newest one applied to that order, by at, is stale and changes nothing. onPaid fires exactly once per order, however many distinct paid events arrive, and never for a stale event. status(orderId) returns the type of the newest applied event.

The signature, sent with both

export type PaymentEvent = { id: string; orderId: string; type: 'paid' | 'refunded' | 'failed'; at: number } export function createPaymentProcessor(onPaid: (orderId: string) => void): { apply: (event: PaymentEvent) => void; status: (orderId: string) => string | undefined }

Work out a billing date

The vague brief

Write nextBillingDate. A subscription started on anchor and bills monthly. Return the date of bill number periods.

The right words

Write nextBillingDate. A subscription started on anchor and bills monthly; return the date of bill number periods. Handle month-end clamping without anchor drift: always work from the anchor's day, not from the previous result. If the target month is too short, use its last day. So Jan 31 gives Feb 28 then Mar 31. Work in UTC only, so the host's timezone and daylight saving cannot shift the date, and keep the UTC time of day. Never mutate the input Date.

The signature, sent with both

export function nextBillingDate(anchor: Date, periods: number): Date

Build a search query

The vague brief

Write buildSearch. It builds a SQLite query for our products table (columns name, price, created_at) that finds rows whose name contains term, sorted by the sortBy column, up to limit rows. Use ? placeholders.

The right words

Write buildSearch. It builds a SQLite query for the products table (columns name, price, created_at) and must be safe from SQL injection. Use ? placeholders for values. Placeholders cannot hold identifiers, so allowlist sortBy against the known columns, ignoring anything else. Escape LIKE wildcards in term, % and _ and the escape character itself, and add an ESCAPE clause, so a search for 50% finds a literal 50%. Coerce limit to an integer, clamp it between 1 and 100, and bind it as a parameter.

The signature, sent with both

export function buildSearch(filters: { term?: string; sortBy?: string; limit?: number }): { sql: string; params: unknown[] }

What we found

A mechanical gate can’t catch what nobody asked for

payment-events asks for a handler for a payment provider’s webhooks. The vague brief says:

Write createPaymentProcessor. apply(event) updates an order's status from a payment provider's webhook event. It calls onPaid when an order gets paid, so we can ship it.

It doesn’t say that webhooks arrive more than once and out of order. With that brief, 27% of one-shot runs passed, and 33% with the harness. With the right words, 100% and 100%.

Only 2 of its 9 scenarios ever failed, and they are the same situation: a refund arrives, then an older paid event turns up late. The order should stay refunded and not ship. “a late, older paid does not undo a refund” failed 11 of 15 one-shot runs and 10 with the harness. “a late, older paid does not ship” failed 8 and 10.

The mechanical gates had almost nothing to say. Of the 11 broken one-shot runs, types stopped 0 and lint stopped 0, and 9 passed all four gates. The other 2 failed because the agent’s own test about late events disagreed with its own code. Retrying can’t fix what no check mentions: 5 of 15 passed with the harness, against 4 of 15 without.

Reading did better than running, with a catch. The neutral reviewer rejected all 15 vague-brief runs of this task, including the 4 that passed every scenario, and 0 of the 15 worded ones. By a keyword count on the stored text, 10 of the 11 reviews of broken code mention ordering or stale events, but 4 lead with something else: if onPaid throws, the event is already marked as seen, so a retry never ships the order. So do all 4 rejections of the passing runs. That’s a real hazard the holdout never tests, so calling those rejections false alarms is generous to our holdout.

So the line is narrower than it first sounds, which is why the heading says mechanical. A check that only runs what was specified can’t catch what wasn’t. Someone reading the code and thinking about webhooks sometimes can.

Most of the gap comes from this one task. Without it, the one-shot gap between the briefs falls from 10 points to 1, and with the harness on, from 9 points to 1. Dropping a task after seeing the results is a choice too, so read that as a check on how fragile the headline is, and not as the real answer. 3 tasks scored 100% in every cell, so they couldn’t have shown any effect at all.

Passed every hidden scenario, by task

Task	Vague, off	Vague, on	Words, off	Words, on
split-bill	93	93	93	100
safe-redirect	87	100	100	100
resolve-upload	100	100	100	100
render-comment	93	100	93	100
memo-async	100	100	100	100
payment-events	27	33	100	100
next-billing	100	100	100	100
search-query	100	100	93	100

Percentages of 15 runs. Off and on mean the harness. Redder means more failures.

The hidden scenario: a late, older paid does not undo a refund

(m) => { const { p } = processor(m); p.apply(ev("2", "o", "refunded", 2)); p.apply(ev("1", "o", "paid", 1)); eq(p.status("o"), "refunded"); }

The hidden scenario: a late, older paid does not ship

(m) => { const { p, paid } = processor(m); p.apply(ev("2", "o", "refunded", 2)); p.apply(ev("1", "o", "paid", 1)); eq(paid, []); }

What the harness really changed

Final code passing strict types plus unchecked index access went from 80% to 99%. Lint went from 85% to 100%, and the average mutation score from 80% to 90%, at an average of 1.6 tries.

Those are real changes to the code, and they’re partly circular: the harness tells the agent to raise exactly those numbers, which is Goodhart’s law waiting to happen. The number nobody told the agent about, the hidden scenarios, moved 3 points, from 93% to 95%.

140 of 240 harness runs passed the gates first time, and the most any took was 4 tries. 2 ran out of tries with a type error still there, which is why types stops short of 100%. Per run, not counting the reviewer, the harness cost $0.0041 against $0.0022.

None of that makes the harness pointless. Code with stricter types and tests that notice breakage may be much easier to change next month. This experiment ends when the function is written, so it can’t see that.

What the final code did, across all 480 runs

One shot Harness on

Passes types80% to 99%

Passes lint85% to 100%

Own tests pass91% to 100%

Average mutation score80% to 90%

Every hidden scenario93% to 95%

The gates are weak detectors here

To judge a gate as a detector, take the 240 one-shot runs, where no gate had any say in the result. 18 failed a hidden scenario, so call them broken. 222 passed them all, so call them good. Then ask what each gate would have stopped.

Base rates matter. With only 18 broken in 240, most stops are false alarms, even from a gate that works. A gate saying no at random would be right 7.5% of the time. Types managed 2% and lint 6%, which is worse than random in this sample. Tests and the neutral reviewer did best, at about 24% and 25%.

Types stopped 1 broken and 47 good, but that gate was stricter than its name, because it included noUncheckedIndexedAccess. Plain strict alone stops 0 broken and 3 good of those 48.

Lint stopped 34 good runs, and 32 of them tripped one rule, noControlCharactersInRegex, which objects to regular expressions that look for control characters, and looking for them is what a careful URL check does. Our own reference solution needs an exemption from that rule too.

Mutation isn’t independent of tests. A run whose tests fail scores zero, so the 5 broken runs it stopped are the same 5 that tests stopped. Among runs whose tests passed, it stopped 0 broken and 18 good. It’s also a coarse instrument here: the median program had 10 mutants, 120 of 459 tested programs had 5 or fewer, and 2 had none and scored zero.

All four mechanical gates together stopped 8 broken and 99 good.

A stop isn’t a diagnosis either. The broken run types stopped had a complaint about a possibly undefined value, and what it really failed was refusing empty weights. The 2 that lint stopped were flagged by noControlCharactersInRegex, and really failed on upper-case hostnames. They were right to stop for the wrong reason.

And “false alarm” only means it passed the hidden scenarios. A warning that a value may be undefined can be perfectly true. So this table says these gates were weak predictors of these scenarios on one-function tasks, and nothing wider.

What each gate stopped, of 240 one-shot runs

Broken code stopped Good code stopped

Types1 broken, 47 good

Lint2 broken, 34 good

Tests5 broken, 16 good

Mutation5 broken, 34 good

Reviewer, neutral prompt13 broken, 39 good

Reviewer, told to find a defect16 broken, 126 good

18 broken and 222 good. Saying no at random would be right 7.5% of the time.
Gate	Broken stopped	Good stopped	Stops that were right
Types	1 of 18	47 of 222	2%
Lint	2 of 18	34 of 222	6%
Tests	5 of 18	16 of 222	24%
Mutation	5 of 18	34 of 222	13%
Reviewer, neutral prompt	13 of 18	39 of 222	25%
Reviewer, told to find a defect	16 of 18	126 of 222	11%

The reviewer’s prompt decides what you hear

We took one correct function and one with a real bug, and had fresh agents review each 40 times under each of two prompts. On the correct code, the neutral prompt approved 33 of 40. The prompt told to find a defect approved 1 of 40. On the bug, both rejected 40 of 40.

Some warnings about that. The prompts differ by more than one sentence, as the comparison shows, so we can’t say which words did it. And the bug was an easy one: the request shown to the reviewer says not to cache failures, and the code caches them. Note too that the neutral prompt said no 7 times to the same correct code, so a single verdict is flaky whichever prompt you use.

On real code the same pattern shows up. Over the 240 one-shot runs, the neutral prompt stopped 13 broken and 39 good. Told to find a defect, it stopped 16 and 126: 3 more catches for 87 more false alarms. In the harness-on, right-words cell, where every run passed every hidden scenario, that prompt still rejected 66 of 120.

The reviewer looks like the best detector in the table, and much of that record is one task. Without payment-events there are 7 broken runs and 203 good. The neutral prompt stops 2 and 35, and the other prompt 5 and 108.

One more oddity worth reading. The neutral reviewer approved 2 of the 3 empty files. Both times it wrote the missing code itself and then approved what it had written.

How often each prompt approved

Approved Rejected

Correct code

Neutral prompt33 of 40

Told to find a defect1 of 40

Code with a real bug

Neutral prompt0 of 40

Told to find a defect0 of 40

The neutral prompt, exactly

You are a senior engineer reviewing a change. You cannot run the code. Finish with one final line that is exactly VERDICT: APPROVE or VERDICT: REJECT.

The prompt told to find a defect, exactly

You are a senior engineer reviewing code another agent wrote. You did not write it. You cannot run it. Find a real defect if there is one, and do not approve out of politeness. Finish with one final line that is exactly VERDICT: APPROVE or VERDICT: REJECT.

Underlined words are in that prompt and not the other. The underlining is ours, and everything else is the text as sent.

Our own holdout missed a real hole

In the security chapter we went back to the recorded comment renderers with an attack added afterwards. With a vague brief, 7 of 29 produced a javascript: link. With the right words, 0 of 29. The count is 29 and not 30 because one empty file in each brief is left out.

All 7 had passed every hidden scenario. 5 of them had passed every gate as well, and the harness didn’t help: 3 came from one-shot runs and 4 from harness runs.

The attack puts a control character in front of the scheme. A hand-written check doesn’t see javascript:, and the browser’s URL parser throws the control character away. Our scenarios tried plain and mixed-case javascript:, and didn’t try this. So our own holdout missed it, and “good” on this page only means it passed the hidden scenarios. Treat every pass rate here as an upper limit.

A correction to our own wording: the exact string above doesn’t run. These renderers stop reading a link at the first closing bracket, so the script gets cut to alert(1, which is a syntax error, in all 7. Swap in a payload with no brackets, [click](\u0001javascript:alert`1`), and we get the link from the same 7 programs and 0 from the right words, and clicking it in Chromium really ran the script. So the hole is real, and the attack as recorded only shows that the link gets through. The worded brief states an allowlist of schemes and all 29 held, though this design can’t say that’s why. The wider lesson is encoding for the place the value lands.

Comment renderers that produced a javascript: link

Vague brief7 of 29

Right words0 of 29

A solid lamp is one program that produced the link. Harness on and off are counted together.

The attack, with its control character shown

[click](\u0001javascript:alert(1))

What one recorded renderer made of it

<a href="\u0001javascript:alert(1">click</a>)

What we got wrong

And how we found out. A harness on the experiment caught some of this, and looking at strange results caught the rest.

Our first task set was too easy
Claude Haiku 5.5 passed nearly everything from vague briefs on famous problems. So we replaced the tasks with trap-laden ones and kept two as controls. That’s a researcher degree of freedom, and it biases towards finding an effect: we changed the test after seeing it didn’t show what we hoped. The first set’s results aren’t in the published data, so you can’t check this one.
The first reviewer prompt was biased, by us
We wrote “find a real defect” and treated the answers as neutral. We only noticed when the results looked strange: it was rejecting 57% of good code. That’s why the reviewer test exists, and why every one-shot run was reviewed again under a neutral prompt. The neutral reviews are a later pass over the one-shot runs only, so harness-on runs have only the biased review.
Our holdout had a false positive
Our reference solution failed “a quote cannot break out of the link”, because the regular expression matched encoded quotes, which are safe. The instrument test caught it: every reference solution has to pass every scenario before a run means anything.
Our workspaces slowed down our own runs
The experiment’s temporary workspaces lived in the repo, so the dev server restarted each time one appeared, and runs slowed down. They moved outside the repo.
The broken pile isn’t what it sounds like
3 of the 18 broken runs are empty files, and 11 of 18 are payment-events. That leaves very little else. We also can’t say why the files are empty, because we didn’t save the raw replies. They cost $0.0014 to $0.0026 against a median of $0.0021, so the model probably wrote something, and our parser may have lost it. Leave them out and the one-shot figures are 88% and 99%.
The worded briefs state the rules as well
The worded payment brief doesn’t just say idempotent. It also says: “An event older than the newest one applied to that order, by at, is stale and changes nothing.” So this doesn’t isolate what the word itself does. It tests a word plus the rule that goes with it, and the rule alone might do all of the work.
Some agent-written tests really hang
3 attempts, all on memo-async, had tests that never finish. We ran them again to be sure, and each printed the test runner’s banner and then nothing until killed. They count as test failures, and 1 of them is among the good runs the tests gate stopped.

What would change our mind

About the claim

Tasks picked by someone with no stake, where vague briefs do as well as worded ones. Then the words claim shrinks to “sometimes”.
A test where the word alone adds nothing beyond stating the rule. Then the honest advice is to say the rule, and the vocabulary is only a shortcut.
Real repos over months, where gated work breaks as often as ungated work. Then the gates claim fails where it matters most. If gates catch much more on multi-file changes, our gate finding is just a small-task result.

A better experiment

Fix the tasks and the analysis in writing before any run, and publish that.
Have someone else sample tasks from real issues, and write the holdout too.
Split the brief into three arms: vague, the rule in plain words, and the word alone.
Use several models, a real repo, tools and multi-file changes.
Treat the task as the unit of analysis, with enough tasks for that to mean something.
Save every raw reply, and attack the results afterwards as routine.
Follow the code past the first version, to see whether gated code is cheaper to change.

Check it yourself

Nothing here needs taking on trust. The figures on this page are worked out from these files when it renders, and a check in the repo fails if any of them drifts.

main.jsonlEvery run, byte for byte what the experiment wrote: each attempt’s code, tests and gate output, the hidden results, and the first review. 5.1 MB.
review-neutral.jsonlThe neutral re-review of every one-shot run. 310 KB.
review-flake.jsonThe reviewer test: both prompts and the tallies.
tasks.jsonEvery task, both briefs and each hidden scenario with its source.
tasks.ts.txtThe same tasks as the file the experiment imported.

Reproduce the first number in the headline. This prints 105, and 105 of 120 is 88%.

Every task, every scenario

Figures are the share passing every hidden scenario. Open a task to see each scenario and how many runs in each cell failed it.

Split a billsplit-bill93%93%93%100%

Runs that failed each scenario, of 15 per cell.
Scenario	Kind	Vague, off	Vague, on	Words, off
two equal people share evenly	core	0	0	1
three people: no cent is lost	core	0	0	1
awkward weights: no cent is lost	core	0	0	1
shares are whole cents	core	0	0	1
one share per person	core	0	0	1
shares are within a cent of proportional	core	0	0	1
a zero bill is all zeros	core	0	0	1
a large bill still adds up	core	0	0	1
ties go to the earliest index	trap	0	0	1
largest remainder wins the spare cent	trap	0	0	1
weight zero gets nothing	trap	0	0	1
empty weights: refused	trap	1	1	0
negative weight: refused	trap	0	0	0
all weights zero: refused	trap	0	0	0
fractional total: refused	trap	0	0	0
negative total: refused	trap	0	0	0

Redirect after loginsafe-redirect87%100%100%100%

Runs that failed each scenario, of 15 per cell.
Scenario	Kind	Vague, off
a plain path is kept	core	0
a path with a query is kept	core	0
an allowed host is kept	core	0
another host is refused	core	0
javascript: is refused	core	0
protocol-relative is refused	core	0
suffix lookalike is refused	core	0
userinfo trick is refused	core	0
allowed name in the path is refused	core	0
backslash after the slash is refused	core	0
data: is refused	core	0
empty string is refused	core	0
two backslashes are refused	trap	0
leading whitespace is refused	trap	0
a tab inside the slashes is refused	trap	0
hostname case is ignored	trap	2
scheme without slashes is refused	trap	0
ftp is refused	trap	0

Save an uploaded fileresolve-upload100%100%100%100%

Runs that failed each scenario, of 15 per cell.
Scenario	Kind	Vague, off	Vague, on	Words, off	Words, on
a plain name lands in the folder	core	0	0	0	0
dots in a name are fine	core	0	0	0	0
a trailing slash on the folder is fine	core	0	0	0	0
dot-dot cannot climb out	trap	0	0	0	0
dot-dot in the middle cannot climb out	trap	0	0	0	0
an absolute path cannot escape	trap	0	0	0	0
a sibling folder with the same prefix cannot pass	trap	0	0	0	0
a null byte cannot pass	trap	0	0	0	0
an empty name does not return the folder itself	trap	0	0	0	0
a single dot does not return the folder itself	trap	0	0	0	0

Render a commentrender-comment93%100%93%100%

Runs that failed each scenario, of 15 per cell.
Scenario	Kind	Vague, off	Words, off
bold works	core	1	1
a link works	core	1	1
plain words stay	core	1	1
a script tag cannot run	trap	1	1
an image handler cannot run	trap	1	1
a javascript: link is refused	trap	1	1
capitals do not hide javascript:	trap	1	1
a data: link is refused	trap	1	1
a quote cannot break out of the link	trap	1	1
an ampersand is encoded	trap	1	1

Cache an async callmemo-async100%100%100%100%

Runs that failed each scenario, of 15 per cell.
Scenario	Kind	Vague, off	Vague, on	Words, off	Words, on
a later call uses the cache	core	0	0	0	0
keys are kept apart	core	0	0	0	0
concurrent callers share one call (stampede)	trap	0	0	0	0
a failure is not cached	trap	0	0	0	0
every waiter sees the failure	trap	0	0	0	0
the key "constructor" works	trap	0	0	0	0
the key "__proto__" works	trap	0	0	0	0

Handle payment webhookspayment-events27%33%100%100%

Runs that failed each scenario, of 15 per cell.
Scenario	Kind	Vague, off	Vague, on
a paid event sets the status	core	0	0
a paid event calls onPaid	core	0	0
a failed event sets the status	core	0	0
an unknown order has no status	core	0	0
a redelivered event ships only once	trap	0	0
two paid events ship only once	trap	0	0
a late, older paid does not undo a refund	trap	11	10
a late, older paid does not ship	trap	8	10
a late, older failure does not undo a payment	trap	0	0

Work out a billing datenext-billing100%100%100%100%

Runs that failed each scenario, of 15 per cell.
Scenario	Kind	Vague, off	Vague, on	Words, off	Words, on
one month on, mid-month	core	0	0	0	0
twelve months on	core	0	0	0	0
period zero is the anchor	core	0	0	0	0
across a year end	core	0	0	0	0
Jan 31 plus one is the end of February	trap	0	0	0	0
a leap year has a Feb 29	trap	0	0	0	0
no drift: Jan 31 plus two is Mar 31	trap	0	0	0	0
Aug 31 plus one is Sep 30	trap	0	0	0	0
the time of day survives	trap	0	0	0	0
the input date is not changed	trap	0	0	0	0

Build a search querysearch-query100%100%93%100%

Runs that failed each scenario, of 15 per cell.
Scenario	Kind	Words, off
finds by term	core	0
no filters finds everything	core	0
sorts by a column	core	1
respects a limit	core	0
a quote in the term is harmless	core	0
injection through the term fails	trap	0
sortBy cannot read another table	trap	0
sortBy cannot end the statement	trap	0
a bad limit cannot reach the SQL	trap	0
a search for 50% means a literal 50%	trap	0
an underscore means an underscore	trap	0

The 18 broken one-shot runs, one by one

Failed: empty weights: refused.
Stopped by: Types, Reviewer, neutral prompt, Reviewer, told to find a defect.
An empty file: it failed every scenario.
Stopped by: Tests, Mutation, Reviewer, told to find a defect.
Failed: hostname case is ignored.
Stopped by: Lint.
Failed: hostname case is ignored.
Stopped by: Lint, Reviewer, told to find a defect.
An empty file: it failed every scenario.
Stopped by: Tests, Mutation, Reviewer, neutral prompt, Reviewer, told to find a defect.
An empty file: it failed every scenario.
Stopped by: Tests, Mutation, Reviewer, told to find a defect.
Failed: a late, older paid does not undo a refund; a late, older paid does not ship.
Stopped by: Reviewer, neutral prompt, Reviewer, told to find a defect.
Failed: a late, older paid does not undo a refund; a late, older paid does not ship.
Stopped by: Tests, Mutation, Reviewer, neutral prompt, Reviewer, told to find a defect.
Failed: a late, older paid does not undo a refund; a late, older paid does not ship.
Stopped by: Reviewer, neutral prompt, Reviewer, told to find a defect.
Failed: a late, older paid does not undo a refund; a late, older paid does not ship.
Stopped by: Reviewer, neutral prompt, Reviewer, told to find a defect.
Failed: a late, older paid does not undo a refund; a late, older paid does not ship.
Stopped by: Tests, Mutation, Reviewer, neutral prompt, Reviewer, told to find a defect.
Failed: a late, older paid does not undo a refund; a late, older paid does not ship.
Stopped by: Reviewer, neutral prompt, Reviewer, told to find a defect.
Failed: a late, older paid does not undo a refund.
Stopped by: Reviewer, neutral prompt, Reviewer, told to find a defect.
Failed: a late, older paid does not undo a refund.
Stopped by: Reviewer, neutral prompt, Reviewer, told to find a defect.
Failed: a late, older paid does not undo a refund; a late, older paid does not ship.
Stopped by: Reviewer, neutral prompt, Reviewer, told to find a defect.
Failed: a late, older paid does not undo a refund; a late, older paid does not ship.
Stopped by: Reviewer, neutral prompt, Reviewer, told to find a defect.
Failed: a late, older paid does not undo a refund.
Stopped by: Reviewer, neutral prompt, Reviewer, told to find a defect.
Failed: sorts by a column.
Nothing stopped it.

Read the real attempts

Pick a task, a cell and a run, and read the code the model wrote and what each gate said about it, unedited. It starts on the task that decided the result.

Task

Brief and harness

Run

A hollow red dot on a run means it failed a hidden scenario, and a solid green dot means it passed them all.

Loading the recorded runs…

The words

Everything this page leans on has its own entry in the course, with an instrument where one helps.

Brief: What the agent was told. Changing it is one side of the design.
The right word: The claim under test, and the part this design can’t isolate.
Edge case: What most trap scenarios are.
Acceptance criteria: What the worded briefs add, and the reason they aren’t a clean test of a word.
Harness: Everything around the model. Here, the loop that hands failures back.
Quality gate: A check that can say no.
Type checker: The Types gate, run with tsc.
Strict mode: Where the extra index-access flag gets confused for part of strict.
Linter: The Lint gate, run with Biome.
False positives: What most stops in the gate table turned out to be.
Mutation testing: Breaking the code on purpose to see if the tests notice.
Mutation score: The share of those breaks the tests caught. The bar here was on this.
Holdout scenario: The test the agent never saw, and what this page calls the truth, with limits.
Test oracle: What decides right from wrong. Ours had a bug, and it has a blind spot.
Flaky test: The same input with different verdicts, as with the reviewer.
Goodhart’s law: Why gate scores rising under a harness that targets them says little.
Reviewer prompt: The wording that moved approvals so far.
AI reviewer: The second agent, reading code it can’t run.
Fresh eyes: Why that second agent started a new session.
Base rate: Why a gate that stops mostly good code can still look busy.
Confidence interval: The line under each bar in the headline chart.
Honest limits: What this page tries to do at the top and not in a footnote.
Webhook: What the task that decided the result was about.
At-least-once delivery: Why the same webhook can arrive twice.
Idempotency: The word the worded payment brief used.
Cross-site scripting: What a javascript: link in a comment is.
Output encoding: The rule the worded comment brief states.
Sandbox: Where the model’s code ran, with no network.
Model tier: Why one small model is a real limit.

The experiment

The result

What it can’t tell you

How it was run

Split a bill

Redirect after login

Save an uploaded file

Render a comment

Cache an async call

Handle payment webhooks

Work out a billing date

Build a search query

What we found

A mechanical gate can’t catch what nobody asked for

What the harness really changed

The gates are weak detectors here

The reviewer’s prompt decides what you hear

Our own holdout missed a real hole

What we got wrong

Our first task set was too easy

The first reviewer prompt was biased, by us

Our holdout had a false positive

Our workspaces slowed down our own runs

The broken pile isn’t what it sounds like

The worded briefs state the rules as well

Some agent-written tests really hang

What would change our mind

About the claim

A better experiment

Check it yourself

Every task, every scenario

Read the real attempts

The words

Before the agent

The agent

The harness

Everything it can get wrong

The operator

The workshop