Performance and cost

Before changing anything to make it faster, time it, find the slow part and write the number down. Then time it the same way after the change. Without both numbers, nobody can say the change helped.

An agent reads code and spots what looks slow, and what looks slow often isn't what is slow. It reports the improvement in the same confident voice either way. So make the number part of the work you ask for, because it is the one thing it can't produce by reasoning.

Latency is how long one thing takes. Throughput is how many things get done per second. They are separate numbers, and improving one can hurt the other: waiting to send items in a batch gets more done per second, and makes each item wait longer.

Tell an agent to make something faster and it has to guess which you meant. Batching can be a real win for a nightly import and a real slowdown for the person waiting on one request, and both get described as a speed-up. Name the number you want to move.

Sort every timing from fastest to slowest. The 95th percentile, or p95, is the value that 95% of them are at or under. p50 is the middle one. A percentile is a time someone really waited, which an average need not be.

An agent asked if a change helped will often quote an average, because an average is what a quick script prints. A few very slow requests drag it up to a value nobody experienced, and a real regression for 5% of people can barely move it. Ask for the percentiles.

The slowest few requests, far along the end of the distribution. One in a hundred sounds rare until a single page makes dozens of requests and waits for them all, at which point a lot of people meet that one.

Chances stack up: with ten requests per page, about one page view in ten hits a p99. An agent run does the same thing, because a session making 40 calls has a one in three chance of meeting a 1-in-100 slow one. So a slow call that looks rare per request is routine per session, and a retry or a timeout has to expect it.

A repeatable test that times a piece of code many times under the same conditions, so two versions can be compared. One timing isn't a benchmark, because the same code gives a different number on each run.

An agent asked to prove its change helped can write a benchmark that shows what it hopes for: one run, a tiny input, the old code cold and the new code warm. It need not mean any harm by that. So read the benchmark as well as the result, and ask how many runs it took and whether the ranges overlap.

A tool that watches a program run and reports how much time each function took. Self time is what the function spent on its own. Total time also includes everything it called. The answer often surprises people.

This is the most useful thing you can hand an agent for speed work. Without a profile it optimises what looks suspicious. With one it starts from a list of where the time went, and you can check whether its change moved the top line.

A picture of a profile. Each bar is a function, sitting on top of the one that called it, and the wider the bar, the more time it took. You look for the wide bars at the top: they are doing the work themselves, with nothing above them to blame.

A flame graph lets you check an agent's story at a glance. If it says the slowness is in the formatter and the widest bar is somewhere else, you can see that for yourself without reading a line of code, and without taking its word for it.

A way of saying how a program's work grows as its input grows. With O(n), twice the data takes about twice as long. With O(n²), twice the data takes about four times as long. It says nothing about how long any one run takes.

An agent's tests run on ten rows, and at ten rows everything is instant. A loop inside a loop can pass every check and only hurt a year later, when the table has grown. It is also a claim agents love to make, so ask to see it measured at two or three sizes.

The one stage that limits how much the whole line can do. Speeding up any other stage changes nothing at the end, and just makes a bigger pile in front of the narrow part.

Agents make writing code far faster, so for many teams the bottleneck has moved to reading it. Generating twice as many pull requests against the same review capacity gets you a longer queue, and pressure to approve without reading. Find the narrow stage before you buy speed anywhere else.

If part of a job has to be done one step at a time, adding more workers can only speed up the rest. If 5% must run alone, you can never go more than 20 times faster, however many workers you add.

This is the sum behind running many agents at once. Ten agents writing in parallel still queue for one person to read, one merge to main and one deploy. That single-file part sets the limit, and past a point another agent mostly adds cost.

Making code faster, and usually harder to read, before you know that it matters. The cost is real and paid at once in complexity. The gain is a guess until someone has measured that this code is where the time goes.

An agent can do this at no effort to itself, so the old brake has gone: a person got tired of writing a clever cache, and an agent doesn't. You then own the extra code. But the same line is often used to excuse not measuring at all, and it doesn't say that.

Keeping a copy of an answer somewhere quick, so you can skip the slow work next time. A hit is when the copy is there. A miss means doing the work anyway. A cache is small, so something has to decide what to throw out, often the least recently used.

A cache is an easy diff that makes a benchmark look great, so it is a favourite agent fix, and it swaps a speed problem for a correctness problem. Check the hit rate on real traffic first: a cache that rarely hits costs memory and complexity, and gives nothing back.

Getting rid of a stored copy when the real answer changes. Do it too late and people read old data, which is called a stale read. A time limit on each copy, or TTL, is the blunt way: it is simple, and wrong until it runs out.

Tests rarely catch this, because they read a value once and never check again after a write. So an agent's cache can pass everything and still show someone the old price for an hour. Whoever adds a cache has to say what happens on every write path, including the ones they didn't read.

When work arrives at random, it sometimes bunches up and has to wait in line. Utilisation is how busy the server is. As it nears 100%, the line grows faster and faster: at 50% a request takes about twice its work time, at 90% ten times, at 99% a hundred times.

Running everything at 95% looks efficient, and the people waiting pay for it. The same sum applies to you: an agent that hands a reviewer more than they can read doesn't get faster reviews, only a longer line. And a word to keep straight: this is where chapter 9's word has its software meaning.

Sending a system a realistic amount of made-up traffic, on purpose, to see how it behaves before real people do. A stress test goes past the expected load, to find where it breaks.

An agent's checks run one request at a time, so everything it builds looks fast. Load is what finds the queue and the exhausted pool. The test can also lie: a tool that waits for each reply before sending the next quietly stops testing at exactly the moment things go wrong.

Concurrency means juggling many tasks, swapping between them while each waits for something. Parallelism means truly doing several at the same instant, on separate cores, the independent workers inside a CPU. Waiting can be juggled. Crunching numbers needs more cores.

An agent will happily wrap heavy work in Promise.all and call it parallel. That speeds up work that waits, such as network calls, and does nothing for work that computes, because in JavaScript it all still takes turns on one thread. Ask which kind of work it is, then time it.

In a browser or Node, a single line of work, called a thread, takes jobs off a list and finishes each before starting the next. While a job runs, nothing else can: no clicks, no animation, no other requests. A job over 50 ms in a browser is called a long task.

An agent's code works on its small test data and passes everything, and then it sorts 200,000 rows inside a click handler and the page freezes. No test notices a freeze. In Node it is worse: one slow job stalls every other person's request on that server.

A fixed set of open database connections that requests borrow and hand back. Opening a new one each time is slow, so they are reused. If all are in use, a request waits. If code forgets to hand one back, the pool slowly runs out.

This one bites agents because everything works in development, where there is one user. A missing release only shows up under load. A pool per serverless instance can multiply past what the database allows. And the instinct to fix it with a bigger pool can make it worse.

Memory the program no longer needs, and can't release because something still points at it: a list that only grows, a listener never removed. Each bit is small, so it passes every test and takes days to bring a server down.

An agent's session ends long before a leak shows. It adds a cache with no limit or a listener with no cleanup, the tests pass, and nothing fails until your server has been up for a week. Restarting hides it, so it can run for months. You have to go looking.

The part of a runtime that finds memory nothing can reach any more and frees it, so you don't have to. It costs work, and some of that work can pause your program. Those pauses are one reason a fast function sometimes isn't.

Most of the time you can ignore it, and an agent should too. It matters when you see a tail with no obvious cause and code that creates a lot of short-lived objects in a hot loop. Check with a profile first: rewriting code to avoid allocation on a hunch makes it worse to read and may not help.

Three scores for how a page feels. LCP: how soon the main content appears. INP: how soon the page responds when you tap. CLS: how much things jump around. Good means LCP within 2.5 seconds, INP within 200 ms and CLS at 0.1 or less.

An agent can't feel a page. It sees that the tests pass, and it may never load the page on a slow phone. These numbers put what a person feels into a form it can read and act on, and be held to. A fast laptop on good wifi will flatter all three, so look at what real visitors got.

How much code a page makes the browser download. A browser then has to unpack it, read it and run it, so on a cheap phone a big bundle costs time long after the download has finished.

Adding a package is one line for an agent, and it won't notice it just added 200 KB for one date format. Nothing fails, because each addition looks harmless, and no one reviewing the diff sees the size at all. You need something to fail when it grows.

Not loading something until it is needed: an image as it nears the screen, a chart's code when someone opens the chart. The first view gets lighter, and anyone who never scrolls that far never pays for it.

It's an easy thing to ask an agent for, and a common way to do harm. Told to lazy-load the images, it may well do every one of them, including the big one at the top, and that makes the main content appear later, which is the number you were trying to improve.

Sending an image no bigger than it will be shown, in a format that compresses well, with its size stated in the page. Images are often the largest thing a page downloads, so this can be the cheapest big win going.

An agent asked to add a picture adds the file you gave it: a 4000-pixel photo, in a box 400 wide. Nothing complains, and on a fast connection nobody notices. So tell it the display size and the format as part of the job, because neither shows up in a test.

A limit you set in advance on something like bundle size or p95 latency, with a check that fails when it is broken. Speed then stops being something you remember to care about, and becomes something that says no.

This is backpressure for speed. Speed tends to get worse a little at a time, each change too small to notice, and an agent adds them faster than anyone can watch. With a budget in CI the agent gets refused in its own loop, with a number, and has to deal with it.

What it really costs to get one piece of work finished: every try, every check and the failures you paid for and threw away. The price per token is only a price list. This is the number you can compare.

Comparing models on price per token is like comparing cars by the cost of a litre of fuel. A cheap model that takes three tries can lose to a dear one that takes one, and a wrong result that nobody catches can cost more than all of them. Measure it on your own tasks.

Deciding what is worth putting in front of the model. Every turn sends the whole conversation again, so a big log read early isn't paid for once. You pay for it again on each turn until the session ends.

The agent decides what to read, and it has no reason to care what that costs you. Long test output, whole files and old conversation all ride along on every later turn. Trimming these saves money, and leaves room for the code. Chapter 7 is about what that does to quality. This is the bill.

Sending each job to a model that fits it, instead of one model for everything: a cheap one for searching and summaries, an expensive one for planning and hard bugs. The router can be a rule, a setting or a person.

The prices differ by up to a hundred times, so on tokens alone routing nearly always wins. The danger is a cheap model that fails without saying so. A failure you see costs one retry. A failure that passes for success costs whatever it breaks later. Route freely where a check can catch it.

The share of a request's input tokens that came from the prompt cache. Because an agent resends everything each turn, a high rate can cut the input bill several times over, and a low one can quietly cost more than having no cache at all.

One changing byte near the top of the prompt can send this to zero, and nothing fails. The agent works just as well, and you pay more for it. So this is a number to watch: read it from the usage fields, and treat a fall as you would treat a failing check.

Working out the whole bill from its parts: people, tasks per day, cost per task. Once you have those three, you can tell what would double it, what a cut would save, and whether the work was worth what it cost.

Agent spending grows quietly and spreads across people and machines. A loop running overnight costs the same whether anyone looks or not. If you know your cost per task, a surprising bill becomes a question you can answer. It also gives you a real answer when someone asks if this is worth it.