Shipping and running

Where your software runs for real, with real users, real data and real money. Every other copy of it exists to prepare for this one.

It is the one place where a mistake reaches people, and the only place that can tell you what was true. An agent should have no way in unless you gave it one on purpose.

One complete place your software can run, with its own address, its own data and its own settings. Development on your machine, staging and production are the usual three.

The more two environments differ, the less a pass in one tells you about the other. And an agent that can't tell which one it is in will sooner or later run the right command in the wrong place.

An environment that copies production as closely as you can afford, where a release runs before real users get it.

It catches what shows up only when the pieces run together: a bad setting, a missing variable, a migration that won't run. It can't show what only real traffic and real data reveal, and a pass there can feel like more proof than it is.

The step that turns source code into the thing that actually runs: compiling, bundling, and copying everything into one output.

It is the first time all of the code has to fit together. A build also pulls things in from outside, such as dependencies, so it is where a change nobody made can arrive.

A three-part version number, like 2.4.1, that makes a promise. The last number means a bug fix, the middle one a new feature that breaks nothing, and the first means something may break.

It lets tools take safe updates on their own. But it is a promise made by a person, and nothing enforces it, so a patch can still break you.

Fixing the exact version of something you depend on, by number or by hash, so that it changes only when you decide. A range is a rule. A pin is this one and no other.

Without a pin, a build can change or break on a day nobody touched the code. But a pin holds still in both directions: a security fix doesn't arrive until someone moves it. So pin, and have something move the pins for you, with tests in front.

A list of every component in your software, including what your dependencies depend on, with exact versions. An ingredients label, generated from what you really built.

When an advisory lands, the first question is "do we have this?", and the answer is often in a package nobody here chose. With a list it takes seconds. An agent can add a dependency in one line of a diff, so the list is worth more than anyone's memory.

The finished output of a build, stored with a version so that you can move exactly that thing between environments. A bundle, a zip, a container image.

If you build again for production, what you tested isn't what you ship. Build once and move the same artefact along, and a pass in staging says something real about production.

A way to package a program with all the files it needs and run it as an isolated process, so that it behaves the same on your laptop and on a server.

It makes "works on my machine" a smaller problem, because the machine's contents travel with the program. It also gives an agent a place to run commands where a mistake is more likely to stay inside.

You hand a provider your code, and they run it on their machines when requests arrive. There are still servers. They just aren't yours to manage.

There is nothing of your own to patch or keep running. The price is limits you don't choose, code that can't count on anything staying in memory between requests, and a dependence on one provider.

Describing your servers, databases and networks in files kept in the repo, and letting a tool make the real world match them, in place of clicking around a console.

Changes become reviewable and repeatable. It also means one wrong command can change everything at once. Agents are good at writing these files. The danger is in letting them run the result.

Whatever changes between deploys belongs outside the code: addresses, keys, passwords. The same artefact can then run anywhere, reading its settings from wherever it is running.

Hardcoded config means rebuilding to move, and a key in the repo is a key in every clone, every log and every agent's context. Deleting it later doesn't help, because the history remembers: rotate it.

Putting a new version where it can run. The aim is to make it automatic, small and boring, so that it can happen often.

A manual step is one that a tired person or an eager agent will skip. So deploying should be a script, the same every time, and what matters most is what stands in front of that script.

Keeping your software in a state where it could go to production at any moment, so that releasing is a decision and not a project.

Small changes are easier to check, and easier to take back. This matters more with agents, because they produce changes faster, and saving those up into one big release piles up risk that nobody can review.

Everyone merges small changes into one main branch, at least daily, and nobody keeps a branch going for long. Unfinished work stays hidden behind flags.

The longer a branch lives, the more it and main both change, and conflicts grow faster than the time does. Parallel agents sharpen this, because each one is a branch.

A condition in your code that turns something on or off, for some or all users, without a deploy. The code ships switched off, and you switch it on when you are ready.

If a release goes wrong, turning it off takes seconds and not a rollback. A percentage lets you try it on a few people first. The cost is real: each flag is a second version of your code, which has to be tested and later removed.

Sending a small share of real traffic to the new version, comparing it with the old one, and only then sending everyone. If it is bad, few people met it.

Some problems appear only under real traffic, and a canary limits how many people find them. It only helps if you can tell in time, which takes enough requests and a real comparison. A tiny canary on a quiet site tells you nothing.

Two identical production environments. One serves users while you deploy to the other and check it. Then you switch the traffic across, and the old one stays ready to switch back to.

Going back becomes flipping a switch, and nobody meets a half-finished deploy. You pay for two environments while you keep both, and there is usually one database, which both versions have to work with.

Going back to the previous version after a bad release. Rolling forward is the other choice: shipping a new version with the fix. Rolling back is usually faster, so you stop the harm first and fix it calmly.

How quickly you can undo decides how much risk each change carries. The trap is data. Code goes back and data doesn't, so one migration can quietly remove your way back, and you find out on the worst possible day.

How well you can work out what a running system is doing from the outside, using what it emits, including for problems you didn't predict.

It decides whether you find out from your signals or from your users. It also changes what an agent can do: if it can read logs and traces, it can check its own work, and you aren't the one relaying symptoms.

Three kinds of signal. A log is a timestamped event. A metric is a number over time, such as requests a second. A trace follows one request through every call it makes.

They answer different questions: a metric says something is wrong, a trace says where, and a log says what happened there. Give an agent only one of them, and it will guess at the rest.

Writing each log entry as named fields, usually JSON, and not as a sentence. A query can then ask for exact values, without guessing at the wording.

In an incident the question is "which users, which release?". Against free text, every answer is a guess at how someone phrased it, and it is wrong without saying so. An agent reading the logs does far better with fields it can query than with prose it has to interpret.

An SLI is a measure of how your service is doing, like the share of requests that succeed in time. An SLO is the target you set for that measure, like 99.9% over 30 days.

Without an agreed number, "is it reliable enough?" is an argument about feelings, and perfect isn't on offer. With one, you know when to keep building and when to stop and fix things. Choose what counts as good with care, because that choice can hide a real problem.

The failure your SLO allows. At 99.9% over 30 days, that is 43.2 minutes of total outage. While there is budget left you ship, and when it is gone you stop and work on reliability.

It turns an argument into arithmetic, and it applies directly to agents. If they let you ship faster than your budget can absorb, the budget says slow down, whatever the velocity chart says.

A signal that tells a person to act now. A good one fires when users are being hurt and someone can do something about it. A burn-rate alert does this by asking how fast the error budget is going.

An alert that cries wolf gets ignored, including on the day it is right, and one set too high misses the slow leak. Asking an agent to "add monitoring" can produce alerts by the dozen, so ask for few, and ask what each would have caught.

The rota that decides who answers when something breaks, at any hour. That person has to respond quickly, and needs the access and the knowledge to act.

An agent can write the code and it can't be accountable at 3am. Whoever is on call has to understand what shipped, and a codebase nobody has read is a bad thing to be woken up by.

An unplanned event that is hurting users, or about to, and needs people to coordinate. The work has an order: stop the harm first, then understand why.

The instinct is to hunt for the root cause while users keep failing, and an agent's instinct is to start changing things. Someone has to hold the whole picture and decide what gets tried, because a change made during an incident needs the most care of all.

Written steps for a known situation, which someone tired can follow: what you see, what to check, the exact commands, and when to call for help.

Thinking it through while calm beats thinking it through in an emergency. A runbook in the repo is also something an agent can read and follow, and that gives you a cheap way to find out whether it has gone stale.

How long it takes to get back to healthy after something breaks. DORA's current version, failed deployment recovery time, counts only failures caused by a change.

You can't stop every failure, and you can decide how fast you recover. It is why rollbacks and flags pay for themselves. Take care with the mean, because one bad day can dominate it, so report the median as well.

A written record of an incident: what happened, what it cost, why, and what changes so that it doesn't happen again. Blameless means it asks how the system allowed this, and doesn't stop at who.

People who fear blame hide what they did, and then nobody learns. With agents there is a new version of this: "the agent did it" is blame with a new target. Who gave it that access, and which check let it through?

Five measures of software delivery from Google's DORA research: change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate. Speed and stability, side by side.

They stop you fooling yourself with lines written or pull requests merged. Speed with no stability number beside it hides what it cost, and that is the case you need to catch when agents raise the rate.

The share of deployments that need fixing straight away, by a rollback or a hotfix. Five bad deploys out of fifty is 10%.

It is the number that shows whether going faster is costing you. It is also easy to fool: the definition of failed moves it a lot, and adding harmless deploys lowers it with the same number of bad ones.

DORA's description of what AI does to a software team: it makes strong teams stronger, and makes the problems of struggling ones worse. What decides which is what was already around the tool.

So buying the tool doesn't settle what you get. This chapter is the surrounding part: small changes, a way back, real signals and an agreed budget. Without them, a faster rate of change can reach your users as more failures.