Chapter 14: Data and failure

Duplicates are half of that trap. The other half is the one a vague brief almost never gets right.

Out-of-order delivery

Messages can arrive in a different order from the one they were sent in, because a retry of an old one can land after a newer one. If your code treats the latest to arrive as the latest to happen, a late "paid" event can undo a refund.

This is the behaviour nobody asks for, so nobody gets it. In our experiment, an agent told to handle payment webhooks passed the hidden scenarios 27% of the time, and 33% with types, lint, tests and mutation testing all behind it. Told that events arrive at least once and out of order, it passed 100%. One small model and one-function tasks, so read the shape and not the size.

Say it to your agent

“Webhooks arrive at least once and out of order. Dedupe by event id. Don't trust arrival order: on each event, fetch the current state of the payment from the provider's API, and only act on that. Ship exactly once per order, never after a refund. Test all six orderings of paid, refunded and a repeat of paid.”

The workshop

Name a word, or describe what you mean in everyday language.