Delivery guarantees
Also called At-least-once, At-most-once, Exactly-once
What a message system promises. At most once: it may be lost, and it never repeats. At least once: it won't be lost, and it may arrive twice. Most systems you use give you at least once, so your code has to cope with seeing the same thing again.
Webhook handlers and queue workers are written by agents as if each message came once, in order, because that's how a test sends them. In production that assumption is simply false, and it's written in the docs of the service you're using, which the agent didn't read. This is the trap in our experiment.
“Stripe delivers webhooks at least once, so this handler must be idempotent. Record each event id under a unique constraint in the same transaction as the work, and skip ids already there. Add a test that delivers every event twice and asserts we ship once.”