Chapter 16: Shipping and running

Alert

Also called Page, Burn-rate alert, Alert fatigue

A signal that tells a person to act now. A good one fires when users are being hurt and someone can do something about it. A burn-rate alert does this by asking how fast the error budget is going.

An alert that cries wolf gets ignored, including on the day it is right, and one set too high misses the slow leak. Asking an agent to "add monitoring" can produce alerts by the dozen, so ask for few, and ask what each would have caught.

Say it to your agent

“Replace these 14 threshold alerts with a burn-rate alert on the checkout SLO: page at 14.4x over 1 hour and 5 minutes, page at 6x over 6 hours and 30 minutes, ticket at 1x over 3 days and 6 hours. Replay last month, and show me how many pages we would have had and what each one caught.”

The workshop

Name a word, or describe what you mean in everyday language.