Chapter 14: Data and failure

Circuit breaker

A switch around a call to something that might fail. Count the failures, and past a limit stop calling it for a while and fail at once. After that, let one trial call through. If it works, close again. Named after the thing in your fuse box.

A timeout means that every request waits the full time before failing, and when the thing you depend on is down, that's all your capacity waiting. A breaker turns slow failure into fast failure. Agents won't add one unasked, and when they do it deserves a test, since a breaker that's never been open is one you're only hoping works.

Say it to your agent

“Wrap calls to the recommendations service in a circuit breaker: open after 5 failures in 10 seconds, fail at once with the cached list for 30 seconds, then allow one trial call. Log each state change, and add a test that forces it open and checks the page still loads.”

The workshop

Name a word, or describe what you mean in everyday language.