Serguey Asael Shinder: A retry without a budget converts a slow dependency into an outage
Retries are added one layer at a time, by different people, each of whom is right. The HTTP client retries idempotent requests three times. The service wrapper around it retries the operation three times, because the client sometimes gives up. The job that calls the service retries three times, because jobs should be resilient. Nobody has done anything unreasonable and a single user action is now twenty-seven requests.
That multiplication is invisible while the dependency is healthy, because the first attempt succeeds and the other twenty-six never happen. It becomes visible at precisely the moment the dependency is struggling: latency rises, the first attempt times out, and the system responds to a loaded service by multiplying its load by twenty-seven. The dependency was slow. Now it is down, and it stays down after the original cause has passed, because the retry traffic is enough on its own.
Exponential backoff with jitter does not fix this. It is necessary and it is not sufficient. Backoff spreads the retries in time, which stops the synchronised thundering herd; it does not reduce the total number of calls, and it does not compose across layers. A stack of three backing-off retriers still sends twenty-seven requests, just politely.
What is missing is a ceiling on retries as a fraction of traffic, held in one place. Count the successful calls, count the retries, and stop retrying when retries exceed some small share — a tenth is a common starting point. Under normal conditions the budget is never touched, so nothing changes. Under failure it caps the amplification at 1.1× instead of 27×, and the dependency gets a chance to recover.
The organisational half of the fix matters as much as the arithmetic:
Decide which layer owns the retry and remove it from the others. Usually that is the layer that knows whether the operation is idempotent and what the user is waiting for. Retrying below that point is guessing; retrying above it, after the lower layer already tried, is multiplication.
Make the total observable. Emit the attempt number on every call and count attempts, not operations. If your dashboards show requests per second without distinguishing first attempts from retries, you cannot see the amplification building — and during the incident, you will be looking at a graph that says the dependency is being hammered without any indication that it is your own system doing it.
Ask what happens when retries stop. A budget that is exhausted must fail fast with a clear error, not queue. Otherwise the amplification moves from the network to your own memory, and the failure arrives later wearing a different costume.