Anatomy of a retry storm

Exercise context

This is a hypothetical incident review, not a report of a production incident. It examines how retries can turn a small dependency failure into a wider outage.

What happened

A downstream service slowed down. Callers timed out and retried immediately. Each layer had its own retry policy, multiplying traffic exactly when the dependency had the least spare capacity.

Illustrative timeline

Relative time	Observation
T+0	Downstream latency increases.
T+2 min	Immediate retries increase concurrency.
T+5 min	Worker pools saturate and queues grow.
T+8 min	Responders reduce retries and shed optional work.
T+15 min	Queues drain; successful requests recover.

Contributing conditions

The system had per-request timeouts but no end-to-end deadline. Retries were not bounded by a shared budget. Identical retry intervals synchronized callers.

Corrective actions

1. Assign one layer ownership of retries and cap attempts within an end-to-end deadline.
2. Use exponential backoff with jitter and honor server retry guidance.
3. Bound concurrency and queue length; reject excess work early.
4. Require idempotency keys for retried mutations.
5. Exercise dependency slowdown under load and confirm the service degrades predictably.

Verification

Repeat the fault injection with the new policy. Compare attempted requests with original requests, queue age, and completed work. A successful mitigation bounds amplification while preserving useful throughput.