Understanding Retry Storms in Distributed Systems

What is a Retry Storm?

A retry storm occurs when multiple clients repeatedly attempt to retry failed requests to a service. This can lead to overwhelming the service, which is already struggling, creating a cycle of failure.

The Failure Amplification Loop

• Failure: A service fails to respond.
• Retry: Clients retry the request.
• More Load: The retries generate additional traffic.
• More Failure: The service becomes even more overloaded, leading to further failures.
• More Retries: This cycle continues, amplifying the initial failure.

Why Retry Storms Happen

• Multiple Services: Many services may be retrying the same failing dependency.
• Too Many Attempts: Excessive retry attempts can flood the service.
• Synchronized Retries: Clients retrying on the same schedule can worsen the load.
• Capacity Consumption: Retries consume resources while the service is already under stress.

Ways to Control Retry Storms

1. Bounded Retries: Set a maximum number of retry attempts.
2. Exponential Backoff: Gradually increase the wait time between retries.
3. Jitter: Introduce randomness in retry timing to distribute traffic.
4. Circuit Breaker: Temporarily stop calls to a failing service to allow recovery.

Important Considerations

• Not Every Failure Should Be Retried: Retries are beneficial for transient failures but not for validation or authentication errors.
• Idempotency: Use idempotent operations or idempotency keys to avoid duplicate effects from retries.

Conclusion

When implementing retries, always consider the potential load on the system. Ensure retries, timeouts, circuit breakers, and idempotency work together to maintain system stability.