In distributed systems, failures aren't the exception — they're routine. The network drops, a service gets slow, a call times out. Resilience is designing so that these inevitable failures don't take the whole system down. For a senior, mastering idempotency, retries and circuit breakers is essential — and a frequent interview topic.

Failures happen

The premise that changes everything: at scale, something is always failing. A disk, an instance, a network link. The goal isn't to avoid 100% of failures (impossible), but to contain the impact and recover automatically. Resilience is the difference between "a slow service" and "the entire system down".

Idempotency: the basis of safe retries

An operation is idempotent when running it several times has the same effect as running it once. This is crucial because, when a request fails, you want to be able to try again — but you can only do so safely if the retry doesn't cause damage. Imagine charging a customer: if the response got lost but the charge went through, a blind retry would charge twice.

The solution is the idempotency key: the client sends a unique identifier for the operation, and the server guarantees it processes each key only once:

POST /charges
Idempotency-Key: 7f3a-9c21

// server:
if alreadyProcessed("7f3a-9c21"):
    return savedResult("7f3a-9c21")   // doesn't charge again
result = processCharge(...)
save("7f3a-9c21", result)
return result

Retry with backoff and jitter

When something fails transiently, retrying makes sense. But retrying immediately and in a loop can make things worse — you hammer a service that's already overloaded. That's why exponential backoff is used: each attempt waits longer than the previous one (1s, 2s, 4s, 8s). And jitter (a random variation) is added to prevent a thousand clients from retrying at the exact same instant, creating a synchronized wave.

wait = min(base * 2^attempt, max) + random(0, jitter)
// attempt 0: ~1s    attempt 1: ~2s
// attempt 2: ~4s    attempt 3: ~8s

And always with a timeout: never wait indefinitely for a response. Without a timeout, a slow dependency ties up your threads and the problem spreads. Also remember that retrying is only safe on idempotent operations.

Circuit breaker

If a service is clearly down, continuing to try (even with backoff) only wastes resources and delays responses. The circuit breaker works like an electrical breaker: after many consecutive failures, it "opens" and starts failing fast, without even attempting the call — giving the service time to recover.

   success                 many failures
  +---------+  failures   +---------+
  | CLOSED  |  -------->  |  OPEN   |  (fails fast,
  +---------+             +---------+   doesn't call)
       ^                       |
       | success               | after timeout
       |                       v
       |  failure        +-------------+
       +---------------- |  HALF-OPEN  |  (tests 1 call)
                         +-------------+

The three states: closed (normal, lets calls through), open (fails fast after the error threshold), and half-open (after a while, lets one test call through — if it succeeds, it closes; if it fails, it reopens).

Retry says "try again, calmly". Circuit breaker says "stop trying, it's clearly broken". Idempotency is what makes the first one safe. The three work together.

Isolation with bulkhead

Finally, the bulkhead pattern (in one sentence): just as a ship is divided into watertight compartments so one hole doesn't sink the whole thing, you isolate resources — separate connection or thread pools per dependency — so that the failure of one part doesn't consume the resources of the others.

Conclusion

Resilience is designing on the assumption that everything will fail. Make operations idempotent to allow safe retries, use exponential backoff with jitter and timeouts so you don't worsen the load, and protect yourself with circuit breakers and bulkheads to contain the damage. That combination is what keeps distributed systems standing in the real world.