Resiliency Patterns

Availability patterns (page 1) keep the system serving traffic through a failure. Resiliency patterns are how one call between two services behaves when the thing it depends on is slow or broken, so one failing dependency doesn't cascade into a bigger outage.

TL;DR — in 30 seconds:

• Retry with backoff — retry a failed call, but wait longer between each attempt (exponential backoff), so retries don't pile onto an already-struggling dependency.
• Circuit breaker — after enough failures, stop calling the failing dependency for a while and fail fast instead, giving it room to recover instead of hammering it with retries.
• Bulkhead — isolate resources (e.g. connection pools, threads) per dependency, so one dependency exhausting its pool can't also starve calls to a healthy one.
• Timeout — never wait indefinitely for a response; give up after a bounded time so a slow dependency can't hang the caller forever.

Pattern	Protects against	Mechanism
Retry with backoff	A transient failure that would succeed on a second try	Retry the call, waiting longer between each attempt, up to a capped number of attempts
Circuit breaker	A retry storm hammering an already-failing dependency	Stop calling it once a failure threshold is crossed; periodically test with a limited call to see if it recovered
Bulkhead	One slow dependency starving calls to a healthy one	Give each dependency its own resource pool (connections, threads) instead of one shared pool
Timeout	A hung call tying up the caller's own resources indefinitely	Give up and free the resource after a bounded maximum wait

1. Retry with backoff

sequenceDiagram
    participant C as Caller
    participant D as Dependency
    C->>D: request (attempt 1)
    D-->>C: fails
    Note over C: wait 1s
    C->>D: request (attempt 2)
    D-->>C: fails
    Note over C: wait 2s (backoff)
    C->>D: request (attempt 3)
    D-->>C: succeeds

• A transient failure (a dropped packet, a momentary blip) often succeeds on retry — but retrying immediately, at high volume, can turn a brief blip into a sustained overload on the dependency.
• Exponential backoff doubles (or otherwise increases) the wait between attempts, so retries space themselves out instead of piling on. A retry policy also needs a cap — a maximum number of attempts — so a permanently-failing dependency doesn't get retried forever.

2. Circuit breaker: stop calling a failing dependency

State	Behavior
Closed (normal)	Calls pass through; failures are counted
Open	Failure count crossed the threshold — calls fail fast immediately, no network call is even attempted
Half-open	After a cooldown, a limited number of calls are let through to test if the dependency has recovered

stateDiagram-v2
    state "Half-open" as HalfOpen
    [*] --> Closed
    Closed --> Open: failure count crosses threshold
    Open --> HalfOpen: cooldown elapses
    HalfOpen --> Closed: test calls succeed
    HalfOpen --> Open: a test call fails

• The point isn't just to protect the caller from wasting time on doomed calls — it's to stop battering an already-struggling dependency, giving it room to recover instead of drowning in retried traffic from every caller.
• This is the direct answer to a retry storm: retries without a circuit breaker can keep hammering a recovering dependency back down; the breaker's "open" state gives it a real cooldown window.

3. Bulkhead: isolate the blast radius

flowchart LR
    App["Application"] --> PoolA["Connection pool A<br/>(Dependency A)"]
    App --> PoolB["Connection pool B<br/>(Dependency B)"]
    PoolA -.->|"exhausted by A's slowness"| A["Dependency A (slow)"]
    PoolB --> B["Dependency B (healthy)"]

• Named for a ship's watertight bulkheads: one compartment flooding doesn't sink the whole ship.
• Applied to software: give each dependency its own pool of resources (connections, threads) instead of one shared pool for everything. If Dependency A goes slow and exhausts its own pool, calls to healthy Dependency B still have their own pool to draw from and keep working.
• Without this isolation, one slow dependency can quietly starve calls to every other dependency that happens to share the same resource pool.

4. Timeout: never wait forever

• Every outbound call needs a bounded maximum wait. Without one, a hung dependency can tie up the caller's resources (a thread, a connection) indefinitely, which is often what actually turns one slow dependency into a full outage of the caller.
• A timeout works together with retry-with-backoff (retry only after the timeout fires) and the circuit breaker (repeated timeouts are exactly the kind of failure that should trip the breaker open).

Common gotchas

• Retrying without a cap or backoff. Fix: unbounded, immediate retries are how a small blip becomes a self-inflicted overload — always cap attempts and back off between them.
• No timeout at all. Fix: a hung call with no timeout can exhaust the caller's own resources — every outbound call needs a bounded maximum wait.
• One shared resource pool for every dependency. Fix: that's exactly what a bulkhead prevents — split pools per dependency so one slow one can't starve the rest.
• Stacking retries at every layer that touches the call. Fix: as with a service mesh (module 22), retries at multiple layers can multiply load during an incident — decide which single layer owns retries for a given call path.

Check yourself — a dependency starts failing intermittently, and the caller retries every failed call immediately, with no cap. What goes wrong, and which two patterns from this page fix it?

Immediate, uncapped retries can turn a brief, partial failure into a full overload on the already- struggling dependency (a retry storm). Retry-with-backoff (space out and cap the retries) and a circuit breaker (stop calling it entirely once it's clearly failing, giving it room to recover) both address this.

Done when: you can explain what each of the four patterns protects against, and describe how retry, timeout, and circuit breaker work together on a single failing call.