Skip to content

Crisis & Incident Response

Something is broken in production. What you do in the first five minutes matters more than being clever — and it's a different skill than debugging on your own time.

When this matters

The moment you know (or strongly suspect) something is down or degraded for real users — an alert fires, an error rate spikes, someone reports the app is broken. Also useful for smaller "this is on fire" moments: a failed deploy, a broken pipeline, a stuck process nobody can explain.

The practice

Detect, don't debate. A fast, dumb check that reliably fires beats a slow, clever one that doesn't. If a signal crosses a real threshold — errors spiking, a service unreachable, a report from an actual user — treat it as real until you've confirmed otherwise. Don't spend the first five minutes arguing about whether it's really broken.

Triage by severity before you do anything else. Classify what you're looking at: is this full outage or data loss (highest — drop everything), a degraded-but-working state (middle — urgent but not all-hands), or cosmetic (log it, move on)? Severity decides how fast you move and who else needs to know right now.

Mitigate first — restore before you understand. Your first job is getting the system back to a good state, not finding out why it broke. Roll back the bad deploy, restart the stuck process, fail over to the backup — whatever gets users back to green fastest. Mitigation is not the fix. It buys you time to diagnose properly instead of diagnosing under pressure with things still on fire.

Keep people informed on a cadence, even with no news. The moment you're investigating something real, say so — and keep saying so on a short, predictable cadence ("still investigating, next update in 15 min") rather than going quiet until you have The Answer. Silence during an incident reads as "nobody's on it," even when someone is. This is the same discipline as the daily status update, compressed to incident speed (see Communication & Status Updates).

Diagnose only once stable. Root-cause the problem after the bleeding has stopped, not during. Rushing to a fix while the system is still unstable usually makes things worse or muddies what actually happened.

Escalate in order, not at random. Try what's in front of you first — check the obvious cause, read the relevant runbook or docs, see if it's a known failure mode. If it's genuinely beyond what you can fix or diagnose alone, escalate to whoever owns the system with a crisp statement of what's broken and what you've tried (see Reporting Blockers & Asking for Help). Climbing a defined ladder — self-check, docs, owner, full incident — beats randomly pinging people until someone answers.

Run a blameless postmortem afterward. Once resolved, write down what happened, the timeline, and the systemic contributing factors — never "X was careless" or "Y should have known better." People operate inside the systems and information they had at the time; blame just teaches people to hide the next incident instead of surfacing it. The postmortem's job is to produce a durable guard — a monitor, a test, a process change — that makes the same failure detectable or impossible next time. An incident with no new guard afterward is only half-handled.

Checklist

  • Did I confirm the signal was real instead of arguing about it first?
  • Did I classify severity before taking any other action?
  • Did I restore service before trying to understand root cause?
  • Did I send a status update as soon as I knew something — even "still investigating"?
  • Did I keep updating on a predictable cadence instead of going quiet?
  • Did I diagnose only after the system was stable again?
  • Did the postmortem name a systemic cause, not a person?
  • Did I walk away with one concrete, durable guard — not just a one-off manual fix?

Template

Incident status update (post this on your cadence — every 10–15 min for a real outage):

INCIDENT: <what's broken, one line>
Severity: <sev1 / sev2 / sev3>
Status: <investigating / mitigating / mitigated / resolved>
Impact: <who or what is affected, and since when>
Next update: <time>

Postmortem skeleton (fill in once resolved and stable):

## What happened
<timeline: detected at X, mitigated at Y, resolved at Z>

## Impact
<who/what was affected, and for how long>

## Root cause
<the systemic cause — a process/design gap, not a person>

## What went well / what slowed us down

## Durable guard
<the concrete change — monitor, test, runbook step — that makes this
detectable or impossible next time>

Adapted from

  • SE's incident-response skill — the detect → triage by severity → mitigate first → diagnose → blameless-postmortem loop, and its anti-patterns: diagnosing before mitigating, blame in the postmortem, and a one-off manual fix with no durable guard.
  • role-runtime/docs/book/12-run-and-recover.md's escalation ladder — a real example of climbing a defined order instead of escalating at random or all at once: check the traces first, then hand-clear stuck state, then a full halt, then (last resort) revert to the last known-good state. This guide generalizes that shape — self-check, docs, owner, full incident — for a reader outside SE's own recovery tooling, not its specific steps.
  • The capstone module's Operate gate question — every SE joiner is eventually asked "it's deployed and broken at 2am, walk me through your first five minutes"; this guide is the practice for that answer.

Practice it

Ask the onboarding-tutor: "Walk me through my first five minutes if <a system I work on> went down right now — quiz me on detect, triage, and mitigate-first before I ever hit a real one."