WSS
On this page
Reliability & Incident ResponseRecommendedUpdated

Blameless postmortems

The practice of investigating outages without assigning human fault, focusing instead on the systemic vulnerabilities that allowed the failure to occur.

Reference Card

  • Origin/Prior Art: Aviation safety (NTSB), Healthcare incident reporting.
  • Related Practices: Root Cause Analysis, 5 Whys, Psychological Safety.
  • Key Concepts: Systemic failure, Action items, Timeline construction.
  • Primary Failure Modes: Witch hunts, Action item rot, Shallow root causes.

Introduction

A postmortem is a written record of an incident, its impact, the actions taken to mitigate it, and the root cause.

A blameless postmortem is a cultural imperative that dictates that individuals are never the root cause of an incident. If an engineer accidentally drops a production database table, the blameless postmortem does not conclude that “the engineer made a mistake and needs to be more careful.” Instead, it assumes the engineer was acting logically based on the tools and information they had, and asks: “Why did the system allow a single command to drop a production table without a secondary approval or a dry-run validation?”

Mental Model

Imagine a driver crashes their car into a wall on a dark, winding mountain road.

A blame-oriented culture fires the driver and tells the next driver to “be more careful.” The next driver inevitably crashes in the exact same spot because the road is inherently dangerous.

A blameless culture assumes the driver was trying their best not to crash. It investigates the road. It discovers there were no streetlights, no guardrails, and the warning sign was obscured by trees. The action items are to install streetlights, build a guardrail, and trim the trees. The systemic environment is fixed, ensuring no future driver can make the same mistake.

Operational Pattern

The postmortem process must be formalized and triggered automatically after any significant incident (e.g., any Sev-1 or Sev-2 outage).

  1. Information Gathering: Within 48 hours of the incident, while memories are fresh, the Scribe (from the Incident Command System) complies the raw timeline of events, Slack transcripts, and metric graphs.
  2. The Document: A lead author (often the Incident Commander, but specifically not the person who triggered the outage, to maintain objectivity) writes the draft. The document must include:
    • Executive Summary
    • Impact (User pain, revenue lost)
    • Timeline (Exact UTC timestamps)
    • Root Cause Analysis
    • Action Items
  3. The Meeting: The team gathers to review the document. The tone must be strictly objective. The phrase “Why did you do X?” is banned. It is replaced with “What information led to the decision to do X?”
  4. Remediation: The action items are converted into Jira tickets or GitHub issues, prioritized, and tracked to completion.

Failure Modes

Action Item Rot

The most common failure of a postmortem culture is writing excellent documents but never actually executing the action items. If the action items are dumped into a backlog and ignored in favor of new feature work, the postmortem was a waste of time, and the exact same outage will happen again. Organizations must mandate that P0 action items from a postmortem halt all feature work until they are resolved.

Shallow Root Causes

Teams often stop investigating too early, concluding that “the CPU spiked.” CPU spiking is a symptom, not a root cause. The “5 Whys” technique must be applied to drill down to the systemic flaw.

  1. Why did the service crash? (CPU spiked).
  2. Why did the CPU spike? (A regex evaluation took too long).
  3. Why did the regex take too long? (A user submitted a 10MB string).
  4. Why did the system accept a 10MB string? (The API gateway lacked a payload size limit).
  5. Why did the gateway lack a limit? (The default configuration was deployed without security review). The final answer is the true root cause.

Weasel Words

Documents that use passive-aggressive language or subtle blame (e.g., “The database was dropped inadvertently,” or “A rogue script was executed”) destroy psychological safety. Names should generally be omitted from the timeline (“An operator executed the script” rather than “Bob executed the script”), focusing entirely on the technical actions.

Security

Security breaches require blameless postmortems just as much as reliability outages. If an employee falls for a phishing email, firing the employee does not secure the company. The blameless postmortem must investigate why the email filter failed to flag the message, and why the employee’s compromised credentials allowed access to internal systems without requiring hardware-backed Multi-Factor Authentication (MFA).

Operational Guidance

Publish postmortems widely. Do not hide them in a private folder. Engineering culture improves when a team publicly presents their worst failures and explains the systemic fixes they implemented. Some of the most mature engineering organizations (like Cloudflare and GitHub) publish their internal postmortems publicly on their corporate blogs to build trust with their users.

Diagnostics

To determine if a postmortem culture is truly blameless, observe the behavior of engineers during an active incident. If an engineer realizes they caused the outage and immediately speaks up (“Stop, I just ran a command that dropped the table, that’s what caused the spike”), the culture is healthy. If the engineer stays silent out of fear of being fired, forcing the rest of the team to spend two hours debugging a mystery, the culture is broken and inherently dangerous.

Related topics

Sources & further reading