WSS
On this page
Reliability & Incident ResponseRecommendedUpdated

Runbooks and playbooks

Explicit, step-by-step technical manuals designed to guide engineers through mitigating specific alerts and routine operational tasks.

Reference Card

  • Origin/Prior Art: ITIL, System Administration practices.
  • Related Practices: Incident Management, Toil reduction, Automation.
  • Key Concepts: Executable documentation, Mitigation steps, Escalation paths.
  • Primary Failure Modes: Stale documentation, Ambiguous instructions, Missing permissions.

Introduction

When an engineer is paged at 3:00 AM because the UserDatabaseReplicationLag alert is firing, they are not operating at peak cognitive capacity. They do not have the mental energy to navigate complex architectures, search through Wiki pages, or safely experiment with database failover commands.

A runbook (sometimes called a playbook) is a concise, highly specific document attached directly to an alert. It provides the sleep-deprived engineer with the exact context of what the alert means, how to verify the failure is real, and the explicit commands required to mitigate the issue immediately.

Mental Model

Think of a runbook like the emergency procedures manual in an airplane cockpit.

When an engine catches fire, the pilot does not pull out an engineering textbook on jet propulsion to understand why it failed. They pull out a laminated checklist specifically titled “Engine Fire.” The checklist contains exact, sequential actions: 1. Throttle Idle. 2. Fuel Cutoff. 3. Fire Extinguisher Discharge. The goal is to stabilize the situation quickly and safely without requiring deep cognitive deduction.

Operational Pattern

Every alert that triggers a page must have a linked runbook. If an alert has no runbook, it should not page a human.

A high-quality runbook contains:

  1. Context: What does this service do, and what does this specific alert mean? (e.g., “The Redis cache is out of memory; the API will fall back to the slow database.”)
  2. Verification: How can the responder confirm the alert is real and not a false positive? (e.g., “Run this PromQL query to check the eviction rate.”)
  3. Mitigation: Step-by-step, copy-pasteable commands to stop the bleeding. (e.g., “Execute kubectl scale deployment cache --replicas=5 to add capacity.”)
  4. Escalation: If the mitigation fails, exactly who should be contacted next? (e.g., “Page the #data-platform team.”)

Failure Modes

Stale Documentation

The most dangerous runbook is an outdated one. A runbook written two years ago might instruct an engineer to restart a service using a bash script that no longer exists, or worse, execute a command that causes a catastrophic failure in the new architecture. Runbooks must be treated as code. They should live in the version control repository alongside the application code and be reviewed during every pull request that alters the service’s behavior.

Ambiguous Instructions

A runbook that says “Check the logs and restart the server if it looks unhealthy” is useless. “Unhealthy” is subjective. Runbooks must be deterministic. They must state: “If the log contains the phrase OutOfMemoryError, execute this exact command: systemctl restart myapp.”

Permission Roadblocks

A perfect runbook is useless if the on-call engineer does not have the IAM permissions to execute the mitigation commands. Organizations often write runbooks assuming a senior engineer is executing them, but when a junior engineer is on-call, they hit permission denied errors. The permissions required to execute the runbook must be explicitly documented and granted to all on-call responders in advance.

Security

Runbooks dealing with security incidents (e.g., “Malicious IP detected”) must include strict chains of custody. If the mitigation step is “Isolate the compromised container,” the runbook must explicitly forbid the engineer from deleting the container, as that destroys forensic evidence. The runbook must dictate how to pause the container and snapshot its memory for the security team to analyze later.

Operational Guidance

Automate your runbooks away. If a runbook’s mitigation step is always “clear the /tmp directory and restart the service,” a human should not be doing it. The monitoring system should trigger an automated webhook to execute that script, and only page a human if the automated script fails to resolve the alert. This eliminates toil.

Diagnostics

During postmortems, the team must explicitly evaluate the runbook that was used during the incident.

  • Was the runbook linked in the alert?
  • Was it accurate?
  • Did the mitigation steps work? If the answer to any of these is no, updating the runbook is the highest priority action item of the postmortem.

Related topics

Sources & further reading