Incident command system (ICS)

Reference Card

• Origin/Prior Art: California wildfires (FIRESCOPE, 1970s), FEMA.
• Related Practices: War rooms, Blameless culture, Communication plans.
• Key Concepts: Incident Commander, Operations, Communications, Scribe.
• Primary Failure Modes: Bystander effect, Executive meddling, Chaos.

Introduction

When a catastrophic failure occurs - such as a primary database corruption taking down the entire business - panic ensues. Engineers jump into a Slack channel, talking over each other. Three different people attempt to run database recovery scripts simultaneously, colliding and causing further damage. Executives jump in demanding ETAs every five minutes, distracting the engineers trying to fix the problem.

The Incident Command System (ICS), originally developed by firefighters to manage massive wildfires, is adapted by tech organizations to impose strict order on this chaos. It establishes clear roles, a chain of command, and a unified communication strategy, ensuring the team acts as a coordinated unit rather than a mob.

Mental Model

Think of an emergency room dealing with a massive trauma patient.

If ten doctors crowd around the patient and all try to shout orders, the patient dies. Instead, there is one Attending Physician (the Incident Commander). They stand back; their hands do not touch the patient. They look at the monitors, take input from the specialists, and make the final decisions. The surgeons (Operations) focus solely on operating, without worrying about talking to the family. The charge nurse (Communications) handles logistics and updates the family.

ICS in tech works exactly the same way.

Operational Pattern

During a major outage (e.g., a Sev-1), an ad-hoc organizational structure is instantly formed, superseding normal company hierarchy.

1. Incident Commander (IC): The single source of truth and authority. The IC holds all state. They do not look at code, they do not run commands, and they do not fix the problem. They coordinate the response, make the hard decisions (e.g., “We will failover to the secondary region and accept 5 minutes of data loss”), and direct the other roles.
2. Operations Lead: The person actually executing the technical work. They run the commands, reboot the servers, and query the logs, reporting their findings back to the IC.
3. Communications Lead: Handles all external communication. They update the public status page, write emails to executives, and field questions from customer support, actively shielding the Operations team from distraction.
4. Scribe: A strictly observational role. They maintain a timeline of events in a shared document, noting exactly when decisions were made and when commands were executed. This is vital for the postmortem.

Failure Modes

Executive Meddling

During an outage, a VP or CEO joins the incident channel and begins giving technical orders. Engineers naturally defer to the executive, destroying the IC’s authority and plunging the response back into chaos. ICS rules dictate that during an incident, the IC outranks everyone, including the CEO, regarding the incident response. The Communications Lead must politely but firmly direct the executive to a separate status channel.

The Working Commander

The most common mistake is an engineer declaring themselves the IC, but then immediately diving into a terminal to debug a database query. Because they are looking at code, they lose the high-level picture, miss updates from other teams, and fail to coordinate. The IC must keep their hands off the keyboard.

The Bystander Effect

Fifty engineers join the incident Zoom call “just to listen.” When the IC asks a general question (“Can someone check the Redis metrics?”), silence follows because everyone assumes someone else will do it. The IC must assign tasks to specific, named individuals (“Sarah, please check the Redis metrics and report back in two minutes”).

Security

During a security incident (a breach), the ICS structure remains identical, but the communication channels must be fundamentally altered. Standard incident Slack channels and Zoom bridges must be abandoned because the attacker may have compromised the corporate network and could be actively monitoring the incident response.

Security incidents require predetermined out-of-band communication methods (like encrypted Signal groups or separate isolated tenants) to coordinate the response securely.

Operational Guidance

Any responder can request to escalate the incident or hand over the IC role if they become overwhelmed. Handover must be explicit: “I am handing over Incident Command to David. David, do you accept?” “I accept. I am now the IC.”

Establish criteria for declaring an incident. Do not wait for absolute confirmation of a total outage to spin up the ICS structure. If a severe anomaly is detected, declare an incident immediately. It is always better to spin up an ICS response and spin it down 5 minutes later because it was a false alarm, than to waste 45 minutes in chaotic debugging before finally organizing.

Diagnostics

To diagnose the health of your ICS process, review the chat transcripts of past incidents.

• Is there a clear, timestamped moment where someone explicitly states: “I am taking Incident Command”?
• Are there periods of more than 15 minutes where the Communications Lead failed to provide an update to stakeholders?
• Did multiple people execute conflicting commands simultaneously?

Related topics

• Blameless postmortems
• Runbooks and playbooks

Sources & further reading

• Incident Management at Google - Google
• FEMA Incident Command System - FEMA