WSS
On this page
Reliability & Incident ResponseRecommendedUpdated

Mean Time To Recovery (MTTR)

A critical metric that measures how quickly a team can diagnose, mitigate, and resolve a production outage.

Reference Card

  • Origin/Prior Art: ITIL, DevOps Research and Assessment (DORA).
  • Related Practices: MTBF (Mean Time Between Failures), Rollbacks, Feature Flags.
  • Key Concepts: Time to Detect (MTTD), Time to Resolve (MTTR), Mitigation vs Resolution.
  • Primary Failure Modes: Optimizing for MTBF over MTTR, Manual deployments, Lack of observability.

Introduction

In traditional IT, the primary metric of success was Mean Time Between Failures (MTBF). The goal was to prevent systems from ever breaking, which led to glacial release cycles, massive QA phases, and a pervasive fear of change.

Modern DevOps recognizes that in complex distributed systems, failure is inevitable. Instead of optimizing to prevent failure, high-performing teams optimize to recover from failure as fast as possible. Mean Time To Recovery (MTTR) is the average time it takes a team to restore service to users after an incident begins.

Mental Model

Think of a tightrope walker. Optimizing for MTBF means trying to build a wider, stiffer rope, outfitting the walker with heavy armor, and only allowing them to walk when there is zero wind. It is expensive and slow.

Optimizing for MTTR means putting a massive, bouncy safety net exactly three feet beneath the rope. The walker will inevitably fall, but the fall is harmless, and they can bounce right back up in seconds. Because the cost of failure is essentially zero, the walker can move incredibly fast.

Operational Pattern

MTTR is a composite metric. To lower it, an organization must optimize the four distinct phases of an incident:

  1. Mean Time to Detect (MTTD): The time between the failure occurring and the alerting system paging an engineer. Optimized by implementing Symptom-based alerting and tight Service Level Objectives (SLOs).
  2. Mean Time to Identify (MTTI): The time it takes the engineer to open the dashboard, read the traces, and find the root cause. Optimized by structured logging, distributed tracing, and the Three Pillars of Observability.
  3. Mean Time to Mitigate (MTTM): The time it takes to stop the user pain. If a bad code deployment caused the outage, mitigation means clicking a button to rollback the deployment or toggling a feature flag off.
  4. Mean Time to Resolve (MTTR): The time it takes to actually fix the underlying bug in the code and deploy the permanent solution.

Note: In modern practice, “MTTR” is often used colloquially to mean “Mean Time to Mitigate,” because returning the system to a healthy state for the user is the operational priority, even if the root cause takes weeks to fully resolve.

Failure Modes

Roll-Forward Mentality

When a deployment breaks production, a developer’s instinct is often to write a hotfix, run it through the CI pipeline, and deploy a new version to fix the bug (“rolling forward”). This inevitably takes 15 to 45 minutes, resulting in a terrible MTTR. The operational rule must be: if a deployment breaks the system, you instantly rollback to the previous known-good state. You debug and write the hotfix in the development environment later.

Manual Intervention

If mitigating an incident requires an engineer to SSH into five different servers, manually edit configuration files, and restart daemon processes, the MTTR will always be measured in hours. Mitigation must be a one-click automated action (e.g., an automated infrastructure pipeline, or automated failover scripts).

The “Sunk Cost” Debugging Trap

During an active incident, engineers often want to figure out why a node is broken before they kill it. They spend 30 minutes reading memory dumps on a live, burning production server while users suffer. Optimizing for MTTR requires treating infrastructure immutably: if a node is broken, terminate it immediately and let the orchestrator replace it. Investigate the failure later using logs.

Security

MTTR applies critically to security incidents. The “Time to Remediate” a zero-day vulnerability measures how fast an organization can patch a CVE across its entire fleet. If patching requires a two-week manual QA cycle, the company remains exposed to active exploitation for two weeks. Low deployment MTTR (the ability to confidently ship code in minutes) is a fundamental security requirement.

Operational Guidance

Measure MTTR rigorously, but do not tie engineer compensation or bonuses to it. If bonuses are tied to MTTR, engineers will game the system by delaying the official “Start” time of an incident or prematurely marking incidents as “Resolved” before the system is actually stable, rendering the metric useless.

Diagnostics

To diagnose a high MTTR, analyze past incident timelines and find the longest gap:

  • If the gap is between the outage starting and the page firing, your alerting (MTTD) is broken.
  • If the gap is between the page firing and the engineer finding the cause, your observability (MTTI) is broken.
  • If the gap is between finding the cause and restoring service, your deployment and rollback pipelines (MTTM) are broken.

Related topics

Sources & further reading