Alert fatigue and actionable alerting

Designing alerts that require human intervention and eliminating noisy warnings that train engineers to ignore paging systems.

Reference Card

Origin/Prior Art: Google Site Reliability Engineering (SRE) practices.
Related Practices: Service Level Objectives (SLOs), Incident Response, PagerDuty culture.
Key Concepts: Actionability, Urgency, Signal-to-noise ratio, Symptom-based alerting.
Primary Failure Modes: The “Boy Who Cried Wolf” syndrome, Burnout, Ignored critical outages.

Introduction

Alert fatigue is the psychological and operational breakdown that occurs when engineers are bombarded with too many monitoring alerts, the vast majority of which do not require actual intervention. When a pager goes off ten times a day for minor CPU spikes that resolve themselves, engineers unconsciously train themselves to ignore the pager.

When a genuine, catastrophic database failure finally triggers the pager, the on-call engineer assumes it is just another false alarm, mutes it, and goes back to sleep. Effective alerting is not about measuring everything; it is about alerting only when a human must take immediate action to prevent user pain.

Mental Model

Think of a home smoke detector.

If your smoke detector went off every time you burned a piece of toast, every time you took a hot shower, and every time you lit a candle, you would eventually take the battery out because the noise is unbearable. When a real fire starts, your house burns down because the detector is disabled.

A smoke detector should only sound an alarm when there is a critical, life-threatening fire that requires you to immediately evacuate or grab an extinguisher. Everything else (burnt toast) should just be a minor note you handle yourself without an alarm.

Operational Pattern

Actionable alerting relies on shifting from cause-based alerting to symptom-based alerting.

Bad (Cause-based): “Alert: Database CPU is at 90%.” Why is this bad? If the CPU is at 90% but every user query is returning in 50 milliseconds, the users don’t care. The system is just utilizing its resources efficiently. Paging someone at 3 AM for this is actively harmful.
Good (Symptom-based): “Alert: 5% of user checkout requests are failing or taking longer than 3 seconds.” This is a symptom of user pain. The on-call engineer is paged because users cannot buy things. The engineer logs in, investigates, and discovers that the database CPU is at 99%, causing the latency.

The alert focuses on the broken user experience; the metrics (CPU, memory) are used strictly for debugging during the incident, not for triggering the page.

Failure Modes

Warning Alerts

Many organizations configure “Warning” alerts that send emails or Slack messages when thresholds are slightly elevated. These are universally ignored. If an alert does not require someone to stop what they are doing and fix it immediately, it should not be an alert. It should be a dashboard widget that engineers review during weekly operational meetings.

Flapping Alerts

An alert triggers when memory hits 80%. A minute later, memory drops to 79%, and the alert resolves. A minute later, it hits 80% again, and the pager fires again. This flapping destroys on-call sanity. Alerts must require a sustained duration (e.g., “Memory > 80% for 10 consecutive minutes”) to trigger, proving the condition is a genuine trend, not a momentary micro-spike.

Redundant Paging

If a core database crashes, it takes down the Cart service, the User service, and the Inventory service. If the alerting system is poorly configured, the on-call engineer receives four separate pages simultaneously. This causes panic and scatters the investigation. Modern alerting tools must support dependency mapping and alert grouping, rolling those four alerts into a single incident: “Database outage affecting multiple downstream services.”

Security

Security alerts are highly susceptible to fatigue. If a Web Application Firewall (WAF) alerts the security team every time it blocks an automated SQL injection scan (which happens thousands of times a day on the public internet), the team will mute the channel. Security alerts must be tuned to page only when a defense mechanism fails, or when a highly anomalous pattern suggests a targeted, successful breach is underway.

Operational Guidance

Audit your alerts ruthlessly. If an alert fires, and the on-call engineer closes it with the note “Self-resolved, no action taken,” that alert is a defect. It must be tuned, delayed, or deleted.

Every alert that triggers a pager must have an attached runbook. When the engineer is woken up, the alert payload must contain a link providing explicit instructions on how to validate the alert, common root causes, and safe mitigation steps to stabilize the system.

Diagnostics

To diagnose an unhealthy alerting culture, audit your incident management tool (e.g., PagerDuty). Calculate the “Actionability Ratio”: the number of alerts that resulted in a configuration change, code rollback, or manual mitigation divided by the total number of alerts fired.

If this ratio is less than 50%, your on-call engineers are suffering from alert fatigue, and your system’s actual reliability is in severe danger.

Sources & further reading

My Philosophy on Alerting - Rob Ewaschuk (Google SRE)
Site Reliability Engineering: Practical Alerting - Google