On this page
The four golden signals
The essential metrics - Latency, Traffic, Errors, and Saturation - that provide a complete picture of a service's health.
Reference Card
- Origin/Prior Art: Google Site Reliability Engineering (SRE).
- Related Practices: USE Method (Utilization, Saturation, Errors), RED Method (Rate, Errors, Duration).
- Key Concepts: Latency, Traffic, Errors, Saturation.
- Primary Failure Modes: Blind spots, Alerting on averages instead of percentiles, Ignoring saturation limits.
Introduction
When an engineer builds a new microservice, they often wonder what exactly they need to monitor. A modern telemetry system can expose thousands of metrics per service, creating overwhelming dashboards full of useless noise.
The “Four Golden Signals” framework cuts through this noise. Developed by Google’s SRE teams, it asserts that if you can only measure four metrics for a user-facing system, they must be Latency, Traffic, Errors, and Saturation. If all four signals are healthy, the service is healthy. If any signal is abnormal, an incident is occurring.
Mental Model
Think of managing a popular restaurant.
- Traffic: How many customers are walking through the door right now? (Are we busy?)
- Latency: How long does it take from the moment a customer orders until they receive their food? (Are we slow?)
- Errors: How many orders were cooked incorrectly or dropped on the floor? (Are we failing?)
- Saturation: Every table is full, the kitchen grills are entirely covered in meat, and the waitstaff is running. (Are we at maximum physical capacity?)
If traffic is high, but latency is low, errors are zero, and the kitchen isn’t saturated, the restaurant is operating perfectly.
Operational Pattern
Dashboards for every service must prioritize these four signals at the very top.
1. Latency
The time it takes to service a request. Crucially, you must distinguish between the latency of successful requests and the latency of failed requests. An HTTP 500 error that fails instantly (in 10ms) will artificially lower your average latency, making the system look incredibly fast exactly when it is breaking completely.
2. Traffic
A measure of how much demand is being placed on the system. For a web API, this is usually measured in HTTP requests per second. For an audio streaming system, it might be network I/O rate or concurrent sessions.
3. Errors
The rate of requests that fail. This includes explicit failures (HTTP 500s), implicit failures (an HTTP 200 response that contains a malformed JSON payload), and policy failures (a request that succeeded but took 4 seconds, violating a 1-second Service Level Objective).
4. Saturation
How “full” your service is. This measures the most constrained resource in the system. For a CPU-bound application, it is CPU utilization. For a database, it might be the number of available connection pool threads. Saturation is a leading indicator; if saturation hits 100%, latency will immediately spike, and errors will follow.
Failure Modes
The Averages Trap
Measuring the average (mean) latency is extremely dangerous. If 99 users get a response in 10ms, but 1 user waits 10,000ms (10 seconds), the average latency is 109ms. The dashboard looks fine, but that 1 user is experiencing a complete outage.
Golden signals must always be measured using percentiles (p50, p90, p99). Tracking the p99 latency ensures that you are monitoring the experience of the slowest 1% of your users.
Misidentifying the Saturation Point
Engineers often assume CPU or Memory is the saturation point for every service. If a Node.js service is I/O bound, its CPU might sit at 10% and memory at 20%, but it is completely saturated because it has exhausted its limited pool of outgoing network sockets. Identifying the correct bottleneck requires deep load testing.
Ignoring Client-Side Signals
Measuring the four golden signals exclusively at the server level misses the network. If the server logs show a latency of 50ms, but a massive BGP routing issue on the internet is adding 5000ms of lag, the user thinks the site is broken while the server dashboards show everything is green. Telemetry should ideally include client-side metrics (e.g., from the browser or mobile app).
Security
Traffic spikes are the primary indicator of Denial of Service (DoS) attacks. A sudden 10,000% increase in traffic, immediately followed by 100% saturation and high latency, requires automated traffic shaping and rate limiting.
A sudden spike in Errors (specifically HTTP 401 Unauthorized or 403 Forbidden) is the golden signal for a brute-force credential stuffing attack or a vulnerability scanner probing the perimeter.
Operational Guidance
Set alerts on the derivatives of these signals, not just flat thresholds. A slow, steady increase in saturation over a month indicates organic growth and prompts capacity planning. A massive spike in saturation in three seconds indicates a software bug or an attack, requiring immediate incident response.
Diagnostics
When paged for an incident, read the Golden Signals dashboard left to right:
- Did Traffic spike? (We are being overwhelmed by demand).
- Did Errors spike while traffic remained flat? (A bad code deployment just broke the logic).
- Did Saturation hit 100%? (We have a resource leak or a blocked database query).
- Is Latency high but everything else normal? (A downstream dependency is slowing us down).