The three pillars of observability

Reference Card

• Origin/Prior Art: Twitter’s Observability Engineering team, Google SRE.
• Related Practices: Incident Response, SRE, Telemetry Collection.
• Key Concepts: Metrics (aggregration), Logs (events), Traces (context).
• Primary Failure Modes: Data silos, Missing correlation IDs, Alert fatigue.

Introduction

Monitoring tells you if a system is broken. Observability tells you why it is broken.

In a monolithic application, debugging is straightforward: you SSH into the server and read the single log file. In a microservices architecture, a single user request might traverse twenty different services, databases, and message queues. When that request fails, no single dashboard or log file holds the answer.

True observability requires instrumenting systems to emit three distinct types of telemetry - Metrics, Logs, and Traces - and crucially, tying them together.

Mental Model

Imagine a vast highway system.

• Metrics are the traffic cameras showing overall speed. They tell you instantly that traffic on Interstate 5 has ground to a halt. They alert you to the problem, but they don’t explain it.
• Logs are the detailed police reports. They contain the specific details of a multi-car collision at mile marker 42. They explain what happened, but finding the right report among thousands is difficult.
• Traces are the GPS tracking data of a single car. They show exactly when the car entered the highway, where it slowed down, and exactly which collision it was involved in. Traces provide the path that connects the metric anomaly to the specific log error.

Operational Pattern

A fully observable system utilizes all three pillars simultaneously, often leveraging the OpenTelemetry standard for unified collection.

1. Metrics (The “What”): Systems continuously emit numerical data: CPU usage, HTTP 500 error rates, database connection counts. These are cheap to store and aggregate over long periods. Dashboards display these metrics, and when the “Checkout Error Rate” spikes above 2%, an alert wakes up an engineer.
2. Traces (The “Where”): The engineer opens the tracing dashboard to investigate the spike. Traces track the lifecycle of a request. The engineer sees that requests hitting the CheckoutService are failing precisely when they attempt to call the downstream InventoryService. The trace pinpoints the exact service causing the bottleneck.
3. Logs (The “Why”): The trace contains a unique trace_id. The engineer queries the centralized logging system for that specific ID. They immediately see the raw event: Connection refused: Database timeout on table inventory_locks. The root cause is identified.

Failure Modes

Uncorrelated Pillars (Data Silos)

If an organization buys three separate enterprise tools - one for metrics, one for logs, one for traces - and they do not share context, the pillars collapse. An engineer gets an alert (metrics), but has to manually guess which logs to search for, wasting critical minutes during an outage. Logs must automatically inject the active Trace ID so the tools can cross-link seamlessly.

Over-Logging (The Cost Trap)

Metrics are highly compressed (just numbers and timestamps). Logs are heavy strings of text. If developers log every minor event (“User clicked button,” “Database query started,” “Database query finished”), log aggregation bills will skyrocket into millions of dollars. Use metrics for counting events, use traces for timing flows, and reserve logs strictly for detailed, anomalous events.

Sampling Blind Spots

To save money, tracing systems often sample data, recording only 1 in every 100 requests. If a bug only affects 0.1% of users, the tracing system might discard the exact traces needed to debug it. Modern systems use “tail-based sampling,” where they initially buffer all traces, but only permanently store the ones that contain errors or exhibit high latency.

Security

Logs frequently become the most dangerous vector for data leaks. Developers accidentally log HTTP request bodies that contain plain-text passwords, credit card numbers, or Personally Identifiable Information (PII). Because logs are aggregated centrally and viewed by all engineers, this creates a massive compliance violation.

Implement automated secret scrubbing at the edge. The log forwarding agent (e.g., Fluentd or Promtail) must run regex filters to redact sensitive patterns (like \b(?:\d[ -]*?){13,16}\b for credit cards) before the data ever leaves the secure network boundary.

Operational Guidance

Instrument applications proactively, not reactively. Do not wait for an outage to realize a microservice has no metrics. Define standard libraries that automatically wrap all incoming HTTP requests to emit metrics and start a trace span without requiring developers to write custom telemetry code.

Diagnostics

During an incident, follow the golden path of observability:

1. Look at the Metrics dashboard. Identify the service where the error rate spiked or latency breached the SLA.
2. Filter Traces originating from that specific time window and service. Find a trace that resulted in a 500 error.
3. Follow the trace waterfall down to the lowest-level span that failed. Copy the trace_id.
4. Search the Logs for that exact trace_id to read the stack trace or database driver error that caused the failure.

Related topics

• Distributed tracing and trace IDs
• High-cardinality metrics

Sources & further reading

• Distributed Systems Observability - Cindy Sridharan (O'Reilly)