WSS
On this page
Monitoring & ObservabilityRecommendedUpdated

High-cardinality metrics

Understanding the cost and performance impact of attaching highly unique labels, like user IDs, to time-series metrics data.

Reference Card

  • Origin/Prior Art: Time-Series Databases (Prometheus, InfluxDB).
  • Related Practices: Dimensional metrics, Observability cost control, Event logging.
  • Key Concepts: Cardinality, Dimensions, Time-Series explosion, Indexing limits.
  • Primary Failure Modes: Database OOM kills, astronomical billing, slow dashboards.

Introduction

Modern metrics are dimensional. Instead of just tracking http_requests_total, a system tracks http_requests_total and attaches labels (dimensions) to it, such as method="GET" and status="200". This allows operators to slice and filter data, for example, querying the error rate specifically for POST requests.

Cardinality refers to the number of unique combinations of these labels. “Low cardinality” means a label has very few possible values (e.g., status has ~5 common values: 200, 404, 500). “High cardinality” means a label has a massive or infinite number of possible values (e.g., user_id could have millions). Inserting high-cardinality data into a metric system will destroy it.

Mental Model

Think of a time-series database (TSDB) as a giant filing cabinet. Every unique combination of labels creates a brand new physical folder in the cabinet.

If you track requests by status (5 values) and method (4 values), you create 20 folders. The TSDB opens those 20 folders and efficiently slides new numbers into them every second.

If you decide to add user_id as a label, and you have 10 million users, you just commanded the TSDB to instantly construct 10 million new physical folders. The TSDB runs out of memory tracking all these open folders and crashes.

Operational Pattern

Time-Series Databases (like Prometheus) are optimized for tracking the performance of infrastructure and services, not individual users.

  1. An application emits a low-cardinality metric: http_requests_total{service="checkout", status="500"} 42
  2. The TSDB reads this, finds the single time-series in memory, and appends the value 42 at the current timestamp.
  3. Dashboards query this efficiently because there are only a few hundred active time-series to aggregate.

To track high-cardinality data (like “which specific user experienced this error?”), operators must shift that data away from Metrics and into Logs or Traces.

  1. The application emits a log event or trace span, which is designed to handle wide, highly unique events.
  2. The log contains: {"event": "checkout_failed", "user_id": "u-987654", "cart_value": 150.00}.
  3. The observability platform indexes these logs, allowing engineers to search for specific users without exploding a metric time-series index.

Failure Modes

The Cardinality Explosion

A developer well-intentionally adds the HTTP request path to a metric: http_requests_total{path="/api/users/1234"}. Because the user ID is embedded in the URL path, every single request generates a brand new time-series. The Prometheus server attempts to hold millions of unique series in RAM, exhausts its memory limit, and is OOMKilled by the kernel. The entire monitoring infrastructure goes blind.

Astronomical Vendor Billing

Managed observability vendors (like Datadog or New Relic) often charge based on the number of active custom metrics or time-series. A cardinality explosion caused by a bad label (like embedding a dynamically generated UUID) can instantly spike a monthly monitoring bill from $500 to $50,000 in a matter of hours.

Dashboard Timeouts

Even if the database survives the memory pressure, the sheer volume of data makes queries impossibly slow. When an engineer opens a dashboard during a critical incident, the query attempts to aggregate across 500,000 unique series, times out after 60 seconds, and returns no data.

Security

High-cardinality labels often accidentally capture sensitive data. If a developer uses a raw, un-parameterized URL path as a label, and a user submits a password reset request via a GET parameter (e.g., /reset?token=secret123), that sensitive token becomes permanently indexed as a metric label. Metric systems rarely have the granular redaction tooling that logging systems possess, making this data difficult to scrub.

Operational Guidance

Strictly govern what labels are allowed on metrics. Implement CI/CD linters that block code if it attempts to pass variables like user_id, email, ip_address, or session_id into a metric tracking function.

Always normalize URL paths before using them as labels. The path /api/users/9942/profile must be collapsed by the web framework’s router into a parameterized route like /api/users/{id}/profile before it is attached to a metric.

Diagnostics

To diagnose a cardinality explosion in Prometheus, query the top series by count:

topk(10, count by (__name__) ({__name__=~".+"}))

This reveals which specific metric has exploded into thousands of series.

To find which label is causing the explosion on a specific metric (e.g., http_requests_total), count the unique values per label:

count(count by (path) (http_requests_total))

If the result is exceptionally high, the path label is the culprit and must be removed or normalized in the application code.

Related topics

Sources & further reading