Log aggregation and indexing

Centralizing logs from distributed ephemeral containers into a searchable, indexed database.

Reference Card

Origin/Prior Art: The ELK Stack (Elasticsearch, Logstash, Kibana), Splunk.
Related Practices: Structured logging, DaemonSets, Log rotation.
Key Concepts: Shippers, Forwarders, Indexers, Inverted Index.
Primary Failure Modes: Dropped logs during network partitions, Index mapping conflicts, Storage exhaustion.

Introduction

In a single-server architecture, viewing logs involves SSHing into the machine and running tail -f /var/log/syslog. In a Kubernetes cluster with 500 ephemeral Pods spinning up and dying every hour, SSH is impossible. When a Pod terminates, its local filesystem (and its logs) are destroyed instantly.

Log aggregation solves this by decoupling log generation from log storage. Applications write logs to standard output, a background agent immediately ships those logs off the physical node, and a central database indexes them, allowing engineers to search across the entire fleet from a single web interface.

Mental Model

Think of a massive international bank. Every branch (a container) generates thousands of transaction receipts (logs) a day. If an auditor wants to find a specific transaction, they cannot physically visit every branch.

Instead, every branch has a dedicated mail clerk (the log forwarder) who collects the receipts and mails them to the central headquarters (the aggregator). At headquarters, clerks read the receipts and file them in massive filing cabinets organized by account number and date (the indexer). When the auditor needs to find a transaction, they just ask the clerk at headquarters, who retrieves it instantly.

Operational Pattern

A robust log aggregation pipeline typically follows a three-stage architecture:

The Shipper (Agent): Applications write JSON logs directly to STDOUT. The container runtime writes these streams to local disk files. A specialized daemon (like Fluentd, Fluent Bit, or Promtail) runs on every physical node. It tails these files, enriches the logs with Kubernetes metadata (adding the Pod name, namespace, and node name), and batches them.
The Aggregator (Buffer/Router): The agents send their batches over the network to a central aggregator cluster. The aggregator buffers the logs (often using a message queue like Kafka) to handle massive spikes in log volume without dropping data. It also performs filtering, dropping debug logs or redacting sensitive PII.
The Indexer (Database): The aggregator pushes the cleaned logs into a search-optimized database (like Elasticsearch, OpenSearch, or Grafana Loki). The database parses the JSON fields and builds an inverted index, allowing sub-second searches across billions of log lines.

Failure Modes

Agent Resource Starvation

Log forwarding agents run on every node. If an application suddenly starts logging 100,000 lines per second due to an infinite loop, the forwarding agent will consume all available CPU on the node trying to parse and ship those logs. This starves the actual applications running on the node. Log agents must be strictly constrained with CPU and memory limits.

Index Mapping Conflicts

If a central database like Elasticsearch expects the field status_code to be an integer (because the first log it received looked like {"status_code": 404}), it creates a strict schema. If another application later sends {"status_code": "404 Not Found"}, Elasticsearch will reject the log entirely because a string cannot be indexed as an integer. This causes silent data loss.

Network Partitions

If the network connection between the worker nodes and the central indexing database goes down, the forwarding agents cannot ship logs. They will buffer logs locally on the node’s disk. If the outage lasts too long, the local disk will fill up, and the agent will be forced to drop newer logs permanently to avoid crashing the node.

Security

Log aggregators become the ultimate honeypot. They contain a complete history of system activity, internal IP addresses, user behaviors, and often accidental secrets. Access to the log aggregator must be strictly controlled with Role-Based Access Control (RBAC).

Engineers should only be granted access to query logs for the specific namespaces or applications they own. The aggregation pipeline must enforce TLS encryption in transit, ensuring that logs are not sent in plaintext across the internal network where a compromised container could sniff them.

Operational Guidance

Separate log retention into tiers. Store the last 7 to 14 days of logs in the expensive, fast-search index (RAM and SSDs) for active incident response. Automatically archive older logs to cheap, slow object storage (like Amazon S3 or Glacier) for long-term compliance and security auditing.

Diagnostics

If logs are missing from the central dashboard, start at the source and follow the pipeline:

View the raw container output directly to ensure the application is actually emitting logs: kubectl logs <pod-name>.
Check the logs of the forwarding agent (e.g., Fluent Bit) on that specific node to see if it is throwing parsing errors or network connection timeouts.
Check the central message queue or aggregator to see if it is dropping logs due to schema conflicts or rate limiting.

Sources & further reading

Log Aggregation Pattern - Microservices.io
Fluentd Architecture - Fluentd