On this page
Secrets management patterns
Patterns for dynamically issuing and revoking temporary credentials, eliminating hardcoded secrets and secret sprawl.
Reference Card
- Origin/Prior Art: Key management systems, HashiCorp Vault.
- Related Practices: Encryption at rest, Identity and Access Management, Zero Trust.
- Key Concepts: Dynamic secrets, Secret zero, Lease revocation, Encryption as a Service.
- Primary Failure Modes: Secret sprawl, Hardcoded credentials, The Secret Zero problem.
Introduction
Secrets management systems store, distribute, and rotate sensitive information like API keys, database passwords, TLS certificates, and SSH keys. They fundamentally separate credentials from application code and configuration files. By centralizing secrets in a specialized vault, organizations gain fine-grained access control, detailed audit logging, and the ability to instantly revoke compromised credentials without redeploying code.
Mental Model
Instead of giving an application a permanent, hardcoded database password (like writing a PIN number on a sticky note), modern secrets management gives the application an identity.
The application authenticates to the secret store using its identity. The secret store acts as a secure broker. If the application is authorized, the store provides a temporary, leased password valid only for a short time (like a hotel keycard). When the lease expires, the password becomes useless. If the application still needs access, it must request a new keycard.
Operational Pattern
Modern secrets management heavily favors dynamic secrets over static, long-lived secrets.
- The application starts up in its environment (e.g., a Kubernetes pod or an AWS EC2 instance).
- The orchestrator or cloud provider provides the application with a cryptographically signed identity token.
- The application sends this token to the centralized secrets manager.
- The secrets manager verifies the token against the trusted platform.
- Instead of reading a static password from a database, the secrets manager connects directly to the target system (e.g., PostgreSQL) and generates a brand new, unique user and password.
- The secrets manager issues this newly generated credential to the application in memory, tied to a strict time-to-live (TTL) lease.
- The application uses the credential. If it needs it longer than the TTL, it requests a lease renewal.
- When the lease expires, the secrets manager automatically drops the user from the PostgreSQL database.
Failure Modes
The Secret Zero Problem
To get a secret from the secrets manager, the application needs a secret to authenticate to the secrets manager. Distributing this initial secret (Secret Zero) securely is inherently difficult. If you bake Secret Zero into the machine image, anyone who copies the image has access to all secrets. Most systems solve this by delegating trust to the underlying platform’s identity provider (like AWS IAM or Kubernetes Service Accounts), eliminating the need to distribute an initial secret manually.
Secret Sprawl
When applications copy secrets into application log files, environment variables, or local disk caches, the secrets manager loses control over their lifecycle. Rotating or revoking the secret in the central manager does not update or remove the leaked copies, leaving the system vulnerable.
Outage Amplification
Centralized secrets managers become single points of failure. If the secrets manager goes offline, applications cannot retrieve credentials to connect to databases or APIs, bringing down the entire environment. High availability and aggressive caching strategies are required to mitigate this risk.
Security
Never store secrets in version control, even if they are obfuscated. Do not pass secrets as command-line arguments to processes, as they remain visible to all users on the host machine via process listing commands (like ps aux).
Inject secrets directly into application memory or expose them as temporary, memory-backed files (like tmpfs) that never touch physical disk storage.
Use Encryption as a Service (EaaS) patterns for handling user data. Instead of applications encrypting data locally, they send plaintext data to the secrets manager, which returns ciphertext. The application never actually possesses the encryption keys, dramatically reducing the impact of a potential breach.
Operational Guidance
Implement strict, short lease times for all dynamic secrets. A lease should ideally last minutes or hours, not months. If an application requires a secret for a long-running process, it must actively and continuously renew the lease.
Audit all access to the secrets manager. The logs show exactly which application identity requested which secret and at what precise time. Export these logs to a central Security Information and Event Management (SIEM) system for anomaly detection.
Diagnostics
If an application reports authentication failures against a downstream service, check the secrets manager audit log first. Verify if the application successfully authenticated to the manager. Check if the secret lease expired before the application could renew it.
Verify that the application’s assigned identity has the correct access policies attached in the secrets manager, and that the underlying platform (e.g., the Kubernetes cluster) is correctly configured as a trusted identity provider.