TLS certificate rotation

Reference Card

• Origin/Prior Art: X.509 standard (1988), Let’s Encrypt / ACME protocol (2015).
• Related Practices: Secret management, Automated provisioning.
• Key Concepts: ACME protocol, Zero-downtime reloads, Expiry monitoring.
• Primary Failure Modes: Manual renewal processes being forgotten, Load balancers dropping active connections during reload.

Introduction

Transport Layer Security (TLS) certificates do not last forever. Historically, they lasted for three to five years, allowing operations teams to treat certificate renewal as a rare, manual chore. Today, industry standards cap validity at 398 days, and free authorities like Let’s Encrypt enforce 90-day lifespans. Google has proposed reducing the maximum lifespan to just 90 days across the entire internet.

This compressed timeline makes manual rotation mathematically impossible for large organizations to sustain. Automated TLS certificate rotation is the process of having a machine continually monitor expiry dates, request new certificates via API, and inject them into load balancers without dropping active user traffic.

Mental Model

Think of a driver’s license that expires every 3 months. If you have to physically drive to the DMV, fill out paperwork, and wait in line four times a year, you will eventually forget or miss a deadline, and you will be arrested for driving with an expired license (the browser blocking your website).

Automated certificate rotation is like the DMV mailing you a new license 30 days before the old one expires, automatically transferring it to your wallet while you sleep.

Operational Pattern

The modern standard for certificate rotation is the ACME (Automated Certificate Management Environment) protocol.

1. The Agent: A daemon (like certbot, or Kubernetes cert-manager) runs continuously on your infrastructure.
2. The Challenge: 30 days before expiration, the agent contacts the Certificate Authority (CA). The CA issues a challenge to prove ownership of the domain (e.g., “Put this specific random string at http://yourdomain.com/.well-known/acme-challenge/” or “Create a specific DNS TXT record”).
3. The Proof: The agent automatically fulfills the challenge. The CA verifies it and issues the new certificate.
4. The Reload: The agent replaces the cryptographic files on disk and signals the web server (e.g., NGINX, Envoy) to gracefully reload its configuration in memory, applying the new certificate to all new connections while allowing existing connections to finish using the old certificate.

Failure Modes

The Hard Restart

When the new certificate arrives, the web server must begin using it. If the automation script executes systemctl restart nginx, it instantly drops every active HTTP connection and WebSocket on the server, causing a brief but highly destructive outage. Rotation scripts must use a graceful reload command (e.g., systemctl reload nginx or sending a SIGHUP signal), which instructs the server to spawn new worker processes with the new certificate while letting the old workers gracefully drain.

The Forgotten Internal CA

Many organizations master automated rotation for their public-facing websites using Let’s Encrypt, but entirely ignore the internal certificates used for service-to-service communication or VPN gateways. When an internal Root CA expires after 10 years, it instantly destroys all internal communication, taking down the entire company. Internal PKI (Public Key Infrastructure) requires the exact same aggressive automation as public certificates.

Security

Certificate rotation involves high-stakes secrets (the private keys). When a new certificate is generated, the new private key must be stored securely (e.g., in HashiCorp Vault, AWS Secrets Manager, or Kubernetes Secrets). It must never be written to a generic text file in a publicly accessible directory or committed to a Git repository.

Furthermore, aggressively rotating certificates reduces the blast radius of a compromised key. If an attacker steals a private key, it is only useful until the next automated rotation replaces it and invalidates the old one.

Operational Guidance

Never alert on “Certificate expired.” By the time it has expired, it is too late; your customers are already seeing browser warnings and your site is down. You must monitor and alert on “Days until expiration.” Set a SEV-2 alert to fire when a certificate hits 14 days remaining. If the automation was going to work, it would have renewed it at 30 days. If you hit 14 days, the automation is broken and requires immediate human intervention.

Sources & further reading

• Let's Encrypt: How It Works - ISRG