Rolling updates

A rolling update replaces instances of an application incrementally. This allows deployments to proceed without taking the entire service offline, maintaining capacity while transitioning to a new version.

Introduction

A rolling update replaces instances of an application incrementally rather than all at once. The deployment system stops a subset of the running instances, deploys the new version to them, and starts them back up. Once those instances pass their health checks and begin serving traffic, the system proceeds to the next subset. This continues until all instances run the new version.

Mental Model

Think of replacing the wheels on a moving car, one at a time. The car keeps driving. You take one old wheel off, put one new wheel on, ensure it rolls smoothly, and move to the next. At no point does the car stop, though it might drive slightly differently while holding mismatched wheels.

Operational Pattern

The deployment orchestrator (such as Kubernetes) manages the rollout process. It calculates the batch size, often expressed as a percentage of the total instances or a fixed maximum number of unavailable instances (e.g., maxUnavailable: 1).

The orchestrator signals the first batch of old instances to shut down gracefully. It provisions new instances running the updated artifact. The load balancer removes the shutting down instances from its pool and adds the new instances once they pass readiness probes.

If a new instance fails its readiness probe, the orchestrator halts the rollout. The remaining old instances continue serving traffic. This prevents a bad update from taking down the entire service.

Failure Modes

Capacity drops during the rollout. If a service requires ten instances to handle peak load, shutting down three instances simultaneously during a rolling update causes the remaining seven to overload and fail.

Data format changes break mixed-version environments. A rolling update guarantees that old and new versions run concurrently. If the new version alters a shared database schema in a way the old version cannot handle, the old instances crash.

Deployments take too long. Replacing instances one by one across a fleet of thousands takes hours. This delays the delivery of urgent bug fixes.

Security

Mixed-version environments complicate security boundaries. If a new version introduces a stricter access control policy, attackers can bypass it by routing requests to the older instances still running during the update window.

Secret rotation requires careful orchestration. If you rotate a database password, you must ensure both the old and new application versions possess the correct credentials until the rollout completes.

Operational Guidance

Set appropriate surge and unavailability limits. Use a maxSurge value to provision new instances before terminating old ones. This preserves total serving capacity during the rollout, at the cost of temporarily increasing infrastructure usage.

Implement robust readiness probes. The orchestrator relies entirely on these probes to determine when an instance is ready to receive traffic. A probe that returns success before the application fully initializes causes dropped requests.

Enforce backward compatibility. Application code must tolerate running alongside previous versions. Database schemas must support both the old and new code simultaneously.

Diagnostics

Monitor the number of available replicas. A sudden drop during a deployment indicates that new instances are failing their readiness checks while old instances are already terminated.

Check deployment duration. A rollout stuck in progress usually means instances are entering a crash loop before they become ready.

Examine load balancer 5xx error rates. Errors isolated to the deployment window suggest inadequate graceful shutdown handling or premature readiness reporting.

Sources & further reading

Kubernetes Documentation: Deployments - The Linux Foundation