WSS
On this page
Infrastructure & ProvisioningRecommendedUpdated

Configuration drift and reconciliation

Detecting and automatically correcting drift between the desired state defined in version control and the actual state of the running environment.

Reference Card

  • Origin/Prior Art: System administration practices, Kubernetes Controllers.
  • Related Practices: Infrastructure as Code, GitOps, Continuous Deployment.
  • Key Concepts: Desired state, Actual state, Convergence loop, Remediation.
  • Primary Failure Modes: Revert wars, Silent failures, Alert fatigue.

Introduction

Configuration drift occurs when the actual state of a system diverges from its defined desired state. This divergence happens through manual intervention, failed automated updates, external API changes, or malicious activity. If left unchecked, drift undermines the value of Infrastructure as Code, making deployments unpredictable and rollbacks impossible.

Reconciliation is the automated process of detecting this drift and correcting it. Modern orchestrators and GitOps tools operate on continuous reconciliation loops, constantly forcing the live environment to match the definitions stored in version control.

Mental Model

Think of a modern thermostat. You set the desired temperature to 72 degrees. The thermostat constantly measures the actual room temperature. If someone opens a window and the temperature drops (drift), the thermostat activates the heater to reconcile the difference. It does not run on a schedule; it runs as a continuous control loop.

Modern orchestration tools follow this exact pattern. The desired state (the codebase) is the temperature setting. The reconciliation loop constantly measures the cloud environment. If an operator manually changes a firewall rule, the system detects the drift and immediately reverts the rule to match the code.

Operational Pattern

Reconciliation loops are typically implemented by autonomous software agents running inside the target environment.

  1. The agent queries the source of truth (usually a Git repository) to determine the desired state.
  2. The agent queries the live environment (via cloud APIs or local system calls) to determine the actual state.
  3. The agent calculates the delta between the two states.
  4. If a delta exists, the agent executes the precise commands necessary to modify the environment, converging the actual state toward the desired state.
  5. The agent sleeps briefly, then repeats the loop indefinitely.
[Desired State]      [Actual State]
      \                  /
       \                /
     [Calculate Delta]
            |
      [Apply Changes]
            |
      (Repeat Loop)

Failure Modes

Revert Wars

Two conflicting automation systems might attempt to reconcile the same resource to different states. For example, a legacy cron job attempts to scale down instances at night, while a modern GitOps agent attempts to keep instances scaled up according to its configuration. The two systems will continuously overwrite each other’s changes, creating a “revert war” that exhausts API rate limits and masks real issues.

Silent Failures

If the reconciliation agent loses network connectivity to the source of truth, it cannot detect new commits. The system appears stable and healthy on dashboards, but it is actually drifting from the intended configuration. Teams may merge critical security patches into the main branch, assuming they are deployed, while the disconnected agent continues enforcing an outdated state.

Alert Fatigue

Drift detection tools that report every minor, inconsequential change (like timestamp updates on cloud resources) quickly overwhelm operators. If drift alerts are noisy, operators will route them to a muted channel, guaranteeing that malicious or catastrophic drift goes unnoticed.

Security

Drift is often the first indicator of unauthorized access. A manual firewall rule change might be a well-intentioned temporary fix by a tired engineer, but it is indistinguishable from a malicious backdoor established by an attacker.

Continuous reconciliation automatically reverts these changes, closing backdoors minutes after they are created. This aggressive remediation forces all actors - legitimate or otherwise - to make changes through the audited version control system, where they can be reviewed and blocked.

Operational Guidance

Run drift detection frequently. A pipeline that only checks for drift during a daily scheduled run leaves a 24-hour window where manual changes can cause incidents or obscure security breaches.

Always send alerts when drift is detected, even if the tool automatically fixes it. Automatic remediation resolves the technical issue, but the alert highlights a broken human process where team members are attempting to bypass standard operating procedures.

For legacy systems that cannot support continuous reconciliation, use drift detection in a “report-only” mode. This provides visibility into technical debt without risking disruptive automated changes to fragile infrastructure.

Diagnostics

Check the reconciliation logs of the controller or agent. They specify exactly which properties drifted, what the system expected, and what actions were taken to remediate the issue.

Compare the exact timestamp of the drift detection against audit logs in the cloud provider. This allows you to identify the specific IAM user, service account, or IP address that initiated the out-of-band manual change.

Related topics

Sources & further reading