On this page
GitOps as a deployment model
An operational framework where a Git repository acts as the single source of truth for declarative infrastructure and applications, relying on software agents to continuously pull and synchronize the desired state.
Reference Card
- Origin/Prior Art: Coined by Weaveworks in 2017.
- Related Practices: Infrastructure as Code (IaC), Continuous Deployment.
- Key Concepts: Desired state vs Actual state, Pull-based vs Push-based deployments, Reconciliation loops.
- Primary Failure Modes: Storing plaintext secrets in Git, Infinite reconciliation loops caused by mutating admission controllers.
Introduction
In a traditional CI/CD pipeline, when a developer merges code, a central server (like Jenkins or GitHub Actions) builds a container image and then actively reaches out to the production cluster to run a deployment script (a “Push” model). This requires giving the CI/CD server administrative credentials to production, which is a massive security vulnerability.
GitOps flips this architecture. Instead of an external server pushing changes, the production cluster contains a software agent that continuously monitors a Git repository. When a developer merges a change to a YAML file in that repository (e.g., updating an image tag from v1.0 to v1.1), the agent detects the commit, pulls the new configuration, and applies it to the cluster (a “Pull” model). The Git repository becomes the absolute, undeniable source of truth for what should be running in production.
Mental Model
Think of a thermostat in a house. In a Push model, you act as the CI/CD pipeline: you feel cold, so you walk to the basement, open the furnace, and manually ignite the burners until the house is warm. In a GitOps (Pull) model, you simply turn the dial on the wall to 72 degrees (updating the Git repository with your desired state). The thermostat (the GitOps agent) continuously monitors the room temperature (the actual state). If the room is 68 degrees, the thermostat automatically fires the furnace until the room matches the dial.
Operational Pattern
GitOps relies on four strict principles, defined by the CNCF OpenGitOps project:
- Declarative: The entire system must be described declaratively (e.g., Kubernetes YAML, Terraform HCL), not via imperative scripts (
do X, then Y). - Versioned and Immutable: The desired state is stored in a version control system (Git) that enforces immutability, versioning, and an auditable history of changes.
- Pulled Automatically: Software agents (like ArgoCD or Flux) automatically pull declarations from Git.
- Continuously Reconciled: The agents continuously compare the actual state of the system against the desired state in Git. If they drift, the agent automatically acts to force the system back to the desired state.
Failure Modes
The Manual Hotfix Override
If a critical outage occurs at 2:00 AM, an engineer might log directly into the Kubernetes cluster using kubectl edit to manually roll back a deployment to a previous image. This fixes the outage for exactly 60 seconds. Then, the GitOps agent wakes up, notices the actual state (the hotfix) no longer matches the Git repository (the broken code), and “helpfully” overwrites the engineer’s hotfix, breaking the system again. In a strict GitOps environment, manual edits to production are impossible; all hotfixes must be executed via a Git commit.
Secret Sprawl
Because GitOps demands that everything required to rebuild the cluster must live in Git, engineers often make the fatal mistake of committing database passwords and API keys in plaintext to the repository. Secrets must be encrypted before entering Git (using tools like Sealed Secrets or SOPS), or the GitOps agent must be configured to fetch them dynamically from a secure vault (like AWS Secrets Manager) at deployment time.
Security
GitOps radically shrinks the attack surface of production environments. Because the deployment agent lives inside the cluster and pulls from Git via HTTPS, you do not need to open inbound firewall ports for an external CI/CD server, and you do not need to grant the CI/CD server production credentials. If your GitHub account is compromised, the attacker can change the source code, but they do not gain direct root access to the production servers.
Operational Guidance
Separate your application source code (Node.js, Python) from your deployment configurations (Helm, Kubernetes YAML). Use two distinct repositories.
If they share the same repository, your CI pipeline will trigger an infinite loop: a developer merges code, the CI pipeline builds an image and automatically commits the new image tag back to the repository, which triggers the CI pipeline to run again, ad infinitum. The application repo should build the artifact; the infrastructure repo should declare where it runs.