Skip to content

Drift Detection and Rollback by Revert

Continuous reconciliation (page 1) is the mechanism; this page covers the two payoffs a DevOps junior needs to be able to explain: what happens when the live cluster drifts from git, and why rollback becomes a git operation instead of an operational scramble.

TL;DR — in 30 seconds:

  • Drift is any difference between what git declares and what's actually running — usually caused by a manual, out-of-band change (a kubectl edit, a hotfix applied by hand).
  • A GitOps operator detects drift on its normal reconciliation loop and, depending on configuration, either flags it (visible, requires a human decision) or auto-heals it (reverts the live state back to match git without waiting for a human).
  • Rollback by revert: because live state always tracks git, undoing a bad deploy means reverting the git commit that introduced it — the operator reconciles the cluster back to the prior state on its own; there is no separate "remember and re-run the old deploy command" step.

1. Drift: what it is and how it's caught

flowchart LR
    A["Someone runs<br/>kubectl edit directly"] --> B["Live state now differs<br/>from git"]
    B --> C["Next reconciliation loop"]
    C -->|"drift detected"| D{"Auto-heal enabled?"}
    D -->|"yes"| E["Operator reverts live state<br/>back to match git"]
    D -->|"no"| F["Operator flags the drift<br/>for a human to resolve"]

Drift is the direct consequence of git being the source of truth but not being the only thing with write access to the cluster. Anyone with cluster access can still run kubectl edit or kubectl apply by hand — GitOps doesn't prevent that at the API-server level. What it does is notice: on the next reconciliation pass, the operator compares live state to git, finds a difference that wasn't caused by a git change, and surfaces it as drift.

Whether the operator auto-heals (reverts silently) or just flags (waits for a human) is a configuration choice, and it's a real tradeoff:

  • Auto-heal guarantees the cluster always matches git, but can undo an emergency manual fix before anyone's had a chance to also fix it properly in git.
  • Flag-only avoids that surprise, but requires someone to notice and act — drift can persist longer.

2. Rollback by revert

Because the operator's whole job is "make live state match git," rolling back a bad deploy doesn't require remembering what command deployed the previous version, or manually reconstructing the old state. It requires one thing: revert the git commit that introduced the bad change (or merge a new commit restoring the prior manifests). The operator picks up that change on its next reconciliation pass and applies it exactly like any other change — no special "rollback mode."

flowchart LR
    A["Bad commit merged<br/>to GitOps repo"] --> B["Revert commit<br/>pushed to git"]
    B --> C["Next reconciliation pass"]
    C --> D["Operator applies the revert<br/>exactly like any other change"]
    D --> E["Live state matches<br/>the prior git state"]

This is a meaningfully different story than a traditional push-based pipeline, where "roll back" often means finding the last known-good deploy command or artifact version and re-running a deploy manually.

Common gotchas

  • Manually patching a live issue and considering it resolved. Fix: the fix doesn't persist under GitOps — it's drift, and depending on configuration it will be flagged or reverted. The permanent fix has to land in git.
  • Not knowing whether a cluster is set to auto-heal or flag-only. Fix: this is a specific, checkable configuration setting on the operator — know which mode a given environment runs in before relying on either behavior.
  • Trying to roll back by remembering the previous deploy command. Fix: revert the git commit and let the operator reconcile — there's no other rollback mechanism to remember.
Check yourself — an on-call engineer applies an emergency manual fix directly to a production cluster managed by a GitOps operator set to auto-heal. What happens, and what should they do differently?

On the next reconciliation pass, the operator detects that the live state no longer matches git and reverts the manual fix, since auto-heal is enabled — the emergency fix disappears. The engineer should instead land the fix as a commit to the GitOps repo (even under time pressure, via a fast-tracked PR), so the operator applies and keeps it, rather than fighting the operator's own reconciliation.

Done when: you can explain what triggers drift, what auto-heal versus flag-only does differently, and why rollback means reverting a git commit rather than re-running a deploy command.