Infra Notes

Argo CD rollback: revert Git instead of patching with kubectl

Roll back an Argo CD application by reverting the failing Git commit, verifying the diff, and letting reconciliation restore the declared state.

Published · Updated

GitOpsArgoCDRollbackKubernetes

Problem

A deployment fails and the first response is to patch or roll back the live resource with kubectl.

Why it happens

Manual intervention can restore service quickly, but it also creates drift. Argo CD may later overwrite the emergency change because Git still describes the broken state.

Recommended approach

Revert the failing commit or restore the last known-good manifest, review the diff, and let Argo CD reconcile the cluster. If break-glass action is unavoidable, follow it immediately with a matching Git change.

Example workflow

bash

git revert <commit>
git push origin main
argocd app sync <app-name>

Use this when

  • GitOps-managed applications
  • Argo CD-managed clusters

Avoid this when

  • Emergency break-glass situations where immediate cluster-level action is required

Quick rollback workflow

First identify the application revision and the Git commit that introduced the failure:

bash

argocd app get <app-name>
argocd app history <app-name>
git log --oneline --decorate -n 10

Revert the change in Git instead of changing the live Deployment directly:

bash

git revert <bad-commit-sha>
git push origin main
argocd app diff <app-name>
argocd app sync <app-name>
argocd app wait <app-name> --health --sync

Use the branch and sync mechanism appropriate to the repository. If automated sync is enabled, the explicit argocd app sync command may not be necessary.

Why kubectl rollout undo is temporary under GitOps

kubectl rollout undo deployment/<name> changes the cluster but does not change the manifest, Helm values, or Kustomize source in Git. Argo CD still sees Git as the desired state and can reapply the revision that caused the incident.

That makes a live rollback useful only as a break-glass measure. Record the exact action, then create the matching Git revert immediately so the desired and live states converge again.

Verify the rollback

bash

argocd app get <app-name>
kubectl rollout status deployment/<deployment> -n <namespace>
kubectl get pods -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Confirm all of the following:

  • Argo CD reports the application as Synced
  • Application health returns to Healthy
  • The live image, configuration, and manifest revision match the reverted Git state
  • The original failure signal—health check, error rate, or unavailable endpoint—has recovered

Common rollback mistakes

  • Reverting a generated manifest without reverting its Helm or Kustomize source
  • Syncing before reviewing the diff
  • Rolling back the cluster while leaving the broken commit at the branch tip
  • Using argocd app rollback without understanding that Git may reconcile forward again
  • Reverting several unrelated commits when only one change caused the incident

References