Infra Notes

Kubernetes CrashLoopBackOff: debug the actual container failure

Diagnose Kubernetes CrashLoopBackOff by checking pod events, previous container logs, exit codes, probes, configuration, and resource limits in order.

Published

KubernetesCrashLoopBackOffkubectlPod DebuggingContainer

Problem

A Pod repeatedly starts, exits, and appears as CrashLoopBackOff. This status is the retry delay, not the root cause. The useful evidence is in the terminated container state, its previous logs, and Pod events.

Quick answer

Run these checks before restarting or editing the Deployment:

bash

kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> -c <container> --previous
kubectl get pod <pod> -n <namespace> \
  -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'

--previous matters because the current container may have already restarted with an empty log buffer. Record the exit code, reason, message, restart count, and recent events.

Interpret the exit reason

  • Exit Code: 1 usually means the application rejected its configuration or failed during startup.
  • Reason: OOMKilled means the container exceeded its memory limit or the node encountered memory pressure.
  • Exit Code: 126 or 127 points to an invalid command, missing executable, or permission problem.
  • Probe failures mean the process may be running but Kubernetes keeps restarting it because liveness checks fail.

Inspect the effective command and configuration rather than assuming the image is broken:

bash

kubectl get pod <pod> -n <namespace> -o yaml
kubectl get configmap <name> -n <namespace> -o yaml
kubectl get secret <name> -n <namespace> -o yaml

Do not print decoded Secret values into shared terminals, CI logs, or tickets.

Debugging order

  1. Read lastState.terminated and the previous logs.
  2. Inspect warning events at the bottom of kubectl describe pod.
  3. Compare the container command, arguments, environment variables, mounted files, and ports with a known-good release.
  4. Check liveness and startup probes. A startup probe can protect slow-starting applications from premature liveness failures.
  5. Compare requests and limits with actual memory use if the container was OOM-killed.
  6. Confirm dependencies such as DNS, databases, and external APIs are reachable from the namespace.

For a distroless image or a process that exits too quickly, use kubectl debug to create a debugging copy instead of modifying the production image merely to add a shell.

Avoid these shortcuts

  • Do not delete the Pod repeatedly; the Deployment recreates the same failure and removes useful evidence.
  • Do not remove probes permanently to make the Pod appear healthy.
  • Do not increase memory blindly without confirming OOMKilled and understanding normal working-set usage.
  • Do not replace the image tag until you know whether configuration or runtime dependencies caused the crash.

Prevention

Emit actionable startup errors, validate configuration before deployment, use immutable image tags, define realistic startup probes, and alert on restart-rate changes rather than only the current Pod phase.

For scheduling failures rather than restart loops, use the separate Kubernetes Pending Pod checklist. For monitoring gaps, see PodMonitor exists but Prometheus does not scrape.

References