Diagnose a CrashLoopBackOff Pod

Pods entering a CrashLoopBackOff state indicate a recurring problem preventing one or more containers within the Pod from starting successfully. This…

Pods entering a CrashLoopBackOff state indicate a recurring problem preventing one or more containers within the Pod from starting successfully. This isn't an issue with Kubernetes itself, but rather a symptom of an underlying application or configuration error. Understanding the diagnostic process for these failures is critical for maintaining stable Kubernetes deployments.

This article outlines a systematic approach to identifying the root cause of CrashLoopBackOff, leveraging standard kubectl commands and exploring common failure patterns.

Understanding CrashLoopBackOff

When a container within a Pod terminates, Kubernetes attempts to restart it according to its restartPolicy. If the container repeatedly terminates shortly after starting, Kubernetes introduces an exponential back-off delay between restart attempts. This behavior is what's signaled by the CrashLoopBackOff status.

The Pod status will cycle through phases like ContainerCreating, Running (briefly), Terminating, and then CrashLoopBackOff. The back-off delay increases with each failed restart, up to a maximum (typically 5 minutes for most Kubernetes versions), to prevent resource exhaustion from rapid, consecutive restarts.

Initial Triage with kubectl get events

Before diving into individual Pod details, a quick check of Kubernetes events can often reveal high-level issues or recent changes that might be impacting your deployment. Events provide a chronological log of actions and states within your cluster.

kubectl get events --field-selector involvedObject.name=<pod-name> --sort-by='.lastTimestamp'

Replace <pod-name> with the name of your failing Pod. Look for events related to FailedSync, FailedScheduling, FailedMount, or any BackOff events that explicitly mention why a container might not be starting. For example, you might see:

  • Failed to pull image "my-private-registry/my-app:v1.0": Indicates an image pull secret issue or incorrect image name.
  • Error: ImagePullBackOff: Similar to above, but often indicates transient network issues or invalid image references.
  • Liveness probe failed: HTTP GET http://...: Suggests the application started but failed its health check.

Detailed Pod Analysis with kubectl describe pod

The kubectl describe pod command provides a comprehensive overview of a Pod's configuration, status, events, and container states. This is your primary tool for gathering specific details about the crash.

kubectl describe pod <pod-name> -n <namespace>

Key sections to examine:

Containers Status

Scroll down to the Containers: section. For the crashing container, you'll see details like:

  • State: Waiting or State: Terminated
  • Reason: CrashLoopBackOff (if Waiting) or Reason: Error (if Terminated)
  • Last State: Terminated
    • Reason: Error or Reason: Completed (if a job completed with an error)
    • Exit Code: <number>: This is crucial. Common exit codes include:
      • 0: Successful termination (unlikely for a crash).
      • 1: General application error or unhandled exception.
      • 127: Command not found (e.g., wrong entrypoint in Dockerfile).
      • 137: Container received SIGKILL (often OOMKilled or manual kill).
      • 139: Segmentation fault (SIGSEGV).
      • 143: Container received SIGTERM (graceful shutdown, but could indicate a problem if unexpected).
    • Started: ... and Finished: ...: These timestamps help determine how long the container ran before crashing.

Image Pull Errors

Also within kubectl describe pod, under the Events: section at the bottom, look for messages indicating problems pulling the container image. Examples:

  • Failed to pull image "my-app:nonexistent": Image does not exist in the registry.
  • Failed to pull image "my-private-registry/my-app:v1.0": rpc error: code = Unknown desc = Error response from daemon: unauthorized: authentication required: Issue with image pull secrets.
  • ErrImagePull, ImagePullBackOff: Generic image pull failures, often due to network issues or incorrect image name/tag.

Verify the image name and tag are correct, and that any required imagePullSecrets are properly configured and referenced in the Pod's YAML.

Accessing Container Logs with kubectl logs

The most direct way to understand why a container crashed is to examine its logs. Since the Pod is in a CrashLoopBackOff, the current container instance may not be running or might have very sparse logs. You need to retrieve the logs from the previous failed instance.

kubectl logs <pod-name> -n <namespace> --previous

If your Pod has multiple containers, specify the container name:

kubectl logs <pod-name> -n <namespace> --previous -c <container-name>

This command outputs the standard output and standard error streams from the last terminated container instance. Look for:

  • Application-specific error messages (e.g., "Cannot connect to database," "Configuration file not found," "NullPointerException").
  • Stack traces, especially for Java, Python, or Node.js applications.
  • Information about dependencies that failed to initialize.
  • Messages indicating failed health checks or unhandled exceptions during startup.

What if logs are empty or unhelpful?

If --previous yields no logs or generic "container terminated" messages, the issue might be happening even before the application writes to standard output/error. Consider these possibilities:

1. Liveness/Readiness Probe Failures

If your application starts but fails its liveness probe, Kubernetes will restart it. The logs from the application itself might not show a crash, but rather that the HTTP endpoint or command checked by the liveness probe isn't responding correctly.

# Example Pod spec snippet with probes
apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
spec:
  containers:
  - name: my-app
    image: my-app:v1.0
    ports:
    - containerPort: 8080
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5

Check the livenessProbe and readinessProbe configuration in the Pod's YAML. Does the path exist? Is the application listening on the specified port? Is it giving an expected HTTP status code (e.g., 200 OK)? Sometimes, probes are configured to check a resource that isn't yet available, causing premature restarts.

2. Resource Limits Exceeded (OOMKilled)

If a container attempts to use more memory than specified in its resources.limits.memory, the Kubernetes scheduler (via the Kubelet) will terminate it with an Out-Of-Memory (OOM) error. The exit code for an OOMKill is typically 137 (SIGKILL). You'll see this in kubectl describe pod:

State:          Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      ...
  Finished:     ...

To verify OOMKills, also check the node's system logs (e.g., journalctl -u kubelet on Linux) for OOM killer messages referencing your container. Adjust your Pod's memory limits upwards or optimize your application's memory usage.

# Example Pod spec snippet with resource limits
apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
spec:
  containers:
  - name: my-app
    image: my-app:v1.0
    resources:
      limits:
        memory: "256Mi"
        cpu: "500m"
      requests:
        memory: "128Mi"
        cpu: "250m"

3. Volume Mount Issues

If your application depends on specific files or directories from a mounted volume, and that volume fails to mount or is misconfigured, the application might crash at startup without clear logs. Examine the Events: section of kubectl describe pod for FailedMount or related errors.

Verify that the PersistentVolumeClaim (PVC) exists, is bound to a PersistentVolume (PV), and that the storage class is correctly configured and provisioned.

4. Entrypoint/Command Errors

The command and args defined in your container spec (or the Dockerfile's ENTRYPOINT and CMD) might be incorrect. This could lead to an "executable not found" error (exit code 127) or incorrect command syntax.

To debug, try running the command directly in a similar base image: 

docker run --rm <your-image> ls -l /app
This helps verify if files exist or the path is correct. You can also temporarily override the entrypoint to a shell to inspect the environment:

# Temporary change in Pod spec for debugging
apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
spec:
  containers:
  - name: my-app
    image: my-app:v1.0
    command: ["/bin/sh"]
    args: ["-c", "while true; do sleep 3600; done"] # Keep container running
    # ... other configurations

Once the Pod is running with this override, you can kubectl exec -it <pod-name> -- /bin/sh to investigate the container's filesystem and environment.

Common Pitfalls

  • Ignoring exit codes: Always check the Exit Code in kubectl describe pod. It's often the quickest indicator of the problem type.
  • Forgetting --previous with kubectl logs: This is the single most common mistake when debugging CrashLoopBackOff.
  • Incorrect image pull secrets: If your image is in a private registry, ensure imagePullSecrets are correctly configured and referenced in the Pod spec.
  • Misconfigured probes: Liveness probes that are too aggressive or check an endpoint that isn't ready can cause an application that's otherwise fine to restart repeatedly.
  • Insufficient resource limits: OOMKilled containers often have misleadingly sparse logs because the kernel terminated them forcefully. Always suspect resource exhaustion if logs are inconclusive and the exit code is 137.
  • Environmental dependencies not met: Applications crashing due to missing environment variables, unavailable databases, or external services failing to connect will produce logs, but they might be subtle.

Back to the knowledge base · Ask the AI assistant