Healthchecks That Actually Restart Broken Containers

Reliable containerized applications demand more than just processes staying alive. A process might be running, but the application within could be…

Reliable containerized applications demand more than just processes staying alive. A process might be running, but the application within could be deadlocked, unresponsive, or serving stale data. Docker's HEALTHCHECK instruction and orchestration platforms leverage this to detect and react to genuinely unhealthy services. This article dives into configuring robust healthchecks to ensure your containers are not just alive, but truly available and performing as expected, ultimately leading to automated restarts of non-functional instances.

The core concept is to define a check that accurately reflects the application's operational state. When this check fails consistently, the Docker daemon or orchestrator can be configured to take corrective action, typically restarting the container. This moves beyond simple process monitoring to application-level health awareness.

Defining Effective Healthchecks in Docker

The Docker HEALTHCHECK instruction in a Dockerfile specifies how to test a container to check if it's still working. This check can be a command that returns a zero exit code for success or a non-zero exit code for failure. It's crucial to make this command lightweight and fast, as it will be executed frequently.

Dockerfile Syntax and Parameters

The basic syntax for a HEALTHCHECK instruction is:

HEALTHCHECK [OPTIONS] CMD command

Key options include:

  • --interval=DURATION: How often to run the check (default: 30s).
  • --timeout=DURATION: Maximum time the check can take before being considered failed (default: 30s).
  • --start-period=DURATION: Initial grace period for containers to bootstrap. During this period, failures won't count towards the unhealthy threshold (default: 0s, Docker Engine 1.12+).
  • --retries=N: Number of consecutive failures before the container is marked unhealthy (default: 3).
  • CMD command: The actual command to execute.

For a web service, a common healthcheck involves an HTTP request:

# Dockerfile snippet
FROM nginx:1.24-alpine

# ... other instructions ...

# Healthcheck for Nginx, checking if it can serve a basic page
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
    CMD wget --quiet --tries=1 --spider http://localhost/ || exit 1

EXPOSE 80

In this example, wget --quiet --tries=1 --spider http://localhost/ attempts to fetch the root page without downloading it (--spider). If the HTTP status is 200-399, it returns 0. Any other status or connection error will result in a non-zero exit code. The || exit 1 ensures an explicit failure if wget itself fails.

For database containers, you might check connectivity or even run a simple query:

# Dockerfile snippet for PostgreSQL
FROM postgres:16.1-alpine

# ... other instructions ...

HEALTHCHECK --interval=10s --timeout=5s --start-period=30s --retries=5 \
    CMD pg_isready -U $POSTGRES_USER -h localhost -p 5432 || exit 1

pg_isready is a utility provided by PostgreSQL to check the connection status of a PostgreSQL server. It's a perfect lightweight choice for a healthcheck.

Testing and Monitoring Healthchecks

After building and running a container with a healthcheck, you can monitor its status:

docker ps

The output will show the health status in the "STATUS" column (e.g., (healthy), (unhealthy), (health: starting)).

For more detailed information, use docker inspect:

docker inspect --format='{{json .State.Health}}' my_container_name_or_id

This will show the health status, consecutive failures, and a log of the healthcheck command's output.

Automated Restarts with Docker Compose

While the Docker daemon marks containers as healthy or unhealthy, it doesn't automatically restart them by default solely based on healthcheck status. For this, you typically rely on orchestration tools. However, Docker Compose offers a simple way to combine healthchecks with restart policies.

Configuring Restart Policies in Compose

In a docker-compose.yml file, you define the healthcheck section under a service and then set a restart policy. The restart: unless-stopped (or always) directive tells Docker to restart the container if it stops for any reason, including an unhealthy status that leads to a container exit (which typically requires a separate mechanism or an orchestrator to explicitly stop the container).

Crucially, for a healthcheck failure to trigger a restart in a standalone Docker Compose setup (without Swarm mode enabled), you need an external mechanism or to configure the application itself to exit if it fails its own internal health checks. The Docker daemon itself doesn't stop a container purely because its healthcheck fails, it only updates its status. Orchestrators use this status to act.

However, if your application within the container is configured to terminate when it detects an unrecoverable internal error (which often aligns with a healthcheck failure condition), then the restart policy will kick in.

# docker-compose.yml
version: '3.8'
services:
  web:
    image: my-web-app:latest
    ports:
      - "80:80"
    restart: unless-stopped # Or 'always'
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost/health"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 20s
    environment:
      # Assuming the app exposes /health endpoint
      APP_HEALTHCHECK_URL: http://localhost/health
    # A hypothetical application might internally monitor its state and exit
    # if it becomes unrecoverable, which restart: unless-stopped would then handle.

For Docker Compose environments where the Docker daemon explicitly stops unhealthy containers, you would need Docker Swarm mode enabled (even for a single-node swarm) or a custom script. In Swarm, the orchestrator actively uses healthcheck status to manage services.

Orchestrators and Health-Driven Restarts

Orchestration platforms like Docker Swarm and Kubernetes fully integrate healthcheck status into their service management. They actively monitor health, and if a container fails its checks consistently, the orchestrator will automatically stop the unhealthy instance and schedule a new one.

Docker Swarm

In Docker Swarm, the deploy section of a service in docker-compose.yml (or docker stack deploy) allows you to specify healthcheck parameters directly, which Swarm then uses to manage service lifecycle.

# docker-compose.yml for Swarm
version: '3.8'
services:
  frontend:
    image: myorg/myfrontend:1.0
    ports:
      - "8080:80"
    deploy:
      replicas: 3
      placement:
        constraints: [node.role == worker]
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure # Important: 'on-failure' or 'any' combined with healthcheck
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:80/ping"]
      interval: 5s
      timeout: 3s
      retries: 3
      start_period: 15s

Swarm will actively restart tasks (containers) that become unhealthy according to these definitions. The restart_policy.condition: on-failure ensures that if the healthcheck fails enough times to mark the container as unhealthy, Swarm will eventually stop it (causing an exit code, which then triggers the restart policy).

Kubernetes Liveness and Readiness Probes

Kubernetes uses a more sophisticated two-pronged approach: liveness probes and readiness probes. Both are crucial for robust application management.

  • Liveness Probe: Determines if the container is running and healthy enough to continue serving. If a liveness probe fails, Kubernetes will restart the container. This is analogous to Docker's healthcheck for restart purposes.
  • Readiness Probe: Determines if the container is ready to serve traffic. If a readiness probe fails, Kubernetes will remove the Pod's IP address from the Endpoints of all associated Services. This means no traffic will be routed to the Pod until the probe succeeds again. It does NOT restart the container.

Types of probes:

  • exec: Executes a command inside the container.
  • httpGet: Performs an HTTP GET request.
  • tcpSocket: Attempts to open a TCP socket.

Example Kubernetes Deployment with both probes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-web-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-web-app
  template:
    metadata:
      labels:
        app: my-web-app
    spec:
      containers:
      - name: web
        image: myorg/myfrontend:1.0
        ports:
        - containerPort: 80
        livenessProbe:
          httpGet:
            path: /healthz
            port: 80
          initialDelaySeconds: 15 # Wait 15s before first check
          periodSeconds: 10      # Check every 10s
          timeoutSeconds: 5      # 5s timeout for the check
          failureThreshold: 3    # 3 failures to restart
        readinessProbe:
          httpGet:
            path: /ready
            port: 80
          initialDelaySeconds: 5 # Wait 5s before first check
          periodSeconds: 5       # Check every 5s
          timeoutSeconds: 2      # 2s timeout for the check
          failureThreshold: 2    # 2 failures to mark unready

In this setup, /healthz should be a very lightweight check to ensure the application process is running and responding minimally. /ready might perform deeper checks, such as database connectivity or external service availability, to determine if it's truly ready to handle requests.

Designing Healthcheck Endpoints

The endpoint for a healthcheck should be:

  • Fast: It runs frequently, so it shouldn't add significant load.
  • Lightweight: Avoid complex database queries or external API calls if possible for a basic liveness check.
  • Deterministic: It should return a reliable status.
  • Specific: A generic "200 OK" from a web server might only mean Nginx is running, not your application behind it. Ideally, the application itself exposes an endpoint that checks its internal state.

A good strategy is to have multiple healthcheck endpoints:

  1. Liveness (/healthz): A very simple check (e.g., return 200 OK if the application process is alive and responsive). This determines if the container needs a restart.
  2. Readiness (/ready): A more thorough check that might verify database connectivity, message queue access, or other critical dependencies. This determines if the container should receive traffic.
  3. Startup (/startup, Kubernetes 1.18+): If your application has a slow startup, a startup probe can defer liveness and readiness checks until the application is truly initialized.

Common Pitfalls

  • Overly Complex Checks: A healthcheck that itself consumes too many resources or takes too long can cause false negatives or overload the application during critical periods. Keep liveness checks very simple.
  • Ignoring start-period (Docker) or initialDelaySeconds (Kubernetes): Without a grace period, containers with slow startup times might be prematurely marked unhealthy and restarted in a loop.
  • Using the Same Endpoint for Liveness and Readiness: While convenient, this often leads to poor behavior. If your database goes down, a combined probe will cause restarts (liveness failure) instead of just routing traffic away (readiness failure), potentially exacerbating issues.
  • Dependency on External Services: A healthcheck that depends on an external service (e.g., a third-party API) will fail if that service is down, leading to unnecessary restarts of your own application. Only critical, unrecoverable internal dependencies should cause a liveness failure.
  • Missing || exit 1 in Dockerfile CMD: The healthcheck command must return a non-zero exit code on failure for Docker to mark it unhealthy. Commands like curl -f return non-zero on HTTP errors, but other commands might need explicit || exit 1.

Back to the knowledge base · Ask the AI assistant