Stackvora.tech

Health Checks That Actually Detect Failure

Load balancers rely on health checks to determine the availability and readiness of backend servers. Properly configured health checks are critical for…

Load balancers rely on health checks to determine the availability and readiness of backend servers. Properly configured health checks are critical for maintaining service availability and preventing traffic from being directed to unhealthy instances. This article explores strategies for designing robust health checks that accurately reflect the operational state of your application, distinguishing between superficial checks and those that genuinely detect failure, and discusses common pitfalls to avoid.

Implementing effective health checks goes beyond simple port availability; it requires a holistic view of the application's dependencies and internal state. We will cover various types of health checks, recommended configurations for common load balancers, and how to tune thresholds for optimal reliability.

Beyond Basic TCP Checks: Layer 7 and Dependency-Aware Health Checks

While a basic TCP port check (e.g., checking if port 80 or 443 is open) can confirm network reachability and that a service is listening, it doesn't guarantee the application itself is functioning correctly. A more comprehensive approach involves layer 7 (application layer) checks that exercise critical application paths and validate backend dependencies.

HTTP/HTTPS Health Checks

HTTP/HTTPS health checks are the most common and effective layer 7 checks. Instead of just checking if a port responds, these checks send an actual HTTP request to a defined endpoint and expect a specific HTTP status code (typically 200 OK) within a timeout period.

  • Dedicated Health Check Endpoint: Expose a specific URI (e.g., /healthz or /status) that your application can use to report its health. This endpoint should be lightweight but also perform critical internal checks.
  • Dependency Verification: Within this health check endpoint, your application should perform quick, non-blocking checks of its essential dependencies. This might include:
    • Pinging a database (e.g., executing a SELECT 1 query).
    • Checking connectivity to a caching layer (e.g., a GET from Redis or Memcached).
    • Verifying connectivity to message queues or external APIs that are critical for the application's core functionality.
    The health check endpoint should return a 200 OK only if all critical dependencies are met. If a critical dependency is down, it should return a 5xx status code (e.g., 503 Service Unavailable).
  • Example Endpoint Logic (Pseudocode): 
    function healthCheckEndpoint():
    try:
        db_connection = connect_to_database()
        db_connection.execute("SELECT 1")
        redis_client = connect_to_redis()
        redis_client.ping()
        # Add checks for other critical dependencies
        return HTTP 200 OK
    except Exception as e:
        log_error("Health check failed due to dependency issue: " + str(e))
        return HTTP 503 Service Unavailable

Deep Checks vs. Shallow Checks

A trade-off exists between the depth of a health check and its performance impact. A "deep" check might simulate a user transaction, while a "shallow" check only validates immediate dependencies. For load balancing, shallow, dependency-aware checks are generally preferred:

  • Shallow, Dependency-Aware: Fast, non-blocking, and checks only critical dependencies. It should return a 5xx if a critical dependency (e.g., primary database) is down. This is the recommended approach for load balancer health checks.
  • Deep Checks (Synthetic Transactions): More comprehensive but often too slow or resource-intensive for frequent load balancer checks. These are better suited for external monitoring systems (e.g., Prometheus, Datadog) that report on application performance and end-to-end user experience, rather than for immediate load balancer traffic shifting.

Configuring Health Check Parameters

Load balancers typically offer several parameters to tune health check behavior. Careful configuration prevents "flapping" (rapid state changes) and ensures timely detection of failures.

Thresholds: Healthy and Unhealthy Counts

These parameters define how many consecutive successful or failed checks are required to change an instance's state.

  • Unhealthy Threshold (Failure Count): The number of consecutive failed health checks after which an instance is marked as unhealthy. A value of 2 or 3 is common. Setting this too low (e.g., 1) can lead to flapping due to transient network issues or brief application hiccups.
  • Healthy Threshold (Success Count): The number of consecutive successful health checks after which an instance is marked as healthy. A value of 2 or 3 is also common here. This prevents an instance from being immediately brought back into service after a single success, ensuring it's genuinely stable.

Intervals and Timeouts

These parameters control the frequency and duration of health checks.

  • Interval: The time in seconds between consecutive health checks for a given instance. A common range is 5-30 seconds. Shorter intervals detect failures faster but increase load on the backend and load balancer.
  • Timeout: The maximum time in seconds the load balancer will wait for a response from the health check endpoint. If no response is received within this time, the check is considered a failure. This should be slightly less than the interval to allow for retries within the interval, or ideally, set based on the expected response time of your health check endpoint, with a small buffer. For a simple dependency-aware HTTP check, 2-5 seconds is often sufficient.

Example Configuration Snippets

HAProxy (version 2.x)

For a backend pool named webservers:

backend webservers
    balance roundrobin
    option httpchk GET /healthz
    http-check expect status 200
    default-server inter 5s fall 3 rise 2 timeout 3s

    server srv1 192.168.1.10:80 check
    server srv2 192.168.1.11:80 check
    # ... more servers
  • option httpchk GET /healthz: Specifies an HTTP GET request to /healthz for health checks.
  • http-check expect status 200: Expects an HTTP 200 OK status code.
  • inter 5s: Health check interval of 5 seconds.
  • fall 3: Mark server down after 3 consecutive failures.
  • rise 2: Mark server up after 2 consecutive successes.
  • timeout 3s: Health check timeout of 3 seconds.

AWS Application Load Balancer (ALB)

Configured via console or AWS CLI for a Target Group:

aws elbv2 create-target-group \
    --name my-app-tg \
    --protocol HTTP \
    --port 80 \
    --vpc-id vpc-xxxxxxxxxxxxxxxxx \
    --health-check-protocol HTTP \
    --health-check-port 80 \
    --health-check-path /healthz \
    --health-check-interval-seconds 10 \
    --health-check-timeout-seconds 5 \
    --healthy-threshold-count 3 \
    --unhealthy-threshold-count 2 \
    --matcher HttpCode=200
  • --health-check-path /healthz: The specific URI for the health check.
  • --health-check-interval-seconds 10: Check every 10 seconds.
  • --health-check-timeout-seconds 5: 5-second timeout for each check.
  • --healthy-threshold-count 3: 3 consecutive successes to be healthy.
  • --unhealthy-threshold-count 2: 2 consecutive failures to be unhealthy.
  • --matcher HttpCode=200: Expects an HTTP 200 response.

Handling Optional Dependencies

Not all application dependencies are equally critical. If a caching layer or an analytics service goes down, the core application might still be functional, albeit with degraded performance or features. It's crucial not to mark the entire application as unhealthy if an optional dependency fails, as this can lead to unnecessary service outages.

For optional dependencies, your health check endpoint should:

  • Gracefully degrade: The application should be designed to operate, potentially with reduced functionality, if an optional dependency is unavailable.
  • Log, but don't fail: The health check endpoint should log errors for optional dependencies but still return an HTTP 200 OK if the core functionality and critical dependencies are operational. This allows the load balancer to keep sending traffic while monitoring alerts are triggered for the optional dependency.
  • Expose separate metrics: Use internal monitoring (e.g., Prometheus metrics) to report the status of optional dependencies, rather than relying on the load balancer's health check.

A simple way to manage this in your health check endpoint is to differentiate checks:

function healthCheckEndpoint():
    all_critical_ok = true
    
    # Critical dependency checks (e.g., primary DB)
    try:
        db_connection = connect_to_database()
        db_connection.execute("SELECT 1")
    except Exception as e:
        log_error("Critical DB check failed: " + str(e))
        all_critical_ok = false

    # Optional dependency checks (e.g., secondary cache)
    try:
        cache_client = connect_to_cache()
        cache_client.get("some_key")
    except Exception as e:
        log_warning("Optional cache check failed: " + str(e))
        # Do NOT set all_critical_ok to false for optional dependencies

    if all_critical_ok:
        return HTTP 200 OK
    else:
        return HTTP 503 Service Unavailable

Common Pitfalls and Troubleshooting

  • Overly Aggressive Thresholds: Setting unhealthy-threshold-count to 1 or health-check-interval-seconds too low (e.g., 1 second) can cause instances to flap in and out of service due to minor network blips or temporary resource spikes, leading to connection resets for users. Aim for at least 2-3 failures and 5-10 second intervals.
  • Too Lenient Timeouts: A health check timeout that's too long (e.g., 30 seconds for a simple check) will delay detection of an unresponsive service. Tune this to be slightly longer than the expected worst-case response time of your health endpoint.
  • Health Check Endpoint Contention: If your /healthz endpoint itself performs heavy operations or acquires locks, it can become a bottleneck, potentially causing the health check to fail even when the application is healthy. Ensure the endpoint is lightweight and non-blocking.
  • Ignoring Load Balancer Logs: Load balancers provide detailed logs on health check failures. Regularly review these logs (e.g., ALB access logs, HAProxy logs) to identify patterns in failures, such as specific error codes or timeouts, which can indicate underlying application or infrastructure issues.
  • Lack of Alerting: While load balancer health checks handle traffic routing, you still need separate monitoring and alerting for instances marked unhealthy, especially if they stay unhealthy for extended periods. Integrate with your monitoring systems (e.g., PagerDuty, Slack) to notify on persistent failures.
  • Security Concerns with /healthz: While /healthz is typically public, ensure it doesn't expose sensitive information. If security is a major concern, consider using a separate, internal load balancer or network ACLs to restrict access to health check endpoints, though this adds complexity.

Back to the knowledge base · Ask the AI assistant