WSS
On this page
Infrastructure & ProvisioningRequiredUpdated

Load balancer health checks

Automated probes used by load balancers to determine if a backend server is capable of handling traffic, preventing requests from being routed to failed instances.

What it is

A load balancer sits in front of multiple backend servers and distributes incoming network traffic across them. A health check is a continuous, automated test performed by the load balancer against each of those backend servers. If a server fails the test, the load balancer marks it as “unhealthy” and stops sending traffic to it. When the server begins passing the test again, it is marked “healthy” and rejoins the active pool.

Why it matters

Servers fail constantly. Hardware degrades, memory leaks exhaust resources, and network cables disconnect. If a load balancer blindly distributed traffic across a pool of servers without checking their status, a percentage of user requests would invariably land on a dead server and fail.

Health checks guarantee that traffic is only routed to instances that are actively capable of serving it. They are the fundamental mechanism that allows distributed systems to achieve high availability and self-heal during localized outages.

How to implement

Network-Level (Layer 4) Checks

The simplest health check attempts to open a TCP connection to the server’s listening port. If the server completes the TCP handshake (SYN, SYN-ACK, ACK), the load balancer considers it healthy. This check is fast and has low overhead, but it is superficial. A server might successfully accept TCP connections while the application process behind it is deadlocked and unable to process actual requests.

Application-Level (Layer 7) Checks

Modern environments use HTTP health checks. The load balancer sends an HTTP GET request to a specific endpoint, commonly /health or /ping.

The server evaluates its own internal state - checking its connection to the database, verifying disk space, and ensuring background threads are running. If everything is functional, it returns an HTTP 200 OK status code. If any critical component is failing, it returns an HTTP 503 Service Unavailable status code. The load balancer parses this status code to determine the server’s health.

Configuration Parameters

Health checks require precise tuning to balance responsiveness against false positives:

  • Interval: How often the check runs (e.g., every 10 seconds).
  • Timeout: How long the load balancer waits for a response before considering it a failure (e.g., 5 seconds).
  • Healthy Threshold: How many consecutive successful checks are required to mark a newly booted server as healthy (e.g., 3 consecutive successes).
  • Unhealthy Threshold: How many consecutive failed checks are required to remove a server from rotation (e.g., 2 consecutive failures).

Common mistakes

Deep vs. Shallow Checks

A “shallow” check simply returns 200 OK as long as the web framework is running, without checking external dependencies. A “deep” check queries the database and external APIs before returning 200 OK. Deep checks are dangerous because a minor database blip might cause every single web server to fail its health check simultaneously. The load balancer will then mark the entire fleet as unhealthy, taking the entire application offline, even though the web servers are perfectly fine. Checks should only validate dependencies that are strictly required for the instance to function.

Thundering Herd

If you configure a health check interval of 1 second, and the check performs an expensive database query, a pool of 100 web servers will execute 100 database queries every single second just for health checks. This self-inflicted DDoS attack will crash the database. Health check endpoints must be aggressively cached and highly optimized.

Zombie Instances

If a server is marked unhealthy, the load balancer stops sending user traffic. If the underlying orchestrator (like an Auto Scaling Group) does not actively terminate the instance, the dead server will sit idle, consuming resources and costing money forever. Load balancer health checks must be linked to orchestration lifecycle hooks to terminate and replace failed instances.

Verification

To verify a health check endpoint, query it directly using curl and inspect the HTTP status code.

curl -i http://api.internal/health

The response should include a 200 OK status.

HTTP/1.1 200 OK
Content-Type: application/json

{
  "status": "pass",
  "version": "1.4.2"
}

Check the load balancer’s metrics dashboard. The number of healthy instances should exactly match the number of provisioned instances. If instances frequently flap between healthy and unhealthy states, increase the health check timeout or interval to accommodate minor latency spikes.

Related topics

Sources & further reading