{"id":1237,"date":"2026-11-22T11:12:29","date_gmt":"2026-11-22T11:12:29","guid":{"rendered":"https:\/\/ictbank.ir\/blog\/?p=1237"},"modified":"2026-11-22T11:12:32","modified_gmt":"2026-11-22T11:12:32","slug":"load-balancer-health-checks-how-they-work-and-how-to-choose-the-right-one","status":"publish","type":"post","link":"https:\/\/ictbank.ir\/blog\/load-balancer-health-checks-how-they-work-and-how-to-choose-the-right-one\/","title":{"rendered":"Load Balancer Health Checks: How They Work and How to Choose the Right One"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">A load balancer is only as good as its knowledge of which backend servers are actually able to serve traffic. Send a request to a dead server and the user sees an error. Keep sending requests to a half-dead one and you get timeouts that are far harder to diagnose. Health checks are the mechanism that gives the load balancer this knowledge. In this article we look at how they work, the main types, their trade-offs, and how to choose between them.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What Is a Health Check?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A health check is a probe the load balancer runs against each backend server to decide whether it should receive traffic. If the probe succeeds, the server stays in the pool. If it fails, the server is marked unhealthy and removed from rotation until it recovers.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This is what makes high availability practical: a failed node is taken out automatically, without a human noticing at 3 a.m.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The Core Parameters<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Almost every load balancer, whether HAProxy, NGINX, F5, AWS ELB or a cloud-native service, exposes the same handful of settings:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Interval:<\/strong> how often the probe runs (for example, every 5 seconds).<\/li>\n\n\n\n<li><strong>Timeout:<\/strong> how long the load balancer waits for a response before counting the probe as failed.<\/li>\n\n\n\n<li><strong>Unhealthy threshold:<\/strong> how many consecutive failures are needed before a server is removed.<\/li>\n\n\n\n<li><strong>Healthy threshold:<\/strong> how many consecutive successes are needed before a server is added back.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">These four values determine how quickly you detect a failure and how likely you are to react to a false alarm. Detection time is roughly <em>interval \u00d7 unhealthy threshold<\/em>, so a 5-second interval with a threshold of 3 means a failure can take up to about 15 seconds to be noticed.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Types of Health Checks<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">1. TCP (Layer 4) Checks<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The load balancer simply tries to open a TCP connection to the server&#8217;s port. If the handshake completes, the server is considered healthy.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Strengths:<\/strong> very cheap, protocol-agnostic, works for databases, message brokers and any TCP service.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Weakness:<\/strong> it only proves that something is listening on the port. An application that has hung, lost its database connection or is returning errors on every request will still pass.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">2. HTTP\/HTTPS (Layer 7) Checks<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The load balancer sends an HTTP request, typically <code>GET \/health<\/code>, and checks the response status code (and sometimes the body or a header).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Strengths:<\/strong> confirms the web server and application process are responding, and can validate TLS.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Weakness:<\/strong> a shallow <code>\/health<\/code> endpoint that always returns <code>200 OK<\/code> tells you the process is alive, not that the service is useful.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">3. Deep (Application-Level) Checks<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The health endpoint itself exercises the service&#8217;s dependencies: it runs a trivial query against the database, pings the cache, or checks that a downstream API is reachable. A 200 is returned only if the service can really do its job.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Strengths:<\/strong> the most accurate picture of whether the server can serve real requests.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Weakness:<\/strong> the most expensive and the most dangerous (see the pitfalls below). A slow or failing dependency can make every server look unhealthy at the same time.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">4. Passive Checks<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Instead of sending probes, the load balancer watches real traffic. If a server returns errors or times out on live requests, it is marked unhealthy. This is sometimes called outlier detection.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Strengths:<\/strong> no extra load, reflects what users actually experience.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Weakness:<\/strong> real users hit the failure before it is detected, and a server that receives no traffic can&#8217;t be evaluated. Passive checks are best combined with active ones.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Comparing the Methods<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><th>Method<\/th><th>Accuracy<\/th><th>Cost<\/th><th>Main risk<\/th><\/tr><tr><td>TCP<\/td><td>Low<\/td><td>Very low<\/td><td>Misses application-level failures<\/td><\/tr><tr><td>HTTP\/HTTPS<\/td><td>Medium<\/td><td>Low<\/td><td>Shallow endpoint gives false confidence<\/td><\/tr><tr><td>Deep check<\/td><td>High<\/td><td>Medium to high<\/td><td>Cascading removal when a shared dependency fails<\/td><\/tr><tr><td>Passive<\/td><td>Medium<\/td><td>None<\/td><td>Users experience the failure first<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Common Pitfalls<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>1. Cascading failure from deep checks.<\/strong> Imagine every web server&#8217;s health check queries the same database. The database slows down, all health checks time out, and the load balancer removes every server at once. A partial slowdown has become a full outage. Deep checks should be used with care, and many teams deliberately avoid checking <em>shared<\/em> dependencies in the check that controls pool membership.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>2. Overly aggressive settings.<\/strong> A 1-second interval with a threshold of 1 will react to every network blip. Servers flap in and out of the pool, which can be worse than a short outage.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>3. Overly relaxed settings.<\/strong> A 60-second interval with a threshold of 5 means five minutes of sending users to a dead server.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>4. A health endpoint that lies.<\/strong> If <code>\/health<\/code> is served by a different code path than real traffic (for example, a static response from the web server), it can stay green while the application is broken.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>5. Probes that are too heavy.<\/strong> A health check that performs an expensive query, multiplied by several load balancer nodes and a short interval, can become a meaningful load on its own.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>6. No graceful handling of recovery.<\/strong> A server that has just restarted may pass its health check before caches are warm or connection pools are filled. Slow-start and connection draining features exist to smooth this out.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Practical Recommendations<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Match the check to the service.<\/strong> A database proxy may be fine with a TCP check; a customer-facing API likely deserves an HTTP check against a meaningful endpoint.<\/li>\n\n\n\n<li><strong>Separate liveness from readiness.<\/strong> Liveness answers &#8220;is the process alive?&#8221; and readiness answers &#8220;can it take traffic right now?&#8221; Using both avoids restarting servers that are merely busy.<\/li>\n\n\n\n<li><strong>Keep the check fast and lightweight,<\/strong> and set the timeout well below the interval.<\/li>\n\n\n\n<li><strong>Tune thresholds deliberately.<\/strong> Choose them based on how much detection delay and how many false positives your service can tolerate.<\/li>\n\n\n\n<li><strong>Combine active and passive checks<\/strong> where your load balancer supports it.<\/li>\n\n\n\n<li><strong>Monitor the health checks themselves.<\/strong> Alert on servers flapping and on the number of healthy backends dropping below a safe minimum.<\/li>\n\n\n\n<li><strong>Test failure scenarios.<\/strong> Stop a service, block a port, slow a dependency, and verify the load balancer behaves as you expect.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Health checks are small in configuration but large in consequence. The methods differ in accuracy, cost and risk, and the deepest method is not automatically the best one. A deep check is more accurate but costs more and can cause cascading failures, while a simple TCP check is cheap but blind to most real problems. The right choice depends on what the service actually needs. Start from the failure modes you care about, pick the simplest check that catches them, and tune it so that it protects your users without overreacting.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A load balancer is only as good as its knowledge of which backend servers are actually able to serve traffic. Send a request to a dead server and the user sees an error. Keep sending requests to a half-dead one and you get timeouts that are far harder to diagnose. Health checks are the mechanism [&hellip;]<\/p>\n","protected":false},"author":24,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[50],"tags":[],"class_list":["post-1237","post","type-post","status-publish","format-standard","hentry","category-os"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/ictbank.ir\/blog\/wp-json\/wp\/v2\/posts\/1237","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ictbank.ir\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ictbank.ir\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ictbank.ir\/blog\/wp-json\/wp\/v2\/users\/24"}],"replies":[{"embeddable":true,"href":"https:\/\/ictbank.ir\/blog\/wp-json\/wp\/v2\/comments?post=1237"}],"version-history":[{"count":1,"href":"https:\/\/ictbank.ir\/blog\/wp-json\/wp\/v2\/posts\/1237\/revisions"}],"predecessor-version":[{"id":1238,"href":"https:\/\/ictbank.ir\/blog\/wp-json\/wp\/v2\/posts\/1237\/revisions\/1238"}],"wp:attachment":[{"href":"https:\/\/ictbank.ir\/blog\/wp-json\/wp\/v2\/media?parent=1237"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ictbank.ir\/blog\/wp-json\/wp\/v2\/categories?post=1237"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ictbank.ir\/blog\/wp-json\/wp\/v2\/tags?post=1237"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}