How do health checks work in a load balancer?
The load balancer probes each server regularly, for example an HTTP request to /health. After several failures it stops sending traffic to that server, and after several successes it adds it back.
Explain it simply
Without health checks, a balancer would keep sending requests to a crashed server and users would see errors. Think of a canteen manager who checks each counter is staffed before sending students there.
Every few seconds the balancer sends a probe to each backend. It can be a TCP connect (is the port open?) or an HTTP call to an endpoint such as /health that expects a 200 response.
To avoid flapping, thresholds are used: mark unhealthy after, say, 3 failed probes; mark healthy again after 2 successful ones.
Design details that interviewers like:
- A shallow check only proves the process is up. A deep check also tests the database or cache connection.
- Do not make the deep check too strict: if a shared dependency is down, every server fails and the balancer removes them all.
- Allow connection draining so in-flight requests finish when a server is taken out for a deploy.
Example
Probe config:
interval: 5 s
unhealthy after: 3 consecutive failures
healthy after: 2 consecutive successes
t=0 probe fails (1)
t=5 probe fails (2)
t=10 probe fails (3) -> server removed from rotationWorst case, about 10 to 15 seconds of requests may still reach the dead server.
What to say in the interview
"The balancer sends periodic probes, usually HTTP to a health endpoint, and uses consecutive-failure and success thresholds to remove and restore servers. I keep the check meaningful but not dependent on every downstream service, and I enable connection draining for safe deploys."
Common mistakes
- Making the health endpoint depend on a shared database so one outage removes every server.