Health checks and failover

A health check finds a dead server before users do, and failover moves its work somewhere else.

Use it when#

Several copies of a service sit behind a load balancer, and one can die without anyone noticing.
A single data store has a replacement waiting in another zone.
Slow detection is costing errors, and you can see them in the gap between a failure and the first healthy answer.

How it works#

The load balancer calls an endpoint on every server at a fixed interval. A server that fails stays out of rotation until it passes again. A shallow check asks only whether the process is up. A deep check also asks whether its database answers, which catches a server that is alive but useless. The interval sets the cost of a failure: a 30-second interval leaves a dead server in rotation for up to 30 seconds. Failover for data works the same way, with a standby that takes over when the primary stops responding.

What it costs#

A short interval adds probe load, and a deep check can take every server out at once when the shared database blips. A standby is a second machine you pay for and rarely use, and a failover that has never been tested often does not work. Keep checks cheap and make failure of the check itself safe.

In Quorum#

Drag to pan. Hold Ctrl or Cmd and scroll to zoom.

At 3.00k Users the Simulation solves this Design to Apdex 1.00, p99 248 ms, 0.0% errors and $2.72/s Burn.

Where	Setting	Value	Why
Load Balancer	Health check interval	2	A dead App Server stays in rotation until the next check, so a short interval shortens the errors.
Load Balancer	Deep health check	On	The check also asks the Database, so an App Server that cannot reach it leaves rotation too.
App Server	Instance size	small	Each App Server can carry the load alone if its twin dies.
App Server	Zone	a	One zone failing takes only one of the two App Servers.
App Server	Instance size	small	Each App Server can carry the load alone if its twin dies.
App Server	Zone	b	The twin sits in a different zone from the first.
Database	Instance size	medium	Sized for the whole load.
Database	Standby	On	A second box in another zone takes over if the Database zone fails, for double the Burn.

Research this Design needs: Load balancing, Health checks, Multiple zones.

Two settings on the Load Balancer shorten the gap. Health check interval sets how soon it notices a dead server, and Deep health check also removes App Servers whose Database is down or overloaded. Both need Health checks, and the deep check also needs the Health check endpoint task done by the team. The two App Servers sit in different Zones, and the Database has a Standby, which takes over within seconds when its zone fails and doubles that Node's Burn. Zones and standbys need Multiple zones. The Zone outage incident tests this Design. The glossary defines health check and availability zone.