Drop dead nodes with health checks
Have the Load Balancer probe each server and stop sending traffic to one that has failed, so a dead host costs seconds instead of a share of requests.
Tighten the Load Balancer's health check so a failed App Server leaves the rotation within seconds.
Move up when#
An App Server dies and a fixed share of requests fail until someone notices.
The Load Balancer still lists a host that no longer answers.
The Design#
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| Load Balancer | Health check interval (s) | 30 | The Load Balancer looks at each server this rarely, so a host that died moments ago still gets its share. |
| App Server | Instance size | small | Three equal servers. |
| App Server | Instance size | small | Three equal servers. |
| App Server | State | Down | This host died ten seconds before the picture. The Design shows what the Load Balancer does in those seconds. |
| App Server | Instance size | small | Three equal servers. |
| Database | Instance size | medium | Medium, shared by all three. |
Research this Design needs: Load balancing, Health checks.
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| Load Balancer | Health check interval (s) | 2 | The Load Balancer takes the dead host out of rotation at its next look, a couple of seconds after it fails. |
| App Server | Instance size | small | Three equal servers. |
| App Server | Instance size | small | Three equal servers. |
| App Server | State | Down | The same host, down for the same ten seconds as in the Design before. |
| App Server | Instance size | small | Three equal servers. |
| Database | Instance size | medium | Medium, shared by all three. |
Research this Design needs: Load balancing, Health checks.
Both pictures show the same moment, ten seconds after one of three servers died. With a slow check the Load Balancer still sends that host a third of the traffic. With a fast check it has already dropped the host and the other two share the load.
Why it works#
The interval is how long a dead host keeps its share. Errors in the first picture are exactly the share the dead server would have taken. In the second they are gone, and the two healthy servers absorb the load with room to spare.
A deep check goes further. It also takes out any server whose Database is down or overloaded. That needs the Health check endpoint task from your Team, and it is sharp: if the Database fails, every server fails the check.
What it costs#
A short interval means more probe traffic and more chance of removing a host over a blip. Real teams keep deep checks narrow for this reason. A check also only helps if you run spare capacity, because the survivors must carry the dead host's share.
In Quorum#
Research Health checks to change the interval or turn on deep checks. The same research unlocks draining a Node. An Instance failure shows the gap. The glossary defines a health check. The next rung is a pooler between servers and database.