Hot keys and hot shards
When one key or one shard takes a large share of the traffic, adding servers does not help, so spread that key or change the shard key.
A hot key or a hot shard saturates one machine while the rest of the fleet idles, so more machines buy nothing.
Use it when#
One shard or one cache node runs red while the others sit near idle.
A single item, such as a celebrity profile or a flash-sale product, draws a large share of reads.
New rows always land on the same shard, because the key grows with time.
How it works#
Hashing spreads keys evenly, but it cannot spread one key. Two repairs exist. Pick a shard key that matches how the traffic is spread, and for a key that is hot anyway, split it: store copies under suffixed keys (item:7#0 to item:7#9) and read a random one. A cache in front absorbs read-heavy keys too.
A range key such as an auto-increment ID sends every new user to the last shard. A hash of the ID does not. A hot key that takes writes is harder. A counter can be split into suffixed parts that a read adds up, but a value that must stay in step cannot.
What it costs#
Splitting a read-heavy key makes writes touch every copy, and splitting a counter makes a read sum every part. You also have to know which keys are hot, usually from metrics, and the hot set changes. A tenant key keeps a tenant together, which is good for queries and bad when one tenant is huge.
In Quorum#
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | large | Large, so the App Server is not the limit and the shards are. |
| Database | Shards | 4 | Four shards, the same as the hash Design, so only the key differs. |
| Database | Shard key | range | A range of IDs sends every new user to the last shard, so that shard takes the writes while the others idle. |
| Database | Instance size | small | Small for each shard, so a shard that takes too much of the load shows. |
Research this Design needs: Sharding.
The range key piles new users onto one shard, so one box takes the writes. Against the hash Design on Sharding, the same shards, Users and sizes give a lower Apdex and a longer p99.
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | large | Large, so the App Server is not the limit. |
| Cache | Instance size | medium | Medium, as in the split Design, so only the setting differs. |
| Database | Instance size | medium | Medium. It serves only what the Cache misses. |
Research this Design needs: Caching, Caching, Hybrid fan-out.
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | large | Large, so the App Server is not the limit. |
| Cache | Split hot keys | On | A key that takes a large share of reads is spread across the Cache shards, so the Cache keeps its full capacity while the hot key runs. |
| Cache | Instance size | medium | Medium, so the hot key has shards to spread over. |
| Database | Instance size | medium | Medium. It serves only what the Cache misses. |
Research this Design needs: Caching, Caching, Hybrid fan-out.
Both Boards run a Hot key incident on the Cache. Read the Load row on each card: without splitting, the Cache's capacity falls while the hot key runs, and with it the capacity stays whole. Nothing fails at this load, so the capacity figure is the evidence. On a Cache, turn on key splitting so a hot key spreads over the cache's shards. It needs Hybrid fan-out. A Wide-column Store has the same switch.
The Hot key incident sends a large share of reads to one key, and Hot shard lands the biggest tenant on one shard. The glossary defines the hot key.