Sharding
Split one dataset across several databases by a key, so writes and storage scale past what one machine holds.
Sharding gives each database a slice of the data, chosen by a key in every row.
Use it when#
The primary is at its write limit or disk limit after replicas, caching and a bigger machine.
Most queries name one customer or one user, so each touches one slice.
How it works#
A shard key such as user_id goes through a hash, and the result picks the shard. With 4 shards, user 8191 might land on shard 3, and every read and write for that user goes there. Writes divide across the shards, so two shards take about twice the writes of one.
The key decides everything. A good key spreads load evenly and keeps related rows together. A bad one builds a hot shard: shard by country and the largest country carries most of the traffic. Consistent hashing is how a real system adds a shard without moving most of the data.
What it costs#
Queries that span shards, like a report across all users, become slow scatter-gather jobs. A transaction across shards is slower and fails more often, and joins are hard or impossible. Changing the key or the shard count later means copying the data, which takes engineering time and calendar time. Treat sharding as the last step, after read replicas, caching and query fixes have run out.
In Quorum#
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | large | Large, so the App Server is not the limit and the shards are. |
| Database | Shards | 4 | Each shard is another copy of the Database, and writes divide across them. |
| Database | Shard key | hash | A hash of the user ID sends every user to one shard and spreads users evenly, so no shard runs hot. |
| Database | Instance size | small | Small for each shard, so a shard that takes too much of the load shows. |
Research this Design needs: Sharding.
The Database card has a shard count and a shard key. A hash key spreads users evenly, a range key piles new users onto the last shard, a tenant key keeps a tenant together but lets a big one run hot, and a region key keeps users near their data at the price of a far round trip for the rest. Hot keys and hot shards puts the range key on a Board.
Research Sharding to unlock it. The Hot shard incident lands the biggest tenant on one shard. The data tier shows where it fits, and the glossary defines sharding.