Distributed SQL and leaderless stores
Replace hand-made shards with a store that replicates by itself, either leaderless with tunable quorums or SQL on Raft, and pick by whether you need every read correct.
Let the store handle replication and splitting: a leaderless store for speed, or distributed SQL for correctness.
Move up when#
Resharding by hand costs Dev Hours you do not have, and cross-shard transactions fail.
A feed or log workload writes far more than a single primary can take.
A checkout must be right everywhere, and a replica that lags is not acceptable.
The Design#
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | medium | Medium, so the store is the tier under test. |
| Wide-column Store | Replicas per key (N) | 3 | Three copies of each key survive the loss of one. |
| Wide-column Store | Replicas a read asks (R) | 2 | Two of three replies, so a read overlaps any write that waited for two. |
| Wide-column Store | Replicas a write waits for (W) | 2 | R plus W is above N, so every read sees the latest write. |
| Database | Instance size | medium | Searches and uploads stay on a Database, because the store takes only reads, writes and transactions. |
Research this Design needs: Leaderless replication, Read replicas, Sharding, Leaderless replication.
The leaderless Wide-column Store replicates each key to N nodes. A read asks R of them and a write waits for W. A Database stays beside it for the kinds the store does not take.
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | medium | Medium, so the store is the tier under test. |
| Distributed SQL | Instance size | small | Each size step doubles the nodes behind it, and every one is billed. Small is enough to show the write cost. |
| Database | Instance size | medium | Searches and uploads stay on a Database, because the store takes only reads, writes and transactions. |
Research this Design needs: Consensus databases, Read replicas, Synchronous replication, Consensus databases.
Distributed SQL puts a SQL store on Raft. Each write waits for a majority of its replicas, so every read sees it.
Why it works#
When R plus W is above N, a read always meets a node that took the latest write. Lower R or W makes it faster and lets stale reads through. Distributed SQL gives that guarantee for transactions and costs more latency per write.
Even a strong quorum is not a transaction. A checkout on the wide-column store still goes wrong now and then, so use distributed SQL for money and a leaderless store for feeds.
What it costs#
Both bill several nodes at once, and distributed SQL bills the most. Neither takes searches or uploads, so a Database stays for them. A split can lose a replica, and a quorum that needs every replica then refuses. Both can also Spread across regions, which the multi-region rung uses. The game does not model a lightweight transaction on the wide-column store.
In Quorum#
Place a Wide-column Store or Distributed SQL. Research Leaderless replication or Consensus databases. Read quorums, Raft and The data tier. The next rung, logs, metrics and traces, shows where these stores spend their time.