The data tier · Quorum Wiki

App Servers are easy to copy. They hold no data, so ten copies are as correct as one. Data is different. Every copy of data must agree with the others, or users see the wrong answer. That is why the data tier becomes the long-term challenge of every Company.

The Database is the source of truth#

The Database answers read queries, write queries, and transactions. Until you link a better Component, it also does other jobs badly. It runs full-text search, stores uploaded files as rows, and runs report scans. Each of these costs it far more CPU than a normal read. A Search Engine, Object Storage, and a Data Warehouse take those jobs away.

Three limits cap a Database:

CPU. Every query costs CPU. A cold Database, just restarted, needs more of it per query.
Connections. Postgres runs one process per connection and accepts a fixed number. Read Errors and retries for why more App Servers make this worse.
The buffer cache. Report scans fill memory with cold data. Other reads then go to disk, and writes wait behind the scans. This is why analytics and transactions should not share a Database. See OLTP and OLAP.

Slow queries hide until you grow#

Two code problems cost little at first and a lot later.

An unindexed query scans the table. Its cost grows with the number of rows, and the rows grow with your users. It is invisible in the Garage and becomes the bottleneck later. The Slow query incident is this problem. Query analysis shows it, and the Add index task fixes it.

An N+1 query comes from an ORM that sends one query per item in a list. Laravel, Rails, and Django make extra reads on page views and API reads until you ship Eager loading. See N+1 query.

The Cache: memory in front of the Database#

A Cache answers read queries from memory and sends misses on to the Database. Its hit ratio decides how much load it takes off:

The square root models skewed access. Most reads ask for a few hot keys, so a small Cache already holds the hottest ones. The first gigabyte of Cache is worth more than the tenth. Hot data grows with your users, so a Cache that was big enough last Era may not be now.

Writes invalidate keys, so a write-heavy mix lowers the hit ratio. A longer TTL keeps more hits but serves more stale data. A Cache that restarts comes back cold, and every miss goes to the Database at once. That is a cache stampede. Stampede protection and Cache warming soften it.

Read Replicas: more reads, older data#

A Read Replica runs like a Database but answers only reads. Writes still go to the primary and then stream to the replica. The replica is always a little behind, and the lag grows with your write rate. See replication lag.

A read from a lagging replica can return old data. Whether that matters depends on the kind:

A page view accepts slightly old data.
An API read that comes back stale fails as stale.
A checkout that reads its own write fails as stale, which costs real money.

Read-your-writes routing sends the reads that care to the primary. That fixes most stale errors and keeps some read load on the primary.

SandboxStale CheckoutKeep Apdex above 0.9 for 90 seconds.

Quorums: the game's namesake#

A Wide-column Store keeps each key on N replicas. You choose two numbers: R, the replicas a read asks, and W, the replicas a write waits for.

Smaller R and W are faster, because each operation waits for an earlier reply, and they cost fewer replica operations. But a weak quorum serves stale reads. A quorum that needs every replica fails while one replica is down. See quorum.

Check yourself

A Wide-column Store keeps N = 3 replicas. Which setting makes every read see the latest write?

Overlap is not the same as a single, ordered history. Concurrent writes can still conflict, so even a strong quorum gets an occasional stale read or duplicate on checkouts. A Distributed SQL Node fixes that with consensus: every write waits for a majority of its replicas. Inside one region, that is cheap. Across regions, every write pays a long round trip. See Raft.

Hot keys#

Spreading data across nodes only helps if the load spreads too. In the Hot key incident, a large share of reads asks for one key, such as a celebrity's post. That key lives on one shard, which saturates while the others idle. Hybrid fan-out lets you split hot keys across shards. See hot key.

The pattern behind all of it#

Every data choice in Quorum trades among three things: speed, freshness, and survival. A Cache and a replica buy speed with freshness. A strong quorum and consensus buy freshness with speed. Copies in other zones and regions buy survival with Burn. The Data track shows the research behind each tool:

For practical steps, see Scale the database and Fix slow queries.