A cache in front of the graph: Meta
Meta put a write-through memcached tier in front of sharded MySQL, and kept one primary region for each shard.
Facebook built TAO in 2013 to answer the social graph's read-heavy queries without sending every one to MySQL. TAO is a write-through cache: a read is served from memcached, and a write goes through the cache to the shard that owns the object. The data set is split into hundreds of thousands of shards, and each shard has one primary region.
The system#
The API is small and fixed: get an object, get its outgoing edges, count them, and iterate a range. TAO servers keep the cache consistent across regions, and MySQL replication carries each shard's updates from its primary region to the secondary regions. A secondary region forwards writes to the primary, and its write-through cache keeps read-after-write consistency for the member who made the change. By 2022 TAO served more than 10 billion requests a second.
The board#
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | large | Large, because the app tier answers the graph reads. |
| App Server | Copies | 3 | Three copies stand for the app tier over the graph. |
| Cache | Engine | memcached | Memcached, the cache TAO puts in front of MySQL. |
| Cache | Cache strategy | write-through | A write goes through the cache and on to the database, so the cache never holds an old object. TAO is a write-through cache over MySQL. |
| Cache | TTL (seconds) | 60 | A minute bounds staleness for an object nobody writes. |
| Cache | Instance size | xlarge | Extra large, because the hot graph is the point of the cache. |
| Database | Shards | 8 | TAO splits the graph into many shards. Eight is the widest the board offers, and it stands for hundreds of thousands. |
| Database | Shard key | hash | A hash spreads objects and associations evenly, so no shard owns the hot end of the graph. |
| Database | Instance size | large | Large for each shard, because each shard is a real MySQL database. |
| Database | Engine | mysql | MySQL, the engine TAO keeps the graph in. |
Research this Design needs: Caching, Caching, Cache strategies, Sharding.
The Cache is TAO's memcached tier, on write-through, so a write updates the cache and the database. The Database has eight shards on a hash key, standing for the hundreds of thousands of TAO shards. The board shows one region. A second region reads through its own cache and forwards writes to the primary, which the board does not draw.
Why it works#
A read-heavy graph spends most of its queries on the same popular objects. A write-through cache keeps those objects in memory and keeps the cache correct on a write, so the database sees the writes and the misses. Write-through and write-behind and shard the database are the two halves.
Where the game stops#
The game has one primary region per Database, not one per shard, and the board draws one region. It does not show TAO's API, its per-shard primary map, or the memcached fleet. The caption is Quorum's result, not Meta's 10 billion requests a second.
Sources#
TAO: The power of the graph, Engineering at Meta: the API, the write-through cache, the shards, and the per-shard primary region.
Open-sourcing TAOBench, Engineering at Meta: the 10 billion requests a second figure and the workload shape.
Scaling Memcache at Facebook, NSDI 2013: the cache fleet and the consistency model behind it.