Index the blobs: Dropbox
Dropbox put a sharded MySQL Block Index in front of Magic Pocket's immutable blocks, so a read was one hash lookup and one block fetch.
In 2016 Dropbox moved its file data off S3 into Magic Pocket, its own multi-exabyte store. A file is cut into blocks of up to 4 MB, each block is hashed with SHA-256, and the hash goes into a sharded MySQL Block Index. A read looks up the hash and fetches the block.
The system#
Blocks are packed into 1 GB buckets and written to volumes across machines, in replicated or erasure-coded form, in three US zones. The Block Index is a sharded MySQL cluster behind an RPC layer, and it decides which volumes hold a block. The Watcher samples 1% of written blocks and fetches them after a minute, an hour, a day, a week, and a month. The metadata scanner walks the index at about a million blocks a second.
The board#
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | large | Large, because the app hashes each block and looks up the index. |
| App Server | Copies | 3 | Three copies stand for the API tier. |
| Object Storage | Storage class | standard | The block store keeps the immutable blocks. |
| Object Storage | Lifecycle (days) | 30 | Older blocks move to a cheaper class after a month. |
| Database | Shards | 4 | The Block Index is a sharded MySQL cluster. Four is the board’s widest, and it stands for the fleet. |
| Database | Shard key | hash | A block is keyed by its SHA-256 hash, which spreads blocks evenly. |
| Database | Engine | mysql | MySQL, the engine of the Block Index. |
| Database | Instance size | medium | Medium for each index shard, so a busy shard shows. |
Research this Design needs: Object storage, Object storage, Sharding, Caching.
The Object Storage is the block store: immutable blocks, keyed by their hash. The Database is the Block Index, four shards on a hash key. The App Server hashes the block, writes it to storage, and writes its hash and location to the index.
Why it works#
The blocks are immutable, so the index only appends and never updates a row. The index is tiny next to the bytes, so a sharded MySQL cluster can hold it while the blocks live on cheap disks. A read is one hash lookup and one fetch. Object storage and lifecycle rules and shard the database are the two halves.
Where the game stops#
The game's Object Storage is a managed service, not Magic Pocket's own hardware. The board cannot show 4 MB blocks, 1 GB buckets, erasure coding, or the three zones. It shows the split: a hash index in front of an immutable block store. The caption is Quorum's result, not Dropbox's exabytes.
Sources#
Inside the Magic Pocket, Dropbox: the blocks, buckets, volumes, the sharded MySQL Block Index, and the multi-zone design.
Scaling to exabytes and beyond, Dropbox: why Dropbox left S3 and what the store had to do.
Pocket watch: Verifying exabytes of data, Dropbox: the scanner and the Watcher that sample and re-fetch blocks.