# Design a file sync service like Dropbox

> Design a file sync service like Dropbox: chunking, object storage, a metadata database, sync and dedup, plus what Quorum teaches.

A file sync service splits every file into chunks, stores each chunk once in object storage under its hash, and keeps the file tree in a metadata database. Clients upload only chunks the server lacks and learn about changes through a notification channel. Bytes are cheap and the metadata database is the hard part.

This question is in the [System Design Primer](https://github.com/donnemartin/system-design-primer)'s [list of additional questions](https://github.com/donnemartin/system-design-primer#additional-system-design-interview-questions).

## Requirements

Functional:

- Upload, download, and delete files of up to several gigabytes.
- Sync a folder across a user's devices, and share files with other users.
- Keep version history, so an overwrite can be undone.

Non-functional:

- Durability above all. A lost file is the worst outcome.
- A small edit to a large file should not re-upload the whole file.
- A change shows up on other devices within seconds.
- Uploads resume after a dropped connection.

## Estimate the load

Assume 50 million users, 10 million active a day. Each user stores 10 GB on average and changes 20 files a day, at 1 MB average change.

- Storage: 50M x 10 GB = 500 PB. With 3x replication, or erasure coding at about 1.5x, that is 750 PB to 1.5 EB.
- Upload bandwidth: 10M x 20 x 1 MB = 200 TB a day, or 200 TB / 86,400 = about 2.3 GB a second.
- Metadata: a 4 MB chunk size gives 10 GB / 4 MB = 2,500 chunks per user, so 50M x 2,500 = 125 billion chunk records. At 100 bytes each, that is 12.5 TB. Metadata is small in bytes and large in rows.
- Sync checks: 10M users polling every 30 seconds is 330,000 requests a second unless you push changes.

## The design

### Chunking

The client splits a file into fixed chunks of about 4 MB, or content-defined chunks that move boundaries with the content. It hashes each chunk with SHA-256. Fixed chunks are simple, and content-defined chunks keep an insert at the top of a file from changing every following chunk.

### Block storage

Chunks go to [Object Storage](https://wiki.playquorum.dev/components/storage.md), keyed by hash. Upload goes directly from the client with a pre-signed URL, so the App Server never carries the bytes. A chunk is immutable, which makes caching, replication, and [lifecycle rules](https://wiki.playquorum.dev/patterns/object-storage-and-lifecycle-rules.md) trivial.

### Metadata database

A relational [Database](https://wiki.playquorum.dev/components/database.md), sharded by user or folder, holds the file tree. Each file row has a version and an ordered list of chunk hashes. Metadata needs transactions, because a rename or a commit of a new version must be atomic. Read the [sharding](https://wiki.playquorum.dev/patterns/sharding.md) pattern for how to split it.

### Dedup

Before uploading, the client asks which hashes the server already has, then sends only the missing ones. If two users upload the same installer, the second upload sends nothing. Dedup across users leaks whether a file exists, so some services dedup only within an account.

### Sync

Each device keeps a cursor, the last change ID it has seen. A notification service holds a connection per device, over [long polling or SSE](https://wiki.playquorum.dev/patterns/long-polling-and-sse.md) or a WebSocket, and tells it that something changed. The device then calls an API for changes after its cursor and downloads the new chunks. Conflicts, where two devices edit the same file, produce a "conflicted copy" file and not a lost edit.

### Download path

Hot, shared files sit behind a [CDN](https://wiki.playquorum.dev/components/cdn.md). Private files are fetched from Object Storage with short-lived signed URLs.

## Where it breaks

**Metadata hotspots.** One enterprise tenant with millions of files lands on one shard, and sync checks pile onto it. Shard by folder tree within big tenants, and cache listings. See [hot keys and hot shards](https://wiki.playquorum.dev/patterns/hot-keys-and-hot-shards.md) and the [Hot shard](https://wiki.playquorum.dev/incidents/hot-shard.md) incident.

**Slow uploads.** A large file on a poor connection holds a server connection for minutes. Upload chunks straight to storage, and send post-processing, such as virus scans and thumbnails, through a queue. See [slow work to a queue and workers](https://wiki.playquorum.dev/patterns/slow-work-to-a-queue-and-workers.md).

**Storage cost.** Every version of every file piles up, and most are never read again. Move old versions to a cheaper class, expire them by policy, and reference-count chunks so deletion is safe. Cost grows as a runaway line item, as in the [Runaway bill](https://wiki.playquorum.dev/incidents/runaway-bill.md) incident.

## Build it in Quorum

Two boards cover the storage half of the answer. The first is the move that comes before any chunking: take file bytes out of the Database. Look at the Database load and p99, which both fall once files leave its rows. It uses an [App Server](https://wiki.playquorum.dev/components/server.md), a [Database](https://wiki.playquorum.dev/components/database.md) and [Object Storage](https://wiki.playquorum.dev/components/storage.md), and the only change from the board before it is where files go. [Stateless app servers](https://wiki.playquorum.dev/patterns/stateless-app-servers.md) is the pattern.



The second board is the real Dropbox shape, as Dropbox described it for its Magic Pocket store. Object Storage holds immutable blocks keyed by hash, and the Database is the block index, set to four shards on a hash key. The App Server hashes a block, writes it to storage, then writes the hash and its location to the index. Look at how small the index is next to the bytes, and at whether any shard runs hotter than the others. That is the "bytes are cheap, metadata is hard" point from the design above, and [index the blobs](https://wiki.playquorum.dev/patterns/index-the-blobs.md) walks through it.



Research [Object storage](https://wiki.playquorum.dev/research/object-storage.md), then [Storage lifecycle rules](https://wiki.playquorum.dev/research/lifecycle-rules.md), and set a lifecycle window and a storage class. Watch Burn as the store fills, and note that a request for an archived object fails as a timeout. Send uploads through a [Queue](https://wiki.playquorum.dev/components/queue.md) and [Workers](https://wiki.playquorum.dev/components/worker.md), and run the [Slow Uploads](https://wiki.playquorum.dev/sandboxes/slow-uploads.md) Sandbox. For metadata growth, work through [Scale the database](https://wiki.playquorum.dev/guides/scale-the-database.md).

The game has no chunking, hashing, dedup, or sync protocol. It models the store, its classes, the upload path, and the database that holds the rest.

## Common questions

### How does Dropbox store files?

It splits files into blocks of a few megabytes, stores each block once in a block store keyed by its hash, and keeps the file tree and block lists in a metadata database. Clients upload only blocks the server lacks.

### Why split files into chunks?

Chunks make uploads resumable, let a small edit upload one chunk and not the whole file, and enable dedup. They also parallelize transfers.

### How does file deduplication work?

Hash each chunk and look the hash up before uploading. If the store already has it, add a reference and skip the transfer. Delete a chunk only when its reference count reaches zero.

### How do other devices learn that a file changed?

Each device keeps a cursor and holds a connection to a notification service. A change sends a ping, and the device fetches changes after its cursor from the metadata service.

Source: https://wiki.playquorum.dev/guides/design-a-file-storage-service
