Partition messages by channel: Discord
Discord stored trillions of messages in a wide-column store, partitioned by channel and time, and read with a quorum.
Discord moved its messages from MongoDB to Cassandra in 2017, when it had 12 nodes and billions of messages. By early 2022 the cluster had 177 nodes and trillions of messages, and the team migrated it again, to ScyllaDB. The schema partitions messages by channel and a time bucket, and replicates each partition three times.
The system#
Every message row is keyed by channel and bucket, so a channel's recent messages sit together and a read is a range scan. Reads and writes use quorum consistency. The weak spot is the hot partition: a server with hundreds of thousands of members sends far more traffic to one channel than a small server does. Discord added data services in front of the store, with consistent-hash routing by channel, so requests for one channel coalesce on one service instance.
The board#
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| Load Balancer | Policy | least-conn | A gateway connection is long-lived, so the load balancer sends it to a hub with room. |
| App Server | Instance size | large | Large, because the API tier does the routing and the reads. |
| App Server | Copies | 3 | Three copies stand for the API tier in front of the message store. |
| Wide-column Store | Replicas per key (N) | 3 | Three copies of each key, as the store keeps. |
| Wide-column Store | Replicas a read asks (R) | 2 | Two of three replies, the quorum Discord reads with. |
| Wide-column Store | Replicas a write waits for (W) | 2 | Two of three, so a read that asks two overlaps the latest write. |
| Wide-column Store | Instance size | medium | Medium for each node, so a hot partition shows. |
| Cache | Engine | redis | Redis is the store Discord uses for presence and hot state beside the message store. |
| Cache | Cache strategy | aside | The API fills the cache on a miss, so a popular channel is read from memory. |
| Cache | TTL (seconds) | 60 | A short TTL keeps hot channel reads current. |
| Cache | Instance size | small | Small, because it holds hot state and not the message log. |
| Realtime Hub | Stack | beam | BEAM holds a lightweight process per connection, which is why Discord’s gateway fits so many on a box. |
| Realtime Hub | Clustering | On | Several hubs sit behind one pub/sub backbone, so one box does not take every connection. |
| Realtime Hub | Reconnect backoff | On | Clients reconnect with jitter after a restart, so the gateway is not hit by every client at once. |
| Realtime Hub | Presence | On | The hub tracks who is online, which costs CPU on each join. |
| Link Load Balancer → App Server | Kinds carried | page,read,write,login | Messages and reads go to the API tier. |
| Link Load Balancer → Realtime Hub | Kinds carried | rt | Realtime events go to a hub, not to an app server. |
Research this Design needs: Load balancing, Leaderless replication, Caching, WebSockets, Leaderless replication, WebSockets, Caching, Elixir.
The Wide-column Store is the message cluster, with three replicas per key and a read and write quorum of two. The Realtime Hub is the gateway tier, on BEAM because a lightweight process per connection is what lets one box hold many. The Cache holds hot channel state.
Why it works#
A quorum read of two replicas overlaps the latest write of two, so a read sees the last message even after a node fails. Partitioning by channel keeps a conversation's history contiguous. Quorum reads and writes and hot keys and hot shards are the two ideas the schema leans on.
Where the game stops#
The board has one Wide-column Store, not 72 ScyllaDB nodes, and it does not show the data services or the per-channel partitions. The caption is Quorum's result, not Discord's p99. The game's Hot key incident is the same failure the data services were built to absorb.
Sources#
How Discord Stores Trillions of Messages, Discord: the Cassandra to ScyllaDB migration, the 177 to 72 node change, and the latency numbers.
How Discord Stores Billions of Messages, Discord: the original schema and why it bucketed messages by time.
How Discord Supercharges Network Disks for Extreme Low Latency, Discord: the storage topology behind the ScyllaDB cluster.