A queue in front of the queue: Slack
Slack put Kafka in front of Redis, so a burst of jobs waited in a durable log instead of filling the queue's memory.
By 2017, Slack's job queue ran on Redis, and every action that could not finish inside a web request went through it: unfurling a link, sending a notification, indexing a message. A Redis cluster ran out of memory, and the queue could not take new work. The team added Kafka in front of Redis instead of replacing it.
The system#
A small service called Kafkagate writes each job to Kafka and acknowledges it. A stateless relay, JQRelay, reads the Kafka topic and writes the job to the Redis cluster that runs it. The change kept the existing scheduling and de-duplication code and added a durable buffer for bursts. At the time, the system carried 1.4 billion jobs a day across 50 Redis clusters and 1,600 Kafka partitions.
The board#
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | large | Large, because the web tier only enqueues the slow work. |
| App Server | Copies | 3 | Three copies stand for the web tier that hands work to Kafka. |
| Database | Instance size | medium | Medium, because the workers read and write it too. |
| Event Stream | Partitions | 12 | Many partitions let many consumers read at once, as Kafka partitions do at Slack. |
| Event Stream | Acks | all | A producer waits for every replica, so an acknowledged job is not lost. |
| Event Stream | Keyed by entity | On | One job keeps its order in one partition. |
| Event Stream | Retention | 3600 | An hour of retention is enough to replay a failed consumer. |
| Queue | Dead-letter queue | On | A job that fails every time is parked, so it cannot block the line. |
| Queue | Max length | 10000 | A full queue refuses work instead of hiding a backlog. |
| Worker | Instance size | small | Small, because a job is one message at a time. |
| Worker | Copies | 4 | Several copies drain the queue faster than one box could. |
Research this Design needs: Event streaming, Message queues, Background jobs, Event streaming, Message queues, Background jobs, Backoff with jitter.
The Event Stream is Kafka: a durable log that holds a job after the web tier has acknowledged it. The Queue is Redis, the Worker is the job runner, and the Database holds the state the job reads and writes. The web tier only enqueues, so the user waits for the enqueue and not the work.
Why it works#
The two hops do different jobs. Kafka absorbs a burst and keeps the job until a consumer reads it. Redis holds the short-lived scheduling state that the workers poll. Slow work to a queue and workers and queue load leveling describe the two effects.
Where the game stops#
The board has one Event Stream and one Queue, not 50 Redis clusters and 1,600 partitions. It does not show JQRelay or the rate limits it reads from Consul. The caption is Quorum's result, not Slack's 1.4 billion jobs a day.
Sources#
Scaling Slack's Job Queue, Slack Engineering: the Redis incident, Kafkagate, JQRelay, and the scale figures.
Building self-driving Kafka clusters using open source components, Slack Engineering: the Kafka fleet, the throughput, and the operational work.
Apache Kafka documentation: partitions, consumer groups, and acknowledgements.