Publish and subscribe
Publish each event once to a topic and let every interested service read it, so producers stop knowing who consumes their events.
Publish and subscribe sends each event to a topic, and every subscriber gets its own copy.
Use it when#
More than one service must react to the same event, such as a search index, a warehouse, and an email sender.
The producer should not change when a new consumer appears.
A new consumer must be able to replay history.
How it works#
A queue gives a task to one worker. A topic gives an event to every subscriber group, and inside a group the members share the work. With a log such as Kafka, the events stay for a retention window, so a new subscriber can read from the start, and a slow one falls behind without slowing anyone else.
A topic splits into partitions, and a group can use at most one consumer per partition. Keying events by entity keeps one entity's events in order inside a partition.
What it costs#
Delivery is at least once, so consumers must tolerate duplicates. Order holds only inside a partition. A slow subscriber's lag grows without any visible failure, so watch lag, not latency. A key that carries most of the traffic overloads its partition whatever the partition count.
In Quorum#
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | medium | Medium, so the App Server is not the limit. |
| Event Stream | Partitions | 6 | A consumer group uses one consumer per partition, so partitions cap how many consumers read at once. |
| Event Stream | Partition key | On | Keyed by entity, so one entity's events stay in order inside a partition. |
| Event Stream | Retention | 10080 | Events stay for the window after they are read, so a new or slow consumer can still catch up. |
| Event Stream | Acknowledgements | all | A producer waits for every replica before it moves on, the safest and slowest setting. |
| Worker | Instance size | small | Small. |
| Worker | Instance size | small | Small. |
| Database | Instance size | medium | Medium. It serves the reads and writes users wait for. |
Research this Design needs: Event streaming, Background jobs, Message queues, Background jobs, Event streaming.
An Event Stream stands for Kafka. The App Server appends to it, and its consumers read at their own pace while the events stay. Partitions cap how many consumers read at once, the key decides order, retention decides how far behind a consumer can fall, and acknowledgements trade write latency for durability.
Workers linked to one stream split its tasks between them, so the game does not give each its own copy. The game's Push Provider sends push and email messages, which is a different thing. Research Event streaming after Message queues. Hot key shows one key filling one partition. The glossary defines at-least-once delivery.