Slow work to a queue and workers
Hand work the user does not wait for to a Queue, and let Workers drain it at their own pace.
Have the App Server enqueue slow work and answer at once, and let Workers do the work later.
Move up when#
Exports, uploads, or webhooks hold App Server slots, and page latency climbs when they run.
A spike in slow work makes fast requests time out.
The work can happen seconds later without the user minding.
The Design#
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | small | It only enqueues exports now, so it stays small. |
| Database | Instance size | medium | Workers read from it while users do, so give it room. |
| Queue | Dead-letter queue | On | A task that keeps failing is parked after its attempts, so it cannot block the line forever. |
| Queue | Max attempts | 3 | Three tries cover a blip. Beyond that the task is broken and a retry only burns Worker time. |
| Queue | Max length | 10000 | A full Queue refuses new tasks instead of hiding a backlog that grows without limit. |
| Worker | Instance size | small | An export reads many rows, so a Worker needs room beyond the smallest box. |
| Worker | Copies | 3 | Several copies drain the backlog faster than one box could, and Queue depth tells you when to add more. |
| Worker | Prefetch | 10 | Each process takes ten tasks ahead of time, which spares broker round trips. A crash loses only those ten. |
Research this Design needs: Message queues, Background jobs, Background jobs, Message queues.
The App Server answers pages and reads itself, and for an export it hands a task to the Queue and returns. The Queue holds tasks, and the Workers take them and read the Database. Both the App Server and the Workers call the same Database, so it has to take their combined load.
Why it works#
The user waits only for the enqueue, which is fast, so slow work no longer holds a request slot. Queue depth becomes the first gauge to watch. When it grows, the Workers are too slow, and adding copies fixes it without touching the App Server.
The settings keep it honest. A dead-letter queue parks a task that keeps failing, so it cannot loop forever. Max attempts says how many tries a task gets. Max length refuses new tasks when full, instead of letting a backlog grow unseen. A Worker's Prefetch lets each process take tasks ahead of time, which saves broker round trips and loses those tasks if it crashes.
What it costs#
You add two Nodes and a delay the user can see, since the result is not ready when the request returns. Delivery is at least once, so a task can run twice and the work must be idempotent. A backlog hides trouble until it hits the limit. The Workers also read the Database, so it can become the bottleneck.
In Quorum#
Place a Queue and a Worker after Background jobs and Message queues. The pattern is Queue-based load leveling, with competing consumers and a dead-letter queue. The glossary has at-least-once delivery. The next rung sends one task to many receivers: Message platform for fan-out.