Worker

What it does#

The Worker stands for a background job runner such as Sidekiq: your code, off the request path.

It pulls tasks from a Queue and runs the async part of each kind. It sends read and write queries, push messages, and calls to customer servers. It runs the same runtimes and frameworks as an App Server, with an instance type and size. Users never wait for it. Its throughput decides whether the Queue drains or grows.

An export runs many read queries in its task, so link the Worker to the data it reads: the Database, a Cache, or a Read Replica. Link it to the Push Provider for notifications.

Prefetch sets how many tasks each process takes ahead of time. Higher prefetch spends less time waiting on the broker. A crash loses every task a process had taken.

When to place it#

Place a Worker when you place a Queue: without one, tasks have nowhere to go. Add copies when Queue depth keeps growing.

How it fails#

• Too slow, and the Queue backlog grows until its max length refuses new tasks.

• A crash loses the tasks it had prefetched. A planned restart hands them back first.

• Poison message keeps Workers busy on one failing task.

• Bad deploy, Bad config push, and Memory leak hit Workers like App Servers.

• Each process holds database connections, which counts toward Connection exhaustion.

Tips#

• Link Workers to an Autoscaler to follow the backlog.

• Keep prefetch low on Workers that crash often.