# Continuous batching

> A serving technique for language models that adds each new request to the running batch as soon as a slot frees, instead of waiting for the whole batch to finish. Throughput rises and the wait for a first token stays short.

Performance

## In Quorum

Continuous batching research adds batch 8 and batch 32 to a GPU Node: 1.6× and 2.2× the prompts a second with the time to the first token unchanged. Without it the larger batches still raise throughput, and they add 150 ms or 500 ms to that wait.

## Design

[GPU Node](https://wiki.playquorum.dev/design/05-components.md#gpu-node)

## Read next

[Model inference](https://wiki.playquorum.dev/glossary/model-inference.md), [Quantization](https://wiki.playquorum.dev/glossary/quantization.md), [Percentile (p50, p95, p99)](https://wiki.playquorum.dev/glossary/percentile-p50-p95-p99.md), [Utilization](https://wiki.playquorum.dev/glossary/utilization.md)

## Learn more

- [vLLM docs: Continuous batching](https://docs.vllm.ai/en/latest/)

## See also

- [Glossary](https://wiki.playquorum.dev/glossary.md)

Source: https://wiki.playquorum.dev/glossary/continuous-batching