ReferenceGlossary
Continuous batching
A serving technique for language models that adds each new request to the running batch as soon as a slot frees, instead of waiting for the whole batch to finish. Throughput rises and the wait for a first token stays short.
Performance
In Quorum#
Continuous batching research adds batch 8 and batch 32 to a GPU Node: 1.6× and 2.2× the prompts a second with the time to the first token unchanged. Without it the larger batches still raise throughput, and they add 150 ms or 500 ms to that wait.
Design#
Read next#
Model inference, Quantization, Percentile (p50, p95, p99), Utilization