ReferenceResearch tree
Era 4 · Series BContinuous batching
A GPU Node batches prompts as they arrive, so a larger batch raises throughput without raising the time to the first token.
Prerequisites#
Research these first: Model serving.
A GPU Node batches prompts as they arrive, so a larger batch raises throughput without raising the time to the first token.
Research these first: Model serving.