ReferenceGlossary
Quantization
Storing a model’s weights at lower precision, such as 8-bit or 4-bit integers rather than 16-bit floats, so the model uses less memory and computes faster, at some cost in answer quality.
Performance
In Quorum#
Quantization research adds FP16, INT8 and INT4 to a GPU Node. INT8 serves 2× the tokens and INT4 3×, and each answer is worth 5% or 15% less. It pays when the GPU is the bottleneck and the answers can afford to be slightly worse.
Design#
Read next#
Model inference, Continuous batching, Utilization, Cold start