Model inference
Running a trained machine learning model to produce an answer, as opposed to training it. For a large language model, a GPU generates the answer token by token, so cost and speed are measured in tokens per second and time to the first token. You can rent GPUs and serve the model yourself, or pay a hosted API per token.
Runtimes
In Quorum#
An ai request sends one prompt of 400 tokens to an inference Node. The LLM API (Hosted models Research) needs no GPUs: it costs $2.50 per 1,000 tokens, takes 600 ms to the first token, and throttles prompts above 100 per second per usage tier. A GPU Node (Model serving Research) takes 300 ms to the first token and generates 200 tokens per second per T4, 900 per A100, or 2,000 per H100. Its Burn is the same whether it is busy or idle, so an H100 is cheaper than the LLM API only while it stays about two-thirds busy.