QuorumWiki
Jugar
Patrones

Un LLM detrás de un gateway en GPU

Sirve prompts de IA desde tus propios Nodos GPU junto a una API de LLM alojada, y ajusta modelo, precisión y tamaño de lote, porque una GPU sale más barata por prompt solo mientras se mantiene ocupada.

Put a gateway in front of AI prompts, and run your own GPUs only when the traffic keeps them busy.

Move up when#

  • The hosted LLM API bill grows faster than revenue, or its rate limit throttles prompts.

  • AI prompts are a large share of requests and you want control of the model.

  • First-token latency is too high on the hosted API.

The Design#

Drag to pan. Hold Ctrl or Cmd and scroll to zoom.

At 1.50k Users the Simulation solves this Design to Apdex 1.00, p99 354 ms, 0.0% errors and $35/s Burn.
WhereSettingValueWhy
API GatewayRate limit1000Caps all requests a second, so a burst of prompts cannot queue behind a fixed number of GPUs. Keep it above real demand.
App ServerInstance sizemediumMedium runs the code and hands each prompt on.
DatabaseInstance sizemediumMedium, to match the App Server.
GPU NodeGPU typea100An A100 costs less per second than an H100 and serves far more than a T4.
GPU NodeModelmediumA medium model answers fast, and a larger one earns more per prompt but serves a fraction as many.
GPU NodePrecisionint8INT8 weights serve faster and take half the memory, and the answer is worth a little less.
GPU NodeBatch size8A bigger batch raises throughput, and continuous batching keeps the first-token wait low.

Research this Design needs: API gateway, Model serving, Model serving, Quantization, Continuous batching.

The gateway rate-limits all requests and passes them to the App Server. The App Server links to one GPU Node, so every prompt runs on it. A hosted LLM API is the alternative, and linking both splits prompts between them by capacity.

Why it works#

A GPU Node serves tokens a second. GPU type, model, precision and batch size all multiply that. INT8 weights serve faster and fit in less memory, and their answers are worth a little less. A larger batch raises throughput and raises the wait for a first token, and continuous batching keeps the throughput without the wait. A model too big for the GPU's memory fails every prompt as out of memory, and lower precision shrinks it until it fits.

What it costs#

A GPU Node bills all the time and is cheaper per prompt only while it stays busy. Link a GPU and a hosted API to the same App Server and the split goes by capacity, so the hosted API takes most prompts, the GPU idles, and Burn pays for both. This Design links the GPU alone, which keeps it busy enough to cost less. Models that are larger earn more per prompt and serve a fraction as many. H100s can be missing during a GPU shortage. The gateway's limit counts every request kind, so set it above real demand. The game has no router that prefers the GPU first.

In Quorum#

Place a GPU Node and an LLM API. Research Hosted models, Model serving, Quantization and Continuous batching. Read model inference, quantization and continuous batching. The next step, vector search and embeddings, gives the model your own data.