QuorumWiki
Jugar

API de LLM

Sirve prompts de IA sin GPUs. Se paga por token y tiene límite de tasa.

Esta página aún no está traducida al español.

The LLM API is a hosted model, someone else's GPUs behind an API.

When to place it#

  • Place it when you ship your first AI Feature. It needs no operations work, and it costs nothing while traffic is low. It unlocks with Hosted models.

  • Skip it for steady load once the token bill grows large. Compare the bill with a busy GPU Node's Burn before you switch. The LLM API stays cheaper than a GPU Node that runs mostly idle.

  • Resize it before peak when a Traffic spike could push you over its limit.

How it works#

It bills for each token it serves, and it is a Managed Component, so it costs nothing to place and serves every region. Inference calls come in from your App Servers and no Component sits downstream of it. Each prompt waits a fixed time for the first token, longer than a GPU Node. See model inference.

It is rate limited. Each size step is a usage tier that doubles how many prompts per second it takes, and past the limit prompts fail at once as throttled. See rate limiting.

How it fails#

  • Past its tier, prompts fail as throttled.

  • Its cost grows with every prompt. Watch third-party costs in the Ledger.

  • Every prompt waits for the first token, so the AI prompt p99 stays high.

It also protects you. During a GPU shortage, an App Server linked to both a GPU Node and the LLM API keeps serving prompts.

Instance failure#

  • Signs: The LLM API card goes grey and its callout reads "The host under this VM died. It still bills." Requests sent to it are stranded.

  • Do now: Select the Node and click Replace. It costs Cash and comes back cold.

  • Prevent: Run a second LLM API Node, so prompts have another place to go.

Tips#

  • Keep an LLM API linked as a backup even after you place GPU Nodes.

Reglas#

  • Cada copia adicional cuesta un 25% más que la anterior.
  • Al venderlo se devuelve el 50% de lo que costó su última copia.
  • Tiene 8 tamaños. Cada paso duplica su capacidad y su Gasto.

Funciona con#

Nodo GPU

Términos de ingeniería#