Vector search and embeddings
Keep embeddings of your documents in a Vector Index and look up the nearest ones before each prompt, so answers use your own data instead of only the model's.
Retrieve the closest documents by meaning, and hand them to the model with the prompt.
Move up when#
AI answers sound generic, because the model has never seen your documents.
Keyword search misses results that mean the same thing in other words.
The model invents facts about your product.
The Design#
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | medium | Medium runs the code that retrieves and then prompts. |
| Database | Instance size | medium | Medium keeps the rows. The index holds only the embeddings. |
| Vector Index | Index type | hnsw | A graph walk answers twice the queries of an exact scan, and gives up a little recall for it. |
Research this Design needs: Embeddings, Hosted models, Hosted models.
The App Server calls the Database, the Vector Index and the LLM API. For a prompt it looks up the nearest embeddings first, then asks the model.
Why it works#
The index holds each document as a vector and returns the nearest ones. An exact index compares the query to every vector, which is perfect but slows as the index grows. HNSW walks a graph instead, so it answers more queries and keeps a high share of true neighbours.
A prompt answered with retrieved documents earns more. A Python Stack gets a further bonus, since it can run small models in process, though PyPy cannot.
What it costs#
The index lives in memory, so HNSW bills more memory than an exact index. An approximate index misses a neighbour now and then, which takes part of the lift and returns a stale result. The caption's errors are those misses, counted as stale results. Embedding quality, chunking and reranking are outside the game, which models only the lookup and its recall.
In Quorum#
Place a Vector Index and link the App Server to it. Set Index type. Research Embeddings and Hosted models. Read model inference.