RAG - retrieval-augmented generation

RAG combines search and generation: the model only answers after pulling the relevant passages from an external base, so the answer is anchored in material that exists and can be cited.

LangChainLlamaIndexHaystackPinecone

What it is

RAG (Retrieval-Augmented Generation) is an architecture pattern in which the language model receives, along with the user's prompt, a set of documents coming from a search. Retrieval happens before generation, and what the LLM reads is this mix: the question plus the retrieved context. The answer comes from the model, but anchored in what retrieval brought back.

The central idea is simple: a pure LLM answers based on what it learned in training, so it gets recent facts wrong, mixes domains and invents references. With RAG, the system first pulls the relevant part of a current base (a wiki, an index of manuals, a collection of PDFs) and hands it to the model as evidence. The final text now has a source.

In a short definition: retrieval decides what the model talks with, and generation decides how it answers.

How it works

The flow has five steps, all deterministic and auditable:

1. the user sends a question; the system writes a clean version of it (sometimes expanded by a query expansion LLM).
2. the question becomes a vector (embedding) and the system searches for the k closest passages in the vector index.
3. lexical search (exact words) runs in parallel and enters a fusion with the vector results, forming the hybrid ranking.
4. a reranker reorders the candidates by perceived relevance, discarding those that do not contribute.
5. the best passages become context in the LLM's prompt, together with the question; the model answers citing the sources.

In practice

The example below shows the minimal pipeline in TypeScript: it receives a question, retrieves the k chunks, builds the prompt and calls the LLM. It is not production code, but it is the skeleton that shows up in every project.

rag/pipeline.tstypescript

type Chunk = { id: string; text: string; source: string };

async function retrieve(query: string, k = 5): Promise<Chunk[]> {
  // 1. embedding of the question
  const qVec = await embedder.embed(query);

  // 2. dense search
  const dense = await vectorStore.search(qVec, { topK: k * 2 });

  // 3. lexical search (BM25) + fusion
  const lexical = await bm25.search(query, { topK: k * 2 });
  const fused = rrf([dense, lexical]);

  // 4. rerank
  const reranked = await reranker.rerank(query, fused, { topK: k });

  return reranked;
}

async function answer(query: string) {
  const chunks = await retrieve(query);
  const context = chunks.map((c) => `[${c.source}] ${c.text}`).join("\n\n");
  const prompt = `Context:\n${context}\n\nQuestion: ${query}\nAnswer citing the sources.`;
  return await llm.generate(prompt);
}

Common variations

Three variations show up in almost every serious architecture. The first is agentic RAG, in which the agent decides whether to call retrieval, how many times and with which filters, instead of retrieval always running. The second is multi-hop RAG, in which the answer of one round becomes the query of the next, building up from several chained searches. The third is GraphRAG, in which the index is a graph of entities and the search follows edges.

All of them share the same principle: generation is restricted to what retrieval can find, so answer quality depends on index quality. Swapping the LLM without reviewing retrieval rarely changes the result.

Strengths

source anchoring: every answer has a verifiable reference, so factual hallucination drops and auditing becomes possible
updatable knowledge: changing the base means reindexing; the model needs no retraining or fine-tuning
scope control: you can limit search to a domain (company manuals, case law, contracts) and block the rest
lower cost than fine-tuning: indexing and searching costs orders of magnitude less than training or adapting weights

Limitations

quality inherited from retrieval: bad retrieval becomes bad context, and bad context becomes a bad answer, even with an excellent LLM
limited : too many chunks force a cut, and what gets cut is often precisely the relevant part
maintenance cost: a stale index, unreviewed embeddings and poorly chosen chunking degrade silently
multi-document reasoning remains hard: the model reads chunks, it does not connect facts between them; GraphRAG tackles part of this

Context windowcontext windowThe maximum number of tokens the model can read at once: the prompt, the entire concatenated history and the output it will generate. It is the LLM's short-term memory. Huge windows seem to solve RAG, but they amplify cost, latency and the risk of the model getting lost in the middle of the text, so good context is small, well-chosen context.see in glossary →

connections on the map

Retrievalfeeds contextRAG - retrieval-augmented generation
RAG - retrieval-augmented generationinjects contextLLM - large language model
Evals - quality measured in productionmeasures retrievalRAG - retrieval-augmented generation

updated on 14/10/2026 · sources to go deeper · report an error

Study before

test what you learned

Loading questions…