Retrieval

Retrieval is where search becomes an answer: dense, sparse, hybrid and reranking decide whether the context reaching the LLM is useful or junk disguised as evidence.

Cohere RerankBM25ColBERTTavily

What it is

Retrieval is the step that decides which passages from the base become context for the LLM. In RAG, it is the stage where most of the quality is won or lost: good context becomes a good answer, wrong context becomes a wrong answer, even with a perfect prompt and an excellent model. Bad retrieval cannot be fixed afterwards.

A retrieval system has three decisions: which algorithm to use (dense, sparse, hybrid), how to rerank the raw result, and how to fuse lists coming from different algorithms. All three decisions depend on the type of question and the type of corpus.

Dense, sparse and hybrid

Dense search uses embeddings: the question becomes a vector, the corpus becomes vectors, similarity is geometric. It is good for paraphrases, conceptual questions and non-technical terms. It is weak for rare acronyms, identifiers, and literal terms that exist in the document but not in the model's vocabulary.

Sparse search uses term frequency ( is the standard in practice). It is good for exact words, proper names and acronyms. It is weak for paraphrases and queries that mix terms. BM25 scores each term by TF (how many times it appears in the document) and by IDF (how rare it is in the corpus), and normalizes by document length. The result is a list ranked by lexical relevance.

Hybrid search runs both in parallel and fuses the lists. The most common fusion method is RRF (Reciprocal Rank Fusion): each list assigns 1/(k + rank) to each candidate, and the candidates are reordered by the sum. The result inherits the best of both searches.

Reranking

After the fused top, noise still remains. The reranker steps in to reorder the top-N with a model that looks at the question and the candidate together. Cohere Rerank, BGE Reranker and ColBERT do this with a smaller language model that scores each (question, document) pair with better quality than .

The cost is proportional to the number of reranked candidates, so the input list is typically 50 to 200 items. The latency cost stays within budget because the reranker is smaller and cheaper than the final LLM.

In practice

The example combines dense retrieval (Qdrant), BM25 (Pyserini) and a Cross-Encoder reranker in a single pipeline. The fusion is RRF, and the reranker cuts the best 50 down to the final 5.

retrieval/pipeline.pypython

from typing import Iterable
from qdrant_client import QdrantClient
from pyserini.search.lucene import LuceneSearcher
from sentence_transformers import CrossEncoder

qdrant = QdrantClient(url="http://localhost:6333")
searcher = LuceneSearcher.from_prebuilt_index("manual_pt")
reranker = CrossEncoder("cross-encoder/mmarco-mMiniLMv2-L12-H384-v1")

def rrf(ranks: Iterable[list[str]], k: int = 60) -> list[tuple[str, float]]:
    scores: dict[str, float] = {}
    for r in ranks:
        for i, doc_id in enumerate(r):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + i + 1)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

def retrieve(query: str, embedding: list[float], top_k: int = 5):
    # 1. dense (Qdrant, top 50)
    dense_hits = qdrant.search(
        collection_name="chunks",
        query_vector=embedding,
        limit=50,
        with_payload=False,
    )
    dense_ids = [str(h.id) for h in dense_hits]

    # 2. sparse (BM25, top 50)
    bm25_hits = searcher.search(query, k=50)
    bm25_ids = [h.docid for h in bm25_hits]

    # 3. RRF fusion
    fused = rrf([dense_ids, bm25_ids])

    # 4. rerank the 50 candidates
    candidates = [(doc_id, lookup_text(doc_id)) for doc_id, _ in fused[:50]]
    pairs = [(query, text) for _, text in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)

    return [doc_id for (doc_id, _), _ in ranked[:top_k]]

Strengths

double coverage: hybrid covers both paraphrase (dense) and exact term (BM25), so both blind spots are mitigated
reranking the top-N delivers a boost: the small reranker filters what got through, so the LLM only sees what really matters
amortized cost: reranking the top-50 is cheap, and the rest of the raw list never even gets processed

Limitations

operational complexity: three subsystems (vector, BM25, reranker) must be healthy together; if one fails, the cluster goes down
fusion is heuristic: RRF does not learn an optimal fusion, and the default k=60 is not universal; tuning is production work
the reranker becomes a theme of its own: the reranker is updated per model, so checking whether a new model beats the previous one is a mandatory regression

BM25The classic ranking function of text search: it scores each document by TF-IDF with saturation (important terms weigh more up to a ceiling) and normalizes by length. It is the sparse search of hybrid retrieval and captures exact term matches (error code, API name) where embeddings alone fail badly.see in glossary →Cosine similarityA measure of how aligned two vectors are in embedding space: the cosine of the angle between them, independent of magnitude. For normalized embeddings the inner product is enough and stays fast, which is why it became the default distance in almost every vector database.see in glossary →

connections on the map

Vector DBqueriesRetrieval
Retrievalfeeds contextRAG - retrieval-augmented generation

updated on 14/10/2026 · sources to go deeper · report an error

Study before

Unlocks

RAG - retrieval-augmented generation

test what you learned

Loading questions…