Retrievers — Dense, Sparse, Hybrid

1. Why does this topic exist?

We have a vector store. We could just call vector_store.similarity_search(query). Why do we need a Retriever abstraction?

Three real-world reasons:

1. Vector stores are one of many knowledge sources. Real apps retrieve from BM25 indexes, SQL databases, web search APIs, knowledge graphs. We need a uniform interface.

2. Vector search has a blind spot. Embedding-based retrieval misses exact terms (product codes, named entities, error codes). Hybrid retrieval (dense + sparse) fixes this.

3. Retrievers compose. Need multi-query? Wrap a base retriever. Need reranking? Wrap again. The retriever abstraction makes pipelines hot-swappable.

Industry pain example: A SaaS company built dense-only RAG. Users searched "Error 0x80070005" — vector retrieval returned random pages. Adding BM25 (which exactly matches 0x80070005) into a hybrid retriever fixed it overnight. Recall@5 on error-code queries: 0.3 → 0.95.

---

2. What is it?

Simple explanation

A Retriever is a function: given a query string, return a list of relevant documents. The implementation can be vector search, keyword search, web search, SQL, or any combination.

Technical explanation

A Retriever is any class implementing the BaseRetriever interface with .invoke(query: str) → List[Document]. It's a Runnable — composable into LCEL pipelines.

from langchain_core.retrievers import BaseRetriever

class MyRetriever(BaseRetriever):
    def _get_relevant_documents(self, query, *, run_manager):
        return [Document(page_content="...")]

Industry definition

The "retriever" abstraction comes from information retrieval (decades old). In RAG-era frameworks (LangChain, LlamaIndex), it's the standard interface between knowledge sources and LLM pipelines.

Mental model

A retriever is a library card catalog: you give it a topic, it tells you which books (chunks) to grab. Different catalogs use different organization (alphabetical/keyword vs subject/semantic) — but the patron sees the same interface.

Analogy

Like a search bar: it doesn't matter if Google uses link analysis, semantic embeddings, or freshness signals — you type a query, you get results. The retriever abstracts away how the matching happens.

---

3. How does it work?

The three retrieval paradigms

flowchart TD
    A[Retriever] --> B[Dense - Semantic Search]
    A --> C[Sparse - Keyword Search]
    A --> D[Hybrid - Both, fused]
    B --> B1[Vector embeddings, cosine similarity]
    C --> C1[BM25, TF-IDF, exact term match]
    D --> D1[Reciprocal Rank Fusion of both]

---

Paradigm 1: Dense retrieval (vector embeddings)

This is what we covered in Chapters 4-5. Embed query → cosine similarity → top-k.

retriever = vector_store.as_retriever(search_kwargs={"k": 5})
docs = retriever.invoke("What is RAG?")

Pros: semantic match, paraphrase-friendly, multilingual. Cons: misses exact terms (codes, IDs, named entities).

---

Paradigm 2: Sparse retrieval (BM25 / TF-IDF)

Sparse retrieval scores documents by term overlap with the query. The vectors are sparse (one dimension per vocabulary word, mostly zeros).

TF-IDF (the classic, 1972)

Formula:

\[ \text{tf-idf}(t, d, D) = \text{tf}(t, d) \times \log\frac{N}{|\{d' \in D : t \in d'\}|} \]

Where: - \(\text{tf}(t, d)\) — count of term \(t\) in document \(d\) - \(N\) — total number of documents - The denominator: number of documents containing \(t\)

Intuition: A term that appears often in one doc (high TF) but rarely in others (high IDF) is important for that doc.

BM25 — the improved standard (1994)

Formula:

\[ \text{BM25}(q, d) = \sum_{t \in q} \text{IDF}(t) \cdot \frac{\text{tf}(t, d) \cdot (k_1 + 1)}{\text{tf}(t, d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\text{avgdl}})} \]

Where: - \(k_1 \approx 1.2-2.0\) — term saturation parameter - \(b \approx 0.75\) — document length normalization - avgdl — average document length

BM25 corrects TF-IDF's main flaws: 1. Term saturation — 100 occurrences of "the" don't make it 100× more important. 2. Length normalization — long docs don't unfairly win because they contain more words.

Implementation:

from langchain_community.retrievers import BM25Retriever

bm25 = BM25Retriever.from_documents(chunks)
bm25.k = 5

results = bm25.invoke("Error 0x80070005 troubleshooting")

flowchart LR
    Q[Query: error 0x80070005] --> T[Tokenize: error, 0x80070005, troubleshooting]
    T --> BM[BM25 score per doc]
    BM --> R[Top-k by score]

Pros (sparse): - Exact terms match perfectly — product codes, error codes, named entities. - Fast (inverted index). - Interpretable — you can see WHY a doc matched. - No embedding cost.

Cons: - Doesn't capture synonyms ("car" vs "automobile"). - Doesn't capture paraphrases ("how to bake" vs "baking instructions"). - Vocabulary mismatch problem.

---

Paradigm 3: Hybrid retrieval (dense + sparse)

The killer combo. Run both. Merge results.

flowchart LR
    Q[Query] --> D[Dense retriever embeddings]
    Q --> S[Sparse retriever BM25]
    D --> L1[Ranked list A]
    S --> L2[Ranked list B]
    L1 --> RRF[Reciprocal Rank Fusion]
    L2 --> RRF
    RRF --> R[Final ranked top-k]

Reciprocal Rank Fusion (RRF)

Formula:

\[ \text{RRF}(d) = \sum_{i \in \text{retrievers}} \frac{w_i}{k + \text{rank}_i(d)} \]

• \(\text{rank}_i(d)\) — position of \(d\) in retriever \(i\)'s ranking (1-indexed)
• \(k\) — smoothing constant (typically 60)
• \(w_i\) — weight per retriever (often 0.5/0.5 or 0.7/0.3)

Implementation:

from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever

bm25 = BM25Retriever.from_documents(chunks)
bm25.k = 5
dense = vector_store.as_retriever(search_kwargs={"k": 5})

hybrid = EnsembleRetriever(
    retrievers=[bm25, dense],
    weights=[0.4, 0.6],   # bias toward semantic
)

docs = hybrid.invoke("Find SKU-A4-12 documentation")

Why hybrid wins:

Query type	Dense wins	Sparse wins	Hybrid wins
"How does X work?"	✅		✅
"Error code 0x80070005"		✅	✅
"Refund policy"	✅	✅	✅ (slight edge)
"Product SKU-A4-12 specs"		✅	✅
"Synonyms of efficient"	✅		✅

Hybrid covers all bases. Production RAG uses it by default.

---

Other retriever variations

Threshold-based retrieval

retriever = vector_store.as_retriever(
    search_type="similarity_score_threshold",
    search_kwargs={"score_threshold": 0.7, "k": 10},
)

Returns at most k results, only those above 0.7. Use when "no answer" is better than "bad answer."

MMR (Maximum Marginal Relevance)

retriever = vector_store.as_retriever(
    search_type="mmr",
    search_kwargs={"k": 4, "fetch_k": 20, "lambda_mult": 0.6},
)

Picks diverse top-k from a larger pool. Avoids returning 4 near-duplicates.

Metadata-filtered retrieval

retriever = vector_store.as_retriever(
    search_kwargs={
        "k": 5,
        "filter": {"customer_id": "acme", "doc_type": "policy"},
    },
)

Multi-tenant safe; auditable.

---

4. Visual Learning

Architecture — retrievers as plug-ins

flowchart LR
    Q[Query] --> R[Retriever interface]
    R --> A[Dense vector]
    R --> B[Sparse BM25]
    R --> C[Web search API]
    R --> D[SQL]
    R --> E[Hybrid ensemble]
    A & B & C & D & E --> DOCS[List of Documents]

The retriever interface lets any knowledge source plug in.

Workflow — hybrid retrieval

flowchart LR
    A[Query] --> B[Run both dense + sparse]
    B --> C1[Dense top-10]
    B --> C2[Sparse top-10]
    C1 & C2 --> M[Merge with RRF]
    M --> R[Top-k final]

Sequence — query path

sequenceDiagram
    participant App
    participant Hybrid as Ensemble Retriever
    participant Dense
    participant Sparse
    App->>Hybrid: invoke query
    par Run in parallel
        Hybrid->>Dense: vector search
        Dense-->>Hybrid: ranked list A
    and
        Hybrid->>Sparse: BM25 search
        Sparse-->>Hybrid: ranked list B
    end
    Hybrid->>Hybrid: RRF merge
    Hybrid-->>App: final top-k

Real-world example — customer support hybrid

flowchart LR
    Q[user query] --> D[Dense: semantic match on intent]
    Q --> S[Sparse: exact match on error codes/product IDs]
    D --> R1[Conceptually related articles]
    S --> R2[Articles with specific terms]
    R1 & R2 --> F[Fuse RRF]
    F --> A[Best of both signals]

---

5. Pros

Benefit	Detail
Uniform interface	Vector, BM25, web, SQL all look the same to the app
Composable	Wrap retrievers in retrievers (multi-query, ensemble)
LCEL-friendly	Pipes into prompt | model | parser
Pluggable	Swap implementation without changing pipeline
Hybrid power	Dense + sparse covers complementary failure modes

6. Cons

Limitation	Detail
More moving parts	Hybrid has two retrieval calls per query
Tuning hybrid weights	Empirical, depends on corpus
BM25 needs separate index	Memory or disk overhead alongside the vector store
Score normalization	Dense scores ∈ [-1,1]; BM25 unbounded — need RRF (rank-based, not score-based)

7. Trade-offs

Choice	Trade-off
Dense only	Simpler, misses exact terms
Sparse only	Misses semantics
Hybrid	Best recall, 2× cost & latency
Threshold-based	Safer, may return empty
MMR	Diverse, may miss the actual best

Default: hybrid retrieval (dense + BM25) for any non-trivial production RAG.

---

8. Real-world Industry Usage

OpenAI

• File search in Assistants API uses hybrid retrieval internally.
• ChatGPT memory retrieval is dense-only (smaller corpus, simpler match).

Anthropic

• Contextual Retrieval research recommends dense + BM25 + reranker as the standard stack.
• Anthropic's reference architecture uses hybrid by default.

Google

• Vertex AI Search offers both dense and sparse, configurable.
• Google Search itself is hybrid (BM25 + neural ranking + many signals).

Enterprise

• Bloomberg Terminal uses hybrid: BM25 for tickers/codes, dense for narrative.
• GitHub Code Search uses hybrid: BM25 for symbol names, dense for natural language.
• Notion AI uses pure dense (their content is mostly prose).

Production patterns

Pattern	Where
Pure dense	Notion AI, NotebookLM (narrative-heavy)
Hybrid (default)	Most enterprise RAG
Dense + reranker	High-precision use cases
Hybrid + reranker	Best-in-class (Anthropic Contextual Retrieval)

---

9. Interview Questions

Beginner

1. What's a retriever? — Function: query → list of relevant documents.
2. Dense vs sparse? — Dense = embeddings (semantic). Sparse = BM25/TF-IDF (keyword).
3. What's BM25 for? — Keyword matching with frequency normalization.

Intermediate

1. Why hybrid retrieval beats pure dense? — Dense misses exact terms (codes, names). Hybrid covers both signals.
2. What's RRF? — Reciprocal Rank Fusion: combines ranked lists by 1/(k + rank) summed across retrievers.
3. TF-IDF vs BM25 — what changed? — BM25 added term saturation (k1) and document length normalization (b).
4. What's MMR for? — Diverse top-k. Avoids returning many near-duplicates.

Advanced

1. Why does score-based fusion fail and rank-based (RRF) succeed? — Dense cosine ∈ [0,1]; BM25 unbounded. Direct score sum is biased toward whichever has higher scale. RRF uses ranks (1, 2, 3) which are comparable across retrievers.
2. How tune hybrid weights? — Build eval set, grid-search weights (0.0-1.0 for one retriever, complement for the other), pick best recall@k. Often 0.6/0.4 dense-favoring works.
3. When pure BM25 still wins? — Legal contracts, code search by exact symbol, error-code lookup — anywhere terminology matters more than meaning.

System design

1. Design retrieval for a code-search product. — Dual indexes: BM25 (exact symbols, identifier matches) + dense (natural language queries → code). Hybrid + reranker. Tree-sitter splitting for code chunks.
2. Plan A/B test of dense vs hybrid retrieval. — Build both retrievers; run 1000 queries through each; compute recall@5, precision@5, MRR; compare; statistical significance test.

---

10. Common Mistakes

Beginners

• ❌ Calling .get_relevant_documents() (deprecated) instead of .invoke().
• ❌ Hardcoding k=4 without measuring.
• ❌ Skipping hybrid retrieval → missing exact-term queries.
• ❌ Confusing MMR with reranking — they're different.

Production teams

• ❌ No retrieval logging — can't debug bad answers.
• ❌ BM25 in-memory at 100M chunks → OOM. Use a proper BM25 store (Elasticsearch, OpenSearch, Vespa).
• ❌ Not tuning hybrid weights → suboptimal recall.
• ❌ Using dense for queries with exact identifiers (error codes, SKUs).

How to avoid

• Log (query, retrieved_chunks, scores) to LangSmith.
• For corpora >10M: serve BM25 from Elasticsearch alongside the vector store.
• Build a RAGAS eval set; tune weights via grid search.
• Detect identifier-heavy queries; route to BM25-favored retrieval.

---

11. Best Practices

Industry standards

• Hybrid retrieval (dense + BM25) is the default for production.
• Use .invoke() (modern API); not .get_relevant_documents().
• Log every retrieval — query, retriever name, returned chunks, scores.
• Always filter by metadata in multi-tenant.

Production

• Tune k per use case — 3-5 for chat, 8-15 for summarization.
• Threshold filtering for safety-critical paths.
• Retrieve broadly (k=20), rerank to k=5 (Chapter 7) for high quality.
• Cache common queries at the retriever layer.

Optimization

• Pre-build the BM25 index offline; reload at startup.
• Use async retrievers in async pipelines.
• Batch retrieval if processing many queries.

---

12. Evolution Story

flowchart LR
    A[TF-IDF<br/>keyword frequency] --> B[BM25<br/>term saturation + length norm]
    B --> C[Dense embedding retrievers<br/>semantic match]
    C --> D[Hybrid dense + sparse<br/>RRF fusion]
    D --> E[Multi-query retrievers<br/>paraphrase expansion]
    E --> F[Self-query retrievers<br/>NL filters]
    F --> G[Reranking<br/>cross-encoder precision]

Where we are: Retrievers are a composable layer. Choose dense, sparse, or hybrid based on workload.

Where we're going (next chapter): Production RAG needs more than just dense+sparse. We'll cover Advanced Retrievers — Multi-Query (paraphrase coverage), Self-Query (NL → metadata filter), Parent-Document (small match + big context), Contextual Compression (filter irrelevant sentences), and the king of all: Reranking with cross-encoders.

---

Practice

What does this print?

Expected: True

retrieval_paradigms = ["dense", "sparse", "hybrid"]
print(len(retrieval_paradigms) == 3)

Use hybrid (not pure dense) for queries with error codes / SKUs

Expected: True

strategy = "dense_only"           # bug: misses exact-term matches
is_hybrid = strategy == "hybrid"
print(not is_hybrid)

Quiz — Quick check

What you remember

Q1. Why hybrid retrieval?

• Combines semantic (dense) with exact-keyword (sparse) — each catches what the other misses
• Pure dense is deprecated
• BM25 is faster
• Required by LangChain

Q2. What's the smoothing constant k in RRF set to?

• 1
• 10
• 60 (from the original paper)
• 1000

Q3. When does BM25 outperform dense retrieval?

• Queries with exact terms (error codes, identifiers, named entities)
• Multilingual queries
• Paraphrased queries
• Long documents

Common doubts

Hybrid is more expensive — when worth it?

For any production RAG. The +10-30% recall easily justifies 2× retrieval cost (retrieval is the cheapest part of the pipeline — LLM dominates).

Can I do score-based fusion instead of RRF?

Dangerous. Dense and sparse scores live on different scales (cosine ∈ [0,1] vs BM25 unbounded). Direct sum is biased. RRF uses ranks → fair across retrievers.

Should I always use MMR?

Only when your corpus has redundancy (multiple chunks restating same fact). For diverse, non-repetitive corpora, pure similarity is fine.

→ Advanced Retrievers — Reranking, Multi-Query, Compression