Skip to content

HyDE — Hypothetical Document Embeddings

1. Why does this topic exist?

Vector search rests on one fragile assumption: a query embeds in a similar way to its answer. That's often false.

Query: "What is RAG?"            (10 words, question-form)
Doc:   "Retrieval-Augmented Generation (RAG) is a technique
       that combines a retrieval step with an LLM..."
       (50+ words, declarative paragraph)

These two texts have different style, length, structure. They sit in different regions of embedding space. Cosine similarity between them is moderate at best — even though they're semantically a perfect match.

Industry pain example: A research team built a semantic search over scientific papers. Users typed conversational questions; papers used technical declarative language. Recall@5 plateaued at 0.55. Adding HyDE: 0.55 → 0.83. Same corpus, same embedder, different query strategy.

The asymmetry problem is the root cause. HyDE flips the script: instead of searching with the query, search with a hypothetical answer.

2. What is it?

Simple explanation

Don't search with your QUESTION. Search with a FAKE ANSWER. The fake answer sounds like real answers in the corpus → much better matching.

Technical explanation

HyDE (Hypothetical Document Embeddings) — proposed by Gao et al. (2022) — works as follows:

  1. Given a user query, ask the LLM to invent a hypothetical document that would answer it.
  2. Embed the hypothetical document.
  3. Search the vector store with the hypothetical's embedding (not the query's).
  4. Use the retrieved (real) chunks as context for the final answer.

The hypothetical is thrown away — we only use it to bridge the query-style ↔ doc-style gap.

Industry definition

Stanford / Meta introduced HyDE for zero-shot retrieval — situations where you have no labeled (query, doc) pairs to fine-tune a retriever. It works surprisingly well even when the hypothetical is partially wrong, because the style matches the corpus.

Mental model

Imagine you're looking up "how to bake bread" in a recipe book whose pages all say "Mix flour..." rather than "Question: how do I bake bread?" If you ask the AI to write a fake recipe paragraph, then look up books with paragraphs similar to that — you find real recipes much faster than searching with the question itself.

Analogy

Style-translation. The query is in "user language"; the corpus is in "expert language." HyDE machine-translates the query into expert language first, then searches.

3. How does it work?

Workflow

flowchart LR
    Q[User Query] --> LLM1[LLM generates hypothetical doc]
    LLM1 --> HYPO[Hypothetical: a fake doc-style paragraph]
    HYPO --> EMB[Embed the hypothetical]
    EMB --> VS[Vector store search]
    VS --> REAL[Top-k REAL chunks]
    REAL --> P[Prompt with real chunks + original query]
    Q --> P
    P --> LLM2[Final LLM call]
    LLM2 --> ANS[Answer with citations]

Key: Only LLM2's answer is grounded in real chunks. The hypothetical is throwaway.

Why it works — the geometry

Imagine embedding space with two clusters:

flowchart LR
    QC[Cluster of queries: question-form, short]
    DC[Cluster of documents: declarative, long]
    QC -. distance is large .-> DC

A direct cosine between a query and a doc is moderate. But if we generate a "fake doc" from the query, the fake doc lives in the document cluster → cosine with real docs is high.

Code

from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.retrievers import BaseRetriever
from langchain_openai import ChatOpenAI


class HyDERetriever:
    def __init__(self, llm_chain, retriever: BaseRetriever):
        self.llm_chain = llm_chain
        self.retriever = retriever

    @classmethod
    def from_llm(cls, llm, retriever):
        prompt = ChatPromptTemplate.from_messages([
            ("system",
             "You are an expert document writer. Given a user query, "
             "generate a hypothetical document that would thoroughly answer it. "
             "Write as if it were a real authoritative passage from a knowledge base — "
             "NOT a response to the user. Factual, declarative tone. "
             "Output only the document text."),
            ("human", "query: {query}"),
        ])
        return cls(llm_chain=prompt | llm, retriever=retriever)

    def _generate_hypothetical(self, query: str) -> str:
        return self.llm_chain.invoke({"query": query}).content

    def invoke(self, query: str) -> list[Document]:
        hypothetical = self._generate_hypothetical(query)
        return self.retriever.invoke(hypothetical)


# Usage
hyde = HyDERetriever.from_llm(
    llm=ChatOpenAI(model="gpt-4o-mini", temperature=0),
    retriever=vector_store.as_retriever(search_kwargs={"k": 5}),
)
docs = hyde.invoke("How does pgvector handle filtering?")

4. Visual Learning

Architecture

flowchart LR
    subgraph EXP[Hypothesis generation]
        Q[Query] --> LLM[Cheap LLM]
        LLM --> H[Hypothetical doc]
    end
    subgraph RET[Retrieval with hypothesis]
        H --> EMB[Embed]
        EMB --> VS[Vector store]
        VS --> R[Real top-k chunks]
    end
    subgraph GEN[Final generation]
        R --> P[Prompt with real chunks]
        Q --> P
        P --> FLLM[Strong LLM]
        FLLM --> ANS[Final answer]
    end

Sequence

sequenceDiagram
    actor U as User
    participant App
    participant CHEAP as Cheap LLM
    participant VS
    participant FLLM as Final LLM
    U->>App: query
    App->>CHEAP: generate hypothetical doc
    CHEAP-->>App: hypothetical paragraph
    App->>VS: search with hypothetical
    VS-->>App: top-k real chunks
    App->>FLLM: prompt with chunks + original query
    FLLM-->>App: answer
    App-->>U: answer with citations

Multi-hypothesis HyDE — combining with RAG Fusion

flowchart LR
    Q[Query] --> LLM[LLM generates N hypotheticals]
    LLM --> H1[Hyp 1]
    LLM --> H2[Hyp 2]
    LLM --> H3[Hyp 3]
    H1 & H2 & H3 --> R[Retriever]
    R --> L1[List 1]
    R --> L2[List 2]
    R --> L3[List 3]
    L1 & L2 & L3 --> RRF[RRF]
    RRF --> FINAL[Top-k real chunks]

Generate multiple hypotheticals, retrieve per hypothesis, fuse with RRF. Maximum recall.

5-7. Pros / Cons / Trade-offs

Pros Cons Trade-off
Closes query-doc style gap One extra LLM call per query Use cheap fast model
+20-40% recall on vague queries Hypothesis may be wrong Doesn't matter — only style needs to match
Zero-shot — no training needed Latency overhead Cache hypothesis per query
Works across languages Cost scales with traffic Skip on well-formed queries

When HyDE helps most:

  • ✅ Conversational queries against formal corpus (legal, medical, scientific).
  • ✅ Short queries against long documents.
  • ✅ No labeled data for fine-tuning a retriever.

When HyDE doesn't help:

  • ❌ Query already in doc-style ("definition of RAG").
  • ❌ Specific identifiers (codes, IDs) — fake hypothesis won't have them.
  • ❌ Cost-sensitive paths.

8. Real-world Industry Usage

OpenAI

  • ChatGPT browse uses query-rewriting (close cousin of HyDE) before web search.

Anthropic

  • Contextual Retrieval research compares HyDE against contextual chunking; finds them complementary.

Enterprise

  • Legal tech (Harvey) uses HyDE for conversational queries against case law (heavily formal style).
  • Medical Q&A (UpToDate AI) uses HyDE for patient-style queries against clinical literature.
  • Research search (Elicit, Consensus) uses HyDE for academic paper retrieval.

Production patterns

  • HyDE + reranking is a strong combo: HyDE broadens, reranker narrows.
  • Caching hypotheticals by query hash — saves cost on repeated queries.
  • Conditional HyDE — only invoke for queries detected as "short" or "vague".

9. Interview Questions

Beginner

  1. What does HyDE do? — Generates a hypothetical document that would answer the query, then searches with that hypothetical's embedding.
  2. Why? — Closes the stylistic gap between queries and documents.
  3. Is the hypothetical the final answer? — No — thrown away; only used for retrieval.

Intermediate

  1. Why does it work even if the hypothesis is partially wrong? — Only the STYLE needs to match. The corpus has real docs with correct facts; we retrieve those.
  2. HyDE vs query rewriting? — Query rewriting produces a SHORT rewritten query. HyDE produces a FULL paragraph. HyDE typically wins on recall.
  3. Cost? — One extra LLM call per query. Use gpt-4o-mini to keep cheap.

Advanced

  1. HyDE + RAG Fusion — how combine? — Generate N hypotheticals, retrieve per hypothesis, RRF the lists.
  2. When does HyDE underperform plain retrieval? — When the query is already declarative or contains specific identifiers (codes, names) that won't be in a hallucinated hypothesis.
  3. Can you use a local LLM for the hypothesis? — Yes — quality of hypothesis doesn't need to be top-tier; even a 7B local model works for the style-translation job.

System design

  1. Plan HyDE rollout at scale. — Detect query type with a classifier; invoke HyDE only on conversational queries; cache by query hash; budget per-query LLM cost; A/B test against baseline.

10. Common Mistakes

  • ❌ Using HyDE on every query — wastes money on already-clean queries.
  • ❌ Hypothetical with high temperature → wildly off-topic → bad retrieval.
  • ❌ Mismatch between hypothesis embedder and corpus embedder.
  • ❌ Discarding original query — sometimes a fallback with original is safer.

11. Best Practices

  • Use temperature=0 for hypothesis generation — deterministic + focused.
  • Cache hypotheticals by (query_hash, model_version).
  • Detect query type; route conversational queries to HyDE, declarative to plain retrieval.
  • Combine with reranking for best results.

12. Evolution Story

flowchart LR
    A[Pure query retrieval<br/>style mismatch hurts] --> B[Query rewriting<br/>short rewrite]
    B --> C[HyDE<br/>full hypothetical paragraph]
    C --> D[Multi-HyDE<br/>N hypotheses + RRF]
    D --> E[HyDE + reranking<br/>broaden then narrow]

Where we are: HyDE is a clean trick that often gives 20-40% recall lift.

Where we're going (next chapter): All techniques so far are LINEAR pipelines — retrieve once, generate once. Agentic RAG puts the LLM in charge, letting it decide WHICH knowledge source to query, WHEN to query again, and HOW to combine results. We'll also cover the variants: Adaptive RAG, Corrective RAG, Self-Reflective RAG, Multi-Hop RAG, Multi-Agent RAG.

Practice

What does this print?

Expected: True

embed_target = "hypothetical_document"
print(embed_target == "hypothetical_document")

Use the SAME embedder for hypothesis and corpus

Expected: True

corpus_embedder = "openai-3-small"
hyde_embedder = "openai-3-large"     # bug: different models = different spaces
consistent = corpus_embedder == hyde_embedder
print(not consistent)

Quiz — Quick check

What you remember

Q1. HyDE embeds…

  • The user query
  • A hypothetical document generated by an LLM
  • Both, averaged
  • The previous answer

Q2. Why does HyDE help even when the hypothesis is partially wrong?

  • Style matters; the corpus has real facts and we retrieve those
  • LLM is always correct
  • HyDE corrects errors
  • Doesn't help if wrong

Q3. Downside?

  • Extra LLM call per query (latency + cost)
  • Requires GPT-4
  • Doesn't work with FAISS
  • Only Spanish

Common doubts

Will HyDE replace embeddings tuning?

No — they're complementary. Fine-tuning your embedder for your domain still helps. HyDE is the cheap zero-shot fix that works without fine-tuning.

Can I cache HyDE outputs?

Yes. Cache by hash(query, model_version). Saves cost dramatically on repeated queries.

When NOT to use HyDE?

Specific-identifier queries ("Error 0x80070005"), already declarative queries, or extreme latency budgets.

→ Agentic RAG and variants