Agentic RAG and Variants¶

1. Why does this topic exist?¶

So far, every RAG technique we've seen is a linear pipeline: retrieve → generate. The same operations run for every query.

Real production queries don't fit one pipeline:

Query	What it really needs
"What's our refund policy?"	Single source (policy docs), single retrieval
"Did our refund policy change recently?"	Two retrievals: policy + recent updates, then comparison
"Compute the difference between this quarter and last quarter"	Retrieval + computation tool
"What does our CEO think about X?"	Web search (recent statement) + internal docs (past statements)
"Hi, how are you?"	NO retrieval needed

Industry pain example: A customer service team built linear RAG. Users typed multi-part questions like "What's our refund policy, and has it changed in the past 30 days?" — the system retrieved policy text but completely missed "past 30 days" because retrieval was one-shot. After switching to an agentic architecture that decomposes the query into sub-tasks, accuracy on multi-part queries went from 35% → 87%.

Agentic RAG puts the LLM in charge. It decides: - Which knowledge source to query. - Whether to retrieve at all. - Whether retrieved info is sufficient or needs follow-up retrieval. - When to stop and answer.

2. What is it?¶

Simple explanation¶

Agentic RAG = LLM-as-orchestrator. Instead of "retrieve then answer", the LLM loops: "think → act (retrieve/use tool) → observe → think again → ..." until it has enough info.

Technical explanation¶

Agentic RAG is a pattern where: - Knowledge sources and tools are wrapped as tools the LLM can call. - The LLM operates a ReAct loop (Reason + Act): Thought → Action → Observation → Thought. - The control flow is dynamic — different queries take different paths through the graph.

Industry definition¶

The pattern emerged from the "ReAct" paper (Yao et al., 2022) combined with LangChain's tool-calling abstraction (2023) and LangGraph's graph-based orchestration (2024).

Mental model¶

A research assistant with access to many libraries, databases, and search tools. They don't follow a fixed procedure — they reason about what they need, fetch it from the right source, evaluate sufficiency, and iterate.

3. How does it work?¶

The ReAct loop¶

flowchart LR
    T[Thought: what do I need next] --> A[Action: call a tool]
    A --> O[Observation: tool result]
    O --> T

Each iteration, the LLM decides whether it has enough information. If not → next action. If yes → final answer.

Retrievers as tools¶

from langchain_core.tools import tool

@tool
def search_policy_handbook(query: str) -> str:
    """Search the HR policy handbook for company rules and benefits."""
    docs = policy_retriever.invoke(query)
    return "\n\n".join(d.page_content for d in docs)

@tool
def search_product_docs(query: str) -> str:
    """Search the product documentation and pricing info."""
    docs = product_retriever.invoke(query)
    return "\n\n".join(d.page_content for d in docs)

@tool
def search_web(query: str) -> str:
    """Search the public web — use for current events or external info."""
    return tavily.run(query)

The LLM reads each tool's docstring to decide which to call. Good docstrings = good routing.

Build with LangGraph¶

from langgraph.prebuilt import create_react_agent
from langchain_openai import ChatOpenAI

agent = create_react_agent(
    model=ChatOpenAI(model="gpt-4o-mini", temperature=0),
    tools=[search_policy_handbook, search_product_docs, search_web],
    prompt="""You are a helpful research assistant. Use tools to find
information. When you have enough info, answer concisely with citations.""",
)

result = agent.invoke({"messages": [("human", "What's our refund policy?")]})
print(result["messages"][-1].content)

The graph under the hood¶

flowchart TD
    START --> AGENT[Agent node - calls LLM]
    AGENT -->|tool_calls present| TOOLS[Tool node - executes]
    TOOLS --> AGENT
    AGENT -->|no tool calls| END[End]

The agent decides "call tool" or "answer". Tool node runs the tool. Repeats until done.

Agentic RAG Variants¶

There are five recognized variants, each solving a specific failure mode.

Variant 1: Adaptive RAG¶

The problem: Some queries need retrieval; some don't. Always retrieving wastes time and money.

The solution: LLM-driven router decides per query.

flowchart LR
    Q[Query] --> R[Router LLM]
    R -->|chitchat| D[Direct answer]
    R -->|knowledge question| RAG[RAG path]
    R -->|computation| TOOL[Tool path]
    R -->|complex| MULTI[Multi-hop]

def adaptive_route(query):
    classification = router_llm.invoke(f"""
Classify the query:
- "chitchat": small talk, no retrieval needed
- "factual": needs knowledge retrieval
- "computation": needs a tool/calculator
- "multi-hop": needs multiple retrievals
Query: {query}
Return only the label.
""").content.strip()
    return classification

# Then dispatch based on classification

Variant 2: Corrective RAG (CRAG)¶

The problem: Retrieval sometimes returns irrelevant chunks. The LLM hallucinates from bad context.

The solution: Add a critic node that evaluates retrieved chunks. If quality is low, re-retrieve (with a refined query) or fall back to web search.

flowchart LR
    Q[Query] --> R1[Retrieve]
    R1 --> EVAL[Critic LLM scores relevance]
    EVAL -->|good| GEN[Generate]
    EVAL -->|ambiguous| REWRITE[Rewrite query, retry]
    EVAL -->|bad| WEB[Fall back to web search]
    REWRITE --> R1
    WEB --> GEN
    GEN --> ANS[Answer]

def corrective_rag(query):
    chunks = retriever.invoke(query)
    grade = critic_llm.invoke(f"""
Rate the relevance of these chunks to the query:
- "high" if directly relevant
- "medium" if partially relevant
- "low" if irrelevant

Query: {query}
Chunks: {format_chunks(chunks)}

Return only the label.
""").content.strip()

    if grade == "high":
        return generate(query, chunks)
    elif grade == "medium":
        return generate(query, chunks + web_search(query))
    else:
        return generate(query, web_search(query))

Variant 3: Self-Reflective RAG (Self-RAG)¶

The problem: Even with good retrieval, the LLM may produce an answer that isn't fully grounded or addresses only part of the question.

The solution: The LLM self-evaluates its own output and decides whether to retrieve more or revise.

flowchart LR
    Q[Query] --> R[Retrieve]
    R --> GEN[Generate draft]
    GEN --> REF[Self-reflect: is this grounded?]
    REF -->|yes| DONE[Final]
    REF -->|no| REVISE[Revise / re-retrieve]
    REVISE --> GEN

def self_reflective_rag(query, max_iters=3):
    for i in range(max_iters):
        chunks = retriever.invoke(query)
        draft = llm.invoke(f"Answer based on context.\n{format_chunks(chunks)}\n{query}")
        reflect = llm.invoke(f"""
Is the following answer grounded in the context AND complete?
Answer: {draft}
Context: {format_chunks(chunks)}
Reply yes/no with reason.
""")
        if "yes" in reflect.content.lower():
            return draft
        # Else: refine query and retry
        query = llm.invoke(f"Rephrase to better target what's missing: {query}").content
    return draft

Variant 4: Multi-Hop RAG¶

The problem: "Who was the manager of the person who founded Acme?" — needs TWO retrievals: founder of Acme, then their manager.

The solution: Decompose the question into a sequence of dependent sub-queries.

flowchart LR
    Q[Multi-hop query] --> D[Decompose into sub-questions]
    D --> Q1[Sub-question 1]
    Q1 --> R1[Retrieve A1]
    R1 --> Q2[Sub-question 2 uses A1]
    Q2 --> R2[Retrieve A2]
    R2 --> SYNTH[Synthesize final answer]

def multi_hop_rag(query):
    # 1. Decompose
    plan = llm.invoke(f"""
Decompose this question into a sequence of dependent sub-questions:
Question: {query}
Output JSON list.
""").content
    sub_questions = parse_json(plan)

    # 2. Iterate
    context = []
    for sq in sub_questions:
        # Fill in references to previous answers
        sq_filled = fill_references(sq, context)
        chunks = retriever.invoke(sq_filled)
        partial = llm.invoke(f"Answer briefly.\n{format_chunks(chunks)}\n{sq_filled}").content
        context.append({"q": sq_filled, "a": partial})

    # 3. Synthesize
    return llm.invoke(f"Combine into final answer.\n{context}\n{query}").content

Variant 5: Multi-Agent RAG¶

The problem: Some tasks span distinct expertise — research + coding + writing. One agent with 30 tools confuses itself.

The solution: Multiple specialized agents coordinated by a supervisor.

flowchart TD
    Q[Query] --> SUP[Supervisor Agent]
    SUP -->|research| R[Researcher Agent]
    SUP -->|code| C[Coder Agent]
    SUP -->|write| W[Writer Agent]
    R --> SUP
    C --> SUP
    W --> SUP
    SUP --> ANS[Final answer]

research_agent = create_react_agent(model, tools=[search_web, search_papers])
coder_agent   = create_react_agent(model, tools=[python_repl, code_search])
writer_agent  = create_react_agent(model, tools=[doc_retriever])

def supervisor(query):
    plan = supervisor_llm.invoke(f"Which agent should handle: {query}").content
    if "research" in plan: return research_agent.invoke({"messages": [...]})
    if "code"     in plan: return coder_agent.invoke(...)
    if "write"    in plan: return writer_agent.invoke(...)

4. Visual Learning — variants side by side¶

flowchart TD
    A[Agentic RAG] --> AD[Adaptive: route per query type]
    A --> CR[Corrective: critic loops on bad retrieval]
    A --> SR[Self-Reflective: critic loops on bad answer]
    A --> MH[Multi-Hop: decompose into dependent steps]
    A --> MA[Multi-Agent: specialists + supervisor]

Decision matrix¶

Variant	When to use
Plain Linear RAG	Single-source, single-hop queries
Adaptive	Mix of chitchat + factual + computational queries
Corrective (CRAG)	Retrieval is unreliable; web fallback OK
Self-Reflective	High-stakes correctness (legal, medical)
Multi-Hop	"X via Y" type queries; cross-source reasoning
Multi-Agent	Wildly different sub-tasks (research + code + UI)

5-7. Pros / Cons / Trade-offs¶

Pros¶

Handles complex, multi-part queries.
Dynamic — no wasted retrievals on chitchat.
Self-correcting — bad retrieval can be detected and fixed.
Composable — easy to add a new tool/source.

Cons¶

3-5× more LLM calls than linear RAG.
Latency: 1-10 seconds (not 100ms).
Debugging complexity — graph state, multiple LLM calls.
Recursion limits to prevent runaway loops.

Trade-offs¶

Choice	Trade-off
Agentic vs Linear	Flexibility vs cost/latency
Plain agent vs Corrective	Simpler vs higher quality
Single agent vs Multi-Agent	Cheaper vs broader expertise
Cheap router LLM vs strong LLM	Speed vs routing accuracy

8. Real-world Industry Usage¶

OpenAI¶

ChatGPT Plus with browsing + DALL-E + code interpreter is multi-agent / multi-tool agentic.

Anthropic¶

Claude's "computer use" is agentic — Claude decides which tool to invoke and reasons about screenshots.

Google¶

Gemini Agents for productivity (Workspace integration) — schedule, search, write.

Enterprise¶

Perplexity AI uses adaptive routing: chitchat → bypass; factual → web RAG; technical → academic RAG.
Microsoft Copilot 365 is multi-agent — different agents for Word, Excel, Outlook contexts.
Notion AI Q&A is corrective — retrieves, evaluates, optionally re-retrieves.
JPMorgan's COiN uses multi-hop for cross-filing analysis.

9. Interview Questions¶

Beginner¶

What's agentic RAG? — LLM-as-orchestrator over tools.
Why use it over linear RAG? — Handles complex queries; chooses sources dynamically.
What's the ReAct loop? — Thought → Action → Observation, repeated.

Intermediate¶

Corrective RAG — what's the "correction" step? — Critic evaluates retrieved chunks; if bad, re-retrieve with refined query or fall back to web.
Self-Reflective — what gets reflected on? — The LLM's own answer — is it grounded, complete?
Multi-Hop — give an example. — "Who is the CEO of the company that acquired Acme?" — needs two retrievals (find acquirer, then find their CEO).

Advanced¶

Why set recursion_limit? — Prevent infinite loops if the agent gets confused.
Multi-Agent supervisor — how does it route? — Supervisor LLM reads tool descriptions of each agent; selects via LLM-as-router.
Adaptive RAG vs Corrective RAG — what's different? — Adaptive routes BEFORE retrieval based on query type. Corrective evaluates AFTER retrieval based on results.
What's the cost trap? — Each agent step = one LLM call. 5-step loop × 1000 queries/day × $0.001 = $5/day. Looks small until you have 100K queries/day.

System design¶

Design Agentic RAG for a multi-source enterprise. — Tools: SQL agent for structured data; vector RAG for docs; web search for current events. Supervisor routes. Tracing via LangSmith.
An agent loops 20 times before finding the answer. Diagnose. — Likely: vague tool descriptions causing wrong tools; or query genuinely complex; or tool returning useless results. Fix: sharpen descriptions, set lower recursion_limit, add a "guard" critic.

10. Common Mistakes¶

❌ Vague tool descriptions → wrong routing.
❌ Overlapping tools (two "search docs" tools) confuse the LLM.
❌ No recursion_limit → cost spirals.
❌ Using a weak LLM as agent (it picks wrong tools).
❌ Treating agentic as "magic" — it amplifies LLM mistakes.

11. Best Practices¶

Sharp tool docstrings — describe WHAT and WHEN (with examples).
Bound recursion — recursion_limit=10-25.
Use a cheap fast model for routing, strong model for synthesis.
LangSmith tracing — agents are unobservable without it.
Add critic nodes for high-stakes paths (Corrective / Self-Reflective).

12. Evolution Story¶

flowchart LR
    A[Linear RAG] --> B[Adaptive: route per query]
    B --> C[Corrective: critic on retrieval]
    C --> D[Self-Reflective: critic on answer]
    D --> E[Multi-Hop: decomposed queries]
    E --> F[Multi-Agent: specialists]

Where we are: Agentic RAG is the production pattern for non-trivial RAG. Pick the simplest variant that handles your traffic.

Where we're going: Most RAG so far has been on text chunks. The next chapter introduces Graph RAG — what if your data has RELATIONSHIPS (people, companies, dates, links)? Knowledge graphs unlock multi-hop reasoning over structured facts.

Practice¶

What does this print?

Expected: 3

react = ["thought", "action", "observation"]
print(len(react))

Set recursion_limit so the agent can't loop forever

Expected: True

config = {}                  # bug: no recursion_limit
safe = "recursion_limit" in config
print(not safe)

Quiz — Quick check¶

What you remember

Q1. Corrective RAG adds…

A critic that evaluates retrieved chunks; falls back/re-retrieves if bad
A second embedder
More tools
Streaming

Q2. Adaptive RAG decides…

Whether/how to retrieve per query
Chunk size
Embedder
Tool count

Q3. Multi-Hop RAG requires…

Sequential, dependent retrievals
Multiple LLMs
Multilingual data
Graph databases

Common doubts¶

Should I use Agentic RAG by default?

No. For single-source factual Q&A, linear RAG is faster and cheaper. Agentic shines when queries span sources or need self-correction.

Adaptive + Corrective — can I combine?

Yes — common pattern. Adaptive routes the query → if "factual", path goes through corrective retrieval; if "computation", goes through tool path.

How do I debug a confused agent?

LangSmith. Trace each Thought/Action/Observation. Look for: wrong tool picked → fix docstring. Tool returns garbage → fix data. Agent loops → set recursion_limit + investigate.

→ Graph RAG