Graph RAG — Knowledge Graphs for Relational Reasoning¶

1. Why does this topic exist?¶

Vector RAG retrieves chunks by semantic similarity. It's great at "find me text about X". It's terrible at:

Query type	Why vector RAG fails
Multi-hop	"Who is the manager of the engineer who built X?" — vector retrieval can't traverse relationships
Relational	"Companies in our portfolio that share an investor" — needs JOIN-like logic
Aggregations	"Count employees per division reporting to manager Z"
Provenance	"Which paper cites Smith 2023?" — needs explicit edges

Industry pain example: A pharma research team built vector RAG over 100K scientific papers. Users asked "Find papers that cite Smith 2023 AND use the BERT model." Vector RAG returned random matches. After building a knowledge graph with citation edges, the query took one Cypher statement and returned exactly the right 47 papers.

Graph RAG uses knowledge graphs — entities as nodes, relationships as edges — to enable structured retrieval that vector RAG simply cannot do.

2. What is it?¶

Simple explanation¶

Graph RAG stores your data as a network of entities (people, companies, dates) connected by relationships (works_at, manages, cites). You can query both by similarity AND by traversing connections.

Technical explanation¶

Graph RAG is a RAG pattern where: 1. Documents are processed by an LLM to extract entities and relationships into a knowledge graph (typically Neo4j, Memgraph, or Amazon Neptune). 2. Queries are translated by an LLM into Cypher (or SPARQL) and executed against the graph. 3. Results are subgraphs (entities + edges), not just text chunks. 4. The final LLM synthesizes the answer from the subgraph + optionally text chunks.

Industry definition¶

The term "Graph RAG" was popularized by Microsoft Research in their 2024 paper (and LangChain's implementations). The underlying technology (knowledge graphs + LLMs) goes back to the IBM Watson era — but the cheap-LLM-as-extractor approach made it accessible.

Mental model¶

Think of your data as a detective's wall of evidence: entities pinned up with strings (relationships) connecting them. To answer "who knows whom", you trace the strings — you don't read every note.

Analogy¶

LinkedIn's "people you may know" — it's a graph traversal of (you)-[KNOWS]->(someone)-[KNOWS]->(target). Vector search would never give you that result.

3. How does it work?¶

Two-phase pipeline¶

Phase 1 — Ingestion: text → graph¶

flowchart LR
    A[Raw Documents] --> B[LLM Entity Extractor]
    B --> C[Triples: entity-rel-entity]
    C --> D[Graph DB Neo4j]

An LLM reads each chunk and extracts (subject, predicate, object) triples:

"Alice works at Acme Corp"  → (Alice, WORKS_AT, Acme Corp)
"Acme is in Mumbai"          → (Acme Corp, LOCATED_IN, Mumbai)
"Alice manages Bob"          → (Alice, MANAGES, Bob)

These become nodes and edges.

Phase 2 — Retrieval: query → subgraph → answer¶

flowchart LR
    Q[User Query] --> EX[Extract entities from query]
    EX --> CY[LLM generates Cypher query]
    CY --> NEO[Neo4j executes]
    NEO --> SUB[Relevant subgraph]
    SUB --> P[Prompt with graph context]
    P --> LLM[LLM]
    LLM --> ANS[Answer]

Example knowledge graph¶

flowchart LR
    A[Person: Alice<br/>role: CEO] -- WORKS_AT --> C[Company: Acme<br/>industry: SaaS]
    B[Person: Bob<br/>role: Engineer] -- WORKS_AT --> C
    A -- MANAGES --> B
    C -- LOCATED_IN --> D[City: Mumbai]
    C -- INVESTED_IN_BY --> E[Investor: VC1]
    F[Company: BetaCo] -- INVESTED_IN_BY --> E

Query: "Companies sharing an investor with Acme":

MATCH (a:Company {name: "Acme"})-[:INVESTED_IN_BY]->(i:Investor)<-[:INVESTED_IN_BY]-(other:Company)
RETURN other.name

Returns: BetaCo. Vector RAG could never do this.

Build with LangChain + Neo4j¶

import os
from langchain_neo4j import Neo4jGraph
from langchain_experimental.graph_transformers import LLMGraphTransformer
from langchain_openai import ChatOpenAI
from langchain_community.document_loaders import PyPDFLoader

# 1. Connect to Neo4j Aura
graph = Neo4jGraph(
    url=os.environ["NEO4J_URI"],
    username=os.environ["NEO4J_USERNAME"],
    password=os.environ["NEO4J_PASSWORD"],
)

# 2. Load + extract
docs = PyPDFLoader("companies.pdf").load()
transformer = LLMGraphTransformer(
    llm=ChatOpenAI(model="gpt-4o", temperature=0),
    allowed_nodes=["Person", "Company", "City", "Investor"],
    allowed_relationships=["WORKS_AT", "MANAGES", "LOCATED_IN", "INVESTED_IN_BY"],
)
graph_documents = transformer.convert_to_graph_documents(docs)
graph.add_graph_documents(graph_documents, baseEntityLabel=True, include_source=True)

# 3. Natural-language query
from langchain_neo4j import GraphCypherQAChain

chain = GraphCypherQAChain.from_llm(
    llm=ChatOpenAI(model="gpt-4o-mini", temperature=0),
    graph=graph,
    verbose=True,
    allow_dangerous_requests=True,
)
answer = chain.invoke({"query": "Which companies share an investor with Acme?"})
print(answer)

The chain runs two LLM calls: one to translate NL → Cypher, one to format the result.

Cypher 101¶

Cypher is to graphs what SQL is to tables.

// Find all companies Alice manages, and their industries
MATCH (p:Person {name: "Alice"})-[:MANAGES]->(c:Company)
RETURN c.name, c.industry

// Multi-hop: who manages people working at Acme
MATCH (m:Person)-[:MANAGES]->(e:Person)-[:WORKS_AT]->(c:Company {name: "Acme"})
RETURN m.name, e.name

// Counts: employees per company
MATCH (p:Person)-[:WORKS_AT]->(c:Company)
RETURN c.name, COUNT(p) AS employees
ORDER BY employees DESC

4. Visual Learning¶

Architecture — Graph RAG pipeline¶

flowchart LR
    subgraph INGEST[Ingestion]
        D[Docs] --> LLM1[LLM Entity Extractor]
        LLM1 --> N[Nodes & Edges]
        N --> GDB[Graph DB Neo4j]
    end
    subgraph QUERY[Query]
        Q[NL Query] --> LLM2[LLM NL → Cypher]
        LLM2 --> CY[Cypher]
        CY --> GDB
        GDB --> SUB[Subgraph result]
        SUB --> LLM3[LLM formats answer]
        LLM3 --> ANS[Answer]
    end

Sequence — single query¶

sequenceDiagram
    actor U as User
    participant App
    participant LLM1 as LLM Cypher Generator
    participant Neo as Neo4j
    participant LLM2 as LLM Answer Formatter
    U->>App: query
    App->>LLM1: generate Cypher
    LLM1-->>App: MATCH ... RETURN ...
    App->>Neo: execute Cypher
    Neo-->>App: rows / subgraph
    App->>LLM2: format result as NL answer
    LLM2-->>App: natural language answer
    App-->>U: answer

Hybrid Graph RAG — vector + graph¶

flowchart LR
    Q[Query] --> V[Vector retrieval - narrative chunks]
    Q --> G[Graph retrieval - structured facts]
    V --> M[Merge]
    G --> M
    M --> LLM[LLM final answer]

You get structural facts from the graph + narrative explanations from chunks.

Real-world example — pharmaceutical research¶

flowchart LR
    P[Research papers] --> EX[LLM extracts: drugs, diseases, mechanisms, citations]
    EX --> KG[Knowledge graph]
    Q[Find drugs targeting protein X with citations after 2020] --> CY[Cypher]
    CY --> KG
    KG --> SUB[Drug list + citation papers]
    SUB --> ANS[Answer with sources]

Vector RAG couldn't filter "after 2020" by edge; graph natively can.

5-7. Pros / Cons / Trade-offs¶

Pros¶

Multi-hop reasoning — traverse relationships natively.
Exact relational queries — JOIN-like semantics.
Auditable — every edge is a fact, traceable to source.
Aggregations — COUNT, SUM, GROUP BY via Cypher.
Schema-validated — entities have types, relationships have constraints.

Cons¶

Expensive ingestion — one LLM call per chunk to extract triples (10K chunks × $0.001/chunk ≈ $10).
Schema drift — without explicit constraints, LLM invents random entity types.
Cypher generation can fail — LLM produces malformed queries.
Less rich for narrative — descriptions live in chunks, not as edges.

Trade-offs¶

Choice	Trade-off
Pure Graph RAG	Best for relational queries; weak for narrative
Vector + Graph hybrid	Best of both; +ingestion cost
LLM-extracted vs hand-curated graph	Cheap but noisy vs expensive but clean
Strict schema vs free-form	Cleaner queries vs more flexibility

8. Real-world Industry Usage¶

Microsoft¶

GraphRAG library (2024) — open-source from MSR. Used internally for Bing-like research.

Google¶

Knowledge Graph is the foundation of Google Search's structured answers.
Gemini is reportedly enhanced with retrieval from internal knowledge graphs.

Enterprise¶

Pharma (Pfizer, AstraZeneca) — drug-target-disease graphs for research.
Finance (Bloomberg) — entity graphs of companies, executives, subsidiaries.
Legal (Harvey) — citation graphs of cases.
Healthcare (Mayo Clinic) — symptom-disease-treatment graphs for clinical support.

Production patterns¶

Hybrid RAG — vector for narrative, graph for relations.
Schema-first — define allowed nodes and relationships before extraction.
Incremental update — re-extract only changed documents.
Graph-aware reranking — boost chunks that connect to query entities.

9. Interview Questions¶

Beginner¶

What's a knowledge graph? — Nodes (entities) + edges (relationships).
Why use Graph RAG? — Multi-hop and relational queries that vector RAG can't handle.
What's Cypher? — Graph query language (the SQL of graphs).

Intermediate¶

How are triples extracted? — LLM reads each chunk and emits (subject, predicate, object) triples.
Why constrain allowed_nodes / allowed_relationships? — Prevents LLM from inventing random schema, keeps the graph clean.
Hybrid Graph RAG — what's the merge? — Vector chunks + graph subgraph go into the LLM prompt together.

Advanced¶

How does entity resolution work? — Merge duplicates ("OpenAI Inc." vs "OpenAI"). Run after extraction; use embeddings for canonical-name matching.
How handle relationship updates? — Re-extract on doc change; upsert by (doc_id, triple_id); tombstone removed triples.
GraphRAG vs LightRAG vs Microsoft GraphRAG? — All knowledge-graph-RAG variants; differ in extraction strategy and retrieval algorithms (community detection, hierarchical).
Cypher generation safety? — Use with_structured_output + read-only DB user; never execute write queries from LLM-generated Cypher.

System design¶

Design Graph RAG for 1M legal cases. — Extract entities (parties, judges, cited cases). Build citation graph in Neo4j. Hybrid: vector for case narrative + graph for citation traversal. Cypher generation with strict schema.
A team's Graph RAG returns empty results for valid queries. Diagnose. — Likely: LLM generated wrong Cypher (try a simpler model + few-shot examples); OR schema is too sparse (run extraction on more docs); OR entity resolution failed (duplicates).

10. Common Mistakes¶

❌ Letting LLM invent unrestricted schema → chaos.
❌ Not setting baseEntityLabel=True → can't query across types.
❌ Skipping include_source=True → can't cite the original doc.
❌ Using Graph RAG for narrative-heavy questions (vector wins there).
❌ Cypher chain without allow_dangerous_requests=True (LangChain blocks).
❌ Forgetting entity resolution → "Microsoft" and "MS" are separate nodes.

11. Best Practices¶

Schema first — define allowed_nodes and allowed_relationships.
Hybrid by default — combine vector + graph for richness.
Cite sources — every node should link back to the chunk it came from.
Read-only DB user for Cypher chain — defense in depth against malicious queries.
Cache extraction — same chunk content → same triples.
Monitor extraction quality — sample 50 docs; verify triples match human judgment.

12. Evolution Story¶

flowchart LR
    A[Vector RAG only] --> B[Multi-hop misses]
    B --> C[Manual knowledge graphs<br/>hand-curated, expensive]
    C --> D[LLM-extracted graphs<br/>cheap, noisy]
    D --> E[Schema-constrained extraction<br/>quality + scale]
    E --> F[Hybrid Vector + Graph RAG]
    F --> G[Microsoft GraphRAG<br/>community detection over graph]

Where we are: Graph RAG is essential for relational use cases. Hybrid graph+vector is the production sweet spot.

Where we're going (final chapter): We've built every RAG variant — now we need to measure them. RAGAS gives us 4 metrics (Faithfulness, Answer Relevance, Context Precision, Context Recall) and a framework for evaluating RAG quality scientifically.

Practice¶

What does this print?

Expected: True

nodes_and_edges = True
print(nodes_and_edges)

Constrain extraction with allowed_nodes / allowed_relationships

Expected: True

config = {}                # bug: no constraints
has_constraints = "allowed_nodes" in config
print(not has_constraints)

Quiz — Quick check¶

What you remember

Q1. Graph RAG is best at…

Multi-hop and relational queries
Plain free-text similarity
Image generation
Translation

Q2. What does LLMGraphTransformer do?

Uses LLM to extract entities + relationships from chunks
Stores text in Neo4j
Embeds chunks
Generates Cypher

Q3. Why hybrid (graph + vector)?

Graph for structural facts, vector for narrative
Faster
Cheaper
Required by Neo4j

Common doubts¶

Always need a Graph DB?

For prototyping with <10K nodes, in-memory NetworkX is fine. For production scale + concurrent access, use Neo4j, Memgraph, or Neptune.

How accurate is LLM extraction?

Decent (~80-90%). Plan for noise: schema constraints, entity resolution, sampling-based human review. Not "perfect" but usable.

When skip Graph RAG?

Narrative content (essays, blog posts, opinion). Forcing a graph onto unstructured prose gets you noisy extraction with little payoff.

→ RAGAS Evaluation