Document Loaders

1. Why does this topic exist?

In Chapter 1, we said RAG starts with "load your documents." But your documents live everywhere:

• 📄 PDFs on a file server
• 🌐 HTML pages on the internet
• 📊 CSV exports from spreadsheets
• 🗄️ Rows in a Postgres database
• 📓 Notion pages, Slack threads, GitHub issues
• 📧 Gmail archives
• 🎥 Audio transcripts (Whisper output)

Each format has its own parsing library, its own quirks, its own "gotcha" of how to extract clean text. Before RAG frameworks, teams wrote bespoke ETL pipelines per source. Pain points:

1. Inconsistent output shape. PyPDF returns one thing; BeautifulSoup another; pyodbc rows another. Downstream code branched per source.
2. Lost metadata. Where did this text come from? Which page? Which row? Who authored it? Gone unless explicitly tracked.
3. Re-inventing the wheel. Every team writes "load PDF" code that subtly mis-handles columns, headers, footers.
4. Encoding hell. UTF-8 vs Latin-1 vs Windows-1252 vs Mac-Roman.

Industry pain example: A legal-tech startup spent 3 weeks writing PDF parsers for various law-firm document formats — and they STILL got tables wrong. They reinvented the wheel four different ways.

Document Loaders are the answer: a uniform abstraction that says "give me any file/URL/DB row, I'll give you back a Document."

---

2. What is it?

Simple explanation

A Document Loader is a translator. It reads files in any format and outputs them in one standard shape that the rest of the RAG pipeline understands.

Technical explanation

A Document Loader is a class implementing BaseLoader. Its .load() method returns a List[Document], where each Document has:

class Document:
    page_content: str       # the actual text
    metadata: dict          # source, page, author, tags...

Industry definition

The pattern comes from LangChain (2022). Today, every RAG framework (LlamaIndex, Haystack, Spring AI) has the same abstraction — a clear signal of "this is the right shape."

Mental model

Think of Document as a business card for a chunk of knowledge:

• Front (page_content): the actual text — what the embedder will read.
• Back (metadata): who, when, where, why — the context the embedder ignores but retrieval and citations use.

Analogy

A loader is like a postal worker: regardless of whether your parcel came from a PDF, a database, or a website, they unbox it, slap a uniform label on it, and put it in the same conveyor belt for downstream processing.

---

3. How does it work?

flowchart LR
    A[Raw source: PDF, HTML, CSV, DB, URL] --> B[Loader-specific parser]
    B --> C[Extracted text + structure]
    C --> D[Document objects with page_content + metadata]
    D --> E[Downstream: splitter]

The Document shape

from langchain_core.documents import Document

doc = Document(
    page_content="Pyodide is Python compiled to WebAssembly.",
    metadata={
        "source": "guide.pdf",
        "page": 12,
        "author": "Smith",
        "ingested_at": "2026-06-03",
        "doc_type": "tutorial",
    }
)

Field	What it does
page_content	Goes into the embedding model. Should be clean text only.
metadata	Used for filtering, citations, debugging. Can have any keys.

Loader execution flow

sequenceDiagram
    participant App
    participant Loader
    participant Parser as Format-specific parser
    participant FS as File / URL / DB
    App->>Loader: load(path)
    Loader->>FS: read raw bytes
    FS-->>Loader: bytes
    Loader->>Parser: parse(bytes)
    Parser-->>Loader: text + structure
    Loader->>Loader: wrap into Documents
    Loader-->>App: List[Document]

The major loaders by source type

flowchart TD
    BL[BaseLoader] --> TF[Text files]
    BL --> PDF[PDFs]
    BL --> HTML[Web pages]
    BL --> CSV[CSV / Excel]
    BL --> DB[Databases]
    BL --> API[SaaS APIs]
    BL --> AV[Audio / video]
    TF --> TXT[TextLoader]
    TF --> MD[UnstructuredMarkdownLoader]
    PDF --> PYPDF[PyPDFLoader]
    PDF --> UNS[UnstructuredPDFLoader]
    PDF --> PDFPLUMB[PDFPlumberLoader]
    HTML --> WBL[WebBaseLoader]
    HTML --> SEL[SeleniumURLLoader]
    HTML --> PLAY[PlaywrightURLLoader]
    CSV --> CSVLD[CSVLoader]
    CSV --> XL[UnstructuredExcelLoader]
    DB --> SQL[SQLDatabaseLoader]
    DB --> MONGO[MongoDB]
    API --> NOTION[NotionDBLoader]
    API --> SLACK[SlackDirectoryLoader]
    API --> GH[GitHubLoader]
    AV --> WHISPER[OpenAIWhisperParser]

LangChain ships 300+ loaders; you pick the right one for the source and they all return the same Document shape.

Code — the five workhorses

# 1. Plain text
from langchain_community.document_loaders import TextLoader
docs = TextLoader("notes.txt", encoding="utf-8").load()

# 2. PDF (one Document per page)
from langchain_community.document_loaders import PyPDFLoader
docs = PyPDFLoader("paper.pdf").load()

# 3. Web page
from langchain_community.document_loaders import WebBaseLoader
docs = WebBaseLoader(["https://example.com/article"]).load()

# 4. CSV (one Document per row)
from langchain_community.document_loaders import CSVLoader
docs = CSVLoader("products.csv").load()

# 5. Directory (recursive)
from langchain_community.document_loaders import DirectoryLoader
docs = DirectoryLoader(
    "./knowledge-base", glob="**/*.md", loader_cls=TextLoader,
).load()

Lazy loading for huge sources

loader = PyPDFLoader("huge.pdf")
for doc in loader.lazy_load():
    process_one(doc)    # one page at a time; doesn't hold whole PDF in RAM

Essential for multi-GB sources.

Custom loaders

from langchain_core.document_loaders import BaseLoader
from langchain_core.documents import Document

class MyAPILoader(BaseLoader):
    def __init__(self, api_url):
        self.api_url = api_url

    def lazy_load(self):
        import requests
        for record in requests.get(self.api_url).json():
            yield Document(
                page_content=record["text"],
                metadata={"id": record["id"], "source": self.api_url},
            )

Implementing lazy_load() is enough; the base class derives .load() automatically.

---

4. Visual Learning

Architecture — loader as the boundary

flowchart LR
    subgraph WORLD[Outside world]
        F1[PDFs]
        F2[Websites]
        F3[Databases]
        F4[SaaS APIs]
    end
    subgraph BOUNDARY[Loader boundary]
        L1[PyPDFLoader]
        L2[WebBaseLoader]
        L3[SQLLoader]
        L4[NotionDBLoader]
    end
    subgraph RAG[RAG pipeline]
        DOCS[Documents] --> SPL[Splitter]
        SPL --> EMB[Embedder]
        EMB --> VS[Vector Store]
    end
    F1 --> L1 --> DOCS
    F2 --> L2 --> DOCS
    F3 --> L3 --> DOCS
    F4 --> L4 --> DOCS

Loaders are the boundary layer between your data sources and the rest of RAG.

Workflow — typical ingestion pipeline

flowchart LR
    A[Discover sources] --> B[Pick right loader per type]
    B --> C[Load with lazy_load]
    C --> D[Augment metadata]
    D --> E[Stream to splitter]
    E --> F[Split + embed + store]

Sequence — bulk ingestion job

sequenceDiagram
    participant Cron
    participant Worker
    participant Storage as Object Storage
    participant Loader
    participant Embedder
    participant VS as Vector Store
    Cron->>Worker: trigger ingestion
    Worker->>Storage: list new files since last run
    Storage-->>Worker: file list
    loop For each file
        Worker->>Loader: load(file)
        Loader-->>Worker: List[Document]
        Worker->>Embedder: embed chunks
        Embedder-->>Worker: vectors
        Worker->>VS: upsert by doc_id
    end
    Worker->>Cron: done, log stats

Real-world example — a legal contract pipeline

flowchart LR
    S[S3 bucket: contracts] --> WORK[Ingestion worker]
    WORK --> PDF[PyPDFLoader]
    PDF --> META[Augment metadata:<br/>client_id, contract_type, signed_date]
    META --> NER[Extract entities: parties, amounts, dates]
    NER --> SPL[Splitter]
    SPL --> EMB[Embedder]
    EMB --> VS[Vector Store]

Heavy enrichment at load time pays off forever at query time (you can filter "show me only NDAs signed in 2024").

---

5. Pros

Benefit	Detail
Uniform shape	Downstream code is loader-agnostic. Swap PDF for HTML, no changes.
300+ pre-built loaders	Notion, Slack, GitHub, S3, Postgres, MongoDB — community-maintained.
Metadata propagation	Every chunk knows where it came from. Citations are automatic.
Lazy loading	Memory-efficient for huge corpora.
Composable	DirectoryLoader(loader_cls=X) parallelizes load across files.
Async support	Most loaders have aload() for concurrent ingestion.

6. Cons

Limitation	Detail
Quality varies	PyPDFLoader is fast but mangles tables. UnstructuredPDFLoader is layout-aware but slow.
Encoding pitfalls	TextLoader("file.txt") fails on non-UTF-8. Explicit encoding required.
Web JS	WebBaseLoader doesn't execute JavaScript. SPAs need Selenium/Playwright.
Rate limits	API loaders (Notion, Slack) hit rate limits on large dumps.
Memory	.load() returns all docs at once — bad for huge files. Use .lazy_load().
PDF tables	Almost all PDF loaders struggle with multi-column tables.
Incremental updates	Loading is one-shot; you'll write logic to detect "what's new".

7. Trade-offs

Decision	Trade-off
PyPDFLoader vs UnstructuredPDFLoader	Fast & lossy vs slow & layout-aware
.load() vs .lazy_load()	Simple code vs memory efficiency
Eager metadata enrichment	Indexing-time cost vs query-time speed
WebBaseLoader vs Playwright	Speed vs JS-rendering capability
Custom loader vs community loader	Control vs maintenance

When to use what:

• PyPDFLoader: simple PDFs (papers, manuals).
• UnstructuredPDFLoader: forms, invoices, anything with tables.
• PDFPlumberLoader: precise table extraction is critical.
• WebBaseLoader: static HTML pages.
• Playwright/Selenium: JS-rendered SPAs (only when needed — slow).
• SQLDatabaseLoader: structured row-by-row data.
• NotionDBLoader / SlackDirectoryLoader: knowledge inside SaaS tools.

---

8. Real-world Industry Usage

OpenAI

• The Assistants File Search uses internal multi-format loaders that detect PDF/DOCX/HTML/CSV and route to the right parser.

Anthropic

• Claude Projects accept up to 200 files and run them through document loaders before chunking.

Google

• Vertex AI Search Datastore loads from Cloud Storage with auto-format detection.

Enterprise

• Bloomberg uses custom loaders for financial filings (10-K, 10-Q) that understand the XBRL format.
• Salesforce Einstein has a Knowledge Article loader that respects Salesforce's permission model.
• GitHub Copilot loads from your repo via Tree-sitter aware loaders (it knows function boundaries).

Production patterns

Pattern	Where
Multi-format auto-detection	Most consumer apps
Source-versioning	Bloomberg, JPMorgan, audited workflows
Permission-aware loaders	Notion, Slack, Salesforce
Streaming ingestion	High-throughput pipelines (Kafka → loader)
Custom enrichment	Legal: parse clauses; Medical: extract drugs/dosages

---

9. Interview Questions

Beginner

1. What's the output of a LangChain loader? — A List[Document] with page_content and metadata.
2. Why metadata matters? — Filtering at retrieval, citations, debugging.
3. What's the loader for PDFs? — PyPDFLoader (fast) or UnstructuredPDFLoader (better layout).

Intermediate

1. PyPDFLoader vs UnstructuredPDFLoader — when which? — Simple text PDFs → PyPDF; complex layouts (tables, multi-column) → Unstructured.
2. How do you load 10 GB of PDFs without OOM? — lazy_load() — stream pages one at a time.
3. A web page returns empty content via WebBaseLoader. Why? — Probably SPA with JS-rendered content. Switch to Selenium/Playwright.
4. How to load only new files since last run? — Track file mtimes / hashes; pass only "new" set to loader.

Advanced

1. Design a multi-source loader that respects user permissions. — Per-source permission_filter callable; metadata carries ACL fields; vector-store query layered with ACL.
2. How to handle PDF tables for RAG? — Use UnstructuredPDFLoader with mode="elements"; extract tables separately; embed table as Markdown to preserve structure.
3. OCR — when needed and how? — Scanned PDFs (no text layer). Use UnstructuredPDFLoader(strategy="hi_res") or pre-run Tesseract.

System design

1. Build incremental indexing for a corpus that gets 1000 new docs/day. — Kafka → consumer → loader (lazy) → splitter → embedder → upsert into vector store with stable IDs. Idempotency via doc fingerprint.
2. Design a loader registry for a SaaS product where customers bring their own sources. — Plugin architecture; each loader implements BaseLoader; tested via golden fixtures; sandboxed execution (no arbitrary code).

---

10. Common Mistakes

What beginners do wrong

• ❌ Calling .load() on a 5 GB PDF. OOMs. Use .lazy_load().
• ❌ Forgetting encoding="utf-8". Fails on European or Asian text.
• ❌ Using WebBaseLoader for JS-heavy sites. Returns empty body.
• ❌ Not augmenting metadata — losing the ability to filter later.
• ❌ Loading the wrong format — Word docs with PyPDFLoader, CSVs with TextLoader.

What production teams do wrong

• ❌ No idempotency — re-runs duplicate vectors. Use stable IDs.
• ❌ No deletion handling — files removed from source still in index.
• ❌ No content fingerprint — re-embeds unchanged files (wasted $$).
• ❌ No error tracking — one bad PDF kills the whole batch silently.
• ❌ Synchronous loading in a web handler — should be async or background job.

How to avoid

• Track (source_id, content_hash, last_modified) per doc.
• Use LangChain's Indexing API which handles deduplication, deletion, and incremental upsert.
• Run loaders in workers, not request paths.
• Test each loader with golden fixtures of malformed input.

---

11. Best Practices

Industry standards

• One loader per source type. Don't mix formats in one loader class.
• Metadata is sacred. Set source, id, ingested_at, version for EVERY document.
• Stream when possible. lazy_load() over .load() for >100 MB sources.
• Log loading stats — file count, byte size, errors per file.

Production recommendations

• Use Object Storage (S3, GCS) for raw files; loaders read from there.
• Async loaders for I/O-bound sources (APIs, web).
• Validate content — empty page_content is a bug; emit metric.
• Sandbox custom loaders — they can execute arbitrary code via Python.

Optimization

• Parallel directory loading — DirectoryLoader with use_multithreading=True.
• Cached parsing — fingerprint the file; reuse last parse if unchanged.
• OCR only when needed — check for text layer first; OCR is 100× slower.

---

12. Evolution Story

Where we were, and where we're going.

flowchart LR
    A[Bespoke per-source ETL<br/>everyone writes own PDF parser]
    A --> B[Format libraries<br/>PyPDF, BeautifulSoup, pyodbc]
    B --> C[Inconsistent shapes<br/>downstream branching]
    C --> D[Document Loaders<br/>uniform Document interface]
    D --> E[Pre-built ecosystem<br/>300+ loaders]
    E --> F[Indexing API<br/>idempotent + incremental]
    F --> G[Permission-aware<br/>multi-tenant safe loading]

Where we are: Loading is "solved" — pick a loader, get a uniform output, augment metadata.

Where we're going (next chapter): Even with clean Documents, we can't embed them whole — they're too long. We need a Text Splitter that chops them into the right granularity for embedding and retrieval. And not all splitters are created equal: Fixed, Recursive, Semantic, Agentic, Hierarchical, Contextual, Late chunking — each solves a specific failure mode.

---

Practice

What does this print?

Expected: 2

documents = [
    {"page_content": "Page 1", "metadata": {"page": 1}},
    {"page_content": "Page 2", "metadata": {"page": 2}},
]
print(len(documents))

Set encoding='utf-8' to handle a UTF-8 file with accents/emoji

Expected: True

encoding = "ascii"           # bug: fails on non-ASCII
is_safe = encoding in ("utf-8", "utf8")
print(not is_safe)

Quiz — Quick check

What you remember

Q1. What does a Document Loader return?

• A List[Document] with page_content + metadata
• Just a string
• Raw bytes
• A Pandas DataFrame

Q2. For PDFs with complex tables and multi-column layouts, which is better?

• PyPDFLoader
• UnstructuredPDFLoader (layout-aware)
• TextLoader
• CSVLoader

Q3. Why use lazy_load()?

• Stream documents one at a time — memory-efficient for huge sources
• Faster than .load() always
• Required by LangChain
• Provides better metadata

Common doubts

Should I write my own loader or use community ones?

Community first. With 300+ loaders, there's likely one for your source. Write your own only for proprietary APIs or oddball formats.

How do I handle incremental updates?

Use LangChain's Indexing API. It tracks fingerprints (hash of content) per doc ID; on each ingestion run, it upserts changed docs and deletes removed ones. No more dedup logic in your app.

Should I clean text at load time or splitting time?

Both. Load: strip boilerplate (headers, footers, page numbers) the loader missed. Pre-split: normalize whitespace, fix encoding. Don't over-strip — you may remove signal the retriever needs.

→ Text Splitters & Chunking Strategies