Document Loaders¶

1. Why this matters¶

Real data lives in PDFs, HTML pages, S3 buckets, Confluence, GitHub, Notion, CSVs. Each format has its own parsing quirks. Without loaders you'd write file-format-specific code in every project.

LangChain ships ~100+ loaders, all returning the same list[Document] shape — so the rest of your pipeline (splitter, embedder, vector store) doesn't care where the data came from.

2. Mental model¶

A loader is a source-specific adapter that produces a standard Document object:

Document(
    page_content="The actual text...",
    metadata={"source": "report.pdf", "page": 3, "author": "Alice", ...}
)

The contract:

flowchart LR
    S[Source<br/>PDF / Web / CSV / DB] --> L[Loader<br/>.load .lazy_load]
    L --> D["List of Document<br/> page_content, metadata "]
    D --> NEXT[next step:<br/>text splitter]

metadata is critical — it's what lets you cite sources, filter retrieval, and debug RAG.

3. Architecture / Flow¶

flowchart TD
    A[TextLoader] --> Z[Document]
    B[PyPDFLoader] --> Z
    C[WebBaseLoader] --> Z
    D[CSVLoader] --> Z
    E[DirectoryLoader] --> Z
    F[YoutubeLoader] --> Z
    G[NotionDBLoader] --> Z
    H[GitLoader] --> Z
    Z --> Next[Text Splitter]
    Next --> Embed[Embedding Model]
    Embed --> VS[Vector Store]

4. Core concepts¶

Document — the universal output type: page_content: str + metadata: dict.
.load() — return all documents eagerly (fine for small sources).
.lazy_load() — generator; one document at a time. Use for large sources (a 5 GB log directory).
.aload() — async variant.
Loaders are NOT Runnables in the LCEL pipe sense — they're called explicitly during indexing, not at query time.
Loaders live in langchain_community.document_loaders (community contrib) — except a few core ones in langchain_core.

5. Code — minimal working example¶

from langchain_community.document_loaders import TextLoader

docs = TextLoader("./notes.txt", encoding="utf-8").load()

print(len(docs))                  # usually 1 doc for a single file
print(docs[0].page_content[:200])
print(docs[0].metadata)           # {"source": "./notes.txt"}

PDFs:

from langchain_community.document_loaders import PyPDFLoader

docs = PyPDFLoader("./report.pdf").load()
# One Document per PAGE
for d in docs[:2]:
    print(d.metadata["page"], "→", d.page_content[:80])

6. Code — real-world pattern¶

Load a whole folder of mixed formats with DirectoryLoader:

from langchain_community.document_loaders import (
    DirectoryLoader,
    PyPDFLoader,
    TextLoader,
    UnstructuredMarkdownLoader,
)

loader = DirectoryLoader(
    "./knowledge_base",
    glob="**/*.{pdf,md,txt}",
    loader_cls=lambda path: (
        PyPDFLoader(path) if path.endswith(".pdf")
        else UnstructuredMarkdownLoader(path) if path.endswith(".md")
        else TextLoader(path)
    ),
    show_progress=True,
    use_multithreading=True,
)
docs = loader.load()
print(f"Loaded {len(docs)} documents")

Web pages:

from langchain_community.document_loaders import WebBaseLoader

loader = WebBaseLoader([
    "https://python.langchain.com/docs/introduction/",
    "https://python.langchain.com/docs/concepts/",
])
docs = loader.load()

YouTube transcript:

from langchain_community.document_loaders import YoutubeLoader

loader = YoutubeLoader.from_youtube_url(
    "https://www.youtube.com/watch?v=pSVk-5WemQ0",
    add_video_info=True,
    language=["en", "hi"],
)
docs = loader.load()

Large source — stream with lazy_load:

loader = DirectoryLoader("./logs", glob="**/*.log")
splitter = ...  # see next chapter
chunks = []
for doc in loader.lazy_load():           # one at a time, no full-RAM blowup
    chunks.extend(splitter.split_documents([doc]))

Add custom metadata at load time (useful for citations):

docs = PyPDFLoader("./report.pdf").load()
for d in docs:
    d.metadata["doc_type"] = "annual_report"
    d.metadata["year"]     = 2025
    d.metadata["tenant"]   = "acme_corp"

7. Common pitfalls¶

❗ Loading PDFs with default settings on scanned/image-only PDFs. PyPDFLoader extracts text — if the PDF is just images, you get nothing. Use UnstructuredPDFLoader with mode="elements" or run OCR (e.g. tesseract) first.
❗ Not preserving source metadata. Without it, retrieved chunks can't be cited. Always check that metadata["source"] (or equivalent) survives through to retrieval.
❗ WebBaseLoader returning navigation/cookie-banner junk. It uses BeautifulSoup with default rules. For better results: bs_kwargs={"parse_only": SoupStrainer("article")}.
❗ DirectoryLoader silently skipping files. If loader_cls raises on one file, the whole batch can fail. Add silent_errors=True for debugging, then fix the real issue.
❗ Loading huge sources eagerly. A 10 GB log folder with .load() will OOM. Always use .lazy_load() for large sources.

8. When to use vs not use¶

Loader	Source
`TextLoader`	`.txt`, `.log`, anything plain text
`PyPDFLoader`	text-based PDFs
`UnstructuredPDFLoader`	scanned PDFs, complex layouts
`WebBaseLoader`	HTML pages — quick & simple
`RecursiveUrlLoader`	crawl a whole site / docs
`UnstructuredHTMLLoader`	local HTML files with rich structure
`CSVLoader`	each row → a Document
`JSONLoader`	structured JSON files; needs a `jq` schema
`DirectoryLoader`	bulk-load a folder with mixed formats
`NotionDBLoader`, `ConfluenceLoader`, `SlackLoader`, …	SaaS sources
`YoutubeLoader`	video transcripts
`GitLoader`	scan a git repo (filtered by extension)

When to not use LangChain loaders: when you already have a streaming pipeline (Kafka, Spark, dbt) that produces clean text — just create Document objects yourself.

9. Cheatsheet¶

# Document construction (do this manually for custom sources)
from langchain_core.documents import Document
doc = Document(page_content="...", metadata={"source": "...", "tag": "..."})

# Most common loaders
from langchain_community.document_loaders import (
    TextLoader,
    PyPDFLoader,
    PyMuPDFLoader,           # faster, better for tables
    UnstructuredPDFLoader,   # scanned PDFs (needs unstructured + tesseract)
    WebBaseLoader,
    RecursiveUrlLoader,
    DirectoryLoader,
    CSVLoader,
    JSONLoader,
    UnstructuredMarkdownLoader,
    UnstructuredHTMLLoader,
    YoutubeLoader,
    GitLoader,
    NotionDBLoader,
    ConfluenceLoader,
    SlackDirectoryLoader,
    S3DirectoryLoader,
)

# Common methods
docs = loader.load()              # eager
for d in loader.lazy_load(): ...  # streaming
docs = await loader.aload()       # async

# DirectoryLoader options
DirectoryLoader(
    path="./data",
    glob="**/*.pdf",
    loader_cls=PyPDFLoader,
    loader_kwargs={"extract_images": False},
    show_progress=True,
    use_multithreading=True,
    silent_errors=True,
)

10. Q&A — recall test¶

Q: What's the universal output type of every loader? A: list[Document], where each Document has page_content: str and metadata: dict.
Q: Why is metadata["source"] important? A: It's how you cite which document a retrieved chunk came from. Without it, your RAG app can't show "according to report.pdf page 3".
Q: When should you use lazy_load instead of load? A: When the source is large enough that holding all documents in memory at once would OOM — lazy_load yields one at a time so you can stream into the splitter.
Q: A PDF returns blank page_content. Why? A: Probably a scanned/image-only PDF. PyPDFLoader extracts text streams; if there's no text layer, you need OCR. Use UnstructuredPDFLoader with OCR enabled or pre-process with Tesseract.
Q: Does a loader work inside an LCEL pipe (loader | splitter | ...)? A: No — loaders run at indexing time, not at query time. They aren't typically composed into the query-time LCEL chain. The query-time chain starts with a retriever, which IS a Runnable.

Practice¶

What does this print?

Expected: 2

# A LangChain Document has `page_content` (str) and `metadata` (dict)
docs = [
    {"page_content": "doc 1", "metadata": {"source": "a.txt"}},
    {"page_content": "doc 2", "metadata": {"source": "b.txt"}},
]
print(len(docs))

Preserve source metadata (currently lost after splitting)

Expected: True

# When splitting docs, metadata should be COPIED to each chunk
original_metadata = {"source": "report.pdf", "page": 1}
chunks = ["chunk 1", "chunk 2", "chunk 3"]
chunks_with_meta = [{"text": c, "metadata": {}} for c in chunks]   # bug: metadata empty
print(all(c["metadata"] != {} for c in chunks_with_meta))

Quiz — Quick check¶

What you remember

Q1. When does a document loader run in a RAG pipeline?

At indexing time (once, before users query)
At query time (every user request)
During training
On the LLM

Why: Loaders + splitters + embedders + vector store build the index ONCE. Queries hit the retriever (which IS in the runtime chain). Loaders aren't usually piped into runtime chains.

Q2. Why is metadata so important on Documents?

To make them bigger
Lets you filter retrievals (e.g., "only docs from this user's company") and cite sources
Required by the splitter
Speeds up search

Why: Without metadata, retrieved chunks are anonymous. With metadata (source, page, section, author), you can filter at query time, cite responses, and debug retrieval quality.

Q3. What's the typical loader chain order?

Loader → Splitter → Embedder → Vector Store
Embedder → Loader → Splitter
Vector Store → Loader
Loader → Vector Store directly

Why: Load raw text → split into chunks → embed each chunk → store embeddings. Each step is a separate concern; you can swap any one independently.

Common doubts¶

Should I load all my data into one vector store?

For small datasets, yes. For multi-tenant apps, use namespaces or collections to isolate (each customer has their own). For very large corpora, consider hierarchical indexing — a smaller "summary" index that routes to the right detailed sub-index.

PyPDFLoader or UnstructuredPDFLoader?

PyPDFLoader is fast but loses some layout (tables, multi-column). UnstructuredPDFLoader handles complex layouts better but is slower and heavier. Start with PyPDFLoader; switch to Unstructured if your PDFs have complex tables/layouts.

How do I handle large PDFs (1000+ pages)?

Stream them — most loaders support iterating page-by-page. Process and embed pages as you go rather than loading the whole thing into memory. For really huge corpora, run loading in a separate background job.