Skip to content

Text Splitters

1. Why this matters

You can't embed or stuff a 100-page PDF into a prompt — embedding models cap around 8k tokens; LLM contexts have practical limits too. You must split into chunks.

Bad chunking destroys RAG. If you split mid-sentence, the embedding is meaningless and retrieval misses the answer. If chunks are too big, you waste context window and dilute relevance. If too small, the LLM gets fragments without enough context to answer.

Getting the splitter right is one of the highest-leverage things in a RAG pipeline.

2. Mental model

A splitter is a function: Document(big) → list[Document(small)].

It tries to split at natural boundaries (paragraph → sentence → word → character) and includes some overlap between adjacent chunks so context isn't lost at boundaries.

flowchart LR
    D[Big Document<br/>10,000 chars] --> S[RecursiveCharacterTextSplitter<br/>chunk_size=1000<br/>chunk_overlap=200]
    S --> C1[Chunk 1<br/>chars 0-1000]
    S --> C2[Chunk 2<br/>chars 800-1800]
    S --> C3[Chunk 3<br/>chars 1600-2600]
    S --> C4[...]

3. Architecture / Flow

The recursive strategy in pictures — try big separators first, fall back to smaller ones:

flowchart TD
    A[Try splitting on '\\n\\n' paragraph] -->|chunks still too big| B[Try '\\n' line]
    B -->|still too big| C[Try ' ' word]
    C -->|still too big| D[Hard split on char]
    A -->|chunks ≤ chunk_size| E[Add overlap, emit]
    B -->|chunks ≤ chunk_size| E
    C -->|chunks ≤ chunk_size| E
    D --> E

4. Core concepts

  • chunk_size — target maximum length per chunk (in characters by default, or tokens if you opt in).
  • chunk_overlap — chars from the END of chunk N that are repeated at the START of chunk N+1. Keeps context across boundaries.
  • length_function — defaults to len (chars). Pass tiktoken_len to count tokens instead.
  • separators — ordered list the recursive splitter tries (default: ["\n\n", "\n", " ", ""]).
  • add_start_index — if True, adds start_index to metadata (useful for citing exact position in source).

5. Code — minimal working example

from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import PyPDFLoader

docs = PyPDFLoader("./report.pdf").load()    # list[Document]

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    add_start_index=True,
)
chunks = splitter.split_documents(docs)

print(f"{len(docs)} docs → {len(chunks)} chunks")
print(chunks[0].page_content[:300])
print(chunks[0].metadata)   # carries forward source + adds start_index

6. Code — real-world pattern

Token-aware splitting (matches the embedding model's tokenizer, more accurate budgeting):

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Counts in tokens using the OpenAI embedding tokenizer
splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base",
    chunk_size=512,        # tokens, not chars
    chunk_overlap=64,
)
chunks = splitter.split_documents(docs)

Code splitting (knows about language structure — splits at function/class boundaries):

from langchain_text_splitters import RecursiveCharacterTextSplitter, Language

py_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON,
    chunk_size=600,
    chunk_overlap=80,
)
py_chunks = py_splitter.create_documents([open("main.py").read()])

Markdown header-aware splitting (preserves section context):

from langchain_text_splitters import MarkdownHeaderTextSplitter

md_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "H1"), ("##", "H2"), ("###", "H3")],
    strip_headers=False,
)
md_chunks = md_splitter.split_text(open("README.md").read())
# Each chunk's metadata now includes the H1/H2/H3 hierarchy

Semantic splitting (advanced — splits where the meaning shifts):

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

splitter = SemanticChunker(
    OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95,
)
chunks = splitter.create_documents([long_text])

7. Common pitfalls

  • ❗ Using CharacterTextSplitter (non-recursive) by mistake. It splits ONLY on its single separator, often creating wildly uneven chunks. The recursive one is almost always better.
  • ❗ Setting chunk_overlap=0. You'll lose context at every boundary. Use ~10–20% of chunk_size (e.g., 200 for 1000-char chunks).
  • ❗ Measuring chunk_size in chars when budgeting against an embedding model with token limits. A 1500-char chunk can be 200–500 tokens depending on language and content. Use from_tiktoken_encoder for accurate budgeting.
  • ❗ One-size-fits-all chunk size. Q&A over short FAQ entries wants small chunks (200–400). Legal contracts want larger (1500–2000). Tune to your data.
  • ❗ Splitting code with the default splitter. It'll cut mid-function. Use Language.PYTHON / JS / GO / ... variants.

8. When to use vs not use

Splitter When
RecursiveCharacterTextSplitter Default — use unless you have a specific reason
from_tiktoken_encoder(...) variant When chunks must fit a token budget exactly
RecursiveCharacterTextSplitter.from_language(...) Splitting code
MarkdownHeaderTextSplitter Markdown / docs where section hierarchy matters for retrieval
HTMLHeaderTextSplitter HTML with semantic headers
SemanticChunker (experimental) High-quality RAG where embedding cost is acceptable
CharacterTextSplitter (non-recursive) Almost never

9. Cheatsheet

from langchain_text_splitters import (
    RecursiveCharacterTextSplitter,
    CharacterTextSplitter,
    TokenTextSplitter,
    MarkdownHeaderTextSplitter,
    HTMLHeaderTextSplitter,
    Language,
)

# THE 90% answer
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    length_function=len,
    add_start_index=True,
    separators=["\n\n", "\n", ". ", " ", ""],  # default
)

# Token-aware (recommended for production RAG)
RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base",   # OpenAI embeddings/GPT-4 tokenizer
    chunk_size=512,
    chunk_overlap=64,
)

# Code
RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON,   # JS / TS / GO / JAVA / CPP / RUST / ...
    chunk_size=600, chunk_overlap=80,
)

# Apply
chunks = splitter.split_documents(docs)        # list[Document] → list[Document]
chunks = splitter.create_documents([text])     # raw strings → list[Document]
chunks = splitter.split_text(big_string)       # raw → list[str]

Rule-of-thumb starting points:

Content type chunk_size (chars) chunk_overlap
FAQs / short Q&A 200–400 50
General docs / articles 800–1200 150–250
Long PDFs / books 1500–2000 200–400
Code 500–1000 (use language splitter) 50–100

10. Q&A — recall test

  • Q: What's the difference between CharacterTextSplitter and RecursiveCharacterTextSplitter? A: CharacterTextSplitter splits on ONE separator only. RecursiveCharacterTextSplitter tries a hierarchy (\n\n → \n → → ``) so it splits at the most natural boundary that keeps chunks under size. Always prefer the recursive one.

  • Q: Why have chunk_overlap at all? A: Without overlap, a sentence split exactly at the chunk boundary becomes incomprehensible in both chunks. Overlap preserves enough context across boundaries that retrieval still works.

  • Q: Should chunk_size be in characters or tokens? A: Tokens are more accurate for budgeting against model limits, but chars are simpler and fine when chunks are well under the limit. Use from_tiktoken_encoder for token-based.

  • Q: Picking chunk_size? A: Start at 1000 chars with 200 overlap. If retrieval quality is poor: try smaller (more granular) for short factoid queries, larger for narrative content. Measure with a small eval set.

  • Q: When does Markdown-aware splitting matter? A: When your retrieval needs section context. MarkdownHeaderTextSplitter records H1/H2/H3 in metadata, so a chunk pulled from "Refund Policy → Eligibility → Time Window" can be cited and filtered accordingly.

Practice

What does this print?

Expected: True

# chunk_size = max characters per chunk; chunk_overlap = chars shared between consecutive chunks
chunk_size = 1000
chunk_overlap = 200
print(chunk_overlap < chunk_size)

Choose a chunk_overlap that's about 10-20% of chunk_size (not 0)

Expected: True

chunk_size = 1000
chunk_overlap = 0                  # bug: 0 overlap → chunks can cut sentences in half with no recovery
good_overlap = 100 <= chunk_overlap <= 200
print(not good_overlap)

Quiz — Quick check

What you remember

Q1. Why use chunk_overlap between chunks?

  • Ensures concepts straddling a chunk boundary appear in BOTH chunks — retrieval can still find them
  • To save memory
  • Speeds up embedding
  • Required by the vector store

Why: Without overlap, a sentence might be split — retrieval misses it. With overlap, the same text appears in two consecutive chunks, increasing recall at the cost of some redundancy.

Q2. Which splitter should you use for code?

  • CharacterTextSplitter
  • RecursiveCharacterTextSplitter.from_language(language="python") or CodeTextSplitter
  • MarkdownHeaderTextSplitter
  • TokenTextSplitter

Why: Code-aware splitters respect function/class boundaries — they don't split in the middle of a function body. Generic splitters can chop a def in half, breaking semantic coherence.

Q3. What's a good default chunk_size for general documents?

  • 100
  • 5000
  • 500-1500 (tokens or chars depending on the splitter)
  • 10000

Why: Too small (100) loses context — chunks become semantically thin. Too large (10000) reduces retrieval precision — each chunk covers many topics, hurting relevance. ~1000 is a common sweet spot.

Common doubts

Should chunk_size be in tokens or characters?

Tokens are more accurate (LLMs work in tokens), but characters are simpler. RecursiveCharacterTextSplitter uses chars by default. For precise context-window planning, use TokenTextSplitter with the same tokenizer the model uses.

What's the right chunk size for my use case?

Start with 1000 chars / 200 overlap. Tune by retrieval quality, not theory. Try a few sizes (500, 1000, 1500) and measure end-to-end answer quality. Bigger chunks = more context per result; smaller chunks = more focused matches.

How do I preserve document structure (headers, sections) in chunks?

Use structural splitters: MarkdownHeaderTextSplitter for Markdown, HTMLHeaderTextSplitter for HTML. These record header hierarchy in metadata so each chunk knows what section it belongs to — invaluable for citations and filtering.