Chunking documents

Three ways to cut

There are three common ways to cut a document into chunks.

1. Fixed size with overlap. Take a set number of words (or tokens) for each chunk. Then start the next chunk a little before the last one ended. So each chunk repeats the end of the one before.

Think of taking a panorama on your phone. Each photo overlaps the last one a little, so nothing falls in a gap between shots. Overlap does the same for text. A sentence cut at a chunk boundary still appears whole in one of the two chunks.

Two numbers control this. The size is how many words go in each chunk. The overlap is how many words neighbors share. From those you get the step, how far each chunk starts after the one before:

step = size − overlap

In plain words: move forward by the chunk size, then back up by the overlap. That's where the next chunk starts.

Here's a small example. Take ten words, a to j. Use a size of 4 and an overlap of 1. The step is 4 − 1 = 3, so chunks start at word 0, then 3, then 6. (Python counts positions from 0.)

• Start at 0: a b c d
• Start at 3: d e f g
• Start at 6: g h i j

That third chunk reaches the last word, so you stop. Neighbors share one word: d, then g.

In Python, range(0, len(words), step) gives the start positions 0, 3, 6 and so on. The slice words[start:start + size] takes the chunk. Step through it below.

words = "a b c d e f g h i j".split()
size, overlap = 5, 2

2. Along the document's structure. Cut where the writer already made breaks: at headings, or at blank lines between paragraphs. Each chunk is then a natural unit, like one recipe in a cookbook. Paragraphs vary in size, though. So you merge paragraphs that are too small and split ones that are too big.

3. Recursive splitting. "Recursive" means the same rule is applied again on smaller and smaller pieces. Think of cutting a book. You try chapters first. If a chapter is still too long, you cut it into sections. Then paragraphs, then sentences, then words, only as far as you need. Most RAG libraries do this by default.

How big?

There's no single right size. Common starting points are 200 to 500 words per chunk. That's roughly 250 to 700 tokens. Use 10 to 20% overlap.

In plain words: the overlap is a slice of the chunk size. For 15% overlap, multiply the size by 0.15.

For example, take 300-word chunks with 15% overlap. The overlap is 300 × 0.15 = 45 words. So the step is 300 − 45 = 255 words.

size = 300                     # words per chunk
overlap = round(size * 0.15)   # 15% overlap
print("overlap:", overlap, "words")
print("step:", size - overlap, "words")

This prints an overlap of 45 words and a step of 255 words. round turns the result into a whole number of words.

Size is a trade-off. Think of a map app. Zoom in and you see exact streets, but not the town around them. Zoom out and you see the whole town, but the details blur.

• Smaller chunks give sharper search, because each embedding is about one thing. But each passage carries less context, and a fact can get split.
• Bigger chunks carry more context. But their embeddings blur, because they mix many topics.

The best size depends on your documents and your questions. Test it with real questions, as you'll learn in the evals lesson.

Keep track of where each chunk came from

A library book has a label on its spine. It tells you which shelf it belongs on. Each chunk needs a label like that too. Extra information stored with a chunk is called metadata. At the very least, record which document it came from and its position in that document. The title or section heading helps too.

You'll use metadata in three ways:

• To cite sources. An answer can say "from Returns and refunds".
• To filter. You can search only this year's reports.
• To rebuild context. When a chunk matches, you can fetch the chunks next to it.

The worked example below does this for every article in the practice help center.

import requests

docs = requests.get("https://api.gradient.test/v1/docs").json()["data"]

chunks = []
for doc in docs:
    for position, paragraph in enumerate(p for p in doc["text"].split("\n\n") if p.strip()):
        chunks.append({
            "id": f"{doc['id']}#{position}",
            "doc_id": doc["id"],
            "title": doc["title"],
            "text": paragraph.strip(),
        })

print(len(docs), "documents became", len(chunks), "chunks")
print(chunks[-1]["id"], "|", chunks[-1]["title"], "|", chunks[-1]["text"][:60])