Metadata
Facts stored alongside a piece of content, such as which document it came from, its date or its section, used to filter and cite.
A library book has a label on its spine. It tells you which shelf it belongs on. Each chunk needs a label like that too. Extra information stored with a chunk is called metadata. At the very least, record which document it came from and its position in that document. The title or section heading helps too.
Try it now
Each chunk starts 3 words after the one before (size minus overlap), so neighbors share one word. The loop stops once a chunk reaches the end.
Learn it properly
Chunking documents · Split long documents so the right piece can be retrieved. About 25 minutes.
Other terms from the same lesson
- chunking: Cutting long documents into smaller pieces so the most relevant pieces can be found and sent to a model.
- overlap: In chunking, the words repeated at the end of one chunk and the start of the next, so an idea split across a boundary still appears whole somewhere.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.