Glossary · Phase 5 · Transformers
Pre-tokenization
Splitting text into rough chunks, such as words and spaces, before the tokenizer merges pieces inside each chunk.
Try it now
Common words end up as one token and rare ones as several, which explains a lot of odd LLM behavior.
Learn it properly
Tokenization and BPE · How tokenizers are built from data. About 35 minutes.
Other terms from the same lesson
- BPE: Byte-pair encoding: a way to build a token vocabulary by repeatedly merging the most common pairs of characters.
- byte-level BPE: BPE that starts from the 256 possible bytes instead of letters, so it can spell out any text at all, emoji included.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.