Glossary · Phase 5 · Transformers
BPE
Byte-pair encoding: a way to build a token vocabulary by repeatedly merging the most common pairs of characters.
Try it now
Common words end up as one token and rare ones as several, which explains a lot of odd LLM behavior.
Learn it properly
Tokenization and BPE · How tokenizers are built from data. About 35 minutes.
Other terms from the same lesson
- byte-level BPE: BPE that starts from the 256 possible bytes instead of letters, so it can spell out any text at all, emoji included.
- pre-tokenization: Splitting text into rough chunks, such as words and spaces, before the tokenizer merges pieces inside each chunk.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.