Glossary · Phase 1 · Building with LLMs
Tokenization
Splitting text into tokens, and turning each into a number the model can use.
Try it now
This splitter copies a real tokenizer's habits, but it isn't one. Delete the link and watch the characters per token go up. Plain English packs more characters into each token.
Learn it properly
Tokens and the context window · How text becomes tokens, and what fits in one request. About 25 minutes.
Other terms from the same lesson
- context window: The most text a model can take in at once, measured in tokens: its short-term memory.
- proxy: A server that passes your requests on to another service, often adding the secret key so it never reaches the browser.
- token: A chunk of text a model reads and writes, often a word or part of a word. Models count and charge by tokens.
- token ID: The number a tokenizer gives each token. Models work with these numbers, not with the text itself.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.