Glossary · Phase 5 · Transformers
Pretraining
The first, huge training run where a model learns language by predicting the next token across vast amounts of text.
Try it now
Most of the engineering effort here goes into data, not into the model.
Learn it properly
Pretraining at scale · Data, compute and what models learn from the internet. About 35 minutes.
Other terms from the same lesson
- base model: A model straight out of pretraining: it continues text but doesn't yet follow instructions.
- Jaccard similarity: How alike two sets are: the number of items they share, divided by the number of different items in both. 1 means identical, 0 means nothing in common.
- MinHash: A hashing trick that estimates Jaccard similarity from short signatures, so near-duplicate documents can be found without comparing every pair.
- shingle: A short run of words in a row, such as three, taken from a document. Comparing two documents' shingles shows how much text they share.
- utilization: The share of a chip's peak speed that a training run actually gets. Waiting on memory and on other chips keeps it well below 100%.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.