Glossary · Phase 5 · Transformers
Utilization
The share of a chip's peak speed that a training run actually gets. Waiting on memory and on other chips keeps it well below 100%.
Try it now
Most of the engineering effort here goes into data, not into the model.
Learn it properly
Pretraining at scale · Data, compute and what models learn from the internet. About 35 minutes.
Other terms from the same lesson
- base model: A model straight out of pretraining: it continues text but doesn't yet follow instructions.
- Jaccard similarity: How alike two sets are: the number of items they share, divided by the number of different items in both. 1 means identical, 0 means nothing in common.
- MinHash: A hashing trick that estimates Jaccard similarity from short signatures, so near-duplicate documents can be found without comparing every pair.
- pretraining: The first, huge training run where a model learns language by predicting the next token across vast amounts of text.
- shingle: A short run of words in a row, such as three, taken from a document. Comparing two documents' shingles shows how much text they share.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.