Glossary · Phase 5 · Transformers
Base model
A model straight out of pretraining: it continues text but doesn't yet follow instructions.
The result is a base model: a very good document completer. It works like your phone's autocomplete, at a far higher level. Give it the start of a product page, and it writes a believable rest of the page. Ask it to write a tagline and it doesn't obey: it continues the text as if it were a document, which may or may not contain a tagline.
Try it now
Most of the engineering effort here goes into data, not into the model.
Learn it properly
Pretraining at scale · Data, compute and what models learn from the internet. About 35 minutes.
Other terms from the same lesson
- Jaccard similarity: How alike two sets are: the number of items they share, divided by the number of different items in both. 1 means identical, 0 means nothing in common.
- MinHash: A hashing trick that estimates Jaccard similarity from short signatures, so near-duplicate documents can be found without comparing every pair.
- pretraining: The first, huge training run where a model learns language by predicting the next token across vast amounts of text.
- shingle: A short run of words in a row, such as three, taken from a document. Comparing two documents' shingles shows how much text they share.
- utilization: The share of a chip's peak speed that a training run actually gets. Waiting on memory and on other chips keeps it well below 100%.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.