Transformers

What you'll learn

Language modeling · 5 lessons, about 3.5 hours

1. Language modeling FreePredict the next token, and score it with cross-entropy.
2. Tokenization and BPEHow tokenizers are built from data.
3. A bigram modelThe simplest language model, built from counts.
4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.
5. Neural language modelsWhy counting fails, and what embeddings fix.

Attention · 5 lessons, about 3 hours

1. Attention: the intuitionEach token decides which other tokens matter to it.
2. Queries, keys and valuesThe three projections behind attention.
3. Scaled dot-product attentionThe attention formula, line by line.
4. Causal maskingStop tokens from seeing the future.
5. Multi-head attentionSeveral attention patterns in parallel.

The transformer · 5 lessons, about 3.5 hours

1. Positional encodingsTell the model where each token sits.
2. The transformer blockAttention, MLP, residuals and norms together.
3. The GPT architectureA stack of decoder blocks that predicts the next token.
4. Build a tiny GPTWrite and train a small GPT from scratch.
5. Encoders and embedding modelsBERT-style models and where they fit.

Scale & inference · 6 lessons, about 4 hours

1. Pretraining at scaleData, compute and what models learn from the internet.
2. Scaling lawsHow loss falls predictably with size, data and compute.
3. Inference and the KV cacheWhy generation is fast after the first token.
4. Long contextEfficient attention and million-token windows.
5. Mixture of expertsRoute each token to a few specialist sub-networks.
6. Multimodal transformersHow models see images and hear audio.

Before you start

Lessons here build on earlier phases:

• Phase 1: Building with LLMs
• Phase 2: Math, when you need it
• Phase 4: Deep learning

The placement test can mark off what you already know, so you only study what's new to you.

What you'll build

Build a tiny GPT

Write and train a small GPT from scratch. Read the tutorial.

• Prepare a corpus for a language model: tokenize, split, and batch with shifted targets
• Train a small GPT from scratch and watch its loss fall
• Generate text from it and judge what it has and hasn't learned

When you finish

Finish every lesson in this phase, or show you know it in the placement test, and you earn a certificate of completion, "Transformers (Phase 5)", with its own link you can share.