Index Ai Academy
Free courses · 20 lessons · about 13 hours

Build an LLM from scratch: a free course

Go from counting letter pairs to a small GPT you train yourself: tokenizers, attention, transformer blocks and the GPT architecture, written line by line.

Create a free account Read the start of the first lesson

Already have an account? Sign in

What you'll be able to do

  • Run byte-pair encoding, and see how a tokenizer's training data shapes what's cheap to encode
  • Build a bigram model from counts, then see what neural language models fix
  • Turn probabilities into text with temperature, top-k and top-p
  • Describe attention as a weighted average, and compute it from queries, keys and values
  • Assemble a transformer block from attention, an MLP, residual connections and layer norms
  • Prepare a corpus, train a small GPT from scratch and watch its loss fall
  • Generate text from it, and judge what it has and hasn't learned

Who it's for

For people who use language models and want to see how they work inside. It's the deepest course here: it assumes you know the basics of neural networks and the math behind them.

Before you start

Some lessons here build on earlier ones in the academy. The placement test can mark off what you already know, and you can open any of them first:

The 20 lessons

One is free with no account, and a free account opens the rest.

  1. PyTorch tensorsNumPy-like arrays that run on GPUs.
  2. PyTorch autograd and modulesLet PyTorch compute gradients; build models with nn.Module.
  3. The training loopForward, loss, backward, step, repeat.
  4. Embedding layersLearned lookup tables that turn tokens into vectors.
  5. Residual connectionsSkip connections that let gradients flow.
  6. Normalization layersBatch and layer norm keep training stable.
  7. Language modeling FreePredict the next token, and score it with cross-entropy.
  8. Tokenization and BPEHow tokenizers are built from data.
  9. A bigram modelThe simplest language model, built from counts.
  10. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.
  11. Neural language modelsWhy counting fails, and what embeddings fix.
  12. Attention: the intuitionEach token decides which other tokens matter to it.
  13. Queries, keys and valuesThe three projections behind attention.
  14. Scaled dot-product attentionThe attention formula, line by line.
  15. Causal maskingStop tokens from seeing the future.
  16. Multi-head attentionSeveral attention patterns in parallel.
  17. Positional encodingsTell the model where each token sits.
  18. The transformer blockAttention, MLP, residuals and norms together.
  19. The GPT architectureA stack of decoder blocks that predicts the next token.
  20. Build a tiny GPTWrite and train a small GPT from scratch.

When you finish

Finish every lesson, or show you know it in the placement test, and you earn a certificate of completion, "LLM from scratch course", with its own link you can share or add to LinkedIn.

Questions

Is it really free?
Yes. There's nothing to pay, here or anywhere in the academy. A free account saves your progress and issues your certificate.
Do I need to install anything?
No. The Python in these lessons runs in your browser.
How long does it take?
About 13 hours in all, at your own pace. Your progress stays where you left it.
What does the certificate show?
That your account recorded every lesson as finished or known. It's a certificate of completion from Index Ai Academy, not an accredited qualification.

Start the course

A free account saves your progress across devices and issues your certificate when you finish.

Create a free account