Index Ai Academy
Phase 5 · Scale & inference

The block in today's open models

RMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.

About 35 minutes · Optional lesson: deeper material that no phase or certificate requires · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers9

Why it matters

Three lessons in this phase name RMSNorm, SwiGLU, RoPE and grouped-query attention, and none assembles them. This one does. The block of today's open models is the block you built with four small swaps, and each swap has a reason you can measure: fewer parameters, a cleaner position signal, a smaller cache or a gate in the MLP.

You will write each part in a few lines, check the property that justifies it, put all four in one block next to the classic one, and read the config of a real model line by line.

What you'll be able to do

  • Name the four changes open models make to the transformer block, and what each one changes in the block's shapes and counts
  • Write RMSNorm, a SwiGLU layer and rotary positions in PyTorch, and show that RoPE scores depend only on the gap
  • Assemble them with grouped-query attention into a block, and compare its parameters and cache with the classic block
  • Read a Llama-style config.json and size a KV cache from it
Quick check
A vector (3, 4) has a root mean square of sqrt((9 + 16) / 2). A guess: what does this print, to 2 decimal places?
print(round(((3 ** 2 + 4 ** 2) / 2) ** 0.5, 2))
  1. 12.5
  2. 3.54
  3. 3.5
  4. 5.0
Show the answer

3.54

The mean of the squares is (9 + 16) / 2 = 12.5, and its square root is 3.54. RMSNorm divides every number by this, so (3, 4) becomes (0.85, 1.13).

The idea: four small switches on the block you built

This rests on The transformer block, the block with a norm, attention, an MLP and two residual adds, on Positional encodings, where positions were added at the input, and on Inference and the KV cache, which counted what a server keeps for every token. Open models today use the same block with four changes. None adds a new idea. Each swaps one part for a cheaper or better-behaved one.

  1. RMSNorm replaces LayerNorm. It divides by the root mean square and scales, with no mean subtracted and no bias.
  2. RoPE replaces the position table. Positions are not added at the input. Each pair of numbers in q and k is rotated by an angle that grows with the token's position, so the dot product of a query and a key depends only on how far apart the two tokens are.
  3. Grouped-query attention (GQA) keeps every query head but fewer key and value heads, each shared by a group of queries. The cache shrinks in proportion.
  4. SwiGLU replaces the MLP's single bend with a gate. One projection goes through SiLU and multiplies a second projection, then a third brings the width back. It has three matrices instead of two, so its hidden width is made narrower to keep the parameter count close.
Try it now
Quick check
From the KV cache lesson: a server keeps keys and values for every token it has read. What does a longer prompt do to the cache?
  1. It shrinks as the model sees more text
  2. It grows with the square of the number of tokens
  3. It stays the same size
  4. It grows in proportion to the number of tokens
Show the answer

It grows in proportion to the number of tokens

Each token adds its own key and value rows, in every layer, so the cache grows in step with the number of tokens. It is the attention scores that grow with the square.

In your own words
In one sentence each: what does RMSNorm drop from LayerNorm, and what does RoPE do in place of a position table?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: each switch, then the block

Each switch is a few lines. The cells write it, test the property that makes it worth using, and then assemble all four into a block that you can compare with the one from the build lesson.

Worked example

RMSNorm against LayerNorm

Both norms on the same row. RMSNorm leaves the mean alone and sets the root mean square to 1.

import torch

x = torch.tensor([[3.0, 4.0, 1.0, 0.0]])
ln = torch.nn.LayerNorm(4)
rms = x * torch.rsqrt((x ** 2).mean(-1, keepdim=True) + 1e-6)
print("LayerNorm:", [round(v, 2) for v in ln(x)[0].tolist()])
print("RMSNorm:  ", [round(v, 2) for v in rms[0].tolist()])
print("mean of the squares after RMSNorm:", round(float((rms ** 2).mean()), 3))
It runs right here in your browser, with nothing to install.

LayerNorm subtracts the mean, 2.0, and divides by the spread, so its row has mean 0. RMSNorm keeps the row's shape: it only divides by the root mean square, 2.55, so the numbers stay in their original proportions and their mean square becomes 1. It does one pass fewer over the data, and its weight has no bias.

Quick check
The RMSNorm of the row (2, 2) divides each number by its root mean square, 2. A guess: what does this print?
x = [2.0, 2.0]
rms = (sum(v * v for v in x) / len(x)) ** 0.5
print([v / rms for v in x])
  1. [1.0, 1.0]
  2. [2.0, 2.0]
  3. [0.0, 0.0]
  4. [0.5, 0.5]
Show the answer

[1.0, 1.0]

The mean of the squares is 4, its root is 2, and 2 / 2 = 1 for each number. LayerNorm would give [0.0, 0.0], because it subtracts the mean first; RMSNorm does not.

Worked example

A SwiGLU layer and its count

The gate, up and down projections on a row, with the parameter count next to the classic MLP at the same width.

import torch
import torch.nn as nn

d, hidden = 32, 88
gate, up = nn.Linear(d, hidden, bias=False), nn.Linear(d, hidden, bias=False)
down = nn.Linear(hidden, d, bias=False)
x = torch.randn(3, d)
y = down(nn.functional.silu(gate(x)) * up(x))
count = lambda *ms: sum(p.numel() for m in ms for p in m.parameters())
print(tuple(y.shape), "SwiGLU parameters:", count(gate, up, down))
classic = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
print("classic MLP parameters:", count(classic))
In the lesson you can change this and run it.

The output is (3, 32), the same as the classic MLP. SwiGLU has 8,448 weights in three matrices and the classic MLP 8,352 in two, with their biases: the narrower width of 88 instead of 128 keeps them close. What changed is the gate, which lets the layer switch hidden units on and off by their input.

Quick check
A SwiGLU hidden unit's gate value is negative and large. After SiLU, what does it do to that unit's up value?
  1. It doubles it
  2. It nearly switches it off
  3. It leaves it unchanged
  4. It flips its sign and keeps its size
Show the answer

It nearly switches it off

SiLU of a large negative number is close to 0, so multiplying the up value by it removes that unit's contribution. That is the gate.

Worked example

RoPE: the gap is what counts

The same query and key vector placed at different positions. Their dot product depends only on how far apart the positions are.

import torch

def rope(x, base=10000.0):
    B, H, T, dh = x.shape
    half = dh // 2
    freq = base ** (-torch.arange(half) / half)
    angle = torch.arange(T).view(T, 1) * freq.view(1, half)
    cos, sin = torch.cos(angle), torch.sin(angle)
    x1, x2 = x[..., :half], x[..., half:]
    return torch.cat([x1 * cos - x2 * sin, x1 * sin + x2 * cos], dim=-1)

torch.manual_seed(0)
q, k = torch.randn(1, 1, 1, 8), torch.randn(1, 1, 1, 8)
Q = rope(q.expand(1, 1, 10, 8).contiguous())     # the same vector at 10 positions
K = rope(k.expand(1, 1, 10, 8).contiguous())
dot = lambda i, j: round(float((Q[:, :, i] * K[:, :, j]).sum()), 4)
print("positions 5 and 2, gap 3:", dot(5, 2))
print("positions 8 and 5, gap 3:", dot(8, 5))
print("positions 3 and 3, gap 0:", dot(3, 3), dot(7, 7))
In the lesson you can change this and run it.

Moving both tokens by 3 places leaves the score unchanged: -0.8022 for the pair (5, 2) and for (8, 5), in real PyTorch. The same holds at a gap of 0, where the two positions give the same score wherever they sit. The score is a function of the gap alone, which is what a language model needs from position, and no position table is added to the input.

Quick check
Two query heads share one key-value head. How many key-value heads must a layer's cache store for 8 query heads?
  1. 2
  2. 4
  3. 8
  4. 16
Show the answer

4

8 query heads in groups of 2 need 8 / 2 = 4 key-value heads, so the cache stores 4 keys and 4 values per token instead of 8.

Worked example

The four switches in one block

ModernBlock next to the course's Block: the same width and heads, with GQA, RoPE, RMSNorm and SwiGLU. It still never lets a token read the ones after it.

import math
import torch
import torch.nn as nn
from tinygpt import Block

class RMSNorm(nn.Module):
    def __init__(self, d, eps=1e-6):
        super().__init__()
        self.eps, self.weight = eps, nn.Parameter(torch.ones(d))

    def forward(self, x):
        return x * torch.rsqrt((x ** 2).mean(-1, keepdim=True) + self.eps) * self.weight

def rope(x, base=10000.0):
    B, H, T, dh = x.shape
    half = dh // 2
    freq = base ** (-torch.arange(half) / half)
    angle = torch.arange(T).view(T, 1) * freq.view(1, half)
    cos, sin = torch.cos(angle), torch.sin(angle)
    x1, x2 = x[..., :half], x[..., half:]
    return torch.cat([x1 * cos - x2 * sin, x1 * sin + x2 * cos], dim=-1)

def repeat_kv(x, g):                     # each key-value head serves g query heads
    B, kv, T, dh = x.shape
    return x.unsqueeze(2).expand(B, kv, g, T, dh).reshape(B, kv * g, T, dh)

class ModernBlock(nn.Module):
    def __init__(self, d, heads, kv_heads, hidden):
        super().__init__()
        self.h, self.kv, self.dh = heads, kv_heads, d // heads
        self.n1, self.n2 = RMSNorm(d), RMSNorm(d)
        self.q, self.o = nn.Linear(d, d, bias=False), nn.Linear(d, d, bias=False)
        self.k = nn.Linear(d, kv_heads * self.dh, bias=False)
        self.v = nn.Linear(d, kv_heads * self.dh, bias=False)
        self.gate = nn.Linear(d, hidden, bias=False)
        self.up = nn.Linear(d, hidden, bias=False)
        self.down = nn.Linear(hidden, d, bias=False)

    def forward(self, x):
        B, T, d = x.shape
        a, g = self.n1(x), self.h // self.kv
        q = self.q(a).view(B, T, self.h, self.dh).transpose(1, 2)
        k = self.k(a).view(B, T, self.kv, self.dh).transpose(1, 2)
        v = self.v(a).view(B, T, self.kv, self.dh).transpose(1, 2)
        q, k = rope(q), rope(k)
        k, v = repeat_kv(k, g), repeat_kv(v, g)
        s = q @ k.transpose(-2, -1) / math.sqrt(self.dh)
        s = s.masked_fill(~torch.tril(torch.ones(T, T)).bool(), float("-inf"))
        y = (torch.softmax(s, dim=-1) @ v).transpose(1, 2).reshape(B, T, d)
        x = x + self.o(y)
        m = self.n2(x)
        hidden = nn.functional.silu(self.gate(m)) * self.up(m)
        return x + self.down(hidden)

torch.manual_seed(0)
old, new = Block(32, 4), ModernBlock(32, 4, 2, 88)
count = lambda m: sum(p.numel() for p in m.parameters())
print("parameters: classic", count(old), " modern", count(new))
x = torch.randn(1, 6, 32)
x2 = x.clone()
x2[:, 5] = torch.randn(32)
print("earlier tokens ignore a changed last token:",
      torch.allclose(new(x)[:, :5], new(x2)[:, :5], atol=1e-6))
print("cache numbers per token per layer: classic", 2 * 32, " grouped-query", 2 * 2 * 8)
In the lesson you can change this and run it.

The modern block has 11,584 parameters against the classic block's 12,704: it has no biases, its norms have one weight vector, and its key and value layers are half the width. The causality check passes, so changing the last token leaves every earlier output the same: the mask and RoPE together still obey the rule that a token cannot read the future. The cache holds 32 numbers per token per layer instead of 64, because there are 2 key-value heads of 8 numbers instead of 4 full-width ones.

Quick check
In the lesson's block, d = 32, 4 query heads, 2 key-value heads. A guess: how many numbers does the key layer produce per token?
d, heads, kv_heads = 32, 4, 2
print(kv_heads * (d // heads))
  1. 8
  2. 64
  3. 16
  4. 32
Show the answer

16

Each key-value head has width 32 / 4 = 8, and there are 2 of them: 2 x 8 = 16 numbers per token, half the 32 of a full-width key.

Drill

Write RMSNorm

Write rms_norm(x, eps=1e-6) for a list of numbers: divide every number by the square root of (the mean of the squares plus eps) and return a list. There is no mean subtraction.

def rms_norm(x, eps=1e-6):
    return x


print([round(v, 3) for v in rms_norm([3.0, 4.0])])
In the lesson, your code is checked as you write it.
Drill

Size a KV cache

Write kv_cache_bytes(layers, kv_heads, head_dim, tokens, bytes_per_number=2): the bytes a server holds for tokens tokens. Each layer keeps one key and one value, each kv_heads * head_dim numbers, per token.

def kv_cache_bytes(layers, kv_heads, head_dim, tokens, bytes_per_number=2):
    return 0


print(kv_cache_bytes(32, 8, 128, 1))
In the lesson, your code is checked as you write it.
In your own words
Why can a grouped-query model handle a longer context on the same GPU?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: reading a Llama-style config

A modern open model's config.json is these four switches written as numbers. Read it from the top: hidden_size and num_attention_heads give the head width; num_key_value_heads below num_attention_heads means grouped-query attention; intermediate_size over hidden_size, near 2.7 instead of 4, means a SwiGLU layer; hidden_act of silu is its gate; rms_norm_eps names RMSNorm; and a rope setting names the rotary positions.

Where this shows up in an engineer's week

On the job: these four switches decide what a model costs to serve. The cache size per token, 2 x layers x kv_heads x head_dim x bytes, sets how many users fit on a GPU, so num_key_value_heads is the config line to read first when you size a deployment. Check the norm and the position scheme when you convert weights between libraries, since a wrong epsilon or a wrong rotation convention gives plausible nonsense with no error. And when two models differ in size and speed, compare their configs before their benchmarks.

Goes deeper in

In your own words
A deployment runs out of GPU memory at long prompts. Which config line do you check first, and what could you change?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

What does RMSNorm leave out of LayerNorm?

  1. The mean subtraction and the bias
  2. The division by a scale
  3. The learned weight
  4. The residual add

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write