Index Ai Academy
Phase 5 · How transformers think

Embeddings and positional waves

From three IDs to the 3 × 8 matrix that enters the first block.

About 35 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers11

Why it matters

The tokenizer left us with three integers, 104, 324 and 782. An integer can't be reasoned with: 324 isn't "more" than 104, and the model can't multiply "cat" by a weight. So the first layer of every transformer swaps each integer for a row of numbers, its embedding, and then stamps each row with where in the sentence it sits. The result is the matrix XX that the attention, feed-forward and LM-head lessons of this track start from.

This lesson follows the three IDs into that matrix, one number at a time, with a model small enough to print: 8 numbers per token instead of GPT-2's 768.

What you'll be able to do

  • Look up a token's embedding as one row of the V × d_model matrix, and give the shape of the result
  • Compute a sine-cosine positional encoding for any position and column from its formula
  • Add the positional encoding to the embeddings and explain what the sum carries into the model
  • Count the parameters of an embedding layer, and say why its rows change during training
Each ID picks one row of the embedding matrix, a positional wave is added, and the boards show the three rows of E, the positional matrix and the sum for the " cat" row. Transcript
Quick check
An embedding lookup is NumPy fancy indexing: rows picked by number. A guess: what does this print?
import numpy as np

E = np.arange(20).reshape(5, 4)    # a tiny table: 5 tokens, 4 numbers each
print(E[[3, 1]])
  1. [[3 3 3 3] [1 1 1 1]]
  2. [[12 13 14 15] [ 4 5 6 7]]
  3. [[ 4 5 6 7] [12 13 14 15]]
  4. [3 1]
Show the answer

[[12 13 14 15] [ 4 5 6 7]]

A list of row numbers picks those rows, in the order given: row 3 is 12 to 15, row 1 is 4 to 7. That is all an embedding lookup does, with V rows instead of 5.

The idea: an ID becomes a row, and a position is stamped on it

This rests on Matrices: tables of numbers, where a row is picked out of a table by its number, and on Tokenization: text to integer IDs, which produced the ids.

From an ID to a row of numbers

The model keeps a table called the embedding matrix, EE, with one row per vocabulary entry and dmodeld_{\text{model}} numbers in each row. Token ID ii is simply row ii: the lookup xi=E[i]∈Rdmodelx_i = E[i] \in \mathbb{R}^{d_{\text{model}}}. For a whole prompt of TT tokens, stacking the rows gives a matrix:

Xemb=E[x]∈RT×dmodel,E∈RV×dmodelX_{\text{emb}} = E[x] \in \mathbb{R}^{T \times d_{\text{model}}}, \qquad E \in \mathbb{R}^{V \times d_{\text{model}}}

Our small model has V=1000V = 1000 and dmodel=8d_{\text{model}} = 8, so EE is 1,000 rows of 8 numbers (8,000 parameters) and "The cat sat" becomes a 3 × 8 matrix. GPT-2 has V=50,257V = 50{,}257 and dmodel=768d_{\text{model}} = 768: a table of 38,597,376 numbers, about 31% of the whole model.

Those numbers are learned. At the start of training they're random; training nudges row 324 until it's a useful summary of how " cat" behaves, which is why similar words end up with similar rows. Nothing about the lookup itself is clever. The row number is the only thing the token contributes.

Worked example

Look up the three rows

A seeded random matrix stands in for a trained one. The same code, with the same seed, prints the same numbers in every lesson of this section.

import numpy as np
np.set_printoptions(precision=2, suppress=True)
rng = np.random.default_rng(42)

V, d_model = 1000, 8
# the embedding matrix: V rows of d_model numbers
E = rng.normal(0.0, 0.5, size=(V, d_model)).round(2)
ids = [104, 324, 782]                                    # "The", " cat", " sat"
X_emb = E[ids]
print(E.shape, "->", X_emb.shape)
print(X_emb)
print("row 324 again:", np.array_equal(E[324], X_emb[1]))
It runs right here in your browser, with nothing to install.

Three row numbers in, a (3, 8) matrix out: the first row is E[104]E[104], the second E[324]E[324], the third E[782]E[782]. Row 324 starts 0.23, 0.36, -0.80. If " cat" appeared twice in a prompt, both of its rows would be this same vector: at this point the model can't yet tell the two apart.

Quick check
A prompt uses the token " cat" twice, at positions 2 and 6. Right after the embedding lookup, how do its two rows compare?
  1. The second row is all zeros until attention runs
  2. They differ, because the lookup also reads the neighbors
  3. They're identical: both are row 324 of E
  4. The second is scaled by 6/2 = 3
Show the answer

They're identical: both are row 324 of E

The lookup only knows the ID. Same ID, same row. Telling the two apart is the job of the position signal added next.

Try it now

The embeddings of "The", " cat" and " sat" (top), the positional matrix (middle), and their sum (bottom): the same 3 × 8 shape before and after. Columns 6 and 7 barely change, because their waves are thousands of positions long.

Positions: the same word in two places

Here is the problem the position signal solves. The attention layer you'll meet in Inside the attention matrix mixes rows with weighted sums, and a weighted sum doesn't care about order: "the cat sat on the mat" and "the mat sat on the cat" would give the same rows the same treatment. So the model needs each row to carry its position as well as its token.

The original transformer did this with fixed waves. For position pospos (0, 1, 2, …) and column pair ii (0, 1, 2, …), the matrix PE∈RT×dmodelPE \in \mathbb{R}^{T \times d_{\text{model}}} is:

PE(pos,  2i)=sin⁡ ⁣(pos10000 2i/dmodel),PE(pos,  2i+1)=cos⁡ ⁣(pos10000 2i/dmodel)PE_{(pos,\;2i)} = \sin\!\left(\frac{pos}{10000^{\,2i/d_{\text{model}}}}\right), \qquad PE_{(pos,\;2i+1)} = \cos\!\left(\frac{pos}{10000^{\,2i/d_{\text{model}}}}\right)

Read it as: even columns hold a sine, odd columns the matching cosine, and each pair has its own speed. Pair 0 divides by 100000=110000^0 = 1, so it's a wave with wavelength 2π≈6.32\pi \approx 6.3 positions. Pair 1 divides by 100002/8=1010000^{2/8} = 10: wavelength 62.8. Pair 2: 628.3. Pair 3: 6,283.2. Fast waves on the left tell nearby positions apart; slow waves on the right separate positions that are far apart. Every position gets a pattern no other position has, and the same formula works for any TT without new parameters.

Worked example

The positional matrix for three positions

The formula, with NumPy doing all four pairs at once. pos is a column of positions and i a row of even column indices, so angle is 3 × 4.

import numpy as np
np.set_printoptions(precision=2, suppress=True)

T, d_model = 3, 8
pos = np.arange(T)[:, None]              # (3, 1): the positions 0, 1, 2
i = np.arange(0, d_model, 2)[None, :]    # (1, 4): the even columns 0, 2, 4, 6
angle = pos / 10000 ** (i / d_model)     # (3, 4): one angle per position and pair
PE = np.zeros((T, d_model))
PE[:, 0::2] = np.sin(angle)              # even columns
PE[:, 1::2] = np.cos(angle)              # odd columns
print(PE)
print("wavelength of each pair:", (2 * np.pi * 10000 ** (i / d_model)).round(1))
In the lesson you can change this and run it.

Position 0 is 0, 1, 0, 1, …: sin 0 and cos 0 in every pair. Position 1, pair 0 is sin 1 = 0.84 and cos 1 = 0.54; pair 1 is sin 0.1 = 0.10 and cos 0.1 = 1.00. By pair 3 the angle is 0.001 and the columns read 0.00 and 1.00 for all three positions: that wave needs thousands of positions to turn.

Quick check
Two entries of the matrix above, computed by hand. What does this print?
import numpy as np

print(round(float(np.sin(1 / 10000 ** 0.25)), 2), round(float(np.cos(2)), 2))
  1. 0.1 0.98
  2. 0.1 -0.42
  3. 0.0 1.0
  4. 0.84 0.54
Show the answer

0.1 -0.42

sin(1/10) is position 1's entry in column 2 (pair 1, exponent 2/8 = 0.25), and cos(2) is position 2's entry in column 1 (pair 0). Both appear in the printed PE: 0.1 and -0.42.

In your own words
What does the embedding matrix know about a token, and what does it not know?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: the input to the first block

The formula for position is a drill away from code. Write it, draw it, add it to the rows, then see the same lookup as a PyTorch layer and read it on the trained tiny GPT.

Drill

Write the positional encoding

Write positional_encoding(T, d_model) that returns the T × d_model matrix from the formula: sines in the even columns, cosines in the odd ones, the pair i dividing the position by 10000 ** (2i / d_model).

import numpy as np

def positional_encoding(T, d_model):
    return np.zeros((T, d_model))

print(positional_encoding(3, 8).round(2))
In the lesson, your code is checked as you write it.
Worked example

See the waves

The same formula for 64 positions and 32 dimensions, drawn as a heatmap with dimensions down the side. Red is +1, blue is −1.

import numpy as np
import matplotlib.pyplot as plt

T, d_model = 64, 32
pos = np.arange(T)[:, None]
i = np.arange(0, d_model, 2)[None, :]
angle = pos / 10000 ** (i / d_model)
PE = np.zeros((T, d_model))
PE[:, 0::2] = np.sin(angle)
PE[:, 1::2] = np.cos(angle)

fig, ax = plt.subplots(figsize=(6.5, 3.6))
# one row per dimension, positions left to right
im = ax.imshow(PE.T, aspect="auto", cmap="RdBu")
ax.set_xlabel("position")
ax.set_ylabel("dimension")
ax.set_title("Positional encoding: fast waves at the top, slow waves at the bottom")
fig.colorbar(im, ax=ax)
In the lesson you can change this and run it.

The top rows flip between red and blue every few positions; the bottom rows change so slowly that 64 positions don't finish a single swing. Read down any one column and you get that position's signature. Read across and you see why positions 10 and 11 look alike while 10 and 50 don't: the fast rows have moved on, the slow rows have barely changed.

Add, don't append

The two matrices have the same shape, so they're added:

X=Xemb+PE∈RT×dmodelX = X_{\text{emb}} + PE \in \mathbb{R}^{T \times d_{\text{model}}}

Adding keeps dmodeld_{\text{model}} fixed, so every matrix after this point keeps the shape it was designed for. It also works: the fast columns on the left carry mostly position, the slow columns on the right are nearly constant and carry mostly token, and the layers in between learn to read both out of the sum. Row 2 of XX, for " cat", is E[324]E[324] plus the wave values for position 1: 0.23+0.84=1.070.23 + 0.84 = 1.07, 0.36+0.54=0.900.36 + 0.54 = 0.90, −0.80+0.10=−0.70-0.80 + 0.10 = -0.70, and so on.

Not every model uses the sine-cosine waves. GPT-2 learns its position rows instead (a second table, 1,024 positions by 768), and most recent models rotate the queries and keys by position inside the attention layer (RoPE). The purpose is the same every time: give each row a usable sense of where it is. XX is the matrix that enters the first transformer block, the XX in the Q=XWQQ = XW_Q of Inside the attention matrix.

Quick check
Why is the positional signal added to the embedding rather than appended as extra columns?
  1. Appended columns would make the sines negative and break the wave pattern
  2. Addition is the only operation NumPy supports on matrices of that size
  3. Adding keeps d_model fixed, and the layers learn to read both from the sum
  4. Appending would lose the position information held inside the embedding
Show the answer

Adding keeps d_model fixed, and the layers learn to read both from the sum

Appending would work in principle but changes the width that every later weight matrix depends on. Adding costs nothing in shape, and the fast and slow columns carry position and token in a way the layers learn to separate.

Worked example

The input to the first block

Lookup, positional matrix, sum. The printed X is the starting point of the attention lesson.

import numpy as np
np.set_printoptions(precision=2, suppress=True)
rng = np.random.default_rng(42)

V, d_model, T = 1000, 8, 3
E = rng.normal(0.0, 0.5, size=(V, d_model)).round(2)
ids = [104, 324, 782]

pos = np.arange(T)[:, None]
i = np.arange(0, d_model, 2)[None, :]
angle = pos / 10000 ** (i / d_model)
PE = np.zeros((T, d_model))
PE[:, 0::2] = np.sin(angle)
PE[:, 1::2] = np.cos(angle)

X = E[ids] + PE
print(X)
print(X.shape)
In the lesson you can change this and run it.

Row 0 is "The" at position 0: its embedding plus 0, 1, 0, 1, …, so -0.56 stays and -0.19 becomes 0.81. Row 1 is " cat" at position 1: 1.07, 0.90, -0.70, …. The shape is still (3, 8): three tokens, eight numbers each, now with their positions inside the numbers.

Drill

Build the model's input

Write model_input(E, ids): look up the rows of E for ids, add positional_encoding(len(ids), E.shape[1]) (given), and return the sum.

import numpy as np

def positional_encoding(T, d_model):
    pos = np.arange(T)[:, None]
    i = np.arange(0, d_model, 2)[None, :]
    angle = pos / 10000 ** (i / d_model)
    pe = np.zeros((T, d_model))
    pe[:, 0::2] = np.sin(angle)
    pe[:, 1::2] = np.cos(angle)
    return pe

def model_input(E, ids):
    return E[:len(ids)]

E = np.arange(40, dtype=float).reshape(10, 4) / 10
print(model_input(E, [7, 2]).round(2))
In the lesson, your code is checked as you write it.
Worked example

The same lookup in PyTorch

nn.Embedding is the matrix E with the lookup built in. This runs on the course's teaching version of PyTorch: the same code as the real library, on the CPU and slower, and its random starting values differ from a real install's.

import torch

torch.manual_seed(0)
emb = torch.nn.Embedding(1000, 8)            # V = 1000 rows, d_model = 8
ids = torch.tensor([104, 324, 782])
x = emb(ids)
print(tuple(emb.weight.shape), "->", tuple(x.shape))
print(sum(p.numel() for p in emb.parameters()), "parameters")
print(torch.equal(x[1], emb.weight[324]))     # the lookup really is "row 324"
In the lesson you can change this and run it.

The weight is the (1000, 8) table and the output is (3, 8): 8,000 parameters, all of them trained. The last line confirms that the layer does nothing but pick row 324. GPT-2's version of this table has 50,257 × 768 = 38,597,376 parameters, learned the same way.

Quick check
You double the vocabulary of a model from 1,000 tokens to 2,000 and keep d_model = 8. What happens to the embedding layer?
  1. Nothing changes, because the vocabulary size isn't part of this layer
  2. The output for a 3-token prompt becomes (6, 8), two rows per token
  3. It gains 1,000 rows: 16,000 parameters, and the output is still (3, 8)
  4. Each row doubles in length to 16 numbers, and the output becomes (3, 16)
Show the answer

It gains 1,000 rows: 16,000 parameters, and the output is still (3, 8)

The table has one row per vocabulary entry, so V sets the number of rows and d_model the length of each. The output shape depends only on T and d_model.

Quick check
GPT-2's tokenizer has V = 50,257 tokens and its d_model is 768. How many numbers are in its token embedding matrix?
  1. 50,257 + 768 = 51,025
  2. 1,024 × 768, one row per position
  3. 768, one per dimension
  4. 50,257 × 768 = 38,597,376
Show the answer

50,257 × 768 = 38,597,376

One row per vocabulary entry, 768 numbers in each: 50,257 × 768 = 38,597,376. That's about 31% of GPT-2's 124 million parameters, spent on the lookup table alone.

The trained tiny GPT's tables

The tiny GPT has the same two tables, with its own sizes: model.tok holds one row of 32 numbers for each of its 195 tokens, and model.pos one row for each of its 64 positions. Its position rows are learned, not waves, as in GPT-2. The cell adds them for "Describe the Lark headlamp ." and confirms that the lookup is row selection.

Worked example

Token rows plus position rows on the trained model

The model's own tok and pos layers do the lookup. The sum is the matrix that enters block 0.

import torch
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
ids = tok.encode("Describe the Lark headlamp .")
idx = torch.tensor([ids])
print(tuple(model.tok.weight.shape), tuple(model.pos.weight.shape))
e = model.tok(idx)
p = model.pos(torch.arange(len(ids)))
x = e + p
print(tuple(e.shape), tuple(p.shape), tuple(x.shape))
print(torch.equal(e[0, 1], model.tok.weight[tok.token_id("the")]))
share = model.tok.weight.numel() / sum(q.numel() for q in model.parameters())
print(round(100 * share), "% of the parameters are the token table")
In the lesson you can change this and run it.

The token table is (195, 32) and the position table (64, 32). Five ids give e and p of shape (5, 32) and a sum x of the same shape: five rows, one per token. The check confirms that "the" is row 175 of the table. The token table holds 6,240 of the model's 40,195 parameters, 16%. GPT-2 spends 31% on the same table.

In your own words
Describe the matrix X that leaves this layer: its shape, what each row holds, and which two things are mixed into every row.

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: the tables in GPT-2, and where this bites

GPT-2 keeps the same two tables, named wte for tokens and wpe for positions. The map shows where this step sits: after the ids, before any block.

Where this shows up in an engineer's week

On the job: adding tokens to a model, such as new special tokens for a chat format, means growing the embedding table with resize_token_embeddings. The new rows start random and mean nothing until fine-tuning trains them. A model also cannot read a longer prompt than its position table has rows for, unless it uses a scheme such as RoPE that has no table.

Embedding tables are also why memory is tight with a big vocabulary: they cost V times d_model numbers before any block runs.

Goes deeper in

In your own words
Why can an engineer not simply type a new token into a trained model and expect it to work?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

What is the embedding matrix, and what does the lookup for token ID 324 return?

  1. A V × d_model table of learned numbers; the lookup returns its row 324
  2. A T × T table of similarities; the lookup returns column 324
  3. A d_model × d_model table; the lookup multiplies it by 324
  4. A list of V strings; the lookup returns the string " cat"

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Part of a free course 5 lessons

How Transformers Think: an interactive inference walkthrough

Follow one prompt, "The cat sat", through a language model with every number shown: tokens to IDs, embeddings and positional waves, the attention matrix, the feed-forward layer, and the probability over the next word.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write