Sequence to sequence, and the invention of attention
Build an encoder and decoder, measure the bottleneck of a fixed-size handoff, add attention in three lines, read an attention map with care, and see how one mistake compounds after teacher forcing.
What it covers6
Why it matters
Attention was not invented for transformers. It was invented in 2014 to repair a specific failure in translation models built from recurrent networks, and for three years it was an add-on to them. Building that original version is the best way to understand what attention is for. You will watch a model fail for the reason attention was invented, add three lines, and watch it recover. The lesson also covers how every sequence generator is trained and used, teacher forcing and step-by-step decoding, which applies unchanged to the largest language models.
What you'll be able to do
- Build an encoder and a decoder from recurrent cells, and train them to turn one sequence into another
- Measure the bottleneck: what a fixed-size handoff loses as inputs get longer
- Add attention in three lines, and explain scores, weights and context
- Read an attention map with care: what it shows and what it doesn't
- Explain teacher forcing, and measure how one mistake compounds when the model works alone
Show the answer
The parts read earliest are remembered worst
With the book closed, everything must be held in memory at once, and what was read first has been held longest and overwritten most. A better translator would keep the book open and glance back at the relevant words while writing each part. The first translation networks worked with the book closed. Attention is what opened it.
The idea: two networks and a handoff
Rests on Attention as matrix math (Foundation lesson 113), where five words looked at one another by hand.
Translation can't be done one word at a time. The output has a different length from the input, and a different order. In 2014 two research groups proposed the same answer, now called sequence to sequence, or the encoder-decoder:
- An encoder reads the whole input, one symbol at a time, and ends with a state.
- A decoder starts from that state and writes the output one symbol at a time. At each step it is given the symbol it wrote last, beginning with a special START token.
The only thing passed between them is the encoder's final state.
Real translation is hard to grade, so this lesson uses a stand-in with the same shape: the output is the input written backward. The order changes, every symbol matters, and an answer is simply right or wrong. Both networks are the GRU cell from LSTMs and GRUs, built and trained.
Show the answer
The encoder's final state
The decoder starts from that one state and writes the output from it. Everything it knows about the input must fit inside it.
Attention by hand
The encoder keeps a state for every input symbol. Take three states, each two numbers: (1, 0), (0, 1) and (1, 1). The decoder's own state, the query, is (2, 0). Attention does three things.
- Scores. A dot product of the query with each state: 2, 0 and 2.
- Weights. A softmax of the scores. e² is about 7.39 and e⁰ is 1, so the weights are 7.39, 1 and 7.39 shared out: about 0.468, 0.063 and 0.468.
- Context. The states blended by the weights: 0.468 × (1, 0) + 0.063 × (0, 1) + 0.468 × (1, 1) is about (0.936, 0.531).
The context leans toward the two states that matched the query, and keeps a trace of the third.
Setup (runs before every cell) · 102 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np import torch import torch.nn as nn import torch.nn.functional as F # symbols are 0 to 9. Token 10 tells the decoder to begin START = 10 def pairs(n, length, rng): """A stand-in for translation: the output is the input written backward.""" source = rng.integers(0, 10, (n, length)) return torch.tensor(source), torch.tensor(source[:, ::-1].copy()) class GRUCell(nn.Module): def __init__(self, n_in, n_hidden): super().__init__() self.gates = nn.Linear(n_in + n_hidden, 2 * n_hidden) self.candidate = nn.Linear(n_in + n_hidden, n_hidden) def forward(self, x, h): z, r = torch.sigmoid(self.gates(torch.cat([x, h], dim=1))).chunk(2, dim=1) new = torch.tanh(self.candidate(torch.cat([x, r * h], dim=1))) return z * h + (1 - z) * new class Seq2Seq(nn.Module): def __init__(self, n_hidden=32, attention=False): super().__init__() self.embed = nn.Embedding(11, 16) self.encoder, self.decoder = GRUCell(16, n_hidden), GRUCell(16, n_hidden) self.out = nn.Linear(2 * n_hidden if attention else n_hidden, 10) self.n_hidden, self.attention = n_hidden, attention def encode(self, source): """Read the input one symbol at a time. Return the state after every symbol, and the last state.""" h, states = torch.zeros(source.shape[0], self.n_hidden), [] for t in range(source.shape[1]): h = self.encoder(self.embed(source[:, t]), h) states.append(h) return torch.stack(states, dim=1), h def step(self, token, h, states): """One step of writing: take the previous output symbol, update the state, score the 10 possible next symbols.""" h = self.decoder(self.embed(token), h) if not self.attention: return self.out(h), h, None # dot product of h with each encoder state scores = torch.bmm(states, h.unsqueeze(2)).squeeze(2) # one weight for each input position, adding up to 1 weights = torch.softmax(scores, dim=1) # the encoder states, blended by those weights context = torch.bmm(weights.unsqueeze(1), states).squeeze(1) return self.out(torch.cat([h, context], dim=1)), h, weights def forward(self, source, target): """Training. At each step the decoder is given the correct previous symbol: teacher forcing.""" states, h = self.encode(source) token, scores = torch.full((source.shape[0],), START, dtype=torch.long), [] for t in range(target.shape[1]): out, h, weights = self.step(token, h, states) scores.append(out) token = target[:, t] return torch.stack(scores, dim=1) def translate(self, source, steps): """Use. At each step the decoder is given its own previous choice.""" with torch.no_grad(): states, h = self.encode(source) token = torch.full((source.shape[0],), START, dtype=torch.long) written, looked = [], [] for t in range(steps): out, h, weights = self.step(token, h, states) token = out.argmax(dim=1) written.append(token) looked.append(weights) return torch.stack(written, dim=1), looked def train(model, length, batches=300): rng = np.random.default_rng(0) optimizer = torch.optim.Adam(model.parameters(), lr=0.01) for k in range(batches): source, target = pairs(64, length, rng) loss = F.cross_entropy(model(source, target).reshape(-1, 10), target.reshape(-1)) optimizer.zero_grad() loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) optimizer.step() return model def report(model, length, n=500): source, target = pairs(n, length, np.random.default_rng(99)) written, looked = model.translate(source, length) right = (written == target).float() print(" symbols right:", f"{100 * right.mean().item():.1f}%", "| whole sequences right:", f"{100 * (right.sum(dim=1) == length).float().mean().item():.1f}%") print(" by output position:", " ".join(f"{right[:, t].mean().item():.2f}" for t in range(length))) return [right[:, t].mean().item() for t in range(length)]
import numpy as np states = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]) query = np.array([2.0, 0.0]) scores = states @ query weights = np.exp(scores) / np.exp(scores).sum() print(scores, weights.round(3), (weights @ states).round(3))
The printed numbers are the three above.
The three attention lines on three encoder states. The context has the size of one state.
Show the answer
About 0.79
e² ≈ 7.39 and the others give 1 each, so the first gets 7.39 ÷ 9.39 ≈ 0.79. Softmax is soft: the others keep a share.
import math scores = [2.0, 0.0, 0.0] total = sum(math.exp(s) for s in scores) print([round(math.exp(s) / total, 2) for s in scores])
Show the answer
[0.79, 0.11, 0.11]
The softmax raises e to each score and divides by the total: e² is about 7.39, and e⁰ is 1 for each of the others, so the weights are 7.39 / 9.39 ≈ 0.79, and 0.11 twice. Attention is soft: most of the weight goes to the best match, and every position keeps a little.
weights = [0.9, 0.1] states = [[2.0, 0.0], [0.0, 4.0]] context = [sum(w * s[k] for w, s in zip(weights, states)) for k in range(2)] print([round(c, 2) for c in context])
Show the answer
[1.8, 0.4]
The context is a weighted average of the states: 0.9 × [2, 0] plus 0.1 × [0, 4], which is [1.8, 0.4]. It is mostly the first state, with a trace of the second. The context always has the size of one state, however many states it blends.
Write a sentence or two, then compare it with a good answer in the lesson.
From scratch: an encoder and a decoder
The cell below builds both networks and trains the version with no attention on sequences of 4 symbols. The same definitions are loaded for every later cell, from the lesson's setup.
Uses the setup above, 102 lines.
import numpy as np import torch import torch.nn as nn import torch.nn.functional as F # symbols are 0 to 9. Token 10 tells the decoder to begin START = 10 def pairs(n, length, rng): """A stand-in for translation: the output is the input written backward.""" source = rng.integers(0, 10, (n, length)) return torch.tensor(source), torch.tensor(source[:, ::-1].copy()) class GRUCell(nn.Module): def __init__(self, n_in, n_hidden): super().__init__() self.gates = nn.Linear(n_in + n_hidden, 2 * n_hidden) self.candidate = nn.Linear(n_in + n_hidden, n_hidden) def forward(self, x, h): z, r = torch.sigmoid(self.gates(torch.cat([x, h], dim=1))).chunk(2, dim=1) new = torch.tanh(self.candidate(torch.cat([x, r * h], dim=1))) return z * h + (1 - z) * new class Seq2Seq(nn.Module): def __init__(self, n_hidden=32, attention=False): super().__init__() self.embed = nn.Embedding(11, 16) self.encoder, self.decoder = GRUCell(16, n_hidden), GRUCell(16, n_hidden) self.out = nn.Linear(2 * n_hidden if attention else n_hidden, 10) self.n_hidden, self.attention = n_hidden, attention def encode(self, source): """Read the input one symbol at a time. Return the state after every symbol, and the last state.""" h, states = torch.zeros(source.shape[0], self.n_hidden), [] for t in range(source.shape[1]): h = self.encoder(self.embed(source[:, t]), h) states.append(h) return torch.stack(states, dim=1), h def step(self, token, h, states): """One step of writing: take the previous output symbol, update the state, score the 10 possible next symbols.""" h = self.decoder(self.embed(token), h) if not self.attention: return self.out(h), h, None # dot product of h with each encoder state scores = torch.bmm(states, h.unsqueeze(2)).squeeze(2) # one weight for each input position, adding up to 1 weights = torch.softmax(scores, dim=1) # the encoder states, blended by those weights context = torch.bmm(weights.unsqueeze(1), states).squeeze(1) return self.out(torch.cat([h, context], dim=1)), h, weights def forward(self, source, target): """Training. At each step the decoder is given the correct previous symbol: teacher forcing.""" states, h = self.encode(source) token, scores = torch.full((source.shape[0],), START, dtype=torch.long), [] for t in range(target.shape[1]): out, h, weights = self.step(token, h, states) scores.append(out) token = target[:, t] return torch.stack(scores, dim=1) def translate(self, source, steps): """Use. At each step the decoder is given its own previous choice.""" with torch.no_grad(): states, h = self.encode(source) token = torch.full((source.shape[0],), START, dtype=torch.long) written, looked = [], [] for t in range(steps): out, h, weights = self.step(token, h, states) token = out.argmax(dim=1) written.append(token) looked.append(weights) return torch.stack(written, dim=1), looked def train(model, length, batches=300): rng = np.random.default_rng(0) optimizer = torch.optim.Adam(model.parameters(), lr=0.01) for k in range(batches): source, target = pairs(64, length, rng) loss = F.cross_entropy(model(source, target).reshape(-1, 10), target.reshape(-1)) optimizer.zero_grad() loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) optimizer.step() return model def report(model, length, n=500): source, target = pairs(n, length, np.random.default_rng(99)) written, looked = model.translate(source, length) right = (written == target).float() print(" symbols right:", f"{100 * right.mean().item():.1f}%", "| whole sequences right:", f"{100 * (right.sum(dim=1) == length).float().mean().item():.1f}%") print(" by output position:", " ".join(f"{right[:, t].mean().item():.2f}" for t in range(length))) return [right[:, t].mean().item() for t in range(length)] torch.manual_seed(0) model = train(Seq2Seq(), length=4) source, target = pairs(3, 4, np.random.default_rng(5)) written, looked = model.translate(source, 4) for row in range(3): print("input", source[row].tolist(), "-> written", written[row].tolist(), "| wanted", target[row].tolist()) print() print("encoder and decoder, no attention, sequences of 4 symbols:") report(model, 4) print("numbers handed from the encoder to the decoder:", model.n_hidden)
Read step and translate in the code before the results. step takes the previous symbol and the state, and scores the ten possible next symbols. translate calls it repeatedly, feeding each choice back in.
On sequences of 4 symbols it works. The three examples are right, 97.8% of symbols are right, and 95.2% of whole sequences come out perfect. Everything the decoder knows about the input reached it as 32 numbers.
nn.Embedding(11, 16). What does that layer do with a symbol such as 7?Show the answer
Looks up row 7 of a learned table, and returns its 16 numbers
An embedding layer is a table with one row for each token: here 11 rows (ten symbols and START) of 16 numbers. A token's number is used as an index, never as a quantity, so 7 is not "more" than 3. The rows are learned along with every other weight, and both the encoder and the decoder read their symbols through this table.
The bottleneck
Now make the sequences 8 symbols long. The model, its size and the 300 training batches stay the same.
Uses the setup above, 102 lines.
import matplotlib.pyplot as plt torch.manual_seed(0) model = train(Seq2Seq(), length=8) print("the same model and the same 300 batches, " "sequences of 8 symbols (this takes a few seconds):") acc = report(model, 8) plt.plot(acc, marker="o") plt.xlabel("output position") plt.ylabel("share of symbols right")
Whole sequences right: 11.2%, down from 95.2%.
The second line says where it breaks. The first output symbol is right 97% of the time. Then 0.85, 0.70, 0.60, and by the end 0.36.
Remember what the task is. The first symbol to write is the last one the encoder read, which is fresh in its state. The last symbol to write is the first one it read, eight steps earlier. The model fails in the same order as its memory fades.
This is the bottleneck. However long the input, the decoder receives one vector of fixed size, and the encoder has to decide what to keep before it knows what will be asked. A larger state moves the problem to longer inputs without removing it. Translation systems of 2014 showed exactly this: quality fell away as sentences grew.
One of the first papers found a trick that tells the same story: feeding the source sentence in reversed improved translation, since it put the first source words next to the first words to be written. That helps the start of a sentence at the expense of its end, and the next idea made it unnecessary.
Show the answer
They depend on the earliest inputs, which have faded most from the handoff
Writing backward means the last output is the first input. By the time the encoder finishes, that symbol has passed through seven more updates of the state, and little of it survives in the 32 numbers handed over. The decoder can't recover what was never passed to it.
Attention: looking back at the input
The encoder produced a state after every input symbol, and all but the last were thrown away. The idea of attention, published late in 2014, is to keep them all and let the decoder consult them at each step. In the model's step it is three lines:
- Scores. Compare the decoder's current state with each encoder state, by a dot product. A high score means "this one looks relevant now".
- Weights. A softmax turns the scores into positive weights that add up to 1.
- Context. Blend the encoder states, using the weights. The output layer then sees the decoder's state and this context together.
Nothing tells the model where to look. The weights come from the two networks' own states, and they are trained by the same loss as everything else.
In the code, torch.bmm is a batched matrix product: it does one matrix product for each sequence in the batch, so (batch, n, m) times (batch, m, p) gives (batch, n, p). h.unsqueeze(2) stands the decoder's state up as a column for it, and squeeze drops the extra axis of size 1 afterward.
Uses the setup above, 102 lines.
import matplotlib.pyplot as plt torch.manual_seed(0) model = train(Seq2Seq(attention=True), length=8) print("with attention, sequences of 8 symbols (this takes a few seconds):") acc = report(model, 8) plt.plot(acc, marker="o") plt.xlabel("output position") plt.ylabel("share of symbols right")
Whole sequences right: 82.2%, up from 11.2%. The last output positions, which were at 0.36, are at 0.88. The model is the same size apart from a wider output layer, and it had the same 300 batches.
What changed is where the information travels. The early input symbols no longer have to survive inside one vector. They wait in their own encoder states, and the decoder reaches back for them when it needs them.
The first version used a small network to compute the scores. The dot product used here came a year later, and it is the form that transformers kept.
Attention for one decoder step
Write attend(query, states) in NumPy. query is the decoder's state, of shape (hidden,). states holds the encoder's states, of shape (positions, hidden). Score each state by its dot product with the query, turn the scores into weights with a softmax, and blend the states. Return (weights, context). Subtract the largest score before np.exp, so that big scores don't overflow.
Uses the setup above, 102 lines.
import numpy as np def attend(query, states): return np.zeros(len(states)), np.zeros(states.shape[1]) states = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]) weights, context = attend(np.array([2.0, 0.0]), states) print(weights.round(3), context.round(3))
What is it looking at?
The weights can be printed, which makes attention unusual among the parts of a network: you can see where it looked. The cell prints the weights for one sequence, then asks a question about all 2,000.
Uses the setup above, 102 lines.
torch.manual_seed(0) model = train(Seq2Seq(attention=True), length=8) source, target = pairs(2000, 8, np.random.default_rng(99)) written, looked = model.translate(source, 8) # (sequence, output step, input position) weights = np.stack([w.numpy() for w in looked], axis=1) print("one sequence. Each row is one output step: " "how its attention was shared over the 8 input positions") print(" input:", " ".join(str(s) for s in source[0].tolist())) for t in range(8): print("step", t, "wrote", written[0, t].item(), "|", " ".join(f"{w:.2f}" if w >= 0.005 else " . " for w in weights[0, t])) # where each step looked hardest strongest = weights.argmax(axis=2) steps = np.arange(1, 8) # the input position of the symbol each step has to write needed = 7 - steps print() print("over 2,000 sequences, steps 1 to 7: where the strongest weight falls") print(" on the symbol it is about to write: ", f"{100 * (strongest[:, 1:] == needed).mean():.1f}%") print(" one place to the right, on the one it just wrote:", f"{100 * (strongest[:, 1:] == needed + 1).mean():.1f}%") right = (written == target).numpy() S = source.numpy() once, repeated = [], [] for t in steps: just_wrote = S[:, 8 - t] # how many times that symbol appears in the input copies = (S == just_wrote[:, None]).sum(axis=1) once.append(right[:, t][copies == 1]) repeated.append(right[:, t][copies > 1]) print("next symbol right, when the one just written appears once in the input:", f"{100 * np.concatenate(once).mean():.1f}%", "| more than once:", f"{100 * np.concatenate(repeated).mean():.1f}%")
Follow the large weights down the grid. They start at the right-hand end of the input and move left as the output is written. The model was never told the task was reversal. It found the order for itself.
Show the answer
Large weights sweep from right to left
To write the input backward, the decoder must read the last input first and the first input last. The large weights sweeping from the right to the left is that order, learned from the loss alone.
Now look again, more carefully. At step 4 the model writes the 5 from input position 3, and its strongest weight, 0.83, is on position 4. At step 6 it writes the 5 from position 1 and looks at position 2. It keeps looking one place to the right of the symbol it needs.
The summary confirms it. The strongest weight falls on the symbol about to be written 3.4% of the time, and on the one just written 88.9% of the time.
The reason is that an encoder state is not one symbol. It is a summary of the input so far, and it remembers the symbol before it well. So the decoder has learned a routine: find the place in the input that matches what I just wrote, and read from that state what came before.
Two lessons follow. An attention map shows where information was read from, and what was stored there is a separate question. And this attention finds things by content. When the symbol just written appears once in the input, the next one is right 95.7% of the time. When it appears more than once, 90.3%: the match is ambiguous.
Show the answer
It read mostly from that position's state, which may summarize nearby input too
A weight says how much of a state went into the context. In a recurrent encoder each state summarizes the input up to that point, so the information taken from it may concern a neighboring symbol, as it did in this lesson's model nearly nine times in ten. Attention maps are useful evidence about where a model looked. They are not a statement of which input caused the output.
Write a sentence or two, then compare it with a good answer in the lesson.
Teacher forcing, and working alone
There is a gap between how this model was trained and how it is used, and every model that writes one piece at a time has the same gap.
In training, forward gives the decoder the correct previous symbol at each step, whatever it would have chosen itself. This is called teacher forcing. It makes training fast and stable, since each step can be graded on its own.
In use, translate gives the decoder its own previous choice. Nobody is there to correct it.
Uses the setup above, 102 lines.
torch.manual_seed(0) model = train(Seq2Seq(attention=True), length=8) source, target = pairs(2000, 8, np.random.default_rng(99)) written, looked = model.translate(source, 8) # fed its own previous choices alone = (written == target).numpy() with torch.no_grad(): # fed the correct previous symbol every time helped = (model(source, target).argmax(dim=2) == target).numpy() print("symbols right, by output position") print(" given the correct previous symbol:", " ".join(f"{v:.2f}" for v in helped.mean(axis=0)), "| overall", f"{100 * helped.mean():.1f}%") print(" given its own previous choice: ", " ".join(f"{v:.2f}" for v in alone.mean(axis=0)), "| overall", f"{100 * alone.mean():.1f}%") slipped = (~alone).any(axis=1) # the step of each sequence's first mistake first = (~alone).argmax(axis=1) after_alone = np.concatenate([alone[i, first[i] + 1:] for i in np.flatnonzero(slipped)]) after_helped = np.concatenate([helped[i, first[i] + 1:] for i in np.flatnonzero(slipped)]) print() print("sequences with at least one mistake when working alone:", f"{100 * slipped.mean():.1f}%") print("in those, symbols right after the first mistake:", f"{100 * after_alone.mean():.1f}%", "| the same steps, given the correct previous symbol:", f"{100 * after_helped.mean():.1f}%")
Given the correct previous symbol, the model is right 97.5% of the time, and about equally at every position. Working alone it is right 93.7% of the time, and it gets worse along the sequence: 0.89 at the last position against 0.97.
The last two lines explain the gap. When working alone, 17.3% of sequences contain a mistake. In those sequences, the symbols after the first mistake are right only 31.0% of the time. At those very steps, given the correct previous symbol, the model is right 95.0% of the time.
So it has not forgotten how to do the task. It was thrown off by its own mistake. A wrong symbol goes back in as input, the model looks for its place using that symbol, lands somewhere else, and carries on from there. It never practiced recovering, since in training the previous symbol was always right. The gap has a name: exposure bias.
Large language models are trained with teacher forcing and used alone, in exactly this way. It is one reason a long generation can drift once it has gone wrong.
What the decoder is given under teacher forcing
Write decoder_inputs(target, start). Under teacher forcing, the decoder's input at each step is the correct symbol from the step before, and at the first step it is the start token. Return a new list, the same length as target. Don't change target.
Uses the setup above, 102 lines.
def decoder_inputs(target, start): return list(target) print(decoder_inputs([8, 0, 8, 6], 10))
Show the answer
A mistake fed back as input throws off the steps after it
At the same steps, with a correct previous symbol, the model does well, so its skill is intact. What differs is its input: its own wrong symbol, a situation it never met in training. One error changes what it sees next, and the damage spreads. Remedies include training partly on the model's own choices, and keeping several candidate outputs alive while decoding.
Worked example: why did it get this one wrong?
The attention weights and the idea of compounding mistakes together make a diagnostic tool. The cell finds the first sequence the model got wrong, locates its first wrong step, and prints the attention around it.
Uses the setup above, 102 lines.
torch.manual_seed(0) model = train(Seq2Seq(attention=True), length=8) source, target = pairs(2000, 8, np.random.default_rng(99)) written, looked = model.translate(source, 8) weights = np.stack([w.numpy() for w in looked], axis=1) wrong = (written != target).numpy() # the first sequence with a mistake row = int(np.flatnonzero(wrong.any(axis=1))[0]) # and its first wrong step step = int(wrong[row].argmax()) print("input: ", source[row].tolist()) print("wanted: ", target[row].tolist()) print("written:", written[row].tolist()) print() print("first mistake at step", step, ": wrote", written[row, step].item(), "where", target[row, step].item(), "was wanted. The symbol it needed is at input position", 7 - step) for t in range(max(0, step - 1), min(8, step + 2)): marks = " ".join(f"{w:.2f}" if w >= 0.005 else " . " for w in weights[row, t]) print(" step", t, "wrote", written[row, t].item(), "| attention:", marks, "| strongest at position", int(weights[row, t].argmax())) previous = written[row, step - 1].item() if step > 0 else None print("the symbol written just before the mistake:", previous, "| it appears in the input at positions", [p for p in range(8) if source[row, p].item() == previous])
The input is 2, 6, 7, 3, 2, 2, 3, 3, and backward it should be 3, 3, 2, 2, 3, 7, 6, 2. The model wrote 3, 3, 2, 2 correctly and then 7, where a 3 was wanted. Look at what it wrote after that: 7, 6, 2. Those are the right symbols in the right order, from input positions 2, 1 and 0. The model skipped one symbol, the 3 at position 3, and then carried on perfectly, one place early. Every later symbol is marked wrong for that single slip, and the sequence ends with an extra 3 to fill the space. Why the slip? The symbol written just before it was a 2, which appears at input positions 0, 4 and 5. At step 4 the attention is split, 0.52 on position 4 and 0.38 on position 5, across the two neighboring 2s. With two 2s followed by two 3s in the input, the model lost count of where it was. All three ideas from this lesson are in this one failure: attention that finds its place by content, a repeated symbol that makes the place ambiguous, and one mistake that shifts everything after it.
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: the same three lines in one call
Purpose. Check the hand-written attention against PyTorch's own function, which transformers use.
PyTorch's F.scaled_dot_product_attention does scores, softmax and blend in one call. Transformers divide the scores by √d; passing scale=1.0 turns that off, which makes it the same attention as the lesson's three lines. (The page's PyTorch is a CPU-only teaching subset.)
Uses the setup above, 102 lines.
import torch import torch.nn.functional as F torch.manual_seed(0) states = torch.randn(1, 5, 8) # (batch, positions, hidden) query = torch.randn(1, 1, 8) # one decoder step scores = query @ states.transpose(1, 2) # (1, 1, 5) weights = torch.softmax(scores, dim=-1) by_hand = weights @ states # (1, 1, 8) library = F.scaled_dot_product_attention(query, states, states, scale=1.0) print(tuple(by_hand.shape), (by_hand - library).abs().max().item() < 1e-5)
The shapes are (1, 1, 8) and the two agree to within 1e-5. This is the whole idea of attention. Scaled dot-product attention adds the 1/√d factor and separate query, key and value projections.
From here to the transformer
For three years, attention was an addition to recurrent networks. By 2016 a model of this design, far larger, was translating for the public. Its weaknesses were the ones you have just measured.
- The encoder still read one symbol at a time, so long inputs couldn't be processed in parallel.
- Each encoder state was a summary, so what attention fetched was blurred by its neighbors.
- Matching by content alone could not tell two copies of a symbol apart.
Show the answer
Reading one symbol at a time
Without recurrence every position can be processed at once, in parallel. Telling copies apart needs position information, which the transformer adds on purpose, and the pairwise scoring cost remains.
In 2017 a paper titled "Attention Is All You Need" removed the recurrence altogether. Every position attends directly to every other position, in the input and in the output so far, and information about position is added to each token since nothing else supplies it. That model is the transformer, and it is the subject of Phase 5.
The vocabulary carries over directly. The decoder's state, which asks the question, is the query. The encoder states are scored against it as keys, and blended as values. Here keys and values are the same vectors. In a transformer each gets its own learned projection.
One cost carries over too. Attention computes a score for every pair of an output step and an input position, so the work grows with the product of the two lengths.
print(8 * 8, 30 * 40, 1000 * 1000)
Show the answer
64 1200 1000000
Every output step looks at every input position, so the count is the product of the two lengths: 64, then 1,200, then a million. Doubling both lengths makes four times the work. This is the price of letting any step read from anywhere, and it is the reason long contexts are expensive for transformers.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 8 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 13 review cards for spaced repetition
Try a quiz question
In an encoder-decoder without attention, what does the decoder receive from the encoder?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 4 · about 24 hours of study
Deep learning
Neural networks, backprop and training them well
31 lessons 1 free
- The artificial neuron FreeWeighted sum plus a nonlinearity.35 min
- Activation functionsReLU, sigmoid, GELU, and why nonlinearity matters.35 min
- Multi-layer networksStack layers to learn complex functions.35 min
- The forward passFollow numbers through a network to a prediction.36 min
- Computation graphsEvery calculation as a graph of small operations.40 min
- BackpropagationThe chain rule applied backwards to get every gradient.42 min
- Build an autograd engineWrite backprop yourself in ~100 lines (micrograd).42 min
- PyTorch tensorsNumPy-like arrays that run on GPUs.38 min
- PyTorch autograd and modulesLet PyTorch compute gradients; build models with nn.Module.42 min
- The training loopForward, loss, backward, step, repeat.40 min
- Mini-batches and SGDTrain on small random batches instead of everything at once.40 min
- OptimizersMomentum and Adam.41 min
- Learning-rate schedules in PyTorchWarmup then cosine by hand, as a function of the step, then LambdaLR and where scheduler.step() goes; digits trained with and without warmup.25 min
- Vanishing and exploding gradientsWhy deep networks were hard to train.40 min
- Weight initializationStart weights at the right scale.33 min
- Normalization layersBatch and layer norm keep training stable.40 min
- Residual connectionsSkip connections that let gradients flow.40 min
- DropoutRandomly switch off neurons to prevent overfitting.35 min
- Debugging trainingRead a failing run from its numbers, and fix what's wrong.40 min
- Looking inside a networkRead four numbers per layer (spread, dead units, gradient norm, update-to-weight ratio), collect them in NumPy, then with forward and tensor hooks, and diagnose three planted faults.30 min
- Transfer learningReuse a trained network: freeze the backbone, train a new head, fine-tune with two learning rates.42 min
- Convolutional networksPattern detectors for images.44 min
- Embedding layersLearned lookup tables that turn tokens into vectors.38 min
- Recurrent networksProcess sequences step by step, and their limits.43 min
- LSTMs and GRUs, built and trainedBuild an LSTM and a GRU from a linear layer, train them on the adding problem beside a plain RNN, read what the gates learned, measure the gradient's path, and carry state across a stream.40 min
- Sequence to sequence, and the invention of attentionBuild an encoder and decoder, measure the bottleneck of a fixed-size handoff, add attention in three lines, read an attention map with care, and see how one mistake compounds after teacher forcing.45 min
- AutoencodersEncoder, bottleneck and decoder, reconstruction error for anomalies, denoising, and the latent space image generators use.42 min
- GPUs and computeWhy deep learning runs on graphics cards.40 min
- Training at scale: precision, accumulation, data and many GPUsGradient accumulation and what it really saves, scaling the learning rate with the batch, mixed precision and loss scaling, a data loader that starves the GPU, and why eight GPUs aren't eight times as fast.45 min
- A training script, start to finishConfig, seed, data, model, schedule, loop and checkpoint as one script: what a checkpoint must hold, a resume that matches an unbroken run, and buffers, collate and torch.save in PyTorch.40 min
- Capstone: train a seam-tear spotterTrain and compare three models on fabric images, choose the alarm level from costs, and score once.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write