Index Ai Academy
Phase 4 · Architectures

Sequence to sequence, and the invention of attention

Build an encoder and decoder, measure the bottleneck of a fixed-size handoff, add attention in three lines, read an attention map with care, and see how one mistake compounds after teacher forcing.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Attention was not invented for transformers. It was invented in 2014 to repair a specific failure in translation models built from recurrent networks, and for three years it was an add-on to them. Building that original version is the best way to understand what attention is for. You will watch a model fail for the reason attention was invented, add three lines, and watch it recover. The lesson also covers how every sequence generator is trained and used, teacher forcing and step-by-step decoding, which applies unchanged to the largest language models.

What you'll be able to do

  • Build an encoder and a decoder from recurrent cells, and train them to turn one sequence into another
  • Measure the bottleneck: what a fixed-size handoff loses as inputs get longer
  • Add attention in three lines, and explain scores, weights and context
  • Read an attention map with care: what it shows and what it doesn't
  • Explain teacher forcing, and measure how one mistake compounds when the model works alone
Quick check
A guess before you start. A translator must read a whole sentence, close the book, and then write the translation from memory. As sentences get longer, what goes wrong first?
  1. The translator forgets the target language
  2. The last few words read are forgotten
  3. Short words are mistranslated more than long ones
  4. The parts read earliest are remembered worst
Show the answer

The parts read earliest are remembered worst

With the book closed, everything must be held in memory at once, and what was read first has been held longest and overwritten most. A better translator would keep the book open and glance back at the relevant words while writing each part. The first translation networks worked with the book closed. Attention is what opened it.

The idea: two networks and a handoff

Rests on Attention as matrix math (Foundation lesson 113), where five words looked at one another by hand.

Translation can't be done one word at a time. The output has a different length from the input, and a different order. In 2014 two research groups proposed the same answer, now called sequence to sequence, or the encoder-decoder:

  • An encoder reads the whole input, one symbol at a time, and ends with a state.
  • A decoder starts from that state and writes the output one symbol at a time. At each step it is given the symbol it wrote last, beginning with a special START token.

The only thing passed between them is the encoder's final state.

Real translation is hard to grade, so this lesson uses a stand-in with the same shape: the output is the input written backward. The order changes, every symbol matters, and an answer is simply right or wrong. Both networks are the GRU cell from LSTMs and GRUs, built and trained.

Quick check
In the plain encoder-decoder, what is the only thing passed from the encoder to the decoder?
  1. The encoder's final state
  2. The loss
  3. Every input symbol, unchanged
  4. The encoder's weights
Show the answer

The encoder's final state

The decoder starts from that one state and writes the output from it. Everything it knows about the input must fit inside it.

Attention by hand

The encoder keeps a state for every input symbol. Take three states, each two numbers: (1, 0), (0, 1) and (1, 1). The decoder's own state, the query, is (2, 0). Attention does three things.

  1. Scores. A dot product of the query with each state: 2, 0 and 2.
  2. Weights. A softmax of the scores. e² is about 7.39 and e⁰ is 1, so the weights are 7.39, 1 and 7.39 shared out: about 0.468, 0.063 and 0.468.
  3. Context. The states blended by the weights: 0.468 × (1, 0) + 0.063 × (0, 1) + 0.468 × (1, 1) is about (0.936, 0.531).

The context leans toward the two states that matched the query, and keeps a trace of the third.

Setup (runs before every cell) · 102 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

# symbols are 0 to 9. Token 10 tells the decoder to begin
START = 10

def pairs(n, length, rng):
    """A stand-in for translation: the output is the input written backward."""
    source = rng.integers(0, 10, (n, length))
    return torch.tensor(source), torch.tensor(source[:, ::-1].copy())

class GRUCell(nn.Module):
    def __init__(self, n_in, n_hidden):
        super().__init__()
        self.gates = nn.Linear(n_in + n_hidden, 2 * n_hidden)
        self.candidate = nn.Linear(n_in + n_hidden, n_hidden)

    def forward(self, x, h):
        z, r = torch.sigmoid(self.gates(torch.cat([x, h], dim=1))).chunk(2, dim=1)
        new = torch.tanh(self.candidate(torch.cat([x, r * h], dim=1)))
        return z * h + (1 - z) * new

class Seq2Seq(nn.Module):
    def __init__(self, n_hidden=32, attention=False):
        super().__init__()
        self.embed = nn.Embedding(11, 16)
        self.encoder, self.decoder = GRUCell(16, n_hidden), GRUCell(16, n_hidden)
        self.out = nn.Linear(2 * n_hidden if attention else n_hidden, 10)
        self.n_hidden, self.attention = n_hidden, attention

    def encode(self, source):
        """Read the input one symbol at a time. Return the state after every symbol,
        and the last state."""
        h, states = torch.zeros(source.shape[0], self.n_hidden), []
        for t in range(source.shape[1]):
            h = self.encoder(self.embed(source[:, t]), h)
            states.append(h)
        return torch.stack(states, dim=1), h

    def step(self, token, h, states):
        """One step of writing: take the previous output symbol, update the state,
        score the 10 possible next symbols."""
        h = self.decoder(self.embed(token), h)
        if not self.attention:
            return self.out(h), h, None
        # dot product of h with each encoder state
        scores = torch.bmm(states, h.unsqueeze(2)).squeeze(2)
        # one weight for each input position, adding up to 1
        weights = torch.softmax(scores, dim=1)
        # the encoder states, blended by those weights
        context = torch.bmm(weights.unsqueeze(1), states).squeeze(1)
        return self.out(torch.cat([h, context], dim=1)), h, weights

    def forward(self, source, target):
        """Training. At each step the decoder is given the correct previous symbol:
        teacher forcing."""
        states, h = self.encode(source)
        token, scores = torch.full((source.shape[0],), START, dtype=torch.long), []
        for t in range(target.shape[1]):
            out, h, weights = self.step(token, h, states)
            scores.append(out)
            token = target[:, t]
        return torch.stack(scores, dim=1)

    def translate(self, source, steps):
        """Use. At each step the decoder is given its own previous choice."""
        with torch.no_grad():
            states, h = self.encode(source)
            token = torch.full((source.shape[0],), START, dtype=torch.long)
            written, looked = [], []
            for t in range(steps):
                out, h, weights = self.step(token, h, states)
                token = out.argmax(dim=1)
                written.append(token)
                looked.append(weights)
            return torch.stack(written, dim=1), looked

def train(model, length, batches=300):
    rng = np.random.default_rng(0)
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
    for k in range(batches):
        source, target = pairs(64, length, rng)
        loss = F.cross_entropy(model(source, target).reshape(-1, 10),
                               target.reshape(-1))
        optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
    return model

def report(model, length, n=500):
    source, target = pairs(n, length, np.random.default_rng(99))
    written, looked = model.translate(source, length)
    right = (written == target).float()
    print("   symbols right:", f"{100 * right.mean().item():.1f}%",
          "| whole sequences right:",
          f"{100 * (right.sum(dim=1) == length).float().mean().item():.1f}%")
    print("   by output position:",
          " ".join(f"{right[:, t].mean().item():.2f}" for t in range(length)))
    return [right[:, t].mean().item() for t in range(length)]
import numpy as np

states = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
query = np.array([2.0, 0.0])
scores = states @ query
weights = np.exp(scores) / np.exp(scores).sum()
print(scores, weights.round(3), (weights @ states).round(3))
It runs right here in your browser, with nothing to install.

The printed numbers are the three above.

Try it now

The three attention lines on three encoder states. The context has the size of one state.

Quick check
From the Foundation: attention scores three words as 2, 0 and 0. After a softmax, about how much weight does the first get?
  1. About 0.67
  2. 2.0
  3. 1.0
  4. About 0.79
Show the answer

About 0.79

e² ≈ 7.39 and the others give 1 each, so the first gets 7.39 ÷ 9.39 ≈ 0.79. Softmax is soft: the others keep a share.

Quick check
A decoder state gives these scores against three encoder states. What does this print: the attention weights?
import math

scores = [2.0, 0.0, 0.0]
total = sum(math.exp(s) for s in scores)
print([round(math.exp(s) / total, 2) for s in scores])
  1. [1.0, 0.0, 0.0]
  2. [0.67, 0.17, 0.17]
  3. [0.79, 0.11, 0.11]
  4. [2.0, 0.0, 0.0]
Show the answer

[0.79, 0.11, 0.11]

The softmax raises e to each score and divides by the total: e² is about 7.39, and e⁰ is 1 for each of the others, so the weights are 7.39 / 9.39 ≈ 0.79, and 0.11 twice. Attention is soft: most of the weight goes to the best match, and every position keeps a little.

Quick check
The weights over two encoder states are 0.9 and 0.1. What does this print: the context vector?
weights = [0.9, 0.1]
states = [[2.0, 0.0], [0.0, 4.0]]
context = [sum(w * s[k] for w, s in zip(weights, states)) for k in range(2)]
print([round(c, 2) for c in context])
  1. [1.8, 0.4]
  2. [2.0, 4.0]
  3. [1.0, 2.0]
  4. [2.0, 0.0]
Show the answer

[1.8, 0.4]

The context is a weighted average of the states: 0.9 × [2, 0] plus 0.1 × [0, 4], which is [1.8, 0.4]. It is mostly the first state, with a trace of the second. The context always has the size of one state, however many states it blends.

In your own words
What does attention compute at each step, in three words?

Write a sentence or two, then compare it with a good answer in the lesson.

From scratch: an encoder and a decoder

The cell below builds both networks and trains the version with no attention on sequences of 4 symbols. The same definitions are loaded for every later cell, from the lesson's setup.

Uses the setup above, 102 lines.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

# symbols are 0 to 9. Token 10 tells the decoder to begin
START = 10

def pairs(n, length, rng):
    """A stand-in for translation: the output is the input written backward."""
    source = rng.integers(0, 10, (n, length))
    return torch.tensor(source), torch.tensor(source[:, ::-1].copy())

class GRUCell(nn.Module):
    def __init__(self, n_in, n_hidden):
        super().__init__()
        self.gates = nn.Linear(n_in + n_hidden, 2 * n_hidden)
        self.candidate = nn.Linear(n_in + n_hidden, n_hidden)

    def forward(self, x, h):
        z, r = torch.sigmoid(self.gates(torch.cat([x, h], dim=1))).chunk(2, dim=1)
        new = torch.tanh(self.candidate(torch.cat([x, r * h], dim=1)))
        return z * h + (1 - z) * new

class Seq2Seq(nn.Module):
    def __init__(self, n_hidden=32, attention=False):
        super().__init__()
        self.embed = nn.Embedding(11, 16)
        self.encoder, self.decoder = GRUCell(16, n_hidden), GRUCell(16, n_hidden)
        self.out = nn.Linear(2 * n_hidden if attention else n_hidden, 10)
        self.n_hidden, self.attention = n_hidden, attention

    def encode(self, source):
        """Read the input one symbol at a time. Return the state after every symbol,
        and the last state."""
        h, states = torch.zeros(source.shape[0], self.n_hidden), []
        for t in range(source.shape[1]):
            h = self.encoder(self.embed(source[:, t]), h)
            states.append(h)
        return torch.stack(states, dim=1), h

    def step(self, token, h, states):
        """One step of writing: take the previous output symbol, update the state,
        score the 10 possible next symbols."""
        h = self.decoder(self.embed(token), h)
        if not self.attention:
            return self.out(h), h, None
        # dot product of h with each encoder state
        scores = torch.bmm(states, h.unsqueeze(2)).squeeze(2)
        # one weight for each input position, adding up to 1
        weights = torch.softmax(scores, dim=1)
        # the encoder states, blended by those weights
        context = torch.bmm(weights.unsqueeze(1), states).squeeze(1)
        return self.out(torch.cat([h, context], dim=1)), h, weights

    def forward(self, source, target):
        """Training. At each step the decoder is given the correct previous symbol:
        teacher forcing."""
        states, h = self.encode(source)
        token, scores = torch.full((source.shape[0],), START, dtype=torch.long), []
        for t in range(target.shape[1]):
            out, h, weights = self.step(token, h, states)
            scores.append(out)
            token = target[:, t]
        return torch.stack(scores, dim=1)

    def translate(self, source, steps):
        """Use. At each step the decoder is given its own previous choice."""
        with torch.no_grad():
            states, h = self.encode(source)
            token = torch.full((source.shape[0],), START, dtype=torch.long)
            written, looked = [], []
            for t in range(steps):
                out, h, weights = self.step(token, h, states)
                token = out.argmax(dim=1)
                written.append(token)
                looked.append(weights)
            return torch.stack(written, dim=1), looked

def train(model, length, batches=300):
    rng = np.random.default_rng(0)
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
    for k in range(batches):
        source, target = pairs(64, length, rng)
        loss = F.cross_entropy(model(source, target).reshape(-1, 10),
                               target.reshape(-1))
        optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
    return model

def report(model, length, n=500):
    source, target = pairs(n, length, np.random.default_rng(99))
    written, looked = model.translate(source, length)
    right = (written == target).float()
    print("   symbols right:", f"{100 * right.mean().item():.1f}%",
          "| whole sequences right:",
          f"{100 * (right.sum(dim=1) == length).float().mean().item():.1f}%")
    print("   by output position:",
          " ".join(f"{right[:, t].mean().item():.2f}" for t in range(length)))
    return [right[:, t].mean().item() for t in range(length)]

torch.manual_seed(0)
model = train(Seq2Seq(), length=4)

source, target = pairs(3, 4, np.random.default_rng(5))
written, looked = model.translate(source, 4)
for row in range(3):
    print("input", source[row].tolist(), "-> written", written[row].tolist(),
          "| wanted", target[row].tolist())
print()
print("encoder and decoder, no attention, sequences of 4 symbols:")
report(model, 4)
print("numbers handed from the encoder to the decoder:", model.n_hidden)

Read step and translate in the code before the results. step takes the previous symbol and the state, and scores the ten possible next symbols. translate calls it repeatedly, feeding each choice back in.

On sequences of 4 symbols it works. The three examples are right, 97.8% of symbols are right, and 95.2% of whole sequences come out perfect. Everything the decoder knows about the input reached it as 32 numbers.

Quick check
From the lesson on embedding layers: the model begins with nn.Embedding(11, 16). What does that layer do with a symbol such as 7?
  1. Looks up row 7 of a learned table, and returns its 16 numbers
  2. Multiplies 7 by each of 16 learned weights
  3. Turns 7 into a list of eleven 0s with a single 1
  4. Rounds 7 to the nearest of 16 allowed values
Show the answer

Looks up row 7 of a learned table, and returns its 16 numbers

An embedding layer is a table with one row for each token: here 11 rows (ten symbols and START) of 16 numbers. A token's number is used as an index, never as a quantity, so 7 is not "more" than 3. The rows are learned along with every other weight, and both the encoder and the decoder read their symbols through this table.

The bottleneck

Now make the sequences 8 symbols long. The model, its size and the 300 training batches stay the same.

Uses the setup above, 102 lines.

import matplotlib.pyplot as plt

torch.manual_seed(0)
model = train(Seq2Seq(), length=8)
print("the same model and the same 300 batches, "
      "sequences of 8 symbols (this takes a few seconds):")
acc = report(model, 8)
plt.plot(acc, marker="o")
plt.xlabel("output position")
plt.ylabel("share of symbols right")

Whole sequences right: 11.2%, down from 95.2%.

The second line says where it breaks. The first output symbol is right 97% of the time. Then 0.85, 0.70, 0.60, and by the end 0.36.

Remember what the task is. The first symbol to write is the last one the encoder read, which is fresh in its state. The last symbol to write is the first one it read, eight steps earlier. The model fails in the same order as its memory fades.

This is the bottleneck. However long the input, the decoder receives one vector of fixed size, and the encoder has to decide what to keep before it knows what will be asked. A larger state moves the problem to longer inputs without removing it. Translation systems of 2014 showed exactly this: quality fell away as sentences grew.

One of the first papers found a trick that tells the same story: feeding the source sentence in reversed improved translation, since it put the first source words next to the first words to be written. That helps the start of a sentence at the expense of its end, and the next idea made it unnecessary.

Quick check
The model without attention gets the first output symbol right 97% of the time and the last only 36%. In this task, why are the later outputs worse?
  1. The decoder gets tired after writing several symbols
  2. Later symbols are drawn from a larger set of possibilities
  3. They depend on the earliest inputs, which have faded most from the handoff
  4. The loss function ignores the last few output positions
Show the answer

They depend on the earliest inputs, which have faded most from the handoff

Writing backward means the last output is the first input. By the time the encoder finishes, that symbol has passed through seven more updates of the state, and little of it survives in the 32 numbers handed over. The decoder can't recover what was never passed to it.

Attention: looking back at the input

The encoder produced a state after every input symbol, and all but the last were thrown away. The idea of attention, published late in 2014, is to keep them all and let the decoder consult them at each step. In the model's step it is three lines:

  1. Scores. Compare the decoder's current state with each encoder state, by a dot product. A high score means "this one looks relevant now".
  2. Weights. A softmax turns the scores into positive weights that add up to 1.
  3. Context. Blend the encoder states, using the weights. The output layer then sees the decoder's state and this context together.

Nothing tells the model where to look. The weights come from the two networks' own states, and they are trained by the same loss as everything else.

In the code, torch.bmm is a batched matrix product: it does one matrix product for each sequence in the batch, so (batch, n, m) times (batch, m, p) gives (batch, n, p). h.unsqueeze(2) stands the decoder's state up as a column for it, and squeeze drops the extra axis of size 1 afterward.

Uses the setup above, 102 lines.

import matplotlib.pyplot as plt

torch.manual_seed(0)
model = train(Seq2Seq(attention=True), length=8)
print("with attention, sequences of 8 symbols (this takes a few seconds):")
acc = report(model, 8)
plt.plot(acc, marker="o")
plt.xlabel("output position")
plt.ylabel("share of symbols right")

Whole sequences right: 82.2%, up from 11.2%. The last output positions, which were at 0.36, are at 0.88. The model is the same size apart from a wider output layer, and it had the same 300 batches.

What changed is where the information travels. The early input symbols no longer have to survive inside one vector. They wait in their own encoder states, and the decoder reaches back for them when it needs them.

The first version used a small network to compute the scores. The dot product used here came a year later, and it is the form that transformers kept.

Drill

Attention for one decoder step

Write attend(query, states) in NumPy. query is the decoder's state, of shape (hidden,). states holds the encoder's states, of shape (positions, hidden). Score each state by its dot product with the query, turn the scores into weights with a softmax, and blend the states. Return (weights, context). Subtract the largest score before np.exp, so that big scores don't overflow.

Uses the setup above, 102 lines.

import numpy as np


def attend(query, states):
    return np.zeros(len(states)), np.zeros(states.shape[1])


states = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
weights, context = attend(np.array([2.0, 0.0]), states)
print(weights.round(3), context.round(3))
In the lesson, your code is checked as you write it.

What is it looking at?

The weights can be printed, which makes attention unusual among the parts of a network: you can see where it looked. The cell prints the weights for one sequence, then asks a question about all 2,000.

Uses the setup above, 102 lines.

torch.manual_seed(0)
model = train(Seq2Seq(attention=True), length=8)

source, target = pairs(2000, 8, np.random.default_rng(99))
written, looked = model.translate(source, 8)
# (sequence, output step, input position)
weights = np.stack([w.numpy() for w in looked], axis=1)

print("one sequence. Each row is one output step: "
      "how its attention was shared over the 8 input positions")
print("            input:", "    ".join(str(s) for s in source[0].tolist()))
for t in range(8):
    print("step", t, "wrote", written[0, t].item(), "|",
          " ".join(f"{w:.2f}" if w >= 0.005 else "  . " for w in weights[0, t]))

# where each step looked hardest
strongest = weights.argmax(axis=2)
steps = np.arange(1, 8)
# the input position of the symbol each step has to write
needed = 7 - steps
print()
print("over 2,000 sequences, steps 1 to 7: where the strongest weight falls")
print("   on the symbol it is about to write:        ",
      f"{100 * (strongest[:, 1:] == needed).mean():.1f}%")
print("   one place to the right, on the one it just wrote:",
      f"{100 * (strongest[:, 1:] == needed + 1).mean():.1f}%")

right = (written == target).numpy()
S = source.numpy()
once, repeated = [], []
for t in steps:
    just_wrote = S[:, 8 - t]
    # how many times that symbol appears in the input
    copies = (S == just_wrote[:, None]).sum(axis=1)
    once.append(right[:, t][copies == 1])
    repeated.append(right[:, t][copies > 1])
print("next symbol right, when the one just written appears once in the input:",
      f"{100 * np.concatenate(once).mean():.1f}%", "| more than once:",
      f"{100 * np.concatenate(repeated).mean():.1f}%")

Follow the large weights down the grid. They start at the right-hand end of the input and move left as the output is written. The model was never told the task was reversal. It found the order for itself.

Quick check
The model was never told the task was reversal. What in the attention grid shows that it found the order?
  1. The weights add up to more than 1
  2. The first row has all its weight on the last output
  3. Large weights sweep from right to left
  4. Every weight is equal on every row
Show the answer

Large weights sweep from right to left

To write the input backward, the decoder must read the last input first and the first input last. The large weights sweeping from the right to the left is that order, learned from the loss alone.

Now look again, more carefully. At step 4 the model writes the 5 from input position 3, and its strongest weight, 0.83, is on position 4. At step 6 it writes the 5 from position 1 and looks at position 2. It keeps looking one place to the right of the symbol it needs.

The summary confirms it. The strongest weight falls on the symbol about to be written 3.4% of the time, and on the one just written 88.9% of the time.

The reason is that an encoder state is not one symbol. It is a summary of the input so far, and it remembers the symbol before it well. So the decoder has learned a routine: find the place in the input that matches what I just wrote, and read from that state what came before.

Two lessons follow. An attention map shows where information was read from, and what was stored there is a separate question. And this attention finds things by content. When the symbol just written appears once in the input, the next one is right 95.7% of the time. When it appears more than once, 90.3%: the match is ambiguous.

Quick check
An attention map shows that when a model wrote one word, its largest weight was on a particular input position. What can you conclude?
  1. The input word at that position is the translation of the output word
  2. The model ignored every other part of the input
  3. It read mostly from that position's state, which may summarize nearby input too
  4. The model was certain about the word it wrote
Show the answer

It read mostly from that position's state, which may summarize nearby input too

A weight says how much of a state went into the context. In a recurrent encoder each state summarizes the input up to that point, so the information taken from it may concern a neighboring symbol, as it did in this lesson's model nearly nine times in ten. Attention maps are useful evidence about where a model looked. They are not a statement of which input caused the output.

In your own words
This model finds its place by matching the symbol it just wrote. What kind of input makes that fail, and what would the model need in order to do better?

Write a sentence or two, then compare it with a good answer in the lesson.

Teacher forcing, and working alone

There is a gap between how this model was trained and how it is used, and every model that writes one piece at a time has the same gap.

In training, forward gives the decoder the correct previous symbol at each step, whatever it would have chosen itself. This is called teacher forcing. It makes training fast and stable, since each step can be graded on its own.

In use, translate gives the decoder its own previous choice. Nobody is there to correct it.

Uses the setup above, 102 lines.

torch.manual_seed(0)
model = train(Seq2Seq(attention=True), length=8)

source, target = pairs(2000, 8, np.random.default_rng(99))
written, looked = model.translate(source, 8)
# fed its own previous choices
alone = (written == target).numpy()
with torch.no_grad():
    # fed the correct previous symbol every time
    helped = (model(source, target).argmax(dim=2) == target).numpy()

print("symbols right, by output position")
print("   given the correct previous symbol:",
      " ".join(f"{v:.2f}" for v in helped.mean(axis=0)), "| overall",
      f"{100 * helped.mean():.1f}%")
print("   given its own previous choice:    ",
      " ".join(f"{v:.2f}" for v in alone.mean(axis=0)), "| overall",
      f"{100 * alone.mean():.1f}%")

slipped = (~alone).any(axis=1)
# the step of each sequence's first mistake
first = (~alone).argmax(axis=1)
after_alone = np.concatenate([alone[i, first[i] + 1:] for i in np.flatnonzero(slipped)])
after_helped = np.concatenate([helped[i, first[i] + 1:]
                               for i in np.flatnonzero(slipped)])
print()
print("sequences with at least one mistake when working alone:",
      f"{100 * slipped.mean():.1f}%")
print("in those, symbols right after the first mistake:",
      f"{100 * after_alone.mean():.1f}%",
      "| the same steps, given the correct previous symbol:",
      f"{100 * after_helped.mean():.1f}%")

Given the correct previous symbol, the model is right 97.5% of the time, and about equally at every position. Working alone it is right 93.7% of the time, and it gets worse along the sequence: 0.89 at the last position against 0.97.

The last two lines explain the gap. When working alone, 17.3% of sequences contain a mistake. In those sequences, the symbols after the first mistake are right only 31.0% of the time. At those very steps, given the correct previous symbol, the model is right 95.0% of the time.

So it has not forgotten how to do the task. It was thrown off by its own mistake. A wrong symbol goes back in as input, the model looks for its place using that symbol, lands somewhere else, and carries on from there. It never practiced recovering, since in training the previous symbol was always right. The gap has a name: exposure bias.

Large language models are trained with teacher forcing and used alone, in exactly this way. It is one reason a long generation can drift once it has gone wrong.

Drill

What the decoder is given under teacher forcing

Write decoder_inputs(target, start). Under teacher forcing, the decoder's input at each step is the correct symbol from the step before, and at the first step it is the start token. Return a new list, the same length as target. Don't change target.

Uses the setup above, 102 lines.

def decoder_inputs(target, start):
    return list(target)


print(decoder_inputs([8, 0, 8, 6], 10))
In the lesson, your code is checked as you write it.
Quick check
A model is right about 97% of the time at every step when given the correct previous symbol, but only about 31% of the time on the steps after its own first mistake. What does that show?
  1. The model was trained on too few examples of long sequences
  2. The model's attention stops working after a few steps
  3. A mistake fed back as input throws off the steps after it
  4. Teacher forcing makes a model less accurate overall
Show the answer

A mistake fed back as input throws off the steps after it

At the same steps, with a correct previous symbol, the model does well, so its skill is intact. What differs is its input: its own wrong symbol, a situation it never met in training. One error changes what it sees next, and the damage spreads. Remedies include training partly on the model's own choices, and keeping several candidate outputs alive while decoding.

Worked example

Worked example: why did it get this one wrong?

The attention weights and the idea of compounding mistakes together make a diagnostic tool. The cell finds the first sequence the model got wrong, locates its first wrong step, and prints the attention around it.

Uses the setup above, 102 lines.

torch.manual_seed(0)
model = train(Seq2Seq(attention=True), length=8)

source, target = pairs(2000, 8, np.random.default_rng(99))
written, looked = model.translate(source, 8)
weights = np.stack([w.numpy() for w in looked], axis=1)
wrong = (written != target).numpy()

# the first sequence with a mistake
row = int(np.flatnonzero(wrong.any(axis=1))[0])
# and its first wrong step
step = int(wrong[row].argmax())
print("input:  ", source[row].tolist())
print("wanted: ", target[row].tolist())
print("written:", written[row].tolist())
print()
print("first mistake at step", step, ": wrote", written[row, step].item(), "where",
      target[row, step].item(), "was wanted. The symbol it needed is at input position",
      7 - step)
for t in range(max(0, step - 1), min(8, step + 2)):
    marks = " ".join(f"{w:.2f}" if w >= 0.005 else "  . " for w in weights[row, t])
    print("  step", t, "wrote", written[row, t].item(), "| attention:", marks,
          "| strongest at position", int(weights[row, t].argmax()))
previous = written[row, step - 1].item() if step > 0 else None
print("the symbol written just before the mistake:", previous,
      "| it appears in the input at positions",
      [p for p in range(8) if source[row, p].item() == previous])
In the lesson you can change this and run it.

The input is 2, 6, 7, 3, 2, 2, 3, 3, and backward it should be 3, 3, 2, 2, 3, 7, 6, 2. The model wrote 3, 3, 2, 2 correctly and then 7, where a 3 was wanted. Look at what it wrote after that: 7, 6, 2. Those are the right symbols in the right order, from input positions 2, 1 and 0. The model skipped one symbol, the 3 at position 3, and then carried on perfectly, one place early. Every later symbol is marked wrong for that single slip, and the sequence ends with an extra 3 to fill the space. Why the slip? The symbol written just before it was a 2, which appears at input positions 0, 4 and 5. At step 4 the attention is split, 0.52 on position 4 and 0.38 on position 5, across the two neighboring 2s. With two 2s followed by two 3s in the input, the model lost count of where it was. All three ideas from this lesson are in this one failure: attention that finds its place by content, a repeated symbol that makes the place ambiguous, and one mistake that shifts everything after it.

In your own words
Why is teacher forcing used at all, if it leaves the model unprepared for its own mistakes?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: the same three lines in one call

Purpose. Check the hand-written attention against PyTorch's own function, which transformers use.

PyTorch's F.scaled_dot_product_attention does scores, softmax and blend in one call. Transformers divide the scores by √d; passing scale=1.0 turns that off, which makes it the same attention as the lesson's three lines. (The page's PyTorch is a CPU-only teaching subset.)

Uses the setup above, 102 lines.

import torch
import torch.nn.functional as F

torch.manual_seed(0)
states = torch.randn(1, 5, 8)                    # (batch, positions, hidden)
query = torch.randn(1, 1, 8)                     # one decoder step
scores = query @ states.transpose(1, 2)          # (1, 1, 5)
weights = torch.softmax(scores, dim=-1)
by_hand = weights @ states                        # (1, 1, 8)
library = F.scaled_dot_product_attention(query, states, states, scale=1.0)
print(tuple(by_hand.shape), (by_hand - library).abs().max().item() < 1e-5)

The shapes are (1, 1, 8) and the two agree to within 1e-5. This is the whole idea of attention. Scaled dot-product attention adds the 1/√d factor and separate query, key and value projections.

From here to the transformer

For three years, attention was an addition to recurrent networks. By 2016 a model of this design, far larger, was translating for the public. Its weaknesses were the ones you have just measured.

  • The encoder still read one symbol at a time, so long inputs couldn't be processed in parallel.
  • Each encoder state was a summary, so what attention fetched was blurred by its neighbors.
  • Matching by content alone could not tell two copies of a symbol apart.
Quick check
Which of those three weaknesses does dropping the recurrence remove outright?
  1. Reading one symbol at a time
  2. Not telling two copies apart
  3. Summaries blurred by neighbors
  4. The cost of scoring every pair
Show the answer

Reading one symbol at a time

Without recurrence every position can be processed at once, in parallel. Telling copies apart needs position information, which the transformer adds on purpose, and the pairwise scoring cost remains.

In 2017 a paper titled "Attention Is All You Need" removed the recurrence altogether. Every position attends directly to every other position, in the input and in the output so far, and information about position is added to each token since nothing else supplies it. That model is the transformer, and it is the subject of Phase 5.

The vocabulary carries over directly. The decoder's state, which asks the question, is the query. The encoder states are scored against it as keys, and blended as values. Here keys and values are the same vectors. In a transformer each gets its own learned projection.

One cost carries over too. Attention computes a score for every pair of an output step and an input position, so the work grows with the product of the two lengths.

Quick check
Attention computes one score for each pair of an output step and an input position. What does this print: the number of scores for 8 symbols in and 8 out, for a 30-word sentence translated into 40 words, and for 1,000 tokens in and 1,000 out?
print(8 * 8, 30 * 40, 1000 * 1000)
  1. 64 900 1000
  2. 64 1200 1000000
  3. 16 70 2000
  4. 8 40 1000
Show the answer

64 1200 1000000

Every output step looks at every input position, so the count is the product of the two lengths: 64, then 1,200, then a million. Doubling both lengths makes four times the work. This is the price of letting any step read from anywhere, and it is the reason long contexts are expensive for transformers.

In your own words
Summarize the path from the 2014 encoder-decoder to the transformer in three steps: what each design could not do, and what was changed.

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 8 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 13 review cards for spaced repetition

Try a quiz question

In an encoder-decoder without attention, what does the decoder receive from the encoder?

  1. One state of fixed size, produced after the last input symbol
  2. A separate state for each input symbol, to consult at will
  3. The input symbols themselves, in their original order
  4. The encoder's weights, copied into the decoder

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 4 · about 24 hours of study

Deep learning

Neural networks, backprop and training them well 31 lessons 1 free
  1. The artificial neuron FreeWeighted sum plus a nonlinearity.35 min
  2. Activation functionsReLU, sigmoid, GELU, and why nonlinearity matters.35 min
  3. Multi-layer networksStack layers to learn complex functions.35 min
  4. The forward passFollow numbers through a network to a prediction.36 min
  5. Computation graphsEvery calculation as a graph of small operations.40 min
  6. BackpropagationThe chain rule applied backwards to get every gradient.42 min
  7. Build an autograd engineWrite backprop yourself in ~100 lines (micrograd).42 min
  8. PyTorch tensorsNumPy-like arrays that run on GPUs.38 min
  9. PyTorch autograd and modulesLet PyTorch compute gradients; build models with nn.Module.42 min
  10. The training loopForward, loss, backward, step, repeat.40 min
  11. Mini-batches and SGDTrain on small random batches instead of everything at once.40 min
  12. OptimizersMomentum and Adam.41 min
  13. Learning-rate schedules in PyTorchWarmup then cosine by hand, as a function of the step, then LambdaLR and where scheduler.step() goes; digits trained with and without warmup.25 min
  14. Vanishing and exploding gradientsWhy deep networks were hard to train.40 min
  15. Weight initializationStart weights at the right scale.33 min
  16. Normalization layersBatch and layer norm keep training stable.40 min
  17. Residual connectionsSkip connections that let gradients flow.40 min
  18. DropoutRandomly switch off neurons to prevent overfitting.35 min
  19. Debugging trainingRead a failing run from its numbers, and fix what's wrong.40 min
  20. Looking inside a networkRead four numbers per layer (spread, dead units, gradient norm, update-to-weight ratio), collect them in NumPy, then with forward and tensor hooks, and diagnose three planted faults.30 min
  21. Transfer learningReuse a trained network: freeze the backbone, train a new head, fine-tune with two learning rates.42 min
  22. Convolutional networksPattern detectors for images.44 min
  23. Embedding layersLearned lookup tables that turn tokens into vectors.38 min
  24. Recurrent networksProcess sequences step by step, and their limits.43 min
  25. LSTMs and GRUs, built and trainedBuild an LSTM and a GRU from a linear layer, train them on the adding problem beside a plain RNN, read what the gates learned, measure the gradient's path, and carry state across a stream.40 min
  26. Sequence to sequence, and the invention of attentionBuild an encoder and decoder, measure the bottleneck of a fixed-size handoff, add attention in three lines, read an attention map with care, and see how one mistake compounds after teacher forcing.45 min
  27. AutoencodersEncoder, bottleneck and decoder, reconstruction error for anomalies, denoising, and the latent space image generators use.42 min
  28. GPUs and computeWhy deep learning runs on graphics cards.40 min
  29. Training at scale: precision, accumulation, data and many GPUsGradient accumulation and what it really saves, scaling the learning rate with the batch, mixed precision and loss scaling, a data loader that starves the GPU, and why eight GPUs aren't eight times as fast.45 min
  30. A training script, start to finishConfig, seed, data, model, schedule, loop and checkpoint as one script: what a checkpoint must hold, a resume that matches an unbroken run, and buffers, collate and torch.save in PyTorch.40 min
  31. Capstone: train a seam-tear spotterTrain and compare three models on fabric images, choose the alarm level from costs, and score once.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write