Index Ai Academy
Phase 4 · Architectures

LSTMs and GRUs, built and trained

Build an LSTM and a GRU from a linear layer, train them on the adding problem beside a plain RNN, read what the gates learned, measure the gradient's path, and carry state across a stream.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

A plain recurrent network rewrites its whole state at every step. So it forgets. The LSTM keeps a second state, a memory, and changes it only in two gentle ways: it scales it down, and it adds to it. That one idea made learning across long sequences possible. The same idea, a path that information and gradients can travel along unchanged, returns as the residual connection in every transformer. And recurrent models still run wherever data arrives as a stream and memory is tight: sensors, audio on a phone, forecasting.

What you'll be able to do

  • Work one LSTM step by hand with one unit: three gates, a candidate, a new memory and a new output
  • Build an LSTM cell and a GRU cell from two linear layers, and train them beside a plain RNN
  • Read what the gates learned, and measure how far a gradient travels back through each cell
  • Check your cell against nn.LSTM and nn.GRU, and carry (h_n, c_n) across pieces of a stream
  • Count a cell's parameters and choose between an LSTM and a GRU
Quick check
A guess before you start. A network reads 40 numbers, one at a time. At the end it must output the sum of the 3rd and the 31st. What must its memory do during the 27 steps in between?
  1. Replace its memory with each new number
  2. Average all the numbers it has seen so far
  3. Store all 40 numbers and add two at the end
  4. Hold the 3rd number unchanged, and ignore what arrives
Show the answer

Hold the 3rd number unchanged, and ignore what arrives

Its memory has a small, fixed size, so it can't keep everything. It has to write the 3rd number down, protect that note through 27 steps that don't matter, and then add the 31st. Writing on purpose, keeping on purpose and ignoring on purpose are three separate skills. A plain recurrent network has no part for any of them. An LSTM has a gate for each.

The idea: a memory that can stay the same

This lesson rests on Recurrent networks: a state that is carried from step to step, and a gradient that shrinks on the way back. Here is that weakness in one question.

Quick check
Why does a plain RNN struggle to learn from something that happened many steps ago?
  1. Its learning rate falls as the sequence gets longer
  2. It reads the sequence in the wrong order
  3. It has too few weights to store a long sequence
  4. The gradient shrinks at every step back in time
Show the answer

The gradient shrinks at every step back in time

To blame an early step, backpropagation multiplies by one factor for every step in between: the recurrent weights and the slope of tanh. When those factors are below 1, the product shrinks fast, and the early steps get no usable signal. You will measure it below.

The LSTM's answer is a second state. Besides h, which the cell shows to the world, it keeps a memory, c. The memory is changed in two gentle ways only: scaled down, and added to.

A gate is a number between 0 and 1, made by a sigmoid, that multiplies something. At 1 it lets the thing through. At 0 it blocks it. Each gate is a blend weight, like the one in Weighted and moving averages (Foundation lesson 27). The cell has three, and one more number, the candidate g, which is new content made with tanh:

  • the forget gate f: how much of the old memory to keep;
  • the input gate i: how much of the candidate to write;
  • the output gate o: how much of the memory to show.
cnew=f×c+i×gh=o×tanh⁡(cnew)c_{\text{new}} = f \times c + i \times g \qquad h = o \times \tanh(c_{\text{new}})

In plain words: keep a share of the old memory, add a share of the new content, then show a share of the result.

Quick check
Which gate decides how much of the old memory survives a step?
  1. The input gate
  2. The forget gate
  3. The candidate
  4. The output gate
Show the answer

The forget gate

The forget gate multiplies the old memory. At 1 all of it survives the step, and at 0 none does. The input gate scales what is written, and the output gate scales what is shown.

Take one unit, so every gate is one number. The layer that makes them is a dot product plus a bias, as in The dot product (Foundation lesson 50). Suppose it hands us these four numbers for this step: 0 for the input gate, 2.2 for the forget gate, 0.7 for the candidate and 0 for the output gate. A sigmoid turns 0 into 0.5 and 2.2 into 0.9. A tanh turns 0.7 into 0.6. The old memory is 0.8. Then:

  • new memory: 0.9 × 0.8 + 0.5 × 0.6 = 0.72 + 0.3 = 1.02;
  • output: 0.5 × tanh(1.02) = 0.5 × 0.77 = 0.38.

With f at 1 and i at 0, the memory passes a step untouched. No matrix and no squashing touch it. The plain RNN had no such path. Step through five steps of one memory and watch the gates decide.

Try it now

Gates at 1 and 0 hold a memory. A gate at 0.5 halves it.

Quick check
A memory unit holds 0.8. At this step the forget gate is 0.9, the input gate is 0.5 and the candidate is 0.6. What does this print: the new memory?
c, f, i, candidate = 0.8, 0.9, 0.5, 0.6
print(round(f * c + i * candidate, 2))
  1. 0.6
  2. 1.02
  3. 0.72
  4. 0.3
Show the answer

1.02

The forget gate keeps 0.9 of the old memory: 0.72. The input gate writes half of the candidate: 0.3. The new memory is their sum, 1.02. Memory isn't confined between −1 and 1. It can add up, which is what a task about sums needs.

In your own words
In your own words: what can the LSTM's memory do that a plain RNN's state can't?

Write a sentence or two, then compare it with a good answer in the lesson.

From scratch: build the cells and train them

You just did one step with one unit. The code below is the same step, written once. Step through it.

A real cell does this for many units at once, and gets the four numbers from layers: one on the input x and one on the old h. The four parts come out of one layer, and chunk(4) cuts them apart. The cells live in this lesson's setup block, folded above the first cell.

To test memory we need a task that can't be solved without it: the adding problem, from the 1997 paper that introduced the LSTM. Each step gives the network a number between 0 and 1 and a marker. Two steps are marked. The target is the sum of the two marked numbers. The steps between them are noise.

Setup (runs before every cell) · 115 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np
import torch
import torch.nn as nn

def adding(n, steps, rng):
    """The adding problem. Each step has a number from 0 to 1 and a marker.
    Two steps are marked. The target is the sum of the two marked numbers."""
    values, marks = rng.random((n, steps)), np.zeros((n, steps))
    first = rng.integers(0, steps // 2, n)
    second = rng.integers(steps // 2, steps, n)
    marks[np.arange(n), first] = 1
    marks[np.arange(n), second] = 1
    target = values[np.arange(n), first] + values[np.arange(n), second]
    x = np.stack([values, marks], axis=2).astype(np.float32)
    return torch.tensor(x), torch.tensor(target.astype(np.float32))

class RNNCell(nn.Module):
    """The plain step: a squashed mix of the input and the old state."""
    n_states = 1
    def __init__(self, n_in, n_hidden):
        super().__init__()
        self.ih, self.hh = nn.Linear(n_in, n_hidden), nn.Linear(n_hidden, n_hidden)

    def forward(self, x, state):
        h = torch.tanh(self.ih(x) + self.hh(state[0]))
        return h, (h,)

class LSTMCell(nn.Module):
    """Two states: h, what the cell shows, and c, its memory. Gates guard c."""
    n_states = 2
    def __init__(self, n_in, n_hidden, forget_bias=1.0):
        super().__init__()
        self.ih = nn.Linear(n_in, 4 * n_hidden)
        self.hh = nn.Linear(n_hidden, 4 * n_hidden)
        with torch.no_grad():
            # start with the forget gates mostly open (the parts: i, f, g, o)
            self.ih.bias[n_hidden:2 * n_hidden] = forget_bias

    def forward(self, x, state):
        h, c = state
        i, f, g, o = (self.ih(x) + self.hh(h)).chunk(4, dim=1)
        i, f, o = torch.sigmoid(i), torch.sigmoid(f), torch.sigmoid(o)
        c = f * c + i * torch.tanh(g)         # keep a share of c, write a share of g
        h = o * torch.tanh(c)                 # show a share of c
        self.seen = (i, f, o)
        return h, (h, c)

class GRUCell(nn.Module):
    """One state, two gates. z is the share of the old state that is kept."""
    n_states = 1
    def __init__(self, n_in, n_hidden):
        super().__init__()
        self.ih = nn.Linear(n_in, 3 * n_hidden)
        self.hh = nn.Linear(n_hidden, 3 * n_hidden)

    def forward(self, x, state):
        h = state[0]
        xr, xz, xn = self.ih(x).chunk(3, dim=1)
        hr, hz, hn = self.hh(h).chunk(3, dim=1)
        r, z = torch.sigmoid(xr + hr), torch.sigmoid(xz + hz)
        n = torch.tanh(xn + r * hn)           # r: how much of h the candidate reads
        h = z * h + (1 - z) * n               # keep z of the old state, take 1 - z of n
        self.seen = (z, r)
        return h, (h,)

class Reader(nn.Module):
    """Runs a cell along a sequence, then predicts one number from the last state."""
    def __init__(self, cell, n_hidden=16):
        super().__init__()
        self.cell, self.out = cell, nn.Linear(n_hidden, 1)
        self.n_hidden, self.n_states = n_hidden, cell.n_states

    def forward(self, x, state=None, keep=None):          # x: (batch, steps, 2)
        if state is None:
            state = tuple(torch.zeros(x.shape[0], self.n_hidden)
                          for k in range(self.n_states))
        for t in range(x.shape[1]):
            h, state = self.cell(x[:, t], state)
            if keep is not None:
                keep.append(state)
        self.state = state
        return self.out(h).squeeze(1)

def test_error(model, steps):
    x, target = adding(1000, steps, np.random.default_rng(99))
    with torch.no_grad():
        return float(((model(x) - target) ** 2).mean().item())

def train(model, steps, rounds=300, seed=0, curve=None):
    rng = np.random.default_rng(seed)
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
    for k in range(1, rounds + 1):
        x, target = adding(64, steps, rng)
        loss = ((model(x) - target) ** 2).mean()
        optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
        if curve is not None and k % 25 == 0:
            curve.append(test_error(model, steps))
    return model

def gradient_reaching(model, steps=40):
    """One batch, one backward pass, no training: how strongly does the loss at the
    end pull on the memory at each step?"""
    x, target = adding(64, steps, np.random.default_rng(1))
    kept = []
    prediction = model(x, keep=kept)
    for state in kept:
        state[-1].retain_grad()
    ((prediction - target) ** 2).mean().backward()
    return [float(np.sqrt((s[-1].grad.numpy() ** 2).sum())) for s in kept]

def count(model):
    return sum(p.numel() for p in model.parameters())
x, target = adding(3, 10, np.random.default_rng(1))
for row in range(3):
    print("numbers:", " ".join(f"{v:.2f}" for v in x[row, :, 0].tolist()))
    marks = " ".join("  ^ " if m else "  . " for m in x[row, :, 1].tolist())
    print("marked: ", marks, "-> target", round(target[row].item(), 2))

x, target = adding(1000, 20, np.random.default_rng(99))
baseline = float(((target - target.mean()) ** 2).mean())
print("error of always answering the average:", round(baseline, 4))

The rows show three examples. The score for knowing nothing, always answering the average, is a squared error of 0.1658. Now train all three cells on sequences 20 steps long, 300 batches each, and plot the error on new sequences as they learn.

Uses the setup above, 115 lines.

import matplotlib.pyplot as plt

x, target = adding(1000, 20, np.random.default_rng(99))
baseline = float(((target - target.mean()) ** 2).mean())
makers = {"plain RNN": RNNCell, "LSTM": LSTMCell, "GRU": GRUCell}
curves = {}
for name, cell in makers.items():
    torch.manual_seed(0)
    curves[name] = []
    model = train(Reader(cell(2, 16)), steps=20, curve=curves[name])
    print(name.ljust(9), "error on new sequences:", round(test_error(model, 20), 4))
    plt.plot(range(25, 301, 25), curves[name], marker="o", label=name)
plt.axhline(baseline, color="gray", linestyle="--", label="always the average")
plt.xlabel("batches trained")
plt.ylabel("squared error on new sequences")
plt.legend()

The plain RNN ends at 0.1529, barely under the 0.1658 of knowing nothing. The LSTM ends at 0.0024 and the GRU at 0.001: both solved it. The plain RNN rewrites its whole state at every step, so a number stored early is squashed and mixed many times before the end. The gated cells have a path that skips that.

Be careful what you take from one run, of one size, on one task. Large comparisons across many tasks have found no consistent winner between the LSTM and the GRU. This run shows that a plain RNN is the one that struggles as sequences grow, and that a GRU, being smaller, is a fair first choice when data or compute is short.

Quick check
In this run the GRU ends at an error of 0.001 and the LSTM at 0.0024. What is the soundest conclusion?
  1. GRUs are more accurate than LSTMs in general
  2. Both solved the task, and one run can't rank them
  3. The GRU learns better because it has more parameters
  4. The LSTM failed to learn the task
Show the answer

Both solved the task, and one run can't rank them

Both errors are far below the 0.1658 of knowing nothing. One run shows what happened once: a different seed or budget could swap them. Learning speed, sensitivity to the starting weights and final accuracy are different things, and broad comparisons across tasks have not found either cell consistently ahead.

What the gates learned

Nobody told the network what its gates were for. The names describe what the equations allow, so we measure. The cell trains the LSTM on 10-step sequences, runs 1,000 new ones, and records the gates at marked steps and at the rest. Then it plots the gates over time for one sequence.

Uses the setup above, 115 lines.

import matplotlib.pyplot as plt

torch.manual_seed(0)
lstm = train(Reader(LSTMCell(2, 16)), steps=10)

x, target = adding(1000, 10, np.random.default_rng(7))
marked = x[:, :, 1].numpy() == 1                 # (sequence, step): was it marked?
state = (torch.zeros(1000, 16), torch.zeros(1000, 16))
writes, keeps = [], []
with torch.no_grad():
    for t in range(10):
        h, state = lstm.cell(x[:, t], state)
        i, f, o = lstm.cell.seen
        writes.append(i.numpy())
        keeps.append(f.numpy())
writes, keeps = np.stack(writes, axis=1), np.stack(keeps, axis=1)   # (seq, step, unit)

print("average over 16 units  | at marked steps | at other steps")
print("input gate (write)     |", f"{writes[marked].mean():15.2f}", "|",
      f"{writes[~marked].mean():.2f}")
print("forget gate (keep)     |", f"{keeps[marked].mean():15.2f}", "|",
      f"{keeps[~marked].mean():.2f}")
gap = writes[marked].mean(axis=0) - writes[~marked].mean(axis=0)
u = int(np.argmax(gap))
print("unit", u, "reacts most to the marker:")
print("  input gate", round(float(writes[marked][:, u].mean()), 2), "at marked steps,",
      round(float(writes[~marked][:, u].mean()), 2), "otherwise")
print("  forget gate", round(float(keeps[marked][:, u].mean()), 2), "at marked steps,",
      round(float(keeps[~marked][:, u].mean()), 2), "otherwise")

row = 0
plt.plot(range(1, 11), keeps[row].mean(axis=1), marker="o", label="forget gate")
plt.plot(range(1, 11), writes[row].mean(axis=1), marker="s", label="input gate")
for t in np.nonzero(marked[row])[0]:
    plt.axvline(t + 1, color="gray", linestyle=":")
plt.xlabel("step (dotted lines: the two marked steps)")
plt.ylabel("average over 16 units")
plt.legend()

Averaged over all 16 units, the input gate is at 0.51 on marked steps and 0.44 on the others: a small gap. The forget gate is at 0.66 on marked steps and 0.72 on the others.

The average hides how sharp some units are. Unit 13's input gate is at 0.74 on marked steps and 0.29 otherwise. Its forget gate is at 0.84 on ordinary steps and dips to 0.55 on marked ones: it writes when told to, makes room as it writes, and holds on in between.

Two honest observations. The gates are soft, not clean switches at 0 and 1, and the work is spread across several units. Training found a way to use the gates that works. It was not obliged to find the one in the textbook diagram.

Quick check
In a trained LSTM, one unit's input gate averages 0.74 on marked steps and 0.29 on unmarked steps. What has that unit learned?
  1. To write mostly when the marker says the number matters
  2. To forget its memory whenever a marker appears
  3. To copy every input number into its memory
  4. To ignore the marker and use the numbers alone
Show the answer

To write mostly when the marker says the number matters

The input gate scales what is written into memory. A gate that opens to 0.74 when the marker is 1 and closes to 0.29 when it is 0 lets marked numbers in and keeps most of the noise out. Nobody programmed that. Gradient descent found that this setting lowered the error.

Why the memory survives training

The LSTM didn't only store a number better. It learned to, so a gradient from the loss at the end reached the early steps. The cell measures that. For each fresh network it runs one batch of 40-step sequences, does one backward pass, and reports the size of the gradient that reaches the state at each step.

Uses the setup above, 115 lines.

print("size of the gradient reaching the state at each step, from a loss at step 40")
print("                    | step 40 | step 30 | step 20 | step 10 | step 1")
makers = (("plain RNN", lambda: Reader(RNNCell(2, 16))),
          ("LSTM, forget bias 1", lambda: Reader(LSTMCell(2, 16, forget_bias=1.0))),
          ("LSTM, forget bias 0", lambda: Reader(LSTMCell(2, 16, forget_bias=0.0))))
for name, make in makers:
    torch.manual_seed(0)
    g = gradient_reaching(make())
    row = " | ".join(f"{g[t - 1]:.1e}" for t in (40, 30, 20, 10, 1))
    print(name.ljust(19), "|", row)

For the plain RNN the gradient is 1.6e-01 at step 40 and 4.5e-04 at step 30. At step 1 it is 6.1e-12: ten orders of magnitude smaller. Whatever happened at step 1, the network can't be told.

For the LSTM with a forget bias of 1, the gradient at step 1 is 1.3e-05. That is smaller than at the end, but about two million times what the plain RNN delivers. The third row is the same LSTM with the bias at 0. Its gradient at step 1 is 1.6e-09, about 8,000 times smaller than with the bias at 1.

Going back one step along the memory multiplies the gradient by the forget gate and nothing else. With the bias at 0 the gates start at 0.5, so half the gradient is lost at every step. An LSTM can only learn to remember if it starts out remembering. That is why the cell sets the forget bias to 1. PyTorch's nn.LSTM doesn't.

Drill

One LSTM step in NumPy

Write lstm_step(x, h, c, W, U, b) for one sequence (1-D arrays), with hidden size H = len(h). First a = W @ x + U @ h + b, which has length 4H. Its four slices of length H are, in order, the input gate (sigmoid), the forget gate (sigmoid), the candidate (tanh) and the output gate (sigmoid). Return the new (h, c).

Uses the setup above, 115 lines.

import numpy as np

def sigmoid(v):
    return 1 / (1 + np.exp(-v))

def lstm_step(x, h, c, W, U, b):
    return h, c

H = 2
x, h, c = np.array([1.0]), np.array([0.5, -0.5]), np.array([0.8, 0.2])
print(lstm_step(x, h, c, np.zeros((4 * H, 1)), np.zeros((4 * H, H)), np.zeros(4 * H)))
In the lesson, your code is checked as you write it.

The GRU: the same idea with fewer parts

In 2014 the GRU (gated recurrent unit) asked if an LSTM needs all its parts. It keeps one state and two gates:

  • the update gate z: how much of the old state to keep;
  • the reset gate r: how much of the old state the candidate may look at.
hnew=z×h+(1−z)×nh_{\text{new}} = z \times h + (1 - z) \times n

Here n is the candidate. Keeping and writing are tied together: whatever share isn't kept is replaced. The path that lets a value through unchanged is still there, with z near 1. There is no separate memory and no output gate, so a GRU has three parts where an LSTM has four. This cell counts what each has to learn.

Uses the setup above, 115 lines.

print("numbers to learn, with 2 inputs and 16 hidden units")
print("(this includes the 17 of the output layer, and two bias vectors per part)")
for name, cell in (("plain RNN", RNNCell), ("LSTM", LSTMCell), ("GRU", GRUCell)):
    print(" ", name.ljust(9), count(Reader(cell(2, 16))))
Quick check
A plain RNN cell and an LSTM cell each have 2 inputs and 16 hidden units. Each part has a weight for every input and every hidden unit, and two biases, for each of its 16 units. What does this print: the parameter counts for an LSTM (4 parts), a GRU (3 parts) and a plain RNN (1 part), not counting the output layer?
per_part = 16 * (2 + 16) + 2 * 16
print(4 * per_part, 3 * per_part, per_part)
  1. 64 48 16
  2. 1216 912 304
  3. 320 320 320
  4. 1280 960 320
Show the answer

1280 960 320

One part is a layer from the 2 inputs to 16 units and a layer from the 16 hidden values to 16 units, with a bias on each: 16 × 18 + 32 = 320 numbers. An LSTM has four such parts, so 1,280. A GRU has three: 960. Gated cells cost three to four times the parameters of a plain RNN of equal width.

Drill

Count a recurrent cell's parameters

Write count_params(kind, n_in, n_hidden) for the cells of this lesson. A part is two layers: one from n_in inputs to n_hidden units, one from n_hidden to n_hidden, with a bias on each (2 * n_hidden biases in all). A "rnn" has 1 part, a "gru" has 3 and an "lstm" has 4. Raise a ValueError for any other kind.

Uses the setup above, 115 lines.

def count_params(kind, n_in, n_hidden):
    return 0
print(count_params("rnn", 2, 16), count_params("gru", 2, 16),
      count_params("lstm", 2, 16))
In the lesson, your code is checked as you write it.
In your own words
What did the GRU give up compared with the LSTM, and what did it keep?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: nn.LSTM and nn.GRU

You will rarely write a cell by hand. PyTorch has each as one layer that runs the whole sequence, and it must give the same numbers as your cells. The cell below loads your weights into nn.LSTM and nn.GRU and compares.

PyTorch keeps the two layers of a part as weight_ih and weight_hh, and two biases, bias_ih and bias_hh. That is why the cells in this lesson have two layers, ih and hh. With batch_first=True the input has shape (batch, steps, features).

Uses the setup above, 115 lines.

torch.manual_seed(0)
x, target = adding(4, 10, np.random.default_rng(5))
pairs = (("LSTM", LSTMCell(2, 16), nn.LSTM(2, 16, batch_first=True)),
         ("GRU", GRUCell(2, 16), nn.GRU(2, 16, batch_first=True)))
for name, mine, real in pairs:
    with torch.no_grad():
        real.weight_ih_l0.copy_(mine.ih.weight)
        real.weight_hh_l0.copy_(mine.hh.weight)
        real.bias_ih_l0.copy_(mine.ih.bias)
        real.bias_hh_l0.copy_(mine.hh.bias)
    state = tuple(torch.zeros(4, 16) for k in range(mine.n_states))
    steps = []
    for t in range(10):
        h, state = mine(x[:, t], state)
        steps.append(h)
    mine_out = torch.stack(steps, dim=1)
    output, last = real(x)
    h_n = last[0] if name == "LSTM" else last
    print(name, "output", tuple(output.shape), "h_n", tuple(h_n.shape))
    print("   same numbers as my cell:", torch.allclose(output, mine_out, atol=1e-6))
    print("   h_n is my last h:", torch.allclose(h_n[0], state[0], atol=1e-6))
    print("   parameters:", count(mine), "mine,",
          sum(p.numel() for p in real.parameters()), "in PyTorch")

Both layers match your cells to within 1e-6, h_n is the last h, and the parameter counts agree: 1,280 for the LSTM and 960 for the GRU, not counting the output layer.

The LSTM returns (output, (h_n, c_n)): the output holds the state at every step, and h_n and c_n hold the final h and c, for each layer. The GRU has no c, so it returns (output, h_n).

Worked example

Worked example: a stream arrives in pieces

A recurrent model's whole memory of the past is its state: here 16 numbers for each sequence, however long the past is. So a long input can be fed in pieces, as it arrives, with the state handed from one call to the next. The cell trains a GRU on 20-step sequences, then runs 500 new ones three ways.

Uses the setup above, 115 lines.

torch.manual_seed(0)
gru = train(Reader(GRUCell(2, 16)), steps=20)
x, target = adding(500, 20, np.random.default_rng(42))
with torch.no_grad():
    whole = gru(x)                        # all 20 steps in one call
    state = None
    for start in range(0, 20, 5):         # four pieces of 5 steps
        carried = gru(x[:, start:start + 5], state)
        state = gru.state                 # hand the state on to the next piece
    for start in range(0, 20, 5):
        forgetful = gru(x[:, start:start + 5])   # the bug: every piece starts at zero

def error(prediction):
    return round(float(((prediction - target) ** 2).mean().item()), 4)

print("all 20 steps at once:            error", error(whole))
print("in 4 pieces, state carried over: error", error(carried))
print("   largest difference from the first way:", float((carried - whole).abs().max()))
print("in 4 pieces, state reset:        error", error(forgetful))
In the lesson you can change this and run it.

Feeding all 20 steps at once gives an error of 0.0011. Feeding four pieces of 5 steps, and passing the state from each call into the next, gives 0.0011 again, and the largest difference between the two sets of predictions is exactly 0.0: the model can't tell how its input was cut up. The third way starts every piece from zeros, a bug that is easy to write. Its error is 0.1844, slightly worse than the 0.1658 of answering the average, since the two marked numbers never fall in the same piece. The memory that holds them is the state, and it was thrown away.

Quick check
A recurrent model processes a live audio stream in pieces of 100 milliseconds. What must be passed from one piece to the next for it to behave as if it had heard the stream whole?
  1. All of the audio received so far
  2. The weights it learned on the last piece
  3. Its final state from the previous piece
  4. Nothing: each piece can be handled separately
Show the answer

Its final state from the previous piece

The state summarizes everything the model has kept from the past, in a fixed number of values. Starting the next piece from that state gives the result of one long run, as the worked example showed. No earlier audio needs to be stored, so the memory needed doesn't grow with the length of the stream. With nn.LSTM that state is the pair (h_n, c_n).

On the job

A few things to know when a recurrent layer shows up in a real model:

  • Clip gradients. Gates cure vanishing gradients, not exploding ones. The training loop in this lesson calls clip_grad_norm_ at every step, and recurrent training usually should.
  • Padding. Sequences in a batch have different lengths, so the short ones are padded. Read each row's state at its own last real step, not at the last column. The exercise below does this.
  • Both directions. bidirectional=True runs a second cell from the end to the start. It helps when the whole sequence is available, as in tagging, and it can't be used on a live stream.
  • Where they live. Transformers replaced LSTMs for language, since attention reaches any earlier step directly. Gated cells remain a sound choice for streams, small devices and time series. Their central idea, an additive path that leaves information alone, is the one residual connections carry into every transformer.
In your own words
A colleague asks why anyone would still use a GRU when transformers exist. Give two situations where you would, and one where you wouldn't.

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 7 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 14 review cards for spaced repetition

Try a quiz question

What lets an LSTM's memory survive many steps unchanged?

  1. It is only scaled by a gate and added to, never remixed
  2. It is stored in a separate file between steps
  3. It is passed through tanh twice, which sharpens it
  4. It has more hidden units than a plain RNN of equal size

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 4 · about 24 hours of study

Deep learning

Neural networks, backprop and training them well 31 lessons 1 free
  1. The artificial neuron FreeWeighted sum plus a nonlinearity.35 min
  2. Activation functionsReLU, sigmoid, GELU, and why nonlinearity matters.35 min
  3. Multi-layer networksStack layers to learn complex functions.35 min
  4. The forward passFollow numbers through a network to a prediction.36 min
  5. Computation graphsEvery calculation as a graph of small operations.40 min
  6. BackpropagationThe chain rule applied backwards to get every gradient.42 min
  7. Build an autograd engineWrite backprop yourself in ~100 lines (micrograd).42 min
  8. PyTorch tensorsNumPy-like arrays that run on GPUs.38 min
  9. PyTorch autograd and modulesLet PyTorch compute gradients; build models with nn.Module.42 min
  10. The training loopForward, loss, backward, step, repeat.40 min
  11. Mini-batches and SGDTrain on small random batches instead of everything at once.40 min
  12. OptimizersMomentum and Adam.41 min
  13. Learning-rate schedules in PyTorchWarmup then cosine by hand, as a function of the step, then LambdaLR and where scheduler.step() goes; digits trained with and without warmup.25 min
  14. Vanishing and exploding gradientsWhy deep networks were hard to train.40 min
  15. Weight initializationStart weights at the right scale.33 min
  16. Normalization layersBatch and layer norm keep training stable.40 min
  17. Residual connectionsSkip connections that let gradients flow.40 min
  18. DropoutRandomly switch off neurons to prevent overfitting.35 min
  19. Debugging trainingRead a failing run from its numbers, and fix what's wrong.40 min
  20. Looking inside a networkRead four numbers per layer (spread, dead units, gradient norm, update-to-weight ratio), collect them in NumPy, then with forward and tensor hooks, and diagnose three planted faults.30 min
  21. Transfer learningReuse a trained network: freeze the backbone, train a new head, fine-tune with two learning rates.42 min
  22. Convolutional networksPattern detectors for images.44 min
  23. Embedding layersLearned lookup tables that turn tokens into vectors.38 min
  24. Recurrent networksProcess sequences step by step, and their limits.43 min
  25. LSTMs and GRUs, built and trainedBuild an LSTM and a GRU from a linear layer, train them on the adding problem beside a plain RNN, read what the gates learned, measure the gradient's path, and carry state across a stream.40 min
  26. Sequence to sequence, and the invention of attentionBuild an encoder and decoder, measure the bottleneck of a fixed-size handoff, add attention in three lines, read an attention map with care, and see how one mistake compounds after teacher forcing.45 min
  27. AutoencodersEncoder, bottleneck and decoder, reconstruction error for anomalies, denoising, and the latent space image generators use.42 min
  28. GPUs and computeWhy deep learning runs on graphics cards.40 min
  29. Training at scale: precision, accumulation, data and many GPUsGradient accumulation and what it really saves, scaling the learning rate with the batch, mixed precision and loss scaling, a data loader that starves the GPU, and why eight GPUs aren't eight times as fast.45 min
  30. A training script, start to finishConfig, seed, data, model, schedule, loop and checkpoint as one script: what a checkpoint must hold, a resume that matches an unbroken run, and buffers, collate and torch.save in PyTorch.40 min
  31. Capstone: train a seam-tear spotterTrain and compare three models on fabric images, choose the alarm level from costs, and score once.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write