LSTMs and GRUs, built and trained
Build an LSTM and a GRU from a linear layer, train them on the adding problem beside a plain RNN, read what the gates learned, measure the gradient's path, and carry state across a stream.
What it covers6
Why it matters
A plain recurrent network rewrites its whole state at every step. So it forgets. The LSTM keeps a second state, a memory, and changes it only in two gentle ways: it scales it down, and it adds to it. That one idea made learning across long sequences possible. The same idea, a path that information and gradients can travel along unchanged, returns as the residual connection in every transformer. And recurrent models still run wherever data arrives as a stream and memory is tight: sensors, audio on a phone, forecasting.
What you'll be able to do
- Work one LSTM step by hand with one unit: three gates, a candidate, a new memory and a new output
- Build an LSTM cell and a GRU cell from two linear layers, and train them beside a plain RNN
- Read what the gates learned, and measure how far a gradient travels back through each cell
- Check your cell against nn.LSTM and nn.GRU, and carry (h_n, c_n) across pieces of a stream
- Count a cell's parameters and choose between an LSTM and a GRU
Show the answer
Hold the 3rd number unchanged, and ignore what arrives
Its memory has a small, fixed size, so it can't keep everything. It has to write the 3rd number down, protect that note through 27 steps that don't matter, and then add the 31st. Writing on purpose, keeping on purpose and ignoring on purpose are three separate skills. A plain recurrent network has no part for any of them. An LSTM has a gate for each.
The idea: a memory that can stay the same
This lesson rests on Recurrent networks: a state that is carried from step to step, and a gradient that shrinks on the way back. Here is that weakness in one question.
Show the answer
The gradient shrinks at every step back in time
To blame an early step, backpropagation multiplies by one factor for every step in between: the recurrent weights and the slope of tanh. When those factors are below 1, the product shrinks fast, and the early steps get no usable signal. You will measure it below.
The LSTM's answer is a second state. Besides h, which the cell shows to the world, it keeps a memory, c. The memory is changed in two gentle ways only: scaled down, and added to.
A gate is a number between 0 and 1, made by a sigmoid, that multiplies something. At 1 it lets the thing through. At 0 it blocks it. Each gate is a blend weight, like the one in Weighted and moving averages (Foundation lesson 27). The cell has three, and one more number, the candidate g, which is new content made with tanh:
- the forget gate
f: how much of the old memory to keep; - the input gate
i: how much of the candidate to write; - the output gate
o: how much of the memory to show.
In plain words: keep a share of the old memory, add a share of the new content, then show a share of the result.
Show the answer
The forget gate
The forget gate multiplies the old memory. At 1 all of it survives the step, and at 0 none does. The input gate scales what is written, and the output gate scales what is shown.
Take one unit, so every gate is one number. The layer that makes them is a dot product plus a bias, as in The dot product (Foundation lesson 50). Suppose it hands us these four numbers for this step: 0 for the input gate, 2.2 for the forget gate, 0.7 for the candidate and 0 for the output gate. A sigmoid turns 0 into 0.5 and 2.2 into 0.9. A tanh turns 0.7 into 0.6. The old memory is 0.8. Then:
- new memory: 0.9 × 0.8 + 0.5 × 0.6 = 0.72 + 0.3 = 1.02;
- output: 0.5 × tanh(1.02) = 0.5 × 0.77 = 0.38.
With f at 1 and i at 0, the memory passes a step untouched. No matrix and no squashing touch it. The plain RNN had no such path. Step through five steps of one memory and watch the gates decide.
Gates at 1 and 0 hold a memory. A gate at 0.5 halves it.
c, f, i, candidate = 0.8, 0.9, 0.5, 0.6 print(round(f * c + i * candidate, 2))
Show the answer
1.02
The forget gate keeps 0.9 of the old memory: 0.72. The input gate writes half of the candidate: 0.3. The new memory is their sum, 1.02. Memory isn't confined between −1 and 1. It can add up, which is what a task about sums needs.
Write a sentence or two, then compare it with a good answer in the lesson.
From scratch: build the cells and train them
You just did one step with one unit. The code below is the same step, written once. Step through it.
A real cell does this for many units at once, and gets the four numbers from layers: one on the input x and one on the old h. The four parts come out of one layer, and chunk(4) cuts them apart. The cells live in this lesson's setup block, folded above the first cell.
To test memory we need a task that can't be solved without it: the adding problem, from the 1997 paper that introduced the LSTM. Each step gives the network a number between 0 and 1 and a marker. Two steps are marked. The target is the sum of the two marked numbers. The steps between them are noise.
Setup (runs before every cell) · 115 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np import torch import torch.nn as nn def adding(n, steps, rng): """The adding problem. Each step has a number from 0 to 1 and a marker. Two steps are marked. The target is the sum of the two marked numbers.""" values, marks = rng.random((n, steps)), np.zeros((n, steps)) first = rng.integers(0, steps // 2, n) second = rng.integers(steps // 2, steps, n) marks[np.arange(n), first] = 1 marks[np.arange(n), second] = 1 target = values[np.arange(n), first] + values[np.arange(n), second] x = np.stack([values, marks], axis=2).astype(np.float32) return torch.tensor(x), torch.tensor(target.astype(np.float32)) class RNNCell(nn.Module): """The plain step: a squashed mix of the input and the old state.""" n_states = 1 def __init__(self, n_in, n_hidden): super().__init__() self.ih, self.hh = nn.Linear(n_in, n_hidden), nn.Linear(n_hidden, n_hidden) def forward(self, x, state): h = torch.tanh(self.ih(x) + self.hh(state[0])) return h, (h,) class LSTMCell(nn.Module): """Two states: h, what the cell shows, and c, its memory. Gates guard c.""" n_states = 2 def __init__(self, n_in, n_hidden, forget_bias=1.0): super().__init__() self.ih = nn.Linear(n_in, 4 * n_hidden) self.hh = nn.Linear(n_hidden, 4 * n_hidden) with torch.no_grad(): # start with the forget gates mostly open (the parts: i, f, g, o) self.ih.bias[n_hidden:2 * n_hidden] = forget_bias def forward(self, x, state): h, c = state i, f, g, o = (self.ih(x) + self.hh(h)).chunk(4, dim=1) i, f, o = torch.sigmoid(i), torch.sigmoid(f), torch.sigmoid(o) c = f * c + i * torch.tanh(g) # keep a share of c, write a share of g h = o * torch.tanh(c) # show a share of c self.seen = (i, f, o) return h, (h, c) class GRUCell(nn.Module): """One state, two gates. z is the share of the old state that is kept.""" n_states = 1 def __init__(self, n_in, n_hidden): super().__init__() self.ih = nn.Linear(n_in, 3 * n_hidden) self.hh = nn.Linear(n_hidden, 3 * n_hidden) def forward(self, x, state): h = state[0] xr, xz, xn = self.ih(x).chunk(3, dim=1) hr, hz, hn = self.hh(h).chunk(3, dim=1) r, z = torch.sigmoid(xr + hr), torch.sigmoid(xz + hz) n = torch.tanh(xn + r * hn) # r: how much of h the candidate reads h = z * h + (1 - z) * n # keep z of the old state, take 1 - z of n self.seen = (z, r) return h, (h,) class Reader(nn.Module): """Runs a cell along a sequence, then predicts one number from the last state.""" def __init__(self, cell, n_hidden=16): super().__init__() self.cell, self.out = cell, nn.Linear(n_hidden, 1) self.n_hidden, self.n_states = n_hidden, cell.n_states def forward(self, x, state=None, keep=None): # x: (batch, steps, 2) if state is None: state = tuple(torch.zeros(x.shape[0], self.n_hidden) for k in range(self.n_states)) for t in range(x.shape[1]): h, state = self.cell(x[:, t], state) if keep is not None: keep.append(state) self.state = state return self.out(h).squeeze(1) def test_error(model, steps): x, target = adding(1000, steps, np.random.default_rng(99)) with torch.no_grad(): return float(((model(x) - target) ** 2).mean().item()) def train(model, steps, rounds=300, seed=0, curve=None): rng = np.random.default_rng(seed) optimizer = torch.optim.Adam(model.parameters(), lr=0.01) for k in range(1, rounds + 1): x, target = adding(64, steps, rng) loss = ((model(x) - target) ** 2).mean() optimizer.zero_grad() loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) optimizer.step() if curve is not None and k % 25 == 0: curve.append(test_error(model, steps)) return model def gradient_reaching(model, steps=40): """One batch, one backward pass, no training: how strongly does the loss at the end pull on the memory at each step?""" x, target = adding(64, steps, np.random.default_rng(1)) kept = [] prediction = model(x, keep=kept) for state in kept: state[-1].retain_grad() ((prediction - target) ** 2).mean().backward() return [float(np.sqrt((s[-1].grad.numpy() ** 2).sum())) for s in kept] def count(model): return sum(p.numel() for p in model.parameters())
x, target = adding(3, 10, np.random.default_rng(1)) for row in range(3): print("numbers:", " ".join(f"{v:.2f}" for v in x[row, :, 0].tolist())) marks = " ".join(" ^ " if m else " . " for m in x[row, :, 1].tolist()) print("marked: ", marks, "-> target", round(target[row].item(), 2)) x, target = adding(1000, 20, np.random.default_rng(99)) baseline = float(((target - target.mean()) ** 2).mean()) print("error of always answering the average:", round(baseline, 4))
The rows show three examples. The score for knowing nothing, always answering the average, is a squared error of 0.1658. Now train all three cells on sequences 20 steps long, 300 batches each, and plot the error on new sequences as they learn.
Uses the setup above, 115 lines.
import matplotlib.pyplot as plt x, target = adding(1000, 20, np.random.default_rng(99)) baseline = float(((target - target.mean()) ** 2).mean()) makers = {"plain RNN": RNNCell, "LSTM": LSTMCell, "GRU": GRUCell} curves = {} for name, cell in makers.items(): torch.manual_seed(0) curves[name] = [] model = train(Reader(cell(2, 16)), steps=20, curve=curves[name]) print(name.ljust(9), "error on new sequences:", round(test_error(model, 20), 4)) plt.plot(range(25, 301, 25), curves[name], marker="o", label=name) plt.axhline(baseline, color="gray", linestyle="--", label="always the average") plt.xlabel("batches trained") plt.ylabel("squared error on new sequences") plt.legend()
The plain RNN ends at 0.1529, barely under the 0.1658 of knowing nothing. The LSTM ends at 0.0024 and the GRU at 0.001: both solved it. The plain RNN rewrites its whole state at every step, so a number stored early is squashed and mixed many times before the end. The gated cells have a path that skips that.
Be careful what you take from one run, of one size, on one task. Large comparisons across many tasks have found no consistent winner between the LSTM and the GRU. This run shows that a plain RNN is the one that struggles as sequences grow, and that a GRU, being smaller, is a fair first choice when data or compute is short.
Show the answer
Both solved the task, and one run can't rank them
Both errors are far below the 0.1658 of knowing nothing. One run shows what happened once: a different seed or budget could swap them. Learning speed, sensitivity to the starting weights and final accuracy are different things, and broad comparisons across tasks have not found either cell consistently ahead.
What the gates learned
Nobody told the network what its gates were for. The names describe what the equations allow, so we measure. The cell trains the LSTM on 10-step sequences, runs 1,000 new ones, and records the gates at marked steps and at the rest. Then it plots the gates over time for one sequence.
Uses the setup above, 115 lines.
import matplotlib.pyplot as plt torch.manual_seed(0) lstm = train(Reader(LSTMCell(2, 16)), steps=10) x, target = adding(1000, 10, np.random.default_rng(7)) marked = x[:, :, 1].numpy() == 1 # (sequence, step): was it marked? state = (torch.zeros(1000, 16), torch.zeros(1000, 16)) writes, keeps = [], [] with torch.no_grad(): for t in range(10): h, state = lstm.cell(x[:, t], state) i, f, o = lstm.cell.seen writes.append(i.numpy()) keeps.append(f.numpy()) writes, keeps = np.stack(writes, axis=1), np.stack(keeps, axis=1) # (seq, step, unit) print("average over 16 units | at marked steps | at other steps") print("input gate (write) |", f"{writes[marked].mean():15.2f}", "|", f"{writes[~marked].mean():.2f}") print("forget gate (keep) |", f"{keeps[marked].mean():15.2f}", "|", f"{keeps[~marked].mean():.2f}") gap = writes[marked].mean(axis=0) - writes[~marked].mean(axis=0) u = int(np.argmax(gap)) print("unit", u, "reacts most to the marker:") print(" input gate", round(float(writes[marked][:, u].mean()), 2), "at marked steps,", round(float(writes[~marked][:, u].mean()), 2), "otherwise") print(" forget gate", round(float(keeps[marked][:, u].mean()), 2), "at marked steps,", round(float(keeps[~marked][:, u].mean()), 2), "otherwise") row = 0 plt.plot(range(1, 11), keeps[row].mean(axis=1), marker="o", label="forget gate") plt.plot(range(1, 11), writes[row].mean(axis=1), marker="s", label="input gate") for t in np.nonzero(marked[row])[0]: plt.axvline(t + 1, color="gray", linestyle=":") plt.xlabel("step (dotted lines: the two marked steps)") plt.ylabel("average over 16 units") plt.legend()
Averaged over all 16 units, the input gate is at 0.51 on marked steps and 0.44 on the others: a small gap. The forget gate is at 0.66 on marked steps and 0.72 on the others.
The average hides how sharp some units are. Unit 13's input gate is at 0.74 on marked steps and 0.29 otherwise. Its forget gate is at 0.84 on ordinary steps and dips to 0.55 on marked ones: it writes when told to, makes room as it writes, and holds on in between.
Two honest observations. The gates are soft, not clean switches at 0 and 1, and the work is spread across several units. Training found a way to use the gates that works. It was not obliged to find the one in the textbook diagram.
Show the answer
To write mostly when the marker says the number matters
The input gate scales what is written into memory. A gate that opens to 0.74 when the marker is 1 and closes to 0.29 when it is 0 lets marked numbers in and keeps most of the noise out. Nobody programmed that. Gradient descent found that this setting lowered the error.
Why the memory survives training
The LSTM didn't only store a number better. It learned to, so a gradient from the loss at the end reached the early steps. The cell measures that. For each fresh network it runs one batch of 40-step sequences, does one backward pass, and reports the size of the gradient that reaches the state at each step.
Uses the setup above, 115 lines.
print("size of the gradient reaching the state at each step, from a loss at step 40") print(" | step 40 | step 30 | step 20 | step 10 | step 1") makers = (("plain RNN", lambda: Reader(RNNCell(2, 16))), ("LSTM, forget bias 1", lambda: Reader(LSTMCell(2, 16, forget_bias=1.0))), ("LSTM, forget bias 0", lambda: Reader(LSTMCell(2, 16, forget_bias=0.0)))) for name, make in makers: torch.manual_seed(0) g = gradient_reaching(make()) row = " | ".join(f"{g[t - 1]:.1e}" for t in (40, 30, 20, 10, 1)) print(name.ljust(19), "|", row)
For the plain RNN the gradient is 1.6e-01 at step 40 and 4.5e-04 at step 30. At step 1 it is 6.1e-12: ten orders of magnitude smaller. Whatever happened at step 1, the network can't be told.
For the LSTM with a forget bias of 1, the gradient at step 1 is 1.3e-05. That is smaller than at the end, but about two million times what the plain RNN delivers. The third row is the same LSTM with the bias at 0. Its gradient at step 1 is 1.6e-09, about 8,000 times smaller than with the bias at 1.
Going back one step along the memory multiplies the gradient by the forget gate and nothing else. With the bias at 0 the gates start at 0.5, so half the gradient is lost at every step. An LSTM can only learn to remember if it starts out remembering. That is why the cell sets the forget bias to 1. PyTorch's nn.LSTM doesn't.
One LSTM step in NumPy
Write lstm_step(x, h, c, W, U, b) for one sequence (1-D arrays), with hidden size H = len(h). First a = W @ x + U @ h + b, which has length 4H. Its four slices of length H are, in order, the input gate (sigmoid), the forget gate (sigmoid), the candidate (tanh) and the output gate (sigmoid). Return the new (h, c).
Uses the setup above, 115 lines.
import numpy as np def sigmoid(v): return 1 / (1 + np.exp(-v)) def lstm_step(x, h, c, W, U, b): return h, c H = 2 x, h, c = np.array([1.0]), np.array([0.5, -0.5]), np.array([0.8, 0.2]) print(lstm_step(x, h, c, np.zeros((4 * H, 1)), np.zeros((4 * H, H)), np.zeros(4 * H)))
The GRU: the same idea with fewer parts
In 2014 the GRU (gated recurrent unit) asked if an LSTM needs all its parts. It keeps one state and two gates:
- the update gate
z: how much of the old state to keep; - the reset gate
r: how much of the old state the candidate may look at.
Here n is the candidate. Keeping and writing are tied together: whatever share isn't kept is replaced. The path that lets a value through unchanged is still there, with z near 1. There is no separate memory and no output gate, so a GRU has three parts where an LSTM has four. This cell counts what each has to learn.
Uses the setup above, 115 lines.
print("numbers to learn, with 2 inputs and 16 hidden units") print("(this includes the 17 of the output layer, and two bias vectors per part)") for name, cell in (("plain RNN", RNNCell), ("LSTM", LSTMCell), ("GRU", GRUCell)): print(" ", name.ljust(9), count(Reader(cell(2, 16))))
per_part = 16 * (2 + 16) + 2 * 16 print(4 * per_part, 3 * per_part, per_part)
Show the answer
1280 960 320
One part is a layer from the 2 inputs to 16 units and a layer from the 16 hidden values to 16 units, with a bias on each: 16 × 18 + 32 = 320 numbers. An LSTM has four such parts, so 1,280. A GRU has three: 960. Gated cells cost three to four times the parameters of a plain RNN of equal width.
Count a recurrent cell's parameters
Write count_params(kind, n_in, n_hidden) for the cells of this lesson. A part is two layers: one from n_in inputs to n_hidden units, one from n_hidden to n_hidden, with a bias on each (2 * n_hidden biases in all). A "rnn" has 1 part, a "gru" has 3 and an "lstm" has 4. Raise a ValueError for any other kind.
Uses the setup above, 115 lines.
def count_params(kind, n_in, n_hidden): return 0 print(count_params("rnn", 2, 16), count_params("gru", 2, 16), count_params("lstm", 2, 16))
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: nn.LSTM and nn.GRU
You will rarely write a cell by hand. PyTorch has each as one layer that runs the whole sequence, and it must give the same numbers as your cells. The cell below loads your weights into nn.LSTM and nn.GRU and compares.
PyTorch keeps the two layers of a part as weight_ih and weight_hh, and two biases, bias_ih and bias_hh. That is why the cells in this lesson have two layers, ih and hh. With batch_first=True the input has shape (batch, steps, features).
Uses the setup above, 115 lines.
torch.manual_seed(0) x, target = adding(4, 10, np.random.default_rng(5)) pairs = (("LSTM", LSTMCell(2, 16), nn.LSTM(2, 16, batch_first=True)), ("GRU", GRUCell(2, 16), nn.GRU(2, 16, batch_first=True))) for name, mine, real in pairs: with torch.no_grad(): real.weight_ih_l0.copy_(mine.ih.weight) real.weight_hh_l0.copy_(mine.hh.weight) real.bias_ih_l0.copy_(mine.ih.bias) real.bias_hh_l0.copy_(mine.hh.bias) state = tuple(torch.zeros(4, 16) for k in range(mine.n_states)) steps = [] for t in range(10): h, state = mine(x[:, t], state) steps.append(h) mine_out = torch.stack(steps, dim=1) output, last = real(x) h_n = last[0] if name == "LSTM" else last print(name, "output", tuple(output.shape), "h_n", tuple(h_n.shape)) print(" same numbers as my cell:", torch.allclose(output, mine_out, atol=1e-6)) print(" h_n is my last h:", torch.allclose(h_n[0], state[0], atol=1e-6)) print(" parameters:", count(mine), "mine,", sum(p.numel() for p in real.parameters()), "in PyTorch")
Both layers match your cells to within 1e-6, h_n is the last h, and the parameter counts agree: 1,280 for the LSTM and 960 for the GRU, not counting the output layer.
The LSTM returns (output, (h_n, c_n)): the output holds the state at every step, and h_n and c_n hold the final h and c, for each layer. The GRU has no c, so it returns (output, h_n).
Worked example: a stream arrives in pieces
A recurrent model's whole memory of the past is its state: here 16 numbers for each sequence, however long the past is. So a long input can be fed in pieces, as it arrives, with the state handed from one call to the next. The cell trains a GRU on 20-step sequences, then runs 500 new ones three ways.
Uses the setup above, 115 lines.
torch.manual_seed(0) gru = train(Reader(GRUCell(2, 16)), steps=20) x, target = adding(500, 20, np.random.default_rng(42)) with torch.no_grad(): whole = gru(x) # all 20 steps in one call state = None for start in range(0, 20, 5): # four pieces of 5 steps carried = gru(x[:, start:start + 5], state) state = gru.state # hand the state on to the next piece for start in range(0, 20, 5): forgetful = gru(x[:, start:start + 5]) # the bug: every piece starts at zero def error(prediction): return round(float(((prediction - target) ** 2).mean().item()), 4) print("all 20 steps at once: error", error(whole)) print("in 4 pieces, state carried over: error", error(carried)) print(" largest difference from the first way:", float((carried - whole).abs().max())) print("in 4 pieces, state reset: error", error(forgetful))
Feeding all 20 steps at once gives an error of 0.0011. Feeding four pieces of 5 steps, and passing the state from each call into the next, gives 0.0011 again, and the largest difference between the two sets of predictions is exactly 0.0: the model can't tell how its input was cut up. The third way starts every piece from zeros, a bug that is easy to write. Its error is 0.1844, slightly worse than the 0.1658 of answering the average, since the two marked numbers never fall in the same piece. The memory that holds them is the state, and it was thrown away.
Show the answer
Its final state from the previous piece
The state summarizes everything the model has kept from the past, in a fixed number of values. Starting the next piece from that state gives the result of one long run, as the worked example showed. No earlier audio needs to be stored, so the memory needed doesn't grow with the length of the stream. With nn.LSTM that state is the pair (h_n, c_n).
On the job
A few things to know when a recurrent layer shows up in a real model:
- Clip gradients. Gates cure vanishing gradients, not exploding ones. The training loop in this lesson calls
clip_grad_norm_at every step, and recurrent training usually should. - Padding. Sequences in a batch have different lengths, so the short ones are padded. Read each row's state at its own last real step, not at the last column. The exercise below does this.
- Both directions.
bidirectional=Trueruns a second cell from the end to the start. It helps when the whole sequence is available, as in tagging, and it can't be used on a live stream. - Where they live. Transformers replaced LSTMs for language, since attention reaches any earlier step directly. Gated cells remain a sound choice for streams, small devices and time series. Their central idea, an additive path that leaves information alone, is the one residual connections carry into every transformer.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 7 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 14 review cards for spaced repetition
Try a quiz question
What lets an LSTM's memory survive many steps unchanged?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 4 · about 24 hours of study
Deep learning
Neural networks, backprop and training them well
31 lessons 1 free
- The artificial neuron FreeWeighted sum plus a nonlinearity.35 min
- Activation functionsReLU, sigmoid, GELU, and why nonlinearity matters.35 min
- Multi-layer networksStack layers to learn complex functions.35 min
- The forward passFollow numbers through a network to a prediction.36 min
- Computation graphsEvery calculation as a graph of small operations.40 min
- BackpropagationThe chain rule applied backwards to get every gradient.42 min
- Build an autograd engineWrite backprop yourself in ~100 lines (micrograd).42 min
- PyTorch tensorsNumPy-like arrays that run on GPUs.38 min
- PyTorch autograd and modulesLet PyTorch compute gradients; build models with nn.Module.42 min
- The training loopForward, loss, backward, step, repeat.40 min
- Mini-batches and SGDTrain on small random batches instead of everything at once.40 min
- OptimizersMomentum and Adam.41 min
- Learning-rate schedules in PyTorchWarmup then cosine by hand, as a function of the step, then LambdaLR and where scheduler.step() goes; digits trained with and without warmup.25 min
- Vanishing and exploding gradientsWhy deep networks were hard to train.40 min
- Weight initializationStart weights at the right scale.33 min
- Normalization layersBatch and layer norm keep training stable.40 min
- Residual connectionsSkip connections that let gradients flow.40 min
- DropoutRandomly switch off neurons to prevent overfitting.35 min
- Debugging trainingRead a failing run from its numbers, and fix what's wrong.40 min
- Looking inside a networkRead four numbers per layer (spread, dead units, gradient norm, update-to-weight ratio), collect them in NumPy, then with forward and tensor hooks, and diagnose three planted faults.30 min
- Transfer learningReuse a trained network: freeze the backbone, train a new head, fine-tune with two learning rates.42 min
- Convolutional networksPattern detectors for images.44 min
- Embedding layersLearned lookup tables that turn tokens into vectors.38 min
- Recurrent networksProcess sequences step by step, and their limits.43 min
- LSTMs and GRUs, built and trainedBuild an LSTM and a GRU from a linear layer, train them on the adding problem beside a plain RNN, read what the gates learned, measure the gradient's path, and carry state across a stream.40 min
- Sequence to sequence, and the invention of attentionBuild an encoder and decoder, measure the bottleneck of a fixed-size handoff, add attention in three lines, read an attention map with care, and see how one mistake compounds after teacher forcing.45 min
- AutoencodersEncoder, bottleneck and decoder, reconstruction error for anomalies, denoising, and the latent space image generators use.42 min
- GPUs and computeWhy deep learning runs on graphics cards.40 min
- Training at scale: precision, accumulation, data and many GPUsGradient accumulation and what it really saves, scaling the learning rate with the batch, mixed precision and loss scaling, a data loader that starves the GPU, and why eight GPUs aren't eight times as fast.45 min
- A training script, start to finishConfig, seed, data, model, schedule, loop and checkpoint as one script: what a checkpoint must hold, a resume that matches an unbroken run, and buffers, collate and torch.save in PyTorch.40 min
- Capstone: train a seam-tear spotterTrain and compare three models on fabric images, choose the alarm level from costs, and score once.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write