Training at scale: precision, accumulation, data and many GPUs
Gradient accumulation and what it really saves, scaling the learning rate with the batch, mixed precision and loss scaling, a data loader that starves the GPU, and why eight GPUs aren't eight times as fast.
What it covers6
Why it matters
A training loop that works on a laptop does not simply run faster on bigger hardware. The batch you want doesn't fit in memory. Half the GPU's time goes on waiting for data. Eight GPUs turn out to be five times as fast as one, at eight times the price. The four techniques in this lesson are what engineers reach for in every real training run, from a fine-tune on one rented card to a model trained on thousands. Each rests on a small piece of arithmetic that you can check here in the page.
What you'll be able to do
- Say what gradient accumulation costs and what memory it does and does not save
- Scale the learning rate with the batch size, and find where the rule stops working
- Explain mixed precision: what 16-bit numbers lose, and how loss scaling repairs the gradients
- Diagnose a data loader that starves the GPU, and size its workers
- Explain how several GPUs share one batch, and why the speedup falls short of their number
Show the answer
Add up the gradients from 4 batches of 16, then update once
A batch's gradient is an average over its examples, and an average can be built in pieces. Run 16 examples, keep the gradient, run the next 16, and so on. After four, the sum is the gradient of all 64, and a single update uses it. Memory only ever held 16. This is gradient accumulation.
The idea: four shortages, four fixes
This lesson rests on GPUs and compute: training takes about 16 bytes a parameter, and a GPU's speed is a share of its peak. It also rests on Mini-batches and SGD, where gradient accumulation is built. Here the question is what each fix costs. A run in the real world runs short of four things, one at a time.
- Memory for the batch you want. Fix: accumulate gradients.
- Updates when the batch grows. Fix: scale the learning rate.
- Room and speed in each number. Fix: mixed precision.
- Data reaching the GPU, and GPUs to share the work. Fix: loader workers and data-parallel training.
Start with the memory, by hand. Four batches of 16 make an effective batch of 64. An epoch of 4,096 examples still runs 4,096 / 16 = 256 small batches, so it takes as long as before. But the weights change once for every four: 64 updates. What shrinks is the activations, the in-between results kept for the backward pass, because they grow with the batch in memory: 16 examples' worth instead of 64. What does not shrink is the 16 bytes a parameter for weights, gradients, the master copy and Adam's averages. They are there whatever the batch.
Show the answer
So that tiny updates aren't rounded away
The 16 bytes are 2 for the 16-bit weight used in the arithmetic, 2 for its gradient, 4 for the 32-bit master copy, and 8 for Adam's two running averages. Updates are added to the master copy, where a change of one part in ten thousand survives.
print(16 * 4, 4096 // 16, 4096 // 16 // 4)
Show the answer
64 256 64
Four batches of 16 make an effective batch of 64. An epoch still runs all 256 small batches. But the weights change only once for every four: 64 updates. Accumulation changes how many examples stand behind each update. It doesn't reduce the work.
Small numbers lose to the grid
A 16-bit float has about three decimal digits. Near 1, the next number above 1 is away, which is 0.000977. Anything closer rounds to a neighbor. A typical weight update is 0.0001, less than half that gap, so 1 + 0.0001 rounds back to exactly 1: the update vanishes. In plain words: a number only has so many digits, and a tiny change to a big number is lost. Step through it.
The gap near 1 is 2 to the power of minus 10, about 0.001.
The same trouble hits gradients: a gradient of 0.00000002 is below the smallest 16-bit number and becomes exactly 0. Loss scaling multiplies the loss by a large number, 65,536 (which is ), before the backward pass, so every gradient comes out that much larger. Then it divides by the same number in 32 bits before the update.
Waiting for data, and for each other
A GPU can only train on batches it has been given. Suppose it needs 20 ms for a batch, and one loader worker needs 80 ms to prepare one. Four workers deliver a batch every 80 / 4 = 20 ms: just enough.
Write a sentence or two, then compare it with a good answer in the lesson.
From scratch: measure each fix
The network below is small enough for its gradient to be written out by hand in NumPy. Its code is in this lesson's setup, folded above the first cell. The first cell compares one batch of 64 with the same rows cut into four batches of 16, then trains three ways.
Setup (runs before every cell) · 44 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np def examples(n, seed): rng = np.random.default_rng(seed) X = rng.normal(0, 1, (n, 8)) return X, np.sin(X[:, 0]) + 0.5 * X[:, 1] * X[:, 2] + 0.1 * rng.normal(0, 1, n) def new_weights(hidden=32): rng = np.random.default_rng(0) return {"W1": rng.normal(0, 0.3, (8, hidden)), "b1": np.zeros(hidden), "W2": rng.normal(0, 0.3, hidden), "b2": 0.0} def loss_and_gradient(w, X, y): """A network with one hidden layer, and the gradient of its mean squared error, worked out by hand.""" h = np.tanh(X @ w["W1"] + w["b1"]) error = h @ w["W2"] + w["b2"] - y d_out = 2 * error / len(y) d_h = np.outer(d_out, w["W2"]) * (1 - h ** 2) return (float((error ** 2).mean()), {"W1": X.T @ d_h, "b1": d_h.sum(axis=0), "W2": h.T @ d_out, "b2": float(d_out.sum())}) X, y = examples(4096, 0) X_new, y_new = examples(2000, 1) def train(batch, rate, epochs, accumulate=1): """Plain gradient descent. With accumulate > 1, gradients from several batches are added up before each update.""" w, rng, updates = new_weights(), np.random.default_rng(0), 0 for epoch in range(epochs): order = rng.permutation(len(y)) total, held = None, 0 for start in range(0, len(y) - batch + 1, batch): rows = order[start:start + batch] loss, g = loss_and_gradient(w, X[rows], y[rows]) total = g if total is None else {k: total[k] + g[k] for k in g} held += 1 if held == accumulate: for k in w: # the average of the gradients held w[k] = w[k] - rate * total[k] / accumulate total, held, updates = None, 0, updates + 1 return loss_and_gradient(w, X_new, y_new)[0], updates, w
w = new_weights() # one batch of 64 loss, big = loss_and_gradient(w, X[:64], y[:64]) # the same rows, as four batches of 16 parts = [loss_and_gradient(w, X[i:i + 16], y[i:i + 16])[1] for i in range(0, 64, 16)] average = {k: sum(part[k] for part in parts) / 4 for k in big} difference = max(float(np.abs(big[k] - average[k]).max()) for k in big) print("the batch-of-64 gradient and the average of the four agree to " "12 decimal places:", difference < 1e-12) print("largest difference from the first batch of 16 alone:", round(max(float(np.abs(big[k] - parts[0][k]).max()) for k in big), 3)) print() print("training for 5 epochs at a learning rate of 0.05 | updates | " "error on new data") a, b, c = train(64, 0.05, 5), train(16, 0.05, 5, accumulate=4), train(16, 0.05, 5) print("batches of 64 |", str(a[1]).rjust(7), "|", round(a[0], 4)) print("batches of 16, gradients added up 4 at a time |", str(b[1]).rjust(7), "|", round(b[0], 4)) print("batches of 16, an update after every one |", str(c[1]).rjust(7), "|", round(c[0], 4)) print("the first two end with the same weights, to 12 decimal places:", max(float(np.abs(a[2][k] - b[2][k]).max()) for k in a[2]) < 1e-12)
The gradient of the batch of 64 and the average of the four small gradients agree to 12 decimal places. They are the same sum, added up in a different order. One small batch alone is a different matter: its gradient differs by as much as 0.768.
Batches of 64, and batches of 16 with gradients added up 4 at a time, make the same 320 updates, reach the same error of 0.1984, and end with the same weights. The third row is the surprise. Batches of 16 with an update after each one reach an error of 0.0456, far better. It made 1,280 updates from the same data, four times as many.
So accumulation reproduces a big batch, with a big batch's smoother gradient and its smaller number of updates. It saves activation memory. It saves no time, and it does not make training better than small batches would.
Bigger batches need bigger steps
Why want a big batch, if small ones made more updates and did better? On many GPUs a big batch is the whole point: it is the work that gets shared out. The price is fewer updates for each pass through the data. The usual compensation is to multiply the learning rate by the same factor as the batch. A gradient averaged over more examples is less noisy, so it can be trusted further. Every run below sees the same data eight times.
Uses the setup above, 44 lines.
import matplotlib.pyplot as plt batches, unscaled, scaled = (16, 64, 256, 1024), [], [] print("batch | updates | error at a rate of 0.02 | rate scaled with the batch | " "error at that rate") for batch in batches: same = train(batch, 0.02, 8) rate = 0.02 * batch / 16 with np.errstate(all="ignore"): big = train(batch, rate, 8) unscaled.append(same[0]) scaled.append(min(big[0], 1.0)) result = f"{big[0]:.4f}" if big[0] < 1000 else "blew up" print(str(batch).rjust(5), "|", str(same[1]).rjust(7), "|", f"{same[0]:23.4f}", "|", f"{rate:26.2f}", "|", result) plt.plot(batches, unscaled, marker="o", label="rate fixed at 0.02") plt.plot(batches, scaled, marker="s", label="rate scaled (capped at 1.0 for the plot)") plt.xscale("log", base=2) plt.xlabel("batch size") plt.ylabel("error on new data") plt.legend() plt.show()
With the rate left at 0.02, the error climbs as the batch grows: 0.0557 at a batch of 16, then 0.2424, 0.2950 and 0.3115. Nothing is wrong with those runs except that they took too few steps of an unchanged size: 32 updates at a batch of 1,024, against 2,048.
At a batch of 64 with the rate scaled to 0.08, the error is 0.0580, almost all the way back. At 256 with a rate of 0.32 it is 0.1269: better than unscaled, and no longer a full recovery. At 1,024 the rule asks for a rate of 1.28, and training blows up. The scaling rule holds over a moderate range, and then runs into the largest step the loss surface will tolerate. Large runs push the limit out a little by raising the rate gradually over the first steps, called warmup. On a problem this small it makes no useful difference.
Show the answer
Raise the learning rate, toward 8 times its old value
With a batch eight times larger, each epoch makes an eighth as many updates. Steps of the old size no longer cover the same ground. Scaling the learning rate with the batch restores most of the progress, up to the largest rate the model can take, and a gradual warmup helps reach it safely.
Half the bits: mixed precision
A 16-bit number takes half the memory, and modern GPUs multiply 16-bit matrices considerably faster. Mixed precision runs most of the forward and backward pass in 16 bits. The word "mixed" is there for a reason, and the cell shows it.
Uses the setup above, 44 lines.
half = np.finfo(np.float16) print("16-bit floats: largest", float(half.max), "| smallest normal number", float(half.tiny), "| gap between 1 and the next number", float(half.eps)) with np.errstate(over="ignore"): print("300 x 300 in 16 bits:", np.float16(300) * np.float16(300)) print("1 + 0.0001 in 16 bits:", np.float16(1.0) + np.float16(0.0001), "| in 32 bits:", np.float32(1.0) + np.float32(0.0001)) print("a gradient of 0.00000002 in 16 bits:", np.float16(2e-8)) # multiply the loss by 65,536 first, so the gradient is that much bigger scaled = np.float16(2e-8 * 65536) print("the same gradient with the loss scaled up:", scaled, "| divided back down in 32 bits:", np.float32(scaled) / np.float32(65536)) loss, g = loss_and_gradient(new_weights(), X[:64], y[:64]) print() print("typical size of gradient | entries that become exactly 0 in 16 bits | " "with the loss scaled by 65,536") # deeper networks and larger batches give smaller gradients for shrink in (1e-3, 1e-4, 1e-5, 1e-6): small = g["W1"] * shrink print(f"{np.abs(small).mean():24.1e}", "|", f"{100 * (small.astype(np.float16) == 0).mean():38.1f}%", "|", f"{100 * ((small * 65536).astype(np.float16) == 0).mean():.1f}%") weights = np.zeros(1_000_000, dtype=np.float32) print() print("a million numbers: 32-bit", weights.nbytes, "bytes | 16-bit", weights.astype(np.float16).nbytes, "bytes")
A 16-bit float runs out of room at both ends, and in the middle.
- At the top. The largest value is 65,504. 300 × 300 is already
inf. - In the middle. Near 1, the gap to the next number is about 0.001. Adding 0.0001 to a weight of 1 changes nothing. So the weights themselves stay in 32 bits, and only a 16-bit copy is used for the arithmetic.
- At the bottom. A gradient of 0.00000002 becomes exactly 0, and a weight with a zero gradient doesn't learn.
When gradient entries are typically around 7e-07 in size, 11.7% of them vanish in 16 bits. At 7e-08, 42.2% do. With loss scaling, the table's last column is 0.0% throughout, and the gradient of 0.00000002 comes back as 1.9994332e-08.
So mixed precision is three things together: 16-bit arithmetic for speed and memory, 32-bit weights for small updates, and loss scaling for small gradients. A newer 16-bit format, bfloat16, keeps the range of 32-bit numbers and gives up precision, so it needs no loss scaling.
import numpy as np print(np.float16(1.0) + np.float16(0.0001) == np.float16(1.0), np.float32(1.0) + np.float32(0.0001) == np.float32(1.0))
Show the answer
True False
Near 1, 16-bit floats are about 0.001 apart, so 1.0001 rounds back to 1: the addition does nothing. In 32 bits the spacing near 1 is about a ten-millionth, and 1.0001 is stored without trouble.
Feeding the GPU
The cell simulates a loader. The GPU needs 20 ms to train on a batch. One worker needs 80 ms to prepare one.
Uses the setup above, 44 lines.
def run(step_ms, load_ms, workers, batches=400): """Each worker takes load_ms to prepare a batch. The GPU takes step_ms to train on one, and can't start until it is ready.""" gpu_free, waiting = 0.0, 0.0 for k in range(batches): # with several workers, that many batches are ready at once ready = (k // workers + 1) * load_ms start = max(ready, gpu_free) waiting += start - gpu_free gpu_free = start + step_ms return 100 * (1 - waiting / gpu_free), 1000 * batches / gpu_free print("the GPU needs 20 ms for a batch. " "One worker needs 80 ms to load and prepare one.") print("workers | GPU busy | batches a second") for workers in (1, 2, 4, 8): busy, rate = run(20, 80, workers) print(str(workers).rjust(7), "|", f"{busy:7.0f}%", "|", round(rate, 1)) print() print("a faster GPU, 10 ms for a batch, with the same 4 workers:") busy, rate = run(10, 80, 4) print(" GPU busy", f"{busy:.0f}%", "| batches a second", round(rate, 1))
With one worker the GPU is busy 25% of the time, and trains 12.5 batches a second. Two workers double the speed. Four bring the GPU to 99% busy and 49.5 batches a second. Eight add nothing: the GPU is now the limit, at one batch every 20 ms.
The last lines are the expensive mistake. Swap in a GPU twice as fast and keep the four workers. It is busy 50% of the time, and the speed is 49.8 batches a second: the same as before. The faster GPU bought nothing, since the data arrives no sooner.
So before paying for more compute, look at GPU utilization as monitoring tools report it: the share of time the GPU has any work to do. (The lesson on GPUs used the word for the share of peak speed.) If the GPU is often idle, the problem is upstream. The cures, cheapest first: more workers until the CPU cores run out, doing slow preparation once ahead of time, smaller files, and a few prepared batches waiting in a queue.
How many workers keep the GPU busy?
Write workers_needed(step_ms, load_ms): the smallest whole number of workers for which batches arrive at least as fast as the GPU uses them. The GPU takes step_ms for a batch, and one worker takes load_ms to prepare one, so n workers deliver a batch every load_ms / n. Always return at least 1.
Uses the setup above, 44 lines.
def workers_needed(step_ms, load_ms): return 1 print(workers_needed(20, 80), workers_needed(10, 80), workers_needed(100, 80))
Many GPUs, one batch
Four batches of 16 can be run one after another on one GPU, with the gradients added up. Or they can be run at the same moment on four GPUs, each holding a copy of the model, with the gradients averaged. The arithmetic is identical, and so is the result: one update from a batch of 64. This is data-parallel training. Every GPU must apply the same update, so at each step the gradients are exchanged and averaged. That takes time, and some work can't be shared.
Uses the setup above, 44 lines.
def step_time(compute_ms, fixed_ms, sync_ms, gpus): """Time for one update on a batch of fixed total size. The compute is shared out, and sharing has a cost.""" return compute_ms / gpus + fixed_ms + (sync_ms if gpus > 1 else 0) # ms: work that splits across GPUs, work that doesn't, and exchanging gradients compute, fixed, sync = 120, 4, 6 print("a simple model of one update: 120 ms of work that splits, 4 ms that doesn't, " "6 ms to exchange gradients") print("GPUs | ms for each update | times faster than one GPU | share of the ideal") one = step_time(compute, fixed, sync, 1) counts, speedups = (1, 2, 4, 8, 16), [] for gpus in counts: t = step_time(compute, fixed, sync, gpus) speedups.append(one / t) print(str(gpus).rjust(4), "|", f"{t:18.1f}", "|", f"{one / t:25.2f}", "|", f"{100 * one / t / gpus:.0f}%") print("the limit, however many GPUs:", round(one / (fixed + sync), 1), "times faster") import matplotlib.pyplot as plt plt.plot(counts, counts, linestyle="--", color="gray", label="ideal") plt.plot(counts, speedups, marker="o", label="this model") plt.xlabel("GPUs") plt.ylabel("times faster than one GPU") plt.legend() plt.show()
This is a model with made-up but plausible costs, and its shape is the point. Two GPUs are 1.77 times as fast as one, and four are 3.10 times as fast. Eight give 4.96, and sixteen give 7.09: 44% of what sixteen GPUs ideally would. The 4 ms that doesn't split and the 6 ms of exchange stay as the shared work shrinks. With sixteen GPUs the fixed costs are 10 ms, more than half of every step. However many GPUs are added, this job can never be more than 12.4 times faster. The principle is known as Amdahl's law.
Two consequences. To use many GPUs well, each must be given enough work that the fixed costs stay small, which means a bigger total batch, and that leads back to the learning rate. And layers that compute statistics over a batch, such as batch normalization, see only their own GPU's share unless they are told to combine across GPUs.
Combine gradients from uneven batches
Write combine(gradients, sizes). Each entry of gradients is a list of numbers: the mean gradient over one batch. sizes gives the number of examples in each batch. Return the mean gradient over all the examples together, as a list. A larger batch must count for more: weight each gradient by its size. Return [] when there are no batches.
Uses the setup above, 44 lines.
def combine(gradients, sizes): return [] print(combine([[1.0, 2.0], [3.0, 6.0]], [16, 16]), combine([[0.0], [10.0]], [30, 10]))
Worked example: planning a run
The four ideas combine into a plan. The job: a model with 300 million weights, a target batch of 512 that earlier experiments say it needs, and GPUs with 24 GB of memory. Each example's activations take 0.55 GB in 32-bit. How should the batch be made up?
Uses the setup above, 44 lines.
import math weights = 300e6 gpu_gb, target = 24, 512 # 16 bytes for each weight: weights, gradients and Adam's two averages fixed_gb = weights * 16 / 1e9 # activations kept for the backward pass, for one example, in 32-bit per_example_gb = 0.55 def fits(per_example): return math.floor((gpu_gb - fixed_gb) / per_example) print("weights, gradients and optimizer:", fixed_gb, "GB, whatever the batch") print("largest batch on one GPU, 32-bit:", fits(per_example_gb)) print("largest batch on one GPU, mixed precision (activations in 16-bit):", fits(per_example_gb / 2)) per_gpu = 64 # a round number under the limit print() print("GPUs | batch on each | batches added up before an update | effective batch | " "updates in an epoch of 1,000,000 examples") for gpus in (1, 2, 4, 8): accumulate = math.ceil(target / (per_gpu * gpus)) effective = per_gpu * gpus * accumulate print(str(gpus).rjust(4), "|", str(per_gpu).rjust(13), "|", str(accumulate).rjust(33), "|", str(effective).rjust(15), "|", format(1_000_000 // effective, ","))
The weights, gradients and optimizer take 4.8 GB whatever happens: 16 bytes for each of 300 million weights. That leaves 19.2 GB for activations. In 32-bit, 34 examples fit. With mixed precision the activations take half the space and 69 fit, so a batch of 64 on each GPU is a comfortable round number. From there the table reaches 512 four ways. One GPU adds up 8 batches before each update. Two GPUs add up 4, four add up 2, and eight need no accumulation at all. Every row makes the same 1,953 updates in an epoch of a million examples, and to the model they are the same training run. What differs is the wall-clock time and the bill: one GPU takes roughly eight times as long as eight, less whatever the eight lose to exchanging gradients. Notice that accumulation never touched the 4.8 GB: it only lets the activations stay small. Two cautions belong with any such plan. The learning rate has to suit a batch of 512, whichever row is chosen. And every row needs enough data workers for the GPUs it uses.
import math print(math.floor((24 - 4.8) / 0.55), math.ceil(512 / (64 * 2)))
Show the answer
34 4
After the fixed 4.8 GB, 19.2 GB is left, and 19.2 / 0.55 is about 34.9, so 34 examples fit: memory rounds down. Two GPUs with 64 each make 128 examples a step, and 512 / 128 = 4 steps must be added up for each update.
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: the loop that holds all four
Each technique is a few lines in a real training loop. The card below runs on real PyTorch. On a machine with a GPU it uses float16 and the GradScaler. On a CPU it falls back to bfloat16, which needs no scaling, so the same code runs everywhere. It runs 8 batches, adds up 4 at a time, and counts the updates, as the from-scratch cell did.
On the job
- Many GPUs use
DistributedDataParallel: one process for each GPU, each given a different share of the data by aDistributedSampler, with gradients averaged at every step. It needs several GPUs to run, so this page has no card for it. On a cluster it is started withtorchrun. - Measure first. Read GPU utilization before you buy anything. A job at 30 percent is waiting for data.
- Order of attack. Memory first, since a job that crashes makes no progress: batch, mixed precision, accumulation. Then data loading. Only then more GPUs, with the learning rate scaled to the total batch.
- What breaks. Gradients that are not divided by the number of accumulation steps give an update that is 4 times too large. A loss scale that is too big makes gradients
inf, and the scaler skips that step and lowers the scale. - Two kinds of batch. Batch norm sees only its own GPU's share. Tell it to combine across GPUs, or use layer norm.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 6 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 13 review cards for spaced repetition
Try a quiz question
A run adds up gradients over 8 batches of 32 before each update. What has changed, compared with updating after every batch of 32?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 4 · about 24 hours of study
Deep learning
Neural networks, backprop and training them well
31 lessons 1 free
- The artificial neuron FreeWeighted sum plus a nonlinearity.35 min
- Activation functionsReLU, sigmoid, GELU, and why nonlinearity matters.35 min
- Multi-layer networksStack layers to learn complex functions.35 min
- The forward passFollow numbers through a network to a prediction.36 min
- Computation graphsEvery calculation as a graph of small operations.40 min
- BackpropagationThe chain rule applied backwards to get every gradient.42 min
- Build an autograd engineWrite backprop yourself in ~100 lines (micrograd).42 min
- PyTorch tensorsNumPy-like arrays that run on GPUs.38 min
- PyTorch autograd and modulesLet PyTorch compute gradients; build models with nn.Module.42 min
- The training loopForward, loss, backward, step, repeat.40 min
- Mini-batches and SGDTrain on small random batches instead of everything at once.40 min
- OptimizersMomentum and Adam.41 min
- Learning-rate schedules in PyTorchWarmup then cosine by hand, as a function of the step, then LambdaLR and where scheduler.step() goes; digits trained with and without warmup.25 min
- Vanishing and exploding gradientsWhy deep networks were hard to train.40 min
- Weight initializationStart weights at the right scale.33 min
- Normalization layersBatch and layer norm keep training stable.40 min
- Residual connectionsSkip connections that let gradients flow.40 min
- DropoutRandomly switch off neurons to prevent overfitting.35 min
- Debugging trainingRead a failing run from its numbers, and fix what's wrong.40 min
- Looking inside a networkRead four numbers per layer (spread, dead units, gradient norm, update-to-weight ratio), collect them in NumPy, then with forward and tensor hooks, and diagnose three planted faults.30 min
- Transfer learningReuse a trained network: freeze the backbone, train a new head, fine-tune with two learning rates.42 min
- Convolutional networksPattern detectors for images.44 min
- Embedding layersLearned lookup tables that turn tokens into vectors.38 min
- Recurrent networksProcess sequences step by step, and their limits.43 min
- LSTMs and GRUs, built and trainedBuild an LSTM and a GRU from a linear layer, train them on the adding problem beside a plain RNN, read what the gates learned, measure the gradient's path, and carry state across a stream.40 min
- Sequence to sequence, and the invention of attentionBuild an encoder and decoder, measure the bottleneck of a fixed-size handoff, add attention in three lines, read an attention map with care, and see how one mistake compounds after teacher forcing.45 min
- AutoencodersEncoder, bottleneck and decoder, reconstruction error for anomalies, denoising, and the latent space image generators use.42 min
- GPUs and computeWhy deep learning runs on graphics cards.40 min
- Training at scale: precision, accumulation, data and many GPUsGradient accumulation and what it really saves, scaling the learning rate with the batch, mixed precision and loss scaling, a data loader that starves the GPU, and why eight GPUs aren't eight times as fast.45 min
- A training script, start to finishConfig, seed, data, model, schedule, loop and checkpoint as one script: what a checkpoint must hold, a resume that matches an unbroken run, and buffers, collate and torch.save in PyTorch.40 min
- Capstone: train a seam-tear spotterTrain and compare three models on fabric images, choose the alarm level from costs, and score once.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write