Gradient descent
Why this matters
Lesson 75 found minimums by solving f′ = 0. That works for a parabola, but a neural network's loss has millions of inputs and no formula to solve. So training does something humbler and far more general: start somewhere, look at the gradient, take a small step downhill, and repeat until the loss stops falling. That's gradient descent, and it is how every model in this academy is trained, from a straight line to a large language model. This lesson takes its first steps by hand, writes it as a loop, shows what the learning rate does, and fits a line with it, checking the result against lesson 60's exact answer.
- Walk downhill to a low point: start with a guess, step against the gradient, repeat
- Use the learning rate to set the step, and recognize steps that are too small, right, and too big
- Write a training loop that updates variables inside a for loop and prints the loss
- Fit a line by gradient descent, compare it with the exact answer from elimination, and train it again with PyTorch's zero_grad(), backward() and step()
The math: step against the gradient, repeat
Purpose. Walk to a low point when no formula gives it.
Start with a guess θ (theta, standing for all the weights). Then repeat one rule:
In plain words: new weights = old weights minus the learning rate times the gradient.
The gradient points uphill (lesson 78), so subtracting it moves downhill. The learning rate η scales the step. Near the bottom the gradient shrinks, so the steps shrink too, and the walk settles.
For L(x) = (x − 2)² + 1 from x = 0 with η = 0.25: the slope is −4, so x = 0 − 0.25 × (−4) = 1; then the slope is −2 and x = 1.5; then 1.75, then 1.875, closing in on the bottom at 2.