Gradient descent, visualized

Step down a curve yourself and see why the learning rate matters, how valleys trap you, and why plain gradient descent zig-zags.

Training a model means finding the weights that make its loss small. Gradient descent does that with one repeated move: find which way is downhill from where you are, and take a step that way. The size of the step is the learning rate, and these pieces run the real calculation.

Press Step a few times at 0.1. Try 0.9: it bounces across the valley but still settles. Then try 1.05: each step overshoots further, and it flies off.

Local valleys

Gradient descent only sees the slope where it's standing, so where you start decides which valley you find.

Start at 2.8 and run 10 steps twice. You settle in the right-hand valley, even though the left one is deeper. Move the start below 0 and you find the deeper one.

Why it zig-zags

Real losses are often steep in one direction and gentle in another. A step big enough for the gentle direction overshoots in the steep one, so the path bounces from wall to wall. Momentum and Adam are optimizers designed for this problem.

Click to move the start, then step. This valley is 8 times steeper one way than the other.

Learn how this works, properly

A free account opens all 244 lessons, with Python that runs in your browser, saved progress and certificates.

Create a free account