Curvature and step sizes

That factor, 1 − η × c, tells the whole story:

• if η × c is below 1, the factor is between 0 and 1: w shrinks steadily toward the bottom;
• if η × c is between 1 and 2, the factor is between −1 and 0: w overshoots, flips sides, but still shrinks;
• if η × c is above 2, the factor is below −1: every step overshoots by more, and training blows up.

So the largest learning rate that converges is 2 ÷ c. A sharply curved loss needs small steps; a gently curved one can take big ones. This is where the "too big a learning rate diverges" rule from gradient descent comes from.

w = 1.0
for step in range(2):
    w = w - 0.6 * 4 * w
print(round(w, 2))

Ravines: two curvatures at once

Real losses have many weights, and the curvature can be very different in different directions. Picture a long, narrow valley: steep side walls, a gently sloping floor. The learning rate must stay under 2 ÷ c for the steepest direction, or the walls send training bouncing out. But a rate that small barely moves along the gentle floor.

Take L(a, b) = 0.5 × (10 × a² + b²): curvature 10 along a, curvature 1 along b. The largest safe rate is 2 ÷ 10 = 0.2. At 0.1, a reaches the bottom in one step, but b only shrinks by 10% per step:

a, b = 1.0, 1.0
for step in range(5):
    a, b = a - 0.1 * 10 * a, b - 0.1 * b
    print(step + 1, round(a, 3), round(b, 3))

This is the most common reason training is slow, and it's what momentum and Adam were invented for. Momentum builds up speed along the consistent floor direction while the bouncing across the walls cancels out. Adam scales each weight's step by how large its recent gradients have been, a cheap stand-in for knowing each direction's curvature.

Convex: one valley only

A function is convex if a straight line between any two points on its curve never dips below the curve. In one dimension that's the same as the second derivative never being negative: the curve bends up, or is straight, everywhere. Convex losses have a wonderful property: any minimum you find is the lowest one there is. There are no false valleys to get stuck in, so every starting point leads to the same answer.

The losses of linear regression (mean squared error) and logistic regression are convex in their weights. That's why those models train reliably, and why two people fitting the same data get the same coefficients. A neural network's loss is not convex: it has many valleys, ridges and flat saddle points, places where the curve bends up in some directions and down in others. Different random starts end in different places. In practice, very large networks mostly reach valleys that are equally good, but there's no guarantee.

def chord_above(f, x1, x2):
    middle = (x1 + x2) / 2
    return (f(x1) + f(x2)) / 2 >= f(middle)

print(chord_above(lambda x: x ** 2, -3, 1), chord_above(lambda x: x ** 3, -3, 1))

Newton's method: divide by the curvature

If you know the curvature, you can choose the step size for each point instead of guessing a learning rate. Newton's method steps by the slope divided by the curvature:

w ← w − slope ÷ curvature

In a perfect bowl this lands exactly on the bottom in one step. On other curves it converges remarkably fast. Here it is on f(w) = eʷ − 2w, whose slope is eʷ − 2 and curvature eʷ; the minimum is at w = log 2 ≈ 0.6931:

import math
w = 0.0
for step in range(4):
    w = w - (math.exp(w) - 2) / math.exp(w)
    print(step + 1, round(w, 4))

So why doesn't every model train this way? With many weights, curvature becomes a table, the Hessian, with one entry for every pair of weights. A model with a billion weights would need a billion billion entries. Deep learning uses first-order methods, slope only, with tricks like momentum and Adam to make up for the missing curvature.

Curvature in AI

You'll meet curvature in quieter forms too. Learning-rate warmup starts training with tiny steps because the curvature early in training can be sharp. Researchers compare sharp and flat minima: a model sitting in a wide, gently curved valley tends to cope better with small changes to its inputs than one in a narrow, steep one. And when a training loss suddenly spikes, a step that was too large for the local curvature is often the cause, which is why a lower learning rate is the first thing to try.

import random


def _simulate(seed=5):
    """Build the house data: 200 sales with floor area, rooms and price."""
    rng = random.Random(seed)
    rows = []
    for _ in range(200):
        size = rng.uniform(50, 250)                                   # floor area in square meters
        rooms = min(6, max(1, round(size / 45 + rng.gauss(0, 0.7))))
        price = 3 * size + 15 * rooms + 50 + rng.gauss(0, 20)         # price in thousands
        rows.append((round(size, 1), rooms, round(price, 1)))
    return rows


houses = _simulate()
sizes = [h[0] for h in houses]
rooms = [h[1] for h in houses]
prices = [h[2] for h in houses]
print(houses[:3])
print(len(houses), "houses; sizes", min(sizes), "to", max(sizes), "| rooms", min(rooms), "to", max(rooms))

# The model: price ≈ w_size × size + w_rooms × rooms + b, fitted by gradient descent on the mean squared error.
# Your study starts here: the loss and its slopes, then the curvatures, then the learning-rate sweep.