Second derivative
The slope of the slope: how fast a curve's slope is changing. Positive means the curve bends up like a bowl; negative, down like a hilltop.
The derivative tells you the slope of a curve at each point. The second derivative asks the same question about the slope itself: how fast is the slope changing? For f(x) = x², the slope is 2x, and the slope of 2x is 2, everywhere. So x² has a second derivative of 2: its slope rises by 2 for every step of 1 to the right.
Try it now
The shrink factor is how much of the distance to the bottom survives each step. Below 1, training converges; above 1, it blows up.
Learn it properly
Curvature and step sizes · Second derivatives, minimum or maximum, why curvature caps the learning rate, convex losses and Newton's method. About 25 minutes.
Other terms from the same lesson
- convex: Bending up everywhere, so a straight line between two points on the curve never dips below it. A convex loss has one valley: any minimum is the lowest.
- curvature: How sharply a curve bends, measured by the second derivative. For a loss, it sets the largest learning rate that still converges: 2 ÷ curvature.
- Hessian: The table of second derivatives for a function of many inputs, one entry for every pair. Far too big to store for a model with billions of weights.
- Newton's method: An optimizer that divides the slope by the curvature to choose each step. It lands on a bowl's bottom in one step, but needs the Hessian.
- saddle point: A flat point that curves up in some directions and down in others, like a horse's saddle. Neural network losses have many.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.