Glossary · Phase 2 · Math, when you need it
Hessian
The table of second derivatives for a function of many inputs, one entry for every pair. Far too big to store for a model with billions of weights.
Try it now
The shrink factor is how much of the distance to the bottom survives each step. Below 1, training converges; above 1, it blows up.
Learn it properly
Curvature and step sizes · Second derivatives, minimum or maximum, why curvature caps the learning rate, convex losses and Newton's method. About 25 minutes.
Other terms from the same lesson
- convex: Bending up everywhere, so a straight line between two points on the curve never dips below it. A convex loss has one valley: any minimum is the lowest.
- curvature: How sharply a curve bends, measured by the second derivative. For a loss, it sets the largest learning rate that still converges: 2 ÷ curvature.
- Newton's method: An optimizer that divides the slope by the curvature to choose each step. It lands on a bowl's bottom in one step, but needs the Hessian.
- saddle point: A flat point that curves up in some directions and down in others, like a horse's saddle. Neural network losses have many.
- second derivative: The slope of the slope: how fast a curve's slope is changing. Positive means the curve bends up like a bowl; negative, down like a hilltop.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.