Saddle point
A flat point that curves up in some directions and down in others, like a horse's saddle. Neural network losses have many.
The losses of linear regression (mean squared error) and logistic regression are convex in their weights. That's why those models train reliably, and why two people fitting the same data get the same coefficients. A neural network's loss is not convex: it has many valleys, ridges and flat saddle points, places where the curve bends up in some directions and down in others. Different random starts end in different places. In practice, very large networks mostly reach valleys that are equally good, but there's no guarantee.
Try it now
The shrink factor is how much of the distance to the bottom survives each step. Below 1, training converges; above 1, it blows up.
Learn it properly
Curvature and step sizes · Second derivatives, minimum or maximum, why curvature caps the learning rate, convex losses and Newton's method. About 25 minutes.
Other terms from the same lesson
- convex: Bending up everywhere, so a straight line between two points on the curve never dips below it. A convex loss has one valley: any minimum is the lowest.
- curvature: How sharply a curve bends, measured by the second derivative. For a loss, it sets the largest learning rate that still converges: 2 ÷ curvature.
- Hessian: The table of second derivatives for a function of many inputs, one entry for every pair. Far too big to store for a model with billions of weights.
- Newton's method: An optimizer that divides the slope by the curvature to choose each step. It lands on a bowl's bottom in one step, but needs the Hessian.
- second derivative: The slope of the slope: how fast a curve's slope is changing. Positive means the curve bends up like a bowl; negative, down like a hilltop.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.