How stretched a problem is: the ratio of the largest to the smallest stretch. A large one makes training slow and numbers unstable.
Try it now
Learn it properly
Why training is fast or slow · Second derivatives, minimum or maximum, why curvature caps the learning rate, convex losses and Newton's method. About 40 minutes.
convex: Bending up everywhere, so a straight line between two points on the curve never dips below it. A convex loss has one valley: any minimum is the lowest.
curvature: How sharply a curve bends, measured by the second derivative. For a loss, it sets the largest learning rate that still converges: 2 ÷ curvature.
gradient descent: Improving a model by taking small steps in the direction that lowers the error the most, over and over.
Hessian: The table of second derivatives for a function of many inputs, one entry for every pair. Far too big to store for a model with billions of weights.
Newton's method: An optimizer that divides the slope by the curvature to choose each step. It lands on a bowl's bottom in one step, but needs the Hessian.
saddle point: A flat point that curves up in some directions and down in others, like a horse's saddle. Neural network losses have many.
second derivative: The slope of the slope: how fast a curve's slope is changing. Positive means the curve bends up like a bowl; negative, down like a hilltop.
Keep learning
More in plain English.
1286 terms in the glossary, each with the lesson that teaches it.