Index Ai Academy
Glossary · Phase 2 · Math for ML engineers

Condition number

How stretched a problem is: the ratio of the largest to the smallest stretch. A large one makes training slow and numbers unstable.

Try it now

Learn it properly

Why training is fast or slow · Second derivatives, minimum or maximum, why curvature caps the learning rate, convex losses and Newton's method. About 40 minutes.

Other terms from the same lesson

  • convex: Bending up everywhere, so a straight line between two points on the curve never dips below it. A convex loss has one valley: any minimum is the lowest.
  • curvature: How sharply a curve bends, measured by the second derivative. For a loss, it sets the largest learning rate that still converges: 2 ÷ curvature.
  • gradient descent: Improving a model by taking small steps in the direction that lowers the error the most, over and over.
  • Hessian: The table of second derivatives for a function of many inputs, one entry for every pair. Far too big to store for a model with billions of weights.
  • Newton's method: An optimizer that divides the slope by the curvature to choose each step. It lands on a bowl's bottom in one step, but needs the Hessian.
  • saddle point: A flat point that curves up in some directions and down in others, like a horse's saddle. Neural network losses have many.
  • second derivative: The slope of the slope: how fast a curve's slope is changing. Positive means the curve bends up like a bowl; negative, down like a hilltop.
Keep learning

More in plain English.

1286 terms in the glossary, each with the lesson that teaches it.

1286 terms

The glossary

Every word you'll meet in AI and machine learning, explained in everyday language.

Phase 2 · lesson 40 min

Why training is fast or slow

Second derivatives, minimum or maximum, why curvature caps the learning rate, convex losses and Newton's method.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write