Index Ai Academy
Glossary · Phase 4 · Deep learning

Amdahl's law

The speedup from adding processors is limited by the part of the work that can't be shared between them.

Try it now

The gap near 1 is 2 to the power of minus 10, about 0.001.

Learn it properly

Training at scale: precision, accumulation, data and many GPUs · Gradient accumulation and what it really saves, scaling the learning rate with the batch, mixed precision and loss scaling, a data loader that starves the GPU, and why eight GPUs aren't eight times as fast. About 45 minutes.

Other terms from the same lesson

  • data-parallel training: Training one model on several GPUs at once: each runs a share of the batch on its own copy, and the gradients are averaged for one shared update.
  • gradient accumulation: Adding up the gradients of several small batches before one update, to get a big batch's update with a small batch's memory.
  • loss scaling: Multiplying the loss by a large number before the backward pass so that small gradients survive in 16-bit numbers, then dividing it back out before the update.
Keep learning

More in plain English.

1286 terms in the glossary, each with the lesson that teaches it.

1286 terms

The glossary

Every word you'll meet in AI and machine learning, explained in everyday language.

Phase 4 · lesson 45 min

Training at scale: precision, accumulation, data and many GPUs

Gradient accumulation and what it really saves, scaling the learning rate with the batch, mixed precision and loss scaling, a data loader that starves the GPU, and why eight GPUs aren't eight times as fast.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write