Glossary · Phase 4 · Deep learning
Bf16
A 16-bit number format with the same range as normal 32-bit numbers but less precision. It halves memory and speeds up training.
Try it now
The rules of thumb from this lesson, live. Time scales with parameters times tokens, so doubling both takes four times as long.
Learn it properly
GPUs and compute · Why deep learning runs on graphics cards. About 30 minutes.
Other terms from the same lesson
- FLOPs: Floating-point operations: the count of basic multiply-and-add steps. Training cost and chip speed are both measured in them.
- GPU: A graphics chip that does thousands of multiplications at once, which is why it trains neural networks fast.
- mixed precision: Doing most of the math in smaller, faster number formats like bf16, while keeping a full-precision copy where accuracy matters.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.