Index Ai Academy
Glossary · Phase 6 · Modern LLM engineering

Per-channel scale

A separate quantization scale for each output channel, which loses less accuracy than one scale for the whole layer.

Try it now

The largest possible error is half a gap, 0.2. Zero sits exactly on a mark.

Learn it properly

Quantization in practice: zero points, calibration and formats · Scale and zero point, per-channel scales, calibration sets, error on a layer's output, quantization-aware training and the file formats you meet at work. About 35 minutes.

Other terms from the same lesson

  • calibration set: A small sample of real inputs run through a model to find the ranges its values take, so quantization scales fit them.
  • quantization-aware training: Training with the rounding of quantization built in, so the model learns weights that survive it.
  • straight-through estimator: A trick that lets gradients pass through rounding as if it were the identity, so quantization-aware training can learn.
  • zero point: The integer that stands for the real value 0 in a quantized range, so both signs fit the integers.
Keep learning

More in plain English.

1286 terms in the glossary, each with the lesson that teaches it.

1286 terms

The glossary

Every word you'll meet in AI and machine learning, explained in everyday language.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write