Index Ai Academy
Glossary · Phase 2 · Math for ML engineers

forward KL

KL(p‖q) with p the data: it punishes a model q for missing anything p does, so q spreads out to cover it.

Try it now

Which fit is closer to p depends on which KL you ask. The next part measures both.

Learn it properly

KL divergence at work · Forward against reverse KL, KL between normals, F.kl_div's argument order, a distillation loss with temperature, and a KL penalty against a base model. About 36 minutes.

Other terms from the same lesson

  • reverse KL: KL(q‖p) with p the data: it punishes q for putting weight where p has none, so q locks onto one mode.
Keep learning

More in plain English.

1286 terms in the glossary, each with the lesson that teaches it.

1286 terms

The glossary

Every word you'll meet in AI and machine learning, explained in everyday language.

Phase 2 · lesson 36 min

KL divergence at work

Forward against reverse KL, KL between normals, F.kl_div's argument order, a distillation loss with temperature, and a KL penalty against a base model.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write