Glossary · Phase 3 · Machine learning
Weight decay
Shrinking every weight a tiny bit on each step, so the model keeps its weights small and is less likely to overfit.
Try it now
Slide the penalty and watch the error on new points fall, then rise again when the penalty is too strong.
Learn it properly
Regularization · Penalize complexity so models generalize. About 30 minutes.
Other terms from the same lesson
- elastic net: Regularization that mixes the ridge and lasso penalties.
- lasso: Regularization that adds the plain size of every weight to the loss. It pushes unhelpful weights to exactly zero, so it also picks features.
- regularization: Gently penalizing a model for being too complicated, to stop it overfitting.
- ridge regression: Regularization that adds the squared size of every weight to the loss, so weights stay small but rarely reach exactly zero.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.