Glossary · Phase 5 · Transformers
Dense model
An ordinary model where every token uses every parameter, as opposed to a mixture of experts.
Try it now
This is how to read "671B total, 37B active". Memory follows the total, and compute follows the active number.
Learn it properly
Mixture of experts · Route each token to a few specialist sub-networks. About 35 minutes.
Other terms from the same lesson
- active parameters: The parameters each token actually uses in a mixture-of-experts model. They set the compute per token; the total sets the memory needed.
- expert: One of the several feed-forward networks in a mixture-of-experts layer. Each token only goes through the few the router picks.
- load-balancing loss: An extra training penalty that pushes the router to spread tokens evenly, so no expert is overloaded and none sits unused.
- mixture of experts: A model with many specialist sub-networks, where only a few are switched on for each token, saving compute.
- router: The small layer in a mixture-of-experts model that scores the experts for each token and sends the token to the top few.
- shared expert: An expert that every token goes through, alongside the ones the router picks.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.