Glossary · Phase 6 · Modern LLM engineering
Speculative decoding
Speeding up generation by letting a small model draft several tokens and a big model check them at once.
Try it now
The first rejected token ends the round, so acceptance rate matters far more than draft length. This is a simplified model of the speed-up.
Learn it properly
Speculative decoding · Draft with a small model, verify with a big one. About 35 minutes.
Other terms from the same lesson
- draft model: In speculative decoding, the small fast model that guesses the next few tokens for the big model to check.
- lossless: Changing how something is computed without changing the result at all, such as speculative decoding producing exactly what the big model would have.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.