Glossary · Phase 6 · Modern LLM engineering
Lossless
Changing how something is computed without changing the result at all, such as speculative decoding producing exactly what the big model would have.
Try it now
The first rejected token ends the round, so acceptance rate matters far more than draft length. This is a simplified model of the speed-up.
Learn it properly
Speculative decoding · Draft with a small model, verify with a big one. About 35 minutes.
Other terms from the same lesson
- draft model: In speculative decoding, the small fast model that guesses the next few tokens for the big model to check.
- speculative decoding: Speeding up generation by letting a small model draft several tokens and a big model check them at once.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.