Glossary · Phase 6 · Modern LLM engineering
Rank
In LoRA, the width of the two thin tables that make up the update. Higher rank can learn more but costs more.
Try it now
A rank-16 LoRA on a 4,096 × 4,096 table trains under 1% of the numbers. Raise the rank and see the cost grow only in a straight line.
Learn it properly
LoRA · Fine-tune a big model by training small low-rank updates. About 40 minutes.
Other terms from the same lesson
- adapter: A small set of extra trained weights added to a frozen model to change its behavior. Easy to swap, share and remove.
- LoRA: Low-rank adaptation: fine-tuning by training a small add-on instead of all the weights, which is far cheaper.
- QLoRA: LoRA on top of a model whose frozen weights are stored in 4 bits, so a big model fits on one GPU for fine-tuning.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.