Glossary · Phase 6 · Modern LLM engineering
Throughput
How much work a system gets through per second, such as tokens generated across all users. Different from how fast one user gets a reply.
Try it now
This is the simple model from this section. Bigger batches slow each user a little and raise the total a lot.
Learn it properly
Serving models · Batching, throughput and latency in production. About 35 minutes.
Other terms from the same lesson
- continuous batching: Letting new requests join a running batch the moment a slot frees up, instead of waiting for the whole batch to finish. It keeps the GPU busy.
- PagedAttention: Storing each conversation's KV cache in small fixed-size blocks, like pages of memory, so little memory is wasted and more conversations fit at once.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.