Glossary · Phase 6 · Modern LLM engineering
Continuous batching
Letting new requests join a running batch the moment a slot frees up, instead of waiting for the whole batch to finish. It keeps the GPU busy.
Try it now
This is the simple model from this section. Bigger batches slow each user a little and raise the total a lot.
Learn it properly
Serving models · Batching, throughput and latency in production. About 35 minutes.
Other terms from the same lesson
- PagedAttention: Storing each conversation's KV cache in small fixed-size blocks, like pages of memory, so little memory is wasted and more conversations fit at once.
- throughput: How much work a system gets through per second, such as tokens generated across all users. Different from how fast one user gets a reply.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.