The glossary
Every word you'll meet in AI and machine learning, explained in everyday language.
Saved keys and values for a prompt start that many requests share, so the server skips that work.
The first token is out at 270 ms. The prefill's cost is 150 ms of a whole GPU. The decode's cost is shared by 10 requests.
One request through a serving stack: latency and cost · Follow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost. About 35 minutes.
1286 terms in the glossary, each with the lesson that teaches it.
Every word you'll meet in AI and machine learning, explained in everyday language.
Follow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write