Index Ai Academy
Glossary · Phase 11 · Professional practice

Online inference

Also written: online prediction, real-time inference

Running a model when a request arrives, while the caller waits for the answer.

Try it now

Ask when the inputs exist and how old an answer may be. If neither forces your hand, choose batch.

Learn it properly

Batch and online inference · Predict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit. About 40 minutes.

Other terms from the same lesson

  • batch inference: Running a model ahead of time over many rows and storing the predictions, to be looked up later.
  • freshness limit: The greatest age at which a stored prediction may still be served before the model must be run again.
  • micro-batching: Scoring several waiting requests in one model call, so the call's fixed overhead is shared between them.
Keep learning

More in plain English.

1286 terms in the glossary, each with the lesson that teaches it.

1286 terms

The glossary

Every word you'll meet in AI and machine learning, explained in everyday language.

Phase 11 · lesson 40 min

Batch and online inference

Predict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write