Index Ai Academy
Phase 8 · FastAPI services

Streaming from the server

Server-sent events written by a server: an ASGI app that sends a body in pieces, FastAPI's StreamingResponse, heartbeats, and what a closed connection looks like.

About 40 minutes · Model names checked November 2026 · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers7

Why it matters

Reading a stream has come up three times: in Streaming responses, in API styles and in A chat front end in the browser. Writing one has not. A model takes seconds to write a reply, and a service that waits for the whole reply before it answers leaves the user looking at nothing. A service that sends each token as it is made shows the first word at once. It also has a duty the reading side never had: when the client leaves, the server must notice and stop, or it keeps paying for tokens that no one will read.

What you'll be able to do

  • Write an event stream: the content type, events ended by a blank line, named events and a heartbeat comment
  • Send a response in pieces from a bare ASGI app, then with FastAPI's StreamingResponse over an async generator
  • Read the stream back with httpx, and tell what a client leaving looks like to the server
  • Stream a small model's tokens as events, and stop generating when the client has gone
Quick check
A chat service takes 6 seconds to write a reply. The user sees the first word after 6 seconds. What change makes the first word appear almost at once?
  1. Compress the reply so that it travels over the network faster
  2. Cache the reply for the next user
  3. Send each token as it is made, instead of the whole reply at the end
  4. Use a faster HTTP version
Show the answer

Send each token as it is made, instead of the whole reply at the end

The delay is waiting for the whole reply to be written. Sending each piece as it is made puts the first word on screen at once, though the reply takes as long to finish.

The idea: one response, sent in pieces

This rests on API styles, where you parsed an event stream, and Build an API server from scratch, where an ASGI app answered with send. A streamed reply is one ordinary HTTP response that does not end at once. The server sends the status and headers, with Content-Type: text/event-stream, and then keeps sending events. An event is a few lines ended by a blank line: data: {...}, optionally event: name before it. A line that starts with a colon is a comment, which parsers ignore, so a server uses it as a heartbeat to stop a proxy closing a connection that looks idle. There is no "end" marker in the format; the server just stops, or sends its own final event, such as data: [DONE].

The length is not known when the headers go out, so a streamed reply has no Content-Length, and a status line, once sent, cannot be changed. The other side of a stream is the one that goes wrong. The client can leave at any moment, by closing the tab or pressing Stop, and nothing tells your generator unless it asks. In ASGI the news is a message of type http.disconnect that receive returns. The picture shows what a user sees with and without streaming, for the same reply, and what it costs the server in work to do it: nothing more, since the tokens are made either way.

Try it now

The reply takes the same time to finish. Streaming shows the first word at 400 ms; not streaming shows nothing until the end.

Quick check
Recall: a parser cuts a buffer into events at blank lines and keeps the unfinished tail. What does this print?
buffer = 'data: a\n\ndata: b\n\ndata: c'
pieces = buffer.split("\n\n")
tail = pieces.pop()
print(len(pieces), repr(tail))
  1. 2 'data: c'
  2. 3 'data: c'
  3. 2 ''
  4. 3 ''
Show the answer

2 'data: c'

Two blank lines end two complete events. The last piece has no blank line after it, so it is the tail, kept for the next chunk.

Quick check
A server streams a reply, and halfway through the model call fails. The status line was already sent as 200. What can the server still do?
  1. Send an error event and end the stream
  2. Change the status to 500
  3. Resend the status line
  4. Nothing: the client sees nothing
Show the answer

Send an error event and end the stream

A status goes out once, before the first piece, and cannot be changed after. The server can only say something inside the stream, such as event: error, and then stop. The client must handle that event.

In your own words
In your own words: what is an event stream, and how does a server notice that the client has left?

Write a sentence or two, then compare it with a good answer in the lesson.

In Python: send in pieces, then let FastAPI do it

An ASGI response is made of messages. The first, http.response.start, sends the status and headers. Each http.response.body carries bytes, and while its more_body is true the response is not finished. That is all streaming is at the lowest level. The cell runs a bare ASGI app that sends three events, with a fake receive and send, and prints each message it sent.

Worked example

A bare ASGI app that sends three events

Each event is its own body message with more_body: True; a last empty message ends the response.

Setup (runs before every cell) · 18 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import json
import torch
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt-chat")
STOP = tok.token_id("<end>")
PROMPT = "<user> Describe the River tent . <assistant>"


def token_stream(prompt=PROMPT, max_new=10):
    """One token at a time: the words a small chat model writes, greedily."""
    idx = torch.tensor([tok.encode(prompt)])
    for _ in range(max_new):
        idx = model.generate(idx, 1, temperature=0)
        nxt = int(idx[0, -1])
        if nxt == STOP:
            return
        yield tok.vocab[nxt]
import asyncio

async def app(scope, receive, send):
    await send({"type": "http.response.start", "status": 200,
                "headers": [(b"content-type", b"text/event-stream")]})
    for n in range(3):
        body = f"data: {n}\n\n".encode()
        await send({"type": "http.response.body", "body": body, "more_body": True})
    await send({"type": "http.response.body", "body": b"", "more_body": False})

sent = []

async def fake_receive():
    return {"type": "http.request", "body": b"", "more_body": False}

async def fake_send(message):
    sent.append(message)

asyncio.run(app({"type": "http", "path": "/"}, fake_receive, fake_send))
for m in sent:
    print(m["type"], m.get("body"), m.get("more_body"))
In the lesson you can change this and run it.

Five messages: the start, three events with more_body true, and an empty last body with more_body false that closes the response. A server writes each one to the socket as it gets it, which is why the client sees the events one by one.

FastAPI hides the messages. Return a StreamingResponse over a generator, sync or async, and it sends each value the generator yields as one body message. The cell serves a small app and reads it back with httpx, line by line. The generator is an ordinary async function with yield; the stepper after the cell shows what it does between yields.

Uses the setup above, 18 lines.

import asyncio
import httpx
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

api = FastAPI()

async def numbers(count: int):
    for n in range(count):
        yield f"data: {json.dumps({'n': n})}\n\n"
        await asyncio.sleep(0)
    yield "data: [DONE]\n\n"

@api.get("/count")
async def count(count: int = 3):
    return StreamingResponse(numbers(count), media_type="text/event-stream")

async def main():
    transport = httpx.ASGITransport(app=api)
    async with httpx.AsyncClient(transport=transport, base_url="http://t") as c:
        async with c.stream("GET", "/count", params={"count": 3}) as r:
            print(r.status_code, r.headers["content-type"])
            lines = [line async for line in r.aiter_lines() if line]
            print(lines)

asyncio.run(main())
Quick check
What does this print?
def numbers():
    for n in range(3):
        yield f"data: {n}\n\n"

pieces = list(numbers())
print(len(pieces), "".join(pieces).count("\n\n"))
  1. 3 1
  2. 3 3
  3. 6 6
  4. 1 3
Show the answer

3 3

The generator yields three separate strings, one per event, and each ends with one blank line, so the joined text has three blank-line endings.

Drill

Cut a buffer into events

Write split_events(buffer): return (events, rest), where events is the list of complete events (each without its ending blank line) and rest is the unfinished tail. An event ends at a blank line, "\n\n".

Uses the setup above, 18 lines.

def split_events(buffer):
    return [buffer], ""


print(split_events('data: a\n\ndata: b\n\ndata: c'))
In the lesson, your code is checked as you write it.
Drill

Add a heartbeat

Write with_heartbeat(events, every): given a list of event strings, return a new list with the comment ": ping\n\n" inserted after every every events. Do not add one after the last event if the list ends exactly on a multiple, only between events.

Uses the setup above, 18 lines.

def with_heartbeat(events, every):
    return list(events)


print(with_heartbeat(['a', 'b', 'c', 'd'], 2))
In the lesson, your code is checked as you write it.
Quick check
Which header does a streamed event reply carry that a plain JSON reply does not?
  1. Retry-After
  2. Content-Type: application/json
  3. Content-Length, set to the final size of the whole reply
  4. Content-Type: text/event-stream, and no Content-Length
Show the answer

Content-Type: text/event-stream, and no Content-Length

The type says "events", and there is no Content-Length because the server does not know how long the stream will run.

In your own words
In your own words: how does a FastAPI generator turn into a streamed response?

Write a sentence or two, then compare it with a good answer in the lesson.

In AI and ML systems: stream a model's tokens, and stop when the client leaves

The tiny chat model from the Phase 6 lessons is loaded once in this lesson's setup, and token_stream yields its words one at a time. Wrapped in a StreamingResponse, each word leaves the server as one event. Loading once matters as much as streaming: a route that loaded the model for each request would pay that cost every time (APIs with FastAPI introduced lifespan for it).

Worked example

Stream a tiny GPT's tokens as events

The chat model writes up to ten words greedily. Each becomes one data: event with a JSON delta, and the stream ends with [DONE]. The client reads them back line by line.

Uses the setup above, 18 lines.

import asyncio
import httpx
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

api = FastAPI()

async def chat_events():
    for word in token_stream():
        yield f"data: {json.dumps({'delta': word + ' '})}\n\n"
        await asyncio.sleep(0)
    yield "data: [DONE]\n\n"

@api.get("/chat")
async def chat():
    return StreamingResponse(chat_events(), media_type="text/event-stream")

async def main():
    transport = httpx.ASGITransport(app=api)
    async with httpx.AsyncClient(transport=transport, base_url="http://t") as c:
        async with c.stream("GET", "/chat") as r:
            lines = [line async for line in r.aiter_lines() if line]
    deltas = [json.loads(line[6:])["delta"] for line in lines if "delta" in line]
    print(len(lines), lines[-1])
    print("".join(deltas))

asyncio.run(main())
In the lesson you can change this and run it.

Each line is one event, and the last is the [DONE] marker. The words joined back together are the reply, built piece by piece, as a browser would build the bubble's text.

Stop when the client leaves

Generating tokens nobody will read costs money. In ASGI the server learns that the client left by receiving a message of type http.disconnect. The generator cannot wait for it, because it has tokens to send, so it polls: between tokens it asks receive with a timeout of almost zero. The cell fakes a client that leaves after the third piece, and counts what the app sent.

Uses the setup above, 18 lines.

import asyncio

async def client_gone(receive):
    try:
        message = await asyncio.wait_for(receive(), timeout=0.001)
    except asyncio.TimeoutError:
        return False
    return message["type"] == "http.disconnect"

async def app(scope, receive, send):
    await send({"type": "http.response.start", "status": 200,
                "headers": [(b"content-type", b"text/event-stream")]})
    for word in token_stream(max_new=10):
        if await client_gone(receive):
            return
        body = f"data: {json.dumps({'delta': word})}\n\n".encode()
        await send({"type": "http.response.body", "body": body, "more_body": True})

async def main():
    gone = asyncio.Event()
    bodies = []
    first = True

    async def receive():
        nonlocal first
        if first:
            first = False
            return {"type": "http.request", "body": b"", "more_body": False}
        await gone.wait()
        return {"type": "http.disconnect"}

    async def send(message):
        if message["type"] == "http.response.body":
            bodies.append(message["body"])
            if len(bodies) == 3:
                gone.set()          # the client closes the tab

    await app({"type": "http"}, receive, send)
    print("pieces sent before the client left:", len(bodies))

asyncio.run(main())

The app would have sent up to ten words. It sent three, because the fourth check saw the disconnect and returned, so the generator stopped and no more tokens were made. In Starlette the same check is await request.is_disconnected(), and a StreamingResponse also cancels its generator when the connection closes.

Where this shows up in an engineer's week

A streaming service fails in ways a plain one does not. A buffering proxy holds the whole reply and releases it at the end, so the stream looks broken; the cure is headers such as Cache-Control: no-cache, a heartbeat, and telling the proxy not to buffer. A client that leaves keeps a worker busy; the check between tokens is the fix. An error halfway cannot change a status that has already been sent, so the stream ends with an event: error and the client must handle it. Goes deeper in: Serving a model behind FastAPI adds a concurrency cap and a token budget in front of a stream, Capstone: a streaming chat service builds the whole service, and One request through a serving stack: latency and cost shows what the GPU does between tokens.

Quick check
What does this print?
bodies = []
for word in ["The", "River", "tent"]:
    if len(bodies) == 2:
        break
    bodies.append(word)
print(len(bodies), bodies[-1])
  1. 2 The
  2. 3 River
  3. 3 tent
  4. 2 River
Show the answer

2 River

The check before each word stops the loop once two have been sent, so the third is never appended. The disconnect check in the cell works the same way: it runs before each piece.

In your own words
In your own words: why must a streaming server check for a disconnect between tokens, and what ends up costing money if it does not?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

What marks the end of one server-sent event?

  1. A blank line
  2. A closing brace
  3. The word DONE
  4. A change of content type

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 8 · about 23 hours of study

APIs

Design, build, secure and scale web APIs from first principles 27 lessons 1 free
  1. HTTP from the wire up FreeWhat really travels between client and server: bytes, headers and bodies.40 min
  2. API stylesREST, RPC, GraphQL, gRPC and streaming: what each is for.35 min
  3. Build an API server from scratchRouting, handlers and responses, with no framework.45 min
  4. A web framework, from the insideDecorators, middleware and error handling: how Flask and FastAPI work.40 min
  5. Designing a REST APIResources, methods, status codes, errors and pagination done well.45 min
  6. Validating requests with schemasCheck every request against a JSON Schema and say exactly what's wrong.45 min
  7. A database-backed APICRUD endpoints on SQLite, with safe concurrent updates.45 min
  8. Authentication and authorizationAPI keys, sessions, tokens and OAuth: who's calling, and what may they do?45 min
  9. Rate limits, retries and idempotencyKeep an API up under load and make retries safe.45 min
  10. Async HTTP clientsMany requests at once with asyncio and httpx: limits, timeouts and retries.45 min
  11. Webhooks and eventsPush events to other systems: signed, retried and deduplicated.45 min
  12. Documenting APIs with OpenAPIA machine-readable contract for docs, SDKs and AI tools.40 min
  13. Running APIs in productionVersioning, CORS, caching, request ids and health checks.45 min
  14. Project: a reviews APIAn authenticated, validated, paginated API with tests.45 min
  15. Web pages with FlaskRoutes, Jinja templates, a shared layout, links and error pages, tested with the test client.45 min
  16. Forms and redirectsGET and POST, validation messages, post/redirect/get, flash messages and CSRF tokens.45 min
  17. Accounts and loginsPassword hashing with bcrypt, sessions, pages that need a login, and safe logout.45 min
  18. Project: a trail log web appA multi-page app with a database, accounts, forms and tests.45 min
  19. JavaScript for Python programmersVariables, functions, arrays and objects mapped from Python, and the traps: truthy empty arrays, for...in and string sort.35 min
  20. Async JavaScript: promises, await and fetchThe event loop, async and await, fetch with res.ok checks, Promise.all, and timeouts with AbortController.35 min
  21. Modules, Node and reading TypeScriptimport and export, package.json, classes, and types as notes the compiler checks, read through a small tool server.30 min
  22. A chat front end in the browserThe DOM and events, model text kept out of innerHTML, a streamed reply parsed from server-sent events, and a Stop button.40 min
  23. APIs with FastAPITyped routes, pydantic models, automatic validation and docs, tested with httpx.45 min
  24. Streaming from the serverServer-sent events written by a server: an ASGI app that sends a body in pieces, FastAPI's StreamingResponse, heartbeats, and what a closed connection looks like.40 min
  25. Project: a FastAPI service with accountsA session per request, bcrypt sign-ups, signed tokens, ownership checks and tests on a fresh database.45 min
  26. Serving a model behind FastAPILoad once with lifespan, a typed predict route, a batch route, a cap on work in flight, 429 with Retry-After, a readiness route and a 202 job.45 min
  27. Capstone: a streaming chat serviceA FastAPI service with key auth, a per-user bucket, a token budget, request ids and an event-stream reply, tested on fresh state.45 min
Part of a free course 127 lessons

Free Python developer course

Go from your first line of code to working as a Python developer: projects, testing, data, databases, web apps and APIs, all in your browser.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write