Free Python developer course
Go from your first line of code to working as a Python developer: projects, testing, data, databases, web apps and APIs, all in your browser.
Server-sent events written by a server: an ASGI app that sends a body in pieces, FastAPI's StreamingResponse, heartbeats, and what a closed connection looks like.
Reading a stream has come up three times: in Streaming responses, in API styles and in A chat front end in the browser. Writing one has not. A model takes seconds to write a reply, and a service that waits for the whole reply before it answers leaves the user looking at nothing. A service that sends each token as it is made shows the first word at once. It also has a duty the reading side never had: when the client leaves, the server must notice and stop, or it keeps paying for tokens that no one will read.
Send each token as it is made, instead of the whole reply at the end
The delay is waiting for the whole reply to be written. Sending each piece as it is made puts the first word on screen at once, though the reply takes as long to finish.
This rests on API styles, where you parsed an event stream, and Build an API server from scratch, where an ASGI app answered with send. A streamed reply is one ordinary HTTP response that does not end at once. The server sends the status and headers, with Content-Type: text/event-stream, and then keeps sending events. An event is a few lines ended by a blank line: data: {...}, optionally event: name before it. A line that starts with a colon is a comment, which parsers ignore, so a server uses it as a heartbeat to stop a proxy closing a connection that looks idle. There is no "end" marker in the format; the server just stops, or sends its own final event, such as data: [DONE].
The length is not known when the headers go out, so a streamed reply has no Content-Length, and a status line, once sent, cannot be changed. The other side of a stream is the one that goes wrong. The client can leave at any moment, by closing the tab or pressing Stop, and nothing tells your generator unless it asks. In ASGI the news is a message of type http.disconnect that receive returns. The picture shows what a user sees with and without streaming, for the same reply, and what it costs the server in work to do it: nothing more, since the tokens are made either way.
The reply takes the same time to finish. Streaming shows the first word at 400 ms; not streaming shows nothing until the end.
buffer = 'data: a\n\ndata: b\n\ndata: c' pieces = buffer.split("\n\n") tail = pieces.pop() print(len(pieces), repr(tail))
2 'data: c'
Two blank lines end two complete events. The last piece has no blank line after it, so it is the tail, kept for the next chunk.
Send an error event and end the stream
A status goes out once, before the first piece, and cannot be changed after. The server can only say something inside the stream, such as event: error, and then stop. The client must handle that event.
Write a sentence or two, then compare it with a good answer in the lesson.
An ASGI response is made of messages. The first, http.response.start, sends the status and headers. Each http.response.body carries bytes, and while its more_body is true the response is not finished. That is all streaming is at the lowest level. The cell runs a bare ASGI app that sends three events, with a fake receive and send, and prints each message it sent.
Each event is its own body message with more_body: True; a last empty message ends the response.
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import json import torch from tinygpt import load_pretrained model, tok = load_pretrained("tiny-gpt-chat") STOP = tok.token_id("<end>") PROMPT = "<user> Describe the River tent . <assistant>" def token_stream(prompt=PROMPT, max_new=10): """One token at a time: the words a small chat model writes, greedily.""" idx = torch.tensor([tok.encode(prompt)]) for _ in range(max_new): idx = model.generate(idx, 1, temperature=0) nxt = int(idx[0, -1]) if nxt == STOP: return yield tok.vocab[nxt]
import asyncio async def app(scope, receive, send): await send({"type": "http.response.start", "status": 200, "headers": [(b"content-type", b"text/event-stream")]}) for n in range(3): body = f"data: {n}\n\n".encode() await send({"type": "http.response.body", "body": body, "more_body": True}) await send({"type": "http.response.body", "body": b"", "more_body": False}) sent = [] async def fake_receive(): return {"type": "http.request", "body": b"", "more_body": False} async def fake_send(message): sent.append(message) asyncio.run(app({"type": "http", "path": "/"}, fake_receive, fake_send)) for m in sent: print(m["type"], m.get("body"), m.get("more_body"))
Five messages: the start, three events with more_body true, and an empty last body with more_body false that closes the response. A server writes each one to the socket as it gets it, which is why the client sees the events one by one.
FastAPI hides the messages. Return a StreamingResponse over a generator, sync or async, and it sends each value the generator yields as one body message. The cell serves a small app and reads it back with httpx, line by line. The generator is an ordinary async function with yield; the stepper after the cell shows what it does between yields.
Uses the setup above, 18 lines.
import asyncio import httpx from fastapi import FastAPI from fastapi.responses import StreamingResponse api = FastAPI() async def numbers(count: int): for n in range(count): yield f"data: {json.dumps({'n': n})}\n\n" await asyncio.sleep(0) yield "data: [DONE]\n\n" @api.get("/count") async def count(count: int = 3): return StreamingResponse(numbers(count), media_type="text/event-stream") async def main(): transport = httpx.ASGITransport(app=api) async with httpx.AsyncClient(transport=transport, base_url="http://t") as c: async with c.stream("GET", "/count", params={"count": 3}) as r: print(r.status_code, r.headers["content-type"]) lines = [line async for line in r.aiter_lines() if line] print(lines) asyncio.run(main())
def numbers(): for n in range(3): yield f"data: {n}\n\n" pieces = list(numbers()) print(len(pieces), "".join(pieces).count("\n\n"))
3 3
The generator yields three separate strings, one per event, and each ends with one blank line, so the joined text has three blank-line endings.
Write split_events(buffer): return (events, rest), where events is the list of complete events (each without its ending blank line) and rest is the unfinished tail. An event ends at a blank line, "\n\n".
Uses the setup above, 18 lines.
def split_events(buffer): return [buffer], "" print(split_events('data: a\n\ndata: b\n\ndata: c'))
Write with_heartbeat(events, every): given a list of event strings, return a new list with the comment ": ping\n\n" inserted after every every events. Do not add one after the last event if the list ends exactly on a multiple, only between events.
Uses the setup above, 18 lines.
def with_heartbeat(events, every): return list(events) print(with_heartbeat(['a', 'b', 'c', 'd'], 2))
Content-Type: text/event-stream, and no Content-Length
The type says "events", and there is no Content-Length because the server does not know how long the stream will run.
Write a sentence or two, then compare it with a good answer in the lesson.
The tiny chat model from the Phase 6 lessons is loaded once in this lesson's setup, and token_stream yields its words one at a time. Wrapped in a StreamingResponse, each word leaves the server as one event. Loading once matters as much as streaming: a route that loaded the model for each request would pay that cost every time (APIs with FastAPI introduced lifespan for it).
The chat model writes up to ten words greedily. Each becomes one data: event with a JSON delta, and the stream ends with [DONE]. The client reads them back line by line.
Uses the setup above, 18 lines.
import asyncio import httpx from fastapi import FastAPI from fastapi.responses import StreamingResponse api = FastAPI() async def chat_events(): for word in token_stream(): yield f"data: {json.dumps({'delta': word + ' '})}\n\n" await asyncio.sleep(0) yield "data: [DONE]\n\n" @api.get("/chat") async def chat(): return StreamingResponse(chat_events(), media_type="text/event-stream") async def main(): transport = httpx.ASGITransport(app=api) async with httpx.AsyncClient(transport=transport, base_url="http://t") as c: async with c.stream("GET", "/chat") as r: lines = [line async for line in r.aiter_lines() if line] deltas = [json.loads(line[6:])["delta"] for line in lines if "delta" in line] print(len(lines), lines[-1]) print("".join(deltas)) asyncio.run(main())
Each line is one event, and the last is the [DONE] marker. The words joined back together are the reply, built piece by piece, as a browser would build the bubble's text.
Generating tokens nobody will read costs money. In ASGI the server learns that the client left by receiving a message of type http.disconnect. The generator cannot wait for it, because it has tokens to send, so it polls: between tokens it asks receive with a timeout of almost zero. The cell fakes a client that leaves after the third piece, and counts what the app sent.
Uses the setup above, 18 lines.
import asyncio async def client_gone(receive): try: message = await asyncio.wait_for(receive(), timeout=0.001) except asyncio.TimeoutError: return False return message["type"] == "http.disconnect" async def app(scope, receive, send): await send({"type": "http.response.start", "status": 200, "headers": [(b"content-type", b"text/event-stream")]}) for word in token_stream(max_new=10): if await client_gone(receive): return body = f"data: {json.dumps({'delta': word})}\n\n".encode() await send({"type": "http.response.body", "body": body, "more_body": True}) async def main(): gone = asyncio.Event() bodies = [] first = True async def receive(): nonlocal first if first: first = False return {"type": "http.request", "body": b"", "more_body": False} await gone.wait() return {"type": "http.disconnect"} async def send(message): if message["type"] == "http.response.body": bodies.append(message["body"]) if len(bodies) == 3: gone.set() # the client closes the tab await app({"type": "http"}, receive, send) print("pieces sent before the client left:", len(bodies)) asyncio.run(main())
The app would have sent up to ten words. It sent three, because the fourth check saw the disconnect and returned, so the generator stopped and no more tokens were made. In Starlette the same check is await request.is_disconnected(), and a StreamingResponse also cancels its generator when the connection closes.
A streaming service fails in ways a plain one does not. A buffering proxy holds the whole reply and releases it at the end, so the stream looks broken; the cure is headers such as Cache-Control: no-cache, a heartbeat, and telling the proxy not to buffer. A client that leaves keeps a worker busy; the check between tokens is the fix. An error halfway cannot change a status that has already been sent, so the stream ends with an event: error and the client must handle it. Goes deeper in: Serving a model behind FastAPI adds a concurrency cap and a token budget in front of a stream, Capstone: a streaming chat service builds the whole service, and One request through a serving stack: latency and cost shows what the GPU does between tokens.
bodies = [] for word in ["The", "River", "tent"]: if len(bodies) == 2: break bodies.append(word) print(len(bodies), bodies[-1])
2 River
The check before each word stops the loop once two have been sent, so the third is never appended. The disconnect check in the cell works the same way: it runs before each piece.
Write a sentence or two, then compare it with a good answer in the lesson.
What marks the end of one server-sent event?
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Go from your first line of code to working as a Python developer: projects, testing, data, databases, web apps and APIs, all in your browser.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write