Async HTTP clients

Limits: be a good client

Starting a thousand requests at once is rude to the server and usually counterproductive: it answers 429 Too Many Requests, or slows down for everyone. Most APIs document a limit, like "at most 5 requests at a time". An asyncio.Semaphore enforces one in your own code: it holds a number of permits, and async with sem: waits until a permit is free.

import asyncio, time

in_flight = 0
busiest = 0

async def fetch(i, sem):
    global in_flight, busiest
    async with sem:
        in_flight += 1
        busiest = max(busiest, in_flight)
        await asyncio.sleep(0.1)  # the request
        in_flight -= 1
        return i * i

async def main():
    sem = asyncio.Semaphore(3)
    start = time.perf_counter()
    results = await asyncio.gather(*(fetch(i, sem) for i in range(12)))
    print(results)
    print(f"at most {busiest} at once, {time.perf_counter() - start:.2f}s")

asyncio.run(main())

Twelve requests of 0.1 seconds, three at a time, take about 0.4 seconds: four rounds. All twelve tasks start at once, but only three hold a permit; the rest wait at async with sem without using any resources. Create the semaphore inside the async code (here in main), so it belongs to the running event loop.

Timeouts and failures

In a batch of 200 requests, something will go wrong: a timeout, a 500, a refused connection. Decide what one failure should do to the rest:

• asyncio.gather(*tasks) raises the first exception, and you lose the other results.
• asyncio.gather(*tasks, return_exceptions=True) puts exceptions in the result list in place of values, so you can report each failure and keep every success.
• async with asyncio.TaskGroup() as tg: (Python 3.11 and later) cancels the remaining tasks when one fails. That's right when the results only make sense together, like the pieces of one page.

Often the clearest design is a small function that fetches one item and never raises: it catches the errors it expects and returns a value or an error message. Then a plain gather collects everything.

For a deadline on the whole batch, wrap it in async with asyncio.timeout(30):, which cancels whatever is still running after 30 seconds and raises TimeoutError.

Retries with backoff

Some failures are temporary, and the same request a moment later succeeds. Retry timeouts, connection errors, 429 and 502, 503, 504. Don't retry most other 4xx errors: a 404 or a 422 will fail the same way every time. Be careful with POST: retrying a payment that actually went through charges twice, unless the API supports idempotency keys (the reliability lesson).

Wait longer before each retry, exponential backoff: 0.5 s, then 1 s, then 2 s, so a struggling server gets room to recover. Add a little random jitter, so a thousand clients that failed together don't all retry in the same instant. If a 429 or 503 reply carries a Retry-After header, wait that long instead. And always cap the number of attempts.

import asyncio
import httpx

async def services(request):
    name = request.url.host.split(".")[0]
    if name == "slow":
        await asyncio.sleep(1)
    if name == "down":
        return httpx.Response(503)
    if name == "gone":
        raise httpx.ConnectError("name not resolved", request=request)
    return httpx.Response(200, json={"status": "ok"})

async def check(client, name):
    try:
        async with asyncio.timeout(0.3):
            reply = await client.get(f"https://{name}.example/health")
        reply.raise_for_status()
        return f"{name}: up"
    except (TimeoutError, httpx.TimeoutException):
        return f"{name}: too slow (no answer in 0.3s)"
    except httpx.HTTPStatusError as e:
        return f"{name}: answered {e.response.status_code}"
    except httpx.RequestError as e:
        return f"{name}: unreachable ({e})"

async def main():
    async with httpx.AsyncClient(transport=httpx.MockTransport(services)) as client:
        names = ["shop", "slow", "down", "gone", "search"]
        async with asyncio.timeout(5):
            for line in await asyncio.gather(*(check(client, n) for n in names)):
                print(line)

asyncio.run(main())

Async code outside the academy

The cells here end with asyncio.run(main()): one call that starts the event loop, runs main and closes the loop. A program should call it once, at the top. Inside a Jupyter notebook a loop is already running, so there you write await main() directly in a cell instead.

In a FastAPI app, the event loop belongs to the server, so an async def endpoint simply awaits an async client, which the app usually creates once at startup and closes at shutdown. And when a library you need has only blocking functions, await asyncio.to_thread(func, ...) runs it in a thread so the loop keeps going.