Rate limits, retries and idempotency
Keep an API up under load and make retries safe.
Create a free account Try part of it here
Why it matters
Networks drop packets, servers restart and providers rate-limit you, all on an ordinary day. Model APIs add long, expensive calls and frequent 429 and 529 answers when demand peaks. What separates a fragile integration from a solid one is how it behaves when things go wrong. It needs limits that keep one client from swamping everyone, retries that help instead of making things worse, and retries that can't charge a customer twice.
What you'll be able to do
- Protect an API with a token-bucket rate limiter and answer 429 with Retry-After
- Set timeouts, and retry only failures that can succeed on a second try
- Space retries with exponential backoff and jitter
- Make POST requests safe to retry with idempotency keys
What it covers
- Rate limiting: fair shares
- Timeouts, always
- A backoff schedule
- Idempotency keys: retries that can't double-charge
- When a dependency is down
- How retries make outages worse
Rate limiting: fair shares
Without limits, one buggy script in a loop can take your API down for everyone, or run up your model bill overnight. A rate limiter caps how many requests each key, user or IP address may make.
The most common algorithm is the token bucket. Picture a jar of arcade tokens. You spend one per game, and an attendant drops a fresh one in every few seconds. If you've saved some up, you can play several games in a row. Once the jar is empty, you wait.
- Each client has a bucket holding up to
capacitytokens. It starts full. - Tokens drip back in at a steady
rateper second, never above capacity. - Each request spends one token. If the bucket is empty, the request is refused.
That allows short bursts (up to the capacity) while holding the long-run average to the rate.
In plain words: you can save up for a short burst, but over time you can only go as fast as the tokens come back.
class TokenBucket: def __init__(self, capacity, rate, now): self.capacity, self.rate = capacity, rate self.tokens, self.updated = capacity, now def allow(self, now): elapsed = now - self.updated self.tokens = min(self.capacity, self.tokens + elapsed * self.rate) self.updated = now if self.tokens >= 1: self.tokens -= 1 return True return False def retry_after(self): return max(0.0, (1 - self.tokens) / self.rate) bucket = TokenBucket(capacity=3, rate=1.0, now=0.0) # bursts of 3, then 1 per second for t in [0.0, 0.1, 0.2, 0.3, 0.4, 1.5, 1.6]: ok = bucket.allow(t) print(f"t={t:.1f}s", "allowed" if ok else f"429, retry after {bucket.retry_after():.1f}s")
A refused request gets 429 Too Many Requests with a Retry-After header, in seconds. It's the "back in 5 minutes" sign on a shop door, and well-behaved clients use it to know exactly when to return. Many APIs also send headers showing the limit and what's left, so clients can slow down before they hit it.
With several servers, the buckets must live in a shared store such as Redis, a fast in-memory database. Otherwise each server would hand out its own full allowance.
Show the answer
10
The bucket never holds more than its capacity, so a full bucket allows a burst of 10. The other 5 get 429s.
Timeouts, always
Nobody stays on hold forever, but a request without a timeout can. It hangs, ties up a worker and freezes your app. In the requests library the default is no timeout at all. Always pass one:
requests.post(url, json=body, timeout=(3.05, 60)) # 3 s to connect, 60 s to read the reply
Model calls can legitimately take a minute or more, so long read timeouts are normal there. The connect timeout stays short, because a server that won't even connect is down.
Retry only what can succeed
When a phone line is busy, you redial. When the number has been disconnected, redialing won't help. In the same way, retrying helps with temporary failures and only wastes time on permanent ones:
| retry? | responses |
|---|---|
| yes | connection errors and timeouts, 429, 502, 503, 504, and model APIs' 529 "overloaded" |
| usually | 500 (sometimes temporary) |
| no | 400, 401, 403, 404, 422: the same request will fail the same way |
Backoff and jitter
Retrying instantly hammers a server that's already struggling. So wait longer each time: exponential backoff doubles the delay (0.5 s, 1 s, 2 s, 4 s...) up to a cap.
That's not enough on its own. Picture a shop reopening after a power cut, with a thousand people outside, all pushing through the door at once. If a thousand clients failed at the same moment, they'd also retry at the same moments, in synchronized waves. That's the thundering herd.
Adding randomness, called jitter, spreads them out. With full jitter, each delay is a random amount between zero and the backoff value. If the server sent Retry-After, wait that long instead, since it knows better than your formula.
Each wait doubles. Without jitter, a thousand clients retry at the same instant and knock the service over again, so real code randomizes each delay.
A backoff schedule
The delays for five attempts, with and without jitter. The random generator is seeded so the output is repeatable.
import random base, cap = 0.5, 8.0 rng = random.Random(7) for attempt in range(5): backoff = min(cap, base * 2 ** attempt) jittered = backoff * rng.random() print(f"attempt {attempt + 1}: backoff {backoff:4.1f}s, with full jitter {jittered:4.2f}s")
The backoff doubles until it reaches the 8-second cap. Jitter picks a random point below it, so many clients that failed together come back at different times. The Anthropic and OpenAI SDKs do this for you, with a few automatic retries, and max_retries lets you change how many.
422 Unprocessable Content from an API. What should the retry logic do?Show the answer
Not retry: fix the request, because the same body will fail the same way
4xx errors other than 429 mean the request itself is wrong. Retrying only adds load and delay.
Idempotency keys: retries that can't double-charge
Here's the hard case. Your client POSTs "charge $50". The server charges the card, but the response is lost. The client sees a timeout. Did the charge happen? Retrying might charge twice. Not retrying might mean no charge. You've felt this on a checkout page that freezes after you press Pay.
The fix, popularized by Stripe, works like an order number on a ticket. However many times that ticket reaches the kitchen, that number gets one meal. The client makes a unique idempotency key for each logical operation, often a random UUID (a long id that's effectively never repeated), and sends it in a header:
POST /v1/charges Idempotency-Key: 8e03978e-40d5-43e8-bc93-6894a57f9324
The server remembers each key with the response it gave:
- New key: do the work, save the key with the response, return it.
- Same key, same body: don't do the work again. Return the saved response.
- Same key, different body: that's a client bug, so refuse it with an error.
In plain words: the key says "this is the same attempt as before", so the server hands back the first answer instead of doing the work twice.
Now retrying is always safe: the operation happens at most once, however many times the request arrives. Keys expire after a while, often 24 hours. Two requests with the same key arriving at the same moment need care too: the second should wait or get a 409 while the first is still running.
seen = set() charges = 0 for key in ["k1", "k1", "k2"]: if key not in seen: seen.add(key) charges += 1 print(charges)
Show the answer
2
k1 and k2 each charge once, and the repeat of k1 is skipped, so charges ends at 2. The retry of k1 does nothing, so the customer is only charged once for it.
When a dependency is down
Two more patterns keep failures from spreading:
- A circuit breaker works like the breaker in your home's fuse box. It notices that a dependency keeps failing and stops calling it for a while, returning a quick error or a fallback, then tries again later.
- Graceful degradation means doing something useful anyway, like a restaurant that's out of fish offering the chicken. If the recommendation model is down, show bestsellers. If the premium model is overloaded, fall back to a smaller one.
POST /payments and retries with the same Idempotency-Key. The first request did succeed. What should the server do?Show the answer
Return the stored result of the first request, without charging again
The key ties the retry to the first attempt. The client gets the same answer, and the customer is charged once.
How retries make outages worse
Retries feel like the responsible thing to do, and they can bring a service down. Think of a busy phone line where everyone hangs up and redials at once. Say a service is struggling, and 1,000 clients each retry 3 times, right away. It now gets 4,000 requests when it could barely handle 1,000. This is a retry storm, and it's how a small slowdown becomes a long outage.
Three rules keep retries safe. Wait longer each time, with some randomness so that clients don't move in lockstep. Stop after a few attempts. And don't retry errors that can't succeed: a 400 will fail the same way forever.
Never retry a POST without an idempotency key, or a timeout may charge someone twice. And always set a timeout, since a call with none can hang forever.
Practice what you just read
- 3 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 5 review cards for spaced repetition
Try a quiz question
What does a token bucket allow that a simple "N requests per minute" counter handles poorly?