Caches, queues and background jobs

The hard parts

• Stale data. A cached value can be out of date until it expires, like an old number on your note. Choose TTLs by how stale is acceptable: seconds for stock levels, hours for a product description.
• Invalidation. When data changes, delete or update its cache entry. Missing one place means users see old data. Versioned keys (product:42:v7) make old entries unreachable.
• Stampedes. Picture a shelf running empty, and a thousand shoppers all heading for the stockroom at once. A popular entry expires, and a thousand requests miss at once and all hit the database. Fixes: let one request reload while others wait (request coalescing), add random jitter to TTLs so entries don't expire together, or refresh hot entries early.
• Keys. Include everything that changes the answer: user, language, version. A cache key without the user id can show one user's private data to another.

Caching AI calls

Model calls are slow and costly, so caching pays. Two layers:

• Prompt caching at the API: reuses the processed prefix of a request (Phase 1 and Phase 10).
• Response caching in your app: store the answer for an identical request, such as the same document summarized with the same prompt and model. Include the model and prompt version in the key, and don't cache replies that depend on the user unless the key includes the user.

Queues and workers

A busy kitchen keeps orders on a rail of tickets. The waiter adds a ticket and goes back to the guests, and the cooks take tickets as they're free. A queue holds jobs; producers add them and workers take them. Use one when work is slow (sending email, video processing, a batch of model calls), when it can fail and should retry, or when traffic comes in bursts the workers can smooth out.

A good queue gives each job a life cycle:

1. A worker pulls a job. The job becomes invisible to other workers for a visibility timeout, say 30 seconds.
2. If the worker finishes, it acknowledges the job, which removes it.
3. If it fails, or crashes and never acknowledges, the job becomes visible again after a delay and is retried.
4. After too many attempts, the job moves to a dead-letter queue for a person to look at, so one bad job can't retry forever. A dead-letter queue holds jobs that failed too many times, set aside for a person to inspect.

At-least-once delivery

A cook finishes a dish and then gets called away before marking the ticket done, so another cook makes it again. Because a crashed worker's job is retried, a job can run more than once: the worker may have done the work and crashed before acknowledging. Most queues promise at-least-once delivery, not exactly-once. So workers must be idempotent: running a job twice has the same effect as once. Record finished job ids, use idempotency keys with payment APIs, or write with upserts, which insert a row or update it if it's already there.

In plain words: assume every job can run twice, and make sure running it twice does no harm.

Common tools: Redis-based job libraries, Amazon SQS, RabbitMQ, and Kafka (a log that consumers read at their own pace).

def run(idempotent):
    charges, done = [], set()
    deliveries = [("job-1", "crash"), ("job-1", "ack")]   # delivered twice
    for job, then in deliveries:
        if idempotent and job in done:
            continue
        charges.append(job)
        done.add(job)
        if then == "crash":
            continue    # the acknowledgment never reaches the queue
    return charges

print("no check:", run(False))
print("with a check:", run(True))

from collections import OrderedDict

cache = OrderedDict()
capacity = 2
for key in ["a", "b", "a", "c"]:
    if key in cache:
        cache.move_to_end(key)
        print("hit ", key)
    else:
        print("miss", key)
        cache[key] = key.upper()
        if len(cache) > capacity:
            evicted, _ = cache.popitem(last=False)
            print("  evicted", evicted)
print(list(cache))

from collections import OrderedDict

cache = OrderedDict()
for key in ["a", "b", "c", "a", "d"]:
    if key in cache:
        cache.move_to_end(key)
    else:
        cache[key] = True
        if len(cache) > 3:
            cache.popitem(last=False)
print(list(cache))

Mistakes that reach production

Leaving the user out of a cache key. If /account is cached by its URL alone, the first person to load it fills the cache, and the next person sees their data. This has caused real and embarrassing leaks. Any personalized response needs the user in its key, or it shouldn't be cached at all.

Retrying forever. A job that always fails, perhaps because of a bad input, will keep going round and using up workers. After a few attempts, move it to a dead-letter queue, where a person can look at it.

A visibility timeout shorter than the job. A queue hides a job while a worker has it, for a set time. If the job takes longer than that, the queue assumes the worker died, and hands the job to another worker. Now two workers are doing it at once. Set the timeout well above your slowest job.