Performance and profiling

The fixes, biggest first

1. A better algorithm or data structure. O(n²) to O(n) beats any amount of tuning. In the profile above, a Counter or a set fixes it.
2. Stop repeating work. Move calculations out of loops, and cache results you'll need again.
3. Batch your I/O. One trip to the store with a list beats 100 trips for one item each. One request for 100 rows beats 100 requests for one row. Network and disk round trips dominate (remember the speed ladder).
4. Overlap the waiting. Run independent I/O concurrently.
5. Use faster building blocks. NumPy, built-in functions and compiled libraries.
6. Only then consider a faster language for a hot loop.

The N+1 query problem

A classic case of number 3. A page lists 50 orders, and for each one the code runs another query for its customer: 1 query for the list plus N for the details. Each query is a round trip, like walking to the filing room once per order. So the page gets slower with every row. One query with a join, or one IN (...) query, does it in one or two trips:

import sqlite3
db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE customers (id INTEGER PRIMARY KEY, email TEXT);
CREATE TABLE orders (id INTEGER PRIMARY KEY, customer_id INTEGER, total REAL);
""")
db.executemany("INSERT INTO customers VALUES (?, ?)", [(i, f"c{i}@shop.test") for i in range(50)])
db.executemany("INSERT INTO orders VALUES (?, ?, ?)", [(i, i % 50, 10.0 + i) for i in range(50)])

queries = 0
def q(sql, args=()):
    global queries
    queries += 1
    return db.execute(sql, args).fetchall()

# N+1: one query for the orders, then one per order.
for oid, cid, total in q("SELECT id, customer_id, total FROM orders"):
    q("SELECT email FROM customers WHERE id = ?", (cid,))
print("N+1 version:", queries, "queries")

queries = 0
rows = q("SELECT o.id, c.email, o.total FROM orders o JOIN customers c ON c.id = o.customer_id")
print("join version:", queries, "query for", len(rows), "rows")

ORMs make N+1 easy to write by accident, because order.customer.email looks like an attribute but runs a query.

Caching

You keep the takeout menu on the fridge so you don't have to look it up every time. A cache keeps results you've already computed or fetched so the next request is instant. It works when the same requests come back often and the answer doesn't change too quickly.

Caches need a size limit, or they grow until memory runs out. The most common rule is LRU: when full, throw away the least recently used entry. Python has one built in:

from functools import lru_cache
import time

@lru_cache(maxsize=128)
def product_info(sku):
    time.sleep(0.05)              # pretend this is a slow API call
    return {"sku": sku, "price": 249}

start = time.perf_counter()
for _ in range(20):
    product_info("TENT-2")        # only the first call is slow
print(round(time.perf_counter() - start, 2), "seconds for 20 calls")
print(product_info.cache_info())

There are only two hard things in computer science, goes the old joke: cache invalidation and naming things. That menu on your fridge may have last year's prices. A cache can serve stale data after the source changes. Common answers are a TTL (time to live: each entry expires after so many seconds) or clearing entries whenever the data is written.

LLM apps cache at several levels: whole responses for repeated questions, embeddings for documents that haven't changed, and the provider's prompt caching, which reuses the work on a long, unchanged prompt prefix.

from functools import lru_cache

calls = 0

@lru_cache(maxsize=None)
def price(sku):
    global calls
    calls += 1
    return len(sku)

for s in ["a", "bb", "a", "a", "bb"]:
    price(s)
print(calls)

import sys

squares_list = [n * n for n in range(100_000)]
squares_gen = (n * n for n in range(100_000))

print("list:     ", sys.getsizeof(squares_list), "bytes (plus the numbers themselves)")
print("generator:", sys.getsizeof(squares_gen), "bytes")
print(sum(squares_gen) == sum(squares_list))

How to compare two versions fairly

Timing a single run tells you very little. Other programs are running on your computer, and small differences come and go. So run each version many times, and take the fastest or the median, since the slow runs are mostly noise from outside your code. If the difference is only a few percent, it's probably not real, and it's probably not worth more complicated code anyway.

Measure the whole request before you tune a small piece. Say a page takes 800 ms: 700 ms in the database, 95 ms in the model call, and 5 ms in your Python. Making that Python ten times faster saves under 5 ms, and nobody will ever notice. Fixing the query might save 600 ms.

Costs you can't see

One line can hide a query. With an object-relational mapper, order.customer.name looks like reading a field. It may run a database query, and inside a loop over 500 orders, that's 501 queries. This is called the N+1 problem, and it's among the most common causes of slow web apps. Your profiler will show the same call hundreds of times. The fix is to load the related rows in one query.

A cache without a plan is a bug in waiting. A cache makes things fast by serving old answers. Before adding one, ask two questions: how old may this answer be, and what happens when the data changes? A price that's five minutes stale may be fine. A permission that's five minutes stale may not be. If you can't answer both questions, don't cache it yet.

When to stop

Nobody polishes the underside of a table. Stop once it's fast enough for its users. Further tuning costs time and clarity, and nobody will notice.