System design fundamentals

Start with requirements

Before designing a house, an architect asks how many people will live there and whether it has to survive hurricanes. Before drawing boxes, pin down two kinds of requirements:

• Functional: what it does. "Users upload a photo; followers see it in their feed."
• Non-functional: how well. Latency, meaning how long people wait ("feed loads in under 200 ms"), scale ("50 million daily users"), availability ("99.9% of the time"), consistency ("a like may take a few seconds to show"), cost and security.

The non-functional ones drive the design. A feed for 1,000 people is one server and a database. For 50 million, it's a different system. Ask, and write the answers down.

Rough numbers

Planning a party, you work out guests times slices before ordering pizza. Back-of-the-envelope math is the same habit, and it tells you which problems are real:

• Requests per second: daily users × requests per user ÷ 86,400 seconds. Peaks are often 2 to 3 times the average.
• Storage: items per day × size × how long you keep them.
• Servers: peak requests per second ÷ what one server handles, plus spares.

Know a few speeds by their order of size:

Think of memory as the pen on your desk, disk as a cabinet down the hall, and the network as a letter across town. Memory is about 1,000 times faster than disk, and the network is slower still. That's why caches exist, and why you avoid many sequential network calls on one request.

users = 200_000
requests_each = 25
peak_factor = 3
per_server = 500       # requests a second one server handles
order_size_kb = 2
orders_a_day = 8_000

average = users * requests_each / 86_400
peak = average * peak_factor
servers = int(peak // per_server) + 1 + 1   # round up, plus one spare

print("average requests a second:", round(average))
print("peak requests a second:", round(peak))
print("servers:", servers)
print("order storage a year (GB):", round(orders_a_day * order_size_kb * 365 / 1_000_000, 1))

The standard parts

Most systems are built from the same parts, the way most restaurants have a host, waiters, a ledger and a storeroom. The host at the door is your load balancer, sending each guest to a free waiter. Here's the full list:

Stateless servers are the key idea. Any waiter can serve any table if orders live in a shared system instead of one waiter's head. If any server can handle any request, because sessions live in the database or a cache, scaling is "add more servers behind the load balancer".

In plain words: add a part when you have the problem it solves, and not before.

reads = 2000     # reads a second

for hit_rate in [0, 0.5, 0.9, 0.99]:
    to_database = reads * (1 - hit_rate)
    print(f"hit rate {hit_rate:>4.0%}: {to_database:>6.0f} reads a second reach the database")

When one database isn't enough

First add read replicas, which are like photocopies of a master ledger. Replicas lag a little behind, so a user might not see their own write for a moment; read from the primary right after a write if that matters.

Next comes sharding: splitting data across several databases by a key, such as user id. A library might keep surnames A to H in one branch and I to Z in another. The simple way, hash(key) % n, has a catch: when n changes, almost every key moves. Consistent hashing places servers and keys on a ring, and each key goes to the next server clockwise. Adding a server then moves only about 1/n of the keys.

Availability math

Availability is the share of time a system works. Old holiday lights on one wire go dark when a single bulb fails. Parts in a chain, all needed, behave the same way, and their availabilities multiply: a 99.9% load balancer, 99.9% servers and a 99.9% database give 99.7%.

Carrying two flashlights is different. You're only in the dark if both die. Redundant copies, any one enough, fail only if all fail: two 99% servers give 1 − 0.01² = 99.99%.

So every part that's needed and has no backup, a single point of failure, caps the whole system. Find them, and add a spare where the cost is worth it.

Consistency trade-offs

Say two bank branches lose their phone line. They can refuse withdrawals until it's back, or keep paying out and risk mistakes. When the network between data centers breaks, a system must make the same choice: refuse some requests to stay consistent, or keep answering with possibly stale data to stay available. That's the CAP theorem in practice. Bank balances choose consistency. Like counts choose availability, with eventual consistency: copies agree soon, not instantly.

a = 0.99
print(round(1 - (1 - a) ** 3, 6))

import hashlib

def h(text):
    return int(hashlib.sha256(text.encode()).hexdigest()[:8], 16)

keys = [f"user-{i}" for i in range(1000)]
before = {k: h(k) % 4 for k in keys}
after = {k: h(k) % 5 for k in keys}
moved = sum(before[k] != after[k] for k in keys)
print(f"{moved} of 1000 keys moved")

Don't build for scale you don't have

You don't build a stadium for a backyard game. Ask how big it needs to be before designing anything. And never use Python's hash() to place data across servers: it changes each time Python starts.

Replicas lag

A photocopy can't include what was written after it was made. A replica may be a moment behind, so someone can save a change and not see it on their next read.