Reliability and incidents
SLOs, error budgets, alerts, on-call and blameless postmortems.
Create a free account Try part of it here
Why it matters
Every system fails sometimes. What separates professional teams is how they decide how reliable to be, how quickly they notice problems, and what they do next. Site reliability engineering turned this into a practice with numbers: targets everyone agrees on, alerts that only wake people for real problems, and a habit of learning from each failure without blame. This lesson teaches that practice.
What you'll be able to do
- Define SLIs and SLOs, and turn an SLO into an error budget
- Alert on how fast the error budget is burning, not on every metric that twitches
- Run an incident: declare it, assign roles, mitigate first, communicate
- Write blameless postmortems that lead to real fixes
What it covers
- SLIs, SLOs and SLAs
- Measuring an SLI
- Error budgets
- Alerts worth waking up for
- Page on two windows
- When something breaks
- Burn rate
- Why not 100%
SLIs, SLOs and SLAs
Think about driving. Your speedometer shows what's happening, you pick a speed to hold, and the law sets a limit with a fine attached. Reliability has the same three pieces:
- An SLI (service level indicator) is the speedometer: a measurement of what users feel, for example the share of requests that succeed in under 300 ms.
- An SLO (service level objective) is your target for it: 99.9% of requests are good, over 30 days.
- An SLA (agreement) is a promise to customers with penalties, usually looser than the SLO.
Pick SLIs from the user's side: success rate, latency, freshness of data. CPU use is not an SLI; users don't feel it unless it causes slow or failed requests.
Measuring an SLI
An SLI is a share of good events. Decide what counts as good, then count. Notice what the average would have told you for the same requests.
latencies = [120, 180, 95, 210, 140, 3000, 160, 130, 4200, 150] # ms errors = [False] * 9 + [True] good = sum(ms <= 300 and not bad for ms, bad in zip(latencies, errors)) print("good requests:", good, "of", len(latencies)) print("SLI:", good / len(latencies)) print("average latency:", sum(latencies) / len(latencies), "ms")
A count like this matches what users felt: 7 in 10 requests were fine. The average of over 800 ms describes a request nobody actually had, which is why SLOs are written as a share of requests under a limit.
Error budgets
An SLO of 99.9% means 0.1% of requests may fail. That 0.1% is the error budget, and it works like a monthly allowance that you spend on taking risks.
| SLO | downtime allowed per 30 days |
|---|---|
| 99% | 7.2 hours |
| 99.9% | 43 minutes |
| 99.99% | 4.3 minutes |
Each extra nine costs far more effort. And 100% is the wrong target: it would forbid any change. The budget turns arguments into a rule: while there's budget left, ship features; when it's spent, pause risky launches and work on reliability.
In plain words: some failure is allowed, so spend that allowance on shipping. When it's gone, fix things first.
A burn rate of 1 uses the budget exactly over 30 days. A burn rate of 10 uses it in 3 days, and that's worth waking someone up for.
Show the answer
10,000
0.1% of 10 million is 10,000.
Alerts worth waking up for
A smoke alarm that goes off for toast teaches people to pull out the battery. Every alert that pages a person should be urgent, real and actionable. Alerting on causes (high CPU, one server down) creates noise; people learn to ignore pages, and miss the real one. Alert on symptoms: the SLO is in danger.
The burn rate measures how fast you're spending budget, like checking how quickly your allowance is going. It's the error rate now divided by the error rate the SLO allows. A burn rate of 1 uses the budget exactly over the month. A burn rate of 14.4 would spend 2% of a month's budget in one hour. A common rule pages when both the last hour and the last 5 minutes burn faster than 14.4: the long window shows it matters, the short one shows it's still happening. Slower burns open a ticket instead of waking anyone.
Each alert should link to a runbook, like the emergency card in a hotel room: what it means, how to check, and the first steps to try.
Page on two windows
A short spike shouldn't wake anyone, and a long-ago problem shouldn't either. Alerting when both a long and a short window are burning fast avoids both false alarms.
def should_page(burn_1h, burn_5m, threshold=14.4): return burn_1h > threshold and burn_5m > threshold cases = { "brief spike, over already": (2.0, 30.0), "bad hour, now recovered": (20.0, 1.0), "bad hour, still happening": (20.0, 25.0), } for name, (long_window, short_window) in cases.items(): print(f"{name:<28} page: {should_page(long_window, short_window)}")
Only the third needs a person right now. The long window shows that it matters, and the short one shows that it's still going on. Alerts that fire for no reason teach people to ignore alerts, and that's when a real one gets missed.
When something breaks
A fire crew doesn't stop to debate how the fire started, and the chief isn't the one holding the hose. Incidents run the same way:
- Declare an incident early. It's cheap to downgrade a false alarm.
- Assign roles: an incident commander coordinates and decides; someone else does the hands-on work; someone else communicates to users and the company.
- Mitigate first. Stop the harm before finding the root cause: roll back, turn off the flag, fail over, add capacity.
- Communicate on a schedule, even "no news yet", on a status page and in one channel.
- Resolve, then write the postmortem.
Blameless postmortems
A postmortem records what happened and how to stop it happening again: impact, timeline, contributing factors, what went well, and action items with owners and dates.
Blameless means it asks how the system let this happen, not who to punish. Air travel got far safer partly because pilots can report mistakes without fear. "An engineer ran the wrong command" leads nowhere. "The command had no confirmation, ran against production by default, and the runbook was out of date" leads to three fixes. People share the truth when they're not afraid, and the truth is what you need.
For AI systems, include quality incidents: a model update or prompt change that starts producing bad answers is an outage too, even if every request returns 200. Monitor quality metrics, not only errors.
Show the answer
The incident commander, while others do the hands-on work and someone handles communication
One coordinator avoids people working at cross purposes. Separate roles keep fixing and communicating both going.
Burn rate
How fast is the budget going, compared with the pace that would last the month?
slo = 0.999 allowed_error_rate = 1 - slo for errors, requests in [(10, 10_000), (150, 10_000), (2, 10_000)]: rate = errors / requests burn = rate / allowed_error_rate print(f"{rate:.2%} errors -> burn rate {burn:.1f}")
1% errors against a 0.1% budget burns 10 times too fast; 1.5% burns 15 times too fast and would page.
slo = 0.995 errors, requests = 30, 1000 print(round((errors / requests) / (1 - slo), 1))
Show the answer
6.0
A 3% error rate against a 0.5% budget is 0.03 / 0.005 = 6.0, which burns it six times faster than the month can afford.
Why not 100%
Aiming for 100% sounds right, and it's a mistake. Users can't tell 100% from 99.99%, because their own Wi-Fi fails more often than that. And the last fraction costs a huge amount, while freezing every change you might make. An error budget makes the trade explicit: you can spend what's left on shipping.
Page on what users feel
A noisy engine doesn't matter if the car still drives well. High CPU isn't a problem if every request is still fast. And low CPU doesn't help if every request is failing. So alert on symptoms, such as error rate and latency, and leave causes for the dashboards you check once you're awake.
"Human error" isn't a root cause
If one person's typo can take down production, the problem is the system that allowed it. Ask what made the mistake easy and the damage large, and fix that. People stop reporting problems honestly if they get blamed for them. Then you lose the information you need most.
Practice what you just read
- 3 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 5 review cards for spaced repetition
Try a quiz question
Which is a good SLI?