Index Ai Academy
Phase 11 · Shipping

Incident lab: from the page to the postmortem

One incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers8

Why it matters

An incident tests a team's habits at the worst time. The habits are simple to state and hard to keep: stop the harm before finding the cause, say plainly what is happening, and write down what the system allowed so it cannot happen again. This lab runs one incident from the page to the postmortem, with the numbers, so each habit has something to practice on.

What you'll be able to do

  • Run an incident in the right order: acknowledge, mitigate first, then find the cause, then write it down
  • Work out an incident's impact in failed requests and in the share of an error budget it spent
  • Write a blameless postmortem with impact, a timeline, contributing factors and owned actions
  • Write a status update and a runbook entry for the same incident
Quick check
A guess before you start. Right after a bad deploy, the error ratio is high and the page has just fired. What is the best first step?
  1. Wait five minutes to see whether it passes
  2. Write the postmortem now, while it is fresh
  3. Roll back the last change, then look for the cause
  4. Read the code of the new version until you find the bug
Show the answer

Roll back the last change, then look for the cause

Customers are failing now. A rollback ends that in minutes, and the cause can be studied afterward with no one waiting.

The practice: one incident, in order

You met error budgets and on-call in Reliability and incidents, and the numbers behind an alert in Metrics, dashboards and alert rules. This lab puts them to work on one incident.

An incident goes through the same stages every time: detect (an alert fires), respond (a person acknowledges and takes charge), mitigate (the harm stops), resolve (the service is healthy again) and review (a postmortem). The picture runs one from the deploy to the all-clear. Step through it, and watch the roles: one person is the incident commander for the whole time, and a second engineer does the rollback.

Try it now

Step through the events. The follow bars show how fast the error budget burns while the incident runs.

Quick check
From Reliability and incidents: an alert on the error ratio fires. Which of these is the usual order for the response?
  1. Write it down, find the cause, mitigate, acknowledge
  2. Acknowledge, mitigate, find the cause, write it down
  3. Find the cause, mitigate, acknowledge, write it down
  4. Mitigate, write it down, acknowledge, find the cause
Show the answer

Acknowledge, mitigate, find the cause, write it down

Someone takes the page, the harm is stopped, the cause is studied once things are safe, and the postmortem records it.

Quick check
What does this print: the minutes to detect, to acknowledge and to mitigate?
start, alert, page, ack, fix = 7, 12, 12, 14, 20
print(alert - start, ack - page, fix - page)
  1. 7 2 8
  2. 5 2 6
  3. 12 2 6
  4. 5 2 8
Show the answer

5 2 8

Detect is the alert (12) minus the start of the failures (7): 5. Acknowledge is 14 − 12 = 2. Mitigate is the fix (20) minus the page (12): 8.

One strong postmortem and one weak one

blankWeakStrong
Impact"Some users had errors.""8,460 requests failed over 23 minutes (5% of traffic at the worst). 3.3% of the month's error budget."
Timeline"We found it and fixed it.""00:00 deploy v42. 00:12 alert. 00:14 acknowledged. 00:20 rolled back. 00:30 resolved."
Cause"Alex deployed a bad change.""v42 queried a feature table that was not yet populated in production. Nothing checked for that before release."
Actions"Be more careful.""Add a release check that the table has rows (owner: Dee, due 2026-12-04)."
ToneNames a personDescribes the system and the gap

A strong postmortem has numbers in the impact, times in the timeline, a cause that names a missing safeguard and not a person, and actions with an owner and a date.

Quick check
Which of these is the strongest postmortem action?
  1. Remind the team about the rollback command in the next standup, and thank the on-call
  2. Improve monitoring
  3. Be more careful with deploys
  4. Add a release check that the table has rows (owner: Dee, due 2026-12-04)
Show the answer

Add a release check that the table has rows (owner: Dee, due 2026-12-04)

It says what changes, who owns it and by when, and it prevents this specific failure.

The numbers an incident report needs

A postmortem is mostly arithmetic done carefully. Impact is how many requests failed, over how many minutes. Budget spent is those failures as a share of the failures the target allowed for the month. Time to detect is from the start of the failures to the alert, time to acknowledge from the page to a person, and time to mitigate from the page to the end of the harm. The ladder does each.

Who does what

A good incident has a few clear roles. The incident commander owns the whole response: they decide, and they say out loud what is happening. They do not have to be the best engineer present. A responder does the hands-on work, such as the rollback. A communicator writes the updates for customers and the rest of the company, so the commander is not interrupted. On a small team one person may wear two hats, but the commander and the person typing commands should not be the same person during a serious incident: someone has to watch the whole picture.

Keep a running note in a shared place, with a time on every line. It becomes the timeline of the postmortem, and it is far more accurate than anyone's memory the next day.

In your own words
In your own words: why does the team roll back before it knows the cause?

Write a sentence or two, then compare it with a good answer in the lesson.

Your turn: decide, then write it down

Three decisions matter in the lab. The checks below ask for each, and then you write the postmortem and the status update yourself, against a rubric you can see.

What makes a good action

A postmortem produces actions, and actions are where its value is. A weak action says "be more careful" or "improve monitoring". A strong one says what changes, who owns it and when, and it prevents or shortens a specific part of this incident. There are three kinds. Prevent actions stop the failure from happening, such as a release check that the table has rows. Detect actions shorten the time before anyone knows, such as an alert on the feature lookup. Mitigate actions shorten the time to stop the harm, such as a one-command rollback that has been practiced. A good postmortem has at least one prevent action, because faster detection alone leaves the same failure waiting for next time.

Quick check
At minute 16 the team knows only that v42 was the last change. Which mitigation is best?
  1. Hot-fix v42 and deploy it again
  2. Scale the service to twice the machines
  3. Roll back to v41
  4. Turn off the alert
Show the answer

Roll back to v41

A rollback goes to a version known to work. A hot-fix is untested, more machines do not fix a broken lookup, and silencing the alert hides the problem.

Quick check
When should the incident be called resolved?
  1. When the on-call person goes home
  2. When the error ratio has been back under its threshold for a set time
  3. As soon as the rollback command finishes running and the deploy page shows green
  4. When the postmortem is written
Show the answer

When the error ratio has been back under its threshold for a set time

A rollback finishing is not the same as the service being healthy. Watch the metric stay normal, here for ten minutes.

Quick check
What does this print: the share of a 259,200-request budget that 8,460 failures spent, as a percent?
failed, allowed = 8460, 259200
print(round(failed / allowed * 100, 1))
  1. 33.0
  2. 3.3
  3. 0.033
  4. 30.6
Show the answer

3.3

8,460 / 259,200 is 0.0326, which is 3.3 percent of the budget.

Worked example

Worked example: the incident in numbers

Every number a postmortem needs comes out of the event times and the error ratios. Run them and read the report.

events = {"start": 7, "alert": 12, "page": 12, "ack": 14, "fix": 20, "resolve": 30}
ratios = [0.002, 0.03, 0.08, 0.09, 0.07, 0.01, 0.002]
per_step, per_minute, target = 30_000, 6000, 0.999
failed = round(sum(r * per_step for r in ratios[:-1]))
allowed = per_minute * 30 * 24 * 60 * (1 - target)
print("time to detect:", events["alert"] - events["start"], "min")
print("time to acknowledge:", events["ack"] - events["page"], "min")
print("time to mitigate:", events["fix"] - events["page"], "min")
print("failed requests:", failed)
print("share of the month's budget:", round(failed / allowed * 100, 1), "percent")
It runs right here in your browser, with nothing to install.

The team found out five minutes after the failures began, someone took the page in two, and the rollback finished eight minutes after the page. 8,460 requests failed, which is 3.3% of the month's allowance. Those numbers belong in the Impact section of the postmortem, and they are what the actions should move: the five-minute detection is the first thing to shorten.

Drill

Count the failed requests

Write failed_requests(ratios, per_step). ratios is the error ratio at the start of each step, and per_step is the number of requests in each step. The last ratio is a reading after the incident, so it does not count as a step. Return the failed requests, rounded to a whole number.

def failed_requests(ratios, per_step):
    return 0


print(failed_requests([0.002, 0.03, 0.08, 0.09, 0.07, 0.01, 0.002], 30_000))
In the lesson, your code is checked as you write it.
Drill

How much of the month's budget did it spend?

Write budget_used(failed, requests_per_minute, target). The budget is the failures the target allows over 30 days: all requests in 30 days times 1 - target. Return the failed requests as a percent of that budget, rounded to 1 place.

def budget_used(failed, requests_per_minute, target):
    return 0.0


print(budget_used(8460, 6000, 0.999))
In the lesson, your code is checked as you write it.
Written brief · Postmortem

Write the postmortem

Version 42 of the scoring service was deployed at 00:00 and queried a feature table that was not populated in production. The error ratio passed 5% by minute 7, the alert fired at minute 12, on-call rolled back at minute 20 and the incident ended at minute 30. 8,460 requests failed. Write the postmortem.

It must

  • Gives the failed requests and the minutes affected
  • Lists at least four events with times
  • Names a missing safeguard, not a person
  • Gives every action an owner and a date

Go further

  • Says what went well

The template

## Impact
Who was affected, by how much, for how long.

## Timeline
The events, each with a time.

## Contributing factors
What in the system allowed it. No names.

## Actions
Each with an owner and a due date.
In the lesson, you write over it, 120 to 300 words, and the tutor can mark it.
In your own words
A draft postmortem says "the engineer deployed without checking". What would you change, and why?

Write a sentence or two, then compare it with a good answer in the lesson.

On the job: the runbook, the status page and the habit

Two documents outlive the incident. A status update is for customers: what they see, what you have done and when you will say more, in words with no internal names. A runbook entry is for the next person on call at 3 a.m.: the symptom, how to confirm it, the first safe step and who to call.

For this incident, the runbook entry says: symptom error ratio above 5% soon after a deploy; confirm the last deploy time against the start of the failures on the dashboard; first step roll back to the previous version with one command; escalate to the service's owner if the ratio does not fall within ten minutes of the rollback.

Written brief · Status update

Write the status-page update

It is minute 22 of the incident. The rollback is done and the error ratio is falling but is not yet back to normal. Write the update for the public status page.

It must

  • Says what customers see
  • Says what has been done
  • Gives the time of the next update

Go further

  • Avoids internal names

The template

## What is happening
In plain words, for a customer.

## What we are doing
The current step.

## What to expect
The next update, and when.
In the lesson, you write over it, 40 to 120 words, and the tutor can mark it.

Goes deeper in

In your own words
What goes in a runbook entry, and who is it for?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 1 code example you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

What should a postmortem's contributing factors describe?

  1. What in the system allowed the failure
  2. Which person made the mistake
  3. How long the team worked on it
  4. Which customers complained

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write