Metrics, dashboards and alert rules
Counters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.
What it covers6
Why it matters
A service that is up can still be failing: slow, erroring on one route, or running out of memory. Nobody can watch a log of a million lines, so the service keeps a few numbers instead and a system reads them every 15 seconds. This lesson builds those numbers from scratch: the three kinds of metric, a rate from a counter, a percentile from buckets, and an alert rule. The tools that do it for real, Prometheus and its query language, come after you have built the parts.
What you'll be able to do
- Tell a counter, a gauge and a histogram apart, and say which one a question needs
- Turn a counter into a rate over a window, and handle a counter that resets
- Estimate a percentile from histogram buckets, and say what that estimate can and cannot tell you
- Write an alert rule that fires on a sustained condition, with a threshold and a duration
Show the answer
10
The counter rose by 3,000 over 300 seconds: 10 a second.
The idea: three metric types and a window
A metric is a number a service keeps about itself, with a name and a few labels, such as http_requests_total{route="/score", status="200"}. A separate system scrapes it, which means it reads every metric on a fixed schedule, say every 15 seconds, and stores each reading with its time. There are three types, and each answers a different kind of question.
- A counter only goes up: requests served, errors, bytes sent. It resets to zero when the process restarts.
- A gauge goes up and down: requests in flight, memory used, queue length. It is the value right now.
- A histogram counts observations into buckets: how many requests took up to 0.1 s, up to 0.25 s, and so on. It is how you get a percentile without storing every request.
A counter's value is rarely interesting. A service that has served four million requests since Tuesday tells you nothing about this minute. What matters is its rate: the increase between two readings, divided by the seconds between them. The picture shows it.
A counter never goes down, so its value tells you little. Its slope, the rate, is the number you watch.
import math def percentile(values, p): ordered = sorted(values) return ordered[math.ceil(p / 100 * len(ordered)) - 1] ms = [9, 8, 9, 10, 8, 9, 120, 9, 8, 10] print(percentile(ms, 50), percentile(ms, 95), sum(ms) / len(ms))
Show the answer
9 120 20.0
The median of the ten latencies is 9, the 95th percentile is the slowest value, 120, and the mean is pulled up to 20.0 by that one slow request.
Three rules for reading a counter
- Use differences, not values. A rate is (new − old) / seconds. A counter that reads 4,900 says nothing; one that rose by 300 in a minute says 5 a second.
- A drop is a reset. The process restarted, and the counter began again from 0. A rate function treats the new reading as the increase since the restart, not as a negative number.
- Pick the window to match the question. A rate over one minute reacts fast and jumps around. A rate over an hour is smooth and slow. An alert usually looks at five minutes.
samples = [(0, 100), (60, 700), (120, 1300)] (t0, v0), (t1, v1) = samples[0], samples[-1] print((v1 - v0) / (t1 - t0))
Show the answer
10.0
The counter rose by 1,200 over 120 seconds: 10.0 a second.
Write a sentence or two, then compare it with a good answer in the lesson.
In code: the three metrics and a percentile from buckets
Everything below uses the Counter, Gauge, Histogram and quantile classes in the setup. They are about 40 lines in all, which is what makes the big libraries less mysterious: a counter is an integer, a gauge is a number, and a histogram is a list of integers.
The cell runs a thousand requests through them. Each request adds one to the counter, 1.2% of requests are errors, and each latency goes into the histogram's buckets: up to 0.1, 0.25, 0.5 and 1.0 seconds.
Setup (runs before every cell) · 51 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import math import random class Counter: def __init__(self): self.value = 0 def inc(self, n=1): self.value += n class Gauge: def __init__(self): self.value = 0 def set(self, value): self.value = value class Histogram: def __init__(self, bounds): self.bounds = bounds self.counts = [0] * (len(bounds) + 1) def observe(self, x): for i, bound in enumerate(self.bounds): if x <= bound: self.counts[i] += 1 return self.counts[-1] += 1 def cumulative(self): out, total = [], 0 for c in self.counts: total += c out.append(total) return list(zip(self.bounds + [float("inf")], out)) def quantile(buckets, q): """The q-th quantile from cumulative buckets, as Prometheus estimates it.""" total = buckets[-1][1] rank = q * total lower, below = 0.0, 0 for bound, count in buckets: if count >= rank: if bound == float("inf"): return lower return lower + (bound - lower) * (rank - below) / (count - below) lower, below = bound, count
rng = random.Random(0) requests, errors, inflight = Counter(), Counter(), Gauge() latency = Histogram([0.1, 0.25, 0.5, 1.0]) for i in range(1000): requests.inc() if rng.random() < 0.012: errors.inc() latency.observe(rng.lognormvariate(-1.6, 0.7)) inflight.set(7) print(requests.value, errors.value, inflight.value) print(latency.cumulative()) print( round(quantile(latency.cumulative(), 0.5), 3), round(quantile(latency.cumulative(), 0.95), 3), )
The counter shows 1,000 requests and 16 errors, an error ratio of 0.016. The histogram is stored as cumulative counts: 157 requests took 0.1 s or less, 615 took 0.25 s or less, 892 took 0.5 s or less, 983 took 1.0 s or less, and all 1,000 are counted in the last bucket. The median comes out near 0.212 s and the 95th percentile near 0.819 s.
The percentile is an estimate. The histogram knows that the 950th request fell between 0.5 s and 1.0 s, and the function assumes the requests in a bucket are spread evenly across it. A coarse bucket gives a coarse answer, so put bucket boundaries around the number you care about, such as your latency target.
errors, requests = 16, 1000 print(errors / requests)
Show the answer
0.016
16 / 1,000 is 0.016, an error ratio of 1.6%.
Show the answer
The one up to 0.5 s
The cumulative count reaches 900 at 0.25 s, which is short of 950, and 980 at 0.5 s, which passes it. So the 950th request is in the bucket from 0.25 to 0.5 s.
A rate that survives a restart
Write rate(samples). samples is a list of (seconds, value) readings of one counter, in time order. Return the increase divided by the seconds from the first reading to the last. Between two readings, if the value went down, the counter restarted: count the new reading as the increase for that step.
Uses the setup above, 51 lines.
def rate(samples): return 0.0 print(rate([(0, 100), (60, 700), (120, 1300)])) print(rate([(0, 9800), (60, 9900), (120, 150), (180, 750)]))
A quantile from cumulative buckets
Write bucket_quantile(buckets, q). buckets is a list of (upper_bound, cumulative_count) in order, ending with (float('inf'), total). Find the first bucket whose cumulative count is at least q * total, and return lower + (bound - lower) * (rank - below) / (count - below), where lower and below are the previous bucket's bound and count (0 and 0 for the first). If the rank falls in the infinite bucket, return the last finite bound.
Uses the setup above, 51 lines.
def bucket_quantile(buckets, q): return 0.0 print(bucket_quantile([(0.1, 600), (0.25, 900), (0.5, 980), (float('inf'), 1000)], 0.9))
Worked example: an alert on a sustained error ratio
A rule that fires the moment one reading is high would page someone for every blip. A good rule needs the condition to hold for several scrapes in a row. This one fires when the error ratio is over 5% for three scrapes running.
Uses the setup above, 51 lines.
ratios = [0.004, 0.006, 0.06, 0.07, 0.08, 0.02] threshold, needed = 0.05, 3 streak, fired_at = 0, None for scrape, ratio in enumerate(ratios): streak = streak + 1 if ratio > threshold else 0 print("scrape", scrape, "| ratio", ratio, "| streak", streak) if streak == needed and fired_at is None: fired_at = scrape print("alert fired at scrape", fired_at)
The ratio first passes 5% at scrape 2, but the rule waits. It has been over for 1, then 2, then 3 scrapes in a row at scrape 4, so the alert fires there. A single high reading would have reset to 0 on the next low one. The wait costs two scrapes of delay and saves a page for every blip, which is the trade the duration in an alert rule sets.
A dashboard panel is one of these series drawn over time. This one shows the 95th percentile per minute, with a target line at 0.9 s, from ten minutes of simulated traffic that gets slower from minute 6:
Uses the setup above, 51 lines.
import matplotlib.pyplot as plt rng = random.Random(1) p95s = [] for minute in range(10): hist = Histogram([0.1, 0.25, 0.5, 1.0, 2.0]) shift = 0.5 if minute >= 6 else 0.0 for _ in range(300): hist.observe(rng.lognormvariate(-1.6 + shift, 0.7)) p95s.append(round(quantile(hist.cumulative(), 0.95), 3)) plt.plot(range(10), p95s, marker="o") plt.axhline(0.9, linestyle=":", color="gray") plt.xlabel("minute") plt.ylabel("p95 latency (s)") plt.show() print(p95s)
The series stays under the line until minute 6 and then crosses it and stays there. That crossing is what an alert on the percentile would catch.
Show the answer
Minute 6
The p95 is 0.986 s at minute 6 and above 0.9 s for every minute after it. Minute 4 is 0.818 s, under the line.
Write a sentence or two, then compare it with a good answer in the lesson.
On the job: Prometheus, PromQL and an alert rule
In a real service these classes come from a client library, and a server called Prometheus scrapes them. Your service exposes its metrics as plain text at a path, usually /metrics: one line per series, with its labels and its current value, and a # TYPE line saying whether it is a counter, a gauge or a histogram. Prometheus reads that page on a schedule and stores each value with its time. A dashboard tool such as Grafana then draws queries over what it stored.
Where it comes up in a week
You are on call and the p95 panel for the scoring service has climbed. You ask three questions of the metrics: is the request rate up (a traffic change), is the error ratio up (a failure), and which bucket moved (a slow dependency or the model)? Then you open the alert rule behind the page and check that its threshold and its for: duration match what the team agreed, because a rule that is too twitchy or too slow is a bug in the monitoring, not in the service.
Goes deeper in
- Reliability and incidents: SLOs, error budgets and burn-rate alerts built on these numbers.
- Incident lab: one incident from the page to the postmortem.
- Metrics, logs and traces: the three signals and when each is the right one.
- Monitoring a model: the same ideas applied to a model's inputs and scores.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 3 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 9 review cards for spaced repetition
Try a quiz question
Which metric type is right for "how many requests are being handled right now"?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 11 · about 34 hours of study
Professional practice
Design, ship and run real systems, work well in a team, and get hired
43 lessons 1 free
- System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
- Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
- Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
- Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
- CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
- Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
- Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
- Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
- Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
- Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
- Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
- Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
- A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
- Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
- Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
- Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
- Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
- Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
- Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
- Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
- Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
- A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
- Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
- Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
- Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
- Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
- Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
- Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
- Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
- Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
- Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
- A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
- What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
- Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
- Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
- Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
- Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
- Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
- Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
- Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
- Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
- Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
- Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write