Index Ai Academy
Phase 11 · Shipping

Metrics, dashboards and alert rules

Counters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

A service that is up can still be failing: slow, erroring on one route, or running out of memory. Nobody can watch a log of a million lines, so the service keeps a few numbers instead and a system reads them every 15 seconds. This lesson builds those numbers from scratch: the three kinds of metric, a rate from a counter, a percentile from buckets, and an alert rule. The tools that do it for real, Prometheus and its query language, come after you have built the parts.

What you'll be able to do

  • Tell a counter, a gauge and a histogram apart, and say which one a question needs
  • Turn a counter into a rate over a window, and handle a counter that resets
  • Estimate a percentile from histogram buckets, and say what that estimate can and cannot tell you
  • Write an alert rule that fires on a sustained condition, with a threshold and a duration
Quick check
A guess before you start. A service keeps one number, the total requests served, which only goes up. It reads 12,000 at 12:00 and 15,000 at 12:05. How many requests a second did it serve in those five minutes?
  1. 15,000
  2. 600
  3. 3,000
  4. 10
Show the answer

10

The counter rose by 3,000 over 300 seconds: 10 a second.

The idea: three metric types and a window

A metric is a number a service keeps about itself, with a name and a few labels, such as http_requests_total{route="/score", status="200"}. A separate system scrapes it, which means it reads every metric on a fixed schedule, say every 15 seconds, and stores each reading with its time. There are three types, and each answers a different kind of question.

  • A counter only goes up: requests served, errors, bytes sent. It resets to zero when the process restarts.
  • A gauge goes up and down: requests in flight, memory used, queue length. It is the value right now.
  • A histogram counts observations into buckets: how many requests took up to 0.1 s, up to 0.25 s, and so on. It is how you get a percentile without storing every request.

A counter's value is rarely interesting. A service that has served four million requests since Tuesday tells you nothing about this minute. What matters is its rate: the increase between two readings, divided by the seconds between them. The picture shows it.

Try it now

A counter never goes down, so its value tells you little. Its slope, the rate, is the number you watch.

Quick check
From Metrics, logs and traces: what does this print?
import math


def percentile(values, p):
    ordered = sorted(values)
    return ordered[math.ceil(p / 100 * len(ordered)) - 1]


ms = [9, 8, 9, 10, 8, 9, 120, 9, 8, 10]
print(percentile(ms, 50), percentile(ms, 95), sum(ms) / len(ms))
  1. 9 10 20.0
  2. 20.0 120 9
  3. 9 120 9.0
  4. 9 120 20.0
Show the answer

9 120 20.0

The median of the ten latencies is 9, the 95th percentile is the slowest value, 120, and the mean is pulled up to 20.0 by that one slow request.

Three rules for reading a counter

  1. Use differences, not values. A rate is (new − old) / seconds. A counter that reads 4,900 says nothing; one that rose by 300 in a minute says 5 a second.
  2. A drop is a reset. The process restarted, and the counter began again from 0. A rate function treats the new reading as the increase since the restart, not as a negative number.
  3. Pick the window to match the question. A rate over one minute reacts fast and jumps around. A rate over an hour is smooth and slow. An alert usually looks at five minutes.
Quick check
A counter reads 100, 700, 1,300 at 0, 60 and 120 seconds. What does this print: the rate over the two minutes?
samples = [(0, 100), (60, 700), (120, 1300)]
(t0, v0), (t1, v1) = samples[0], samples[-1]
print((v1 - v0) / (t1 - t0))
  1. 10
  2. 1300.0
  3. 10.0
  4. 600.0
Show the answer

10.0

The counter rose by 1,200 over 120 seconds: 10.0 a second.

In your own words
In your own words: why do you watch a counter's rate and not its value, and what does a drop in a counter mean?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: the three metrics and a percentile from buckets

Everything below uses the Counter, Gauge, Histogram and quantile classes in the setup. They are about 40 lines in all, which is what makes the big libraries less mysterious: a counter is an integer, a gauge is a number, and a histogram is a list of integers.

The cell runs a thousand requests through them. Each request adds one to the counter, 1.2% of requests are errors, and each latency goes into the histogram's buckets: up to 0.1, 0.25, 0.5 and 1.0 seconds.

Setup (runs before every cell) · 51 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import math
import random


class Counter:
    def __init__(self):
        self.value = 0

    def inc(self, n=1):
        self.value += n


class Gauge:
    def __init__(self):
        self.value = 0

    def set(self, value):
        self.value = value


class Histogram:
    def __init__(self, bounds):
        self.bounds = bounds
        self.counts = [0] * (len(bounds) + 1)

    def observe(self, x):
        for i, bound in enumerate(self.bounds):
            if x <= bound:
                self.counts[i] += 1
                return
        self.counts[-1] += 1

    def cumulative(self):
        out, total = [], 0
        for c in self.counts:
            total += c
            out.append(total)
        return list(zip(self.bounds + [float("inf")], out))


def quantile(buckets, q):
    """The q-th quantile from cumulative buckets, as Prometheus estimates it."""
    total = buckets[-1][1]
    rank = q * total
    lower, below = 0.0, 0
    for bound, count in buckets:
        if count >= rank:
            if bound == float("inf"):
                return lower
            return lower + (bound - lower) * (rank - below) / (count - below)
        lower, below = bound, count
rng = random.Random(0)
requests, errors, inflight = Counter(), Counter(), Gauge()
latency = Histogram([0.1, 0.25, 0.5, 1.0])
for i in range(1000):
    requests.inc()
    if rng.random() < 0.012:
        errors.inc()
    latency.observe(rng.lognormvariate(-1.6, 0.7))
inflight.set(7)
print(requests.value, errors.value, inflight.value)
print(latency.cumulative())
print(
    round(quantile(latency.cumulative(), 0.5), 3),
    round(quantile(latency.cumulative(), 0.95), 3),
)

The counter shows 1,000 requests and 16 errors, an error ratio of 0.016. The histogram is stored as cumulative counts: 157 requests took 0.1 s or less, 615 took 0.25 s or less, 892 took 0.5 s or less, 983 took 1.0 s or less, and all 1,000 are counted in the last bucket. The median comes out near 0.212 s and the 95th percentile near 0.819 s.

The percentile is an estimate. The histogram knows that the 950th request fell between 0.5 s and 1.0 s, and the function assumes the requests in a bucket are spread evenly across it. A coarse bucket gives a coarse answer, so put bucket boundaries around the number you care about, such as your latency target.

Quick check
The first cell counted 16 errors in 1,000 requests. What does this print?
errors, requests = 16, 1000
print(errors / requests)
  1. 16
  2. 0.016
  3. 0.16
  4. 62.5
Show the answer

0.016

16 / 1,000 is 0.016, an error ratio of 1.6%.

Quick check
A histogram's cumulative buckets are (0.1, 600), (0.25, 900), (0.5, 980) and (inf, 1000). Which bucket holds the 950th request?
  1. The one up to 0.5 s
  2. The one up to 0.1 s
  3. The last one, up to infinity
  4. The one up to 0.25 s
Show the answer

The one up to 0.5 s

The cumulative count reaches 900 at 0.25 s, which is short of 950, and 980 at 0.5 s, which passes it. So the 950th request is in the bucket from 0.25 to 0.5 s.

Drill

A rate that survives a restart

Write rate(samples). samples is a list of (seconds, value) readings of one counter, in time order. Return the increase divided by the seconds from the first reading to the last. Between two readings, if the value went down, the counter restarted: count the new reading as the increase for that step.

Uses the setup above, 51 lines.

def rate(samples):
    return 0.0


print(rate([(0, 100), (60, 700), (120, 1300)]))
print(rate([(0, 9800), (60, 9900), (120, 150), (180, 750)]))
In the lesson, your code is checked as you write it.
Drill

A quantile from cumulative buckets

Write bucket_quantile(buckets, q). buckets is a list of (upper_bound, cumulative_count) in order, ending with (float('inf'), total). Find the first bucket whose cumulative count is at least q * total, and return lower + (bound - lower) * (rank - below) / (count - below), where lower and below are the previous bucket's bound and count (0 and 0 for the first). If the rank falls in the infinite bucket, return the last finite bound.

Uses the setup above, 51 lines.

def bucket_quantile(buckets, q):
    return 0.0


print(bucket_quantile([(0.1, 600), (0.25, 900), (0.5, 980), (float('inf'), 1000)], 0.9))
In the lesson, your code is checked as you write it.
Worked example

Worked example: an alert on a sustained error ratio

A rule that fires the moment one reading is high would page someone for every blip. A good rule needs the condition to hold for several scrapes in a row. This one fires when the error ratio is over 5% for three scrapes running.

Uses the setup above, 51 lines.

ratios = [0.004, 0.006, 0.06, 0.07, 0.08, 0.02]
threshold, needed = 0.05, 3
streak, fired_at = 0, None
for scrape, ratio in enumerate(ratios):
    streak = streak + 1 if ratio > threshold else 0
    print("scrape", scrape, "| ratio", ratio, "| streak", streak)
    if streak == needed and fired_at is None:
        fired_at = scrape
print("alert fired at scrape", fired_at)
It runs right here in your browser, with nothing to install.

The ratio first passes 5% at scrape 2, but the rule waits. It has been over for 1, then 2, then 3 scrapes in a row at scrape 4, so the alert fires there. A single high reading would have reset to 0 on the next low one. The wait costs two scrapes of delay and saves a page for every blip, which is the trade the duration in an alert rule sets.

A dashboard panel is one of these series drawn over time. This one shows the 95th percentile per minute, with a target line at 0.9 s, from ten minutes of simulated traffic that gets slower from minute 6:

Uses the setup above, 51 lines.

import matplotlib.pyplot as plt

rng = random.Random(1)
p95s = []
for minute in range(10):
    hist = Histogram([0.1, 0.25, 0.5, 1.0, 2.0])
    shift = 0.5 if minute >= 6 else 0.0
    for _ in range(300):
        hist.observe(rng.lognormvariate(-1.6 + shift, 0.7))
    p95s.append(round(quantile(hist.cumulative(), 0.95), 3))
plt.plot(range(10), p95s, marker="o")
plt.axhline(0.9, linestyle=":", color="gray")
plt.xlabel("minute")
plt.ylabel("p95 latency (s)")
plt.show()
print(p95s)

The series stays under the line until minute 6 and then crosses it and stays there. That crossing is what an alert on the percentile would catch.

Quick check
In the panel, from which minute does the 95th percentile stay above the 0.9 s line?
  1. Minute 4
  2. Minute 9
  3. Minute 6
  4. Minute 0
Show the answer

Minute 6

The p95 is 0.986 s at minute 6 and above 0.9 s for every minute after it. Minute 4 is 0.818 s, under the line.

In your own words
A rate jumps around from one scrape to the next. What would you change, and what would you give up?

Write a sentence or two, then compare it with a good answer in the lesson.

On the job: Prometheus, PromQL and an alert rule

In a real service these classes come from a client library, and a server called Prometheus scrapes them. Your service exposes its metrics as plain text at a path, usually /metrics: one line per series, with its labels and its current value, and a # TYPE line saying whether it is a counter, a gauge or a histogram. Prometheus reads that page on a schedule and stores each value with its time. A dashboard tool such as Grafana then draws queries over what it stored.

Where it comes up in a week

You are on call and the p95 panel for the scoring service has climbed. You ask three questions of the metrics: is the request rate up (a traffic change), is the error ratio up (a failure), and which bucket moved (a slow dependency or the model)? Then you open the alert rule behind the page and check that its threshold and its for: duration match what the team agreed, because a rule that is too twitchy or too slow is a bug in the monitoring, not in the service.

Goes deeper in

In your own words
An alert fires on every short spike and the team has started to ignore it. What do you change in the rule?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 3 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

Which metric type is right for "how many requests are being handled right now"?

  1. A gauge
  2. A counter
  3. A histogram
  4. A log line

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write