Index Ai Academy
Phase 11 · Data engineering

Datasheets: documenting a dataset

The document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers7

Why it matters

A dataset outlives the work it was built for. A year on, someone finds the file, trains on it and ships a model, without knowing that every row came from one quarter, that one region hardly appears, or that the labels were one person's judgment. Nothing in the file says so. A datasheet is the short document that does. Half of it can be computed from the data and checked on every change, and the other half has to be written down by the people who know. This lesson builds both halves.

What you'll be able to do

  • List the questions a datasheet answers, and say which a program can answer and which only a person can
  • Profile a dataset by column and by group, and spot the group a model will know little about
  • Write a card's claims as data and check them against the data, so the card can't go stale unnoticed
  • Find rows that leak between a training and a test split, and split by group with a stable hash
  • Scan for personal data, and record a dataset's license and the uses it is and isn't fit for
Quick check
A guess before you start. A colleague is about to train on a dataset that someone else built last year. Which of these facts can't be recovered from the data file, however carefully it is analyzed?
  1. How many rows each of the classes contains
  2. What share of the rows has a missing value
  3. The earliest and the latest date among the rows
  4. Who assigned the labels, and what rule they followed
Show the answer

Who assigned the labels, and what rule they followed

Counts, ranges and missing values can all be computed from the file. How the labels were made can't: the file holds the labels, and nothing about the people, the guideline or the agreement behind them. If nobody wrote that down, it is gone.

The idea: what the file can't tell you

A datasheet answers the questions its next user will ask. Its numbers are counts by group, so this part uses Describing data and the share arithmetic of Percentages.

What the file can't tell you

A datasheet, also called a data card, is a short document that travels with a dataset and answers the questions its next user will have. The name comes from electronics, where every component ships with a sheet saying what it does and the conditions it was tested under.

Here is the dataset this lesson documents: support tickets exported from a help desk, each labeled urgent or routine.

Setup (runs before every cell) · 23 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import random, re, zlib

SUBJECTS = ["I can't log in", "I was charged twice", "The app crashes on start",
            "Please cancel my plan", "Where is my invoice", "The export is empty"]


def tickets(n, seed, months=("01", "02", "03"), regions=("eu", "us", "asia"),
            weights=(70, 25, 5), urgent=0.12):
    rng, rows = random.Random(seed), []
    for i in range(n):
        region = rng.choices(regions, weights=weights)[0]
        contact = f". Reach me at user{seed}{i}@example.com"
        customer = f"c{rng.randrange(300)}"
        created = f"2026-{rng.choice(months)}-{rng.randrange(1, 29):02d}"
        language = None
        if rng.random() >= 0.04:
            language = rng.choices(["en", "de"], weights=[85, 15])[0]
        text = rng.choice(SUBJECTS) + (contact if rng.random() < 0.03 else "")
        rate = 0.02 if region == "asia" else urgent
        label = "urgent" if rng.random() < rate else "routine"
        rows.append({"customer": customer, "created": created, "region": region,
                     "language": language, "text": text, "label": label})
    return rows
rows = tickets(2400, 0)
for row in rows[:3]:
    print(row)
print(len(rows), "rows")

From the file you can count rows and read values. You can't tell why it was collected, who labeled it, whether these customers agreed to this use, or what it shouldn't be used for. A datasheet holds both kinds of fact, in sections like these.

SectionThe question it answers
PurposeWhy was this made, and for which task?
CompositionWhat is in it: rows, columns, classes, groups, time span, missing values?
CollectionWhere did the rows come from, and what was left out?
LabelingWho assigned the labels, under what guideline, with what agreement?
SplitsHow were training and test rows separated, and can anything leak between them?
Uses and limitsWhat is it fit for, where is it known to be weak, what must it not be used for?
Personal data and licenseDoes it hold personal data, and on what terms may it be used?
MaintenanceWho owns it, and how are corrections made?

A model card, which you met in the churn capstone, does the same job for a trained model. The two belong together: a model card's section on training data should point at a datasheet.

Try it now

A program can count. Only a person knows why, how, and what for.

The half a program can write

Start with the half that costs nothing. A profile is a summary of every column: how much is missing, how many distinct values there are, the range, and the most common value. It is the composition section of the card, and it takes a dozen lines.

Uses the setup above, 23 lines.

rows = tickets(2400, 0)

def profile(rows, column):
    values = [row[column] for row in rows if row[column] is not None]
    counts = {}
    for v in values:
        counts[v] = counts.get(v, 0) + 1
    top = max(sorted(counts), key=counts.get)
    return {"missing": round(1 - len(values) / len(rows), 3), "distinct": len(counts),
            "min": min(
                values
            ), "max": max(values), "top": (top, round(counts[top] / len(values), 3))}

for column in ("customer", "created", "region", "language", "text", "label"):
    p = profile(rows, column)
    lo, hi = str(p["min"])[:12].ljust(12), str(p["max"])[:12].ljust(12)
    print(column.ljust(8), "| missing", str(p["missing"]).ljust(5),
          "| distinct", str(p["distinct"]).rjust(3),
          "| from", lo, "to", hi, "| most common", p["top"])

print()
print("region | rows | share of rows | urgent | share urgent")
for region in ("eu", "us", "asia"):
    part = [row for row in rows if row["region"] == region]
    urgent = sum(row["label"] == "urgent" for row in part)
    print(
        region.ljust(6), "|", str(len(part)).rjust(4), "|",
        str(round(len(part) / len(rows), 3)).ljust(13), "|", str(urgent).rjust(6), "|",
        round(urgent / len(part), 3),
    )

Read the first table as a new user would. The tickets run from 2026-01-01 to 2026-03-28: one quarter, so nothing here has seen a summer or a year end. The language is missing on 0.039 of the rows. Seven in ten tickets, 0.704, are from one region. And 2,400 tickets hold only 82 distinct texts, so the same sentences recur many times, which will matter when the data is split.

The second table is the one teams skip. An overall figure hides the groups inside it, so a profile should be broken down by every group the model will be used on. Here the asia region has 95 tickets, and not one of them is urgent. A model trained on this file has never seen an urgent ticket from that region. Whatever accuracy it reports overall, nobody knows how it behaves there.

That sentence goes in the card, under known limits. Finding it took a count per group.

Quick check
What does this print: the number of rows in each group, then the share of each group's rows that is missing a score?
rows = [
    ("north", 0.7), ("north", None), ("south", 0.4), ("north", 0.9), ("south", None),
    ("north", 0.2), ("south", None), ("north", 0.5),
]

for group in ("north", "south"):
    part = [score for g, score in rows if g == group]
    print(
        group, len(part), round(sum(s is None for s in part) / len(part), 2), end="  ",
    )
  1. north 5 0.8 south 3 0.33
  2. north 4 0.25 south 2 1.0
  3. north 5 0.38 south 3 0.38
  4. north 5 0.2 south 3 0.67
Show the answer

north 5 0.2 south 3 0.67

North has five rows with one score missing: 0.2. South has three rows with two missing: 0.67. Overall 3 of 8 scores are missing, 0.38, a figure that describes neither group. The breakdown shows the problem sits in the south.

In your own words
Why does a datasheet need facts that can't be computed from the data?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: a profile, claims as data and splits that don't leak

Everything computed here can be recomputed, which is the point: the card stays true because a program checks it.

Drill

Profile one column

Write profile(rows, column). rows is a list of dictionaries. A value is missing when it is None or the key isn't in the row. Return a dictionary with "rows" (the number of rows), "missing" (the share of rows missing a value, rounded to 3 places), "distinct" (the number of distinct values that aren't missing) and "top": a tuple of the most common value and its share of the values present, rounded to 3 places. If several values tie for most common, take the smallest. If every value is missing, or there are no rows, "top" is None and "missing" is 1.0 for rows that exist and 0.0 for no rows.

Uses the setup above, 23 lines.

def profile(rows, column):
    return {"rows": 0, "missing": 0.0, "distinct": 0, "top": None}


rows = [{"region": "eu"}, {"region": "us"}, {"region": None}, {"region": "eu"}, {}]
print(profile(rows, "region"))
In the lesson, your code is checked as you write it.

A card that can be checked

Documentation has a well-known way of failing: it is right on the day it is written, and the data moves on. A new batch is appended, a region is added, and the card still says what it said.

The defense is the one from the lesson on data quality. Write the card's factual claims as data, and let a program compare them with the dataset every time it changes. A card that disagrees with its data fails the build, the way a broken contract does.

Uses the setup above, 23 lines.

card = {                                     # the checkable part of the datasheet
    "rows": 2400,
    "created": ("2026-01-01", "2026-03-28"),
    "regions": ["asia", "eu", "us"],
    "urgent_share": 0.12,
    "max_missing": {"language": 0.05},
    "personal_data": "none",
}

def check(card, rows):
    problems = []
    if len(rows) != card["rows"]:
        problems.append(f"rows: the card says {card['rows']}, the data has {len(rows)}")
    dates = [row["created"] for row in rows]
    if (min(dates), max(dates)) != card["created"]:
        problems.append(
            f"created: the card says {card['created']}, the data runs from "
            f"{min(dates)} to {max(dates)}"
        )
    regions = sorted({row["region"] for row in rows})
    if regions != card["regions"]:
        problems.append(
            f"regions: the card says {card['regions']}, the data has {regions}"
        )
    urgent = sum(row["label"] == "urgent" for row in rows) / len(rows)
    if abs(urgent - card["urgent_share"]) > 0.01:
        problems.append(
            f"urgent_share: the card says {card['urgent_share']}, the data has "
            f"{round(urgent, 3)}"
        )
    for column, most in card["max_missing"].items():
        missing = sum(row[column] is None for row in rows) / len(rows)
        if missing > most:
            problems.append(
                f"{column}: the card allows {most} missing, the data has "
                f"{round(missing, 3)}"
            )
    emails = sum(
        1 for row in rows if re.search(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)+", row["text"])
    )
    if card["personal_data"] == "none" and emails:
        problems.append(
            f"personal_data: the card says none, {emails} rows contain an email address"
        )
    return problems

rows = tickets(2400, 0)
print("the card as first written:")
for problem in check(card, rows):
    print(" -", problem)

april = tickets(
    600, 1, months=("04",), regions=("eu", "us", "latam"), weights=(30, 20, 50),
    urgent=0.25,
)
print("after a new batch is appended:")
for problem in check(card, rows + april):
    print(" -", problem)

The check earned its keep before any data changed. The author wrote "personal data: none" in good faith, and 76 tickets contain an email address that a customer typed into the message. A claim nobody tests is a guess.

Then the April batch arrives, from a help desk that now serves a fourth region during a month with an outage. Five claims are wrong at once: the row count, the dates, the regions, the share of urgent tickets and the personal data. Without the check, the card would have gone on describing a dataset that no longer exists.

A failed check isn't an error to silence. It is a prompt: look at what changed, decide whether the new rows belong, then update the card and give the dataset a new version, as in Versioning data and models.

Quick check
A card says urgent_share: 0.12, checked with a tolerance of 0.01. What does this print for three versions of the data?
claimed, tolerance = 0.12, 0.01
for urgent, total in ((286, 2400), (310, 2400), (436, 3000)):
    share = urgent / total
    print(
        round(share, 3), "ok" if abs(share - claimed) <= tolerance else "stale",
        end="  ",
    )
  1. 0.119 ok 0.129 stale 0.145 stale
  2. 0.119 ok 0.129 ok 0.145 stale
  3. 0.119 ok 0.129 ok 0.145 ok
  4. 0.119 stale 0.129 stale 0.145 stale
Show the answer

0.119 ok 0.129 ok 0.145 stale

The first two shares are within 0.01 of the claim: 0.119 is off by 0.001 and 0.129 by 0.009. The third, 0.145, is off by 0.025, so the card is stale. A tolerance lets ordinary variation through and catches a real shift.

Quick check
A dataset's card check fails after a new batch is added: the batch contains a region the card doesn't list. What should happen next?
  1. Skip the check for this batch and re-enable it later
  2. Widen the tolerance until the check passes again
  3. Delete the failing claim from the card to unblock the build
  4. Review the batch, then update the card and the version together
Show the answer

Review the batch, then update the card and the version together

The check has done its job: it noticed that the dataset is no longer what the card describes. Someone has to decide whether a new region belongs in this dataset, and if it does, the card and the version change with it, so every model trained on the data can say which description applied.

Splits that leak

One section of the card deserves a check of its own: how the training and test rows were separated. A test set measures what a model does on data it hasn't seen. If the same thing sits on both sides of the split, the test measures memory.

Rows leak between splits in three ways.

  • Exact duplicates. The same row on both sides. The tickets above hold 82 distinct texts in 2,400 rows, so any random split shares nearly all of them.
  • Near duplicates. The same text with different capitals or spacing, or the same photo resized.
  • Shared groups. Different rows from the same customer, patient, author or session. They aren't copies, and they resemble each other far more than rows from strangers do.

The third is the easiest to miss. Below, 200 customers each write 10 tickets in a style of their own, and a model that predicts the label of the most similar training row is tested twice: on rows held out at random, and on customers held out whole. A last line gives the real answer, 200 customers nobody has seen.

Uses the setup above, 23 lines.

import numpy as np
from sklearn.neighbors import KNeighborsClassifier

def make(customers, per, seed):
    g = np.random.default_rng(seed)
    # each customer writes in a style of their own
    style = g.normal(size=(customers, 5))
    leaning = (style[:, 0] - style[:, 1] + g.normal(size=customers) > 0).astype(int)
    who = np.repeat(np.arange(customers), per)
    X = style[who] + g.normal(scale=0.15, size=(customers * per, 5))
    y = np.where(g.random(customers * per) < 0.95, leaning[who], 1 - leaning[who])
    return X, y, who

X, y, who = make(200, 10, 0)
# customers nobody has seen
X_new, y_new, _ = make(200, 10, 1)

g = np.random.default_rng(2)
by_row = np.zeros(2000, dtype=bool)
# 400 rows picked at random
by_row[g.permutation(2000)[:400]] = True
# the last 40 customers, whole
by_customer = who >= 160

for name, test in (("split by row", by_row), ("split by customer", by_customer)):
    shared = len(set(who[test]) & set(who[~test]))
    model = KNeighborsClassifier(n_neighbors=1).fit(X[~test], y[~test])
    print(
        name.ljust(17), "| customers on both sides:", str(shared).rjust(3),
        "| test accuracy:", round(float(model.score(X[test], y[test])), 3),
    )

model = KNeighborsClassifier(n_neighbors=1).fit(X, y)
print(
    "trained on all of it, scored on 200 new customers:",
    round(float(model.score(X_new, y_new)), 3),
)
It runs right here in your browser, with nothing to install.

Split by row, 178 customers have tickets on both sides, and the test reports 0.865. Split by customer, it reports 0.677. On customers the model has never met, it scores 0.695.

The row split overstated the model by 17 points. Nearly every test row had siblings from the same customer in training, so finding the right answer was a matter of recognizing the customer. The split by customer asks the question the product will ask, and lands within two points of the truth.

The rule: split by the unit you will meet new in production. If the model will serve new customers, whole customers go to one side.

Quick check
From the lesson on train and test splits: what is data leakage?
  1. A model's accuracy falling slowly after it has been deployed
  2. Rows being lost when a file is copied between two systems
  3. Personal data being published outside the company by mistake
  4. Information from the test data reaching the model during training
Show the answer

Information from the test data reaching the model during training

Leakage is anything that lets training see what the test was meant to hide: the test rows themselves, a statistic computed over them, or, as here, other rows from the same customer. The test score then overstates what the model will do on new data.

To split by group, don't shuffle. Shuffling gives a different split every time the data grows, and a customer who was in training last month can land in the test set this month. Derive the side from the group's own id with a hash: a function that turns the id into a number that looks random and is always the same for the same id.

Uses the setup above, 23 lines.

import zlib

for customer in ("c1", "c2", "c3", "c4", "c1"):
    # a number from 0 to 99, fixed for this id
    bucket = zlib.crc32(customer.encode()) % 100
    print(
        customer, "-> bucket", str(bucket).rjust(2), "->",
        "test" if bucket < 20 else "train",
    )

Each customer gets a bucket from 0 to 99, and buckets below 20 form a test set of about a fifth of the customers. The customer c1 gets bucket 57 both times it appears, and will get it next year too. New rows from an old customer join that customer's side, and nothing already assigned ever moves. Python's built-in hash() won't do here: for text it gives different numbers each time Python starts.

The card's section on splits then says three things: the unit of the split, the rule, and the result of a check that no unit appears on both sides.

Drill

Split by group with a stable hash

Write split_by_group(rows, key, test_percent). Each row is a dictionary, and row[key] names its group. A row goes to the test set when zlib.crc32(str(row[key]).encode()) % 100 is less than test_percent, and to the training set otherwise. Return (train, test), two lists that keep the rows in their original order.

Uses the setup above, 23 lines.

import zlib


def split_by_group(rows, key, test_percent):
    return rows, []


rows = [
    {"customer": c, "n": i} for i, c in enumerate(["c1", "c4", "c2", "c4", "c8", "c1"])
]
train, test = split_by_group(rows, "customer", 20)
print([r["n"] for r in train], [r["n"] for r in test])
In the lesson, your code is checked as you write it.

Personal data, license and use

Three more sections decide whether a dataset may be used at all.

Personal data is anything that identifies a living person: a name, an email address, a phone number, an account id. It is often there uninvited, typed by a customer into a free-text field. A scan with patterns finds the kinds that have a recognizable shape. Privacy engineering goes further: what to do about the columns that identify nobody alone and almost everybody together.

Uses the setup above, 23 lines.

rows = tickets(2400, 0)
# some characters, an @, then words joined by dots
EMAIL = re.compile(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)+")

with_email = [row for row in rows if EMAIL.search(row["text"])]
print(len(with_email), "of", len(rows), "tickets contain an email address")
print("before:", with_email[0]["text"])
print("after: ", EMAIL.sub("[email]", with_email[0]["text"]))
text = ("Write to ana@example.com or Ben.Ortiz+help@mail.example.org. "
        "Signed, Ana Ruiz, flat 4")
print(EMAIL.findall(text))

The scan finds 76 tickets and replaces each address with a placeholder, which is called redaction. The last line shows the limit. Both addresses are caught. The name and the flat number, which identify a person just as well, are not: they have no fixed shape. A pattern scan can show that personal data is present. It can't show that none is, so the card should say what was scanned for and how, not "none".

The license is the set of terms under which the data may be used. Data from your own product is governed by what users agreed to. A public dataset carries a license of its own: some allow any use with credit, some forbid commercial use, and data with no license at all gives no permission. The card records the source and the terms, since a model inherits the obligations of what it was trained on.

Intended use says what the dataset is fit for, and out-of-scope use says what it isn't. These tickets suit ranking a support queue. They would be a poor basis for judging individual customers, and the card should say so before someone tries.

Quick check
What does this print: how many of the four messages contain something the pattern takes for an email address?
import re

EMAIL = re.compile(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)+")
messages = [
    "Mail me: li@example.com", "I'm Li Wei, call me at home", "See you @ the office",
    "billing@shop.example.org bounced",
]
print(sum(1 for m in messages if EMAIL.search(m)))
  1. 2
  2. 1
  3. 3
  4. 4
Show the answer

2

The first and last messages hold an address: characters, an @, then a domain with a dot. The third has an @ with a space on each side and nothing that looks like a domain. The second holds a full name, which is personal data too, and the pattern can't see it.

Quick check
A card for the ticket dataset lists "intended use: ranking the support queue" and "out of scope: decisions about individual customers". A team wants to use the data to pick customers to drop. What does the card tell them?
  1. That they only need to redact the email addresses first
  2. That the model would be accurate enough for that use
  3. That the use is fine if the license allows commercial use
  4. That the dataset wasn't built or checked for that use
Show the answer

That the dataset wasn't built or checked for that use

An out-of-scope entry is the author saying: nobody collected, labeled or evaluated this data with that decision in mind, and the consequences for the people in it weren't considered. It doesn't forbid by force. It tells the next team they would be starting from nothing, with other people's data.

Worked example

Worked example: write the datasheet, and prove its facts

Put the two halves together for the ticket dataset: the facts are computed from the data, the rest is written by the people who know, and the result is printed as the Markdown file that ships beside the data.

Uses the setup above, 23 lines.

rows = tickets(2400, 0)
EMAIL = re.compile(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)+")
test = [row for row in rows if zlib.crc32(row["customer"].encode()) % 100 < 20]
train = [row for row in rows if zlib.crc32(row["customer"].encode()) % 100 >= 20]
shared = {row["customer"] for row in train} & {row["customer"] for row in test}

written = {                                  # the half only people can supply
    "Purpose": "Rank the support queue so urgent tickets are read first.",
    "Collection": ("Exported from the help desk. Tickets raised by phone are not "
                   "included."),
    "Labeling": ("Two annotators per ticket under guideline v3. Split votes settled "
                 "by a team lead."),
    "Out of scope": ("Decisions about individual customers. Any region with fewer "
                     "than 200 tickets."),
    "License": "Internal use only, under the product's privacy policy.",
    "Maintainer": "The support-ml team. Corrections by pull request against this file.",
}

print("# Datasheet: support tickets, version 1")
for heading in ("Purpose", "Collection", "Labeling"):
    print(f"\n## {heading}\n{written[heading]}")
print("\n## Composition")
print(
    f"{len(rows)} tickets created from {min(r['created'] for r in rows)} to "
    f"{max(r['created'] for r in rows)}."
)
for region in sorted({r["region"] for r in rows}):
    part = [r for r in rows if r["region"] == region]
    print(
        f"- {region}: {len(part)} tickets, {sum(r['label'] == 'urgent' for r in part)} "
        f"urgent"
    )
print("\n## Splits")
print(
    f"By customer, with a hash of the customer id: {len(train)} training and "
    f"{len(test)} test tickets."
)
print(f"Customers on both sides: {len(shared)}.")
print("\n## Personal data")
print(
    f"Scanned for email addresses with a pattern: found in "
    f"{sum(1 for r in rows if EMAIL.search(r['text']))} tickets, not yet redacted."
)
print("Names and postal addresses were not scanned for.")
for heading in ("Out of scope", "License", "Maintainer"):
    print(f"\n## {heading}\n{written[heading]}")
In the lesson you can change this and run it.

Every number in the Composition, Splits and Personal data sections was computed in the same run that printed it, so it can't disagree with the data. The composition is broken down by region, which puts the weakest point in plain view: 95 tickets from asia and no urgent one among them. The written half turns that into guidance, by ruling out any region with fewer than 200 tickets. The split is by customer with a hash, and the card states the result of the check, 0 customers on both sides, instead of asking to be believed. The personal data section says what was scanned for, what was found, 76 tickets, and what wasn't looked for. A reader can now tell what this data can support, and the next change to the data will either keep these lines true or fail the check.

The rest of the file is the half people write. Try it for the ticket dataset below.

Written brief · Datasheet

Write the half a program cannot

The ticket dataset's computed half is done. Write the sections only a person can: why it was made, who labeled it and what it must not be used for.

It must

  • Names one task the data suits
  • Gives the labeling guideline and an agreement number
  • Rules out at least one use and says why

Go further

  • Says how the version number changes when a correction is made

The template

## Purpose
What the dataset is for, in one sentence.

## Labeling
Who labeled it, under which guideline, with what agreement.

## Out of scope
Two uses it must not be put to, and why.

## Maintenance
Who owns it and how a correction is made.
In the lesson, you write over it, 90 to 220 words, and the tutor can mark it.
In your own words
What is gained by writing a card's claims as data that a program checks, compared with a paragraph of text?

Write a sentence or two, then compare it with a good answer in the lesson.

On the job: dataset cards in the hub and in the build

On a team, the card lives in the repository next to the data, and a failing claim fails the build.

Where it comes up in a week

Someone asks if the support tickets can be used to train a new model. You open the card before you open the file: what it is for, what it must not be used for, which license applies, what was scanned for personal data. When the April batch lands, the card check fails on five claims and the pull request says which ones. You review the batch, update the card, and the dataset gets a new version in Versioning data and models.

Goes deeper in

In your own words
You inherit a dataset with no documentation. What do you compute, and what do you go and ask?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 7 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 12 review cards for spaced repetition

Try a quiz question

Which of these belongs in a datasheet and can't be produced by analyzing the data?

  1. Which tickets were never exported, and why
  2. The number of rows in each region
  3. The share of rows with a missing language
  4. The range of dates the rows cover

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write