Index Ai Academy
Phase 11 · ML in production

Feature pipelines and training-serving skew

Why a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

A model can pass every offline test and still fail the day it goes live, with no error anywhere. The usual reason is quiet: the numbers the model is fed in production are not computed the way its training numbers were. A different default for a missing value, a statistic recalculated on the wrong rows, a count that accidentally included next week. This is called training-serving skew. It is among the most common ways an ML system fails in production, and the offline score can't see it. This lesson shows two real cases, with measurements, and the habits that prevent them.

What you'll be able to do

  • Say what training-serving skew is, and name its three usual causes
  • Fit a feature's statistics once on training data, save them, and apply the same ones when serving
  • Build training features as of the moment each prediction would have been made, with nothing from the future
  • Catch skew with a parity check that runs the same raw rows through both paths
Quick check
A guess before you start. A fraud model scores 0.89 on its test set. In production its inputs are scaled by a line of code that recomputes the mean and spread from each incoming batch of requests. On an ordinary afternoon, what will the team see?
  1. Better accuracy than offline, since each batch is scaled to fit itself
  2. Accuracy near 0.50, since the scaling is wrong for every single row
  3. Nothing wrong: the bug only shows when a batch differs from the training data
  4. An error message as soon as the first batch of requests arrives
Show the answer

Nothing wrong: the bug only shows when a batch differs from the training data

An afternoon batch looks like the training data, so its own mean and spread are close to the saved ones and the model does about as well as offline. The damage appears when traffic shifts, as it does at night in this lesson's data, which is exactly when the model is needed most.

The idea: two code paths and one definition

A feature is a function of rows, and standardizing is the idea from Describing data: subtract a mean, divide by a spread. This part is about where those statistics come from.

The same model, fed differently

A model never sees your data. It sees features: the numbers a piece of code computed from your data. That code runs in two places. In training it runs once over a big table of past rows. In serving it runs again, usually as different code, on one request at a time. Training-serving skew is any difference between what those two compute for the same input.

Here is a fraud model with two inputs, the amount of a payment and the age of the account. Its inputs are standardized, as Data preparation taught: subtract the mean, divide by the standard deviation.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

def make(n, seed, night=False):
    rng = np.random.default_rng(seed)
    # dollars: mostly small, a few large
    amount = rng.lognormal(mean=4.5 if night else 3.5, sigma=0.8, size=n)
    # how old the account is
    age_days = rng.integers(1, 1000, size=n)
    risk = 0.02 * amount - 0.004 * age_days + rng.normal(0, 1, n)
    return np.column_stack([amount, age_days]).astype(float), (risk > 0.5).astype(int)

X_train, y_train = make(2000, 0)
X_test, y_test = make(1000, 1)
# night traffic: larger payments, more fraud
X_night, y_night = make(1000, 2, night=True)

# mean and spread, learned once from training rows
scaler = StandardScaler().fit(X_train)
model = LogisticRegression().fit(scaler.transform(X_train), y_train)

print("offline test          ",
      round(float(model.score(scaler.transform(X_test), y_test)), 3))
print("night, saved scaler   ",
      round(float(model.score(scaler.transform(X_night), y_night)), 3))
print("night, scaler refit   ",
      round(float(model.score(StandardScaler().fit_transform(X_night), y_night)), 3))
print("daytime, scaler refit ",
      round(float(model.score(StandardScaler().fit_transform(X_test), y_test)), 3))
It runs right here in your browser, with nothing to install.

The second line is the model served correctly: at night, with the scaler that was saved from training, it gets 0.861 right. The third line is the bug. Someone wrote fit_transform in the serving code, so each batch is scaled by its own mean and spread, and accuracy falls to 0.669.

The fourth line is why nobody noticed. On daytime traffic the refit scaler scores 0.892, as good as the offline test, because a daytime batch has nearly the same mean and spread as the training data. The bug is invisible until the inputs shift.

What went wrong at night? A large payment is a warning sign. But scaled against a batch in which every payment is large, it looks average:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

def make(n, seed, night=False):
    rng = np.random.default_rng(seed)
    amount = rng.lognormal(mean=4.5 if night else 3.5, sigma=0.8, size=n)
    age_days = rng.integers(1, 1000, size=n)
    risk = 0.02 * amount - 0.004 * age_days + rng.normal(0, 1, n)
    return np.column_stack([amount, age_days]).astype(float), (risk > 0.5).astype(int)

X_train, y_train = make(2000, 0)
X_night, y_night = make(1000, 2, night=True)
scaler = StandardScaler().fit(X_train)
model = LogisticRegression().fit(scaler.transform(X_train), y_train)

print("share of night payments that are fraud:", round(float(y_night.mean()), 3))
print("flagged with the saved scaler:         ",
      round(float(model.predict(scaler.transform(X_night)).mean()), 3))
print("flagged with the scaler refit:         ",
      round(float(model.predict(StandardScaler().fit_transform(X_night)).mean()), 3))
print("one request, scaled by itself:         ",
      StandardScaler().fit_transform(X_night[:1]))

At night 43% of payments are fraud. Served correctly the model flags 41% of them. With the refit scaler it flags 10%, because the refit told it this was an ordinary batch. And the last line shows the bug at its purest: a single request scaled by its own statistics is all zeros, whatever it contained.

Quick check
A serving function standardizes each batch with the batch's own mean and standard deviation. What does this print?
import numpy as np

quiet = np.array([10.0, 20.0, 30.0])
busy = np.array([1010.0, 1020.0, 1030.0])

def scale_by_self(batch):
    return (batch - batch.mean()) / batch.std()

print(np.allclose(scale_by_self(quiet), scale_by_self(busy)),
      scale_by_self(busy).round(2))
  1. True [1010. 1020. 1030.]
  2. True [-1.22 0. 1.22]
  3. False [123.7 124.9 126.1]
  4. False [-1.22 0. 1.22]
Show the answer

True [-1.22 0. 1.22]

Both batches have the same spread, and subtracting each batch's own mean removes the 1,000 that separates them. After scaling they are identical, so a model downstream can't tell a quiet batch from a busy one.

Three causes of skew

Nearly every case of skew is one of three things.

  1. State refit at serving. A statistic that should have been learned once from training data, a mean, a median for filling gaps, a vocabulary, is recomputed from whatever arrives. That was the scaler.
  2. Two code paths. Training features come from a notebook or a SQL query, and serving features from a hand-written copy in another language. They disagree on a default, a unit, a time zone or a rounding rule.
  3. Features from the future. A training feature was computed with data that wouldn't have existed yet at the moment of the prediction. The second half of this lesson measures one.

None of them raises an error, and the offline score is blind to all of them, because the offline score is computed with the training path on both sides.

Try it now

Pick a case. Only the scaler's statistics differ between them.

Features from the future

The third cause is the hardest to see, because the code can be identical on both sides and still be wrong.

A subscription shop wants to predict, on day 60, which customers will churn: place no order in the next 30 days. A natural feature is "orders in the last 30 days". The analyst builds the training table on day 75, from the orders table as it stands that day, and "the last 30 days" quietly means days 45 to 74.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)
n_users = 1500
# how often each customer orders
activity = rng.gamma(2.0, 1.0, size=n_users)
# some stop for good
quits = np.where(rng.random(n_users) < 0.35, rng.integers(40, 80, size=n_users), 999)
user, day = [], []
for u in range(n_users):
    for d in rng.integers(0, 120, size=rng.poisson(activity[u] * 6)):
        if d < quits[u]:
            user.append(u)
            day.append(int(d))
user, day = np.array(user), np.array(day)

def orders(start, end):
    """Each customer's number of orders on days start, ..., end - 1."""
    return np.bincount(user[(day >= start) & (day < end)],
                       minlength=n_users).astype(float).reshape(-1, 1)

predict_day = 60
churned = (orders(predict_day, predict_day + 30)[:, 0] == 0).astype(int)
# days 30 to 59: known on day 60
honest = orders(predict_day - 30, predict_day)
# days 45 to 74: counted on export day
leaky = orders(75 - 30, 75)

train, test = slice(0, 1000), slice(1000, None)
for name, F in (("as of the prediction day", honest),
                ("as of the export day    ", leaky)):
    model = LogisticRegression().fit(F[train], churned[train])
    print(name, "| offline AUC",
          round(float(roc_auc_score(churned[test],
                                    model.predict_proba(F[test])[:, 1])), 3))

The feature built on export day scores an AUC of 0.858 offline. The honest one scores 0.692. Days 60 to 74 are inside the period the label is about, so the leaky feature has seen part of the answer: a customer with no orders in those days is already half way to "churned".

In production, on a real day 60, nobody has days 60 to 74. The only feature that can be served is the honest one, and the model's real performance is 0.692, not the 0.858 the team was promised.

The rule is called point-in-time correctness: every training row's features are computed as of the moment its prediction would have been made, using only events with an earlier timestamp. It needs two things from your data. Every event keeps the time it happened, and every training example keeps the time it was predicted for.

Go deeper What a feature store is for

A feature store is a system built around these two rules. A feature is defined once. An offline side serves training, answering "what was this feature's value for this customer as of this timestamp?" An online side serves the model, answering "what is it right now?" in a few milliseconds. Because both read the same definition, the two code paths can't drift, and because the offline side is asked "as of when", the future stays out. You don't need one to start. You need its two rules.

Quick check
A function counts a customer's orders in the 30 days before a prediction day. What does this print?
order_days = [12, 31, 44, 58, 60, 61, 75]
predict_day = 60

as_of = [d for d in order_days if predict_day - 30 <= d < predict_day]
at_export = [d for d in order_days if 75 - 30 <= d < 75]
print(len(as_of), len(at_export))
  1. 4 3
  2. 3 4
  3. 3 3
  4. 4 5
Show the answer

3 3

As of day 60 the window is days 30 to 59, which holds 31, 44 and 58. The order on day 60 itself hasn't happened yet when the prediction is made. Counted at export, the window is days 45 to 74, which holds 58, 60 and 61: the same count here, but two of those three orders came after the prediction.

In your own words
In your own words: what is training-serving skew, and why doesn't the offline test score reveal it?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: fit once, one function, features as of a date

The cells below build the fix for each of the three causes, then a test that would have caught them.

One function, and its saved state

The fix for the first two causes is a discipline with two parts.

Fit once, save, apply everywhere. Whatever a feature learns from data is computed on the training rows, saved as an artifact next to the model, and only ever applied afterward. Fitting and applying are two separate functions, and the serving code has no access to the fitting one.

One function computes the features, and both sides call it. If training and serving import the same make_features, they can't disagree about a default or a unit.

import json, statistics

def fit_state(rows):
    """Learn from training rows only: the fill value and the scale for each column."""
    state = {}
    for column in ("amount", "income"):
        known = [r[column] for r in rows if r[column] is not None]
        state[column] = {"fill": statistics.median(known),
                         "mean": statistics.mean(known),
                         "std": statistics.pstdev(known)}
    return state

def make_features(row, state):
    """The one place features are computed. Training and serving both call this."""
    features = []
    for column in ("amount", "income"):
        value = row[column] if row[column] is not None else state[column]["fill"]
        features.append(
            round((value - state[column]["mean"]) / state[column]["std"], 3))
    return features

training_rows = [
    {"amount": 20.0, "income": 3000.0}, {"amount": 35.0, "income": None},
    {"amount": 50.0, "income": 5000.0}, {"amount": 95.0, "income": 4000.0},
]
state = fit_state(training_rows)
# stored beside the model, under its hash
saved = json.dumps(state)
print(saved)

# the server loads it; it never refits
state_at_serving = json.loads(saved)
print(make_features({"amount": 400.0, "income": None}, state_at_serving))

The request for 400 dollars comes out more than ten standard deviations above the training mean, which is the truth about it, whatever else is in the batch. Its missing income is filled with 4,000, the training median, which standardizes to exactly 0.

The saved state is part of the model. It gets a hash and a place in the model's record like the weights do, since a model served with another model's state is skewed by construction. scikit-learn's Pipeline, from the workflow lesson, does this packaging for you: the fitted steps and the model are one object, saved and loaded as one file.

Quick check
From the scikit-learn workflow lesson: why are the scaler and the model put into one Pipeline?
  1. So preprocessing is fitted on training rows only
  2. So the scaler can be refit on each new batch of requests
  3. So the test set can be used to choose the scaling
  4. So the model trains faster than it would on unscaled data
Show the answer

So preprocessing is fitted on training rows only

A Pipeline fits its preprocessing steps on the rows it is trained on and applies those fitted steps to anything it later predicts on. That prevents leakage in cross-validation and skew in serving, which are the same mistake at two different moments.

Drill

Fit once, apply anywhere

Write two functions. fit_state(values) takes a column of training values, some of them None, and returns {"fill": ..., "mean": ..., "std": ...}: the median, mean and population standard deviation (statistics.pstdev) of the values that aren't None. apply(value, state) returns one standardized value: replace None with the fill value, subtract the mean, divide by the std. If the std is 0, return 0.0. apply must use only value and state.

import statistics


def fit_state(values):
    return {"fill": 0.0, "mean": 0.0, "std": 1.0}


def apply(value, state):
    return value


state = fit_state([20.0, 35.0, None, 50.0, 95.0])
print(state)
print(apply(400.0, state), apply(None, state))
In the lesson, your code is checked as you write it.
Quick check
A model predicts at 9 a.m. whether a delivery will be late. Which feature is safe to train on?
  1. The number of orders the courier had been assigned by 9 a.m.
  2. The time the delivery was finally marked complete that day
  3. The total distance the courier ended up driving that day
  4. The customer's rating of the delivery, left that evening
Show the answer

The number of orders the courier had been assigned by 9 a.m.

A feature is safe if its value was known at the moment of the prediction. The courier's load at 9 a.m. was. The other three were only known after the delivery, so in training they would leak the outcome and in production they wouldn't exist.

Drill

Count only what was known

Write orders_as_of(order_days, predict_day, window): how many of a customer's orders fall in the window days before predict_day. An order on predict_day itself is not known yet, so it doesn't count. An order exactly window days earlier does. order_days is a list of day numbers in any order, possibly with repeats (two orders on one day count twice).

def orders_as_of(order_days, predict_day, window):
    return len(order_days)


print(orders_as_of([12, 31, 44, 58, 60, 61, 75], 60, 30))
In the lesson, your code is checked as you write it.

Catching skew before customers do

Skew doesn't announce itself, so you go looking for it. Three checks do most of the work.

A parity test. Take a few hundred raw rows, run them through the training path and through the serving path, and compare the features value by value. It belongs in the test suite from the CI/CD lesson, and it fails the build the day someone edits one path and forgets the other.

def train_path(row):                 # the batch job's version
    amount = row["amount_cents"] / 100
    return {"amount": round(amount, 2), "is_new": int(row["age_days"] < 30)}

def serve_path(row):                 # the service's version, written by another team
    amount = row["amount_cents"] / 100
    return {"amount": round(amount, 2), "is_new": int(row["age_days"] <= 30)}

rows = [{"amount_cents": 1999, "age_days": 12},
        {"amount_cents": 45000, "age_days": 30},
        {"amount_cents": 800, "age_days": 400}]
for i, row in enumerate(rows):
    a, b = train_path(row), serve_path(row)
    diffs = [name for name in a if a[name] != b[name]]
    print("row", i, "ok" if not diffs else "MISMATCH in " + ", ".join(diffs))

Two paths, one character apart: < against <=. Only an account that is exactly 30 days old shows it, which is why a handful of hand-picked rows isn't enough and the test rows should include the edges.

Log what was served. Have the service write down the feature values it actually fed the model, next to each prediction. Those logs are the only record of what the model really saw. They let you compare the distribution of each served feature with its training distribution, the subject of Monitoring a model: drift and decay, and they make the best training data for the next model, since they are point-in-time correct by construction.

Distrust a score that is too good. A sudden jump in an offline metric, like 0.692 to 0.858 from one new feature, is more often a leak than a breakthrough. Ask of every feature: at the moment of the prediction, would this value have existed?

Quick check
A parity test feeds the same raw rows to the training path and the serving path. What does this print?
def train_path(row):
    return {"spend": row["cents"] / 100, "weekend": int(row["weekday"] >= 5)}

def serve_path(row):
    return {"spend": row["cents"] / 100, "weekend": int(row["weekday"] > 5)}

rows = [{"cents": 500, "weekday": 2}, {"cents": 1250, "weekday": 5},
        {"cents": 99, "weekday": 6}]
bad = [(i, name) for i, row in enumerate(rows) for name in ("spend", "weekend")
       if train_path(row)[name] != serve_path(row)[name]]
print(bad)
  1. [(1, 'weekend'), (2, 'weekend')]
  2. []
  3. [(1, 'spend'), (1, 'weekend')]
  4. [(1, 'weekend')]
Show the answer

[(1, 'weekend')]

The two paths agree on spend everywhere. They disagree on weekend only when the weekday is exactly 5: >= 5 says yes and > 5 says no. That's row 1. On weekday 6 both say yes, and on weekday 2 both say no.

The same bars as a plot cell, with the coin-flip line at 0.5:

import matplotlib.pyplot as plt

names = ["built on export day", "as of each day"]
auc = [0.858, 0.692]
plt.bar(names, auc, color=["tab:red", "tab:blue"])
plt.axhline(0.5, linestyle=":", color="gray")
plt.ylabel("AUC")
plt.ylim(0.4, 0.9)
plt.show()
Worked example

Worked example: a training table built as of each prediction

Real training sets hold many prediction dates, not one. Each row gets the features as of its own date. Here three customers are each predicted on two different days, and the same function that builds the training rows answers a live request.

events = [                      # (customer, day, dollars), in any order
    ("ana", 3, 20.0), ("ana", 41, 35.0), ("ana", 58, 15.0), ("ana", 75, 60.0),
    ("ben", 10, 80.0), ("ben", 12, 40.0),
    ("cy", 55, 25.0), ("cy", 59, 25.0), ("cy", 60, 30.0), ("cy", 88, 10.0),
]

def features_as_of(events, customer, predict_day, window=30):
    mine = [(d, amount) for c, d, amount in events if c == customer and d < predict_day]
    recent = [amount for d, amount in mine if d >= predict_day - window]
    last = max((d for d, amount in mine), default=None)
    return {"orders": len(recent), "spend": sum(recent),
            "days_since_last": None if last is None else predict_day - last}

for customer in ("ana", "ben", "cy"):
    for predict_day in (60, 90):
        print(customer, "as of day", predict_day,
              features_as_of(events, customer, predict_day))

print("live request, day 95:", features_as_of(events, "cy", 95))
In the lesson you can change this and run it.

Read cy's two rows. As of day 60 there are two orders in the window, days 55 and 59, and the order placed on day 60 itself is correctly left out. As of day 90 the window has moved to days 60 to 89 and holds the orders on days 60 and 88. The same customer gives two different, equally valid training rows, because each is a picture of what was known on its own day. Ben's rows show the other edge: no order in either window, with days_since_last growing from 48 to 78. And the last line is the serving path: the live request calls the very same function with today's date, so there is no second implementation to drift.

In your own words
Why can a feature that includes data from after the prediction time look excellent offline?

Write a sentence or two, then compare it with a good answer in the lesson.

On the job: a feature store, parity in CI and a saved pipeline

On a team the feature code is shared, its fitted state is versioned with the model, and a parity test runs on every pull request.

Where it comes up in a week

A model that scored well offline underperforms for a week after launch. You compare the features it was served, from the logs, with the same rows pushed through the training path: a parity test. When one column differs at an edge (< against <=), the fix is one function imported by both sides. You also ask of the newest feature: at the moment of the prediction, would this value have existed? That question is the point-in-time rule from the Streams lesson, applied to a training table.

Goes deeper in

In your own words
Name two checks that would catch training-serving skew, and what each one compares.

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 7 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 11 review cards for spaced repetition

Try a quiz question

A model does well offline and badly in production, with no errors in the logs. Which explanation fits training-serving skew?

  1. A feature is computed differently in serving than it was in training
  2. The model has too few parameters to fit the training data well
  3. The test set was smaller than the training set by a wide margin
  4. The learning rate was set too high during the final training run

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Part of a free course 11 lessons

Free MLOps course: running machine learning in production

Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write