Index Ai Academy
Phase 11 · ML in production

Hyperparameter search

What a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

A model's settings, its hyperparameters, can be worth more accuracy than a change of model, and nobody knows the right values in advance. So you search. A search is a loop that trains many models, which makes it the most expensive thing most ML teams do, and the easiest place to fool yourself: try enough settings and one of them will look good by luck. This lesson is about spending a search budget well and reading its result honestly.

What you'll be able to do

  • Work out what a grid search costs, and say why it spends most of that on settings that don't matter
  • Run a random search, and sample rates on a log scale
  • Cut a search's cost with successive halving, and name the case where it drops the wrong configuration
  • Explain why the winner's validation score is too high, and report a search honestly
Quick check
A guess before you start. You can afford 9 training runs to tune two settings. One of the two turns out to matter a great deal and the other hardly at all, but you don't know which. A 3 × 3 grid tries how many different values of the setting that matters?
  1. 9
  2. 3
  3. 1
  4. 6
Show the answer

3

A grid repeats each value of one setting for every value of the other. Nine runs, but only three values of each setting, so only three looks at the one that matters. Nine runs with each value drawn at random would have tried nine.

The idea: a search space, a budget and a winner's curse

A search is a budget spent on settings, and its cost is a product, so it leans on Counting. The check below recalls how a validation score is used.

What a grid costs

You met GridSearchCV in the scikit-learn lessons: list a few values for each setting and it trains every combination. The trouble is in the word "every". The number of combinations is the product of the list lengths, and with cross-validation each combination is trained once per fold.

from itertools import product

space = {
    "learning_rate": [0.01, 0.1, 1.0],
    "subsample": [0.5, 0.75, 1.0],
    "max_depth": [2, 3, 4],
    "n_estimators": [50, 100, 200],
}
combos = list(product(*space.values()))
print(len(combos), "combinations")
print(len(combos) * 5, "training runs with 5-fold cross-validation")
print("first:", dict(zip(space, combos[0])))
It runs right here in your browser, with nothing to install.

product yields every way of picking one value from each list. Four settings with three values each is already 81 combinations and 405 training runs. Add a fifth setting and it's 1,215. The cost multiplies with every setting you add, and three values is a coarse look at each one.

A grid also assumes you know which values are worth listing. If the best learning rate is 0.3 and your list says 0.01, 0.1 and 1.0, the grid will never find it, however many runs it spends.

Try it now

Each new setting multiplies the bill. Each new value raises the base.

Quick check
What does this print?
from itertools import product

grid = list(product([0.1, 1.0], ["a", "b", "c"], [True, False]))
print(len(grid), grid[0])
  1. 12 [0.1, 'a', True]
  2. 12 (0.1, 'a', True)
  3. 7 (0.1, 'a', True)
  4. 3 (0.1, 1.0)
Show the answer

12 (0.1, 'a', True)

The lists have 2, 3 and 2 values, so there are 2 × 3 × 2 = 12 combinations. Each one is a tuple with one value from each list, and the first takes the first value of each.

Random search

Here is a real search problem. A gradient boosting model (the boosting lesson built one) has a learning_rate, how big a step each new tree takes, and a subsample, the share of rows each tree sees. evaluate trains one model and returns its accuracy on 300 validation rows.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=900, n_features=10, n_informative=4,
                           flip_y=0.08, random_state=1)
X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=300, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X_dev, y_dev, test_size=300, random_state=0)

def evaluate(lr, subsample, trees=40):
    model = GradientBoostingClassifier(n_estimators=trees, learning_rate=lr,
                                       subsample=subsample, max_depth=2, random_state=0)
    return round(float(model.fit(X_tr, y_tr).score(X_val, y_val)), 4)

print("learning rate | subsample 0.5, 0.75, 1.0")
for lr in (0.001, 0.01, 0.1, 0.3, 1.0):
    print(lr, [evaluate(lr, ss) for ss in (0.5, 0.75, 1.0)])

Read down the rows and accuracy climbs from 0.51 to 0.84, then falls. Read across a row and, except at the extreme rate of 1.0, it barely moves. The learning rate matters here and the subsample mostly doesn't. That pattern is typical: of the settings you tune, a few carry most of the effect, and you rarely know which beforehand.

A grid handles this badly, as the opening question showed. Random search draws each setting independently for every run, so every run tests a new value of every setting:

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=900, n_features=10, n_informative=4,
                           flip_y=0.08, random_state=1)
X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=300, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X_dev, y_dev, test_size=300, random_state=0)

def evaluate(lr, subsample, trees=40):
    model = GradientBoostingClassifier(n_estimators=trees, learning_rate=lr,
                                       subsample=subsample, max_depth=2, random_state=0)
    return round(float(model.fit(X_tr, y_tr).score(X_val, y_val)), 4)

grid = [(lr, ss) for lr in (0.01, 0.1, 1.0) for ss in (0.5, 0.75, 1.0)]
rng = np.random.default_rng(0)
rand = [(round(float(10 ** rng.uniform(-3, 0)), 4),
         round(float(rng.uniform(0.5, 1.0)), 2)) for trial in range(9)]

for name, configs in (("grid", grid), ("random", rand)):
    scored = [(evaluate(lr, ss), lr, ss) for lr, ss in configs]
    print(name, "| learning rates tried:", sorted({lr for lr, ss in configs}))
    print(name, "| best (score, rate, subsample):", max(scored))

Same budget, nine runs each. The grid looked at three learning rates and the random search at nine, five of them between 0.15 and 0.39, where the table above peaks. The scores are 0.8367 and 0.84. That gap is one validation row out of 300, so don't read it as a win: on this problem the grid's list happened to include a good value. What random search buys is that it doesn't depend on that luck. You give it a range, not a list.

Rates live on a log scale

The random search above drew the learning rate as 10 ** rng.uniform(-3, 0), not as rng.uniform(0.001, 1.0). The reason is that a learning rate matters by its order of magnitude: going from 0.001 to 0.01 is as big a change as going from 0.1 to 1. Drawing the exponent evenly gives every factor of ten the same share of the runs. This is called sampling on a log scale, or log-uniform sampling.

import numpy as np

rng = np.random.default_rng(0)
plain = rng.uniform(0.001, 1.0, size=1000)
logged = 10 ** rng.uniform(-3, 0, size=1000)
for name, draws in (("uniform    ", plain), ("log-uniform", logged)):
    print(name, "| below 0.01:", round(float((draws < 0.01).mean()), 3),
          "| below 0.1:", round(float((draws < 0.1).mean()), 3))

Drawn evenly between 0.001 and 1, fewer than 1 draw in 100 lands below 0.01, and 9 in 10 land above 0.1: the search would never look at two of the three factors of ten. On the log scale each gets about a third. Use it for anything you'd naturally write as a power of ten: learning rates, regularization strengths, weight decay.

Go deeper Searches that learn as they go

Random search ignores what earlier runs found. Bayesian optimization uses it: after each run it fits a cheap model of "score as a function of settings" and picks the next settings where that model expects a good score or is most unsure. Libraries such as Optuna do this. It helps most when each run is expensive and the budget is a few dozen runs. Random search remains the baseline any smarter method has to beat, and it runs in parallel with no coordination.

Quick check
A search draws a regularization strength on a log scale from 0.001 to 1. What does this print: the middle of that range, the value half the draws fall below?
low, high = -3, 0
middle = (low + high) / 2
print(middle, round(10 ** middle, 4))
  1. -1.5 0.0316
  2. -1.5 0.15
  3. 0.5 0.0316
  4. -1.5 0.5005
Show the answer

-1.5 0.0316

The exponents run from -3 to 0, so their middle is -1.5, and 10 to the power -1.5 is about 0.0316. On a log scale the midpoint of 0.001 and 1 is 0.0316, not 0.5.

In your own words
In your own words: why does a grid search get expensive so quickly?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: log sampling, halving and an honest final score

Each cell trains small gradient boosting models on 900 rows, so a full search runs in the page.

Drill

Draw one configuration from a search space

A search space maps each setting's name to how to draw it: ("uniform", low, high), ("log", low, high) or ("choice", [options]). Write sample_config(space, rng): a dictionary with one drawn value per setting. For "uniform" use rng.uniform(low, high). For "log" draw the exponent evenly between math.log10(low) and math.log10(high) and return 10 to that power. For "choice" use rng.choice(options).

import math
import numpy as np


def sample_config(space, rng):
    config = {}
    # one drawn value for each setting in the space
    return config


space = {"lr": ("log", 0.001, 1.0), "subsample": ("uniform", 0.5, 1.0),
         "depth": ("choice", [2, 3, 4])}
print(sample_config(space, np.random.default_rng(0)))
In the lesson, your code is checked as you write it.

Stop the losers early

Most configurations in a search are bad, and many are visibly bad long before training finishes. Successive halving uses that. Give every configuration a small budget, keep the better half, double the budget, and repeat until one is left. For boosting, the budget is the number of trees. For a neural network it would be epochs.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=900, n_features=10, n_informative=4,
                           flip_y=0.08, random_state=1)
X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=300, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X_dev, y_dev, test_size=300, random_state=0)

def evaluate(lr, subsample, trees=40):
    model = GradientBoostingClassifier(n_estimators=trees, learning_rate=lr,
                                       subsample=subsample, max_depth=2, random_state=0)
    return round(float(model.fit(X_tr, y_tr).score(X_val, y_val)), 4)

rng = np.random.default_rng(0)
configs = [(round(float(10 ** rng.uniform(-3, 0)), 4),
            round(float(rng.uniform(0.5, 1.0)), 2)) for trial in range(16)]

alive, budget, cost = list(configs), 5, 0
while len(alive) > 1:
    ranked = sorted(alive, key=lambda c: evaluate(c[0], c[1], trees=budget),
                    reverse=True)
    cost += len(alive) * budget
    print(len(alive), "configs at", budget, "trees | best so far:", ranked[0])
    alive = ranked[: len(alive) // 2]
    budget *= 2
print("winner:", alive[0], "| cost:", cost, "trees")

full = sorted(configs, key=lambda c: evaluate(c[0], c[1], trees=80), reverse=True)
print("best of all 16 at 80 trees:", full[0], "| cost:", 16 * 80, "trees")

Sixteen configurations, four rounds, 320 trees trained in all. Training every one of the sixteen to 80 trees costs 1,280, four times as much, and here it names the same winner.

Notice the leader changed from round to round. That is the method's weak point: it assumes that whoever is ahead early will be ahead late. A configuration that starts slowly and finishes strongly, a low learning rate for instance, can be cut before it shows what it can do. With a different draw of sixteen configurations it goes wrong:

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=900, n_features=10, n_informative=4,
                           flip_y=0.08, random_state=1)
X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=300, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X_dev, y_dev, test_size=300, random_state=0)

def evaluate(lr, subsample, trees=40):
    model = GradientBoostingClassifier(n_estimators=trees, learning_rate=lr,
                                       subsample=subsample, max_depth=2, random_state=0)
    return round(float(model.fit(X_tr, y_tr).score(X_val, y_val)), 4)

rng = np.random.default_rng(5)
configs = [(round(float(10 ** rng.uniform(-3, 0)), 4),
            round(float(rng.uniform(0.5, 1.0)), 2)) for trial in range(16)]
alive, budget = list(configs), 5
while len(alive) > 1:
    alive = sorted(alive, key=lambda c: evaluate(c[0], c[1], trees=budget),
                   reverse=True)[: len(alive) // 2]
    budget *= 2
best = max(configs, key=lambda c: evaluate(c[0], c[1], trees=80))
print("halving's winner:", alive[0], "scores",
      evaluate(alive[0][0], alive[0][1], trees=80), "at 80 trees")
print("best at 80 trees:", best, "scores", evaluate(best[0], best[1], trees=80))

This time halving kept a configuration that scores 0.8333 at 80 trees and dropped one that reaches 0.86. The two met in the last round, at 40 trees, and the order there was not the order at 80. Halving is a trade: a quarter of the cost, for a winner that is usually near the best and sometimes isn't the best.

Quick check
Successive halving starts 32 configurations at a budget of 1 epoch, keeps the better half each round and doubles the budget, until one is left. What does this print?
alive, budget, cost, rounds = 32, 1, 0, 0
while alive > 1:
    cost += alive * budget
    alive = alive // 2
    budget *= 2
    rounds += 1
print(rounds, cost, 32 * budget)
  1. 5 62 1024
  2. 6 192 1024
  3. 5 160 512
  4. 5 160 1024
Show the answer

5 160 1024

Each round costs 32 epochs: 32 × 1, 16 × 2, 8 × 4, 4 × 8, 2 × 16. Five rounds make 160. By then the budget has doubled five times to 32 epochs, and training all 32 configurations for 32 epochs would cost 1,024.

Quick check
Which configuration is successive halving most likely to throw away by mistake?
  1. One with a large learning rate that scores well from the first round
  2. One that scores worst at every budget, small or large
  3. One with a small learning rate that improves slowly but ends highest
  4. One that is identical to another configuration in the search
Show the answer

One with a small learning rate that improves slowly but ends highest

Halving judges everyone at a small budget first. A slow starter looks bad there and is cut, even if it would have overtaken the rest with a full budget.

Drill

Successive halving

Write successive_halving(configs, evaluate, budget). While more than one configuration is alive: score each with evaluate(config, budget), add alive × budget to the cost, keep the better half (len(alive) // 2 of them, and on a tie the one that came earlier in the list stays ahead), and double the budget. Return a tuple: the last configuration standing, and the total cost.

def successive_halving(configs, evaluate, budget):
    alive = list(configs)
    cost = 0
    # rounds: score, count the cost, keep the better half, double the budget
    return alive[0], cost


def toy(config, budget):
    return config * budget          # a made-up score: bigger configs do better


print(successive_halving([3, 1, 4, 2], toy, 1))
In the lesson, your code is checked as you write it.

A plot of the first table shows why a learning rate wants a log axis:

import matplotlib.pyplot as plt

rates = [0.001, 0.01, 0.1, 0.3, 1.0]
scores = [0.5067, 0.8, 0.8367, 0.84, 0.6933]
plt.plot(rates, scores, marker="o")
plt.xscale("log")
plt.xlabel("learning rate (log scale)")
plt.ylabel("validation accuracy")
plt.show()

The curve climbs from 0.001 to 0.3 and falls at 1.0. The good values sit in one narrow band.

The winner's score is too high

Here is the trap that catches every search. Suppose 30 settings are all exactly as good as each other: each would get 80% of new cases right. Score each one on 300 validation rows. By chance some will get a few more right and some a few fewer. Now pick the best.

import numpy as np

rng = np.random.default_rng(0)
scores = rng.binomial(300, 0.80, size=30) / 300
print("30 settings, each truly 0.80 | lowest", round(float(scores.min()), 3),
      "| highest", round(float(scores.max()), 3))

best = [rng.binomial(300, 0.80, size=30).max() / 300 for search in range(1000)]
print("mean best score over 1,000 searches:", round(float(np.mean(best)), 3))

rng.binomial(300, 0.80) counts how many of 300 rows a setting gets right when each has an 80% chance. The "best" of thirty identical settings scores 0.847, and across a thousand such searches the winner averages 0.846. None of them is better than 0.80. Choosing the maximum selects for luck as well as skill, so the winner's validation score is too high, and the more settings you try the more too high it is.

The same thing happens in a real search:

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=900, n_features=10, n_informative=4,
                           flip_y=0.08, random_state=1)
X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=300, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X_dev, y_dev, test_size=300, random_state=0)

def fit(lr, subsample):
    return GradientBoostingClassifier(n_estimators=40, learning_rate=lr,
                                      subsample=subsample, max_depth=2,
                                      random_state=0).fit(X_tr, y_tr)

rng = np.random.default_rng(0)
trials = [(float(10 ** rng.uniform(-3, 0)), float(rng.uniform(0.5, 1.0)))
          for trial in range(12)]
models = [fit(lr, ss) for lr, ss in trials]
val = [round(float(m.score(X_val, y_val)), 4) for m in models]
i = val.index(max(val))
print("winner on validation:", val[i], "| the same model on the test set:",
      round(float(models[i].score(X_test, y_test)), 4))

The search's winner scores 0.8433 on the rows that chose it and 0.8233 on rows it has never met. One search is one draw, as Experiment tracking showed, so don't read the size of that gap too closely. The direction is what you can rely on.

So the rule from the cross-validation lesson carries over, with more force. The validation score picks the winner. The test score, measured once on the winner alone, is the number you report. And every trial goes in the run log from Experiment tracking, losers included: a result that says "best of 12" means something different from one that says "best of 1,200".

Quick check
From the cross-validation lesson: a search reports a best validation score of 0.80, and the untouched test set gives the chosen model 0.78. Which number estimates how it will do on new data?
  1. 0.80: a search always reports its true best score
  2. 0.80: it comes from more rows of data in total
  3. 0.79: the average of the two is fairer than either
  4. 0.78: the test set played no part in choosing the model
Show the answer

0.78: the test set played no part in choosing the model

The validation rows chose the winner, so its score on them is flattered by the choice. The test rows had no say, so the test score is the honest estimate.

Quick check
Two teams tune the same model on the same data. Team A tries 10 configurations and Team B tries 1,000. Each reports its winner's validation score. What should you expect?
  1. A's reported score is flattered more, since 10 trials is a small sample
  2. B's score is exact, since 1,000 trials average the luck away
  3. B's reported score is flattered more, since the best of 1,000 has more luck in it
  4. The two scores are equally trustworthy, since the data is the same
Show the answer

B's reported score is flattered more, since the best of 1,000 has more luck in it

The maximum of many noisy scores drifts further above the truth as the number of scores grows. B may well have found a better model, but its validation score overstates it by more. Only a test set neither team searched on can compare them.

Worked example

Worked example: a search you could hand to a teammate

Put the pieces together: a seeded random search on a log scale, every trial kept, the winner picked on validation and scored once on the test set.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=900, n_features=10, n_informative=4,
                           flip_y=0.08, random_state=1)
X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=300, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X_dev, y_dev, test_size=300, random_state=0)

rng = np.random.default_rng(3)                    # the search has a seed too
log = []
for trial in range(10):
    config = {"learning_rate": round(float(10 ** rng.uniform(-3, 0)), 4),
              "subsample": round(float(rng.uniform(0.5, 1.0)), 2)}
    model = GradientBoostingClassifier(n_estimators=40, max_depth=2, random_state=0,
                                       **config).fit(X_tr, y_tr)
    log.append({"trial": trial, "config": config,
                "validation": round(float(model.score(X_val, y_val)), 4),
                "model": model})

winner = max(log, key=lambda t: t["validation"])
print("trials:", len(log), "| validation scores from",
      min(t["validation"] for t in log), "to", winner["validation"])
print("winner: trial", winner["trial"], winner["config"])
print("report: test accuracy", round(float(winner["model"].score(X_test, y_test)), 4),
      "(best of", len(log), "trials)")
In the lesson you can change this and run it.

Four habits are in those lines. The search itself is seeded, so the same ten configurations can be drawn again. Every trial is kept, not only the winner, so anyone can see how many were tried and how the scores spread. The winner is chosen on validation data only. And the test set is touched exactly once, by the winner, for the number that goes in the report, next to the count of trials behind it.

In your own words
What does successive halving assume, and what does it cost you when the assumption fails?

Write a sentence or two, then compare it with a good answer in the lesson.

On the job: Optuna, scikit-learn and a search you can rerun

A search is a job: it has a seed, a trial log and a budget, and it ends with a test score measured once.

Where it comes up in a week

Someone asks for a better score before the end of the week. You set a budget in runs, not days, draw the learning rate and weight decay on log scales, log every trial in Experiment tracking's format, and report the test score of the winner measured once. If the leader changes from round to round, you do not use halving on that model.

Goes deeper in

In your own words
A search tries 200 configurations and the best scores 0.91 on validation. What do you report, and what do you record?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 10 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 11 review cards for spaced repetition

Try a quiz question

A grid search covers 5 settings with 4 values each, using 3-fold cross-validation. How many models does it train?

  1. 3,072
  2. 60
  3. 1,024
  4. 625

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Part of a free course 11 lessons

Free MLOps course: running machine learning in production

Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write