Index Ai Academy
Phase 11 · ML in production

Experiment tracking and reproducible runs

A run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Training a model once is easy. The job of an ML engineer is the fortieth run: the one where a teammate asks why last month's model scored higher, and nobody can say which data, settings or code produced it. Every team that ships models keeps a record of each run for this reason, and tools such as MLflow and Weights & Biases exist to hold that record. The work of running models this way has a name, MLOps, and it starts here. This lesson builds the record by hand, so you know what those tools store and why, and then uses it for the question a record exists to answer: is this model better, or did it get lucky?

What you'll be able to do

  • List what a training run must record before its score means anything: settings, seed, data, code and metrics
  • Fingerprint a configuration and a dataset with a hash, so two runs can be proved to match or differ
  • Keep runs in an append-only log, load them back, and reproduce one from its record
  • Compare two settings across several seeds, and say whether the gap is bigger than the noise
Quick check
A guess before you start. A teammate reports that making a model's trees one level deeper raised accuracy from 0.927 to 0.947. They trained each version once. What is the most useful thing to ask first?
  1. Whether 0.947 is above the industry average for this task
  2. Whether a third decimal place would make the gap clearer
  3. How long each of the two training runs took to finish
  4. How much the score moves when nothing changes except the random seed
Show the answer

How much the score moves when nothing changes except the random seed

A score from one run is one draw. If runs with identical settings differ by 0.03 from seed to seed, a gap of 0.02 says nothing yet. This lesson measures that spread for exactly this pair of models.

The idea: one run is one draw

A score from one run is one draw from a spread, so the standard error from Sampling and the central limit theorem is the tool that tells a real gap from luck. First the recall.

One run is one draw

A training run has randomness in it: which rows land in the test set, which rows each tree samples, where the weights start. A seed is the number that fixes those choices. Same seed, same choices, same score.

Here is the comparison from the opening question, run for real. run trains a small random forest on a synthetic dataset of 600 rows and returns its accuracy on the held-out quarter.

Setup (runs before every cell) · 12 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import hashlib, json, math, os, statistics
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=600, n_features=8, n_informative=4, random_state=0)


def run(params, seed):
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=seed)
    model = RandomForestClassifier(**params, random_state=seed).fit(X_tr, y_tr)
    return round(float(model.score(X_te, y_te)), 4)
shallow = {"n_estimators": 20, "max_depth": 5}
deeper = {"n_estimators": 20, "max_depth": 6}
print("seed 1:      ", run(shallow, 1), "->", run(deeper, 1))
print("seed 1 again:", run(shallow, 1), "->", run(deeper, 1))
print("seed 0:      ", run(shallow, 0), "->", run(deeper, 0))

Three things to read off:

  • With seed 1, the deeper trees win by 0.02. That's the teammate's result.
  • Run seed 1 again and you get the same two numbers, to the last digit. The run is reproducible: anyone with the same data, code, settings and seed gets the same score.
  • With seed 0, the order flips. The shallower trees win.

Nothing about the models changed between the first line and the third. Only the seed did. So a score is a fact about a model and a seed, and a run that doesn't record its seed can't be checked by anyone.

Quick check
You rerun a colleague's training script with the settings they sent you and get 0.941 where they reported 0.953. Their run recorded no seed. What can you conclude?
  1. Nothing yet: the seed, the data or the code could differ
  2. The model has gotten worse since they trained it
  3. Your machine is slower, which lowers accuracy a little
  4. Their number was wrong, since yours used the same settings
Show the answer

Nothing yet: the seed, the data or the code could differ

Identical settings are not an identical run. Without the seed, a fingerprint of the data and the version of the code, a different score has several possible causes and no way to separate them.

Try it now

The first cell's output, drawn. The outlined bar wins in each pair.

What a run has to carry

For a score to be checked later, the run needs to keep everything that produced it. That's a short list:

FieldWhat it answers
ParametersWhich settings? Tree depth, learning rate, batch size, every one.
SeedWhich random choices?
Data versionExactly which rows and columns, in which state?
Code versionWhich version of the training script? In practice, the git commit.
MetricsWhat came out: accuracy, loss, time taken.

Together these are a run record. A dictionary holds one, and JSON turns it into a line of text you can save:

Uses the setup above, 12 lines.

import json

record = {
    "params": {"n_estimators": 20, "max_depth": 6},
    "seed": 1,
    "data": "6fdb5b36182d",
    "code": "a41c9e7",
    "metrics": {"accuracy": 0.9467},
}
print(json.dumps(record))
It runs right here in your browser, with nothing to install.

Two of those fields are short strings of letters and digits. "code" is a git commit, which you met in the git lessons: a name for one exact state of the files. "data" is the same idea applied to a dataset, and the next part shows how to make one.

Go deeper What the big tools add

MLflow, Weights & Biases and similar trackers store this same record: parameters, metrics, the code version, and files the run produced (they call the files artifacts). On top of it they add a web page that charts runs side by side, and a server so a whole team writes to one place. The record is the part that matters. If you understand the five fields, each tool is a different screen over the same table.

Quick check
What does this print?
import json

a = {"lr": 0.1, "depth": 3}
b = {"depth": 3, "lr": 0.1}
print(
    a == b, json.dumps(a) == json.dumps(b),
    json.dumps(a, sort_keys=True) == json.dumps(b, sort_keys=True),
)
  1. False False True
  2. True False False
  3. True False True
  4. True True True
Show the answer

True False True

The dictionaries are equal: same keys, same values. Written as text they differ, because each keeps the order its keys were added in. With sort_keys=True both are written in alphabetical key order, so the text matches and so would a hash of it.

In your own words
In your own words: what does it mean for a run to be reproducible, and what has to be written down to make it so?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: a hash, a log and a paired comparison

The tracker below is a dozen lines of Python. Real tools add screens and a server, not a different record.

A fingerprint for data and settings

"We used the March export" is not a data version. Files get fixed, re-exported and overwritten under the same name. What you want is a short code that changes whenever the content does. A hash function gives you one: it reads any amount of bytes and returns a fixed-length string, the hash or fingerprint. The same bytes always give the same hash. Change one character and the hash changes completely.

Uses the setup above, 12 lines.

import hashlib

print(hashlib.sha256(b"hello").hexdigest()[:12])
print(hashlib.sha256(b"hellp").hexdigest()[:12])
print(len(hashlib.sha256(b"hello").hexdigest()), "characters in the full hash")

sha256 is the hash function. .hexdigest() writes its result as 64 characters, and the first 12 are plenty to tell runs apart. The b before the quotes makes the text bytes, which is what a hash function reads.

For a NumPy array, .tobytes() hands over the raw numbers:

Uses the setup above, 12 lines.

import hashlib
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=600, n_features=8, n_informative=4, random_state=0)
print("data:           ", hashlib.sha256(X.tobytes() + y.tobytes()).hexdigest()[:12])

X2 = X.copy()
X2[0, 0] += 1e-9          # one cell out of 4,800, moved by a billionth
print("one cell nudged:", hashlib.sha256(X2.tobytes() + y.tobytes()).hexdigest()[:12])

A change far too small to see in a table gives an unrelated fingerprint. That is the property you need: if two runs show the same data hash, they trained on the same numbers.

Settings get a fingerprint too. It gives every configuration a short name to group runs by. There's one trap. Two dictionaries with the same keys in a different order are equal in Python, but they turn into different text, and different text hashes differently. sort_keys=True writes the keys in a fixed order:

Uses the setup above, 12 lines.

import hashlib, json

def fingerprint(config):
    text = json.dumps(config, sort_keys=True)
    return hashlib.sha256(text.encode()).hexdigest()[:12]

print(fingerprint({"n_estimators": 20, "max_depth": 5}))
print(fingerprint({"max_depth": 5, "n_estimators": 20}))
print(fingerprint({"max_depth": 6, "n_estimators": 20}))

The first two match, because they are the same settings. The third differs in one value and shares nothing.

Drill

A fingerprint that ignores key order

Write fingerprint(config): the first 12 characters of the SHA-256 hash of the configuration written as JSON. Two configurations with the same content must get the same fingerprint, whatever order their keys were written in, including inside nested dictionaries. Any change to a value must change it.

Uses the setup above, 12 lines.

import hashlib, json


def fingerprint(config):
    # write the config as text in a fixed key order, then hash the text
    return ""


print(fingerprint({"max_depth": 5, "n_estimators": 20}))
In the lesson, your code is checked as you write it.

A tracker in a dozen lines

Now somewhere to keep the records. The simplest store that works is a text file with one JSON record per line, a format called JSON Lines. A run appends its line when it finishes and nothing is ever edited or deleted. That rule is the point: a log you can rewrite is a log you can't trust.

Uses the setup above, 12 lines.

def log_run(path, record):
    with open(path, "a") as f:                 # "a" appends: earlier lines stay
        f.write(json.dumps(record, sort_keys=True) + "\n")

def load_runs(path):
    with open(path) as f:
        return [json.loads(line) for line in f if line.strip()]

if os.path.exists("runs.jsonl"):
    # start this demonstration from an empty log
    os.remove("runs.jsonl")
for depth in (5, 6):
    params = {"n_estimators": 20, "max_depth": depth}
    for seed in range(8):
        log_run(
            "runs.jsonl",
            {"params": params, "seed": seed, "accuracy": run(params, seed)},
        )

runs = load_runs("runs.jsonl")
print(len(runs), "runs logged")
print(runs[0])

Sixteen runs: two settings, eight seeds each. Every line can be read back into the dictionary that wrote it.

Reproducing a run is now a procedure, not a memory test. Take a record, check that your data has the hash it names, check out the commit it names, and run with its parameters and seed. If the score matches the record, the result is confirmed. If it doesn't, something that should have been in the record wasn't, and you've found a gap in your tracking before it cost you anything.

Quick check
A log file holds three lines, one JSON record each. What does this print?
import json

lines = [
    '{"seed": 0, "accuracy": 0.91}',
    '{"seed": 1, "accuracy": 0.95}',
    '{"seed": 2, "accuracy": 0.93}',
]
runs = [json.loads(line) for line in lines]
best = max(runs, key=lambda r: r["accuracy"])
print(len(runs), best["seed"])
  1. 3 0.95
  2. 3 1
  3. 3 2
  4. 1 1
Show the answer

3 1

Each line becomes one dictionary, so there are 3 runs. max with key= picks the run with the highest accuracy, 0.95, and the code prints that run's seed, which is 1.

Quick check
A team's tracking file lets anyone edit earlier lines to "fix" a metric they think was mistyped. What is lost?
  1. Trust that a recorded score is the one the run produced
  2. The speed of loading the file back into Python
  3. The option of using JSON for each record
  4. The ability to add new runs to the end of the file
Show the answer

Trust that a recorded score is the one the run produced

A tracking log is evidence. Once past lines can change, a record no longer proves what happened, and a reproduction that disagrees with it can't tell you whose number is wrong. Corrections go in as new lines that say what they correct.

Is it better, or was it lucky?

The log holds eight seeds for each setting. Look at all of them before deciding anything:

Uses the setup above, 12 lines.

for depth in (3, 5, 6):
    scores = [run({"n_estimators": 20, "max_depth": depth}, seed) for seed in range(8)]
    print(
        "depth", depth, "mean", round(statistics.mean(scores), 4), "low", min(scores),
        "high", max(scores),
    )

Depth 5 and depth 6 average 0.9533 and 0.9542: a gap of about 0.001, with each one ranging over 0.06 from its worst seed to its best. The teammate's 0.02 was seed 1. Depth 3 is different. Its average is 0.05 lower.

Compare seed by seed. Each seed gives both settings the same train and test rows, so the fair comparison is the gap within each seed. Some test splits are easier than others for every model, and taking the gap within a seed removes that. This is a paired comparison.

Uses the setup above, 12 lines.

scores = {}
for d in (3, 5, 6):
    scores[d] = [run({"n_estimators": 20, "max_depth": d}, s) for s in range(8)]
for a, b in ((5, 6), (3, 6)):
    gaps = [round(q - p, 4) for p, q in zip(scores[a], scores[b])]
    se = statistics.stdev(gaps) / math.sqrt(len(gaps))
    print("depth", a, "->", b, "| deeper wins", sum(g > 0 for g in gaps), "of 8",
          "| mean gap", round(
              statistics.mean(gaps), 4,
          ), "| standard error", round(se, 4))

From 5 to 6, the deeper trees win 4 seeds of 8, and the mean gap of 0.0009 is a small fraction of its standard error, 0.006. That is no evidence of a difference. From 3 to 6 they win 8 of 8, and the mean gap of 0.0475 is about eight standard errors. That one is real.

A working rule: call a gap real only when it is more than two standard errors from zero. Below that, write "no difference found" in the log and keep the simpler model.

Go deeper Why two standard errors is a floor here

These eight splits all draw from the same 600 rows, so they overlap and aren't eight independent experiments. The true uncertainty is somewhat larger than the standard error says. With only a handful of seeds the bar should also be higher than two. A/B testing a model change gives the careful version. When a gap only just clears the rule, run more seeds before you believe it.

Quick check
From the lesson on samples and confidence: eight gaps have a standard deviation of s. What is the standard error of their mean?
  1. s itself
  2. s ÷ 8
  3. s ÷ √8
  4. s × √8
Show the answer

s ÷ √8

The standard error of a mean is the standard deviation divided by the square root of the count. Averaging eight values makes the mean steadier than any one value, by a factor of √8, about 2.8.

A standard error of the mean gap divides the spread of the gaps by the square root of the number of seeds. Plot the gap seed by seed to see it:

Uses the setup above, 12 lines.

import matplotlib.pyplot as plt

gaps = [run({"n_estimators": 20, "max_depth": 6}, s)
        - run({"n_estimators": 20, "max_depth": 5}, s) for s in range(8)]
plt.bar(range(8), gaps)
plt.axhline(0, color="black")
plt.xlabel("seed")
plt.ylabel("depth 6 minus depth 5")
plt.show()
print([round(g, 4) for g in gaps])

The bars straddle zero: some seeds favor depth 6, some depth 5. A real improvement is positive on nearly every seed.

Quick check
Two settings were each run with seeds 0 to 4. What does this print?
a = [0.90, 0.92, 0.91, 0.93, 0.90]
b = [0.92, 0.93, 0.90, 0.95, 0.93]
gaps = [round(q - p, 2) for p, q in zip(a, b)]
print(sum(g > 0 for g in gaps), max(gaps))
  1. 5 0.03
  2. 4 0.03
  3. 1 0.03
  4. 4 0.02
Show the answer

4 0.03

The gaps are 0.02, 0.01, -0.01, 0.02 and 0.03. Four are above zero, and the largest is 0.03.

Drill

Gaps on the seeds two settings share

Runs get lost: a job crashes, a seed is skipped. Write paired_gaps(a, b). Each argument is a dictionary from seed to score for one setting. Return the list of gaps b - a, rounded to 4 places, for the seeds that appear in both, in order of seed. A seed that only one setting ran can't be paired, so it's left out.

Uses the setup above, 12 lines.

def paired_gaps(a, b):
    # gaps b - a on the seeds both settings ran, in seed order
    return []


print(paired_gaps({0: 0.90, 1: 0.92, 2: 0.91}, {0: 0.92, 2: 0.90, 3: 0.99}))
In the lesson, your code is checked as you write it.
Worked example

Worked example: reproduce a run from its record

A record from last week says a depth-6 forest scored 0.9467 with seed 1 on data with the hash 6fdb5b36182d. Before building on that number, confirm it: check the data first, then rerun.

Uses the setup above, 12 lines.

import hashlib
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

record = {"params": {"n_estimators": 20, "max_depth": 6}, "seed": 1,
          "data": "6fdb5b36182d", "metrics": {"accuracy": 0.9467}}

X, y = make_classification(n_samples=600, n_features=8, n_informative=4, random_state=0)
have = hashlib.sha256(X.tobytes() + y.tobytes()).hexdigest()[:12]
print("data matches the record:", have == record["data"])

seed = record["seed"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=seed)
model = RandomForestClassifier(**record["params"], random_state=seed).fit(X_tr, y_tr)
again = round(float(model.score(X_te, y_te)), 4)
print(
    "recorded:", record["metrics"]["accuracy"], "| reproduced:", again, "| match:",
    again == record["metrics"]["accuracy"],
)
In the lesson you can change this and run it.

The data check comes first on purpose. If the hash didn't match, a different score would be expected and the rerun would tell you nothing about the record. Here the data matches and the score comes back identical, so the number is confirmed and safe to build on. Try changing the seed in the record to 2: the data still matches, the score doesn't, and that is exactly the situation of a run that never wrote its seed down.

One more place seeds matter: PyTorch. torch.manual_seed fixes the random numbers the model's weights start from. This academy's torch is a teaching subset (same code, CPU only, and its seeded numbers differ from real PyTorch), so the cell prints only that two runs match:

Uses the setup above, 12 lines.

import torch

torch.manual_seed(0)
first = torch.randn(3)
torch.manual_seed(0)
second = torch.randn(3)
print(torch.equal(first, second))
In your own words
A single run of a new setting beats the old one by 0.02. What do you do before reporting it as an improvement?

Write a sentence or two, then compare it with a good answer in the lesson.

On the job: MLflow, runs and a team's single table

A team keeps one place where every run lands, so an answer to "which setting is better" is a query, not a search through notebooks.

Where it comes up in a week

A colleague posts a chart: the new setting beats the old one by two points. You ask for the seeds, then compare the two settings seed by seed. If the gap is less than two standard errors, the entry in the tracker says "no difference found" and the simpler setting stays. When a run does win, its record names the commit and data hash that the model registry will point at.

Goes deeper in

In your own words
Describe the steps for reproducing a run from its record.

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 11 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 11 review cards for spaced repetition

Try a quiz question

Which set of fields makes a training run reproducible?

  1. Parameters, seed, data version, code version and the metrics
  2. The model's name, the date and the final accuracy
  3. The parameters and the accuracy to four decimal places
  4. The seed and the name of the person who ran it

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Part of a free course 11 lessons

Free MLOps course: running machine learning in production

Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write