A model's settings, its hyperparameters, can be worth more accuracy than a change of model, and nobody knows the right values in advance. So you search. A search is a loop that trains many models, which makes it the most expensive thing most ML teams do, and the easiest place to fool yourself: try enough settings and one of them will look good by luck. This lesson is about spending a search budget well and reading its result honestly.
What you'll be able to do
Work out what a grid search costs, and say why it spends most of that on settings that don't matter
Run a random search, and sample rates on a log scale
Cut a search's cost with successive halving, and name the case where it drops the wrong configuration
Explain why the winner's validation score is too high, and report a search honestly
Quick check
A guess before you start. You can afford 9 training runs to tune two settings. One of the two turns out to matter a great deal and the other hardly at all, but you don't know which. A 3 × 3 grid tries how many different values of the setting that matters?
9
3
1
6
Show the answer
3
A grid repeats each value of one setting for every value of the other. Nine runs, but only three values of each setting, so only three looks at the one that matters. Nine runs with each value drawn at random would have tried nine.
The idea: a search space, a budget and a winner's curse
A search is a budget spent on settings, and its cost is a product, so it leans on Counting. The check below recalls how a validation score is used.
What a grid costs
You met GridSearchCV in the scikit-learn lessons: list a few values for each setting and it trains every combination. The trouble is in the word "every". The number of combinations is the product of the list lengths, and with cross-validation each combination is trained once per fold.
It runs right here in your browser, with nothing to install.
product yields every way of picking one value from each list. Four settings with three values each is already 81 combinations and 405 training runs. Add a fifth setting and it's 1,215. The cost multiplies with every setting you add, and three values is a coarse look at each one.
A grid also assumes you know which values are worth listing. If the best learning rate is 0.3 and your list says 0.01, 0.1 and 1.0, the grid will never find it, however many runs it spends.
Try it now
Each new setting multiplies the bill. Each new value raises the base.
The lists have 2, 3 and 2 values, so there are 2 × 3 × 2 = 12 combinations. Each one is a tuple with one value from each list, and the first takes the first value of each.
Random search
Here is a real search problem. A gradient boosting model (the boosting lesson built one) has a learning_rate, how big a step each new tree takes, and a subsample, the share of rows each tree sees. evaluate trains one model and returns its accuracy on 300 validation rows.
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=900, n_features=10, n_informative=4,
flip_y=0.08, random_state=1)
X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=300, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X_dev, y_dev, test_size=300, random_state=0)
defevaluate(lr, subsample, trees=40):
model = GradientBoostingClassifier(n_estimators=trees, learning_rate=lr,
subsample=subsample, max_depth=2, random_state=0)
returnround(float(model.fit(X_tr, y_tr).score(X_val, y_val)), 4)
print("learning rate | subsample 0.5, 0.75, 1.0")
for lr in (0.001, 0.01, 0.1, 0.3, 1.0):
print(lr, [evaluate(lr, ss) for ss in (0.5, 0.75, 1.0)])
Read down the rows and accuracy climbs from 0.51 to 0.84, then falls. Read across a row and, except at the extreme rate of 1.0, it barely moves. The learning rate matters here and the subsample mostly doesn't. That pattern is typical: of the settings you tune, a few carry most of the effect, and you rarely know which beforehand.
A grid handles this badly, as the opening question showed. Random search draws each setting independently for every run, so every run tests a new value of every setting:
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=900, n_features=10, n_informative=4,
flip_y=0.08, random_state=1)
X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=300, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X_dev, y_dev, test_size=300, random_state=0)
defevaluate(lr, subsample, trees=40):
model = GradientBoostingClassifier(n_estimators=trees, learning_rate=lr,
subsample=subsample, max_depth=2, random_state=0)
returnround(float(model.fit(X_tr, y_tr).score(X_val, y_val)), 4)
grid = [(lr, ss) for lr in (0.01, 0.1, 1.0) for ss in (0.5, 0.75, 1.0)]
rng = np.random.default_rng(0)
rand = [(round(float(10 ** rng.uniform(-3, 0)), 4),
round(float(rng.uniform(0.5, 1.0)), 2)) for trial inrange(9)]
for name, configs in (("grid", grid), ("random", rand)):
scored = [(evaluate(lr, ss), lr, ss) for lr, ss in configs]
print(name, "| learning rates tried:", sorted({lr for lr, ss in configs}))
print(name, "| best (score, rate, subsample):", max(scored))
Same budget, nine runs each. The grid looked at three learning rates and the random search at nine, five of them between 0.15 and 0.39, where the table above peaks. The scores are 0.8367 and 0.84. That gap is one validation row out of 300, so don't read it as a win: on this problem the grid's list happened to include a good value. What random search buys is that it doesn't depend on that luck. You give it a range, not a list.
Rates live on a log scale
The random search above drew the learning rate as 10 ** rng.uniform(-3, 0), not as rng.uniform(0.001, 1.0). The reason is that a learning rate matters by its order of magnitude: going from 0.001 to 0.01 is as big a change as going from 0.1 to 1. Drawing the exponent evenly gives every factor of ten the same share of the runs. This is called sampling on a log scale, or log-uniform sampling.
Drawn evenly between 0.001 and 1, fewer than 1 draw in 100 lands below 0.01, and 9 in 10 land above 0.1: the search would never look at two of the three factors of ten. On the log scale each gets about a third. Use it for anything you'd naturally write as a power of ten: learning rates, regularization strengths, weight decay.
Go deeper Searches that learn as they go
Random search ignores what earlier runs found. Bayesian optimization uses it: after each run it fits a cheap model of "score as a function of settings" and picks the next settings where that model expects a good score or is most unsure. Libraries such as Optuna do this. It helps most when each run is expensive and the budget is a few dozen runs. Random search remains the baseline any smarter method has to beat, and it runs in parallel with no coordination.
Quick check
A search draws a regularization strength on a log scale from 0.001 to 1. What does this print: the middle of that range, the value half the draws fall below?
The exponents run from -3 to 0, so their middle is -1.5, and 10 to the power -1.5 is about 0.0316. On a log scale the midpoint of 0.001 and 1 is 0.0316, not 0.5.
In your own words
In your own words: why does a grid search get expensive so quickly?
Write a sentence or two, then compare it with a good answer in the lesson.
In code: log sampling, halving and an honest final score
Each cell trains small gradient boosting models on 900 rows, so a full search runs in the page.
Drill
Draw one configuration from a search space
A search space maps each setting's name to how to draw it: ("uniform", low, high), ("log", low, high) or ("choice", [options]). Write sample_config(space, rng): a dictionary with one drawn value per setting. For "uniform" use rng.uniform(low, high). For "log" draw the exponent evenly between math.log10(low) and math.log10(high) and return 10 to that power. For "choice" use rng.choice(options).
import math
import numpy as np
defsample_config(space, rng):
config = {}
# one drawn value for each setting in the spacereturn config
space = {"lr": ("log", 0.001, 1.0), "subsample": ("uniform", 0.5, 1.0),
"depth": ("choice", [2, 3, 4])}
print(sample_config(space, np.random.default_rng(0)))
In the lesson, your code is checked as you write it.
Stop the losers early
Most configurations in a search are bad, and many are visibly bad long before training finishes. Successive halving uses that. Give every configuration a small budget, keep the better half, double the budget, and repeat until one is left. For boosting, the budget is the number of trees. For a neural network it would be epochs.
Sixteen configurations, four rounds, 320 trees trained in all. Training every one of the sixteen to 80 trees costs 1,280, four times as much, and here it names the same winner.
Notice the leader changed from round to round. That is the method's weak point: it assumes that whoever is ahead early will be ahead late. A configuration that starts slowly and finishes strongly, a low learning rate for instance, can be cut before it shows what it can do. With a different draw of sixteen configurations it goes wrong:
This time halving kept a configuration that scores 0.8333 at 80 trees and dropped one that reaches 0.86. The two met in the last round, at 40 trees, and the order there was not the order at 80. Halving is a trade: a quarter of the cost, for a winner that is usually near the best and sometimes isn't the best.
Quick check
Successive halving starts 32 configurations at a budget of 1 epoch, keeps the better half each round and doubles the budget, until one is left. What does this print?
Each round costs 32 epochs: 32 × 1, 16 × 2, 8 × 4, 4 × 8, 2 × 16. Five rounds make 160. By then the budget has doubled five times to 32 epochs, and training all 32 configurations for 32 epochs would cost 1,024.
Quick check
Which configuration is successive halving most likely to throw away by mistake?
One with a large learning rate that scores well from the first round
One that scores worst at every budget, small or large
One with a small learning rate that improves slowly but ends highest
One that is identical to another configuration in the search
Show the answer
One with a small learning rate that improves slowly but ends highest
Halving judges everyone at a small budget first. A slow starter looks bad there and is cut, even if it would have overtaken the rest with a full budget.
Drill
Successive halving
Write successive_halving(configs, evaluate, budget). While more than one configuration is alive: score each with evaluate(config, budget), add alive × budget to the cost, keep the better half (len(alive) // 2 of them, and on a tie the one that came earlier in the list stays ahead), and double the budget. Return a tuple: the last configuration standing, and the total cost.
defsuccessive_halving(configs, evaluate, budget):
alive = list(configs)
cost = 0# rounds: score, count the cost, keep the better half, double the budgetreturn alive[0], cost
deftoy(config, budget):
return config * budget # a made-up score: bigger configs do betterprint(successive_halving([3, 1, 4, 2], toy, 1))
In the lesson, your code is checked as you write it.
A plot of the first table shows why a learning rate wants a log axis:
The curve climbs from 0.001 to 0.3 and falls at 1.0. The good values sit in one narrow band.
The winner's score is too high
Here is the trap that catches every search. Suppose 30 settings are all exactly as good as each other: each would get 80% of new cases right. Score each one on 300 validation rows. By chance some will get a few more right and some a few fewer. Now pick the best.
import numpy as np
rng = np.random.default_rng(0)
scores = rng.binomial(300, 0.80, size=30) / 300print("30 settings, each truly 0.80 | lowest", round(float(scores.min()), 3),
"| highest", round(float(scores.max()), 3))
best = [rng.binomial(300, 0.80, size=30).max() / 300for search inrange(1000)]
print("mean best score over 1,000 searches:", round(float(np.mean(best)), 3))
rng.binomial(300, 0.80) counts how many of 300 rows a setting gets right when each has an 80% chance. The "best" of thirty identical settings scores 0.847, and across a thousand such searches the winner averages 0.846. None of them is better than 0.80. Choosing the maximum selects for luck as well as skill, so the winner's validation score is too high, and the more settings you try the more too high it is.
The same thing happens in a real search:
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=900, n_features=10, n_informative=4,
flip_y=0.08, random_state=1)
X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=300, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X_dev, y_dev, test_size=300, random_state=0)
deffit(lr, subsample):
return GradientBoostingClassifier(n_estimators=40, learning_rate=lr,
subsample=subsample, max_depth=2,
random_state=0).fit(X_tr, y_tr)
rng = np.random.default_rng(0)
trials = [(float(10 ** rng.uniform(-3, 0)), float(rng.uniform(0.5, 1.0)))
for trial inrange(12)]
models = [fit(lr, ss) for lr, ss in trials]
val = [round(float(m.score(X_val, y_val)), 4) for m in models]
i = val.index(max(val))
print("winner on validation:", val[i], "| the same model on the test set:",
round(float(models[i].score(X_test, y_test)), 4))
The search's winner scores 0.8433 on the rows that chose it and 0.8233 on rows it has never met. One search is one draw, as Experiment tracking showed, so don't read the size of that gap too closely. The direction is what you can rely on.
So the rule from the cross-validation lesson carries over, with more force. The validation score picks the winner. The test score, measured once on the winner alone, is the number you report. And every trial goes in the run log from Experiment tracking, losers included: a result that says "best of 12" means something different from one that says "best of 1,200".
Quick check
From the cross-validation lesson: a search reports a best validation score of 0.80, and the untouched test set gives the chosen model 0.78. Which number estimates how it will do on new data?
0.80: a search always reports its true best score
0.80: it comes from more rows of data in total
0.79: the average of the two is fairer than either
0.78: the test set played no part in choosing the model
Show the answer
0.78: the test set played no part in choosing the model
The validation rows chose the winner, so its score on them is flattered by the choice. The test rows had no say, so the test score is the honest estimate.
Quick check
Two teams tune the same model on the same data. Team A tries 10 configurations and Team B tries 1,000. Each reports its winner's validation score. What should you expect?
A's reported score is flattered more, since 10 trials is a small sample
B's score is exact, since 1,000 trials average the luck away
B's reported score is flattered more, since the best of 1,000 has more luck in it
The two scores are equally trustworthy, since the data is the same
Show the answer
B's reported score is flattered more, since the best of 1,000 has more luck in it
The maximum of many noisy scores drifts further above the truth as the number of scores grows. B may well have found a better model, but its validation score overstates it by more. Only a test set neither team searched on can compare them.
Worked example
Worked example: a search you could hand to a teammate
Put the pieces together: a seeded random search on a log scale, every trial kept, the winner picked on validation and scored once on the test set.
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=900, n_features=10, n_informative=4,
flip_y=0.08, random_state=1)
X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=300, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X_dev, y_dev, test_size=300, random_state=0)
rng = np.random.default_rng(3) # the search has a seed too
log = []
for trial inrange(10):
config = {"learning_rate": round(float(10 ** rng.uniform(-3, 0)), 4),
"subsample": round(float(rng.uniform(0.5, 1.0)), 2)}
model = GradientBoostingClassifier(n_estimators=40, max_depth=2, random_state=0,
**config).fit(X_tr, y_tr)
log.append({"trial": trial, "config": config,
"validation": round(float(model.score(X_val, y_val)), 4),
"model": model})
winner = max(log, key=lambda t: t["validation"])
print("trials:", len(log), "| validation scores from",
min(t["validation"] for t in log), "to", winner["validation"])
print("winner: trial", winner["trial"], winner["config"])
print("report: test accuracy", round(float(winner["model"].score(X_test, y_test)), 4),
"(best of", len(log), "trials)")
In the lesson you can change this and run it.
Four habits are in those lines. The search itself is seeded, so the same ten configurations can be drawn again. Every trial is kept, not only the winner, so anyone can see how many were tried and how the scores spread. The winner is chosen on validation data only. And the test set is touched exactly once, by the winner, for the number that goes in the report, next to the count of trials behind it.
In your own words
What does successive halving assume, and what does it cost you when the assumption fails?
Write a sentence or two, then compare it with a good answer in the lesson.
On the job: Optuna, scikit-learn and a search you can rerun
A search is a job: it has a seed, a trial log and a budget, and it ends with a test score measured once.
Where it comes up in a week
Someone asks for a better score before the end of the week. You set a budget in runs, not days, draw the learning rate and weight decay on log scales, log every trial in Experiment tracking's format, and report the test score of the winner measured once. If the leader changes from round to round, you do not use halving on that model.
03Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
04Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
05CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
06Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
07Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
08Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
09Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
11Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
13A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
14Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
15Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
16Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
17Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
18Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
19Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
20Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
21Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
22A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
23Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
24Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
25Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
26Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
27Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
28Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
29Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
30Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
31Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
32A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
33What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
34Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
35Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
36Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
37Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
43Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write