A classifier that is right 95 times out of 100 sounds good, until you learn that only 5 of the 100 cases were yeses and it said no to every one of them. Accuracy hides which mistakes a model makes. So professional evaluation counts the mistakes by kind, in a small table called the confusion table, and reads a few numbers from it: precision, recall and F1. This lesson builds the table from the arrow detector of lesson 102, works each number by hand and in code, and moves the threshold to watch them trade against each other.
What you'll be able to do
Fill a confusion table from a classifier's calls: true positives, false negatives, false positives, true negatives
Compute accuracy, precision, recall and F1 from the table, and say which one matters when a miss costs more than a false alarm
Count the four cells in Python with a loop, then with array comparisons, and in PyTorch with (pred == 1) & (y == 1)
Move a threshold and predict which scores rise and which fall, and read a three-class table row by row
Quick check
Have a guess before you start. 100 trail signs, 5 of them arrows. A detector says "not an arrow" for every sign. How accurate is it?
0%, because it finds no arrows
95%, though it never finds an arrow
50%, a coin toss
5%, the share of arrows
Show the answer
95%, though it never finds an arrow
It's right about all 95 boards and wrong about the 5 arrows: 95 of 100. A high accuracy, from a detector that is useless for the one thing it's for.
The math: four cells and four scores
Purpose. Count a classifier's right and wrong answers by kind, and read four scores from the count.
The four cells. Each time a yes-or-no classifier meets an example, one of four things happens. Call the yes class "1" (an arrow) and the no class "0" (anything else).
It says 1 and the answer is 1: a true positive, TP. A hit.
It says 0 and the answer is 1: a false negative, FN. A miss.
It says 1 and the answer is 0: a false positive, FP. A false alarm.
It says 0 and the answer is 0: a true negative, TN.
"True" or "false" says whether the call was right; "positive" or "negative" says what was called. Count each kind and lay the counts out in a 2 × 2 table, rows for what really is and columns for what the model said. That's the confusion table (often called the confusion matrix): it shows exactly what the model confuses with what.
Quick check
From lesson 90: a classifier called 12 signs "tent", and 8 of them really were tents. What is P(tent | called tent)?
0.4
0.333
1.5
0.667
Show the answer
0.667
Given the 12 called tent, 8 are tents: 8 ÷ 12 ≈ 0.667. That lesson named it precision; here it comes from the said-1 column of the table.
The picture. Here are the ten signs from lesson 102: the arrow detector's chance for each, and the real answer. At a threshold of 0.5 each chance becomes a call. Then the signs drop into the table one at a time, each lighting its cell; the totals sit in the "all" row and column. Last, each score lights the cells it reads and writes itself as a fraction. Watch which cells precision reads, and which recall reads.
Try it now
Then set threshold to 0.7 and press Use these numbers: precision rises to 1, recall falls to 0.6, and accuracy stays at 0.8.
In plain words: accuracy is the share of all calls that were right. Precision: of the yes calls, how many were right. Recall: of the real yeses, how many were found. F1 blends the two.
Why F1 is a harmonic mean. A plain average would let one good score hide a bad one. A detector with precision 1 and recall 0.1 finds one arrow in ten, but averages (1 + 0.1) ÷ 2 = 0.55. Its F1 is 2 × 1 × 0.1 ÷ (1 + 0.1) = 0.18. The harmonic mean sits near the smaller of the two, so F1 is high only when both are. It is also 2TP ÷ (2TP + FP + FN): true negatives don't count at all, which is why F1 suits a rare yes class.
Quick check
A detector says yes 8 times and 6 of those are right. There were 10 real yeses. What are its precision and recall?
Both 0.6
Precision 0.75, recall 0.6
Precision 0.75, recall 0.8
Precision 0.6, recall 0.75
Show the answer
Precision 0.75, recall 0.6
Precision is 6 of its 8 yes calls, 0.75. Recall is 6 of the 10 real yeses, 0.6.
Quick check
Precision is 0.9 and recall is 0.1. Which is F1 closest to?
0.18
0.5
0.9
0.1
Show the answer
0.18
2 × 0.9 × 0.1 ÷ (0.9 + 0.1) = 0.18 ÷ 1 = 0.18: near the weak score, far below the plain average of 0.5.
Moving the threshold. Every call came from a threshold, and moving it moves the cells. Raise it and the model says yes less often: fewer false alarms, so precision usually rises, but more misses, so recall falls. Lower it and the trade runs the other way. Below, 200 cases are placed by a model's score, real arrows above and boards below, with the table and the four scores for the threshold you choose.
In your own words
In your own words: why can't accuracy alone tell you whether a detector is any good?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: count the cells, then score them
Purpose. Count the four cells with a loop, then with array comparisons, and compute the four scores.
The loop asks of each sign, in turn, which of the four cases it is:
The calls are 1, 0, 1, 0 and so on, and the counts are 4, 1, 1 and 4: the table the flow filled. The order of the branches matters. The first catches the hits; by the second, y is 1 but the call wasn't, so it's a miss; by the third, y is 0, so a 1 call is a false alarm; whatever is left is a true negative.
Code flow. Step through the same loop on four examples, one of each kind, and watch which branch each one takes and which counter goes up.
Array comparisons. NumPy can ask all ten signs at once. y == 1 is an array of Trues and Falses, and & joins two of them place by place: True only where both are. np.sum counts the Trues. Each comparison needs its own brackets, because & is worked out before ==.
[1] * 5 + [0] * 95 is a list of five 1s then ninety-five 0s. The model is right about every 0: accuracy 0.95. But its recall is 0.0: it found none of the 5 arrows. Its precision can't even be worked out, since it never said 1 and 0 ÷ 0 has no answer; libraries report 0 and print a warning. Whenever one class is rare, check recall before you trust an accuracy.
A named mistake: swapping precision and recall. Both divide TP by something, and the two somethings are easy to mix up. Swap them at a threshold of 0.3 and a detector that misses nothing (recall 1.0) gets reported as one that never raises a false alarm. One way to keep them apart: Precision divides by the Predicted yeses, the said-1 column, TP + FP; Recall divides by the Real yeses, the real-1 row, TP + FN.
The first divides by the real yeses, TP + FN = 3: recall 1.0, nothing missed. The second divides by the yes calls, TP + FP = 5: precision 0.6.
Drill
Fix the four scores
scores(y, pred) has three slips: two of its scores are swapped, F1 is a plain average, and a model that never says 1 crashes precision with 0 ÷ 0. Fix it so it returns accuracy, precision, recall and F1 as floats, with precision 0.0 when nothing was said 1 and F1 0.0 when precision and recall are both 0.
In the lesson, your code is checked as you write it.
In your own words
Why do the four cells have to be counted from the same calls before any score is worked out?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: scoring the sign detector
Purpose. Choose a detector's threshold by what its mistakes cost, and read a three-way sign reader's table row by row.
The arrow detector at three thresholds. The same ten chances, called at 0.3, 0.5 and 0.7. Each line prints the threshold, then accuracy, precision, recall and F1:
Accuracy doesn't move: 0.8 at every threshold. Underneath, the mistakes change places. At 0.3 the detector finds every arrow (recall 1.0) but two of its seven yes calls are boards (precision 0.714). At 0.7 every yes call is an arrow (precision 1.0), but it finds only 3 of the 5 (recall 0.6). F1 is best at 0.3 here, 0.833, but the right threshold depends on what each mistake costs, which no score knows.
Quick check
A trail-rescue alert flags hikers who may be in trouble. A miss could cost a life; a false alarm costs a phone call. Which score should it keep high?
Recall, even if precision falls
F1, since it blends both
Precision, even if recall falls
Accuracy, since it counts everything
Show the answer
Recall, even if precision falls
Recall counts the real cases found. When a miss is far worse than a false alarm, lower the threshold until recall is high, and accept the extra calls.
A three-class table, read row by row. The sign reader of the capstone (lesson 118) tells three kinds apart: arrows, tent signs and trail markers. Its table is 3 × 3, rows for the real kind and columns for what it said. Each kind gets its own recall from its row, the right calls (on the diagonal) over the row's total, and its own precision from its column. T.diagonal() reads the diagonal, and T.sum(axis=1) adds each row:
Accuracy is 0.8, 24 right of 30. Row by row, the recalls are 0.9 for arrows, 0.7 for tent signs and 0.8 for markers: tent signs are the ones it misses, 2 called arrows and 1 a marker. Column by column, the precisions are 0.818, 0.7 and 0.889. One number would hide all of that.
Quick check
In the sign reader's table, which kind's precision is lowest, and what does that mean?
Markers, 0.889: it calls markers least
Tent signs, 0.7: it finds only 7 in 10 tents
Tent signs, 0.7: of its tent calls, 3 in 10 are wrong
Arrows, 0.818: it misses arrows most
Show the answer
Tent signs, 0.7: of its tent calls, 3 in 10 are wrong
The tent column holds 1 + 7 + 2 = 10 calls, 7 of them right: precision 0.7, so 3 of every 10 tent calls are another kind.
In PyTorch. The same comparisons work on tensors. (pred == 1) & (y == 1) is a tensor of Trues and Falses, .sum() counts them into a one-number tensor, and .item() takes the number out:
The same calls, the same TP 4, FP 1 and FN 1, and the same precision and recall, 0.8 each, as the NumPy cells. In a training loop this is how a model's held-back calls get scored at the end of each pass.
Drill
Which kind does the reader miss?
A second run of the sign reader gave the table T (rows: real arrow, tent, marker; columns: what it said). Set recalls to each kind's recall, rounded to 3 places, as a list, worst to the name of the kind with the lowest recall, and marker_precision to the markers' precision.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write