Scoring a classifier: confusion table, precision, recall, F1
Why this matters
A classifier that is right 95 times out of 100 sounds good, until you learn that only 5 of the 100 cases were yeses and it said no to every one of them. Accuracy hides which mistakes a model makes. So professional evaluation counts the mistakes by kind, in a small table called the confusion table, and reads a few numbers from it: precision, recall and F1. This lesson builds the table from the arrow detector of lesson 102, works each number by hand and in code, and moves the threshold to watch them trade against each other.
- Fill a confusion table from a classifier's calls: true positives, false negatives, false positives, true negatives
- Compute accuracy, precision, recall and F1 from the table, and say which one matters when a miss costs more than a false alarm
- Count the four cells in Python with a loop, then with array comparisons, and in PyTorch with (pred == 1) & (y == 1)
- Move a threshold and predict which scores rise and which fall, and read a three-class table row by row
The math: four cells and four scores
Purpose. Count a classifier's right and wrong answers by kind, and read four scores from the count.
The four cells. Each time a yes-or-no classifier meets an example, one of four things happens. Call the yes class "1" (an arrow) and the no class "0" (anything else).
- It says 1 and the answer is 1: a true positive, TP. A hit.
- It says 0 and the answer is 1: a false negative, FN. A miss.
- It says 1 and the answer is 0: a false positive, FP. A false alarm.
- It says 0 and the answer is 0: a true negative, TN.
"True" or "false" says whether the call was right; "positive" or "negative" says what was called. Count each kind and lay the counts out in a 2 × 2 table, rows for what really is and columns for what the model said. That's the confusion table (often called the confusion matrix): it shows exactly what the model confuses with what.