Precision, recall and F1

Four kinds of right and wrong

Every yes/no prediction lands in one of four boxes. Here "positive" means "churns", the thing you're hunting for. "True" or "false" says whether the model's guess was right.

In plain terms:

• TP, true positive: you flagged them and they really churn. A hit.
• FP, false positive: you flagged them but they stay. A false alarm.
• FN, false negative: you didn't flag them, but they churn. A miss.
• TN, true negative: you didn't flag them and they stay. Correctly left alone.

This grid of four counts is the confusion matrix. It shows which mistakes a model makes, not only how many.

Watch out: scikit-learn prints the grid in a different order. It puts class 0, "stays", first. So its top-left number is TN and its bottom-right number is TP. In the cell above, the bottom row holds the churners: missed ones on the left, caught ones on the right.

Here's a small example to carry through this section. A shop has 100 customers, and 20 of them churn. A model flags 25 customers. Of those 25, 15 really churn.

• TP = 15, the churners it flagged.
• FP = 25 − 15 = 10, the happy customers it flagged.
• FN = 20 − 15 = 5, the churners it missed.
• TN = 100 − 15 − 10 − 5 = 70, the rest.

Precision: how many of your flags were right

In plain words: look only at the customers you flagged. Precision is the share of them who really churn.

In the example, the model flagged 25, and 15 were real. Precision is 15 / 25 = 0.6.

tp, fp = 15, 10
precision = tp / (tp + fp)
print(precision)

It prints 0.6. Low precision wastes retention offers on happy customers.

Recall: how many of the real ones you found

In plain words: look only at the customers who really churn. Recall is the share of them you flagged.

In the example, 20 customers churn, and the model caught 15. Recall is 15 / 20 = 0.75.

tp, fn = 15, 5
recall = tp / (tp + fn)
print(recall)

It prints 0.75. Low recall means churners leave without anyone trying to keep them. (You met recall for each label in Phase 1's evals lesson.)

Tip: both have TP on top. Only the bottom differs. Precision divides by what you flagged. Recall divides by what was really there.

Think of a fishing net. Recall asks how many of the fish in the lake you caught. Precision asks how much of your catch is fish, and not old boots.

F1: one number that needs both

Sometimes you want one score to compare models. F1 blends precision and recall. Its symbol is $F_1$.

The dot means multiply.

In plain words: multiply precision by recall, double it, then divide by their sum. The result is a kind of average that stays low unless both are high.

In the example: 2 × 0.6 × 0.75 = 0.9, and 0.6 + 0.75 = 1.35. So F1 = 0.9 / 1.35 ≈ 0.667.

precision, recall = 0.6, 0.75
f1 = 2 * precision * recall / (precision + recall)
print(round(f1, 3))

It prints 0.667. F1 is handy for comparing models. But it treats both kinds of mistake as equally bad. It doesn't know what your mistakes cost.

precision, recall = 1.0, 0.1
print((precision + recall) / 2, round(2 * precision * recall / (precision + recall), 3))

The threshold trades one for the other

The model doesn't really say "churns" or "stays". It gives each customer a probability of churning, a number between 0 and 1. You then pick a cut-off called the threshold. Anyone at or above it gets flagged.

model.predict uses a threshold of 0.5. That cutoff is a choice, not a rule.

Think of a metal detector's sensitivity dial. Turn it up and you find more coins, but it also beeps at more bottle caps. Lowering the threshold works the same way:

• You catch more churners, so recall goes up.
• You also flag more happy customers, so precision usually goes down.

Raise the threshold and the opposite happens.

Choose the threshold from what mistakes cost

Say a retention offer costs $15 per customer. Sending it to someone who was about to churn is worth $80 on average. (Some of them stay, and keep buying.) So a false alarm costs $15. A missed churner costs the $80 you could have kept.

You can score any threshold in dollars:

Here "flagged" is the number of customers who get the offer, which is TP + FP.

In plain words: every churner you reach earns $80. Every offer you send costs $15, whether it reaches a churner or a happy customer.

Say you flag 10 customers and 4 of them really churn. That's 80 × 4 − 15 × 10 = 320 − 150 = $170.

tp, flagged = 4, 10
print(80 * tp - 15 * flagged)

It prints 170.

The cell below does this on the real test set. First it prints a table. Each row is one threshold, with how many customers it flags and the three scores. Then it tries every threshold from 0.05 to 0.8, in steps of 0.05. It works out the net value for each, and reports the best.

import numpy as np
import pandas as pd
import requests
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import f1_score, precision_score, recall_score

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
df["email_open_rate"] = df["email_open_rate"].fillna(0)
features = ["tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order",
            "discount_share", "email_open_rate", "support_tickets", "trail_club"]
X_tr, X_te, y_tr, y_te = train_test_split(df[features], df["churned"], test_size=0.3,
                                          random_state=0, stratify=df["churned"])
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_tr, y_tr)
probs = model.predict_proba(X_te)[:, 1]

print("threshold  flagged  precision  recall    F1")
for t in [0.1, 0.2, 0.3, 0.4, 0.5, 0.6]:
    p = (probs >= t).astype(int)
    print(f"   {t:.1f}      {p.sum():>4}     {precision_score(y_te, p, zero_division=0):.2f}      "
          f"{recall_score(y_te, p):.2f}    {f1_score(y_te, p):.2f}")

OFFER_COST, VALUE_PER_CHURNER_REACHED = 15, 80
best = None
for t in np.arange(0.05, 0.81, 0.05):
    p = probs >= t
    tp = int(np.sum(p & (y_te == 1)))
    value = VALUE_PER_CHURNER_REACHED * tp - OFFER_COST * int(p.sum())
    if best is None or value > best[1]:
        best = (round(t, 2), value)
print(f"\nbest threshold for value: {best[0]}  (net value on this test set: ${best[1]})")

Read the table from the bottom up. As the threshold drops, more customers are flagged and recall climbs. Precision mostly falls.

F1 peaks at 0.3 to 0.4. But the threshold worth the most money is 0.15. Missing a churner costs much more than a wasted offer, so it pays to flag generously.

Change the two numbers and the answer moves. That's the point. The model gives probabilities. The business decides the threshold.

probs = [0.9, 0.6, 0.4, 0.2]   # predicted chance of churning
truth = [1, 0, 1, 0]            # 1 = really churns
t = 0.3
tp = sum(1 for p, y in zip(probs, truth) if p >= t and y == 1)
print(tp / sum(truth))

Reporting for several classes

Some models pick from more than two labels. A support model might sort tickets into "billing", "shipping" and "legal". Then you get a precision, recall and F1 for each label.

classification_report(y_true, y_pred) prints all of them, one row per label. Here y_true holds the real labels and y_pred holds the model's guesses. It also prints two averages:

• Macro average: every label counts equally, so rare labels matter as much as common ones.
• Weighted average: each label counts by how many examples it has, so common labels dominate.

Here's how far apart they can be. Take a simpler model with only two labels. "Billing" has 90 tickets, with recall 0.9. "Legal" has 10 tickets, with recall 0.5.

• Macro recall: (0.9 + 0.5) / 2 = 0.7.
• Weighted recall: 0.9 × 90/100 + 0.5 × 10/100 = 0.81 + 0.05 = 0.86.

The weighted number hides the weak legal recall. When rare labels matter, read the macro average or the rows for each label.

truth = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]          # 1 = spam
probs = [0.95, 0.8, 0.55, 0.3, 0.7, 0.45, 0.35, 0.2, 0.15, 0.1, 0.05, 0.02]

for t in [0.5, 0.25]:
    flagged = [p >= t for p in probs]
    tp = sum(f and y == 1 for f, y in zip(flagged, truth))
    fp = sum(f and y == 0 for f, y in zip(flagged, truth))
    fn = sum(not f and y == 1 for f, y in zip(flagged, truth))
    precision = tp / (tp + fp)
    recall = tp / (tp + fn)
    f1 = 2 * precision * recall / (precision + recall)
    print(f"threshold {t}: tp={tp} fp={fp} fn={fn}   precision {precision:.2f}   recall {recall:.2f}   F1 {f1:.2f}")