Loss functions
One number that says how wrong the model is.
Create a free account Try part of it here
Why it matters
Training a model means making it a little less wrong, again and again. To do that, the computer needs one number that says how wrong the model is right now. That number is the loss. Choosing it is a real decision. Squared error chases rare extreme values, absolute error mostly ignores them, and log loss punishes confident mistakes hard. The loss is how you tell a model what "good" means. So it should match what mistakes really cost you.
What you'll be able to do
- Explain what a loss function is and why training needs one
- Compute mean squared error, mean absolute error and Huber loss
- Compute log loss for probability predictions
- Choose a loss that matches what errors really cost
What it covers
- One number for "how wrong"
- How the losses differ
- One number for everyone
- Losses for probabilities
- Match the loss to the cost of errors
- Worked example: one late delivery
One number for "how wrong"
Think of a game of darts. After each throw, you can measure how far the dart landed from the bullseye. Add up those distances over a round and you get one score. A lower score means better aim. Practice means changing your throw to bring that score down.
A loss function does the same job for a model. It takes the true values and the model's predictions, and it returns one number. Lower is better. Training adjusts the model's parameters, the numbers it learns, to make that number as small as possible on the training data. You'll also hear it called a "cost function" or an "objective".
You've met one already. In linear regression, least squares picked the line with the smallest sum of squared errors. That sum was the loss. This lesson looks at other choices, and at why the choice matters.
Losses for numbers
Start with models that predict a number, like sales or a delivery time. This is called regression. For each prediction, the error is the true value minus the prediction. An error of 3 means the model guessed 3 too low. An error of −3 means it guessed 3 too high.
You need to turn a list of errors into one score. If you added the errors as they are, the plus and minus signs would cancel out. There are two common fixes.
Mean squared error
The first fix: square each error, then take the average. Squaring makes every error positive. This is the mean squared error, or MSE:
Here is the true value for example number . Its prediction is , said "y-hat". The letter is how many examples there are. As in the last lesson, means "add up". Dividing the total by turns it into an average.
In plain words: square every miss, add the squares up, and divide by how many there are.
print((1 ** 2 + 3 ** 2) / 2)
Show the answer
5.0
Square each miss, giving 1 and 9, then average them: 10 / 2 = 5.0. An average miss of 2 scores 5 because the larger miss counts for more.
Mean absolute error
The second fix: drop the minus signs, then take the average. A number with its sign dropped is its absolute value. It's written between two bars, so . This gives the mean absolute error, or MAE:
In plain words: measure how far off each prediction is, ignoring the direction. Then take the average.
A small example
A bakery predicts how many dozen muffins it will sell. Over three days it predicted 5, 6 and 7 dozen. It really sold 4, 6 and 10. So the errors are 4 − 5 = −1, then 0, then 10 − 7 = 3.
- MSE: square the errors to get 1, 0 and 9. The average is 10 / 3, about 3.33.
- MAE: drop the signs to get 1, 0 and 3. The average is 4 / 3, about 1.33.
y = [4, 6, 10] # dozens really sold y_hat = [5, 6, 7] # dozens predicted errors = [a - p for a, p in zip(y, y_hat)] mse = sum(e ** 2 for e in errors) / len(errors) mae = sum(abs(e) for e in errors) / len(errors) print(errors, round(mse, 2), round(mae, 2))
It prints [-1, 0, 3] 3.33 1.33. Look at where the MSE comes from. The miss of 3 adds 9 of the total of 10. Squaring makes big misses count far more than small ones.
errors = [2, -2, 2, -6] print(sum(e ** 2 for e in errors) / 4, sum(abs(e) for e in errors) / 4)
Show the answer
12.0 3.0
The squares are 4, 4, 4 and 36, which add to 48, so the MSE is 48 / 4 = 12.0. Without signs the errors are 2, 2, 2 and 6, which add to 12, so the MAE is 3.0. Averaging the raw errors would give −1.0, because the signs cancel.
How the losses differ
An outlier is a value far from the rest, like one huge order among many small ones. The big difference between losses is how they treat outliers.
- MSE lets big misses dominate. One error of 10 costs 100, as much as a hundred errors of 1. MSE is also smooth: the score changes gently as you nudge a prediction. That makes it easy to optimize, and it's the usual default.
- MAE charges each error in proportion to its size. An error of 10 costs ten times an error of 1, and no more. So a few wild values matter much less.
- Huber loss mixes the two. It's squared for small errors and grows in a straight line for big ones. You get the smoothness of MSE near zero, and outliers can't take over.
Go deeper The Huber formula
Huber loss uses a cut-off that you choose, called (the Greek letter delta). For each error , the cost is:
In plain words: if the miss is within the cut-off, square it and halve it. If it's bigger, charge it in a straight line, so doubling the miss only doubles the cost.
The halves make the two pieces meet without a jump at the cut-off. Take . An error of 0.4 is inside the cut-off, so it costs ½ × 0.4² = 0.08. An error of 2 is past it, so it costs 1 × (2 − 0.5) = 1.5. Plain squared error would charge it 2² = 4.
delta = 1.0 for e in [0.4, 2.0]: if abs(e) <= delta: loss = 0.5 * e ** 2 else: loss = delta * (abs(e) - 0.5 * delta) print(e, round(loss, 3))
It prints each error next to its cost: 0.08 for the small one and 1.5 for the big one. The Huber loss of a whole model is the average of these costs over all its errors.
Squared error punishes one big miss far more than several small ones. That's why a single outlier can drag a squared-error model.
One number for everyone
Here's a neat way to see the difference. Suppose the model may only predict one number, the same for every customer. Which number gives the lowest loss?
- Under MSE, the best single number is the mean: add the values up and divide by how many there are.
- Under MAE, the best single number is the median: the middle value once they're sorted.
The code below tries this on eight online orders. Seven are ordinary, between $38 and $58. One is a wholesale order of $950. np.linspace(0, 300, 3001) makes 3,001 guesses from $0 to $300, spaced 10 cents apart. The code works out the MSE and the MAE of each guess. Then np.argmin finds the guess with the lowest loss.
import numpy as np order_values = np.array([38, 42, 45, 47, 51, 55, 58, 950]) # one wholesale order candidates = np.linspace(0, 300, 3001) mse = [np.mean((order_values - c) ** 2) for c in candidates] mae = [np.mean(np.abs(order_values - c)) for c in candidates] print("best constant under MSE:", candidates[np.argmin(mse)], " mean:", order_values.mean()) print("best constant under MAE:", candidates[np.argmin(mae)], " median:", np.median(order_values))
The best guess under MSE is about $161. The search lands on 160.7, right next to the true mean of 160.75. (It may print 160.70000000000002. The long tail is a tiny rounding slip inside the computer, and you can ignore it.) No ordinary customer spends anything like that. The one wholesale order dragged the guess up.
The best guess under MAE stays with the typical orders. Every value from $47 to $51, the two middle orders, gives the same MAE. So the search could land on any of them, and here it lands on 47.1. The median, $49, sits in that range too.
Go deeper Why the median wins under MAE
Start at any guess and move it up by $1. Under MAE, every order above the guess now costs $1 less. Every order below it costs $1 more. If more orders sit above the guess, moving up helps, so you keep going. You stop when the orders are split evenly, half above and half below. That point is the median. How far away the big order is doesn't matter, only which side it's on.
Under MSE, each order's pull grows with its distance from the guess. The far-off order pulls hardest, and the balance point is the mean.
Show the answer
MAE or Huber
Squared error lets a few huge errors dominate. Absolute-style losses like MAE and Huber don't, so the big orders can't drag typical predictions toward them.
Losses for probabilities
Some models predict a chance instead of a number. A churn model gives each customer a probability, written , that they'll churn: cancel and leave. Later you find out what happened. Write if the customer churned and if they stayed.
Think of a weather forecaster. One who says "50% chance of rain" every day is never badly wrong. But they're not much help either. One who says "99% sure it'll be dry" right before a downpour has made a bad call. A good score should reward confident forecasts that turn out right. It should punish confident forecasts that turn out wrong, and punish them hard.
The standard loss for this is log loss. It's the two-outcome version of cross-entropy from Phase 2:
The log here is the natural log: ln on a calculator, and np.log in NumPy.
In plain words: for each customer, find the chance the model gave to what really happened. Take minus its log. Then average over all the customers.
The formula looks long because it covers both cases in one line. When , the second part is multiplied by 0 and drops out. The loss is , minus the log of the chance of churning. When , the first part drops out instead. The loss is , minus the log of the chance of staying.
A small example
Minus a log is small when the chance is near 1. It gets huge as the chance nears 0:
import math for p in [0.99, 0.9, 0.5, 0.1, 0.01]: print(p, round(-math.log(p), 3))
Next to each chance it prints the loss: 0.01, 0.105, 0.693, 2.303 and 4.605. Say the model gives a customer a 0.6 chance of churning, and they churn. The loss is −ln 0.6, about 0.51. Now say the model says 0.99 for a customer who stays. The chance it gave to staying was only 0.01. The loss is −ln 0.01, about 4.6. That's nine times bigger. Confident mistakes are punished hard, and that pushes models toward honest probabilities.
The code below scores three models on the same four customers. The first and last churned. The middle two stayed.
import numpy as np y = np.array([1, 0, 0, 1]) for name, p in [("cautious", np.array([0.6, 0.4, 0.4, 0.6])), ("confident and right", np.array([0.95, 0.05, 0.05, 0.95])), ("confident and wrong once", np.array([0.95, 0.05, 0.99, 0.95]))]: loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p)) print(f"{name:<26} log loss {loss:.3f}")
The cautious model always gives 0.6 to what really happened, so its loss is −ln 0.6, about 0.511. The confident and right model scores far better, at 0.051. The third model matches it except for one customer. It said 0.99 for churning, and that customer stayed. That one miss costs about 4.6. Averaged over four customers, it lifts the loss to 1.190, worse than the cautious model.
Watch out: if a model ever gives exactly 0 to what happened, the loss is minus the log of 0, which is infinite. Code guards against this by clipping.
np.clipnudges every probability to stay a tiny step inside 0 and 1.
import math p = 0.8 # predicted chance of churn y = 0 # the customer stayed print(round(-(y * math.log(p) + (1 - y) * math.log(1 - p)), 3))
Show the answer
1.609
The customer stayed, so y = 0 and only the second part counts. The chance the model gave to staying was 1 − 0.8 = 0.2, and −ln 0.2 ≈ 1.609. The answer 0.223 is −ln 0.8, the loss if the customer had churned.
Match the loss to the cost of errors
The loss quietly encodes your priorities. Before you pick one, ask what a mistake really costs.
| Situation | What errors cost | Loss to consider |
|---|---|---|
| Forecasting revenue, a few huge accounts | big misses matter most | MSE |
| Typical order value, with rare wholesale orders | shouldn't be swayed by outliers | MAE or Huber |
| Inventory: running out costs 3× more than overstocking | under-forecasting is worse | an asymmetric (quantile) loss |
| Churn probability used to rank outreach | probabilities must be honest | log loss |
All the losses so far treat a miss of 10 too high the same as a miss of 10 too low. Sometimes one direction hurts more. A shop that runs out of stock loses sales. A shop with a few spare items only has to discount them. An asymmetric loss charges more for misses in one direction. A quantile loss is a common one. It teaches the model to aim high (or low) on purpose.
The loss you use for training doesn't have to be the measure you report. A model can be trained with log loss and judged by precision and recall, two measures you'll meet later in this phase. Log loss is easy to optimize. Precision is what the business reads.
Worked example: one late delivery
A food delivery app promised five customers a delivery time. Four orders arrived within 2 minutes of the promise. The fifth got stuck in traffic and arrived 55 minutes late. Compare how MSE, MAE and Huber loss (with a cut-off of 5 minutes) score these promises.
import numpy as np actual = np.array([30, 25, 35, 28, 90]) # minutes each delivery really took predicted = np.array([28, 27, 33, 30, 35]) # minutes the app promised errors = actual - predicted delta = 5 squared = errors ** 2 absolute = np.abs(errors) huber = np.where(absolute <= delta, 0.5 * errors ** 2, delta * (absolute - 0.5 * delta)) print("errors: ", errors) print("squared: ", squared) print("absolute:", absolute) print("huber: ", huber) print("MSE", squared.mean(), " MAE", absolute.mean(), " Huber", huber.mean())
The errors are 2, −2, 2, −2 and 55. Squared, the late order costs 3,025, while each of the others costs 4. That one order makes up over 99% of the MSE of 608.2. Without signs, it costs 55 against 2 for each of the others, and the MAE is 12.6. Huber charges the small errors ½ × 2² = 2 each. The late order is past the cut-off, so it costs 5 × (55 − 2.5) = 262.5, and the Huber loss is 54.1. So which loss should the app train with? If traffic jams are rare flukes it can't predict, MAE or Huber stops one jam from pushing every future promise later. If late orders are common and costly, the extra attention MSE gives them may be what you want.
Practice what you just read
- 6 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 4 review cards for spaced repetition
Try a quiz question
Which constant prediction minimizes mean absolute error?