Huber loss: what it is, with an example

A loss that treats small misses like squared error and big misses like absolute error, so a few wild values can't dominate training.

What it means

Huber loss is a loss function for predicting numbers that combines the best of two others. For small errors it's squared, like mean squared error (MSE), which makes it smooth and easy to optimize. For big errors it grows in a straight line, like mean absolute error (MAE), so a few wild values can't dominate training. A setting called delta marks where it switches.

It's named after the statistician Peter Huber, who introduced it in 1964, and it's usually said "HOO-ber".

An example in Python

def huber(e, delta=1.0):
    e = abs(e)
    return 0.5 * e ** 2 if e <= delta else delta * (e - 0.5 * delta)

clean = [0.5, -0.8, 0.3, -0.4, 0.6]
with_outlier = clean + [20.0]

def mean(values):
    return sum(values) / len(values)

for name, errors in [("clean", clean), ("one outlier", with_outlier)]:
    mse = mean([e ** 2 for e in errors])
    mae = mean([abs(e) for e in errors])
    print(f"{name:<12} MSE {mse:>7.3f}   MAE {mae:>6.3f}   Huber {mean([huber(e) for e in errors]):>6.3f}")

What it prints

clean        MSE   0.300   MAE  0.520   Huber  0.150
one outlier  MSE  66.917   MAE  3.767   Huber  3.375

A single outlier takes MSE from 0.3 to about 67, so a model trained on it would bend over backward to fit that one point. Huber goes from 0.15 to 3.4, close to MAE.

In practice

In PyTorch it's nn.HuberLoss(delta=1.0), and in scikit-learn there's HuberRegressor. Choose delta around the size of error you still consider normal: errors below it count fully, and errors above it count less. Reach for it when your data has real outliers you can't simply remove, such as a few enormous orders.

Huber loss

What it means

An example in Python

In practice

Try it now

Learn it properly

Other terms from the same lesson

Never trained an AI?