Regularization

Seeing it: a wiggly curve tamed

Here's the effect in a picture. The code makes 15 points that follow a smooth wave, plus random noise. Then it fits a degree-12 polynomial. That's a curve built from x, x², x³ and so on up to x¹², each with its own weight. Twelve weights give the curve a lot of freedom to bend.

It fits the curve twice: once with no penalty, and once with a ridge penalty. The gray dashed line is the true wave the points came from.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler

rng = np.random.default_rng(3)
x = np.sort(rng.uniform(0, 1, 15))
y = np.sin(2 * np.pi * x) + rng.normal(0, 0.25, 15)
grid = np.linspace(0, 1, 300).reshape(-1, 1)

for name, reg in [("no penalty", LinearRegression()), ("ridge, λ = 0.1", Ridge(alpha=0.1))]:
    model = make_pipeline(PolynomialFeatures(12), StandardScaler(), reg).fit(x.reshape(-1, 1), y)
    plt.plot(grid, model.predict(grid), label=name)
plt.scatter(x, y, color="gray", s=20, zorder=3)
plt.plot(grid, np.sin(2 * np.pi * grid), "--", color="gray", label="true pattern")
plt.ylim(-2, 2)
plt.legend()

scikit-learn calls the penalty strength alpha. It's the λ above.

Without a penalty, the curve threads through the points with wild swings between them. With a ridge penalty, it follows the trend. The pipeline also rescales the columns first with StandardScaler. You'll see why that matters below.

Ridge (L2) and lasso (L1)

There are two common ways to charge for weights.

• Ridge charges for each weight squared. It's the penalty you've used so far.
• Lasso charges for each weight's size, ignoring its sign. That size is called the absolute value, written $\vert w_j \vert$. It's Python's abs(): abs(-5) is 5.

In plain words: ridge squares every weight and adds them up. Lasso drops every minus sign and adds them up. Both then multiply the total by lambda. "L2" and "L1" are the technical names for these two ways of measuring size.

Go back to the worked example with $\lambda = 0.01$. Lasso charges Model A 0.01 × (2 + 1) = 0.03. It charges Model B 0.01 × (6 + 5) = 0.11.

lam = 0.01
for name, weights in [("A", [2, 1]), ("B", [6, -5])]:
    ridge = lam * sum(w ** 2 for w in weights)
    lasso = lam * sum(abs(w) for w in weights)
    print(name, "ridge", round(ridge, 2), " lasso", round(lasso, 2))

w = [0.5, -1.5]
lam = 0.2
print(round(lam * sum(abs(x) for x in w), 2))

Why lasso lands on exactly zero

Ridge cares a lot about big weights and hardly at all about small ones. A weight of 0.1 costs only 0.1² = 0.01 under ridge. So ridge barely bothers to push it any further. Lasso charges the same rate all the way down to zero.

The code shrinks three weights by 0.1 each. It prints how much penalty each method saves, before multiplying by λ.

for w in [1.0, 0.5, 0.1]:
    new = w - 0.1
    print(w, "->", round(new, 1), "  ridge saves", round(w ** 2 - new ** 2, 2), "  lasso saves", round(abs(w) - abs(new), 2))

Ridge saves 0.19 on the big weight but only 0.01 on the small one. Lasso saves 0.1 every time. So for big weights, ridge pushes harder. For small weights, lasso does.

Now picture a feature that barely helps. Under lasso, its small weight still pays the full rate. That fee is bigger than the little the weight adds to the fit. So lasso pushes it all the way to zero, and the feature drops out of the model. That's why lasso does automatic feature selection.

Lasso on real data

The next code uses the churn data from earlier lessons: will a customer leave? It takes five real features, such as days since the last order and support tickets. Then it adds 15 columns of pure random noise. Those columns carry no information, so a good model should ignore them.

It fits two logistic regressions. Here C sets the penalty strength backwards: a big C means a weak penalty (more on that below). So C=100 is nearly unpenalized. The second model uses a lasso penalty, set by l1_ratio=1.

import numpy as np
import pandas as pd
import requests
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
real = ["days_since_last_order", "orders_last_year", "support_tickets", "trail_club", "tenure_months"]
rng = np.random.default_rng(0)
noise = pd.DataFrame(rng.normal(size=(len(df), 15)), columns=[f"noise_{i}" for i in range(15)])
X = StandardScaler().fit_transform(pd.concat([df[real], noise], axis=1))
y = df["churned"]

for name, model in [("no strong penalty", LogisticRegression(C=100, max_iter=2000)),
                    ("lasso (L1)", LogisticRegression(l1_ratio=1, C=0.05, solver="liblinear"))]:
    coefs = model.fit(X, y).coef_[0]
    print(f"{name:<18} real features kept: {np.sum(coefs[:5] != 0)}/5   noise features kept: {np.sum(np.abs(coefs[5:]) > 1e-9)}/15")

The nearly unpenalized model gives every noise column some weight. Lasso keeps the real signals and zeros out the noise.

The knobs in scikit-learn and deep learning

• Ridge(alpha=...) and Lasso(alpha=...) do regression with a ridge or lasso penalty. A larger alpha means a stronger penalty.
• LogisticRegression(C=...): C is the inverse of the penalty strength. So a smaller C means a stronger penalty. The default penalty is ridge (L2). That means every logistic regression you've fit in this phase was lightly regularized.
• Decision trees: max_depth and min_samples_leaf are regularization too. They limit how complicated a tree can get.
• Neural networks (Phase 4): weight decay is the ridge idea. It shrinks every weight a little on each step, and it's the "W" in AdamW, which is the optimizer most large models are trained with. Dropout randomly switches off parts of the network during training. Early stopping ends training once the validation score stops improving.

Scale your features first

A weight's size depends on the units of its feature. Say a price in dollars gets a weight of 2. Measure the same price in cents and the weight becomes 0.02, because the numbers are 100 times bigger. The model predicts exactly the same either way. But ridge charges 2² = 4 for the dollars version and only 0.02² = 0.0004 for the cents version.

So a penalty is only fair when features share a scale. Standardize them first, as you did in the data-prep lesson. Both code examples above do this with StandardScaler.

Choosing λ

Think of a dog on a leash. Too loose, and the dog chases every squirrel. Too tight, and it can barely walk. The penalty works the same way. Too little, and the model still overfits the noise. Too much, and the model is forced so flat that it underfits. It then misses the real pattern too, and both the training and validation errors are high.

You can't know the right λ in advance, so you test. Try values spread on a log scale. Each value is a fixed multiple of the last, often ten times: 0.001, 0.01, 0.1, 1, 10. Fit a model on the training data with each one. Keep the one with the lowest validation error. (Cross-validation, which is a sturdier version of this test, has its own lesson.) It's the same routine you used to choose a tree's depth in the overfitting lesson. In the exercise, you'll do it yourself for ridge.