Random forests and boosting

Random forests: each split gets random questions

Bagged trees still tend to look alike. Suppose one feature, such as days since last order, is the strongest signal. Almost every tree will ask about it first. Then the trees agree too much, and their errors overlap.

A random forest adds a second source of difference. At each split, a tree may only consider a random handful of the features. Sometimes the dominant feature isn't on the list. The tree has to find the best question among the others. The trees end up asking different questions, so they disagree more usefully.

How big is the handful? For classification, scikit-learn's default is the square root of the number of features. With 8 features, that's 2 per split, since the square root of 8 is about 2.8 and it rounds down.

Here's the whole idea for one customer:

tree 1: bootstrap sample A, random features at each split  →  P(churn) = 0.62
tree 2: bootstrap sample B, random features at each split  →  P(churn) = 0.35
...
tree 300                                                     →  P(churn) = 0.51
forest prediction = average                                  →  P(churn) = 0.48

Boosting: trees that fix each other's mistakes

Think of a golfer on the green. The first shot gets the ball close. The next putt aims at the distance that's left. Each putt after that fixes a smaller leftover gap.

Boosting builds trees the same way, one after another. It starts with a simple guess, such as the average. Then each new tree is trained on what the ensemble so far gets wrong. These trees are small, often 3 to 6 questions deep. One small tree is a weak model on its own. Hundreds of them, each fixing a bit of the leftover error, add up to a strong one.

Boosting doesn't add each new tree in full. It adds a fraction of it, set by a learning rate. That's the same idea as the step size in the learning rate lesson. Small steps make each tree's mistakes matter less.

One symbol at a time:

• $x$ is one customer's features.
• $F_{\text{old}}(x)$ is the ensemble's prediction so far.
• $h(x)$ is the new tree's prediction of the leftover error.
• $\eta$, pronounced "eta", is the learning rate, a number such as 0.1.
• $F_{\text{new}}(x)$ is the ensemble's prediction after adding the new tree.

In plain words: new prediction = old prediction + learning rate × the new tree's correction.

Worked example. A customer really spent $100. Boosting predicts spend, a number, so here it starts from the average customer's spend: $60. The learning rate is 0.5. To keep the numbers small, pretend each new tree predicts the leftover error exactly.

1. Leftover: 100 − 60 = 40. Add half: 60 + 0.5 × 40 = 80.
2. Leftover: 100 − 80 = 20. Add half: 80 + 0.5 × 20 = 90.
3. Leftover: 100 − 90 = 10. Add half: 90 + 0.5 × 10 = 95.

Each tree closes half of the gap that's left. The same steps in Python:

target = 100            # what the customer really spent
prediction = 60.0       # starting guess: the average customer's spend
learning_rate = 0.5

for tree in range(1, 5):
    leftover = target - prediction                   # the error the new tree learns
    prediction = prediction + learning_rate * leftover
    print(f"tree {tree}: leftover {leftover:4.1f} -> new prediction {prediction}")

It prints predictions of 80.0, 90.0, 95.0 and 97.5. This is a simplification. A real tree learns the leftovers of every training customer at once, so it only gets them roughly right.

The full name is gradient boosting. Each tree fits the downhill direction of the loss. That's the direction the predictions need to move to shrink the loss. It's the opposite of the gradient, which points uphill. For the squared-error loss, that direction is the leftover error itself, as above.

Bagging and boosting fix different problems

• Bagging and random forests mainly cut variance. Each tree is deep and jumpy, and averaging calms them down.
• Boosting mainly cuts bias, the error from a model too simple for the pattern. Each small tree adds a bit more detail.

Boosting needs care not to overfit. Each new tree chases the leftover errors, and some of those are noise. Use a small learning rate. Use early stopping, too: keep checking the score on validation data, and stop adding trees when it stops improving.

target, prediction = 50, 30.0
for _ in range(2):
    prediction = prediction + 0.5 * (target - prediction)
    print(prediction)

Comparing them on churn

The code below loads the shop's customers. It fills missing email open rates with 0. It holds 30% of the customers back as a test set. Then it trains three models on the rest:

• a single deep tree, with no depth limit;
• a random forest of 300 trees, with at least 3 customers in every leaf;
• gradient boosting with 150 trees, each 2 questions deep, and a learning rate of 0.05.

It scores each model on the test customers. At the end it prints the forest's feature importances. These say how much the forest leaned on each feature when it made its splits.

import pandas as pd
import requests
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
df["email_open_rate"] = df["email_open_rate"].fillna(0)
features = ["tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order",
            "discount_share", "email_open_rate", "support_tickets", "trail_club"]
X_tr, X_te, y_tr, y_te = train_test_split(df[features], df["churned"], test_size=0.3, random_state=0, stratify=df["churned"])

models = {
    "single deep tree": DecisionTreeClassifier(random_state=0),
    "random forest (300 trees)": RandomForestClassifier(n_estimators=300, min_samples_leaf=3, random_state=0),
    "gradient boosting": GradientBoostingClassifier(n_estimators=150, learning_rate=0.05, max_depth=2, random_state=0),
}
for name, model in models.items():
    model.fit(X_tr, y_tr)
    auc = roc_auc_score(y_te, model.predict_proba(X_te)[:, 1])
    print(f"{name:<27} test AUC {auc:.3f}")

forest = models["random forest (300 trees)"]
importance = pd.Series(forest.feature_importances_, index=features).sort_values(ascending=False)
print("\nfeature importances:\n" + importance.round(3).to_string())

The score is AUC. It measures how well a model ranks churners above stayers. An AUC of 0.5 is a coin flip, and 1.0 is perfect. It gets its own lesson.

The single deep tree is barely better than chance on new customers. The two ensembles are far better. The importances show which features the forest leaned on most. Here that's recency (days since last order) and order frequency, matching what you've seen all phase.

Practical notes

• Random forests are hard to break. The defaults are sensible, they need little tuning, and features don't need rescaling. They make a great first strong model.
• Gradient boosting usually wins on accuracy once you tune it. The main settings are the number of trees, the learning rate and the tree depth. For real projects, use the dedicated libraries: XGBoost, LightGBM or CatBoost. They're much faster than scikit-learn's version.
• Importances aren't explanations of cause. When two features are closely related, they split the credit between them.
• Ensembles have a cost. Hundreds of trees are slower to predict with than one tree or a logistic regression. They're also much harder to read. If a simple model is nearly as good, the simple one is often the better product.

For later: with LLMs, you can ask the model the same question several times and take the majority answer. That's called self-consistency, and it's an ensemble too. It helps for the same reason: independent errors cancel.