Feature engineering

Taming huge numbers with a log

Lifetime value is the total a customer has spent with you. Most customers spend a little. A few spend thousands. Data like this has a long tail: a big crowd of small values and a thin trail of giant ones.

Many models, especially simple ones, get pulled around by those few giants. The biggest spenders, often called whales, end up steering the whole fit.

The usual fix is a log. You met logs in the exponentials and logarithms lesson. The natural log, written $\ln$, asks one question: what power of $e$ (about 2.718) gives this number? It's the log that NumPy's np.log uses. Here is the feature:

Here $x$ is the original value, such as a customer's lifetime value.

In plain words: add 1 to the value, then take its natural log. Big values shrink a lot. Small values shrink only a little.

NumPy has a function for exactly this, np.log1p. The name means "log of 1 plus". Python's math module has one with the same name. Try $0, then three lifetime values that each grow about tenfold:

import math

for value in [0, 49, 499, 4999]:
    print(value, round(math.log1p(value), 2))

The values become 0, 3.91, 6.21 and 8.52. Before, the top spender was about 100 times bigger than the $49 customer. After the log, it's 8.52 against 3.91. Each tenfold jump now adds about 2.3, so the whale is still on top, but no longer drowns everyone else out.

import math

print(round(math.log(10 * 100), 4), round(math.log(10) + math.log(100), 4))

RFM: a marketer's feature set

Recency, frequency, monetary (RFM) is a way to score customers. Shops used it long before machine learning. It asks three questions:

• Recency: how recently did they last buy?
• Frequency: how often do they buy?
• Monetary: how much do they spend? (This dataset uses the average order value.)

Each question gets a score from 1 to 5, where 5 is best. To score one question, line all the customers up from worst to best. Then cut the line into five equal groups, called quintiles (fifths). The bottom fifth scores 1 and the top fifth scores 5. Finally, add the three scores:

$R$ is the recency score. $F$ is the frequency score. $M$ is the monetary score. Each runs from 1 to 5, so the total runs from 3 to 15.

In plain words: an RFM score is the three 1-to-5 scores added up. A higher total means a more active, more valuable customer.

r, f, m = 5, 4, 2
rfm = r + f + m
if rfm <= 6:
    band = "low"
elif rfm <= 10:
    band = "mid"
else:
    band = "high"
print(rfm, band)

RFM on 600 customers

The practice shop has 600 customers. The churned column is 1 if a customer left and 0 if they stayed. The code below builds the three scores, adds them up and sorts customers into bands. Then it prints the churn rate in each band:

import numpy as np
import pandas as pd
import requests

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])

df["recency_score"] = pd.qcut(-df["days_since_last_order"], 5, labels=False) + 1   # 5 = most recent
df["frequency_score"] = pd.qcut(df["orders_last_year"].rank(method="first"), 5, labels=False) + 1
df["monetary_score"] = pd.qcut(df["avg_order_value"], 5, labels=False) + 1
df["rfm"] = df["recency_score"] + df["frequency_score"] + df["monetary_score"]
df["rfm_band"] = pd.cut(df["rfm"], bins=[2, 6, 10, 15], labels=["low (3-6)", "mid (7-10)", "high (11-15)"])

print(df.groupby("rfm_band", observed=True)["churned"].agg(["mean", "count"]).round(2))

Here's what each piece does:

• pd.qcut(column, 5, labels=False) cuts a column into five equal-sized groups. It numbers them 0 to 4, so adding 1 makes them 1 to 5.
• pd.cut sorts the summed score into bands, using fixed edges. The edges 2, 6, 10 and 15 mean: above 2 up to 6, above 6 up to 10, and above 10 up to 15.
• The last line prints two things per band. mean is the share of churners, since the average of a 0/1 column is the share of 1s. count is how many customers are in the band.

Churn falls sharply from the low band to the high band. One engineered number tells much of the story.

Does the feature help? Measure it

A new feature can sound clever and still not help. It can even add noise. So test it. A feature belongs in the model only if it improves results on data the model didn't train on. This is called held-out data. A model can memorize its training data, so a better training score proves little.

The test below uses logistic regression as a measuring stick. You'll learn how that model works soon. First, the code builds three new columns:

• never_subscribed is a flag. The email open rate is missing for people who never signed up for emails, so a missing value becomes 1.
• open_filled is the open rate with those gaps filled in as 0.
• log_days is the log of (1 + days since the last order).

Then it compares four versions of the feature set. The first uses only the starting three columns. Each of the others adds more. trail_club is 1 for members of the shop's loyalty club and 0 for everyone else.

import numpy as np
import pandas as pd
import requests
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
df["never_subscribed"] = df["email_open_rate"].isna().astype(int)
df["open_filled"] = df["email_open_rate"].fillna(0)
df["log_days"] = np.log1p(df["days_since_last_order"])

start = ["orders_last_year", "avg_order_value", "support_tickets"]
versions = {
    "starting features": start,
    "+ recency (days)": start + ["days_since_last_order"],
    "+ recency (log of days)": start + ["log_days"],
    "+ recency + engagement": start + ["days_since_last_order", "trail_club", "open_filled", "never_subscribed", "tenure_months"],
}
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
for name, cols in versions.items():
    auc = cross_val_score(model, df[cols], df["churned"], cv=5, scoring="roc_auc").mean()
    print(f"{name:<26} AUC {auc:.3f}")

Two terms in that code get their own lessons later. AUC is a score of how well a model ranks churners above customers who stay. A score of 0.5 is no better than guessing, and 1 is perfect. Cross-validation (cv=5) splits the customers into five parts. Each part takes a turn as the held-out data, and the five scores are averaged.

Recency is the big win. The log version scores no better than the plain number here, so keep the simpler one. A log isn't automatically better: it has to prove itself like any other feature. Engagement signals (club membership, email opens, never subscribing) add a bit more. That's the habit to build. Every feature earns its place on held-out data, or it stays out.

Watch out: only build features from information you'd have at the moment you predict. A column like "clicked cancel" would score brilliantly in a test. In real use, you'd only learn it after the customer had already left.

Where this matters most

Hand-made features matter most for tabular data. That means rows and columns, like a spreadsheet or a database table. Most business data looks like this.

For images, audio and text, deep learning finds its own features. An embedding is one example: a list of numbers that a model learned to describe something. Even then, teams often give a model an embedding plus a few table columns, such as how long someone has been a customer and which plan they're on. That mix is a common design, and it works well.