Preparing data

Scaling numbers

Picture comparing two shoppers. Their order values differ by $10. Their discount shares differ by 0.7, which is a huge gap: one rarely uses a discount and the other usually does. A model that adds up differences sees 10 and 0.7. The dollar gap wins, only because dollars are bigger numbers.

Columns on very different scales cause trouble for models that measure distances between rows. They also slow down gradient descent, the step-by-step training method from Phase 2. The fix is to put every number column on the same ruler.

Standardization rescales a column so its average is 0 and its standard deviation is 1. If you met z-scores in Phase 2, this is the same calculation:

Here $x$ is the original value, such as an order value in dollars. The answer, $z$, is the scaled value. $\mu_{\text{train}}$ (mu, pronounced "mew") is the column's average over the training rows. $\sigma_{\text{train}}$ (sigma) is the column's standard deviation over the same rows. The standard deviation measures how spread out the values are.

In plain words: subtract the training average, then divide by the training spread. The answer says how many standard deviations a value sits above (positive) or below (negative) the training average.

A small example. Your training rows have order values of $20, $40 and $60. The average is $40. The standard deviation is $20. A new order of $80 sits $40 above average. Divide by 20 and you get $z = 2$. An order of $20 gives $z = -1$.

Not every model needs this. Tree-based models (later in this phase) don't need scaling. Linear models, nearest neighbors and neural networks do.

import statistics

train = [20, 40, 60]                 # order values in the training rows
mean = statistics.mean(train)        # 40
std = statistics.stdev(train)        # 20.0 (divides by n - 1, like pandas)

for x in [20, 60, 80]:
    print(x, "->", (x - mean) / std)

mean, std = 40, 20        # learned from the training rows
print((50 - mean) / std)

Fit on training data, apply everywhere

To check a model fairly, you keep some rows aside. These rows are the test set. The model never learns from them. The other rows are the training data. Later, real new customers arrive that nobody has seen yet.

Every preparation step learns something from data:

• filling learns the median;
• one-hot encoding learns the list of categories;
• scaling learns an average and a standard deviation.

The rule: learn these numbers from the training data only. Then reuse the same numbers on the test set and on all future data.

Here's why. Think of a student preparing for an exam. If they peek at the exam while studying, their score looks better than it would on a fresh exam. If you compute the scaling average over all the data, the test rows sneak into training in the same way. It's a mild version of the leakage from the last lesson. Your test score comes out a little too good.

There's a second reason. Once the model is in use, new customers arrive one at a time. You'll only ever have the training numbers, so the model must be built around those.

The code below uses the first 450 customers for training and keeps the last 150 as the test set.

import numpy as np
import pandas as pd
import requests

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
train, test = df.iloc[:450], df.iloc[450:]

median_open = train["email_open_rate"].median()          # learned from training rows only
mean, std = train["avg_order_value"].mean(), train["avg_order_value"].std()

def prepare(part):
    out = pd.DataFrame(index=part.index)
    out["open_missing"] = part["email_open_rate"].isna().astype(int)
    out["email_open_rate"] = part["email_open_rate"].fillna(median_open)
    out["aov_z"] = (part["avg_order_value"] - mean) / std
    return out

print(prepare(train).describe().round(2).loc[["mean", "std"]])
print(prepare(test).head(3).round(2))

The code learns three numbers from the training rows. The median open rate is 0.35. The average order value is about $69.89, with a standard deviation of about $35. Over all 600 customers the average would be about $70.49. That small gap is exactly the test-set information you keep out.

The prepare function then uses those three numbers on any group of rows. The first printout shows the training rows: the scaled order value aov_z has an average of 0 and a standard deviation of 1, as standardization promises. The second printout shows three test customers. Their order values are around $30 below the training average, so their aov_z values are negative, around −0.9.

In scikit-learn (the workflow lesson later in this phase), each step becomes a transformer. A transformer has a fit method, which learns from training data, and a transform method, which applies what it learned anywhere. You chain transformers in a pipeline, so you can't get the order wrong.

Watch out: one-hot encoding must produce the same columns in training and in use. A category that only appears later, such as a new channel, needs a plan. You could add an "other" column, or ignore it.