Nearest neighbors

import math
from collections import Counter

new = (4, 1)  # orders last year, support tickets
stored = {
    "A": ((5, 1), "stay"),
    "B": ((1, 3), "churn"),
    "C": ((4, 3), "churn"),
    "D": ((3, 0), "stay"),
    "E": ((0, 4), "churn"),
}

def distance(a, b):
    return math.sqrt(sum((x - y) ** 2 for x, y in zip(a, b)))

ranked = sorted(stored, key=lambda name: distance(new, stored[name][0]))
for name in ranked:
    point, label = stored[name]
    print(name, round(distance(new, point), 2), label)

k = 3
votes = [stored[name][1] for name in ranked[:k]]
print("votes:", votes)
print("prediction:", Counter(votes).most_common(1)[0][0])

Scaling isn't optional here

Imagine comparing houses by price and number of bedrooms. A $1,000 price gap is tiny in real life. But as a number, 1,000 dwarfs a gap of 3 bedrooms. A distance that adds up raw gaps would treat the bedrooms as if they barely mattered.

The same thing happens with customers. In this shop's data, days since last order runs from 6 to

1. Membership in the Trail Club loyalty program (trail_club) is 0 or 1. Say two customers

are 100 days apart, and only one is a member. The days add 100 × 100 = 10,000 to the squared distance. Membership adds 1 × 1 = 1. So the neighbors are chosen almost entirely by recency, and membership is ignored.

The fix is to standardize each feature first. That rescales a column so its average is 0 and its typical spread (its standard deviation) is 1. After that, a typical gap in days counts about as much as a typical gap in membership. Every feature gets a fair say.

The code below loads the shop's 600 customers and holds back 30% of them for testing (test_size=0.3). Then it trains two k-NN models with $k = 15$. One uses the raw numbers. The other puts StandardScaler in front. make_pipeline chains the two steps. Because the pipeline is fit on the training rows, the scaler learns its averages from the training data only.

import pandas as pd
import requests
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
features = ["days_since_last_order", "orders_last_year", "support_tickets", "trail_club", "tenure_months"]
X_tr, X_te, y_tr, y_te = train_test_split(df[features], df["churned"], test_size=0.3, random_state=0, stratify=df["churned"])

raw = KNeighborsClassifier(n_neighbors=15).fit(X_tr, y_tr)
scaled = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=15)).fit(X_tr, y_tr)
for name, model in [("unscaled", raw), ("scaled", scaled)]:
    auc = roc_auc_score(y_te, model.predict_proba(X_te)[:, 1])
    print(f"{name:<9} accuracy {model.score(X_te, y_te):.3f}   AUC {auc:.3f}")

The unscaled model scores accuracy 0.817 and AUC 0.756. The scaled one scores 0.822 and 0.813.

Accuracy barely moves. About 81% of the customers stay, so a model that says "stays" for everyone already scores about 0.81. So accuracy hardly tells these two models apart.

AUC looks at ranking instead. Each model gives every customer a churn score: here, the share of their 15 neighbors who churned. AUC is the chance that a random churner gets a higher score than a random customer who stayed. 0.5 is guessing and 1 is perfect. It gets its own lesson later. Scaling lifts AUC from 0.756 to 0.813. The scaled model is clearly better at putting likely churners at the top of the list.

Choosing k

Back to the restaurant. Ask one friend, and their personal quirks become your answer. Poll the whole town, and you get the most popular place, whatever your own taste.

• Small k (like 1): the prediction copies the single nearest customer, noise and all. It's very flexible, and it tends to overfit. That means it memorizes quirks of the training data and does worse on new customers.
• Large k: the vote averages over many customers, so it drifts toward the overall majority. It's stable, but it can underfit, which means it's too blunt to see the real pattern. Say $n$ is the number of stored examples. With $k = n$, it always predicts the majority class.

With two classes, an odd k like 3 or 15 means a vote can never end in a tie.

The code below trains scaled k-NN models for six values of k. For each one, it prints the accuracy (the share of customers predicted correctly) on the training data and on the test data.

import pandas as pd
import requests
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
features = ["days_since_last_order", "orders_last_year", "support_tickets", "trail_club", "tenure_months"]
X_tr, X_te, y_tr, y_te = train_test_split(df[features], df["churned"], test_size=0.3, random_state=0, stratify=df["churned"])
for k in [1, 3, 9, 25, 75, 200]:
    model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=k)).fit(X_tr, y_tr)
    print(f"k={k:<4} train {model.score(X_tr, y_tr):.3f}  test {model.score(X_te, y_te):.3f}")

With $k = 1$, training accuracy is a perfect 1.000, but test accuracy is only 0.717. Every training customer is its own nearest neighbor, at distance 0. So the model looks up each one's stored answer, and the perfect score means nothing.

Test accuracy climbs as k grows and peaks at 0.833 with $k = 25$. At $k = 75$ and $k = 200$, the scores fall to 0.807 on training and 0.806 on test. That's the same score you'd get by predicting "stays" for everyone.

Pick k by how well it does on validation data (data held back from training), the same way you'd pick a tree's depth.

The same idea at scale: vector search

k-NN's weakness is speed. Every prediction compares the new example with every stored one. With a million stored customers, that's a million distances for each prediction. That's too slow for a live app.

Vector databases, from Phase 1, solve this problem. They use approximate nearest-neighbor indexes. These find neighbors that are usually the true nearest ones, or very close, far faster than checking everything.

Retrieval in RAG is k-NN. You embed the question, find the k nearest document chunks, and hand them to the model.

Other places it appears:

• Lookalike audiences: find the people most similar to your best customers.
• Recommendations: suggest items whose embedding vectors are nearest to what someone liked.
• Anomaly detection: an example that's far from all its neighbors is unusual.

For marketers: a k-NN prediction comes with a ready explanation: "these 15 similar customers, 11 of whom churned". That transparency is often worth more than a point of accuracy.