Dimensionality reduction

t-SNE: a map of neighborhoods

Think of a subway map. It shows which stops are next to each other and which lines connect. It does not show true distances. Two stops that look far apart on the map might be a short walk apart. Nobody minds, because the map's job is to show what's next to what.

t-SNE draws your data like a subway map. It ignores the big directions that PCA cares about. Instead it tries to keep points that are close in 64 dimensions close in 2D. (The name is short for "t-distributed stochastic neighbor embedding". You don't need to remember that.)

Measuring how many neighbors survive

To compare the two methods fairly, you need a score. Take one image. List its 10 nearest images in the original 64 numbers. Then list its 10 nearest on the 2D picture. Count how many appear on both lists:

In plain words: out of the neighbors you checked, what share are still neighbors in the picture? 1 means the picture kept every neighbor. 0 means it kept none.

Here's a small case with 4 neighbors instead of 10. In the full data, an image's nearest neighbors are A, B, C and D. On the map they're A, C, B and F. Three names appear on both lists (A, B and C), so the overlap is 3 ÷ 4 = 0.75, or 75%. The order inside each list doesn't matter.

original = ["A", "B", "C", "D"]   # nearest neighbors in the full data
picture = ["A", "C", "B", "F"]    # nearest neighbors on the 2D map
shared = len(set(original) & set(picture))
print(shared, "of", len(original), "kept:", shared / len(original))

This prints 3 of 4 kept: 0.75. The & between two sets keeps only the items in both.

The code below does this for every image, then averages the scores. It uses 300 images and checks 10 neighbors each. It also draws the PCA picture and the t-SNE picture side by side.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.neighbors import NearestNeighbors

digits = load_digits()
X, y = digits.data[:300], digits.target[:300]          # 300 images keeps t-SNE quick in the browser
views = {"PCA": PCA(n_components=2).fit_transform(X),
         "t-SNE": TSNE(n_components=2, perplexity=30, init="pca", max_iter=500, random_state=0).fit_transform(X)}

def neighbors(A, k=10):
    return NearestNeighbors(n_neighbors=k + 1).fit(A).kneighbors(A, return_distance=False)[:, 1:]

fig, axes = plt.subplots(1, 2, figsize=(10, 4.2))
for ax, (name, Z) in zip(axes, views.items()):
    kept = np.mean([len(set(a) & set(b)) / 10 for a, b in zip(neighbors(X), neighbors(Z))])
    print(f"{name:<6} keeps {kept:.0%} of each image's 10 nearest neighbors")
    ax.scatter(Z[:, 0], Z[:, 1], c=y, cmap="tab10", s=6)
    for d in range(10):
        ax.text(*np.median(Z[y == d], axis=0), str(d), fontsize=13, weight="bold")
    ax.set_title(name)

The neighbors function asks for 11 neighbors and drops the first one, because each image's nearest neighbor is itself.

The t-SNE map splits the digits into clean islands. It also keeps far more of the true neighbors than PCA does.

Reading a t-SNE map

Like a subway map, a t-SNE map is useful only if you read it the right way. Three rules:

• Island sizes and the gaps between islands mean little. t-SNE keeps who is near whom. It doesn't keep distances. Two islands far apart on the map aren't necessarily more different than two close ones. An island's size doesn't reliably tell you how many items it holds, or how varied they are.
• Settings change the picture. One setting is called perplexity. Roughly, it's how many neighbors each point pays attention to. (It shares a name with the perplexity of language models, but here it's a dial you set.) Perplexity and the random seed can both reshape the map. Look at more than one map before you conclude anything.
• It can't place new points, and its coordinates aren't features. t-SNE positions only the points it was given. Never feed t-SNE output into a model.

UMAP is a popular alternative. It draws similar pictures, runs faster on large data, and keeps a bit more of the overall layout. It lives in a separate package, umap-learn, which isn't available in this browser.

PCA before a model

Compressing features can also help a model. Think of a short summary of a long report. If it keeps the main points, a reader can decide almost as well, and faster. For a model, fewer inputs can mean less noise and faster training.

Whether it helps is something you measure, not guess. You measure it with cross-validation, from the Evaluation lessons. Here's the churn data from earlier lessons, with its 8 customer features. The pipeline scales the features, runs PCA to keep k components, then fits a logistic regression. The score is AUC, where 0.5 is a coin flip and 1 is perfect.

import pandas as pd
import requests
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
df["email_open_rate"] = df["email_open_rate"].fillna(0)
features = ["tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order",
            "discount_share", "email_open_rate", "support_tickets", "trail_club"]
for k in [1, 2, 4, 6, 8]:
    model = make_pipeline(StandardScaler(), PCA(n_components=k), LogisticRegression())
    auc = cross_val_score(model, df[features], df["churned"], cv=5, scoring="roc_auc").mean()
    print(f"k = {k}: cross-validated AUC {auc:.3f}")

Four components do as well as all eight. With 8 features, that saving doesn't matter. With thousands of embedding dimensions, it can.

Notice that PCA sits inside the pipeline. So on each fold it's fit again, using only that fold's training rows. If you fit PCA on all the rows first, the test rows would help choose the components. That's a quiet form of leakage: information from the test data sneaking into training, which makes the score look better than it really is.

Shorter embeddings

The same idea applies to embeddings, and here the savings are real. Say you store ten million documents, each with a 3,072-number embedding. That's about 31 billion numbers. At the usual 4 bytes per number, it's roughly 123 GB to store and search. Cut each embedding to 256 numbers and it drops to about 10 GB.

Your options, roughly from most to least convenient:

• Ask for shorter vectors. Some embedding models are trained so the first numbers carry the most meaning. They're called Matryoshka embeddings, after Russian nesting dolls: a smaller embedding sits inside the bigger one. Their APIs let you request fewer dimensions directly.
• Use PCA. Fit it on a sample of your embeddings, then project everything to fewer dimensions. It works on documents you add later, too.
• Quantize. Many vector databases can store each number in fewer bits. It's like rounding prices to the nearest dollar: a little less exact, much smaller.

Whichever you choose, check retrieval quality on your own evaluation set before and after. Do it the same way you tested PCA on the churn model above.