Strange geometry in high dimensions

Distances bunch together

Take a query point and a crowd of random points, and measure the distance to each. In 2 dimensions, some points are much closer than others. In 500 dimensions, the nearest and the farthest are almost the same distance away. A useful measure is the relative contrast: (farthest − nearest) ÷ nearest.

import random
random.seed(1)

def dist(a, b):
    return sum((x - y) ** 2 for x, y in zip(a, b)) ** 0.5

for d in [2, 500]:
    query = [random.random() for _ in range(d)]
    ds = [dist(query, [random.random() for _ in range(d)]) for _ in range(100)]
    print(d, round((max(ds) - min(ds)) / min(ds), 2))

Real data is less extreme than these random points, because real features are correlated and the data lies close to a much smaller surface inside the space. That's exactly why embeddings and PCA work at all. Still, when everything is roughly equally far away, "the nearest neighbor" stops meaning much. This is a big part of the curse of dimensionality, and it's why search over raw high-dimensional data needs care.

import random

def dist(a, b):
    return sum((x - y) ** 2 for x, y in zip(a, b)) ** 0.5

def contrast(d):
    random.seed(2)
    # a query point, 200 random points, and (farthest - nearest) / nearest
    return 0

low_d = contrast(2)
high_d = contrast(300)
print(round(low_d, 2), round(high_d, 2))

Why embeddings use cosine similarity

An embedding's direction carries its meaning; its length often reflects things like how long or common a text is. Comparing directions with cosine similarity ignores those lengths. And because random directions in high dimensions sit near cosine 0, any noticeably positive cosine stands out: in 1,536 dimensions, a cosine of 0.3 or 0.4 is far above what chance produces, even though it would look small on a 2-D diagram.

That's also why the absolute numbers differ between embedding models. Each model spreads its texts differently, so "similar" might mean 0.8 in one model and 0.4 in another. Judge cosines by comparing them with each other, or with a few known pairs, rather than against a fixed cutoff. Many vector databases also scale every embedding to length 1 before storing it; then the plain dot product equals the cosine, which is cheaper to compute.

The curse of dimensionality

Each new feature multiplies the amount of space data has to fill. Cut every feature into 10 bins: 1 feature makes 10 cells, 2 make 100, and 6 make a million. A million data points would put 100,000 in each cell of a 1-feature grid but about 1 in each cell of a 6-feature grid. In high dimensions, data is always sparse: most of the space is empty, and every point is far from every other.

That's the curse of dimensionality: methods that rely on nearby examples, like k-nearest neighbors, need far more data as features grow; models can find patterns in the emptiness that aren't real; and pictures are impossible. Dimensionality reduction, like PCA, helps by keeping the few directions where the data actually varies. Collecting more data helps less than you'd hope: each extra feature would need about ten times as much to keep the grid as full.

points = 1_000_000
features = [1, 2, 3, 6]

per_cell = []

print(per_cell)

High dimensions in AI

Embedding models produce vectors with hundreds or thousands of numbers, and vector databases find neighbors among millions of them with approximate methods, because exact search through a sparse, high-dimensional space is slow. A surprising result, the Johnson-Lindenstrauss lemma, says you can project high-dimensional points onto far fewer random directions and still keep their distances nearly intact, which is why random projections and compressed embeddings work at all. And inside a transformer, attention scores are divided by the square root of the dimension, because dot products of high-dimensional vectors grow with the dimension and would otherwise be too large.

a = [2, 0]
b = [5, 0]
dot = sum(x * y for x, y in zip(a, b))
print(dot / ((sum(x * x for x in a) ** 0.5) * (sum(y * y for y in b) ** 0.5)))