Autoencoders

Choosing the bottleneck size

Too small a code and the reconstructions blur away real detail; too large and the network drifts back toward copying. The practical way to choose is the same as for the number of principal components: train a few sizes, plot the reconstruction error on held-out data against the code size, and pick the point where adding numbers stops helping much. Here, with two hidden factors behind the readings, the error drops sharply from 1 to 2 numbers and barely moves after that.

What the code is for matters too. For anomaly detection, a smaller code is often better, because a tight model of normal data reconstructs unusual data worse. For compression you want the smallest code that keeps the detail you care about, and for features you'll feed to another model, test the downstream model with each size.

Anomaly detection by reconstruction

An autoencoder trained on normal data learns to reconstruct normal data. Show it something unlike anything it trained on, and it reconstructs badly, because its few code numbers have no way to describe the new pattern. So the reconstruction error of each item is an anomaly score:

import numpy as np
import torch
from torch import nn

rng = np.random.default_rng(0)
mixing = rng.normal(size=(2, 8))
normal = rng.normal(size=(300, 2)) @ mixing + rng.normal(scale=0.1, size=(300, 8))
X = torch.tensor(normal, dtype=torch.float32)

torch.manual_seed(0)
model = nn.Sequential(nn.Linear(8, 4), nn.ReLU(), nn.Linear(4, 2), nn.Linear(2, 4), nn.ReLU(), nn.Linear(4, 8))
optimizer = torch.optim.Adam(model.parameters(), lr=0.02)
for _ in range(300):
    loss = nn.MSELoss()(model(X), X)
    optimizer.zero_grad(); loss.backward(); optimizer.step()

typical = rng.normal(size=(3, 2)) @ mixing
broken = typical.copy(); broken[:, 5] += 4          # one sensor stuck high
for name, rows in [("typical", typical), ("broken sensor", broken)]:
    t = torch.tensor(rows, dtype=torch.float32)
    errors = ((model(t) - t) ** 2).mean(dim=1)
    print(name, [round(e, 2) for e in errors.tolist()])

The broken readings break the pattern the network learned, so their errors are many times larger, about 2.3 against nearly 0. This works with no labeled faults at all, which is why it's common for machines, transactions and network traffic. The threshold comes from the same review-budget reasoning as any anomaly detector.

errors = [0.02, 0.01, 2.31, 0.04]
print(max(range(len(errors)), key=lambda i: errors[i]))

def row_errors(original, reconstructed):
    return []


print(row_errors([[1, 2], [3, 4]], [[1, 2], [3, 6]]))

Denoising, and other ways to stop copying

A bottleneck isn't the only way to force learning. A denoising autoencoder gets a corrupted input, with noise added or values blanked out, and must reconstruct the clean original. Copying no longer works, since the input is wrong, so the network has to learn the structure that tells signal from noise. Trained this way, it cleans up noisy sensor data and scanned documents, and the masked-input versions of the idea gave us BERT-style pretraining for text.

Other constraints work too: penalize the code for being large or for having many non-zero values (a sparse autoencoder). Researchers now use sparse autoencoders on the inside of large language models, to pull apart the many concepts mixed into each layer's activations and see which ones a model is using.

From autoencoders to generating images

A plain autoencoder's codes can be scattered: pick a random code and the decoder may produce garbage, because nothing made the space between training codes meaningful. A variational autoencoder (VAE) fixes this by training the encoder to produce a small cloud of codes for each input, all close to a standard shape, so the whole code space fills in and any point in it decodes to something plausible. Sampling a code and decoding it then generates new data.

That idea sits inside today's image generators. Stable Diffusion doesn't paint pixels directly: an autoencoder compresses each 512 by 512 image into a much smaller grid of codes, the diffusion model works in that latent space, where it's far cheaper, and the decoder turns the final codes back into pixels. The same compress-then-reconstruct pattern shows up in audio and video models too.

That also explains a familiar quirk of generated images: fine details such as small text and fingers depend on what the autoencoder's small grid can hold. Detail the decoder can't rebuild from the codes never appears in the final image, however good the diffusion model is.