Support vector machines

Soft margins and C

Real data overlaps, so a street with no points on it usually doesn't exist. The soft-margin SVM allows points inside the street or on the wrong side, and charges for each with the hinge loss:

with labels $y = \pm 1$. A point comfortably on its side, scoring beyond ±1, costs nothing. A point inside the street costs a little, and one on the wrong side costs more, growing with how far it is. Training minimizes the size of $w$ plus $C$ times the total hinge loss.

C sets the trade. A large C punishes every mistake, so the street narrows to fit the training data closely, which risks overfitting. A small C tolerates mistakes for a wider street and a smoother, more general boundary. As with any setting that controls fit, choose C by cross-validation.

When one class is much rarer than the other, such as fraud among payments, the hinge losses of the common class outnumber the rare one's and push the boundary toward it. Setting class_weight="balanced" makes each mistake on the rare class cost more, in proportion to how rare it is, so the margin stops ignoring it.

def hinge(score, y):
    return 0.0


print(hinge(2.5, 1), hinge(0.4, 1), hinge(0.4, -1))

Seeing C at work

scikit-learn's SVC trains an SVM and reports its support vectors. Here two overlapping clouds are fitted with a small and a large C:

import numpy as np
from sklearn.svm import SVC

rng = np.random.default_rng(0)
X = np.vstack([rng.normal([2, 2], 1.0, (40, 2)), rng.normal([4, 4], 1.0, (40, 2))])
y = np.array([0] * 40 + [1] * 40)
for C in (0.01, 100):
    model = SVC(kernel="linear", C=C).fit(X, y)
    width = 2 / np.linalg.norm(model.coef_[0])
    print(f"C={C}: {len(model.support_)} support vectors, margin width {width:.2f}, training accuracy {model.score(X, y):.2f}")

With a small C, the street is wide and many points lie on or inside it, so many points are support vectors. With a large C, the street shrinks to the few points that are hardest to separate. Neither is right in general: C is chosen on held-out data.

Kernels: curved boundaries without new features

A straight line can't separate a ring of one class around a blob of the other. One fix is to add features, such as $x_1^2 + x_2^2$, the squared distance from the center, after which a straight boundary in the bigger feature space is a circle in the original. The kernel trick gets the same effect without building those features: the SVM's math only ever needs similarities between pairs of points, so you can swap in a similarity that behaves like a richer space.

The common choice is the RBF (radial basis function) kernel, $\exp(-\gamma \lVert x - x' \rVert^2)$: points close together are similar, far apart are not. Its $\gamma$ sets how far each training point's influence reaches. A small $\gamma$ gives smooth boundaries; a large one wraps tightly around individual points and overfits.

from sklearn.datasets import make_moons
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC

X, y = make_moons(n_samples=300, noise=0.2, random_state=0)
for name, model in [("linear", SVC(kernel="linear")), ("RBF", SVC(kernel="rbf", gamma=1.0))]:
    print(name, round(cross_val_score(model, X, y, cv=5).mean(), 2))

The curved RBF boundary separates the crescents far better than any straight line can.

import math
gamma, squared_distance = 1.0, 2.0
print(round(math.exp(-gamma * squared_distance), 3))

When to reach for an SVM

SVMs do well with many features and modest data: text classification on TF-IDF features is a classic strength, as are small scientific and tabular problems. A linear SVM on text trains in seconds and is hard to beat as a baseline. SVMs separate two classes at a time; with more, scikit-learn trains one per pair or one per class behind the scenes, and LinearSVC handles hundreds of thousands of rows when the boundary can be straight.

Three practical notes. Scale your features: the margin is a distance, so a feature measured in thousands swamps one measured in fractions; put a StandardScaler in front. Kernel SVMs grow slowly with data: training compares pairs of points, so beyond some tens of thousands of rows, a linear SVM, logistic regression or a tree ensemble is the practical choice. SVMs don't give probabilities natively: their scores are distances from the boundary. When you need calibrated probabilities, logistic regression gives them directly, or calibrate the SVM's scores on held-out data.