Anomaly detection

Compared with its own past

A value is often only strange for this thing at this time: 400 orders an hour is a quiet Saturday for the shop but a flood at 3 a.m. Seasonal data needs a baseline that matches, such as the same hour last week or a rolling median of the last few hours.

One rule matters more than the formula: the value being judged must not be part of its own baseline. If the 3 a.m. flood is included in the numbers it's compared against, it raises the mean and the spread and partly hides itself, the same masking as before. Build the baseline from history only, then score the new value against it.

Baselines also drift. After a successful launch, "normal" traffic doubles, and a fixed baseline would alert forever. Rolling windows adapt; the trade-off is that a slow, steady attack can teach the baseline to accept it. For data with strong daily and weekly cycles, compare with the same hour on each of the last four weeks and take their median: one unusual week then can't distort the baseline.

import statistics as st


def is_spike(history, value):
    window = history + [value]
    mean, sd = st.mean(window), st.pstdev(window)
    return (value - mean) / sd > 3


print(is_spike([100, 102, 98, 101, 99], 300))

Many features at once: isolation forests

One column at a time misses combinations: an order of 3 items isn't strange, and neither is a total of 900 dollars, but 3 items for 900 dollars at a gear shop might be. For rows with several features, an isolation forest works well with no labels. It builds many random trees, each splitting on a random feature at a random value, and asks how many splits it takes to isolate each point on its own. Normal points sit in crowds and need many splits; anomalies sit alone and get cut off in a few. Fewer splits on average means more unusual.

import numpy as np
from sklearn.ensemble import IsolationForest

rng = np.random.default_rng(1)
items = rng.integers(1, 6, 200)
total = items * rng.normal(60, 12, 200)
orders = np.column_stack([items, total])
orders = np.vstack([orders, [[3, 900], [1, 5], [5, 300]]])

forest = IsolationForest(random_state=0).fit(orders)
score = -forest.score_samples(orders)   # higher means more unusual
top = np.argsort(score)[::-1][:3]
print("most unusual rows:", sorted(int(i) for i in top))
print("their items and totals:", [[int(orders[i, 0]), round(float(orders[i, 1]))] for i in sorted(top)])

Three orders were planted at the end, rows 200 to 202. Two come out on top: 3 items for 900 dollars, and 1 item for 5 dollars. The third flagged row, 145, is a real order from the random crowd, 5 items for 409 dollars, expensive enough to be worth a look. The third planted order, 5 items for 300 dollars, isn't flagged at all. Always read what a detector flags before trusting it: here, it was right.

flagged, real = 20, 6
print(real / flagged)

Running a detector people can live with

Set the threshold from capacity: if reviewers can check 20 alerts a day, flag the 20 highest scores, and measure precision at 20 (how many were real) and, when you learn about misses later, recall. Every alert that turns out to be nothing costs attention, and teams that get too many false alarms start ignoring all of them, which is worse than having no detector.

Feedback, drift and explanations

Feed the reviewers' decisions back. Each confirmed or dismissed alert is a label; after a few hundred, you can measure detectors properly, tune them, and eventually train a classifier on the confirmed cases. Expect the anomalies to change, especially when they're caused by people trying not to be caught: retire rules that no longer fire, and watch for new patterns in what reviewers dismiss.

Finally, explain each alert. A score of 7.2 helps nobody; a sentence that names the feature and compares it with the usual value lets a reviewer decide in seconds, and makes it easy to spot a detector that has started flagging the wrong thing, such as every order from a new country after a launch there.