Integrals: totals and areas

Smooth curves: thinner and thinner strips

For a smoothly changing rate, cut the range into many thin strips, treat the rate as constant across each strip, and add up the rectangles. Thinner strips give a better estimate, and in the limit you get the exact area, written ∫ (an S for "sum") with the range below and above it. In ∫₀¹ x² dx, the x² is the height of each strip and the dx stands for its width, shrinking toward zero: the notation is a recipe for exactly this sum.

Take the area under y = x² from 0 to 1. Its exact value is 1/3:

def area(f, a, b, n):
    width = (b - a) / n
    return sum(f(a + i * width) * width for i in range(n))   # left edge of each strip

for n in [4, 40, 400]:
    print(n, round(area(lambda x: x ** 2, 0, 1, n), 5))

With 4 strips the estimate is 0.219, with 400 it's 0.332: closing in on 1/3. Each strip's height comes from its left edge, so for a rising curve every strip falls a little short, and more strips shrink that shortfall.

Simple curves have exact formulas: the area under xⁿ from 0 to 1 is 1 / (n + 1), which is where the 1/3 comes from. But strips work for any curve at all, including one you only have as a list of measured points, which is the usual situation with real data.

def trapezoid(f, a, b, n):
    width = (b - a) / n
    total = 0
    # add width * (f(left) + f(right)) / 2 for each strip
    return total

est = trapezoid(lambda x: x ** 2, 0, 2, 100)
print(est)

Probability is area

A continuous quantity, like a delivery time or a model's score, has a density curve. The chance of a value falling between a and b is the area under the curve between them, and the whole area under a density is 1, since the value has to land somewhere.

This is the continuous version of adding up the probabilities of separate outcomes, as you did for dice, with areas taking the place of sums. For the normal distribution, the area within one standard deviation of the mean is about 0.68, the "about two thirds" from the normal distribution lesson. You can check it by adding thin strips under its curve:

import math
def normal(x):
    return math.exp(-x * x / 2) / math.sqrt(2 * math.pi)

width = 0.001
print(round(sum(normal(-1 + i * width) * width for i in range(2000)), 3))

A density's height isn't itself a probability, and it can even be above 1 for a narrow curve; only areas under it are probabilities. A single exact value, like exactly 30 minutes, has zero area: a strip of zero width. That's why continuous probabilities are always asked about ranges, such as "between 25 and 35 minutes", and never about one exact value.

Integrals undo derivatives

Keep a running total of a rate, and you get a curve of the total so far. Its slope at any moment is the rate at that moment: the total grows fast when the rate is high, and stays flat when the rate is zero. So the derivative of a running total gives back the rate, and the integral of a rate gives the total. They undo each other, which is the fundamental theorem of calculus. It also gives a shortcut: the area between a and b is just the running total at b minus the running total at a, so one running total answers every "how much between these two points?".

rates = [2, 3, 5, 0, 4]          # sign-ups per hour
totals = []
running = 0
for r in rates:
    running += r
    totals.append(running)
print(totals)
print([totals[0]] + [totals[i] - totals[i - 1] for i in range(1, len(totals))])   # differences give the rates back

rates = [2, 3, 5]
running = 0
totals = []
for r in rates:
    running += r
    totals.append(running)
print(totals)

roc = [(0.0, 0.0), (0.1, 0.5), (0.3, 0.8), (0.6, 0.95), (1.0, 1.0)]

auc = 0

print(round(auc, 4))

Integrals in AI

AUC is the area under the ROC curve, computed exactly this way from the points a classifier's thresholds trace out. The expected value of a continuous quantity is an integral: each value times its density, added up. A cumulative distribution, the chance of a value at or below x, is the running area under a density. Training compute is the integral of power over time, and charts of a model's performance often report the area under a learning curve to sum up how fast and how far it learned. Whenever you see "total", "area" or "cumulative", look for a rate being added up.

In many dimensions, strips stop working: a grid fine enough to cover 50 dimensions would have more cells than there are atoms in the world. There, integrals are estimated by averaging random samples instead, a Monte Carlo method, which is the same trick the bootstrap and the simulations in this phase use.