Lesson 70 measured change across a gap. The derivative, next lesson, asks for change at a single point, where the gap is zero, and dividing by zero makes no sense. Limits are the way around it: instead of asking what happens at the point, ask what the values head toward as you get closer and closer. That one idea makes the derivative precise. It also gives words for what can go wrong with a function: a hole, a corner, a jump. Those words matter in ML, because activation functions are judged by them, and because code that gets too close to dividing by zero needs a guard.
What you'll be able to do
Read a limit from a table, approaching a point from both sides
See that a limit can exist where the formula itself breaks, at a hole
Tell smooth points, corners and jumps apart, and say which have a limit
Judge activation functions by the same test, in plain Python and with PyTorch's F.relu and torch.sigmoid
Add a tiny epsilon so code never divides by zero, as real training code does with torch.sqrt(v + eps)
Quick check
Have a guess before you start. What do the numbers 1.9, 1.99, 1.999, 1.9999 head toward?
They never settle
1.9
2
1.99999
Show the answer
2
Each one is closer to 2 than the last, and they never pass it: they head toward 2.
The math: getting closer without arriving
Purpose. Make sense of "as the gap shrinks to nothing".
Take f(x) = (x² − 1)/(x − 1). At x = 1 the formula gives 0 ÷ 0: it breaks. But try values close to 1: f(0.9) = 1.9, f(0.99) = 1.99, f(1.01) = 2.01. From both sides the values close in on 2. We say the limit of f as x approaches 1 is 2:
x→1limx−1x2−1=2
In plain words: as x gets as close to 1 as you like, from either side, f(x) gets as close to 2 as you like.
The point itself doesn't matter, only the approach. Here there's a hole at x = 1: the graph is the line y = x + 1 with one point missing.
Quick check
From lesson 15: what is (x² − 1) factored?
x(x − 1)
(x + 1)²
(x − 1)²
(x − 1)(x + 1)
Show the answer
(x − 1)(x + 1)
A difference of squares. So (x² − 1)/(x − 1) = x + 1 everywhere except x = 1, which is why its limit there is 1 + 1 = 2.
The picture. Step through the table: the same function at 1 − h and 1 + h, for h = 0.1, 0.01 and 0.001. The drawing shows the graph, a straight line with a ring where x = 1 has no value. Each step puts a point on each side of the ring, labeled with its height. From the second step on, the drawing zooms in ten times: the last pair slides out toward the edges, and the new pair lands closer in. However far you zoom, there's a point on each side, and both heights hug 2. The PyTorch tab works out all three h at once.
Try it now
Then set the x it approaches to 3: the formula works there, and the limit is just f(3) = 4.
Symbols and formula. A limit exists when the values from the left and from the right head to the same number:
x→a−limf(x)=x→a+limf(x)=L⇒x→alimf(x)=L
In plain words: a − means approaching from below, a + from above. Both must agree.
Continuous. When the limit also equals f(a), so there's no hole and no jump, the function is continuous at a: you could draw it through that point without lifting your pen. x², eˣ and the sigmoid are continuous everywhere.
Quick check
From the left, f heads to 3; from the right, f heads to 5. What is the limit at that point?
3, from the left side
4, the average of the two sides
There is none: the two sides disagree
5, from the right side
Show the answer
There is none: the two sides disagree
A limit needs both sides to head to the same number. Here they don't: it's a jump.
Three kinds of trouble. Look at a function near a point and one of these usually describes it:
Smooth: values close in from both sides and the graph has no sharp point. The sigmoid at 0.
Corner: the values close in, so the function is continuous, but the graph turns sharply: relu(x) = max(0, x) at 0, or |x| at 0. The slope jumps from one side to the other.
Jump: the two sides head to different values, like a step from 0 to 1. No limit at all.
A fourth kind, blowing up, happens with 1/x near 0: the values grow without bound. At x = −0.1, −0.01 and −0.001, 1/x is −10, −100 and −1000; at 0.1, 0.01 and 0.001 it is 10, 100 and 1000.
In your own words
In your own words: how can a limit exist at a point where the formula gives 0 ÷ 0?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: tables from both sides, and epsilon
Purpose. Estimate a limit by computing the approach, and keep code away from dividing by zero.
deff(x):
return (x ** 2 - 1) / (x - 1)
for h in [0.1, 0.01, 0.001]:
print(h, round(f(1 - h), 6), round(f(1 + h), 6))
It runs right here in your browser, with nothing to install.
Both columns close in on 2. Calling f(1) itself would stop with "ZeroDivisionError": the formula really does break at the point, and the table is how you see past it. Rounding hides float noise: f(1.001) is really 2.0009999999999177.
Code flow. Here is that loop again, with each side stored and one more column: gap, how far apart the two sides are. Step through it one line at a time. Watch left climb toward 2 and right fall toward it, while gap gets ten times smaller on every pass.
A named mistake: trusting the point. Putting the point into the formula gives the limit only when the function is continuous there. At a hole you get an error; at a jump you get one side's value, which hides the fact that the other side disagrees. Always check both sides.
Epsilon. Real code often computes things like a logarithm of a probability, which breaks at exactly 0. The usual guard adds a tiny number, epsilon, so the input is never exactly 0:
import numpy as np
p = np.array([0.5, 0.0, 0.25])
print(np.log(p + 1e-9).round(3).tolist())
Without + 1e-9, the middle entry would be minus infinity and break everything computed from it. With it, the result is a large but finite −20.723.
Worked example
Worked example: estimate three limits by table
Write approach(f, a) that returns the values at a − 0.001 and a + 0.001, and use it on three functions.
import math
defapproach(f, a):
returnround(f(a - 0.001), 4), round(f(a + 0.001), 4)
defhole(x):
return (x ** 2 - 4) / (x - 2)
defsinc(x):
return math.sin(x) / x
defstep(x):
if x >= 0:
return1return0print(approach(hole, 2), approach(sinc, 0), approach(step, 0))
In the lesson you can change this and run it.
The first two come back as two nearly equal numbers, about 4 and 1, so those limits exist. The step comes back as (0, 1): the two sides disagree by a whole unit no matter how close you get, so it has no limit at 0. A function can return in more than one place; step returns 1 as soon as x >= 0, and 0 otherwise. And a function can be handed to another function: approach(hole, 2) passes the function hole itself, without brackets, so that approach can call it at two points.
(x² − 9)/(x − 3) = x + 3 away from 3, so the values are 5.999 and 6.001: the limit is 6.
Quick check
What does this print?
import numpy as np
p = np.array([0.0, 1.0])
print(np.log(p + 1e-9).round(2).tolist())
[-20.72, 0.0]
[-20.72, 1.0]
[-inf, 0.0]
[0.0, 0.0]
Show the answer
[-20.72, 0.0]
log(1e-9) ≈ −20.72, large but finite thanks to epsilon. log(1 + 1e-9) rounds to 0.0.
Drill
Does the limit exist?
Fix has_limit(f, a) so it compares the values at a − 0.0001 and a + 0.0001 and returns True when they're within 0.01 of each other. Then set results for the hole, the step and relu, in that order.
In the lesson, your code is checked as you write it.
In your own words
Why does has_limit look at a − 0.0001 and a + 0.0001 instead of at a?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: judging activation functions
Purpose. An activation function's behavior near 0 decides how training sees it.
Neural networks pass each sum through an activation function. Training needs to know which way to nudge the weights, which means slopes, and slopes need the function to behave near every point:
step:jumprelu:cornerσ(x)=1+e−x1:smooth
In plain words: a jump has no slope at all at the jump, and slope 0 everywhere else; a corner has a slope everywhere but at the corner; a smooth function has one everywhere.
import math
defsigmoid(x):
return1 / (1 + math.exp(-x))
for h in [0.1, 0.01, 0.001]:
print(h, round(sigmoid(-h), 6), round(sigmoid(h), 6))
The sigmoid closes in on 0.5 from both sides, smoothly. That's why it replaced the step function in early neural networks: a step's slope is 0 everywhere it exists, so training gets no signal about which way to move.
Quick check
relu is continuous at 0 but has a corner there. What does that mean for its slope?
Its slope is 0.5 everywhere
0 on the left, 1 on the right: it jumps at 0
It has the same slope on both sides
It has no slope anywhere
Show the answer
0 on the left, 1 on the right: it jumps at 0
relu is flat (slope 0) for negative x and rises at slope 1 for positive x; at the corner the slope switches.
The picture. Here are relu, the bent line, and the sigmoid, the S-shaped curve, drawn together near 0. Drag the point along relu: the line through it is its tangent, and the readout gives relu's slope there.
Why corners are fine and jumps aren't. A corner affects only one exact point, and code simply picks a slope there, usually 0. Everywhere else relu has a clear slope, 0 or 1, which is cheap to compute, so it's the most common activation today. A jump gives no useful slope at all, so nothing tells training which way to move.
In PyTorch. The activations come ready-made, and they work on a whole tensor at once: F.relu and torch.sigmoid. Here they are at the closest pair from the tables, −0.001 and 0.001. dtype=torch.float64 keeps plain Python's precision, and .round(decimals=6) rounds every entry, like round:
import torch
import torch.nn.functional as F
x = torch.tensor([-0.001, 0.001], dtype=torch.float64)
print(F.relu(x).tolist())
print(torch.sigmoid(x).round(decimals=6).tolist())
v = torch.tensor([4.0, 0.0])
g = torch.tensor([2.0, 0.0])
print((g / torch.sqrt(v)).tolist())
print((g / torch.sqrt(v + 1e-8)).tolist())
The sigmoid gives 0.49975 and 0.50025, the same pair the cell above printed for h = 0.001, and relu gives 0.0 and 0.001. The last two lines are epsilon in real training code. Layer norm, a step inside every transformer, divides by torch.sqrt(v + eps), where v is an average of squares that can be exactly 0. Without eps, the second entry is 0 ÷ 0, which PyTorch writes as nan, "not a number": the hole again, and a nan spreads to everything computed from it. With eps = 1e-8 it comes out 0.0. Adam, a way of training (lesson 80), guards the same division with its eps just outside the root, torch.sqrt(v) + eps.
Drill
Smooth, corner or jump?
For each activation, set its label in kinds to "smooth", "corner" or "jump", judged at 0: step, relu, sigmoid, and leaky (x for x ≥ 0, 0.1x below). Use approach to check the values, and the slopes on each side for corners.
import math
defapproach(f, a):
returnround(f(a - 0.001), 4), round(f(a + 0.001), 4)
defstep(x):
if x >= 0:
return1return0defrelu(x):
returnmax(0, x)
defsigmoid(x):
return1 / (1 + math.exp(-x))
defleaky(x):
if x >= 0:
return x
return0.1 * x
kinds = ["smooth", "smooth", "smooth", "smooth"]
print(approach(step, 0), approach(relu, 0), approach(sigmoid, 0), approach(leaky, 0), kinds)
In the lesson, your code is checked as you write it.
In your own words
Why would a network built from step functions be hard to train by following slopes?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write