Index Ai Academy
Phase 7 · Writing good code

Testing numeric and random code

Close enough for floats, seeds and what they fix, properties that hold for any input, a gradient check and overfit-one-batch.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Ordinary tests say assert f(2) == 4. Numeric code breaks that habit twice. A float result is almost never exactly what you wrote down, and a function that uses random numbers gives a different answer each run. The usual reaction is to skip the test, and then a change that quietly breaks the model reaches production. This lesson gives you the four tools that fix both: a tolerance for equality, a generator you control, properties that hold for every input, and one test that trains a tiny model on purpose.

What you'll be able to do

  • Compare floating-point numbers with a tolerance, and pick a sensible rtol and atol
  • Make a random test repeatable by passing the generator in, and say what a seed does and does not fix
  • Write tests for properties that hold for any input: shapes, sums, invariances, and a gradient that matches a finite difference
  • Use "overfit one batch" as a test, and torch.testing.assert_close with a tiny model in a real codebase
Quick check
A guess before we start. In Python, what does 0.1 + 0.2 == 0.3 give?
  1. False
  2. 0.3
  3. True
  4. An error
Show the answer

False

0.1 and 0.2 have no exact binary form, so their sum is 0.30000000000000004, a hair above 0.3. Comparing floats with == is a trap. This lesson shows the fix.

The idea: close enough, repeatable, and true for every input

This rests on Testing with pytest: a test is a function that asserts what should be true, and a fixture builds what it needs.

Numeric code asks three new questions of a test.

What does "equal" mean? A float is a rounded number. Two correct calculations of the same value can differ in the last digits, so a test compares with a tolerance: the numbers are equal if the gap between them is small. A relative tolerance (rtol) scales with the size of the numbers, and an absolute tolerance (atol) covers values near zero. The common rule is |a − b| ≤ atol + rtol × |b|.

Is it repeatable? A function that draws random numbers gives new answers every run, which makes a test that passes on Monday and fails on Tuesday. A seed starts a generator from a fixed place, so it makes the same draws every time. It fixes only the generator you seeded: a different library, the order of a set, or a thread race still varies.

What is true for any input? When there is no single right answer to write down, test a property: the output shape, a sum that must be 1, an answer that must not change when the input is shifted.

Quick check
From the lesson on pytest: a test is a function whose name starts with test_ and that uses assert. Which of these is a complete test?
  1. def test_double(): assert 2 * 2 == 4
  2. assert 2 * 2 == 4, with no function
  3. def test_double(): return 2 * 2
  4. def double(): assert 2 * 2 == 4
Show the answer

def test_double(): assert 2 * 2 == 4

pytest collects functions named test_something and runs them. A passing test returns without an exception: a failing assert raises, which pytest reports. A function that only returns a value checks nothing.

The picture. One sum, checked three ways. Step through what == sees, the size of the gap, and what a tolerance says about it.

Try it now

Step through. The question is never are they identical; it is whether the gap is small enough to ignore.

In your own words
In your own words: why is a test of a random function written with a seed, and what does a seed not fix?

Write a sentence or two, then compare it with a good answer in the lesson.

In Python: tolerances, generators and properties

Purpose. Four tools, each a few lines: a tolerance, a generator you pass in, a property, and a gradient check.

A tolerance. np.allclose and pytest.approx both compare with the rule from above. They accept numbers, lists and arrays:

import numpy as np
import pytest

total = 0.1 + 0.2
print(total == 0.3, np.allclose(total, 0.3), total == pytest.approx(0.3))
print(np.allclose([1.0, 2.0], [1.0, 2.1]), np.allclose([1.0, 2.0], [1.0, 2.0000001]))
print(np.allclose(0.0, 1e-12), np.allclose(0.0, 1e-12, atol=0))
It runs right here in your browser, with nothing to install.

The last line shows why atol exists. Next to zero the relative tolerance shrinks to nothing, so a value of 1e-12 would never be close to 0 without an absolute tolerance.

A generator you pass in. If a function draws its own random numbers, a test cannot control them. Give it the generator as an argument, and the test passes a seeded one. The production code passes a fresh one:

import numpy as np

def add_noise(x, rng):
    return x + rng.normal(0, 0.1, size=x.shape)

x = np.zeros(3)
a = add_noise(x, np.random.default_rng(7))
b = add_noise(x, np.random.default_rng(7))
print(np.array_equal(a, b), a.shape)
Quick check
What does this print?
import numpy as np

near = np.allclose(100.0, 100.001, rtol=1e-3, atol=0)
far = np.allclose(100.0, 100.5, rtol=1e-3, atol=0)
print(near, far)
  1. True False
  2. True True
  3. False True
  4. False False
Show the answer

True False

The tolerance is 1e-3 × 100.001 ≈ 0.1. A gap of 0.001 is far inside it, and a gap of 0.5 is well outside. A relative tolerance grows with the size of the numbers.

Code flow. A gradient check compares a hand-derived slope with a finite difference: nudge x up and down by a tiny h and see how much the function moved. It is how you test a derivative you wrote by hand, which is what every custom layer is. For f(x) = x³ the slope is 3x², which at x = 2 is 12.

Quick check
A function draws from random.Random(5) inside its body. A test calls it twice. What does this print?
import random

def pick(items):
    return random.Random(5).choice(items)

print(pick([1, 2, 3, 4]) == pick([1, 2, 3, 4]))
  1. An error
  2. True
  3. False
  4. It varies from run to run
Show the answer

True

The generator is made fresh from the same seed on every call, so both calls make the same draw. A seed inside the function makes it repeatable, but the test cannot choose a different seed. Passing the generator in is more flexible.

Quick check
Which test is the stronger check of a custom layer's backward pass?
  1. Print the gradient and look at it
  2. Check that the output has the right shape
  3. Compare its gradient with a finite difference at many random inputs
  4. Check that backward() returns without raising an error
Show the answer

Compare its gradient with a finite difference at many random inputs

A finite difference measures the slope straight from the forward function, so it catches a wrong formula anywhere in the range. Running without an error or giving the right shape says nothing about the numbers.

Drill

Write isclose from the rule

Write is_close(a, b, rtol=1e-5, atol=1e-8): True when abs(a - b) <= atol + rtol * abs(b). That is the rule NumPy uses.

def is_close(a, b, rtol=1e-5, atol=1e-8):
    ...


print(is_close(0.1 + 0.2, 0.3), is_close(1.0, 1.1))
In the lesson, your code is checked as you write it.
Drill

A gradient check

Write grad_ok(f, df, x, h=1e-5, tol=1e-6): compare the central difference (f(x + h) - f(x - h)) / (2 * h) with df(x), and return True when they differ by less than tol.

def grad_ok(f, df, x, h=1e-5, tol=1e-6):
    ...


print(grad_ok(lambda x: x ** 3, lambda x: 3 * x ** 2, 2.0))
print(grad_ok(lambda x: x ** 3, lambda x: 2 * x ** 2, 2.0))
In the lesson, your code is checked as you write it.

A property test. For softmax there is no single "right" output to write down for random inputs, but three things are true of every output. The probabilities add up to 1, the shape is unchanged, and adding the same number to every score changes nothing. The cell checks all three on made-up scores, and then a naive version on a large input, to see what a property test catches that a hand-picked example can miss:

import numpy as np

def softmax(x):
    e = np.exp(x - np.max(x, axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

def naive(x):
    e = np.exp(x)
    return e / e.sum(axis=-1, keepdims=True)

scores = np.array([[1.0, 2.0, 3.0], [0.0, 0.0, 0.0]])
probs = softmax(scores)
print(np.allclose(probs.sum(axis=-1), 1.0), probs.shape == scores.shape)
print(np.allclose(softmax(scores), softmax(scores + 100)))
big = np.array([1000.0, 1000.0])
print(softmax(big).tolist(), np.isnan(naive(big)).any())

(NumPy may print a warning for the naive version. That is the bug being caught.) The naive version is correct on small inputs and returns nothing but NaN on large ones. A property such as "the output sums to 1, for scores of every size" finds it.

Quick check
A test checks only softmax([1, 2, 3]) against numbers copied from the output. What could it miss?
  1. A version that is correct on [1, 2, 3]
  2. A version that overflows on large inputs
  3. Nothing: one example is enough
  4. A version that returns the right shape
Show the answer

A version that overflows on large inputs

A fixed example pins down only that one input. A naive softmax passes it and returns NaN for [1000, 1000]. A test for a property over many inputs, including large ones, catches it.

Overfit one batch. The best single test of a training loop is to train a tiny model on a handful of examples and expect the loss to fall to nearly zero. If it cannot memorize eight examples, something is broken: the loss, the gradient, the optimizer or the data. The cell fits a linear model to eight points:

import numpy as np

rng = np.random.default_rng(0)
X = rng.standard_normal((8, 3))
w_true = np.array([1.0, -2.0, 0.5])
y = X @ w_true
w = np.zeros(3)
losses = []
for step in range(200):
    grad = 2 * X.T @ (X @ w - y) / len(X)
    w -= 0.1 * grad
    losses.append(float(np.mean((X @ w - y) ** 2)))
print(losses[0] > losses[-1], losses[-1] < 0.001)

The loss started high and ended below 0.001. The test is cheap, it runs in a fraction of a second, and it fails loudly if a change breaks the learning.

Worked example

Worked example: a test file for softmax

The three property tests as plain pytest functions, with the generator passed in. Run directly here, as a list of results.

import numpy as np
import pytest

def softmax(x):
    e = np.exp(x - np.max(x, axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

def test_sums_to_one():
    x = np.random.default_rng(0).standard_normal((4, 6))
    assert softmax(x).sum(axis=-1) == pytest.approx(1.0)

def test_shift_invariant():
    x = np.random.default_rng(1).standard_normal(5)
    assert softmax(x) == pytest.approx(softmax(x + 1000))

def test_shape_kept():
    assert softmax(np.zeros((8, 5))).shape == (8, 5)

for test in (test_sums_to_one, test_shift_invariant, test_shape_kept):
    test()
    print(test.__name__, "passed")
In the lesson you can change this and run it.

Each test builds its own seeded generator, so it repeats. The comparisons use pytest.approx, never ==. And no test needs a number copied from the output: they say what must be true, so they keep passing when the implementation changes, and fail when it breaks.

In your own words
Which would you trust more as a test of a new activation function's gradient: a hand-picked example with the expected number written down, or a finite-difference check at many random points? Why?

Write a sentence or two, then compare it with a good answer in the lesson.

In a real codebase: tests for a model library

Purpose. In a real repository these tools sit in a tests/ folder and run on every push.

With PyTorch the same ideas have their own names. torch.manual_seed(0) seeds PyTorch's generator, and a test seeds it first. torch.testing.assert_close is allclose for tensors: it checks shape, dtype and values, and its message says how far apart they are. A tiny-model fixture builds a small network with a fixed seed, so every test starts from the same weights. And a golden file stores an output you have inspected and approved, so a later change that moves it is noticed.

import torch

torch.manual_seed(0)
a = torch.randn(3)
torch.manual_seed(0)
b = torch.randn(3)
print(torch.equal(a, b))

torch.testing.assert_close(torch.tensor([1.0, 2.0]), torch.tensor([1.0, 2.0000001]))
print("close enough")
try:
    torch.testing.assert_close(torch.tensor([1.0, 2.0]), torch.tensor([1.0, 2.1]))
except AssertionError as error:
    print(str(error).splitlines()[0])

(This page runs a teaching subset of PyTorch: same calls, CPU only. The message wording matches the real library's first line.)

A flaky test, traced. A test that shuffles its data with an unseeded generator passes most days and fails some. The fix is the one from the top of the lesson: the function takes the generator, the test passes a seeded one.

import random

def first_after_shuffle(items, rng):
    items = list(items)
    rng.shuffle(items)
    return items[0]

a = first_after_shuffle([1, 2, 3, 4, 5], random.Random(0))
b = first_after_shuffle([1, 2, 3, 4, 5], random.Random(0))
print(a, a == b)

With a seed the first item is always the same, so a test can name it. Without one, any answer between 1 and 5 is possible, and the test would fail four times in five.

On the job: add three tests to every model library before anything else. A shape test for each public function, an overfit-one-batch test for the training loop, and a gradient check for every custom layer. Seed everything a test uses. When a test is flaky, look for the unseeded random call first.

In your own words
A test of your data pipeline fails about one run in five. What do you suspect first, and how do you confirm and fix it?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 7 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

Why does a test of a function that uses random numbers pass the generator in as an argument?

  1. So the test can pass a seeded generator and get the same result each run
  2. Because random numbers cannot be used inside any function
  3. To make the function run faster
  4. So the function can be used without NumPy

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 7 · about 48 hours of study

Software engineering

How software is built, from bits to tested, well-designed programs 59 lessons 1 free
  1. Iterators and generators FreeThe iteration protocol, yield, lazy pipelines and itertools, in small memory.45 min
  2. Context managersSetup and cleanup that always run: with, contextlib, transactions and atomic writes.35 min
  3. Closures and decoratorsFunctions that remember, wrap and extend other functions.40 min
  4. Exceptions in depthError hierarchies, chaining, notes, exception groups and warnings.40 min
  5. Pattern matchingmatch and case: literal, sequence, mapping and class patterns with guards.35 min
  6. Text, bytes and encodingsstr and bytes, UTF-8, decoding errors, normalization and base64.35 min
  7. Binary filesBytes as numbers, endianness, struct records, file signatures and checking uploads.45 min
  8. Testing your codeAutomated tests that prove code works and keep it working.45 min
  9. Testing with pytestFixtures, parametrize, mocks, temporary files and coverage.45 min
  10. Testing numeric and random codeClose enough for floats, seeds and what they fix, properties that hold for any input, a gradient check and overfit-one-batch.40 min
  11. Debugging systematicallyFind any bug with a method instead of guesswork.35 min
  12. Readable code and refactoringWrite code people can understand, and improve it safely.35 min
  13. Code style and toolsPEP 8, formatters, linters and type checkers, and how they fit together.35 min
  14. Types and data modelingType hints, dataclasses and checking data at the edges of a program.35 min
  15. Typing in depthGenerics, protocols, TypedDict, Literal and narrowing, checked by mypy.45 min
  16. Classes in depthProperties, alternative constructors, abstract base classes and tuned dataclasses.40 min
  17. Make your classes feel built inSpecial methods for printing, equality, ordering, arithmetic and containers.45 min
  18. Software designModules, interfaces and dependencies that keep big programs changeable.45 min
  19. LoggingLoggers, levels, handlers and configuration, without leaking secrets.40 min
  20. How computers run codeBits, memory, the processor, and what really happens when Python runs.40 min
  21. Arrays in memoryBuffers, dtypes, shapes and strides: why a transpose is free, and which operations copy.40 min
  22. Algorithm complexityCount the work code does, and see why some programs stop scaling.40 min
  23. Concurrency and asyncDo many things at once: processes, threads and the event loop.45 min
  24. Performance and profilingMeasure before you optimize, then fix the part that matters.45 min
  25. Profiling numeric code and a training loopWhere a training step spends its time, Amdahl's rule by hand, and cProfile on a loop with a slow batch function.40 min
  26. Reading C, C++, CUDA and RustPointers and loops in C, references and templates in C++, one-thread-per-element CUDA kernels, and Rust's Option, Result and ownership.30 min
  27. Core data structuresArrays, linked lists, stacks, queues and the hash table behind every dict.45 min
  28. Heaps and priority queuesA list read as a tree: the best k of a million without sorting, and where top-k hides in AI work.35 min
  29. Recursion, trees and graphsFunctions that call themselves, and the shapes most real data takes.45 min
  30. Searching, sorting and dynamic programmingThe classic algorithms, and the ideas behind them.45 min
  31. Dynamic programming in depthA method in five questions: state, choices, recurrence, base case and order. Coins where greedy fails, rebuilding the solution, packing a token budget, the likeliest path behind noisy readings, and what it costs.45 min
  32. Binary search trees and traversalsInsert, search and delete in a sorted tree, walk it four ways, and see why balance and B-trees matter.45 min
  33. Patterns for coding problemsTwo pointers, counting with dictionaries, prefix sums and sliding windows: O(n) answers to classic problems.45 min
  34. Project: a spell checker, three waysA sorted list, a hash set and a trie, measured for speed and memory, with suggestions.45 min
  35. Searching for solutionsBacktracking with pruning, Dijkstra's algorithm and A*, ending with an optimal sliding-puzzle solver.45 min
  36. Project: an unbeatable game opponentGame state and rules, minimax with a cache, and an exhaustive test that proves it never loses.45 min
  37. Databases and SQLStore, query and join data with the language every database speaks.40 min
  38. SQL for analysisSubqueries, CASE, common table expressions, outer joins and window functions on real order data.45 min
  39. Designing a databaseFrom one big table to related tables: keys, constraints, many-to-many and a loader that is safe to rerun.40 min
  40. Databases with an ORMSQLAlchemy models, sessions, relationships, transactions and the N+1 problem.45 min
  41. Changing a database safelyNumbered migrations in transactions, required columns and backfills, and expand-and-contract changes.40 min
  42. Secure codingInjection, escaping, secrets and passwords: the mistakes attackers look for.40 min
  43. Projects, packaging and CILay out a real project, pin its dependencies and test it on every push.35 min
  44. Config files: YAML and TOMLWrite YAML and TOML that parse as you meant, dodge the type traps, and validate what you load.25 min
  45. Command-line programsargparse subcommands, stdin, exit codes and testable main functions.40 min
  46. Packaging your codepyproject.toml, the src layout, versions, entry points and wheels.45 min
  47. Pipes and text tools in the shellgrep, wc, sort, uniq, head and cut, chained with pipes to answer questions about logs and data.25 min
  48. JSON on the command line: curl and jqCall APIs with curl, fail on HTTP errors, and pull values out of JSON with jq paths and select.25 min
  49. Shell scripts that fail safelyTurn commands into scripts with variables, set -euo pipefail and clear exit codes, and know when to switch to Python.25 min
  50. Reading and writing XMLElementTree, attributes and text, namespaces, building XML safely, and untrusted input.35 min
  51. Project: a word-guessing gamePlan a game with a flowchart, then build it from tested functions and one input loop.40 min
  52. Project: a card game of 21Turn written rules into functions, and make randomness testable.40 min
  53. Project: a drinks kioskA program with state and commands, built with functions, then refactored into classes.45 min
  54. Project: a flashcard trainerCards from a file, review boxes, and progress saved between runs.45 min
  55. Project: a price-watch alert botAn API client, alert rules, digest emails and a state file, so reruns never repeat.45 min
  56. Project: Pack Train, an arcade gameA grid game on the turtle canvas: game state as classes, a timer loop, key handlers and a saved best score.45 min
  57. Project: a private password-breach checkHash locally, send five characters, match locally: k-anonymity, safe logging and failure handling.40 min
  58. Capstone: a command-line appDesign, build and test a complete program with a database.45 min
  59. Capstone: ship a small model libraryA typed, tested and profiled package with a train and an eval command, a YAML config, logging and a wheel.45 min
Part of a free course 127 lessons

Free Python developer course

Go from your first line of code to working as a Python developer: projects, testing, data, databases, web apps and APIs, all in your browser.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write