There's no dividing by a matrix. To free X from AX = B, you multiply both sides by A⁻¹, the matrix that undoes A. Since AB and BA are usually different, A⁻¹ has to go on the same side as A. You'll use that move on systems, the eigenvalue equation, a model's best weights and a layer run backward, and check its conditions each time.
What you'll be able to do
Free X from AX = B or XA = B by multiplying by A⁻¹ on the side where A is, and show that the other side gives a wrong answer
Undo sums and number multiples as usual, and flip the order to undo a product: (AB)⁻¹ = B⁻¹A⁻¹ and (AB)ᵀ = BᵀAᵀ
State the conditions: a square matrix, a determinant that isn't 0, and shapes that fit
Rearrange Ax = λx into det(A − λI) = 0, and say why that determinant must be 0
Solve with np.linalg.solve, check with np.allclose, and turn around the normal equations and a linear layer
Quick check
Have a guess before you start. For numbers, 3x = 12 gives x = 12 ÷ 3. For 2 × 2 matrices, AX = B. What is X?
BA⁻¹
B − A
A⁻¹B
B ÷ A
Show the answer
A⁻¹B
There's no dividing by a matrix. Multiply both sides by A⁻¹ on the left, where A sits: A⁻¹AX = IX = X, so X = A⁻¹B. BA⁻¹ is usually a different matrix.
The math: undo a matrix from the side it's on
Purpose. Free an unknown matrix or vector from a matrix equation, and know when you can.
The inverse A⁻¹ undoes A: A⁻¹A = AA⁻¹ = I (lesson 59). The identity I changes nothing: IX = X (lesson 57). And order matters: AB and BA are usually different (lesson 56).
So there's no dividing by a matrix: "B ÷ A" could mean A⁻¹B or BA⁻¹. Instead, multiply both sides by A⁻¹, on the side where A is:
Equation
X on its own
A + X = B
X = B − A: take A away
kX = B
X = B/k: divide by the number k (k not 0)
AX = B
X = A⁻¹B: A⁻¹ on the left, where A is
XA = B
X = BA⁻¹: A⁻¹ on the right, where A is
Xᵀ = B
X = Bᵀ: transpose both sides
On A's side, A⁻¹ meets A and cancels it: A⁻¹AX = IX = X. On the far side, the two never meet.
Quick check
From an earlier lesson: what is the inverse of [[2, 1], [1, 1]]?
[[1, −1], [−1, 2]]
[[1, 1], [1, 2]]
[[−2, −1], [−1, −1]]
[[0.5, 1], [1, 1]]
Show the answer
[[1, −1], [−1, 2]]
The determinant is 2 × 1 − 1 × 1 = 1. Swap a and d, flip the signs of b and c, and divide by 1. Check: [[2, 1], [1, 1]] times [[1, −1], [−1, 2]] is I.
Here's lesson 56's shopping, made small. Hiker 1 buys 2 bars and 1 apple, and hiker 2 buys one of each: A = [[2, 1], [1, 1]], a row per hiker. Their bills at two shops are C = [[7, 6], [4, 4]], a column per shop. The prices P are unknown (a row per item, a column per shop), and AP = C. P is on the right of A, so A⁻¹ goes on the left: P = A⁻¹C.
Try it now
Then swap them: make the left matrix [[7, 6], [4, 4]] and the right matrix [[1, -1], [-1, 2]], and press Use these numbers. That's CA⁻¹, the wrong side.
Symbols and formula. The two sides, and what happens to a product:
AX=B⇒X=A−1BXA=B⇒X=BA−1(AB)−1=B−1A−1(AB)T=BTAT
In plain words: multiply both sides by the inverse, on the side where the matrix is. A product comes apart from the outside in, so its order flips.
In ABx, B acts on x first and A acts last. So A comes off first, as in lesson 30: when ABx = c, multiply by A⁻¹ and then by B⁻¹, and x = B⁻¹A⁻¹c. A transpose flips the same way.
Conditions. A⁻¹ exists only when A is square and det A ≠ 0. A singular matrix, with det A = 0, squashes the plane flat, so AX = B has no answer or endless ones (lesson 60). The shapes must fit too: A's columns must match X's rows.
Quick check
A is 5 × 2 and B is 2 × 3. What is (AB)ᵀ, and what shape is it?
AᵀBᵀ, 3 × 5
AᵀBᵀ, 2 × 2
BᵀAᵀ, 5 × 3
BᵀAᵀ, 3 × 5
Show the answer
BᵀAᵀ, 3 × 5
AB is 5 × 3, so its transpose is 3 × 5. Bᵀ is 3 × 2 and Aᵀ is 2 × 5, and they fit in the order BᵀAᵀ.
Named mistakes.
Dividing: writing B/A. Multiply by A⁻¹ instead, on A's side.
The wrong side: X = BA⁻¹ from AX = B. In the shop, it made an apple free.
Keeping the order: (AB)⁻¹ = A⁻¹B⁻¹. A product comes apart in reverse.
Splitting a sum: (A + B)⁻¹ isn't A⁻¹ + B⁻¹, just as 1/(2 + 3) isn't 1/2 + 1/3.
Sums and numbers undo as usual: A + X = B gives X = B − A, and 4X = B gives X = B/4.
Quick check
XAB = C, with A and B invertible. What is X?
X = A⁻¹B⁻¹C
X = CB⁻¹A⁻¹
X = B⁻¹A⁻¹C
X = CA⁻¹B⁻¹
Show the answer
X = CB⁻¹A⁻¹
X is on the left, so the inverses go on the right. B is the outside matrix on that side, so it comes off first: XABB⁻¹ = XA, then XAA⁻¹ = X. So X = CB⁻¹A⁻¹.
The eigenvalue equation.lesson 62 looked for vectors v and numbers λ (lambda) with Av = λv. Call the unknown vector x and rearrange Ax = λx. Take λx away: Ax − λx = 0. A number can't be taken from a matrix, so write λx as λIx and factor: (A − λI)x = 0.
Now try to free x with (A − λI)⁻¹. You'd get x = 0, and the zero vector isn't an eigenvector. So an eigenvector exists only when A − λI has no inverse:
Ax=λx⇒(A−λI)x=0⇒det(A−λI)=0
In plain words: λ is an eigenvalue when A − λI squashes some direction to 0, so its determinant is 0.
For A = [[4, 1], [2, 3]], that's (4 − λ)(3 − λ) − 2 = λ² − 7λ + 10 = 0, so λ = 5 or 2.
In your own words
In your own words: why does the inverse have to go on the same side as the matrix it undoes?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: solve, then put the answer back
Purpose. Let NumPy find X, and prove it right by putting it back.
np.linalg.solve(A, B) solves AX = B by elimination, without building A⁻¹, so it's faster and loses less to rounding than np.linalg.inv(A) @ B. Then check with np.allclose(A @ X, B).
import numpy as np
A = np.array([[2, 1], [1, 1]])
C = np.array([[7, 6], [4, 4]])
P = np.linalg.solve(A, C)
print(P.tolist(), np.allclose(A @ P, C))
wrong = C @ np.linalg.inv(A)
print(wrong.tolist(), np.allclose(A @ wrong, C))
It runs right here in your browser, with nothing to install.
The prices come back, and the check says True. The wrong side, CA⁻¹, fails the same check.
Worked example
Worked example: the unknown on the left
Now the prices are known and the counts aren't: AP = C, so A = CP⁻¹. np.linalg.solve only solves from the left, so start by transposing both sides. The order flips, (AP)ᵀ = PᵀAᵀ, so PᵀAᵀ = Cᵀ, and the unknown Aᵀ now sits on the right of Pᵀ.
import numpy as np
P = np.array([[3, 2], [1, 2]])
C = np.array([[7, 6], [4, 4]])
A = np.linalg.solve(P.T, C.T).T
print(np.round(A, 6).tolist(), np.allclose(A @ P, C))
print(np.round(np.linalg.solve(P, C), 6).tolist())
In the lesson you can change this and run it.
Solving PᵀAᵀ = Cᵀ gives Aᵀ, and one more .T gives the counts, [[2.0, 1.0], [1.0, 1.0]]. The check multiplies on the correct side, A @ P. The last line solves PA = C, the wrong side, and gets 1.25 bars for hiker 2. It all works because det P = 4, not 0.
Quick check
Here AB ≠ BA. What does this print?
import numpy as np
A = np.array([[1, 2], [3, 4]])
B = np.array([[0, 1], [1, 0]])
X1 = np.linalg.inv(A) @ B
X2 = B @ np.linalg.inv(A)
print(np.allclose(A @ X1, B), np.allclose(A @ X2, B), np.allclose(X2 @ A, B))
True False False
False False True
True True True
True False True
Show the answer
True False True
X1 = A⁻¹B solves AX = B, so A @ X1 gives B back. X2 = BA⁻¹ solves XA = B instead: X2 @ A is B, but A @ X2 isn't.
Quick check
Why prefer np.linalg.solve(A, B) to np.linalg.inv(A) @ B?
solve also works when A is singular
solve doesn't need A to be square
inv(A) @ B is on the wrong side
It's faster and loses less to rounding
Show the answer
It's faster and loses less to rounding
Both mean A⁻¹B, but solve runs elimination directly instead of building the whole inverse first. That's less work and less float rounding. Neither helps when A is singular or not square.
Drill
Solve on the correct side
solve_left(A, B) should solve AX = B, and solve_right(A, B) should solve XA = B. Each has its inverse on the wrong side. Fix them with np.linalg.solve, and return None when A isn't square or its determinant is within 1e-9 of 0.
import numpy as np
defsolve_left(A, B):
return B @ np.linalg.inv(A)
defsolve_right(A, B):
return np.linalg.inv(A) @ B
A = np.array([[3, 1], [5, 2]])
B = np.array([[13, 5], [5, 2]])
print(solve_left(A, B), solve_right(A, B))
In the lesson, your code is checked as you write it.
In your own words
Why check a matrix answer with np.allclose(A @ X, B), and not just trust the code that found it?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: best weights, and a layer run backward
Purpose. Turn around two of a model's matrix equations: its best weights and its layers.
Least-squares weights. The trail club's sign-ups over four weeks were 3, 5, 6 and 8. No line hits all four, so take the one whose squared misses add up to the least (lesson 101 builds it). Its weights w solve the normal equations, XᵀXw = Xᵀy, where each row of X is [1, week]: the 1 carries the line's starting value. X is 4 × 2, with no inverse, but XᵀX is 2 × 2, so peel it off from the left:
XTXw=XTy⇒w=(XTX)−1XTy
In plain words: X isn't square, but XᵀX is, and it's invertible as long as no column of X is a mix of the others.
import numpy as np
X = np.array([[1, 1], [1, 2], [1, 3], [1, 4]])
y = np.array([3, 5, 6, 8])
XtX = X.T @ X
Xty = X.T @ y
w = np.linalg.solve(XtX, Xty)
print(XtX.tolist(), Xty.tolist(), round(np.linalg.det(XtX), 6))
print(np.round(w, 6).tolist(), np.round(X @ w, 6).tolist())
The line is 1.5 + 1.6 × week, and it misses each week by 0.3 or less.
Quick check
A data table X has 50 rows, one per day, and 3 columns. Why not solve Xw = y as w = X⁻¹y?
X isn't square, so it has no inverse
y has the wrong length for X
50 rows is too few to fit 3 weights
X⁻¹ would be on the wrong side
Show the answer
X isn't square, so it has no inverse
Only a square matrix can have an inverse, and X is 50 × 3. XᵀX is 3 × 3, so the normal equations peel that off instead: w = (XᵀX)⁻¹Xᵀy. With 50 equations and 3 weights, no w fits every day exactly anyway.
A layer run backward. A linear layer computes y = Wx + b. If W is square and invertible, peel in reverse: take the bias away first, then multiply by W⁻¹ on the left, so x = W⁻¹(y − b).
import numpy as np
W = np.array([[1, 2], [1, 3]])
b = np.array([1, -1])
x = np.array([2, 1])
y = W @ x + b
print(y.tolist(), np.round(np.linalg.solve(W, y - b), 6).tolist())
flat = np.array([[1, 2], [2, 4]])
print(np.linalg.matrix_rank(flat), (flat @ np.array([2, 0])).tolist(), (flat @ np.array([0, 1])).tolist())
Most real layers can't be undone. One from 784 pixels to 10 scores has a 10 × 784 W: not square, so no inverse, and many pictures give the same 10 scores. A square W fails too when its rank is below its size (lesson 61): above, flat sends (2, 0) and (0, 1) to the same place.
Quick check
What does this print?
import numpy as np
W = np.array([[1, 2], [1, 3]])
b = np.array([1, -1])
y = np.array([5, 4])
print(np.linalg.solve(W, y - b).tolist(), (np.linalg.solve(W, y) - b).tolist())
[6.0, 0.0] [2.0, 1.0]
[4.0, 5.0] [6.0, 0.0]
[2.0, 1.0] [2.0, 1.0]
[2.0, 1.0] [6.0, 0.0]
Show the answer
[2.0, 1.0] [6.0, 0.0]
b was added last, so it comes off first: W⁻¹(y − b) = (2, 1), the real input. Taking b away after undoing W gives (6, 0), which is wrong: W(6, 0) + b = (7, 5), not y.
Drill
Run a layer backward
Fix undo_layer(W, b, y) so it returns the input x with Wx + b = y. Return None when W isn't square or its rank is less than its size (use np.linalg.matrix_rank).
import numpy as np
defundo_layer(W, b, y):
return np.linalg.solve(W, y) - b
W = np.array([[1, 2], [1, 3]])
b = np.array([1, -1])
print(undo_layer(W, b, np.array([5, 4])))
In the lesson, your code is checked as you write it.
In your own words
A layer maps 784 pixels to 10 scores. Why can't any formula get the picture back from the scores?
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
lesson 59 and lesson 60: inverses, determinants, and the elimination inside np.linalg.solve.
lesson 61: rank, and why a squashed direction can't come back.
lesson 101: the least-squares line that the normal equations solve.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
Learn to make any letter of any formula the subject: layers undone in reverse order, fractions, a letter that appears twice, powers, roots, quadratics, logs, inequalities, trigonometry, vectors, matrices, calculus, probability, statistics and the formulas inside AI models, each checked in Python.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write