Phase 2 · Math, when you need it · Linear algebra

Transpose, identity and inverse

The matrix operations you'll see in every formula.

About 25 minutes · A free account opens the whole lesson

Create a free account Try part of it here

Already have an account? Sign in to open this lesson

Why it matters

Three small operations turn up in almost every AI formula. The transpose flips a table so rows become columns. The identity is the "do nothing" matrix. The inverse undoes a transformation, and it can solve a set of equations in one step. Knowing these three lets you read the formulas in AI papers and documentation, instead of skipping over them.

What you'll be able to do

Transpose a matrix and use the rule for transposing a product
Write a dot product as aᵀb and recognize XᵀX and QKᵀ in formulas
Explain the identity matrix and what an inverse undoes
Solve a small system of equations, and know when no inverse exists

Transpose: turn the table on its side

Picture a spreadsheet with 2 rows and 3 columns. Now rotate it so the first row becomes the first column, the second row becomes the second column, and so on. That flip is the transpose. A 2 × 3 table becomes 3 × 2.

[[1, 2, 3],        [[1, 4],
 [4, 5, 6]]   →     [2, 5],
                    [3, 6]]

In NumPy, add .T to any array:

import numpy as np

A = np.array([[1, 2, 3], [4, 5, 6]])
print(A.T)
print(A.shape, "->", A.T.shape)

It runs right here in your browser, with nothing to install.

In plain words: the transpose swaps rows and columns. A table that was tall becomes wide.

Go deeper The notation

The transpose of $A$ is written $A^\top$ , with a small T:

A = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix} \qquad A^\top = \begin{bmatrix} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{bmatrix}

Read $(A^\top)_{ij} = A_{ji}$ as: "the number in row i, column j of the flipped table is the number in row j, column i of the original."

Quick check

What does this print?

m = [[1, 2, 3], [4, 5, 6]]
t = list(zip(*m))
print(t[0])

(1, 2)
(4, 5, 6)
(1, 2, 3)
(1, 4)

Show the answer

(1, 4)

zip(*m) lines up the rows and pairs their numbers by position, so rows become columns. The first new row is the old first column: 1 and 4.

Quick check

A table has shape (5, 2). What's the shape of its transpose?

(2, 5)
(5, 5)
(5, 2)
(10,)

Show the answer

(2, 5)

Transposing swaps rows and columns: 5 rows and 2 columns become 2 rows and 5 columns.

Why AI formulas are full of transposes

Mostly, a transpose makes shapes fit for matrix multiplication. Remember the shape rule: the inside numbers must match. Flipping one table often makes them match.

The famous example is attention, the core of models like ChatGPT and Claude. Each word gets a "what am I looking for" vector (a query) and a "what do I offer" vector (a key). Stack the queries as rows of a table Q, and the keys as rows of a table K. To compare every word's query with every word's key, you multiply Q by the transpose of K:

import numpy as np

rng = np.random.default_rng(0)
Q = rng.normal(size=(4, 8))     # 4 tokens, 8-dimensional queries
K = rng.normal(size=(4, 8))     # 4 tokens, 8-dimensional keys
scores = Q @ K.T
print(scores.shape)             # (4, 4): every token against every token
print(np.isclose(scores[1, 2], Q[1] @ K[2]))

Shapes: (4 × 8) @ (8 × 4) = (4 × 4). Without the .T, the shapes (4 × 8) @ (4 × 8) wouldn't fit. The result is a 4 × 4 table of "how relevant is word j to word i". The last line confirms that one cell is just one dot product.

Go deeper Three facts you'll see in formulas

A dot product is a transpose times a vector. If vectors are written as columns, $\mathbf{a} \cdot \mathbf{b} = \mathbf{a}^\top \mathbf{b}$ .
Transposing a product reverses the order: $(AB)^\top = B^\top A^\top$ , like taking off shoes before socks.
$X^\top X$ compares every feature with every feature. If $X$ is (examples × features), $X^\top X$ is (features × features). Each entry is the dot product of two feature columns. Statistics uses it to see which features rise and fall together.

The identity: do nothing

Multiplying a number by 1 leaves it alone. Matrices have the same thing: the identity matrix, written I. It has 1s down the diagonal and 0s everywhere else:

[[1, 0],
 [0, 1]]

Its columns are [1, 0] and [0, 1]: "one step right" stays one step right, and "one step up" stays one step up. So it moves nothing. Any matrix times I is unchanged.

In plain words: the identity matrix is the matrix version of the number 1.

The inverse: undo

If a matrix turns the page 30° clockwise, its inverse turns it 30° back. Do a matrix and then its inverse, and you're back where you started: the combination is the identity.

But some motions can't be undone. A matrix with determinant 0 squashes the page flat, so two different points land on the same spot. Afterwards, you can't tell which point came from where. Such a matrix has no inverse. Mathematicians call it singular. It's the algebra version of "you can't unsquash a pancake". In short: determinant 0 means the matrix flattens space.

In plain words: the inverse undoes a matrix. Only matrices that don't squash the page flat (determinant not 0) have one.

Go deeper The 2 × 2 inverse formula

\begin{bmatrix} a & b \\ c & d \end{bmatrix}^{-1} = \frac{1}{ad - bc} \begin{bmatrix} d & -b \\ -c & a \end{bmatrix}

Read it as: "swap a and d, flip the signs of b and c, then divide everything by the determinant." When the determinant $ad - bc$ is 0, that's a division by zero: no inverse exists.

The identity and inverse in symbols: $AI = IA = A$ and $A^{-1}A = AA^{-1} = I$ .

Try it now

A matrix moves the grid. Its inverse moves it back. Try a matrix whose square collapses to a line: area 0 means nothing can undo it.

Quick check

A matrix squashes the whole page onto a single line. Does it have an inverse?

Yes, its inverse stretches the line back out
Yes, every matrix has an inverse
Only if the line is horizontal
No: different points landed on the same spot, so it can't be undone

Show the answer

No: different points landed on the same spot, so it can't be undone

Squashing loses information (its determinant is 0), so no matrix can undo it.

Solving a puzzle with an inverse

A shop sold bundles on two days.

Monday: 3 hoodies and 2 caps for $270.
Tuesday: 1 hoodie and 4 caps for $290.

What does one hoodie cost, and one cap? Put the counts in a matrix, the totals in a vector, and the inverse finds the prices:

[[3, 2],     [hoodie,      [270,
 [1, 4]]  @   cap]     =    290]

import numpy as np

A = np.array([[3, 2], [1, 4]])
b = np.array([270, 290])
print(np.linalg.inv(A) @ b)        # with the inverse
print(np.linalg.solve(A, b))       # what you'd use in practice: faster and more accurate

A hoodie costs $50 and a cap $60. Check: 3 × 50 + 2 × 60 = 270. ✓

In real code, use np.linalg.solve rather than working out the inverse first. It gets the same answer more accurately. In Phase 3, linear regression finds its best-fit line with this same idea, applied to thousands of equations at once.

Go deeper The equations in math notation

\begin{aligned} 3h + 2c &= 270 \\ h + 4c &= 290 \end{aligned} \quad\Longleftrightarrow\quad A\mathbf{x} = \mathbf{b} \quad\Rightarrow\quad \mathbf{x} = A^{-1}\mathbf{b}

Read it as: "the two equations are the same as A times x equals b, so x is the inverse of A times b." Linear regression's formula, $\mathbf{w} = (X^\top X)^{-1} X^\top \mathbf{y}$ , is this pattern with a transpose added so the shapes fit.

Quick check

Q has shape (10, 64) and K has shape (10, 64). What's the shape of Q @ K.T?

(10, 64)
It's an error
(64, 64)
(10, 10)

Show the answer

(10, 10)

(10 × 64) @ (64 × 10) = (10 × 10): every query against every key.

Easy to mix up

A transpose and an inverse aren't the same. A transpose flips a table on its side and always exists. An inverse undoes a matrix, and many square matrices don't have one.

The order flips when you transpose a product: $(AB)^\top$ is $B^\top A^\top$ . In real code, use np.linalg.solve(A, b), which is faster and more accurate than computing the inverse first.

A quick check

Multiply a matrix by its inverse: you should get the identity, give or take rounding.

Practice what you just read

3 code examples you can edit and run
a practice exercise that checks your code
a 5-question quiz
5 review cards for spaced repetition

Create a free account

Try a quiz question

Which is equal to (AB)ᵀ?

BᵀAᵀ
AᵀBᵀ
AB
BA

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Matrix multiplication