Free Python developer course
Go from your first line of code to working as a Python developer: projects, testing, data, databases, web apps and APIs, all in your browser.
Buffers, dtypes, shapes and strides: why a transpose is free, and which operations copy.
Every tensor you have used in this course is an array in memory: one block of equal-size numbers, plus a small rule that says how to read it as rows and columns. Nothing teaches that rule, and it explains many things that otherwise look like magic: why a transpose costs nothing, why view sometimes fails and reshape does not, why changing one array changes another, and why a "best model so far" saved with a plain assignment quietly follows the model it was meant to remember. This lesson builds the rule by hand, then reads it off real NumPy and PyTorch arrays.
b = a.T, the transpose of a big array, and then change b[0, 1] = 99. What happens to a?One of a's elements changes too
The transpose is a view: it reads the same numbers in the same block of memory, in a different order. So a change through b is a change in a. This lesson shows why.
This rests on How computers run code: memory is a long row of numbered bytes, and a program names a place in it by its address.
A Python list of numbers is a row of pointers. Each slot holds the address of a separate number object somewhere else. That is flexible, since the objects can be any type, and slow, since every read follows a pointer. An array is different: one block of memory that holds the numbers themselves, one after another, all the same size. Three small facts describe it:
float32, says how many bytes each number takes (4, here) and how to read them;(3, 4), says how to see the block as rows and columns;(16, 4), say how many bytes to move to reach the next row, and the next column.The block has 3 × 4 = 12 numbers, so it is 12 × 4 = 48 bytes. The element at row i, column j starts at byte i × 16 + j × 4. That one line, offset = i × stride₀ + j × stride₁, is the whole secret of an array.
104
The first number uses addresses 100, 101, 102 and 103, so the next one starts at 104. An array is exactly this: numbers laid end to end, so any element's address is the start plus a multiple of the size.
The picture. Twelve bytes, read three ways. The block never changes: only the rule for reading it does. A transpose is a new rule over the same bytes, and so is taking every second column.
Step through. The numbers in the block never change; each step only changes how they are read.
Write a sentence or two, then compare it with a good answer in the lesson.
Purpose. Read the rule off real arrays, and learn which operations leave the data where it is.
import numpy as np a = np.arange(12, dtype=np.uint8).reshape(3, 4) print(a.shape, a.strides, a.itemsize, a.nbytes) print(a.T.shape, a.T.strides) print(a[:, ::2].shape, a[:, ::2].strides) print(np.shares_memory(a, a.T), np.shares_memory(a, a.copy()))
itemsize is the bytes in one element and nbytes is the whole block. The transpose has swapped strides, and the slice has doubled the column step. np.shares_memory answers the question the whole lesson is about: do two arrays read the same bytes? A view does. A copy, from .copy(), gets a block of its own.
Writing through a view changes the original, because there is only one block:
import numpy as np a = np.arange(6).reshape(2, 3) b = a.T b[0, 1] = 99 print(a.tolist()) c = a.T.copy() c[0, 0] = -1 print(a[0, 0])
b[0, 1] is a[1, 0], so that is the element that became 99. The copy c is separate, so writing to it leaves a alone.
import numpy as np a = np.arange(6) v = a[::2] v[0] = 50 print(a[0], np.shares_memory(a, v))
50 True
A basic slice is a view, so writing to v writes to the block that a also reads: a[0] is now 50, and the two share memory.
Code flow. The rule from the picture, written by hand. Flat numbers, a shape and strides, and the offset formula. Then NumPy reads the same element from the same block, and the two must agree.
Write offset(strides, index): the byte offset of an element, the sum of each index times its stride. Then set row1_col2 to the offset of element (1, 2) in a float32 array of shape (3, 4), whose strides are (16, 4).
def offset(strides, index): ... row1_col2 = None print(offset((16, 4), (2, 1)), row1_col2)
Write transposed(shape, strides): return the shape and strides of the transpose of a 2-D array, as a pair of tuples. No data is involved: swap the two entries of each.
def transposed(shape, strides): ... print(transposed((3, 4), (4, 1)))
Which operations copy? A short list is worth knowing. Slicing with : and steps, .T, reshape on a contiguous array, and ravel on a contiguous array are views. Indexing with a list or a boolean mask, .copy(), and flatten() make copies. The cell asks NumPy rather than trusting memory:
import numpy as np a = np.arange(12).reshape(3, 4) tests = { "a.T": a.T, "a[:, ::2]": a[:, ::2], "a.reshape(4, 3)": a.reshape(4, 3), "a[[0, 2]]": a[[0, 2]], "a[a > 5]": a[a > 5], "a.flatten()": a.flatten(), } for name, other in tests.items(): print(f"{name:16} shares memory: {np.shares_memory(a, other)}")
a[[0, 2]], indexing with a list of rows
A list of indexes picks rows that are not evenly spaced, so no single shape and strides describe them. NumPy has to build a new block. A slice, a transpose and a reshape of a contiguous array can all be described by a new shape and strides.
import numpy as np a = np.arange(12).reshape(3, 4) t = a.T print(t.shape, t.strides == tuple(reversed(a.strides)), t.flags["C_CONTIGUOUS"])
(4, 3) True False
The transpose has shape (4, 3) and the strides swapped, so the strides equal the original's reversed. It is no longer in C order, because walking along its rows now jumps through memory.
Why strides matter for speed. An array in the usual order (C order) stores each row in one stretch, so walking along a row visits neighboring bytes, and the processor's cache loves that. Walking down a column jumps a whole row each step. The cell sums the rows and the columns of a square array and draws both lines as the array grows:
from academy import measure import numpy as np sums = { "sum each row": lambda a: a.sum(axis=1), "sum each column": lambda a: a.sum(axis=0), } measure( sums, [250, 500, 1000, 2000], setup=lambda n: np.ones((n, n), dtype=np.float32), repeat=2, xlabel="n (the array is n by n)", )
Both lines climb as the array grows, because there are n² numbers to add. When n doubles, the number of elements goes up four times, so each line should rise about that much. Compare the two lines' heights on your machine: how far apart they sit depends on the processor.
4 times
n × n becomes 2n × 2n, which is four times as many numbers, so a sum over every element takes about four times as long. On log axes that is a slope of 2.
About 9.6
4,816,896 × 2 bytes is 9,633,792 bytes, about 9.6 MB: half of the 19.3 MB it takes in float32.
A batch of 32 images, 3 channels, 224 by 224 pixels, in float32. Work out its size in megabytes from the shape and the dtype alone, then check it against NumPy.
import numpy as np shape = (32, 3, 224, 224) count = 1 for size in shape: count *= size by_hand = count * 4 batch = np.zeros(shape, dtype=np.float32) print(count, by_hand, batch.nbytes, batch.strides) print(round(by_hand / 1_000_000, 1), "MB")
The number of elements is the product of the shape, 32 × 3 × 224 × 224, and each takes 4 bytes in float32. The by-hand count and NumPy's nbytes agree. Memory in the page is 32-bit Python, but the array's own size is the same arithmetic on any machine. Halving the precision to float16 would halve the bytes, and that one line is the first thing to try when a batch does not fit.
Write a sentence or two, then compare it with a good answer in the lesson.
Purpose. A PyTorch tensor is this same block, shape and strides. Its views, its errors and a very common saving bug all follow from the rule.
In PyTorch the same words appear: x.stride(), x.is_contiguous(), x.view, x.reshape. view promises not to copy, so it works only when the existing strides can describe the new shape. A transposed tensor cannot be flattened that way, and view says so. reshape copies only when it has to. .contiguous() makes the copy by hand.
import torch x = torch.arange(12.0).reshape(3, 4) t = x.t() print(x.stride(), t.stride(), x.is_contiguous(), t.is_contiguous()) try: t.view(12) except RuntimeError as error: print("view failed:", str(error)[:41]) print(t.reshape(12).shape, t.contiguous().view(12).shape)
(This page runs a teaching subset of PyTorch: same calls, CPU only. The error text is the real one.) The first error message every PyTorch user meets is this one. The fix is not to fight it: call reshape, or .contiguous() first.
x.numpy() on a CPU tensor is a view too. The tensor and the NumPy array read one block, so a write through either shows in both:
import torch x = torch.arange(6.0) n = x.numpy() n[0] = 99.0 print(x[0].item(), n[0])
On the job: when a training loss looks fine but the "best" checkpoint behaves like the last one, check whether the saved state is a copy. When a batch will not fit, compute its size from the shape and dtype before blaming the GPU. And when an in-place operation fails or changes something it should not, ask whether two tensors share a block.
Write a sentence or two, then compare it with a good answer in the lesson.
Why is a.T instant even for a huge array?
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Go from your first line of code to working as a Python developer: projects, testing, data, databases, web apps and APIs, all in your browser.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write