Index Ai Academy
Phase 7 · How computers run code

Arrays in memory

Buffers, dtypes, shapes and strides: why a transpose is free, and which operations copy.

About 40 minutes · API rules checked November 2026 · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Every tensor you have used in this course is an array in memory: one block of equal-size numbers, plus a small rule that says how to read it as rows and columns. Nothing teaches that rule, and it explains many things that otherwise look like magic: why a transpose costs nothing, why view sometimes fails and reshape does not, why changing one array changes another, and why a "best model so far" saved with a plain assignment quietly follows the model it was meant to remember. This lesson builds the rule by hand, then reads it off real NumPy and PyTorch arrays.

What you'll be able to do

  • Say how many bytes an array takes from its shape and dtype, and where element (i, j) sits from its strides
  • Tell a view from a copy, and predict which NumPy and PyTorch operations move no data
  • Explain why transposing is free but reading a transposed array in order can be slow
  • Avoid two real bugs: a view that changes data you meant to keep, and a saved model state that is not a copy
Quick check
A guess before we start. You make b = a.T, the transpose of a big array, and then change b[0, 1] = 99. What happens to a?
  1. Nothing: b is an independent copy
  2. Python raises an error
  3. a is deleted
  4. One of a's elements changes too
Show the answer

One of a's elements changes too

The transpose is a view: it reads the same numbers in the same block of memory, in a different order. So a change through b is a change in a. This lesson shows why.

The idea: one block of numbers, and a rule for reading it

This rests on How computers run code: memory is a long row of numbered bytes, and a program names a place in it by its address.

A Python list of numbers is a row of pointers. Each slot holds the address of a separate number object somewhere else. That is flexible, since the objects can be any type, and slow, since every read follows a pointer. An array is different: one block of memory that holds the numbers themselves, one after another, all the same size. Three small facts describe it:

  • the dtype, such as float32, says how many bytes each number takes (4, here) and how to read them;
  • the shape, such as (3, 4), says how to see the block as rows and columns;
  • the strides, such as (16, 4), say how many bytes to move to reach the next row, and the next column.

The block has 3 × 4 = 12 numbers, so it is 12 × 4 = 48 bytes. The element at row i, column j starts at byte i × 16 + j × 4. That one line, offset = i × stride₀ + j × stride₁, is the whole secret of an array.

Quick check
From the lesson on how computers run code: memory is a row of numbered bytes. A 4-byte number is stored at address 100. At what address can the next 4-byte number start, if the numbers sit one after another?
  1. 104
  2. 101
  3. 108
  4. 400
Show the answer

104

The first number uses addresses 100, 101, 102 and 103, so the next one starts at 104. An array is exactly this: numbers laid end to end, so any element's address is the start plus a multiple of the size.

The picture. Twelve bytes, read three ways. The block never changes: only the rule for reading it does. A transpose is a new rule over the same bytes, and so is taking every second column.

Try it now

Step through. The numbers in the block never change; each step only changes how they are read.

In your own words
In your own words: why is a transpose free, however big the array is?

Write a sentence or two, then compare it with a good answer in the lesson.

In Python: strides, views and copies

Purpose. Read the rule off real arrays, and learn which operations leave the data where it is.

import numpy as np

a = np.arange(12, dtype=np.uint8).reshape(3, 4)
print(a.shape, a.strides, a.itemsize, a.nbytes)
print(a.T.shape, a.T.strides)
print(a[:, ::2].shape, a[:, ::2].strides)
print(np.shares_memory(a, a.T), np.shares_memory(a, a.copy()))
It runs right here in your browser, with nothing to install.

itemsize is the bytes in one element and nbytes is the whole block. The transpose has swapped strides, and the slice has doubled the column step. np.shares_memory answers the question the whole lesson is about: do two arrays read the same bytes? A view does. A copy, from .copy(), gets a block of its own.

Writing through a view changes the original, because there is only one block:

import numpy as np

a = np.arange(6).reshape(2, 3)
b = a.T
b[0, 1] = 99
print(a.tolist())
c = a.T.copy()
c[0, 0] = -1
print(a[0, 0])

b[0, 1] is a[1, 0], so that is the element that became 99. The copy c is separate, so writing to it leaves a alone.

Quick check
What does this print?
import numpy as np

a = np.arange(6)
v = a[::2]
v[0] = 50
print(a[0], np.shares_memory(a, v))
  1. 0 False
  2. 50 False
  3. 0 True
  4. 50 True
Show the answer

50 True

A basic slice is a view, so writing to v writes to the block that a also reads: a[0] is now 50, and the two share memory.

Code flow. The rule from the picture, written by hand. Flat numbers, a shape and strides, and the offset formula. Then NumPy reads the same element from the same block, and the two must agree.

Drill

Where does element (i, j) live?

Write offset(strides, index): the byte offset of an element, the sum of each index times its stride. Then set row1_col2 to the offset of element (1, 2) in a float32 array of shape (3, 4), whose strides are (16, 4).

def offset(strides, index):
    ...


row1_col2 = None
print(offset((16, 4), (2, 1)), row1_col2)
In the lesson, your code is checked as you write it.
Drill

Transpose without moving anything

Write transposed(shape, strides): return the shape and strides of the transpose of a 2-D array, as a pair of tuples. No data is involved: swap the two entries of each.

def transposed(shape, strides):
    ...


print(transposed((3, 4), (4, 1)))
In the lesson, your code is checked as you write it.

Which operations copy? A short list is worth knowing. Slicing with : and steps, .T, reshape on a contiguous array, and ravel on a contiguous array are views. Indexing with a list or a boolean mask, .copy(), and flatten() make copies. The cell asks NumPy rather than trusting memory:

import numpy as np

a = np.arange(12).reshape(3, 4)
tests = {
    "a.T": a.T,
    "a[:, ::2]": a[:, ::2],
    "a.reshape(4, 3)": a.reshape(4, 3),
    "a[[0, 2]]": a[[0, 2]],
    "a[a > 5]": a[a > 5],
    "a.flatten()": a.flatten(),
}
for name, other in tests.items():
    print(f"{name:16} shares memory: {np.shares_memory(a, other)}")
Quick check
Which of these makes a copy of the data?
  1. a[:, ::2], taking every second column
  2. a.reshape(4, 3) on a contiguous array
  3. a[[0, 2]], indexing with a list of rows
  4. a.T, the transpose
Show the answer

a[[0, 2]], indexing with a list of rows

A list of indexes picks rows that are not evenly spaced, so no single shape and strides describe them. NumPy has to build a new block. A slice, a transpose and a reshape of a contiguous array can all be described by a new shape and strides.

Quick check
What does this print?
import numpy as np

a = np.arange(12).reshape(3, 4)
t = a.T
print(t.shape, t.strides == tuple(reversed(a.strides)), t.flags["C_CONTIGUOUS"])
  1. (4, 3) True False
  2. (3, 4) True True
  3. (4, 3) True True
  4. (4, 3) False True
Show the answer

(4, 3) True False

The transpose has shape (4, 3) and the strides swapped, so the strides equal the original's reversed. It is no longer in C order, because walking along its rows now jumps through memory.

Why strides matter for speed. An array in the usual order (C order) stores each row in one stretch, so walking along a row visits neighboring bytes, and the processor's cache loves that. Walking down a column jumps a whole row each step. The cell sums the rows and the columns of a square array and draws both lines as the array grows:

from academy import measure
import numpy as np

sums = {
    "sum each row": lambda a: a.sum(axis=1),
    "sum each column": lambda a: a.sum(axis=0),
}
measure(
    sums,
    [250, 500, 1000, 2000],
    setup=lambda n: np.ones((n, n), dtype=np.float32),
    repeat=2,
    xlabel="n (the array is n by n)",
)

Both lines climb as the array grows, because there are n² numbers to add. When n doubles, the number of elements goes up four times, so each line should rise about that much. Compare the two lines' heights on your machine: how far apart they sit depends on the processor.

Quick check
On the chart, n goes from 1,000 to 2,000. By how much does the number of elements in an n by n array grow?
  1. 2 times
  2. It stays the same
  3. 4 times
  4. 8 times
Show the answer

4 times

n × n becomes 2n × 2n, which is four times as many numbers, so a sum over every element takes about four times as long. On log axes that is a slope of 2.

Quick check
A float16 array takes 2 bytes per number. The batch above held 4,816,896 numbers. How many megabytes is it in float16?
  1. About 38.5
  2. About 19.3
  3. About 4.8
  4. About 9.6
Show the answer

About 9.6

4,816,896 × 2 bytes is 9,633,792 bytes, about 9.6 MB: half of the 19.3 MB it takes in float32.

Worked example

Worked example: how much memory does a batch take?

A batch of 32 images, 3 channels, 224 by 224 pixels, in float32. Work out its size in megabytes from the shape and the dtype alone, then check it against NumPy.

import numpy as np

shape = (32, 3, 224, 224)
count = 1
for size in shape:
    count *= size
by_hand = count * 4
batch = np.zeros(shape, dtype=np.float32)
print(count, by_hand, batch.nbytes, batch.strides)
print(round(by_hand / 1_000_000, 1), "MB")
In the lesson you can change this and run it.

The number of elements is the product of the shape, 32 × 3 × 224 × 224, and each takes 4 bytes in float32. The by-hand count and NumPy's nbytes agree. Memory in the page is 32-bit Python, but the array's own size is the same arithmetic on any machine. Halving the precision to float16 would halve the bytes, and that one line is the first thing to try when a batch does not fit.

In your own words
A colleague says "I only sliced the array, so my original is safe." What would you check, and how?

Write a sentence or two, then compare it with a good answer in the lesson.

In AI and ML engineering: tensors, views and a saved model that was not a copy

Purpose. A PyTorch tensor is this same block, shape and strides. Its views, its errors and a very common saving bug all follow from the rule.

In PyTorch the same words appear: x.stride(), x.is_contiguous(), x.view, x.reshape. view promises not to copy, so it works only when the existing strides can describe the new shape. A transposed tensor cannot be flattened that way, and view says so. reshape copies only when it has to. .contiguous() makes the copy by hand.

import torch

x = torch.arange(12.0).reshape(3, 4)
t = x.t()
print(x.stride(), t.stride(), x.is_contiguous(), t.is_contiguous())
try:
    t.view(12)
except RuntimeError as error:
    print("view failed:", str(error)[:41])
print(t.reshape(12).shape, t.contiguous().view(12).shape)

(This page runs a teaching subset of PyTorch: same calls, CPU only. The error text is the real one.) The first error message every PyTorch user meets is this one. The fix is not to fight it: call reshape, or .contiguous() first.

x.numpy() on a CPU tensor is a view too. The tensor and the NumPy array read one block, so a write through either shows in both:

import torch

x = torch.arange(6.0)
n = x.numpy()
n[0] = 99.0
print(x[0].item(), n[0])

On the job: when a training loss looks fine but the "best" checkpoint behaves like the last one, check whether the saved state is a copy. When a batch will not fit, compute its size from the shape and dtype before blaming the GPU. And when an in-place operation fails or changes something it should not, ask whether two tensors share a block.

In your own words
Your training loop saves best = model.state_dict() whenever validation loss improves, but loading it later gives the same model as the last epoch. What went wrong and how do you fix it?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 7 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

Why is a.T instant even for a huge array?

  1. It only swaps the shape and the strides; no data moves
  2. It runs on the GPU
  3. NumPy caches the result of the last transpose
  4. It copies in a background thread

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 7 · about 48 hours of study

Software engineering

How software is built, from bits to tested, well-designed programs 59 lessons 1 free
  1. Iterators and generators FreeThe iteration protocol, yield, lazy pipelines and itertools, in small memory.45 min
  2. Context managersSetup and cleanup that always run: with, contextlib, transactions and atomic writes.35 min
  3. Closures and decoratorsFunctions that remember, wrap and extend other functions.40 min
  4. Exceptions in depthError hierarchies, chaining, notes, exception groups and warnings.40 min
  5. Pattern matchingmatch and case: literal, sequence, mapping and class patterns with guards.35 min
  6. Text, bytes and encodingsstr and bytes, UTF-8, decoding errors, normalization and base64.35 min
  7. Binary filesBytes as numbers, endianness, struct records, file signatures and checking uploads.45 min
  8. Testing your codeAutomated tests that prove code works and keep it working.45 min
  9. Testing with pytestFixtures, parametrize, mocks, temporary files and coverage.45 min
  10. Testing numeric and random codeClose enough for floats, seeds and what they fix, properties that hold for any input, a gradient check and overfit-one-batch.40 min
  11. Debugging systematicallyFind any bug with a method instead of guesswork.35 min
  12. Readable code and refactoringWrite code people can understand, and improve it safely.35 min
  13. Code style and toolsPEP 8, formatters, linters and type checkers, and how they fit together.35 min
  14. Types and data modelingType hints, dataclasses and checking data at the edges of a program.35 min
  15. Typing in depthGenerics, protocols, TypedDict, Literal and narrowing, checked by mypy.45 min
  16. Classes in depthProperties, alternative constructors, abstract base classes and tuned dataclasses.40 min
  17. Make your classes feel built inSpecial methods for printing, equality, ordering, arithmetic and containers.45 min
  18. Software designModules, interfaces and dependencies that keep big programs changeable.45 min
  19. LoggingLoggers, levels, handlers and configuration, without leaking secrets.40 min
  20. How computers run codeBits, memory, the processor, and what really happens when Python runs.40 min
  21. Arrays in memoryBuffers, dtypes, shapes and strides: why a transpose is free, and which operations copy.40 min
  22. Algorithm complexityCount the work code does, and see why some programs stop scaling.40 min
  23. Concurrency and asyncDo many things at once: processes, threads and the event loop.45 min
  24. Performance and profilingMeasure before you optimize, then fix the part that matters.45 min
  25. Profiling numeric code and a training loopWhere a training step spends its time, Amdahl's rule by hand, and cProfile on a loop with a slow batch function.40 min
  26. Reading C, C++, CUDA and RustPointers and loops in C, references and templates in C++, one-thread-per-element CUDA kernels, and Rust's Option, Result and ownership.30 min
  27. Core data structuresArrays, linked lists, stacks, queues and the hash table behind every dict.45 min
  28. Heaps and priority queuesA list read as a tree: the best k of a million without sorting, and where top-k hides in AI work.35 min
  29. Recursion, trees and graphsFunctions that call themselves, and the shapes most real data takes.45 min
  30. Searching, sorting and dynamic programmingThe classic algorithms, and the ideas behind them.45 min
  31. Dynamic programming in depthA method in five questions: state, choices, recurrence, base case and order. Coins where greedy fails, rebuilding the solution, packing a token budget, the likeliest path behind noisy readings, and what it costs.45 min
  32. Binary search trees and traversalsInsert, search and delete in a sorted tree, walk it four ways, and see why balance and B-trees matter.45 min
  33. Patterns for coding problemsTwo pointers, counting with dictionaries, prefix sums and sliding windows: O(n) answers to classic problems.45 min
  34. Project: a spell checker, three waysA sorted list, a hash set and a trie, measured for speed and memory, with suggestions.45 min
  35. Searching for solutionsBacktracking with pruning, Dijkstra's algorithm and A*, ending with an optimal sliding-puzzle solver.45 min
  36. Project: an unbeatable game opponentGame state and rules, minimax with a cache, and an exhaustive test that proves it never loses.45 min
  37. Databases and SQLStore, query and join data with the language every database speaks.40 min
  38. SQL for analysisSubqueries, CASE, common table expressions, outer joins and window functions on real order data.45 min
  39. Designing a databaseFrom one big table to related tables: keys, constraints, many-to-many and a loader that is safe to rerun.40 min
  40. Databases with an ORMSQLAlchemy models, sessions, relationships, transactions and the N+1 problem.45 min
  41. Changing a database safelyNumbered migrations in transactions, required columns and backfills, and expand-and-contract changes.40 min
  42. Secure codingInjection, escaping, secrets and passwords: the mistakes attackers look for.40 min
  43. Projects, packaging and CILay out a real project, pin its dependencies and test it on every push.35 min
  44. Config files: YAML and TOMLWrite YAML and TOML that parse as you meant, dodge the type traps, and validate what you load.25 min
  45. Command-line programsargparse subcommands, stdin, exit codes and testable main functions.40 min
  46. Packaging your codepyproject.toml, the src layout, versions, entry points and wheels.45 min
  47. Pipes and text tools in the shellgrep, wc, sort, uniq, head and cut, chained with pipes to answer questions about logs and data.25 min
  48. JSON on the command line: curl and jqCall APIs with curl, fail on HTTP errors, and pull values out of JSON with jq paths and select.25 min
  49. Shell scripts that fail safelyTurn commands into scripts with variables, set -euo pipefail and clear exit codes, and know when to switch to Python.25 min
  50. Reading and writing XMLElementTree, attributes and text, namespaces, building XML safely, and untrusted input.35 min
  51. Project: a word-guessing gamePlan a game with a flowchart, then build it from tested functions and one input loop.40 min
  52. Project: a card game of 21Turn written rules into functions, and make randomness testable.40 min
  53. Project: a drinks kioskA program with state and commands, built with functions, then refactored into classes.45 min
  54. Project: a flashcard trainerCards from a file, review boxes, and progress saved between runs.45 min
  55. Project: a price-watch alert botAn API client, alert rules, digest emails and a state file, so reruns never repeat.45 min
  56. Project: Pack Train, an arcade gameA grid game on the turtle canvas: game state as classes, a timer loop, key handlers and a saved best score.45 min
  57. Project: a private password-breach checkHash locally, send five characters, match locally: k-anonymity, safe logging and failure handling.40 min
  58. Capstone: a command-line appDesign, build and test a complete program with a database.45 min
  59. Capstone: ship a small model libraryA typed, tested and profiled package with a train and an eval command, a YAML config, logging and a wheel.45 min
Part of a free course 127 lessons

Free Python developer course

Go from your first line of code to working as a Python developer: projects, testing, data, databases, web apps and APIs, all in your browser.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write