Index Ai Academy
Phase 7 · Software engineeringPython in depth

Iterators, generators and lazy pipelines

About 45 minLooking ahead: some prerequisites aren't done yet

Why this matters

A for loop over a list is fine for a thousand items. A log file with ten million lines, a stream of API pages, or batches of training data for a model won't fit in memory as one list. Generators produce values one at a time, only when asked, so a program can process data far bigger than memory with a few lines of ordinary-looking code. PyTorch's data loaders, file reading and many APIs work this way.

  • Say what iter and next do, and why an iterator is used up
  • Write a generator with yield and chain generators into a pipeline
  • Plot peak memory for a list and a generator, and read the difference
  • Stream a JSONL file through an IterableDataset into batches
Quick check
A generator hands out values one at a time. A guess: what does this print?
def countdown(n):
    while n > 0:
        yield n
        n -= 1

print(list(countdown(3)))

The idea: One item at a time

This part rests on Comprehensions, where a comprehension built a list in one go, and on Loops. A generator is the same loop, paused after each item.

What a for loop really does

A for loop works on anything iterable. It calls iter() to get an iterator, then calls next() on it until the iterator raises StopIteration:

skus = ["TE-101", "PA-108", "FO-120"]
it = iter(skus)
print(next(it))
print(next(it))
print(next(it))
try:
    next(it)
except StopIteration:
    print("done: that's how a for loop knows to stop")

with open("stock.txt", "w") as f:
    f.write("TE-101 4\nPA-108 0\nFO-120 12\n")
with open("stock.txt") as f:
    print(next(f).strip())
    for line in f:
        print("rest:", line.strip())

A list is iterable but isn't an iterator itself: every iter(skus) call makes a fresh iterator that starts at the beginning, which is why you can loop over a list as often as you like. An iterator remembers where it is, and it's used up once it's done. A file object is its own iterator: after next(f) reads the first line, the for loop carries on from the second. That's also why a file can be read line by line without loading the whole thing.