Iterators, generators and lazy pipelines
Why this matters
A for loop over a list is fine for a thousand items. A log file with ten million lines, a stream of API pages, or batches of training data for a model won't fit in memory as one list. Generators produce values one at a time, only when asked, so a program can process data far bigger than memory with a few lines of ordinary-looking code. PyTorch's data loaders, file reading and many APIs work this way.
- Say what iter and next do, and why an iterator is used up
- Write a generator with yield and chain generators into a pipeline
- Plot peak memory for a list and a generator, and read the difference
- Stream a JSONL file through an IterableDataset into batches
def countdown(n): while n > 0: yield n n -= 1 print(list(countdown(3)))
The idea: One item at a time
This part rests on Comprehensions, where a comprehension built a list in one go, and on Loops. A generator is the same loop, paused after each item.
What a for loop really does
A for loop works on anything iterable. It calls iter() to get an iterator, then calls next() on it until the iterator raises StopIteration:
skus = ["TE-101", "PA-108", "FO-120"] it = iter(skus) print(next(it)) print(next(it)) print(next(it)) try: next(it) except StopIteration: print("done: that's how a for loop knows to stop") with open("stock.txt", "w") as f: f.write("TE-101 4\nPA-108 0\nFO-120 12\n") with open("stock.txt") as f: print(next(f).strip()) for line in f: print("rest:", line.strip())
A list is iterable but isn't an iterator itself: every iter(skus) call makes a fresh iterator that starts at the beginning, which is why you can loop over a list as often as you like. An iterator remembers where it is, and it's used up once it's done. A file object is its own iterator: after next(f) reads the first line, the for loop carries on from the second. That's also why a file can be read line by line without loading the whole thing.