Iterators and generators

Generator expressions, and what memory costs

A generator expression looks like a list comprehension with round brackets. It produces items lazily instead of building a list. tracemalloc measures how much memory Python allocates, so the difference can be seen:

import tracemalloc

tracemalloc.start()
squares = [n * n for n in range(1_000_000)]
total = sum(squares)
_, peak_list = tracemalloc.get_traced_memory()
del squares
tracemalloc.reset_peak()

total2 = sum(n * n for n in range(1_000_000))
_, peak_gen = tracemalloc.get_traced_memory()
tracemalloc.stop()

print(total == total2)
print(f"list: {peak_list / 1e6:.1f} MB, generator: {peak_gen / 1e6:.3f} MB")

The list holds a million numbers at once; the generator holds one at a time. When a function like sum, max, any or "".join is the only argument's consumer, you can even drop the extra brackets: sum(n * n for n in range(10)).

The trade-off: a generator can be read only once, and has no len() or [i]. If you need the items several times, or need to index them, make a list.

Pipelines

Generators chain together. Each stage takes an iterable and yields transformed items, so data flows through the whole pipeline one item at a time:

def read_lines(lines):
    for line in lines:
        yield line.rstrip("\n")

def parse(lines):
    for line in lines:
        if line and not line.startswith("#"):
            date, sku, qty = line.split(",")
            yield {"date": date, "sku": sku, "qty": int(qty)}

def only(prefix, rows):
    yield from (r for r in rows if r["sku"].startswith(prefix))

raw = ["# date,sku,qty\n", "2025-06-01,TE-101,1\n", "2025-06-01,PA-108,2\n", "\n", "2025-06-02,TE-104,1\n"]
tents = only("TE", parse(read_lines(raw)))
for row in tents:
    print(row)

yield from hands out every item of another iterable, one by one. Swap raw for an open file and the same pipeline handles a 20 GB log in a few kilobytes of memory. Each stage stays small and testable on a short list, which is how data-loading code for machine learning is usually built.

def batches(items, size):
    pass


print(list(batches(range(7), 3)))

itertools: the standard pipeline parts

The itertools module has the stages you'd otherwise write by hand:

from itertools import accumulate, chain, groupby, islice, count

print(list(islice(count(1000), 3)))
print(list(chain(["TE-101"], ("PA-108", "FO-120"))))
print(list(accumulate([120, 45, 300])))

rows = [("2025-06-01", 3), ("2025-06-01", 1), ("2025-06-02", 5), ("2025-06-02", 2), ("2025-06-03", 4)]
for day, group in groupby(rows, key=lambda r: r[0]):
    print(day, sum(qty for _, qty in group))

• islice takes a slice of any iterator, even an endless one like count().
• chain joins iterables end to end.
• accumulate gives running totals.
• groupby groups neighboring items with the same key. Unlike pandas' groupby, it only works on input that's already sorted by that key; an unsorted day would start a new group. Logs written in time order are sorted already, which is why groupby suits them.

Where laziness pays, and where it doesn't

Lazy iteration is everywhere once you look:

• Files are read a line at a time with for line in f. Never call f.readlines() on a file that might be big.
• APIs with pages: a client generator yields items across pages, so callers write one simple loop and only the pages they actually reach get fetched.
• Databases: cursors hand back rows in batches, and ORMs can stream results instead of loading a whole table.
• Machine learning: training code loops over a data loader that yields one mini-batch at a time, because a dataset of images or text rarely fits in memory.
• Built-ins: range, zip, map, enumerate and dict.items() are all lazy already.

Laziness has costs too. A generator can be used only once, so if two parts of a program need the same data, give them a list or create the generator twice. Errors appear late, when the item that triggers them is reached, not when the generator is created. And a pipeline is harder to inspect in the middle: when debugging, list(islice(stage, 5)) shows the first five items any stage produces. Use generators when the data is big, endless or expensive to fetch, and plain lists when it's small and used more than once. Most real programs mix both: a lazy pipeline that reads and filters, ending in a small list of results.

import random
import tracemalloc
from itertools import groupby

def order_log(days, per_day, seed=1):
    rng = random.Random(seed)
    for d in range(days):
        date = f"2025-01-{d % 28 + 1:02d}" if d < 28 else f"2025-02-{d - 27:02d}"
        for _ in range(per_day):
            yield f"{date},{rng.choice(['TE-101', 'PA-108', 'FO-120'])},{rng.randint(1, 3)},{rng.choice([20, 55, 200])}"

def parse(lines):
    for line in lines:
        date, sku, qty, price = line.split(",")
        yield date, int(qty) * int(price)

def daily_totals(rows):
    for date, group in groupby(rows, key=lambda r: r[0]):
        yield date, sum(amount for _, amount in group)

tracemalloc.start()
totals = list(daily_totals(parse(order_log(days=40, per_day=5000))))
_, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
print(len(totals), "days; first:", totals[0], "last:", totals[-1])
print(f"peak memory: {peak / 1e6:.2f} MB for 200,000 lines")