A generator produces a sequence without building it. That one sentence is the whole idea, and it is
what lets you process a four-gigabyte file in a few megabytes of memory. This post explains what the
for loop is really doing, then how yield lets you take part in it.
What a for loop actually does
Looping is not indexing. for asks the object for an iterator, then asks
that iterator for the next item until it says stop:
accounts = ["Checking", "Savings"]
it = iter(accounts)
print(next(it)) # Output: Checking
print(next(it)) # Output: Savings
Every for loop you have written is that, with the bookkeeping hidden. When the items
run out, next() raises StopIteration and the loop ends quietly:
it = iter(["only one"])
next(it)
next(it)
This is why any object that can answer next() works in a for loop — a
list, a file, a dict, a database cursor. They do not share a base class; they share a protocol.
yield
Put yield in a function and it stops being a function that returns a value and becomes
one that produces a series of them:
def transactions():
print("starting")
yield 100
print("between")
yield -25
print("finishing")
for amount in transactions():
print(amount)
# Output: starting
# Output: 100
# Output: between
# Output: -25
# Output: finishing
Read that output carefully, because it is the whole mechanism. starting did not print
when the generator was created — it printed when the loop asked for the first item. Each
yield hands a value out and freezes the function where it stands, local
variables intact. The next request resumes it on the following line.
A normal function runs to completion and returns once. A generator runs in slices, on demand.
Why that matters
def squares_list(n):
return [i * i for i in range(n)] # builds all n in memory
def squares_gen(n):
for i in range(n):
yield i * i # builds one at a time
import sys
print(sys.getsizeof(squares_list(100_000)) > 400_000) # Output: True
print(sys.getsizeof(squares_gen(100_000)) < 250) # Output: True
The list needs hundreds of kilobytes. The generator is a couple of hundred bytes no matter what
n is, because it stores a position rather than the results.
The other half is that a generator can start producing immediately. A function returning a list of ten thousand database rows gives you nothing until every row has been fetched; a generator gives you the first one straight away.
Generator expressions
Swap a comprehension's brackets for parentheses and you get a generator instead of a list:
amounts = [100, -25, 250, -40]
print([n * 2 for n in amounts]) # Output: [200, -50, 500, -80]
print(sum(n for n in amounts if n > 0)) # Output: 350
print(any(n < 0 for n in amounts)) # Output: True
print(max(abs(n) for n in amounts)) # Output: 250
When a generator expression is the only argument to a function, its parentheses double as the
call's, so you write sum(n for n in ...) rather than sum((n for n in ...)).
The rule: if you are going to consume it once and immediately — sum, any,
max, join, a for loop — use a generator expression. Use a list
comprehension when you need to keep the result, index into it, or look at it more than once.
A generator is used up
This is the one that catches people. Once consumed, it is empty:
numbers = (n for n in [1, 2, 3])
print(sum(numbers)) # Output: 6
print(sum(numbers)) # Output: 0
print(list(numbers)) # Output: []
The second sum is not a bug in Python. The generator reached the end and there is no
rewind — the values were never stored, so there is nothing to go back to. If you need the data twice,
call list() on it and keep that.
The related trap is len(), which does not work on a generator at all. Knowing the
length would mean running it to the end, which is exactly what you were avoiding.
Reading a large file
This is the pattern you will actually use. A file object is already an iterator over its lines:
from pathlib import Path
Path("transactions.csv").write_text(
"id,amount\n1,100.00\n2,-25.00\n3,250.00\n"
)
def amounts(path):
with open(path) as handle:
next(handle) # skip the header
for line in handle:
yield float(line.split(",")[1])
print(sum(a for a in amounts("transactions.csv") if a > 0)) # Output: 350.0
That totals the file without ever holding more than one line in memory, and it works identically on
three rows or thirty million. Replace yield with building a list and the second case stops
working.
itertools, before you write it yourself
The standard library ships the generators people most often reinvent:
from itertools import islice, chain, count
print(list(islice(count(1), 5))) # Output: [1, 2, 3, 4, 5]
print(list(chain([1, 2], [3, 4]))) # Output: [1, 2, 3, 4]
def all_transactions():
yield from range(1, 1000000)
print(list(islice(all_transactions(), 3))) # Output: [1, 2, 3]
count() is an infinite generator — perfectly safe, because islice only
ever asks it for five. That combination is worth internalising: an unbounded source plus a limit,
where a list would have to be finite before you could take anything from it.
islice is also how you peek at a generator without consuming all of it, and
chain joins several iterables into one stream without concatenating them in memory.
Chaining them
Generators compose. Each stage pulls from the one before it, one item at a time, so a pipeline of five costs no more memory than one:
def read():
yield from ["100.00", "-25.00", "bad", "250.00"]
def to_float(rows):
for row in rows:
try:
yield float(row)
except ValueError:
continue
def credits(numbers):
for n in numbers:
if n > 0:
yield n
print(sum(credits(to_float(read())))) # Output: 350.0
yield from delegates to another iterable — it is shorthand for looping and yielding
each item. Nothing runs until sum starts pulling, and then one value travels the whole
chain before the next one starts.
Next: Decorators.