Iterator protocol in Python - Deep Dive

Overview - Iterator protocol

What is it?

The iterator protocol is a way Python lets you loop over items one by one without loading them all at once. It uses two special methods: __iter__() to get the iterator object and __next__() to get each item. When there are no more items, __next__() raises a StopIteration exception to end the loop. This protocol makes it easy to work with sequences, files, or any collection in a simple, consistent way.

Why it matters

Without the iterator protocol, Python would have to load entire collections into memory before looping, which can be slow or impossible for large data. This protocol allows Python to handle data efficiently, even if it’s huge or infinite, by fetching one item at a time. It also makes custom objects behave like built-in collections, so you can use loops and other tools naturally.

Where it fits

Before learning the iterator protocol, you should understand basic Python loops and functions. After this, you can explore generators, comprehensions, and advanced data processing techniques that rely on iterators for efficiency.

Mental Model

Core Idea

An iterator is like a bookmark that remembers your place as you move through a collection one item at a time.

Think of it like...

Imagine reading a book with a bookmark. The bookmark shows where you stopped reading, so next time you open the book, you continue from that exact page without starting over.

Collection (list, file, etc.)
  │
  ▼
Iterator object (has __next__ method)
  │
  ▼
Calls __next__() repeatedly → returns next item
  │
  ▼
Raises StopIteration when no items left

Build-Up - 7 Steps

1

Foundation

Understanding basic iteration in Python

Concept: Learn how Python loops over collections using for loops.

In Python, a for loop goes through each item in a collection like a list or string. For example: numbers = [1, 2, 3] for num in numbers: print(num) This prints each number one by one.

Result

1 2 3

Knowing how for loops work with collections sets the stage for understanding how Python gets each item behind the scenes.

2

Foundation

What is an iterator object?

Concept: Introduce the iterator object that controls the sequence of items.

An iterator is an object that remembers where you are in a collection. You get it by calling iter() on a collection: numbers = [1, 2, 3] it = iter(numbers) print(next(it)) # prints 1 print(next(it)) # prints 2 print(next(it)) # prints 3 Calling next(it) gives the next item each time.

Result

1 2 3

Understanding that iter() returns an object that controls the sequence helps you see how loops fetch items one by one.

3

Intermediate

How __iter__ and __next__ work

🤔Before reading on: do you think __iter__ returns the collection itself or a new object? Commit to your answer.

Concept: Learn the two special methods that define the iterator protocol.

The iterator protocol requires two methods: - __iter__(): returns the iterator object itself. This allows the object to be used in a for loop. - __next__(): returns the next item or raises StopIteration when done. For example, a list has __iter__ that returns an iterator, and that iterator has __next__ to get items.

Result

Calling iter(collection) calls collection.__iter__(), which returns an iterator with __next__ method.

Knowing these two methods explains how Python loops work under the hood and how to make your own iterable objects.

4

Intermediate

Creating a custom iterator class

🤔Before reading on: do you think your custom iterator needs to store state? Commit to your answer.

Concept: Build a simple class that follows the iterator protocol by implementing __iter__ and __next__.

Here is a class that counts from 1 to 3: class Counter: def __init__(self): self.current = 1 self.end = 3 def __iter__(self): return self def __next__(self): if self.current <= self.end: val = self.current self.current += 1 return val else: raise StopIteration for num in Counter(): print(num)

Result

1 2 3

Understanding that the iterator object must keep track of its position (state) is key to building custom iterators.

5

Intermediate

Using StopIteration to end loops

Concept: Learn how raising StopIteration signals the end of iteration.

When __next__ runs out of items, it raises StopIteration. Python’s for loop catches this exception and stops looping. Example: class Simple: def __init__(self): self.count = 0 def __iter__(self): return self def __next__(self): if self.count < 2: self.count += 1 return self.count else: raise StopIteration for i in Simple(): print(i)

Result

1 2

Knowing that StopIteration controls loop termination helps you understand how iteration ends cleanly.

6

Advanced

Iterators vs iterables distinction

🤔Before reading on: Is every iterable also an iterator? Commit to your answer.

Concept: Clarify the difference between objects you can loop over and the iterator objects themselves.

An iterable is any object you can loop over (like a list). It has an __iter__ method that returns an iterator. An iterator is the object returned by __iter__, which has the __next__ method. Example: lst = [1, 2] print(hasattr(lst, '__iter__')) # True it = iter(lst) print(hasattr(it, '__next__')) # True The list is iterable but not an iterator; the iterator is separate.

Result

True True

Understanding this difference prevents confusion when implementing or using iteration in Python.

7

Expert

Iterator protocol in generator functions

🤔Before reading on: Do generator objects implement __iter__ and __next__? Commit to your answer.

Concept: Explore how Python’s generator functions automatically implement the iterator protocol.

Generators are special functions that yield values one at a time. When called, they return a generator object that implements __iter__ and __next__. Example: def gen(): yield 1 yield 2 g = gen() print(next(g)) # 1 print(next(g)) # 2 The generator object is its own iterator, so __iter__ returns self.

Result

1 2

Knowing that generators are iterators under the hood explains their efficiency and ease of use for lazy evaluation.

Under the Hood

When Python executes a for loop, it calls the iterable’s __iter__ method to get an iterator object. Then it repeatedly calls the iterator’s __next__ method to get each item. Internally, Python uses a try-except block to catch the StopIteration exception, which signals the end of the loop. This design separates the iterable (the data source) from the iterator (the stateful object that tracks progress).

Why designed this way?

The iterator protocol was designed to unify looping over many types of data sources with a simple, consistent interface. By separating the iterable and iterator, Python allows multiple independent loops over the same data without interference. The StopIteration exception cleanly signals loop end without extra flags or checks, making code simpler and more Pythonic.

┌───────────────┐       calls       ┌───────────────┐
│   Iterable    │──────────────────▶│   Iterator    │
│  (has __iter__)│                   │ (has __next__)│
└───────────────┘                   └───────────────┘
       ▲                                  │
       │                                  │ calls
       │                                  ▼
       │                         ┌─────────────────┐
       │                         │  Returns next    │
       │                         │  item or raises  │
       │                         │  StopIteration   │
       │                         └─────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does calling iter() on an iterator return a new iterator or the same one? Commit to your answer.

Common Belief: Calling iter() on any iterable always returns a new iterator object.

Reality: Calling iter() on an iterator returns the same iterator object, not a new one.

Why it matters: Assuming a new iterator is returned can cause bugs when reusing iterators, leading to unexpected behavior or infinite loops.

Quick: Does every object with __next__ method automatically work in a for loop? Commit to your answer.

Common Belief: If an object has a __next__ method, it can be used directly in a for loop.

Reality: An object must also have an __iter__ method that returns itself to be used in a for loop.

Why it matters: Missing __iter__ causes TypeError in loops, confusing beginners who think __next__ alone is enough.

Quick: Does StopIteration mean an error you should always fix? Commit to your answer.

Common Belief: StopIteration is an error indicating something went wrong during iteration.

Reality: StopIteration is a normal signal to end iteration, not an error to fix.

Why it matters: Misunderstanding this leads to unnecessary try-except blocks and complicated code instead of clean loops.

Quick: Can you reset an iterator by calling iter() on it? Commit to your answer.

Common Belief: Calling iter() on an iterator resets it to the start of the collection.

Reality: Calling iter() on an iterator returns the same iterator at its current position; it does not reset.

Why it matters: Expecting reset behavior causes bugs when looping multiple times over the same iterator.

Expert Zone

1

Iterators can be infinite, so StopIteration may never occur, requiring careful use to avoid infinite loops.

2

Some iterators maintain internal state that can be affected by external changes to the underlying data, leading to subtle bugs.

3

The iterator protocol allows chaining and composing iterators efficiently, enabling powerful data pipelines without intermediate storage.

When NOT to use

Avoid custom iterator classes when simple generators can express the logic more clearly and concisely. For very large or infinite data, consider using itertools or built-in lazy evaluation tools instead of manual iterator management.

Production Patterns

In production, iterators are used to process streams of data like files, network packets, or database query results lazily. They enable memory-efficient pipelines, such as reading large logs line-by-line or chaining transformations without loading all data at once.

Connections

Generators

Generators implement the iterator protocol automatically.

Understanding iterators clarifies how generators produce values lazily and why they can be used directly in loops.

Streams in functional programming

Streams and iterators both represent sequences of data processed one element at a time.

Knowing iterator protocol helps grasp how streams enable lazy, composable data processing in other languages.

Bookmarking in reading

Both iterators and bookmarks remember a position to continue later.

This cross-domain idea shows how remembering state is key to managing sequences efficiently.

Common Pitfalls

#1 Trying to loop over an iterator twice expecting it to restart.

Wrong approach:

it = iter([1, 2, 3])
for x in it:
print(x)
for x in it:
print(x) # prints nothing

Correct approach:

lst = [1, 2, 3]
for x in lst:
print(x)
for x in lst:
print(x) # prints all items both times

Root cause:

Misunderstanding that iterators do not reset; only iterables can produce new iterators.

#2 Defining __next__ but missing __iter__ in a custom iterator class.

Wrong approach:

class BadIter:
def __next__(self):
return 1

for x in BadIter():
print(x) # TypeError: 'BadIter' object is not iterable

Correct approach:

class GoodIter:
def __iter__(self):
return self
def __next__(self):
return 1

for x in GoodIter():
print(x) # prints 1 infinitely

Root cause:

For loops require __iter__ method; __next__ alone is not enough.

#3 Catching StopIteration inside __next__ instead of letting it propagate.

Wrong approach:

def __next__(self):
try:
return next(self.data)
except StopIteration:
return None # breaks iteration protocol

Correct approach:

def __next__(self):
return next(self.data) # let StopIteration propagate

Root cause:

Swallowing StopIteration breaks the protocol and confuses loops that rely on it to stop.

Key Takeaways

The iterator protocol uses __iter__ and __next__ methods to allow looping over data one item at a time.

Iterables produce iterator objects, which keep track of the current position during iteration.

Raising StopIteration signals the end of data and cleanly stops loops without errors.

Custom iterators must implement both __iter__ (returning self) and __next__ to work with for loops.

Understanding iterators unlocks efficient data processing and is the foundation for generators and lazy evaluation.