Iterators

An iterable is any object that can be iterated through, or gone through one element at a time. One construct that we've used to iterate through an iterable is a for loop:

for elem in iterable:
    # do something

for loops work on any object that is iterable. We previously described it as working with any sequence -- all sequences are iterable, but there are other objects that are also iterable! We define an iterable as an object on which calling the built-in function iter function returns an iterator. An iterator is another type of object that allows us to iterate through an iterable by keeping track of which element is next in the sequence.

To illustrate this, consider the following block of code, which does the exact same thing as a the for statement above:

iterator = iter(iterable)
try:
    while True:
        elem = next(iterator)
        # do something
except StopIteration:
    pass

Here's a breakdown of what's happening:

• First, the built-in iter function is called on the iterable to create a corresponding iterator.
• To get the next element in the sequence, the built-in next function is called on this iterator.
• When next is called but there are no elements left in the iterator, a StopIteration error is raised. In the for loop construct, this exception is caught and execution can continue.

Calling iter on an iterable multiple times returns a new iterator each time with distinct states (otherwise, you'd never be able to iterate through a iterable more than once). You can also call iter on the iterator itself, which will just return the same iterator without changing its state. However, note that you cannot call next directly on an iterable.

Let's see the iter and next functions in action with an iterable we're already familiar with -- a list.

>>> lst = [1, 2, 3, 4]
>>> next(lst)             # Calling next on an iterable
TypeError: 'list' object is not an iterator
>>> list_iter = iter(lst) # Creates an iterator for the list
>>> list_iter
<list_iterator object ...>
>>> next(list_iter)       # Calling next on an iterator
1
>>> next(list_iter)       # Calling next on the same iterator
2
>>> next(iter(list_iter)) # Calling iter on an iterator returns itself
3
>>> list_iter2 = iter(lst)
>>> next(list_iter2)      # Second iterator has new state
1
>>> next(list_iter)       # First iterator is unaffected by second iterator
4
>>> next(list_iter)       # No elements left!
StopIteration
>>> lst                   # Original iterable is unaffected
[1, 2, 3, 4]

Since you can call iter on iterators, this tells us that that they are also iterables! Note that while all iterators are iterables, the converse is not true - that is, not all iterables are iterators. You can use iterators wherever you can use iterables, but note that since iterators keep their state, they're only good to iterate through an iterable once:

>>> list_iter = iter([4, 3, 2, 1])
>>> for e in list_iter:
...     print(e)
4
3
2
1
>>> for e in list_iter:
...     print(e)

Analogy: An iterable is like a book (one can flip through the pages) and an iterator for a book would be a bookmark (saves the position and can locate the next page). Calling iter on a book gives you a new bookmark independent of other bookmarks, but calling iter on a bookmark gives you the bookmark itself, without changing its position at all. Calling next on the bookmark moves it to the next page, but does not change the pages in the book. Calling next on the book wouldn't make sense semantically. We can also have multiple bookmarks, all independent of each other.

Iterable Uses

We know that lists are one type of built-in iterable objects. You may have also encountered the range(start, end) function, which creates an iterable of ascending integers from start (inclusive) to end (exclusive).

>>> for x in range(2, 6):
...     print(x)
...
2
3
4
5

Ranges are useful for many things, including performing some operations for a particular number of iterations or iterating through the indices of a list.

There are also some built-in functions that take in iterables and return useful results:

• map(f, iterable) - Creates iterator over f(x) for each x in iterable
• filter(f, iterable) - Creates iterator over x for each x in iterable if f(x)
• zip(iter1, iter2) - Creates iterator over co-indexed pairs (x, y) from both input iterables
• reversed(iterable) - Creates iterator over all the elements in the input iterable in reverse order
• list(iterable) - Creates a list containing all the elements in the input iterable
• tuple(iterable) - Creates a tuple containing all the elements in the input iterable
• sorted(iterable) - Creates a sorted list containing all the elements in the input iterable

Generators

We can create our own custom iterators by writing a generator function, which returns a special type of iterator called a generator. Generator functions have yield statements within the body of the function instead of return statements. Calling a generator function will return a generator object and will not execute the body of the function.

For example, let's consider the following generator function:

def countdown(n):
    print("Beginning countdown!")
    while n >= 0:
        yield n
        n -= 1
    print("Blastoff!")

Calling countdown(k) will return a generator object that counts down from k to 0. Since generators are iterators, we can call iter on the resulting object, which will simply return the same object. Note that the body is not executed at this point; nothing is printed and no numbers are output.

>>> c = countdown(5)
>>> c
<generator object countdown ...>
>>> c is iter(c)
True

So how is the counting done? Again, since generators are iterators, we call next on them to get the next element! The first time next is called, execution begins at the first line of the function body and continues until the yield statement is reached. The result of evaluating the expression in the yield statement is returned. The following interactive session continues from the one above.

>>> next(c)
Beginning countdown!
5

Unlike functions we've seen before in this course, generator functions can remember their state. On any consecutive calls to next, execution picks up from the line after the yield statement that was previously executed. Like the first call to next, execution will continue until the next yield statement is reached. Note that because of this, Beginning countdown! doesn't get printed again.

>>> next(c)
4
>>> next(c)
3

The next 3 calls to next will continue to yield consecutive descending integers until 0. On the following call, a StopIteration error will be raised because there are no more values to yield (i.e. the end of the function body was reached before hitting a yield statement).

>>> next(c)
2
>>> next(c)
1
>>> next(c)
0
>>> next(c)
Blastoff!
StopIteration

Separate calls to countdown will create distinct generator objects with their own state. Usually, generators shouldn't restart. If you'd like to reset the sequence, create another generator object by calling the generator function again.

>>> c1, c2 = countdown(5), countdown(5)
>>> c1 is c2
False
>>> next(c1)
5
>>> next(c2)
5

Here is a summary of the above:

• A generator function has a yield statement and returns a generator object.
• Calling the iter function on a generator object returns the same object without modifying its current state.
• The body of a generator function is not evaluated until next is called on a resulting generator object. Calling the next function on a generator object computes and returns the next object in its sequence. If the sequence is exhausted, StopIteration is raised.

• A generator "remembers" its state for the next next call. Therefore,

  • the first next call works like this:

    1. Enter the function and run until the line with yield.
    2. Return the value in the yield statement, but remember the state of the function for future next calls.

  • And subsequent next calls work like this:

    1. Re-enter the function, start at the line after the yield statement that was previously executed, and run until the next yield statement.
    2. Return the value in the yield statement, but remember the state of the function for future next calls.
• Calling a generator function returns a brand new generator object (like calling iter on an iterable object).
• A generator should not restart unless it's defined that way. To start over from the first element in a generator, just call the generator function again to create a new generator.

Another useful tool for generators is the yield from statement (introduced in Python 3.3). yield from will yield all values from an iterator or iterable.

>>> def gen_list(lst):
...     yield from lst
...
>>> g = gen_list([1, 2, 3, 4])
>>> next(g)
1
>>> next(g)
2
>>> next(g)
3
>>> next(g)
4
>>> next(g)
StopIteration

Object-Oriented Programming

Object-oriented programming (OOP) is a style of programming that allows you to think of code in terms of "objects." Here's an example of a Car class:

class Car(object):
    num_wheels = 4

    def __init__(self, color):
        self.wheels = Car.num_wheels
        self.color = color

    def drive(self):
        if self.wheels <= Car.num_wheels:
            return self.color + ' car cannot drive!'
        return self.color + ' car goes vroom!'

    def pop_tire(self):
        if self.wheels > 0:
            self.wheels -= 1

Here's some terminology:

• class: a blueprint for how to build a certain type of object. The Car class (shown above) describes the behavior and data that all Car objects have.

• instance: a particular occurrence of a class. In Python, we create instances of a class like this:

  >>> my_car = Car('red')

  my_car is an instance of the Car class.

• attribute or field: a variable that belongs to the class. Think of an attribute as a quality of the object: cars have wheels and color, so we have given our Car class self.wheels and self.color attributes. We can access attributes using dot notation:

  >>> my_car.color
  'red'
  >>> my_car.wheels
  4

• method: Methods are just like normal functions, except that they are tied to an instance or a class. Think of a method as a "verb" of the class: cars can drive and also pop their tires, so we have given our Car class the methods drive and pop_tire. We call methods using dot notation:

  >>> my_car = Car('red')
  >>> my_car.drive()
  'red car goes vroom!'

• constructor: As with data abstraction, constructors describe how to build an instance of the class. Most classes have a constructor. In Python, the constructor of the class defined as __init__. For example, here is the Car class's constructor:

  def __init__(self, color):
      self.wheels = Car.num_wheels
      self.color = color

  The constructor takes in one argument, color. As you can see, the constructor also creates the self.wheels and self.color attributes.

• self: in Python, self is the first parameter for many methods (in this class, we will only use methods whose first parameter is self). When a method is called, self is bound to an instance of the class. For example:

  >>> my_car = Car('red')
  >>> car.drive()

  Notice that the drive method takes in self as an argument, but it looks like we didn't pass one in! This is because the dot notation implicitly passes in car as self for us.