{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<small><small><i>\n",
"All the IPython Notebooks in **Python Advanced Topics** lecture series by Dr. Milaan Parmar are available @ **[GitHub](https://github.com/milaan9/07_Python_Advanced_Topics)**\n",
"</i></small></small>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Python Iterators\n",
"\n",
"Iterators are objects that can be iterated upon. In this tutorial, you will learn how iterator works and how you can build your own iterator using **`__iter__`** and **`__next__`** methods."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Iterators in Python\n",
"\n",
"Iterators are everywhere in Python. They are elegantly implemented within `for` loops, comprehensions, generators etc. but are hidden in plain sight.\n",
"\n",
"Iterator in Python is simply an **[object](https://github.com/milaan9/06_Python_Object_Class/blob/main/002_Python_Classes_and_Objects.ipynb)** that can be iterated upon. An object which will return data, one element at a time.\n",
"\n",
"Technically speaking, a Python **iterator object** must implement two special methods, **`__iter__()`** and **`__next__()`**, collectively called the **iterator protocol**.\n",
"\n",
"An object is called **iterable** if we can get an iterator from it. Most built-in containers in Python like: **[string](https://github.com/milaan9/02_Python_Datatypes/blob/main/002_Python_String.ipynb)**, **[list](https://github.com/milaan9/02_Python_Datatypes/blob/main/003_Python_List.ipynb)**, **[tuple](https://github.com/milaan9/02_Python_Datatypes/blob/main/004_Python_Tuple.ipynb)** etc. are iterables.\n",
"\n",
"The **`iter()`** function (which in turn calls the **`__iter__()`** method) returns an iterator from them."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Iterating Through an Iterator\n",
"\n",
"We use the **`next()`** function to manually iterate through all the items of an iterator. When we reach the end and there is no more data to be returned, it will raise the **`StopIteration`** Exception. Following is an example."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"ExecuteTime": {
"end_time": "2021-06-22T05:59:09.342273Z",
"start_time": "2021-06-22T05:59:08.896940Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"6\n",
"9\n",
"0\n",
"3\n"
]
},
{
"ename": "StopIteration",
"evalue": "",
"output_type": "error",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[1;31mStopIteration\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-1-4b26ab49e29c>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m 18\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 19\u001b[0m \u001b[1;31m# This will raise error, no items left\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 20\u001b[1;33m \u001b[0mnext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmy_iter\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;31mStopIteration\u001b[0m: "
]
}
],
"source": [
"# Example 1:\n",
"\n",
"# define a list\n",
"my_list = [6, 9, 0, 3] # 4 elements\n",
"\n",
"# get an iterator using iter()\n",
"my_iter = iter(my_list)\n",
"\n",
"# iterate through it using next()\n",
"\n",
"print(next(my_iter)) # Output: 6\n",
"print(next(my_iter)) # Output: 9\n",
"\n",
"# next(obj) is same as obj.__next__()\n",
"\n",
"print(my_iter.__next__()) # Output: 0\n",
"print(my_iter.__next__()) # Output: 3\n",
"\n",
"# This will raise error, no items left\n",
"next(my_iter)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A more elegant way of automatically iterating is by using the **[for loop](https://github.com/milaan9/03_Python_Flow_Control/blob/main/005_Python_for_Loop.ipynb)**. Using this, we can iterate over any object that can return an iterator, for example list, string, file etc."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2021-06-22T05:59:13.861477Z",
"start_time": "2021-06-22T05:59:13.849761Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"6\n",
"9\n",
"0\n",
"3\n"
]
}
],
"source": [
"for element in my_list: # create a function\n",
" print(element)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Working of `for` loop for Iterators\n",
"\n",
"As we see in the above example, the **[for loop](https://github.com/milaan9/03_Python_Flow_Control/blob/main/005_Python_for_Loop.ipynb)** was able to iterate automatically through the list.\n",
"\n",
"In fact the **`for`** loop can iterate over any iterable. Let's take a closer look at how the **`for`** is actually implemented in Python.\n",
"\n",
"```python\n",
">>> for element in iterable:\n",
">>> # do something with element\n",
"```\n",
"\n",
"Is actually implemented as.\n",
"\n",
"```python\n",
"\n",
">>> # create an iterator object from that iterable\n",
">>> iter_obj = iter(iterable)\n",
"\n",
">>> # infinite loop\n",
">>> while True:\n",
">>> try:\n",
">>> # get the next item\n",
">>> element = next(iter_obj)\n",
">>> # do something with element\n",
">>> except StopIteration:\n",
">>> # if StopIteration is raised, break from loop\n",
">>> break\n",
"```\n",
"\n",
"So internally, the **`for`** loop creates an iterator object, **`iter_obj`** by calling **`iter()`** on the iterable.\n",
"\n",
"Ironically, this **`for`** loop is actually an infinite **[while loop](https://github.com/milaan9/03_Python_Flow_Control/blob/main/006_Python_while_Loop.ipynb)**.\n",
"\n",
"Inside the loop, it calls **`next()`** to get the next element and executes the body of the **`for`** loop with this value. After all the items exhaust, **`StopIteration`** is raised which is internally caught and the loop ends. Note that any other kind of exception will pass through."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Building Custom Iterators\n",
"\n",
"Building an iterator from scratch is easy in Python. We just have to implement the **`__iter__()`** and the **`__next__()`** methods.\n",
"\n",
"The **`__iter__()`** method returns the iterator object itself. If required, some initialization can be performed.\n",
"\n",
"The **`__next__()`** method must return the next item in the sequence. On reaching the end, and in subsequent calls, it must raise **`StopIteration`**.\n",
"\n",
"Here, we show an example that will give us the next power of 2 in each iteration. Power exponent starts from zero up to a user set number.\n",
"\n",
"If you do not have any idea about object-oriented programming, visit **[Python Object-Oriented Programming](https://github.com/milaan9/06_Python_Object_Class/blob/main/001_Python_OOPs_Concepts.ipynb)**.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"ExecuteTime": {
"end_time": "2021-06-22T05:59:16.468016Z",
"start_time": "2021-06-22T05:59:16.447511Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1\n",
"2\n",
"4\n",
"8\n",
"16\n"
]
}
],
"source": [
"class PowTwo:\n",
" \"\"\"Class to implement an iterator\n",
" of powers of two\"\"\"\n",
"\n",
" def __init__(self, max=0):\n",
" self.max = max\n",
"\n",
" def __iter__(self):\n",
" self.n = 0\n",
" return self\n",
"\n",
" def __next__(self):\n",
" if self.n <= self.max:\n",
" result = 2 ** self.n\n",
" self.n += 1\n",
" return result\n",
" else:\n",
" raise StopIteration\n",
"\n",
"\n",
"# create an object\n",
"numbers = PowTwo(4)\n",
"\n",
"# create an iterable from the object\n",
"i = iter(numbers)\n",
"\n",
"# Using next to get to the next iterator element\n",
"print(next(i))\n",
"print(next(i))\n",
"print(next(i))\n",
"print(next(i))\n",
"print(next(i))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also use a **`for`** loop to iterate over our iterator class."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"ExecuteTime": {
"end_time": "2021-06-22T05:59:18.986663Z",
"start_time": "2021-06-22T05:59:18.968112Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1\n",
"2\n",
"4\n",
"8\n",
"16\n",
"32\n"
]
}
],
"source": [
"for i in PowTwo(5): # calling the class\n",
" print(i)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Python Infinite Iterators\n",
"\n",
"It is not necessary that the item in an iterator object has to be exhausted. There can be infinite iterators (which never ends). We must be careful when handling such iterators.\n",
"\n",
"Here is a simple example to demonstrate infinite iterators.\n",
"\n",
"The **[built-in function](https://github.com/milaan9/04_Python_Functions/tree/main/002_Python_Functions_Built_in)** **[iter()](https://github.com/milaan9/04_Python_Functions/blob/main/002_Python_Functions_Built_in/037_Python_iter%28%29.ipynb)** function can be called with two arguments where the first argument must be a callable object (function) and second is the sentinel. The iterator calls this function until the returned value is equal to the sentinel.\n",
"\n",
"```python\n",
">>> int()\n",
"0\n",
"\n",
">>> inf = iter(int,1)\n",
">>> next(inf)\n",
"0\n",
">>> next(inf)\n",
"0\n",
"```\n",
"\n",
"We can see that the **`iter()`** function always returns **`0`**. So passing it as **`iter(int,1)`** will return an iterator that calls **`iter()`** until the returned value equals 1. This never happens and we get an infinite iterator.\n",
"\n",
"We can also build our own infinite iterators. The following iterator will, theoretically, return all the odd numbers.\n",
"\n",
"```python\n",
">>> class InfIter:\n",
">>> \"\"\"Infinite iterator to return all\n",
">>> odd numbers\"\"\"\n",
"\n",
">>> def __iter__(self):\n",
">>> self.num = 1\n",
">>> return self\n",
"\n",
">>> def __next__(self):\n",
">>> num = self.num\n",
">>> self.num += 2\n",
">>> return num\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A sample run would be as follows.\n",
"\n",
"```python\n",
">>> a = iter(InfIter())\n",
">>> next(a)\n",
"1\n",
">>> next(a)\n",
"3\n",
">>> next(a)\n",
"5\n",
">>> next(a)\n",
"7\n",
"```\n",
"\n",
"And so on...\n",
"\n",
"Be careful to include a terminating condition, when iterating over these types of infinite iterators.\n",
"\n",
"The advantage of using iterators is that they save resources. Like shown above, we could get all the odd numbers without storing the entire number system in memory. We can have infinite items (theoretically) in finite memory.\n",
"\n",
"There's an easier way to create iterators in Python. To learn more visit: **[Python generators using yield](https://github.com/milaan9/07_Python_Advanced_Topics/blob/main/002_Python_Generators.ipynb)**."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"hide_input": false,
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.8"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
},
"varInspector": {
"cols": {
"lenName": 16,
"lenType": 16,
"lenVar": 40
},
"kernels_config": {
"python": {
"delete_cmd_postfix": "",
"delete_cmd_prefix": "del ",
"library": "var_list.py",
"varRefreshCmd": "print(var_dic_list())"
},
"r": {
"delete_cmd_postfix": ") ",
"delete_cmd_prefix": "rm(",
"library": "var_list.r",
"varRefreshCmd": "cat(var_dic_list()) "
}
},
"types_to_exclude": [
"module",
"function",
"builtin_function_or_method",
"instance",
"_Feature"
],
"window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 4
}