{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/dlukes/dlukes.github.io/source?filepath=content%2Fnotebooks%2Fjupyter_magic.ipynb)\n", "\n", "Python-based [Jupyter](http://jupyter.org/) notebooks mostly consist of Python code, obviously, and some Markdown text. But they also offer some very handy functions and shortcuts which are not available in Python itself, and which are really helpful for interactive work. This is my personal best of / reference.\n", "\n", "The shortcuts fall into two groups:\n", "\n", "- **magics**: special functions with special syntax whose names start with `%` (in which case they apply to the rest of the line → \"line magics\") or `%%` (in which case they apply to the entire cell → \"cell magics\")\n", "- **command line programs**: if you know how to use command line programs, you can do so directly from the notebook by prefixing the command line invocation with `!`\n", "\n", "If you want to follow along, the easiest way is to just [click this link](https://mybinder.org/v2/gh/dlukes/dlukes.github.io/source?filepath=content%2Fnotebooks%2Fjupyter_magic.ipynb) and have Binder launch a Jupyter environment with the notebook loaded for you. Or you can download this post in its original notebook format [here](https://raw.githubusercontent.com/dlukes/dlukes.github.io/source/content/notebooks/jupyter_magic.ipynb) and load it into your own Jupyter instance yourself.\n", "\n", "**NB:** Most of the following also applies to the [IPython REPL](https://ipython.org/). \n", "\n", "## Magics\n", "\n", "The syntax of magic functions is modeled after the syntax of command line programs:\n", "\n", "- to call them, just write their name and evaluate the cell, without any parentheses (unlike regular Python functions, which are called like this: `function()`)\n", "- arguments are separated just by whitespace (in Python, there are commas: `function(arg1, arg)`)\n", "- some have optional arguments (**options**) which tweak their behavior: these are formed by a hyphen and a letter, e.g. `-r`\n", "\n", "### Getting help\n", "\n", "You can read more about the magic function system by calling the `%magic` magic:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%magic" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`%quickref` brings up a useful cheat sheet of special functionality:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "%quickref" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you want more information about an object, `%pinfo` and `%pinfo2` are your friends:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "def foo():\n", " \"This foo function returns bar.\"\n", " return \"bar\"" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# shows the object's docstring\n", "%pinfo foo" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "# shows the full source code\n", "%pinfo2 foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These are so handy that they have their own special syntax: `?` and `??`, placed either before or after the object's name:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "?foo" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "foo?" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "??foo" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "foo??" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Of course, this also works with magic functions:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "?%pylab" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also open a documentation popup by pressing `Shift+Tab` with your cursor placed in or after a variable name. Repeating the command cycles through different levels of detail.\n", "\n", "### Manipulating objects\n", "\n", "The appeal of an interactive environment like Jupyter is that you can inspect any object you're working with by just evaluating it:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "foo = 1" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`%who` and `%whos` will show you all the objects you've defined:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "foo\t \n" ] } ], "source": [ "%who" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Variable Type Data/Info\n", "----------------------------\n", "foo int 1\n" ] } ], "source": [ "%whos" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sometimes though, these objects are large and you don't want to litter your notebook with tons of output you'll delete right afterwards. (Also, if you forget to delete it, your notebook might get [too large to save](https://github.com/jupyter/notebook/issues/650).) That's when you need to use the Jupyter pager, which lets you inspect an object in a separate window." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "foo = \"This is a line of text.\\n\" * 1000" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "%page foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By default, the pager pretty-prints objects using the `pprint()` function from the `pprint` module. This is handy for collections, because it nicely shows the nesting hierarchy, but not so much for strings, because special characters like newlines `\\n` are shown as escape sequences. If you want the string to look like it would if it were a text file, pass the `-r` option (\"raw\") to page through the result of calling `str()` on the object instead:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "%page -r foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you want to inspect the source code of a module, use `%pfile` on the object representing that module, or an object imported from that module:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "import os\n", "from random import choice" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "%pfile os" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "%pfile choice" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sometimes, you create an object which you know you will want to reuse in a different session or maybe in a completely different notebook. A lightweight way to achieve this is using the `%store` magic:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Stored 'foo' (str)\n" ] } ], "source": [ "%store foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can list the values stored in your database by invoking `%store` without arguments:" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Stored variables and their in-db values:\n", "foo -> 'This is a line of text.\\nThis is a line of text.\\\n" ] } ], "source": [ "%store" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To restore a variable from the database into your current Python process, use the `-r` option:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "# restores only `foo`\n", "%store -r foo" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "# restores all variables in the database\n", "%store -r" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And this is how you clear no longer needed variables from storage:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "# removes `foo`\n", "%store -d foo" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "# removes all variables\n", "%store -z" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Stored variables and their in-db values:\n" ] } ], "source": [ "%store" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Working with the file system\n", "\n", "`%ls` lists files in the directory where your notebook is stored:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3foos.py command_line_intro.ipynb pos_tagging.ipynb zipf.ipynb\r\n", "classification.ipynb foo.py regex.ipynb\r\n", "cmudict.ipynb jupyter_magic.ipynb unicode.ipynb\r\n", "collocations.ipynb libraries.ipynb xcorr_vs_conv.ipynb\r\n" ] } ], "source": [ "%ls" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you provide a path as argument, it lists that directory instead:" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0m\u001b[01;34mconf.d\u001b[0m/ koi-utf nginx.conf \u001b[01;34msites-available\u001b[0m/ uwsgi_params\r\n", "fastcgi.conf koi-win proxy_params \u001b[01;34msites-enabled\u001b[0m/ win-utf\r\n", "fastcgi_params mime.types scgi_params \u001b[01;34msnippets\u001b[0m/\r\n" ] } ], "source": [ "%ls /etc/nginx" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you provide a [glob pattern](https://en.wikipedia.org/wiki/Glob_%28programming%29), then only files that match it are listed:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/etc/nginx/fastcgi.conf /etc/nginx/nginx.conf\r\n" ] } ], "source": [ "%ls /etc/nginx/*.conf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`%ll` (\"long listing\") formats the listing as one entry per line with columns providing additional information:" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "total 12805\r\n", "drwxrwsrwt+ 3 lukes 2000147 Oct 4 18:52 \u001b[0m\u001b[30;42mexchange\u001b[0m/\r\n", "drwxrwsr-x+ 21 lukes 2041640 Oct 24 14:44 \u001b[01;34mlukes\u001b[0m/\r\n", "drwxrwsr-x+ 9 lukes 2060567 Dec 18 21:54 \u001b[01;34mmda\u001b[0m/\r\n", "drwxrwsr-x+ 14 lukes 3001102 Oct 31 17:39 \u001b[01;34mpython\u001b[0m/\r\n", "drwxrwsr-x+ 4 lukes 2004559 Feb 23 2017 \u001b[01;34mr\u001b[0m/\r\n", "drwxr-sr-x+ 4 lukes 2002420 Mar 9 2017 \u001b[01;34mtextlink\u001b[0m/\r\n" ] } ], "source": [ "%ll ~/edu/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One of those columns indicates file size, which is great, but they're in bytes, which is less great (hard to read at a glance). The `-h` option makes the file sizes print in human-readable format:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "-rw-rw-r--+ 1 lukes 1.5G Nov 1 2016 /home/lukes/edu/python/syn2015.gz\r\n", "-rw-rw-r--+ 1 lukes 112M Nov 2 2016 /home/lukes/edu/python/syn2015_sample\r\n" ] } ], "source": [ "%ll -h ~/edu/python/syn*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`%%writefile` writes the contents of a cell to a file:" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Writing foo.py\n" ] } ], "source": [ "%%writefile foo.py\n", "\n", "def foo():\n", " \"This foo function returns bar.\"\n", " return \"bar\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`%cat` prints the contents of a file into the notebook:" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r\n", "def foo():\r\n", " \"This foo function returns bar.\"\r\n", " return \"bar\"" ] } ], "source": [ "%cat foo.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`%cat` is called `%cat` because it can also con**cat**enate multiple files (or the same file, multiple times):" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r\n", "def foo():\r\n", " \"This foo function returns bar.\"\r\n", " return \"bar\"\r\n", "def foo():\r\n", " \"This foo function returns bar.\"\r\n", " return \"bar\"" ] } ], "source": [ "%cat foo.py foo.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The output of `%cat` can be saved into a file with `>` (if the file exists, it's overwritten):" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [], "source": [ "%cat foo.py foo.py >3foos.py" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r\n", "def foo():\r\n", " \"This foo function returns bar.\"\r\n", " return \"bar\"\r\n", "def foo():\r\n", " \"This foo function returns bar.\"\r\n", " return \"bar\"" ] } ], "source": [ "%cat 3foos.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Hey! Our `3foos.py` is one foo short. Let's add it by **appending** to the file with `>>`:" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [], "source": [ "%cat foo.py >>3foos.py" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r\n", "def foo():\r\n", " \"This foo function returns bar.\"\r\n", " return \"bar\"\r\n", "def foo():\r\n", " \"This foo function returns bar.\"\r\n", " return \"bar\"\r\n", "def foo():\r\n", " \"This foo function returns bar.\"\r\n", " return \"bar\"" ] } ], "source": [ "%cat 3foos.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There, much better.\n", "\n", "`%less` opens a file in the pager (with nice syntax highlighting if it's a Python source file):" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "%less foo.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`%less` is named after the program `less`, which is used to page through text files at the command line. Why is the original `less` called \"less\"? Because an earlier pager program was called `more` (as in \"show me *more* of this text file\"), and as the saying goes, \"less is more\".\n", "\n", "(Programmers are fond of dad jokes. I like how this one works on multiple levels -- the literal meaning that `less`-the-program is intended to replace `more`-the-program interacts with the figurative meaning that having less is better than having more, and both coalesce into \"use `less` because it's better than `more`\".)\n", "\n", "`%cat` and `%ls` are also named after corresponding command line programs." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Finding out more about your code\n", "\n", "When developing, code often behaves differently from what you intended when you wrote it. The following tools might help you find out why.\n", "\n", "Timing the execution of a piece of code will help you determine if it's slowing you down. The `%timeit` magic has your back, it runs your code repeatedly and thus provides more reliable estimates. It comes in both line and cell variants." ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "61.4 ms ± 7.63 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n" ] } ], "source": [ "%timeit sorted(range(1_000_000))" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "68.8 ms ± 7.57 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n" ] } ], "source": [ "%%timeit\n", "lst = list(range(1_000_000))\n", "sorted(lst)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The cell variant can include initialization code on the first line, which is run only once:" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "28.8 ms ± 4.84 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n" ] } ], "source": [ "%%timeit lst = list(range(1_000_000))\n", "sorted(lst)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you have the [memory_profiler](https://github.com/pythonprofilers/memory_profiler) library installed, you can load its magic extension and use `%memit` in the same way as `%timeit` to get a notion of how much memory your code is consuming." ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [], "source": [ "%load_ext memory_profiler" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "peak memory: 82.43 MiB, increment: 35.71 MiB\n" ] } ], "source": [ "%memit list(range(1_000_000))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Peak memory* is the highest total amount of memory the Python process used when your code ran. *Increment* is peak memory minus the amount of memory Python used *before* your code ran." ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "peak memory: 127.82 MiB, increment: 73.29 MiB\n" ] } ], "source": [ "%%memit\n", "lst = list(range(1_000_000))\n", "even = [i for i in lst if i % 2 == 0]" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "peak memory: 98.02 MiB, increment: -30.69 MiB\n" ] } ], "source": [ "%%memit lst = list(range(1_000_000))\n", "even = [i for i in lst if i % 2 == 0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you have a more involved piece of code where multiple functions are called, you may need more granular information about running times than that provided by `%timeit`. In that case, you can resort to **profiling** using the `%prun` magic. Profiling tells you how fast different parts of your code run relative to each other, in other words, where your bottlenecks are." ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [], "source": [ "import time\n", "\n", "def really_slow():\n", " time.sleep(1)\n", "\n", "def fast():\n", " pass\n", "\n", "def only_slow_because_it_calls_another_slow_function():\n", " fast()\n", " really_slow()" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " " ] } ], "source": [ "%prun only_slow_because_it_calls_another_slow_function()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The results show up in the pager, here's a copy:\n", "\n", "```\n", " 7 function calls in 1.001 seconds\n", "\n", " Ordered by: internal time\n", "\n", " ncalls tottime percall cumtime percall filename:lineno(function)\n", " 1 1.001 1.001 1.001 1.001 {built-in method time.sleep}\n", " 1 0.000 0.000 1.001 1.001 {built-in method builtins.exec}\n", " 1 0.000 0.000 1.001 1.001 :3(really_slow)\n", " 1 0.000 0.000 1.001 1.001 :9(only_slow_because_it_calls_another_slow_function)\n", " 1 0.000 0.000 1.001 1.001 :1()\n", " 1 0.000 0.000 0.000 0.000 :6(fast)\n", " 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}\n", "```\n", "\n", "`%prun` also has a cell variant:" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " " ] } ], "source": [ "%%prun\n", "really_slow()\n", "fast()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```\n", " 6 function calls in 1.001 seconds\n", "\n", " Ordered by: internal time\n", "\n", " ncalls tottime percall cumtime percall filename:lineno(function)\n", " 1 1.001 1.001 1.001 1.001 {built-in method time.sleep}\n", " 1 0.000 0.000 1.001 1.001 {built-in method builtins.exec}\n", " 1 0.000 0.000 1.001 1.001 :2()\n", " 1 0.000 0.000 1.001 1.001 :3(really_slow)\n", " 1 0.000 0.000 0.000 0.000 :6(fast)\n", " 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}\n", "```\n", "\n", "Perhaps the most useful magic for development is `%debug`, which allows you to pause the execution of a piece of code, examine the variables which are defined at that moment in time, resume execution fully or step-by-step etc. You can either pass a statement that you want to debug as argument:" ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [], "source": [ "def foo():\n", " for i in range(10):\n", " print(\"printing\", i)" ] }, { "cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "NOTE: Enter 'c' at the ipdb> prompt to continue execution.\n", "> \u001b[0;32m\u001b[0m(1)\u001b[0;36m\u001b[0;34m()\u001b[0m\n", "\n", "ipdb> help\n", "\n", "Documented commands (type help ):\n", "========================================\n", "EOF cl disable interact next psource rv unt \n", "a clear display j p q s until \n", "alias commands down jump pdef quit source up \n", "args condition enable l pdoc r step w \n", "b cont exit list pfile restart tbreak whatis\n", "break continue h ll pinfo return u where \n", "bt d help longlist pinfo2 retval unalias \n", "c debug ignore n pp run undisplay\n", "\n", "Miscellaneous help topics:\n", "==========================\n", "exec pdb\n", "\n", "ipdb> step\n", "--Call--\n", "> \u001b[0;32m\u001b[0m(1)\u001b[0;36mfoo\u001b[0;34m()\u001b[0m\n", "\u001b[0;32m----> 1 \u001b[0;31m\u001b[0;32mdef\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 2 \u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 3 \u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"printing\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\n", "ipdb> next\n", "> \u001b[0;32m\u001b[0m(2)\u001b[0;36mfoo\u001b[0;34m()\u001b[0m\n", "\u001b[0;32m 1 \u001b[0;31m\u001b[0;32mdef\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m----> 2 \u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 3 \u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"printing\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\n", "ipdb> next\n", "> \u001b[0;32m\u001b[0m(3)\u001b[0;36mfoo\u001b[0;34m()\u001b[0m\n", "\u001b[0;32m 1 \u001b[0;31m\u001b[0;32mdef\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 2 \u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m----> 3 \u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"printing\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\n", "ipdb> i\n", "0\n", "ipdb> next\n", "printing 0\n", "> \u001b[0;32m\u001b[0m(2)\u001b[0;36mfoo\u001b[0;34m()\u001b[0m\n", "\u001b[0;32m 1 \u001b[0;31m\u001b[0;32mdef\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m----> 2 \u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 3 \u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"printing\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\n", "ipdb> next\n", "> \u001b[0;32m\u001b[0m(3)\u001b[0;36mfoo\u001b[0;34m()\u001b[0m\n", "\u001b[0;32m 1 \u001b[0;31m\u001b[0;32mdef\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 2 \u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m----> 3 \u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"printing\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\n", "ipdb> i\n", "1\n", "ipdb> quit\n" ] } ], "source": [ "%debug foo()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or you can invoke plain `%debug` after an exception has been raised to jump directly to the place where the error occurred, so that you can figure out why things went wrong:" ] }, { "cell_type": "code", "execution_count": 53, "metadata": {}, "outputs": [], "source": [ "def foo():\n", " dct = dict(foo=1)\n", " return dct[\"bar\"]" ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [ { "ename": "KeyError", "evalue": "'bar'", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m\u001b[0m in \u001b[0;36mfoo\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0mdct\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfoo\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mdct\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"bar\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;31mKeyError\u001b[0m: 'bar'" ] } ], "source": [ "foo()" ] }, { "cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "> \u001b[0;32m\u001b[0m(3)\u001b[0;36mfoo\u001b[0;34m()\u001b[0m\n", "\u001b[0;32m 1 \u001b[0;31m\u001b[0;32mdef\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 2 \u001b[0;31m \u001b[0mdct\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfoo\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m----> 3 \u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mdct\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"bar\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\n", "ipdb> \"bar\" in dct\n", "False\n", "ipdb> dct.keys()\n", "dict_keys(['foo'])\n", "ipdb> quit\n" ] } ], "source": [ "%debug" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you want to pause one of your functions and explore its state at a particular point, set a *breakpoint* using the `set_trace()` function from the `IPython.core.debugger` module. The debugger will be automatically invoked when the call to `set_trace()` is reached during execution:" ] }, { "cell_type": "code", "execution_count": 56, "metadata": {}, "outputs": [], "source": [ "from IPython.core.debugger import set_trace\n", "\n", "def foo():\n", " for i in range(2):\n", " set_trace()\n", " print(\"printing\", i)" ] }, { "cell_type": "code", "execution_count": 57, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "> \u001b[0;32m\u001b[0m(6)\u001b[0;36mfoo\u001b[0;34m()\u001b[0m\n", "\u001b[0;32m 2 \u001b[0;31m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 3 \u001b[0;31m\u001b[0;32mdef\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 4 \u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 5 \u001b[0;31m \u001b[0mset_trace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m----> 6 \u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"printing\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\n", "ipdb> i\n", "0\n", "ipdb> continue\n", "printing 0\n", "> \u001b[0;32m\u001b[0m(5)\u001b[0;36mfoo\u001b[0;34m()\u001b[0m\n", "\u001b[0;32m 2 \u001b[0;31m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 3 \u001b[0;31m\u001b[0;32mdef\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 4 \u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m----> 5 \u001b[0;31m \u001b[0mset_trace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 6 \u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"printing\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\n", "ipdb> i\n", "1\n", "ipdb> continue\n", "printing 1\n" ] } ], "source": [ "foo()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The Python debugger is called `pdb` and it has some special commands of its own which allow you to step through the execution. They can be listed by typing `help` at the debugger prompt (see above), or you can have a look at the [documentation](https://docs.python.org/3/library/pdb.html#debugger-commands). The examples above also illustrate what a typical debugging session looks like (stepping through the program, inspecting variables). When you want to stop debugging, don't forget to **quit the debugger** with `quit` (or just `q`) at the debugger prompt, or else your Python process will become unresponsive." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Plotting\n", "\n", "Jupyter is tightly integrated with the [matplotlib](https://matplotlib.org/) plotting library. Plotting is enabled by running the `%matplotlib` magic with an argument specifying how the notebook should handle graphical output. `%matplotlib notebook` will generate an interactive plot which you can resize, pan, zoom and more. A word of caution though: when using this variant, once you're **done with the plot**, don't forget to **\"freeze\" it using the ⏻ symbol** in the upper right corner, or else subsequent plotting commands from different cells will all draw into this same plot." ] }, { "cell_type": "code", "execution_count": 58, "metadata": {}, "outputs": [], "source": [ "%matplotlib notebook" ] }, { "cell_type": "code", "execution_count": 59, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt" ] }, { "cell_type": "code", "execution_count": 60, "metadata": {}, "outputs": [ { "data": { "application/javascript": [ "/* Put everything inside the global mpl namespace */\n", "window.mpl = {};\n", "\n", "\n", "mpl.get_websocket_type = function() {\n", " if (typeof(WebSocket) !== 'undefined') {\n", " return WebSocket;\n", " } else if (typeof(MozWebSocket) !== 'undefined') {\n", " return MozWebSocket;\n", " } else {\n", " alert('Your browser does not have WebSocket support.' +\n", " 'Please try Chrome, Safari or Firefox ≥ 6. ' +\n", " 'Firefox 4 and 5 are also supported but you ' +\n", " 'have to enable WebSockets in about:config.');\n", " };\n", "}\n", "\n", "mpl.figure = function(figure_id, websocket, ondownload, parent_element) {\n", " this.id = figure_id;\n", "\n", " this.ws = websocket;\n", "\n", " this.supports_binary = (this.ws.binaryType != undefined);\n", "\n", " if (!this.supports_binary) {\n", " var warnings = document.getElementById(\"mpl-warnings\");\n", " if (warnings) {\n", " warnings.style.display = 'block';\n", " warnings.textContent = (\n", " \"This browser does not support binary websocket messages. \" +\n", " \"Performance may be slow.\");\n", " }\n", " }\n", "\n", " this.imageObj = new Image();\n", "\n", " this.context = undefined;\n", " this.message = undefined;\n", " this.canvas = undefined;\n", " this.rubberband_canvas = undefined;\n", " this.rubberband_context = undefined;\n", " this.format_dropdown = undefined;\n", "\n", " this.image_mode = 'full';\n", "\n", " this.root = $('
');\n", " this._root_extra_style(this.root)\n", " this.root.attr('style', 'display: inline-block');\n", "\n", " $(parent_element).append(this.root);\n", "\n", " this._init_header(this);\n", " this._init_canvas(this);\n", " this._init_toolbar(this);\n", "\n", " var fig = this;\n", "\n", " this.waiting = false;\n", "\n", " this.ws.onopen = function () {\n", " fig.send_message(\"supports_binary\", {value: fig.supports_binary});\n", " fig.send_message(\"send_image_mode\", {});\n", " if (mpl.ratio != 1) {\n", " fig.send_message(\"set_dpi_ratio\", {'dpi_ratio': mpl.ratio});\n", " }\n", " fig.send_message(\"refresh\", {});\n", " }\n", "\n", " this.imageObj.onload = function() {\n", " if (fig.image_mode == 'full') {\n", " // Full images could contain transparency (where diff images\n", " // almost always do), so we need to clear the canvas so that\n", " // there is no ghosting.\n", " fig.context.clearRect(0, 0, fig.canvas.width, fig.canvas.height);\n", " }\n", " fig.context.drawImage(fig.imageObj, 0, 0);\n", " };\n", "\n", " this.imageObj.onunload = function() {\n", " this.ws.close();\n", " }\n", "\n", " this.ws.onmessage = this._make_on_message_function(this);\n", "\n", " this.ondownload = ondownload;\n", "}\n", "\n", "mpl.figure.prototype._init_header = function() {\n", " var titlebar = $(\n", " '
');\n", " var titletext = $(\n", " '
');\n", " titlebar.append(titletext)\n", " this.root.append(titlebar);\n", " this.header = titletext[0];\n", "}\n", "\n", "\n", "\n", "mpl.figure.prototype._canvas_extra_style = function(canvas_div) {\n", "\n", "}\n", "\n", "\n", "mpl.figure.prototype._root_extra_style = function(canvas_div) {\n", "\n", "}\n", "\n", "mpl.figure.prototype._init_canvas = function() {\n", " var fig = this;\n", "\n", " var canvas_div = $('
');\n", "\n", " canvas_div.attr('style', 'position: relative; clear: both; outline: 0');\n", "\n", " function canvas_keyboard_event(event) {\n", " return fig.key_event(event, event['data']);\n", " }\n", "\n", " canvas_div.keydown('key_press', canvas_keyboard_event);\n", " canvas_div.keyup('key_release', canvas_keyboard_event);\n", " this.canvas_div = canvas_div\n", " this._canvas_extra_style(canvas_div)\n", " this.root.append(canvas_div);\n", "\n", " var canvas = $('');\n", " canvas.addClass('mpl-canvas');\n", " canvas.attr('style', \"left: 0; top: 0; z-index: 0; outline: 0\")\n", "\n", " this.canvas = canvas[0];\n", " this.context = canvas[0].getContext(\"2d\");\n", "\n", " var backingStore = this.context.backingStorePixelRatio ||\n", "\tthis.context.webkitBackingStorePixelRatio ||\n", "\tthis.context.mozBackingStorePixelRatio ||\n", "\tthis.context.msBackingStorePixelRatio ||\n", "\tthis.context.oBackingStorePixelRatio ||\n", "\tthis.context.backingStorePixelRatio || 1;\n", "\n", " mpl.ratio = (window.devicePixelRatio || 1) / backingStore;\n", "\n", " var rubberband = $('');\n", " rubberband.attr('style', \"position: absolute; left: 0; top: 0; z-index: 1;\")\n", "\n", " var pass_mouse_events = true;\n", "\n", " canvas_div.resizable({\n", " start: function(event, ui) {\n", " pass_mouse_events = false;\n", " },\n", " resize: function(event, ui) {\n", " fig.request_resize(ui.size.width, ui.size.height);\n", " },\n", " stop: function(event, ui) {\n", " pass_mouse_events = true;\n", " fig.request_resize(ui.size.width, ui.size.height);\n", " },\n", " });\n", "\n", " function mouse_event_fn(event) {\n", " if (pass_mouse_events)\n", " return fig.mouse_event(event, event['data']);\n", " }\n", "\n", " rubberband.mousedown('button_press', mouse_event_fn);\n", " rubberband.mouseup('button_release', mouse_event_fn);\n", " // Throttle sequential mouse events to 1 every 20ms.\n", " rubberband.mousemove('motion_notify', mouse_event_fn);\n", "\n", " rubberband.mouseenter('figure_enter', mouse_event_fn);\n", " rubberband.mouseleave('figure_leave', mouse_event_fn);\n", "\n", " canvas_div.on(\"wheel\", function (event) {\n", " event = event.originalEvent;\n", " event['data'] = 'scroll'\n", " if (event.deltaY < 0) {\n", " event.step = 1;\n", " } else {\n", " event.step = -1;\n", " }\n", " mouse_event_fn(event);\n", " });\n", "\n", " canvas_div.append(canvas);\n", " canvas_div.append(rubberband);\n", "\n", " this.rubberband = rubberband;\n", " this.rubberband_canvas = rubberband[0];\n", " this.rubberband_context = rubberband[0].getContext(\"2d\");\n", " this.rubberband_context.strokeStyle = \"#000000\";\n", "\n", " this._resize_canvas = function(width, height) {\n", " // Keep the size of the canvas, canvas container, and rubber band\n", " // canvas in synch.\n", " canvas_div.css('width', width)\n", " canvas_div.css('height', height)\n", "\n", " canvas.attr('width', width * mpl.ratio);\n", " canvas.attr('height', height * mpl.ratio);\n", " canvas.attr('style', 'width: ' + width + 'px; height: ' + height + 'px;');\n", "\n", " rubberband.attr('width', width);\n", " rubberband.attr('height', height);\n", " }\n", "\n", " // Set the figure to an initial 600x600px, this will subsequently be updated\n", " // upon first draw.\n", " this._resize_canvas(600, 600);\n", "\n", " // Disable right mouse context menu.\n", " $(this.rubberband_canvas).bind(\"contextmenu\",function(e){\n", " return false;\n", " });\n", "\n", " function set_focus () {\n", " canvas.focus();\n", " canvas_div.focus();\n", " }\n", "\n", " window.setTimeout(set_focus, 100);\n", "}\n", "\n", "mpl.figure.prototype._init_toolbar = function() {\n", " var fig = this;\n", "\n", " var nav_element = $('
')\n", " nav_element.attr('style', 'width: 100%');\n", " this.root.append(nav_element);\n", "\n", " // Define a callback function for later on.\n", " function toolbar_event(event) {\n", " return fig.toolbar_button_onclick(event['data']);\n", " }\n", " function toolbar_mouse_event(event) {\n", " return fig.toolbar_button_onmouseover(event['data']);\n", " }\n", "\n", " for(var toolbar_ind in mpl.toolbar_items) {\n", " var name = mpl.toolbar_items[toolbar_ind][0];\n", " var tooltip = mpl.toolbar_items[toolbar_ind][1];\n", " var image = mpl.toolbar_items[toolbar_ind][2];\n", " var method_name = mpl.toolbar_items[toolbar_ind][3];\n", "\n", " if (!name) {\n", " // put a spacer in here.\n", " continue;\n", " }\n", " var button = $('