{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Overview of Python\n",
    "\n",
    "This is a sweeping overview of Python. This is best taken as a supplement to your study, or a reference guide.\n",
    "\n",
    "## Objectives:\n",
    "\n",
    "* Birds-eye overview of Python syntax\n",
    "* History of Python (2 vs. 3)\n",
    "* Packages for Python\n",
    "* Tools for Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Disclamer\n",
    "\n",
    "We are focusing on Python 3 for most of the course, but for reference, I'll note when features became available for this set of slides only. Most things are applicable to both versions. Also, if using Python 2, include the following line:\n",
    "\n",
    "```python\n",
    "from __future__ import print_function, division, absolute_import\n",
    "```\n",
    "\n",
    "> Python 3 came out 10 years ago. Python 2 [is dying in 2020](https://pythonclock.org). Packages are [dropping support](http://python3statement.org). Please convince everyone you work with to go to Python 3 if they can!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Basics of Python\n",
    "\n",
    "* The ultimate source: [Python docs](https://docs.python.org/3.6/index.html)\n",
    "* Variables are dynamically typed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "my_int = 3  # Python 2 has two integers, normal and long\n",
    "my_float = 3.0\n",
    "my_str = \"hi\"  # Strings come in different flavors, like r\"raw\".\n",
    "my_complex = 1j\n",
    "my_bool = True\n",
    "# None is it's own type. Odd.\n",
    "\n",
    "my_list = [0, 1, 2, 3]  # Can be changed later\n",
    "my_tuple = (0, \"anything\")  # Can't be changed later\n",
    "my_dict = {\"one\": 1, \"two\": 2}  # Order preserved in Python 3.6+\n",
    "my_set = {1, 2, 3}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Special values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "special_values = (None, True, False, Ellipsis)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Last uncaptured value: `_`\n",
    "* Indexing arrays: `[value]`\n",
    "    * Multi-dimensional arrays supported by language, but not in standard library\n",
    "* Slicing arrays: `[first:last:step]`\n",
    "    * Can omit `:step` or either/both values\n",
    "    * Does not include final value\n",
    "    * The length of the result is `last - first`\n",
    "    * `...` means \"all the rest\"\n",
    "\n",
    "See [An informal introduction to Python](https://docs.python.org/3.6/tutorial/introduction.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Overview of syntax:\n",
    "* Indented blocks with `:`\n",
    "* Classes and functions create scope\n",
    "* Function parameters can be optional or keyword only\n",
    "* Iteration built into the language\n",
    "\n",
    "```python\n",
    "for x in iterable: ...\n",
    "listing = [x for x in iterable if True]\n",
    "listing = list(iterable)\n",
    "```\n",
    "\n",
    "Common stumbling blocks:\n",
    "* Mutable vs Immutable (only some built-ins immutable)\n",
    "* Assignment *replaces* the variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "x = [1, 2]\n",
    "y = x\n",
    "x[:] = [3, 4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is printed with `print(x,y)` ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = [5, 6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Keywords\n",
    "\n",
    "[Exhaustive and complete](https://docs.python.org/3.6/reference/lexical_analysis.html#identifiers) list:\n",
    "\n",
    "```python\n",
    "False      class      finally    is         return\n",
    "None       continue   for        lambda     try\n",
    "True       def        from       nonlocal   while\n",
    "and        del        global     not        with\n",
    "as         elif       if         or         yield\n",
    "assert     else       import     pass       async\n",
    "break      except     in         raise      await\n",
    "```\n",
    "\n",
    "Python 2: add `print` and `exec`, remove `nonlocal`\n",
    "Python before 3.7: remove `async` and `await`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Built in functions\n",
    "\n",
    "[Every function](https://docs.python.org/3.7/library/functions.html) Python 3.7 offers:\n",
    "\n",
    "|       ...       |      ...      |     ...        |      ...   |    ...      |\n",
    "|-----------------|---------------|----------------|------------|-------------|\n",
    "| `abs()`         | `dict()`      | `help()`       | `min()`    | `setattr()` |\n",
    "| `all()`         | `dir()`       | `hex()`        | `next()`   | `slice()`   |\n",
    "| `any()`         | `divmod()`    | `id()`         | `object()` | `sorted()`  |\n",
    "| `ascii()`       | `enumerate()` | `input()`    | `oct()` | `staticmethod()` |\n",
    "| `bin()`         | `eval()`      | `int()`        | `open()`   | `str()`     |\n",
    "| `bool()`        | `exec()`      | `isinstance()` | `ord()`    | `sum()`     |\n",
    "| `bytearray()`   | `filter()`    | `issubclass()` | `pow()`    | `super()`   |\n",
    "| `bytes()`       | `float()`     | `iter()`       | `print()`  | `tuple()`   |\n",
    "| `callable()`    | `format()`    | `len()`        | `property()` | `type()`  |\n",
    "| `chr()`         | `frozenset()` | `list()`       | `range()`  | `vars()`    |\n",
    "| `classmethod()` | `getattr()`   | `locals()`     | `repr()`   | `zip()`     |\n",
    "| `compile()`     | `globals()`   | `map()`   | `reversed()` | `__import__()` |\n",
    "| `complex()`     | `hasattr()`   | `max()`       | `round()` | `breakpoint()`|\n",
    "| `delattr()`     | `hash()`      | `memoryview()` | `set()`    |             |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "You could even get this list in Python itself!\n",
    "\n",
    "```python\n",
    "# All builtins are available in the builtins module\n",
    "import builtins\n",
    "\n",
    "# Iterate over the contents of the module\n",
    "for item in dir(builtins):\n",
    "\n",
    "    # Items are strings; only look at the ones]\n",
    "    # that start with a lower case letter\n",
    "    if item[0] != item[0].upper():\n",
    "        print(item)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Control statements\n",
    "\n",
    "The `if` statement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = True\n",
    "if x:\n",
    "    print(\"x was true\")\n",
    "else:\n",
    "    print(\"x was not true\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The for each loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = \"abcdefg\"\n",
    "for item in a:\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use `for item in range(N)` to count over a range of values, or `while CONDITION:`.  You can use `for` (and `if`) inside `[]`, `()`, or `{}` to build the data structures mentioned above inplace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bad_way_to_make_a_set_of_tens = {x for x in range(100) if x % 10 == 0}\n",
    "bad_way_to_make_a_set_of_tens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### With\n",
    "\n",
    "With lets you take code like this:\n",
    "\n",
    "```python\n",
    "f = open(filename)\n",
    "txt = f.read()\n",
    "f.close() # Don't forget me!\n",
    "```\n",
    "\n",
    "and write instead:\n",
    "\n",
    "```python\n",
    "with open(filename) as f:\n",
    "    txt = f.read()\n",
    "# File automatically closed here!\n",
    "```\n",
    "\n",
    "With simply runs code at the start and at the end of a block. It also promises the code at the end runs, even if there's an error!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## More Functions\n",
    "\n",
    "You can define your own functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def my_function(value, another=None):\n",
    "    if another is not None:\n",
    "        return another\n",
    "    else:\n",
    "        return value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "print(my_function(2), my_function(3, 4), my_function(3, another=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Special function: iterator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def my_iterator():\n",
    "    for i in range(5):\n",
    "        yield i * 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "for j in my_iterator():\n",
    "    print(j)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Decorators are objects that can modify functions or classes:\n",
    "\n",
    "```python\n",
    "@change_a_function\n",
    "def something():\n",
    "    return 1\n",
    "```\n",
    "\n",
    "is the same as\n",
    "\n",
    "```python\n",
    "def something():\n",
    "    return 1\n",
    "\n",
    "something = change_a_function(something)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Types and classes\n",
    "\n",
    "* Everything in Python is an object (an instance of a class)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "class MyFunctionLike(object):\n",
    "    def __init__(self, value=2):\n",
    "        self.value = value\n",
    "\n",
    "    def __call__(self, value):\n",
    "        return value * self.value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f2 = MyFunctionLike(2)\n",
    "f3 = MyFunctionLike(3)\n",
    "\n",
    "print(f2(2), f3(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Lots of special methods are available\n",
    "* Can customize almost all behavior\n",
    "* Lookup for normal members goes `object -> class -> parent class`\n",
    "* Lookup for special methods starts with the class\n",
    "* Member lookup can be intercepted\n",
    "\n",
    "Creating an object looks like this:\n",
    "\n",
    "1. Capture the local variables in class statement\n",
    "2. Process the variables using metaclass (normally type) (improved in Python 3)\n",
    "2. `class.__new__` `->` create object\n",
    "3. `object.__init__` : prepare object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Special decorators:\n",
    "* `@property`: Make a property function\n",
    "* `@classmethod`: Make a method runnable from the class\n",
    "* `@staticmethod`: Make a method not depend on anything in class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "class MyTest(object):\n",
    "    @property\n",
    "    def x(self):\n",
    "        return 3\n",
    "\n",
    "    @staticmethod\n",
    "    def y():\n",
    "        return 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(MyTest.y(), MyTest().x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Reflection\n",
    "\n",
    "Python knows everything about objects (basically)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def simple(x):\n",
    "    \"This is help\"\n",
    "    return x * 2\n",
    "\n",
    "\n",
    "print(dir(simple))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "help(simple)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import inspect\n",
    "\n",
    "print(inspect.getsource(simple))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Signature details are available in Python 2, but are much more elegant in Python 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "inspect.signature(simple)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "inspect.getfile(inspect)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Exceptions\n",
    "\n",
    "Errors are usually exceptions in Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": [
    "def make_an_error():\n",
    "    raise RuntimeError(\"I'm an error!\")\n",
    "\n",
    "\n",
    "# Try uncommenting this\n",
    "# make_an_error()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    make_an_error()\n",
    "except RuntimeError as e:\n",
    "    print(e, \"But I don't care, continuing anyway\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can catch errors like `ImportError` and then fix the problem, etc.\n",
    "\n",
    "The only exception: segfault (C error) (but can still get some info about it in Python 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two blocks that also show up in Python: `finally` and `else`. Finally will run regardless of if an exception was caught or not, and else will run only if an exception was not caught. You can also use `else` on a loop, where it only runs if `break` was not used to exit the loop. (While we are at it, Python provides `continue` to keep going in the loop as well)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(10):\n",
    "    print(\"Breaking out:\")\n",
    "    # break\n",
    "else:\n",
    "    print(\"Did not break out\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Anatomy of a package\n",
    "\n",
    "Folders and packages are intrinsically linked in Python.\n",
    "\n",
    "```python\n",
    "import one\n",
    "from two import three\n",
    "from four.five import six\n",
    "```\n",
    "\n",
    "* `one` could be `one.py` or `one/__init__.py`\n",
    "* `three` could be in a package or another package\n",
    "* All package **folders** should contain `__init__.py`\n",
    "* Relative imports `from .one` are available **inside** a package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Making an installable package\n",
    "\n",
    "* Running code requires that it be in `sys.path` (made from `$PATH` and `$PYTHONPATH` and the local dir)\n",
    "* Installing a package puts the code or a link to it in system or local `site-packages`\n",
    "\n",
    "To be installable:\n",
    "\n",
    "* Make a setup.py file in root folder, follow examples\n",
    "* Use `pip install -e .` (recommended) or `python setup.py develop` (not recommended)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# History of Python: 2 vs. 3\n",
    "\n",
    "* Python started in 1991 by Guido Van Rossum\n",
    "* Based on the learner's language ABC\n",
    "* Released at the right time\n",
    "* Python 2.4 had a huge user base by the time Python 3 was released\n",
    "\n",
    "So why is Python 3 better and why is Python 2 still common?\n",
    "\n",
    "TL;DR: Python 3 removes bad features and backward incompatible is slow when you have millions of lines of code.\n",
    "\n",
    "See [the great talk](http://www.asmeurer.com/python3-presentation/slides.html#1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Changes\n",
    "\n",
    "* Many of the improvements were backported to 2.6 and 2.7 (from 3.0 and 3.1)\n",
    "* Some stylistic changes (print is a function, etc)\n",
    "* Some removals not back-portable (function iterator unpacking)\n",
    "* Most changes covered by the packages: [six](https://pythonhosted.org/six/) or [future](https://pypi.python.org/pypi/future)\n",
    "\n",
    "## Iterator unpacking\n",
    "* Better support for `*list` and `**dict`, especially in 3.5\n",
    "\n",
    "```\n",
    "first, *rest, last = range(10)\n",
    "```\n",
    "\n",
    "## Keyword only arguments\n",
    "\n",
    "* No longer need to fake this with manual keyword dictionary, more natural"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Unicode\n",
    "* Biggest change for libraries\n",
    "* Strings are unicode (old strings are now bytes)\n",
    "* Unicode variable names!\n",
    "\n",
    "```python\n",
    "π=3.1415926535\n",
    "```\n",
    "\n",
    "## Chained exceptions\n",
    "\n",
    "* Errors are clearer when several occur"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Iterators\n",
    "* Iterators instead of duplicate functions\n",
    "\n",
    "```python\n",
    "for i in range(1000000000000):\n",
    "   print(\"Your computer will die in Python 2!\")\n",
    "```\n",
    "* May require wrapping some functions with `list`\n",
    "\n",
    "* You can now `yield from` another iterator (3.4)\n",
    "\n",
    "## Comparison fixed\n",
    "* You can no longer compare everything\n",
    "* `\"two\" < 1` is now an error instead of the wrong answer!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Type support\n",
    "\n",
    "* Still duck-typed (dynamically typed)\n",
    "* Added function (3.0) and variable (3.6) annotations\n",
    "* Tools like PyCharm and mypy can do static analysis\n",
    "\n",
    "## Matrix multiplication operator (3.5)\n",
    "\n",
    "* The `@` symbol was added for matrix multiplication\n",
    "\n",
    "## F-strings (3.6)\n",
    "\n",
    "* Format strings allow inline python in strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "x = 3\n",
    "print(f\"This is {x} and x+1={x+1}\")\n",
    "print(\"This is {} and x+1={}\".format(x, x + 1))\n",
    "print(\"This is %i and x+1=%i\" % (x, x + 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Easy breakpoints for debugging (3.7)\n",
    "\n",
    "A new breakpoint builtin function was added to make breakpoints easier to set, and easier to customize."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Packages for Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "As you could see, Python is simple. To do real work in Python, you need libraries.\n",
    "\n",
    "The best library for Python: the Python standard library (included).\n",
    "\n",
    "* Usually pretty good, covers many use cases\n",
    "* Does not evolve, better options usually available\n",
    "* Most Python 3 additions available as external libraries for Python 2\n",
    "\n",
    "#### Small sampling\n",
    "\n",
    "* `sys`, `os`: system tools\n",
    "* `re`: regular expressions\n",
    "* `math`, `time` and friends, `pathlib` (3 or backport), `functools`, `itertools`, `pdb`, ...\n",
    "\n",
    "See [PMOTW3 (or PMOTW2)](https://pymotw.com/3/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "There are many more common packages that can be downloaded from [pypi](https://pypi.python.org/pypi) with `pip`:\n",
    "\n",
    "#### For scientists: The big three\n",
    "* `numpy`: Numerical python (matrix and array)\n",
    "* `scipy`: More specialized additions to numpy\n",
    "* `matplotlib`: The classic plotting package\n",
    "* `pandas`: A replacement for `R` or Excel, tabular data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.arange(100)\n",
    "plt.plot(x, np.sin(x))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Other useful libraries\n",
    "* `sympy`: Symbolic mathematics as Python objects\n",
    "* `plumbum`: Set of tools for command line programming (calling other programs, command line arguments, paths, remote connections, color)\n",
    "* `numba`: Compile python into CPU, OpenMP, or CUDA code\n",
    "* `pytest`: Better testing for your code in python\n",
    "* `setuptools`: The continued development of Python packaging\n",
    "* `tensorflow`: Google's computation engine designed for ML\n",
    "* `Cython`: The C++ and Python mix for compiled python-like code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Tools for Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Jupyter\n",
    "\n",
    "The interactive language project, make up of:\n",
    "\n",
    "* Jupyter notebook: A html interface to IPython and other kernels\n",
    "* IPython: a kernel that extends Python\n",
    "* Lots of other kernels being developed (ROOTbook, etc)\n",
    "\n",
    "This is the first project to drop support for Python 2!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "IPython language extensions:\n",
    "```python\n",
    "%line_magic\n",
    "%%cell_magic\n",
    "help?\n",
    "!command\n",
    "```\n",
    "\n",
    "* History (input and output cells stored)\n",
    "* IPython has access to the original text before processed by Python\n",
    "* IPython looks for special methods to print formatted objects\n",
    "* Example: `%debug` after an exception drops you into `pdb`\n",
    "* Example: `%timeit` will run a command multiple times and get the timing info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Jupyter Notebooks\n",
    "\n",
    "* Powerful interactive environment\n",
    "* Great for remote servers or websites\n",
    "* Inline plots, widgets, etc\n",
    "* Cells in markdown, with code and $m \\alpha \\tau h$ support\n",
    "* Converts to PDF, slides, etc.\n",
    "* Great for prototyping\n",
    "* Fledgling project: [nteract](https://github.com/nteract/nteract) for desktop application\n",
    "* Future project: [Juypter Lab](https://github.com/jupyterlab/jupyterlab) for web environment with multiple components\n",
    "\n",
    "#### Useful shortcuts (more in help->keyboard shortcuts)\n",
    "\n",
    "* Shift-tab: Help\n",
    "* Shift-tab-tab: More help (etc.)\n",
    "* Tab: Auto-completion (existing objects only for IPython < 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## JupyterLab\n",
    "\n",
    "The newest and shiniest member of the IPython/Jupyter family. It's a home for notebooks and other types of windows, all in a single browser window. The notebook has some nice updates, too, like drag-n-drop cells and collapsible input. Not ready for major sites yet, but good enough for you to run it on your own."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Packaging in Python\n",
    "\n",
    "### pip\n",
    "\n",
    "The Package Installer for Python, any package should be able to be installed with this.\n",
    "\n",
    "* Easy to get (ensurepip in latest Python 2.7 or Python 3.4+, `getpip.py` before that)\n",
    "* Can download binaries now instead of just source\n",
    "* Interacts with environments nicely\n",
    "\n",
    "```bash\n",
    "pip install a_package\n",
    "pip install --user local_package\n",
    "pip install --upgrade pip\n",
    "pip install -e .\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## [Anaconda](https://www.continuum.io/anaconda-overview)\n",
    "\n",
    "* A science focused distribution of Python\n",
    "* Contains Python 2 or/and 3\n",
    "* Highly optimized Numpy and friends with help from Intel\n",
    "* Contains the Conda package manager and ~100 packages, with ~400 optional packages\n",
    "\n",
    "### Conda\n",
    "\n",
    "* Designed to manage binary packages (mostly Python related)\n",
    "* Built in support for environments\n",
    "* Can manage Python itself!\n",
    "\n",
    "```bash\n",
    "conda create -n my_python_27_env python=2.7 anaconda\n",
    "source activate my_python_27_env\n",
    "conda install paramiko\n",
    "pip install plumbum\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## IDEs for Python\n",
    "\n",
    "### PyCharm\n",
    "\n",
    "* If you need to develop a package, PyCharm is impressive\n",
    "* All the features you could ever want, plus many more\n",
    "* Free community edition is great\n",
    "\n",
    "### Spyder\n",
    "\n",
    "* An open-source Matlab like IDE\n",
    "* Used to be part of Python(x,y) for Windows, now on pip and in anaconda\n",
    "\n",
    "### IDLE\n",
    "\n",
    "* A basic editor that comes with every Python"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "compclass",
   "language": "python",
   "name": "compclass"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}