{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# CS236781: Deep Learning\n", "\n", "# Tutorial 1: Python and tensor basics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this tutorial, we will cover:\n", "\n", "* Environment setup with `conda`\n", "* Jupyter: Using notebooks\n", "* Basic Python: Basic data types (Containers, Lists, Dictionaries, Sets, Tuples), Functions, Classes\n", "* Numpy: Arrays, Array indexing, Datatypes, Array math, Broadcasting\n", "* PyTorch tensors" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Administration and General Info" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "My info:\n", "- Aviv Rosenberg\n", "- avivr@cs.technion.ac.il\n", "- Office hour: Thursdays after the tutorial, Taub 531.\n", "- Any questions about the **homeworks** or **tutorials** -> to me." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Course:\n", "- Website is at https://vistalab-technion.github.io/cs236781/\n", "- Updates will be posted there (but emails will be sent via WebCourse)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Tutorials:\n", "- Structure is usually a short theory reminders part and then step-by-step technical implementation of a real problem.\n", "- Technical, meant to help you understand the implementation details behind deep learning.\n", "- Highly relevant for success in the homework assignments.\n", "- You can (and should!) clone the [tutorials repo](https://github.com/vistalab-technion/cs236781-tutorials), install the conda env and play with the code." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Homework:\n", "- Four HW assignments, quite heavy load. Best to tackle them after you have sufficient programming experience.\n", "- Almost entirely \"wet\" i.e. implementation of real algorithms with real data.\n", "- Can be done in pairs or alone.\n", "- Some will require use of GPUs. We will provide access provide course servers.\n", "- Read the guidelines carefully!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Python is a great general-purpose programming language on its own and with the addition of a few\n", "popular libraries such as `numpy`, `pandas`, `scikit-learn`, `matplotlib` and others it becomes an\n", "effective scientific computing environment.\n", "\n", "Recently many Deep Learning frameworks have emerged for python.\n", "Arguably the most notable ones in 2019 are **TensorFlow** (with the Keras frontend) and **PyTorch**.\n", "In this course we'll use PyTorch, which is currently [the leading DL framework](https://thegradient.pub/state-of-ml-frameworks-2019-pytorch-dominates-research-tensorflow-dominates-industry) for research.\n", "\n", "Many of you may have some experience with Python and numpy; for the rest of you, this section will serve as a quick crash course both on the Python programming language and on the use of Python for scientific computing.\n", "\n", "If you have previous knowledge in Matlab,\n", "we recommend the [numpy for Matlab users](https://docs.scipy.org/doc/numpy-1.15.0/user/numpy-for-matlab-users.html) page as a useful resource." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Credit: Some parts of the Python tutorial here were adapted from the [CS231n Python tutorial](http://cs231n.github.io/python-numpy-tutorial/) by Justin Johnson." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Environment setup" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To install and manage all the necessary packages and dependencies for the\n", "course tutorials and assignments, we use [conda](https://conda.io), a popular package-manager for python.\n", "\n", "- The tutorial notebooks come with an `environment.yml` file which defines which third-party libraries we depend on.\n", "- Conda will use this file to create a virtual environment for you.\n", "- This virtual environment includes python and all other packages and tools we specified, separated from any preexisting\n", "python installation you may have." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Installation\n", "\n", "Install the python3 version of [miniconda](https://conda.io/miniconda.html).\n", "Follow the [installation instructions](https://conda.io/docs/user-guide/install/index.html)\n", "for your platform.\n", "\n", "Install all dependencies with `conda`:\n", "\n", "```shell\n", "conda env update -f environment.yml\n", "```\n", "\n", "To activate the virtual environment (set up `$PATH`):\n", "\n", "```shell\n", "conda activate cs236781\n", "```\n", "\n", "To check what conda environments you have and which is active, run\n", "\n", "```shell\n", "conda env list\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Running Jupyter\n", "\n", "From a terminal, enter the folder contaning the tutorial notebooks.\n", "Make sure that the active conda environment is `cs236781`, and run\n", "\n", "```shell\n", "jupyter lab\n", "```\n", "\n", "This will start a [jupyter lab](https://jupyterlab.readthedocs.io/en/stable/)\n", "server and open your browser at the local server's url. You can now start working with the notebooks.\n", "\n", "If you're new to jupyter notebooks, you can get started by reading the\n", "[UI guide](https://jupyter-notebook.readthedocs.io/en/stable/notebook.html#notebook-user-interface)\n", "and also about how to use notebooks in\n", "[JupyterLab](https://jupyterlab.readthedocs.io/en/latest/user/notebook.html).\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Jupyter basics\n", "\n", "Jupyter notebooks consist mainly of code and markdown cells.\n", "The code cells contain code that is run by a `kernel`, an\n", "interpreter for some programming language, python in our case." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "bar\n" ] }, { "data": { "text/plain": [ "42" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# This is a code cell; it can contain arbitrary python code.\n", "\n", "foo = 'bar'\n", "print(foo)\n", "\n", "def the_answer():\n", " return 42\n", "\n", "# The output of the last expression in a cell is shown\n", "2*the_answer()\n", "the_answer()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Variables and functions defined in a code cell are available in subsequent cells." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "ans = the_answer()" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "42" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ans" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is a markdown cell. You can use markdown syntax to format your text, and also include equations\n", "written in $\\LaTeX$:\n", "\n", "$$\n", "e^{i\\pi} - 1 = 0\n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Other useful things to know about:\n", "* Opening a console for notebook\n", "* Restarting kernel\n", "* Magics" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The autoreload extension is already loaded. To reload it, use:\n", " %reload_ext autoreload\n" ] } ], "source": [ "%matplotlib inline\n", "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Basics of Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Python is a high-level, dynamically typed multiparadigm programming language. Python code is often said to be almost like pseudocode, since it allows you to express very powerful ideas in very few lines of code while being very readable. As an example, here is an implementation of the classic quicksort algorithm in Python:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1, 1, 2, 3, 6, 8, 10]\n" ] } ], "source": [ "def quicksort(arr):\n", " if len(arr) <= 1:\n", " return arr\n", " pivot = arr[len(arr) // 2]\n", " left = [x for x in arr if x < pivot]\n", " middle = [x for x in arr if x == pivot]\n", " right = [x for x in arr if x > pivot]\n", " return quicksort(left) + middle + quicksort(right)\n", "\n", "print(quicksort([3,6,8,10,1,2,1]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Python has great [documentation](https://docs.python.org/3)! Use it often." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Packages and modules" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A python **module** is simply a python file (`.py`), which can contain functions, classes and even top-level code.\n", "\n", "A **package** is a collection of modules within a directory. Python comes with a standard library which\n", "includes many useful packages.\n", "\n", "A package must be imported before use. They can be imported like so:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# Import packages from the python standard library\n", "import math\n", "import sys" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Additionally, we can define our own packages and modules. This tutorial comes with a `demo_package`\n", "which includes a `demo_module`.\n", "\n", "Any object can be imported from a module like so:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "this is a demo, a=17\n", "this is a demo, a=3.141592653589793\n", "FOO\n" ] } ], "source": [ "# Import just a specific function from a specific module from a specific package\n", "from demo_package.demo_module import demo_func\n", "\n", "demo_func(math.pi)\n", "print('FOO', file=sys.stdout)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Basic data types" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Numbers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Integers and floats work as you would expect from other languages:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3 \n" ] } ], "source": [ "x = 3\n", "print(x, type(x))" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "4\n", "2\n", "6\n", "9\n" ] } ], "source": [ "print(x + 1) # Addition;\n", "print(x - 1) # Subtraction;\n", "print(x * 2) # Multiplication;\n", "print(x ** 2) # Exponentiation;" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "4\n", "8\n" ] } ], "source": [ "x += 1\n", "print(x)\n", "x *= 2\n", "print(x)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "2.5 3.5 5.0 6.25 1.25 1.0\n" ] } ], "source": [ "y = 2.5\n", "print(type(y))\n", "print(y, y + 1, y * 2, y ** 2, y / 2, y // 2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that unlike many languages, Python does not have unary increment (x++) or decrement (x--) operators.\n", "\n", "Python also has built-in types for long integers and complex numbers; you can find all of the details in the [documentation](https://docs.python.org/3/library/stdtypes.html#numeric-types-int-float-long-complex)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Booleans" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Python implements all of the usual operators for Boolean logic, but uses English words rather than symbols (`&&`, `||`, etc.):" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "t, f = True, False" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we let's look at the operations:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "False\n", "True\n", "False\n", "True\n" ] } ], "source": [ "print(t and f) # Logical AND\n", "print(t or f ) # Logical OR\n", "print(not t ) # Logical NOT\n", "print(t != f ) # Logical XOR" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Strings" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "('hello', 5)" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "hello = 'hello' # String literals can use single quotes\n", "world = \"world\" # or double quotes; it does not matter.\n", "hello, len(hello)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "'aaa bbb'" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# String concatenation\n", "'aaa ' + 'bbb'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are several way to created formatted strings, here are a couple:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "hello [1, 2, 3]: pi=3.14159\n", "hello [1, 2, 3]: pi=3.14159\n" ] } ], "source": [ "s = 'hello'\n", "a = [1,2,3]\n", "\n", "# sprintf style string formatting\n", "print('%s %s: pi=%.5f' % (s, a, math.pi))\n", "\n", "# formatting with f-string literals (python 3.6+)\n", "print(f'{s} {a}: pi={math.pi:.5f}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "String objects have a bunch of useful methods; for example:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Hello\n", "HELLO\n", " hello\n", " hello \n", "he(ell)(ell)o\n", "world\n" ] } ], "source": [ "s = \"hello\"\n", "print(s.capitalize() ) # Capitalize a string; prints \"Hello\"\n", "print(s.upper() ) # Convert a string to uppercase; prints \"HELLO\"\n", "print(s.rjust(7) ) # Right-justify a string, padding with spaces; prints \" hello\"\n", "print(s.center(7) ) # Center a string, padding with spaces; prints \" hello \"\n", "print(s.replace('l', '(ell)')) # Replace all instances of one substring with another\n", "print(' world '.strip()) # Strip leading and trailing whitespace; prints \"world\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can find a list of all string methods in the [documentation](https://docs.python.org/3/library/stdtypes.html#string-methods)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Containers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Python includes several built-in container types: lists, dictionaries, sets, and tuples." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Lists" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A list is the Python equivalent of an array, but is resizeable and can contain elements of different types:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[3, 1, 2]\n", "2 2\n" ] } ], "source": [ "xs = [3, 1, 2] # Create a list\n", "print(xs)\n", "print(xs[2], xs[-1]) # Negative indices count from the end of the list; prints \"2\"" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[3, 1, 'foo']\n" ] } ], "source": [ "xs[2] = 'foo' # Lists can contain elements of different types\n", "print(xs)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[3, 1, 'foo', 'bar']\n" ] } ], "source": [ "xs.append('bar') # Add a new element to the end of the list\n", "print(xs)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "('bar', [3, 1, 'foo'])" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x = xs.pop() # Remove and return the last element of the list\n", "x, xs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Slicing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In addition to accessing list elements one at a time, Python provides concise syntax to access sublists; this is known as slicing:" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "[0, 1, 2, 3, 4]" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nums = list(range(5))\n", "nums" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "[2, 3]" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nums[2:4] # Get a slice from index 2 to 4 (exclusive)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "[2, 3, 4]" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nums[2:] # Get a slice from index 2 to the end" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "[0, 1]" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nums[:2] # Get a slice from the start to index 2 (exclusive)" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "[0, 1, 2, 3, 4]" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nums[:] # Get a slice of the whole list" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "[0, 1, 2, 3]" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nums[:-1] # Slice indices can be negative" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[0, 2]" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nums[0:4:2] # Can also specify slice step size" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "[0, 1, 8, 9, 4]" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nums[2:4] = [8, 9] # Assign a new sublist to a slice\n", "nums" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[1, 8, 9]" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Delete elements from a list\n", "nums[0:1] = []\n", "del nums[-1]\n", "nums" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Loops" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can loop over the elements of a list like this:" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "cat\n", "dog\n", "monkey\n" ] } ], "source": [ "animals = ['cat', 'dog', 'monkey']\n", "for animal in animals:\n", " print(animal)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you want access to the index of each element within the body of a loop, use the built-in `enumerate` function:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "#1: cat\n", "#2: dog\n", "#3: monkey\n" ] } ], "source": [ "animals = ['cat', 'dog', 'monkey']\n", "for idx, animal in enumerate(animals):\n", " print(f'#{idx+1}: {animal}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### List comprehensions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When programming, frequently we want to transform one type of data into another. As a simple example, consider the following code that computes square numbers:" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "[0, 1, 4, 9, 16]" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nums = [0, 1, 2, 3, 4]\n", "squares = []\n", "for x in nums:\n", " squares.append(x ** 2)\n", "squares" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can make this code simpler using a list comprehension:" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "[0, 1, 4, 9, 16]" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "squares = [x ** 2 for x in nums]\n", "squares" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "List comprehensions can also contain conditions:" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "[0, 4, 16]" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "even_squares = [x ** 2 for x in nums if x % 2 == 0]\n", "even_squares" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "List comprehensions can be nested:" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[0, 0, -1, 1, -2, 2, -3, 3, -4, 4]" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nums2 = [-1, 1]\n", "[x * y for x in nums for y in nums2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Dictionaries" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A dictionary stores (key, value) pairs. In other languages this is known as a `Map` or `Hash`." ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "cute\n", "True\n" ] } ], "source": [ "d = {'cat': 'cute', 'dog': 'furry'} # Create a new dictionary with some data\n", "print(d['cat']) # Get an entry from a dictionary\n", "print('cat' in d) # Check if a dictionary has a given key" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "{'cat': 'cute', 'dog': 'furry', 'fish': 'wet'}" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d['fish'] = 'wet' # Set an entry in a dictionary\n", "d" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "'monkey'\n" ] } ], "source": [ "# Trying to access a non-existing key raises a KeyError\n", "try:\n", " d['monkey']\n", "except KeyError as e:\n", " print(e, file=sys.stderr)" ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "N/A\n", "wet\n" ] } ], "source": [ "print(d.get('monkey', 'N/A')) # Get an element with a default\n", "print(d.get('fish', 'N/A')) # Get an element with a default" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "{'cat': 'cute', 'dog': 'furry'}" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "del d['fish'] # Remove an element from a dictionary\n", "d" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "A person has 2 legs\n", "A cat has 4 legs\n", "A spider has 8 legs\n" ] } ], "source": [ "# Iteration over keys\n", "d = {'person': 2, 'cat': 4, 'spider': 8}\n", "for animal in d:\n", " print(f'A {animal} has {d[animal]} legs')" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "A person has 2 legs\n", "A cat has 4 legs\n", "A spider has 8 legs\n" ] } ], "source": [ "# Iterate over key-value pairs\n", "d = {'person': 2, 'cat': 4, 'spider': 8}\n", "for animal, num_legs in d.items():\n", " print(f'A {animal} has {num_legs} legs')" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'foo': 1, 'bar': 2, 'baz': 3}" ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create a dictionary using the built-in dict() function\n", "dict(foo=1, bar=2, baz=3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Dictionary comprehensions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These are similar to list comprehensions, but allow you to easily construct dictionaries. For example:" ] }, { "cell_type": "code", "execution_count": 45, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "{0: 0, 2: 4, 4: 16}" ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nums = [0, 1, 2, 3, 4]\n", "even_num_to_square = {x: x ** 2 for x in nums if x % 2 == 0}\n", "even_num_to_square" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Sets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A set is an unordered collection of distinct elements" ] }, { "cell_type": "code", "execution_count": 46, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'dog', 'cat'}\n", "True\n", "False\n" ] } ], "source": [ "animals = {'cat', 'dog'}\n", "print(animals)\n", "print('cat' in animals ) # Check if an element is in a set\n", "print('fish' in animals) # prints \"False\"" ] }, { "cell_type": "code", "execution_count": 47, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n" ] }, { "data": { "text/plain": [ "3" ] }, "execution_count": 47, "metadata": {}, "output_type": "execute_result" } ], "source": [ "animals.add('fish') # Add an element to a set\n", "print('fish' in animals)\n", "len(animals) # Number of elements in a set" ] }, { "cell_type": "code", "execution_count": 48, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "{'cat', 'dog', 'fish'}" ] }, "execution_count": 48, "metadata": {}, "output_type": "execute_result" } ], "source": [ "animals.add('cat') # Adding an element that is already in the set does nothing\n", "animals" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "_Loops_: Iterating over a set has the same syntax as iterating over a list; however since sets are unordered, you cannot make assumptions about the order in which you visit the elements of the set:" ] }, { "cell_type": "code", "execution_count": 49, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "#0: fish\n", "#1: dog\n", "#2: cat\n" ] } ], "source": [ "animals = {'cat', 'dog', 'fish'}\n", "for idx, animal in enumerate(animals):\n", " print(f'#{idx}: {animal}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Set comprehensions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Like lists and dictionaries, we can easily construct sets using set comprehensions:" ] }, { "cell_type": "code", "execution_count": 50, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "{0, 1, 2, 3, 4, 5, 6}" ] }, "execution_count": 50, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from math import sqrt\n", "s = {int(sqrt(x)) for x in range(37)}\n", "s" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Tuples" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A tuple is an **immutable** ordered list of values." ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(1, 2, 'three')" ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "t = (1, 2, 'three')\n", "t" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It can be used in ways similar to a list:" ] }, { "cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((1,), (2, 'three'), 'three', 3)" ] }, "execution_count": 52, "metadata": {}, "output_type": "execute_result" } ], "source": [ "t[0:1], t[1:3], t[-1], len(t)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A tuple can be used a key in a dictionary and as an element of a sets, while **lists cannot**." ] }, { "cell_type": "code", "execution_count": 53, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "{(0, 1): 0,\n", " (1, 2): 1,\n", " (2, 3): 2,\n", " (3, 4): 3,\n", " (4, 5): 4,\n", " (5, 6): 5,\n", " (6, 7): 6,\n", " (7, 8): 7,\n", " (8, 9): 8,\n", " (9, 10): 9}" ] }, "execution_count": 53, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = {(x, x + 1): x for x in range(10)} # Create a dictionary with tuple keys\n", "d" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A tuple (and also a list) can be **unpacked**:" ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(1, 2, 'three')" ] }, "execution_count": 54, "metadata": {}, "output_type": "execute_result" } ], "source": [ "one, two, three = t\n", "one, two, three" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that when retuning multiple values from a function (or code block in a jupyter notebook, as above)\n", "your values get wrapped in a tuple, and the tuple is what's returned.\n", "Unpacking the return value of a function can make it seem as if multiple values were returned." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Functions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Python functions are defined using the `def` keyword. For example:" ] }, { "cell_type": "code", "execution_count": 55, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "negative\n", "zero\n", "positive\n" ] } ], "source": [ "def sign(x):\n", " if x > 0:\n", " return 'positive'\n", " elif x < 0:\n", " return 'negative'\n", " else:\n", " return 'zero'\n", "\n", "for x in [-1, 0, 1]:\n", " print(sign(x))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will often define functions to take optional keyword arguments, like this:" ] }, { "cell_type": "code", "execution_count": 56, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Hello, Bob!\n", "HELLO, FRED\n" ] } ], "source": [ "def hello(name, loud=False):\n", " if loud:\n", " print('HELLO, %s' % name.upper())\n", " else:\n", " print('Hello, %s!' % name)\n", "\n", "hello('Bob')\n", "hello('Fred', loud=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Positional and Keyword arguments" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Python functions are very flexible in the way they accept arguments. Both positional (regular) and keyword\n", "arguments are supported and can be mixed in the same definition. Additionally, extra arguments can be passed in with the `*args` and `**kwargs` constructs.\n", "\n", "Here's a function with three positional arguments and three keyword arguments which also accepts extra \n", "positional and keyword arguments." ] }, { "cell_type": "code", "execution_count": 57, "metadata": {}, "outputs": [], "source": [ "def myfunc(a1, a2, a3, *extra_args, kw1='foo', kw2='bar', kw3=3, **extra_kwargs):\n", " print(f'Got positional args: {(a1, a2, a3)}')\n", " print(f'Got keyword args : {dict(kw1=kw1, kw2=kw3, kw3=kw3)}')\n", " print(f'Got extra positional args: {extra_args}')\n", " print(f'Got extra keyword args: {extra_kwargs}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It can be called in many ways:" ] }, { "cell_type": "code", "execution_count": 58, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Got positional args: (1, 2, 3)\n", "Got keyword args : {'kw1': 'foo', 'kw2': 3, 'kw3': 3}\n", "Got extra positional args: (4, 5, 6)\n", "Got extra keyword args: {}\n" ] } ], "source": [ "myfunc(1,2,3,4,5,6)" ] }, { "cell_type": "code", "execution_count": 59, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Got positional args: (1, 2, 3)\n", "Got keyword args : {'kw1': 'foo', 'kw2': 3, 'kw3': 3}\n", "Got extra positional args: (4,)\n", "Got extra keyword args: {}\n" ] } ], "source": [ "my_args = [1,2,3,4]\n", "myfunc(*my_args)" ] }, { "cell_type": "code", "execution_count": 60, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Got positional args: (1, 2, 3)\n", "Got keyword args : {'kw1': 'foo', 'kw2': 3, 'kw3': 3}\n", "Got extra positional args: ()\n", "Got extra keyword args: {'foo': 'bar'}\n" ] } ], "source": [ "myfunc(1,2,3, kw3=3, kw2=2, foo='bar')" ] }, { "cell_type": "code", "execution_count": 61, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Got positional args: (1, 2, 3)\n", "Got keyword args : {'kw1': 1, 'kw2': 3, 'kw3': 3}\n", "Got extra positional args: ()\n", "Got extra keyword args: {'kw4': 4}\n" ] } ], "source": [ "my_kwargs = dict(kw1=1, kw2=2, kw3=3, kw4=4)\n", "myfunc(1,2,3, **my_kwargs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that keyword args can be omitted, while positional args cannot:" ] }, { "cell_type": "code", "execution_count": 62, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "myfunc() missing 1 required positional argument: 'a3'\n" ] } ], "source": [ "try:\n", " myfunc(1,2)\n", "except TypeError as e:\n", " print(e, file=sys.stderr)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Classes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The syntax for defining classes in Python is straightforward:" ] }, { "cell_type": "code", "execution_count": 63, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Hello, Fred\n", "HELLO, FRED!\n" ] } ], "source": [ "class Greeter:\n", "\n", " # Constructor\n", " def __init__(self, name):\n", " self.name = name # Create an instance variable\n", "\n", " # Instance method\n", " def greet(self, loud=False):\n", " if loud:\n", " print('HELLO, %s!' % self.name.upper())\n", " else:\n", " print('Hello, %s' % self.name)\n", "\n", "g = Greeter('Fred') # Construct an instance of the Greeter class\n", "g.greet() # Call an instance method\n", "g.greet(loud=True) # Call an instance method" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Classes can implement special **magic functions** that enable them to be integrated nicely with other python code. Magic functions have special names that start and end with `__`.\n", "\n", "For example, here's a class that can be indexed with `[]` and iterated over with a `for` loop." ] }, { "cell_type": "code", "execution_count": 64, "metadata": {}, "outputs": [], "source": [ "class ExampleCollection(object):\n", " def __init__(self):\n", " self.items = [100, 200, 300]\n", " \n", " def __len__(self):\n", " return len(self.items)\n", " \n", " def __getitem__(self, idx):\n", " return self.items[idx]\n", " \n", " def __iter__(self):\n", " class ExampleIter():\n", " def __init__(self, collection):\n", " self.idx = 0\n", " self.collection = collection\n", " \n", " def __next__(self):\n", " if self.idx >= len(self.collection):\n", " raise StopIteration()\n", " x = self.collection[self.idx]\n", " self.idx += 1\n", " return x\n", " \n", " return ExampleIter(self)\n", " " ] }, { "cell_type": "code", "execution_count": 65, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "length= 3\n", "example[0]= 100\n" ] } ], "source": [ "example = ExampleCollection()\n", "print('length=', len(example)) # invokes __len__\n", "print('example[0]=', example[0]) # invokes __getitem__" ] }, { "cell_type": "code", "execution_count": 66, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "100\n", "200\n", "300\n" ] } ], "source": [ "for x in example: # invokes __iter__ and it's __next__\n", " print(x)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Many other magic functions exist. Consult the docs and see if you can catch 'em all!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Numpy" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Numpy is a core library for scientific computing in Python. It provides a high-performance multidimensional array object, and tools for working with these arrays.\n", "\n", "We'll refer to such n-dimentional arrays as **tensors** in accordance with the deep learning terminology.\n", "\n", "Although we'll mainly use PyTorch tensors for implementing our Deep Learning systems, it's still important to be proficient with `numpy`, since:\n", "1. They concepts are very similar. It will help you understand numpy ndarrays you'll understand Pytorch Tensors.\n", "1. You'll find that you constantly need to switch between the two when working with read DL systems." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To use Numpy, we first need to import the `numpy` package:" ] }, { "cell_type": "code", "execution_count": 67, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "import numpy as np" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Arrays" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A numpy array represents an n-dimentional grid of values, all of the same type, and is indexed by a tuple of nonnegative integers.\n", "\n", "- The number of dimensions is the **rank** of the array\n", "- The **shape** of an array is a tuple of integers giving the size of the array along each dimension" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can initialize numpy arrays from nested Python lists, and access elements using square brackets:" ] }, { "cell_type": "code", "execution_count": 68, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([1, 2, 3])" ] }, "execution_count": 68, "metadata": {}, "output_type": "execute_result" } ], "source": [ "a = np.array([1, 2, 3]) # Create a rank 1 array\n", "a" ] }, { "cell_type": "code", "execution_count": 69, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1" ] }, "execution_count": 69, "metadata": {}, "output_type": "execute_result" } ], "source": [ "a[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Two very important properties of any numpy array are its `shape` and `dtype`." ] }, { "cell_type": "code", "execution_count": 70, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "def print_arr(arr, pre_text=''):\n", " print(f'{pre_text}{arr} shape={arr.shape} dtype={arr.dtype}')" ] }, { "cell_type": "code", "execution_count": 71, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1 2 3] shape=(3,) dtype=int64\n" ] } ], "source": [ "print_arr(a)" ] }, { "cell_type": "code", "execution_count": 72, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([5, 2, 3])" ] }, "execution_count": 72, "metadata": {}, "output_type": "execute_result" } ], "source": [ "a[0] = 5 # Change an element of the array\n", "a" ] }, { "cell_type": "code", "execution_count": 73, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1. 2. 3. ]\n", " [4. 5. 6.7]] shape=(2, 3) dtype=float64\n" ] } ], "source": [ "b = np.array([[1,2,3],[4,5,6.7]]) # Create a rank 2 array\n", "print_arr(b)" ] }, { "cell_type": "code", "execution_count": 74, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "(1.0, 2.0, 4.0)" ] }, "execution_count": 74, "metadata": {}, "output_type": "execute_result" } ], "source": [ "b[0, 0], b[0, 1], b[1, 0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Numpy also provides many functions to create arrays:" ] }, { "cell_type": "code", "execution_count": 75, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([[0., 0.],\n", " [0., 0.]])" ] }, "execution_count": 75, "metadata": {}, "output_type": "execute_result" } ], "source": [ "np.zeros((2,2)) # Create an array of all zeros" ] }, { "cell_type": "code", "execution_count": 76, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]])" ] }, "execution_count": 76, "metadata": {}, "output_type": "execute_result" } ], "source": [ "np.ones((1,10)) # Create an array of all ones" ] }, { "cell_type": "code", "execution_count": 77, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([[7.2, 7.2, 7.2],\n", " [7.2, 7.2, 7.2],\n", " [7.2, 7.2, 7.2]])" ] }, "execution_count": 77, "metadata": {}, "output_type": "execute_result" } ], "source": [ "np.full((3,3), 7.2) # Create a constant array" ] }, { "cell_type": "code", "execution_count": 78, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([[1, 0, 0, 0],\n", " [0, 1, 0, 0],\n", " [0, 0, 1, 0],\n", " [0, 0, 0, 1]])" ] }, "execution_count": 78, "metadata": {}, "output_type": "execute_result" } ], "source": [ "np.eye(4, dtype=np.int) # Create an identity matrix of integers" ] }, { "cell_type": "code", "execution_count": 79, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([[[0.21921227, 0.39286303, 0.89934625],\n", " [0.72141811, 0.1582272 , 0.14031527],\n", " [0.20468505, 0.38861605, 0.44066535],\n", " [0.50377059, 0.06713136, 0.75217024]],\n", "\n", " [[0.40337263, 0.10336229, 0.22444997],\n", " [0.82814678, 0.48742751, 0.53362932],\n", " [0.74819915, 0.63329761, 0.31033589],\n", " [0.8924808 , 0.85084316, 0.50349368]],\n", "\n", " [[0.88498439, 0.74481972, 0.40691951],\n", " [0.34428811, 0.57900335, 0.47587481],\n", " [0.69307823, 0.51344594, 0.92398386],\n", " [0.5347534 , 0.12659492, 0.57205981]],\n", "\n", " [[0.05397933, 0.1507 , 0.92718578],\n", " [0.2778349 , 0.51612588, 0.24952806],\n", " [0.43105188, 0.54110875, 0.07947924],\n", " [0.81656111, 0.87409979, 0.79965167]]])" ] }, "execution_count": 79, "metadata": {}, "output_type": "execute_result" } ], "source": [ "t = np.random.random((4,4,3)) # Create a 3d-array filled with U[0,1] random values\n", "t" ] }, { "cell_type": "code", "execution_count": 80, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.5336293174054009" ] }, "execution_count": 80, "metadata": {}, "output_type": "execute_result" } ], "source": [ "t[1,1,2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Array rank\n", "\n", "In `numpy` **rank** means **number of dimensions**." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**rank-0** arrays are scalars." ] }, { "cell_type": "code", "execution_count": 81, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "17 shape=() dtype=int64\n" ] } ], "source": [ "a0 = np.array(17)\n", "print_arr(a0)" ] }, { "cell_type": "code", "execution_count": 82, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "17" ] }, "execution_count": 82, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Get sclar as a python float\n", "a0.item()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**rank-1** arrays of length `n` have a shape of `(n,)`. " ] }, { "cell_type": "code", "execution_count": 83, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1 2 3] shape=(3,) dtype=int64\n" ] } ], "source": [ "# A rank-1 array\n", "a1 = np.array([1,2,3])\n", "\n", "print_arr(a1)" ] }, { "cell_type": "code", "execution_count": 84, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[3.14] shape=(1,) dtype=float64\n" ] } ], "source": [ "# A rank-1 array scalar\n", "print_arr(np.array([3.14]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**rank-2** arrays have a shape of `(n,m)`. " ] }, { "cell_type": "code", "execution_count": 85, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1 2 3]\n", " [4 5 6]] shape=(2, 3) dtype=int64\n" ] } ], "source": [ "a2 = np.array([[1,2,3], [4,5,6]])\n", "\n", "print_arr(a2)" ] }, { "cell_type": "code", "execution_count": 86, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1]\n", " [2]\n", " [3]] shape=(3, 1) dtype=int64\n" ] } ], "source": [ "# A column vector is also rank-2!\n", "a_col = a1.reshape(-1, 1)\n", "\n", "print_arr(a_col)" ] }, { "cell_type": "code", "execution_count": 87, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1 2 3]] shape=(1, 3) dtype=int64\n" ] } ], "source": [ "# A row vector is also rank-2\n", "a_row = a1.reshape(1, -1)\n", "\n", "print_arr(a_row)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**rank-k** arrays have a shape of `(n1,...,nk)`. " ] }, { "cell_type": "code", "execution_count": 88, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[[0. 0. 0. 0.]\n", " [0. 0. 0. 0.]\n", " [0. 0. 0. 0.]]\n", "\n", " [[0. 0. 0. 0.]\n", " [0. 0. 0. 0.]\n", " [0. 0. 0. 0.]]] shape=(2, 3, 4) dtype=float64\n" ] } ], "source": [ "print_arr(np.zeros((2,3,4)))" ] }, { "cell_type": "code", "execution_count": 89, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[[[1. 1.]\n", " [1. 1.]]\n", "\n", " [[1. 1.]\n", " [1. 1.]]]\n", "\n", "\n", " [[[1. 1.]\n", " [1. 1.]]\n", "\n", " [[1. 1.]\n", " [1. 1.]]]] shape=(2, 2, 2, 2) dtype=float64\n" ] } ], "source": [ "print_arr(np.ones((2,2,2,2)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Array math" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Elementwise operations\n", "Basic mathematical functions **operate elementwise** on arrays, and are available both as operator overloads and as functions in the numpy module:" ] }, { "cell_type": "code", "execution_count": 90, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[ 6. 8.]\n", " [10. 12.]]\n", "[[-4. -4.]\n", " [-4. -4.]]\n", "[[ 5. 12.]\n", " [21. 32.]]\n", "[[0.2 0.33333333]\n", " [0.42857143 0.5 ]]\n" ] } ], "source": [ "x = np.array([[1,2],[3,4]], dtype=np.float64)\n", "y = np.array([[5,6],[7,8]], dtype=np.float64)\n", "\n", "# Elementwise basic math\n", "print(x + y)\n", "print(x - y)\n", "print(x * y)\n", "print(x / y)" ] }, { "cell_type": "code", "execution_count": 91, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1. 1.41421356]\n", " [1.73205081 2. ]]\n", "[[ 2.71828183 7.3890561 ]\n", " [20.08553692 54.59815003]]\n", "[[0. 0.69314718]\n", " [1.09861229 1.38629436]]\n" ] } ], "source": [ "# Elementwise functions\n", "print(np.sqrt(x))\n", "print(np.exp(x))\n", "print(np.log(x))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are of course many more elementwise operations inmplemented by `numpy`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Inner products\n", "\n", "Unlike MATLAB, `*` is elementwise multiplication, not matrix multiplication (as we saw above).\n", "\n", "We can instead use the `dot()` function to:\n", "- compute inner products of vectors,\n", "- multiply a vector by a matrix, and to\n", "- multiply matrices, and more generally n-d tensors.\n", "\n", "The `dot()` function is available both as a function in the numpy module and as an instance\n", "method of array objects." ] }, { "cell_type": "code", "execution_count": 92, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "219\n", "219\n" ] } ], "source": [ "v = np.array([9,10])\n", "w = np.array([11, 12])\n", "\n", "# Inner product of vectors; both produce 219\n", "print(v.dot(w))\n", "print(np.dot(v, w))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Rank-1 arrays arrays are somewhat special in that `numpy` can treat them both as column or as row vectors.\n", "Arrays of different rank have different semantics when using them in vector-vector or vector-matrix products, so always make sure you known what shapes you're working with:" ] }, { "cell_type": "code", "execution_count": 93, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "a1 * a1 =\t14 shape=() dtype=int64\n", "a_row * a1 =\t[14] shape=(1,) dtype=int64\n", "a1 * a_col =\t[14] shape=(1,) dtype=int64\n", "a_row * a_col =\t[[14]] shape=(1, 1) dtype=int64\n", "a_col * a_row =\n", "[[1 2 3]\n", " [2 4 6]\n", " [3 6 9]] shape=(3, 3) dtype=int64\n" ] } ], "source": [ "print_arr(np.dot(a1, a1), 'a1 * a1 =\\t')\n", "\n", "print_arr(np.dot(a_row, a1), 'a_row * a1 =\\t')\n", "\n", "print_arr(np.dot(a1, a_col), 'a1 * a_col =\\t')\n", "\n", "print_arr(np.dot(a_row, a_col), 'a_row * a_col =\\t')\n", "\n", "print_arr(np.dot(a_col, a_row), 'a_col * a_row =\\n')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Non-elementwise operations\n", "\n", "Numpy provides many useful functions for performing computations on arrays." ] }, { "cell_type": "code", "execution_count": 94, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1 2]\n", " [3 4]] shape=(2, 2) dtype=int64\n" ] } ], "source": [ "x = np.array([[1,2],[3,4]])\n", "print_arr(x)" ] }, { "cell_type": "code", "execution_count": 95, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "10\n", "[2. 3.]\n", "[ 2 12]\n" ] } ], "source": [ "print(np.sum(x)) # Compute sum of all elements\n", "print(np.mean(x, axis=0)) # Compute mean of each column\n", "print(np.prod(x, axis=1) ) # Compute product of each row" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can find the full list of mathematical functions provided by numpy in the [documentation](http://docs.scipy.org/doc/numpy/reference/routines.math.html)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Array indexing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Numpy offers several ways to index into arrays." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Slicing**\n", "\n", "Similar to Python lists, numpy arrays can be sliced. Since arrays may be multidimensional, you must specify **a slice for each dimension** of the array:" ] }, { "cell_type": "code", "execution_count": 96, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[ 1 2 3 4]\n", " [ 5 6 7 8]\n", " [ 9 10 11 12]] shape=(3, 4) dtype=int64\n" ] } ], "source": [ "a = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])\n", "\n", "print_arr(a)" ] }, { "cell_type": "code", "execution_count": 97, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[2 3]\n", " [6 7]] shape=(2, 2) dtype=int64\n" ] } ], "source": [ "b = a[:2, 1:3]\n", "\n", "print_arr(b)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A slice of an array is a **view** into the same in-memory data, so modifying it will modify the original array." ] }, { "cell_type": "code", "execution_count": 98, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([[ 1, 77777, 3, 4],\n", " [ 5, 6, 7, 8],\n", " [ 9, 10, 11, 12]])" ] }, "execution_count": 98, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Changing a view\n", "b[0, 0] = 77777\n", "\n", "# ...modifies original\n", "a" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also mix integer indexing with slice indexing.\n", "However, doing so will yield an array of **lower rank** than the original array.\n" ] }, { "cell_type": "code", "execution_count": 99, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([[ 1, 2, 3, 4],\n", " [ 5, 6, 7, 8],\n", " [ 9, 10, 11, 12]])" ] }, "execution_count": 99, "metadata": {}, "output_type": "execute_result" } ], "source": [ "a = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])\n", "a" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Two ways of accessing the data in the middle row of the array.\n", "- Mixing integer indexing with slices yields an array of lower rank\n", "- Using only slices yields an array of the same rank as the original array" ] }, { "cell_type": "code", "execution_count": 100, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[5 6 7 8] shape=(4,) dtype=int64\n", "[[5 6 7 8]] shape=(1, 4) dtype=int64\n", "[[5 6 7 8]] shape=(1, 4) dtype=int64\n" ] } ], "source": [ "row_r1 = a[1, :] # Rank 1 view of the second row of a \n", "row_r2 = a[1:2, :] # Rank 2 view of the second row of a\n", "row_r3 = a[[1], :] # Rank 2 view of the second row of a\n", "\n", "print_arr(row_r1)\n", "print_arr(row_r2)\n", "print_arr(row_r3)" ] }, { "cell_type": "code", "execution_count": 101, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ 2 6 10] shape=(3,) dtype=int64\n", "[[ 2]\n", " [ 6]\n", " [10]] shape=(3, 1) dtype=int64\n" ] } ], "source": [ "# We can make the same distinction when accessing columns of an array:\n", "col_r1 = a[:, 1]\n", "col_r2 = a[:, 1:2]\n", "\n", "print_arr(col_r1)\n", "print_arr(col_r2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Integer array indexing** \n", "\n", "- When you slice, the resulting array view will always be a subarray of the original array.\n", "- Integer array indexing allows you to construct arbitrary arrays using the data from another array.\n" ] }, { "cell_type": "code", "execution_count": 102, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1 2]\n", " [3 4]\n", " [5 6]] shape=(3, 2) dtype=int64\n" ] } ], "source": [ "a = np.array([[1,2], [3, 4], [5, 6]])\n", "print_arr(a)" ] }, { "cell_type": "code", "execution_count": 103, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1 4 5] shape=(3,) dtype=int64\n" ] } ], "source": [ "# An example of integer array indexing.\n", "# The returned array will have shape (3,)\n", "print_arr(a[ [0, 1, 2], [0, 1, 0] ])" ] }, { "cell_type": "code", "execution_count": 104, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1 4 5] shape=(3,) dtype=int64\n" ] } ], "source": [ "# The above example of integer array indexing is equivalent to this:\n", "print_arr(np.array([a[0, 0], a[1, 1], a[2, 0]]))" ] }, { "cell_type": "code", "execution_count": 105, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[2 2]\n", "[2 2]\n" ] } ], "source": [ "# When using integer array indexing, you can reuse the same\n", "# element from the source array:\n", "print(a[[0, 0], [1, 1]])\n", "\n", "# Equivalent to the previous integer array indexing example\n", "print(np.array([a[0, 1], a[0, 1]]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One useful trick with integer array indexing is selecting or mutating one element from each row of a matrix:" ] }, { "cell_type": "code", "execution_count": 106, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([[ 1, 2, 3],\n", " [ 4, 5, 6],\n", " [ 7, 8, 9],\n", " [10, 11, 12]])" ] }, "execution_count": 106, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create a new array from which we will select elements\n", "a = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])\n", "a" ] }, { "cell_type": "code", "execution_count": 107, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([ 1, 6, 7, 11])" ] }, "execution_count": 107, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create an array of indices\n", "b = np.array([0, 2, 0, 1])\n", "\n", "# Select one element from each row of a using the indices in b\n", "a[np.arange(4), b]" ] }, { "cell_type": "code", "execution_count": 108, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([[1001, 2, 3],\n", " [ 4, 5, 1006],\n", " [1007, 8, 9],\n", " [ 10, 1011, 12]])" ] }, "execution_count": 108, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Mutate one element from each row of a using the indices in b\n", "a[np.arange(4), b] += 1000\n", "a" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Boolean array indexing**\n", "\n", "This type of indexing is used to select the elements of an array that satisfy some condition\n", "(similar to MATLAB's logical indexing)." ] }, { "cell_type": "code", "execution_count": 109, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1 2]\n", " [3 4]\n", " [5 6]] shape=(3, 2) dtype=int64\n" ] } ], "source": [ "a = np.array([[1,2], [3, 4], [5, 6]])\n", "print_arr(a)" ] }, { "cell_type": "code", "execution_count": 110, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([[False, False],\n", " [ True, True],\n", " [ True, True]])" ] }, "execution_count": 110, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bool_idx = (a > 2) # Find the elements of a that are bigger than 2;\n", " # this returns a numpy array of Booleans of the same\n", " # shape as a, where each slot of bool_idx tells\n", " # whether that element of a is > 2.\n", "\n", "bool_idx" ] }, { "cell_type": "code", "execution_count": 111, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([3, 4, 5, 6])" ] }, "execution_count": 111, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# We use boolean array indexing to construct a rank 1 array\n", "# consisting of the elements of a corresponding to the True values\n", "# of bool_idx\n", "a[a>2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For brevity we have left out a lot of details about numpy array indexing; if you want to know more you should read the [documentation](https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Datatypes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Every numpy array is a grid of elements of the same type. Numpy provides a large set of numeric datatypes that you can use to construct arrays. Numpy tries to guess a datatype when you create an array, but functions that construct arrays usually also include an optional argument to explicitly specify the datatype. Here is an example:" ] }, { "cell_type": "code", "execution_count": 112, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "(dtype('int64'), dtype('float64'), dtype('int64'))" ] }, "execution_count": 112, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x = np.array([1, 2]) # Let numpy choose the datatype\n", "y = np.array([1.0, 2.0]) # Let numpy choose the datatype\n", "z = np.array([1, 2], dtype=np.int64) # Force a particular datatype\n", "\n", "x.dtype, y.dtype, z.dtype" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can read all about numpy datatypes in the [documentation](http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Changing and adding dimensions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can **transpose** dimensions within an array using arbitrary axis permutations." ] }, { "cell_type": "code", "execution_count": 113, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1. 1. 1.]\n", " [1. 1. 1.]\n", " [1. 1. 1.]\n", " [1. 1. 1.]\n", " [1. 1. 1.]] shape=(5, 3) dtype=float64\n" ] } ], "source": [ "a = np.ones((3, 5))\n", "print_arr(a.transpose()) # also a.T" ] }, { "cell_type": "code", "execution_count": 114, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[[ 1. 1. 1. 1. 1. 1.]\n", " [ 1. 1. 1. 1. 1. 1.]]\n", "\n", " [[ 1. 1. 1. 1. 1. 1.]\n", " [ 1. 1. 1. 1. 1. 1.]]\n", "\n", " [[ 1. 1. 1. 1. 1. 1.]\n", " [ 1. 1. 1. 777. 1. 1.]]\n", "\n", " [[ 1. 1. 1. 1. 1. 1.]\n", " [ 1. 1. 1. 1. 1. 1.]]] shape=(4, 2, 6) dtype=float64\n" ] } ], "source": [ "a = np.ones((2, 4, 6))\n", "a[1,2,3] = 777\n", "\n", "print_arr(a.transpose(1,0,2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that an element `[x,y,z]` moves to position `[y,x,z]` after a transpose with this permutation (1,0,2)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Another important feature is **reshaping** an array into different dimensions." ] }, { "cell_type": "code", "execution_count": 115, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1. 1. 1. 1. 1. 1. 1. 1. 1.]\n", " [1. 1. 1. 1. 1. 1. 1. 1. 1.]] shape=(2, 9) dtype=float64\n" ] } ], "source": [ "a = np.ones((3, 6))\n", "print_arr(np.reshape(a, (2, 9)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When reshaping, we need to make sure to preserve the same number of elements.\n", "Use `-1` in one of the dimensions to tell numpy to \"figure it out\"." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also combine multiple arrays with **concatenation** along an arbitrary axis." ] }, { "cell_type": "code", "execution_count": 116, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1 2]\n", " [3 4]] shape=(2, 2) dtype=int64\n", "[[5 6]] shape=(1, 2) dtype=int64\n" ] } ], "source": [ "a = np.array([[1, 2], [3, 4]])\n", "b = np.array([[5, 6]])\n", "print_arr(a)\n", "print_arr(b)" ] }, { "cell_type": "code", "execution_count": 117, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1 2]\n", " [3 4]\n", " [5 6]] shape=(3, 2) dtype=int64\n" ] } ], "source": [ "print_arr(np.concatenate((a, b), axis=0))" ] }, { "cell_type": "code", "execution_count": 118, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1 2 5]\n", " [3 4 6]] shape=(2, 3) dtype=int64\n" ] } ], "source": [ "print_arr(np.concatenate((a, b.T), axis=1))" ] }, { "cell_type": "code", "execution_count": 119, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1 2 3 4 5 6] shape=(6,) dtype=int64\n" ] } ], "source": [ "print_arr(np.concatenate((a, b), axis=None))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Broadcasting" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Broadcasting is a powerful mechanism that allows numpy to work with arrays of **different shapes** when performing arithmetic operations.\n", "\n", "Frequently we have a smaller array and a larger array, and we want to use the smaller array multiple times to perform some operation on the larger array.\n", "\n", "For example, suppose that we want to add a constant vector to each row of a matrix." ] }, { "cell_type": "code", "execution_count": 120, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "x=\n", "[[ 1 2 3]\n", " [ 4 5 6]\n", " [ 7 8 9]\n", " [10 11 12]] shape=(4, 3) dtype=int64\n", "v=[1 0 1] shape=(3,) dtype=int64\n" ] } ], "source": [ "# We will add the vector v to each row of the matrix x,\n", "# storing the result in the matrix y\n", "x = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])\n", "v = np.array([1, 0, 1])\n", "y = np.empty_like(x) # Create an empty matrix with the same shape as x\n", "\n", "print_arr(x,'x=\\n')\n", "print_arr(v, 'v=')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Naïve approach**: Use a loop." ] }, { "cell_type": "code", "execution_count": 121, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([[ 2, 2, 4],\n", " [ 5, 5, 7],\n", " [ 8, 8, 10],\n", " [11, 11, 13]])" ] }, "execution_count": 121, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Add the vector v to each row of the matrix x with an explicit loop\n", "for i in range(4):\n", " y[i, :] = x[i, :] + v\n", "\n", "y" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This works; however computing explicit loops in Python is **slow**. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Naïve approach 2**: adding the vector v to each row of the matrix `x` is equivalent to forming a matrix `vv` by stacking multiple copies of `v` vertically, then performing elementwise summation of `x` and `vv`.\n", "\n", "We could implement this approach like this:" ] }, { "cell_type": "code", "execution_count": 122, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([[1, 0, 1],\n", " [1, 0, 1],\n", " [1, 0, 1],\n", " [1, 0, 1]])" ] }, "execution_count": 122, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vv = np.tile(v, (4, 1)) # Stack 4 copies of v on top of each other\n", "vv" ] }, { "cell_type": "code", "execution_count": 123, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([[ 2, 2, 4],\n", " [ 5, 5, 7],\n", " [ 8, 8, 10],\n", " [11, 11, 13]])" ] }, "execution_count": 123, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y = x + vv # Add x and vv elementwise\n", "y" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Nice, but a new array was allocated and memory was copied." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Numpy broadcasting** allows us to perform this computation without actually creating multiple copies of v. Consider this version, using broadcasting:" ] }, { "cell_type": "code", "execution_count": 124, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "shapes: (4, 3) (3,)\n" ] }, { "data": { "text/plain": [ "array([[ 2, 2, 4],\n", " [ 5, 5, 7],\n", " [ 8, 8, 10],\n", " [11, 11, 13]])" ] }, "execution_count": 124, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])\n", "v = np.array([1, 0, 1])\n", "\n", "# Add v to each row of x using broadcasting\n", "y = x + v \n", "\n", "print('shapes: ', x.shape, v.shape)\n", "y" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The line `y = x + v` works even though `x` has shape `(4, 3)` and `v` has shape `(3,)` due to broadcasting; this line works **as if** v actually had shape `(4, 3)`, where each row was a copy of `v`, and the sum was performed elementwise." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Broadcasting two arrays together follows these rules:\n", "\n", "1. All input arrays with ndim smaller than the input array of largest ndim, have **1’s prepended to their shapes**.\n", "1. The size in each dimension of the **output shape** is the maximum of all the input sizes in that dimension.\n", "1. An input can be used in the calculation if its size in a particular **dimension either matches** the output size in that dimension, **or has value exactly 1**.\n", "1. If an input has a dimension size of 1 in its shape, the **first data entry in that dimension will be used for all calculations** along that dimension. In other words, the stepping machinery of the ufunc will simply not step along that dimension (the stride will be 0 for that dimension)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In our example:\n", "- `x` has shape `(4,3)`\n", "- `v` has shape `(3,)`.\n", "\n", "Following the Broadcasting logic, we can say the following is equivalent to what happened:\n", "1. `v` has less dims than `x` so a dimension of `1` is **prepended** -> `v` is now `(1, 3)`.\n", "1. Output shape will be `(max(1,4), max(3,3)) = (4,3)`.\n", "1. Dim 1 of `v` matches exactly (3); dim 0 is exactly 1, so we can use the first data entry (row 0) for each time any row is accessed. This is effectively like converting `v` from `(1,3)` to `(4,3)` by replicating." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Broadcasting is incredibly useful and necessary for writing **vectorized** code,\n", "i.e. code that avoids explicit python loops which are very slow.\n", "Instead, this approach leveraged the underlying C implementation of numpy." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For more on broadcasting, see the [documentation](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html) or this [explanation](http://wiki.scipy.org/EricsBroadcastingDoc).\n", "\n", "Functions that support broadcasting are known as universal functions. You can find the list of all universal functions in the [documentation](http://docs.scipy.org/doc/numpy/reference/ufuncs.html#available-ufuncs)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here are some applications of broadcasting:" ] }, { "cell_type": "code", "execution_count": 125, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1 2 3] shape=(3,) dtype=int64\n", "[4 5] shape=(2,) dtype=int64\n" ] } ], "source": [ "# Compute outer product of vectors\n", "v = np.array([1,2,3]) # v has shape (3,)\n", "w = np.array([4,5]) # w has shape (2,)\n", "print_arr(v)\n", "print_arr(w)" ] }, { "cell_type": "code", "execution_count": 126, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([[ 4, 5],\n", " [ 8, 10],\n", " [12, 15]])" ] }, "execution_count": 126, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# To compute an outer product, we first reshape v to be a column\n", "# vector of shape (3, 1); we can then broadcast it against w to yield\n", "# an output of shape (3, 2), which is the outer product of v and w:\n", "\n", "# (3,1) * (2,) -> (3,1) * (1, 2) -> (3, 2) * (3, 2)\n", "np.reshape(v, (3, 1)) * w" ] }, { "cell_type": "code", "execution_count": 127, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "array([[2., 2., 2.],\n", " [2., 2., 2.]])" ] }, "execution_count": 127, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Multiply a matrix by a constant:\n", "x = np.ones((2,3))\n", "\n", "# x has shape (2, 3). Numpy treats scalars as arrays of shape ();\n", "# these can be broadcast together to shape (2, 3).\n", "\n", "# (2,3) * () -> (2,3) * (1,1) -> (2,3) * (2,3)\n", "x * 2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Broadcasting typically makes your code more concise and faster, so you should strive to use it where possible." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This brief overview has touched on many of the important things that you need to know about numpy, but is far from complete. Check out the [numpy reference](http://docs.scipy.org/doc/numpy/reference/) to find out much more about numpy." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## PyTorch Tensors" ] }, { "cell_type": "code", "execution_count": 128, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'1.3.0'" ] }, "execution_count": 128, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import torch\n", "torch.__version__" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "PyTorch is a relatively new yet widely used deep learning framework.\n", "\n", "During the course we'll use it extensively and learn many parts of its API.\n", "You should also familiarize yourself with the [PyTorch Documentation](https://pytorch.org/docs/stable/) as it will greatly assist you when implementing your own models." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This notebook will show only **a small part** of PyTorch's API, the `Tensor` class.\n", "This class is very similar to numpy's `ndarray`, and provides much of the same functionality.\n", "However, it also has two important distinctions:\n", "- Support for GPU computations.\n", "- Can store extra data needed for implementing back propagation:\n", " - A tensor of the same dimentions containing the gradient of this tensor w.r.t. some number (e.g. loss).\n", " - A node representing an operation in the computational graph that produced this tensor.\n", "\n", "In the next tutorials we will examine these concepts further." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This notebook will show some brief examples, just to get a feel for it and compare it to the usual numpy `ndarray`s.\n", "\n", "You will be using both PyTorch tensors and numpy `ndarray`s extensively throughout the course homework assignments, and in general when implementing deep learning algorithms." ] }, { "cell_type": "code", "execution_count": 129, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tensor([[1., 2., 3.],\n", " [4., 5., 6.]]) shape=torch.Size([2, 3]) dtype=torch.float32\n" ] } ], "source": [ "# Basic tensor creation\n", "t = torch.tensor([[1, 2, 3], [4, 5, 6]], dtype=torch.float32)\n", "\n", "print_arr(t)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Some other tensor construction methods:" ] }, { "cell_type": "code", "execution_count": 130, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[0., 0.]])" ] }, "execution_count": 130, "metadata": {}, "output_type": "execute_result" } ], "source": [ "torch.zeros((1,2))" ] }, { "cell_type": "code", "execution_count": 131, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[1., 1.],\n", " [1., 1.],\n", " [1., 1.]])" ] }, "execution_count": 131, "metadata": {}, "output_type": "execute_result" } ], "source": [ "torch.ones((3,2))" ] }, { "cell_type": "code", "execution_count": 132, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[6],\n", " [5],\n", " [1],\n", " [4]])" ] }, "execution_count": 132, "metadata": {}, "output_type": "execute_result" } ], "source": [ "torch.randint(10, (4,1))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As in numpy, tensors can be reshaped" ] }, { "cell_type": "code", "execution_count": 133, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[1., 2.],\n", " [3., 4.],\n", " [5., 6.]])" ] }, "execution_count": 133, "metadata": {}, "output_type": "execute_result" } ], "source": [ "t.reshape(3, 2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In many cases, PyTorch provides the same methods on both the `Tensor` object and via the `torch` package." ] }, { "cell_type": "code", "execution_count": 134, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[1., 2.],\n", " [3., 4.],\n", " [5., 6.]])" ] }, "execution_count": 134, "metadata": {}, "output_type": "execute_result" } ], "source": [ "torch.reshape(t, (3, -1))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As in numpy, many functions are provided for computations over tensors, for example," ] }, { "cell_type": "code", "execution_count": 135, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tensor([2., 5.]) shape=torch.Size([2]) dtype=torch.float32\n" ] } ], "source": [ "print_arr(torch.mean(t, dim=1))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sometime it's necessary to preserve original dimensions:" ] }, { "cell_type": "code", "execution_count": 136, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tensor([[2.],\n", " [5.]]) shape=torch.Size([2, 1]) dtype=torch.float32\n" ] } ], "source": [ "print_arr(torch.mean(t, dim=1, keepdim=True))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Properties of a tensor:" ] }, { "cell_type": "code", "execution_count": 137, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "shape: torch.Size([2, 3])\n", "dimensions: 2\n", "type: torch.FloatTensor\n", "device: cpu\n" ] } ], "source": [ "print('shape:', t.size()) # t.shape also works, like numpy\n", "print('dimensions:', t.dim())\n", "print('type:', t.type())\n", "print('device:', t.device)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Slicing** works just like in numpy:" ] }, { "cell_type": "code", "execution_count": 138, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[1., 2., 3.],\n", " [4., 5., 6.],\n", " [7., 8., 9.]])" ] }, "execution_count": 138, "metadata": {}, "output_type": "execute_result" } ], "source": [ "t = torch.Tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])\n", "t" ] }, { "cell_type": "code", "execution_count": 139, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([3., 6., 9.])" ] }, "execution_count": 139, "metadata": {}, "output_type": "execute_result" } ], "source": [ "t[:, -1]" ] }, { "cell_type": "code", "execution_count": 140, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[1., 2., 3.],\n", " [4., 5., 6.]])" ] }, "execution_count": 140, "metadata": {}, "output_type": "execute_result" } ], "source": [ "t[:2, :]" ] }, { "cell_type": "code", "execution_count": 141, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[9.]])" ] }, "execution_count": 141, "metadata": {}, "output_type": "execute_result" } ], "source": [ "t[-1:, -1:]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Unlike numpy, PyTorch tensors can be moved to **GPU** memory (if a GPU is available on the machine)." ] }, { "cell_type": "code", "execution_count": 142, "metadata": {}, "outputs": [], "source": [ "if torch.cuda.is_available():\n", " t = t.cuda()" ] }, { "cell_type": "code", "execution_count": 143, "metadata": {}, "outputs": [], "source": [ "# Better way to use GPU tensors (and later - models)\n", "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n", "\n", "t = t.to(device)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Broadcasting** also works (with almost identical semantics):" ] }, { "cell_type": "code", "execution_count": 144, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tensor([[ 1, 2, 3],\n", " [ 4, 5, 6],\n", " [ 7, 8, 9],\n", " [10, 11, 12]]) shape=torch.Size([4, 3]) dtype=torch.int64\n", "tensor([1, 0, 1]) shape=torch.Size([3]) dtype=torch.int64\n" ] } ], "source": [ "x = torch.tensor([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])\n", "v = torch.tensor([1, 0, 1])\n", "\n", "print_arr(x)\n", "print_arr(v)" ] }, { "cell_type": "code", "execution_count": 145, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[ 2, 2, 4],\n", " [ 5, 5, 7],\n", " [ 8, 8, 10],\n", " [11, 11, 13]])" ] }, "execution_count": 145, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y = x + v # Add v to each row of x using broadcasting\n", "y" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.4" } }, "nbformat": 4, "nbformat_minor": 4 }