{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "###### The cell below loads the visual style of the notebook when run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "css_file = '../../styles/styles.css'\n",
    "HTML(open(css_file, \"r\").read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Introducing Numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<section class=\"objectives panel panel-warning\">\n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-certificate\"></span>Learning Objectives</h2>\n",
    "</div>\n",
    "</section>\n",
    "\n",
    "> * How to import libraries, and where to find useful libraries. \n",
    "> * Explain why numpy is more useful than Python lists\n",
    "> * Use numpy for simple operations on arrays.\n",
    "> * Learn how to slice numpy arrays\n",
    "> * Read tabular data from a file into a program.\n",
    "\n",
    "---\n",
    "\n",
    "<section class=\"objectives panel panel-warning\">\n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-check\"></span>Prerequisites</h2>\n",
    "</div>\n",
    "</section>\n",
    "\n",
    "> Before starting this lecture, it is assumed you are familiar with\n",
    "> material in part 2 of the Python Bootcamp\n",
    "\n",
    "---\n",
    "\n",
    "## Numpy\n",
    "\n",
    "As a reminder, most of Python's functionatility comes from the use of [libraries](../../resources/reference.html#library). `numpy` is one such library that is designed to work with arrays of data, which are useful to represent vectors and matrices. `numpy` also provides routines to perform operations on the whole array in one line of code. We often import numpy using an alias to save typing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "The core of numpy is the `array` object. We'll start by looking at how numpy's arrays differ from a Python `list`. We can create a numpy array from a Python list like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "a_list = [1,2,3,4] # a python list\n",
    "an_array = np.array( a_list ) # one way to make an array\n",
    "an_array = np.array( [1,2,3,4] ) # another way\n",
    "\n",
    "print(an_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We saw in the [bootcamp](http://nbviewer.ipython.org/github/StuartLittlefair/python-teaching/blob/master/pybootcamp-part2/01-numpy.ipynb) how to access elements of a numpy array or a Python list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "print (a_list[0])\n",
    "print (an_array[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "print (a_list[1:3])\n",
    "print (an_array[1:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Differences between arrays and lists\n",
    "---\n",
    "\n",
    "What's the difference? To start with, Python's lists are very flexible and can contain different [types](../../resources/reference.html#type) of objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "a_list[-1] = 'a string inside a list'\n",
    "print(a_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "whereas numpy arrays can only contain a single type!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "an_array[-1] = 'a string inside an array'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Information about the type of object in a numpy array is stored in the `dtype` [member](../../resources/reference.html#member)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "an_array.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Once the array is created, the `dtype` is fixed, and an array can only hold that type. For example, if we try and put a [floating point number](../../resources/reference.html#floating-point-number) in our array, it gets converted to an [integer](../../resources/reference.html#integer):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "an_array[-1] = 1.3141\n",
    "print(an_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Why use arrays?\n",
    "---\n",
    "\n",
    "Right now, numpy arrays look less useful than Python lists! \n",
    "\n",
    "But numpy arrays are fast, and smart. Numpy has [methods](../../resources/reference.html#method) which allow fast computations on all elements of the array at the same time, and some other tricks up it's sleeve that we'll look at later.\n",
    "\n",
    "As a taster, let's compare two ways of calculating the sum of all the elements in a list or array. Make sure each function makes some sense to you before running the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "def sum_python(a_list):\n",
    "    '''Add up all the values in a list or array using plain Python'''\n",
    "    sum = 0.0\n",
    "    for thing in a_list:\n",
    "        sum += thing\n",
    "    return sum\n",
    "\n",
    "def sum_numpy(a_list):\n",
    "    '''Add up all the values in a list or array using numpy'''\n",
    "    return np.sum(a_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "my_list = np.random.randn(1000) # use numpy to create an array of 1000 random numbers\n",
    "%timeit sum_python(my_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "%timeit sum_numpy(my_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We can see that using numpy allows us to avoid ```for``` loops, and so leads to simpler code, but also that the numpy code is around 40 times faster!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<section class=\"challenge panel panel-success\"> \n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-pencil\"></span>Numpy vs List</h2>\n",
    "</div>\n",
    "</section>\n",
    "\n",
    "> Write two functions to calculate the mean of a list/array of numbers; one should use pure Python and the other should use numpy. Compare your answer to your neighbour. Are your answers similar? Which did you find easier to write?\n",
    "\n",
    "> Hint: the mean of a list of numbers is the sum of all the numbers, divided by the size of the list, so use the functions above as a template."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "# Write your own code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Creating arrays\n",
    "---\n",
    "\n",
    "We have seen that we can create arrays from lists. Let's look at a few more ways of creating arrays. We can create arrays filled with zeros or ones:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "zeros = np.zeros(5)\n",
    "ones  = np.ones(3)\n",
    "\n",
    "print(zeros)\n",
    "print(ones)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Numpy also offers the `arange` function which returns an array of evenly spaced values: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "np.arange(0, 5, 1) # min, max(exclusive), step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "and the very useful `linspace` function to create a linearly-spaced grid, with a fixed number of points and including both ends of the specified interval: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "print (\"A linear grid of 5 points between 0 and 2:\")\n",
    "print (np.linspace(0, 2, 5)) #min, max (inclusive), num_points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Arrays with more than one dimension\n",
    "---\n",
    "\n",
    "We've used 1D numpy arrays so far, but all the methods discussed work with arrays with 2 or more dimensions, e.g:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "np.zeros( (2,2) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "x = np.random.normal(loc=5, scale=2, size=(5,4))\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "You can create a 2D array from a list of lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "list_of_lists = [ [1,2,3], [4,5,6], [7,8,9] ] # each list is a row\n",
    "np.array(list_of_lists)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Arrays can even have their shape changed, as long as we don't change the total number of elements in the array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "x.reshape((10,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<section class=\"challenge panel panel-success\"> \n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-pencil\"></span>Views, not copies</h2>\n",
    "</div>\n",
    "</section>\n",
    "\n",
    "> Reshaping an array, like most numpy operations, provides a **view** of the *same bit* of computer memory. Without running it, what do you expect the code below to print? \n",
    "\n",
    "> ``` python\n",
    "> arr  = np.arange(8)\n",
    "> arr2 = arr.reshape(2, 4)\n",
    "> \n",
    "> arr[0] = 1000\n",
    "> print (arr)\n",
    "> print (arr2)\n",
    "> ```\n",
    "\n",
    "> Now run the code below. Did you get what you expected?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "arr  = np.arange(8)\n",
    "arr2 = arr.reshape(2, 4)\n",
    "\n",
    "arr[0] = 1000\n",
    "print (arr)\n",
    "print (arr2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "> This lack of copying might be confusing at times, but it makes numpy very efficient and fast."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Array methods and properties\n",
    "---\n",
    "\n",
    "One of the great things about numpy arrays is the many [methods](../../resources/reference.html#methods) that allow us to quickly get info about an array, or perform some operation on it. Here are some useful ones:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "print ('Data type                :', arr2.dtype)\n",
    "print ('Total number of elements :', arr2.size)\n",
    "print ('Number of dimensions     :', arr2.ndim)\n",
    "print ('Shape (dimensionality)   :', arr2.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "print ('Minimum and maximum             :', arr.min(), arr.max() )\n",
    "print ('Mean and standard deviation     :', arr.mean(), arr.std() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<section class=\"challenge panel panel-success\"> \n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-pencil\"></span>Array methods</h2>\n",
    "</div>\n",
    "</section>\n",
    "\n",
    "> Use the <kbd>Tab</kbd> completion feature of the Jupyter notebook to browse through all the methods belonging to the `arr2` object. Use the `argmax` method to find the index and value of the largest value in `arr2`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
   ],
   "source": [
    "# Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Array Operations\n",
    "---\n",
    "\n",
    "Arrays support all the normal math operators (+, -, / etc). The numpy library also contains a complete collection of basic mathematical functions that operate on arrays. In general, the math operators act on **every element of an array in turn**. Again, this means you don't have to use for loops to add all the numbers in two arrays together. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "arr1 = np.arange(4)\n",
    "arr2 = np.arange(10, 14)\n",
    "print (arr1, '+', arr2, '=', arr1+arr2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "In particular, multiplication is **not** matrix multiplication, but multiplies each element of the arrays together in turn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "print (arr1, '*', arr2, '=', arr1*arr2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We can also multiply arrays by scalars:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "print (arr1 * 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "This is a first example of *broadcasting*.\n",
    "\n",
    "In principle, to add two arrays their shapes must match. However, numpy will *broadcast* the shapes to match if possible. Broadcasting is a bit beyond the level of this course, but if you find yourself needing to understand it, it is described in the companion notebook ```01a-broadcasting.ipynb```.\n",
    "\n",
    "As we mentioned before, numpy ships with many mathematical functions that work on entire arrays, including logarithms, exponentials, and trigonometric functions. For example, calculating the sine function at 100 points between $0$ and $2\\pi$ is as simple as: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "x = np.linspace(0.0,2.0*np.pi,100)\n",
    "y = np.sin(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Using Numpy to do calculations on lots of values at once is a great way to *speed up your code* and make it *simpler to understand*. If you ever find yourself looping over an array to do some calculation with each element in turn, stop, and see if Numpy can do the hard work for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Linear Algebra in numpy\n",
    "\n",
    "numpy has a `dot` method which gives the dot product between two vectors (one-dimensional arrays), or matrix multiplication when one or both of its arguments are matrices (two-dimensional arrays): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "v1 = np.array([2, 3, 4])\n",
    "v2 = np.array([1, 0, 1])\n",
    "result = np.dot(v1,v2)\n",
    "\n",
    "print (v1, '.', v2, '=', result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Here is a regular matrix-vector multiplication, note that the array `v1` should be viewed as a column vector in traditional linear algebra notation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "A = np.arange(6).reshape(2, 3)\n",
    "result = np.dot(A, v1)\n",
    "\n",
    "print (A, 'x', v1, '=', result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We can calculate transposes of arrays with `arr.T`, and `np.linalg` module contains methods to find eigenvectors, determinants and much more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<section class=\"challenge panel panel-success\"> \n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-pencil\"></span>Numpy Maths Challenge</h2>\n",
    "</div>\n",
    "</section>\n",
    "\n",
    "> A system of linear equations can be written in matrix form.\n",
    ">\n",
    "> $$3x + 6y - 5z = 12$$\n",
    "> $$x - 3y + 2z  = -2$$\n",
    "> $$5x - y + 4z  = 10$$\n",
    ">\n",
    "> $$\\left[\\begin{matrix} 3 & 6 & -5 \\\\ 1 & -3 & 2 \\\\ 5 & -1 & 4 \\end{matrix}\\right]\n",
    " \\left[\\begin{matrix} x \\\\ y \\\\ z \\end{matrix}\\right] = \\left[\\begin{matrix} 12 \\\\ -2\\\\ 10 \\end{matrix}\\right]$$\n",
    ">\n",
    "> Now, representing the matrix system as $\\mathbf{AX} = \\mathbf{B}$, we can see the solution is given by $\\mathbf{X} = \\mathbf{A^{-1}B}$. \n",
    ">\n",
    "> Write some numpy code to find the solution of these equations.\n",
    ">\n",
    "> Hint: `np.linalg.inv` can be used to find a matrices inverse..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "# Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Numpy slicing\n",
    "---\n",
    "\n",
    "[Slicing](../../resources/reference.html#slice) is the term used to refer to extracting part of an array. We already saw some basic slicing in the [bootcamp](http://nbviewer.ipython.org/github/StuartLittlefair/python-teaching/blob/master/pybootcamp-part2/01-numpy.ipynb). As a quick reminder, we use the syntax `arr[lower:upper:step]` like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "A = np.array([1,2,3,4,5,6,7,8])\n",
    "A[1:6:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We can omit any part of the `[lower:upper:step]` notation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "print (A[2:]) #from index 2 to the end, assumed step=1\n",
    "print (A[:3]) # from index 0 to 3\n",
    "print (A[::2]) # from 0 until the end, in steps of 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "And slicing works the same way for arrays with more dimensions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "B = np.arange(36).reshape((6,6))\n",
    "print (B)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "print (B[1:3]) # just rows 1-3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "print (B[1:3, 1:3]) # columns 1-3 and rows 1-3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "print (B[1, :])  # row 1, all columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "print (B[:, 0])  # column 0 (the first column), and all rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Fancy slicing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We can use numpy arrays or python lists to slice numpy arrays as well! This is an extremely powerful technique."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "row_indices = [1, 3, 5]\n",
    "print (B[row_indices])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Why is this so useful? We can also use arrays of `True` and `False` values, known as *masks* as an index. If the mask is `True`, the element will be selected. For example, we can calculate a mask to see if each element of B is odd:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "is_odd = (B % 2 != 0) # does each element of B divide by two with no remainder?\n",
    "print (is_odd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "And use that mask to select only the odd values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "print( B[is_odd] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "You can negate a mask (switch True and False) with `~`, so to find the even numbers we use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "print( B[~is_odd] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<section class=\"challenge panel panel-success\"> \n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-pencil\"></span>Fancy slicing</h2>\n",
    "</div>\n",
    "</section>\n",
    "\n",
    "> Use `np.arange` to make a 1D array of values from 0 to 100 (inclusive) and calculate the sum of only the even values using fancy slicing. Compare your answer to your neighbour. Do you get the same answer? Is your code the same?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "# Write your code here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Reading and writing data to files\n",
    "---\n",
    "\n",
    "It is perfectly possible to read and write files without using numpy. Files are opened using the `open` function. This returns a [file object](../../resources/reference.html#file-object) which is used to read or write the file. It is most often used with two arguments\n",
    "\n",
    "```python\n",
    "file_handle = open(filename,mode)\n",
    "```\n",
    "\n",
    "The second argument is a string containing a few characters describing the way in which the file will be used. mode can be `r` when the file will only be read, `w` for only writing (an existing file with the same name will be erased), and `a` opens the file for appending; any data written to the file is automatically added to the end. The mode argument is optional, if it is missing, `r` is assumed.\n",
    "\n",
    "For example, I have a file with some basic data on temperature in Stockholm. Below I show how you would open a file and read in the lines in pure Python. Notice how the ```open``` function gets what we call a *relative path* for the file; this is how to get to the file from the location of the notebook on the filesystem. The symbol \"..\" means \"go up one directory\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
   ],
   "source": [
    "file_handle = open('../../data/Session1/td_stockholm.dat','r') # open file for reading\n",
    "lines = file_handle.readlines() # read in all the lines into a list\n",
    "for line in lines[0:5]: # iterate over the first few lines\n",
    "    print(line) # print out line\n",
    "\n",
    "file_handle.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "You can see this is a space separated text file. If we wanted to convert this data to a numpy array we can imagine writing the code to do so. It would:\n",
    "\n",
    "* read in the lines into a list\n",
    "* for each line in the list\n",
    " * skip header lines if necessary\n",
    " * split lines into fields at the whitespace\n",
    " * put the fields into a numpy array\n",
    " \n",
    "This code is obviously going to be very similar for each file and it would make sense to write  a function you could re-use. But the first lesson of Python club is to see if someone else has done it for you! In this case they have, and we can look at `np.loadtxt`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "np.loadtxt?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Looking at the help for this function, we can see that we can read in the file above with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "data = np.loadtxt('../../data/Session1/td_stockholm.dat', skiprows=1)\n",
    "print(data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We've got our 7 columns and 77431 rows. We can use slicing to access individual columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "year = data[:, 0] # first column\n",
    "month = data[:, 1]\n",
    "day = data[:, 2]\n",
    "temperature = data[:, 5]\n",
    "# calculate a 'decimal year' for each observation\n",
    "# NOTICE HOW NUMPY ALLOWS US TO CALCULATE EVERY ROW IN ONE LINE.\n",
    "obs_time = year + month/12 + day/31"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "And plot the data (plotting is discussed in another lecture):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "fig, ax = plt.subplots(figsize=(14,4))\n",
    "ax.plot(obs_time,temperature)\n",
    "ax.axis('tight')\n",
    "ax.set_title('tempeatures in Stockholm')\n",
    "ax.set_xlabel('year')\n",
    "ax.set_ylabel('tempature (C)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Writing files\n",
    "---\n",
    "\n",
    "Suppose we want to save our data to a file? `np.savetxt` will take a 2D numpy array and save it to a simple text file. We can stack our 1D columns into a 2D array with `np.column_stack`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
   ],
   "source": [
    "data_out = np.column_stack([obs_time, temperature])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "And we can save this 2D array to a comma-seperated value file with `np.savetxt`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
   ],
   "source": [
    "np.savetxt('output.csv', data_out, delimiter=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "!head output.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<section class=\"challenge panel panel-success\"> \n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-pencil\"></span>File IO</h2>\n",
    "</div>\n",
    "</section>\n",
    "\n",
    "> Read the help file for `np.loadtxt` and try and read only the minimum and maximum\n",
    "> temperature from the file. Place the data *directly* in two numpy arrays. Your call\n",
    "> should start like\n",
    "> ```python\n",
    "> temp_min, temp_max = np.loadtxt(...\n",
    "> ```\n",
    "> Save the min and max temp to a new file in a format of your choice using `np.savetxt`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
   ],
   "source": [
    "#Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "---------\n",
    "<section class=\"challenge panel panel-success\"> \n",
    "<div class=\"panel-heading\">\n",
    "<h1>Homework #1</h1>\n",
    "<h2><span class=\"fa fa-pencil\"></span>Advanced File IO and Slicing</h2>\n",
    "</div>\n",
    "</section>\n",
    "\n",
    "> In the data directory is a file named 'sdss_wds.csv'. This is a comma-seperated values file of data about white dwarfs discovered in the Sloan Digital Sky Survey. Your job is to investigate how the median mass of the white dwarfs depends on temperature.\n",
    "\n",
    ">The code below prints out the first few lines of the file. As you can see, there is one header row which tells us what the columns are."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "lines = open('../../data/Session1/sdss_wds.csv', 'r').readlines()\n",
    "for line in lines[0:4]:\n",
    "    print (line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<section class=\"challenge panel panel-success\"> \n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-question\"></span> Question 1 (1 points)</h2>\n",
    "</div>\n",
    "</section> \n",
    "\n",
    "> Look at the documentation for `np.loadtxt` and work out how to read in the file above. \n",
    "\n",
    "> Use `np.loadtxt` to load the white dwarf file into an array called `data`.\n",
    "\n",
    "> **If you are successful, your code will pass the tests below each question. You can also use these tests to see if your code gives the right answer.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false,
    "deletable": false
   },
   "outputs": [
   ],
   "source": [
    "# WRITE YOUR OWN CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false,
    "deletable": false
   },
   "outputs": [
   ],
   "source": [
    "from nose.tools import assert_tuple_equal, assert_false, assert_true  \n",
    "assert_tuple_equal(data.shape, (13673, 6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<section class=\"challenge panel panel-success\"> \n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-question\"></span>Question 2 (4 points)</h2>\n",
    "</div>\n",
    "</section> \n",
    "\n",
    "> `np.loadtxt` has read in our text files, and we have placed the contents into a 2D array called \"data\". Complete the function outline below to create a function which will take this 2D array and return a user-specified column. The function should return a 1D array of the column values.\n",
    "\n",
    "> Get into the habit of reading the documentation at the start of the function to understand how it should work, and what the function [arguments](../../resources/reference.html#argument) should be.\n",
    "\n",
    "> Use [defensive programming](../../resources/reference.html#defensive-programming) and assert statements to make sure that the function only accepts valid column numbers.\n",
    "\n",
    "> Again, try and get your code to pass the tests that I have written in the cells below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false,
    "deletable": false
   },
   "outputs": [
   ],
   "source": [
    "def get_column(data, col_num):\n",
    "    \"\"\"Filter a numpy array and return the requested column.\n",
    "\n",
    "    Given a 2D numpy array, and a column number, this function returns the \n",
    "    requested column number as a 1D numpy array.\n",
    "\n",
    "    Args:\n",
    "        data (numpy.ndarray): the array to filter\n",
    "        col_num (int): the column number to return (e.g `0` for the first column in the array).\n",
    "    \"\"\"\n",
    "    # here's an example of using defensive programming to make sure data is a 2D array\n",
    "    # notice how we run these checks before doing anything else\n",
    "    assert(len(data.shape) == 2), \"data must be 2D array\"\n",
    "    # ADD YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false,
    "deletable": false
   },
   "outputs": [
   ],
   "source": [
    "from nose.tools import assert_equal, assert_almost_equal\n",
    "teff = get_column(data, 4)\n",
    "mass = get_column(data, 5)\n",
    "assert_equal(len(teff), 13673)\n",
    "assert_equal(len(mass), 13673)\n",
    "assert_almost_equal(teff.mean(), 17103.082278943904)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false,
    "deletable": false
   },
   "outputs": [
   ],
   "source": [
    "from nose.tools import assert_raises\n",
    "assert_raises(AssertionError, get_column, data, -1 )\n",
    "assert_raises(AssertionError, get_column, data, 6 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<section class=\"challenge panel panel-success\"> \n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-question\"></span>Question 3 (1 points)</h2>\n",
    "</div>\n",
    "</section>\n",
    "\n",
    "> What is the median mass of the white dwarfs found in the Sloan Digital Sky Survey? Store your answer in a variable named `median_mass`. Store the mean mass in a variable named `mean_mass`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false,
    "deletable": false
   },
   "outputs": [
   ],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false,
    "deletable": false
   },
   "outputs": [
   ],
   "source": [
    "from nose.tools import assert_almost_equal\n",
    "assert_almost_equal(median_mass, 0.5939999999999999)\n",
    "assert_almost_equal(mean_mass, 0.62164258026768093)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<section class=\"challenge panel panel-success\"> \n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-question\"></span>Question 4 (2 points)</h2>\n",
    "</div>\n",
    "</section>\n",
    "\n",
    "> Why do the mean and median masses differ? What does this tell you about the *shape* of the mass distribution of white dwarfs discovered in the Sloan Digital Sky Survey?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "deletable": false
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<section class=\"challenge panel panel-success\"> \n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-question\"></span>Question 5 (2 points)</h2>\n",
    "</div>\n",
    "</section>\n",
    "\n",
    "> Use fancy slicing to find the mean mass of white dwarfs cooler than 12,000K. Store your answer in the variable ```mean_mass_coolwd```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
   ],
   "source": [
    "from nose.tools import assert_almost_equal\n",
    "assert_almost_equal(mean_mass_coolwd, 0.67178046054102403)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<section class=\"panel panel-warning\"> \n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-question\"></span>Extra Credit (2 points)</h2>\n",
    "</div>\n",
    "</section>\n",
    "\n",
    "> Extra credit questions allow you to make up for marks dropped in this and other homeworks. You can't score more than 100% overall, but if you get 2 extra credit points this week, and lose 2 points next week, you'd still be on course for 100% marks. I don't expect you to answer extra credit questions, *unless you want to*.\n",
    "\n",
    "> For extra credit this week, use the techniques you've learned above and your own statistical knowledge to justify whether the cool white dwarfs are *significantly* different in mass to the hotter white dwarfs. Use the code cell to make any calculations you need, and the markdown cell to justify your answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "collapsed": false,
    "deletable": false
   },
   "outputs": [
   ],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "deletable": false
   },
   "source": [
    "### Write your answer here"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (system-wide)",
   "language": "python",
   "metadata": {
    "cocalc": {
     "description": "Python 3 programming language",
     "priority": 100,
     "url": "https://www.python.org/"
    }
   },
   "name": "python3",
   "resource_dir": "/ext/jupyter/kernels/python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}