{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced Numpy Techniques\n",
    "\n",
    "<img src=\"assets/numpylogo.png\" alt=\"http://www.numpy.org/#\">\n",
    "\n",
    "General, user-friendly [documentation](https://docs.scipy.org/doc/numpy/index.html) with lots of examples.\n",
    "\n",
    "Technical, \"hard\" [reference](https://docs.scipy.org/doc/numpy/reference/index.html#reference).\n",
    "\n",
    "Basic Python knowledge assumed.\n",
    "\n",
    "CPython ~3.6, NumPy ~1.12\n",
    "\n",
    "If you like content like this, you might be interested in [my blog](https://vlad17.github.io/)\n",
    "\n",
    "## What is it?\n",
    "\n",
    "[NumPy](http://www.numpy.org/) is an open-source package that's part of the [SciPy](https://scipy.org/) ecosystem. Its main feature is an array object of arbitrary dimension, but this fundamental collection is integral to any data-focused Python application.\n",
    "\n",
    "\n",
    "<table>\n",
    "<tr>\n",
    "<th>\n",
    "<img src=\"assets/nonsteeplearn.png\" width=\"200\" alt=\"http://gabriellhanna.blogspot.com/2015/03/negatively-accelerated-learning-curve-i.html\">\n",
    "</th><th>\n",
    "<img src=\"assets/steeplearn.jpg\" width=\"200\" alt=\"http://malaher.org/2007/03/pet-peeve-learning-curve-misuse/\">\n",
    "</th></tr></table>\n",
    "\n",
    "Most people learn numpy through assimilation or necessity. I believe NumPy has the latter learning curve (steep/easy to learn), so you can actually invest just a little bit of time now (by going through this notebook, for instance), and reap a lot of reward!\n",
    "\n",
    "# Motivation\n",
    "\n",
    "* Provide a uniform interface for handling numerical structured data\n",
    "* Collect, store, and manipulate numerical data efficiently\n",
    "* Low-cost abstractions\n",
    "* Universal glue for numerical information, used in lots of external libraries! The API establishes common functions and re-appears in many other settings with the same abstractions.\n",
    "\n",
    "\n",
    "<table>\n",
    "<tr>\n",
    "<th>\n",
    "<img src=\"assets/numba.png\" alt=\"http://numba.pydata.org/\" width=\"150\"></th><th><img src=\"assets/pandas.png\" alt=\"http://pandas.pydata.org/\"  width=\"150\"> </th><th><img src=\"assets/tf.png\" alt=\"https://github.com/tensorflow/tensorflow\" width=\"150\"></th><th> <img src=\"assets/sklearn.png\" alt=\"https://github.com/scikit-learn/scikit-learn\" width=\"150\"> </th><th><img src=\"assets/stan.png\" alt=\"http://mc-stan.org/\" width=\"150\"></th>\n",
    "</tr>\n",
    "</table>\n",
    "\n",
    "\n",
    "\n",
    "# Goals and Non-goals\n",
    "\n",
    "### Goals\n",
    "\n",
    "What I'll do:\n",
    "\n",
    "* Give a bit of basics first.\n",
    "* Describe NumPy, with under-the-hood details to the extent that they are useful to you, the user\n",
    "* Highlight some [GOTCHA]s, avoid some common bugs\n",
    "* Point out a couple useful NumPy functions\n",
    "\n",
    "**This is not an attempt to exaustively cover the reference manual (there's too many individual functions to keep in your head, anyway).**\n",
    "\n",
    "Instead, I'll try to...\n",
    "\n",
    "* provide you with an overview of the API structure so next time you're doing numeric data work you'll know where to look\n",
    "* convince you that NumPy arrays offer the perfect data structure for the following (wide-ranging) use case:\n",
    "\n",
    "**RAM-sized general-purpose structured numerical data applications: manipulation, collection, and analysis.**\n",
    "\n",
    "### Non-goals\n",
    "\n",
    "* No emphasis on multicore processing, but will be briefly mentioned\n",
    "* Some NumPy functionality not covered -- mentioned briefly at end\n",
    "* HPC concerns\n",
    "* GPU programming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why not a Python list?\n",
    "\n",
    "A list is a resizing contiguous array of pointers.\n",
    "\n",
    "<img src=\"assets/pylist.png\" alt=\"http://www.laurentluce.com/posts/python-list-implementation/\">\n",
    "\n",
    "Nested lists are even worse - there are two levels of indirection.\n",
    "\n",
    "<img src=\"assets/nestlist.png\" alt=\"http://www.cs.toronto.edu/~gpenn/csc401/401_python_web/pyseq.html\">\n",
    "\n",
    "Compare to NumPy arrays, happy contiguous chunks of memory, even across axes. This image is only illustrative, a NumPy array may not necessarily be in C-order (more on that later):\n",
    "\n",
    "<img src=\"assets/nparr.png\" alt=\"https://www.safaribooksonline.com/library/view/python-for-data/9781491957653/ch04.html\" width=300>\n",
    "\n",
    "**Recurring theme**: NumPy lets us have the best of both worlds (high-level Python for development, optimized representation and speed via low-level C routines for execution)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import time\n",
    "import gc\n",
    "import sys\n",
    "\n",
    "assert sys.maxsize > 2 ** 32, \"get a new computer!\"\n",
    "\n",
    "# Allocation-sensitive timing needs to be done more carefully\n",
    "# Compares runtimes of f1, f2\n",
    "def compare_times(f1, f2, setup1=None, setup2=None, runs=5):\n",
    "    print('    format: mean seconds (standard error)', runs, 'runs')\n",
    "    maxpad = max(len(f.__name__) for f in (f1, f2))\n",
    "    means = []\n",
    "    for setup, f in [[setup1, f1], [setup2, f2]]:\n",
    "        setup = (lambda: tuple()) if setup is None else setup\n",
    "        \n",
    "        total_times = []\n",
    "        for _ in range(runs):\n",
    "            try:\n",
    "                gc.disable()\n",
    "                args = setup()\n",
    "                \n",
    "                start = time.time()\n",
    "                if isinstance(args, tuple):\n",
    "                    f(*args)\n",
    "                else:\n",
    "                    f(args)\n",
    "                end = time.time()\n",
    "                \n",
    "                total_times.append(end - start)\n",
    "            finally:\n",
    "                gc.enable()\n",
    "                \n",
    "        mean = np.mean(total_times)\n",
    "        se = np.std(total_times) / np.sqrt(len(total_times))\n",
    "        print('    {} {:.2e} ({:.2e})'.format(f.__name__.ljust(maxpad), mean, se))\n",
    "        means.append(mean)\n",
    "    print('    improvement ratio {:.1f}'.format(means[0] / means[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bandwidth-limited ops\n",
    "\n",
    "* Have to pull in more cache lines for the pointers\n",
    "* Poor locality causes pipeline stalls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "create a list 1, 2, ... 10000000\n",
      "    format: mean seconds (standard error) 5 runs\n",
      "    create_list  2.86e-01 (8.22e-03)\n",
      "    create_array 3.47e-02 (1.86e-04)\n",
      "    improvement ratio 8.2\n"
     ]
    }
   ],
   "source": [
    "size = 10 ** 7 # ints will be un-intered past 258\n",
    "print('create a list 1, 2, ...', size)\n",
    "\n",
    "\n",
    "def create_list(): return list(range(size))\n",
    "def create_array(): return np.arange(size, dtype=int)\n",
    "\n",
    "compare_times(create_list, create_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deep copies (no pre-allocation)\n",
      "    format: mean seconds (standard error) 5 runs\n",
      "    copy_list  7.64e-02 (9.16e-04)\n",
      "    copy_array 3.29e-02 (7.43e-04)\n",
      "    improvement ratio 2.3\n"
     ]
    }
   ],
   "source": [
    "print('deep copies (no pre-allocation)') # Shallow copy is cheap for both!\n",
    "size = 10 ** 7\n",
    "\n",
    "ls = list(range(size))\n",
    "def copy_list(): return ls[:]\n",
    "\n",
    "ar = np.arange(size, dtype=int)\n",
    "def copy_array(): return np.copy(ar)\n",
    "\n",
    "compare_times(copy_list, copy_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deep copy (pre-allocated)\n",
      "    format: mean seconds (standard error) 5 runs\n",
      "    deep_copy_lists  8.57e-02 (2.97e-03)\n",
      "    deep_copy_arrays 3.09e-02 (5.53e-04)\n",
      "    improvement ratio 2.8\n"
     ]
    }
   ],
   "source": [
    "print('Deep copy (pre-allocated)')\n",
    "size = 10 ** 7\n",
    "\n",
    "def create_lists(): return list(range(size)), [0] * size\n",
    "def deep_copy_lists(src, dst): dst[:] = src\n",
    "\n",
    "def create_arrays(): return np.arange(size, dtype=int), np.empty(size, dtype=int)\n",
    "def deep_copy_arrays(src, dst): dst[:] = src\n",
    "\n",
    "compare_times(deep_copy_lists, deep_copy_arrays, create_lists, create_arrays)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Flop-limited ops\n",
    "\n",
    "* Can't engage VPU on non-contiguous memory: won't saturate CPU computational capabilities of your hardware (note that your numpy may not be vectorized anyway, but the \"saturate CPU\" part still holds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "square out-of-place\n",
      "    format: mean seconds (standard error) 5 runs\n",
      "    square_lists  9.07e-01 (5.35e-03)\n",
      "    square_arrays 2.34e-02 (3.51e-04)\n",
      "    improvement ratio 38.7\n"
     ]
    }
   ],
   "source": [
    "print('square out-of-place')\n",
    "\n",
    "def square_lists(src, dst):\n",
    "    for i, v in enumerate(src):\n",
    "        dst[i] = v * v\n",
    "\n",
    "def square_arrays(src, dst):\n",
    "    np.square(src, out=dst)\n",
    "    \n",
    "compare_times(square_lists, square_arrays, create_lists, create_arrays)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "square in-place\n",
      "    format: mean seconds (standard error) 5 runs\n",
      "    square_list  8.63e-01 (1.84e-02)\n",
      "    square_array 7.61e-03 (5.08e-04)\n",
      "    improvement ratio 113.4\n"
     ]
    }
   ],
   "source": [
    "# Caching and SSE can have huge cumulative effects\n",
    "\n",
    "print('square in-place')\n",
    "size = 10 ** 7\n",
    "\n",
    "def create_list(): return list(range(size))\n",
    "def square_list(ls):\n",
    "    for i, v in enumerate(ls):\n",
    "        ls[i] = v * v\n",
    "\n",
    "def create_array(): return np.arange(size, dtype=int)\n",
    "def square_array(ar):\n",
    "    np.square(ar, out=ar)\n",
    "    \n",
    "compare_times(square_list, square_array, create_list, create_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Memory consumption\n",
    "\n",
    "List representation uses 8 extra bytes for every value (assuming 64-bit here and henceforth)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "list  kb 400\n",
      "array kb 78\n"
     ]
    }
   ],
   "source": [
    "from pympler import asizeof\n",
    "size = 10 ** 4\n",
    "\n",
    "print('list  kb', asizeof.asizeof(list(range(size))) // 1024)\n",
    "print('array kb', asizeof.asizeof(np.arange(size, dtype=int)) // 1024)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Disclaimer\n",
    "\n",
    "Regular python lists are still useful! They do a lot of things arrays can't:\n",
    "\n",
    "* List comprehensions `[x * x for x in range(10) if x % 2 == 0]`\n",
    "* Ragged nested lists `[[1, 2, 3], [1, [2]]]`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The NumPy Array\n",
    "\n",
    "[doc](https://docs.scipy.org/doc/numpy/reference/arrays.ndarray.html#internal-memory-layout-of-an-ndarray)\n",
    "\n",
    "### Abstraction\n",
    "\n",
    "We know what an array is -- a contiugous chunk of memory holding an indexed list of things from 0 to its size minus 1. If the things have a particular type, using, say, `dtype` as a placeholder, then we can refer to this as a `classical_array` of `dtype`s.\n",
    "\n",
    "The NumPy array, an `ndarray` with a _datatype, or dtype,_ `dtype` is an _N_-dimensional array for arbitrary _N_. This is defined recursively:\n",
    "* For _N > 0_, an _N_-dimensional `ndarray` of _dtype_ `dtype` is a `classical_array` of _N - 1_ dimensional `ndarray`s of _dtype_ `dtype`, all with the same size.\n",
    "* For _N = 0_, the `ndarray` is a `dtype`\n",
    "\n",
    "We note some familiar special cases:\n",
    "* _N = 0_, we have a scalar, or the datatype itself\n",
    "* _N = 1_, we have a `classical_array`\n",
    "* _N = 2_, we have a matrix\n",
    "\n",
    "Each _axis_ has its own `classical_array` length: this yields the shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ndim 0 shape ()\n",
      "3.0\n",
      "ndim 1 shape (4,)\n",
      "[3. 3. 3. 3.]\n",
      "ndim 2 shape (2, 4)\n",
      "[[3. 3. 3. 3.]\n",
      " [3. 3. 3. 3.]]\n",
      "ndim 3 shape (2, 2, 4)\n",
      "[[[3. 3. 3. 3.]\n",
      "  [3. 3. 3. 3.]]\n",
      "\n",
      " [[3. 3. 3. 3.]\n",
      "  [3. 3. 3. 3.]]]\n"
     ]
    }
   ],
   "source": [
    "n0 = np.array(3, dtype=float)\n",
    "n1 = np.stack([n0, n0, n0, n0])\n",
    "n2 = np.stack([n1, n1])\n",
    "n3 = np.stack([n2, n2])\n",
    "\n",
    "for x in [n0, n1, n2, n3]:\n",
    "    print('ndim', x.ndim, 'shape', x.shape)\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Axes are read LEFT to RIGHT: an array of shape `(n0, n1, ..., nN-1)` has axis `0` with length `n0`, etc.**\n",
    "\n",
    "### Detour: Formal Representation\n",
    "\n",
    "**Warning, these are pretty useless definitions unless you want to understand [np.einsum](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.einsum.html), which is only at the end anyway.**\n",
    "\n",
    "Formally, a NumPy array can be viewed as a mathematical object. If:\n",
    "\n",
    "* The `dtype` belongs to some (usually field) $F$\n",
    "* The array has dimension $N$, with the $i$-th axis having length $n_i$\n",
    "* $N>1$\n",
    "\n",
    "Then this array is an object in:\n",
    "\n",
    "$$\n",
    "F^{n_0}\\otimes F^{n_{1}}\\otimes\\cdots \\otimes F^{n_{N-1}}\n",
    "$$\n",
    "\n",
    "$F^n$ is an $n$-dimensional vector space over $F$. An element in here can be represented by its canonical basis $\\textbf{e}_i^{(n)}$ as a sum for elements $f_i\\in F$:\n",
    "\n",
    "$$\n",
    "f_1\\textbf{e}_1^{(n)}+f_{2}\\textbf{e}_{2}^{(n)}+\\cdots +f_{n}\\textbf{e}_{n}^{(n)}\n",
    "$$\n",
    "\n",
    "$F^n\\otimes F^m$ is a tensor product, which takes two vector spaces and gives you another. Then the tensor product is a special kind of vector space with dimension $nm$. Elements in here have a special structure which we can tie to the original vector spaces $F^n,F^m$:\n",
    "\n",
    "$$\n",
    "\\sum_{i=1}^n\\sum_{j=1}^mf_{ij}(\\textbf{e}_{i}^{(n)}\\otimes \\textbf{e}_{j}^{(m)})\n",
    "$$\n",
    "\n",
    "Above, $(\\textbf{e}_{i}^{(n)}\\otimes \\textbf{e}_{j}^{(m)})$ is a basis vector of $F^n\\otimes F^m$ for each pair $i,j$.\n",
    "\n",
    "We will discuss what $F$ can be later; but most of this intuition (and a lot of NumPy functionality) is based on $F$ being a type corresponding to a field.\n",
    "\n",
    "# Back to CS / Mutability / Losing the Abstraction\n",
    "\n",
    "The above is a (simplified) view of `ndarray` as a tensor, but gives useful intuition for arrays that are **not mutated**.\n",
    "\n",
    "An `ndarray` **Python object** is a actually a _view_ into a shared `ndarray`. The _base_ is a representative of the equaivalence class of views of the same array\n",
    "\n",
    "<img src=\"assets/ndarrayrep.png\" alt=\"https://docs.scipy.org/doc/numpy/reference/arrays.html\">\n",
    "\n",
    "**This diagram is a lie (the array isn't in your own bubble, it's shared)!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 2 3 4 5 6 7 8 9]\n"
     ]
    }
   ],
   "source": [
    "original = np.arange(10)\n",
    "\n",
    "# shallow copies\n",
    "s1 = original[:]\n",
    "s2 = s1.view()\n",
    "s3 = original[:5]\n",
    "\n",
    "print(original)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s1 [ 0  1 -1  3  4  5  6  7  8  9]\n",
      "s2 [ 0  1 -1  3  4  5  6  7  8  9]\n",
      "s3 [ 0  1 -1  3  4]\n"
     ]
    }
   ],
   "source": [
    "original[2] = -1\n",
    "print('s1', s1)\n",
    "print('s2', s2)\n",
    "print('s3', s3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(140237625897248, 140237625897248, 140237625897248, 140237625897248, None)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id(original), id(s1.base), id(s2.base), id(s3.base), original.base"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dtypes\n",
    "\n",
    "$F$ (our `dtype`) can be ([doc](https://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html)):\n",
    "\n",
    "* boolean\n",
    "* integral\n",
    "* floating-point\n",
    "* complex floating-point\n",
    "* any structure ([record array](https://docs.scipy.org/doc/numpy/user/basics.rec.html)) of the above, e.g. [complex integral values](http://stackoverflow.com/questions/13863523/is-it-possible-to-create-a-numpy-ndarray-that-holds-complex-integers)\n",
    "\n",
    "The `dtype` can also be unicode, a date, or an arbitrary object, but those don't form fields. This means that most NumPy functions aren't usful for this data, since it's not numeric. Why have them at all?\n",
    "\n",
    "* for all: NumPy `ndarray`s offer the tensor abstraction described above.\n",
    "* unicode: consistent format in memory for bit operations and for I/O\n",
    "* [date](https://docs.scipy.org/doc/numpy/reference/arrays.datetime.html): compact representation, addition/subtraction, basic parsing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "i16 296 i64 896\n"
     ]
    }
   ],
   "source": [
    "# Names are pretty intuitive for basic types\n",
    "\n",
    "i16 = np.arange(100, dtype=np.uint16)\n",
    "i64 = np.arange(100, dtype=np.uint64)\n",
    "print('i16', asizeof.asizeof(i16), 'i64', asizeof.asizeof(i64))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(1,  1) (2, -1)]\n",
      "1+1i\n",
      "2-1i\n"
     ]
    }
   ],
   "source": [
    "# We can use arbitrary structures for our own types\n",
    "# For example, exact Gaussian (complex) integers\n",
    "\n",
    "gauss = np.dtype([('re', np.int32), ('im', np.int32)])\n",
    "c2 = np.zeros(2, dtype=gauss)\n",
    "c2[0] = (1, 1)\n",
    "c2[1] = (2, -1)\n",
    "\n",
    "def print_gauss(g):\n",
    "    print('{}{:+d}i'.format(g['re'], g['im']))\n",
    "          \n",
    "print(c2)\n",
    "for x in c2:\n",
    "    print_gauss(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'\\x00\\x05' 0000000000000101\n",
      "b'\\x05\\x00' 0000000000000101\n"
     ]
    }
   ],
   "source": [
    "l16 = np.array(5, dtype='>u2') # little endian signed char\n",
    "b16 = l16.astype('<u2') # big endian unsigned char\n",
    "print(l16.tobytes(), np.binary_repr(l16, width=16))\n",
    "print(b16.tobytes(), np.binary_repr(b16, width=16))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Indexing [doc](https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html)\n",
    "\n",
    "Probably the most creative, unique part of the entire library. This is what makes NumPy `ndarray` better than any other array.\n",
    "\n",
    "And index returns an `ndarray` view based on the other `ndarray`.\n",
    "\n",
    "### Basic Indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 2 3 4 5 6 7 8 9]\n",
      "[2 4]\n",
      "140237625830256 140237625830256\n"
     ]
    }
   ],
   "source": [
    "x = np.arange(10)\n",
    "\n",
    "# start:stop:step\n",
    "# inclusive start, exclusive stop\n",
    "print(x)\n",
    "print(x[2:6:2])\n",
    "print(id(x), id(x[2:6:2].base))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 2]\n",
      "[7 8 9]\n"
     ]
    }
   ],
   "source": [
    "# Default start is 0, default end is length, default step is 1\n",
    "print(x[:3])\n",
    "print(x[7:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 2 3 4 5 6 7 8 9]\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "# Don't worry about overshooting\n",
    "print(x[:100])\n",
    "print(x[7:2:1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[6 7 8]\n"
     ]
    }
   ],
   "source": [
    "# Negatives wrap around (taken mod length of axis)\n",
    "print(x[-4:-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[9 8 7 6 5 4 3 2 1 0]\n",
      "[9 8 7]\n"
     ]
    }
   ],
   "source": [
    "# An array whose index goes up in reverse\n",
    "print(x[::-1]) # default start = n-1 and stop = -1 for negative step [GOTCHA]\n",
    "print(x[::-1][:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x[:5]                [0 1 2 3 4]\n",
      "x[:5][::-1]          [4 3 2 1 0]\n",
      "calling x[:5][::-1].sort()\n",
      "x[:5][::-1] (sorted) [0 1 2 3 4]\n",
      "x[:5] (rev-sorted)   [4 3 2 1 0]\n",
      "x                    [4 3 2 1 0 5 6 7 8 9]\n"
     ]
    }
   ],
   "source": [
    "# What happens if we do an ascending sort on an array with the reverse index?\n",
    "x = np.arange(10)\n",
    "\n",
    "print('x[:5]               ', x[:5])\n",
    "print('x[:5][::-1]         ', x[:5][::-1])\n",
    "x[:5][::-1].sort()\n",
    "print('calling x[:5][::-1].sort()')\n",
    "print('x[:5][::-1] (sorted)', x[:5][::-1])\n",
    "print('x[:5] (rev-sorted)  ', x[:5])\n",
    "print('x                   ', x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x (2, 4, 4)\n",
      "[[[ 0  1  2  3]\n",
      "  [ 4  5  6  7]\n",
      "  [ 8  9 10 11]\n",
      "  [12 13 14 15]]\n",
      "\n",
      " [[16 17 18 19]\n",
      "  [20 21 22 23]\n",
      "  [24 25 26 27]\n",
      "  [28 29 30 31]]]\n",
      "\n",
      "x[1, :, :1] (4, 1)\n",
      "[[16]\n",
      " [20]\n",
      " [24]\n",
      " [28]]\n",
      "\n",
      "x[1, :, 0] (4,)\n",
      "[16 20 24 28]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Multi-dimensional\n",
    "\n",
    "def display(exp):\n",
    "    print(exp, eval(exp).shape)\n",
    "    print(eval(exp))\n",
    "    print()\n",
    "    \n",
    "x = np.arange(4 * 4 * 2).reshape(2, 4, 4)\n",
    "display('x')\n",
    "display('x[1, :, :1]')\n",
    "display('x[1, :, 0]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y (2, 2)\n",
      "[[0 1]\n",
      " [2 3]]\n",
      "\n",
      "y[:, :, np.newaxis] (2, 2, 1)\n",
      "[[[0]\n",
      "  [1]]\n",
      "\n",
      " [[2]\n",
      "  [3]]]\n",
      "\n",
      "y[np.newaxis, :, :, np.newaxis] (1, 2, 2, 1)\n",
      "[[[[0]\n",
      "   [1]]\n",
      "\n",
      "  [[2]\n",
      "   [3]]]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Add as many length-1 axes as you want [we'll see why later]\n",
    "y = np.arange(2 * 2).reshape(2, 2)\n",
    "display('y')\n",
    "display('y[:, :, np.newaxis]')\n",
    "display('y[np.newaxis, :, :, np.newaxis]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "slice 0 2 1\n",
      "x[0, 0, s] (2,)\n",
      "[0 1]\n",
      "\n",
      "x[tuple([0, 0, s])] (2,)\n",
      "[0 1]\n",
      "\n",
      "x[(0, 0, s)] (2,)\n",
      "[0 1]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Programatically create indices\n",
    "def f(): return slice(0, 2, 1)\n",
    "s = f()\n",
    "print('slice', s.start, s.stop, s.step)\n",
    "display('x[0, 0, s]')\n",
    "# equivalent notation\n",
    "display('x[tuple([0, 0, s])]')\n",
    "display('x[(0, 0, s)]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basic indices let us access hyper-rectangles with strides:\n",
    "\n",
    "<img src=\"assets/slices.png\" alt=\"http://www.scipy-lectures.org/intro/numpy/numpy.html\" width=\"300\">\n",
    "\n",
    "### Advanced Indexing\n",
    "\n",
    "Arbitrary combinations of basic indexing. **GOTCHA: All advanced index results are copies, not views**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "m (4, 5)\n",
      "[[ 0  1  2  3  4]\n",
      " [ 5  6  7  8  9]\n",
      " [10 11 12 13 14]\n",
      " [15 16 17 18 19]]\n",
      "\n",
      "m[[1,2,1],:] (3, 5)\n",
      "[[ 5  6  7  8  9]\n",
      " [10 11 12 13 14]\n",
      " [ 5  6  7  8  9]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "m = np.arange(4 * 5).reshape(4, 5)\n",
    "\n",
    "# 1D advanced index\n",
    "display('m')\n",
    "display('m[[1,2,1],:]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "original indices\n",
      "  rows [0 1 2 3]\n",
      "  cols [0 1 2 3 4]\n",
      "new indices\n",
      "  rows [1, 2, 1]\n",
      "  cols [0 1 2 3 4]\n"
     ]
    }
   ],
   "source": [
    "print('original indices')\n",
    "print('  rows', np.arange(m.shape[0]))\n",
    "print('  cols', np.arange(m.shape[1]))\n",
    "print('new indices')\n",
    "print('  rows', ([1, 2, 1]))\n",
    "print('  cols', np.arange(m.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "m (4, 5)\n",
      "[[ 0  1  2  3  4]\n",
      " [ 5  6  7  8  9]\n",
      " [10 11 12 13 14]\n",
      " [15 16 17 18 19]]\n",
      "\n",
      "m[0:1, [[1, 1, 2],[0, 1, 2]]] (1, 2, 3)\n",
      "[[[1 1 2]\n",
      "  [0 1 2]]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 2D advanced index\n",
    "display('m')\n",
    "display('m[0:1, [[1, 1, 2],[0, 1, 2]]]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why on earth would you do the above? Selection, sampling, algorithms that are based on offsets of arrays (i.e., basically all of them).\n",
    "\n",
    "**What's going on?**\n",
    "\n",
    "Advanced indexing is best thought of in the following way:\n",
    "\n",
    "A typical `ndarray`, `x`, with shape `(n0, ..., nN-1)` has `N` corresponding _indices_. \n",
    "\n",
    "`(range(n0), ..., range(nN-1))`\n",
    "\n",
    "Indices work like this: the `(i0, ..., iN-1)`-th element in an array with the above indices  over `x` is:\n",
    "\n",
    "`(range(n0)[i0], ..., range(n2)[iN-1]) == (i0, ..., iN-1)`\n",
    "\n",
    "So the `(i0, ..., iN-1)`-th element of `x` is the `(i0, ..., iN-1)`-th element of \"x with indices `(range(n0), ..., range(nN-1))`\".\n",
    "\n",
    "An advanced index `x[:, ..., ind, ..., :]`, where `ind` is some 1D list of integers for axis `j` between `0` and `nj`, possibly with repretition, replaces the straightforward increasing indices with:\n",
    "\n",
    "`(range(n0), ..., ind, ..., range(nN-1))`\n",
    "\n",
    "The `(i0, ..., iN-1)`-th element is `(i0, ..., ind[ij], ..., iN-1)` from `x`.\n",
    "\n",
    "So the shape will now be `(n0, ..., len(ind), ..., nN-1)`.\n",
    "\n",
    "It can get even more complicated -- `ind` can be higher dimensional."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x (2, 4, 4)\n",
      "[[[ 0  1  2  3]\n",
      "  [ 4  5  6  7]\n",
      "  [ 8  9 10 11]\n",
      "  [12 13 14 15]]\n",
      "\n",
      " [[16 17 18 19]\n",
      "  [20 21 22 23]\n",
      "  [24 25 26 27]\n",
      "  [28 29 30 31]]]\n",
      "\n",
      "x[(0, 0, 1),] (3, 4, 4)\n",
      "[[[ 0  1  2  3]\n",
      "  [ 4  5  6  7]\n",
      "  [ 8  9 10 11]\n",
      "  [12 13 14 15]]\n",
      "\n",
      " [[ 0  1  2  3]\n",
      "  [ 4  5  6  7]\n",
      "  [ 8  9 10 11]\n",
      "  [12 13 14 15]]\n",
      "\n",
      " [[16 17 18 19]\n",
      "  [20 21 22 23]\n",
      "  [24 25 26 27]\n",
      "  [28 29 30 31]]]\n",
      "\n",
      "x[(0, 0, 1)] ()\n",
      "1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# GOTCHA: accidentally invoking advanced indexing\n",
    "display('x')\n",
    "display('x[(0, 0, 1),]') # advanced\n",
    "display('x[(0, 0, 1)]') # basic\n",
    "# best policy: don't parenthesize when you want basic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above covers the case of one advanced index and the rest being basic. One other common situation that comes up in practice is every index is advanced.\n",
    "\n",
    "Recall array `x` with shape `(n0, ..., nN-1)`. Let `indj` be integer `ndarrays` all of the same shape (say, `(m0, ..., mM-1)`).\n",
    "\n",
    "Then `x[ind0, ... indN-1]` has shape `(m0, ..., mM-1)` and its `t=(j0, ..., jM-1)`-th element is the `(ind0[t], ..., indN-1(t))`-th element of `x`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "m (4, 5)\n",
      "[[ 0  1  2  3  4]\n",
      " [ 5  6  7  8  9]\n",
      " [10 11 12 13 14]\n",
      " [15 16 17 18 19]]\n",
      "\n",
      "m[[1,2],[3,4]] (2,)\n",
      "[ 8 14]\n",
      "\n",
      "m[np.ix_([1,2],[3,4])] (2, 2)\n",
      "[[ 8  9]\n",
      " [13 14]]\n",
      "\n",
      "m[0, np.r_[:2, slice(3, 1, -1), 2]] (5,)\n",
      "[0 1 3 2 2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "display('m')\n",
    "display('m[[1,2],[3,4]]')\n",
    "\n",
    "# ix_: only applies to 1D indices. computes the cross product\n",
    "display('m[np.ix_([1,2],[3,4])]')\n",
    "\n",
    "# r_: concatenates slices and all forms of indices\n",
    "display('m[0, np.r_[:2, slice(3, 1, -1), 2]]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[7 2 9 1 0 8 4 5 6 3]\n",
      "[1 0 1 1 0 0 0 1 0 1]\n",
      "[ True False  True  True False False False  True False  True]\n",
      "[2 7 2 2 7 7 7 2 7 2]\n",
      "[7 9 1 5 3]\n"
     ]
    }
   ],
   "source": [
    "# Boolean arrays are converted to integers where they're true\n",
    "# Then they're treated like the corresponding integer arrays\n",
    "np.random.seed(1234)\n",
    "digits = np.random.permutation(np.arange(10))\n",
    "is_odd = digits % 2\n",
    "print(digits)\n",
    "print(is_odd)\n",
    "print(is_odd.astype(bool))\n",
    "print(digits[is_odd]) # GOTCHA\n",
    "print(digits[is_odd.astype(bool)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[7 2 9 1 0 8 4 5 6 3]\n",
      "[0 2 3 7 9]\n",
      "[7 9 1 5 3]\n"
     ]
    }
   ],
   "source": [
    "print(digits)\n",
    "print(is_odd.nonzero()[0])\n",
    "print(digits[is_odd.nonzero()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 1]\n",
      " [2 3]]\n",
      "[[False  True]\n",
      " [False  True]]\n",
      "(array([0, 1]), array([1, 1]))\n",
      "[1 3]\n"
     ]
    }
   ],
   "source": [
    "# Boolean selection in higher dimensions:\n",
    "x = np.arange(2 *2).reshape(2, -1)\n",
    "y = (x % 2).astype(bool)\n",
    "print(x)\n",
    "print(y)\n",
    "print(y.nonzero())\n",
    "print(x[y]) # becomes double advanced index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing Applications"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1.  2.  3. nan  2.  1. nan]\n",
      "[ True  True  True False  True  True False]\n",
      "[1. 2. 3. 2. 1.]\n"
     ]
    }
   ],
   "source": [
    "# Data cleanup / filtering\n",
    "\n",
    "x = np.array([1, 2, 3, np.nan, 2, 1, np.nan])\n",
    "b = ~np.isnan(x)\n",
    "print(x)\n",
    "print(b)\n",
    "print(x[b])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(100, 2)\n",
      "(100,)\n",
      "[0 1]\n"
     ]
    }
   ],
   "source": [
    "# Selecting labelled data (e.g. for plotting)\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# From DBSCAN sklearn ex\n",
    "from sklearn.datasets.samples_generator import make_blobs\n",
    "\n",
    "X, labels = make_blobs(n_samples=100, centers=[[0, 0], [1, 1]], cluster_std=0.4, random_state=0)\n",
    "print(X.shape)\n",
    "print(labels.shape)\n",
    "print(np.unique(labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAF2xJREFUeJzt3X2MXNV5x/HfY2PyB6GiwQ4Y4xWpZCWlTduQFbAUVYto\nE7AiuWnSCCoVhKJsiEAKUiOVFLmx4j9MqyoiFIK1SazEUhoaCRKs1CnBCAsqGcSagMEQEodAsOuA\nMBUvIjIx+/SPewcm45mdO3PP3HvOvd+PtJq3O3Pfdu5zznNextxdAAAsq3sDAABxICAAACQREAAA\nOQICAEASAQEAkCMgAAAkBQgIZrbWzO4zsyfNbL+Zfb7PMmZmN5vZATPbZ2bnlF0vACCsEwJ8xjFJ\n/+Duj5jZyZL2mtk97v5k1zKXSlqX/50n6bb8FgAQidI1BHc/7O6P5Pdfk/SUpDU9i22QtN0zD0o6\nxcxWl103ACCcEDWEt5nZWZI+JOmhnpfWSHq+6/HB/LnDfT5jTtKcJJ100kkf/sAHPhByEwGg0fbu\n3fuSu68a573BAoKZvVvSHZKuc/dXx/0cd5+XNC9J09PTvrCwEGgLAaD5zOy5cd8bpJeRma1QFgy+\n4+539lnkkKS1XY/PzJ8DAEQiRC8jk/RNSU+5+1cGLLZD0hV5b6PzJb3i7seliwAA9QmRMvpzSX8v\n6XEzezR/7p8kTUmSu2+VtFPSekkHJL0h6aoA6wUABFQ6ILj7/0iyIcu4pGvKrgsAMDmMVAYASCIg\nAAByBAQAgCQCAgAgR0AAAEgiIAAAcgQEAIAkAgIAIEdAAMrYs0fasiW7BRIXdPproFX27JEuvlh6\n803pxBOle++VZmaKv3f3bml2tvh7JvEZQBcCAjCu3buzYPDWW9nt7t3FLsxlAknIzwB6kDICxjU7\nm12Mly/Pbmdni72vXyAZVfdnHD0qbdpE2gqlERCAcc3MZCXzzZtHK6GPG0j6fcayZdLiorRrV1Zj\nICigBFJGQBkzM6OnajqBpEz+v/MZmzZlwWBxcbS0FdAHAQGowziBpN9nbNokPfDAO20J49Q2gBwB\nAUhZiNqGRI8lSCIgAOkrW9ugxxJyNCoDbRei1xMagYAAtF2IXk9oBFJGQNuFaodA8ggIQEzqatwN\n0esJySMgALGgcRc1ow0BiAWNu82WwMy41BCAWHQadxlk1jyJ1P4ICEAsaNxtrnFnxq0YAQGICY27\nzZRI7Y+AAKA8pr5Y2ii1vxqPZZCAYGbbJH1M0ovu/sd9Xp+VdJekX+ZP3enuXw6xbgA1SyQ/Xrsi\ntb+aj2WoXkbfknTJkGUecPc/y/8IBkBT0DsqnJqPZZCA4O73S3o5xGcBSAxTX4RT87Gssg3hAjPb\nJ+mQpC+4+/4K1w1gUugdFU7Nx9LcPcwHmZ0l6YcD2hB+T9Kiu79uZuslfdXd1w34nDlJc5I0NTX1\n4eeeey7I9gFAG5jZXnefHue9lYxUdvdX3f31/P5OSSvMbOWAZefdfdrdp1etWlXF5gHVSWC0Ktqr\nkpSRmZ0u6QV3dzM7V1kgOlLFuoFo0BsHkQvV7fS7kmYlrTSzg5K+JGmFJLn7VkmflPQ5Mzsm6TeS\nLvNQuSogFYmMVkV7BQkI7n75kNdvkXRLiHUByUpktCrai5HKQFXojYPIERCAUZSdVoC5ihDKBKa4\nICAARdEoHB5zII1nQv+L/EAOUBRTNITVuaht3JjdNrEr7qS6GU/of5GAABTVPa3ACSdIv/pVMy9i\nVWl6gJ1kwJvQFBcEBKCoTqPwZz4juUtf/3pzS7ZVGHZRS30Q3yQDXud/cfPmoKlL2hCAUczMZF/s\nt95KZzxBrHn6pXpdNaG9ZtLdjCfQQYGAAIwqpfEEsV9YB13UmjCIL8FuxgQEYFQpfdFTvbCmFHSX\nklg3YwICMI6iv35VNmiU/YxUL6wpBd0GISAAkxAiVRPiM0JfWKtsj0isdN0EBARgEkKkakKle0Jd\nWGNvj0BpdDsFJiFEP/HYfpqy6eMGQA0BmIhRUzX9UjGx5dFTbY9AYcF+QnMSpqenfWFhoe7NACYr\npVRMrGMa8LYyP6FJDQGoW0pdQ2nobTTaEACp3mkSYmsrQGtRQwDqTtnE1lZQF9JRtSMgADGkbNqe\niqk7KEMSKSOAlE0M6NIaBWoIACmb+o3apXXU9BLpqEIICIBU3dxE6G+UoDxqeol0VGEEBKAILiqT\nV7QdZdQ2nxjaiBJBGwJQBDnueIza5kMbUWHUEIAimLYhHqO2+dBGVBhTVwBF0YaABDB1BVCFto8V\niA0BOjgCAhAKF6jq0Mg/EUEalc1sm5m9aGZPDHjdzOxmMztgZvvM7JwQ6wWi0blAbdyY3YaYE6nI\n/Ep1zsFUJxr5JyJUDeFbkm6RtH3A65dKWpf/nSfptvwWaIbQXRuLlIDbXEqmkX8igtQQ3P1+SS8v\nscgGSds986CkU8xsdYh1A1EI3bWxSAm4zaXkTs+hzZvbFQgnrKo2hDWSnu96fDB/7nDvgmY2J2lO\nkqampirZOKC00F0bi5SA215KppE/uOgald19XtK8lHU7rXlzgOJCXqCKBBj61yOwqgLCIUlrux6f\nmT8HoJ+iPZbaXEqmV1dwVQWEHZKuNbPblTUmv+Lux6WLAKjZjcWhLuJNPkY1ChIQzOy7kmYlrTSz\ng5K+JGmFJLn7Vkk7Ja2XdEDSG5KuCrFeoJGaOhlb70X8ppukI0fGCw5NPUY1CxIQ3P3yIa+7pGtC\nrAsILbrMQ1Mbi7sv4kePStdcI7mPV8Jv6jGqWXSNykCVosw8NLWxuPsivmxZFhgWF8cr4Tf1GNWM\ngIBWizbz0MTG4u6L+KmnStddV66E38RjVDMCAlqNzEPFui/iH/wgJfzIEBDQamQeakQJPzoEBLQe\n1yUgw09oAkUMm1W0rbOOolGoIQDDDOuKFGVXJWB01BCAYYbNKtrmWUfRKASEAsgGtNywqa1DT30N\n1ISU0RBkAzC0K9ISr0c3ChpYAgFhiGgHLqFaw7oi9XmdwgRSQ8poCLIBGFcSTQsp5UNT2tZEUUMY\ngoFLGFfhUdB15ZVSqsKktK0JIyAUwMCl0ZE7L1iYqPNCl1I+NKVtTRgBAcFRmHvH0MJEqAvdOBE4\npYmcUtrWhBEQEByFuRGEuNCNG4FTyoemtK0JIyAgOApzI+i90ElZw+koF70yETilfGhK25ooAgKC\nozA3os6FbtySPhEYgRAQMBEU5sYwbkmfCIxACAhALMqU9InACICAAMSCkj5qRkAAYkJJHzVi6gpU\nqq2zD7R1v5EWagioTFsHrLV1v2vHcPmRUUNAZZKY7G0CWrnf8/PSRz+a3dahE4U3bsxuqZoVQg0B\nlWlrd/nW7ff8vPTZz2b3f/zj7HZurtptYLj8WAgIqExbO9Eks9+hUix33HH846oDQuuicBhBAoKZ\nXSLpq5KWS/qGu9/Y8/qspLsk/TJ/6k53/3KIdSMtbe1EE/1+h2zo+MQn3qkZdB5XLZkoHJfSAcHM\nlku6VdJfSToo6WEz2+HuT/Ys+oC7f6zs+gBMQMgUS6c2cMcdWTCounbQEX0Ujk+IGsK5kg64+zOS\nZGa3S9ogqTcgIHJ0yhiusccodIplbq6+QICxhQgIayQ93/X4oKTz+ix3gZntk3RI0hfcfX+/DzOz\nOUlzkjQ1NRVg81AEXSOHa/QxIsUCVdft9BFJU+7+J5L+XdIPBi3o7vPuPu3u06tWrapo89DKrpEj\navwxmpmRvvhFgkFoCY1KDFFDOCRpbdfjM/Pn3ubur3bd32lmXzOzle7+UoD1IwA6ZQyXzDFqbF4r\nQYlVK0MEhIclrTOz9ykLBJdJ+rvuBczsdEkvuLub2bnKaiZHAqwbgZAxGC6JY5TYBajxEhsPUTog\nuPsxM7tW0t3Kup1uc/f9ZnZ1/vpWSZ+U9DkzOybpN5Iuc3cvu+6mqquAR6eM4aI/RoldgBovmWpl\nJsg4BHffKWlnz3Nbu+7fIumWEOtqOgp4KCWxC1DjJVGtfAcjlSPTlgIeae4JSewC1ArRVyvfQUCI\nTBsKeK2qBdUR+RK6ACEuBIQI9F4zml7Aa0stqF2RrwSqi9EgINRs0DWjyd+LNtSCJLUo8pVA0IwK\nv4dQs8YPduqjUwvavLnh3/9O5Fu+vOGRr4Q2fgEiRg2hZq0pLfdoei1IUjvyf2W19QsQKQJCDdrW\nZtBqrYh8JfAFiAoBoWJtbDMAlsQXIBq0IVSMlOlwCc0FBjQKNYSKkTJdGp1OgPoQECpGynRp9NQE\n6kNAqAEp08GoQQH1ISAgKtSggPoQEBAdalBAPehlBACQREAYGV0iATQVKaMR0CWyv+6R1xL5fyBV\nBIQRxNIlMqbZgruD5AknSO7Z8ekXMPfskbZvz+5fcUX92x6jmM4t2oeAMIIYukTGVkvpDpKLi9lz\n7scHzD17pIsuko4ezR5v28YYg16xnVu0D20II4hh2ubYpr7oneF5xYr+sz13trvjt78tvu1tabeJ\n7dyifaghjKjuLpEx1FK69Y4bkH435dFJgZx6ara9nRrCihXFtr1Npea+55YcEipEQEhMjAO3eoNk\nd5qo+2J+883ST36SvVa0DSGWdpsqHHdu1aJoiCgQEBJUdy2lqN6L+ZEj0m23jfYZsdWIJu13zu2W\n3e2JhogCAQETE+JiHmONqDJti4aonbl73dsw0PT0tC8sLNS9Ga0UKnVNCrwkDiBGZGZ73X16rPcS\nENCrCQ25jHlAW5UJCKSMcJzUG3IZ8wCMJ8g4BDO7xMyeNrMDZnZ9n9fNzG7OX99nZueEWG9bTbpf\nfid1vWyZZJZ1GU1JmTEPQJuVDghmtlzSrZIulXS2pMvN7OyexS6VtC7/m5M0Yl8TdHTSORs3ZreT\nCAozM9JNN2UDzBYXpeuuS2tQWCegdRQd8wC0XYgawrmSDrj7M+7+pqTbJW3oWWaDpO2eeVDSKWa2\nOsC6W6eq0axHjmTBYHExvVGzMzPSffdJV1+d/ZEuAooJ0YawRtLzXY8PSjqvwDJrJB3u/TAzm1NW\ni9DU1FSAzWuWqnoipt7jMZWxGkBMomtUdvd5SfNS1suo5s2JTlX98lvd/x9oqRAB4ZCktV2Pz8yf\nG3UZFFRV6ZdSNtAuIdoQHpa0zszeZ2YnSrpM0o6eZXZIuiLvbXS+pFfc/bh0EVC1tsykChRRuobg\n7sfM7FpJd0taLmmbu+83s6vz17dK2ilpvaQDkt6QdFXZ9daJwaPFxH6cRhmAN+6+xH4MgG5B2hDc\nfaeyi373c1u77ruka0Ksq25NGMVbhRSOU9EBeOPuSwrHAOjGD+SMiB8xKSaF49T74z6DelKNuy8p\nHAOgW3S9jGKXenfMYUKlOFI4TkV7Uo27LykcA6Abk9uNoal54dApjiYdJ9oQkAomt6tYKt0xu3++\n8siR4Rel0JPapXKcihh3X5p0DNB8BISG6pT2jx7Npp9Ytkx617uWLvVXleKg1AzEiYDQUJ3S/uJi\n9rh7TqKZmf4X5SpGJ9PzBogXAaGhOqX97hpCp9S/1EV50imO1H9rAWgyAkJDdZf2e9sQtmyp76JM\nzxsgXgSEBhtU2q/zosykeUC8CAiJKtMwW/dFmZ43QJwICAkK0TDLRRlAL6auSBBTIgCYBAJCgorO\nwZM6pqYGqkXKKEF1twFUgfEKQPUICIlqehvApMYrMEoaGIyAgChNomsstQ5gaQQERGkSaTFGSQNL\nIyAgGr3pnNBpMUZJA0sjICAKVaRz2tAYD5RBQEAUqkrnNL0xHiiDcQiIQlvGVgAxo4aAKJDOGYyu\nsqgKAQHRIJ1zPLrKokqkjICIMW8VqkRAACJG2wqqRMoIiBhtK6gSAQGIHG0rqEqpgGBm75H0n5LO\nkvSspE+5+//1We5ZSa9JekvSMXefLrNelEfPFQC9ytYQrpd0r7vfaGbX54//ccCyF7n7SyXXhwDo\nuVIcgRNtUjYgbJA0m9//tqTdGhwQEAkmeSuGwIm2KdvL6DR3P5zf/7Wk0wYs55J2mdleM5sruU6U\nRM+VYujyibYZWkMws12STu/z0g3dD9zdzcwHfMyF7n7IzN4r6R4z+6m73z9gfXOS5iRpampq2OZh\nDPRcKYbZUdE25j7oGl7gzWZPS5p198NmtlrSbnd//5D3bJL0urv/27DPn56e9oWFhbG3DyiLNgSk\nxsz2jttxp2wbwg5JV0q6Mb+9q3cBMztJ0jJ3fy2//xFJXy653sbjQhQHunyiTcoGhBslfc/MPi3p\nOUmfkiQzO0PSN9x9vbJ2he+bWWd9/+Hu/11yvY1GYyZCo4CBIkoFBHc/IuniPs//r6T1+f1nJP1p\nmfW0Db2AEBIFDBTFXEYRohcQQqK3FIpi6ooI0QvoeKQ8xkdvKRRFQIgUjZnvIOVRDgUMFEVAQPRo\nUymPAgaKoA0B0aNNBagGNQREj5QHUA0CQgO0ocGVlAcweQSExNHgCiAU2hASRx9zAKEQEBJHgyuA\nUEgZJY4GVwChEBAagAZXACGQMgIASCIgAAByBIRE7dkjbdmS3QJACLQhJIixBwAmgRpCghh7AGAS\nCAgJYuwBgEkgZZQgxh4AmAQCQqIYewAgNFJGAABJBAQAQI6AAACQREAAAOQICAAASQQEAECOgAAA\nkERAAADkSgUEM/tbM9tvZotmNr3EcpeY2dNmdsDMri+zTgDAZJStITwh6W8k3T9oATNbLulWSZdK\nOlvS5WZ2dsn1AgACKzV1hbs/JUlmttRi50o64O7P5MveLmmDpCfLrBsAEFYVcxmtkfR81+ODks4b\ntLCZzUmayx8eNbMnJrhtdVop6aW6N2KC2L+0sX/pev+4bxwaEMxsl6TT+7x0g7vfNe6KB3H3eUnz\n+boX3H1g20TKmrxvEvuXOvYvXWa2MO57hwYEd//LcT88d0jS2q7HZ+bPAQAiUkW304clrTOz95nZ\niZIuk7SjgvUCAEZQttvpx83soKQZSf9lZnfnz59hZjslyd2PSbpW0t2SnpL0PXffX3AV82W2L3JN\n3jeJ/Usd+5eusffN3D3khgAAEsVIZQCAJAICACAXTUBo+jQYZvYeM7vHzH6e3/7+gOWeNbPHzezR\nMt3HqjLsfFjm5vz1fWZ2Th3bOa4C+zdrZq/k5+tRM/vnOrZzHGa2zcxeHDTWpwHnbtj+pXzu1prZ\nfWb2ZH7d/HyfZUY/f+4exZ+kP1Q2oGK3pOkByyyX9AtJfyDpREmPSTq77m0vuH//Kun6/P71kv5l\nwHLPSlpZ9/YW3Keh50PSekk/kmSSzpf0UN3bHXj/ZiX9sO5tHXP//kLSOZKeGPB6sueu4P6lfO5W\nSzonv3+ypJ+F+O5FU0Nw96fc/ekhi709DYa7vympMw1GCjZI+nZ+/9uS/rrGbQmlyPnYIGm7Zx6U\ndIqZra56Q8eU8v/bUO5+v6SXl1gk5XNXZP+S5e6H3f2R/P5rynpwrulZbOTzF01AKKjfNBi9ByFW\np7n74fz+ryWdNmA5l7TLzPbm03jErMj5SPmcFd32C/Iq+Y/M7I+q2bRKpHzuikr+3JnZWZI+JOmh\nnpdGPn9VzGX0tqqnwajaUvvX/cDd3cwG9fe90N0Pmdl7Jd1jZj/NSzqI0yOSptz9dTNbL+kHktbV\nvE0oJvlzZ2bvlnSHpOvc/dWyn1dpQPCGT4Ox1P6Z2QtmttrdD+fVthcHfMah/PZFM/u+srRFrAGh\nyPmI+pwNMXTbu7+E7r7TzL5mZivdvQkTp6V87oZK/dyZ2QplweA77n5nn0VGPn+ppYxSngZjh6Qr\n8/tXSjquRmRmJ5nZyZ37kj6i7DcnYlXkfOyQdEXe4+F8Sa90pc5iN3T/zOx0s2z+dzM7V9l36kjl\nWzoZKZ+7oVI+d/l2f1PSU+7+lQGLjX7+6m4t72oR/7iyHNdRSS9Iujt//gxJO3tazn+mrPfHDXVv\n9wj7d6qkeyX9XNIuSe/p3T9lvVkey//2p7B//c6HpKslXZ3fN2U/kPQLSY9rQA+yWP8K7N+1+bl6\nTNKDki6oe5tH2LfvSjos6bf5d+/TDTt3w/Yv5XN3obL2xn2SHs3/1pc9f0xdAQCQlF7KCAAwIQQE\nAIAkAgIAIEdAAABIIiAAAHIEBACAJAICACD3/2NfUoIvk6XxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8b8d3d1358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for label, color in [(0, 'b'), (1, 'r')]:\n",
    "    xy = X[labels == label]\n",
    "    plt.scatter(xy[:, 0], xy[:, 1], color=color, marker='.')\n",
    "\n",
    "plt.axis([-1, 2, -1, 2])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPgAAAD8CAYAAABaQGkdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztfd+rPUtW31rdZ5/7Ha4jXogjE2dgAgm+CFFymTxMCERQ\njA4mjwr6JNyXJIyJQWLe/AMi8xAfclFJRKMIKohJDBMckQF/zmTU+aFBzIAzES5ijE7g3u853ZWH\nrlW11qdWdfc+Z5+zz9lZH/h++1R1VXV17737s9aqtVZxSokCgcBlYjj3BAKBwMMhfuCBwAUjfuCB\nwAUjfuCBwAUjfuCBwAUjfuCBwAXjak8jZv4CEf0VEU1EdJtSev0hJxUIBE6DXT/wjH+QUvqzB5tJ\nIBA4OUJEDwQuGLzHk42Z/ycR/R9aRPR/l1J602nzBhG9QUTE19d/5/Ce95x4qoFAQHD7v/+cpi//\nX95qt/cH/rUppS8x83uI6GNE9M9SSr/Wa//K+9+f/vr3f99REw4EAvvxv/7NR+mdP/mTzR/4Lh08\npfSlfHyLmX+BiD5IRN0feMHa5Tvn0mqfHX7zW7e8+UhO1OfUuEvIwJ4+u9r4D2DPx9Edf6Vvd9y1\n661+cTbG3TP+KfucAjuvu6mDM/OrzPxu+ZuIvoWIPnOfuQUCgcfBHgb/GiL6BWaW9v8xpfTLR13F\nebmWFy6+VhmOK+Mk75WM/Y5h394r/qkz+A726o6Bj98bC9W45Fd78yg1xzC2lGE89+NJcMTrem07\n5VWGv4tU8YDYJT3Rjh94SumPiehv33M+gUDgDIhlskDggnGMo8tJ0IjmKJKztFMyyJrYrsdaa4vX\nM+dWxiMi3iMB75WZPOwQsdfF4WQOWO326Yi27qIKtC1ifDO+qihTWtruEdVRNGeco/Mxt3PrjLna\nhv16b77HiPW9MY5B76txKiNbIBB4vnhYBi9srOvyq2fANh1G130atndYGc5xpy2vGOgaxl55TZ/S\n/rb6UgaWTw4z1XPAsLlsfB6aNp2jN45UzMvBNcx15ibglfHLORnfG0s+zhnKvbHMdbBtf65NmxWs\nSg+ngHrMewXGYPBA4ILxODr4io6cGkZ3WHnwGZwHfBVTw9AM45YhTB9f50aW93RxVxI4AVKHsbFe\nn6tMLUyL9bpTPswMbYSdHX0d2JIG24dn7wHJhY5wREHmnuG8U1eujSw9qz495qa2bX/prnPeG++B\nltASt9fqIRg8ELhgPLoVfZO5pTykpk9h7EH0a2BpNc7A9lUr7DtAXzMOsj8yuHM7Q+c1vYfZPTYW\nzMjC0Ef3RXZPwNgzsLRpM0gZmFsTuExC6qTNbE8k9ZkxjCOX9pnPsm/D3Cg5qPG508bt02N3TxDE\n/jus9Efp4Ftfjw0379DBA4HAwzJ4XfNWlT1LODK3ZgNh3XLM9YXBq/KETD3kc8jOmnkHYOzapt+n\nXs/W9Rjdw9zo2e1rewZWFtJMDhsj68/zkM/bsq3LfaSt54cw23ErfCbX/XnHOkPPmt0ybh2LJ6mz\nR+xrdHBo2+j8uxg8QVmNv6WDH6OTry1MMFV7yAaCwQOBC8YjWdHrn931btSvNYOPtq4wbmFpxcaZ\nsQdoOxYml3KfwUdg7DXmxnP3YXAsE7Xs7OnmU2ZfZPtpnu15dc9l3FmOy7l5ym31kgGufDSzzCem\ntqqR1PqDtOvdhXGtvq3/RqYdZA543unT6u1qUiABbEoZXt3KV2Hra+KaZ2IdPBAIaMQPPBC4YJxt\nmawa16zIXgxpY2tkQ/F7HG2ZqBXJRRSv5Xx0jGxyrojmhEa3tk+3vCKfzR0LihbRG5FcysRNWxTR\npyKiS72I6rXP7TTme2M4yqDKYJZ5YK4V+lD/MLfVLrd10TFOoaFsj7iNxjdtjGrF92T76nl0DHKt\nMc8R63cY1+4jooejSyAQIKIHDzYBg5r+u+N2ikeiPnOXo3odXo3A1MDYUr5SS2sj2zpk8IHbPmVu\n0PYu8BgdGVyOt2k0ZSKi27IcJm2WsjC4nJ+G2kf+FFa/zWuP1bZW3/1TpjPu8EET3LJ0Oh6NIQvG\n9dxOkblx+UzNozB2Z4nNlxBgWaxndDN1qdvmaDhMnsLRJRAIEJ1DBxd0lsnEicXoux3mFkYV1iaq\nDH0ABr/OCpjH4Ff5lS7XLExOUs599ZwKu6MuvtMDgYjmNEBZ6eBk9Wlpe5umpq0wtLD7KIydH+Y4\nLMebrHfred8UXXwp37ozlXnO+f8sMRR3VEdSE/fVsgS55rnhH3cxLDJ3w+DqMwMdfGiYXNl9ejo+\nMrpmcDiHY93X0aV8XZggcqiPYPBA4ILxKAzuZz8FnRvdRZUOvsXch7EqWsLcUif69XUuC1tbBpdx\nl3MHtrr4IddrPXuANuNe30EHE1kr+PK3WK4tk9/OC+PeKPafB8vuLzMr3w6W0bW0UZg7l2/ZvuuN\n/4YQUxLmXsrlMxJXVmNrEYrbY0aHLj0mN2y5cczMPWgdHFh+mMCK7unrHQb3WH87MOUOyrh6ftr1\nO3TwQCBwjnVw0Ncg5JPBYq7/Fh0Ymfug2LjWLUfRva/HRbtEtiYiemWwfVAnP0CZqDJ2YXDQvY9Z\nB59Sy+DI6jdiPR+W+xB9m4joJrO6WM/lHl9mV1Vh58GxITDXcfT1UmolkrIWL26/oosDk+eT61D3\n2riodpIuejo4Afsic7MyKgzQRsbAen2utbzbvpqVy7gNk6/QLZ6C55Ycl+EINgkEAkQUP/BA4KLx\nKFlV9zi6tJlXqgwygGiOS2HayPZKFsWvsyhbjWtL21dyvbRbzolxzRrgpCzi90FZYUTEHYuxzcpM\n44qIPoEcJga1SRvOinEtL3Gl0Rxv5zqXKxYRfTT3KnN6yfs/5uqroePNl9oZoveKsS1/PjaXPbiq\n7re1OSIuTI76xq+hMaSpPii+d8q6fxXR4ZjseTNfGQcdXjyg+A5GSWu4lB/HhtivEAweCFwwzh5s\ngoEcgxPjjW6nyNwH9ZpG5payGNIKgw83pQ8ydnPMfTQrV8MbMvfxy2XVoKbdQ8G4hkw+VOPY2/OB\niCqrD3Qwcyvx7TsMf8WQpqQiIakhnxsGa2yjWZxl6niVzY+g7p5RzTGyNcEgPaObYtgt5h5uHQa/\nneE6wOSGwdEJRoyPO5i8A8PgksWWOYJNAoHAGTK6lKoSZCIVwtx2+YloLeTTuqES9Zn7XePLXF7q\ntT6NdQcsw5GoMnXRwcmy5R4UN1RqdXBh9S6Dq2UyYeabrIsXaWjuf7y4VIdBLaPK3ybPX7K+TJiF\nVnRyc+/HM/dW1pS1YJBmyctzXgHmHm58/ZqosnkZH8qFlac6qZrjTSQbqXe+E1s6dE0iWOtKLkIO\nV9VAIHBWRxd7xBfWqMM5IeSzOq9YJxYibS33mVuOL5QOLvp4j7GvuWX96uiS5wZW9T2YINhEs/JU\nWF108SvTRsp6Du9kXXzPHCZg7Ovs5jqV0NK5aVscZtC9WBp6qyV3QDdhwpoVHZh7cFxVG0cXOFod\nXJh7hraik+e2xjkmX7zo3nI/1qp+DNi4qsqPg8NVNRAIPNY6uFcHy6QNKxgdfHaPJXBEMRYydY+5\ntRX9BVvLOjL2IZevzTq41cGFwYcjrOgzBHi8VAwuernUCcO+nbKFXLHBkOwcxKpO1gvVXjsNcFw+\nicMogStKB58l7HQZ/1bWnEU3dz6zdh94+4G7+cRxh5OONd20aSztVie3IaCWqZG5h1ulTwubz3Cu\nWNVzWTN4TTZv+tI9GFzr4IXN5yF08EAgcASD8xKV8DtE9KWU0ofve+EmPBTWw3Xe8hJkAska6rH1\nZEO9Gpn7hYpCqHWLvn4NfYTB7Tq4rau6+BE6OKx/H9ScplJndW5h62Gu1xnlY8TXtZDM0Aaz3OZn\ndpssY1/NIh3V8W8wkSVLQgmk2o0bPha9dXHS1mx7rqebE1EJPClebsDkbHTwzNS3lql5mkwfUp8D\niUV9Bj39LgzuhNmmLF3RPD8Ig3+EiD5/RPtAIHBm7PqBM/P7iOjbiehHH3Y6gUDglNgron+UiH6A\niN7da8DMbxDRG0RE42uv9UdCyaPnsurlLZdYbjCuvaJEdMzYgq6paFBb6hbR/EVpI8Y2a2TzlslK\nQAr4Dh4TbDKxzbtGRPQyi+YjGNCKcU+pMG/Dsoy8tuU6L7xc6rgMl91NS/455Qpbtn2aJfOqv7Rp\ncI+MLo2RDR1hvHPdzCutka05ihiu296CiI7Gtdv8XVAielUbcLnMEdHv4OhSrj0Mp3NVZeYPE9Fb\nKaVPrrVLKb2ZUno9pfT6+Oqr+64eCAQeFHsY/ENE9B3M/G1E9IKIvpKZfzKl9N27r+LmZINieWH1\nl8kwb/kVZGIhWgkYYQk2sQY1opa59zD4NWZ0KQy735BSQkKFYZWBrjA1dRhcW5yQuTPTirQiy3Hm\nOeVlt9ssKci5l4XJ6/j43HsG0iPI2gC/HnscXdrMq8mvN3nWpM5ncr1MRmBEQ+Zmh8GLkU3YORvk\nPLZOGwzOrquq+Kqe0MiWUvrBlNL7UkofIKLvJKJfOerHHQgEzoazuariCwrZwNs7DPOWH0Ant+f8\nwJFrx2mlx9xSrlKAXppKpq4y7H7IaBMw+TKOlQxwOU6vxk35norOLTp9PspSm5ZA5O93yjO0R5NB\nluxnMsBnVqDK8rke5dqxxdxaB0cHl8bRZYcOLstkRQdX+rQsh91uMPetEhGAuVNhdGfpdIOBUxON\nRURlC2ymvU/2qB94SulXiehXj+kTCATOh/PtbJLROrpQPrYM0hwdi/uWDo6OL0SVsbeY+6Bo87ow\n3YKxZtNZyiv3PEF5lnBP9VZG5i5WeWmiRYVC6tmhJb/1b1bCXUXnPsyWyeszVWwGzxmDTFZ175IJ\nVPrsUNQ9nZuIVnfy7JQN62OSBmR0k7JJ2BjYHZl7qs80FfdV0L2LfUDbTTYYePAYPCd8GMZI+BAI\nBM6QdLG87Tvxbt5e3AysgtZzsz4tbNvRvQ9w9OqQuV9IWTNsvo9DKed0Ru5dWUifooMnK70QEd3I\n2x9CPyfhf/X4CmPnncVkT7J6P6M5LvMEe8Ysz038CNp9zLzPRtcbbr5LfqIOvKSLBb31cCdlU+lf\nEifC+rS2iANjl8QOk2XpZKzok9/GG38LyfocLHX6CZ/Iih4IBJ4vnoAOno+5jOyg664g8YO3wyd6\nlrXJG1rPs1ZPtzr3oYxV5zSWvjnFUb6D4YjF4BmY2+rtwC55TugFR0T0MveUte0bSWXFUm4TRNbn\nNJtz3jNFW0ePye+Mrs5t631PNrBNQL3dz8zW9RI/EFFhYe7o4I2+rfpUKzrq4scw+NLH7GwirE4U\nOnggEIgfeCBw0Ti7iN4DOyJiKa/EYA8gmo/g8jmUIJFqZJNz1+AWKkthoyNCH4oTSc52IiL6Ee/M\nGcTiWS8D4Y58+dyU56izol5nwxsa20rACjjj6DoUv9dUJCz3XFYfBR3xvRHZtaNLL/jDi9dGsRrE\n78agpupQNE9ajBdsLZOJimFE9DweznUFweCBwAXjbAxenST2G3B6bGNyqMO+XL2yv0uJsL91YjmU\ndqz6+Mxdl8v6784SVFLCQ2VtR91rfkOL1DAXKcIaGJe/rXQylr7J1GupZcj7lY0gBaGxTf99lFFt\ny9a4Zyg0rjlGttoWKhzDXDfLaXExVWzcaZOa+ll1gbbC3I6jS+owOIvFVYQA5ehyl1ieYPBA4IJx\nvrzoR6AwdZNUoQ2dxL3CBtCrm6AN51zNcZ7Pw1KY/rvH3MPq+za3yU1uym3VOYmOLcw9JX+Otg6Y\nHPYm00BJoKdn3xtlnDskfjiibdcZRpeR5Wc46nHBnbXJlIpHoqqXN1lVZWlN6/j+klmaLefyoCUE\nznM7YbhoIBB4vnhyVvQ1a25ts+0wUNld2BjYzXlt12AW8o9KB99i7pFX3p1JLOEyN4nIqH3KcwBd\nvM6pZfBSlnsFltjDyh7bo16O9pO7Jnp4CBRX1bVbLQ4v9uha0Xs6t7DyWjIHZG79eXQZ2H5mmtHL\nPuz7MzYFgwcCl4wnw+Bra6hbaZDW2GwPen3WQj7rtY9gbhlX2pQ3ek6AqAJJx5LGKSdFRLuAnnNy\n6vT1PGkFVhWeA9zdUO7a38Nddh5xA1TQon+8i6orFpVx9nwzFwSDBwIXjPiBBwIXjCcjogeUsY22\nPRk1ivHsRCtby5iPL7rfaWVuow87Yvemk4xT1zi47JlaI6qf8APaiWDwQOCCEQz+hDDd8Q0/pye0\nTvXYuMOty+MqXT2DFtRJ0Mcxn5C4nSYn1uSxEAweCFwwngWDzxuvaX3ey3iyhV4fefEe3LNybbvU\nJUsZa8tlEyybzM4y1pS5YgLOKMk+78nasg/alPx5zp36h8TWLZnzO29fZ0RpmBuxx2MH2wzqOYmr\nKgSM1MyoOxxd1uawYwkWEQweCFwwngyDiyP9DEcPx7CLMJ0wlSRFmJx3WwnwaOrluvWtK8kaxL1U\nXtrifrrHuWEGdvaYXLCmxuHOKOVeyd7rHl3dkwzmMp7f/wzG4S6OETyE3d27arfeycfBHvVnJmwu\nTJ7bSMCIDSTZ+H6Uvu3smHdll1+mtLNdIBB4hnh8Br+H7ogMUve4ru+poluSZbFmLNVH9udGJhSG\n9fKWV/fY5U0sIZ91LXtHwge5fmr1bZEWJD86RjZ6dge856no2e0zn8AVFtl9je1LZOaez/IOn/cx\nXaQt9illo7eXbXPyUdpmJvf039I2N5bwzaHtg+PUfB6WyYnasFAEr+xsQsOwO8InGDwQuGCcTQfv\nqW2oi+Pfuuzp60X/BCvxTVoc9F8m2eWjpi+qOqbNPX5Dlrm1i3/Rx8ulrU5uAkdEQki+zi3MrXX8\nm2JFl/uiPH+5n1ZqqdILSiKt3QGfT9XTV2wTR1Hrxvk9QyE76/yD2L/oyPZzMe16jF3jgev4Y2Zd\nCSBBdpa+Y/1WSO+S4kKSMAKTExGVTWbQViOM7UgIdX/w/Z9DMHggcMGIH3ggcME4+zKZJ5IjGpG8\niIyO6IlLRrhU5BjmJH94zU5q856VjQBNWs/lMBQjWM7sIhlklDyJMQfovFIMaqr+JRjXbsAoNjlG\nNlE/ttQU28ZXd2Yzf19FSp3jo8AzopESnUVU12J3EbOlLxrQ9Oc723NsRfUimpssMKIXQF41ZyPJ\nmgnGxnY3hj7tSJOvycy7HX2CwQOBC8bZGVxQFp0cNkB2mRomb41swlY3hdUsy+mtdEdxL+1kYC1v\nXmMks7nSJPspMroHZPCbZA1qS385J8cBjnX+N+kq9xlM+SU8A21Am3FpDZxZPCOn99mcBF02tvXe\nZVMxeqFxDer1OQkCQSZX7MnZyCabDEqZ8rMsBrWhza5SRpE5OFlavDBWbENEhsHLHHigvRS+yeDM\n/IKZf4uZf5eZP8vMP7Rr5EAgcHbsYfB3iOibUkpfZuYDEX2Cmf9LSuk3Nns6qmtVZ9mWy/mWQVo3\nVuvAYdnMstWcLKtdyxa7qd56Ny96s59tGyzQ5i3P89gRWCiMLaPq3WtF50bmfklyrPf8EpYAyzMA\n+4P3nG5ny9yT8/yRuXuMbu74IfRxb5msx/LIzlSZuyx1laWwPPNZX0D0ctDPR/u5aiYuf8k1ZQvi\nBPr7Hqw5uhyhg2/+wNMiX3w5Fw/53xPyPg4EAj3s0sGZeSSiTxLR3ySiH0kp/abT5g0ieoOIaHzt\ntd0TQAts3TBCMUh+Xd0K62QWvp0zUw3VaQV1bzy+nZbgT50zHPf2KumKJDQzc621XHd2TMnnx5U3\n7ASvR5nJS0dHRuaW+Rs2JrlHq3u/zOUbsK4TEd3kZ1fYfRZGt0ei/kpHdVnt32vdI+wI9uqwsg79\nFCs5Gq6TVZVNsoXSNn84nBk7oTsqUWFqzhKO6NHtXdSfEOc97tIM7JscG04vJxcGl5h7zhLHeGJX\n1ZTSlFL6BiJ6HxF9kJm/3mnzZkrp9ZTS6+Orr+66eCAQeFgcZUVPKf0FM3+ciL6ViD5zpytu6N6e\nPtdbqxV2u1VsdjtnHXtABl9uVazcg8pnPQ6Sc1zyWucT8vpzXrZl3bth8I7+roDBH8jWRMpdVtiY\nfEmEiOjt+drUyb2+M1u293VwkYrAiq7XwcGyPs32syq4r96N3Tt6tleHFnFvSVqYuxBq8UHOzGgW\nSbKEMNrLFTfUMg/NsCIZyPfIui+nOzA4O+Mv6+Gns6J/NTN/Vf77XUT0zUT0B7tGDwQCZ8UeBn8v\nEf2HrIcPRPSzKaVfethpBQKBU2CPFf33iOgb73UVI7rZHN7tHm8iBipxNf99C8d5sAYiIqKr7MDy\ndhZP6/a4dkO+Ud3623LxLFFNEmkmkha3S1PX2fDW255YS+jivopGqibqyxXRwWmF5P6u1fwP5p7R\nECei+jtzvecqttslR3zG+u8J1ajmWO/tmCwvm5K+I6I3RjVxXsnqFhrbiIjSBG1GK98blSMP0BPN\ni8FPLa0x1skzxE0ONfTWR0TW0LcM2v59aiNbIBB4njhDRhcsrztR6L+lTbNcppivnBNjG1sWKwyr\nX21gVMM46htaGP2g1lyE1XvbEXubAyKaTCw6AAacVV6CsdAa2XzGxvobY4z0l8XK0qPjaNRj7D3L\nZHuAWU8xW4sh2I5jS4/ZidTSWTF6iSMKm3oPDZMXZxY1PmwDzZPn4kxQ19lI0HGfLQbEYb+jSzB4\nIHDBOF9Glx4rwHkipR/mN+OYdZuX0/L2u1IupFdls/qsg0v4Zq4XVjMQ5s7jvshvypKBNR9v1HWE\n1UdZdhMmP2JPrzb7KTfnbsBZBZmcqLI5Mvc7wvaODi5/y/ElLJdpp5iig4vtA3XWclQ3d58ls97y\nmLdM1jC2HK2+TaQYXGwrV5ZxByOFdbgP9WwdDCL6dGbuJOfusK+Zq2MXt1kOHTwQCDw0g3svLAhA\nQXfHubBFfZtOwCDCqLfZmeWlalus5XKc4RZLfGetEuZ8ka3nc3473mTXQ8nfds1VB5ckEcV6Dlb6\nPWiyn5okC1b3rrq4dWIhUs4vG8ytGfy2OAmJ1dwGn2greg1AkbmtW9OXghyB3ZOtJtIJGGxYZSMg\naKeVEvIJVnOpz66mydgS8hE+orqIUi8wgGFG9Gv5XtX4ESUhJMvq3C4R0W6s7JeWxmDwQCBA50z4\nAK/nubg/LtV6HXzOr0thlXGwuvitCql7ycstjRubTU9Kd3ohVvM8jiSDOBQGz3ov60ypNkAFr4dW\ndXcOhcGt1X6psyGeGPKpLeLokorM/RKORETvTLlu8nVvzw9hLoEX8JlJwxPFGKI1vTyW1XXw9SMR\nUXGXyBeQUyKZmIUVCezInyvfzra+ZvhQk0q2bo25t9jcsaIXK/2w38QRDB4IXDAeh8Ed66o43mMC\niLl4qdVOEtwwZf3qJrOOWMgHh62HDTrx1tkPgzA2HuV6KkAlj6/zq/fmsmcORDZ5JCauwJBPy+DW\n0t5ayHN5qh+3WM3lnDzTcjQMnp9/R9ryF6r7991DQtJa0cHrkrPvwTYXL7U6kUG+e1k/rwFDuY9j\nF6ibocjA+To55pf1MnY+x1e5jZR3xdV24OyOGlb0QCBARPEDDwQuGudzVYXlk9b9sYogt0Ukp3xc\nGhVRfUUebHONyRJbNZiJaCyuqEU0HySgJJl6ImVkE6MezGFYcXjB7YHcYBOIzy7GtXk7tltEczGc\nFYOaCsoR0VwMly9hmcwYOcHIVkR0Mbq5SWePcHTxHFmoFdmdWJCaNAXTlMv3SKXWEbUQY/ml7KU/\nq9lesrgt6XgkTlzFdcuyWBHfMe7INbZBGR+bJ6JHXvRAIED0WI4uqV+XSl6spSzsMDtheGLsEeau\nL7SOw74CZmS9NWyZXWElMCWz7yEHrFwNbUAJusCiFLHmsjoBg1cpQy8N5jqyjie1rBnc1iGDy1KY\nXiaT8YTd5ZlWBm9dhWfI5NK4FbuOLnDcA3RDFacS4+hij8V4O1iGtZ9CZmpwqClLYcbKlg9ZaCsC\nTcnnJiytvqfI7gkpXA3fMbglNJ6xc46ddh0EgwcCF4wzOrrIEZbNip6ndBsWhrXOB9rBBYF5xCQf\nupQ1g4teKzq3BKxUJodsq6T3IJtN+S5AttZ1uOPIDQSFEOlkDcLUfr41YWndBpn7ZsrSgGrb6t72\nuCqp9aBdPAuz5jLo5BCFuQB0b7meLI95dpnqfCqOTZmFSy41Pb98EF1fzgn5T5bJiaiye2dZ7C5f\nEWPK0C68oYMHAoFHYXCtpzRWzuLpr95ORDRPKgwP3uTWtQSSE5TrWAYXq/AhZ9C8UgEqV5m5hdVK\nyGl+5aI1XZ9Da/mWi6xGk13V0cFxFaAkZFCv8BogYhm7plpqGRyDSpC5javqZBm8PG+wo6wGF+1A\ntZpD0EnRs2vbYj2XOjDDoKXcnCv6tZUcNBuXXVBE/8fjaMt6fpXt7efb3Y9sBTYXvMwNpJkVBIMH\nAheM8+vg5XWaqyVBoX5zlbQ49n1U2Vrvu5xZZlxYVwJSpmwJF3bTSSLGHHYq1nJM1Hg1WNdYjdq2\n/3ZGlsd1cG9vdLQhzDsYvO6+CgwuLO0EkIi1HJlbr2K0696og3tWdNFHV9hdADp3z2XVCzbZkhD0\n96hay+34lZ1V21JnreWNLm7uGV1TyZY9xXljHdx1J2A1+Q0EgwcCF4z4gQcCF4wHdnQRmUvHzPrH\n1BHZlybi5AFZNtww2yzii89BdnwQI5WI7DdKpB4HEcWtiM4QrabF8FJ3hIsqohHVTUYXX0RPYHwj\nqqJ4yToLecxRHCdSmXMgig8NakTV4FkcXFBEB7H13igGNPlyOGJ+x7jWwFlmKrs/yjnH1bYR28GA\n5jmxoDrSaG13cfbR3VsNaBPB4IHABeP8wSby9seXtHpD1SaDqan5rRXbZMYeCjNZp5iy0eBQmVa2\niK3up5axkck1sG4rDl1jhte0a2zbkTcedxyp2U/zGMDWug5dg9GgZq4txs4ZjgnKRMe5qHaMbOiy\n6o4l8T/C5DDG4GwfXBxbitSY603giNTBtUs9jKFxCgYvE3GGD0eXQCBAdIaMLmUXCUyrOnOvS/Pm\nk9xlJb+wvZmcAAAZqElEQVS1YuNBGDw7KkzCvrn+dhaW1o40yNQEZXte1yF2vlgNvJF6zC3Q+jRm\nNe3q7d7SF+jVM5TN3z0XVc/s0LAXPJmVFaOyAlSYO5/Ru5RgZ9GjYYnNxI+Aro3OK1oSbJfDEpTb\niWwy9okYfAk22dc9GDwQuGA8KIOXgHfP+lkM4vKqLe/vZpzmRSj6jwThq/cUsnp1c7VMPqnXLbYp\nR7juYPp0GPwOEQXJeR1jXWFW53wvWUaT/VQvZojlvZQtK68xeCN1OVbuXQ4uMj6wLtbXj1c9//zp\nNPo65jzXedyKBTyfg7KZf8+hpadf6/6de763q2oZiEIHDwQCZ3RVrTs6AnN7TA7MVF6fJVmEw6zZ\npbOyvHRpX8HI4Fv1vTrdZw/WXujI4N4e3N1zwOCli2Z91DuRufX1MXgCyx5zHWNFF8hw8hHJerjT\nVFyaGf0mkPWdEFAMUGks5uQ8547uvcrk7aQ7JxysNWWKYJNAIPBoVnTNxsCSDQvntpNuJH1tOVVa\nVpcSPdqWCdnYU20KC8Ac26Z30rWPQcPgTQOtg+M5GAOY3fTpsbCXfglWOnprw944q49L9hfDxA8r\nTF70aOkLbevun2pKHRZu+mrAvPesbT/wVyPWwQOBwILNHzgzv5+ZP87Mn2PmzzLzRx5jYoFA4P7Y\nI6LfEtH3p5Q+xczvJqJPMvPHUkqfu9MVe0sIJaOLJ0Kj2A7GNt2WYZzSNl/elbd941rvvD3XP3U0\n1kS7FQNNK6J3lqjSSpstxxSnbbMU5hnZOtDDo/tKyaaCy6xrIjSK3WUDQHUdVF2a+pXxoW3v/O5z\n98EpHV1SSn+aUvpU/vuviOjzRPS195lfIBB4HBxlZGPmDxDRNxLRbzrn3iCiN4iIxtdeWyodgk0E\nb300ggHjNn+TZvmOE4DXqVR7VpGOQesYdr6PZeWY5ZNjGGPPuN3loBUGx7JnmCMZp9N3DWh0K/Xq\n0jhe71ZVu9onmbklp+3aOM5U13HM57v3Aqc2sjHzVxDRzxHR96WU/hLPp5TeTCm9nlJ6fXz11b3D\nBgKBB8QuBmfmAy0/7p9KKf380VfRelBzDlwOS8P+q5HXvEm23mxHeKKko1j5lMr4ylVOxfZbbe5g\nDzjK6cMZrhkVlsB2zeWIe26XHnf0zcCV210XPCVOxeC8/Jp+jIg+n1L64fvNKhAIPCb2MPiHiOh7\niOj3mfnTue5fp5T+852uuFuXeSBGPMpb8HFY+cHwwA4Xp0ZXOFn9GNJK6TgcI7Cd+9HuFeQ2f+Ap\npU/QY8mfgUDgpHic3UUdbL4tj3lFHqGXntyN8JTjnfg1epTx9pgHs8c4v2kLeazr7L+v+xi7/Wuf\neDzjq7uvS7iqBgIXjPiBBwIXjEfafNCp7Dhl7GsL469d/AjXwpOqDafCPcTU1RXBZnWJV8+7fVaW\nNvHUqvh7xLjdC2A8uINm2bM3xl3x0GFkGsO+awWDBwIXjLPnRW9cGb1Yk6Yv+X2dtm1wwIo7JV6n\nd/0jgeMfZ/zaX7/FfOY8EjYG46wwePEyZqy3TsnedNfuvV4bvgQrc2qCk1Ylhc6XC6/rneuVvfGP\nQJOHbu8QYWQLBAKPklXVC1NsdpUgaOuE+fUZnI9o60y0y/ZO2w4eK9ZkjbmboIwSOmuK7ni1TWrn\nNGAbuL4T+5NKcvntB7PF3CX0c0UC6bO+4ycN53qZfHTbJoOPF+fUlbYe4Muxc8xg8EDggvHoOdm6\nzI05qlWGS2TqNme1vha5bdqxnDoYYzPY32lzL+zSq3e0RVZ2xm9yjgPrGzZOwI7I6GU3UDXNEtDR\nYfI1K32PuQ2DJ5g/zhHY2WlT8t/jGN6UIBuvG6l8gky7azDPN3TwQCBwNit6w9zIyipDZ3vOrydq\nGbop7+izh/WbcyeAn06q08bRR3sMjoxLpKzmM5SR0XXHZq+wXJQuXgZTKPvMbRt3mVuv/WIbYWyo\nZ5MzX47JHD1WHobkt+2M5bah00KPvjerbzB4IHDBeJRgE09H7jMs6Nn6b2Bub0eKTZZ3+vQt7wnK\n1KLzIl17wR4TFtmyMirW+hyUh05ZtS36M+7B7S1py7Hs8gnr7LrgSQJb6K17CwsbBs9N4BzjUQ+f\n96sbQD8fYB87UydlYHLcidbcRjm3zbLIxN4+dQK9w+yesYmCwQOBi0b8wAOBC8ajG9kah5YiSmfR\nfJKy6rNhZNslzjdjKOPIDkOcaafb9iSlNQmqI4WtuZJ2nUGoitetiM7mPHt95BVfRF6oJ2rupUrx\n4kiTzNRM2z2SZHcpEERzvRWwiNnjbOZdl76sOK7/rkcrho9D/SLJOONgxW0Uv40/UEc0XzOIYdt5\nRURPIaIHAgGNR3J0af/uGcrWWLkckeXX2mambsZ1jGxdBxo0uuE9kc9eTds9zF0GlHMM5XashsFL\nOdnzyiKETJ3yVroJ7pmIyja7ZkNIdb1mVxrsbyaXbNE0gU5gqDNLXsDchY2BpcexftDSZhSmlbK0\nVQwuDDkCK0ubwtbqRnvGtTW2lXNrzC2QNnPiWCYLBAIPvkwmtKPr7BGXxVxHFGTuya9f6nzGLm28\nZTJk+42lPFvX1z/3orJy+xbHcMiGrUkxM7ZpjnWSwtjNZzRSH3JOnh3q/N7nvGeZrFkegwfvLJP1\nmFsYW46aPa/G5UuAjC0sfaUYXM5dsSyh2TYegw8M545g8h48Zp8TlzlvIRg8ELhgPL4VvceKcBw0\nKyNzNwyu3qLSb7blnk5u/gYLe083X875pmXjdLOBhK9X9Vbu6d7VeaW+2YWZW8Zmt56osq30nccd\nVIt2AGByXrG8d3cTUf17R3ReIdLWcp+5C9MqVj7Auauig1u2NnXl3GTHdRj8Kn/ZkF0H56632Nxl\n7vxApsRFWthCMHggcMF4nKSLXiXofmhVX7eM52Nm7lW2L0zesaaTp7dbuiljeVb0Y6zpHfgBJGJt\nBiYfoQ+tMPdomd2wNOjcwjLzitLcTRwhNDG3UsWdgC6qnhUd1rmRua+kPNYvxwEY+1oYF8rLOJax\nhd0PuV6el2bi9pxl2fvq4HP+IGfiWAcPBALnDBftrT07DFu80go7W+bm29q06tzJ9OnVe3UNg6N1\nnYioY3H3gPp6QgpE3ZaoRDHgXugpz9Ho08LqRefOzC07t2KYJxGlzOaoyRUm13OUoIx8bSHqMq4M\n4ungveeyxkAQvukFjiBzF514sMx9MDq41C3H61wW/fpasT0ydluezHWJiA5QN1JryUeMwPITGGY0\ng0/5Ac+JadyZaSQYPBC4YMQPPBC4YJwhq6p/bHK1KckFxesBDGnWyIZtoCyitRLre0a2svwGovoy\n32Tmi1EVR8WDl8Bk3QadV0Rkz2VlMEuTGNXIHmWKYkhTF06QYgVFdeMKidlZ8TNyHF1KTrYdkmTC\ntVNweKmiunY77Yjko4jS+WiMbFYUvx5u83EpvzLWL4WI5K/kNmJ0EzEcxXFdJ2K3qDsohmugIW4G\nEV2L7HWZbIhlskAgcEZHl62caWtGtp7RjWiFuaV8u2ZkS7YsjC31rpENKGpuKauwmSXlisLOKhgE\nmJuFRTNzJ/V8qlFtMOdKGzG2jXVuwgYlQEKYHFiaiOr9yxzgs8PyUmeDS3bZhHqOLsWApj5nMbJB\nWCc6rxyUeCfM/WK8WcrC3JmlLYPbc4W5O0xOVNl4hGUyLB8DzeiTfEZpCFfVQCBwjnDRjC5z4/IZ\ntTryUHTv1LYF5h5ugJ2Bye34op+L8mpZWksK1dfT6uINoysgcyfQvdksTaHOPeQ5ytKX0qeF1TNl\nz1f2SnU1SyUMKA9emFvukXW1mTh+NkVCKIEr1KCwfHuqGR87Yf4zL2fagO6m4LSil74anRuYW87r\ncy+Ghe2RsT0GP2TDDi6TVZ18P4PPDveKPn6Txt1jbTI4M/84M7/FzJ/ZPbtAIPAksIfB/z0R/Vsi\n+omTXNELIdXltXDRzlFb0VHn7jG3ZmO+zSGBhcFtme7L4N7mXVSJC5l8+dvmWZK5FOYe2xSpDBEp\nZWqpHR51cJEeisXasYF0s9o6FvNdzI0AV1jccUQHjsizGxtdHANINIP7zP2u4aWp138LQ7dMbs/r\nuhEYHHVxDWF3dHARTEq8EX18ouF0rqoppV8joj/fNVogEHhSOJkOzsxvENEbRETja68tlY51dVP3\ndtecoS26lmo27jC1lAdhZ6WDC8XJOZpnOy6WiSpdTaALIbN7QCYvDK6VTGB9MW9nZvcs1mJFF/1M\n9DgnFqQmKWwSbiRTr2+lSBG9QCEnHLXn7+AmmGy+BFb31o+tYWoICvF0cGFs1LlfgSNRZexXgLlf\nsJTzGIbBcR3c6uAegxddWj5m4FzN4FMJNhmKdLCFk1nRU0pvppReTym9Pr766qmGDQQC90AskwUC\nF4zHjyYD9LKqGmmma2TzlsnsEUV1LmVlsBHRvBjkJtNHRHQjjoMojgY5g46RDctJG84kF5jUiRFm\ndK7TGGhAVM+ys1lm6jiVFMcaLx6/81ntyn1+D6BbKlE1DmK20yqy2ygw/Tc6seCS2FK3/C0iORrZ\nUFQnqqL4gayovubogqL2BJxrHF3yB3iTrohPFU3GzD9NRL9ORF/HzF9k5u/dNXIgEDg7Nhk8pfRd\nJ72iY3jT9buynnaY3LTpLH0Rls252ZYnqNcMPgN9rTF4D2Bc41mNP4jb6WzKPOd6lf20n4NFmLt1\nFClMLRJPMbrJUT9TYHU0oJFTvguro4uqFJ1dRHB5rLAzGN20q2qPsQ+F0SuDI3O3TJ6XyVSy+OsS\ngJINfWTLGlvx3MUtVXGw1B14iowugUDgjK6qvbDRo5bWMAiFlK4t55rlMqtvExFRYft87lZSlwBz\na4adkMH7CqmEZnJPBwenFn0DRQcvASUtm8kVe66wIpEMeh1Lsqk2zJ2PJumbPRbX3Z7TkpnEyrkO\nWhdVq2/rv3HPsJL9tOjbSgfvBIogk+u6HpN7ji7C2GW5DPK2+ctk+XPoyGGT+k7cFEcXdjO1eggG\nDwQuGGfLyYZlBnZYd1VN7pGIuiGfhYWFraf65kXmZmBw1MWXuqVNanRwxwEhtymzRF/MsgF9G2xS\nlO3RMrd+jE1d7iu6n1jRyXlONFoWXt2xdUPacqWuY9AJOilFVWawnvd0b62Db4V8Cjubug5z17Jm\n8HztJuHDGoNb4LdHO7pI/4n4dFb0QCDwfPFk1sELPAstMsQa28i5yTJrZXTH7bTH3FIWtnYYvLWi\n73MhXC4oDN4mfKBxYe6ehdzo4CVQxLq1pvJ8clnf8yjnRMLJYzihn0XnFmt65/NYgyd5bDVGF1Ve\n0cGbI1mdnGgt5PPWHIk8Xdtn7hd6/OKamnJZGDzPecetC+RbNCibzpivtejg+xAMHghcMB54d1E4\n0oputqrPJb+Ns/bc1b2FvdA7Tf+9xdxKb69WdLu+ntCq7kGs5gS696gWt0VvHzKTl3vMH5lmewkM\nmSAgBXbnZL12XpJZ2HDUNSt6j7HXkmveKWwUUK3oqi4fyy4ieFyxvPeY3AaO2HXuVl/PZaU1X4sn\nGzC2PPZRfWY9Zm118KT+XnCTUujggUAgfuCBwEXjbEa2reyqa+JeuzlgatqWIJDSJ4vDThx3zZ5q\nnVUa0XxqjWxFJNfiO9G6iC5txR1VjGS6DSRPl9vi4ltaz5dspyX7y5znJktgYAgkZTjrLJcZtQfm\ncoxxbRe68d/JHDXQwWUorqr+Upj+eyii9C0ca1sJFBGxXZxYylIYyfn6OYtofihj5OVKKeuceM0O\njhZzatWSqmGlFfdki2DwQOCCcfZlsh58V1Ww2HgGOdiNpA0KcZxWsM00wbE1sqVmmUxYs2XLBvJa\nlrBUYWDdJrsllje1tJktSy/XsrnYynwH6+bKV/UKZclMlstw6dGRoLrBJc7n8BDYs0wmS1Te9r5t\n+Kbto3cgORCyvUgEsBSmqFSY+5A/K2HsAcrL/HxunctSm7B/nf9NEqmBW7fnDoLBA4ELxvkYvKN7\nr+l3yBD+8ozo3sKkBEe7jKb7FL0aA0lK/dT0KXUrji5JrlmYW24g6+AS8IF6PFFVSGVOxe1UM3hm\n+9wmlbxtdhkxGR2c3DZe1tu7OLZsttmrRKqmayGSeK5mMq3PqbK6PVeyn6q2I4R8FtbPx0Mpk+pj\nmfrAknhjMOfXMGaRaio2EO3FJXnRe6EpLYLBA4ELxvmDTY7pA3qigFPLTM2uJA1TqYmgw0zPeUX1\n6TF3WmNyYO6yW+ZsmVzPt4SH4oZjunHv3jCtlLPY0Ps8TqZP4x5le+AkqGiaOI4sRNSEUa7p4CX7\nadGrFYMD27f6urTT15Zriv5smdvTuwfg4rlcV8ra4ahm5wgGDwQCT9CKfox+t7YPWGPxXWHw3jiF\nyR3LO7RZY+52fOnjM/lSKTo3RFwAsxNZCcZcr1fvnGt0csMc/pFRbz8jWh18TV+3OndhdPXlwzRL\n2KasbSsxQ3TvcgTm9nh3ZOBY2V/ODS2VhA+OraaDYPBA4IIRP/BA4ILxdET03hIY3dPgsyfLKYq9\ne5xVxACHovkR12tEdRNMBsZBuK47t66qsabKnMqa9jSwN9voXmxlPzX7Ra4Y05pxUTTH+qLqtWrb\nfIRKFAweCFwwng6D3weOo8uzhDHQjd1mDYrx74g+W4/pDI9xyw9kzVW1B7305eUn122MowtEZg+O\nY8teDMXo9vh8GgweCFwwLoPBA//f4hh9tIep+AqvXOcY39onhGDwQOCCcRkMrl+uO8PoTnZpyYha\nnFR2WNN7c1Q6GqODi0B8GHUmgOEO72nxm+nO5fgh74st80lSbL2XuSfl0js3u7Ba2H3AeuGc6+Wl\nThxoxlxunaDuoo9Pd7AvBYMHAheMp8PgsLOkSerZ7HhxzLg9hmp3EcHUR4Ul5c3vuXr2vAb3SBIl\n6MRhZcTaObxm73hXbHV/QurpMTr5vIPfps7NucydJBBl6SOMWwNH9PdnPX++tJ2dK01HLHEEgwcC\nF4zHZ/D7vO0bJm93BEkoAeCuHx6bNYw3mKOsv+rrVLa3ASNp3v/OLMxdruPo1YOdS3M/3vybCznn\nSx3l+5Ajm7Idxx67+rvpc4cF9TtYxpG5pazrRR9HXVxYWrO1tBGWl10+pY3sODKZAJV8rPnC8v/e\n/WzZA+Y859TUzSnt5vBg8EDgghE/8EDggnE+I9sxhhtoW2xe7LUFg1VpC4a0US1JiatnEYuljPJr\nm0Wl5CuRLYC4XRJpgEskZa6qXjYflHmCOO+L2zAOqDCe4TI16kk73QcJ9z4iJqcm6VGqmCOCE1Vx\nWERsvdw1gyiOIrteUkOR/Cafq9lgZCPACnEUvim7PsqXL2dodQJHELgUpo1sMu7JjWzM/K3M/IfM\n/EfM/K92jx4IBM6KTQZn5pGIfoSIvpmIvkhEv83Mv5hS+twpJoAs7LFFazhb6dNh7JJES4xgOjtL\nYTy7GaBs/CfQl8F3aNlNpLwytwM/uMO45hwa14TR9UaFo+R0s9JLwn1rze59HSObtF1j8j2fwyMt\nnaExbZIy2frlnGXsm5QzmJI1qC3nlp/GQW3Z6x1vDOMuf1+znLNMrjOxbC3nCUvPanypu0nJZPRZ\nwx4G/yAR/VFK6Y9TSi+J6GeI6B/tGj0QCJwVe3TwryWiP1HlLxLR38VGzPwGEb2Ri+984Z//y8/c\nf3qPgr9GRH927kkcgec03+c0V6LnNd+v29PoZEa2lNKbRPQmEREz/05K6fVTjf2QeE5zJXpe831O\ncyV6XvNl5t/Z026PiP4lInq/Kr8v1wUCgSeOPT/w3yaiv8XMf4OZr4noO4noFx92WoFA4BTYFNFT\nSrfM/E+J6L/SYhr+8ZTSZze6vXmKyT0SntNciZ7XfJ/TXIme13x3zZX3mtsDgcDzQ7iqBgIXjPiB\nBwIXjJP+wJ+TSysz/zgzv8XMT369npnfz8wfZ+bPMfNnmfkj557TGpj5BTP/FjP/bp7vD517Tltg\n5pGZ/zsz/9K557IFZv4CM/8+M396a7nsZDp4dmn9H6RcWonou07l0npqMPPfJ6IvE9FPpJS+/tzz\nWQMzv5eI3ptS+hQzv5uIPklE//gJP1smoldTSl9m5gMRfYKIPpJS+o0zT60LZv4XRPQ6EX1lSunD\n557PGpj5C0T0ekpp0ynnlAz+rFxaU0q/RkR/fu557EFK6U9TSp/Kf/8VEX2eFg/DJ4m04Mu5eMj/\nnqw1l5nfR0TfTkQ/eu65nBqn/IF7Lq1P9kv4XMHMHyCibySi3zzvTNaRRd5PE9FbRPSxlNJTnu9H\niegHyE+19hSRiOi/MfMns4t4F2Fke0Zg5q8gop8jou9LKf3lueezhpTSlFL6Blo8Hz/IzE9SDWLm\nDxPRWymlT557Lkfg7+Vn+w+J6J9kddPFKX/g4dL6gMi67M8R0U+llH7+3PPZi5TSXxDRx4noW889\nlw4+RETfkfXanyGib2LmnzzvlNaRUvpSPr5FRL9Ai3rs4pQ/8HBpfSBko9WPEdHnU0o/fO75bIGZ\nv5qZvyr//S5aDK9/cN5Z+Ugp/WBK6X0ppQ/Q8p39lZTSd595Wl0w86vZ0ErM/CoRfQsRdVeCTvYD\nTyndEpG4tH6eiH52h0vr2cDMP01Ev05EX8fMX2Tm7z33nFbwISL6HlrY5dP537ede1IreC8RfZyZ\nf4+WF//HUkpPfvnpmeBriOgTzPy7RPRbRPSfUkq/3GscrqqBwAUjjGyBwAUjfuCBwAUjfuCBwAUj\nfuCBwAUjfuCBwAUjfuCBwAUjfuCBwAXj/wHNiiHJknTfXgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8b8d3c8898>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Contour plots\n",
    "# How to plot sin(x)*sin(y) heatmap?\n",
    "\n",
    "xs, ys = np.mgrid[0:5:100j, 0:5:100j] # genertate mesh\n",
    "Z = np.sin(xs) * np.sin(ys)\n",
    "plt.imshow(Z, extent=(0, 5, 0, 5))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXd4VNXTx7+T0DtC6FVAEBBQQo+IEGqoIhiU9gOMiCDw\n2kAUFBULCoIgShMQAVF6rwFEaQFCr0GlQ6SXQEh23j9mQwrZZLPtbpnP89xnd+89e8/c5O6dc+ZM\nIWaGoiiK4nv4GS2AoiiKYgyqABRFUXwUVQCKoig+iioARVEUH0UVgKIoio+iCkBRFMVHUQWgKIri\no6gCUBRF8VFUASiKovgomYwWIC0KFizIZcqUMVoMRVEUj2H37t3/MXOANW3dWgGUKVMGERERRouh\nKIriMRDRv9a2VROQoiiKj6IKQFEUxUdRBaAoiuKjqAJQFEXxUVQBKIqi+CiqABRFUXwUVQCKoig+\niioARVFcT0wMMGMGcO+e0ZL4NHYrACIqSUThRHSYiA4R0cBU2hARjSeik0S0n4iesbdfRVE8mO++\nA/73P6BXL8BSXXKTCRg/Hjh/3rWy+RCOmAHEAXiLmSsDqAvgDSKqnKJNSwAVzFsYgEkO6Ne72bUL\nOHzYaCkUxfEwA9OmAXnyAHPnAiNHpt5u0SJg4EDg229dK58PYbcCYOYLzLzH/P4WgCMAiqdo1g7A\nLBa2A8hHREXt7dsriYkBBg8GatcGQkKA+HijJVIUx7J1K3D8uIzue/QAPvoImDMneRuTSfYDwPLl\nrpbQZ3DoGgARlQHwNIAdKQ4VB3AmyeezeFRJ+DbMwPbtQGCgjHiaNAH++QdYutRoyRTFsUydKqP/\nF18EJk8GnntOzEFbtya2WbgQOHgQCAoCjhwBoqKMk9eLcZgCIKJcABYAGMTMN+04TxgRRRBRRHR0\ntKPEcy/mzgUmTAB+/x3YvBkYNQqoWhWoVw+4dg1YvRpYswYoUwYYO9ZoaRXFcdy4Afz2G9ClC5Az\nJ5AlC7BgAVC2LNC2rTzsTSbg44+BSpXEVAQAK1YYK7e3wsx2bwAyA1gD4P8sHP8RQJckn48BKJre\neWvWrMlex7ZtzDLeT74FBTF//z3ztWuJbb/5Ro5FRCQ/h8nkWpkVxVFMmiT39K5dyfefOsVcuDBz\nqVLM48ZJmzlz5FilSsxNm7peVg8FQARb+ex2hBcQAZgG4Agzj7HQbCmA7mZvoLoAbjDzBXv79kiG\nDQMCAoC//wYiI2Wk//ffwB9/AK+/DuTLl9i2d28gVy5g3Dj5fOOGrAuUKSOzBEXxNKZOBapVA2rW\nTL6/bFlg5Urg6lVZ+K1UCejcWY61bi0z5Vu3XC+vl+MIE1ADAN0ANCaiSPPWioj6ElFfc5uVAE4B\nOAlgCoB+DujX89iwAdi4EXj/fXmIV68ONGsm71Mjb15xk5s3D9i2DWjQAFi7FsicGWjZUo5dv+7K\nK1AU29m6Fdi9G+jTByB69Pgzz4g5KG9e4PPPAX9/2R8SAsTGAuvXu1ZeH4DYkg+uGxAYGMheUxCG\nWWz8584BJ04A2bJZ972oKKBCBXmfN6/8QBo0ENe5L78U5bFvn9hTFcUdYQamTJGR/WOPAQcOyKsl\n4uMTH/4A8OCBzJo7dkxcE1AsQkS7mTnQmrYaCewqli0DduwARoyw/uEPAOXKAa+8AjzxhHgJNW4M\nZM0KfPYZsGqVKIjRo50nt6LYw+3bwMsvA6+9Bjz7LLBnT9oPfyD5wx+QGW+LFrIQbDI5T1YfRBWA\nKzCZgA8+AMqXF7/njDJzpnhHVKyYfH/TpmIn/eor4MyZ1L+rKEYyciQwf754uq1eDRQubNt5WrcG\nLl2SAEnFYagCcAUrV8q0d/hwGc1kFD+/1G2mgDz8TSZgyBD7ZFQURxMTIyabjh2BoUPlPraVFi3E\nZTQoSEyp774rJqUmTYAiReSzkmFUAbiCsWOB4sWB0FDHn7t0aeDttyWScts2x59fUWxl/nzx6unn\nAJ+PggVlEfntt0WRfPutKJfbt4FSpYBvvpFBlpIhdBHY2ezbB9SoAXzxBfDee87p4/ZtWSMoWVKU\ngD0jLUVxFHXqiOvmoUOWZ7C2EhsLZMok9/qVK+IoERgobtWO7svD0EVgd+Lbb4EcOYCwMOf1kSuX\nuM3t3An8+qvz+lEUa4mIkPuxXz/nPJCzZEkc6BQoIObVdevEMUKxGp0BOJpbt4DcueX9xYtiounT\nB5g40bn9xsdLcM2NG8DRo+IppChG0bu3DEbOnRP3ZWcTGyvpVPz9gf37bVtr8xJ0BmAU+/cD+fNL\ncqt164BJk+TGHPhIiQTH4+8v7qD//ON8ZaMoaXHtmqxJde3qmoc/IDOC0aNl8JMQOa+kiyoAR7Jo\nkXjkREVJhO8nnwBt2oh93hU0bSr9fvqp/AgVxQgmTpRKX6+/7tp+27YFWrUC3nlHTE8xMa7t3wNR\nBeBIVq+WPP5RUcCPPwL160vglysZPVrSQ3z2mWv7VRRAUj2MHAm88IKkOnElRDIIe/ttmX3XqSMz\nAsUiugbgKK5ckXD14cMTC1kYxf/+J1PwY8cs5xlSFEdz65bk87l3T7zf0ov4dSarVwPduomL6O7d\nxslhALoGYATr1knOk5YtjZZETE9+fqKMFMVVDBgAnDoFzJ5t7MMfkMCxN98E9u5Vc2gaqAJwFKtW\niTtaoFWK17mUKCE3/+zZMhJTFGczd66kLPngA3GCcAcaNpRB2Z9/Gi2J26IKwBGYTDLlbNbs0URW\nRjFkiHhgDB1qtCSKtxMdLaP/unWBDz80WppEatcW76AtW4yWxG1RBeAIIiOBy5dl2uku5M8vdQdW\nrQLCw42WRvFm/u//gJs3pdhLpkxGS5NI9uxArVqqANLAIQqAiKYT0WUiOmjheCMiupGkYIx3GacT\nog+bNzdWjpQMGCDpId57T6bCiuJo1q4VU+N77wFVqhgtzaM0bCiLwHfuGC2JW+KoGcAMAOkNf/9g\n5hrmbaSD+nUPVq8W7wdbU906i2zZZEF41y6JDVAUR3LnDtC3r8S5DBtmtDSp07AhEBcntTSUR3CI\nAmDmLQCuOuJcHgOz2D537pQEbO7g/ZMa3bvLNny4LNQpiqMYOVLqWU+enLEiR66kfn3xiPvjD6Ml\ncUtcuQZQn4j2E9EqInLDuWIG2LJFErAVKiTBJvHxEvHrjhDJD7RhQ4kP+OsvoyVSvIGTJyXNeY8e\n7uP1kxp58kg2Xl0HSBVXKYA9AEoxczUA3wFYbKkhEYURUQQRRURHR7tIvAwyb548WMeNA37/XXyN\n69QxWirLZM0KLFwo6wHt2kllJUWxh3feEQ+bzz83WpL0efZZmaXHxhotidvhEgXAzDeZ+bb5/UoA\nmYmooIW2k5k5kJkDAwICXCFexlm7VmrzvvmmVDuqUcNoidKnQAEp0PHff8DSpUZLo3gy4eHA4sXi\nZVa0qNHSpE/DhhKd7GMRwdbgEgVAREWIJCk4EdU293vFFX07nKioxGRvnkaNGvKD3bTJaEkUTyU+\nHhg8WFIsDB5stDTW8eyz8qpmoEdwlBvoXADbAFQkorNE1JuI+hJRX3OTFwEcJKJ9AMYDCGV3TkKU\nFuvWyasnKgAioFEjGcF56J9fMZiffpLo8q++Ej97TyAgAKhUSRVAKjgkaoOZu6RzfAKACY7oy3DW\nrpUiLxUqGC2JbTRqJN5AJ064Lk214h1cuiT+/g0aAJ07Gy1NxmjWDPjhBylQU7y40dK4DRoJnBHi\n4oANG+Rm8tS6o88/L69qBlIySv/+Un968mTPu/8HDhTz1ejRRkviVqgCyAg7dkjIuyeafxIoXx4o\nVkzTQygZ4/ffZRsxAqhc2WhpMs7jj0t66B9/VC+4JKgCyAhr10pQSePGRktiOwnrAJs26TqAYh1X\nrgBvvAE8/bS4f3oq778vrqDffGO0JG6DKoCMsHatJJcyOte5vTRqJAXrjx83WhLFE3j7beDqVWD6\ndM8utl6hAhAaCnz/vbhDK6oArObaNUn74MnmnwR0HUCxlkuXJNlbv36eEe+SHsOGAXfvAt9+a7Qk\nboEqAGvZsEHy/nuDAihXTjwhdB1ASY+ffxbnh75902/rCVSuLMGb48fLrMbHUQVgLb/9JtG07pzy\nwVp0HUCxBmZg2jSgXj3gySeNlsZxDB8u9YvHjjVaEsNRBWANN25I+oTQUM+2gSbl+edlen/okNGS\nKO7Ktm3A0aNA795GS+JYnnoKePFFyeXl47MAVQDWsGCB5BLp1s1oSRxH8+YSydm+vfzIFSUl06cD\nOXN6XtCXNYwYITENPu4RpArAGmbPFv/52rWNlsRxlCghawC3bskUf8MGoyVS3Inbt4Fff5WHf+7c\nRkvjeKpWBTp1krUAH/YIUgWQHmfOiK28a1fPi35Mjzp1JLiteHGpZ6y5UpQE5s8XJeBt5p+kDB8u\nVc3GjDFaEsNQBZAec+bIYljXrkZL4hzKlJEiMblyATNmGC2N4i5Mnw5UrCgVtbyVKlVkhvPddxLh\n74OoAkgLZnGDq1dPXCe9lTx5ZAawYoW4uiq+zcWLwJ9/euesNyVvvSUznTlzjJbEEFQBpMX+/eIl\n462j/6S0bg1cvgxERBgtiWI0K1bIq7uWOXUkgYFA9eqSI8gHXaJVAVji7l1g0CApe/fSS0ZL43xa\ntJA8Rwk/fsV3Wb5cyodWq2a0JM6HCAgLAyIjfbJimCqA1IiJkdq5mzdLIEyBAkZL5HwKFBB77/Ll\nRkuiGMm9e5Lzqk0bh5t/4uKA7dvdcKD9yiviEj15stGSuBxHVQSbTkSXieigheNEROOJ6CQR7Sei\nZxzRr1O4dw/o0EHcIn/6ye3MP8xONNOHhAB79kjRDMU3CQ+X2W/r1g4/9dixspz20UcOP7V95M0r\nQZ5z54pbtA/hqBnADAAt0jjeEkAF8xYGYJKD+nU8I0YAa9YAU6YAPXoYJsaxY8CBA4kP+/h4YN48\ncVyoWdNJo6iEH/3KlU44ueIRLF8O5MiRmDDQgcyeLYH0I0dKRcnUOH1aarh//bVkbnYZYWGyGDxv\nngs7dQOY2SEbgDIADlo49iOALkk+HwNQNL1z1qxZk11OhQrMLVu6vl9mPnSI+YMPmJ98klke8cyP\nPcbcrh1z5cryOSBAXrdudYIAJhNz6dLSoeJ7mEzMJUs65f9/4IDct99+yxwaKu+/+y55m/h45saN\nmf395XjFiswrVzKfPct8+DDzrl3M9+87XDTBZGKuWpU5MNBJHbgOABFs5XPbVWsAxQGcSfL5rHnf\nIxBRGBFFEFFEdHS0S4R7SFSU1Mpt2dKhp716FfjySxndpMRkApYtA4KDZXQ/ahRQpAgwcaK45bdv\nDxw8KCOnX38FTp6UAdqsWQ4VUSCSWcC6dWIKU3yLAwck8NEJ3j9z5gD+/mJpmTVLltgGDJCRfsJs\n9rvvgI0bpXTvihUy623VSoLWK1eWUhyff/7ouW/fdsBsgQjo00e84HypToa1miK9DWnPAJYDCEry\neQOAwPTO6fIZwMSJMvQ4ftwhp4uPZ54+nblgQTltq1bJj9+/z1ynjhwrXpx51CjmixfTP2+3bsx5\n8zLHxDhEzOSsXCkCrVrlhJMrbs2nn8r//vx5h542YWLZvHnivnv3mDt3lu7efJP54EHmbNmYW7eW\n9gltZs1injyZed485ho1Hh2gm0zMVaowt2mT+D2bOXlSBBo/3s4TGQsyMANQE1BS2rRhfvxxB9xJ\nYs6pX1/+wvXrM7/+urzftCmxzZdfyr6JE5ljY60/99q18r3ffrNbzEeJiWHOlYu5Vy8nnFxxa+rW\nZa5Vy+Gn3bpV7tdZs5Lvj49nHjxYjuXIwVygAPOFC5bPM3IkMxFzdHTivgTTEsC8aJEDhC1Xjjkk\nxAEnMg53VAAhAFYBIAB1Aey05pwuVQD37jHnzMncr59dp4mNZf7kE+YsWeSG/uknudHv3pVRfp06\nol9On5bubDG3xsUxFysm+sop9OzJnDs38507TupAcTtOnZLHwSef2H0qkym5rb5fP+bs2Zlv3ky9\n/ZgxzFmzMi9cmPZ5t28XEefOTdz32Weyr3x55lKlmG/ftlP4fv1EG927Z+eJjMPlCgDAXAAXADyA\n2Pd7A+gLoK/5OAGYCCAKwAFrzD/sagWwYYP8OZYutfkU168zP/20nOall5gvXUp+fOpUObZwIfOL\nL8qU9++/bevr3XeZM2VK3ocDJi5CeLgI+ssvDjqh4vYMG8bs5ycjEzs4dIi5dm0ZAIWFMR89KibQ\nl15K+3vWzIDj4sQpomfPxH116kh/W7bILfv++3aJz7xkiZxo40Y7T2QchswAnLG5VAG88w5z5szM\nt27ZfIoZM+Qv+vPPqR9/8IC5UqVETx57BlsJU98xY5gXLGAOCmIuXFh+gHYTH/+o0VbxXh48kCll\nykWqDJ7i88/lwV+wIHPXrvI+wTyzZIljRO3cWUQ1mWSpApClC2ZZG8ucWZSOzdy8KSOr995ziLxG\noArAFp56SnzQ7KBLF3kIx8dbbrNwIT+cstq7iJsw2wCYy5aVvkuWFLc5u/ngAxkRnjvngJMpbs3S\npWyPEf3BA+a2beUUL76YOCs9d475//5PFnYd5b45bZr0s3+/LA4DMhhiFgeKPHmYq1e3cx37uedk\nxdlDUQWQUc6elT/FV1/ZfIq4OLH5d+uWdjuTSbx9du+2uauHLFokIQsLFkj/e/eK6b5aNTFH2cXx\n4/I3+fJL+wVV3Js2bZiLFMmYJ4IZk4n5tdfkVhk3zgmypeDMGelr9GhZqy1bNrnpc9UqWVsrVSpR\nMWSYhIUFa1zy3BBVABkl6bDCRnbuZLcwm69bJzPY+vVlscwuk269ehKB5rDFBcXtOHtWZnpDhtj0\n9VGj5L638es2Ubmy3N9ZszIPHPjo8d27mYsWldnA+vU2dBARwWnact2cjCgATQYHSOqDYsWkTJyN\nrFkjsSRNmzpQLhsIDpZAm/37gS5dgFKlpJrl2rU2nKxHD+DwYckPpHgnM2ZINGKfPhn+6ty5wPvv\nSy61zz5zvGiWaN5cahjdvy8BZSl55hkpdFeqFNCxo8S2ZYinnwYCAuRH7eWoAoiJAVavBtq2tSv7\n4Zo1cuMFBDhQNhvp0gW4dk2CGseNA7JmlWzPw4ZJRkareekl+fLUqU6TVTEQk0my3TZunOGCRzEx\nwODBQN26UjzMz4VPkubN5TVfPiAoKPU2JUsCixfL/d67dwZzZ/n5yUhu7VqvL5CkCmDdOqkL2qGD\nzae4cQPYtk0esu5CpkySNO7NN4Fdu+RHMGqU5Pg6fNjKk+TLJ9lQZ8wAXJ2WQ3E+u3cDf/9tU9LD\nadOAS5eAL76QkhmupGFDyd4cEiIpUixRrpykmli3Tuq9ZIiWLaVA0qpVdsnq9lhrKzJic8kaQM+e\nklfBDjeFBQvEZLhliwPlcgK//CJ2UT8/5j59rPQWOnxYLu6jj5wun+JiPvxQbob//svQ1+7dYy5R\nQlyPjVoe2rXr0Tib1DCZmJs2lYXhkycz0MH9+5KNrlw5J+VccR7QNQAriYsDli6V5FcZGMasWyeJ\n25Ytk89r1gC5c8t02J15+WXJdzdgADBzJlChghUVIJ98UhLETZggeeIV72HZMikClMGCRzNnAmfP\nAh9+aFzJ4MBAoFCh9NsRyWwlUyagUSOZ0H71lRXFv7Jkkex0UVEyjfBWrNUURmxOnwEkRP8uWGD1\nV+7eFdczIvnqoEHictahgxPldAJRUTIb6N7disabN8vFTprkdLkUF5HgT5lBN9/YWOYyZST61pOc\nw9atE7fRkiXlsv39mRcvtuKLnTrZF7JvAFA3UCvp31+SlGQggciIEfJXW72aecAAfhiI9cMPzhPT\nWbz6qqQ9sZSj5SEmkyQJK19eAg4Uz2fSJLlxDx9Ot+n588zLlzMvWyZWIzszphjO5cuSQiJLFlEM\naXL6tPxIPKhGhioAazCZxJDZvr3VX4mKEt/j0NDEfYsWMbdoYZ090t3480+5A6ZPt6Lx/Pn8MJGR\n4vm0amVV5ttr15gLFUoc6ADMzzzjWaP/1LhyRYL/c+SQ30GafPGFXPj27S6RzV4yogB8dw0gIkIM\nmRnw/hk0SLwOkpoE27cXRwFr7JHuRr16wBNPiJNPunToAJQubYM7heJ23LkjNa+tKPz+4YfAf/8B\nCxeKb/3OnfJVo2z/juKxx8TLs1gxWeK6ciWNxq++Kq6hXugR5LsKYOFCWRlKo/j18ePAwIHiV9+o\nkayZDR8OFE+1lpnnQQT07Als2SJrXWmSKRPQubP8+q9fd4V4irNYv16iqNKp/BUZCXz/PdC3r+j/\n2rWlKle+fC6S08kUKQIsWiS38xdfpNHwscdk1dmmaEr3xjcVADPw++/yVH/ssVSbXL8ufv0//iiT\nhdhY4PXXRSF4E926iSKwqsRkx47iOZXg/qR4JsuXA3nyAM8+a7EJM9C/v/w8Pv3UhbK5mKpV5Tcw\nYYIYBCzStKlMf27ccJlsrsA3FcCePVJcNzQ01cPMEhl/5gwQHi5lgv/6S0ZDrg56cTYlSsi9PXOm\nFUGPtWrJFxYscIlsihMwmUQBtGiR5s3888/An39KLev8+V0onwF8/LHUHx45Mo1GzZpJo/Bwl8nl\nChyiAIioBREdI6KTRDQkleONiOgGEUWat+GO6Ndmfv1VjPkW7P8TJ8oz7vPPxU7u7fTsCfz7r0QK\n376dRkM/P+CFFyTwIc2Gituybx9w8aKE0VrgwQNg6FCgTh25N7ydMmXEzDV9ehr14OvWBXLmlCAg\nb8La1WJLGwB/SKWvxwFkAbAPQOUUbRoBWJ7RczvFCyg+Xhz3W7dO9XBEhLiHtW6ddl5/b+LuXSmF\nAEhQ9Ntvp+EaummTNPz1V5fKqDiIMWPk/3fmjMUmCQ5fy5e7UC6DuXhRooU7dUrDwykkRFyh3Ry4\n2AuoNoCTzHyKmWMBzAOQSo4+N2H7duD0aUl0lgoffCBT3hkzXJvgykiyZ5e13W3bJNHWN9+kYfcN\nChKXJzUDeSbh4ZIetkQJi00mTRKHL3fKbeVsChcG3noL+O03oEkTeUw8QrNmYjr+5x9Xi+c0HPGI\nKw4gacLVs+Z9KalPRPuJaBURVXFAv7Yxbx6QLZtk/0zBhQuy0N+rV4aj472CunXFOhYcnMY6r7+/\n+L6uWCEpIRXPIT5eXL6ef95ik6NHRUe89pr8q32JDz+U7LkHD4rpt2NH4N69JA0Scr17kRnIVWPc\nPQBKMXM1AN8BWGypIRGFEVEEEUVEOzoDZXw8MH++2D/z5Hnk8C+/yBqZDckRvYqQEODIEeDUKQsN\nOnYUX3Iv+iH4BJGR4sViVgAPHogj0LRpiU1+/FGWx3r3NkhGA8mUSbLnnjolloCFC2VA9JBKlcQH\n3IvcQR2hAM4BKJnkcwnzvocw801mvm1+vxJAZiIqmNrJmHkyMwcyc2CAo5Prb94sOWxT8f5hFk+Y\nOnWAihUd262nkRAasWKFhQbPPy92MjUDeRYJHiyNGgEADhwAtm6VOKdffpFcfzNmiH73xMBGR5Er\nl3gEVawITJ6c5ACRmIE2bJDBpBfgCAWwC0AFIipLRFkAhAJYmrQBERUhkthBIqpt7jet2DvnMG+e\n/HdbtXrk0N69MvXz9dE/IHnUK1YUb8FUyZxZ/oarVnl9wQyvIjxc/rFFiwJIzIhZrZrc96++KvEv\nr79uoIxuAhEQFibu3wcPJjnQtKlUW/KSKnl2KwBmjgPQH8AaAEcAzGfmQ0TUl4j6mpu9COAgEe0D\nMB5AqHm12nXExcmcrk0bIEeORw7PmiVu0RbWhn2O1q2BTZvS8PZs2VKKxKSbV1dxC+LigD/+eDj6\nB+Rfly+fLAvUrAnMmSNpztOID/MpuneXZ8KUKUl2Nmkir+vXGyKTo3HIGgAzr2TmJ5i5HDN/Zt73\nAzP/YH4/gZmrMHN1Zq7LzH85ot8MsWWLJPx48cVHDj14IDd/27YWA4N9jpAQiX62eJ83by7DJC/M\nj+KV7NkD3LqVbAF4924pY5onj/wb27WTWBBPz/PjKAoWlLCXWbOS+DsUKgQ89ZSYgbwAH3F0hIz+\ns2dPLCiahFWrZDDbvbsBcrkpQUHyYLBoBipYUCKDVQF4Bins/7GxwP79kuIGkIHP4sWpOsf5NGFh\nYhZLttzVpImESSdzEfJMfEMBmEyiAFq2lGi+JMTEiM97oUK+5fecHpkzi65cuTINM3+rVpIi8r//\nXCqbYgObNkl1t8KFAYhdOzZWTD+KZRo1krCJZIvBTZrIw3/bNqPEchi+oQB27BAn/xdeSLbbZAL+\n9z9J9vbDD2kXmPZFWreWP9vevRYatGwp7lNe5BbnlTx4IPb/FOYfQBVAeiQsBv/xh3hNAZCq9P7+\nXmEG8g0FsGCBPN1TpH4eMUL8fL/8MkNlAXyGli3lB9C5syjKCRMkiPohgYFiClIzkHuzY4fEbaRQ\nAPnyAY8/bqBcHkKvXmIO/egj8448eSQ3tioAD4BZzD/BwTDlzouoKIlyffddMf306QO8/bbRQron\nAQGSFqBiRTEFDRggBWTee8+cFdfPT+xEq1erO6g7s2SJDICaNXu4KyJCFoB1wTd9ChSQNBELFwK7\ndpl3Nm4sH27eNFQ2u7E2aZARm0OSwe3ZI5mtpk7lli2Tl7Zr00aKXCvpYzIxnzwpReQB5oIFzXVh\nf/lFduzYYbSISmqYTMzlykndUjP370vCw3feMVAuD+PmTbnnmzY179i4kd21ODK0JGQSFi4E/Pxw\nvWFbrFkDvPyyBHdcuwYsXap2f2shkgCxmTPFfFCokMwITMHN1B3UnTl8GIiKgqlt+4e7dAE44+TO\nDbz/vmQ/CQ+HJAvKls3jzUC+oQAaNsSG/QEwmYB+/eR/5y1l7YzgmWeAYcOkhsAfRwqKPdRi3gjF\nUBZL2q2qQ9tg6lTZlbAAnOACqljH669LEtVhwwDOmk18pVUBuDGnTskIqF07rF2buHaj2E/79pJV\nY9YsSHT1rl3A+fNGi6WkZPFi/F24Do7cKIa+faWWjy4A20a2bFITfNs2WRNDkyYynbp0yWjRbMa7\nFYB5VMosJJUCAAAfiElEQVQhrbFmjazbqMnHMeTIAXTqJPnTY5qZyz9YjBpTDOHMGSAiAjOvt0e7\ndhLA2qmTWOt0Adg2evaUWgmffAJwY3NaCA/Oiuv9CuCJJ3CCy+Pff1MNAlbsoHt3yS6w+EQVGU4u\nWWK0SEpSlkpOxnn322PgwMRa8KdPq/3fVjJnBoYMEc/ajTdqAiVLAnPnGi2WzXivArhzR6IfQ0Ie\nxikl8YJTHEDDhkCpUsCsn0kSyWzYoLWC3YklS3A6e0XElauE556TVPbLl0sN3JYtjRbOc+nZEyhW\nDPh0lB/QrZvY1S5eNFosm/BeBbBhA3D/PtC6NdauFQ8WtXk6Fj/z/b92LXClQVv5e69ZY7RYCgBc\nvw4OD8ecmPbo3TuxvGmNGsDff6dZFExJh2zZJI5o0yZgd+VuUhvAQ2cB3qsAli8HcudGbO0ghIfr\n6N9ZdOsmMWAzo4Iko9jSpel/SXE+ixeD4uKwzK+91rhwAq++KoGSH8yuJJ4lM2caLZJNeKcCYJZl\n+mbNsG13Fty+rfZ/Z1GxorjVfvF1JlwPChHFGxdntFg+j2nmLJzyL4+CIXVQrJjR0ngfOXJIdPDq\n1cCpoO7Avn2yeRgOUQBE1IKIjhHRSSIakspxIqLx5uP7iegZR/RrkX37gHPnHpp//P11yutMfvpJ\n/sZvbW4HXL0qqXIV4/j3X/htCsdP8d3R51V19XEW/frJGnCHeS+BM2cGfv7ZaJEyjN0KgIj8AUwE\n0BJAZQBdiKhyimYtAVQwb2EAJtnbb1qYlok74vgTLTFjhoxQU6kBrziIihWBjRuB8CzNcR9ZcPUn\n9QYykkNDZwMAbrbpmjL/oeJAcucGFi0Cjl8tiK15QsC//OJxs19HzABqAzjJzKeYORbAPADtUrRp\nB2CWOVXFdgD5iKioA/p+hJgYYM8nK7ATtTBwVGEEBEiBZ8W5PPkksHxTLmzJEow7sxci6oQmhzOC\n3RGMzPNmYW/uhvhyfln19XcyNWsC06YBY650B1286HGp0R2hAIoDOJPk81nzvoy2cQjZEYOKWf5G\nlvYhOH8eiIxU84+rqFwZqDTyFZSM/xdDg/7Av/8aLZFvceECMKLlTjzBx/H4xz2QLZvREvkGL78M\nVBwcgjMogTPdhuKfEw+MFslq3G4RmIjCiCiCiCKio6MzfoLs2ZH7xjnU+OUdFHXKHENJi5ID2iM+\nVx60uz4DjRvLUoziGj74AGh9dRZMWbMhb+9Ha18rzuOz0VmwNmQ8Sl7dj+8rjUOvXp6RIcIRCuAc\ngJJJPpcw78toGwAAM09m5kBmDgwICLBNIn9/WaZXXE+OHPDv8hJC/X/D3cu30ayZV5ROdXuOHgXm\n/HQf3TLPhd8LHXTRy8X4+wO9l7VHTHAbfOI3An/M/hf9+xstVfo4QgHsAlCBiMoSURYAoQBSOoMv\nBdDd7A1UF8ANZr7ggL4Vd6RnT/jH3MHasN9x+DAwapTRAnk/H3wAtMu6GjnvX5McHYrrIUL2qd8h\naxZgSan+WLiA3d4MarcCYOY4AP0BrAFwBMB8Zj5ERH2JqK+52UoApwCcBDAFQD97+1XcmHr1gAoV\n8NTuGejWDfjiC+DQIaOF8l527ZKqp+88aU7206SJ0SL5LqVLAyNHonLUcrSlpZg40WiB0oakgIx7\nEhgYyBEREUaLodjCqFHAsGG4sjMKFVs+jooVpbC2n9utOnk+TZsCkXsZl7KVgl/dOsDvvxstkm8T\nFweULo0//Rui9a25OHsWyJnTdd0T0W5mtqrag/4cFefQrRtAhAIrZmHMGKnCNnmy0UJ5H+HhwPr1\nwJjeh+B37izQooXRIimZMgFBQQiM/RPXr5trZrgpqgAU51CyJBAcDPz0E7q9HI/gYCkm76FJE92W\n0aOBIkWA0LzmkpyqANyDBg2Q9dIZhFQ7g3HjJF+WO6IKQHEer70GnD4NWrYU338vQXpDHkkUotjK\n0aNS3KVfPyDz+lVS8aVECaPFUgCgfn0AwJCGf+HYMfeND1MFoDiPdu2kYMC4cahQQZJnzZwpJfUU\n+xk/HsiaFej7yi1g61ZN8u9OVK8O5MiBevwXihcHPv5YclS6G6oAFOeRKRPQvz+weTMQGYlhw6Qo\nSf/+kkJdsZ2rV0WZdu0KBOzfADx4oArAncicGahdG/7b/8TIkcD27cCvvxot1KOoAlCcS58+EpQ3\nfjxy5RKb9Z49kj9FsZ74eClyl8CUKcDdu8DAgRA7UO7cQIMGhsmnpEKDBkBkJHq8eAc1asgaWEyM\n0UIlRxWA4lzy5wd69ADmzAGioxEaKqUk339f6gkr1vH221Jvp08fYP9+YMIEcfd/qiqLAggOllGn\n4j7Urw/Ex8N/906MHSu1mMeONVqo5KgCUJzPm29KucgffwQR8NVXwJUrwIwZRgvmGVy5Avz4o5Q0\nnTNHzMtnzwKDBgE4fBg4c0bNP+5IvXry+tdfaNQI6NBBwmMuuFEOBFUAivOpVElKsn3/PWAyoU4d\n+W2MG6drAdbw449iOvjtNxlFfvwxEBYGtGoFYP58aaTun+5H/vySItdcIOmrr4DYWKBXL+D2bYNl\nM6MKQHENL78sQ5+DBwHI6DUqClixwmC53JzYWDH3NGsGVK0KFCwIDB8uSsEv5o4cbNtW4i4U96NB\nA3F7M5lQvjzw3XfiElq/PvDPP0YLpwpAcRXPPSevmzcDAF54QZ5Z335roEwewK+/it4cPDiVg1On\nijuQBle4L/XrA9evA0eOAJDQmFWrxGpXq5bx1VNVASiuoXRp2cwKIFMmYMAASWUQGWmwbG4Ksywa\nVq4sFrRkxMYCX38tK+oJtmbF/UjwzPrrr4e7mjUDduwA8uYV/wgj4wNUASiu47nngC1bHt7xffpI\nkqxx4wyWy03ZsgXYu1fMZY+UdpwzR1aChw41RDbFSsqXBwoVAjZsSLb7iSfElBcVlUw3uBxVAIrr\naNgQiI6WHAaQNbKePeVZduKEsaK5I198ITb/rl1THDCZgC+/FHegR6YGiltBBLRvDyxbljyQA2IG\nzZFDAvqMQhWA4jpSrAMAEhyTO7f8RjQuIJEtW4DVq+Xvkz17ioPLlokSHTIklamB4naEhkrUXgqP\nh1y5gI4dxZHLqAAxVQCK6yhXDihWTJ5uZkqWlIXOY8fEHuquWRNdCbNYdooXB954I5UGs2cDRYsC\nL2rdX4+gYUP5f82d+8ihHj2AGzdEpxuBXQqAiB4jonVEdML8mt9Cu3+I6AARRRKRVnjxVYjkx7B5\nc7KVryZNZD1z0SItHwkAy5eLXXj48FRG//fuiRtJ27aykq64P/7+QOfOwMqV8rRPQqNGksDVKDOQ\nvTOAIQA2MHMFABvMny3xPDPXsLZSjeKlPPcccP68rH4lYeBAqSHz4YdSOcxXMZmAYcOAChWA//0v\nlQYbN4otuV07l8um2EGXLuK5tWhRst3+/rLGs2aNMbUy7FUA7QAk6K6ZANrbeT7F20lYB0hiBgJk\ncvDDD2ISevNN340QnjsXOHAA+OQTC6l9liwR43Hjxi6XTbGD2rWBsmWBefMeOdS9u9zvqViInI69\nCqAwMydktrgIoLCFdgxgPRHtJqKwtE5IRGFEFEFEEdHR0XaKp7gdlSoBAQHJFoITyJFDTEGRkRLj\n5GtEREhxl5o1gU6dUmlgMgFLl0ren6xZXS6fYgdEshi8fr14wiXhySclKGzsWNdHB6erAIhoPREd\nTGVLNgdlqS5vKaQhiJlrAGgJ4A0iamipP2aezMyBzBwYEBCQkWtRPIGk6wCp0KmTTBKGDQOuXXOx\nbAYSGSkBQgUKiJXAL7Vf5s6dYidQ849nEhoqQ/3ff3/k0MSJ4gX37LMPvaRdQroKgJmDmblqKtsS\nAJeIqCgAmF8vWzjHOfPrZQCLANR23CUoHsdzzwH//iuZzVJAJIFh164BH33ketGM4OBBoGlTsexs\n3JhGWp8lS2Tht1Url8qnOIinngKqVJHpbYrw31q1ZEz04IGMj/budY1I9pqAlgLoYX7fA8CSlA2I\nKCcR5U54D6AZgIN29qt4MkFB8mohEUr16pIzZeJEWRR25YjI1cTEyIA+c2Z5+Jcpk0bjJUtEeeZP\n1dlOcXeIJP/Jnj2pzoCrVRMHiOzZZTboirgYexXAFwCaEtEJAMHmzyCiYkS00tymMICtRLQPwE4A\nK5h5tZ39Kp7MU09J9Fca7j6ffipBrqNGiY20enWZHlevLmH0a9a4UF4nMmoUcOqUREOXL59Gw+PH\nJaGYmn88m+7dZQ1s9OhUD1eoIOWdf/pJfiLOhtgdKxWbCQwM5IgIDRvwSpo3lzSX+/en2ezCBYmU\nXLpUPufOLT+QOnU8P5X0sWOiC0NDgVmz0mn89dfAO++I6axUKZfIpziJkSOBESPE9lelisNPT0S7\nrXW310hgxRiCguQHkM5Kb9GiEiOwYYNsixdLQY21a6VSlqfCLB4/OXNaHAwmZ+1aKQigD3/Pp18/\nsfN8843RkqgCUAwiKEiegtu2ZfirXboAcXHAggVOkMtFzJ0rNv9Ro4DClpynE4iNlfWS5593iWyK\nkylYUEYxs2dLUKSBqAJQjKF2bfFo2bo1w1+tUQOoWNGYwBlHEB0tBV4CA6W0Y7rs3CnJxFQBeA+D\nB4tL6PjxhoqhCkAxhpw5gWeesUkBEMksYPNm4Nw5J8jmRJjFw+n6dWDaNEkFkC7h4XLRCVHUiudT\nrhzQujXwyy+GVoRRBaAYR1CQjG7v38/wV0ND5XeTUBPdU5g9WwK9PvlE3P6sIjxcpj2PPeZU2RQX\n066dFPVJxxHCmagCUIwjKEge/rt3Z/irFSsCTz/tWWagM2eA/v3lst96y8ov3bsnqUHV/ON9JAT0\nLV9umAiqABTjSKiXaoMZCBAz0K5djyQWdUsuXwZeflnMvjNnWmn6AWSR/P59VQDeSJEiEgJsoD+z\nKgDFOAoVkqguGxVAaKi8fv+9A2VyMMxS8KZKFbF2TZkCPP54Bk4QHi6JgZ591mkyKgbSujWwffsj\nCeJchSoAxViCgsTF0YZSYCVLAr17A99+a7MOcSomk4z6Q0MlE/DevTJryRDh4ZIeNG9ep8ioGExI\niIwSVq0ypHtVAIqxBAUBV68Chw/b9PWxY4HSpSXC3t1qCv/2m6R/HzZMzPiVK2fwBHfvAjt2qPnH\nm3n6aYl2NGgdQBWAYixNmsjrhg02fT13buDnnyVDwqBBDpTLTuLipKRjlSrAxx/bWL3xzz8lPaQq\nAO/Fz09mAWvWyP/a1d27vEdFSUqpUpIBa/16m0/RoAEwZAgwfbo8bF1dVCM1Zs2S/G2ffpqBBd+U\nbNwomiMhe6rinYSEADdvGmLHVAWgGE9wMLBpk10joBEjJIXuRx+Jvf3pp4HVTs45GxeX+v7790UR\n1aplR/LOhBqBzz0nhQIU7yU4GMiSxRAzkCoAxXiCg4Hbt8VNxkayZJFZ9MmTklztxg3g1VctP6Tt\n5YcfZF02lRKvmDxZat2MGiUBvDaxYoXYtV5/3S45FQ8gVy6gUSND3EFVASjG8/zz8qS0wwyUQLly\nwNtvi2fQ2bMSdetovvpKnstEQM+eyfVWgtnn+ecTlzdsYuJEoHhxzf/vKzRrJvnBXZzbxC4FQESd\niOgQEZmIyGL+aSJqQUTHiOgkEQ2xp0/FC8mfXzKjOUABJBASIqaglLm2Vq2SRWNrOXJEvHjGjweW\nLZO1hvfeA156CThxQhw4EiL6lywRs098vHgn2Tz6P35c0j+/9pqNq8eKx2GnM4TNMLPNG4AnAVQE\nsAlAoIU2/gCiADwOIAuAfQAqW3P+mjVrsuIjDB3KnCkT882bDjvlmDHMAPPu3fJ5717mrFll32+/\npf1dk4l50iTmbNmkfdKtTx/muDhpd+AAc+7czMWKybHAQOZ//7VT8EGDmDNnZr5wwc4TKR5DfDxz\nwYLM3bvbfSoAEWzlM9yuGQAzH2HmY+k0qw3gJDOfYuZYAPMA6LxWSU5wsBjst2xx2Cl79ZKko+PG\nSYxA585AgQKSibpnT6lHkwAzcOmSBGutXAm88IKYeRo2lJTtly5JVobwcLHxJ3j2VK0qa7XR0UCf\nPlLl0q6aLXfuSD3Ajh0lVYDiG/j5id1wwwaXZgd1xfyyOIAzST6fBVDHUmMiCgMQBgCltPqR71C/\nPpAtm5iBQkIccsq8eeVBP2WKxJpFRckDvHx5Ca5t3148LZcvl3QShw4lfjdzZmDMGKlG5mceJhUq\nlHo/ISGy6Jw9uwOEnjtXTtavnwNOpngUTZpI9ODx45Lt0AWkOwMgovVEdDCVzSmjeGaezMyBzBwY\nEBDgjC4UdyRbNsl3s349EBEBvPKKPMEjI+067YABUlBr+XJJwdywIVCsmFQTO31aoojfeAPImlUe\n+AsXStTuuXNSs8PPyjmyQx7+gCxQVKmivv++iAHrAOnOAJg52M4+zgEomeRzCfM+RUlOcLCssNaq\nJa5xt2/Lj6FGDZtPWbGimGbu3pUF3ATq15fc/OvXSz6h2rXtWLR1FPfuSWKwN990A2EUl1OunNgP\n16932QzQFSagXQAqEFFZyIM/FMDLLuhX8TReekmc+Vu3lqdy5crAvn12n3bKlNT3d+4sm9uwY4dM\nVxo2NFoSxQiIZBawaJG4ktkcQm499rqBdiCiswDqAVhBRGvM+4sR0UoAYOY4AP0BrAFwBMB8Zj5k\n6ZyKD1O6tIz4Bw8G8uQBqld3iALwGDZvloeApn72XYKDpV7o3r0u6c6uGQAzLwLwSKgNM58H0CrJ\n55UAVtrTl+KDVK8u/vD374uR3tvZskWuOV8+oyVRjKJxY3ndsEFiY5yMRgIr7kv16uIaeuSI0ZI4\nn9hYWX1W849vU6SIOAG4aCFYFYDivlSvLq++YAaKiABiYiT5m+LbNGki3m/x8U7vShWA4r5UqCD+\nlb6gABIC4NT+r3z0keQWccEisCYaUdwXf38JtfUFBbB5s3g9aeyLkj+/y7rSGYDi3iR4ArkwPN7l\nxMVJMRA1/yguRhWA4t5Urw5cuSIJebyVyEgJetMFYMXFqAJQ3BtfWAjevFleVQEoLkYVgOLeVKsm\nr96sANavlwx1xYoZLYniY6gCUNybvHmBMmXsTgrntixdKsWLX3nFaEkUH0QVgOL+eGtKiMuXpXBx\njRrA++8bLY3ig6gCUNyfGjWk/uLdu0ZL4jiYgbAwyfvy889S1V5RXIzGASjuT/XqgMkkaT3Ll5e8\nQEFBUkPAU5k+XYoIf/ONxDooigHoDEBxf2rVksosgwZJquimTYFRo4yWyjZOnxZ7f58+QKNGck2K\nYhCqABT3p0QJ4NQpYPduYOdOSZk7ebIkUPMkvv5aKtQsXAgMGwYsW2Z9yTFFcQJ69ymeQenSwDPP\nyGxg8GCp0r54sdFSWc/Fi8C778qo/9gx4NNPpeqZohiIvQVhOhHRISIyEZHF5NVE9A8RHSCiSCKK\nsKdPRUHz5uIaOmmS0ZJYz5IlsvA7erSU/VMUN8DeGcBBAC8A2GJF2+eZuQYzO7/KgeLd+PsDr70G\nbNrkObUCFi2SBewqVYyWRFEeYpcCYOYjzHzMUcIoitX06gVkzuwZs4Dr16XAR4cOWuxdcStctQbA\nANYT0W4iCnNRn4o3U6gQ0KkTMHMmcOeO0dKkzYoVkvGzQwejJVGUZKSrAIhoPREdTGVrl4F+gpi5\nBoCWAN4gIotZr4gojIgiiCgiOjo6A10oPsfrrwM3bwJz5xotSdosWgQULQrUqWO0JIqSjHQVADMH\nM3PVVLYl1nbCzOfMr5chReRrp9F2MjMHMnNggBbHUNKiQQPgySeBWbOMlsQyMTHAqlVA+/bq8qm4\nHU6/I4koJxHlTngPoBlk8VhR7IMI6NoV+OMP4J9/jJYmddaulRQWav5R3BB73UA7ENFZAPUArCCi\nNeb9xYhopblZYQBbiWgfgJ0AVjDzanv6VZSHvPyyvM6ZY6wclli0CMiXT/z/FcXNIHbjUnuBgYEc\nEaFhA0o6NGwIREcDhw+7l5fNvXtA8eJASIh7m6kUr4KIdlvrbq9GScXzeeUV4OhRYO9eoyVJzsyZ\nwNWrQI8eRkuiKKmiCkDxfDp1kpiA2bONliSRuDjgyy+B2rWBxo2NlkZRUkUVgOL5PPaYmFnmzpUH\nrzvw66/A339LoRd3MkspShJUASjeQdeuknBt40ajJZHaBV98AVSuDLRpY7Q0imIRVQCKdxASIjOB\nsDBg+3ZjZVm+HDh4EBg6VH3/FbdG707FO8iWDVi5UswtQUEyAjeZXC9HTIykei5TBggNdX3/ipIB\nVAEo3kOdOuIJ1LGjjL4HD3Zd33fvAmPGAGXLArt2AcOHA5m04qri3ugdqngX+fIB8+bJjGDqVGDk\nSCBvXuf2eeoU8OyzwPnz4vEzf77EJiiKm6MzAMX7IAIGDJBRubMjhOPjgW7dJCPp5s2S9lkf/oqH\noApA8U5q1gRq1JDawY6Mdj99Ovn5vvoK+OsvYMIEffArHocqAMU7IQJefRWIjJRi8o5g/nypTdyy\nJXDokJx7xAjgxRclGllRPAxVAIr38sorQPbswJQpifuYxWxjC5MmAQEBwI4dQPXqUpu4YEHghx80\n2EvxSFQBKN5L3rzASy/JOsDt22Kfr1xZFmwzahY6cUJqEA8aJO/79hW7//TpQIECThFfUZyNKgDF\nu3n1VXn4N2gABAcDly4B27YBy5Zl7DzTpkkx+p49ZdQ/YQJw6xbQooVTxFYUV6AKQPFu6tUDqlUD\njh8Xl9DTp4Fy5eS9tbOABw+AGTMk2rhYscT9avZRPBx7C8KMJqKjRLSfiBYRUT4L7VoQ0TEiOklE\nQ+zpU1EyBBGwbp346n/4IZArFzBsmCwMr1yZ+nf++w84cybx84oVMnPo08c1MiuKi7B3BrAOQFVm\nrgbgOIChKRsQkT+AiZCC8JUBdCGiynb2qyjWU6iQFGVPoGtXSdWQMAswmYCffwaaNgWKFJGF3tKl\nJa/Qf/9JQFmxYuL9oyhehF2RwMy8NsnH7QBeTKVZbQAnmfkUABDRPADtABy2p29FsZnMmSVNc1gY\nMHYssGCB+PI/+STQqhVQtSpw9izw3Xdy7Pp1SS2hqR0UL8ORd3QvAL+msr84gCTzaZwFUMeB/SpK\nxunRA/jkE+Ctt2TEP3267EuavbN3b6B/f3H77NXLOFkVxUmkqwCIaD2AIqkcGsbMS8xthgGIA/CL\nvQIRURiAMAAoVaqUvadTlNTJkkUe+ps3A//3f0D+/I+2qVJF6gvcuSNrB4riZaSrAJg5OK3jRNQT\nQGsATTj1CvPnAJRM8rmEeZ+l/iYDmAxIUfj05FMUmwkOli0tiPThr3gt9noBtQDwLoC2zHzXQrNd\nACoQUVkiygIgFMBSe/pVFEVR7MdeL6AJAHIDWEdEkUT0AwAQUTEiWgkAzBwHoD+ANQCOAJjPzIfs\n7FdRFEWxE3u9gMpb2H8eQKskn1cCsOB0rSiKohiBRgIriqL4KKoAFEVRfBRVAIqiKD6KKgBFURQf\nRRWAoiiKj0Kpx265B0QUDeBfG79eEMB/DhTHE/DFawZ887p98ZoB37zujF5zaWYOsKahWysAeyCi\nCGYONFoOV+KL1wz45nX74jUDvnndzrxmNQEpiqL4KKoAFEVRfBRvVgCTjRbAAHzxmgHfvG5fvGbA\nN6/badfstWsAiqIoStp48wxAURRFSQOvUwC+UoCeiEoSUTgRHSaiQ0Q00Lz/MSJaR0QnzK+pVDrx\nbIjIn4j2EtFy82dfuOZ8RPQ7ER0loiNEVM/br5uIBpvv7YNENJeIsnnjNRPRdCK6TEQHk+yzeJ1E\nNNT8fDtGRM3t6durFICPFaCPA/AWM1cGUBfAG+ZrHQJgAzNXALDB/NnbGAhJLZ6AL1zzOACrmbkS\ngOqQ6/fa6yai4gDeBBDIzFUB+ENqiXjjNc8A0CLFvlSv0/wbDwVQxfyd783PPZvwKgWAJAXomTkW\nQEIBeq+DmS8w8x7z+1uQB0JxyPXONDebCaC9MRI6ByIqASAEwNQku739mvMCaAhgGgAwcywzX4eX\nXzckXX12IsoEIAeA8/DCa2bmLQCuptht6TrbAZjHzPeZ+W8AJyHPPZvwNgWQWgH64gbJ4jKIqAyA\npwHsAFCYmS+YD10EUNggsZzFt5AqdKYk+7z9mssCiAbwk9n0NZWIcsKLr5uZzwH4GsBpABcA3GDm\ntfDia06Bpet06DPO2xSAz0FEuQAsADCImW8mPWau0ew1bl5E1BrAZWbebamNt12zmUwAngEwiZmf\nBnAHKUwf3nbdZpt3O4jyKwYgJxF1TdrG267ZEs68Tm9TABkqQO/pEFFmyMP/F2ZeaN59iYiKmo8X\nBXDZKPmcQAMAbYnoH4h5rzERzYZ3XzMgo7yzzLzD/Pl3iELw5usOBvA3M0cz8wMACwHUh3dfc1Is\nXadDn3HepgB8pgA9ERHEJnyEmcckObQUQA/z+x4AlrhaNmfBzEOZuQQzl4H8bzcyc1d48TUDADNf\nBHCGiCqadzUBcBjefd2nAdQlohzme70JZJ3Lm685KZaucymAUCLKSkRlAVQAsNPmXpjZqzZILeLj\nAKIADDNaHideZxBkWrgfQKR5awWgAMRr4ASA9QAeM1pWJ11/IwDLze+9/poB1AAQYf5/LwaQ39uv\nG8DHAI4COAjgZwBZvfGaAcyFrHM8gMz2eqd1nQCGmZ9vxwC0tKdvjQRWFEXxUbzNBKQoiqJYiSoA\nRVEUH0UVgKIoio+iCkBRFMVHUQWgKIrio6gCUBRF8VFUASiKovgoqgAURVF8lP8Ho4GBopsH1AgA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8b8ae4c080>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Actual problem from my research:\n",
    "\n",
    "# Suppose you have 2 sensors, each of which should take measurements\n",
    "# at even intervals over the day. We want to make a method which can let us\n",
    "# recover from device failure: if a sensor goes down for an extended period,\n",
    "# can we impute the missing values from the other?\n",
    "\n",
    "# Take for example two strongly correlated measured signals:\n",
    "\n",
    "np.random.seed(1234)\n",
    "s1 = np.sin(np.linspace(0, 10, 100)) + np.random.randn(100) * 0.05\n",
    "s2 = 2 * np.sin(np.linspace(0, 10, 100)) + np.random.randn(100) * 0.05\n",
    "plt.plot(s1, color='blue')\n",
    "plt.plot(s2, color='red')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAd8AAAEICAYAAAAeO/7PAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl4VNX5wPHvmawkJAESQiALYUtYZQmiAYEoaFHR2rr8\n1LaorUvFpS61ioq1bliXarWiIqjg3rpXLa2gQZCIssoaZAkJa0ICSYBAljm/P84MmSSTffa8n+eZ\nJ8ncO/eeO0nmveec95yjtNYIIYQQwnMs3i6AEEII0dFI8BVCCCE8TIKvEEII4WESfIUQQggPk+Ar\nhBBCeJgEXyGEEMLDJPj6EaXU60qpR2zfj1dK5bbxOC8ppWa6tnSgjNeUUoeUUt+7+vhCCBEoJPj6\nKa31Uq11enP7KaWuVkotq/fa32utH3ZDsc4AzgaStNZj2noQpVSWUmq364oFSqnzlVLLlFKHlVL7\nlVJzlVJRjewbr5R6Rym1VylVqpT6Vil1Wr19rlRK7VJKHVVKfayU6ubkON2UUkWO779SKs52vGLb\nsXOUUuMctg9VSv1XKXVQKdVgEL7tmB/ZzrtLKXVlve2TlFJblFLHlFJfK6V6O2wLs914HVBKlSil\n/q2USnTFNduO/apSqsz2/t7RyHs7TSmllVLXOjx3uVIq1/baQqXUfKVUdEuv2WG/B2zHnuxsuxC+\nRIKvlyilgr1dBjfoDeRprY96sxCNvLcxwCNAL2AQkAg82cghOgM/ABlAN2A+8LlSqrPt+EOAl4Hf\nAD2AY8BsJ8f5K7C53nNHgGttr+ti2+ffDmWuAv4J/K6Rsr0AVNpe/yvgRVt5UErFAR8CM23lXgm8\n5/DaPwCZwCm29+EQ8LyLrvlBYADmb+BM4E9KqSmOBVdKdQXuBTbWu6blwEStdTTQFwjG/K6avWaH\nY/cDLgX2OXvThPA5Wmt5uOgB5AEzgE2YD7bXgHDbtixgN3A3sB94w/b8VGAtcBjzIXSKw/FGAquB\ncsyH6LvAI47Hc9g3GfPBWwQUA//ABJnjQA3mQ/+wbd/X7cex/XwdsA0oAT4Fejls08DvgZ9sZXwB\nUE6u/Xf1zvUXoCvwma1Mh2zfJzm8ppvtPdpr2/4xEAlUAFbbcY5gAkUY8Kxt372278Oaem+b+V39\nEljfit9tGZBh+/4x4G2Hbf0wwSHK4bmxQA5wDbCskWNagAts73F8vW39zb9nnecibedJc3huAfC4\n7fvrgeX19q8ABtp+fhF4wmH7+UCuK67Z9js5x2H7Q8C79Y73EjAdyAaubeScnW3X9EVLrtnhuYXA\neZj/wcne/iyQhzyae0jN1/V+BfwM8+GUBtzvsC0BE3B6A9crpUYCrwI3ALGYmsWntia8UEwwesP2\nmn8BFzs7oVIqCBPYdgGpmFrdu1rrzZjAmaO17qy17uLktWcBs4DLgJ62Y7xbb7epwKmYGtNltuur\nQ2s9r965/owJLq/ZrjcFEwj+4fCyN4AIYAgQDzyjTa35XGCv7TidtdZ7gfuA04ERwHBgTFPvrbP3\nqZ4JNKyBOaWUGgGEYm5QsJV3ncO1bwdOYH7f9t/HP4CbMYHV2TF/xNysfArM1VoXtqAoaUC11nqr\nw3PrbOVxVq6jtjLbt88DximleimlIjB/q/9p7zXbarQ9HbfXKxdKqTHAaEwAdna+M5RSpZgbzYsx\nN1ctuWaUUpcCJ7TWXzg7thC+SIKv6/1Da12gtS4BHgWucNhmBf6stT6hta7ABImXtdYrtNY1Wuv5\nmA+0022PEOBZrXWV1vp9TLOgM2MwtcO7tNZHtdbHtdbLGtm3vl8Br2qtV2utT2Bq7plKqVSHfR7X\nWh/WWucDX2MCYLO01sVa6w+01se01uWY92MigFKqJybI/l5rfch2jUuaKedDWutCrXURpmb9G4ft\n9d/bRimlzgauAh5o7hpsfY9vAH/RWpfanu4MlNbbtQyw9yHfCqzQWq9q7Lha61OAaOBKoKW/q862\n8zR23ubK9RNQAOyxPT8IU0Otow3X3Nn2c6mTbfabkdnAzVprq7ML01ov01rHAEmY7oC8llyzrd/+\nMUyTuhB+Q4Kv6xU4fL8LExTtirTWxx1+7g3caUsCOqyUOoxpPu5le+zRWjvWnHY1cs5kYJfWuroN\n5e3leFyt9RFMs3Wiwz77Hb4/Ru2HbZOUUhFKqZdtSTJlwDdAF9uHcTJQorU+1JZy0vx721iZTgfe\nBi6pV5tytm8n4N/Ad1rrWQ6bjmACp6MYoFwp1QsTfO9rriy2m6R3gHuUUsOb27+p87Zw+wtAOKaV\nJRLTTVGn5tuWa7Zto952x/NOB37UWn/XxLUBoLXeg2lCtre+NHdND2K6GfKaO7YQvkSCr+slO3yf\ngukLs6vfBFkAPKq17uLwiLB9IO8DEpVSqt7xnCkAUhpJNGpu2aq9mJsAAJRSkZgP5z3NvK4l7gTS\ngdO0SaaZYD8NpszdlFINmsJxXuY65aT597YBWzP/p8BvtdaLm9k3DNPsvxvTLeBoI6bp275vP0wT\n7VZMK0RPYJNSaj/wd2CMLQM4qJHThWASjZqzFQhWSg1weG44tc3n9csVien+sG8fAbymtS6xtXI8\nbytbXHuu2XYDtc9xe71yTQJ+YXsP9mP6w59WSjl2QTgKtpW7Jdc8CbjV4djJwD+VUnc3cmwhfIO3\nO50D6YFpKluPaTrrhmlOfMy2LQuHBCnbc6MxQeg0TECKxCTBRGE+2PIxzWkhmAShKpwkXAFBmH6w\np2zHCAfG2bZNsZUr1OG8rzscZzImIWoEJqnp7zgkCGGCWn9nr3Vy/VfXe+0TmJpVuO39+Mh2vGDb\n9s8xtdCutmucYHt+IKZ/OMbhWI9gEtK6A3G299Zp8lkjZRsKHAD+rwW/xxBM7e9je1nrbR+Cafoc\nb3u/38aWXGR7DxMcHn8AVgAJtu2nY4ZkhQKdMEli5diS3Gx/B+HAYNt7FY4tscy2/V3gHdt5z8A0\n9Q6xbetu+/li2+uewNRg7a99DfgAU3MMwWQe72nvNdu2Pw4ssf0uB2FaS6bYtnWp954sB+6w/34x\nXQoptu97247zYQuvObbesQswWc+dvf15IA95NPXwegEC6UHdbOfDmOEaEbZtTgMEJjj+YNt/Hyax\nyp5BOhpYQ22283s0nu2cYvvgLAYOAs/Zng/FBLkS4KDtudepm+38e2C7bZ/6GcntCb69MJmtRzA1\nmBuoG3ztQ1oOYLKdHT9wX7Vdy2HbccKB52zv0T7b9+FNvbf1yvYadTOojwAbHba/BLxk+36irZzH\n6u0/3mH/KzE3R0eBT4BuLXxPJmJulMpt7/cSbDcdtu2ptnM7PvIctnez/Z6P2s5/Zb3zTQa2YG5e\nsoFUh22xwFtAoe19XQaMccU1Y246XsUE6APAHU38LrJxyHbG5ALsth13NzAHiG3pNTv5H5RsZ3n4\n/ENp3WxrnWghpVQe5kNlkbfLIoQQwndJn68QQgjhYRJ8hRBCCA+TZmchhBDCw6TmK4QQQniYVyb3\nj4uL06mpqd44tRBC+K1Vq1Yd1Fp393Y5RPt5JfimpqaycuVKb5xaCCH8llKqsVnuhJ+RZmchhBDC\nwyT4CiGEEB4mwVcIIYTwMAm+QgghhIdJ8BVCCCE8TIKvEEII4WESfIUQQggPk+ArhGiX7SXb2XFo\nh7eLIYRfkeArhGiXN398kxd/eJFqa7W3iyKE35DgK4RosxprDQeOHqDsRBkrdq/wdnGE8BsSfIUQ\nbVZ4tJAaaw0WZeF/2/9HY6uklVSUsPPQTg+XTgjfJcFXCNFme8r3ADC572T2H9nP+sL1Tvebv3Y+\nT3z7BFsObvFk8YTwWRJ8hRBttq98H0oppqZNJTYilv9u+2+DfcpPlJNbnItGM2fVHA4eO+iFkgrh\nWyT4CiHabE/5HuIj4wkLDmNSn0lsK9nWIPN5zf41aK25IeMGtNa88P0LHK8+DoBVW7FqqzeKLoRX\neWVJQSFEYNhbvpdeUb0AOCPlDD7b+hkLty1k+qnTT+6zau8qenTuwYiEEYQHh/Pciud4eMnDWLWV\nw8cPExcRx8NnPeytSxDCK6TmK4Rok6qaKoqOFp0MvmHBYUzuO5l1+9edrP3am5wzemaglGJQ90Fc\nPeJq4iLiSItNIz0uncKjhRyrOubNSxHC4yT4CiHa5MDRA1i19WTwBZN4FR0WzYebP0RrfbLJOaNX\nxsl9Tks6jdszb+eakdeQlZoFQNHRojrH3lq8lRe+f4Eaa41HrkUIT5PgK4Rok73lewHqBN+w4DAu\nSL+An4p/Yn3helbvW018ZDyJUYlOjxEXEQfQIAlr7f61/HjgR7Yf2u6m0gvhXRJ8hRBtsrd8LxZl\nIT4yvvbJHBj31jh6HO7Bvzb+i9yDuWT0Mk3OzjQWfAuPFgKw/oDzoUtC+DsJvkKINtlbvpeEzgkE\nW2x5mznAJAh6IIiLHriIwp2FWLWVjJ4ZdV+YA8wyX8ODw4kKi6LoWN1m5wNHDgCwoXCD269DCG+Q\nbGdvyMmB7GzIyoLMTG+XRog22VO2h9QuqbVPZAOVQA2MLBhJ3319Odb3GEnRSbX72AI0lUAosNjU\nfh1rvjXWGg4eO0hkaCR7y/dSfKyY2IhYD1yREJ7T7pqvUipZKfW1UmqTUmqjUuoPrihYwMrJgUmT\nYOZM8zUnx9slEqLVKmsqKa4opmdUz9onszABNQhUqOKWrFu4c+yddZucszkZoKk0P8dFxNVJuCqu\nKMaqrYxPGQ/AxqKNbr4aITzPFc3O1cCdWuvBwOnATUqpwS44bmDJyYFZs2DBAqishJoa8zU729sl\nE6JFvs3/lo2FJhDuK9+H1rpuIlUmsBh42HyNCIog+m/RprZrl8XJAE2o+bl7RHdKKkpOTrZhb3I+\npccpxEXESb+vCEjtbnbWWu8D9tm+L1dKbQYSgU3tPbZfcmxSBvN9bCzcdpsJtkFBEGx720NDa/cT\nwof9eOBHFqxbgEVZuHbUtVTWVAJ1M50BE4Azcdq8fHLbYkwNOMv8HJcfh1VbKakoIS4ijgNHTfBN\n6JzAsB7DWJa/jKqaKkKCQtx/oUJ4iEv7fJVSqcBIoMHaYkqp64HrAVJSUlx5Wt9hb1K2B1mloLra\nfLVazQPguusgJaVhn6/0BQsfdPj4YV5f+zrJMcmEBYUxd/VcUmJSCLYE0z2yu/MXZdOgeRn7n3Sm\nw/dw8hgHjx0kLiKOwqOFRIREEBkaydD4oXy982u2Fm9lSPwQ91ygEF7gsuCrlOoMfADcprUuq79d\naz0HmAMwevRo5+uO+bvs7NomZXug1RosltpgHBoK06Y1DK6OgTs0FBYvlgAsvE5rzWtrXqOqporr\nRl1HTHgMz694nm0l20iKTsKiGum5ysLUeO0131hMhnMWdQIv1A43KjpaxMC4gRw4coAenXsAkB6b\nTkhQCBsKN0jwFQHFJcFXKRWCCbxvaa0/dMUx/VJWlgmc9Wu+oaHw7LNQXNx4rdYxcFdWmr5hqQUL\nL/vf9v+x5eAWpg2fdjIg3nLaLbyy6hX6dO1jdsqhTjMyULd5ORa4jYZN0DZdwrsQZAk6mfF84OgB\n0mLTAAgJCiE9Np31heu5TF/W6HhhIfxNu4OvMv8N84DNWuu/tb9Ifiwz09RY6/f5NhVA7U3NsbF1\nA/drr9UGbqkFCy84WnmUf2/9NyN7jmRs8tiTz4cHh3PLabeYHxrr24Xa5uVZNN4EDViUhdhOsRw8\ndpCqmioOVRyiR2SPk9uHJwznrR/fIrc4l4FxA91zsUJ4mCtqvuOA3wDrlVJrbc/dq7X+wgXH9j+Z\nmXUDZVNBs35Ts712nJ8Pr7xSNyNagq/wMHui0wVpFzSscdpru/k0GViBhk3QWQ3P1T2yO0XHik7O\nbGWvZQNkJmXyn5/+w/ub3ue+8fdJ7VcEBFdkOy8DOvZ/Q1sTpeo3NRcXw4wZ5njz59cGZcmIFh5m\n1VaW7FpCWmwaidH15mV2rO0GUfsp0khgdZbhXOdY2RCXHkdeZN7JTOf4LfEw3+wfkhnCRQMv4tU1\nr7JizwpOTzrdFZcohFfJDFft1Z5EKcc+YscgW7/5Wmq9wsPWH1hP8bFiLhl8ScON2dTWdgGuA1Jw\nmkx1UqaTbQ5BPC4tjqPXHCVvZR6EQvxD8VDByabsMaePYfHOxXy85WMyembIsCPh92Ru5/aqX3tt\nzaQZ9iD78MMNg3ZmpqkF25+zT9IhM2IJD/g672u6durKiIQRDTdmUXeijGnADBoPvI3J5mQQ717a\nHZbBpi83EbMohvBj4XWaspVSXDL4Eg5VHGLRjkVtvSwhfIbUfNursdprS9XvI3ZGhiEJD9pXvo/N\nRZu5aOBFzocSNdWM3BpZnOwLjjseBxoKogoYcGiACeyKOk3ZabFpDE8YzsJtC5mYOpGIkIg2nlgI\n75Pg216eaCJ2VruW4CvcJDsvm2BLMGeknNH4Ts6akVvLIYjHdYuDhUAN9DjRA/4BFNMguJ834DzW\n7V/Huv3ryEyW/wHhvyT4tkZjiVUtqb22R3tr10K0kFVb+W73d2T0yiAqLMr9J7QF8U50IjIskqMF\nR+kxogdc4LCPwzji3qf3plunbqzet1qCr/BrEnxbqrFhQZ5IiJIELOEhu8t2c7z6OMPihzXc6Gwy\nDRfq3q87R2OPEn9qfN1zOowjVosVo3qOIjsvm+PVxwkPDnd9QYTwAAm+LeXY9HviBNx8s5lC0lN9\nsO6uXQsBbCvZBkD/bv3rbmhqMg0XiYuII+9wXp0JNhrMEb0ARiWPYlGPRfx44EfGJI5xbSGE8BDJ\ndm4pe9NvUJCZq7mmxnvLAkrms3CTbSXbiI2IpWunrnU3ZNNwMg0XS+icQEhQyMm5noG6mdVBwGvQ\n94G+xHwQw+pvV7u+EEJ4iNR8W8qx6ddxiUBP98FK5rNwE60120q2OZ/CMYtmZ6lqr7P7nc3IniPr\njuF1zKzOB14BVaMYuWck3+Z+y4nqE4QFh7m+MEK4mQTf1nBs+h02zDt9sJL5LNykuKKY0uOldZuc\nHft5XTG8qAnhweEkRSc13OC4RvB8oBIyDmaQ3SubjUUbGdVzlOsLI4SbSfBtK2/1wUrms3CTBv29\nzvp5Z3inbECdWnD/if2JOhLFqr2rJPgKvyTB199I5rNwk20l24gIiaBn557miWyaXzTB02y1YAsW\nRvw4gu/3fE9BaQFJ0Umy4ILwKxJ8/ZFjrbutizoIUc+2km3069avNohl4fZ+3vY4I+UMvtv9HY98\n8wiJ0YmMTxlPVmqWBGHhFyT4+jNJvhIucqTyCPvK95kVgzzYz9seqV1S+evkv7Jy70qWFyzn3Q3v\n0imkk6x6JPyCDDVyxnEojy8P62nPog5CONhesh2A/rv6m37emZiv0LZFEzwkMjSSiakTueeMe+gZ\n1ZMvt3+J1trbxRKiWVLzrc+xNhkUBEpBdbVv1izrJ1/FxpobBWmCFq20rWQbwZZgeq/o7Xv9vC2g\nlOKcfucwf+18thzcwqDug7xdJCGaJMG3PsfapNVqntPaN4f1NDX22NduFITPKakooexEGVZtZfPB\nzfTu0puQM0N8up+3KWMSx/DR5o/4cseXEnyFz5PgW59jbbJ+zdcXh/XYk69mzZLxv6LFjlUd44Gv\nH6Cqpurkc+cNOA8G4tP9vE0JtgRzVp+z+HjLx+wu2+18zLAQPkKCb331h/KAf2QTy/hf0QqbijZR\nVVPFFcOuoHtEdyzrLPT7sB+ciWuWC/QEJws9TOg9gf9s+w+Ldizi6hFXe6tkQjRLgq8z9SfQ8OWg\nayfjf0UrrD+wnsjQSCb0noDlOwv8HLcumuByjSz0EBkaybjkcSzZtYSLBl5El/AuXi2mEI2RbOdA\nkpkJM2ZI4BVN0lqzsWgjQ7oPwaIsHlk0weWyabTME3pPoMZaw/oD671RMiFaRIKvEB1Mfmk+5SfK\nGRo/1DyRRe3KQf6SZJVFo2VO6JxATHgMucW53iiZEC0izc5CdDAbCjeglGJw98HmCceVg7Lw/SZn\naLLMSinSY9PZcnALWmuZ8Ur4JAm+QnQwGwo30DumN1Gro+oGL38Iuo6aKHN6XDrf7/me/Uf20zOq\np0eLJURLSLOzEB3Ikcoj7Dy8k2GHhtWdycoHJ3BrlRxgFievIz02HUCanoXPkuBr58vTSArhIpuK\nNqG1ZuiGof6XZNUYe+azw41EXEQcXTt1JfegBF/hm6TZGWSBAtFhbCjcQFRYFL2zevvtTFYNZNPg\nRkJlmn7fDYUbpN9X+CSp+UJgLlAgNXlRj1Vb2Vi4kcHdB6PGKpOw9DD+Ma63KVk4zXxOj0vnSOUR\n9pbv9VbJhGiU1Hwh8GaHkpq8cGL/kf0cqTzCoDjbvMf+mGTlTCOZz479vonRiV4pmhCNkeALgTc7\nVP2a/IIFgXNtos3yS/MB6N2lt5dL4gZObiRiI2KJi4gj92AuZ/U5yyvFEqIxEnzt6k8p6c/qLw7x\n2mu+uyyi8Jj80nxCikJIeD6hdg7nQGWb9zk9LZ01VWuk31f4HOnzDUT2mvzDD8Nvf2sCbyD1Z4s2\nyf8xn+Q3krE8YAmM4UWNcch+Tv9TOsf2HGPHoR3eLpUQdUjwDVT2eZ6nTTM13qCgwOjPFm2itSZ/\nWz4pJSmBMbyoKdmczH4euncoUXujmLt6LocqDnm5YELU6tjBtyNkBDvWgqXJucMqPFrIiV4n6H2s\nt3/N4dwWWZzMfo5UkfzhjD9wrOoYf1/xd45WHvVy4YQwOm6fb0fKCA6k/mzRJvml+RAPKXNS4Dv8\nZw7ntqiX/ZycmcxNxTfx9+/+zj++/we3Z95OaFCoV4soRMcLvjk5pt8zP7/h2F4JUCJA5ZfmE2wJ\npufEnibZyoXWrDFfR4507XHbpV72c1psGr8d+VvmrJrDD3t+YFzKOK8VTQhwUbOzUupVpVShUmqD\nK47nNvba7syZ8OqrEBzsU32hVVWwaZO3SyH83fHq4zyT8wy7y3affG5X6S6SopMIsgS59Fw1NTBq\nlHl88YVLD+1yo3qOIjw4/OSQKyG8yVV9vq8DU1x0LPdxHP9aUwPXXOPxvtCKCvj8c9PVvHAhaA0n\nTsA//gH9+8OQIZKQLNpnT9kethzcwmdbPwNsyVal+aTEpLj8XErBP/9pvr/oIvj3v5ve/8ABk3zv\nUbZFF9R3iuSYZAm+wie4pNlZa/2NUirVFcdyq/ozWU2b5rGgu2gRzJ4N//0vHDtmnhswALZuNQH4\nscegb18oK4M5c3yiIi78VOmJUgDW7l9L4dFCLMpCRVWFW4KvxQKXXgolJfCzn8HFF8NHH8H559fu\n8+KLsHOn+Ru/8krYuxduuMEE4bIy6NcPrrrK5UUz7MOObHNYJ7+VzLKwZVi1FYvq2Pmmwrs89ten\nlLpeKbVSKbWyqKjIU6etyw2Zv/n58Otfw9lnm2ZjR+XlplYLsHKlafWeNs3UeIuL4dNPzbbwcFi9\nGpYtM8f68EPzYeYWHSHDu4MrPV568vtFOxax6/AuAJcH38OH4a67YNcu6NoVvvzS9Or06VO7z7p1\ncPvtsGGDCdR/+ANYrea5u+4y/4o//GBuQB0VFJib0PrPt1o2dRZdSN6UTGVNJYVHC9t5YCHaSWvt\nkgeQCmxoyb4ZGRk6UHzwgdYhIVqD1n//e91tv/yl1m+8Yb4/dkzrqqrmj/fTT1ovWqR1TY3ry6qX\nL9e6Uyetg4LM1+XL3XAS4W0fbf5I//7fv9evr3ld3/T5TXr++/P1jQ/dqKuWteAPsBVeftn83X//\nfcNtVqvW8+drPXCg1gkJWh84ULutulrrPXu0Litr/O/8hhvMsefPb2chl2utO2mtg8zXgq8K9PWf\nXq+/3+2k0H4AWKld9JktD+8+pN2lDT75BF56yXz/i1+YO//Jk+HBB2trrB9/bGqwu205L506mfyu\n5vTvb2oPFnf8ZgJx9SbRQOnxUqLDojmn3zlU7ati+RvL6bW4F8FnB7t0Vqv582HwYBg9uuG25ctN\nU3JuLrz5JsTH124LCoJevSAqyvyd19SY7hc7rWuTt+64A9rVUGYfdmRbvSlhYgLBlmAKygracVAh\n2k+Cbyvs3Qs//7lJLHntNdN8phT07Al/+xuUlpoW3dJSuOkmOOUUuPPO1p+nqMh86Kxd6+ILsPd5\n+1CGt3C9shNlxITH0DOqJ8P2DUNbNSmHU9o1q9Xhw3DfffD66+bnrVtrA6yzKZPHjTM3oG++aW4m\nm3LLLXDaabWJWErBjz+aAFxWBvfe27Yyn5QJzDBfgy3B9IzqKUlXwutcNdToHcw9dbpSardS6neu\nOK6vufFG06/15JOmf9axdjpsGLz1Ftx9t5nVcf9+mDsXQkJaf56QEJOkMmeO+UDaudNFFyCzXXUI\npSdKiQmLAeCc08+BIOhT2qdNs1ppDe+9BwMHmoSpINtIJXvLz69/3fhrf/5zk2DVnLPOMsF9+fLa\n57p0gXPPNf9Tf/lL68rcnJSYFApKC+zdZUJ4hauyna9wxXF82bFj8L//wfXXwx//6Hyfyy83CViv\nvGISS049tW3n6tIFLrkE3ngDPvsMIiJg82bzQVhSAnFxbb8Ome0q8JUeL6V3jFk2MG1yGnepu0hd\nkdrqlYxOnIArrjDZy6NHm5roqFFmW3S0aQHq1av95T37bNMl8/nncMYZ5m//mmvgggtMJjWY5MWS\nEujtgtUQk6OT+Tb/Ww4fP0zXTl3bf0Ah2kCanVtoxw7Tb+U4hMKZlBRYsQIeeqh957vpJjh+HNLT\nTVO21ua58eOhUBI1RSOs2kp5ZTkx4TEnn+s/qT/B9wa3ejrJd981gfeJJ+C772oDL5hs5bffdk2Z\nY2LM3/Xnn5vM548+gkP11kC4+WZzA7BkSfvPZ8/6ln5f4U0dI/i6YHjN0KGQl2cSq5ozahR07tzm\nUwFw+ummtv3llyapy2IxTXi7dpnxlP/9b8MPKCHKT5SjtT7Z7Nwe06aZoHvXXbXNzXYxMSaJ0FXO\nPx82bjS5E8HBptbr6L77IDbW/P/Zm7zbKik6CaWU9PsKrwr84Os4peSkSW0KwFrXJle5JQu5EfX7\ni8ePNxngY3InAAAgAElEQVTUubkwZQp062aawttMxvwGnLITZQBE50bDLNqU3fzoo2ZcrlImEcoT\nLr/cTERj/3ftWq81OC3NtCidfbbJvfjXv9p4ohwIezKM+EPxFJRKzVd4T+AHXxcMr1m/3mQ0f/ON\ny0vXalOmmGSuxYvNh+SIEeb5FStM0kqLueCmRPie0hOlUAgxV8fATMzsTq341a5aBfffb4YReVJi\nosllKCgws2Q5ExNjhvmNGQP33NOGaSrts13NhOTZyRRsluArvCfwg68Lhtd8/rnpZx0wwOWla5Po\naJMheu+9ph+6osJklp5yiqkZtyiJU8b8BqTS46WwF2KOxJyc1ak1w4seeMC0qNx/v5sK2IQ9e0wC\n10UXNb5PSIjJvs7Obtm4+TqyOTnbVUpxCsW7imV9X+E1gR982zC8pqrKxCS7zz6DjAxT+/VFnTqZ\nqSo7dza1howMM8F9k0FYxvwGpNITpdALonU0BNGq4UXLl5uM5j/9ydQyPW3cOPN33L170/ulpkJy\nsukKWrGiFSfIwrwfQZBckQy9JOlKeE/gB18wAXfGjBYF3poaU6vs1880Mx88aJJOmsty9rYxY8zE\nBPPnm0k+LrywmX4xGfMbkEqPlxKRGEHIlyEnZ3VqaZbzzJmmJeXmm91ZwsbFxJgbx5Z6+mmTmJiS\nYsYE33+/GZLUKIfZrlLeToF42HnIVYPohWgdl4zzDSSzZ5sJNOLjTWJTfr65w5461dsla15wsMlQ\nveIK2LLFTPzRJBnzG3Dss1vVX0y+Ofabzssvh8hItxXPpaZPN1/XrjWZ0rNmmeboJpukbe9LZzrT\n681e/PTeT5w76dxWD8MSor0k+DooKDD9qFOmwAcfmObctWvNUIvW3JF7W0hIbeDV2vn0fw3k5JhP\nrawsCch+rPSEmde5OVarSdizz6n85JNmOI8/iYw0/5t2779vkhFb1BecA2nPpJGTkEPNrBqCFgVJ\nABYeJcHXwZtvmg+l2bPNrFIAI0eahz+65RY4csTMQ90ke+azfZ1jaYb2W6XHS+nbtW+z+730kkmu\nSk42waqiwgOFc7NLLqn9fuVKs+iD/f+4gWxIP5BOdq9sdnXaRd/svhJ8hUcFbp9vC8awVleb2u6a\nNWbqunvuMeuPOq5H6s8sFjMLUbOrwkjmc0DQWpt5ncObzpbas8f8rZ99tpm0ZceOwPmbBzP5zJln\nwuOPN7FTFqSVp4GC3ITcVs95LUR7BWbwbcEY1rIy05SckmJmpFq0yDTP9u/vhfK6yXXXmVi6YEEz\nO0rmc0A4UXOCqpqqZpud1683TbYvvdTCLgk/07WrSZB86qnaJT0byITOCzuTOCaR3PtzpdYrPC4w\ng28zNTmr1SQmbdxoMiY/+shM6B5ohg6FsWPN6khNDjuSzOeAULqsFNZCzNama75TppipUvs23zrt\nt2bNMv/+TY5XzoS0i9LYHr2damtrZ+wQon0CM/g2U5N74w0zU87TT5t1c121Oosvuv56k1TT7NrA\nrRiOJXxQDpT+qhR+gJjrYpzOanXsmGkF0RrCwjxfRE/q0wduu81c7+rVje+XHptOZU0luw7v8lzh\nhCBQg28zNblf/cqsE3rrrV4qnwddeqkZ82ufhvL7780kIiLAZEOppRS0bXar7Ia7vPkmXHWVWTmo\nI7j3XkhIMOP0G5MWmwZAbnGuh0olhBG42c5OxrBWV5tHeHjLFvkOBBER8PHHpm+vtBQmTDDLwc2a\n5e2SifaqsdawYs8KxiSOITgrmNIFpaBss1tl1d1Xa3juOZO539Z1pv1NTAz89JPp39bajH0fNKju\nPpGhkSRFJ5F7MJfzBpznnYJ62apVq+KDg4PnAkMJ1AqZ51mBDdXV1ddmZGQ4XQQ2cIOvE59+apKQ\nli41wxA6CntSTVSUWaXmnXfgsccCM9mmI1lfuJ75a+dTba1mQuYEymaVEbw+mIh7IhokEC1ebHIc\nXn+9Y/3e7ROG/Oc/Jgnrootg7lyzPKFdWmwaS/OXUm2tJtjSoT4SAQgODp6bkJAwqHv37ocsFktL\nZoYXzbBaraqoqGjw/v375wIXOtunQ93lzJ1rJs5IS/N2SbzDYjGJZrt2mSFVjZKlBv2CfUm8JXlL\nzDCj3qVEZ0ZTOlhxwQV1l5t89lkza9vll3upsF6WmQl//rOZ8/yJJ+puS49Lp6qmqiNPNTm0e/fu\nZRJ4Xcdiseju3buXYloTnO/jwfK4XxNBo6AAFi6Ea65pw2ooAeSCC0zN5+OPG9lBlhr0G/ZFAXaX\n7WbHoR0nx/h+/bVZDORnP4P//tckWu3YAb//feAnWjWma1d48MHamq/jpCIDug2AQtj60tY2rX8c\nACwSeF3P9p42GmMDJ/g2EzTsszz97ndeKJsPiY83q8f8+9+N7CATbviNgtIChicMJzw4nOy8bEqP\nlxITFsNZZ5mM/oEDzVKTS5fChg0mmb2ju/lmM6HOu+/WPhe5KpJe7/Zi+2fbW73+sRBtFTjBt4mg\nUVMD8+bB5MlmObKObs4cM6mIUzLhhl84WnmUkooS+nXtx9jksazat4riimKiw6KJiYFf/9osEDJo\nkKnxKWUSDTu6iRPhD3+AIUMcnsyG/kX92d5lO9ZKa6vWPxbu89hjj3VPSUkZqpTK2LdvX8C1VwbO\nBdmDhn1+YoegYbGYJKOO3NzsqH7GZx32YVqyyIJP211mpm5KjklmeMJwvtr5FTXWGqJCY3jiCdO9\nMGhQ7U1WR0qyaopSpv+7jizoN68f34R8w97YvSRlJXmjaKKeiRMnHrn44otLzzrrrHRvl8UdAicc\nNRM0xo71Sql81ttvmzG/DT6IQJYa9AP2/t6k6CSiw6IZGDeQLQe3UFoYw913m+6FQYPqZvWKWlu3\nmvG/06YBmdB/Xn/4BrbdsI2kTAm+nlZWVma58MIL++7bty/UarWqP/3pT3uvu+66Q94ulzsFTvAF\np0Fj3jz4/HMTbKTZrdbmzfD882b6vbg4b5dGtFZBaQExJTFE/82M6c1KzWLLwS0U5HYDYMwY75bP\n182ebR7nnGMm4oidEEuXyi5si91GVgdfZWHMGBrUNH/5S0ruuYei8nIskyYxoP72X/+ag7feSvG+\nfQT//Of0c9z2/fc0O4PJhx9+GJ2QkFCVnZ29DaC4uDioPdfgDwKnz9eJf//bTK949Khpeha1LrrI\nzHH93HMmE7zJuZ+Fz9m9cTfJryTDTGASjNg5gjvH3snutYOJioL0gGyoc53p082EO08/bX5WStGv\nWz+2l2z3bsE6qFGjRlUsXbo0+sYbb0xcuHBh59jY2Bpvl8ndAqvm62DZMrjsMrNi0QcfmG5gUWvU\nKDPl5MMPm8fbb8MVV3i7VKIlqq3V7N25l2GHhkENUAlqiSJtbBo/fA+jR5t8OdG4tDQz1eZzz8FN\nN5lEzP7d+rNq7yoOVRyia6eu3i6i1zRVU42KwtrU9p49qW5JTbe+U0455cTq1as3ffDBBzEzZ85M\nXLRoUdlTTz21r7XH8ScBWR98/HEYPx6SkkyTc+fO3i6R71EKli+Hb7+FF16oXdVpwQIzIX1xsXfL\nJxq3r3wf1l5WkiqSIAgIBbLMnN1btphZzETzHn7Y3KTce6/5uV9X01q6rWSbF0vVMeXl5YVERUVZ\np0+fXnLHHXfsX7t2bYS3y+Ru/l/zzcmB7GwKBp3D0mMZXHCBWUj7iSfMcIv4eG8X0Hd16mQS0RyT\n0bZuNX3BmzebCRrs769kPvuOgrICiIfkV5PNmNQsIBNCMDdNjhNIiMYlJcHdd0NRkemCSY5JJiw4\njG0l2zg1sYNMgO0jVq1a1WnGjBlJFouF4OBgPXv27F2PPPJI/PPPP59QXFwcMnz48MFnnnlm6Xvv\nvRcwy0/5d/C1T6xRWcn7lkPcUZXBjh3mzl/u/tvmkUfMLEgPPAA73l9N32mTaodvyVq/PqGgtICw\n4DC6T+zeYAGF0FDpYmmNP//Z8ScLfbv2lZqvF1x88cVlF1988SbH5yZMmHDs/vvvd7ooQSDw72Zn\nh4k1vqg6h0HdD9Knj7cL5f+uucYkqL36QoXMduWDCsoKSIxK5NFHLNx9t0mWq642N06PPurt0vmn\nZctgyRLT9Lxn2x4qHquQma6EW/l38LVNrHHEEs0SJnD+2ZXeLlFASEqCKVNgwebRWEPCZLYrH6K1\npqC0gB6dknnmGdi5E95/3yTQzZkDq1Z5u4T+x2qFa681U0/23dUf/Zlmx/M7ZKpJ4Vb+HXxtE2ss\nvnIeVYRy3rW9vF2igPHkk/DtD2FYvlpkMlOkydknFFcUc7z6OFt/SObQITNkJiHBrFtbUCDje9vC\nYjFTcG7YAGte74ul2sKmbpugEplqUriNfwdfgMxMViRfQlSUWTBAuMbgwZCcjAm4M2ZI4PURW5ds\nhbXw+dwUBg0ycxWPHw///KdZPP7ss71dQv902WUwdCg88m0YQw8OZ0XyCqrDqhv0qQvhKv4ffDEL\nw//0kySauNq6dWYB8gMHvF0SAUAOLPvrMmqye/DjqhSmn1s7Z/MFF8ChQ5CR4d0i+iuLBR56CHLz\nofTsMygfV876D9eD3HMKNwmI4AvQo4e3SxB4wsLgiy9g/nxvl0QA7P56N9ujtzMubwK/VYrf1Bu/\nLosntM9FF5mWg/jeg+lyeheWdVnm7SKJAOb3wfeFF+DKK022p3CtgQPNmOkHH4Qvv/R2acTSgUsJ\nVsFctjeTeeEQM8XbJQosSpmx7df+zsLY5LFsLNrIoYqAnts/YEVERIwEM3nHlClT+ja170MPPRRf\nXl5+MhZOnDix/8GDB90+R5xLgq9SaopSKlcptU0pdY8rjtlS770HubmyXKC7vPsuDBgAU6fCp09s\ngVmzzPhq4VEnqk+wPOw7NseNZv9NkbAYaRJ1A3vrwdbF49iVr8nZLX/rvqK6DTWs1NTUqoULF+5o\nap+XX365x5EjR07GwiVLlmyLi4tz+9zS7Q6+Sqkg4AXgXGAwcIVSanB7j9uUNa+v46bTVzGwdwVL\nl5rAINwjPh6+/hpG9C/njRmbYOZMM7GJBGCP+n7PD3z59XG+/mgC3wxBAq8bVVbCO/Pi+OqfA/nw\ni2Xox7QMOXKz3Nzc0D59+gy58MIL+/Tt23fIlClT+paXl1sSExOH3XjjjYmDBw8e9Oqrr3bduHFj\n2Pjx4wcMGTJkUEZGRvqaNWvCAbZs2RI6YsSIgWlpaYNvvfXWXo7HHTBgwBAwwfv6669PGjBgwJC0\ntLTBjz76aPwjjzwSX1hYGDJx4sS00047LQ0gMTFx2L59+4IBHnzwwR4DBgwYMmDAgCEPPfRQvP2Y\nffv2HXL55Zf37t+//5Bx48YNOHLkSKs7fVxRXxwDbNNa7wBQSr0L/BzY1OSr2ionh+03PM/8yjlM\ntCzhhlsHMv3eVLecShjdusGiS+cQ+vBMqKmh6oSVkOxsyYD2oAfmfUPuykRm3tKXa67xdmkCW2io\nWRFt5M/O4N3Fc7l8xRYmPzKoQ7Q2zF09t2tBaYFLU1eTY5Irrx11bbPt93l5eeEvv/xy3jnnnHP0\n0ksvTX3yySe7A8TGxlZv2rRpM0BmZmbanDlzdg0bNuzEV199FXnjjTemfPfdd1unT5+ecu211xbd\nfPPNxbNmzeru7PhPP/109/z8/NBNmzZtDAkJ4cCBA0E9evSoefHFF3ssWbJka8+ePetUrZcuXRrx\n9ttvx65atWqz1pqMjIxBkyZNKo+Li6vJz88Pf/PNN3eMHTt213nnndd3wYIFXadPn17SmvfFFc3O\niUCBw8+7bc/VoZS6Xim1Uim1sqioqO1ny87mwuqPKKEbn6up3J7wDmFhbT+caJmon40lLAxKLHGM\n0SuYV36Zt4vUYbz07i6y1+zivCET+MuDklXlCYmJ8PmZI6g8Hs3Phi3gT6pYxvy6WUJCQuU555xz\nFOA3v/lN8fLlyzsDTJs27RBAaWmpZc2aNZ0vvfTSfgMHDhw8ffr03oWFhSEAq1ev7nzdddeVANxw\nww1Ol4X56quvom+44YaDISEhAPTo0aPJpuXs7OzO55133uHo6GhrTEyM9fzzzz/09ddfRwEkJiae\nGDt2bAXAyJEjj+Xl5bU6Cnmsp1RrPQeYAzB69Oi2rx6blUVo2MNQaZVZlzzJNqFJpy+XkfCfFK6d\n1ZWaVLNesnCvee+sp1snxTuXjpGMZg869coQsi+8lemj/8bmcU9TMu6PdKObt4vlVi2pobqLqvfH\nbf85KirKClBTU0NUVFT1li1bnLaqWiwWj61KHhoaevJcQUFBuqKiotUVWVfUfPcAyQ4/J9mecw9b\nEJBZl7wgM5NOD9zFx9ldOfdcMx3funXeLlSAy4HLDudy7YEUos+LkL5HT8qEsZ8m88n42+h3xTFm\nbXuaJ39xiKNfebtggWnfvn2hixYtigR46623uo0dO/aI4/Zu3bpZk5KSKl999dWuAFarlZycnE4A\no0aNOvLKK690A3jllVdinR1/0qRJZS+//HJcVVUVAAcOHAgCiIyMrCktLW0QC88888wjX3zxRZfy\n8nJLWVmZ5Ysvvuh65plnlrvqel0RfH8ABiil+iilQoHLgU9dcNzGyaxLXhUWBm+8YfqCr77aJKgI\n1zt8GIoXVpEXtYMxReky3aE3ZELve3tzW+/b2PXhEf6U9zHzpiA3QW6Qmpp6/Pnnn4/v27fvkMOH\nDwf/8Y9/bNA/+c477+x47bXX4tLT0wcPGDBgyAcffNAFYPbs2flz5syJT0tLG7xnz54QZ8e//fbb\ni5KSkioHDhw4JD09ffC8efO6AVx11VUHp0yZcjLhyu6MM844duWVVxaPGjVqUEZGxqDf/OY3RePG\njXPZgp1K6/bX1JVS5wHPYpb2flVr3eTaKqNHj9YrV65s93mFd33yiVlJ55NPoJdMq+1yt9wC7y3c\nwtQhz3DHilsYWjq0QyT9+KRZUPBkAf9XFs+emjC2PQwh93u+GEqpVVrr0a485rp16/KGDx9+0JXH\nbK3c3NzQqVOnDvjpp582erMcrrZu3bq44cOHpzrb5pJxvlrrL7TWaVrrfs0FXhEgcnL4+aZZfPdM\njgReN8jNhRdfhIypuYRdYKH/Lf0l8HpTFiQfT+YeHUY+8C+3jwIVgU6mphCtl5NjxvpWVhIUGkrx\nh0t4+ptT+ctfIMRpg49orUcfNc37p07JJTIylfDfhXu7SB1bJrAYpn4Fg+bCEx/BFQ/IlJ6ukp6e\nXhlotd7m+P30ksILsrNNR29NDVRW8u3becyaBc884+2CBYb8fHjnHbjmuhMUVe0kPS7d20USAJlg\nuQ/uegCioqCkVaM6hahLgq9ovawsM8wrKAhCQ7nwxiQuusjMAb1zp7cL5/8++sh8veiabVi1lfRY\nCb6+5OqrYelSiHWaUytEy0jwFa3nZLjX88+bWDx9Orggh69D+8MfYMsWKAvNJcgSRL9u/bxdJOHA\n3tS8fz98JcOORBtJ8BVtU2+4V1KS6adcuNAs7C5a5sgRU4uqsSXw2Idt9esHuQdz6dOlD6FBslC1\nL7r+erj0Ujjo1Txh4a8k+Ir2y8mBWbO4aVQOd98NZ5zh7QL5j3ffhQkTzMpRTz0F/fvD7NlQsayC\nXdm7SD8oTc6+6rHHoLQU7r3X2yXxfwcPHgx6/PHHnc7J3BRPLf/nDhJ8RfvYM59nziTonEk8/vMc\nEhvM7C0a87vfwbx5puXgrrugoACGAj/96if0D5r0m9JlQgcfNXSo6SKYOxe+/97bpfFvxcXFQfPm\nzYuv/7x9NqrGeGr5P3eQ4Cvap17mM9nZrFkDN95Y25QqnCsuNv2Hv/0tfPMNrFljasLjD8OOzjuw\nWC30Lewrs1r5sD//GXr2NMua5uZ6uzT+684770wqKCgIGzhw4OChQ4cOysjISD/rrLP6DxgwYCjA\n5MmT+w0ZMmRQ//79hzz11FNx9tfZl/9z1TJ/niTBV7RPvcxnsrLYtg1eegk+/tjbhfNdhw9D376m\nqdluxAj4v/8DdSbkxeaReCSRkOAQyPJaMUUzoqNNzuFFF5l++g5j0aJIZsxIwDYXc3s9/fTTu5OT\nk09s2bJl0+OPP75706ZNEbNnz87Py8vbAPDWW2/lbdy4cfPatWs3vfzyyz3279/foKk5Pz8//NZb\nby3ctm3bxpiYmJoFCxZ0dUXZ3EWCr2gfJ5nPv/yl6bv8618l87kxs2dDWRmcfXbDbfp0Td41efSZ\n3EdmtfJVtjwHcnIYOBDmzIHgYDhwwDwd0BYtimTq1DSeeCKRqVPTXBWAHZ1yyilHBw4ceHLW+L/+\n9a890tPTB2dkZAzav39/yMaNGxvMOuOKZf48SYKvaL96mc9BQfDHP8IPP5hWaVFXRQU8+yycey4M\nH95we+HRQiriKkidliqB1xc55DkwaZL52eaZZ8x96OHDXiyfuy1eHEVVlQWrFaqrLSxeHOXqU0RE\nRFjt33/22WdRS5YsiVq5cuWW3NzcTYMGDapwtoRf/WX+qqurpdlZdDzTpkF8vKkFSO23rnnzoKgI\n7rmn3oYcYBbkLc0DoE/XPp4ummgJJ3kOdpdcYm6u3nnHa6Vzv0mTygkJsRIUBMHBViZNavcyezEx\nMTVHjx51Go8OHz4cFBMTUxMVFWVds2ZN+Lp161xe0/YGmdtZuEWnTmYIxrFj3i6Jb7Fa4bnnYOJE\nGD/eYUMOMAmohJ3DdxJ2cxgJnRO8VErRJHueQ2XlyTwHu4wM05oxd65JOgxIkycf5bPPtrJ4cRST\nJpUzefLR9h4yISGhJiMj48iAAQOGhIWFWbt3734yzfniiy8unTNnTve+ffsO6du37/Hhw4e3+3y+\nwCVLCraWLCnY8Zw4YRYKEGZShtLSegk6s4CZQA08Pv5xgs8J5o/3/9FLJRTNyskxNd6srAbriv/j\nH2Y5yNWrYeRI1542UJcUDFRuX1JQiKasWGECTUe/3yovNzXfuDgnmbFZQChUB1dT0LWAPkOlydmn\n1ctzcPSrX0HXrvDjj14ol/Ab0uws3K5PH7PU4IUXwsaN5oOpI7rhBti718wHbKl/22tbsm7P13uo\n7lFN6phUL5RQuELXrub3HC6rQIomSM1XuF18PHz4oRmG0VGn4luxwiThnHGGk8Brlwl5V+ZBPKR2\nSfVg6YSr2QPvkSPeLYfwXRJ8hWs5jH90NHKkmYrv5ZdNIOpINm+GX/zCzIR09911t2mtOV59/OTP\neYfziAqLolunbh4upWizRv7mr7gCTjlFltkUzknwFa7TxPhHgL/8BXr16lgzX23YYDKbrVb48kuz\nCLuj7Lxs7vzvnazZtwaAnYd3ktolFaV8eoiisGvib/6WW8x438xMk3wlhCMJvsJ1mhj/CCbwrF7d\nAWYActC1KwwebOZuHjKk4fZNRZuotlYzZ9UcsvOy2X9kvzQ5+5Mm/ubHjoVvvzVZ/hMmmO+FsJPg\nK1zHyTzP9cXb1i157z244AJ44w3TF7xnD2zb5tHSulVNjZlcJDHRfB6npTXcR2tN3uE8RvUcRVps\nGu+sfwetNX26SKaz32jmb37QIFMZnjrVfC+ca+uSggAPPfRQfHl5ud/FMr8rsPBhTuZ5bkxFBaxb\nZ2bCSkgwS+plZHiwrG52333wy19CdXXj+5RUlFBWUEb6f9O52XozwxOGExYcJjVff9KCv/levcxq\nVd2kG79RjS0p2BIvv/xyjyNHjvhdLJOhRsK1MjObDLp2V18NV11lkq+++w4iIiAmxv3F84RVq8xq\nRddcYybbb8zOb3bCZ9Dnmz6EPBrCjYtupGJyBREhEZ4rrGi/Fv7Ni8Y5Lik4ceLEsvj4+KqPPvqo\nW2VlpTr//PMPP/PMM3vLysosF154Yd99+/aFWq1W9ac//WnvgQMHQgoLC0MmTpyY1rVr1+oVK1Zs\n9fa1tJQEX+E1SsHpp5uH3cKFJiP6/fdNS56/qaqC3/3ONK8/+WTT++atyiO4KpjEw4mgQC1RRIyV\nwCv8wCIiWUwUkyhnMu2e7vHpp5/ePXXq1E5btmzZ9OGHH0b/61//6vrjjz9u1lozefLk/v/5z386\nHzhwIDghIaEqOzt7G5jacmxsbM2LL77YY8mSJVt79uzZRDuT7/G7qroIbGVlJhv666+9XZK2eeIJ\n05w+ezZ06dL0vjv77CTlaArBKhhCkXV7A0Ejw44CyiIimUoaT5DIVNJYhEsXOli4cGH0N998Ez14\n8ODBQ4YMGbx9+/bwLVu2hI8aNapi6dKl0TfeeGPiwoULO8fGxta48ryeJjVf4VMuuMA0Py9YAJMn\ne7s0rXPsGLz0Elx+uVlcvSlWbWVXl11MuHUCnIMJvNJy6d/sw47sCy40k/fgtxYTRRUWrEA1FhYT\n5Yrar53Wmttuu23fXXfd1WC+6dWrV2/64IMPYmbOnJm4aNGisqeeemqfq87raVLzFT6lUye47DL4\n4AP/mx0oIsL09774YvP77inbQ1VNFX3G9IEZSOANBM0MtQsYkygnBCtBQDBWJuHSJQXPPffcsjfe\neCOutLTUArBz586QPXv2BOfl5YVERUVZp0+fXnLHHXfsX7t2bQRAZGRkjX1ffyI1X+Fzpk2DV14x\nU1JOm+bt0jRPa9NH/Ytf1A6lak7e4TxAppEMKE0sNRhQJnOUz9jqyj5fxyUFzzrrrNJLL7205NRT\nTx0IEBERYX3rrbd2btmyJWzGjBlJFouF4OBgPXv27F0AV1111cEpU6ak9ejRo9KfEq5kSUHhc7Q2\ntd+rrjLjI32Z1vDss3DHHfDaayaLuyUWrFvA2v1refqcp2U2q0DSxFKDriBLCvqXppYUlJqv8DlK\nwb/+5e1SNO+nn8yC6YsXw3nnwW9+0/LX7jy0kz5d+kjgDTQy7Ei0kN+1kws/0s7Mz2PHzHzQFRUu\nLpcLzJ0Lw4bBDz/ACy/Ap5+2fGjU8erj7Duyjz5dZSYrIToqqfkK93BB5ud338GDD5rpJ2fPdk8x\n26KUWLIAAA0QSURBVGrkSLjuOrOeeq9erXttfmk+Wmvp7xW+wmq1WpXFYvF8H2QAs1qtCrA2tl1q\nvsI9XJD5edZZ8Mc/muxhX2uGzsiA559vfeAF2Ll8J6yF1NxUl5dLiDbYUFRUFGMLFsIFrFarKioq\nigE2NLaP1HyFe7go8/PRR82KQNdeayavuP/+2oXKvWH1ahN0n3wS4uLacIAc2PrXrfQI60HnWZ1h\nMTLMKFC5OfnKVaqrq6/dv3//3P379w9FKmSuYgU2VFdXX9vYDpLtLNzHRR8+eXnw29+ar9u3m4Qs\nd1q3Dnr0MAs+ONLatKT/+KMpR1vmoq55rIbbv72d0wpO41ebfgUPY8b5isDipgk33JHtLLxD7nKE\n+2Rmmk7Rdn7opKbCV1/B5s0m8H72GTz+uGuKWN/8+aZJOTMT9u+vu+2LL8y0l3/+c9sXgdh12i5O\nhJ4g/VC6TCkZyDrKhBuizST4Cr8RFma+/u9/MHMm7N5dd7u10dSGluvdG848EwoLzRhj+yxbixaZ\n2veAAXDDDW0/fm5KLkyF9BvTpck5kLVgbWvRsbUr+CqlLlVKbVRKWZVS0hQiPOL2202g/fvfa5/L\nyYHu3VueFf3ZZ5CSArGxpqb7yCPm+aws+PJL+Oc/Yc0aM9lHdTXcdZfp4/3kE/NZ2la5xbn06teL\nqHujJPAGslasbS06pvbWfDcAvwS+cUFZhGiRPn3g0kvN0oOlpbBvH1x8MRw+DLfc0vyw4n/+0yzg\n0KUL/N//mf7d99+H3Nzafc4/3yyScP75Zk3eTz6B77+HQYPaXu5qazXbS7aTHpfe9oMI/+GibhcR\nmNqV7ay13gzILD2idVyQiHXXXfDee2aCiy++MEH4229NZvSpp5p9tIYlS+Ctt2DHDjjtNHjsMbjw\nQrPY/c031zZlO3PddbXfp6S0qZi1ciDv6zwqe1SSPlKCrxAdnceGGimlrgeuB0hp9yeZ8Dv2gBsb\nC7fd1u4s0IwMuOYaSEw0L7/lFjj9dPOwn+6KK2DXLoiONrNR2YcohYfDnXe67tKalQNMMv29aqAi\nLS0Nenrw/EIIn9Ns8FVKLQISnGy6T2v9SUtPpLWeA8wBM9SoxSUU/s9x2IVSpsPWaq3NAm1j7ffV\nV50/X11t5lzu3dt0uV18sVnuz2uygUrI7ZZLYmkikcsiYbwXyyOE8Lpmg6/W2s+WNBc+x3HYhcVi\nMkCVclsWaHCwSZbymd6QLKgKr2J7t+1M3DtRhhcJIWSGK+EB9We7evZZKC5268w/PhN4ATJh58c7\nqc6pJv20dMlyFkK0L/gqpX4BPA90Bz5XSq3VWv/MJSUTgcM+7MIPptpzl01Jm1AjFQPOHODtoghv\n8JOpJoXntDfb+SPgIxeVRQSyDrzOaVVNFcvylzG4+2AiQrzZ+Sy8wk1TTQr/JjNcCeFmywuWU36i\nnCn9p3i7KMIbZKpJ4YQEXyHcyKqt/G/7/+jTtQ8DukmTc4ckU00KJyThSgg3WrV3FQePHeTSIZfK\nZDQdleQ8CCck+ArhJlprFm5bSELnBIb3GO7t4ghv6sA5D8I5aXYWwk02FW1id9luftb/Z1LrFULU\nIcFXCDdZmr2U6LXRjMkf4+2iCCF8jARfIdyg+ttqNr2xiREfjCD47GAzv7MQQthI8BXCDX5a8hMn\n1AmG7R8GlZj5nYUQwkaCrxBusH7QeoJVMOmH0iEUmc9ZCFGHZDsL73Kcdg/8fzhGDpAN65PWM/DK\ngYSdHmYCr59ejhDCPST4Cu9xnHbPvtJRdbX/TcF3cq3iqXDbMA6EHKDwrEImXTsJZni7cEIIXyTB\nV3iP47R7Vqt5Tut2r/PrUXXWKq4A61DW91wPNTBs3TA439sFFEL4IunzFd7jOO1eSIh/TsFX5wbi\nKwiqZn3CenpW9CT2zFhvl04I4aOk5iu8p/60e+B/fb511ipezfFn1vFT4U9MGjZJ+nmFEI2S4Cu8\nq/60e/4SdO3q3UBsTg2iZmUNp4w9xdslE0L4MAm+QrSXww3E+nULiAiJoF+3fl4ulBDCl0nwFb7J\ncQiSH9WGh8UPIyk6CYuSdAohROMk+Arf45hB7GfDjkb2HOntIggh/IDcngvf45hBbB92JIQQAUSC\nr/A9jkOQfHXYUU4OzJplvgohRCtJs7PwPfWHIPlak7MfN4sLIXyDBF/hm+oPQfIlzprFfbWsQgif\nJM3OQrSWPzSLCyF8mtR8hWgtX28WF0L4PAm+wvf54phfX24WF0L4PAm+wrf5UnKTL94ECCH8kgRf\n4dsaG/Pr6SDoSzcBQgi/J8FX+LY6qwaFQmysZ4Ogvbabny8ZzkIIl5HgK3xb/eQmTw7zcaztBgVB\nsO3fRTKchRDtJMFX+L76yU2ONWFXBUFn/bmOgR7guusgJUX6fIUQ7SbBV/gXdwzzaaw/t36T97Rp\nEnSFEC4hwVf4n8aG+bQ1G7l+U/aCBbXHkfG8Qgg3kOArAkN7spEda7hBQfDaa1BdXXucGTPcWnQh\nRMcjwVf4N1dkIzs2ZefnwyuvSFazEMKtJPgK/+XKbGR7U3ZODsyf7/qELiGEcCDBV/ivlmYjO/YF\n21/XWB+uzNsshPCAdgVfpdSTwAVAJbAduEZrfdgVBROiWS3JRq5fO1aqbn9uYwFYgq4Qwo3aW/P9\nEpihta5WSv0VmAHc3f5iCdECTdVSnfUFW61mm9YNs5ol2AohPKhdwVdr/T+HH78DLmlfcYRoJWe1\n1Mb6gh1rvs6ymiUACyE8xJV9vr8F3mtso1LqeuB6gJSUFBeeVoh6muoLtm+XrGYhhBc1G3yVUouA\nBCeb7tNaf2Lb5z6gGnirseNorecAcwBGjx6t21RaIVqiub5gyWoWQnhZs8FXaz25qe1KqauBqcAk\nrbUEVeF9LclYlqxmIYQXqfbES6XUFOBvwEStdVFLXzd69Gi9cuXKNp9XCCE6IqXUKq31aG+XQ7Sf\npZ2v/wcQBXyplFqrlHrJBWUSQgghAlp7s537u6ogQgghREfR3pqvEEIIIVpJgq8QQgjhYRJ8hRBC\nCA+T4CuEEEJ4mARfIYQQwsPaNc63zSdVqgjY1c7DxAEHXVAcf9LRrrmjXS/INXcUbb3m3lrr7q4u\njPA8rwRfV1BKrexog8072jV3tOsFueaOoiNes6hLmp3/v737CbGqjMM4/n2YyUqF/i3E1HAWQzEF\nZUTYH0IyyEqyVU0gSNEuSKMIrUW0aBdRiwrCLKFQwqQkKAoLamVRLjInSTR0bPwD0R9aZNHT4hzx\nMnhbzZzDee/z2cx933Mv/J6ZO+/vznnPnRsREdGwNN+IiIiGdbn5vt52AS0YtMyDlheSeVAMYubo\n0dk934iIiK7q8l++ERERnZTmGxER0bDONV9JqyQdkHRQ0sa265kNkpZI+lzSfknfS1pfz18q6VNJ\nP9ZfL2m71pkkaUjSXkkf1uOi8wJIuljSDkk/SJqQdFPJuSU9Xj+n90naJumCEvNK2iLppKR9PXN9\nc0raVK9pByTd2U7V0aRONV9JQ8ArwF3AGPCgpLF2q5oV/wBP2B4DlgOP1jk3ArttjwK763FJ1gMT\nPePS8wK8DHxs+yrgWqr8ReaWtAh4DLjB9jXAEDBOmXnfAlZNmztnzvp3exy4un7Mq/VaFwXrVPMF\nbgQO2j5k+zSwHVjTck0zzvaU7W/r239QLciLqLJure+2FbivnQpnnqTFwD3A5p7pYvMCSLoIuA14\nA8D2adu/UnbuYeBCScPAXOBnCsxr+wvgl2nT/XKuAbbb/sv2YeAg1VoXBeta810EHO0ZT9ZzxZK0\nFFgG7AEW2J6qDx0HFrRU1mx4CXgK+LdnruS8ACPAKeDN+nT7ZknzKDS37WPAC8ARYAr4zfYnFJr3\nHPrlHLh1LbrXfAeKpPnAe8AG27/3HnP1HrEi3icmaTVw0vY3/e5TUt4ew8D1wGu2lwF/Mu2Ua0m5\n6z3ONVQvOi4H5kla23ufkvL+n0HJGf11rfkeA5b0jBfXc8WRdB5V433H9s56+oSkhfXxhcDJtuqb\nYbcA90r6iWor4XZJb1Nu3jMmgUnbe+rxDqpmXGruO4DDtk/Z/hvYCdxMuXmn65dzYNa1OKtrzfdr\nYFTSiKQ5VBcp7Gq5phknSVT7gBO2X+w5tAtYV99eB3zQdG2zwfYm24ttL6X6mX5mey2F5j3D9nHg\nqKQr66mVwH7KzX0EWC5pbv0cX0l1PUOpeafrl3MXMC7pfEkjwCjwVQv1RYM69x+uJN1NtT84BGyx\n/XzLJc04SbcCXwLfcXYP9Gmqfd93gSuoPpLxftvTL+roNEkrgCdtr5Z0GeXnvY7qIrM5wCHgIaoX\nxUXmlvQc8ADVFf17gUeA+RSWV9I2YAXVRweeAJ4F3qdPTknPAA9TfV822P6ohbKjQZ1rvhEREV3X\ntdPOERERnZfmGxER0bA034iIiIal+UZERDQszTciIqJhab4RERENS/ONiIho2H9VJuFw8mVxGAAA\nAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8b8ad61e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Simulate a failure in sensor 2 for a random 40-index period\n",
    "\n",
    "def holdout(): # gives arbitrary slice from 0 to 100 width 40\n",
    "    width = 40\n",
    "    start = np.random.randint(0, len(s2) - width)\n",
    "    missing = slice(start, start + width)\n",
    "    return missing, np.r_[:start, missing.stop:len(s2)]\n",
    "\n",
    "# Find the most likely scaling for reconstructing s2 from s1\n",
    "def factor_finder(train_ix):\n",
    "    return np.mean((s2[train_ix] + 0.0001) / (s1[train_ix] + 0.0001))\n",
    "\n",
    "test, train = holdout()\n",
    "f = factor_finder(train)\n",
    "\n",
    "def plot_factor(factor):\n",
    "    times = np.arange(len(s1))\n",
    "    test, train = holdout()\n",
    "    plt.plot(times, s1, color='blue', ls='--', label='s1')\n",
    "    plt.scatter(times[train], s2[train], color='red', marker='.', label='train')\n",
    "    plt.plot(times[test], s1[test] * factor, color='green', alpha=0.6, label='prediction')\n",
    "    plt.scatter(times[test], s2[test], color='magenta', marker='.', label='test')\n",
    "    plt.legend(bbox_to_anchor=(1.05, 0.6), loc=2)\n",
    "    plt.title('prediction factor {}'.format(factor))\n",
    "    plt.show()\n",
    "\n",
    "plot_factor(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100, 15)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Cubic kernel convolution and interpolation\n",
    "# Complicated example; take a look on your own time!\n",
    "\n",
    "import scipy\n",
    "import scipy.sparse\n",
    "\n",
    "# From Cubic Convolution Interpolation (Keys 1981)\n",
    "# Computes a piecewise cubic kernel evaluated at each data point in x\n",
    "def cubic_kernel(x):\n",
    "    y = np.zeros_like(x)\n",
    "    x = np.fabs(x)\n",
    "    if np.any(x > 2):\n",
    "        raise ValueError('only absolute values <= 2 allowed')\n",
    "    q = x <= 1\n",
    "    y[q] = ((1.5 * x[q] - 2.5) * x[q]) * x[q] + 1\n",
    "    q = ~q\n",
    "    y[q] = ((-0.5 * x[q] + 2.5) * x[q] - 4) * x[q] + 2\n",
    "    return y\n",
    "\n",
    "# Everything is 1D\n",
    "# Given a uniform grid of size grid_size\n",
    "# and requested samples of size n_samples,\n",
    "# generates an n_samples x grid_size interpolation matrix W\n",
    "# such that W.f(grid) ~ f(samples) for differentiable f and samples\n",
    "# inside of the grid.\n",
    "def interp_cubic(grid, samples):\n",
    "    delta = grid[1] - grid[0]\n",
    "    factors = (samples - grid[0]) / delta\n",
    "    # closest refers to the closest grid point that is smaller\n",
    "    idx_of_closest = np.floor(factors)\n",
    "    dist_to_closest = factors - idx_of_closest # in units of delta\n",
    "\n",
    "    grid_size = len(grid)\n",
    "    n_samples = len(samples)\n",
    "    csr = scipy.sparse.csr_matrix((n_samples, grid_size), dtype=float)\n",
    "    for conv_idx in range(-2, 2): # sliding convolution window\n",
    "        coeff_idx = idx_of_closest - conv_idx\n",
    "        coeff_idx[coeff_idx < 0] = 0 # threshold (no wraparound below)\n",
    "        coeff_idx[coeff_idx >= grid_size] = grid_size - 1 # threshold (no wraparound above)\n",
    "        \n",
    "        relative_dist = dist_to_closest + conv_idx\n",
    "        data = cubic_kernel(relative_dist)\n",
    "        col_idx = coeff_idx\n",
    "        ind_ptr = np.arange(0, n_samples + 1)\n",
    "        csr += scipy.sparse.csr_matrix((data, col_idx, ind_ptr),\n",
    "                                       shape=(n_samples, grid_size))\n",
    "    return csr\n",
    "    \n",
    "lo, hi = 0, 1\n",
    "fine = np.linspace(lo, hi, 100)\n",
    "coarse = np.linspace(lo, hi, 15)\n",
    "W = interp_cubic(coarse, fine)\n",
    "W.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdMAAAD8CAYAAAAsc076AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8TNf/x/HXmURENoSINYk1sbSoCC0ttRTd7LUWpRSl\nSr9oS/1arS7owrfW4lutoNWqvdTSWqpK7HutiSUhtsiGJHN+f0yoqiXMTO4sn+fjMY/M3Lm5531J\n5pN777nnKK01QgghhHhwJqMDCCGEEM5OiqkQQghhJSmmQgghhJWkmAohhBBWkmIqhBBCWEmKqRBC\nCGElKaZCCCGElaSYCiGEEFaSYiqEEEJYydPoAHdTuHBhHRYWZnQMIYRwGlu3bj2ntQ4yOoe7cehi\nGhYWRkxMjNExhBDCaSilYo3O4I7kNK8QQghhJSmmQgghhJWkmAohhBBWkmIqhBBCWEmKqRBCCGEl\nKaZCCCGElaSYCiGEEFaSYiqEEEJYSYqpEEIIYSUppjmgtUZrbXQMIYQQDkqK6T1kZGTQo0cPxo4d\nC8CsWZqwMDCZICwMoqMNjSeEEMIBSDG9Bw8PD9LT00lLSyM6Gnr0mMe52EagE4mNhV69pKAK60RH\nc/s/0C5cgLVr2TJkCK9UrkxiixbQrBk/RETwaP78nGrSBLp0YU/v3nz18sskr18PGRn33q4QwuaU\nI5++jIyM1EYNdJ+RkYGnpydKKcxmMyaTidKhZmrENcWDlUTSjiHMARShoXD8uCExhZOLjrb8QZaW\nBqCpyRZqekzld485zLmWRkVgMfCSUmwMCaFCUBC/ZWby0fHjRBcvTuHUVEbHxTFUay4B+X19WRYR\nwdKMIH49OJr9Vx+60ZaPD0ydCp06GbOvIncopbZqrSONzuF2rl8PdMRHjRo1tFHGjRunw8PD9fnz\n5y0LUlP1j7TUGvRmIrUGPYz3NWitlGExhZMLDdU6gNP6cVrrbymtNeijKP2w8tebevbUevlynRUX\np7XZfMdtZKWl6diVK7WeO1frvn314MBAHQo6C/TvPKrb8KYOIFaDpT3h2oAY7QCf3+72kNO8d1Cm\nTBnq1atHYGCg5bChfn1asIDXGEcUm/mCloTyDp2YRUiI0WmFM9IXLtArdhhHKMc+fmQl6fRgGjU4\nx24uU2vqVGjSBFOpUqDUHbdjypePkEaNoF07mDCB0efPY+Ygb/IJflxmNx9TnTJ8yFtciT2Ti3so\nhBsxuprf7WHkkek/TJ+uNej1r87RPj5ag9Ym3tR5UPo4nnrZyC1GJxTO5MoV/fkzz+imnp7aDHo2\n7XUVVmjQNx7WHkGGhl7fllmH87UeSQOdhdIXyas/qVdPp8TH22JPhANCjkwNeciR6W38+uuvpKen\n/71g2jSoWJG6/23H1KkQGgpm3qRM0R0Uy+dDs8P/NS6scC4bNkC1auRfuhSfwoX5ccRGXvaZwx6e\nurGKjw+MGmVdM6NGWbYDioN0ZQSrqe59gLkVH2Xo2rX8WbkyzJplqbdCCOsZXc3v9jDiyDQ+Pl57\neHjot956y7Jgzx6tQetPP739N/Tooc0+PlqnpOReSOF0LsfH684VKuh51w87f/75xnuzZlkWKWX5\nOmuWbdq803b3zpmjdc2aWoNe9PDDOu6PP2zToHAIyJGpIQ/DA9ztYUQxNZvN+tdff9WxsbGWBQMG\naJ0nj9Znz952/a5NmujeYLtPQOF69uzRVyMi9KOgP3/iCa2Tk41OpHVWlk777391EaV0a09Prb/5\n5q6dnITzkGJqzENO895CKUX9+vUJCQmBK1fg22+hZUsICrrt+sEPP0yRgADLekLcRGvN/15+mfSa\nNfG6cIH1y5fz+tq14OdndDQwmcjXrx8bV69mXLVq0KULye3akZyQYHQyIZySFNObbN26lffff5+L\nFy9aFvz0k+XG+Z497/g9n4wezXv9+8PKlRAfn0tJhcPLyuLPjh3pPn0635QqBTt24NGkidGp/qXs\nk09SYtMmeP99es6bR53Spbm2f7/RsYRwOlJMb/L777/zwQcf4OnpaVkwbRqULg0NGtz9G198kdNm\nM8yZY/+QwuHplBRo3Zrac+eyoW1beu3dC8WKGR3rzjw8YPhwen70EX1MJrweewx++cXoVEI4FSmm\nN3nttddITEzE39/fMqTRmjXQo4dlPLa7eG/uXMqbTKR+/XWu5BSO63hMDDWDg9m5aBGMH0+d779H\nXf/jzME1fPNN+uzeDSEhxDRrxqQuXYyOJITTkGJ6i4CAAMuT5cstX9u2vef3PP/884xu1Qp274Y9\ne+yYTji006dRHTpgTk8n9eOPoX9/oxPdvzJlYP16JhctyuhvvyVl2DC5fUaIHJBimm316tV07tyZ\nM2eyR4j55RfL6ODly9/ze6tXr86rEybgqxTMn2/foMIhnd22Df3444QmJLB1zRoeGzLE6EgPLiCA\nKUeOsKFNG/w+/BDduzdZNw2gL4T4N5sUU6VUU6XUQaXUYaXUm7d5P0Ip9YdS6qpS6j+2aNPWTp06\nxYYNG8ifP79l5o3Vq+Gpp+46jNvNrubPz8KwMC7JtSa3E7dxIw/VrMnY06dh5UpU/fpGR7Kah7c3\nJb7/HoYO5Z2pU+lYoQKZV68aHUsIh2V1MVVKeQATgGZAJaCDUqrSLatdAF4Dxlrbnr106dKFY8eO\n4e3tDZs3w+XLlmKaQ7t27aLFsWMs3LQJbh49Sbi2hARKdOlC9zx5eG7WLKhd2+hEtqMUfPQRBZ96\nivzHj2Pq1u0fU7wJIf5miyPTKOCw1vqo1voaMBdofvMKWuuzWustgEP/JqrrR6G//GLpdHSvXrw3\niYyM5NePPqJ9VhZs3GinhMKRJB46xIUGDfBISOCj334jonVroyPZnlK8sWIFUz7+GNPcueyq3ZYy\nIekyR6oQt7BFMS0BnLjp9cnsZQ9EKdVLKRWjlIpJTEy0OlxO/Pjjj9SsWZOTJ09aFvzyC0RFQcGC\nOd6GUor6r75KXg8PSy9g4ZKuT7jtq1JoFP4ITfcfwDx/vmsdkd6GGjqUje0+5sVtCyl5oipoM7Gx\nlrlYpaAK4YAdkLTWU7XWkVrryKA7jDpka3nz5iUwMJCiRYvCxYuW07z3cYr3ugsZGYwtUYL9S5fa\nIaUw2vWJvE/EZjGbTozVqeTzHMacxPv/WXFGHTcNJYjGjOAQ/6U/oElLg2HDjE4mhPFsUUxPAaVu\nel0ye5nTePbZZ1mxYoVlsIY1a8BsfqBimpmZyZC4OH7dtQuSk+2QVBhp2DDL1LZD6UlzFrGQ8azL\nfN9tiklcHKxmBdsYzKtM5D/0u7FcCHdni2K6BSivlCqtlPIC2gOLbLDdXJGZmYnZbP57wS+/QECA\n5TTvfSpSpAjx8+bRV2tYv96GKYUjiIuDhnRnLP/jddoywc2KSUgIgGIonzCU5oxnIg3pmb1cCPdm\ndTHVWmcC/YAVwH7ge631XqVUb6VUbwClVFGl1ElgEDBcKXVSKRVgbdu2MH/+fAoWLMihQ4csN6ev\nWGHpeJQnzwNtL/iZZ8DLS66buqD2QauZy0xaU57xzLqx3F2Kyc1zpI5lDi0oy1ym87+2y4yOJoTh\nbHLNVGu9TGtdQWtdVms9KnvZZK315OznCVrrklrrAK11geznl23RtrXCwsLo3LkzoaGhcPgwxMZC\n48YPvL1zqan0LFSI1QsX2jClMNqlffuYlt6esyqCxWxD4wXYZiJvZ9GpE0ydCqGhoFU+dpfagSm0\nGk9MbMvJn382Op4QhnK4Dki5LSoqigkTJuDl5QVr11oW3sctMbfy9/dnWUoKRw4ftsw4I5xeZno6\nz9WuTbf0Sxz55EcKh/qhlKWoTJ1qKTLuolMny7DVZjPsi/MjcOMSXlWKx597juQjR4yOJ4Rh3LqY\nms3mv4cPBMt1zqAgCA9/4G3mzZuXk0uX0gvgt9+sjSgcgMdbb9EpOZlWffvy3OCIG8Xk+HH3KqS3\nVbw43SdN4j8mE37dusG1a0YnEsIQbl1MDx06RNGiRYm+fqPc+vXw+OM5HkLwTlStWpA3L/z+uw1S\nCiOZ589HjRtH79deo/24cUbHcUhRL77IqzNnojZs4HK/fkbHEcIQbl1MCxQowGeffUadOnXg5Ek4\ndsxSTK2UlJ7OE3nzMkOumzq1U1u38vALL7AhPBzGjDE6jmPr0IG/unenwldfMatXL6PTCJHr3LqY\nBgcHM3DgQMLCwv6+leWJJ6zebkBAAH6BgXjHxsppL2dlNpMyYAB+ZjNFPv/c0kNb3FXpL7/kueLF\nqT5zpmU6QiHciFsX0x07dpCSkmJ5sX49+PtD1apWb1cpxbLRo+mYmQk7d1q9PWGACRMI//13/pgw\ngQrNmhmdxinkyZePr7Zvp3KhQtC2LebLDtFhX4hc4bbFNDMzkzp16jDs+vA169bBY4+Bh4dtGqhd\nGw1kynVTp7Pthx94a+BArj39NKp3b6PjOJciRWD2bN4+eJDu1aujbx4QRQgX5rbFVGvN999/T/fu\n3eH8edi71ybXS687pRRFTSZmffedzbYpckFWFstef51vtSb1s8+s7ozmlurXx6t+fbyPHiVr+nSj\n0wiRKzyNDmCUPHny8Mwzz1heLMoe/dAG10uvK1a8OC1CQigdG2uzbYpc8PnnDD91ir5ffUVBK26R\ncnf/t3IlqmlTeP11qFcPKlQwOpIQduW2R6YbN25kz549lhfr11s6mNSsabPtm0wmpvTtS734eDh7\n1mbbFfaze/Fijg0bBi1aENijh9FxnJry9ISZMzmSJw+v1K3LtdRUoyMJYVduW0wHDhxI//79LS/W\nrbMMbO/tbdtGatfmPJB6fWQl4bB0VhavdOrE05mZmL/8Uk7v2kKJEuzq2ZPvExPZN3Cg0WmEsCu3\nLabR0dF89tlnkJoK27bZ9BTvdft8fCgMLJTZkx2emjKFucnJzHj7bUwlHnhue3GLlmPGcLRDB6pN\nny6DmAiX5rbFtFy5clSvXh02bYLMTJt2ProuvFo1PixenBoJCTbftrCd9CNH4K23CGnUiEdHjjQ6\njsspOGUKhIWxoHVrLsfHGx1HCLtwy2IaExPDvHnzyMzMtJziNZkst8XYmIeHB2+1bEn43r2QlWXz\n7Qvrmc1mGteuTf+UFJg0SU7v2oO/PwdHjqT1mTP8t1Uro9MIYRduWUxnzpxJjx49MJlMlmJavbpl\nQnA7yIiM5M+UFJL+/NMu2xfWyVq6lEbnzlGrZUsoV87oOC4rvFMnfmnZkqGbNskEEMIluWUxHTNm\nDFu2bMGUkWE5zWuH66XXbfX1pTawcuZMu7UhHlBaGnlee413IyLoLNe17a7hrFl4litHWteuJMvp\nXuFi3LKYent7Ex4eDjExcOWKXYtpjebNmefnx5NpaXZrQzyYkU2bsun4cZg82TLLj7AvHx+uTplC\nzbg4BjZsaHQaIWzK7YrpiRMn+PDDDzl58qTlFC9A3bp2ay+PlxdtHnuMQrt22a0Ncf8uxsQwdf16\nlj30kGVQAZEr8jZoQM8nnqD9/v3Su1e4FLcrptu3b2fYsGGcO3fOUkwrV4bChe3a5tnKlflm925S\nZfAGx6A1Bd95hwP+/rwl0+TluteXLqVRaCi8/DJcvWp0HCFswu2K6fPPP8+lS5d4qGJFy1/GdjzF\ne12Mnx9dtWbz3Ll2b0vc2/4vv8S8fDl+I0eSr3Rpo+O4Hz8/9KRJfHHgACObNDE6jRA24XbFFCB/\n/vx47NkDycm5Ukzrde3KLqBeRobd2xJ3lxgbS+0BAxhauDD062d0HLelmjVjT9mybF+3DrNcAhEu\nwK2Kqdaafv36sWrVqr+vl9phsIZb+ZYty0MlS2KKibF7W+LuCk2fzjit6fn55+DptvM8OISJ69Yx\nPzAQ0yuvgEzVJpycTYqpUqqpUuqgUuqwUurN27yvlFLjs9/fpZR6xBbt3q+LFy8yb9489u3bZymm\nZctCLg0dtzs8nHdXrMAsHxrGiY3FNGYM3dq1o0LnzkancXtexYujPvuMxE2bWPOf/xgdRwirWF1M\nlVIewASgGVAJ6KCUqnTLas2A8tmPXsAka9t9EIGBgSQkJPBqnz6WmWJy4RTvddsDA/ng4kWObt2a\na22Kv2mt6fTEE8w3m2HMGKPjiOtefJGX/AvT7vMvKKLiCAsDueVXOCNbHJlGAYe11ke11teAuUDz\nW9ZpDnyjLTYBBZRSxWzQ9n1TSuFx8KBlQvBcLKZtunUjCSh34UKutSn+dmHxYg7ExXH2qaegVCmj\n44hs0bMVx67OYhUejOEdYmOhVy8pqML52KKYlgBO3PT6ZPay+10HAKVUL6VUjFIqJjEx0Qbx/vbO\nO+8watQo+OUXy4IGDWy6/bvxqVMHX4DNm3OtTZEtK4tCI0awJSSEnnPmGJ1G3GTYMNh3rQmLGUpX\nvuFxVpCWZlkuhDNxuA5IWuupWutIrXVkUFCQTbd9+PBhjhw5AsuWWe4vDQmx6fbvKn9+lpQsSd/p\n03OvTQHAn8OGkbZzJ6ZPP8XDz8/oOOImcXGWr6MYxhsU5AzN8SDtxnIhnIUtiukp4ObzZiWzl93v\nOnY3Z84cZowfb7le2qxZbjfPweBgfj5xguTLl3O9bXcUHQ2VSp6kySejaWsqQvSV1kZHEre4/vfs\nFfKxjn48xVX6MC5X/84VwhZsUUy3AOWVUqWVUl5Ae2DRLessArpk9+qtDSRprY0Z6XrNGrh2zZBi\n+vqLL3LMbMY/KSnX23Y30dGWa29dT/2XBWiOmyfS6xUl1+IczKhR4ONjeR7DezSjGaP4iE8HyxzA\nwrlYXUy11plAP2AFsB/4Xmu9VynVWynVO3u1ZcBR4DDwFdDX2nbv14QJE3jmmWfIWLIE/PzsOh7v\nnXjUrm15smVLrrftboYNg+C0o7zOFxynK/toLdfiHFCnTjB1KoSGWjoHjin+BbGmdE5/28LoaELc\nF6W1NjrDHUVGRuoYGw10MHHiRH7++WcW794N1arBggU22e59uXKFL3x92RYRwTd79+Z++27EZIKG\nuixPcoIJHON0dn83pWR8AEc3qGZNvomJ4dCKFRR86imj4zgdpdRWrXWk0TncjcN1QLKXvn37snj0\naIiNNeQULwDe3qQWL87FhAQc+Y8YV9A8aC0FOMrP1L5RSCF3+5yJB/PuggUcCA6m4PDh8pePcBpu\nUUxvFK5lyyxfjSqmwLBWrVicno7KyjIsg8vTmikFhjFeFWMrS28s9vGxXKMTji2gRAkKjx4NW7aQ\nOGWK0XGEyBG3KKZz586lbNmyxP30E1SqZOzhyaOPQno6yODedrNv0iSy/vqduG7/R5FQf5SyXJOb\nOtVyjU44gc6dGRAURK3+/bly8aLRaYS4J7copkWKFCGyWjWKb94MTz9tbJjatekBdOjVy9gcrsps\n5pXBg3kyTx6iJr/E8eOWM4XHj0shdSomEy3efpv+WVmY/vtfo9MIcU9uUUwbNmzId61b45mRYXwx\nDQ2ltJ8f5a5dMzaHq5ozh6/T0pg0dCjKy8voNMIKT77+OgNbtsRrzBhIkFtlhGNz+d68mZmZZFy7\nRr769SEpCfbvt3T1NFLLlrBnDxw6ZGwOV3PtGlSsCAEBsHWr8f/PwnqHDrG8YkV21ajBkD//NDqN\nU5DevMZw+U+bzZs3ExAQwK9btsCAAY7xAVu7NvrwYa6cPGl0EpcS3bs33Y8eJXXECMf4fxbWK1+e\nxZUr8/XmzVzdscPoNELckct/4hQpUoTB5ctTOSAAunQxOo7Fo49SDejXo4fRSVzH1aucnj+fg/7+\n+DS/ddIi4cw+XriQnf7+5H33XaOjCHFHLltMo6MhLAwalM/LyAOHOFe3l2XkI0dQowadlaKRr6/R\nSVzH9OkMTkpi/Q8/oOSo1KX4h4WRZ+hQMhcuJGHRrSOVCuEYXPJT5/q4rLGx0JaRaMy0WtPPccZl\n9fVlcPXqtJcB723ialISu957D+rWxdS4sdFxhD28/jqNvbx44cUX0TKQg3BALllMhw2DtDTw5i8+\nYxq9qcrBK6GONS5r7dpc/vNPzp05Y3QSpzetZ0+qnj3Lri5dLOMFCtfj60ufLl0YePkyLF5sdBoh\n/sUli+n1uRAbs4n/ATt46x/LHcHVGjUokpLCpw5V4Z3QlSt0XL+eKeXK8ZBcg3ZpL0ycSMsKFVBv\nvw0ygphwMC5ZTK8PcLSYLnzKLrbR9h/LHUHeJ55gPNA6ONjoKM5t2jQKJiTQa8oUuVbq6vLkIfO9\n95i4bx8/DRpkdBoh/sElP31uniNxDw8ByvHGZS1bll7BwUTGxhqdxGmlXrhAuyFD2FmtGjz5pNFx\nRC4wtWnDtHz5+GnGDMt9xUI4CJcspv+cI9FBx2VViqz69dm+fDknT5wwOo1T2j92LL+lp5Mq10rd\nhsnTk1X/+x8zU1Jgxgyj4whxg8uPgOTIEj//nCKDBvHRwIG8+dlnRsdxLhkZUKECVwoXxnvzZimm\n7kRrqFuXy0ePkmfvXvIFBhqdyKHICEjGcMkjU2cR1KIFC4CuhQoZHcXpxI4fjz5+HO8RI6SQuhul\nOPOf/1AmIYEvHWUgFuH2pJgaqXRpmpcuTbGtW41O4lRSkpKIHDqUN4KC4NlnjY4jDBDcsiX9ypSh\n4YYNkJxsdBwhpJga7XLdusxZsYK4Y8eMjuI0vBYt4sOsLDq+8YYclbqxd2fP5pGkJPjyS6OjCCHF\n1GjnHnmEjmlpLJ061egozsFsxmv0aHpWrEjk4MFGpxFGqlWL840aMXzkSC450k3kwi1JMTVY6Xbt\n2Ab0DAgwOopTmDVoEN/t2YN++22ZGUYQ160bH1+5wsohQ4yOItyc9OZ1BFWqQIkSsGKF0Ukcm9Y0\nyJ8fz8xMfrl8GTw9jU4kHMDJp56i5JYtcOwYFChgdBzDSW9eY1j1p71SKlAptVIpdSj7a8E7rDdD\nKXVWKbXHmvZcVVzNmry3Zg2njx83Oopj++UXViUnM3vUKCmk4oaSn3wCly6ROmaM0VGEG7P2PNmb\nwGqtdXlgdfbr2/kaaGplWy7rl4yHeS8zk6dKzyUsDMeZ3caBXLt2jWsjR2IqWZLCr75qdBzhSKpX\n5+tHHiHko484f/So0WmEm7K2mDYHZmY/nwm0uN1KWut1wAUr23JJ0dHwzo9dOIeiHanExlqmj5OC\n+k+T//MfKmzcSGKfPuDlZXQc4WBqvv02bbUma/Jko6MIN2XVNVOl1CWtdYHs5wq4eP31bdYNA5Zo\nravkdPvucM00LMwy7+pv1KMAl6jGTsAyBKKc9f3bmho1WLB/P+PPn4d8+YyOIxxRixawdq3lFyd/\nfqPTGMZe10y3bt1axNPTcxpQBffsvGoG9mRmZr5co0aNs7e+ec8LT0qpVUDR27z1j7nDtNZaKWV1\nbyalVC+gF0CII03zYifXe/RPoyaZfEpJ/uQktRxqujjDbdpEg23baDB6tBRScWcjRnCgRg329+1L\nSzm1Y3Oenp7TihYtWjEoKOiiyWRy3J6rdmI2m1ViYmKlhISEacDzt75/z78utNaNtNZVbvNYCJxR\nShUDyP76r2p9v7TWU7XWkVrryKCgIGs35/Cu/73wK3VZCzzK7H8sd3fXrl1jQo8epAYGQp8+RscR\njuyRRxhetCj95swh44JcVbKDKkFBQZfdsZACmEwmHRQUlITlyPzf71u5/UVA1+znXYGFVm7P7Vyf\nLu4UzVlOZfqy0/GmizPQigkT6LdvH+uffRb8/IyOIxzcp1OmsENr8sggKPZgctdCel32/t+2blpb\nTD8GGiulDgGNsl+jlCqulFp2fSWl1BzgDyBcKXVSKdXDynZdxt/TxSkW0ZLHWc/Xn553rOniDPTc\n+vXE+PnRZNw4o6MIJxD6/PMENWsGY8eiU1KMjiPciFXFVGt9XmvdUGtdPvt08IXs5ae11k/ftF4H\nrXUxrXUerXVJrfV0a4O7kk6dLH0m6k4uSU3M1LnyrdGRHILetQt++okagwah5GZ8kUNXhgzhufPn\nGdO+vdFRhI1Vr1494l7rjBw5skhycnKud5Byxx5ZDqtAVBQFvLy4uGSJ0VEMl5mZSa369fkqb14Y\nMMDoOMKJeNevj39wMPnWrYMrV4yOI2xo+/btB+61zpQpU4JTUlLuq7ZlZmY+eKhsUkwdSLXq1Vnz\n8stU3rgR0tKMjmOoy5s2EXrxIoWffRZk8mdxn2bPmUP/5GSGl5iByYQMhuIifHx8qgMsWbLEPyoq\nKrxp06ZlSpcuXfn5558vbTab+eCDD4qcPXs2T7169SrUqlWrAsD8+fMDqlWrFlGpUqWKzZo1K5OU\nlGQCKFGixEN9+vQpUalSpYozZswoGBUVFf7SSy+VioiIqFS+fPnKv/76q8/9ZJMx2RxNixZkTZyI\nx6pV8Py/el+7jcAvvmBeQIDlgrIQ9yn6VH1Kq0d5+MJ7eNCF2Fg/evWyvCf9EWyge/dS7NlzX8Xm\nnqpUSWPGjBM5XX3//v35duzYcTQsLCyjRo0aEStXrvQbPnz42UmTJgWvXbv2r2LFimXGx8d7fvjh\nh8XWrVv3V0BAgHnYsGFF33///eCxY8fGAxQqVChz3759+wGmTZtWJD093XTgwIF9P//8s1+vXr1K\nHzp0aG9O88iRqYOZf/EiBYHTc+YYHcUw27/7jvgff7Sc3pWjUvEAhg1XDNKtaMdZavMaYDnZM2zY\nPb5ROI2HHnootWzZshkeHh5Urlw57ciRI/8aGu23337zPXLkiHdUVFREREREpblz5xaKi4u7sV6X\nLl0u3rx+x44dLwA0a9YsJSUlxXTu3DmPnOaRI1MHE165Mp3KlSPjl18gKws8cvx/6RK01vR45RW0\nycS2119Hpv4WDyIuDmIZxPtMoAW/UY1MsvCUwVBs5T6OIO0lb968N27T8fDwIDMz818fF1pr6tat\ne3nx4sXHbrcNf39/882vLQP53fn13ciRqYOpXLkykz74gNALF+CPP4yOk+vUzp3MS0piYrduKDkq\nFQ/IMuiJiW18ShWO0YYfblouXJmvr2/W9eui9evXT42JifHbs2dPXoDLly+bdu3alfdO3ztnzpyC\nACtWrPDUwSUrAAAeSklEQVTz9/fPKlSoUFZO25Vi6oiaNeOEpyd6wQKjk+S+d96hbIECPPrpp0Yn\nEU7s+mAoC2jBdEqSTl/yeZtlMBQ30LVr13NNmzatUKtWrQrFixfPnDJlyvH27duXqVChQqXIyMiI\n3bt3e9/pe729vXXFihUr9evXL3TKlCnH76ddmRzcAUVHR9O5c2f2h4YScewY3MepBmcUHW25luUT\nO5ZaDKZ7i3d4/KeRRscSTu76z1XJ2O6c4X981GcWbSa6fu8jew10v3PnzuNVq1Y9Z+vtOoqoqKjw\nsWPHnnjiiSfueivFzp07C1etWjXs1uVyZOqA6tWrx+dt2hAYGwsH7nlblVOLjrZMORcbq2nEZP7A\ng5bLB8ltDMJq1wdD+S31Sw6UKEGbPVOMjiRcmBRTB1SyZEle/+ILigAsdO3hjocNs/SybMIKxnOE\nJ/mM81cKSK9LYTOePj54DB5M5vr1nFu61Og4wkFt3rz54L2OSu9GiqmDSg8MZGX58lydP9/oKHZl\n6V2ZxSsM4iilmU7vm5YLYRu6Rw8e8/CgVw8ZFlzYhxRTB7Vq1SqeOnSIjVu2QHy80XHsJiQEHmcE\nrdhPT1qTgdeN5ULYivLzo/dzz/HimTOwN8f34QuRY1JMHdSTTz7JsokTqQWweLHRcezmw/ezGMM8\nXqcIv/IegExBJ+yi+7RptPTxgbFjjY4iXJAUUwfl5+dHs9698SlbFn76yeg4dtOR2dTiEFcKTwDl\nQ2ioZQRBGfJN2FyhQqR16cKEb74hYft2o9MIFyPF1IGdTUzky9BQzq1aBUlJRsexOX3tGgP79+fP\ncuWYdKYVZrOl96UUUmEvp9q2pb/ZzI+DBxsdRTiYJUuW+K9cudL3Qb9fiqkDi4uLo/+aNazNzAQX\n7IV4ctw4opOS2NG4MZjkR1HYX/kGDdjbrBl9N2+Gy5eNjiMcyJo1a/zXr1/v96DfL59gDqx69eoc\n/usvWhct6nqneq9do9SECRx75BFe+vxzo9MIN1Lx/fdRycnoKXLfqTOaOHFi4EMPPVQxIiKiUseO\nHUP/+usvr9DQ0Crx8fGeWVlZ1KhRI3z+/PkBAI0aNSpbuXLliuXKlas8duzYwte38cMPPwRUqlSp\nYnh4eKVHH320wsGDB72++eaboMmTJwdHRERUWr58+X0XVSmmDszDw4Oy5ctDixbw88+Qnm50JJs5\nP348OjYW31Gj8Mp7x6EyhbC9GjX4pmJFIocNI9PN5w22RlRUVPj48eMLAVy9elVFRUWFT5w4MRAg\nOTnZFBUVFf7VV18VBDh//rxHVFRU+MyZMwsAxMfHe0ZFRYXPnj07P0BcXFyOJl3Ztm2b9w8//BAY\nExNz4MCBA/tMJpP+5Zdf/AcMGJDQvXv3kHfffTc4PDz8SqtWrS4DREdHH9+7d+/+HTt27JsyZUpw\nQkKCx+nTpz379esXNn/+/CMHDx7ct2DBgiPh4eHXunTpkti7d+8zBw4c2Ne0adOU+/33kGLq4BIT\nE+l36hQbU1Nh5Uqj49iEvnKFpm+/TftChaBJE6PjCDcU+MILlMzI4ML//md0FHEfli9f7r9nzx6f\nqlWrVoyIiKi0YcOGgKNHj+YdNGjQueTkZI+vv/46aMKECTdmtPnkk0+Cw8PDK9WoUaNiQkJCnr17\n93r/9ttvvlFRUckRERHXAIKDg3M8mP3dyBRsDs7Hx4e5GzdSLV8+HvvpJ5eYMNw8fTqvZGRQoE8f\nlx93WDimZ0eM4Nnvv4fp06FvX/k5fACbN28+eP153rx59c2v/f39zTe/LlSoUNbNr4sVK5Z58+uQ\nkJDMnLSptVZt27Y9P2HChFM3L09OTjYlJCR4AVy+fNmjYMGC5iVLlvivXbvWPyYm5oC/v785Kioq\nPD093W4HkHJk6uB8fX1JSEjg5datYdEiyMzRz5zjunoVj48/5uXHHqPNSBnMXhjEZIJBgzi7fTtH\nZ882Oo3IoaZNm15esmRJwVOnTnkCnDlzxuOvv/7y6tevX4k2bdqcf/vtt09369YtFODSpUse+fPn\nz/L39zdv377de+fOnb5gmZZt8+bN/gcOHPC6vg0Af3//rOTk5AeeQFqKqRPw9PSEli3hwgVYt87o\nOFb5+Y03mHvyJOYRI+RoQBgqq0MHaphMDHzjDaOjiByqUaPGleHDh59q2LBhhQoVKlRq0KBBhUOH\nDnnt2LHD94MPPkjo06fPhTx58uhx48YVat26dVJmZqYqU6ZM5cGDB5eoWrVqKkDx4sUzx48ff7xl\ny5blwsPDK7Vs2bIMQOvWrS8tXbq0wIN2QLJqCjalVCDwHRAGHAde0FpfvGWdUsA3QDCggala63E5\n2b67TsF2q8zMTJ57+mme+PVX3urVCyZMMDrSg7l6lVYFCnDIZGLn5cuYPB74j0AhbGJxx46UnzOH\niP37ISLC6Dg2IVOw2Ze9pmB7E1ittS4PrM5+fatM4A2tdSWgNvCqUqqSle26FU9PTwoHB+NfpQr8\n+CNk2eR6ee6bMYN5V67w87RpUkiFQ3juiy+IyJsX5PYsYSVri2lzYGb285lAi1tX0FrHa623ZT9P\nBvYDJaxs1+18++239Bs2DM6cgbVrjY5z365dvszVDz7Ao04dSrZvb3QcISyKFOFEq1b0mz6d83/9\nZXQa4cSsLabBWuvrU5okYDmVe0dKqTCgOvCnle26Jd2sGRd9fOC774yOct8mdO1K+OnTJL7xhlwr\nFQ4lqV07pmVl8cf77xsdRTixexZTpdQqpdSe2zya37yetlx8veMFWKWUH/Aj8LrW+o7jeCmleiml\nYpRSMYmJifexK67v6TZteN7Hx3Kq15l69aalUX3tWlqWLElQy5ZGpxHiH6o0b07Ck0/y7OrVcO2a\n0XGEk7pnMdVaN9JaV7nNYyFwRilVDCD769nbbUMplQdLIY3WWt91tmut9VStdaTWOjIoKOj+98iF\ndevWjR4dOqDPn4c1a4yOk3OTJ1P/4kU+l1sQhIMqMHgwxMdzbc4co6MIJ2Xtad5FQNfs512Bhbeu\noJRSwHRgv9b6Myvbc2vt2rWj2+jRqIAApznVe/rQIUaPGEF6gwbw+ONGxxHi9po04Y2CBXn81VfR\nZrPRaYQTsraYfgw0VkodAhplv0YpVVwptSx7nTrAi0ADpdSO7MfTVrbrtlKzsvipenX0jz86xSmp\nBf/5D8NTU4nv29foKELcmclE9eefp3FqKpnr1xudRlhp9OjRQV9++WWhW5cfPHjQq3z58pXt0aZV\nwwlqrc8DDW+z/DTwdPbzDYD0OLGR2bNn02vtWrYB1Vetgqcd+O+SCxfou24dT9WvT5nWrY1OI8Rd\ndZ4wARYuhC+/hHr1jI4jHlBGRgZDhgzJ9Q43Mjavk2nbti0RZctStXVrmD3bYYup1poL//d/FEpK\notz48UbHEeLefH3RPXuyaexYQv78kxK1ahmdyKlNnkzgyJGUSEjAq2hRro0Ywanevblg7XYHDx5c\nbN68eYUKFSqUUbx48WvVq1dPW758eYEqVaqkbd682a9169YXkpOTPfz8/LJGjhx5Zv369T4vv/xy\nGED9+vXtNomtDCfoZAoUKMDjDRpg6tDB0qs3KcnoSLe1eMYMwr78km3PPAMPPWR0HCFy5MwLL/C4\n1nw5YIDRUZza5MkEDhxIaHw8XlpDfDxeAwcSOnkygdZsd+3atT6LFy8uuG/fvr2rVq06tGvXLt/r\n7127dk3t2bNn/3vvvXfm5u/p0aNH2BdffBF38ODBfda0fS9STJ1QcnIyI7Oy+P3KFYftiFRx2TK6\nmkw89MUXRkcRIseKRkaypE4dhh08CDLX6QMbOZISV678s75cuYJp5EjrBuxZu3atX7NmzS75+Pjo\nggULmhs3bnzp+nsdOnT411HvuXPnPJKTkz2aNWuWAtC9e/fz1rR/N1JMnZCXlxfjfviBDUWKgCPO\nx7h7N+V/+okvBw0iT9myRqcR4r40/egj/C5dglmzjI7itBIS8Lqf5bbg7+9vaDdsKaZOKG/evBw+\nfJihQ4bApk2wf7/RkW44dfIkfZ95hvN+fvDWW0bHEeL+1a3Lr2XL0nXIEMzOOg62wYoW5ba3Gtxp\neU7Vq1cvZcWKFfnT0tJUUlKSadWqVQXutn7hwoWz/P39s1asWOEH8PXXX1t1mvlupJg6qYIFC0Ln\nzmSaTA51dLrus8+YdeIESQMGQKDdfm6FsB+lOPnkk6xLSuLUvHlGp3FKI0ZwytubfxwpentjHjGC\nU3f6npyoV69eWtOmTZMqVapUuUGDBuXDw8PT8+fPf9e/eKZPn378tddeC4mIiKiktbbbnSVWTcFm\nbzIF291NmTKFjwYO5EBAAN4nT4KnwZ2zU1OhUiUu+vlRcOdO4/MI8YAyU1JQYWF41K0LCxYYHee+\nOMoUbPbqzZuUlGTKnz+/OTk52fToo4+GT548ObZu3bq5doH7TlOwyaedE4uIiKBRnTqkrFqF9/Ll\n8OyzhmVJT0/n4IABVIuLo+C6dVJIhVPz9POD3r0xjxpF+t69+Fa2y33+Lq13by7YonjeqnPnzqGH\nDh3Kd/XqVdW+ffvzuVlI70Y+8ZxYvXr1qPfYY1CyJEyebGgxHffOOwyfPp0DLVpQToYNFC4go0cP\nqo8aRaMuXfhi61aj44hsixcvPmZ0htuRa6bOLk8ejrRvz19LlxrXESkri96bNjHF25tykycbk0EI\nG8tTujTtKlfmCblN5jqz2Wx269Hssvf/tr2GpZg6uYyMDGpHR/OWyQSf5f48AhkZGZg/+YQCv/9O\nj4kTIfiuU9oK4VTemTSJVqmpEB1tdBRHsCcxMTG/uxZUs9msEhMT8wN7bve+dEByAStXruThWbMI\n/u47iI3N1YI2tEsXYmbN4udWrfCaN08m/hauRWvSqlZl4aVLtD9+HGVy/OMPe3VA2rp1axFPT89p\nQBXc80DMDOzJzMx8uUaNGv+ablSumbqAxo0bQ2gofPstTJgAI0fmTsMpKYSvWEGWry9eX30lhVS4\nHqX4oWZNus6YQakJE6jbv7/RiQyTXUCeNzqHo3LHvy5c0oXChWkRHMyPn39u1+s70dEQFgYeysyC\n4Fd4KTGRsUuWQMGCdmtTCCO9MHYsa/39qfPbb0ZHEQ5MiqmLyJ8/P4mFC3MxJcVugzhER0OvXhAb\nm0k1KpI3bTbven5A9EmZrkq4Lu+CBXmiTx/UwoVw4oTRcYSDkmumLkSbzah69eCvv+DgQShw15G2\n7ltYmOWS7CCGsprRlKchP7CS0FDF8eM2bUoIh6KPHePDMmXwbtiQN1atMjrOXdnrmqm4OzkydSHK\nZILx4/k5MZFDr71m8+3HxcEAvuBTRtOXF/mBFYAiLs7mTQnhUFTp0mwvVowdGzbA1atGxxEOSIqp\ni7lctiydvbz44NtvYedO221Ya/r69iaLgXxPS/oyA/AAICTEds0I4ajmzpjBt1evgozXK25DiqmL\nCQgIYOXPPzMlMJCzL/QjLFRjMllO0T7wrXJaw5AhlEyZwjz86cBEsrI7gvv4wKhRNosvhMPyfOop\nkoqFs777F9b/TgmXI8XUBT3y5JPsaPkJBf7aQPW4D9Hacq2zV68H+OW/ehX90kswdixv9uvH+1Pi\nKBVaFKUsd+NMnQqdOtllN4RwKNFzTPQ824gmGVsJ1988+O+UcEnSAclFlQ41Uz6uKJtJ5GF+ZD2t\nAEsBzHFnofh4/nzqKQbs2cOs/v0pN26c3Esq3FZYGFyIPcELlKE8z/MmPwL3+TuVC6QDkjHkyNRF\nxZ4wsYVfeYdSLKMLtfkDIOedhf74AyIj8T98mIzSpcno00cKqXBrcXGQTCkeoScDWEog528sF8Kq\nYqqUClRKrVRKHcr++q8795VS3kqpzUqpnUqpvUqp96xpU+RMSAhcojJj+ZMEijKSpwjlTUqVuus8\nunD+PKdefJGpjz0GXl5U+vNPYo4coWLFirkTXAgHdb2j3RRe4SRXqce7/1gu3Ju1R6ZvAqu11uWB\n1dmvb3UVaKC1rgpUA5oqpWpb2a64h1GjLJ2DEihGQ1bzOVnk4xNWlegMhw//+xuOHUOPGQPh4XwV\nHU1/Dw9OLVsGDz+MkiNSIW78Tu2iKi/gx3a+Ip+3WTrgCcD6YtocmJn9fCbQ4tYVtEVK9ss82Q/H\nvVDrIjp1snQOCg2FEyqUkyVOMKlxP8rvXERWeDghefMyqUoV6NiRwxERhJUpw/whQ6ByZQZt2MCB\nQ4coIUejQtxw8+9UeQbwO1f58e1t0gFPAFZ2QFJKXdJaF8h+roCL11/fsp4HsBUoB0zQWg/Nyfal\nA5IdJCRw+YMPGDh/Pq19fHjabOZKiRK8lJJC36FDebx9e6MTCuH4Ll2CYsWgWzeYNMnoNP8gHZCM\ncc9iqpRaBRS9zVvDgJk3F0+l1EWt9R1HPFdKFQB+AvprrW8/J5xSvYBeACEhITViY2PvuRNCCJHb\nDrdowfClS/lk925CIyKMjnODFFNj3PM0r9a6kda6ym0eC4EzSqliANlf/zXH2y3bugT8CjS9yzpT\ntdaRWuvIoKCg+9sbIYTIJV4dOrA6M5O906YZHUU4AGuvmS4CumY/7wosvHUFpVRQ9hEpSql8QGPg\ngJXtCiGEoUJeeIHTZcrw9NatRkcRDsDaYvox0FgpdQholP0apVRxpdSy7HWKAb8qpXYBW4CVWusl\nVrYrhBDGUoo8PXrAb7+Ruue2V62EG5ERkIQQ4kHFxtIxLIzTYWH8duyY0WkAuWZqFE+jAwghhNMK\nDeWpChW4lJRkmU/YJIPKuSv5nxdCCCt0GziQ18+cQe3aZXQUYSAppkIIYY22bcny8GDp+++Tmppq\ndBphECmmQghhjUKF2FSrFs/On88P339vdBphECmmQghhpcdefZWlQEcZ9d5tSTEVQggrqebNedrX\nlzzffWd0FGEQKaZCCGEtX19o0YIZs2bx8QcfGJ1GGECKqRBC2ELHjmxIT2fZnDk48v37wj7kPlMh\nhLCFRo2YULAg3lWryhzAbkiOTIUQwha8vMjXpg1q0SKuXbokR6duRoqpEELYSvv2/JmaSqmwMDZv\n3mx0GpGL5DSvEELYSr16VAwKop6XF3nz5jU6jchFcmQqhBC24uFBQPv2fH/+PNXKlDE6jchFUkyF\nEMKW2reHK1c4Fx3N9u3bjU4jcomc5hVCCFuqXRtCQnj+7bdJDQlhx44d0rvXDciRqRBC2JLJBC+8\nwJjkZGZPnCiF1E1IMRVCCFvr0IE6WVlU3rvX6CQil0gxFUIIW6teHSIiOD59OgMHDiQpKcnoRMLO\npJgKIYStKQUdO3J+82YmTpzIpk2bjE4k7EyKqRBC2EPHjtQA4t96iyZNmhidRtiZFFMhhLCHsmWh\nVi0CFywAkOEFXZwUUyGEsJdOnWDnTjo9+yx9+/Y1Oo2wIymmQghhLy+8AB4elLpwgeLFixudRtiR\nVYM2KKUCge+AMOA48ILW+uId1vUAYoBTWutnrWlXCCGcQnAwNGrExwcPwvDhRqcRdmTtkembwGqt\ndXlgdfbrOxkA7LeyPSGEcC6dOsHx4+h169i9e7fRaYSdWFtMmwMzs5/PBFrcbiWlVEngGWCale0J\nIYRzadUK/P2ZNXw4Dz/8sIzX66KsLabBWuv47OcJQPAd1vsCGAKY77VBpVQvpVSMUiomMTHRynhC\nCGEwX1/o0IFnY2KY+OmnlC1b1uhEwg7uWUyVUquUUntu82h+83ra0u/7X32/lVLPAme11ltzEkhr\nPVVrHam1jgwKCsrpfgghhOPq0YOCV67Qx9eXgIAAo9MIO7hnByStdaM7vaeUOqOUKqa1jldKFQPO\n3ma1OsDzSqmnAW8gQCk1S2vd+YFTCyGEM6lZE6pUwTxtGj8VLkxAQACNGzc2OpWwIWtP8y4CumY/\n7wosvHUFrfVbWuuSWuswoD2wRgqpEMKtKAU9eqBiYnj7jTeYNGmS0YmEjVlbTD8GGiulDgGNsl+j\nlCqulFpmbTghhHAZnTuj8uRhRYMGzJs3z+g0wsaUIw9xFRkZqWNiYoyOIYQQttGuHaxaBadOgbe3\nXZpQSm3VWkfaZePijmQEJCGEyC09e8KFCywZOpTHH3+cq1evGp1I2IgUUyGEyC0NG0LVquT56Sey\nsrKIj4+/9/cIpyDFVAghcotSMGQITU6cYONbbxEWFmZ0ImEjUkyFECI3tW0LoaEwZgzp6ekkJSUZ\nnUjYgBRTIYTITXnywKBBpKxfT6lixRg7dqzRiYQNSDEVQojc1qMHfoGBDClZkqZNmxqdRtiAFFMh\nhMhtvr7w6qsM2bePOv7+RqcRNiDFVAghjDBgABQqxKmXXmLed98ZnUZYSYqpEEIYoVAh+PRTxm3b\nRqdOnbh48aLRiYQVpJgKIYRRXnyRDhXq8mdWPqoEphMWBtHRRocSD0KKqRBCGCR6tqJr3AwqksHn\nvE5sLPTqJQXVGUkxFUIIgwwbBruvlGc4QyjLPJ5kKWlpluXCudxzPlMhhBD2ERdn+Tqe4aQTzAYa\n/2O5cB5yZCqEEAYJCbF8zcCLibxKBl7/WC6chxRTIYQwyKhR4OPzz2U+PpblwrlIMRVCCIN06gRT\np1qG6lXK8nXqVMty4VzkmqkQQhioUycpnq5AjkyFEEIIK0kxFUIIIawkxVQIIYSwkhRTIYQQwkpS\nTIUQQggrWdWbVykVCHwHhAHHgRe01v+a+kApdRxIBrKATK11pDXtCiGEEI7E2iPTN4HVWuvywOrs\n13fypNa6mhRSIYQQrsbaYtocmJn9fCbQwsrtCSGEEE7H2mIarLWOz36eAATfYT0NrFJKbVVK9brb\nBpVSvZRSMUqpmMTERCvjCSGEEPantNZ3X0GpVUDR27w1DJiptS5w07oXtdYFb7ONElrrU0qpIsBK\noL/Wet09wymVCMTea717KAycs3IbzkT217XJ/rouW+1rqNY6yAbbEffhnh2QtNaN7vSeUuqMUqqY\n1jpeKVUMOHuHbZzK/npWKfUTEAXcs5ja4gdCKRXjTtdpZX9dm+yv63KnfXVF1p7mXQR0zX7eFVh4\n6wpKKV+llP/158BTwB4r2xVCCCEchrXF9GOgsVLqENAo+zVKqeJKqWXZ6wQDG5RSO4HNwFKt9XIr\n2xVCCCEchlX3mWqtzwMNb7P8NPB09vOjQFVr2rHSVAPbNoLsr2uT/XVd7rSvLueeHZCEEEIIcXcy\nnKAQQghhJZcppkqppkqpg0qpw0qpf43EpCzGZ7+/Syn1iBE5bSUH+9spez93K6U2KqWMPNVutXvt\n703r1VRKZSql2uRmPlvLyf4qpeorpXYopfYqpdbmdkZbycHPcn6l1GKl1M7sfX3JiJy2opSaoZQ6\nq5S6bUdMV/uschtaa6d/AB7AEaAM4AXsBCrdss7TwM+AAmoDfxqd2877+xhQMPt5M1ff35vWWwMs\nA9oYndvO/78FgH1ASPbrIkbntuO+vg18kv08CLgAeBmd3Yp9fgJ4BNhzh/dd5rPKnR6ucmQaBRzW\nWh/VWl8D5mIZ6vBmzYFvtMUmoED2vbHO6J77q7XeqP+edGATUDKXM9pSTv5/AfoDP3KH+52dSE72\ntyMwX2sdB5Z7uHM5o63kZF814K+UUoAflmKambsxbUdbBqy5cJdVXOmzym24SjEtAZy46fXJ7GX3\nu46zuN996YHlL11ndc/9VUqVAFoCk3Ixl73k5P+3AlBQKfVb9jCdXXItnW3lZF+/BCoCp4HdwACt\ntTl34hnClT6r3IZVt8YIx6eUehJLMa1rdBY7+wIYqrU2Ww5gXJ4nUAPLrWn5gD+UUpu01n8ZG8su\nmgA7gAZAWWClUmq91vqysbGE+JurFNNTQKmbXpfMXna/6ziLHO2LUuphYBrQTFvuCXZWOdnfSGBu\ndiEtDDytlMrUWi/InYg2lZP9PQmc11qnAqlKqXVY7ud2tmKak319CfhYa62Bw0qpY0AElkFgXJEr\nfVa5DVc5zbsFKK+UKq2U8gLaYxnq8GaLgC7ZPeVqA0n67xlvnM0991cpFQLMB150gaOVe+6v1rq0\n1jpMax0G/AD0ddJCCjn7eV4I1FVKeSqlfIBawP5czmkLOdnXOLIHh1FKBQPhwNFcTZm7XOmzym24\nxJGp1jpTKdUPWIGld+AMrfVepVTv7PcnY+nh+TRwGEjD8teuU8rh/o4ACgETs4/WMrWTDqKdw/11\nGTnZX631fqXUcmAXYAamaa2dbszrHP7fvg98rZTajaWH61CttdPOJKOUmgPUBworpU4C/wfkAdf7\nrHInMgKSEEIIYSVXOc0rhBBCGEaKqRBCCGElKaZCCCGElaSYCiGEEFaSYiqEEEJYSYqpEEIIYSUp\npkIIIYSVpJgKIYQQVvp/idfn/FF1pPYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8b8ae3a400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def f(x):\n",
    "    a = np.sin(2 / (x + 0.2)) * (x + 0.1)\n",
    "    #a = a * np.cos(5 * x)\n",
    "    a = a * np.cos(2 * x)\n",
    "    return a\n",
    "\n",
    "known = f(coarse) # only use coarse\n",
    "interp = W.dot(known)\n",
    "\n",
    "plt.scatter(coarse, known, color='blue', label='grid')\n",
    "plt.plot(fine, interp, color='red', label='interp')\n",
    "plt.plot(fine, f(fine), color='black', label='exact', ls=':')\n",
    "plt.legend(bbox_to_anchor=(1.05, 0.6), loc=2)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Array Creation and Initialization\n",
    "\n",
    "[doc](https://docs.scipy.org/doc/numpy-dev/reference/routines.array-creation.html)\n",
    "\n",
    "If unspecified, default dtype is usually float, with an exception for arange."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "np.linspace(4, 8, 2) (2,)\n",
      "[4. 8.]\n",
      "\n",
      "np.arange(4, 8, 2) (2,)\n",
      "[4 6]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "display('np.linspace(4, 8, 2)')\n",
    "display('np.arange(4, 8, 2)') # GOTCHA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD8CAYAAACcjGjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt4XPV95/H3VxfLkm+yLFk2vmBjjI3lcBXGQEO4xJeG\nNGa3CetsU9yUB9KFbZOUNglp2jzblD5pd3Ojz8LWzY00F2JIGrwkWDYmxOmmGOyAsUbCF3w3Hl18\nkWzJljSa7/4xR2Ys2VjSSHNmRp/X88wzZ37zOzPf4wPno/M7M/Mzd0dERCRZXtgFiIhI5lE4iIhI\nHwoHERHpQ+EgIiJ9KBxERKQPhYOIiPShcBARkT4UDiIi0ofCQURE+igIu4DBKi8v91mzZoVdhohI\nVtm6dWuzu1dcrF/WhsOsWbPYsmVL2GWIiGQVM9vfn34aVhIRkT4UDiIi0ofCQURE+lA4iIhIHwoH\nERHp46LhYGbfNrNGM6tNaiszsw1mtiu4n5j03CNmttvMdpjZsqT2681se/DcY2ZmQXuRmf04aN9s\nZrOGdhNFRGSg+nPm8F1gea+2zwEb3X0usDF4jJktAFYCVcE6j5tZfrDOE8D9wNzg1vOa9wHH3f1y\n4GvAPwx2Y0REZGhcNBzcfRNwrFfzCuDJYPlJ4O6k9qfcvcPd9wK7gUVmNhUY7+4ve2Je0u/1Wqfn\ntZ4B7uw5qxARkXfE486jP6+j9nDLsL/XYK85VLr7kWA5ClQGy9OAg0n9DgVt04Ll3u3nrOPuMaAF\nmHS+NzWzB8xsi5ltaWpqGmTpIiLZaduhE/zLr/eyq/HksL9XyhekgzMBH4Ja+vNeq9292t2rKyou\n+u1vEZGcUhNpoCDPuGN+5cU7p2iw4dAQDBUR3DcG7YeBGUn9pgdth4Pl3u3nrGNmBcAE4Ogg6xIR\nyUnuzrraI9w0ZxITiguH/f0GGw5rgVXB8irg2aT2lcEnkGaTuPD8SjAE1Wpmi4PrCff2WqfntT4M\nvBicjYiISGBnwyn2HW1n+cIpaXm/i/7wnpn9CLgNKDezQ8AXgS8Da8zsPmA/cA+Au0fMbA1QB8SA\nh9y9O3ipB0l88qkYeD64AXwL+Fcz203iwvfKIdkyEZEcUhOJYgZLFgz/kBL0Ixzc/aMXeOrOC/R/\nFHj0PO1bgIXnaT8DfORidYiIjGTraqNcP3Mik8eNTsv76RvSIiIZ7uCxduqOtKZtSAkUDiIiGa8m\nEgVgWZXCQUREAutqoyyYOp4ZZSVpe0+Fg4hIBms8eYatB46ndUgJFA4iIhltQ10D7ukdUgKFg4hI\nRltXG2V2+RiuqByb1vdVOIiIZKiW9i7+462jLKuaQrp/j1ThICKSoV7c0UAs7iyrSs8X35IpHERE\nMtS62ihTxo/m6umlaX9vhYOISAY63dnNr3Y2sayqkry89E9xo3AQEclAv9rZxJmueNo/pdRD4SAi\nkoFqIlFKSwpZNLsslPdXOIiIZJjOWJyN9Q0subKSgvxwDtMKBxGRDPPynqO0nomFNqQECgcRkYyz\nLhKlZFQ+vzO3PLQaFA4iIhmkO+6sjzRw+7zJjC7MD60OhYOISAZ57cBxmk91sCzNP7TXm8JBRCSD\nrKuNMio/j9vnVYRah8JBRCRDuDs1dVFuuXwS40YXhlqLwkFEJEPUHWnl4LHTaZ+74XwUDiIiGaKm\nNkqewfuvTP8P7fWmcBARyRA1kQZumFXGpLFFYZeicBARyQR7mk6xo+FkRgwpgcJBRCQj1EQagPRP\nB3ohCgcRkQxQE4ly1fQJXFJaHHYpgMJBRCR0R1pO8/rBExlz1gAKBxGR0K0PhpQy5XoDKBxEREJX\nE4ly+eSxzKkYG3YpZykcRERCdKytk817j7E8g4aUQOEgIhKqF+ob6I57Rg0pQYrhYGafNrOImdWa\n2Y/MbLSZlZnZBjPbFdxPTOr/iJntNrMdZrYsqf16M9sePPeYmaV/Nm0RkRCsj0SZVlpM1SXjwy7l\nHIMOBzObBvwZUO3uC4F8YCXwOWCju88FNgaPMbMFwfNVwHLgcTPr+bHyJ4D7gbnBbflg6xIRyRan\nOmJs2tXMsqopZNrfxKkOKxUAxWZWAJQAbwMrgCeD558E7g6WVwBPuXuHu+8FdgOLzGwqMN7dX3Z3\nB76XtI6ISM56aUcjnbE4y6rC/y2l3gYdDu5+GPhfwAHgCNDi7uuBSnc/EnSLAj1bPQ04mPQSh4K2\nacFy73YRkZxWE2lg0phRVM8qC7uUPlIZVppI4mxgNnAJMMbMPpbcJzgT8JQqPPc9HzCzLWa2pamp\naaheVkQk7c50dfNifQNLqyrJz8usISVIbVjp/cBed29y9y7gp8DNQEMwVERw3xj0PwzMSFp/etB2\nOFju3d6Hu69292p3r66oCHeWJBGRVPzmrWbaOrtZmmEfYe2RSjgcABabWUnw6aI7gXpgLbAq6LMK\neDZYXgusNLMiM5tN4sLzK8EQVKuZLQ5e596kdUREclJNbQPjigq4ec6ksEs5r4LBrujum83sGeC3\nQAx4DVgNjAXWmNl9wH7gnqB/xMzWAHVB/4fcvTt4uQeB7wLFwPPBTUQkJ8W642yob+COKydTVJB/\n8RVCMOhwAHD3LwJf7NXcQeIs4nz9HwUePU/7FmBhKrWIiGSLV/cd51hbZ0b90F5v+oa0iEia1USi\nFBXk8b4rMvfaqcJBRCSN3J2aSJRbr6hgTFFKgzfDSuEgIpJGbxxq4UjLmYweUgKFg4hIWtVEouTn\nGe+/cnLYpbwrhYOISJq4O+tqo9x02SRKS0aFXc67UjiIiKTJ7sZT7Gluy8jfUupN4SAikiY1kShA\nxn4rOpnCQUQkTdZFolw7s5TK8aPDLuWiFA4iImlw8Fg7tYdbM2460AtROIiIpMH6ugaAjP8Iaw+F\ng4hIGtTURpk/ZRyzyseEXUq/KBxERIZZ08kOXt1/LGvOGkDhICIy7F6ob8Adli9UOIiISGBdbZSZ\nZSXMnzIu7FL6TeEgIjKMWs908Zu3mlm+cAqJ+cyyg8JBRGQY/fLNRrq6PauuN4DCQURkWK2rjTJ5\nXBHXzigNu5QBUTiIiAyTM13dvLSjiaVVleTlZc+QEigcRESGzaadTZzu6mZ51dSwSxkwhYOIyDBZ\nF4kyobiQGy8rC7uUAVM4iIgMg67uOC/UNXDnlZMpzM++Q232VSwikgU27zlG65lY1vzQXm8KBxGR\nYbAucoTiwnxuvaIi7FIGReEgIjLE4nFnfaSB2+ZVMLowP+xyBkXhICIyxF47eILGkx1Z98W3ZAoH\nEZEhVhOJUphv3D5/ctilDJrCQURkCLk7NZEoN88pZ0JxYdjlDJrCQURkCL0ZPcn+o+1ZPaQECgcR\nkSG1rjaKGSxZUBl2KSlROIiIDKGaSJQbLi2jYlxR2KWkROEgIjJE9jW38Wb0JEursvusAVIMBzMr\nNbNnzOxNM6s3s5vMrMzMNpjZruB+YlL/R8xst5ntMLNlSe3Xm9n24LnHLJtmxBARCdREogBZf70B\nUj9z+Aawzt3nA1cD9cDngI3uPhfYGDzGzBYAK4EqYDnwuJn1fDvkCeB+YG5wW55iXSIiaVcTibJw\n2nhmlJWEXUrKBh0OZjYBuBX4FoC7d7r7CWAF8GTQ7Ung7mB5BfCUu3e4+15gN7DIzKYC4939ZXd3\n4HtJ64iIZIWG1jP89sAJli3I/rMGSO3MYTbQBHzHzF4zs2+a2Rig0t2PBH2iQM/g2zTgYNL6h4K2\nacFy7/Y+zOwBM9tiZluamppSKF1EZGitD4aUli9UOBQA1wFPuPu1QBvBEFKP4EzAU3iPc7j7anev\ndvfqiors/DErEclNNZEGLqsYw+WTx4ZdypBIJRwOAYfcfXPw+BkSYdEQDBUR3DcGzx8GZiStPz1o\nOxws924XEckKJ9o7+Y89R1lWNYVc+TzNoMPB3aPAQTObFzTdCdQBa4FVQdsq4NlgeS2w0syKzGw2\niQvPrwRDUK1mtjj4lNK9SeuIiGS8F+ob6Y571s7dcD4FKa7/p8APzGwUsAf4OInAWWNm9wH7gXsA\n3D1iZmtIBEgMeMjdu4PXeRD4LlAMPB/cRESyQk0kytQJo7lq+oSwSxkyKYWDu78OVJ/nqTsv0P9R\n4NHztG8BFqZSi4hIGNo6Ymza2cRHF83MmSEl0DekRURS8qudTXTE4jnxxbdkCgcRkRTURKJMLCnk\nhlkTL945iygcREQGqSPWzYv1jSxZUElBfm4dTnNra0RE0ug3bx3lZEcsZ774lkzhICIySOsjUcaM\nyufmOeVhlzLkFA4iIoPQHXfWRxq4ff5kRhfmX3yFLKNwEBEZhC37jnG0rTMnh5RA4SAiMig1kQZG\nFeRx27zJYZcyLBQOIiID5O7URKK89/Jyxhal+kMTmUnhICIyQLWHWzl84jTLcnRICRQOIiIDVhOJ\nkmfw/iuzf67oC1E4iIgM0LpIlBtnT6JszKiwSxk2CgcRkQHY3XiK3Y2ncvZTSj0UDiIiA1ATTAe6\ntCp3h5RA4SAiMiA1kShXzyhl6oTisEsZVgoHEZF+OnziNG8camFZjp81gMJBRKTf1gdDSrk0HeiF\nKBxERPppXW2UKyrHclnF2LBLGXYKBxGRfjh6qoNX9x3LuRnfLkThICLSDy/UNxB3FA4iIvKOdbVR\npk8spuqS8WGXkhYKBxGRi9i85ygv7WziQ1dfgpmFXU5aKBxERN7FqY4YDz+9jZllJTx0++Vhl5M2\nuflbsyIiQ+Tvnqvj7ROnWfOJmxiToz/PfT46cxARuYCN9Q089epBPvG+OVTPKgu7nLRSOIiInMex\ntk4++5PtzJ8yjk+9f27Y5aTdyDlHEhHpJ3fnr/5tOy2nO/nX+xZRVJAfdklppzMHEZFenn39bZ6v\njfLnS+Zx5dSR8dHV3hQOIiJJjrSc5q+freX6SyfywK2XhV1OaBQOIiKBeNz5y6ffINbtfOUjV5Of\nNzK+03A+KYeDmeWb2Wtm9lzwuMzMNpjZruB+YlLfR8xst5ntMLNlSe3Xm9n24LnHbKR8y0REMsr3\nN+/n33c381d3Xcms8jFhlxOqoThz+CRQn/T4c8BGd58LbAweY2YLgJVAFbAceNzMeq7yPAHcD8wN\nbsuHoC4RkX7b03SKv/9FPbdeUcEf3Dgz7HJCl1I4mNl04C7gm0nNK4Ang+UngbuT2p9y9w533wvs\nBhaZ2VRgvLu/7O4OfC9pHRGRYRfrjvPw09soKsjnH3//qhHzExnvJtUzh68DnwHiSW2V7n4kWI4C\nPVMmTQMOJvU7FLRNC5Z7t4uIpMU/b9rDawdO8KW7FzJlwuiwy8kIgw4HM/sg0OjuWy/UJzgT8MG+\nx3ne8wEz22JmW5qamobqZUVkBKs93MLXNuzkg1dN5UNXXxJ2ORkjlTOHW4APmdk+4CngDjP7PtAQ\nDBUR3DcG/Q8DM5LWnx60HQ6We7f34e6r3b3a3asrKipSKF1EBM50dfPwmm2UjRnFl1YsDLucjDLo\ncHD3R9x9urvPInGh+UV3/xiwFlgVdFsFPBssrwVWmlmRmc0mceH5lWAIqtXMFgefUro3aR0RkWHz\ntQ072dFwkn/48FVMHDMq7HIyynD8fMaXgTVmdh+wH7gHwN0jZrYGqANiwEPu3h2s8yDwXaAYeD64\niYgMm1f2HmP1r/fwX2+cye3zJoddTsaxxGWB7FNdXe1btmwJuwwRyUKnOmL87jc2YRjPf/K9I+qn\nuM1sq7tXX6zfyPkXEREJPPrzOg4dP83TI2yOhoHQz2eIyIjy4psN/OiVg3zi1pE3R8NAKBxEZMQ4\nnjRHw6eXjLw5GgZC51MiMiK4O1/4WS0n2jt58uMjc46GgdCZg4iMCGu3vc3Ptx/hU++/ggWXjMw5\nGgZC4SAiOS/acoa//lkt180s5RMjeI6GgVA4iEhOc3f+8pltdHU7X7nnGgryddjrD/0riUhO+/7m\nA/x6VzOfv+tKZo/wORoGQuEgIjlrb3Mbf//zxBwNH9McDQOicBCRnBTrjvPwmtcpzDfN0TAI+iir\niOSkf960h98eOME3Vl6jORoGQWcOIpJzIm+38PUXdnKX5mgYNIWDiOSUjlg3f/7jbZSWjOLvVizU\ncNIgaVhJRHLKV4M5Gr7zRzdojoYU6MxBRHLGq/uOsXrTHj66aCa3z9ccDalQOIhITmjriPHwmm3M\nmFjCF+66Muxysp6GlUQkJzz6i3oOHm/nxw9ojoahoDMHEcl6v9zRyA83H+CB917Gotmao2EoKBxE\nJKsdb+vks8+8wbzKcXx6yRVhl5MzdO4lIlntr5+t5Xh7J9/5+A2MLtQcDUNFZw4ikrXWbnub595I\nzNFQdcmEsMvJKQoHEclKPXM0XKs5GoaFwkFEso6785mfvEFnLM5XNUfDsNC/qIhknR9sPsCmnU18\n/gPzNUfDMFE4iEhW2dfcxqM/r+e9c8v52OJLwy4nZykcRCRrdMedh5/elpij4cOao2E46aOsIpI1\nVm/aw9b9x/nGymuYOqE47HJyms4cRCQr1B9p5asbdnDXezRHQzooHEQk43XEuvn0j1+ntGQUX7pb\nczSkg4aVRCTjff2FXbwZPcm3/6iaMs3RkBY6cxCRjLZl3zH++VdvsfKGGdwxvzLsckaMQYeDmc0w\ns1+aWZ2ZRczsk0F7mZltMLNdwf3EpHUeMbPdZrbDzJYltV9vZtuD5x4znTOKjHjuzg83H2DVt19h\n2sRivvDBBWGXNKKkcuYQAx529wXAYuAhM1sAfA7Y6O5zgY3BY4LnVgJVwHLgcTPr+ZWsJ4D7gbnB\nbXkKdYlIljvScppV33mVz//bdq6eUcqP7l/MWM3RkFaD/td29yPAkWD5pJnVA9OAFcBtQbcngZeA\nzwbtT7l7B7DXzHYDi8xsHzDe3V8GMLPvAXcDzw+2NhHJTu7OM1sP8bfP1RHrdr60ooo/uPFS8vI0\nmJBuQxLFZjYLuBbYDFQGwQEQBXoGCacBLyetdiho6wqWe7eLyAjS2HqGz//bdl6ob2TRrDL+50eu\n4tJJ+mmMsKQcDmY2FvgJ8Cl3b02+XODubmae6nskvdcDwAMAM2fOHKqXFZEQuTtrt73NF9dGON3Z\nzRfuupI/vmW2zhZCllI4mFkhiWD4gbv/NGhuMLOp7n7EzKYCjUH7YWBG0urTg7bDwXLv9j7cfTWw\nGqC6unrIQkdEwnH0VAdf+Fktz9dGuWZGKV+552rmVIwNuywhtU8rGfAtoN7dv5r01FpgVbC8Cng2\nqX2lmRWZ2WwSF55fCYagWs1scfCa9yatIyI5al3tEZZ+bRMb6xv5zPJ5PPMnNykYMkgqZw63AH8I\nbDez14O2zwNfBtaY2X3AfuAeAHePmNkaoI7EJ50ecvfuYL0Hge8CxSQuROtitEiOOtHeyRfXRnj2\n9bdZOG08P/zINcybMi7ssqQXc8/O0Znq6mrfsmVL2GWIyAC8+GYDn/vJdo61dfKnd8zlwdvnUKiJ\netLKzLa6e/XF+umDwyIy7FrPdPGl/1vH01sPMX/KOL79RzewcJrmfM5kCgcRGVa/3tXEZ595g2jr\nGR66fQ5/dudcigryL76ihErhICLDoq0jxt//op4fbD7AnIox/PTBW7hmRmnYZUk/KRxEZMi9vOco\nf/nMNg4dP839753Nw0vnMbpQZwvZROEgIkPmdGc3/1jzJt/5f/u4dFIJaz5xEzfMKgu7LBkEhYOI\nDImt+4/zF09vY29zG6tuupTP/u58SkbpEJOttOdEJCVnurr52gs7+ZdNe5g6oZgf3n8jN88pD7ss\nSZHCQUQG7Y1DJ3h4zTZ2NZ7io4tm8ld3Xamf1s4R2osiMmCdsTj/9OIuHn/pLSrGFvHkHy/ifVdU\nhF2WDCGFg4gMSN3brTz89Dbqj7Ty+9dN529+bwETigvDLkuGmMJBRPol1h3niZfe4rEXdzGheBT/\ncm81SxZoTudcpXAQkYva1XCSh5/exhuHWvi9qy/hbz9UxcQxo8IuS4aRwkFELqg77nzz13v4yoad\njC0q4PE/uI4PvGdq2GVJGigcRKSP7rizZd8x/rFmB1v3H2dZVSWP/qf3UD62KOzSJE0UDiICJL6v\n8O+7mllfF+WF+kaOtXUyobiQr/+Xa1hxzSUkTwEsuU/hIDKCtbR38eKOBtZHGvjVzibaO7sZN7qA\nO+ZPZumCKbxvXoW+tzBCaa+LjDBHWk6zoa6BmkiUzXuOEYs7leOL+M/XTWNZ1RRunD2JUQWagGek\nUziI5Dh3Z3fjKWoiUdbXNfDGoRYA5lSM4f5bL2NZ1RSumjaBvDwNG8k7FA4iOSged147eJz1kQbW\n1zWwt7kNgGtmlPLZ5fNZsqCSyyePDblKyWQKB5Ec0RHr5jdvHWV9pIENdQ00n+qgMN+4aU459/3O\nbJYsqKRy/Oiwy5QsoXAQyWKtZ7p4aUcT6yNRXtrRxKmOGGNG5XPb/MksXVDJbfMm66ctZFAUDiJZ\nprH1DBvqG6iJNPAfbzXT1e2Ujx3F7109laULpnDz5ZM0R7OkTOEgkgX2NJ2iJtLA+roorx04AcCl\nk0r4+C2zWbqgkmtnTiRfF5RlCCkcRDJQPO68cbiF9cEnjHY3ngLgPdMm8BdLr2Bp1RTmTh6rL6bJ\nsFE4iIQoHnfebjnNvuZ29jafYm9zO/uOthF5u4WG1g7y84wbZ5fxh4svZcmCSi4pLQ67ZBkhFA4i\nw8zdaTzZwd7mNvY2t7EvuN/b3Mb+Y+10xuJn+xYX5nPppBJumFXGHfMnc8f8yZSW6NdPJf0UDiJD\nwN051tb5TgAcbWNfczt7mtvYf7SN9s7us31H5ecxc1IJs8vHcPv8ycyaNIbZ5Ylb5fgiDRVJRlA4\niAxAS3sXe4+e+9f/vqOJ+5NnYmf75ecZM8tKmDWphMWXlZ09+M+aNIZLSot18VgynsJBpJe2jljS\nX/9t7AmGgvYdbedYW+fZfmYwrbSY2eVjuPuaae8EQPkYpk8spjBfv08k2UvhIDmrMxan5XTX2Vvr\n6S5OnO6kpb2LltOxxHLQ3nK6ixPtXRxv76T5VOc5rzNl/GhmlZewrGoKs8tLzg4DzSgrYXShvk8g\nuSljwsHMlgPfAPKBb7r7l0MuSTJAd9yDg3rXOQf6lvbOs8sn2ns9F9ySx/nPZ2xRAROKC8/e5lSM\npbSkkBllJWfPAi6dVELJqIz530QkbTLiv3ozywf+N7AEOAS8amZr3b0u3Mrk3cS643TEem7ddHQl\nLcfiwePu8z/fq++Zrm5Onon1CoAuTnbE3rWG4sJ8JhQXUlpSyPjixIH9PUkH/AklhecEQKLvKMaP\nLqBAwz4iF5QR4QAsAna7+x4AM3sKWAFkZTi4O+7gQNydeM9jf+dx3BP9YnGnO+kWO2c5fsHnBtM3\n7k6s2+mOxxPPu9Pd7XQN6CAfp6MrsRyLe0r/TnkGowvzKSrIo6ggn7GjCygtLmTK+NHMqxzX58Be\nes7jUYwvLtDPRIgMk0wJh2nAwaTHh4Abh+ON1rx6kNW/3nP2gN373oMDd88BHJIexx3n3IN88kG/\nJww8tWNmWhTkGfnBbVRB3tkDdFFBHkWF7yyPLUocgBNtffuMLjz/eu+2TlHwfvrLXSRzZUo49IuZ\nPQA8ADBz5sxBvUZpSSHzKsdhBmZGnkGeWeIx7zzOywOSHwf97QKPjf7163k9gv4F+YkDdOJgnUd+\nHuTn5b1z8DYjPz943oK++UHfcx73er5nOS+P/HMemyZ1EZGLypRwOAzMSHo8PWg7h7uvBlYDVFdX\nD+rv86VVU1haNWUwq4qIjBiZcl7/KjDXzGab2ShgJbA25JpEREasjDhzcPeYmf13oIbER1m/7e6R\nkMsSERmxMiIcANz9F8Avwq5DREQyZ1hJREQyiMJBRET6UDiIiEgfCgcREelD4SAiIn2YZ8NvPZyH\nmTUB+we5ejnQPITlhEnbknlyZTtA25KpUtmWS9294mKdsjYcUmFmW9y9Ouw6hoK2JfPkynaAtiVT\npWNbNKwkIiJ9KBxERKSPkRoOq8MuYAhpWzJPrmwHaFsy1bBvy4i85iAiIu9upJ45iIjIu8jZcDCz\nb5tZo5nVXuB5M7PHzGy3mb1hZtelu8b+6se23GZmLWb2enD7m3TX2B9mNsPMfmlmdWYWMbNPnqdP\nVuyXfm5LtuyX0Wb2ipltC7blf5ynT7bsl/5sS1bsFwAzyzez18zsufM8N7z7JDHfce7dgFuB64Da\nCzz/AeB5wIDFwOawa05hW24Dngu7zn5sx1TgumB5HLATWJCN+6Wf25It+8WAscFyIbAZWJyl+6U/\n25IV+yWo9c+BH56v3uHeJzl75uDum4Bj79JlBfA9T3gZKDWzqempbmD6sS1Zwd2PuPtvg+WTQD2J\n+cOTZcV+6ee2ZIXg3/pU8LAwuPW+GJkt+6U/25IVzGw6cBfwzQt0GdZ9krPh0A/TgINJjw+Rpf9z\nB24OTi2fN7OqsIu5GDObBVxL4i+7ZFm3X95lWyBL9kswfPE60AhscPes3S/92BbIjv3ydeAzQPwC\nzw/rPhnJ4ZBLfgvMdPergH8CfhZyPe/KzMYCPwE+5e6tYdeTiotsS9bsF3fvdvdrSMzfvsjMFoZd\n02D1Y1syfr+Y2QeBRnffGlYNIzkcDgMzkh5PD9qyjru39pxKe2JGvUIzKw+5rPMys0ISB9MfuPtP\nz9Mla/bLxbYlm/ZLD3c/AfwSWN7rqazZLz0utC1Zsl9uAT5kZvuAp4A7zOz7vfoM6z4ZyeGwFrg3\nuOK/GGhx9yNhFzUYZjbFzCxYXkRivx4Nt6q+ghq/BdS7+1cv0C0r9kt/tiWL9kuFmZUGy8XAEuDN\nXt2yZb9cdFuyYb+4+yPuPt3dZwErgRfd/WO9ug3rPsmYOaSHmpn9iMSnEsrN7BDwRRIXp3D3/0Ni\nvuoPALuBduDj4VR6cf3Ylg8D/83MYsBpYKUHH2fIMLcAfwhsD8aEAT4PzISs2y/92ZZs2S9TgSfN\nLJ/EgXKNuz9nZn8CWbdf+rMt2bJf+kjnPtE3pEVEpI+RPKwkIiIXoHAQEZE+FA4iItKHwkFERPpQ\nOIiISB9QazzxAAAAF0lEQVQKBxER6UPhICIifSgcRESkj/8Pwc6WdGfT4P4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8b8abd96d8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.linspace(1, 4, 10), np.logspace(1, 4, 10))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0. 0.]\n",
      " [0. 0.]\n",
      " [0. 0.]\n",
      " [0. 0.]]\n",
      "[[1. 2.]\n",
      " [0. 0.]\n",
      " [0. 0.]\n",
      " [0. 0.]]\n"
     ]
    }
   ],
   "source": [
    "shape = (4, 2)\n",
    "print(np.zeros(shape)) # init to zero. Use np.ones or np.full accordingly\n",
    "\n",
    "# [GOTCHA] np.empty won't initialize anything; it will just grab the first available chunk of memory\n",
    "x = np.zeros(shape)\n",
    "x[0] = [1, 2]\n",
    "del x\n",
    "print(np.empty(shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2],\n",
       "       [3, 4],\n",
       "       [5, 6]])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# From iterator/list/array - can just use constructor\n",
    "np.array([[1, 2], range(3, 5), np.array([5, 6])]) # auto-flatten (if possible)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 1]\n",
      " [2 5]]\n",
      "[[0 1]\n",
      " [2 3]]\n",
      "[[0 0]\n",
      " [0 0]]\n"
     ]
    }
   ],
   "source": [
    "# Deep copies & shape/dtype preserving creations\n",
    "x = np.arange(4).reshape(2, 2)\n",
    "y = np.copy(x)\n",
    "z = np.zeros_like(x)\n",
    "x[1, 1] = 5\n",
    "print(x)\n",
    "print(y)\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extremely extensive [random generation](https://docs.scipy.org/doc/numpy/reference/routines.random.html). Remember to seed!\n",
    "\n",
    "# Transposition\n",
    "\n",
    "**Under the hood**. So far, we've just been looking at the abstraction that NumPy offers. How does it actually keep things contiguous in memory?\n",
    "\n",
    "We have a base array, which is one long contiguous array from 0 to size - 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2, 3, 4)\n",
      "24\n"
     ]
    }
   ],
   "source": [
    "x = np.arange(2 * 3 * 4).reshape(2, 3, 4)\n",
    "print(x.shape)\n",
    "print(x.size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[ 0  1  2  3]\n",
      "  [ 4  5  6  7]\n",
      "  [ 8  9 10 11]]\n",
      "\n",
      " [[12 13 14 15]\n",
      "  [16 17 18 19]\n",
      "  [20 21 22 23]]]\n",
      "[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23]\n"
     ]
    }
   ],
   "source": [
    "# Use ravel() to get the underlying flat array. np.flatten() will give you a copy\n",
    "print(x)\n",
    "print(x.ravel())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "transpose (2, 3, 4) -> (4, 3, 2)\n",
      "rollaxis (2, 3, 4) -> (3, 2, 4)\n",
      "\n",
      "arbitrary permutation [0, 1, 2] [0 2 1]\n",
      "(2, 3, 4) -> (2, 4, 3)\n",
      "moved[1, 2, 0] 14 x[1, 0, 2] 14\n"
     ]
    }
   ],
   "source": [
    "# np.transpose or *.T will reverse axes\n",
    "print('transpose', x.shape, '->', x.T.shape)\n",
    "# rollaxis pulls the argument axis to axis 0, keeping all else the same.\n",
    "print('rollaxis', x.shape, '->', np.rollaxis(x, 1, 0).shape)\n",
    "\n",
    "print()\n",
    "# all the above are instances of np.moveaxis\n",
    "# it's clear how these behave:\n",
    "\n",
    "perm = np.array([0, 2, 1])\n",
    "moved = np.moveaxis(x, range(3), perm)\n",
    "\n",
    "print('arbitrary permutation', list(range(3)), perm)\n",
    "print(x.shape, '->', moved.shape)\n",
    "print('moved[1, 2, 0]', moved[1, 2, 0], 'x[1, 0, 2]', x[1, 0, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sigma 3.19, eig 3.19\n"
     ]
    }
   ],
   "source": [
    "# When is transposition useful?\n",
    "# Matrix stuff, mostly:\n",
    "np.random.seed(1234)\n",
    "\n",
    "X = np.random.randn(3, 4)\n",
    "print('sigma {:.2f}, eig {:.2f}'.format(\n",
    "    np.linalg.svd(X)[1].max(),\n",
    "    np.sqrt(np.linalg.eigvalsh(X.dot(X.T)).max())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQ8AAAD8CAYAAABpXiE9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADYlJREFUeJzt3W+onvV9x/H3Z0ncXKp1mlbT/EHdwphz3Zod0mBly5gt\nGoT0gYw4qCKDg6KshfpAKthHg22MbrWKWaBShaJ7oNWwpSsqZdoHOmOI0WitqRNMzMyqXdQq2rjv\nHpzL7XA8J+fkd1/nvu/o+wU35/rzO9f3yy/h43Vff0yqCkk6Xr8y6gYknZgMD0lNDA9JTQwPSU0M\nD0lNDA9JTZYO8stJTgf+CTgbeBH4s6r6+SzjXgTeAN4DjlbVxCB1JY3eoGceNwAPVdU64KFufS5/\nUlV/YHBIHw6DhscW4I5u+Q7giwMeT9IJIoM8YZrkv6vqtG45wM/fX58x7j+AI0x9bfnHqtp+jGNO\nApMAv/br+cPV5/5qc38fdq88ffKoWxh776xZPuoWxtrR117jvTd/kZbfnfeaR5IHgbNm2XXj9JWq\nqiRzJdGFVXUwySeBB5L8uKoenm1gFyzbAdb93sn1zft/c74WP7L+/rd+Z9QtjL39128cdQtj7eW/\n+4fm3503PKrqorn2JXklycqqOpRkJXB4jmMc7H4eTvI9YAMwa3hIOjEMes1jB3Blt3wlcP/MAUmW\nJznl/WXgC8DTA9aVNGKDhsdfA59P8jxwUbdOkk8l2dmNORP4UZIngX8H/qWq/nXAupJGbKDnPKrq\nVeBPZ9n+MrC5W34B+P1B6kgaPz5hKqmJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnh\nIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEh\nqYnhIamJ4SGpieEhqUkv4ZHk4iTPJdmf5IZZ9ifJzd3+vUnW91FX0ugMHB5JlgC3ApcA5wGXJzlv\nxrBLgHXdZxK4bdC6kkarjzOPDcD+qnqhqt4F7ga2zBizBbizpjwKnJZkZQ+1JY1IH+GxCnhp2vqB\nbtvxjpF0Ahm7C6ZJJpPsSrLryGvvjbodSXPoIzwOAmumra/uth3vGACqantVTVTVxMdPX9JDe5IW\nQx/h8TiwLsk5SU4CtgI7ZozZAVzR3XXZCBypqkM91JY0IksHPUBVHU1yHfADYAlwe1XtS3J1t38b\nsBPYDOwH3gKuGrSupNEaODwAqmonUwExfdu2acsFXNtHLUnjYewumEo6MRgekpoYHpKaGB6Smhge\nkpoYHpKaGB6SmhgekpoYHpKaGB6SmhgekpoYHpKaGB6SmhgekpoYHpKaGB6SmhgekpoYHpKaGB6S\nmhgekpoYHpKaGB6SmhgekpoYHpKaGB6SmhgekpoYHpKaGB6SmvQSHkkuTvJckv1Jbphl/6YkR5Ls\n6T439VFX0ugsHfQASZYAtwKfBw4AjyfZUVXPzBj6SFVdOmg9SeOhjzOPDcD+qnqhqt4F7ga29HBc\nSWNs4DMPYBXw0rT1A8BnZxl3QZK9wEHg+qraN9vBkkwCkwBLV3ycv3x8aw8tfjid//DLo25h7K19\n59CoWxhrPzv5l82/O6wLpruBtVX1aeBbwH1zDayq7VU1UVUTS05dPqT2JB2vPsLjILBm2vrqbtv/\nqarXq+rNbnknsCzJih5qSxqRPsLjcWBdknOSnARsBXZMH5DkrCTpljd0dV/tobakERn4mkdVHU1y\nHfADYAlwe1XtS3J1t38bcBlwTZKjwNvA1qqqQWtLGp0+Lpi+/1Vk54xt26Yt3wLc0kctSePBJ0wl\nNTE8JDUxPCQ1MTwkNTE8JDUxPCQ1MTwkNTE8JDUxPCQ1MTwkNTE8JDUxPCQ1MTwkNTE8JDUxPCQ1\nMTwkNTE8JDUxPCQ1MTwkNTE8JDUxPCQ1MTwkNTE8JDUxPCQ1MTwkNTE8JDUxPCQ16SU8ktye5HCS\np+fYnyQ3J9mfZG+S9X3UlTQ6fZ15fAe4+Bj7LwHWdZ9J4Lae6koakV7Co6oeBl47xpAtwJ015VHg\ntCQr+6gtaTSGdc1jFfDStPUD3bYPSDKZZFeSXe+9/ouhNCfp+I3dBdOq2l5VE1U1seTU5aNuR9Ic\nhhUeB4E109ZXd9sknaCGFR47gCu6uy4bgSNVdWhItSUtgqV9HCTJXcAmYEWSA8DXgWUAVbUN2Als\nBvYDbwFX9VFX0uj0Eh5Vdfk8+wu4to9aksbD2F0wlXRiMDwkNTE8JDUxPCQ1MTwkNTE8JDUxPCQ1\nMTwkNTE8JDUxPCQ1MTwkNTE8JDUxPCQ1MTwkNTE8JDUxPCQ1MTwkNTE8JDUxPCQ1MTwkNTE8JDUx\nPCQ1MTwkNTE8JDUxPCQ1MTwkNTE8JDXpJTyS3J7kcJKn59i/KcmRJHu6z0191JU0Or38Q9fAd4Bb\ngDuPMeaRqrq0p3qSRqyXM4+qehh4rY9jSTox9HXmsRAXJNkLHASur6p9sw1KMglMAqxdtZSf/PEd\nQ2zxxHLJ5j8fdQtj78Vrlo+6hbH27jvtETCsC6a7gbVV9WngW8B9cw2squ1VNVFVE584Y8mQ2pN0\nvIYSHlX1elW92S3vBJYlWTGM2pIWx1DCI8lZSdItb+jqvjqM2pIWRy/XPJLcBWwCViQ5AHwdWAZQ\nVduAy4BrkhwF3ga2VlX1UVvSaPQSHlV1+Tz7b2HqVq6kDwmfMJXUxPCQ1MTwkNTE8JDUxPCQ1MTw\nkNTE8JDUxPCQ1MTwkNTE8JDUxPCQ1MTwkNTE8JDUxPCQ1MTwkNTE8JDUxPCQ1MTwkNTE8JDUxPCQ\n1MTwkNTE8JDUxPCQ1MTwkNTE8JDUxPCQ1MTwkNRk4PBIsibJD5M8k2Rfki/PMiZJbk6yP8neJOsH\nrStptPr4h66PAl+tqt1JTgGeSPJAVT0zbcwlwLru81ngtu6npBPUwGceVXWoqnZ3y28AzwKrZgzb\nAtxZUx4FTkuyctDakkan12seSc4GPgM8NmPXKuClaesH+GDASDqB9BYeST4G3AN8papeH+A4k0l2\nJdn1X6++11d7knrWS3gkWcZUcHy3qu6dZchBYM209dXdtg+oqu1VNVFVE584Y0kf7UlaBH3cbQnw\nbeDZqvrGHMN2AFd0d102Akeq6tCgtSWNTh93Wz4HfAl4KsmebtvXgLUAVbUN2AlsBvYDbwFX9VBX\n0ggNHB5V9SMg84wp4NpBa0kaHz5hKqmJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnh\nIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEh\nqYnhIamJ4SGpieEhqYnhIanJwOGRZE2SHyZ5Jsm+JF+eZcymJEeS7Ok+Nw1aV9JoLe3hGEeBr1bV\n7iSnAE8keaCqnpkx7pGqurSHepLGwMBnHlV1qKp2d8tvAM8CqwY9rqTxlqrq72DJ2cDDwPlV9fq0\n7ZuAe4EDwEHg+qraN8cxJoHJbvV84OneGhzcCuBno25iGvuZ37j1NG79/HZVndLyi72FR5KPAf8G\n/FVV3Ttj36nA/1TVm0k2A9+sqnULOOauqpropcEe2M+xjVs/MH49fZj66eVuS5JlwD3Ad2cGB0BV\nvV5Vb3bLO4FlSVb0UVvSaPRxtyXAt4Fnq+obc4w5qxtHkg1d3VcHrS1pdPq42/I54EvAU0n2dNu+\nBqwFqKptwGXANUmOAm8DW2th35e299Bfn+zn2MatHxi/nj40/fR6wVTSR4dPmEpqYnhIajI24ZHk\n9CQPJHm++/kbc4x7MclT3WPuuxahj4uTPJdkf5IbZtmfJDd3+/cmWd93Dw09De3x/yS3JzmcZNbn\nb0Y0P/P1NNTXIxb4ysbQ5mnRXiGpqrH4AH8L3NAt3wD8zRzjXgRWLFIPS4CfAucCJwFPAufNGLMZ\n+D4QYCPw2CLPy0J62gT885D+nP4IWA88Pcf+oc7PAnsa2vx09VYC67vlU4CfjPLv0QL7Oe45Gpsz\nD2ALcEe3fAfwxRH0sAHYX1UvVNW7wN1dX9NtAe6sKY8CpyVZOeKehqaqHgZeO8aQYc/PQnoaqlrY\nKxtDm6cF9nPcxik8zqyqQ93yfwJnzjGugAeTPNE9yt6nVcBL09YP8MFJXsiYYfcEcEF3+vv9JL+7\niP3MZ9jzs1AjmZ/ulY3PAI/N2DWSeTpGP3Ccc9THcx4LluRB4KxZdt04faWqKslc95AvrKqDST4J\nPJDkx91/eT7KdgNr6/8f/78PmPfx/4+QkcxP98rGPcBXatq7XqMyTz/HPUdDPfOoqouq6vxZPvcD\nr7x/2tb9PDzHMQ52Pw8D32PqtL4vB4E109ZXd9uOd0yf5q1X4/X4/7DnZ16jmJ/5XtlgyPO0GK+Q\njNPXlh3Ald3ylcD9MwckWZ6p/2cISZYDX6Dft24fB9YlOSfJScDWrq+ZfV7RXS3fCByZ9nVrMczb\n05g9/j/s+ZnXsOenq3XMVzYY4jwtpJ+mORrG1ecFXhE+A3gIeB54EDi92/4pYGe3fC5TdxueBPYB\nNy5CH5uZuhr90/ePD1wNXN0tB7i12/8UMDGEuZmvp+u6+XgSeBS4YBF7uQs4BPySqe/pfzEG8zNf\nT0Obn67ehUxdm9sL7Ok+m0c1Twvs57jnyMfTJTUZp68tkk4ghoekJoaHpCaGh6QmhoekJoaHpCaG\nh6Qm/wsrMwkwfCtYpQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8b8ace7f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQ8AAAD8CAYAAABpXiE9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADYVJREFUeJzt3W+onvV9x/H3ZzHq/FPUZtUYI7Vd2OZktVlInZWRMdtq\nENIHMuIDFRkcFIUWakEq2EeDbQ8Kc4pZoFIdRfdAq6FLJypS7QOtMWg0WjV1gjnNmk1t1Pk33XcP\nzuV2OJ6Tc/K7r3Pf99H3C27O9ed3ru+Xn+GT677+mFQVknS4fmfUDUhamgwPSU0MD0lNDA9JTQwP\nSU0MD0lNjhjkl5OcBPwL8FngZeCvqur1Wca9DLwJ/BY4WFXrBqkrafQGPfO4DniwqtYAD3brc/mL\nqjrb4JA+HgYNj03Abd3ybcDXBzyepCUigzxhmuQ3VXVCtxzg9Q/XZ4z7d+AAU19b/qmqth7imBPA\nBMCxx+RP//D3j2zu7+PuhV3HjLqFsXfUH2XULYy1N3/137z7m3ebJmneax5JHgBOmWXX9dNXqqqS\nzJVE51XVZJLPAPcn+UVVPTzbwC5YtgKs+8LR9fP7Vs/X4ifW1049e9QtjL3P//PRo25hrN116fbm\n3503PKrq/Ln2Jfl1kpVVtS/JSmD/HMeY7H7uT/IjYD0wa3hIWhoGveaxDbi8W74cuHfmgCTHJjn+\nw2Xgq8AzA9aVNGKDhsffAl9J8iJwfrdOklOTfHg+dDLwsyRPAT8H/rWq/m3AupJGbKDnPKrqVeAv\nZ9n+K2Bjt/wS8IVB6kgaPz5hKqmJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ\n4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnh\nIamJ4SGpieEhqUkv4ZHkgiTPJ9mT5LpZ9ifJjd3+XUnW9lFX0ugMHB5JlgE3AxcCZwKXJDlzxrAL\ngTXdZwK4ZdC6kkarjzOP9cCeqnqpqt4H7gQ2zRizCbi9pjwKnJBkZQ+1JY1IH+GxCnhl2vrebtvh\njpG0hIzdBdMkE0l2JNnxn6/+dtTtSJpDH+ExCayetn5at+1wxwBQVVural1Vrfu9Ty/roT1Ji6GP\n8HgcWJPkjCRHApuBbTPGbAMu6+66nAMcqKp9PdSWNCJHDHqAqjqY5BrgPmAZcGtV7U5yZbd/C7Ad\n2AjsAd4Grhi0rqTRGjg8AKpqO1MBMX3blmnLBVzdRy1J42HsLphKWhoMD0lNDA9JTQwPSU0MD0lN\nDA9JTQwPSU0MD0lNDA9JTQwPSU0MD0lNDA9JTQwPSU0MD0lNDA9JTQwPSU0MD0lNDA9JTQwPSU0M\nD0lNDA9JTQwPSU0MD0lNDA9JTQwPSU0MD0lNDA9JTQwPSU16CY8kFyR5PsmeJNfNsn9DkgNJnuw+\nN/RRV9LoHDHoAZIsA24GvgLsBR5Psq2qnp0x9JGqumjQepLGQx9nHuuBPVX1UlW9D9wJbOrhuJLG\n2MBnHsAq4JVp63uBL80y7twku4BJ4Nqq2j3bwZJMABMAR3MMXzv17B5a/Hj63Z+ePOoWxt4L3141\n6hbG2nt7H2r+3T7CYyF2AqdX1VtJNgL3AGtmG1hVW4GtAJ/KSTWk/iQdpj6+tkwCq6etn9Zt+z9V\n9UZVvdUtbweWJ1nRQ21JI9JHeDwOrElyRpIjgc3AtukDkpySJN3y+q7uqz3UljQiA39tqaqDSa4B\n7gOWAbdW1e4kV3b7twAXA1clOQi8A2yuKr+SSEtYL9c8uq8i22ds2zJt+Sbgpj5qSRoPPmEqqYnh\nIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEh\nqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpSS/hkeTWJPuTPDPH\n/iS5McmeJLuSrO2jrqTR6evM4wfABYfYfyGwpvtMALf0VFfSiPQSHlX1MPDaIYZsAm6vKY8CJyRZ\n2UdtSaMxrGseq4BXpq3v7bZ9RJKJJDuS7PiA94bSnKTDN3YXTKtqa1Wtq6p1yzlq1O1ImsOwwmMS\nWD1t/bRum6QlaljhsQ24rLvrcg5woKr2Dam2pEVwRB8HSXIHsAFYkWQv8F1gOUBVbQG2AxuBPcDb\nwBV91JU0Or2ER1VdMs/+Aq7uo5ak8TB2F0wlLQ2Gh6QmhoekJoaHpCaGh6QmhoekJoaHpCaGh6Qm\nhoekJoaHpCaGh6QmhoekJoaHpCaGh6QmhoekJoaHpCaGh6QmhoekJoaHpCaGh6QmhoekJoaHpCaG\nh6QmhoekJoaHpCaGh6QmhoekJr2ER5Jbk+xP8swc+zckOZDkye5zQx91JY1OL//QNfAD4Cbg9kOM\neaSqLuqpnqQR6+XMo6oeBl7r41iSloa+zjwW4twku4BJ4Nqq2j3boCQTwATAcaccy+d/fPQQW1xa\nXvj2qlG3MPb2/Zl/fg7lg+fS/LvDumC6Ezi9qv4E+EfgnrkGVtXWqlpXVeuOPvGoIbUn6XANJTyq\n6o2qeqtb3g4sT7JiGLUlLY6hhEeSU5KkW17f1X11GLUlLY5ernkkuQPYAKxIshf4LrAcoKq2ABcD\nVyU5CLwDbK6q6qO2pNHoJTyq6pJ59t/E1K1cSR8TPmEqqYnhIamJ4SGpieEhqYnhIamJ4SGpieEh\nqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGp\nieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpycDhkWR1koeSPJtkd5JvzDImSW5MsifJriRrB60rabT6\n+IeuDwLfqqqdSY4Hnkhyf1U9O23MhcCa7vMl4Jbup6QlauAzj6raV1U7u+U3geeAVTOGbQJurymP\nAickWTlobUmj0+s1jySfBb4IPDZj1yrglWnre/lowEhaQnoLjyTHAXcB36yqNwY4zkSSHUl2vPv6\ne321J6lnvYRHkuVMBccPq+ruWYZMAqunrZ/WbfuIqtpaVeuqat3RJx7VR3uSFkEfd1sCfB94rqq+\nN8ewbcBl3V2Xc4ADVbVv0NqSRqePuy1fBi4Fnk7yZLftO8DpAFW1BdgObAT2AG8DV/RQV9IIDRwe\nVfUzIPOMKeDqQWtJGh8+YSqpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEh\nqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGpieEhqYnhIamJ4SGp\nieEhqYnhIamJ4SGpycDhkWR1koeSPJtkd5JvzDJmQ5IDSZ7sPjcMWlfSaB3RwzEOAt+qqp1Jjgee\nSHJ/VT07Y9wjVXVRD/UkjYGBzzyqal9V7eyW3wSeA1YNelxJ4y1V1d/Bks8CDwNnVdUb07ZvAO4G\n9gKTwLVVtXuOY0wAE93qWcAzvTU4uBXAf426iWnsZ37j1tO49fMHVXV8yy/2Fh5JjgN+CvxNVd09\nY9+ngP+pqreSbAT+oarWLOCYO6pqXS8N9sB+Dm3c+oHx6+nj1E8vd1uSLAfuAn44MzgAquqNqnqr\nW94OLE+yoo/akkajj7stAb4PPFdV35tjzCndOJKs7+q+OmhtSaPTx92WLwOXAk8nebLb9h3gdICq\n2gJcDFyV5CDwDrC5FvZ9aWsP/fXJfg5t3PqB8evpY9NPrxdMJX1y+ISppCaGh6QmYxMeSU5Kcn+S\nF7ufJ84x7uUkT3ePue9YhD4uSPJ8kj1Jrptlf5Lc2O3flWRt3z009DS0x/+T3Jpkf5JZn78Z0fzM\n19NQX49Y4CsbQ5unRXuFpKrG4gP8PXBdt3wd8HdzjHsZWLFIPSwDfgl8DjgSeAo4c8aYjcBPgADn\nAI8t8rwspKcNwI+H9N/pz4G1wDNz7B/q/Cywp6HNT1dvJbC2Wz4eeGGUf44W2M9hz9HYnHkAm4Db\nuuXbgK+PoIf1wJ6qeqmq3gfu7PqabhNwe015FDghycoR9zQ0VfUw8Nohhgx7fhbS01DVwl7ZGNo8\nLbCfwzZO4XFyVe3rlv8DOHmOcQU8kOSJ7lH2Pq0CXpm2vpePTvJCxgy7J4Bzu9PfnyT540XsZz7D\nnp+FGsn8dK9sfBF4bMaukczTIfqBw5yjPp7zWLAkDwCnzLLr+ukrVVVJ5rqHfF5VTSb5DHB/kl90\nf/N8ku0ETq//f/z/HmDex/8/QUYyP90rG3cB36xp73qNyjz9HPYcDfXMo6rOr6qzZvncC/z6w9O2\n7uf+OY4x2f3cD/yIqdP6vkwCq6etn9ZtO9wxfZq3Xo3X4//Dnp95jWJ+5ntlgyHP02K8QjJOX1u2\nAZd3y5cD984ckOTYTP0/Q0hyLPBV+n3r9nFgTZIzkhwJbO76mtnnZd3V8nOAA9O+bi2GeXsas8f/\nhz0/8xr2/HS1DvnKBkOcp4X00zRHw7j6vMArwp8GHgReBB4ATuq2nwps75Y/x9TdhqeA3cD1i9DH\nRqauRv/yw+MDVwJXdssBbu72Pw2sG8LczNfTNd18PAU8Cpy7iL3cAewDPmDqe/pfj8H8zNfT0Oan\nq3ceU9fmdgFPdp+No5qnBfZz2HPk4+mSmozT1xZJS4jhIamJ4SGpieEhqYnhIamJ4SGpieEhqcn/\nAs+f/1HdE4BYAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8b8ae23630>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Check frob norm upper vs lower tri 0.0\n"
     ]
    }
   ],
   "source": [
    "# Create a random symmetric matrix\n",
    "X = np.random.randn(3, 3)\n",
    "plt.imshow(X)\n",
    "plt.show()\n",
    "\n",
    "X += X.T\n",
    "plt.imshow(X)\n",
    "plt.show()\n",
    "\n",
    "print('Check frob norm upper vs lower tri', np.linalg.norm(np.triu(X) - np.tril(X).T))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 1 2]\n",
      " [3 4 5]]\n",
      "[[0 1 2]\n",
      " [3 4 5]]\n"
     ]
    }
   ],
   "source": [
    "# Row-major, C-order\n",
    "# largest axis changes fastest\n",
    "A = np.arange(2 * 3).reshape(2, 3).copy(order='C')\n",
    "\n",
    "# Row-major, Fortran-order\n",
    "# smallest axis changes fastest\n",
    "# GOTCHA: many numpy funcitons assume C ordering\n",
    "B = np.arange(2 * 3).reshape(2, 3).copy(order='F')\n",
    "\n",
    "# Differences in representation don't manifest in abstraction\n",
    "print(A)\n",
    "print(B)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 2 3 4 5]\n",
      "[0 3 1 4 2 5]\n",
      "[[0 3 1]\n",
      " [4 2 5]]\n",
      "[[0 1 2]\n",
      " [3 4 5]]\n"
     ]
    }
   ],
   "source": [
    "# Array manipulation functions with order option\n",
    "# will use C/F ordering, but this is independent of the underlying layout\n",
    "print(A.ravel())\n",
    "print(A.ravel(order='F'))\n",
    "\n",
    "# Reshape ravels an array, then folds back into shape, according to the given order\n",
    "# Note reshape can infer one dimension; we leave it as -1.\n",
    "print(A.ravel(order='F').reshape(-1, 3))\n",
    "print(A.ravel(order='F').reshape(-1, 3, order='F'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "140237307683504 140237307683504 140237307418144\n"
     ]
    }
   ],
   "source": [
    "# GOTCHA: ravel will copy the array so that everything is contiguous\n",
    "# if the order differs\n",
    "print(id(A), id(A.ravel().base), id(A.ravel(order='F')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transposition Example: Kronecker multiplication\n",
    "\n",
    "Based on Saatci 2011 (PhD thesis).\n",
    "\n",
    "Recall the tensor product over vector spaces $V \\otimes W$ from before. If $V$ has basis $\\textbf{v}_i$ and $W$ has $\\textbf{w}_j$, we can define the tensor product over elements $\\nu\\in V,\\omega\\in W$ as follows.\n",
    "\n",
    "Let $\\nu= \\sum_{i=1}^n\\nu_i\\textbf{v}_i$ and $\\omega= \\sum_{j=1}^m\\omega_j\\textbf{w}_j$. Then:\n",
    "$$\n",
    "V \\otimes W\\ni \\nu\\otimes \\omega=\\sum_{i=1}^n\\sum_{j=1}^m\\nu_i\\omega_j(\\textbf{v}_i\\otimes \\textbf{w}_j)\n",
    "$$\n",
    "\n",
    "If $V$ is the vector space of $a\\times b$ matrices, then its basis vectors correspond to each of the $ab$ entries. If $W$ is the vector space of $c\\times d$ matrices, then its basis vectors correspond similarly to the $cd$ entries. In the tensor product, $(\\textbf{v}_i\\otimes \\textbf{w}_j)$ is the basis vector for an entry in the $ac\\times bd$ matrices that make up $V\\otimes W$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'numpy.ndarray'>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAAEICAYAAABoLY4BAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFepJREFUeJzt3W2sXdV95/HvLw6khUQJFMWYh7SEWDOlVXE6iCRTIKWB\nBNAQk5lOgVQEMRMRKogoolGpomkz7YtBiRKkTEksOoNCpQZIm1CsxoUCGuI+DBQTMYAh1IaQgLGh\nJpmkNM2A7/3Pi7NNj2/vtff1Ocdn75PvR1q6+2Htvdbx2f7fdddea+9UFZKk6XvNtCsgSRowIEtS\nRxiQJakjDMiS1BEGZEnqCAOyJHWEAVmSOsKAPEFJ7k3y3SSvm3Zd1D9Jnk7yT0leaq6jryY5dtr1\n0uQYkCckyU8BpwIFvH+qlVGfnVtVrwdWAc8D/33K9dEEGZAn50PAfcAXgIunWxX1XVX9EPgT4IRp\n10WT89ppV2CGfQj4DHA/cF+SlVX1/JTrpJ5KcghwPoNf8ppRBuQJSHIK8JPAl6pqZ5IngQ8C1023\nZuqhP02yCzgU+HvgfVOujybILovJuBj4i6ra2ax/EbsttH/Oq6o3AT8GXAF8LcmRU66TJsSAPGZJ\nfhz4FeDdSXYk2QFcBZyY5MTp1k59VVVzVfUVYA44Zdr10WQYkMfvPAb/aU4A1jTpp4G/ZNCvLC1b\nBtYChwGPT7s+moz4POTxSnIHsLmqrl6w/VeAzwLHVNWuqVROvZLkaWAlg1/wBXwL+G9V9UfTrJcm\nx4AsSR1hl4UkdYQBWdKPpCQ3JnkhyaNL7E+SzybZmuThJD8/tO+sJE80+64ZV50MyJJ+VH0BOGsv\n+88GVjfpUuDzAElWANc3+08ALkwylhmUBmRJP5KqaiPwnb1kWQv8YQ3cB7wpySrgZGBrVT1VVS8D\ntzR5R9bpmXrzO1b3/o7jJd8+ddpVGIubTr4x067D/pqF6+j4Wy+bdhXG4ptXXj3SdfS+0w+tF78z\n1yrvgw//v83AD4c23VBVNyyjuKOBZ4bWn222Lbb9Hcs475I6HZAladiL35njb+98S6u8K1Zt+WFV\nnTThKo2VAVlSbxQwz/yBKm4bMPz86WOabQctsX1k9iFL6o2ieKXmWqUxWA98qBlt8U7ge1W1HXgA\nWJ3kuCQHAxc0eUdmC1lSr4yrhZzkZuAXgSOSPAv8DoPWL1W1DtgAnANsBX4AXNLs25XkCuBOYAVw\nY1VtHkedDMiSeqMo5sY0u7iqLtzH/gIuX2LfBgYBe6wMyJJ6ZZ7eD5pZkgFZUm8UMGdAlqRusIUs\nSR1QwCsz/IRKA7Kk3ijKLgtJ6oSCudmNxwZkSf0xmKk3uwzIknokzNHb51ztkwFZUm8MbuoZkCVp\n6gbjkA3IktQJ87aQJWn6bCFLUkcUYW6GnxpsQJbUK3ZZSFIHFOHlWjHtakyMAVlSbwwmhthlIUmd\n4E09SeqAqjBXs9tCnt1PJmkmzZNWqY0kZyV5IsnWJNcssv9jSR5q0qNJ5pIc3ux7Oskjzb5N4/hs\ntpAl9cbgpt54wlaSFcD1wJnAs8ADSdZX1WOvllf1KeBTTf5zgauq6jtDpzm9qnaOpULYQpbUI7tv\n6rVJLZwMbK2qp6rqZeAWYO1e8l8I3Dz6p1iaAVlSr8xVWqUWjgaeGVp/ttn2LyQ5BDgL+PLQ5gLu\nTvJgkkv38+PswS4LSb2xzJl6Ryzo272hqm7Yz6LPBf56QXfFKVW1LcmbgbuSfKOqNu7n+QEDsqSe\nmW8/ymJnVZ20l/3bgGOH1o9pti3mAhZ0V1TVtubnC0luY9AFMlJAtstCUm8MHi70mlaphQeA1UmO\nS3Iwg6C7fmGmJG8E3g3cPrTt0CRv2L0MvBd4dNTPZwtZUm8U4ZUxTZ2uql1JrgDuBFYAN1bV5iSX\nNfvXNVk/APxFVf3j0OErgduSwCCOfrGq7hi1TgZkSb1RxVgnhlTVBmDDgm3rFqx/AfjCgm1PASeO\nrSINA7KkHmk/6aOPDMiSeqMYbwu5awzIknrFB9RLUgcU8QH1ktQFBbwypmdZdNHsfjJJMyg+D1mS\nuqBY1ky93jEgS+oVW8iS1AFVsYUsSV0wuKnnW6clqQNm+516BmRJvTG4qWcfsiR1gjP1JKkDnKkn\nSR3S8gWmvWRAltQbVfDKvAF5Kk69/CPTrsLIDrnt/mlXYTzmp12B/Xf8rZdNuwoje/L8dfvO1AtX\nj3T0oMvCgCxJneBMPUnqgFkf9ja7bX9JM2jQZdEmtTpbclaSJ5JsTXLNIvt/Mcn3kjzUpN9ue+z+\nsIUsqVfG9U69JCuA64EzgWeBB5Ksr6rHFmT9y6r6d/t57LIYkCX1xmCUxdieZXEysLV5gzRJbgHW\nAm2C6ijHLskuC0m9sXtiSJsEHJFk01C6dMHpjgaeGVp/ttm20L9N8nCSP0/yM8s8dllsIUvqlWV0\nWeysqpNGLO7rwFuq6qUk5wB/Cqwe8ZxLsoUsqTd2j7Jo2ULel23AsUPrxzTb/rm8qu9X1UvN8gbg\noCRHtDl2fxiQJfXKGEdZPACsTnJckoOBC4D1wxmSHJkkzfLJDGLmi22O3R92WUjqjaqwa0wz9apq\nV5IrgDuBFcCNVbU5yWXN/nXALwO/lmQX8E/ABVVVwKLHjlonA7KkXhnnxJCmG2LDgm3rhpZ/H/j9\ntseOyoAsqTdmfaaeAVlSrxiQJakDfEC9JHXIuKZOd5EBWVJvVMEuH1AvSd1gl4UkdYB9yJLUIWVA\nlqRu8KaeJHVAlX3IktQRYc5RFpLUDfYhS1IH+CwLSeqKGvQjzyoDsqRecZSFJHVAeVNPkrrDLgtJ\n6ohZHmUxu21/STOnahCQ26Q2kpyV5IkkW5Ncs8j+X03ycJJHkvxNkhOH9j3dbH8oyaZxfD5byJJ6\nZVzD3pKsAK4HzgSeBR5Isr6qHhvK9k3g3VX13SRnAzcA7xjaf3pV7RxLhTAgS+qZMfYhnwxsraqn\nAJLcAqwFXg3IVfU3Q/nvA44ZW+mLsMtCUm8UYX7+Na0ScESSTUPp0gWnOxp4Zmj92WbbUv4z8Od7\nVAfuTvLgIufeL7aQJfXKMhrIO6vqpHGUmeR0BgH5lKHNp1TVtiRvBu5K8o2q2jhKObaQJfXHeG/q\nbQOOHVo/ptm2hyQ/B/wPYG1VvfhqVaq2NT9fAG5j0AUyEgOypH6plmnfHgBWJzkuycHABcD64QxJ\n3gJ8Bbioqv5uaPuhSd6wexl4L/DoSJ8Luywk9cy4xiFX1a4kVwB3AiuAG6tqc5LLmv3rgN8GfgL4\nXBKAXU03yErgtmbba4EvVtUdo9bJgCypNwqYnx/fxJCq2gBsWLBt3dDyh4EPL3LcU8CJC7ePyoAs\nqT8KmOGZegZkSb3isywkqSsMyJLUBe2fU9FHBmRJ/WILWZI6oKDGOMqiawzIknrGgCxJ3WCXhSR1\nhAFZkjrAiSGS1B1ODJmSv/7yb8zur0IdMN+88uoZuI6unnYFusNRFpLUDbGFLEkd0P5Zx71kQJbU\nI/GmniR1hi1kSeqI+WlXYHJ8p94yJHltkseTbBnjOQ9LUkleatL3kvxJkkPGVYa6Y9Lf9ySu0U7Z\nPQ65TeohA/LyXAa8GXhr82LDRSV5Y5JPJnk0yTeSfC7JqiWyrwFerKrXV9XrgX8N/AJw0dhrry5o\n/X0v8zrardU12mepdqnVuZKzkjyRZGuSaxbZnySfbfY/nOTn2x67PwzILSV5I/A7wBXAHPCze8l3\nL/AD4H3AqcDjwL1Jjl3kkDXA5t0rVbUdeAY4aIzVV3e0+r734zpqfY323pjeOp1kBXA9cDZwAnBh\nkhMWZDsbWN2kS4HPL+PYZTMgt/dfgKeq6mbg74CfWyLfxxm8Svxc4DgGb6z9CIMv8tOL5H87zX/Q\n5s/N84FjgS+Ntfbqirbf93KvI2h/jWrgZGBrVT1VVS8DtwBrF+RZC/xhDdwHvKn5K6XNscvmTb0W\nkrwV+CiDlgoM/kMtdbG/j8Fvza8BXwZ2Ab8J3Ar81iL51wBvS/JB4BDgh8B5VfXC2D6AuqTt972s\n62iZ12ivLWNiyBFJNg2t31BVNwytH83gr5PdngXeseAci+U5uuWxy2ZAbueTwN1VdW+zvhk4Y4m8\nrwVeBrYz+LPxYOBbzfIekrwO+GngXVW1qfkz6DeBmxi0mjRDlvl9t76OGsu5RvurWM7U6Z1VddIE\nazN2BuR9SHIK8B+A7yfZ0Wz+cZYefPM14IPArzF4AMEO4GYGrZoHF+T92eY8DwNU1VySrwGfSHJQ\nVb0yzs+iqVvO9936OtqPa7TfxjcOeRt7/iI8ptnWJs9BLY5dNgPyXiQJ8BlgHfBfh3b9JHBfkrdU\n1bcXHPa7wEbgc8AfM/hP8VvAJ4D3L8j7duCRpg+KJEfR9B0ajGfScr7vVtfRfl6jvTbGZ1k8AKxO\nchyDYHoBg1+Cw9YDVyS5hUGXxPeqanuSv29x7LIZkPfuV4GVwMeq6qXdG5M8D/wDgz66PS72qtqR\n5JeATwG7h8I8AJxbVY8tOP8aYE2Slxj8Kfoig/7C353AZ9H0tf6+l3EdLfsa7b0xBeSq2pXkCuBO\nYAVwY1VtTnJZs38dsAE4B9jKYMTLJXs7dtQ6pWb54aKSZsrrjj22jrnyqlZ5n/rY1Q/ahyxJE7Kc\nSR99ZECW1C8+oF6SusEWsiR1hQF5OuZ3rJ74P/3xt1420fOf9q6Rb7zu08b//TMTL6PP76U7ENfR\nJd8+daLnPxDf8YEw8nVkH7IkdYgBWZK6IbM5/xDwaW+S1Bm2kCX1i10WktQB3tSTpA4xIEtSRxiQ\nJWn6wmyPsjAgS+oP+5AlqUMMyJLUEQZkSeoGuywkqStmOCA7dVpSf9RglEWbNIokhye5K8mW5udh\ni+Q5Nsn/SvJYks1Jrhza94kk25I81KRz2pRrQJbUL9UyjeYa4J6qWg3cwz+/aHbYLuDqqjoBeCdw\neZIThvZfV1VrmrShTaEGZEm9svu9evtKI1oL3NQs3wSctzBDVW2vqq83y/8APA4cPUqhBmRJ/dK+\nhXxEkk1D6dJllLKyqrY3yzuAlXvLnOSngLcD9w9t/miSh5PcuFiXx2K8qSepP5bXHbGzqk5aameS\nu4EjF9n18T2KrKpk6TZ3ktcDXwZ+vaq+32z+PPB7TW1/D/g08J/2VWEDsqTeCOMb9lZVZyxZTvJ8\nklVVtT3JKuCFJfIdxCAY/1FVfWXo3M8P5fkD4M/a1MkuC0m9coD6kNcDFzfLFwO3/4t6JAH+J/B4\nVX1mwb5VQ6sfAB5tU6gBWVK/HJhRFtcCZybZApzRrJPkqCS7R0z8AnAR8EuLDG/7ZJJHkjwMnA5c\n1aZQuywk9csBmBhSVS8C71lk+3PAOc3yXzHoRVns+Iv2p1wDsqT+8GlvktQhBuTpOP7WyyZexpPn\nr5vo+Q/EZzjtXZsnXkafnXr5RyZexiG33b/vTKO4brKn7xMfUC9JHWGXhSR1wXhGUHSWAVlSvxiQ\nJWn6xjlTr4sMyJJ6JfOzG5ENyJL6wz5kSeoOuywkqSsMyJLUDbaQJakrDMiS1AHl1GlJ6gTHIUtS\nl9TsRmQDsqResYUsSV0w4xNDfKeepF7JfLs0UhnJ4UnuSrKl+XnYEvmebt6d91CSTcs9fiEDsqRe\nORABGbgGuKeqVgP3NOtLOb2q1lTVSft5/KsMyJL6oxjc1GuTRrMWuKlZvgk470Acb0CW1Cupdgk4\nIsmmoXTpMopZWVXbm+UdwMol8hVwd5IHF5y/7fF78KaepH5p3/jduaAbYQ9J7gaOXGTXx/corqqS\nJcd2nFJV25K8GbgryTeqauMyjt+DAVlSb4xzYkhVnbFkOcnzSVZV1fYkq4AXljjHtubnC0luA04G\nNgKtjl/ILgtJ/VFF5tulEa0HLm6WLwZuX5ghyaFJ3rB7GXgv8Gjb4xdjQJbUL9UyjeZa4MwkW4Az\nmnWSHJVkQ5NnJfBXSf4P8LfAV6vqjr0dvy+d7rI4auPkR4Afz2UTPf/brrpvoucH2HjdOydeBidP\nvohJee60TL6Q0yb7HTx5/rqJnh/gkm+fOvEyxuFAzNSrqheB9yyy/TngnGb5KeDE5Ry/L50OyJK0\nhwJ8p54kdcTsxmMDsqR+8eFCktQRYxhB0VkGZEn9MeNPezMgS+qNwcSQ2Y3IBmRJ/eI79SSpG2wh\nS1IX2IcsSV0xludUdJYBWVK/2GUhSR1QY3k9U2cZkCX1iy1kSeqI2Y3HBmRJ/ZL52e2zMCBL6o/C\niSGS1AWhnBgiSZ0xwwHZd+pJ6peqdmkESQ5PcleSLc3PwxbJ86+SPDSUvp/k15t9n0iybWjfOW3K\nNSBL6o/dfcht0miuAe6pqtXAPc36nlWpeqKq1lTVGuDfAD8AbhvKct3u/VW1YeHxizEgS+qVzM+3\nSiNaC9zULN8EnLeP/O8Bnqyqb41SqAFZUo+07K4YvZ95ZVVtb5Z3ACv3kf8C4OYF2z6a5OEkNy7W\n5bEYA7Kk/iiWE5CPSLJpKF06fKokdyd5dJG0do8iq/b6jLkkBwPvB/54aPPngbcCa4DtwKfbfDxH\nWUjql/a9ETur6qSldlbVGUvtS/J8klVVtT3JKuCFvZRzNvD1qnp+6NyvLif5A+DP2lS40wH5udMy\n8TKO2jjZITRbr3vnRM8P8Lar7pt4GVw5+SK0tFMv/8jEyzjktvsnXsY4JnUcoHHI64GLgWubn7fv\nJe+FLOiu2B3Mm9UPAI+2KdQuC0n9cmD6kK8FzkyyBTijWSfJUUleHTGR5FDgTOArC47/ZJJHkjwM\nnA5c1abQTreQJWkPVTA3+bnTVfUig5ETC7c/B5wztP6PwE8sku+i/SnXgCypX2Z4pp4BWVK/GJAl\nqQMK8J16ktQFBTW7z980IEvqj+KA3NSbFgOypH6xD1mSOsKALEldMJZJH51lQJbUHwX4klNJ6ghb\nyJLUBQdm6vS0GJAl9UdBOQ5ZkjrCmXqS1BH2IUtSB1Q5ykKSOsMWsiR1QVFzc9OuxMQYkCX1h4/f\nlKQOmeFhb77kVFJvFFDz1SqNIsl/TLI5yXySk/aS76wkTyTZmuSaoe2HJ7kryZbm52FtyjUgS+qP\nah5Q3yaN5lHg3wMbl8qQZAVwPXA2cAJwYZITmt3XAPdU1WrgnmZ9nwzIknql5uZapZHKqHq8qp7Y\nR7aTga1V9VRVvQzcAqxt9q0FbmqWbwLOa1NuaoaHkEiaLUnuAI5omf3HgB8Ord9QVTcss7x7gd+o\nqk2L7Ptl4Kyq+nCzfhHwjqq6Isn/rao3NdsDfHf3+t54U09Sb1TVWeM6V5K7gSMX2fXxqrp9XOVU\nVSVp1fI1IEv6kVRVZ4x4im3AsUPrxzTbAJ5PsqqqtidZBbzQ5oT2IUvS/nkAWJ3kuCQHAxcA65t9\n64GLm+WLgVYtbgOyJC2Q5ANJngXeBXw1yZ3N9qOSbACoql3AFcCdwOPAl6pqc3OKa4Ezk2wBzmjW\n912uN/UkqRtsIUtSRxiQJakjDMiS1BEGZEnqCAOyJHWEAVmSOsKALEkdYUCWpI4wIEtSRxiQJakj\nDMiS1BEGZEnqCAOyJHWEAVmSOsKALEkdYUCWpI4wIEtSRxiQJakj/j/w569CNayTdgAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8b8af707b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Kronecker demo\n",
    "\n",
    "A = np.array([[1, 1/2], [-1/2, -1]])\n",
    "B = np.identity(2)\n",
    "\n",
    "f, axs = plt.subplots(2, 2)\n",
    "\n",
    "# Guess what a 2x2 axes subplot type is?\n",
    "print(type(axs))\n",
    "# Use of numpy for convenience: arbitrary object flattening\n",
    "for ax in axs.ravel():\n",
    "    ax.axis('off')\n",
    "    \n",
    "ax1, ax2, ax3, ax4 = axs.ravel()\n",
    "\n",
    "ax1.imshow(A, vmin=-1, vmax=1)\n",
    "ax1.set_title('A')\n",
    "ax2.imshow(B, vmin=-1, vmax=1)\n",
    "ax2.set_title('B')\n",
    "ax3.imshow(np.kron(A, B), vmin=-1, vmax=1)\n",
    "ax3.set_title(r'$A\\otimes B$')\n",
    "im = ax4.imshow(np.kron(B, A), vmin=-1, vmax=1)\n",
    "ax4.set_title(r'$B\\otimes A$')\n",
    "\n",
    "f.colorbar(im, ax=axs.ravel().tolist())\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "diff 1.5526505380777123e-12\n",
      "Kronecker matrix vector multiplication\n",
      "    format: mean seconds (standard error) 5 runs\n",
      "    kron_mvm   7.13e-04 (2.49e-05)\n",
      "    saatci_mvm 4.77e-05 (2.55e-06)\n",
      "    improvement ratio 14.9\n"
     ]
    }
   ],
   "source": [
    "# Transposition demo: using transpose, you can compute\n",
    "\n",
    "A = np.random.randn(40, 40)\n",
    "B = np.random.randn(40, 40)\n",
    "AB = np.kron(A, B)\n",
    "z = np.random.randn(40 * 40)\n",
    "\n",
    "def kron_mvm():\n",
    "    return AB.dot(z)\n",
    "\n",
    "def saatci_mvm():\n",
    "    # This differs from the paper's MVM, but is the equivalent for\n",
    "    # a C-style ordering of arrays.\n",
    "    x = z.copy()\n",
    "    for M in [B, A]:\n",
    "        n = M.shape[1]\n",
    "        x = x.reshape(-1, n).T\n",
    "        x = M.dot(x)\n",
    "    return x.ravel()\n",
    "\n",
    "print('diff', np.linalg.norm(kron_mvm() - saatci_mvm()))\n",
    "print('Kronecker matrix vector multiplication')\n",
    "compare_times(kron_mvm, saatci_mvm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ufuncs and Broadcasting\n",
    "\n",
    "[doc](https://docs.scipy.org/doc/numpy-dev/reference/ufuncs.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 2 3 4 5]\n",
      "[1 1 1 2 2 2]\n",
      "[1 2 3 5 6 7]\n",
      "[1 2 3 5 6 7]\n"
     ]
    }
   ],
   "source": [
    "# A ufunc is the most common way to modify arrays\n",
    "\n",
    "# In its simplest form, an n-ary ufunc takes in n numpy arrays\n",
    "# of the same shape, and applies some standard operation to \"parallel elements\"\n",
    "\n",
    "a = np.arange(6)\n",
    "b = np.repeat([1, 2], 3)\n",
    "print(a)\n",
    "print(b)\n",
    "print(a + b)\n",
    "print(np.add(a, b))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A (2, 3)\n",
      "[[0 1 2]\n",
      " [3 4 5]]\n",
      "\n",
      "b (2,)\n",
      "[0 1]\n",
      "\n",
      "c (3,)\n",
      "[0 1 2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# If any of the arguments are of lower dimension, they're prepended with 1\n",
    "# Any arguments that have dimension 1 are repeated along that axis\n",
    "\n",
    "A = np.arange(2 * 3).reshape(2, 3)\n",
    "b = np.arange(2)\n",
    "c = np.arange(3)\n",
    "for i in ['A', 'b', 'c']:\n",
    "    display(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A * c (2, 3)\n",
      "[[ 0  1  4]\n",
      " [ 0  4 10]]\n",
      "\n",
      "c.reshape(1, 3) (1, 3)\n",
      "[[0 1 2]]\n",
      "\n",
      "np.repeat(c.reshape(1, 3), 2, axis=0) (2, 3)\n",
      "[[0 1 2]\n",
      " [0 1 2]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# On the right, broadcasting rules will automatically make the conversion\n",
    "# of c, which has shape (3,) to shape (1, 3)\n",
    "display('A * c')\n",
    "display('c.reshape(1, 3)')\n",
    "display('np.repeat(c.reshape(1, 3), 2, axis=0)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "np.diag(c) (3, 3)\n",
      "[[0 0 0]\n",
      " [0 1 0]\n",
      " [0 0 2]]\n",
      "\n",
      "A.dot(np.diag(c)) (2, 3)\n",
      "[[ 0  1  4]\n",
      " [ 0  4 10]]\n",
      "\n",
      "A * c (2, 3)\n",
      "[[ 0  1  4]\n",
      " [ 0  4 10]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "display('np.diag(c)')\n",
    "display('A.dot(np.diag(c))')\n",
    "display('A * c')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# GOTCHA: this won't compile your code to C: it will just make a slow convenience wrapper\n",
    "demo = np.frompyfunc('f({}, {})'.format, 2, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A (2, 3)\n",
      "[[0 1 2]\n",
      " [3 4 5]]\n",
      "\n",
      "b (2,)\n",
      "[0 1]\n",
      "\n",
      "ValueError!\n",
      "operands could not be broadcast together with shapes (2,3) (2,) \n"
     ]
    }
   ],
   "source": [
    "# GOTCHA: common broadcasting mistake -- append instead of prepend\n",
    "display('A')\n",
    "display('b')\n",
    "try:\n",
    "    demo(A, b) # can't prepend to (2,) with 1 to get something compatible with (2, 3)\n",
    "except ValueError as e:\n",
    "    print('ValueError!')\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b[:, np.newaxis] (2, 1)\n",
      "[[0]\n",
      " [1]]\n",
      "\n",
      "np.repeat(b[:, np.newaxis], 3, axis=1) (2, 3)\n",
      "[[0 0 0]\n",
      " [1 1 1]]\n",
      "\n",
      "demo(A, b[:, np.newaxis]) (2, 3)\n",
      "[['f(0, 0)' 'f(1, 0)' 'f(2, 0)']\n",
      " ['f(3, 1)' 'f(4, 1)' 'f(5, 1)']]\n",
      "\n",
      "demo(b[:, np.newaxis], A) (2, 3)\n",
      "[['f(0, 0)' 'f(0, 1)' 'f(0, 2)']\n",
      " ['f(1, 3)' 'f(1, 4)' 'f(1, 5)']]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# np.newaxis adds a 1 in the corresponding axis\n",
    "display('b[:, np.newaxis]')\n",
    "display('np.repeat(b[:, np.newaxis], 3, axis=1)')\n",
    "display('demo(A, b[:, np.newaxis])')\n",
    "# note broadcasting rules are invariant to order\n",
    "# even if the ufunc isn't \n",
    "display('demo(b[:, np.newaxis], A)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b (2,)\n",
      "[0 1]\n",
      "\n",
      "np.diag(b) (2, 2)\n",
      "[[0 0]\n",
      " [0 1]]\n",
      "\n",
      "b[:, np.newaxis] * A (2, 3)\n",
      "[[0 0 0]\n",
      " [3 4 5]]\n",
      "\n",
      "np.diag(b).dot(A) (2, 3)\n",
      "[[0 0 0]\n",
      " [3 4 5]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Using broadcasting, we can do cheap diagonal matrix multiplication\n",
    "display('b')\n",
    "display('np.diag(b)')\n",
    "# without representing the full diagonal matrix.\n",
    "display('b[:, np.newaxis] * A')\n",
    "display('np.diag(b).dot(A)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "demo.outer(a, b) (4, 4)\n",
      "[['f(0, 4)' 'f(0, 5)' 'f(0, 6)' 'f(0, 7)']\n",
      " ['f(1, 4)' 'f(1, 5)' 'f(1, 6)' 'f(1, 7)']\n",
      " ['f(2, 4)' 'f(2, 5)' 'f(2, 6)' 'f(2, 7)']\n",
      " ['f(3, 4)' 'f(3, 5)' 'f(3, 6)' 'f(3, 7)']]\n",
      "\n",
      "np.bitwise_or.accumulate(b) (4,)\n",
      "[4 5 7 7]\n",
      "\n",
      "np.bitwise_or.reduce(b) ()\n",
      "7\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# (Binary) ufuncs get lots of efficient implementation stuff for free\n",
    "a = np.arange(4)\n",
    "b = np.arange(4, 8)\n",
    "display('demo.outer(a, b)')\n",
    "display('np.bitwise_or.accumulate(b)')\n",
    "display('np.bitwise_or.reduce(b)') # last result of accumulate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accumulation speed comparison\n",
      "    format: mean seconds (standard error) 5 runs\n",
      "    manual_accum 2.47e-01 (6.68e-03)\n",
      "    np_accum     1.59e-03 (1.32e-05)\n",
      "    improvement ratio 155.2\n"
     ]
    }
   ],
   "source": [
    "def setup(): return np.arange(10 ** 6)\n",
    "\n",
    "def manual_accum(x):\n",
    "    res = np.zeros_like(x)\n",
    "    for i, v in enumerate(x):\n",
    "        res[i] = res[i-1] | v\n",
    "        \n",
    "def np_accum(x):\n",
    "    np.bitwise_or.accumulate(x)\n",
    "    \n",
    "print('accumulation speed comparison')\n",
    "compare_times(manual_accum, np_accum, setup, setup)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Aliasing\n",
    "\n",
    "You can save on allocations and copies by providing the output array to copy into.\n",
    "\n",
    "**Aliasing** occurs when all or part of the input is repeated in the output\n",
    "\n",
    "[Ufuncs allow aliasing](https://github.com/numpy/numpy/pull/8043)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[3 5 0]\n",
      " [7 7 9]\n",
      " [4 0 8]]\n",
      "[[ 6 12  4]\n",
      " [12 14  9]\n",
      " [ 4  9 16]]\n"
     ]
    }
   ],
   "source": [
    "# Example: generating random symmetric matrices\n",
    "A = np.random.randint(0, 10, size=(3,3))\n",
    "print(A)\n",
    "A += A.T # this operation is WELL-DEFINED, even though A is changing\n",
    "print(A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[12, 24,  8],\n",
       "       [24, 28, 18],\n",
       "       [ 8, 18, 32]])"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Above is sugar for\n",
    "np.add(A, A, out=A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 2 3 4 5 6 7 8 9]\n",
      "[-5 -5 -5 -5 -5  5  6  7  8  9]\n"
     ]
    }
   ],
   "source": [
    "x = np.arange(10)\n",
    "print(x)\n",
    "np.subtract(x[:5], x[5:], x[:5])\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[GOTCHA]: If it's not a ufunc, aliasing is VERY BAD**: [Search for \"In general the rule\" in this discussion](https://github.com/numpy/numpy/issues/8440). Ufunc aliasing is safe since [this pr](https://github.com/numpy/numpy/pull/8043)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output array is not acceptable (must have the right datatype, number of dimensions, and be a C-Array)\n"
     ]
    }
   ],
   "source": [
    "x = np.arange(2 * 2).reshape(2, 2)\n",
    "try:\n",
    "    x.dot(np.arange(2), out=x)\n",
    "    # GOTCHA: some other functions won't warn you!\n",
    "except ValueError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Configuration and Hardware Acceleration\n",
    "\n",
    "NumPy works quickly because it _can_ perform vectorization by linking to C functions that were built for **your** particular system.\n",
    "\n",
    "**[GOTCHA] There are two different high-level ways in which NumPy uses hardware to accelerate your computations**.\n",
    "\n",
    "### Ufunc\n",
    "\n",
    "When you perform a built-in ufunc:\n",
    "* The corresponding C function is called directly from the Python interpreter\n",
    "* It is not **parallelized**\n",
    "* It may be **vectorized**\n",
    "\n",
    "In general, it is tough to check whether your code is using vectorized instructions (or, in particular, which instruction set is being used, like SSE or AVX512.\n",
    "\n",
    "* If you installed from pip or Anaconda, you're probably not vectorized.\n",
    "* If you compiled NumPy yourself (and select the correct flags), you're probably fine.\n",
    "* If you're using the [Numba](http://numba.pydata.org/) JIT, then you'll be vectorized too.\n",
    "* If have access to `icc` and MKL, then you can use [the Intel guide](https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl) or [Anaconda](https://docs.continuum.io/mkl-optimizations/)\n",
    "\n",
    "### BLAS\n",
    "\n",
    "These are optimized linear algebra routines, and are only called when you invoke operations that rely on these routines.\n",
    "\n",
    "This won't make your vectors add faster (first, NumPy doesn't ask BLAS to nor could it, since bandwidth-limited ops are not the focus of BLAS). It will help with:\n",
    "* Matrix multiplication (`np.dot`)\n",
    "* Linear algebra (SVD, eigenvalues, etc) (`np.linalg`)\n",
    "* Similar stuff from other libraries that accept NumPy arrays may use BLAS too.\n",
    "\n",
    "There are different implementations for BLAS. Some are free, and some are proprietary and built for specific chips (MKL). You can check which version you're using [this way](http://stackoverflow.com/questions/9000164/how-to-check-blas-lapack-linkage-in-numpy-scipy), though you can only be sure [by inspecting the binaries manually](http://stackoverflow.com/questions/37184618/find-out-if-which-blas-library-is-used-by-numpy).\n",
    "\n",
    "Any NumPy routine that uses **BLAS** will use, by default **ALL AVAILABLE CORES**. This is a departure from the standard parallelism of `ufunc` or other numpy transformations. You can change BLAS parallelism with the `OMP_NUM_THREADS` environment variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stuff to Avoid\n",
    "\n",
    "NumPy has some cruft left over due to backwards compatibility. There are some edge cases when you would (maybe) use these things (but probably not). In general, avoid them:\n",
    "\n",
    "* `np.chararray`: use an `np.ndarray` with `unicode` `dtype`\n",
    "* `np.MaskedArrays`: use a boolean advanced index\n",
    "* `np.matrix`: use a 2-dimensional `np.ndarray`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stuff Not Mentioned\n",
    "\n",
    "* [General array manipulation](https://docs.scipy.org/doc/numpy-dev/reference/routines.array-manipulation.html)\n",
    "    * Selection-related convenience methods `np.sort, np.unique`\n",
    "    * Array composition and decomposition `np.split, np.stack`\n",
    "    * Reductions many-to-1 `np.sum, np.prod, np.count_nonzero`\n",
    "    * Many-to-many array transformations `np.fft, np.linalg.cholesky`\n",
    "* [String formatting](https://docs.scipy.org/doc/numpy/reference/generated/numpy.array2string.html) `np.array2string`\n",
    "* [IO](https://docs.scipy.org/doc/numpy/reference/routines.io.html#string-formatting) `np.loadtxt, np.savetxt`\n",
    "* [Polynomial interpolation](https://docs.scipy.org/doc/numpy/reference/routines.polynomials.html) and related [scipy integration](https://docs.scipy.org/doc/scipy/reference/tutorial/integrate.html)\n",
    "* [Equality testing](https://docs.scipy.org/doc/numpy/reference/routines.testing.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Takeaways\n",
    "\n",
    "* Use NumPy arrays for a compact, cache-friendly, in-memory representation of structured numeric data.\n",
    "* Vectorize, vectorize, vectorize! Less loops!\n",
    "   * Expressive\n",
    "   * Fast\n",
    "   * Concise\n",
    "* Know when copies happen vs. when views happen\n",
    "   * Advanced indexing -> copy\n",
    "   * Basic indexing -> view\n",
    "   * Transpositions -> usually view (depends if memory order changes)\n",
    "   * Ufuncs/many-to-many -> copy (possibly with overwrite\n",
    "* Rely on powerful indexing API to avoid almost all Python loops\n",
    "* Rolling your own algorithm? Google it, NumPy probably has it built-in!\n",
    "* Be concious of what makes copies, and what doesn't\n",
    "\n",
    "Downsides. Can't optimize across NumPy ops (like a C compiler would/numpy would). But do you need that? Can't parallelize except BLAS, but is it computaitonal or memory bandwidth limited?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cherry on Top: Einsum\n",
    "\n",
    "[doc](https://docs.scipy.org/doc/numpy/reference/generated/numpy.einsum.html)\n",
    "\n",
    "Recall the Kronecker product $\\otimes$ from before? Let's recall the fully general tensor product.\n",
    "\n",
    "If $V$ has basis $\\textbf{v}_i$ and $W$ has $\\textbf{w}_j$, we can define the tensor product over elements $\\nu\\in V,\\omega\\in W$ as follows.\n",
    "\n",
    "Let $\\nu= \\sum_{i=1}^n\\nu_i\\textbf{v}_i$ and $\\omega= \\sum_{j=1}^m\\omega_j\\textbf{w}_j$. Then:\n",
    "$$\n",
    "V \\otimes W\\ni \\nu\\otimes \\omega=\\sum_{i=1}^n\\sum_{j=1}^m\\nu_i\\omega_j(\\textbf{v}_i\\otimes \\textbf{w}_j)\n",
    "$$\n",
    "\n",
    "But what if $V$ is itself a tensor space, like a matrix space $F^{m\\times n}$, and $W$ is $F^{n\\times k}$. Then $\\nu\\otimes\\omega$ is a tensor with shape $(m, n, n, k)$, where the $(i_1, i_2,i_3,i_4)$-th element is given by $\\nu_{i_1i_2}\\omega_{i_3i_4}$ (the corresponding cannonical basis vector being $\\textbf{e}^{(m)}_{i_1}(\\textbf{e}^{(n)}_{i_2})^\\top\\otimes \\textbf{e}^{(n)}_{i_3}(\\textbf{e}^{(k)}_{i_4})^\\top$, where $\\textbf{e}^{(m)}_{i_1}(\\textbf{e}^{(n)}_{i_2})^\\top$, the cannonical matrix basis vector, is not that scary - here's an example in $2\\times 3$:\n",
    "$$\n",
    "\\textbf{e}^{(2)}_{1}(\\textbf{e}^{(3)}_{2})^\\top=\\begin{pmatrix} 0 & 1 & 0\\\\ 0 & 0 & 0 \\end{pmatrix}\n",
    "$$\n",
    "\n",
    "What happens if we **contract** along the second and third axis, both of which have length $n$, where **contraction** in this example builds a tensor with shape $(m, k)$ such that the $(i_1,i_4)$-th entry is the sum of all entries in the tensor product $\\nu\\otimes \\omega$ which have the same values $i_2=i_3$. In other words:\n",
    "$$\n",
    "[\\text{contract}_{12}(\\nu\\otimes\\omega)]_{i_1i_4}=\\sum_{i_2=1}^n(\\nu\\otimes\\omega)_{i_1,i_2,i_2,i_4}=\\sum_{i_2=1}^n\\nu_{i_1i_2}\\omega_{i_2,i_4}\n",
    "$$\n",
    "\n",
    "Does that last term look familiar? It is, it's the matrix product! Indeed, **a matrix product is a generalized trace of the outer product of two compatible matrices**.\n",
    "\n",
    "That's one way of thinking about einsum: it lets you do generalized matrix products; in that you take in an arbitrary number of matrices, compute their outer product, and then specify which axes to trace. But then it also lets you arbitrarily transpose and select diagonal elements of your tensors, too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[ 5  9]\n",
      "  [-4  2]]\n",
      "\n",
      " [[10  5]\n",
      "  [ 7 -1]]]\n"
     ]
    }
   ],
   "source": [
    "# Great resources to learn einsum:\n",
    "# https://obilaniu6266h16.wordpress.com/2016/02/04/einstein-summation-in-numpy/\n",
    "# http://ajcr.net/Basic-guide-to-einsum/\n",
    "\n",
    "# Examples of how it's general:\n",
    "\n",
    "np.random.seed(1234)\n",
    "x = np.random.randint(-10, 11, size=(2, 2, 2))\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[ 5 10]\n",
      "  [-4  7]]\n",
      "\n",
      " [[ 9  5]\n",
      "  [ 2 -1]]]\n"
     ]
    }
   ],
   "source": [
    "# Swap axes\n",
    "print(np.einsum('ijk->kji', x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "33 33\n"
     ]
    }
   ],
   "source": [
    "# Sum [contraction is along every axis]\n",
    "print(x.sum(), np.einsum('ijk->', x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Multiply (pointwise) [take the diagonal of the outer product; don't sum]\n",
    "y = np.random.randint(-10, 11, size=(2, 2, 2))\n",
    "np.array_equal(x * y, np.einsum('ijk,ijk->ijk', x, y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Already, an example where einsum is more clear: multiply pointwise along different axes:\n",
    "print(np.array_equal(x * y.transpose(), np.einsum('ijk,kji->ijk', x, y)))\n",
    "print(np.array_equal(x * np.rollaxis(y, 2), np.einsum('ijk,jki->ijk', x, y)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Outer (tensor) product\n",
    "x = np.arange(4)\n",
    "y = np.arange(4, 8)\n",
    "np.array_equal(np.outer(x, y), np.einsum('i,j->ij', x, y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14.0 14\n"
     ]
    }
   ],
   "source": [
    "# Arbitrary inner product\n",
    "a = np.arange(2 * 2).reshape(2, 2)\n",
    "print(np.linalg.norm(a, 'fro') ** 2, np.einsum('ij,ij->', a, a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(1234)\n",
    "x = np.random.randn(2, 2)\n",
    "y = np.random.randn(2, 2)\n",
    "\n",
    "# Matrix multiply\n",
    "print(np.array_equal(x.dot(y), np.einsum('ij,jk->ik', x, y)))\n",
    "\n",
    "# Batched matrix multiply\n",
    "x = np.random.randn(3, 2, 2)\n",
    "y = np.random.randn(3, 2, 2)\n",
    "print(np.array_equal(\n",
    "    np.array([i.dot(j) for i, j in zip(x, y)]),\n",
    "    np.einsum('bij,bjk->bik', x, y)))\n",
    "\n",
    "# all of {np.matmul, np.tensordot, np.dot} are einsum instances\n",
    "# The specializations may have marginal speedups, but einsum is\n",
    "# more expressive and clear code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### General Einsum Approach\n",
    "\n",
    "Again, lots of visuals in [this blog post](https://obilaniu6266h16.wordpress.com/2016/02/04/einstein-summation-in-numpy/).\n",
    "\n",
    "**[GOTCHA]**. [You can't use more than 52 different letters.](http://stackoverflow.com/questions/37794245/can-i-use-more-than-26-letters-in-numpy-einsum). But if you find yourself writing `np.einsum` with more than 52 active dimensions, you should probably make two `np.einsum` calls. If you have dimensions for which nothing happens, then `...` can be used to represent an arbitrary amount of missed dimensions.\n",
    "\n",
    "Here's the way I think about an `np.einsum` (the actual implementation is more efficient)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Let the contiguous blocks of letters be words\n",
    "# If they're on the left, they're argument words. On the right, result words.\n",
    "\n",
    "np.random.seed(1234)\n",
    "x = np.random.randint(-10, 11, 3 * 2 * 2 * 1).reshape(3, 2, 2, 1)\n",
    "y = np.random.randint(-10, 11, 3 * 2 * 2).reshape(3, 2, 2)\n",
    "z = np.random.randint(-10, 11, 2 * 3).reshape(2, 3)\n",
    "\n",
    "# Example being followed in einsum description:\n",
    "# np.einsum('ijkm,iko,kp->mip', x, y, z)\n",
    "\n",
    "# 1. Line up each argument word with the axis of the array.\n",
    "#    Make sure that word length == dimension\n",
    "#    Make sure same letters correspond to same lengths\n",
    "# x.shape (3, 2, 2, 1)\n",
    "#          i  j  k  m\n",
    "# y.shape (3, 2, 2)\n",
    "#          i  k  o\n",
    "# z.shape (2, 3)\n",
    "#          k  p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3, 2, 2, 1, 3, 2, 2, 2, 3)\n",
      "(i  j  k  m  i  k  o  k  p)\n"
     ]
    }
   ],
   "source": [
    "# 2. Create the complete tensor product\n",
    "outer = np.tensordot(np.tensordot(x, y, axes=0), z, axes=0)\n",
    "print(outer.shape)\n",
    "print('(i  j  k  m  i  k  o  k  p)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3, 2, 2, 1, 2, 2, 2, 3)\n",
      "(i  j  k  m  k  o  k  p)\n",
      "(3, 2, 2, 1, 2, 3)\n",
      "(i  j  k  m  o  p)\n"
     ]
    }
   ],
   "source": [
    "# 3. Every time a letter repeats, only look at the corresponding \"diagonal\" elements.\n",
    "\n",
    "# Repeat i: (i  j  k  m  i  k  o  k  p)\n",
    "#           (i           i            )\n",
    "# Expected: (i  j  k  m  k  o  k  p)\n",
    "\n",
    "# The expected index corresponds to the above index in the outer product\n",
    "# We can do this over all other values with two advanced indices\n",
    "span_i = np.arange(3)\n",
    "repeat_i = outer[span_i, :, :, :, span_i, ...] # ellipses means \"fill with :\"\n",
    "print(repeat_i.shape)\n",
    "print('(i  j  k  m  k  o  k  p)')\n",
    "\n",
    "# Repeat k: (i  j  k  m  k  o  k  p)\n",
    "#           (      k     k     k   )\n",
    "# Expected: (i  j  k  m  o  p)\n",
    "span_k = np.arange(2)\n",
    "repeat_k = repeat_i[:, :, span_k, :, span_k, :, span_k, :]\n",
    "# GOTCHA: advanced indexing brings shared advanced index to front, fixed with rollaxis\n",
    "repeat_k = np.rollaxis(repeat_k, 0, 2)\n",
    "print(repeat_k.shape)\n",
    "print('(i  j  k  m  o  p)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3, 2, 1, 2, 3)\n",
      "(i  k  m  o  p)\n",
      "(3, 1, 2, 3)\n",
      "(i  m  o  p)\n",
      "(3, 1, 3)\n",
      "(i  m  p)\n"
     ]
    }
   ],
   "source": [
    "# 4. Compare the remaining word to the result word; sum out missing letters\n",
    "\n",
    "# Result word:  (m  i  p)\n",
    "# Current word: (i  j  k  m  o  p)\n",
    "\n",
    "# Sum out j:    (i  k  m  o  p)\n",
    "# The resulting array has at entry (i k m o p) the following:\n",
    "# (i 0 k m o p) + (i 1 k m o p) + ... + (i [axis j length] k m o p)\n",
    "sumj = repeat_k.sum(axis=1)\n",
    "print(sumj.shape)\n",
    "print('(i  k  m  o  p)')\n",
    "\n",
    "# Sum out k:    (i  m  o  p)\n",
    "sumk = sumj.sum(axis=1)\n",
    "print(sumk.shape)\n",
    "print('(i  m  o  p)')\n",
    "\n",
    "# Sum out o:    (i  m  p)\n",
    "sumo = sumk.sum(axis=2)\n",
    "print(sumo.shape)\n",
    "print('(i  m  p)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[  808 -1006 -1139]\n",
      "  [-1672   228   552]\n",
      "  [  806    35  -125]]]\n",
      "[[[  808 -1006 -1139]\n",
      "  [-1672   228   552]\n",
      "  [  806    35  -125]]]\n"
     ]
    }
   ],
   "source": [
    "# 6. Transpose remaining word until it has the same order as the result word\n",
    "\n",
    "# (i  m  p) -> (m  i  p)\n",
    "print(np.moveaxis(sumo, [0, 1, 2], [1, 0, 2]))\n",
    "print(np.einsum('ijkm,iko,kp->mip', x, y, z))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Neural Nets with Einsum\n",
    "\n",
    "[Original post](https://obilaniu6266h16.wordpress.com/2016/02/04/einstein-summation-in-numpy/)\n",
    "\n",
    "<table>\n",
    "<tr>\n",
    "<th>\n",
    "<img src=\"assets/mlp1.png\" alt=\"https://obilaniu6266h16.wordpress.com/2016/02/04/einstein-summation-in-numpy/\" width=\"600\" >\n",
    "</th><th>\n",
    "<img src=\"assets/mlp2.png\" alt=\"https://obilaniu6266h16.wordpress.com/2016/02/04/einstein-summation-in-numpy/\" width=\"600\" >\n",
    "</th></tr></table>\n",
    "\n",
    "Notice how `np.einsum` captures succinctly the tensor flow (yep): the extension to batch is **extremely natural**. You can imagine a similar extension to RGB input (instead of a black/white float, we have an array of 3 values, so our input is now a 4D tensor `(batch_size, height, width, 3)`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Real Application\n",
    "\n",
    "Under certain conditions, a kernel for a Gaussian process, a model for regression, is a matrix with the following form:\n",
    "$$\n",
    "K = \\sum_{i=1}^nB_i\\otimes D_i\n",
    "$$\n",
    "\n",
    "$B_i$ has shape $a\\times a$, and they are small dense matrices. $D_i$ is a $b\\times b$ diagonal matrix, and $b$ is so large that we can't even hold $b^2$ in memory. So we only have a vector to represent $D_i$. A useful operation in Gaussian process modelling is the multiplication of $K$ with a vector, $K\\textbf{z}$. How can we do this efficiently and expressively?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L2 norm of difference 4.012242167593457e-14\n",
      "Matrix-vector multiplication\n",
      "    format: mean seconds (standard error) 5 runs\n",
      "    quadratic_impl 2.83e-02 (9.63e-04)\n",
      "    einsum_impl    6.28e-05 (5.31e-06)\n",
      "    improvement ratio 449.5\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(1234)\n",
    "a = 3\n",
    "b = 300\n",
    "Bs = np.random.randn(10, a, a)\n",
    "Ds = np.random.randn(10, b) # just the diagonal\n",
    "\n",
    "z = np.random.randn(a * b)\n",
    "\n",
    "def quadratic_impl():\n",
    "    K = np.zeros((a * b, a * b))\n",
    "    for B, D in zip(Bs, Ds):\n",
    "        K += np.kron(B, np.diag(D))\n",
    "    return K.dot(z)\n",
    "\n",
    "def einsum_impl():\n",
    "    # Ellipses trigger broadcasting\n",
    "    left_kron_saatci = np.einsum('N...b,ab->Nab', Ds, z.reshape(a, b))\n",
    "    full_sum = np.einsum('Nca,Nab->cb', Bs, left_kron_saatci)\n",
    "    return full_sum.ravel()\n",
    "\n",
    "print('L2 norm of difference', np.linalg.norm(quadratic_impl() - einsum_impl()))\n",
    "# Of course, we can make this arbitrarily better by increasing b...\n",
    "print('Matrix-vector multiplication')\n",
    "compare_times(quadratic_impl, einsum_impl)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}