{ "metadata": { "name": "", "signature": "sha256:1cef4e5868f3307029713a5e8b00de90db2eb3cff529c981aadc53662b2a3f36" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Probabilistic Interpretations of Regression\n", "\n", "# Gaussian Process Summer School, Melbourne, Australia\n", "### 25th-27th February 2015\n", "### Neil D. Lawrence\n", "\n", "\n", "### Overdetermined Systems\n", "\n", "We motivated the introduction of probability by considering systems where there were more observations than unknowns. In particular we thought about the simple fitting of the gradient and an offset of a line,\n", "\n", "$$ y= mx +c $$\n", "\n", "and what happens if we have three pairs of observations of $x$ and $y$, $\\{x_i, y_i\\}_{i=1}^3$. We solved this issue by introducing a type of [slack variable](http://en.wikipedia.org/wiki/Slack_variable), $\\epsilon_i$, known as noise, such that for each observation we had the equation,\n", "\n", "$$y_i = mx_i + c + \\epsilon_i.$$\n", "\n", "The slack variable represented the difference between our actual prediction and the true observation. This is also known as the *residual*. By introducing the slack variable we now have an additional $n$ variables to estimate, one for each data point, $\\{\\epsilon_i\\}$. This actually turns the overdetermined system into an underdetermined system. Introduction of $n$ variables, plus the original $m$ and $c$ gives us $n+2$ parameters to be estimated from $n$ observations, which actually makes the system *underdetermined*. However, we then made a probabilistic assumption about the slack variables, we assumed that the slack variables were distributed according to a probability density. And for the moment we have been assuming that density was the Gaussian,\n", "\n", "$$\\epsilon_i \\sim \\mathcal{N}(0, \\sigma^2),$$\n", "\n", "with zero mean and variance $\\sigma^2$. \n", "\n", "#### Sum of Squares and Probability\n", "\n", "In the overdetermined system we introduced a new set of slack variables, $\\{\\epsilon_i\\}_{i=1}^n$, on top of our parameters $m$ and $c$. We deal with the variables by placing a probability distribution over them. This gives rise to the likelihood and for the case of Gaussian distributed variables, it gives rise to the sum of squares error. It was Gauss who first made this connection in his volume on \"Theoria Motus Corprum Coelestium\" (written in Latin)" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import pods\n", "pods.notebook.display_google_book(id='ORUOAAAAQAAJ', page='213')" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stderr", "text": [ "/Users/neil/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pytz/__init__.py:29: UserWarning: Module pods was already imported from /Users/neil/sods/ods/pods/__init__.pyc, but /Users/neil/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages is being added to sys.path\n", " from pkg_resources import resource_stream\n" ] }, { "html": [ "" ], "metadata": {}, "output_type": "display_data", "text": [ "" ] } ], "prompt_number": 1 }, { "cell_type": "markdown", "metadata": {}, "source": [ "The relevant section roughly translates as\n", "\n", "... It is clear, that for the product $\\Omega = h^\\mu \\pi ^{-frac{1}{2}\\mu} e^{-hh(vv + v^\\prime v^\\prime + v^{\\prime\\prime} v^{\\prime\\prime} + \\dots)}$ to be maximised the sum $vv + v ^\\prime v^\\prime + v^{\\prime\\prime} v^{\\prime\\prime} + \\text{etc}.$ ought to be minimized. *Therefore, the most probable values of the unknown quantities $p , q, r , s \\text{etc}.$, should be that in which the sum of the squares of the differences between the functions $V, V^\\prime, V^{\\prime\\prime} \\text{etc}$, and the observed values is minimized*, for all observations of the same degree of precision is presumed.\n", "\n", "It's on the strength of this paragraph that the density is known as the Gaussian, despite the fact that four pages later Gauss credits the necessary integral for the density to Laplace, and it was also Laplace that did a lot of the original work on dealing with these errors through probability. [Stephen Stigler's book on the measurement of uncertainty before 1900](http://www.hup.harvard.edu/catalog.php?isbn=9780674403413) has a nice chapter on this." ] }, { "cell_type": "code", "collapsed": false, "input": [ "pods.notebook.display_google_book(id='ORUOAAAAQAAJ', page='217')" ], "language": "python", "metadata": {}, "outputs": [ { "html": [ "" ], "metadata": {}, "output_type": "display_data", "text": [ "" ] } ], "prompt_number": 2 }, { "cell_type": "code", "collapsed": false, "input": [ "%matplotlib inline\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import pods\n", "from IPython.display import display" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 3 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here `numpy` gives us the numerical array facility, `matplotlib` is for plotting and `pods` is a python open data science library which provides some convenient access to a range of data sets.\n", "\n", "## Iterative Solution\n", "\n", "Now we will code linear regression in python. We will do it in two ways, once using iterative updates (coordinate ascent) and then using linear algebra. \n", "For this part we are going to load in some data, we will use the example from the Olympics: the pace of Marathon winners. To load in the data we are going to use the `pods` library, an 'open data science' library for access to a range of data sets and with other facilities." ] }, { "cell_type": "code", "collapsed": true, "input": [ "data = pods.datasets.olympic_marathon_men()\n", "x = data['X']\n", "y = data['Y']" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 4 }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can see what these values are by typing:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "print(x)\n", "print(y)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "[[ 1896.]\n", " [ 1900.]\n", " [ 1904.]\n", " [ 1908.]\n", " [ 1912.]\n", " [ 1920.]\n", " [ 1924.]\n", " [ 1928.]\n", " [ 1932.]\n", " [ 1936.]\n", " [ 1948.]\n", " [ 1952.]\n", " [ 1956.]\n", " [ 1960.]\n", " [ 1964.]\n", " [ 1968.]\n", " [ 1972.]\n", " [ 1976.]\n", " [ 1980.]\n", " [ 1984.]\n", " [ 1988.]\n", " [ 1992.]\n", " [ 1996.]\n", " [ 2000.]\n", " [ 2004.]\n", " [ 2008.]\n", " [ 2012.]]\n", "[[ 4.47083333]\n", " [ 4.46472926]\n", " [ 5.22208333]\n", " [ 4.15467867]\n", " [ 3.90331675]\n", " [ 3.56951267]\n", " [ 3.82454477]\n", " [ 3.62483707]\n", " [ 3.59284275]\n", " [ 3.53880792]\n", " [ 3.67010309]\n", " [ 3.39029111]\n", " [ 3.43642612]\n", " [ 3.20583007]\n", " [ 3.13275665]\n", " [ 3.32819844]\n", " [ 3.13583758]\n", " [ 3.0789588 ]\n", " [ 3.10581822]\n", " [ 3.06552909]\n", " [ 3.09357349]\n", " [ 3.16111704]\n", " [ 3.14255244]\n", " [ 3.08527867]\n", " [ 3.10265829]\n", " [ 2.99877553]\n", " [ 3.03392977]]\n" ] } ], "prompt_number": 5 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Plotting the Data\n", "\n", "And you can make a plot of $y$ vs $x$ with the following command:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "plt.plot(x, y, 'rx')" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 6, "text": [ "[]" ] }, { "metadata": {}, "output_type": "display_data", "png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEACAYAAACwB81wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFFRJREFUeJzt3X+s5Xdd5/HnqwzaH6JXAxmQGTKTSBNL1HarQ9NBe9as\naGe0ugGTZldgMXEnoIEoVF3KpuPGQWLX0CVxoQluwu4a1g2VLjCzS/l1ujDB1sq0FMYCDXdi2y3F\npL0g1kRq3/vH+d72zJ3745y533vP99zv85Hc3O/5fj/3nPe9ud/X+Z7391eqCknSznfBrAuQJG0P\nA1+SesLAl6SeMPAlqScMfEnqCQNfknpi1ySDkpwBvgX8E/CdqjqwYvkA+F/A15pZt1XV77dXpiRp\nsyYKfKCAQVU9vs6YO6vquhZqkiRtgWlaOtnkcknSDE0a+AV8Isk9SX5tjeVXJ7kvyYkkl7VXoiSp\nDZO2dA5W1aNJXgB8PMkDVfWZseWfB/ZW1ZNJrgVuBy5tu1hJ0vnLtNfSSXIT8O2q+qN1xiwCV473\n/JN40R5JOg9V1UrLfMOWTpKLkzyvmb4EeCVw/4oxu5OkmT7A6I3knB28VTW3XzfddNPMa+hr/fNc\nu/XP/mve62/TJC2d3cCHmjzfBfxpVd2R5EgT4rcCrwbekOQp4Eng+larlCRt2oaBX1WLwOWrzL91\nbPqPgT9utzRJUps803ZCg8Fg1iVsyjzXP8+1g/XP2rzX36apd9qe9wsltV2vJUk7RRJqu3baSpJ2\nBgNfknrCwJeknjDwJaknDHxJ6gkDX5J6wsCXpJ4w8CWpJwx8SeoJA1+SesLAl6SeMPA36/hxWFo6\ne97S0mi+JHWIgb9ZBw/CjTc+G/pLS6PHBw/Oti5JWsGrZbZhOeRvuAFuvhmOHYOFhVlXJWkHaPNq\nmQZ+W86cgf37YXER9u2bdTWSdggvj9w1S0ujLfvFxdH3lT19SeoAA3+zlts5x46NtuyPHTu7py9J\nHWFLZ7OOHx/toB3v2S8twcmTcPjw7OqStCPYw5eknrCHL0mamoEvST1h4EtSTxj4ktQTBr4k9YSB\nL0k9YeBLUk9MFPhJziT5QpJTSe5eY8y7k3w1yX1Jrmi3TEnSZu2acFwBg6p6fLWFSQ4BP1RVL03y\ncuA9wFUt1ShJasE0LZ31zvS6Dng/QFXdBSwk2b2ZwiRJ7Zo08Av4RJJ7kvzaKstfDDw09vhhYM9m\ni5MktWfSls7Bqno0yQuAjyd5oKo+s2LMyk8AXjhHkjpkosCvqkeb73+b5EPAAWA88B8B9o493tPM\nO8vRo0efmR4MBgwGg6kLlqSdbDgcMhwOt+S5N7xaZpKLgedU1d8luQS4A/i9qrpjbMwh4Deq6lCS\nq4BbquqqFc/j1TIlaUptXi1zki383cCHkiyP/9OquiPJEYCqurWqTiQ5lORB4O+B17dRnCSpPV4P\nX5I6zOvhS5KmZuBLUk8Y+JLUEwa+JPWEgS9JPWHgS1JPGPiS1BMGviT1hIEvST1h4EtSTxj4ktQT\nBr4k9YSBL0k9YeBLUk8Y+JLUEwa+JPWEgS9JPWHgS1JPGPiS1BMGviT1hIEvST1h4EtSTxj4ktQT\nBr4k9YSBL0k9YeBLUk8Y+JLUEwa+JPWEgS9JPTFR4Cd5TpJTST6yyrJBkm82y08leXv7ZUqSNmvX\nhOPeDJwGnrfG8jur6rp2SpIkbYUNt/CT7AEOAe8DstawNouSJLVvkpbOu4AbgKfXWF7A1UnuS3Ii\nyWWtVbdVjh+HpaWz5y0tjeZL0g61bksnyc8D36iqU0kGawz7PLC3qp5Mci1wO3DpagOPHj36zPRg\nMGAwWOspt9jBg3DjjXDsGCwsjMJ++fG448dHYxcWnp23tAQnT8Lhw9tbs6ReGA6HDIfDLXnuVNXa\nC5N3AK8BngIuBL4XuK2qXrvOzywCV1bV4yvm13qvte2WQ/6GG+Dmm58N/9XGrPbGsHKsJG2BJFRV\nK23zdQN/xYteA7y1qn5hxfzdjD4FVJIDwP+sqn2r/Hy3Ah/gzBnYvx8WF2HfvtXHTPLGIElbpM3A\nn/QonWXVFHAEoKpuBV4NvCHJU8CTwPVtFLbllpZGAb64uH6QLyyMwn75jcGwlzSnJt7C3/QLdWkL\nf5pWjVv4kmZoJi2dTb9QlwJ/0p2x9vAlzZiBv108SkfSjBn4ktQTbQa+F0+TpJ4w8CWpJwx8SeoJ\nA1+SesLAl6SeMPAlqScMfEnqCQNfknrCwJeknjDwJaknDHxJ6gkDX5J6wsCXpJ4w8CWpJwx8SeoJ\nA3+7HD8+unnKuKWl0XxJ2gYG/nY5eHB0e8Tl0F++XeLBg7OtS1JveMer7eQN0SVNyVsczrMzZ2D/\nflhchH37Zl2NpI7zFofzamlptGW/uDj6vrKnL0lbyMDfLsvtnGPHRlv2x46d3dOXpC1mS2e7HD8+\n2kE73rNfWoKTJ+Hw4dnVJanT7OFLUk/Yw5ckTc3Al6SemCjwkzwnyakkH1lj+buTfDXJfUmuaLdE\nSVIbJt3CfzNwGjinCZ/kEPBDVfVS4N8C72mvPElSWzYM/CR7gEPA+4DVdhxcB7wfoKruAhaS7G6z\nSEnS5k2yhf8u4Abg6TWWvxh4aOzxw8CeTdYlSWrZrvUWJvl54BtVdSrJYL2hKx6vevzl0aNHn5ke\nDAYMBus9pST1z3A4ZDgcbslzr3scfpJ3AK8BngIuBL4XuK2qXjs25r3AsKr+R/P4AeCaqnpsxXN5\nHL4kTWnbjsOvqrdV1d6q2g9cD3xqPOwbHwZe2xR2FbC0MuwlSbO3bktnFQWQ5AhAVd1aVSeSHEry\nIPD3wOtbrlGS1AIvrdA1XnNH0hgvrbCTeWcsSVvELfwu8s5YkhpeLbMPvDOWJGzp7HzeGUvSFjDw\nu8Y7Y0naIgZ+15w8eXbPfmFh9Pjkya15vePHz30zWVoazZe0o9jD77vxTxQLC+c+ljRT7rRVuzwq\nSOosA1/t86ggqZM8Skft8qggqRcM/L7zqCCpN2zp9J3X7pE6zR6+JPWEPXxJ0tQMfEnqCQNfknrC\nwJeknjDw1Q9eM0gy8NUT3klM8rBM9YjXDNIc8jh8bb+dcoKW1wzSnPE4fG2/ndAS8ZpB6jkDX5NZ\nvhHLjTeOtpJXu2Z+l3eMes0gyZaOprReS6TLN1PZKS0p9Y4tHU2m7S3ujVoik3wKmJXDh8+tY2HB\nsFevuIW/k7W5xT3Nc7ljVGqNW/iaTJtb3JPeXN0do1JnuYXfB9u1xT3ppwD76dLEtnULP8mFSe5K\ncm+S00n+YJUxgyTfTHKq+Xp7G8WpBdu5xT3pp4CdcIinNIcm2sJPcnFVPZlkF/BZ4K1V9dmx5QPg\nt6rqunWewy387dblo2Y861WayMzOtE1yMXAn8LqqOj02fwC8pap+YZ2fNfC3W9dbJ+7clTa07Ttt\nk1yQ5F7gMeDT42HfKODqJPclOZHksjaK0yZ1+VBEd+5K227XJIOq6mng8iTfB3wsyaCqhmNDPg/s\nbdo+1wK3A5eufJ6jR48+Mz0YDBgMBudfuebXytbS8pFEtnUkhsMhw+FwS5576qN0kvx74B+q6j+u\nM2YRuLKqHh+bZ0tHI11vNUkdst1H6Tw/yUIzfRHwM8CpFWN2J0kzfYDRG8nj5zyZBN1uNU2iy9cM\nktYxSQ//RcCnmh7+XcBHquqTSY4kOdKMeTVwfzPmFuD6rSlX6gAPK9Wc8sQr6Xx4WKm2iTdAkbrA\nw0q1DbyWjrQVpunNe1ip5pCBLy2btDfvzVQ0p2zpSOMm6c17WKm2kT18aSvZm1eH2MOXtkoXe/Me\n96+WGPjSsq725j3uXy2xpSMt63Jv3uP+e8sevtRH7lvoJXv4Ut90cd+C5o6BL3VdV/ctaO7Y0pG6\nrsv7FrTl7OFLUk/Yw5ckTc3Al7aCJ0upgwx8aSt4spQ6yB6+tFU8WUotcKetNC88WUqb5E5baR54\nstR03O+x5Qx8aSvM4mSpeQ/MSfZ7zPvvOGtVtS1fo5eSeuKjH6164omz5z3xxGj++YybxBNPVL3x\njc8+38rH82C55sXF1WvfCb/jlJrsbCeH23qiDV/IwJfO1XaAbRSY82BxcRRNi4urL98Jv+MUDHxp\nJ2k7wDYKzEm0+cljGpP+Ldr4Hatm93tOwcCXdpq2AqytN482P3lMGqqTvuYkv2PbrzlDBr60k2x3\nSE8bhl2qa9o3hUmCvOMtIgNf2im6vCVdtfEnj+1+85imBTPNa7b1CWsLGPjSTtHlXvk0Y9p489gK\nk7ymW/gGvrTjrReGbbdEZhGqbb9hbWSL3ry3LfCBC4G7gHuB08AfrDHu3cBXgfuAK9YYs6lfWlKL\nNgrDacOrrTePtrS9P6PN15zStm7hAxc333cBfwG8YsXyQ8CJZvrlwF+s8Tyb+qUltWS7j/2fRduq\ny62yKbUZ+BNfPC3JxcCdwOuq6vTY/PcCn66qP2sePwBcU1WPrfj5mvS1JG2hNm+ZOH4JiYWFcx/v\nBNP+vVq+YN62XjwtyQVJ7gUeYxTsp1cMeTHw0Njjh4E9bRQnaQscPnxuGC8snN/9cU+ePDvcFxZG\nj0+e3HydXTHNvQ06fsG8DQO/qp6uqssZhfhPJRmsMmzlu4+b8lIftPnm0VXLb2I33jjael/rE8ws\nLpg3pV2TDqyqbyY5Dvw4MBxb9Aiwd+zxnmbeOY4ePfrM9GAwYDAYTF6pJM3KwsLoRjbLrZrV2lXr\nfdqZ4g1wOBwyHA7bqXuFdXv4SZ4PPFVVS0kuAj4G/F5VfXJszCHgN6rqUJKrgFuq6qpVnssevqT5\nNMO7l21nD/9FwKeaHv5dwEeq6pNJjiQ5AlBVJ4CvJXkQuBV4YxuFSVInzEGrZlLe4lCS1tPmUU3n\nwXvaSlJPeE9bSdLUDHxJ6gkDX5J6wsCXpJ4w8CWpJwx8SeoJA1+SesLAl6SeMPAlqScMfEnqCQNf\nknrCwJeknjDwJaknDHxJ6gkDX5J6wsCXpJ4w8CWpJwx8SeoJA1+SesLAl6SeMPAlqScMfEnqCQNf\nknrCwJeknjDwJaknDHxJ6gkDX5J6YsPAT7I3yaeTfCnJF5O8aZUxgyTfTHKq+Xr71pQrSTpfk2zh\nfwf4zap6GXAV8OtJfniVcXdW1RXN1++3WmUHDIfDWZewKfNc/zzXDtY/a/Nef5s2DPyq+npV3dtM\nfxv4a+AHVxmalmvrlHn/p5nn+ue5drD+WZv3+ts0VQ8/yT7gCuCuFYsKuDrJfUlOJLmsnfIkSW3Z\nNenAJN8DfBB4c7OlP+7zwN6qejLJtcDtwKXtlSlJ2qxU1caDkucCHwX+d1XdMsH4ReDKqnp8bN7G\nLyRJOkdVtdIy33ALP0mAPwFOrxX2SXYD36iqSnKA0RvJ4+Nj2ipYknR+JmnpHAR+BfhCklPNvLcB\nLwGoqluBVwNvSPIU8CRw/RbUKknahIlaOpKk+XfeZ9om+S9JHkty/9i8A0nubk6++sskP9HMvzDJ\nB5J8IcnpJL879jNXJrk/yVeT/KfN/Tqbrv/HknyuqfPDSZ43tuzfNTU+kOSV81R/kp9Jck8z/54k\n/3ye6h9b/pIk307ylnmrP8mPNsu+2Cz/rlnVP+X/ThfX3VVPBk3yA0k+nuQrSe5IsjD2M51Zf6et\nv9X1t6rO6wv4SUaHaN4/Nm8I/GwzfS3w6Wb63wAfaKYvAhaBlzSP7wYONNMngJ8735paqP8vgZ9s\npl8P/Idm+jLgXuC5wD7gQZ79dDQP9V8OvLCZfhnw8NjPdL7+seUfBP4MeMs81c+odXof8CPN4+8H\nLphV/VPW3sV194XA5c309wBfBn4Y+EPgt5v5vwO8s5nu1Pp7HvW3tv6e9xZ+VX0GeGLF7EeB72um\nF4BHxuZfkuQ5wCXAPwLfSvIi4HlVdXcz7r8Cv3S+NU1jjfpf2swH+ATwqmb6Fxn903+nqs4w+od5\n+bzUX1X3VtXXm/mngYuSPHde6gdI8kvA1xjVvzxvXup/JfCFqrq/+dknqurpWdU/Ze1dXHdXOxn0\nxcB1wPubYe8fq6dT6++09be5/rZ98bTfBf4oyd8ANzPauUtVfQz4FqN/njPAzVW1xOiXfHjs5x9p\n5s3Kl5L8YjP9y8DeZvoHObvOhxnVuXJ+V+sf9yrgr6rqO8zJ3z+jc0B+Gzi6Yvxc1M/onJRK8n+S\n/FWSG5r5Xap/1dq7vu7m7JNBd1fVY82ix4DdzXRn198J6x+3qfW37cD/E+BNVfUS4DebxyT5FUYf\nB18E7AfemmR/y6/dhl8F3pjkHkYftf5xxvVMa936k7wMeCdwZAa1TWKt+o8C76qqJ+n2JTzWqn8X\n8ArgXzXf/2WSn2Z0hnpXrFp7l9fdZkPgNkYng/7d+LIa9Ti69Pc9x7T1t7H+Tnym7YQOVNW/aKY/\nCLyvmb4a+FBV/RPwt0lOAlcCnwX2jP38Hp5tA227qvoy8LMASS4FDjeLHuHsreU9jN5ZH2E+6ifJ\nHuDPgddU1WIzu+v1H2oWHQBeleQPGbUKn07yD4x+ny7Xv/z3fwj4v9Wcm5LkBPDPgP9OR+pf52/f\nyXU3o5NBbwP+W1Xd3sx+LMkLq+rrTbvjG838zq2/U9bf2vrb9hb+g0muaaZ/GvhKM/1A85gklzC6\n6uYDTV/qW0leniTAaxhdlmEmkryg+X4B8HbgPc2iDwPXJ/muZuvmpcDd81J/s7f/OPA7VfW55fFV\n9Sjdrv+9TZ0/VVX7q2o/cAtwrKr+87z8/YGPAT+S5KIku4BrgC91qf61/vZ0cN1tXm+1k0E/DLyu\nmX7dWD2dWn+nrb/V9XcTe5o/APw/Rh/9HmK0Z//HGfWi7gU+B1zRjP1uRlsz9wNf4uyjLK5s5j8I\nvPt862mh/l8F3sRoj/mXgXesGP+2psYHaI5Empf6Ga3A3wZOjX09f17qX/FzNwG/NU9//2b8vwa+\n2NT6zlnWP+X/ThfX3VcATzc5s/z//HPADzDa4fwV4A5gYexnOrP+Tlt/m+uvJ15JUk94i0NJ6gkD\nX5J6wsCXpJ4w8CWpJwx8SeoJA1+SesLAl6SeMPAlqSf+P7P3QdRZ9dimAAAAAElFTkSuQmCC\n", "text": [ "" ] } ], "prompt_number": 6 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Maximum Likelihood: Iterative Solution\n", "\n", "Now we are going to fit a line, $y_i=mx_i + c$, to the data you've plotted. We are trying to minimize the error function:\n", "\n", "$$E(m, c, \\sigma^2) = \\frac{n}{2} \\log \\sigma^2 + \\frac{1}{2\\sigma^2} \\sum_{i=1}^n(y_i-mx_i-c)^2$$\n", "\n", "with respect to $m$, $c$ and $\\sigma^2$. We can start with an initial guess for $m$, " ] }, { "cell_type": "code", "collapsed": true, "input": [ "m = -0.4\n", "c = 80" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 7 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then we use the maximum likelihood update, derived in the lecture, to find an estimate for the offset, $c$,\n", "\n", "$$c^* = \\frac{\\sum_{i=1}^n(y_i-m^*x_i)}{n}$$" ] }, { "cell_type": "code", "collapsed": false, "input": [ "c = (y - m*x).mean()\n", "print c" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "786.019771145\n" ] } ], "prompt_number": 8 }, { "cell_type": "markdown", "metadata": {}, "source": [ "And now we can make an estimate for the gradient of the line,\n", "\n", "$$m^* = \\frac{\\sum_{i=1}^n ((y_i - c)*x_i))}{\\sum_{i=1}^n x_i^2}$$" ] }, { "cell_type": "code", "collapsed": false, "input": [ "m = ((y - c)*x).sum()/(x**2).sum()\n", "print m" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "-0.3998724073\n" ] } ], "prompt_number": 9 }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can have a look at how good our fit is by computing the prediction across the input space. First create a vector of 'test points'," ] }, { "cell_type": "code", "collapsed": false, "input": [ "x_test = np.linspace(1890, 2020, 130)[:, None]" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 10 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now use this vector to compute some test predictions," ] }, { "cell_type": "code", "collapsed": false, "input": [ "f_test = m*x_test + c" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 11 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now plot those test predictions with a blue line on the same plot as the data," ] }, { "cell_type": "code", "collapsed": false, "input": [ "plt.plot(x_test, f_test, 'b-')\n", "plt.plot(x, y, 'rx')" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 12, "text": [ "[]" ] }, { "metadata": {}, "output_type": "display_data", "png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEACAYAAAC6d6FnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHe1JREFUeJzt3XuUFPWZ//H3AwPilQFU7qyoA4o3FCUq+rO9hngDYwxq\nohgvSSR74ibGCJgcJxcFYc1u1DXuqtmQZOM1RsErxLXVuAqa5T7AjAlsxAjGOESJZkXn+f3xrZZm\nlsvMdPVUddfndc6cqarurnqYQ/fTVfX9VJm7IyIi2dMl6QJERCQZagAiIhmlBiAiklFqACIiGaUG\nICKSUWoAIiIZFUsDMLOuZrbQzOZE873NbJ6ZNZrZXDOrjWM7IiISn7j2AK4CGoBCqGAyMM/dhwFP\nR/MiIpIiJTcAMxsEnA7cBVi0+GxgVjQ9Cxhf6nZERCRecewB/BNwDdBStKyvu6+PptcDfWPYjoiI\nxKikBmBmZwJvuvtCNn/734KHa03oehMiIilTU+LrjwXONrPTgR7AHmb2M2C9mfVz93Vm1h94s/UL\nzUxNQUSkA9x9q1+426ukPQB3n+rug919KHA+8J/ufhEwG5gYPW0i8PA2Xl+xP9dff33iNaj+5OtQ\n/ZX3U8m1u8f7vTnuHEChuunAqWbWCJwUzYuISIqUegjoY+7+LPBsNP02cEpc6xYRkfgpCdxBuVwu\n6RJKovqTpfqTU8m1x83iPqbU5g2beVLbFhGpVGaGp+EksIiIVC41ABGRjFIDEBHJKDUAEZGMUgMQ\nEckoNQARkYxSAxARySg1ABGRjFIDEBHJKDUAEZGMUgMQEckoNQARkYxSAxARySg1ABGRjFIDEBHJ\nKDUAEZGMUgMQEckoNQARkYxSAxARySg1ABGRjFIDEBHJqJIagJn1MLP5ZrbIzBrMbFq0vLeZzTOz\nRjOba2a18ZQrIiJxMXcvbQVmu7j7e2ZWA/wG+AZwNvCWu88ws2uBXu4+udXrvNRti4hkjZnh7hbH\nuko+BOTu70WT3YGuQDOhAcyKls8Cxpe6nfbX1dlbFBGpLCU3ADPrYmaLgPXAM+6+HOjr7uujp6wH\n+pa6nfZYuxZGjoSHHlIjEBHZlppSV+DuLcBIM+sJPGVmJ7Z63M1sqx/D9fX1H0/ncjlyuVyp5QAw\ncCBMnw5TpsBNN4XpE0/c8etERNImn8+Tz+fLsu6SzwFssTKzbwPvA5cDOXdfZ2b9CXsGB7R6btnP\nAbS0wL33wre/DXV1MG0aHH54WTcpIlJWqTkHYGZ7Fkb4mNnOwKnAQmA2MDF62kTg4VK201FdusCF\nF8KKFXDWWXD66XDBBfDqq0lUIyKSLqWeA+gP/Gd0DmA+MMfdnwamA6eaWSNwUjSfmO7d4StfgaYm\nOOggOPpomDQJ3ngjyapERJIV6yGgdm04wWGgb70VDgf95Cfw5S/DNddArZIKIlIBUnMIqFLtuSfc\nfDMsXBj2AoYNg3/8R3j//aQrExHpPJlsAAVDhsCPfwz5PLzwQmgEd98NH36YdGUiIuWXyUNA2/LS\nSzB5MqxfDzfcAOecAxbLjpaISDziPASkBtCKOzz1VGgEO+2kDIGIpIsaQCdoaYH77oNvfQv23z+c\nND7iiKSrEpGs00ngTtClS8gMrFgB48bBmWfC+ecrQyAi1UMNYAe6dw+ZgaYmOOSQkCG48kplCESk\n8qkBtNGuu8J118GqVWH64INh6lTYsCHpykREOkYNoJ369AmZgYULw2ihYcNg5kxlCESk8qgBdNCQ\nISEzkM/Diy+GRnDXXcoQiEjl0CigmBQyBOvWhQzBpz+tDIGIxE/DQFPKHebODY2gW7eQITjppKSr\nEpFqogaQci0tcP/9IUOw335w440walTSVYlINVAOIOW6dAmZgYYGGD8+3ItgwoQwlFREJC3UAMqo\ne/eQGWhqgsMOg2OOCZef/uMfk65MREQNoFPsumvIDKxaBbvtFjIEU6YoQyAiyVID6ESFDMHixfDm\nm+E+xTNmKEMgIslQA0jA4MEhQ/DcczB/vjIEIpIMjQJKgfnzw9DRN95QhkBEtk/DQKtQIUMwZQrU\n1ITLT598ctJViUjaqAFUseIMwb77hkagDIGIFCgHUMUKGYIVK8ItKQsZgsbGpCsTkWqjBpBS3bpt\nmSE49lhlCEQkXiU1ADMbbGbPmNlyM1tmZl+Nlvc2s3lm1mhmc82sNp5ys6c4Q7D77uGmNFOmQHNz\n0pWJSKUrdQ9gE/A1dz8IOBr4ipkdCEwG5rn7MODpaF5K0KdPuO/AokXwpz+FoaPKEIhIKUpqAO6+\nzt0XRdMbgRXAQOBsYFb0tFnA+FK2I5sNHhwyA88/H4aP1tXBnXcqQyAi7RfbKCAz2wd4FjgY+IO7\n94qWG/B2Yb7o+RoFFIMFC0KG4PXXQ4bg3HOVIRCpZnGOAqqJYyVmthvwS+Aqd3/Xij6B3N3NbKuf\n9PX19R9P53I5crlcHOVkyujR8PTTMG9eaAQ33RTuQ6AMgUh1yOfz5PP5sqy75D0AM+sGPAo84e7/\nHC1bCeTcfZ2Z9QeecfcDWr1OewAxa2mBBx4IGYJ99gmNQBkCkeqSmhxAdHjnbqCh8OEfmQ1MjKYn\nAg+Xsh1pmy5dQmagoSEcCjrrLPjsZ5UhEJGtK2kPwMyOA54DlgCFFU0BFgD3A0OANcBn3X1Dq9dq\nD6DM/vpXuOUWuPnm0BCuvx4GDEi6KhEphS4FIe3y5z+HcwN33w1XXAHXXgu9eu34dSKSPqk5BCSV\noU+fkBlYvDg0g2HDQkN4772kKxORJKkBZMigQSEz8Pzz8PLLoRH8278pQyCSVToElGHFGYLvfx8+\n8xllCETSTucAJDbumzMEXbqEoaOnnJJ0VSKyLWoAErvWGYJp0+DII5OuSkRa00lgiV1xhuAzn4Fx\n45QhEKl2agCyhW7d4EtfCvchOOIIGDMmzL/+etKViUjc1ABkq3bZJZwXWLUKamvh0EPDvO5DIFI9\n1ABku3r3DpmBxYvh7beVIRCpJmoA0iaDBoXMQCFDUFcH//qvsGlT0pWJSEdpFJB0yMsvh0NCr722\nOUPQRV8nRMpOw0AlFdzh178OjcBMGQKRzqAGIKnS0gIPPhgyBEOGhEagDIFIeSgHIKnSpUvIDCxf\nDuedFzIE550XRhCJSHqpAUhsijMEo0bBccfBF7+oDIFIWqkBSOyKMwS9eoUMwbXXhmGkIpIeagBS\nNoUMwZIlIUA2fHg4P6AMgUg6qAFI2Q0cGDIEv/kN/Pa3yhCIpIVGAUmnU4ZApOM0DFSqQiFDAOHy\n06eemmw9IpVADUCqRksL/PKXcN11IUMwbRocdVTSVYmkl3IAUjW6dAmZgeXLQ5Zg/HhlCEQ6ixqA\npEK3biEz0NQUUsSFDMHatUlXJlK9Sm4AZvZjM1tvZkuLlvU2s3lm1mhmc82sttTtSDbsskvIDDQ2\nhmGkhx0G3/ymMgQi5RDHHsC/A2NbLZsMzHP3YcDT0bxIm/XqFTIDS5bAX/4S7kMwbZoyBCJxKrkB\nuPvzQOv7RJ0NzIqmZwHjS92OZNPAgSEz8MILsHBhyBDccYcyBCJxKNc5gL7uvj6aXg/0LdN2JCOG\nD4f774dHHgmjhkaMgPvuC6OIRKRjasq9AXd3M9vqeM/6+vqPp3O5HLlcrtzlSIU78kiYN29zhmDG\njM0ZAotlYJxIuuTzefL5fFnWHUsOwMz2Aea4+yHR/Eog5+7rzKw/8Iy7H9DqNcoBSEncw30IrrsO\nBg8OjWD06KSrEimvSsgBzAYmRtMTgYfLtB3JMLPNGYIJE+Ccc8JlJVauTLoykcoQxzDQe4D/Aoab\n2Wtm9gVgOnCqmTUCJ0XzImVRnCE46ig4/ni44gplCER2RJeCkKrT3BwuQ33nnXDZZeFcQe/eSVcl\nEo9KOAQkkpjiDME774QRRNOmwV//mnRlIumiBiBVa+DAkBl44QVYtCiEyZQhENlMDUCq3rBhITPw\nyCPw0EPKEIgU6ByAZM7TT4fzAi0tyhBI5dH9AERK5L75PgQDB4ZG8IlPJF2VyI7pJLBIicxCZmD5\ncrjgAjj33PCjDIFkiRqAZFpNTcgMNDaGPYDjj4fLL1eGQLJBDaAcHnsMNmzYctmGDWG5pNIuu4T7\nDjQ2wl57hfsQXHON7kMg1U0NoBzGjAkHlwtNYMOGMD9mTLJ1yQ716hXOByxdCu++G0YQ3XijMgRS\nndQAyqG2Fm64IXzor1kTft9wQ1guFWHAgJAZePFFWLw43IfgRz9ShkCqixpAQVsO27Tn0E5tbTiG\nMHRo+K0P/4pUVxcyA3PmwK9+BQceCPfeqwyBVAc1gIK2HLZpz6GdDRtg5kxYvTr8bt04dJ6goowa\nBXPnhruT/eAH4b4ETz0VhpOKVCx3T+QnbDplmpvdJ01yX706/G5uLu05hcdaz7f1Oe7ujz76f5c1\nN4fl5ZLENjtbCf/Glhb3Bx5wHzbMPZdzf+mlMtUoshXRZ2c8n8NxrajdG05jA3APH+wQfnf0OW39\ncImrmbRVe+va3jbbuq62PC+JhhNDA960yf3OO90v6vOof+6MZm9o6MT6JbPUAMolrj2A9mhLw4lr\nm+1pJjvaZlvXFefeUNxiasDv/bHZ//uYSb5fn2a/7DL315aWsDeXRGNN6zbTuq62KtMXGzWAckji\ng6o9H+xtaRSdvc22riuJxtpWcTXg5mb/22WT/KYrV/tdO03yb/19s7/11jbWE8f/sc5eV6XXH/ff\noi0f7mX6YqMGUA6dfaiiPf854v5wjHOvo62NKY5Da3GLuwFHz1n30mr/8pfd+/Rxv+EG940b27nN\nJBprWreZ9nW1taHE+MVGDaAaxHk8vj2S+M+dxj2AuBvwVp7T2Og+YYJ7//7ut9/u/sEH0XPjbIad\nva4ktpnWdcX9JamN1ACyJIm9jjh3b+PcPY9TnA14B8955RX3005z328/9wfubPaWKyv0W28S20zr\nugriOkzaDmoA0jFJnOBK6yigtoqx/mcfafb7957kxx3c7E884d7ydgnNMM7GmtZtpnVdrZ9b6kCJ\ndlIDEKlEjz7qLW83+4MPug8fHjIEC+Z2sBlqFFBy6yos29GHewWMAtINYUQS8OGH8JOfwHe+A0cd\nFS4VdeCBSVclbfbYY+EKAMWXeNmwIdyA+owzyrpp3RFMpEq8/z7cdlu4WshZZ0F9PQwenHRVkmYV\ncUcwMxtrZivNrMnMri3XdkQq2c47h2sFNjZC374wciR84xvw5z8nXZlkQVkagJl1BW4DxgIjgAvM\nTDu4IttQWxvuO7BsWbj3wPDh4bCQ7kMg5VSuPYDRwKvuvsbdNwH3AuPKtC2RqtG/f7jvwIsvhpvS\n1NXB7bfDBx8kXZlUo3I1gIHAa0Xza6NlItIGdXXhvgOPPgqzZ8OIEXDPPboPgcSrpkzrbdPZ3fr6\n+o+nc7kcuVyuTOWIVKYjjoAnn4RnnoHJk2HGjHDLyk9+EiyW04CSdvl8nnw+X5Z1l2UUkJkdDdS7\n+9hofgrQ4u43FT1Ho4BE2sE93JVs6lTo1w+mT4ejj066KulslTAK6BWgzsz2MbPuwARgdpm2JZIJ\nZvDpT4cTxRddBOedB+ecAytWJF2ZVKqyNAB3/xD4e+ApoAG4z93131QkBjU1cNllYejomDFwwglw\n6aXw2ms7fq1IMQXBRCpc4fbTd9wBl1wSDhH16ZN0VVIulXAISEQ6SW1tyAwsWxaSxcOHw/e/Dxs3\nJl2ZpJ0agEiV6N8/ZAZeegmWLw9DSf/lX5QhkG1TAxCpMvvvHzIDjz8Oc+aEi8z94hfKEMj/pXMA\nIlWukCH43/8NGYKxY5UhqGS6GqiItIs7PPxwOEHct68yBJVMJ4FFpF3MQmZg6VK4+OLNGYKGhqQr\nkySpAYhkSE1NyAwUMgS5XJj/wx+SrkySoAYgkkE77xzuO9DYCAMGwOGHw9e/Dm+9lXRl0pnUAEQy\nrLY2ZAaWLQsniQ84QBmCLFEDEBH69w+ZgZdeCucFlCHIBjUAEfnY/vuHzMDjj4d7EShDUN00DFRE\ntimfDxmCv/1NGYK0UA5ARDpNcYZg771DhuCYY5KuKruUAxCRTlOcIZg4ESZMgPHjw/WGpLKpAYhI\nmxRnCI4/Hk48Eb7wBfif/0m6MukoNQARaZcePeDqq6GpCQYODPctVoagMqkBiEiH9OwZMgPLl2/O\nEHzve8oQVBI1ABEpSb9+ITMwf364P3FdHdx2mzIElUANQERisd9+ITPwxBPw2GNhj+A//kMZgjTT\nMFARKYtChuD990OG4FOfUoYgDsoBiEhFcIdHHgkZgj33DBmCY49NuqrKphyAiFQEs5AZWLIkDBk9\n/3wYN04ZgrRQAxCRsqupCQ2gsRFOOCFkCC65RBmCpHW4AZjZeWa23Mw+MrMjWj02xcyazGylmZ1W\nepkiUg169AiZgaYmGDw4ZAi+9jVlCJJSyh7AUuAc4LnihWY2ApgAjADGArebmfY0RORjPXuGzMDy\n5bBpUxgx9N3vKkPQ2Tr8wezuK929cSsPjQPucfdN7r4GeBUY3dHtiEj16tcvZAbmz4dVq5Qh6Gzl\n+GY+AFhbNL8WGFiG7YhIldhvv5AZeOKJcC+CAw6An/9cGYJyq9neg2Y2D+i3lYemuvucdmxnq+M9\n6+vrP57O5XLkcrl2rFJEqs3IkaEBPPtsyBDMnAk33ginn57dDEE+nyefz5dl3SXnAMzsGeBqd//v\naH4ygLtPj+afBK539/mtXqccgIhskzvMnh0yBH36KENQkMYcQHExs4Hzzay7mQ0F6oAFMW1HRDLC\nLGQGliwJl6G+4IIwv2xZ0pVVj1KGgZ5jZq8BRwOPmdkTAO7eANwPNABPAJP0VV9EOqpr15AZWLUK\ncjk4+WRlCOKiS0GISEX5y1/g5pvDFUgvvjgcItprr6Sr6jxpPAQkItIpevYMmYGGhpAhOPDAMP/u\nu0lXVnnUAESkIvXtGzIDCxZszhDcemu4OY20jRqAiFS0ffcNGYInnww/hQzBRx8lXVn66RyAiFSV\n554LGYKNG8N9CKotQ6D7AYiIbEdxhqB375AhGDMm6arioZPAIiLbUZwhuOwyuPBCOPtsZQhaUwMQ\nkapVnCE48cSQIZg4URmCAjUAEal6PXqE+w40NcHf/V24D8E//AP86U9JV5YsNQARyYw99ticIfjo\nozBi6DvfyW6GQA1ARDKnb9+QGXj55bBXUFcHt9ySvQyBGoCIZNa++4bMwFNPhZ8DDoCf/Sw7GQIN\nAxURiRRnCG68Ec44I30ZAuUARETKxB3mzAkZgl690pchUA5ARKRMzEJmYPFiuPzyzRmCpUuTrix+\nagAiIlvRtWvIDKxaBSedBKecEubXrEm6svioAYiIbEePHiEz0NQE++wDo0aF+TffTLqy0qkBiIi0\nwR57hMzAihXQ0hLuQ1BfX9kZAjUAEZF22HvvkBl45RX43e9ChuCHP6zMDIEagIhIBwwdGjIDc+fC\nvHkwfDj89KeVlSHQMFARkRg8/3zIELzzTrgPQbkyBMoBiIikUHGGoLY2ZAiOOy7ebSgHICKSQsUZ\ngiuugM99Ds46K70ZAjUAEZGYFTIEjY0hP3DqqXDxxbB6ddKVbanDDcDMZprZCjNbbGYPmVnPosem\nmFmTma00s9PiKVVEpLLstBNcdVVoBPvuC0ceGebTkiEoZQ9gLnCQux8GNAJTAMxsBDABGAGMBW43\nM+1piEhm7bFHyAysWBHmCxmCd95JsqoSGoC7z3P3lmh2PjAomh4H3OPum9x9DfAqMLqkKkVEqsDe\ne4fMwCuvwO9/D8OGJZshiOub+aXA49H0AGBt0WNrgYExbUdEpOINHRoyA0lnCGq296CZzQP6beWh\nqe4+J3rOdcAH7v6L7axqq+M96+vrP57O5XLkcrkdlCsiUj0OPRQefXRzhmDmzHAfgjPP3JwhyOfz\n5PP5smy/pByAmV0CXAGc7O5/i5ZNBnD36dH8k8D17j6/1WuVAxARibiHZjB1KvTsue0MQSpyAGY2\nFrgGGFf48I/MBs43s+5mNhSoAxaUVqaISHUzC5mBRYvgi1+Ez38+zC9ZUr5tlnIO4FZgN2CemS00\ns9sB3L0BuB9oAJ4AJumrvohI23TtGjIDq1aFDMFpp8FFF5UnQ6BLQYiIpNi778LNN8Ott4Zk8a23\npuAQkIiIlN/uu2/OEMR9cTntAYiIVJBUnAQWEZHKpgYgIpJRagAiIhmlBiAiklFqACIiGaUGICKS\nUWoAIiIZpQYgIpJRagAiIhmlBiAiklFqACIiGaUGICKSUWoAIiIZpQYgIpJRagAiIhmlBiAiklFq\nACIiGaUGICKSUWoAIiIZpQYgIpJRHW4AZvY9M1tsZovM7GkzG1z02BQzazKzlWZ2WjyliohInErZ\nA5jh7oe5+0jgYeB6ADMbAUwARgBjgdvNrOr2NPL5fNIllET1J0v1J6eSa49bhz+Y3f3dotndgLei\n6XHAPe6+yd3XAK8CoztcYUpV+n8i1Z8s1Z+cSq49bjWlvNjMbgAuAt5n84f8AOCloqetBQaWsh0R\nEYnfdvcAzGyemS3dys9ZAO5+nbsPAf4d+OftrMpjrFlERGJg7qV/NpvZEOBxdz/YzCYDuPv06LEn\ngevdfX6r16gpiIh0gLtbHOvp8CEgM6tz96ZodhywMJqeDfzCzH5AOPRTByxo/fq4/gEiItIxpZwD\nmGZmw4GPgN8BVwK4e4OZ3Q80AB8CkzyO3QwREYlVLIeARESk8sQ6Pt/Mfmxm681sadGy0Wa2wMwW\nmtnLZnZUtLyHmd1jZkvMrKFw7iB6bFR0srnJzH4YZ43trP0wM3sxqnG2me1e9NhWw25J1N7e+s3s\nVDN7JVr+ipmdWEn1Fz0+xMw2mtnVlVa/mR0aPbYserx7pdSfwvfuYDN7xsyWR3/Pr0bLe0cDWRrN\nbK6Z1Ra9JjXv3/bWH+v7191j+wGOBw4HlhYtywOfjKY/BTwTTV9CyAsA7AysBoZE8wuA0dH048DY\nOOtsR+0vA8dH018AvhtNjwAWAd2AfQhZB0uq9g7UPxLoF00fBKwtek3q6y96/EHgPuDqSqqfcOh1\nMXBINN8L6FJB9aftvdsPGBlN7wasAg4EZgDfjJZfC0yPplP1/u1A/bG9f2PdA3D354HmVovfAHpG\n07XA60XLdzWzrsCuwAfAO2bWH9jd3Qsnjn8KjI+zzq3ZRu110XKAXwPnRtNbC7t9IqnaoX31u/si\nd18XLW8AdjazbpVSP4CZjQd+T6i/sKxS6j8NWOLuS6PXNrt7SwXVn7b37jp3XxRNbwRWEAagnA3M\nip42q6iWVL1/21t/nO/fzrhEw2TgZjP7AzATmArg7k8B7xD+M60BZrr7BsI/fG3R618nuSDZcjMb\nF02fBxSudzSALWsshN1aL0+ydth2/cXOBX7r7ptI198etlG/me0GfBOob/X8iqgfGAa4mT1pZr81\ns2ui5RVRf5rfu2a2D2FPZj7Q193XRw+tB/pG06l9/7ax/mIlvX87owHcDXzVQ2Dsa9E8ZvZ5wu5j\nf2Ao8A0zG9oJ9bTHpcAkM3uFsGv2QcL1tNd26zezg4DpwJcSqK0ttlV/PfBP7v4ekObhxNuqvwY4\nDrgw+n2OmZ1E+gKTW60/re/d6IvBL4GrfMtL1eDhmEja/r5baG/9cbx/S7oURBuNdvdToukHgbui\n6WOBX7n7R8CfzOwFYBTwG2BQ0esHsfmwUady91XAJwHMbBhwRvTQ62z5bXoQofO+Tkpqh+3Wj5kN\nAh4CLnL31dHitNd/evTQaOBcM5tBOKzYYmbvE/49aa6/8Pd/DXjO3d+OHnscOAL4Oemuv/D3T917\n18y6ET48f+buD0eL15tZP3dfFx0eeTNanrr3bzvrj+392xl7AK+a2QnR9ElAYzS9MprHzHYFjgZW\nRse23jGzT5iZEa419DAJMLO9ot9dgG8BP4oemg2cb2bdo28+dcCCNNUO264/Gk3wGHCtu79YeL67\nv0G6678jqvP/uftQdx9KuATJDe5+e6X8/YGngEPMbGczqwFOAJZXQP13RA+l6r0bbetuoMHdiy9J\nMxuYGE1PLKolVe/f9tYf6/s35rPZ9wB/JOwqvkYYOXAk4XjWIuBF4PDouTsRvvEsBZaz5UiOUdHy\nV4Fb4qyxHbVfCnyVcEZ+FXBjq+dPjepbSTTKKana21s/4c28kZDeLvzsWSn1t3rd9cDXK+nvHz3/\nc8CyqNbplVR/Ct+7xwEt0WdM4f/zWKA34eR1IzAXqC16TWrev+2tP873r4JgIiIZVXU3ahERkbZR\nAxARySg1ABGRjFIDEBHJKDUAEZGMUgMQEckoNQARkYxSAxARyaj/D10ke1Zz54oLAAAAAElFTkSu\nQmCC\n", "text": [ "" ] } ], "prompt_number": 12 }, { "cell_type": "markdown", "metadata": {}, "source": [ "The fit isn't very good, we need to iterate between these parameter updates in a loop to improve the fit, we have to do this several times," ] }, { "cell_type": "code", "collapsed": false, "input": [ "for i in np.arange(10):\n", " m = ((y - c)*x).sum()/(x*x).sum()\n", " c = (y-m*x).sum()/y.shape[0]\n", "print(m)\n", "print(c)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "-0.398725964251\n", "783.527379727\n" ] } ], "prompt_number": 13 }, { "cell_type": "markdown", "metadata": {}, "source": [ "And let's try plotting the result again" ] }, { "cell_type": "code", "collapsed": false, "input": [ "f_test = m*x_test + c\n", "plt.plot(x_test, f_test, 'b-')\n", "plt.plot(x, y, 'rx')" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 14, "text": [ "[]" ] }, { "metadata": {}, "output_type": "display_data", "png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEACAYAAAC6d6FnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH2lJREFUeJzt3XuUFNW59/HvwwBJXnWJoIeLSsAjykUQBoJjkNh4ARQv\nYDwh+mqMLnyNAyJe4jUuxrjMQfBuBPXEHIlJeEMOCeIViTJqiAExXB1QNHCO8gbUCCoCivC8f+wa\nHccBe6arp6q7f5+1ZlFd1V31MND99H527b3N3RERkdLTIukAREQkGUoAIiIlSglARKREKQGIiJQo\nJQARkRKlBCAiUqJySgBm9nUzW2hmS82sxsz+Pdrf1szmmdlrZva0mbWJJ1wREYmL5ToOwMz+l7tv\nNbOWwJ+BK4HTgHfdfbKZXQ3s5+7X5B6uiIjEJecSkLtvjTZbA2XAJkICmB7tnw6MzPU6IiISr5wT\ngJm1MLOlwEZgvru/ArR3943RUzYC7XO9joiIxKtlridw911AXzPbF5hrZkPqHXcz03wTIiIpk3MC\nqOXu75vZ40B/YKOZdXD3DWbWEXi7/vOVFEREmsbdLY7z5HoX0P61d/iY2TeAE4ElwBzgvOhp5wGz\nG3q9u+f1Z9UqJ5NxysudRYviPffEiRPzHn8+fxS/4i/V+As5dvd4vzfn2gfQEXg26gNYCDzq7s8A\nk4ATzew14LjocbPr3h2efRYuvRROPRUuuQTefz+JSERE0ienBODuK9y93N37unsfd58S7X/P3U9w\n98Pcfai7b44n3MYzgx/8AF55BbZvh5494fe/h5gTqYhIwSmZkcDt2sF//Af87ndw440wYgT8/e9N\nP18mk4kttiQo/mQp/uQUcuxxy3kgWJMvbOZJXfuTT+D22+HWW+GKK8JP69aJhCIi0ihmhsfUCVyS\nCaDW2rUwdiz893/DfffB4MGJhiMi8pWUAGLkDrNmwYQJMGwYTJ4cykUiImkUZwIomT6A3TGDM8+E\nmhrYay/o1QumT1cnsYgUv5JvAdS3eDFcdBHss08oC3XvnnREIiKfUwsgjwYMgEWL4Iwz4Jhj4IYb\nYNu2pKMSEYmfEkADyspg/HhYtgxWrYLevWHevKSjEhGJl0pAWXj8cRg3Do4+Otw+2qFD0hGJSKlS\nCaiZjRgRRhJ37hxaA9Omwa5dSUclIpIbtQAaacUK+NGPYOfO0Enct2/SEYlIKVELIEG9e8MLL8CY\nMTB0aBhFvGVL0lGJiDSeEkATtGgREsDKlfDOO2HswCOPJB2ViEjjqAQUg2efhYsvhh494O67Q1+B\niEg+qASUMscdB8uXQ3l5+LntNvj006SjEhHZM7UAYrZmTWgNvPsu3H8/HHVU0hGJSDFRCyDFunUL\ng8auugpGjoTKStic2HI4IiK7pwSQB2Zw9tlhgrldu8IqZDNmaII5EUkXlYCawYsvhgnmOnSAqVPh\n0EOTjkhECpVKQAXm6KPh5ZfDuIGKCrjpJvj446SjEpFSpwTQTFq1giuvDIngpZfCCOLq6qSjEpFS\nphJQAtxh9uww4+jxx8OUKXDAAUlHJSKFQCWgAmcGo0aFTuK2beGII+DBBzXBnIg0L7UAUmDJktBJ\n/LWvhQnmevVKOiIRSSu1AIpMv37hTqHvfx8yGbjuOti6NemoRKTYKQGkRFkZjB0bViH7+99DWejJ\nJ5OOSkSKmUpAKfXUUyEh9O8Pd94JnTolHZGIpIFKQCVg+PAw3fRhh0GfPvDzn4dFaERE4pJTC8DM\nDgZ+BfwL4MAD7n63mbUFfgd8E1gHfM/dN9d7rVoAWaqpCauQbdsWJpgrL086IhFJSppaADuAy9y9\nF1ABjDWzHsA1wDx3Pwx4JnosTdSzJzz3XJhY7qSTYMIE+PDDpKMSkUKXUwJw9w3uvjTa3gKsAg4E\nTgOmR0+bDozM5ToSxg6cf35YnP7990NSmDVLE8yJSNPF1glsZl2A54AjgP9x9/2i/Qa8V/u4zvNV\nAsrBc8+FdQcOOST0D3TpknREItIc0lQCAsDM9gZmAZe6+xeKE9GnvD7pY3bssbB0aZhobsAAmDwZ\nduxIOioRKSQtcz2BmbUifPg/7O6zo90bzayDu28ws47A2w29tqqq6rPtTCZDJpPJNZyS0ro1XH99\nGEBWWQkPPxw6ib/97aQjE5G4VFdXU52nmSNzvQvICDX+f7r7ZXX2T4723WJm1wBt3P2aeq9VCShG\n7jBzJlx+OYwYAZMmhXmGRKS4pKkENAg4BxhiZkuin+HAJOBEM3sNOC56LHlkBqNHh07iVq3CfEK/\n/rU6iUVk9zQSuEgtWhQmmGvbFqZNCwPKRKTwpakFICk1cGBYeOaUU0KfQFUVbN+edFQikiZKAEWs\nZUu47LIw3fSyZWFKiWeeSToqEUkLlYBKyJw5cMklMHgw3H47/Mu/JB2RiDSWSkDSJKedFjqJO3YM\n000/8IBWIRMpZWoBlKhly8IEc2Zh7EDv3klHJCLZUAtAcnbkkbBgAZx3Hhx3HFx1FXz0UdJRiUhz\nUgIoYS1ahFtFV66E9evD2IHHHks6KhFpLioByWfmzQtTSvTpA3fdBQcdlHREIlKfSkCSFyeeCCtW\nhA7ivn3DUpSffpp0VCKSL2oBSINWrw7TTb//fugk/ta3ko5IREAtAGkG3bvDs8+GgWSnngrjxoVk\nICLFQwlAdssMzj03rEn88cdhFbKZMzXBnEixUAlIsvbnP4exAwcfDPfeG1YjE5HmpRKQJOKYY+Bv\nf4NMJkw297OfwSefJB2ViDSVWgDSJGvXhn6BdevgvvvC/EIikn9xtgCUAKTJ3GHWLJgwAYYNC+sS\nt2uXdFQixU0lIEkFMzjzzNBJvNdeoZP4oYfUSSxSKNQCkNgsXhw6iffeO5SFundPOiKR4qMWgKTS\ngAGwcCGccUboML7hBti2LemoRGR3lAAkVmVlMH58mG569eowzfTTTycdlYg0RCUgyavHHw93C1VU\nwB13QIcOSUckUthUApKCMWJEWIXsm98MrYFp07QKmUhaqAUgzWblyrD+wM6doZO4b9+kIxIpPGoB\nSEE64gh44QUYMwaGDoUrroAtW5KOSqR0KQFIs2rRIiSAlSvhnXfC2IHZs5OOSqQ0qQQkiZo/P4wd\n6N4d7rkHOndOOiKRdFMJSIrGkCGwfDn07w/l5XDbbVqFTKS5qAUgqbFmTViF7N13QydxRUXSEYmk\nj1oAUpS6dQsL0191FYwaFZLB5s1JRyVSvHJOAGb2SzPbaGYr6uxra2bzzOw1M3vazNrkeh0pDWZw\n9tlhgjn30Ek8Y4YmmBPJh5xLQGY2GNgC/Mrde0f7JgPvuvtkM7sa2M/dr6n3OpWA5Cu9+GIYO9Ch\nA0ydCocemnREIslKVQnI3V8ANtXbfRowPdqeDozM9TpSmo4+Gl5+OYwbqKiAm24K6xOLSO7y1QfQ\n3t03RtsbgfZ5uo6UgFat4MorQyJ46SU48kiork46KpHC1zLfF3B3N7MGaz1VVVWfbWcyGTKZTL7D\nkQL2zW/CI4+Enx/8INxCeuutcMABSUcmkj/V1dVU5+kbTyy3gZpZF+DROn0Aq4GMu28ws47AfHfv\nXu816gOQJvvwQ5g4EX7zm7A4/fnnh1HGIsUuVX0AuzEHOC/aPg/QYH+J1T77wO23w1NPwf33w7HH\nhllHRSR7cdwGOgP4C3C4mb1pZucDk4ATzew14LjosUjs+vULdwqddRZkMnDttbB1a9JRiRQGjQSW\novGPf8Bll8GiRXDvvXDSSUlHJBK/OEtASgBSdObOhcrKML/QnXdCp05JRyQSn0LoAxBJzLBhYbrp\nww6DPn3CLKM7dyYdlUj6qAUgRa2mJswptHVr6CwuL086IpHcqAUgkqWePcOgscrK0CcwYUK4hVRE\nlACkBJiFcQKvvAIffBCSwqxZmmBORCUgKTnPPRfKQoccAj//OXTpknREItlTCUgkB8ceC0uXhonm\n+veHW26BHTuSjkqk+akFICXtjTdg7FhYvz50En/720lHJLJnGgcgEiN3mDkTLr8cRoyASZOgbduk\noxJpmEpAIjEyg9GjQydxq1ahk/jhh9VJLMVPCSAfHn/8y4vZbt4c9ktqtWkTppCYMydMNHfCCfDq\nq0lHJZI/SgD5MGgQXH/950lg8+bweNCgZOOSrAwcGBaeOeWU8E9WVQXbtycdlUj81AeQL7Uf+j/+\nMUyZAjffHL5iSkF5800YPz6Uh6ZNg+OPTzoiKXXqA8iHbMo2jSnttGkTPvy7dg1/6sO/IB18MPzx\nj2HlsQsugHPOgY0bv/p1IoVACaBWNmWbxpR2Nm8O3/zXrg1/1k8c6icoKKedFloBHTtC797wwAOw\na1fSUYnkyN0T+QmXTplNm9wrK93Xrg1/btqU23Nqj9V/nO1z3N0fe+zL+zZtCvvzJYlrFpBly9wr\nKtyPPtp9+fKko5FSE312xvM5HNeJGn3hNCYA9/DBDuHPpj4n2w/QuJJJthob156ume25snleASa5\nnTvd77vPff/93R884zHf8pYSpjQPJYB8iasF0BjZJJy4rtmYZPJV18z2XHG2huIUUwtswwb3MWdu\n8ul7V/qTM5qxNVcKrbRS+Ds2gRJAPiTxQdWYD/ZsEkVzXzPbcyWRWLMRYwts/h83+cP7Vvr/GbrW\nP/xBDudKayst2+fFea44y6Rp/l00khJAPjT3P2ic38YbK85WR7aJKY7SWj7E+LvYvjqcq2+btX7H\nHe47djThXGltpWX7vHxds7n63JL4XTSSEkAxiPObXmPE+K23JFoAtRrRGnrv7EofMWiT9+vnvmhR\nbudKVSst2+fFfc24knSafxeNoARQSpJodcTZPE/wm9Ie5fmb9q6LK33GtE3evr372LHumzdnea5a\naW2lZfu8uM4VZzKMM658XDNLSgDSNEl0Qhb6XUDZJIo9nOuf/3QfM8a9Uyf3P/wyJIZmK3tke664\nnxfXueJM0nH/HeO+ZiMoAYg0l5iS0wsvuF/c+TH/7vGb/I039nCutLbSsn1enOeKM0mn+XfRSEoA\nIgXo44/dJ01yb9fO/eabw+MvSWsrLdvnxX3NbOguoCb/aDI4kWa2di2MGwfr1sF998HgwUlHJIVE\nK4KJFDh3mDULJkyAoUPDdFHt2iUdlRQCzQYqUuDM4MwzoaYG9tkHevWChx7SKmTSvPLWAjCz4cCd\nQBnwC3e/pd5xtQBEIosXw49+BHvvHdYd6NEj6YgkrVLfAjCzMuDnwHCgJ3CWmem/tMhuDBgACxfC\nGWeEPoGf/AS2bUs6Kil2+SoBDQRed/d17r4D+L/A6Xm6lkhRKCsLq48tWxbWIu7dG55+OumopJjl\nKwEcCLxZ5/Fb0T4R+QoHHgi//z3cfTdcdBGcdRZs2JB0VFKMWubpvFkV96uqqj7bzmQyZDKZPIUj\nUnhOPjmsQvbTn4bWwI03hoRQVpZ0ZNKcqqurqa6uzsu589IJbGYVQJW7D48eXwvsqtsRrE5gkeyt\nXBk6iXfsgPvvh759k45IkpL6TmBgMdDNzLqYWWtgNDAnT9cSKXpHHAHPPw8XXhjGDVxxBWzZknRU\nUujykgDc/VNgHDAXqAF+5+6r8nEtkVLRogWMGRNaA++8Az17wuzZSUclhUwjgUUK1Pz5oSzUvTvc\ncw907px0RNIcCqEEJCJ5NmQILF8O/ftDeTncemvoIxDJlloAIkVgzRqorIS33w6dxBUVSUck+aIW\ngIh8QbduYdDY1VfDqFFw8cWweXPSUUnaKQGIFAkzOPvsMMGce+gknjFDE8zJ7qkEJFKkXnwxdBK3\nbw9Tp8KhhyYdkcRBJSAR+UpHHx1mGR06NPQJ3HQTfPxx0lFJmigBiBSxVq3gyivh5ZfhpZfgyCMh\nT7MKSAFSCUikRLjDI4+EGUeHDAm3jR5wQNJRSWOpBCQijWYGI0eGCebatQvTSzz4IOzalXRkkhS1\nAERK1JIloZO4deuwOH2vXklHJNlQC0BEctavH/zlL2G9gUwGrr0Wtm5NOippTkoAIiWsrCyMIF6+\nHNauDWWhJ59MOippLioBichn5s4NCaG8HO66Czp1SjoiqU8lIBHJi2HDwnTThx8OffqEWUZ37kw6\nKskXtQBEpEGrVoVO4q1bwwRz5eVJRySgFoCINIMePcKgscpKOOkkuPRS+OCDpKOSOCkBiMhumcH5\n54exAx9+GG4VnTVLE8wVC5WARCRrzz8fykJdu8K990KXLklHVHpUAhKRRHznO7B0KQwaBAMGwC23\naBWyQqYWgIg0yRtvwNixsH59GEk8aFDSEZWGOFsASgAi0mTuMHMmXH45nHxyaBG0bZt0VMVNJSAR\nSQUzGD06rEL2ta+FVcgeflidxIVCLQARic2iRXDRRaEVMHVqGFAm8VILQERSaeDAsPDMKaeEPoGJ\nE2H79qSjkt1RAhCRWLVsCZddFqabXrEiTCnxpz8lHZU0RCUgEcmrRx+FceNg8GC47bawSL00nUpA\nIlIwTj01jCTu2BF694YHHtAqZGmhFoCINJvly0MnsVkYO9CnT9IRFZ5UtADM7N/M7BUz22lm5fWO\nXWtma8xstZkNzT1MESkGffrAggVw3nlw/PFw1VXw0UdJR1W6cikBrQBGAc/X3WlmPYHRQE9gODDV\nzFRqEhEAWrQIrYCVK8Mo4l69Qj+BNL8mfzC7+2p3f62BQ6cDM9x9h7uvA14HBjb1OiJSnNq3h9/8\nBn7xizCS+Iwz4K23ko6qtOTjm3knoO4/41vAgXm4jogUgRNOCLeL9u4NffvCnXfCp58mHVVpaLmn\ng2Y2D+jQwKHr3L0xjbYGe3urqqo+285kMmQymUacUkSKxde/DjfeCGefDRdfDL/6VegkHqjaAdXV\n1VRXV+fl3DnfBWRm84Er3P1v0eNrANx9UvT4KWCiuy+s9zrdBSQiX+IOv/41/PjHcOaZcPPNsO++\nSUeVHqm4C6ieusHMAb5vZq3NrCvQDVgU03VEpMiZwbnnhgnmPv44TDA3c6YmmMuHJrcAzGwUcDew\nP/A+sMTdT4qOXQdcAHwKXOrucxt4vVoAIvKVFiwIq5AddFBYheyQQ5KOKFlaD0BESsqOHXD77TBl\nSrhj6MoroXXrpKNKRhpLQCIiedOqFVx9dZhpdMGCcLfQ889/9etkz9QCEJGC4g6zZsGECTB0KEye\nDPvvn3RUzUctABEpWWbh7qCaGthnnzCS+KGH1EncFGoBiEhBe/nlMLXEXnuFsQM9eiQdUX6pBSAi\nEunfHxYuDK2CwYPhJz+BbduSjqowKAGISMErK4NLLoFly+DVV8O0Ek8/nXRU6acSkIgUnSeegLFj\n4aij4I47wmI0xUIlIBGRPTj55LAKWdeuYQ2CqVNh586ko0oftQBEpKitXBlGEu/YAfffH8YQFDK1\nAEREsnTEEWHQ2IUXhnEDl18OW7YkHVU6KAGISNFr0QLGjAlloX/+M0wwN3t20lElTyUgESk58+eH\ndQcOPxzuuQc6d046ouypBCQikoMhQ8ItowMGQHk53Hpr6CMoNWoBiEhJW7MGKivh7bdDJ3FFRdIR\n7ZlaACIiMenWLQwau/pqGDUqlIY2bUo6quahBCAiJc8srEdcUxMe9+oFv/1t8U8wpxKQiEg9f/1r\nmGCuffswiOzQQ5OO6HMqAYmI5FFFBSxeHMYNVFTATTeF9YmLjRKAiEgDWrUKS0/+7W8hGRx5ZLh9\ntJioBCQikoXZs2H8+HAL6a23wgEHJBOHSkAiIs1s5MgwkrhduzC9xIMPwq5dSUeVG7UAREQaacmS\nMMFc69ZhFbJevZrv2moBiIgkqF8/+Mtf4KyzIJOBa6+FrVuTjqrxlABERJqgrCyMIF6+HNatC62A\nJ55IOqrGUQlIRCQGc+eGhFBeDnfdBZ065ec6KgGJiKTMsGFh8ZnDDw+rkN1zT/pXIVMLQEQkZqtW\nhU7ijz4KE8z17x/fudUCEBFJsR49oLoaxo0L6xNfeil88EHSUX1ZkxOAmU0xs1VmtszM/mBm+9Y5\ndq2ZrTGz1WY2NJ5QRUQKhxn88Idh7MCHH4ZO4lmz0jXBXJNLQGZ2IvCMu+8ys0kA7n6NmfUEfgt8\nCzgQ+BNwmLvvqvd6lYBEpGQ8/3woC3XtCvfeC126NO08qSgBufu8Oh/qC4GDou3TgRnuvsPd1wGv\nAwNzilJEpMB95zuwdCkMGhRWIrvlluRXIYurD+ACoPYO2E7AW3WOvUVoCYiIlLTWreG662DhwjCx\nXL9+sGBBcvHsMQGY2TwzW9HAz6l1nnM98Im7/3YPp1KtR0Qk8q//Ck8+CTfcAN/7Hlx4Ibz3XvPH\n0XJPB939xD0dN7MfAicDx9fZvR44uM7jg6J9X1JVVfXZdiaTIZPJ7OlyIiJFwwxGj4bhw+H666Fn\nT5gyBc45JxyrVV1dTXV1dX5iyKETeDhwG3Csu79bZ39tJ/BAPu8EPrR+j686gUVEPrdoUViFbL/9\nYNq0MKCsIanoBAbuAfYG5pnZEjObCuDuNcBMoAZ4EqjUJ72IyJ4NHAgvvQSnnRY6iidOhO3b83tN\njQQWEUmZN98Mg8dWrgxrEp9wwufH4mwBKAGIiKTUo4+G0cSDB8Ntt4VF6tNSAhIRkTw69VSoqQkz\ni/buHeYVipNaACIiBWD58tBJ/Ne/qgQkIlJydu2CsjKVgERESk6LmD+xlQBEREqUEoCISIlSAhAR\nKVFKACIiJUoJQESkRCkBiIiUKCUAEZESpQQgIlKilABEREqUEoCISIlSAhARKVFKACIiJUoJQESk\nRCkBiIiUKCUAEZESpQQgIlKilABEREqUEoCISIlSAhARKVFKACIiJUoJQESkRCkBiIiUKCUAEZES\n1eQEYGY3mdkyM1tqZs+Y2cF1jl1rZmvMbLWZDY0nVBERiVMuLYDJ7n6ku/cFZgMTAcysJzAa6AkM\nB6aaWdG1NKqrq5MOISeKP1mKPzmFHHvcmvzB7O4f1nm4N/ButH06MMPdd7j7OuB1YGCTI0ypQv9P\npPiTpfiTU8ixx61lLi82s5uBc4FtfP4h3wn4a52nvQUcmMt1REQkfntsAZjZPDNb0cDPqQDufr27\ndwb+E7hzD6fyGGMWEZEYmHvun81m1hl4wt2PMLNrANx9UnTsKWCiuy+s9xolBRGRJnB3i+M8TS4B\nmVk3d18TPTwdWBJtzwF+a2a3E0o/3YBF9V8f119ARESaJpc+gH83s8OBncAbwMUA7l5jZjOBGuBT\noNLjaGaIiEisYikBiYhI4Yn1/nwz+6WZbTSzFXX2DTSzRWa2xMxeMrNvRfu/bmYzzGy5mdXU9h1E\nx/pHnc1rzOyuOGNsZOxHmtmLUYxzzGyfOscaHOyWROyNjd/MTjSzxdH+xWY2pJDir3O8s5ltMbMr\nCi1+M+sTHVsZHW9dKPGn8L17sJnNN7NXot/n+Gh/2+hGltfM7Gkza1PnNal5/zY2/ljfv+4e2w8w\nGOgHrKizrxoYFm2fBMyPtn9IGC8A8A1gLdA5erwIGBhtPwEMjzPORsT+EjA42j4f+Gm03RNYCrQC\nuhDGOlhSsTch/r5Ah2i7F/BWndekPv46x/8L+B1wRSHFTyi9LgN6R4/3A1oUUPxpe+92APpG23sD\nrwI9gMnAVdH+q4FJ0Xaq3r9NiD+292+sLQB3fwHYVG/3P4B9o+02wPo6+/cyszJgL+AT4AMz6wjs\n4+61Hce/AkbGGWdDdhN7t2g/wJ+A70bbDQ12Oyqp2KFx8bv7UnffEO2vAb5hZq0KJX4AMxsJ/J0Q\nf+2+Qol/KLDc3VdEr93k7rsKKP60vXc3uPvSaHsLsIpwA8ppwPToadPrxJKq929j44/z/dscUzRc\nA9xmZv8DTAGuA3D3ucAHhP9M64Ap7r6Z8Bd/q87r15PcQLJXzOz0aPvfgNr5jjrxxRhrB7vV359k\n7LD7+Ov6LvCyu+8gXb972E38ZrY3cBVQVe/5BRE/cBjgZvaUmb1sZj+O9hdE/Gl+75pZF0JLZiHQ\n3t03Roc2Au2j7dS+f7OMv66c3r/NkQAeBMZ7GDB2WfQYMzuH0HzsCHQFrjSzrs0QT2NcAFSa2WJC\n0+yThONprD3Gb2a9gEnARQnElo3dxV8F3OHuW4E03068u/hbAscAZ0d/jjKz40jfgMkG40/rezf6\nYjALuNS/OFUNHmoiafv9fkFj44/j/ZvTVBBZGujuJ0Tb/wX8Itr+NvBHd98JvGNmC4D+wJ+Bg+q8\n/iA+Lxs1K3d/FRgGYGaHASOiQ+v54rfpgwiZdz0piR32GD9mdhDwB+Bcd18b7U57/CdHhwYC3zWz\nyYSy4i4z20b4+6Q5/trf/5vA8+7+XnTsCaAc+DXpjr/295+6966ZtSJ8eD7s7rOj3RvNrIO7b4jK\nI29H+1P3/m1k/LG9f5ujBfC6mR0bbR8HvBZtr44eY2Z7ARXA6qi29YGZHWVmRphraDYJMLMDoj9b\nAD8BpkWH5gDfN7PW0TefbsCiNMUOu48/upvgceBqd3+x9vnu/g/SHf99UZzfcfeu7t6VMAXJze4+\ntVB+/8BcoLeZfcPMWgLHAq8UQPz3RYdS9d6NrvUgUOPudaekmQOcF22fVyeWVL1/Gxt/rO/fmHuz\nZwD/j9BUfJNw58AAQj1rKfAi0C967tcI33hWAK/wxTs5+kf7XwfujjPGRsR+ATCe0CP/KvCzes+/\nLopvNdFdTknF3tj4CW/mLYTR27U/+xdK/PVeNxG4vJB+/9Hz/zewMop1UiHFn8L37jHArugzpvb/\n83CgLaHz+jXgaaBNndek5v3b2PjjfP9qIJiISIkquoVaREQkO0oAIiIlSglARKREKQGIiJQoJQAR\nkRKlBCAiUqKUAERESpQSgIhIifr/WBTxqw6qEjoAAAAASUVORK5CYII=\n", "text": [ "" ] } ], "prompt_number": 14 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Clearly we need more iterations than 10! \n", "\n", "Try add more iterations above to try and get closer to the solution.\n", "\n", "## Direct Solution with Linear Algebra\n", "\n", "Hopefully, you are now persuaded of the merits of solving the entire system, simultaneously, using linear algebra. To do that, we need to make a design matrix of the data, which includes the $x_0=1$ column, to represent the bias, remember (from the lecture notes) that we are now moving to a system where our prediction is given by an inner product:\n", "\n", "$$f(\\mathbf{x}_i) = \\mathbf{x}_i^\\top\\mathbf{w}$$\n", "\n", "where each vector $\\mathbf{x}_i$ is given by appending a 1 onto the original vector\n", "\n", "$$\\mathbf{x}_i = \n", "\\begin{bmatrix} \n", "1 \\\\\\\n", "x_i\n", "\\end{bmatrix}$$\n", "\n", "We can do this for the entire data set to form a design matrix $\\mathbf{X}$,\n", "\n", "$$\\mathbf{X} = \\begin{bmatrix} \n", "\\mathbf{x}_1^\\top \\\\\\ \n", "\\mathbf{x}_2^\\top \\\\\\ \n", "\\vdots \\\\\\\n", "\\mathbf{x}_N^\\top\n", "\\end{bmatrix} = \\begin{bmatrix}\n", "1 & x_1 \\\\\\\n", "1 & x_2 \\\\\\\n", "\\vdots & \\vdots \\\\\\\n", "1 & x_N \n", "\\end{bmatrix},$$\n", "\n", "which in `numpy` is done with the following commands:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "X = np.hstack((np.ones_like(x), x)) # [ones(size(x)) x]\n", "print(X)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "[[ 1.00000000e+00 1.89600000e+03]\n", " [ 1.00000000e+00 1.90000000e+03]\n", " [ 1.00000000e+00 1.90400000e+03]\n", " [ 1.00000000e+00 1.90800000e+03]\n", " [ 1.00000000e+00 1.91200000e+03]\n", " [ 1.00000000e+00 1.92000000e+03]\n", " [ 1.00000000e+00 1.92400000e+03]\n", " [ 1.00000000e+00 1.92800000e+03]\n", " [ 1.00000000e+00 1.93200000e+03]\n", " [ 1.00000000e+00 1.93600000e+03]\n", " [ 1.00000000e+00 1.94800000e+03]\n", " [ 1.00000000e+00 1.95200000e+03]\n", " [ 1.00000000e+00 1.95600000e+03]\n", " [ 1.00000000e+00 1.96000000e+03]\n", " [ 1.00000000e+00 1.96400000e+03]\n", " [ 1.00000000e+00 1.96800000e+03]\n", " [ 1.00000000e+00 1.97200000e+03]\n", " [ 1.00000000e+00 1.97600000e+03]\n", " [ 1.00000000e+00 1.98000000e+03]\n", " [ 1.00000000e+00 1.98400000e+03]\n", " [ 1.00000000e+00 1.98800000e+03]\n", " [ 1.00000000e+00 1.99200000e+03]\n", " [ 1.00000000e+00 1.99600000e+03]\n", " [ 1.00000000e+00 2.00000000e+03]\n", " [ 1.00000000e+00 2.00400000e+03]\n", " [ 1.00000000e+00 2.00800000e+03]\n", " [ 1.00000000e+00 2.01200000e+03]]\n" ] } ], "prompt_number": 15 }, { "cell_type": "markdown", "metadata": {}, "source": [ "From the multivariate regression solution we derived in the lecture, the maximum likelihood solution for $\\mathbf{w}^*$ is given by\n", "\n", "$$\\mathbf{w}^* = \\left[\\mathbf{X}^\\top \\mathbf{X}\\right]^{-1} \\mathbf{X}^\\top \\mathbf{y}$$ \n", "\n", "First let's persuade ourselves of a few things. We suggested in the lecture that\n", "\n", "$$\\sum_{i=1}^n \\mathbf{x}_i\\mathbf{x}_i^\\top = \\mathbf{X}^\\top\\mathbf{X}$$\n", "\n", "We can show that this is, indeed the case for our data. First we need to know how to do matrix multiplication and transpose using numpy, this is done as" ] }, { "cell_type": "code", "collapsed": false, "input": [ "np.dot(X.T, X)" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 16, "text": [ "array([[ 2.70000000e+01, 5.28200000e+04],\n", " [ 5.28200000e+04, 1.03365648e+08]])" ] } ], "prompt_number": 16 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we will compute the same thing with a for loop" ] }, { "cell_type": "code", "collapsed": false, "input": [ "store = np.zeros((2, 2))\n", "for i in range(X.shape[0]):\n", " store += np.outer(X[i, :], X[i, :])\n", "print store" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "[[ 2.70000000e+01 5.28200000e+04]\n", " [ 5.28200000e+04 1.03365648e+08]]\n" ] } ], "prompt_number": 17 }, { "cell_type": "markdown", "metadata": {}, "source": [ "I hope that you agree that the first version is a little more compact. \n", "\n", "### Solving the System\n", "\n", "The solution for $\\mathbf{w}$ is given in terms of a matrix inverse, but numerically this isn't the best way to compute it. What we actually want python to do is to *solve* the system of linear equations given by\n", "\n", "$$\\mathbf{X}^\\top\\mathbf{X} \\mathbf{w} = \\mathbf{X}^\\top\\mathbf{y}$$\n", "\n", "for $\\mathbf{w}$. This can be done in numpy using the command" ] }, { "cell_type": "code", "collapsed": false, "input": [ "np.linalg.solve?" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 18 }, { "cell_type": "markdown", "metadata": {}, "source": [ "so we can obtain the solution using" ] }, { "cell_type": "code", "collapsed": false, "input": [ "w = np.linalg.solve(np.dot(X.T, X), np.dot(X.T, y))\n", "print w\n", "print c\n", "print m" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "[[ 2.88952457e+01]\n", " [ -1.29806477e-02]]\n", "783.527379727\n", "-0.398725964251\n" ] } ], "prompt_number": 19 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Allowing us to plot the fit as follows" ] }, { "cell_type": "code", "collapsed": false, "input": [ "m = w[1]; c=w[0]\n", "f_test = m*x_test + c\n", "print(m)\n", "print(c)\n", "plt.plot(x_test, f_test, 'b-')\n", "plt.plot(x, y, 'rx')" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "[-0.01298065]\n", "[ 28.89524574]\n" ] }, { "metadata": {}, "output_type": "pyout", "prompt_number": 20, "text": [ "[]" ] }, { "metadata": {}, "output_type": "display_data", "png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEACAYAAACwB81wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHc9JREFUeJzt3XmUVPWZ//H3gxgVt4ajQUdw0IlORuMogyIjGiuOa6PG\nNWpicLeRZMyiTmJ0DpAMSU7MTNT5zU86QR0dHJeoUbRREbXRcESjIi64oY0/UMTkQLkhsj2/P763\npWh6qeVW3Vt1P69z+vTd6tZTfbqe+tZzv/f7NXdHREQaX7+kAxARkdpQwhcRyQglfBGRjFDCFxHJ\nCCV8EZGMUMIXEcmI/sUcZGaLgA+BdcAadx/ZZX8OuBd4K9p0l7v/W3xhiohIpYpK+IADOXdf3ssx\ns939+BhiEhGRKiilpGMV7hcRkQQVm/AdmGVmz5jZBT3sP8jM5pvZDDPbK74QRUQkDsWWdEa7+1Iz\n2xF42MxedfcnCvY/Bwx195VmdgxwD7Bn3MGKiEj5rNSxdMxsAvCxu/97L8d0ACMKa/5mpkF7RETK\n4O6xlMz7LOmY2QAz2zZa3ho4EnixyzGDzcyi5ZGED5JNLvC6e93+TJgwIfEYshp/Pceu+JP/qff4\n41RMSWcw8Icon/cHbnH3mWbWEiXxVuAU4CIzWwusBE6PNUoREalYnwnf3TuA/brZ3lqw/F/Af8Ub\nmoiIxEl32hYpl8slHUJF6jn+eo4dFH/S6j3+OJV80bbsJzLzWj2XiEijMDO8VhdtRUSkMSjhi4hk\nhBK+iEhGKOGLiGSEEr6ISEYo4YuIZIQSvohIRijhi4hkhBK+iEhGKOGLiGSEEr6ISEYo4VeqrQ3y\n+Y235fNhu4hIiijhV2r0aLjiig1JP58P66NHJxuXiEgXGi0zDp1J/rLL4KqrYPJkaGpKOioRaQBx\njpaphB+XRYtgt92gowOGDUs6GhFpEBoeOW3y+dCy7+gIv7vW9EVEUkAJv1Kd5ZzJk0PLfvLkjWv6\nIiIpoZJOpdrawgXawpp9Pg9z5sCYMcnFJSINQTV8EZGMUA1fRERKpoQvIpIRSvgiIhmhhC8ikhFK\n+CIiGaGELyKSEUr4IiIZUVTCN7NFZvaCmc0zs6d7OOZaM3vDzOab2fB4wxQRkUr1L/I4B3Luvry7\nnWbWDHzJ3fcwswOB64BRMcUoIiIxKKWk09udXscDNwG4+1NAk5kNriQwERGJV7EJ34FZZvaMmV3Q\nzf5dgMUF60uAIZUGJyIi8Sm2pDPa3Zea2Y7Aw2b2qrs/0eWYrt8ANHCOiEiKFJXw3X1p9PvPZvYH\nYCRQmPDfAYYWrA+Jtm1k4sSJny/ncjlyuVzJAYuINLL29nba29urcu4+R8s0swHAZu7+kZltDcwE\nJrn7zIJjmoHvunuzmY0Crnb3UV3Oo9EyRURKFOdomcW08AcDfzCzzuNvcfeZZtYC4O6t7j7DzJrN\nbCHwCXBOHMGJiEh8NB6+iEiKaTx8EREpmRK+iEhGKOGLiGSEEr6ISEYo4YuIZIQSvohIRijhi4hk\nhBK+iEhGKOGLiGSEEr6ISEYo4YuIZIQSvohIRijhi4hkhBK+iEhGKOGLiGSEEr6ISEYo4YuIZIQS\nvohIRijhi4hkhBK+iEhGKOGLiGSEEr6ISEYo4YuIZIQSvohIRijhi4hkhBK+iEhGKOGLiGSEEr6I\nSEYo4YuIZERRCd/MNjOzeWZ2Xzf7cmb2QbR/npldGX+YIiJSqf5FHvc9YAGwbQ/7Z7v78fGEJCIi\n1dBnC9/MhgDNwFTAejoszqBERCR+xZR0fgNcBqzvYb8DB5nZfDObYWZ7xRZdtbS1QT6/8bZ8PmwX\nEWlQvZZ0zOxY4H13n2dmuR4Oew4Y6u4rzewY4B5gz+4OnDhx4ufLuVyOXK6nU1bZ6NFwxRUweTI0\nNYVk37leqK0tHNvUtGFbPg9z5sCYMbWNWUQyob29nfb29qqc29y9551mPwe+DawFtgS2A+5y97G9\nPKYDGOHuy7ts996eq+Y6k/xll8FVV21I/t0d090HQ9djRUSqwMxw91jK5r0m/C5Peihwqbsf12X7\nYMK3ADezkcAd7j6sm8enK+EDLFoEu+0GHR0wbFj3xxTzwSAiUiVxJvxie+l08iiAFgB3bwVOAS4y\ns7XASuD0OAKrunw+JPCOjt4TeVNTSPadHwxK9iJSp4pu4Vf8RGlq4ZdSqlELX0QSlEhJp+InSlPC\nL/ZirGr4IpIwJfxaUS8dEUmYEr6ISEbEmfA1eJqISEY0RMJ/801Yvrzv40REsqwhEv4998Duu8NZ\nZ8GTT4IqRyIim2qYGv6f/wz//d/w29/CVltBSwuceSZsv33VnlJEpOp00bYX69fDo49CayvMmgUn\nnwzjxsH++1f9qUVEYqeEX6T33oMbboDf/Q4GDQqJ/4wzYJttahqGiEjZlPBLtG4dPPwwTJkCjz8O\np50Wkv+++yYSjohI0ZTwK7BkCVx/PUydCrvsEmr9p50GAwYkHZmIyKaU8GOwdi088EBo9c+dC9/6\nVkj+e++ddGQiIhvoxqsY9O8Pxx0XRk947rnQm+eII+CQQ2DaNFi1KukIRUTildkWfnfWrIH77gut\n/nnzYOxYuPBC+Nu/TToyEckqtfCrZPPN4aSTYObMUObZfHP46lfhsMPg9tth9eqkIxQRKZ9a+H34\n7LNwJ++UKbBgAZxzTmj177570pGJSBaohV9DW2wRevE89hjMnh1a+SNHwlFHwd13hzKQiEg9UAu/\nDKtWwZ13hlb/W2/BeefBBRfArrsmHZmINBq18BO25ZZhnJ4//jHU+z/4AIYPh2OPDRd9163r5kFt\nbWHylEL5fNguIlIDSvgV+spX4NprYfHiMG7P5MlhvvOf/hTeeafgwNGjw/SInUm/c7rE0aMTiVtE\nskclnSp4/vkweNttt0EuF27oOvJI6PehJkQXkdLoTts68dFHcOutoda/YkXo3XP+4YvYceRu0NEB\nw4YlHaKIpJxq+HVi221Dkn/2WbjjDnh3QZ57D76K7zR3sOR7V7F+eb7vk4iIxEQJvwbM4IA98vzn\ndldw6uuT2at5GKctnMz/7n4F//mzPH/5S9IRikgWqKRTK21t4QJtVLN3h6dn5pnzqzn89NkxNDeH\nIZsPOSR8QIiIgGr4DWf5crj55nCh1yxc5B07FgYOTDoyEUmaEn6DcocnnggXeWfMgBNOCMl/1Ci1\n+kWySgk/A7pOyj5uXBizX5Oyi2RLzXvpmNlmZjbPzO7rYf+1ZvaGmc03s+FxBJZ1O+4Yuuu/9hr8\nx3+EsXyGDQtDODzzTNLRiUg9KraXzveABcAmTXQzawa+5O57ABcC18UXnvTrB4cfDr//PbzySriL\n99RTYcSIMDn7xx8nHaGI1Is+E76ZDQGagalAd18rjgduAnD3p4AmMxscZ5AS7LQT/OQnsHBhuEm3\nrS0M2HbRRTB/ftLRiUjaFdPC/w1wGbC+h/27AIsL1pcAQyqMS3qx2WZw9NFhnP4XXggfBMceGy7u\n3ngjrFyZdIQikkb9e9tpZscC77v7PDPL9XZol/Vur85OnDjx8+VcLkcu19sppRhDhsCECWGIns5J\n2S+9VJOyi9Sr9vZ22tvbq3LuXnvpmNnPgW8Da4Etge2Au9x9bMExU4B2d78tWn8VONTdl3U5l3rp\n1Mjbb8PUqXD99fA3fxMS/ymnhGGdRaS+JNIt08wOBS519+O6bG8GvuvuzWY2Crja3Ud183gl/BrT\npOwi9S/JwdM8CqDFzFoA3H0G8JaZLQRagfFxBCaV06TsIlJIN16lTZcxd4AwWcqcOTBmTMWn75yU\nvbUVXn5Zk7KLpJ2GR25kVZ4Zq3NS9kcfhccfD2WfAw/UpOwiWaAWfhrlazszliZlF0kvjaWTBYsW\nhdtqazwz1ksvhfF7brkF/vEfQw+f5ubQ919Eak8lnUaXz4eWfUdH+J2v3cxYRU/KLiJ1Rwk/bTrL\nOZMnh5b95Mkb1/RrZMCAcEF37lyYPh2WLg0fBieeCA8+COt7uu9aRFJLJZ20qXIvnUqer7tJ2c89\nFwZr5CSRqlENX+JT+I2iqWnT9W64hyGaW1vhrrvgiCNCrf9rXwuje4pIfJTwJV4V9Ar64AOYNi0k\n/08/DYn/7LNhhx2qG7JIVijhS/wq7BXkDk8+GRL/vfeGalBLiyZlF6mUeulIvGLoFWQGBx0EN90U\n+vIfcMCG0TqvuSbU/EUkWWrhZ10ZNfxiaVJ2kcqppCPxqVGvIE3KLlIeJXypW+vXh3F8Wlth1qww\nTn9LC+y/f9KRiaSTEr40hPfegxtuCJOxDxoUWv1nnAHbbJN0ZCLpoYQvDWXdOnj44VDrf/zxMJrn\nuHGw775JRyaSPCV8aVhLloSpGadOhV12CeWe004LQz2IZJESvjS8tWs3TMo+d64mZZfsUj98aXj9\n+8Nxx4VORM89F3rzHHFEuJFr2rQwhn9J2to2vb8gnw/bRTJCCV9S76//Gn72M3j7bfjBD+Dmm2Ho\nULjkEnj99SJPUuWZxETqgUo6UpfefDP07rnxxlDmGTcu3Nj1hS/08qAazyQmEgfV8KX2aj1sc5EK\nJ2VfsCAM3NbrpOwJzSQmUi7V8KX2UloSKZyUffbsPiZlT3AmMZE0UAtfitdXSSQl3wI6J2VvbQ2l\nn/POgwu/kWfolOqMGSRSTSrpSHJ6K4lUcSC2cr38ckj879/Yhh80mjO/27RhUvYUlKRE+qKSjhQn\n7q6IfZVEmpo2zMG7aFHiyR7CBd1rr4Ublo3h6NObNp6U/ZMmJXvJFnevyU94KqmpFSvcx48Pv7tb\nr9a5OjrcIfxOoXnz3MeNc29qcj/hBPcHHnBfty7pqES6F+XOWPKwSjqNLq6uiMXW5+uo66MmZZd6\noBq+lKZWXRGLreGn5OJuJ03KLmlW0xq+mW1pZk+Z2fNmtsDMftHNMTkz+8DM5kU/V8YRnMSgll0R\n58zZOLl31vTnzNn4uJR18TQLUzJOnRo+Gw89NNzR++Uvw69/DX/5SyJhicSvmLoPMCD63R+YCxzc\nZX8OmN7HOapQ3ZJexVnDj1tnLB0d6YmpwPr17nPmuI8d67799u7f/Kb77Nlhu0gtkVQN38wGALOB\ns9x9QcH2HHCJux/Xy2O9lOeSGKSsdLKJOrnrdfnyMH5Pa2v4NtDSAmPHwsCBSUcmWVDzbplm1s/M\nngeWAY8VJvuIAweZ2Xwzm2Fme8URnFRozJhNL5g2paQrYh3d9TpoEHz/+2HohilT4KmnwufU2WfD\nk0+GawAi9aDUFv72wEPAj929vWD7tsA6d19pZscA17j7nl0e6xMmTPh8PZfLkcvlKote6lMKb9Aq\nlSZll2ppb2+nvb398/VJkyYl10vHzP4V+NTdf93LMR3ACHdfXrBNJR0J0l5qKsH69fDYY6Hlr0nZ\npRpq2i3TzHYA1rp73sy2IrTwJ7n7IwXHDAbed3c3s5HAHe4+rMt5lPClMfTwgbXi/jlMWTyG3/5W\nk7JLfGpdw98ZeDSq4T8F3Ofuj5hZi5m1RMecArwYHXM1cHocwYmkUg/dSgceO5rLL4eFC0N1qq0N\ndt0VLroI5s9PNmQR0I1XIuUp8o5iTcouldKdtiJpUEK3Uk3KLuXSaJki1VDK6KIldivtbVL2W24p\nY1J2kTIo4Yt0KnbIh8JupMOGbRgSush7CbpOyn7TTWVMyi5SBpV0RAoVU5uvQrfSsiZll0xQDV+k\nmhIc8qHkSdml4amGL1ItCQ/50N2k7FcOb+Okw/IbT8peycxlkllq4Yt0SumQD6vey/P2mVfw/U8m\nM//tJr7zrTwXL7uCba+tn6EopHwq6YhUQ5qHfIg+fF7/+mUsu/Qqxi6ezN6jm2hpYcOk7NKQlPBF\nsqjg2sLKLw7j9ttDrf/dd+H88+G888LNXdJYVMMXyZou1xYGrM5zzjnhJq7p02HpUvjKV+DEE+HB\nB8OgbiJdqYUvknZFXlsonJQ9n4cLLtCk7I1AJR2RLCnx2oImZW8sSvgiUpQPPoBp00LyX7Uq9Ok/\n+2zYYYekI5NiKeGLSEncw3SMra1w773hi0FLSxjLx2JJJVItumgrknalDMRWA2Zw0EFh3J633oID\nDtgwWuc118CKFYmEJTWmhC9SDcUOxJYATcqeXSrpiFRLkZOkpIEmZU8v1fBF6kWCA7GVQ5Oyp49q\n+CL1IOGB2MrRrx/80z/B738Pr7wSRuk89VQYMSIM3/zxx1V88pRd92hESvgi1VDhJClliTlh7rQT\nXH55GKu/cFL28eOrNCl7Mdc99KFQGXevyU94KpGMuP9+9xUrNt62YkXYXs5xxVixwn38+A3n67oe\ng8WL3SdOdB8yxH3UKPcbb3T/5JPYTr8h5o6O7mOvwWtMmyh3xpOH4zpRn0+khC+yqbgTWF8JMyZr\n1rhPn+7e3Ow+aJD7xRe7v/xyTCfv6AipqaOj+/01eo1poYQv0kjiTmB9JcxilPDNY9Ei9yuvdN95\nZ/eDD3afNs3900/LfN5i/xZxvEb3eL9hVYkSvkijiSuBxfXhUcY3j9Wr3e+6y/2II9x32MH9hz90\nf+01Lz6pFvucxbzGuJ8zQUr4Io2k1km61GRYRlwLF7r/6EfuX/yi+7EHr/A3jhzvny2LIa5SPxSK\nSeQpLxEp4Ys0ijhbmNVo1fb1zaOP51y1yv2220LSv2Gr8f7LcR2eP7OCpFpKCaaURB7XN6wqUMIX\naRRJ1ZCLSYalHFPEh8fCWR3u4MMHdviRR4byz+rVcb6obhSTyNXCV8IXaXi9JcO4SyIFx6xpGe+3\nTVnhBx8cLvReeaX722/H+LrKiauW37BKVLOED2wJPAU8DywAftHDcdcCbwDzgeE9HFPRixaRGPWV\nDEtNXmV+eLz0kvs//3Po2jlmTOjquXZtha+tj+fcSB3cB1HTFj4wIPrdH5gLHNxlfzMwI1o+EJjb\nw3kqetEiEpNa9/0vIql+8on7DTe4H3ig+9Ch7pMmuS9ZUl44xT5nVVShPJRISQcYAPwJ2KvL9inA\naQXrrwKDu3l8xS9cRGKQ8lbtvHnu48a5DxzofsIJ7g8+6L5uXdmnq1yc33bKEGfC73MsHTPrZ2bP\nA8uAx9x9QZdDdgEWF6wvAYb0dV4RSciYMZsO09zU1O38uH2aM2fjYZ+bmsL6nDllh7fffnDddfD2\n23DMMWE8ny99CX7xC1i2rOzTlq+UuQ3SPmBesZ8MwPaEkk6uy/b7gNEF67OAf+jm8bF82olItqxf\n7/700+7nnefe1OR+6qnus2bVuNVf6wvABYixhV/SePhm9q/Ap+7+64JtU4B2d78tWn8VONTdl3V5\nrE+YMOHz9VwuRy6XK/kDSkSyK9FJ2fua26CtLbT6C7895fPh204J357a29tpb2//fH3SpEl4LSZA\nMbMdgLXunjezrYCHgEnu/kjBMc3Ad9292cxGAVe7+6huzuWlfLiIiPTEaz0pe4Kzl9Vsxisz2we4\niTBufj/gf9z9KjNrAXD31ui4/wMcDXwCnOPuz3VzLiV8EYnd8uVw880h+ZuFxD92LAwcGNMTFM5t\n0NS06XqVaYpDEZEu3OGJJ8L0jDNmwAknhOQ/alSFrf6YSjXlUsIXEelFI03KroQvIlKERpiUXQlf\nRKRE770HN94YWv2DBoVW/xlnwDbbJB1Z75TwRUTKtH49zJwZLvLOng2nnx5a/fvum3Rk3VPCFxGJ\nwTvvwPXXw+9+B0OGhMT/jW/AgAFJR7aBEr6ISIzWroUHHgi1/rlz4cwzQ/Lfa6+kI4s34fc5lo6I\nSKPr3x+OOy70wHzuOdhuOzj88HAj1y23hLt6G4Fa+CIi3VizBu67L7T6580LN3O1tMCee9Y2DrXw\nRUSqbPPN4aSTwgXeuXPD+iGHwGGHwR13wOrVSUdYOrXwRUSK9NlncM89oYfPggVh4LYLL4Tdd6/e\nc6qFLyKSgC22gNNOg0cfDV0616yBAw+Eo46Cu+8O62mmFr6ISAVWrYI77wyt/jffhPPOgwsugF13\njef8auGLiKTElluGbpxPPAEPPxzG7B8+HI49Fu6/H9atSzrCDdTCFxGJ2cqVcPvtodX/7rtw/vmh\n5b/LLqWfSy18EZEUGzAAzjkn9O6ZPh2WLoV99oETT4SHHgrDOyRBLXwRkRr46CO49dbQ6l+xItT5\nzz0XBg/u/XFq4YuI1Jlttw1dOJ99NvTjf/NN+PKXw9g9jzxSm1a/WvgiIgkpZlJ2DZ4mItJAepuU\nvV8/JXwRkYbUdVL2V15RwhcRaWidk7IfeqgSvohIJqiXjoiIlEwJX0QkI5TwRUQyQglfRCQjlPBF\nRDJCCV9EJCP6TPhmNtTMHjOzl83sJTO7uJtjcmb2gZnNi36urE64IiJSrmJa+GuAH7j73sAo4Dtm\n9nfdHDfb3YdHP/8Wa5Qp0N7ennQIFann+Os5dlD8Sav3+OPUZ8J39/fc/flo+WPgFeCvujk0lhsD\n0qre/2nqOf56jh0Uf9LqPf44lVTDN7NhwHDgqS67HDjIzOab2Qwz2yue8EREJC79iz3QzLYB7gS+\nF7X0Cz0HDHX3lWZ2DHAPsGd8YYqISKWKGkvHzDYH7gcecPerizi+Axjh7ssLtmkgHRGRMsQ1lk6f\nLXwzM+B6YEFPyd7MBgPvu7ub2UjCB8nywmPiClhERMpTTElnNHAm8IKZzYu2/QTYFcDdW4FTgIvM\nbC2wEji9CrGKiEgFajY8soiIJKvsO23N7AYzW2ZmLxZsG2lmT0c3X/3JzA6Itm9pZrea2QtmtsDM\nflzwmBFm9qKZvWFm11T2ciqOf18zezKKc7qZbVuw7/IoxlfN7Mh6it/MjjCzZ6Ltz5jZ1+op/oL9\nu5rZx2Z2Sb3Fb2Z/H+17Kdr/haTiL/F/J43v3W5vBjWzQWb2sJm9bmYzzayp4DGpef+WGn+s7193\nL+sHOITQRfPFgm3twFHR8jHAY9Hy2cCt0fJWQAewa7T+NDAyWp4BHF1uTDHE/yfgkGj5HOCn0fJe\nwPPA5sAwYCEbvh3VQ/z7ATtFy3sDSwoek/r4C/bfCdwOXFJP8RNKp/OBfaL1gUC/pOIvMfY0vnd3\nAvaLlrcBXgP+DvgV8C/R9h8Bv4yWU/X+LSP+2N6/Zbfw3f0JYEWXzUuB7aPlJuCdgu1bm9lmwNbA\nauBDM9sZ2Nbdn46Ouxk4odyYStFD/HtE2wFmASdHy18n/NOvcfdFhH+YA+slfnd/3t3fi7YvALYy\ns83rJX4AMzsBeIsQf+e2eon/SOAFd38xeuwKd1+fVPwlxp7G9253N4PuAhwP3BQddlNBPKl6/5Ya\nf5zv37gHT/sx8O9m9v+AqwgXd3H3h4APCf88i4Cr3D1PeJFLCh7/TrQtKS+b2dej5VOBodHyX7Fx\nnEsIcXbdntb4C50MPOvua6iTv7+Fe0D+BZjY5fi6iJ9wT4qb2YNm9qyZXRZtT1P83cae9veubXwz\n6GB3XxbtWgYMjpZT+/4tMv5CFb1/40741wMXu/uuwA+idczsTMLXwZ2B3YBLzWy3mJ87DucC483s\nGcJXrdUJx1OqXuM3s72BXwItCcRWjJ7inwj8xt1Xku4hPHqKvz9wMPDN6PeJZnYY4Q71tOg29jS/\nd6OGwF2Em0E/KtznocaRpr/vJkqNP473b9F32hZppLsfHi3fCUyNlg8C/uDu64A/m9kcYATwR2BI\nweOHsKEMVHPu/hpwFICZ7QmMiXa9w8at5SGET9Z3qI/4MbMhwN3At929I9qc9vibo10jgZPN7FeE\nUuF6M/uU8HrSHH/n338x8LhH96aY2QzgH4BppCT+Xv72qXzvWrgZ9C7gf9z9nmjzMjPbyd3fi8od\n70fbU/f+LTH+2N6/cbfwF5rZodHyYcDr0fKr0TpmtjVh1M1Xo7rUh2Z2oJkZ8G3CsAyJMLMdo9/9\ngCuB66Jd04HTzewLUetmD+Dpeok/utrfBvzI3Z/sPN7dl5Lu+KdEcX7V3Xdz992Aq4HJ7v5/6+Xv\nDzwE7GNmW5lZf+BQ4OU0xd/T354Uvnej5+vuZtDpwFnR8lkF8aTq/Vtq/LG+fyu40nwr8C7hq99i\nwpX9/Qm1qOeBJ4Hh0bFbEFozLwIvs3EvixHR9oXAteXGE0P85wIXE66Yvwb8vMvxP4lifJWoJ1K9\nxE94A38MzCv42aFe4u/yuAnAD+vp7x8d/y3gpSjWXyYZf4n/O2l87x4MrI/yTOf/89HAIMIF59eB\nmUBTwWNS8/4tNf4437+68UpEJCM0xaGISEYo4YuIZIQSvohIRijhi4hkhBK+iEhGKOGLiGSEEr6I\nSEYo4YuIZMT/B0w3BHPgeZdvAAAAAElFTkSuQmCC\n", "text": [ "" ] } ], "prompt_number": 20 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introducing a Basis: Quadratic Fit\n", "\n", "Now we will fit a quadratic model using basis functions. Given everything we've learnt above, this is now quite easy to do. Firstly, we need to create a design matrix that contains the quadratic basis, \n", "\n", "$$\\mathbf{\\Phi} = \\left[ \\mathbf{1} \\quad \\mathbf{x} \\quad \\mathbf{x}^2\\right]$$\n", "\n", "where this notation means that each column of $\\mathbf{\\Phi}$ is derived from the entire set of input years." ] }, { "cell_type": "code", "collapsed": false, "input": [ "Phi = np.hstack([np.ones(x.shape), x, x**2])" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 21 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can solve this system for $\\mathbf{w}$ just as we did for the linear case, so we have," ] }, { "cell_type": "code", "collapsed": false, "input": [ "w = np.linalg.solve(np.dot(Phi.T, Phi), np.dot(Phi.T, y))\n", "print(w)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "[[ 6.43641954e+02]\n", " [ -6.42502988e-01]\n", " [ 1.61109704e-04]]\n" ] } ], "prompt_number": 22 }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can plot the solution in two different ways, either we take" ] }, { "cell_type": "code", "collapsed": false, "input": [ "f_test = w[2]*x_test**2 + w[1]*x_test + w[0]\n", "plt.plot(x_test, f_test, 'b-')\n", "plt.plot(x, y, 'rx')" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 23, "text": [ "[]" ] }, { "metadata": {}, "output_type": "display_data", "png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEACAYAAACwB81wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFPW1//H3YYvg1hoVE8FAFKNEDIiiAsr83AGFIHrF\nuKC5MShGE40TTfBG9HHUSPy55LqQqNG4xxBUFhUXxgUVRUFxQ02GuLJEaNyjwLl/fGu0GWbpnqnp\n6p76vJ6nn66uqu4609Cnq099F3N3RESk7WuXdAAiIlIcSvgiIimhhC8ikhJK+CIiKaGELyKSEkr4\nIiIp0SGfncxsMfAhsAb40t0H1NleAdwD/DNaNcXdL4gvTBERaam8Ej7gQIW7r2hkn0fdfUQMMYmI\nSCsopKRjLdwuIiIJyjfhO/CQmc0zsxMb2D7QzF4ws5lm1ju+EEVEJA75lnQGufv7ZrYl8KCZvebu\nj+dsfx7o7u6fmtlQ4G5gh7iDFRGR5rNCx9Ixs3OBj9390kb2qQH659b8zUyD9oiINIO7x1Iyb7Kk\nY2ZdzGzjaHlD4EBgYZ19upqZRcsDCF8k613gdfeyvZ177rmJx5DW+Ms5dsWf/K3c449TPiWdrsDU\nKJ93AG5191lmNi5K4pOBw4GTzWw18CkwJtYoRUSkxZpM+O5eA/StZ/3knOWrgKviDU1EROKknrZ5\nqqioSDqEFinn+Ms5dlD8SSv3+ONU8EXbZh/IzIt1LBGRtsLM8GJdtBURkbZBCV9EJCWU8EVEUkIJ\nX0QkJZTwRURSQglfRCQllPBFRFJCCV9EJCWU8EVEUkIJX0QkJZTwRURSQgm/pWbMgGx23XXZbFgv\nIlJClPBbatAgmDDh66SfzYbHgwYlG5eISB0aLTMOtUm+shImTYKqKshkko5KRNqAOEfLVMKPy+LF\n0LMn1NRAjx5JRyMibYSGRy412Ww4s6+pCfd1a/oiIiVACb+lass5VVXhzL6qat2avohIiVBJp6Vm\nzAgXaHNr9tkszJkDw4cnF5eItAmq4YuIpIRq+CIiUjAlfBGRlFDCFxFJCSV8EZGUUMIXEUkJJXwR\nkZRQwhcRSYm8Er6ZLTazF81svpk908A+V5rZG2b2gpn1izdMERFpqQ557udAhbuvqG+jmQ0Dtnf3\nXma2B3ANsGdMMYqISAwKKek01tNrBHATgLvPBTJm1rUlgYmISLzyTfgOPGRm88zsxHq2bwO8nfP4\nHaBbS4MTEZH45FvSGeTu75vZlsCDZvaauz9eZ5+6vwA0cI6ISAnJK+G7+/vR/XIzmwoMAHIT/rtA\n95zH3aJ165g4ceJXyxUVFVRUVBQcsIhIW1ZdXU11dXWrvHaTo2WaWRegvbt/ZGYbArOA89x9Vs4+\nw4CfufswM9sTuNzd96zzOhotU0SkQHGOlpnPGX5XYKqZ1e5/q7vPMrNxAO4+2d1nmtkwM3sT+AQ4\nIY7gREQkPhoPX0SkhGk8fBERKZgSvohISijhi4ikhBK+iEhKKOGLiKSEEr6ISEoo4YuIpIQSvohI\nSijhi4ikRJtI+FdfDVOnJh2FiEhpy3d45JK2554wdCh07AiHHJJ0NCIipalNnOHvuitMmwY//jE8\n8EDS0YiIlKY2kfABBgyAu++GY4+Fhx9OOhoRkdLTZhI+wMCB8Le/wZgx8OijSUcjIlJa2lTCB9hn\nH7jzTjjiCHjiiaSjEREpHW0u4QPsuy/ccgscdhg8/XTS0YiIlIY2mfABDjwQbrwRRo6EefOSjkZE\nJHltNuEDDBsGf/oTDB8O8+cnHY2ISLLaRDv8xowYAV9+GdrpP/gg9OmTdEQiIslo8wkfYPRoWL0a\nDjoIZs2CnXdOOiIRkeJLRcIHOPJIcIcDDoD774cf/CDpiEREiis1CR9C+/z27cOZ/syZoYeuiEha\npCrhQ2if36FDqOlPnw677550RCIixZG6hA8walRI+sOHw733hsHXRETaujbdLLMxhx4a2umPGAFz\n5iQdjYhI60ttwofQTv+WW8IZ/2OPJR2NiEjrSnXCh9Aj9/bb4fDD4ZFHko5GRKT1pD7hA+y3H9x1\nV2i6+eCDSUcjItI6lPAjQ4aEaRKPPhruuy/paERE4pdXwjez9mY238ym1bOtwsxWRdvnm9k58YdZ\nHIMHwz33wNixMGVK0tGIiMQr32aZPwdeATZuYPuj7j4inpCStddeYZrEYcPgo4/g+OOTjkhEJB5N\nnuGbWTdgGHAdYA3tFmdQSevXD2bPht/+Fv7wh6SjERGJRz4lncuASmBtA9sdGGhmL5jZTDPrHVt0\nrWXGDMhm112XzYb1kR13hMcfhyuvhAsuCOPwiIiUs0ZLOmZ2CLDM3eebWUUDuz0PdHf3T81sKHA3\nsEN9O06cOPGr5YqKCioqGnrJVjZoEEyYAFVVkMmEZF/7OMd3XprBE9MHsf/hGVatgksuAVuVDT21\nhg9PJnYRadOqq6uprq5uldc2b+TU1cwuBI4FVgMbAJsAU9z9uEaeUwP0d/cVddZ7Y8cqutokX1kJ\nkyZ9nfzr2WflmVUcPCbDXjtl+f9dJtDuwnr2FRFpBWaGu8dSNm804dc56BDgTHc/tM76roRfAW5m\nA4C/unuPep5fWgkfYPFi6NkTamqgR4/694mS/scnV/LQQZOYtmcV196RoWPHYgYqImkVZ8IvtB2+\nRwGMM7Nx0brDgYVmtgC4HBgTR2CtLpsNZ/Y1NeG+bk2/ViYDlZVs1KcnBz9cybIvMowaBZ99Vtxw\nRURaKu8z/BYfqJTO8HNr9nVr+A2UdWpLP19OrOK40zIsWRJG2ty4oYaqIiIxSKSk0+IDlVLCnzEj\nXLjNTe7Zei7GNvDFsOb8Kk6ZkGHevPBSXbsW/08QkXRQwi+WRr4YfNhwzjsvjLb5wAOw3XbJhSki\nbZcSfgm59lo4/3yYNg369086GhFpa5TwS8zUqTBuHNx6a5gkXUQkLkm20pF6jBoVBls75hi47bak\noxERqV8q57RtDXvvDQ8/HAZdW7IEzjgj6YhERNalkk7M3noLDj44NPb53e+gnX5DiUgLqIZf4las\ngEMOCS13brgB9coVkWZTDb/Ebb45PPQQrFoFhx4axtUXEUmaEn4r6dIF/v73METP4MHwzjtJRyQi\naaeE34o6dIBrroFjjw0zac2fn3REIpJmquEXyZQpcNJJ8Oc/h/q+iEg+VMMvQ6NHw/Tp8NOfatpE\nEUmGzvCLrKYmNNk88EC49FJo3z7piESklKlZZplbuRIOPxw22ij0zN1ww6QjEpFSpZJOmdtsM7jv\nvtB8c5994L33ko5IRNJACT8hnTqFTlmjR4cWPC++mHREItLWqaRTAu64A049Fa67DkaOTDoaESkl\nKumUoxkz1p83N5uFGTMYMyZsPuUUuOgi0PeiiLQGJfxiGTQoTJdYm/Rrp08cNAiAAQNg7tzQO/eY\nYzRJuojETwm/WDKZMDfuhAmweHG9k6Zvsw089hisXQtDhuhirojESzX8Ylu8GHr2DA3ye/Sodxf3\nUNq55ppwxr/77kWNUERKiGr45SqbhUmTQrKfNGn9mn7EDH7zm9Ajd9gwzaIlIvHQGX6x1Nbsa8s4\ndR83YOHC0HLnsMPg4ovDgGwikh7qaVuOZswIF2hzk3s2C3PmhLEWGvHBB3DUUbBmDdx5J2yxRSvH\nKiIlQwk/hdasCT8I7rgj1PV33TXpiESkGFTDT6H27UNJZ9IkOOgg+Mtfko5IRMqNzvDL0EsvwahR\nMHRoGHFTc+aKtF1FP8M3s/ZmNt/MpjWw/Uoze8PMXjCzfnEEJg3beWd49ln4xz9gv/3g/feTjkhE\nykG+JZ2fA68A652im9kwYHt37wX8FLgmvvCkIZkMTJsG++4L/ftDdXXSEYlIqWsy4ZtZN2AYcB1Q\n38+KEcBNAO4+F8iYWdc4g5T6tWsHEyfCjTfCmDGhxr92bdJRiUipyucM/zKgEmgolWwDvJ3z+B2g\nWwvjkgIceGAo8dx7b2izv2JF0hGJSClqtBuPmR0CLHP3+WZW0diudR7Xe3V24sSJXy1XVFRQUdHY\nS0ohuncPZZ2zzgolnrvugt12SzoqESlUdXU11a1Uo220lY6ZXQgcC6wGNgA2Aaa4+3E5+1wLVLv7\nHdHj14Ah7r60zmuplU6RTJkCJ50E558f7i2W6/sikoREOl6Z2RDgTHc/tM76YcDP3H2Yme0JXO7u\ne9bzfCX8InrjjTBv7k47weTJsOmmSUckIs2RZMcrjwIYZ2bjANx9JvBPM3sTmAyMjyMwaZleveDp\np8O8ubvuCs88k3REIpI0dbwqNS0Yc6chU6bAySdDZSX88pehdY+IlAcNrdCWNTEzVnOMHh1a8Uyd\nGr4zli2LKVYRKStK+KUmj5mxmuM734FHH4V+/cLt4YfjCVdEyodKOqUqj5mxmuuhh2DsWDjuODjv\nPOjUKdaXF5EYqaTT1uU5M1Zz7b8/zJ8fJlcZOBBeey3WlxeREqWEX2pyZ8Lq0ePr8k7MSX+rrcJY\nPD/5Cey9d5g/Vz/ARNo2lXRKTSu00mnqeG9sNYijTs6w9dZw/fXQ9RuteDwRKYhKOm3Z8OHrX6DN\nZFov+Q4aRK8bJ/DkzCx9+8I+u2SpObplrYJEpDTpDF++LiNVVvLe6ZM4+Lkq9hqa4fe/h403Tjo4\nkXTTGb7EK5MJvbJ69uTbl1Xy+MIMq1fDLrvA7NlJBycicVHCl/VaBW3qWa6/Hv73f+HYY+G00+CT\nT5IOUkRaSgk/7RppFTR8eGi6uXIl9O0LTzyRdLAi0hKq4addnq2C7r4bxo+Ho46CCy6Azp0TiFUk\nhRIZHrnFB1LCL3v//jf87GewYAFcdx0MHpx0RCJtnxK+JGrKFDj1VBg1Ci66CDbZJOmIRNoutdKR\nRI0eDS+/DP/5D+y8c6gKiUjp0xm+tMgjj8CJJ8KAAXDFFWHIBhGJj87wpWTsu29oydOtG/TpAzff\nrDF5REqVEr60WJcuoRn/zJlw6aVwwAGwaFHSUdUxY8b6A9Bls6pHSaoo4Uts+veHefNCa85Bg+C3\nv4XPPks6qkgrzCQmUm6U8CVWHTrA6aeHppuvvhou6t53X9JR0WoziYmUE120lfw0c9jm+++HU04J\n0ypefnmo9SeqFWcSE2kNumgrxdfMksjBB8NLL0Hv3mF4ht//Hr74ogjx1qeVZxITKXVK+JKffEoi\nDVwY7fzIDM4/H558Moy+2adPuMBbVEWaSUyklKmkI4VprCSSm1QzmfUfR2bMCHX+HXaAyy6DXr2K\nEHexZxITiYlKOpKfuJsiNlUSyfPC6PDhocwzZAjstRecdRZ89FHzQspbsWcSEylBSvhtWZxNEfMt\nieRMpkJlZYOtYDp1CptfegmWLYMdd4Qbb4Q1awoPTUTyo4TflsXZFHHOnHWfW/vac+asu1+BF0a3\n3hr+/GeYOhX+9KfQlv+hhwoPT0Saphp+GhSrKWKeNfyG6un+xBz+/p/hnH02bL89XHJJuMArkmZF\nreGb2QZmNtfMFpjZK2Z2UT37VJjZKjObH93OiSM4iUExmyLm+yuggVKTDR701Uicw4bB/vvDT34C\n773XeiGLpIq7N3kDukT3HYCngcF1tlcA9zbxGi5FtnKl+/jx4b6+x0mqjaWmpsGYVq50P+ss9803\nd/+f/3Fftar4YYokLcqdeeXqpm551fDd/dNosRPQHlhRz26x/OSQGOV7xp2EPC7uZjJw8cXw/PPw\nr3+F5puTJsGnn9bzeiLSpLwSvpm1M7MFwFJgtru/UmcXBwaa2QtmNtPMescdqDRDKTdFLKDU9J3v\nwE03hU5bc+eGxH/11Qn22BUpUwVdtDWzTYEHgLPdvTpn/cbAGnf/1MyGAle4+w51nuvnnnvuV48r\nKiqoqKhoWfRSnvK9uNuA556Dc84JQzBPnAhHHw3t27d+2CLFUF1dTXV19VePzzvvvOTmtDWz/wE+\nc/ffN7JPDdDf3VfkrPNCjyVtVEy9Xh97LHxPfPBBGIr5iCOU+KXtKXYrnS3MLBMtdwYOAObX2aer\nmVm0PIDwRVJfnV8ktlLTPvuEpH/ZZXDllfD978Mtt8Dq1THGWh9NpiJlKp8a/reAR6Ia/lxgmrs/\nbGbjzGxctM/hwMJon8uBMa0Trsi6zOCgg8KPg6uugj/+EXbaKfTa/fLLVjqoJlORMqWOV9LmVFfD\n+eeH/ma/+Q0cd1wYyiFWtUm+sjJcdNZkKtJK4izpKOFLm/XEEyHxv/oq/OIXcOKJsMkmMR5Ak6lI\nEWi0TJE8DB4Ms2bBPfeEuXa/+1349a/h/fcbeEIhtXlNpiJlSAlf2rxdd4Xbb4dnngnDMPfuHc72\nFy2qs2O+tXlNpiJlSiUdSZ3ly8MF3quvhoED4ec/h4qKcAE4r9q8JlORIlINXyQGn3wCN98cmnR2\n6ACnnQY/+hF0WbZYtXkpGarhi8Rgww3hpJPC6JyXXhpq/X26Z3lq9CTem1NCtXm1+5eYKOFL6pnB\nAQfAtJuzPDd0AvcOqKLPoT0Y+04Vbx83gbUrEk76avcvMVFJR6RWTm3+o49Cuee2q7PstGIOvX4x\nnOOPh622Sig2tftPLdXwRYrEPbTumTw5TMN44IEwbly4yNuu2L+P1e4/lVTDFykSM9hjD7jhhpBn\n99kndOL63vfCSfZbbxUpELX7lxjoDF+kQO5hXP6bboK//hX69oWxY+Gww2CjjVrhgC0cTlrKm0o6\nIiXi889h2rSQ/J94AkaODMk/1pKP2v2nmhK+SAlauhRuuy0k/+XLw/j8Y8aEkpBpAlBpJiV8kRL3\n6quh3HPHHfDZZ/Bf/wVHHhmGeVDyl0Io4YuUuqgM45tmWLgQ7rwTZt6WZbf/zGGLscMZORIGDEig\npY+UHbXSESl1UWcpW5Vll12gqjLL80MncMptg2jXDv77v2GbbUITz5kzw7UAkdamM3yR1tJEZ6k3\n3wzDOdx9N7z4YujtO2JEmMGra9cE45aSopKOSLnIs7PU8uUwfTrcey/Mng3bbRcS/8EHw157QceO\nRYtYSoxKOiLloIDOUltuCSecEHrzLl8Ol18eLu6ecUbYNmoUXHstvP566AfQJmmQuPUsXx7v6ynh\ni7SGFkyS0rEj7L13eMq8eSHJjx4NTz4J++0Xav9HHRWGe1i0KOcLoNwTZj6DxJX739iEJUtgypQw\nR8Muu0CvXjEfwN2LcguHEkmJ6dPdV65cd93KlWF9c/aLrF3r/o9/uF9/vftxx7l37+6+9dbuRx7p\nfu3FK33p4eP9P0tXfv0648ev//qlrDbmmpr6Y6/7N5Xj3xhZu9Z90aLwb3n88e7bb+++2Wbuhxzi\nfvHF7nPnun/5pXuUO2PJw6rhiySphcMmuIfLBNXV8NRT8PKcLGNfn8CsH1Ry+upJ/Pv0Kvrvl6Fb\nt1b/S+LT1HWPMh05dPny8Ivt2WfD/dy5sMEGYe7lvfcO9717r99UVxdtRdqSmBPYRwsXs/EuPbnq\nzBoeWNSDp54KM3r17Qv9+oX7vn1h++0b6QeQ1HAO+b4XcY0c2gp/p3sozSxcCPPnf53gs1no3x92\n3x122y30w9h226ZfL86Er5KOSCmoqXGHcN8S9ZRE1q51/9e/3O+5x33iRPcf/tC9Rw/3jTZyHzjQ\n/aST3K+4wn3WLPe33gqlhlhLJ/mWrfI9ZlNln9Y4ZgNWrXJ/8kn3yZPdTz3VvaLC/ZvfDLeKCvdf\n/ML91lvdX3/dfc2avF5yPcRY0lHCF0laPgmskNdpKnlFyXDFCvfZs92vvNL9jB+v9LP7TPettw5f\nBP37u594xEp/dsB4n3pZjb932Hh//9Xw5dHaca333NwkXeiXQj6JvIn3/+OP3RcscL/rLvcLL3Q/\n4QT3wYPdu3Z179IlvFfHH+9+6aXuDzzg/t573rz3qQFK+CJtRQmeSa9c6f7UU+433OB+8Uk17uAj\ndqnxLbZw79zZfccd3YcODU+549jpfvMfVvrMme7z57svWeK+5oNGjtnSL7VCLnI3ccy1a92z2XD2\nPffO8Hde/asaP+UU90MPde/bN5ypb7CBe+/e7iNHup95Zjibf+QR97ffbv5ZeyHiTPiq4YskqZRr\n5fXs83GHDIsXh9J5TQ2s+GeWwfdP4KpvVfHmvzN88m6WylUT+EPXKjptlWHzzWGzzcJtu/aL+fUf\ne3LrBTW0+24POneGLl3CrXa5c+dwIbNdu/VvZuG2ejV88UX9t48/ho8+gg8/DDf712JO+l1PJo6t\n4c3VPVi+HJYtCxdQly+Hb3wDtvtmlnM+n0D1bpUc9c4kXjyyiq7fy7DtttC9e+gHkeSAd7poKyIt\n19iFz0JaD9X5Yvji3CqWfJ7hgw9gxQpYuRI+fifLLndO4MG+lQx4bBI3f6+KD9Zk+PTTMJpo7v3n\nn8PateHm/vVy7eOOHaFTp/VvHTuGCWg22QQ23hi6fiPLEQsm8Ny+lQx5ZhIv/6iKzXpm2GqrkMS3\n3BI2+DzGyWVa6cu7aBdtgQ2AucAC4BXgogb2uxJ4A3gB6NfAPq3ze0dECtdUiaXA/gGNXnROou18\nnNcN4j5mgShmDR/oEt13AJ4GBtfZPgyYGS3vATzdwOu06I8WkZjEnZji/vKIQxLHrD1GHNcqcsSZ\n8PMu6ZhZF+BRYKy7v5Kz/lpgtrvfGT1+DRji7kvrPN/zPZaItKI4Sw9pmG+30Pcrrj4CkaIOnmZm\n7cxsAbCUkNhfqbPLNsDbOY/fAcqpX59Iugwfvn4yzmSaV2eeM2fd5J7JhMdz5rQ8zlKRzxg/tQoY\nMC8JTSZ8d1/r7n0JSXwfM6uoZ7e63z46lRdJgzi/PEpV7ZfYhAnh7L2pi9fNGDCvWDrku6O7rzKz\nGcBuQHXOpneB7jmPu0Xr1jNx4sSvlisqKqioqMg/UhGRpGQyoRVSbammvnJVY792CvgCrK6uprq6\nOp6462i0hm9mWwCr3T1rZp2BB4Dz3P3hnH2GAT9z92FmtidwubvvWc9rqYYvIuUpwQHbilnD/xbw\nSFTDnwtMc/eHzWycmY0DcPeZwD/N7E1gMjA+jsBEREpCGZRq8qWOVyIijUmqN3REPW1FRFJCc9qK\niEjBlPBFRFJCCV9EJCWU8EVEUkIJX0QkJZTwRURSQglfRCQllPBFRFJCCV9EJCWU8EVEUkIJX0Qk\nJZTwRURSQglfRCQllPBFRFJCCV9EJCWU8EVEUkIJX0QkJZTwRURSQglfRCQllPBFRFJCCV9EJCWU\n8EVEUkIJX0QkJZTwRURSQglfRCQllPBFRFJCCV9EJCWaTPhm1t3MZpvZy2b2kpmdVs8+FWa2yszm\nR7dzWidcERFprnzO8L8ETnf37wN7AqeY2U717Peou/eLbhfEGmUJqK6uTjqEFinn+Ms5dlD8SSv3\n+OPUZMJ39yXuviBa/hh4Ffh2PbtazLGVlHL/T1PO8Zdz7KD4k1bu8cepoBq+mfUA+gFz62xyYKCZ\nvWBmM82sdzzhiYhIXDrku6OZbQT8Dfh5dKaf63mgu7t/amZDgbuBHeILU0REWsrcvemdzDoC04H7\n3P3yPPavAfq7+4qcdU0fSERE1uPusZTMmzzDNzMDrgdeaSjZm1lXYJm7u5kNIHyRrMjdJ66ARUSk\nefIp6QwCjgFeNLP50brfANsCuPtk4HDgZDNbDXwKjGmFWEVEpAXyKumIiEj5a3ZPWzO7wcyWmtnC\nnHUDzOyZqPPVs2a2e7R+AzO73cxeNLNXzOzsnOf0N7OFZvaGmV3Rsj+nxfH/wMyeiuK818w2ztn2\n6yjG18zswHKK38wOMLN50fp5Zvb/yin+nO3bmtnHZvbLcovfzHaJtr0Ube+UVPwF/t8pxc9uvZ1B\nzWxzM3vQzF43s1lmlsl5Tsl8fguNP9bPr7s36wbsTWiiuTBnXTVwULQ8FJgdLR8P3B4tdwZqgG2j\nx88AA6LlmcDBzY0phvifBfaOlk8Azo+WewMLgI5AD+BNvv51VA7x9wW2jpa/D7yT85ySjz9n+9+A\nO4FfllP8hNLpC0Cf6PFmQLuk4i8w9lL87G4N9I2WNwIWATsBlwC/itafBVwcLZfU57cZ8cf2+W32\nGb67Pw6srLP6fWDTaDkDvJuzfkMzaw9sCHwBfGhm3wI2dvdnov3+AvywuTEVooH4e0XrAR4CRkfL\nIwn/6b9098WE/zB7lEv87r7A3ZdE618BOptZx3KJH8DMfgj8kxB/7bpyif9A4EV3Xxg9d6W7r00q\n/gJjL8XPbn2dQbcBRgA3RbvdlBNPSX1+C40/zs9v3IOnnQ1camZvAZMIF3dx9weADwn/eRYDk9w9\nS/gj38l5/rvRuqS8bGYjo+UjgO7R8rdZN853CHHWXV+q8ecaDTzn7l9SJu+/hT4gvwIm1tm/LOIn\n9ElxM7vfzJ4zs8pofSnFX2/spf7ZtXU7g3Z196XRpqVA12i5ZD+/ecafq0Wf37gT/vXAae6+LXB6\n9BgzO4bwc/BbQE/gTDPrGfOx4/BjYLyZzSP81Poi4XgK1Wj8ZvZ94GJgXAKx5aOh+CcCl7n7p5T2\nEB4Nxd8BGAz8KLofZWb7Enqol4p6Yy/lz250IjCF0Bn0o9xtHmocpfT+rqfQ+OP4/Obd0zZPA9x9\n/2j5b8B10fJAYKq7rwGWm9kcoD/wBNAt5/nd+LoMVHTuvgg4CMDMdgCGR5veZd2z5W6Eb9Z3KY/4\nMbNuwN+BY929Jlpd6vEPizYNAEab2SWEUuFaM/uM8PeUcvy17//bwGMe9U0xs5nArsAtlEj8jbz3\nJfnZtdAZdApws7vfHa1eamZbu/uSqNyxLFpfcp/fAuOP7fMb9xn+m2Y2JFreF3g9Wn4teoyZbUgY\ndfO1qC71oZntYWYGHEsYliERZrZldN8OOAe4Jtp0LzDGzDpFZze9gGfKJf7oav8M4Cx3f6p2f3d/\nn9KO/9oozn3cvae79wQuB6rc/epyef+BB4A+ZtbZzDoAQ4CXSyn+ht57SvCzGx2vvs6g9wJjo+Wx\nOfGU1Oe30Phj/fy24Erz7cB7hJ9+bxOu7O9GqEUtAJ4C+kX7foNwNrMQeJl1W1n0j9a/CVzZ3Hhi\niP/HwGk7gsB+AAAAkUlEQVSEK+aLgAvr7P+bKMbXiFoilUv8hA/wx8D8nNsW5RJ/needC5xRTu9/\ntP/RwEtRrBcnGX+B/3dK8bM7GFgb5Zna/88HA5sTLji/DswCMjnPKZnPb6Hxx/n5VccrEZGU0BSH\nIiIpoYQvIpISSvgiIimhhC8ikhJK+CIiKaGELyKSEkr4IiIpoYQvIpIS/wcBL7feDAyMTgAAAABJ\nRU5ErkJggg==\n", "text": [ "" ] } ], "prompt_number": 23 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or we can do the matrix form of this equation which first involves creating a design matrix for the test points,\n" ] }, { "cell_type": "code", "collapsed": false, "input": [ "Phi_test = np.hstack((np.ones_like(x_test), x_test, x_test**2))" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 24 }, { "cell_type": "markdown", "metadata": {}, "source": [ "and then computing the value of the function using a matrix multiply" ] }, { "cell_type": "code", "collapsed": false, "input": [ "f_test = np.dot(Phi_test,w)\n", "plt.plot(x_test, f_test, 'b-')\n", "plt.plot(x, y, 'rx')\n", "w" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 25, "text": [ "array([[ 6.43641954e+02],\n", " [ -6.42502988e-01],\n", " [ 1.61109704e-04]])" ] }, { "metadata": {}, "output_type": "display_data", "png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEACAYAAACwB81wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFPW1//H3YYvg1hoVE8FAFKNEDIiiAsr83AGFIHrF\nuKC5MShGE40TTfBG9HHUSPy55LqQqNG4xxBUFhUXxgUVRUFxQ02GuLJEaNyjwLl/fGu0GWbpnqnp\n6p76vJ6nn66uqu4609Cnq099F3N3RESk7WuXdAAiIlIcSvgiIimhhC8ikhJK+CIiKaGELyKSEkr4\nIiIp0SGfncxsMfAhsAb40t0H1NleAdwD/DNaNcXdL4gvTBERaam8Ej7gQIW7r2hkn0fdfUQMMYmI\nSCsopKRjLdwuIiIJyjfhO/CQmc0zsxMb2D7QzF4ws5lm1ju+EEVEJA75lnQGufv7ZrYl8KCZvebu\nj+dsfx7o7u6fmtlQ4G5gh7iDFRGR5rNCx9Ixs3OBj9390kb2qQH659b8zUyD9oiINIO7x1Iyb7Kk\nY2ZdzGzjaHlD4EBgYZ19upqZRcsDCF8k613gdfeyvZ177rmJx5DW+Ms5dsWf/K3c449TPiWdrsDU\nKJ93AG5191lmNi5K4pOBw4GTzWw18CkwJtYoRUSkxZpM+O5eA/StZ/3knOWrgKviDU1EROKknrZ5\nqqioSDqEFinn+Ms5dlD8SSv3+ONU8EXbZh/IzIt1LBGRtsLM8GJdtBURkbZBCV9EJCWU8EVEUkIJ\nX0QkJZTwRURSQglfRCQllPBFRFJCCV9EJCWU8EVEUkIJX0QkJZTwRURSQgm/pWbMgGx23XXZbFgv\nIlJClPBbatAgmDDh66SfzYbHgwYlG5eISB0aLTMOtUm+shImTYKqKshkko5KRNqAOEfLVMKPy+LF\n0LMn1NRAjx5JRyMibYSGRy412Ww4s6+pCfd1a/oiIiVACb+lass5VVXhzL6qat2avohIiVBJp6Vm\nzAgXaHNr9tkszJkDw4cnF5eItAmq4YuIpIRq+CIiUjAlfBGRlFDCFxFJCSV8EZGUUMIXEUkJJXwR\nkZRQwhcRSYm8Er6ZLTazF81svpk908A+V5rZG2b2gpn1izdMERFpqQ557udAhbuvqG+jmQ0Dtnf3\nXma2B3ANsGdMMYqISAwKKek01tNrBHATgLvPBTJm1rUlgYmISLzyTfgOPGRm88zsxHq2bwO8nfP4\nHaBbS4MTEZH45FvSGeTu75vZlsCDZvaauz9eZ5+6vwA0cI6ISAnJK+G7+/vR/XIzmwoMAHIT/rtA\n95zH3aJ165g4ceJXyxUVFVRUVBQcsIhIW1ZdXU11dXWrvHaTo2WaWRegvbt/ZGYbArOA89x9Vs4+\nw4CfufswM9sTuNzd96zzOhotU0SkQHGOlpnPGX5XYKqZ1e5/q7vPMrNxAO4+2d1nmtkwM3sT+AQ4\nIY7gREQkPhoPX0SkhGk8fBERKZgSvohISijhi4ikhBK+iEhKKOGLiKSEEr6ISEoo4YuIpIQSvohI\nSijhi4ikRJtI+FdfDVOnJh2FiEhpy3d45JK2554wdCh07AiHHJJ0NCIipalNnOHvuitMmwY//jE8\n8EDS0YiIlKY2kfABBgyAu++GY4+Fhx9OOhoRkdLTZhI+wMCB8Le/wZgx8OijSUcjIlJa2lTCB9hn\nH7jzTjjiCHjiiaSjEREpHW0u4QPsuy/ccgscdhg8/XTS0YiIlIY2mfABDjwQbrwRRo6EefOSjkZE\nJHltNuEDDBsGf/oTDB8O8+cnHY2ISLLaRDv8xowYAV9+GdrpP/gg9OmTdEQiIslo8wkfYPRoWL0a\nDjoIZs2CnXdOOiIRkeJLRcIHOPJIcIcDDoD774cf/CDpiEREiis1CR9C+/z27cOZ/syZoYeuiEha\npCrhQ2if36FDqOlPnw677550RCIixZG6hA8walRI+sOHw733hsHXRETaujbdLLMxhx4a2umPGAFz\n5iQdjYhI60ttwofQTv+WW8IZ/2OPJR2NiEjrSnXCh9Aj9/bb4fDD4ZFHko5GRKT1pD7hA+y3H9x1\nV2i6+eCDSUcjItI6lPAjQ4aEaRKPPhruuy/paERE4pdXwjez9mY238ym1bOtwsxWRdvnm9k58YdZ\nHIMHwz33wNixMGVK0tGIiMQr32aZPwdeATZuYPuj7j4inpCStddeYZrEYcPgo4/g+OOTjkhEJB5N\nnuGbWTdgGHAdYA3tFmdQSevXD2bPht/+Fv7wh6SjERGJRz4lncuASmBtA9sdGGhmL5jZTDPrHVt0\nrWXGDMhm112XzYb1kR13hMcfhyuvhAsuCOPwiIiUs0ZLOmZ2CLDM3eebWUUDuz0PdHf3T81sKHA3\nsEN9O06cOPGr5YqKCioqGnrJVjZoEEyYAFVVkMmEZF/7OMd3XprBE9MHsf/hGVatgksuAVuVDT21\nhg9PJnYRadOqq6uprq5uldc2b+TU1cwuBI4FVgMbAJsAU9z9uEaeUwP0d/cVddZ7Y8cqutokX1kJ\nkyZ9nfzr2WflmVUcPCbDXjtl+f9dJtDuwnr2FRFpBWaGu8dSNm804dc56BDgTHc/tM76roRfAW5m\nA4C/unuPep5fWgkfYPFi6NkTamqgR4/694mS/scnV/LQQZOYtmcV196RoWPHYgYqImkVZ8IvtB2+\nRwGMM7Nx0brDgYVmtgC4HBgTR2CtLpsNZ/Y1NeG+bk2/ViYDlZVs1KcnBz9cybIvMowaBZ99Vtxw\nRURaKu8z/BYfqJTO8HNr9nVr+A2UdWpLP19OrOK40zIsWRJG2ty4oYaqIiIxSKSk0+IDlVLCnzEj\nXLjNTe7Zei7GNvDFsOb8Kk6ZkGHevPBSXbsW/08QkXRQwi+WRr4YfNhwzjsvjLb5wAOw3XbJhSki\nbZcSfgm59lo4/3yYNg369086GhFpa5TwS8zUqTBuHNx6a5gkXUQkLkm20pF6jBoVBls75hi47bak\noxERqV8q57RtDXvvDQ8/HAZdW7IEzjgj6YhERNalkk7M3noLDj44NPb53e+gnX5DiUgLqIZf4las\ngEMOCS13brgB9coVkWZTDb/Ebb45PPQQrFoFhx4axtUXEUmaEn4r6dIF/v73METP4MHwzjtJRyQi\naaeE34o6dIBrroFjjw0zac2fn3REIpJmquEXyZQpcNJJ8Oc/h/q+iEg+VMMvQ6NHw/Tp8NOfatpE\nEUmGzvCLrKYmNNk88EC49FJo3z7piESklKlZZplbuRIOPxw22ij0zN1ww6QjEpFSpZJOmdtsM7jv\nvtB8c5994L33ko5IRNJACT8hnTqFTlmjR4cWPC++mHREItLWqaRTAu64A049Fa67DkaOTDoaESkl\nKumUoxkz1p83N5uFGTMYMyZsPuUUuOgi0PeiiLQGJfxiGTQoTJdYm/Rrp08cNAiAAQNg7tzQO/eY\nYzRJuojETwm/WDKZMDfuhAmweHG9k6Zvsw089hisXQtDhuhirojESzX8Ylu8GHr2DA3ye/Sodxf3\nUNq55ppwxr/77kWNUERKiGr45SqbhUmTQrKfNGn9mn7EDH7zm9Ajd9gwzaIlIvHQGX6x1Nbsa8s4\ndR83YOHC0HLnsMPg4ovDgGwikh7qaVuOZswIF2hzk3s2C3PmhLEWGvHBB3DUUbBmDdx5J2yxRSvH\nKiIlQwk/hdasCT8I7rgj1PV33TXpiESkGFTDT6H27UNJZ9IkOOgg+Mtfko5IRMqNzvDL0EsvwahR\nMHRoGHFTc+aKtF1FP8M3s/ZmNt/MpjWw/Uoze8PMXjCzfnEEJg3beWd49ln4xz9gv/3g/feTjkhE\nykG+JZ2fA68A652im9kwYHt37wX8FLgmvvCkIZkMTJsG++4L/ftDdXXSEYlIqWsy4ZtZN2AYcB1Q\n38+KEcBNAO4+F8iYWdc4g5T6tWsHEyfCjTfCmDGhxr92bdJRiUipyucM/zKgEmgolWwDvJ3z+B2g\nWwvjkgIceGAo8dx7b2izv2JF0hGJSClqtBuPmR0CLHP3+WZW0diudR7Xe3V24sSJXy1XVFRQUdHY\nS0ohuncPZZ2zzgolnrvugt12SzoqESlUdXU11a1Uo220lY6ZXQgcC6wGNgA2Aaa4+3E5+1wLVLv7\nHdHj14Ah7r60zmuplU6RTJkCJ50E558f7i2W6/sikoREOl6Z2RDgTHc/tM76YcDP3H2Yme0JXO7u\ne9bzfCX8InrjjTBv7k47weTJsOmmSUckIs2RZMcrjwIYZ2bjANx9JvBPM3sTmAyMjyMwaZleveDp\np8O8ubvuCs88k3REIpI0dbwqNS0Yc6chU6bAySdDZSX88pehdY+IlAcNrdCWNTEzVnOMHh1a8Uyd\nGr4zli2LKVYRKStK+KUmj5mxmuM734FHH4V+/cLt4YfjCVdEyodKOqUqj5mxmuuhh2DsWDjuODjv\nPOjUKdaXF5EYqaTT1uU5M1Zz7b8/zJ8fJlcZOBBeey3WlxeREqWEX2pyZ8Lq0ePr8k7MSX+rrcJY\nPD/5Cey9d5g/Vz/ARNo2lXRKTSu00mnqeG9sNYijTs6w9dZw/fXQ9RuteDwRKYhKOm3Z8OHrX6DN\nZFov+Q4aRK8bJ/DkzCx9+8I+u2SpObplrYJEpDTpDF++LiNVVvLe6ZM4+Lkq9hqa4fe/h403Tjo4\nkXTTGb7EK5MJvbJ69uTbl1Xy+MIMq1fDLrvA7NlJBycicVHCl/VaBW3qWa6/Hv73f+HYY+G00+CT\nT5IOUkRaSgk/7RppFTR8eGi6uXIl9O0LTzyRdLAi0hKq4addnq2C7r4bxo+Ho46CCy6Azp0TiFUk\nhRIZHrnFB1LCL3v//jf87GewYAFcdx0MHpx0RCJtnxK+JGrKFDj1VBg1Ci66CDbZJOmIRNoutdKR\nRI0eDS+/DP/5D+y8c6gKiUjp0xm+tMgjj8CJJ8KAAXDFFWHIBhGJj87wpWTsu29oydOtG/TpAzff\nrDF5REqVEr60WJcuoRn/zJlw6aVwwAGwaFHSUdUxY8b6A9Bls6pHSaoo4Uts+veHefNCa85Bg+C3\nv4XPPks6qkgrzCQmUm6U8CVWHTrA6aeHppuvvhou6t53X9JR0WoziYmUE120lfw0c9jm+++HU04J\n0ypefnmo9SeqFWcSE2kNumgrxdfMksjBB8NLL0Hv3mF4ht//Hr74ogjx1qeVZxITKXVK+JKffEoi\nDVwY7fzIDM4/H558Moy+2adPuMBbVEWaSUyklKmkI4VprCSSm1QzmfUfR2bMCHX+HXaAyy6DXr2K\nEHexZxITiYlKOpKfuJsiNlUSyfPC6PDhocwzZAjstRecdRZ89FHzQspbsWcSEylBSvhtWZxNEfMt\nieRMpkJlZYOtYDp1CptfegmWLYMdd4Qbb4Q1awoPTUTyo4TflsXZFHHOnHWfW/vac+asu1+BF0a3\n3hr+/GeYOhX+9KfQlv+hhwoPT0Saphp+GhSrKWKeNfyG6un+xBz+/p/hnH02bL89XHJJuMArkmZF\nreGb2QZmNtfMFpjZK2Z2UT37VJjZKjObH93OiSM4iUExmyLm+yuggVKTDR701Uicw4bB/vvDT34C\n773XeiGLpIq7N3kDukT3HYCngcF1tlcA9zbxGi5FtnKl+/jx4b6+x0mqjaWmpsGYVq50P+ss9803\nd/+f/3Fftar4YYokLcqdeeXqpm551fDd/dNosRPQHlhRz26x/OSQGOV7xp2EPC7uZjJw8cXw/PPw\nr3+F5puTJsGnn9bzeiLSpLwSvpm1M7MFwFJgtru/UmcXBwaa2QtmNtPMescdqDRDKTdFLKDU9J3v\nwE03hU5bc+eGxH/11Qn22BUpUwVdtDWzTYEHgLPdvTpn/cbAGnf/1MyGAle4+w51nuvnnnvuV48r\nKiqoqKhoWfRSnvK9uNuA556Dc84JQzBPnAhHHw3t27d+2CLFUF1dTXV19VePzzvvvOTmtDWz/wE+\nc/ffN7JPDdDf3VfkrPNCjyVtVEy9Xh97LHxPfPBBGIr5iCOU+KXtKXYrnS3MLBMtdwYOAObX2aer\nmVm0PIDwRVJfnV8ktlLTPvuEpH/ZZXDllfD978Mtt8Dq1THGWh9NpiJlKp8a/reAR6Ia/lxgmrs/\nbGbjzGxctM/hwMJon8uBMa0Trsi6zOCgg8KPg6uugj/+EXbaKfTa/fLLVjqoJlORMqWOV9LmVFfD\n+eeH/ma/+Q0cd1wYyiFWtUm+sjJcdNZkKtJK4izpKOFLm/XEEyHxv/oq/OIXcOKJsMkmMR5Ak6lI\nEWi0TJE8DB4Ms2bBPfeEuXa/+1349a/h/fcbeEIhtXlNpiJlSAlf2rxdd4Xbb4dnngnDMPfuHc72\nFy2qs2O+tXlNpiJlSiUdSZ3ly8MF3quvhoED4ec/h4qKcAE4r9q8JlORIlINXyQGn3wCN98cmnR2\n6ACnnQY/+hF0WbZYtXkpGarhi8Rgww3hpJPC6JyXXhpq/X26Z3lq9CTem1NCtXm1+5eYKOFL6pnB\nAQfAtJuzPDd0AvcOqKLPoT0Y+04Vbx83gbUrEk76avcvMVFJR6RWTm3+o49Cuee2q7PstGIOvX4x\nnOOPh622Sig2tftPLdXwRYrEPbTumTw5TMN44IEwbly4yNuu2L+P1e4/lVTDFykSM9hjD7jhhpBn\n99kndOL63vfCSfZbbxUpELX7lxjoDF+kQO5hXP6bboK//hX69oWxY+Gww2CjjVrhgC0cTlrKm0o6\nIiXi889h2rSQ/J94AkaODMk/1pKP2v2nmhK+SAlauhRuuy0k/+XLw/j8Y8aEkpBpAlBpJiV8kRL3\n6quh3HPHHfDZZ/Bf/wVHHhmGeVDyl0Io4YuUuqgM45tmWLgQ7rwTZt6WZbf/zGGLscMZORIGDEig\npY+UHbXSESl1UWcpW5Vll12gqjLL80MncMptg2jXDv77v2GbbUITz5kzw7UAkdamM3yR1tJEZ6k3\n3wzDOdx9N7z4YujtO2JEmMGra9cE45aSopKOSLnIs7PU8uUwfTrcey/Mng3bbRcS/8EHw157QceO\nRYtYSoxKOiLloIDOUltuCSecEHrzLl8Ol18eLu6ecUbYNmoUXHstvP566AfQJmmQuPUsXx7v6ynh\ni7SGFkyS0rEj7L13eMq8eSHJjx4NTz4J++0Xav9HHRWGe1i0KOcLoNwTZj6DxJX739iEJUtgypQw\nR8Muu0CvXjEfwN2LcguHEkmJ6dPdV65cd93KlWF9c/aLrF3r/o9/uF9/vftxx7l37+6+9dbuRx7p\nfu3FK33p4eP9P0tXfv0648ev//qlrDbmmpr6Y6/7N5Xj3xhZu9Z90aLwb3n88e7bb+++2Wbuhxzi\nfvHF7nPnun/5pXuUO2PJw6rhiySphcMmuIfLBNXV8NRT8PKcLGNfn8CsH1Ry+upJ/Pv0Kvrvl6Fb\nt1b/S+LT1HWPMh05dPny8Ivt2WfD/dy5sMEGYe7lvfcO9717r99UVxdtRdqSmBPYRwsXs/EuPbnq\nzBoeWNSDp54KM3r17Qv9+oX7vn1h++0b6QeQ1HAO+b4XcY0c2gp/p3sozSxcCPPnf53gs1no3x92\n3x122y30w9h226ZfL86Er5KOSCmoqXGHcN8S9ZRE1q51/9e/3O+5x33iRPcf/tC9Rw/3jTZyHzjQ\n/aST3K+4wn3WLPe33gqlhlhLJ/mWrfI9ZlNln9Y4ZgNWrXJ/8kn3yZPdTz3VvaLC/ZvfDLeKCvdf\n/ML91lvdX3/dfc2avF5yPcRY0lHCF0laPgmskNdpKnlFyXDFCvfZs92vvNL9jB+v9LP7TPettw5f\nBP37u594xEp/dsB4n3pZjb932Hh//9Xw5dHaca333NwkXeiXQj6JvIn3/+OP3RcscL/rLvcLL3Q/\n4QT3wYPdu3Z179IlvFfHH+9+6aXuDzzg/t573rz3qQFK+CJtRQmeSa9c6f7UU+433OB+8Uk17uAj\ndqnxLbZw79zZfccd3YcODU+549jpfvMfVvrMme7z57svWeK+5oNGjtnSL7VCLnI3ccy1a92z2XD2\nPffO8Hde/asaP+UU90MPde/bN5ypb7CBe+/e7iNHup95Zjibf+QR97ffbv5ZeyHiTPiq4YskqZRr\n5fXs83GHDIsXh9J5TQ2s+GeWwfdP4KpvVfHmvzN88m6WylUT+EPXKjptlWHzzWGzzcJtu/aL+fUf\ne3LrBTW0+24POneGLl3CrXa5c+dwIbNdu/VvZuG2ejV88UX9t48/ho8+gg8/DDf712JO+l1PJo6t\n4c3VPVi+HJYtCxdQly+Hb3wDtvtmlnM+n0D1bpUc9c4kXjyyiq7fy7DtttC9e+gHkeSAd7poKyIt\n19iFz0JaD9X5Yvji3CqWfJ7hgw9gxQpYuRI+fifLLndO4MG+lQx4bBI3f6+KD9Zk+PTTMJpo7v3n\nn8PateHm/vVy7eOOHaFTp/VvHTuGCWg22QQ23hi6fiPLEQsm8Ny+lQx5ZhIv/6iKzXpm2GqrkMS3\n3BI2+DzGyWVa6cu7aBdtgQ2AucAC4BXgogb2uxJ4A3gB6NfAPq3ze0dECtdUiaXA/gGNXnROou18\nnNcN4j5mgShmDR/oEt13AJ4GBtfZPgyYGS3vATzdwOu06I8WkZjEnZji/vKIQxLHrD1GHNcqcsSZ\n8PMu6ZhZF+BRYKy7v5Kz/lpgtrvfGT1+DRji7kvrPN/zPZaItKI4Sw9pmG+30Pcrrj4CkaIOnmZm\n7cxsAbCUkNhfqbPLNsDbOY/fAcqpX59Iugwfvn4yzmSaV2eeM2fd5J7JhMdz5rQ8zlKRzxg/tQoY\nMC8JTSZ8d1/r7n0JSXwfM6uoZ7e63z46lRdJgzi/PEpV7ZfYhAnh7L2pi9fNGDCvWDrku6O7rzKz\nGcBuQHXOpneB7jmPu0Xr1jNx4sSvlisqKqioqMg/UhGRpGQyoRVSbammvnJVY792CvgCrK6uprq6\nOp6462i0hm9mWwCr3T1rZp2BB4Dz3P3hnH2GAT9z92FmtidwubvvWc9rqYYvIuUpwQHbilnD/xbw\nSFTDnwtMc/eHzWycmY0DcPeZwD/N7E1gMjA+jsBEREpCGZRq8qWOVyIijUmqN3REPW1FRFJCc9qK\niEjBlPBFRFJCCV9EJCWU8EVEUkIJX0QkJZTwRURSQglfRCQllPBFRFJCCV9EJCWU8EVEUkIJX0Qk\nJZTwRURSQglfRCQllPBFRFJCCV9EJCWU8EVEUkIJX0QkJZTwRURSQglfRCQllPBFRFJCCV9EJCWU\n8EVEUkIJX0QkJZTwRURSQglfRCQllPBFRFJCCV9EJCWaTPhm1t3MZpvZy2b2kpmdVs8+FWa2yszm\nR7dzWidcERFprnzO8L8ETnf37wN7AqeY2U717Peou/eLbhfEGmUJqK6uTjqEFinn+Ms5dlD8SSv3\n+OPUZMJ39yXuviBa/hh4Ffh2PbtazLGVlHL/T1PO8Zdz7KD4k1bu8cepoBq+mfUA+gFz62xyYKCZ\nvWBmM82sdzzhiYhIXDrku6OZbQT8Dfh5dKaf63mgu7t/amZDgbuBHeILU0REWsrcvemdzDoC04H7\n3P3yPPavAfq7+4qcdU0fSERE1uPusZTMmzzDNzMDrgdeaSjZm1lXYJm7u5kNIHyRrMjdJ66ARUSk\nefIp6QwCjgFeNLP50brfANsCuPtk4HDgZDNbDXwKjGmFWEVEpAXyKumIiEj5a3ZPWzO7wcyWmtnC\nnHUDzOyZqPPVs2a2e7R+AzO73cxeNLNXzOzsnOf0N7OFZvaGmV3Rsj+nxfH/wMyeiuK818w2ztn2\n6yjG18zswHKK38wOMLN50fp5Zvb/yin+nO3bmtnHZvbLcovfzHaJtr0Ube+UVPwF/t8pxc9uvZ1B\nzWxzM3vQzF43s1lmlsl5Tsl8fguNP9bPr7s36wbsTWiiuTBnXTVwULQ8FJgdLR8P3B4tdwZqgG2j\nx88AA6LlmcDBzY0phvifBfaOlk8Azo+WewMLgI5AD+BNvv51VA7x9wW2jpa/D7yT85ySjz9n+9+A\nO4FfllP8hNLpC0Cf6PFmQLuk4i8w9lL87G4N9I2WNwIWATsBlwC/itafBVwcLZfU57cZ8cf2+W32\nGb67Pw6srLP6fWDTaDkDvJuzfkMzaw9sCHwBfGhm3wI2dvdnov3+AvywuTEVooH4e0XrAR4CRkfL\nIwn/6b9098WE/zB7lEv87r7A3ZdE618BOptZx3KJH8DMfgj8kxB/7bpyif9A4EV3Xxg9d6W7r00q\n/gJjL8XPbn2dQbcBRgA3RbvdlBNPSX1+C40/zs9v3IOnnQ1camZvAZMIF3dx9weADwn/eRYDk9w9\nS/gj38l5/rvRuqS8bGYjo+UjgO7R8rdZN853CHHWXV+q8ecaDTzn7l9SJu+/hT4gvwIm1tm/LOIn\n9ElxM7vfzJ4zs8pofSnFX2/spf7ZtXU7g3Z196XRpqVA12i5ZD+/ecafq0Wf37gT/vXAae6+LXB6\n9BgzO4bwc/BbQE/gTDPrGfOx4/BjYLyZzSP81Poi4XgK1Wj8ZvZ94GJgXAKx5aOh+CcCl7n7p5T2\nEB4Nxd8BGAz8KLofZWb7Enqol4p6Yy/lz250IjCF0Bn0o9xtHmocpfT+rqfQ+OP4/Obd0zZPA9x9\n/2j5b8B10fJAYKq7rwGWm9kcoD/wBNAt5/nd+LoMVHTuvgg4CMDMdgCGR5veZd2z5W6Eb9Z3KY/4\nMbNuwN+BY929Jlpd6vEPizYNAEab2SWEUuFaM/uM8PeUcvy17//bwGMe9U0xs5nArsAtlEj8jbz3\nJfnZtdAZdApws7vfHa1eamZbu/uSqNyxLFpfcp/fAuOP7fMb9xn+m2Y2JFreF3g9Wn4teoyZbUgY\ndfO1qC71oZntYWYGHEsYliERZrZldN8OOAe4Jtp0LzDGzDpFZze9gGfKJf7oav8M4Cx3f6p2f3d/\nn9KO/9oozn3cvae79wQuB6rc/epyef+BB4A+ZtbZzDoAQ4CXSyn+ht57SvCzGx2vvs6g9wJjo+Wx\nOfGU1Oe30Phj/fy24Erz7cB7hJ9+bxOu7O9GqEUtAJ4C+kX7foNwNrMQeJl1W1n0j9a/CVzZ3Hhi\niP/HwGk7gsB+AAAAkUlEQVSEK+aLgAvr7P+bKMbXiFoilUv8hA/wx8D8nNsW5RJ/needC5xRTu9/\ntP/RwEtRrBcnGX+B/3dK8bM7GFgb5Zna/88HA5sTLji/DswCMjnPKZnPb6Hxx/n5VccrEZGU0BSH\nIiIpoYQvIpISSvgiIimhhC8ikhJK+CIiKaGELyKSEkr4IiIpoYQvIpIS/wcBL7feDAyMTgAAAABJ\nRU5ErkJggg==\n", "text": [ "" ] } ], "prompt_number": 25 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note the values of the coefficient $w_2$ in particular. It is relatively small, because it is multiplying a large number (square of 2000 is 4 million). This need to use small coefficients becomes worse as we increase the order of the fit. As an exercise for later, try fitting higher order polynomials to the data. See what happens as you increase the polynomial order.\n", "\n", "## Generalization\n", "\n", "The aim of this notebook is to review the different methods of model selection: hold out validation, leave one out cross validation and cross validation. \n", "\n", "## Training Error\n", "\n", "The first thing we'll do is plot the training error for the polynomial fit. To do this let's set up some parameters." ] }, { "cell_type": "code", "collapsed": false, "input": [ "num_data = x.shape[0]\n", "num_pred_data = 100 # how many points to use for plotting predictions\n", "x_pred = np.linspace(1890, 2016, num_pred_data)[:, None] # input locations for predictions\n", "order = 4 # The polynomial order to use.\n", " " ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 26 }, { "cell_type": "markdown", "metadata": {}, "source": [ "now let's build the basis matrices.\n" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# build the basis set\n", "Phi = np.zeros((num_data, order+1))\n", "Phi_pred = np.zeros((num_pred_data, order+1))\n", "for i in range(0, order+1):\n", " Phi[:, i:i+1] = x**i\n", " Phi_pred[:, i:i+1] = x_pred**i" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 27 }, { "cell_type": "markdown", "metadata": {}, "source": [ "now we can solve for the regression weights and make predictions both for the training data points, and the test data points. That involves solving the linear system given by\n", "\n", "$$\\boldsymbol{\\Phi}^\\top \\boldsymbol{\\Phi} \\mathbf{w}^* = \\boldsymbol{\\Phi}^\\top \\mathbf{y}$$" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# solve the linear system\n", "w_star = np.linalg.solve(np.dot(Phi.T, Phi), np.dot(Phi.T, y))" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 28 }, { "cell_type": "markdown", "metadata": {}, "source": [ "and using the resulting vector to make predictions at the training points and test points,\n", "\n", "$$\\mathbf{f} = \\boldsymbol{\\Phi}\\mathbf{w}.$$\n", "\n", "To implement this in practice we need to use basis matrices for both the predictions and the training points." ] }, { "cell_type": "code", "collapsed": false, "input": [ "# predict at training and test points\n", "f = np.dot(Phi, w_star)\n", "f_pred = np.dot(Phi_pred, w_star)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 29 }, { "cell_type": "markdown", "metadata": {}, "source": [ "These can be used to compute the error\n", "\n", "$$E(\\mathbf{w}) = \\frac{n}{2} \\log \\sigma^2 + \\frac{1}{2\\sigma^2} \\sum_{i=1}^n \\left(y_i - \\mathbf{w}^\\top \\phi(\\mathbf{x}_i)\\right)^2 \\\\\\\n", "E(\\mathbf{w}) = \\frac{n}{2} \\log \\sigma^2 + \\frac{1}{2\\sigma^2} \\sum_{i=1}^n \\left(y_i - f_i\\right)^2$$" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# compute the sum of squares term\n", "sum_squares = ((y-f)**2).sum()\n", "# fit the noise variance\n", "sigma2 = sum_squares/num_data\n", "error = 0.5*(num_data*np.log(sigma2) + sum_squares/sigma2)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 30 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we have the fit and the error, let's plot the fit and the error." ] }, { "cell_type": "code", "collapsed": false, "input": [ "# print the error and plot the predictions\n", "print(\"The error is: %2.4f\"%error)\n", "plt.plot(x_pred, f_pred)\n", "plt.plot(x, y, 'rx')\n", "ax = plt.gca()\n", "ax.set_title('Predictions for Order 5')\n", "ax.set_xlabel('year')\n", "ax.set_ylabel('pace (min/km)')" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "The error is: -29.9371\n" ] }, { "metadata": {}, "output_type": "pyout", "prompt_number": 31, "text": [ "" ] }, { "metadata": {}, "output_type": "display_data", "png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEZCAYAAABmTgnDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFNXZ9/HvrSAgouMWZFNI1ChuIAooohN9giJojLgk\nxpVEUWI0RlHjuGDMqHlM3F6NmqhR4xaXaBDcEB1UghJZFHB/HBRQEMM0ogiy3O8fpwaaYWa6Z6a7\nqnvm97muuqaqurrq7obuu885dc4xd0dERKQ+GyUdgIiIFD4lCxERyUjJQkREMlKyEBGRjJQsREQk\nIyULERHJSMlCCo6Z3WNmV0XrA83s3Uae5zYzuzS30WW85o/NbK6ZLTWzveK8dn3M7FQzeyXpOKR4\nKVlIo5jZHDNbFn0pLjCzv5lZ+xyd3qMFd3/F3XfJIp4Nvgzd/Sx3/32OYsrWH4GR7t7B3d/MxQnN\nbKiZTTGzr8zsCzO738y65OLcTYjpHjNbEf37LzWzL83MkoxJ8kvJQhrLgaHu3gHYG9gH2OBXvJm1\nauT5i+6LJ/qy3B54u5HP3+DzaGbHAA8A1wNbA7sBK4BXzaykjvNs3JjrNyQuwr//H6Kk2MHdN3f1\n8G3WlCykydz9U+BZwhcZZrbGzEaa2QfAe9G+oWY2w8yqzGySme1R/Xwz621m06Jfpw8DbdMeKzWz\nuWnb3czsn2b2efQr+/+Z2S7A7cB+0a/cxdGxa6uzou3TzewDM/uvmf3LzDqlPbbGzEaY2ftRjLek\nPbajmU00s5SZLYpiXI+ZtQGWAhsDb0avHTPb1cwqonPOMrMj0p5zT1RV9rSZfQWU1jinAX8CrnL3\nh919hbsvBH4BfAWcFx13avSeXm9mXwBXmNlWZjbGzJaY2evA92qcexczGx+9F++a2bHZxpV+mjr2\nS3Pk7lq0NHgBKoFDovVuwCzgymh7DfAcUAK0AXoDC4F9CV8wJ0fPbw1sAnwMnEv4oh0GfAv8LjpX\nKTA3Wt8YeJPwBdouOvf+0WOnAK/UiPFvaec5GFgE9IqueTMwMe3YNcAYYPPo9XwODIoeewj4bbS+\nSfU163hf1gDfjdZbAx8CFwOtgB8AXwI7R4/fA6SA/aLtNjXOtUt0vh1quc5o4N/R+qnASuCXhB+A\nbYGHo6UdIYnPA16Ojm8PzI3es42i92QRsGs2caW9t/+NljeAo5P+P6klv4tKFtJYBjxpZlXAK0AF\ncHXa49e4e8rdVwBnAHe4+388uI9QlbIf0B9o5e43uftqd38c+E8d1+wLdAJGufs3Hn5p/zstnvr8\nDLjL3We4+7fAbwklke3TjrnW3b9097nAS4QvUQjJq7uZdXH3b9OumUl/oL27X+vuq9z9JWAs8NO0\nY55098kA0XuVbpvo72e1nHtB2uMAn7r7re6+hpA4jgYuj96n2cC9rHuPhgKV7n6vu69x9xnAP4Fj\n085XX1wQku2OwLbAZcA9ZrZ/3W+FFDslC2ksB37k7lu6e3d3P7vGl8rctPUdgPOjqpiqKMF0JXzx\ndwbm1zj3x3VcsxvwcfSF2FCd0s/r7l8TfhWnNxQvSFtfBnSI1i8kfNFOiaqSTsvymp1Z/30giqFz\ndRi1PJ7ui7TYa+pEKA1USz/PtoSSTPq+T9LWdwD61fj3OAHomGVcuPt0d6+Kks0zhHaVo+t7jhQ3\nJQvJl/TGzk+A8iixVC+bufs/CL+aa97Zs0Md55wLbF9HA26mxtVPge7VG9GdW1uzYaLa8MTuC939\nDHfvAowA/mxm3830vOia3WrcJbRDNteMvEeoPjoufWfU4DwMmJAeZtr6ImAVobG9Wvr6J4QquPR/\njw7u/sss45IWSMlC4vBX4Ewz62tBezMbYmabAf8GVpnZOWbW2syOJrRt1GYKIblca2abmlnbtKqP\nhUBXM2uddryxrurlIeA0M9sraoy+GnjN3dN/cVPjuWHF7Fgz6xptpghfzNmUbl4jlFAujF5bKaEK\nqLqBvN6qM3d34ALgUjP7afR6twPuBDYDbqjjeasJ1UqjzaydmfUktE9UJ5RxwM5mdmIUV2sz2ze6\nUSBjXBDu0jKzzcxsIzMbRKjmG5PpeVK8lCwkH9b7le/uU4HTgVuAxcAHhEZu3L26fv1UQrXQccDj\ntZ0v+hI8glBX/gmhpFH9q3sCMBtYYGafpz2v+rkTCHXrjxN+8fcAflJXzOnPJdwW/JqZLQX+BZzj\n7nMyvfbotR0BDCb82r8FOMnd36/lGrWfzP0R4CTCnU9fRK+xDTDA3avqOc/ZhISyALg7WqrPuRQY\nRHj98wkJ+BpC431WcQHnEEo9VcAfgF+4+8sZniNFzMKPlzxewGwO4Q6Q1cBKd+9b4/FSwgfwo2jX\n4x5/RyoREalHYztMNYQDpe6+uJ5jJrr7kTHEIiIijRBXNVSmOlB17hERKWBxJAsHXjCzN8zs9Doe\n39/M3ox6jPaMISYREWmAOKqhBrj7Z2a2LTDezN519/QB36YB3dx9mZkNBp4Edo4hLhERyVLeG7jX\nu5jZFcBX7v6neo6pBPqkt3GYmQYoExFpBHfPSTV/XquhonvhO0Tr7Qm3682scUzH6k5LZtaXkMA2\naAxPelyUpixXXHFF4jG01PiLOXbFn/xS7PHnUr6roToCT0S5oBXwgLs/b2YjANz9DuAY4CwzW0Xo\nwPSTuk4mIiLJyGuycPdK1g3Glr7/jrT1W4Fb8xmHiIg0jXpwx6C0tDTpEJqkmOMv5thB8Set2OPP\npVgbuBvLzLwY4hQRKSRmhhdDA7eIiDQPShYiIpKRkoWIiGSkZCEiIhkpWYiISEZKFiIikpGShYiI\nZKRkISIiGSlZiIhIRkoWIiKSkZKFiIhkpGSRpHHjIJVaf18qFfaLiBQQJYskDRgAZWXrEkYqFbYH\nDEg2LhGRGjTqbNKqE8SoUXDddVBeDiUlSUclIs1ALkedVbIoBHPmQI8eUFkJ3bsnHY2INBMaorw5\nSaVCiaKyMvyt2YYhIlIAlCySVF0FVV4eShTl5eu3YYiIFAhVQyVp3LjQmJ3eRpFKwaRJMGRIcnGJ\nSLOgNgsREclIbRYiIhIrJQsREclIyUJERDJSshARkYyULEREJKMWnyy+/RaWLUs6ChGRwtbik0V1\nnzgREalb3vtZmNkc4EtgNbDS3fvWcszNwGBgGXCqu0+v8Xje+lnMmwd77QUzZkC3bnm5hIhIIoqt\nn4UDpe7eu45EcTiwo7vvBJwB3BZDTGt17QpnnQWXXBLnVUVEiktc1VD1ZbYjgXsB3P11oMTMOsYS\nVeSii+CFF2Dq1DivKiJSPOIqWbxgZm+Y2em1PN4FmJu2PQ/oGkNca3XoAFdeCeefDxpVRERkQ3Ek\niwHu3pvQJvFLMxtYyzE1Sx6xf2UPHw6LFsGYMXFfWUSk8LXK9wXc/bPo7yIzewLoC7ySdsh8IL1p\nuWu0bz2jR49eu15aWkppaWlO42zVCq6/Hs4+Gw47DNq0yenpRUTyrqKigoqKirycO693Q5nZpsDG\n7r7UzNoDzwNXuvvzacccDpzt7oebWX/gRnfvX+M8sY06O3QoHHRQmOVURKSYFc0Q5WbWA3gi2mwF\nPODu15jZCAB3vyM67hbgMOBr4DR3n1bjPLEli/ffh/33h9mzoWOszewiIrlVNMkiV+Kez+L882HJ\nErjzztguKSKSc0oWeZZKwS67wNNPw957x3ZZEZGcKrZOeUWnpASuugp+9StYsybpaEREkqdkUYfh\nw8Mgg/ffn3QkIiLJUzVUPaZMgaOOgnfegS22iP3yIiJNojaLGJ1+OrRvDzfemMjlRUQaTckiRosW\nwW67wYQJsMceiYQgItIoauCO0bbbhnGjRo5UY7eItFxKFlk44wxYsQLuvTfpSEREkqFqqCxNmwaD\nB4ee3dtsk2goIiJZUZtFQn79a1i6FO66K+lIREQyU7JIyNKl0LMnPPggDKxtoHURkQKiBu6EdOgQ\nbqEdMSK0YYiItBRKFg109NGw885w7bVJRyIiEh9VQzXC/PnQqxdMnBiqpURECpGqoRLWpQv87neh\nd7f6XohIS6Bk0UgjRoAZ3HZb0pGIiOSfqqGa4J134MAD4Y03YIcdko5GRGR9qoYqELvuGmbV+8Uv\noABzmYhIzihZNNEFF4SZ9TQFq4g0Z6qGyoFZs+AHPwhDgnTrlnQ0IiKBqqEKzO67w7nnhrujCjin\niYg0mpJFjlx0Efz3v/CXvyQdiYhI7qkaKoeq74567TX43veSjkZEWjpVQxWoXXeFSy6BU06B1auT\njkZEJHeULHLs3HOhVSv405+SjkREJHdUDZUHc+bAvvvC+PFhDCkRkSSoGqrAde8ON9wAJ5wAy5Yl\nHY2ISNOpZJEn7vCzn8FWW8EttyQdjYi0RJopr0ikUqEa6tZbYciQpKMRkZamqKqhzGxjM5tuZk/V\n8lipmS2JHp9uZpfmO544lZTA3/8exo767LOkoxERabw42izOBd4G6ioaTHT33tHy+xjiidXAgXDm\nmXDiibqdVkSKV16ThZl1BQ4H7gTqKgrlpIhUyC69NCSKP/wh6UhERBon3yWLG4BRQF3zyTmwv5m9\naWZPm1nhT1I6blxojEiXSoX9ddh4Y7j/frj5Zpg0Kc/xiYjkQat8ndjMhgKfu/t0Myut47BpQDd3\nX2Zmg4EngZ1rO3D06NFr10tLSyktreuUeTZgAJSVQXl5aJRIpdZtpxs3LhxbUgJA165wz40p7vzx\nJL7/9hC22SaB2EWkWauoqKCioiIv587b3VBmdjVwErAKaAtsDjzu7ifX85xKoI+7L66xv7DuhqpO\nEKNGwXXXrUsctR1TI6lctlE5U/+vhLFjYSP1chGRPCq6W2fN7CDgAnc/osb+joTSh5tZX+ARd+9e\ny/MLK1lA6KbdowdUVoZeeLWpJamsbF/CwQfDYYeFh0RE8iWXyaLeaigzaw0MAg4EuhPaGD4GXgae\nc/dVDbiWR+ccAeDudwDHAGeZ2SpgGfCTBsafjFQqfPlXVtZdsoCwb9SodUmlpITWwMMPh+FA9t8/\nTJokIlLo6ixZmNllwDBgMjAF+JTQIN4J6Av0Bx6L43bXgipZ1FG9VG9VVC3VVS+8ACefDP/5D3Tp\nksDrEJFmL5ZqKDM7Eniqrm9pM9sIGOruY3IRSH0KKlnUaLgGQlKYNGn9btpZJJVrroExY6CiAtq0\nifdliEjzV3RtFk1VUMkiW1kkFXcYNgw6doTbbksoThFptmJNFma2L3AJoc2iuo3D3X3PXASQjaJM\nFln68kvo2xcuvBCGD086GhFpTuJOFu8DFwCzSOtc5+5zchFANppzsoB107GOHQv9+iUdjYg0F3En\ni0nuPiAXF2us5p4sAJ56Cs46C15/XQ3eIpIbcSeLQcDxwAvAt9Fud/d/5iKAbLSEZAFw9dXw5JMw\ncSK0a5d0NCJS7OJOFg8A3wdms3411Gm5CCAbLSVZuMNPfwqtW8N994E1+yEWRSSf4k4W7wG7JPlt\n3VKSBYRpWA88EI4+Gi65JOloRKSYxT350b+Bwh8NtpnYdNPQ9+L22+HRR5OORkQkyKZk8S7wPaAS\nWBHt1q2zeTZjBgwaFO6Q6ts36WhEpBjFXQ21AxtOUOTu/nEuAshGS0wWEO6QOvPM0I+vrrEKRUTq\nEnc11P+4+5z0BTgrFxeX+h1xBFx8MQweDIsXZz5eRCRfskkWx5jZidUbZnYr8J38hSTpfvWrMDrI\nUUfB8uVJRyMiLVU21VDtgDHA3cBgoMrdz40htvQYWmQ1VLU1a8IttQAPPaRJk0QkO3GNOrtV2mYH\n4F/Aq8DlADVns8unlp4sIJQqDj0U9toLbrpJfTBEJLO4ksUcogmLqnelbbu7fzcXAWRDySJYsgQO\nOgiOPVaz7IlIZnHNlDfA3efn4iKSG1tsAc88AwccANtuC2eckXREItJS1Jcs/mpmWwMvAc8CrzZw\nGlXJg06d4LnnQi/vkhI47rikIxKRlqDOplJ3PxwoBSYCRwOvmdkTZnaGmW0fU3wCYSKlVGrt5o47\nwvOPpPjXGeMYNy7BuESkxaj3vhp3/8bdn3H3c9x9H+B8oDVwq5lNiSVCCTPulZWtSxipFLs/VMZ5\njw3g1FPhpZcSjU5EWoBGT6tqZm3cfUXmI5tODdysm8N71Ci47rq1c3m/9BIcfzz861+w335JByki\nhSTu4T6GAdcCHVk37Ie7++a5CCAbShaROXOgRw+orFxv/I9nn4WTT9Y4UiKyvriH+/hf4Eh339zd\nO0RLbIlCIqlUKFFUVoa/aW0Yhx0Gd98dhgeZOjXBGEWk2comWSxw93fyHonUrboKqrw8lCjKy9dv\nwwCGDoW//AUOPxymTUsuVBFpnrKphroJ2A54Ek2rmoxx40Ijd0nJun2pVBiOdsiQ9Q598kkYMSJU\nSe27b8xxikhBibvN4p5odb0DNa1q4Ro7FoYPV6O3SEsXa7IoBEoWDVfd6P3YY6EDn4i0PHGNDXWR\nu//BzP5fLQ+7u5+TiwCyoWTROBMmhNFq77svNIKLSMsS191Qb0d/p9axZMXMNjaz6Wb2VB2P32xm\nH5jZm2bWO9vzSmaHHBKqok45JZQwREQaq86xodz9qejvPU28xrmExNOh5gNmdjiwo7vvZGb9gNuA\n/k28nqTZb78wltTgwWHU2p//POmIRKQYZbx11sz2jcaEmm5mM6PlrWxObmZdgcOBO9lwHm+AI4F7\nAdz9daDEzDpmH75ko1cvmDgRfv97uOYaUI2eiDRUfaPOVnsAuACYBaxp4PlvAEYBdXXi6wLMTdue\nB3QFFjbwOpLBzjuHO20PPRQWLoTrr9eMeyKSvWySxSJ3H9PQE5vZUOBzd59uZqX1HVpju9bfvaNH\nj167XlpaSmlpfaeU2nTuDC+/DEceGRq+770X2rZNOioRyZWKigoqKirycu5s+lkMAo4HXqABnfLM\n7GrgJGAV0JZQunjc3U9OO+Z2oMLdH4623wUOcveFNc6lu6FyaPnycFvtggWhE99WW2V+jogUn7jH\nhjoF2As4DBgaLUdkepK7X+Lu3dy9B/AT4MX0RBEZA5wMYGb9gVTNRCG517YtPPww9OsH++8PH32U\ndEQiUuiyqYbaB9glBz/tHcDMRgC4+x3u/rSZHW5mHwJfA7H1Cm/pNtoojEfYvXsYSeTRR8N0rSIi\ntcmmGupvwB/dfXY8IdUag6qhstGAMaTSVff2/tOf4KSTYohTRGIRdzXUfsAMM3u/obfOSsxqmVGP\nsrKwvx6HHRZm27v8cvjtb2H16hhiFZGikk3Jontt+919Tu7DqTMGlSyyVceMetlYtAiOPRY22wwe\neAC22CLPsYpIXsU1NlQHd1+aIZCMx+SCkkUD1TGjXjZWroTzzoMXXghDhXz/+3mJUERiEFc11BNm\ndquZDTKztTdXmtnWZnaomd0GPJGLICSH6plRLxutW8Mtt8AFF8DAgfCE/oVFhAzVUGZ2MHACMADo\nHO3+FHgVeMDdK/IdYBSHShbZSJ9Rr6Rkw+0GmjIlVEudcAJcdRW0yubeOREpGJrPQmrXyLuh6rNo\nUejtvWYNPPggbLddfq8nIrkT991QUiyGDNmwBFFS0qQv7m23DaPWDhwIffrAiy+mPdjIu69EpPio\nZCFZGz8+9McYMQIuvTSqlmrC3Vcikl+qhpLEfPppSBgrVoTba7ffnibdfSUi+RN7NZSZDTSz06L1\nbc2sRy4uLsWnc2d4/nkYOhT22QeevKdpd1+JSHHIplPeaKAP8H1339nMugCPuHtsFdMqWRSmN15I\n8e4xZUwaXM61t5ewhTft7isRya24SxY/Bn5EGOgPd59PLVOkSsuzz4pJ/Hh2Ob5FCb16wctvlYRE\nMWlS0qGJSI5lU7KY4u59zWy6u/c2s/bAZHffM54QVbIoBuPGwRlnhH4ZV18Nm26adEQiEnfJ4lEz\nu4MwP/YZwATCnNoiaw0ZAjNnwhdfhDm/VbgQaV6yuhsqmi1vULT5nLuPz2tUG15fJYsi8sQT8Mtf\nwrBhoZTRQZWWIomI9dbZ6M6nBe7+TbTdDuioUWelPosXh/GlJkyA22+HwYOTjkik5Yk7WUwF9nP3\nb6PtNsAkd98nFwFkQ8mieI0fD2eeGW6zvfFG6NQp6YhEWo642yw2rk4UAO6+Amidi4tL8/fDH8Ks\nWbDjjrDnnnDrrUU4udK4cRv2H0mlwn6RFiKbZPGFmf2oeiNa/yJ/IUlz065duKO2oiLM9d23L7z2\nWtJRNYDGwBLJqhpqR+AB1g1RPg84yd0/zHNs6TGoGqqZcIeHHgpDSR16aGgAX28k20KlMbCkCCUy\nNpSZdQDc3b/KxYUbQsmiyNUylPmXn6R47LxJXDhxCBdeCOeeC23aJBhjNjQGlhSZJMaGGgqcBfzG\nzC43s8tzcXFpIWqpxtn8D2UMv2sAkyfDq69Cz57w2GOh5FGQmjgDoUixy5gsog55xwHnABat75Dn\nuKQ5KYmGASkrC7/O08aP2mknGDMGHhs+jpuuTDFgAEyeHD2vUBqR02cc7N593WtRwpAWJJs2i5nu\nvoeZveXue5rZZsCz7n5APCGqGqrZqK8aJ5VizSVlPLx7ORddU8JBe6W4efMytvpzAbQNaEZAKVJx\nV0N9E/1dFo04uwoohiZJaYpc3y6aqRqnpISNri7nhNllfDB+DucvLqPv8+X8/PwSPv64cZfMmTzM\nQChSbLJJFk+Z2ZbAdcA0YA7wUD6DkgKQy9tFs63GKSmBUaNou2sPej84ijc+LGG77WDvveHss8PE\nSyKSEHfPegHaAFs05Dm5WEKYEruqKveRI90rK8PfqqrGnWfs2A2fW1UV9mdxvc8/d//Nb9y33NL9\nnHPc589vXBgiLU303ZmT7+Fs2izaASOBAwAHXgFuc/fl+UxiNWLwTHFKnsR1u2h66aOkZMNt4LPP\nYOxZ4yivGMCRJ5cwahR064baD0TqEHebxX1AT+Bm4BZgN+Dv2ZzczNqa2etmNsPM3jaza2o5ptTM\nlpjZ9Gi5tCEvQPIozttFJ01av6Nb9R1UaWOdd+oEp98zgNk/LmMLT7HXXvCrk1Kkfqne1CJ5l6no\nAbydzb56nr9p9LcV8BpwQI3HS4ExGc6Rs2KZZKm6Sqi6+qjmdpKiWBZPq/Qp+470725V5ccc4z5l\nStKBiRQWclgNlU3JYpqZ7Ve9YWb9gakNSEbLotVNgI2BxbUclpNikuRQFr/0ExM1hG+5dw/2fWQU\nb35cwgEHwDHHwA9+AGPHwpo1SQcp0rxk02bxLrAzMJfQZrE98B7hFlr3DNOrmtlGhLuovkdo67iw\nxuMHAf8kjDk1H7jA3d+ucYxnilNakDrGaVq5Eh55BK6/Hr76Cn79azj5ZGjfPumARZIR93wW3et7\n3LOcBMnMtgCeAy5294q0/R2A1e6+zMwGAze5+841nutXXHHF2u3S0lJKS0uzuaw0N1k0hLvDK6+E\npPHqq3DqqWHmvh49kg1dJN8qKiqoqKhYu33llVfGP5BgTi5mdhnwjbv/sZ5jKoE+7r44bZ9KFhI0\nsDd1ZSX8+c/wt79B//4wcmQY7XbjjWOMWSQhiYw626iTm20DrHL3VHQL7nPAle4+Ie2YjsDn7u5m\n1hd4xN271ziPkoU0ybJl8PDDIXEsXgynnw6nnZbA8OgaOkRiFPuos03QCXjRzGYArwNPufsEMxth\nZiOiY44BZkbH3Aj8JM8xSQu06aYwfDi88Qb84x/w0Uew664wbBg880yMs/dpIiUpUrFWQzWWShaS\nD19+GSZiuuuuMJTIqaeGZccd83xhTaQkMSmaaqhcUbKQfJs5E+6+Gx58MCSLU06B447L43e4JlKS\nGBRTNZRIUdhjD7jhBpg3Dy6+GJ5/HnbYIVRTPfEErFiR4QQNGaVXEylJEVKyEEnTujUccUSYtW/O\nHBg8GG68MQw1Mnw4jB8Pq1bV8sRs2yI0kZIUKVVDiWRh3rzQ4e+hh+CTT+Doo0M11YEHpt2Gm01b\nhO6GkhipzUIkQR99BI8+Gu6qmj8fjjoqJI+DD4bW8+eoLUIKhtosRBL03e/CRRfBtGlhvvCddoLR\no2GnbVOMH3QdT99aybdXF0hbRK5nPJQWSyULkVxIpfj612U8vEc5/3iuhHcmp7htqzI+P7ecQceV\n0LVrcnFlGh5Fmi9VQ4kUmhptEUuWwIv/TPF/903imreG0K1baJIYMgT69Yt5uBH162ixlCxEisiq\nVfDaayGfjBsX2jl++MNwp9WgQeFOq7xTv44WSclCpIjNmwfPPhuWCRNg++3D4IY//CEccAC0a5fj\nC6pk0WIpWYg0E6tWwZQpoRPg+PHw1luhmuqQQ8Ky997QqlUTLqA2ixZNyUKkmVqyBCZOhBdfDKWO\nuXNh4MAwA+DBB8Oee8JGDbmHUf06WjQlC5EW4vPPoaICXnopLJ9/HqqqDjoodAjs3buJJQ9p1pQs\nRFqoBQvg5ZdDAnnlFfj441BtNXBgSCL9+mkaWVlHyUKkOWtA1dHixWHq2EmTwjJ9epinY//9w7Lf\nfqEB3XLydSHFRslCpDlrQqP08uWhZ/m//x2Sx+TJoY2jf/+w9OsHffrAZpvF9FokUUoWIs1djm53\ndQ9VVZMnw+uvh+Wtt8KcHX37wr77hmX33cOIu9K8KFmItAR56kj37bchYUyZEpY33giX2GOPUOro\n0wf22Qd69lTjebFTshBp7mLuSPfVV6H6aurUdcsnn8Buu4W+Hr17h2X33cN85gVHtwjXSslCpDlL\noiNdLV+2X81L8clDk3ihzRBmzAiN5++9Fwo5vXrBXnuFZc89w5AliTaiZ/OetcCEkstkgbsX/BLC\nFGkhxo51r6paf19VVdjfmOOyUVXlPnLkuvPV3I6sWOE+Y4b7vfe6n3ee+yGHuG+zjfvWW7uXlrr/\n6lfuf/2r++TJ7l9+2fAwmqQ65srKWmPP9jU2J9F3Z06+h1WyEClWuS6BNLLqyz30/5g5c90yaxa8\n8w5su22oyurZc93fXXaBzTdvxOvNRqZ2nhY2TpaqoUQkyPWXXy4a1aPqntUdSqishNmz4f+mpljz\nyiQeXDKS/CHNAAAMOklEQVSE996DrbYK/UF22WXd8v3vQ+fOTajOyva9yNWNA0VQraVqKBFZp7LS\nHcLfpshUjdPQ89RR3bN6tftHH7mPG+d+/fXuZ5zhPnCg+3e+477ZZu577+1+/PHul13mPuE3Y33K\n81W+aJH7mjVp569Z1ZZtFVM2rzHb6r0iqNYih9VQiSeCrIJUshCpXUxf8Gs19Iu0gXFVVbm//rr7\n/fe7X365+8+HVfkj3xnp229e5Vts4X7gnlX+7PdG+uhfV/kdd7iPH+/+4YfuK5/MIq6GJpRskkCu\n3v88UbIQkdz+ss3Hr+lMJZ4GXHPNWSN98bRKXzBspD/61yq/6ir34cNDo/oOO7hvsol7t27uBx7o\nfuKJ7mVl7n/5i/szz7jPmuW+ZEkDrpf+urJJArkq2eVBLpOF2ixEilVSdebZtA005JhsGugztDOs\nXBkmlZozJ/RY/+ST8Hfu3LA+d26YyrZrV+jSJSydO4dbfquX7baDjh3DQIxmma+Z9etMkBq4RSRZ\n9X2RNiQJ5CrxZOAeTjN/fkgqn34Kn30WthcsCMtnn8HCheH4HbdJcdnyMsb1HMWJC67jxYPL2bRz\nCVtvDVtuGS6/TasU37u7jOWXldOhWwntV6awSxt5N1qeEn9RJAszawtMBNoAmwD/cvff1nLczcBg\nYBlwqrtPr+UYJQuRQpHpy7uhX3y5Sjw58vX8FKsuLuODU8pZuKKEJR+n6PlAGU/1L2f+1yVUVYUw\nelaO45U1A5j/dQlLloRBHDtvmqK09SQmbzWEdu1Cb/e2baFNm7BsskkYQqVVq1DSMQtL+5Upjp5a\nxqO9ylm6cQmtv05x/Ftl3LNTOf9dXcLy5fDNN6Gn/ddfhyFZxozJ/FqKIlkAmNmm7r7MzFoBrwIX\nuPuraY8fDpzt7oebWT/gJnfvX8t5lCxECkHcfTuSqGpr5DVXr4Zly2Dp0vD3m2/C3+XLYcWKsKxc\nGabSrV5CY0dY2nyTYu9/lvHu0FHs/sx1vHtSeC/atQsJp23bMFpw+/ahn0o2fVWKJlmsvYjZpoRS\nxinu/nba/tuBl9z9H9H2u8BB7r6wxvOVLEQKQS6/vFvC/OC5LGU1Qi6TRUNm820wM9vIzGYACwlJ\n4e0ah3QB5qZtzwO65jMmEWmCIUM2/CIvKWncr/xJk9ZPDCUlYXvSpKbHWSgGDAgJMJUK29UJccCA\nDY9NpULpqrIy/K1+ToHIa7Jw9zXu3ouQAA40s9JaDquZ9VSEEGkJcpl4ClV1AiwrC6WGTA395eWh\nRFH9nAJKGLGMVu/uS8xsHLAPUJH20HygW9p212jfBkaPHr12vbS0lNLS0lyHKSKSeyUloU2munqp\ntiq2+kpZDUieFRUVVFRU5CbuGvJ5N9Q2wCp3T5lZO+A54Ep3n5B2THoDd3/gRjVwi0izkmBfjGJp\ns+gEvBi1WbwOPOXuE8xshJmNAHD3p4GPzOxD4A5gZB7jERGJVxFUL2VLnfJERPIl4ZFpi+7W2aZS\nshARabhiqYYSEZFmQslCREQyUrIQEZGMlCxERCQjJQsREclIyUJERDJSshARkYyULEREJCMlCxER\nyUjJQkREMlKyEBGRjJQsREQkIyULERHJSMlCREQyUrIQEZGMlCxERCQjJQsREclIyUJERDJSshAR\nkYyULEREJCMlCxERyUjJQkREMlKyEBGRjJQsREQkIyULERHJSMlCREQyUrIQEZGM8poszKybmb1k\nZrPNbJaZnVPLMaVmtsTMpkfLpfmMSUREGi7fJYuVwHnuvhvQH/ilme1ay3ET3b13tPw+zzHFrqKi\nIukQmqSY4y/m2EHxJ63Y48+lvCYLd1/g7jOi9a+Ad4DOtRxq+YwjacX+H66Y4y/m2EHxJ63Y48+l\n2NoszKw70Bt4vcZDDuxvZm+a2dNm1jOumEREJDut4riImW0GPAacG5Uw0k0Durn7MjMbDDwJ7BxH\nXCIikh1z9/xewKw1MBZ4xt1vzOL4SqCPuy9O25ffIEVEmil3z0k1f15LFmZmwF3A23UlCjPrCHzu\n7m5mfQkJbHH6Mbl6sSIi0jj5roYaAJwIvGVm06N9lwDbA7j7HcAxwFlmtgpYBvwkzzGJiEgD5b0a\nSkREil8iPbjN7G4zW2hmM9P29TWzKVHHvP+Y2b7R/rZm9pCZvWVmb5vZxWnP6WNmM83sAzO7KeH4\n9zKzyVGcY8ysQ9pjv41ifNfMBhVT/Gb2QzN7I9r/hpn9oJjiT3t8ezP7yszOL7b4zWzP6LFZ0eOb\nJBV/A//vFOJnt9aOwma2lZmNN7P3zex5MytJe07BfH4bGn9OP7/uHvsCDCTcRjszbV8FcGi0Phh4\nKVo/FXgoWm8HVALbR9tTgL7R+tPAYQnG/x9gYLR+GvC7aL0nMANoDXQHPmRdia4Y4u8FbBet7wbM\nS3tOwcef9vhjwD+A84spfkJV8ZvAHtH2lsBGScXfwNgL8bO7HdArWt8MeA/YFfhf4MJo/0XAtdF6\nQX1+GxF/zj6/iZQs3P0VoKrG7s+ALaL1EmB+2v72ZrYx0B74FvjSzDoBHdx9SnTcfcBReQ08Ukf8\nO0X7AV4AhkXrPyJ8YFa6+xzCf7Z+xRK/u89w9wXR/reBdmbWuljiBzCzo4CPCPFX7yuW+AcBb7n7\nzOi5Ve6+Jqn4Gxh7IX52a+so3AU4Erg3OuzetHgK6vPb0Phz+fktpIEELwb+ZGafANcRGsJx9+eA\nLwn/8eYA17l7ivAGzUt7/vxoX1Jmm9mPovVjgW7RemfWj3MeIc6a+ws1/nTDgKnuvpIief8t9PG5\nEBhd4/iiiJ/Q58jN7Fkzm2pmo6L9hRR/rbEX+mfX1u8o3NHdF0YPLQQ6RusF+/nNMv50Tfr8FlKy\nuAs4x923B86LtjGzEwlF2E5AD+ACM+uRWJR1Gw6MNLM3CMXDbxOOp6Hqjd/MdgOuBUYkEFs26op/\nNHCDuy+jsIeVqSv+VsABwAnR3x+b2cGEkQ8KRa2xF/JnN/oR8Tiho/DS9Mc81MsU0vu7gYbGn4vP\nbyw9uLPU193/J1p/DLgzWt8feMLdVwOLzGwS0Ad4Feia9vyurKu6ip27vwccCmBmOwNDoofms/6v\n9K6EjD6f4ogfM+sK/BM4yd0ro92FHv/h0UN9gWFm9r+E6s01ZvYN4fUUcvzV7/9c4GWP+h6Z2dPA\n3sD9FEj89bz3BfnZtdBR+HHg7+7+ZLR7oZlt5+4Loiqaz6P9Bff5bWD8Ofv8FlLJ4kMzOyhaPxh4\nP1p/N9rGzNoTRq99N6qH+9LM+pmZAScRhgpJhJltG/3dCLgUuC16aAzwEzPbJPpVtRMwpVjij+6q\nGAdc5O6Tq493988o7Phvj+I80N17uHsP4Eag3N3/XCzvP/AcsIeZtTOzVsBBwOxCir+u954C/OxG\n16uto/AY4JRo/ZS0eArq89vQ+HP6+c13630dLfoPAZ8SiqtzCXdQ7EOoe5sBTAZ6R8e2IfyKmgnM\nZv27WfpE+z8Ebk4w/uHAOYQ7E94Drq5x/CVRjO8S3fFVLPETPvxfAdPTlm2KJf4az7sC+E0xvf/R\n8T8DZkWxXptk/A38v1OIn90DgDXR90z1/+fDgK0IjfPvA88DJWnPKZjPb0Pjz+XnV53yREQko0Kq\nhhIRkQKlZCEiIhkpWYiISEZKFiIikpGShYiIZKRkISIiGSlZiIhIRkoWIjGIejeLFC39Bxapwcyu\nNLNz07bLzewcMxtlYYKuN81sdNrjT0QTy8wys9PT9n9lZn80sxmEoS5EipaShciG7gZOhrUlguOB\nBcCO7t6XMCx0HzMbGB0/3N33AfYFzjGzLaP9mwKvuXsvd/93rK9AJMcKadRZkYLg7h+b2X/NrBdh\nZrLphEQwyMymR4e1B3YEXgHOjSZYgjBC6U6EWchWE0YHFSl6ShYitbuTMMBlR0JJ4xDgGnf/S/pB\nZlYaPdbf3Zeb2UtA2+jh5a7B16SZUDWUSO2eIIzmuQ/wLGGo8OHRUNuYWZdoaO7NgaooUeyC2iak\nmVLJQqQW7r7SzF4kJAIHxpvZrsDkMPw/S4ETCYnkTDN7mzBE9+T008QctkjeaIhykVpEDdtTgWPc\n/f+SjkckaaqGEqnBzHoCHwAvKFGIBCpZiIhIRipZiIhIRkoWIiKSkZKFiIhkpGQhIiIZKVmIiEhG\nShYiIpLR/wc2nF4QOWe5fQAAAABJRU5ErkJggg==\n", "text": [ "" ] } ], "prompt_number": 31 }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Solution with QR Decomposition\n", "\n", "Performing a solve instead of a matrix inverse is the more numerically stable approach, but we can do even better. A [QR-decomposition](http://en.wikipedia.org/wiki/QR_decomposition) of a matrix factorises it into a matrix which is an orthogonal matrix $\\mathbf{Q}$, so that $\\mathbf{Q}^\\top \\mathbf{Q} = \\mathbf{I}$. And a matrix which is upper triangular, $\\mathbf{R}$. \n", "$$\n", "\\boldsymbol{\\Phi}^\\top \\boldsymbol{\\Phi} \\mathbf{w} = \\boldsymbol{\\Phi}^\\top \\mathbf{y}\n", "$$\n", "$$\n", "(\\mathbf{Q}\\mathbf{R})^\\top (\\mathbf{Q}\\mathbf{R})\\mathbf{w} = (\\mathbf{Q}\\mathbf{R})^\\top \\mathbf{y}\n", "$$\n", "$$\n", "\\mathbf{R}^\\top (\\mathbf{Q}^\\top \\mathbf{Q}) \\mathbf{R} \\mathbf{w} = \\mathbf{R}^\\top \\mathbf{Q}^\\top \\mathbf{y}\n", "$$\n", "$$\n", "\\mathbf{R}^\\top \\mathbf{R} \\mathbf{w} = \\mathbf{R}^\\top \\mathbf{Q}^\\top \\mathbf{y}\n", "$$\n", "$$\n", "\\mathbf{R} \\mathbf{w} = \\mathbf{Q}^\\top \\mathbf{y}\n", "$$\n", "This is a more numerically stable solution because it removes the need to compute $\\boldsymbol{\\Phi}^\\top\\boldsymbol{\\Phi}$ as an intermediate. Computing $\\boldsymbol{\\Phi}^\\top\\boldsymbol{\\Phi}$ is a bad idea because it involves squaring all the elements of $\\boldsymbol{\\Phi}$ and thereby potentially reducing the numerical precision with which we can represent the solution. Operating on $\\boldsymbol{\\Phi}$ directly preserves the numerical precision of the model.\n", "\n", "This can be more particularly seen when we begin to work with *basis functions* in the next session. Some systems that can be resolved with the QR decomposition can not be resolved by using solve directly." ] }, { "cell_type": "code", "collapsed": false, "input": [ "import scipy as sp\n", "Q, R = np.linalg.qr(X)\n", "w = sp.linalg.solve_triangular(R, np.dot(Q.T, y)) \n", "w" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 32, "text": [ "array([[ 2.88952457e+01],\n", " [ -1.29806477e-02]])" ] } ], "prompt_number": 32 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Putting it Together\n", "\n", "Now we look at how we can fit different polynomial basis sets to the olympic data and compute the training error." ] }, { "cell_type": "code", "collapsed": false, "input": [ "# import the time module to allow python to pause.\n", "import time\n", "# import the IPython display module to clear the output.\n", "from IPython.display import clear_output, display\n", "\n", "def polynomial(x, degree):\n", " degrees = np.arange(degree+1)\n", " return x**degrees\n", "\n", "num_data = len(x)\n", "error_list = []\n", "max_degree = 6\n", "sigma2 = 1\n", "fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(12,6))\n", "for degree in range(0, max_degree+1):\n", "\n", " # 1. build the basis set\n", " Phi = polynomial(x, degree)\n", " Phi_pred = polynomial(x_pred, degree)\n", " \n", " # 2. solve the linear system with QR decomposition\n", " Q, R = np.linalg.qr(Phi)\n", " w = sp.linalg.solve_triangular(R, np.dot(Q.T, y)) \n", "\n", " # 3. make predictions at training and test points\n", " f_pred = np.dot(Phi_pred, w)\n", " f = np.dot(Phi, w)\n", "\n", " # 4. compute the error and append it to a list.\n", " error_list.append(((y-f)**2).sum() + num_data/2.*np.log(sigma2))\n", "\n", " # 5. plot the predictions\n", " axes[0].clear()\n", " axes[1].clear() \n", " axes[0].plot(x_pred, f_pred, linewidth=2)\n", " axes[0].plot(x, y, 'rx', markersize=10, linewidth=2)\n", " axes[0].set_ylim((2.5, 5.5))\n", " axes[0].set_title('Predictions for Degree ' + str(degree) + ' model.')\n", " axes[1].plot(np.arange(0, degree+1), np.asarray(error_list), linewidth=2)\n", " axes[1].set_xlim((0, max_degree))\n", " axes[1].set_ylim((0, 10))\n", " axes[1].set_title('Training Error')\n", " display(fig)\n", " time.sleep(1)\n", " clear_output()" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "display_data", "png": "iVBORw0KGgoAAAANSUhEUgAAAsEAAAF6CAYAAAD4VIVfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XmcU/X1//HXYZFVGBEFFAQU97pRBRSVqVutaK2t/aq1\nWq1aFLtYLXbBiv7saFu02rqi1bbWVtS6i1rXsYq7IK64sYiAojBhEVBgzu+PTwJhyDYzSW6W9/Px\nyCM3uTf3nhuGm5NPzufzMXdHRERERKSatIk6ABERERGRYlMSLCIiIiJVR0mwiIiIiFQdJcEiIiIi\nUnWUBIuIiIhI1VESLCIiIiJVR0lwmTGzv5vZRfHl/cxsegv3c62ZnZff6LIe8ygzm2NmS81st2Ie\nW9Izs5PM7Okct1379yci1cHMHjSzE/K9rUjUlAQXgJnNMrPl8WTvYzP7m5l1ydPuPX7D3Z929x1y\niGeDJMfdz3D33+UpplxdCox2943dfVprd2Zm9Wa2wsyWmNliM3vZzH5pZhvlIdaCM7O2ZvY7M5sb\nP4cpZtY96riyWPv3JyKly8yWxT+DlppZY9Jn0lIzO645+3L3w9z9n/netjnMrDZ+Hkub3Ibm+1hS\nPZQEF4YDh7v7xsBgYE9gg1ZXM2vXwv1bK2KLhJkZsBXwVgtfn+pv1YEz3b0b0Bs4BzgWeLClcWaJ\noW2ed3khMAwYFj+H7wMr83yMQii7vz+RauPuXeMNDhsDs4l/JsVvtya2a8XnUBTmJp1D4vZC040s\nrslzzTrPMntfpIWUBBeYu88DHgZ2Boh/kx1tZu8B78SfO9zMXjWzBjObbGa7JF5vZnvEWwiXmNlE\noGPSulozm5P0uJ+Z3WVmC8zsMzO70sx2AK4D9o5/a14U33a9n7XN7DQze8/MFprZvWbWJ2ldo5mN\nMrN34zFelbRukJk9ZWYxM/s0HuN6zKwDsBRoC0yLnztmtmO8NbfBzN4wsyOSXvP3eMnGg2a2DKhN\n8xZb/H1e4e5PAd+Mn+vI+H7MzH5lZu/H35PbzGyTpOOcaGaz4+vOi7fiHxBfd4GZ/cfM/mlmi4Ef\nmFl3M7vRzOaZ2UdmdlFygm5mPzSzt8xskZk9bGZbpQw6xPAz4DR3nxM/h7fc/Ys02//dzK6Jvx9L\nzexpM+ttZn+Ov39vm9nuSdtnem83NbP7LLSevwBs0+RYO5jZo/G/helm9t00772IlJn458ZHZnau\nmc0HbjSzGjN7IP7ZscjM7jezLZNeU29mp8SXTzKzZ8xsfHzbGWZ2aAu3HWhm/4t/vj1qZlebWYta\nkePH/Z2ZTQaWAVtb6s/bbJ91620vlU1JcOEYhMQU+AYwNWndkcBewE5mtgdwI3Aa0AOYANxnZu0t\n/Kx/D/APYBPgDuA7pPg52kIr5QPATKA/sCVwq7tPB0YBz8W/NfeIv2Ttz9rxpO9i4LtAH0KrQdNk\ndiShRXtX4P/M7JD48xcBD7t7TfyYf2kam7t/4e5d4w93dfdtzaw9cD/hC8JmwE+Af5nZdkkvPQ64\nKP7ayU33m3QeyceaA7wM7Bd/6qeExHj/+Lk1AFfHz3un+PJx8XXdgS2a7P+bwB3u3h34N/B34EtC\n4rgHcAhwanx/RwK/Bo4CegJPA7eS2i7AauC7ZjbfzN4xs9Fptk34LjA2vu8vgeeBlwh/N/8B/hSP\nI9t7ezWwnNB6/kPgZNb9LXQBHgVuib/2WOAaM9sxS2wiUj56ET5TtiJ8PrQhfA5tFb+tAK5K2r5p\nGdQQYDqwKfDH+Gtbsu2/CdexHsAFhF/DWlNu9X3C9Xhj4MP4c8mft7l81q3dvhVxSJlQElwYBtxj\nZg2ERKie8B8v4RJ3j8Vb/X4ETHD3lzy4GfgC2JvwU3k7d/+zu69x9zsJSU8qQwj/qcfEW0W/cPdn\nk+LJ5HjgRnd/1d2/JCRyezdpxfy9uy+JJ5lPAolWxy+BAWa2pbt/mXTMbIYBXdz99+6+2t2fJCTx\nybVq97j7cxAS6Rz3CzCPcIEHOB04z93nufsqQgnC0fEvDUcD97n7s/F157PhBfhZd78vvtyd8IXm\n5/H3+FPgCkKimDjWJe7+jrs3ApcAu8e/CDXVN76/bYEB8VguMLOD0pyTA3e5+9T4e3E38Lm73+Lu\nDtxOSMohw3sbP+9vA+fHz+FNwpesxN/I4cBMd/+Huze6+6vAXYQPDRGpDI3AOHdf5e4r3X2Ru98d\nX15G+LwakeH1s939xvi152agj5lt3pxt458vexKuRavdfTJwH5k/r7aI/7qVuC0ys07xdQ783d3f\njl+7VsWfT/68zeWzLnl7qXBKggvDgSPdfRN3H+DuP27yH2pO0nJ/4Jzk/9iEBKkPoVVybpN9z05z\nzH6Ei01jC+JNfCMOwbt/DiwktOwmfJy0vJzwTRvgXMJF68X4z+4n53jMLVj/fSAeQ6Il1lOsz1Vf\nYFF8uT9wd9J7+xahBbYX4bw/SrzI3VcQzjvZR0nL/YH2wPyk/V1HaDFNrP9z0rrEvpLfx4QV8fv/\nF//C8jqhReKwDOe1IGl5ZZPHK4BEa3um97Yn0K7J+g+TlvsDQ5v8PX6P8H6JSGX4NJ4EAmBmnc1s\ngoVysMXAU0B3M0uXkK79PHD35fHFrs3cdgtgkbsn94PIds2fF/9cTdx6xK/bmV6f/Fwun3Ut/dyR\nMqTC72gktzZ+CNS5+8VNNzKzEWyYQPUH3k+xzznAVmbW1t3XZDheKvMIrZGJ43Yh/HTVNAHfgLt/\nQmjNxsyGA4+Z2VPuPiOHY/YzM4u3EEA4txYN+ZYQb3UdTGiFhfD+npxoUW6y7Xxg+6THnQjnnSz5\nvZtDaKXfNM2XjQ8J5RvpSiCSvZbm+XyMvJDpvf2U8CVgK9bVvCW3gnwIPOXuhyAilarpdeYcYDtg\niLsvsNC/YAqhgaNQo8HMB3qYWaekRHarVh4v1WuTn8vls06j31QRtQRH7wbgdDMbYkEXMxtpZl2B\nZ4HVZvbTeI3wtwm1Sqm8SLio/D7+rb6jme0TX/cJ0DdeK5pgrPvZ6VbgZDPbzUIntouB5909uYWQ\nJq8NC2bfNbO+8YcxwgUkl9bo5wktyufGz62W8FN8oj4r1xEIErXXneNfGu4FXnD3xAgR1wEXJ37u\nMrPNzOyb8XX/AY4ws73j9dcXZDquu88HHgH+ZGYbm1kbM9vGzPZPOtZv4rXGWOhEl7KMwN0/IJTK\njDWzjeI1t8cQyhbSnmeOXiDNextP3u8ilF50isf6A9Zd+CcB25nZ9+OvbW9me1noYNncOESkPHQl\n/Jq02Mx6AOMKfUB3n03ov3FB/DqzN+E61ZokNNv1qbmfdVLhlAQXX9OOXK8QOsVdRfgJ/z3gxPi6\nVYT6zZMIP9n8H3Bnqv3FW3+PAAYRWvPmxLcHeBx4E/jYzBYkvS7x2seB38b3PQ8YyLo61w1iTn4t\noabreTNbSkhAf+rus7Kde/zcjiDU2H4aP/8T3P3dFMfI5CozW0L4ye1yQufBQ5PW/5lQZ/ZIfLvn\nCPXTuPtbhE5jE+PnvZRQYpAoXUkVw4nARoSyikXx4/WO7+8e4A/AxPhPiq8DX88Q+3GEFtqFhOT3\nvHj9bipNY0kVW+Lf80syv7c/JnzofQzcFL8Rf+1SQme/YwmtI/MJreqJsZeTO1RuZWGkisSXIBEp\nD02vHVcAnYDPCI0vD6XYJvm1Ka89Ldj2eEL/l4WETta3EfqZpNvXFrbhOMFHZYij6edtcz/rpMLZ\nul9LM2xkNgtYAqwBVrn7kCbrawkJUOIn8Du9+BMxiLRKvPW9ARgUb6UQKWtmdhNhZJcF7r5L/Lke\nhGSjPzAL+D93j0UWpEicmd0GvOXuF0Ydi1SHXFuCHah19z2aJsBJnoqv30MJsJQLMzsiXkrRhTCj\n3WtKgKWC/I31fxkB+BXwqLtvR/iV6FdFj0oEMLM94yVlbczsG4QhKe+JOi6pHs0ph8hWa6NaQSlH\n3yT87D+XMPbvsZk3Fykf7v404deNZN8kDItH/P5bRQ1KZJ3ehCE3lxLK2U5392nRhiTVJNdyiBnA\nYkI5xAR3v6HJ+hGEzjYfEZKJX8TrLUVEJEJmNgC4P6kcosHdN4kvG2GYqk3S70FEpDLlOkTacHef\nb2abAY+a2fR4C0PCFKCfuy+P/6RxD2G4FRERKVHu7mamzkAiUpVySoLjQ0Ph7p+a2d2E3vVPJ61f\nmrT8kJldY2Y93D0xYQG60IpIOXP3Sin5+sTMerv7x2bWh/UnXVlL12wRKXfZrttZa4LjnYY2ji93\nIQyf9HqTbXolZpYxsyGEMotFTffl7mVxGzduXOQxVGq85RRrucVbTrGWW7wV5j7C2NDE79N2RIr6\nfa/Gv7VqOZdKOQ+dS+necpFLS3AvwrSzie3/5e6PmNmo+EVyAnA0cIaZrSYM0q/ORSIiETOzW4ER\nQE8zmwOcD/weuN3MTiE+RFp0EYqIRCdrEuzuM4HdUzw/IWn5auDq/IYmIiKt4e7HpVl1UFEDEREp\nQZoxLoXa2tqoQ2iWcoq3nGKF8oq3nGKF8otXylcl/a1VyrlUynmAzqWc5TREWl4OZObFOpaISD6Z\nGV45HeNyomu2iJSzXK7bagkWERERkaqjJFhEREREqo6SYBERERGpOkqCRURERKTqKAkWERERkaqj\nJFhEREREqo6SYBERERGpOkqCRURERKTqKAkWERERkaqjJFhEREREqo6SYBERERGpOkqCRURERKTq\nKAkWERERkaqjJFhEREREqo6SYBERERGpOkqCRURERKTqKAkWERERkaqjJFhEREREqo6SYBERERGp\nOkqCRURERKTqKAnOxaRJEItl3iYWC9uJiIiISMlTEpyL4cNh7Nj0iXAsFtYPH17cuERERESkRZQE\n56KmBurqUifCiQS4ri5sJyIiIiIlT0lwrlIlwkqARURERMqSuXtxDmTmxTpWQSUS3zFjYPx4JcAi\nVcDMcHeLOo5iqphrtohUpVyu20qCW2LWLBg4EGbOhAEDoo5GRApMSbCISHnJ5bqtcojmisVCC/DM\nmeE+26gRIiIiIlJylAQ3R3IN8IAB6TvLiYiIiEhJUxKcq1Sd4DKNGiEiIiIiJUtJcC4yjQKhRFhE\nRESk7KhjXC4mTQoTYWQaBSIWg8mTYeTI4sUlIkWhjnEiIuVFo0OIiOSBkmARkfKi0SFERERERFJQ\nEiwiIiIiVUdJsIiIiIhUHSXBIiIiIlJ1lASLiIiISNVREiwiIiIiVUdJsIiIiIhUHSXBIiIiIlJ1\nlASLiIiISNVREiwiIiIiVUdJsIiIiIhUHSXBIiIiIlJ1lASLiIiISNXJKQk2s1lm9pqZTTWzF9Ns\n8xcze8/MppnZHvkNU0REREQkf9rluJ0Dte6+KNVKMzsMGOTu25rZUOBaYFieYhQRERERyavmlENY\nhnXfBP4B4O4vADVm1qs1gYmIiIiIFEquSbADj5nZy2Z2Wor1WwJzkh5/BPRtbXAiIiIiIoWQaznE\ncHefb2abAY+a2XR3f7rJNk1bir314YmIiIiI5F9OSbC7z4/ff2pmdwNDgOQkeC7QL+lx3/hz67ng\nggvWLtfW1lJbW9vsgEVECq2+vp76+vqowxARkQIy98wNtmbWGWjr7kvNrAvwCHChuz+StM1hwI/d\n/TAzGwZc4e7DmuzHsx1LRKQUmRnunqlfRMXRNVtEylku1+1cWoJ7AXebWWL7f7n7I2Y2CsDdJ7j7\ng2Z2mJm9D3wOnNzK2EVERERECiZrS3DeDqRWBREpU2oJFhEpL7lctzVjnIiIiIhUHSXBIiIiIlJ1\nlASLiFQhM/u1mb1pZq+b2b/NrEPUMYmIFJOSYBGRKmNmA4DTgMHuvgvQFjg2yphERIot18kyRESk\nciwBVgGdzWwN0JkUY7uLiFQytQSLiFQZd18EXAZ8CMwDYu7+WLRRiYgUl5JgEZEqY2bbAGcBA4At\ngK5mdnykQYmIFJnKIUREqs+ewLPuvhDAzO4C9gH+lbyRproXkXLRkunuNVmGiEgWlTZZhpntRkh4\n9wJWAn8HXnT3q5O20TVbRMqWJssQEZENuPs04GbgZeC1+NPXRxeRiEjxqSVYRCSLSmsJzoWu2SJS\nztQSLCIiIiKSgpJgEREREak6SoJFREREpOooCRYRERGRqqMkWEREUlqwIOoIREQKR0mwiIik9MQT\nUUcgIlI4SoJFRCSlxx+POgIRkcJREiwiIik9+ihoqGARqVRKgkVEJKXZs2HGjKijEBEpDCXBIiKS\nlkoiRKRSKQkWEZG0Hnss6ghERArDijU3vOahF5Fylcsc9JXGzBycTTcNQ6W1UZOJiJSRXK7buqyJ\niEhKW20FCxfCtGlRRyIikn9KgkVEJKUDDwz3qgsWkUqkJFhERFI66KBwr7pgEalEqgkWEcmiWmuC\n5893+vSBzp1h0SLo0CHqqEREcqOaYBERabHeveErX4Hly+H556OORkQkv5QEi4hIWqoLFpFKpSRY\nRETSUl2wiFQq1QSLiGRRrTXB7s6SJdCjR3hu0SLo1i3auEREcqGaYBERaZVu3WDoUFizBp56Kupo\nRETyR0mwiIhkpJIIEalESoJFRCQjdY4TkUqkmmARkSyquSYY4MsvYZNNwlBp8+ZBnz4RBycikoVq\ngkVEpNU22gj23z8sP/FEtLGIiOSLkuC4L76Ajz4KnT9ERGR9qgsWkUpT9UnwmjXw179C//7Qrx90\n6gSDBsHZZ4ef/kREZP26YFW2iUglqOqa4NdegxNPhGnTwuPu3WHx4nXrd9oJbr0Vdt01mvhEpDRU\ne00wQGMj9OoFn30G77wD220XYXAiIlmoJjiDhgY44oiQAPfrF5LdhobQ+vv007DDDvDWWzBkCEyc\nGHW0IiLRatMGDjggLKskQkQqQVUmwe5w+unw4Yew114wfToceyyYhXKIffeFl1+G004LtcI/+AFM\nnhx11CIi0UrUBWuoNBGpBFVZDnHTTXDKKdC1K0ydGmqA0/npT+HKK2HzzeHFF0PtsIhUF5VDBDNn\nwtZbQ01NKIto2zai4EREslA5RArvvw8/+UlYvvrqzAkwwJ/+FFo/FiyAI4+EZcsKH6OISCkaODDc\nYrHQgCAiUs6qLgn+wx9C3e8xx8AJJ2Tfvl07uP122HbbUD/8m98UPkYRkVKlodJEpFJUVRK8cCHc\ncktY/n//L9QA52KTTeCOO8JPf1ddBS+9VLgYRURKmaZQFpFKUVVJ8I03wsqVcOihzR/eZ7fdwtjB\n7jBqFKxeXZgYRURKWWKEiGeeCddTEZFyVTVJ8OrVoQYY1tUEN9e4caFj3NSpoUVYRKTabLZZaBRY\nuRKefTbqaEREWi6nJNjM2prZVDO7P8W6WjNbHF8/1czOy3+YrXf//WFItEGDQktwS3Tpsi75/e1v\nYd68/MUnIlIuVBcsIpUg15bgnwFvAenGOHvK3feI336Xn9Dy68orw/2PfxwGfW+pww9fN0pEXV1+\nYhMRKSeqCxaRSpA1HTSzvsBhwF+BdF3JSnr8zHffhSefDC25J53U+v3V1YVOdTfcALNmtX5/IiLl\nZL/9oH37MKlQQ0PU0YiItEwubaKXA2OAxjTrHdjHzKaZ2YNmtlPeosuHSZN45PYYAN/6FnTvnmKb\nWAwmTcp5lzvvDMcfD6tWwYUX5ilOEZEy0bUrDBsGjY1QXx91NCIiLZMxCTazw4EF7j6V9K29U4B+\n7r4bcCVwT35DbKXhw+l73Vi6E2PkyBTrb7sNfvELGD48/T5SJMkXXBDGEL755jDtsohINdEUyiJS\n7tplWb8P8E0zOwzoCHQzs5vd/cTEBu6+NGn5ITO7xsx6uPuipju74IIL1i7X1tZSW1vbyvCzW9Km\nhlM/ruNixnLosDqgZt3KWAwefTTzDmIxGDt2gwLgbbYJUy9PmADnnx8m1BCRylBfX0+9mjgzOuig\nMGKOOseJSLmypnPDp93QbATwC3c/osnzvQitxW5mQ4Db3X1AitdvMA99Mdx5Jxx9NHx9aIyHvxpP\nZmtqNkxuxyatS0jeJvn5uI8+CqNNfPklvPUW7LBDkU5KRIoqlznoK022a/aqVdCjR+gkPGcO9O1b\nxOBERLLI5brd3HESPL7jUWY2Kv7c0cDrZvYqcAVwbLMjLaAHHwz3I46sCcns2LGhN1tycluTtC4W\n6oezJcAQLvo/+EGYQOOyy4pzPiIipaB9e0j8mKeSCBEpRzm3BLf6QBG0BDc2wpZbwscfw7RpsOuu\nhAR44ECYORMGDFj/BYnEd8wYGD8+YwKc8O67oQW4ffuw6z59CnQyIhIZtQSndsUV8POfwwknhP4R\nIiKlohAtwWVl6tSQAPftC7vsQkhyx48PCfD48etafRNqakICPHBguM+SAEOYfvmoo0JJxF/+Upjz\nEBEpRcmTZkRQ7SYi0ioVnQQnSiEOOwxscVJ5w4ABG5Y/QPYkOY1zzw33114LS5bk9xxERErVzjtD\nr14wf75GyRGR8lPRSXBiVLMjR6So721aBxzLIUlOY+hQ2H9/WLwYrr++cOcjIlJKzNbNHqdRIkSk\n3FRsTfCyZWFijBpizD9lLBv9MU19bywWxgkGuPTSZo0OkWzSpDClcr9+MGNGGENYRCqDaoLTu+mm\nMFzkkUfCPaU1SryIVLGqrgl+5ZXQMe74AZPTJ8DJDj54w21SjRqRxje+AdtuG4YKuu++VgYvIlIm\nEnXBTz4Jq1dHG4uISHNUbBL84ovhftUhIzMnwJMnhxbgY45JvT6RCE+enPF4bdrAj38clq+8sgUB\ni4iUoa22Cg0AS5bAyy9HHY2ISO4qNgl+4YVwP3Rolg1HZkmSIaxPOefy+k46Cbp2hfp6eO21XKIU\nESl/ibpgjRcsIuWkYpPgREvwkCHFO2a3bmHyDICrrirecUVEopQ8VJqISLmoyI5x8+fDFluEpLSh\nIZQqFMv06bDjjtCpU5hWuUeP4h1bRApDHeMyW7QIevYMkwY1NEDnzgUOTkQki6rtGJcohdhrr+Im\nwBBmjzvkEFixIvSaFhGpdD16wODBYdKgZ56JOhoRkdxUZBIcRSlEsjPPDPcTJoQRKkREKp3qgkWk\n3FRkEpxzp7gCGTkyjBf8/vvwxBPRxCAiUkyqCxaRclNxSfCaNfDSS2E5qpbgtm3htNPC8nXXRROD\niEgxDR8OG20EU6fCwoVRRyMikl3FJcHvvANLl4aW2D59oovjlFNCMnzvvaGjnohIKTGzGjP7j5m9\nbWZvmdmw1uyvc+eQCLuHiTNEREpdxSXBUZdCJGyxRZhGdPVqdZATkZL0Z+BBd98R2BV4u7U7TNQF\nqyRCRMpBxSXBiU5xUSfBAKefHu6vvz6UaYiIlAIz6w7s5+43Abj7andf3Nr9JuqC1TlORMpBxSXB\nr74a7r/61WjjgNAqss028OGH8PDDUUcjIrLWQOBTM/ubmU0xsxvMrNWj+371q9C9e+gUPHt2HqIU\nESmgikqC3eGtt8LyV74SbSwQxihOdJC74YZoYxERSdIOGAxc4+6Dgc+BX7V6p+2gtjYsqzVYREpd\nu6gDyKd582DJkjBz0WabRR1NcNJJcN558MADIb4ttog6IhERPgI+cvf4WDr8hxRJ8AUXXLB2uba2\nltpEhpvBQQeFDsGPPQY//GFeYhURyaq+vp76+vpmvaaipk1+9NEwW9v++8NTTxX0UM1y9NFw553w\nu9/B2LFRRyMizVWJ0yab2f+AU939XTO7AOjk7r9MWt+ia/bbb8NOO8Hmm8PHH4NV1LsmIuWi6qZN\nTpRC7LRTtHE0lSiJuPFGzSAnIiXjJ8C/zGwaYXSIi/Ox0x12CL94LVgAb7yRjz2KiBSGkuAiOPhg\n6N8fZs5UnZyIlAZ3n+bue7n7bu7+7XyMDgGh5VdTKItIOVASXARt2sCpp4ZldZATkUqnKZRFpBxU\nTE2wO2y6KTQ0hA5oUc4Wl8rcubDVVmEWuY8+CvVyIlIeKrEmOJvWXLPnzoW+faFrV1i0CNq3z3Nw\nIiJZVFVN8IIFIQGuqYHevaOOZkNbbgkjR8KqVXDzzVFHIyJSOFtuGWqDly1bN4GRiEipqZgkOLkU\nolR7IydKIv7619ByLSJSqVQSISKlriKT4FJ12GGhTOOdd+CZZ6KORkSkcNQ5TkRKnZLgImrXDk4+\nOSz/9a/RxiIiUki1taFT8HPPhbIIEZFSoyS4yBIzKN1xB8Ri0cYiIlIoNTWw556wejU8/XTU0YiI\nbEhJcL5MmpQ9q43F2Gb6JA48EFasgH/9qzihiYhEQXXBIlLKKiIJ/uyzMDpE165hWJ5IDB8e5kRO\nlwjHYmH98OHrjRmsDnIiUqlUFywipawikuC33w73kY4MUVMDdXWpE+FEAlxXBzU1HHVUGNN42jR4\n+eVowhURKbR99oGOHcO1bsGCqKMREVlfRSTBkZdCJKRKhJskwAAdOsCJJ4bVmkFORCpVx46w775h\n+ckno41FRKSpikiC33033G+/fbRxAOsnwrNmbZAAJ5x2Wri/9Vb1nBaRyqW6YBEpVRWRBM+cGe4H\nDow2jrVqamDMmBDQmDEbJMAAO+4YyoiXLYOJEyOIUUSkCBJ1wUqCRaTUVFQSvPXW0caxViwG48eH\nwMaPT9tZLtEarJIIEalUe+wBm2wSfhibMSPqaERE1in7JNh93YW1JFqCk2uABwxI31kO+O53oXt3\nePFFeO214ocqIlJobdvCAQeEZbUGi0gpKfskuKEBliwJw6NtumnEwaToBJdp1IjOneH73w/L119f\n5FhFRIpEQ6WJSCkq+yQ4uR44suHRIHUCnJAhEU6URNxyCyxfXqRYRUSKKNE57vHHobEx2lhERBIq\nJgmOvB548uTUCXBCIhGePHm9p3fbDYYOhcWL4bbbihCniEiRDRoE/frBwoUq/RKR0lH2SXDJ1AOP\nHJk+AU6oqQnbNTFqVLifMKEAcYmIRMxMQ6WJSOkp+yS45IZHa4Fjjgkd5F54IcysJCJSaVQXLCKl\nRklwCejcGU44ISyrNVhEKlEiCf7f/+DLL6ONRUQElASXjERJxC23aAY5Eak8vXvDzjuHDsDPPx91\nNCIiZZ4k/TLLAAAgAElEQVQENzaGAdih/JPgr3wF9tkHli6Ff/876mhERPJPdcEiUkrKOgmeNy/8\nrLb55tClS9TRtN4ZZ4T7a68Nk4CIiFQSJcEiUkrKOgmulFKIhO9+F3r2hFdfheeeizoaEZH82n//\nMIPciy+GSY5ERKKUUxJsZm3NbKqZ3Z9m/V/M7D0zm2Zme+Q3xPQqLQnu0AFOPTUsX3NNtLGIiORb\nt25hXPQ1a+Cpp6KORkSqXa4twT8D3gI2+JHezA4DBrn7tsCPgGvzF15miTGCI58oI49OPz2MqXnH\nHbBgQdTRiIjkl4ZKE5FSkTUJNrO+wGHAX4FUExN/E/gHgLu/ANSYWa98BplOpbUEA/TvD4cfHmqd\nb7wx6mhERPJLdcEiUipyaQm+HBgDpJvxfUtgTtLjj4C+rYwrJ5WYBAOceWa4v+668LOhiEilGDYs\njI3+5pvw8cdRRyMi1axdppVmdjiwwN2nmlltpk2bPE45tsEFF1ywdrm2tpba2ky7zK5kpkzOs4MP\nhkGD4P334b774Kijoo5IpLrU19dTX18fdRgVaaONQge5hx8OJRHHHx91RCJSrcwzjMVlZhcDJwCr\ngY5AN+BOdz8xaZvrgHp3nxh/PB0Y4e6fNNmXZzpWc33xBXTqBG3awIoV0L593nZdEv78ZzjrLBgx\nAvRZLBItM8PdU5WDVax8X7OTXXopjBkDJ58MN91UkEOISJXL5bqdsRzC3X/j7v3cfSBwLPBEcgIc\ndx9wYvyAw4BY0wS4EGbPDmPp9utXeQkwhA+HjTcOPahffTXqaERE8ie5LlhjootIVJo7TrADmNko\nMxsF4O4PAjPM7H1gAjA6vyGmVqn1wAnduoVEGOAvf4k2FhGRfNp11zAm+pw58N57UUcjItUqYzlE\nXg+U55/WFiyAZ54JM8V9/et5221Jef992G67UEP34YdhZjwRKT6VQ+TfMcfA7beHMdETs2WKiORL\nq8shStnmm8O3v125CTCEznEjR4b65wkToo5GRCR/NFSaiEStbJPgsjRpEsRimbeJxcJ2cT/7Wbi/\n5pqQDIuIVILEpBlPPqmhIEUkGkqCi2n4cBg7Nn0iHIuF9cOHr33qwAND/dzHH8MttxQpThGRAtt6\n69Cno6EBpk6NOhoRqUZKgouppgbq6lInwokEuK4ubBdnFoYSgjCsUGO6KUtERMqMplAWkSgpCS62\nVIlwmgQ44ZhjwlBw06evVykhIlLWVBcsIlFSEhyF5ER41qyMCTCEcZDPOissjx9fvDBFRArpgAPC\n/TPPwMqV0cYiItWnbIdIqwizZoWiuJkzYcCAjJsuXRpagxcvhuefh6FDixKhiKAh0gpp991h2rRQ\nEpFIikVEWquih0gre7FYaNadOTPcZxk1YuON4fTTw/Lvf1+E+EREikB1wSISFSXBUUiuAR4wIH1n\nuSbOOgs6doR77oHXXy9OqCIihaS6YBGJipLgYkvVCS7TqBFJeveG004LyxdfXIRYRUQKbL/9Qr+H\nl1/OPoy6iEg+KQkupkyjQOSYCI8ZEz4wbrsN3nknzUYtmJRDRCQKXbvCsGFh+Mf6+qijEZFqoiS4\nmCZPzjgKxNpEePLktLvo1w9OPhnc4ZJL0mzUgkk5RESiopIIEYmCRocoQzNmwHbbheV33w0zL20g\nXatzljGJRWRDGh2isCZPhn33hR12gLffLsohRaTCaXSICrX11vD978OaNXDRRWk2asGkHCIiURgy\nJJRFTJ8Oc+dGHY2IVAslwWXqt7+Fdu3g5psztJw0c1IOEZEotG8PI0aEZQ2VJiLFoiS4TG2zDZx6\nauhMcv75GTasqQm96QYODPdKgEWkBKkuWESKTUlwGTvvvDBu8H/+A6+8kmajZk7KISISheRJM9R9\nRESKQUlwGdtyS/jxj8Pyeeel2KCFk3KIiBTbV74Cm28O8+aF2mARkUJTElzmfvnLMKXyww/DE08k\nrWjFpBwiIsVmpimURaS4lASXuZ49QyIM8POfhxEj8jEph4hIsakuWESKSeMEV4AVK8L4mh9+CDfc\nAKf2mRQmwsjUCS4WC4NzjhxZvEBFypTGCS6ODz+E/v2hWzdYuDCMgCMi0hK5XLeVBFeIiRPhuOOg\nVy94771QIiEi+aEkuHi23Rbefx+efx6GDi364UWkQmiyjCpyzDGw997wyScZplMWESlxKokQkWJR\nElwhzODyy8PyZZfBO+9EG4+ISEuoc5yIFIuS4AoydCj88Ifw5ZcwerTG2hSRzMysrZlNNbP7o44l\n4WtfC1/qJ0+G5cujjkZEKpmS4Arzxz/CppuG4dJuuSXqaESkxP0MeAsoma/Mm24Ke+wRvsxPnhx1\nNCJSyZQEV5hNN4VLLw3L55wDixZFG4+IlCYz6wscBvwVKKlOf6oLFpFiUBJcgX7wAxgxAj79NCTC\nZW/SpOxjGsdiYTsRydXlwBigMepAmlJdsIgUg5LgCmQG110HHTrA3/8O990XdUStNHx45sk9EpOD\nDB9e3LhEypSZHQ4scPeplFgrMMC++8JGG8GUKWG8YBGRQtA4wRXs8svh7LNh883hjTdgs82ijqgV\n0s2Cl2l2PJE8qbRxgs3sYuAEYDXQEegG3OnuJyZt4+PGjVv7mtraWmpra4sW49e+BvX1cMcdcPTR\nRTusiJSp+vp66uvr1z6+8MILNVlGNWtsDD8r1tfDUUfBnXeGVuKy1TThVQIsRVJpSXAyMxsB/MLd\nj2jyfKTX7Lo6OO88OP10uPbayMIQkTKlyTKqXJs2oRxi443h7rvhb3+LOqJWqqkJn4xjx8KsWUqA\nRfKn5Foo1DlORApNLcFV4OabQ2e5jh3huedg97mTQv1spuQxFgvjE40cWbxAczVrFgwcCDNnwoAB\nUUcjVaCSW4LTifqavXp1GO1myZLwX75//8hCEZEypJZgAeDEE+GUU2DlylBbt/grZdzRLBaD8eND\nAjx+fPZRI0SkLLVrF+qCQaNEiEhhKAmuEldeCbvvDh98ACedVYP/ri51IlzKdbbJsQ0YsK40Qomw\nSEXSUGkiUkhKgqtEp06hY1z37nDPPfDby2o2TCLLJQFOxFaT4hxEpGIk6oIff1zTwItI/ikJriJb\nbw0TJ0LbtiF3vPHODB3NSmmCikzJuRJhkYq1ww7Qpw988gm8+WbU0YhIpVESXGUOPRSuuSYsjxoF\nj7xYA2PGhI5mY8asSzJLaYKKyZMzt04nEuHJkwsfi4gUjZlGiRCRwlESXG7y0EL7ox/Br34Fa9bA\nyUfFmH92io5mmVpYi102MXJk9uPU1JTmSBYi0iqqCxaRQlESXG7y1EJbVwejjonxm+Vj2euROp6d\nN2DDpDdVIlzKdcMiUnESSXB9PaxaFWkoIlJhlASXmzy10LZZEuOaHmN59eg65n5ew6GHwnNvp9i3\nJqgQkQj17Rtqg5ctgxdfjDoaEakkSoLLUWtbaOPbtrm4jmtvreHYY2Hp0tDicu9TaRLhVHXDIiJF\noJIIESkEJcHlqjUttEkdzdq1g3/+E374Q1ixAo46Cq78Z5OOZukmqCilESREpGKpc5yIFIKmTS53\neZpC2B1+9zs4//zw+Ec/giuugE5fNGlhTm5xhszJt+qHpUJo2uRoxWJhCuW2bWHRIujaNeqIRKTU\nadrkSpfHKYTN4Le/hZtvhg4d4Prr4eC9YjScmWGCCiidESREpGLV1MCee4aOcU8/HXU0IlIplASX\nqwJNIXzCCfD88zB46xjHvTmWr9xTx5//UcOaNUkbZUuElQCLSJ6pLlhE8i1rEmxmHc3sBTN71cze\nMrNLUmxTa2aLzWxq/HZeYcIVoOBTCO++Ozz9+8lM+24d85bXcNZZMGwYTJmStFHyBBUaQUJECkx1\nwSKSbznVBJtZZ3dfbmbtgGeAX7j7M0nra4Gz3f2bGfZRMvVlZS1bK2ueW2Hvvx/OPBPmzAklE8cc\nAxdeCNttl2LjPNUni5Qa1QRHb+VK2GSTcL9gAWy2WdQRiUgpy1tNsLsvjy9uBLQFFqU6XvPCkxYp\n8hTCRxwBb74JZ58N7dvDxImw006hbOKll5I2zGN9sohIUx07wr77huUnnog2FhGpDDklwWbWxsxe\nBT4BnnT3t5ps4sA+ZjbNzB40s53yHajERTCF8MYbw2WXwXvvwamnhuduuQWGDAllEn+/IsaKs/Nf\nnywikixRF6ySCBHJh2YNkWZm3YH/Ar9y9/qk5zcG1sRLJr4B/Nndt2vy2pL6aU1abuZMuOYauPFG\naGyIUcdYxrWtY8+Dahg5Eg45BLbbPIadp9pgqQwqhygNL78Me+0Vqq5mzIg6GhEpZblct5s9TrCZ\n/RZY4e6XZthmJvBVd1+U9JyPGzdu7Ta1tbXU1tY269hSWpbPi/HhCWM5z+q4p379EST69YMDvxrj\nZwvGsuzXdey4dw2bbhpdrCLNUV9fT319/drHF154oZLgErBmTagFbmiADz6ArbeOOiIRKVV5SYLN\nrCew2t1jZtaJ0BJ8obs/nrRNL2CBu7uZDQFud/cBTfZTchdUaaVJk2D4cKip4bPP4IEH4L//hUcf\nhYULwybdiTGcyTzISHr3hu23h223hW22CZUTAwbAVltB797QRgP2SYlSS3Dp+M534K67wljmp50W\ndTQiUqrylQTvAvyDUD/cBvinu483s1EA7j7BzM4EzgBWA8sJI0U832Q/JXlBlfxrbIS334YXXghj\nDk+bFjrXff55+te0awdbbgl9+4akuF8/6N8//Ow5YEBo8enQoRVBJSXsacVioUNhHuuppTIoCS4d\n114Lo0fD//0f3HZb1NGISKkqSDlEK4IpyQuqFEdjI8yeHTrXvfde+Clz9uwwqtqHH8Jnn2V+fZs2\nIRneYQf4yldgl13CeMY77himUs2qyEPLSWVRElw63n03/KLUsyd88ol+QRKR1JQES9lYuRI++iiM\nR5y4zZ4dOuHNmBGS5cbGDV+38caho8y++4ae48OGwUYbpTlIukRXCbBkoSS4dLiHX4nmzIGpU8OX\nYRGRppQES8X44ouQDL/1Frz+eri98kpIlJN17gwHHwxHHgmHH55iQP2mCa8SYMmBkuDScvLJ8Pe/\nw6WXwjnnRB2NiJQiJcFS8T7+GJ57Durr4fHHQ+1xQtu2ISE+4YSQFHfpEl+RSHzHjAkTeygBliyU\nBJeWf/0Lvv99OPRQeOihqKMRkVKkJFiqR7zj29zPa7jvPrj33pAUr14dVm+8MZx+bIyfDJ5Mv9NH\naopnaRYlwaVl/nzYYovwy09DQ4YSKBGpWnmbNlmk5A0fDmPHsmWXGGecAQ8/HD4or7km1Am3WRqj\n/w1j2eWM4Xz7gBhzzxqPz9AUzyLlqE8f2HlnWL48jEAjItISSoKlMtTUbDBdc8+ecMYZ8NxDMd4/\nZizvn1RH505w4JNj2fneOoYfP4DHD6jDf6MpnkXKzUEHhXtNoSwiLaUkWCpHikQ4Uf/b87o6Lr8c\nZnxvLJ//uo52m9bw3HNw0NE1HPFaHR+fokRYpJwceGC4f/zxzNuJiKSjmmCpPKk6vsF6o0B8/jlc\nfTX8/vehprA7MW4fNJad76ljy53VSU7Wp5rg0rNkCfToEZYXLYJu3aKNR0RKizrGSfVq2vEtzYxx\nsVjIk//0J+iwMsYBHSaz17iRnHOOOtvIOkqCS9M++4TRYe6/PwyJKCKSoI5xUp0Sme3MpI5vI0em\nHAYtUUExfToc9J0a7v5iJL/5Dey5J7z0UgSxi0jOVBcsIq2hJFgqS/LkFwMGbFgjnEb//vCf/8Cj\nj8I224TJOIYNg1/8Isxml7NJk7LXFsdiYTsRaRXVBYtIaygJlsqRava3VJ3lMjjoIHjttVBODHDZ\nZfDVr8KUKTnGEB+qLe2xEjEOH57jDkUknWHDwljBb7wRJs4REWkOJcFSGTJNf9zMRLhzZ/jjH8P4\no9tvH6ZqHjo0dKJrbMzy4kzH0hTNInnVoQPst19YfuKJaGMRkfKjJFgqw+TJmZPLRHI6eXLOu9xr\nr9AC/NOfhpnnfv1r+MY3YMGCLC/MMFSbEmCR/FJdsIi0lEaHEMnBQw/BiSfCZ59B794wcSKMGJHl\nRamGalMCXJY0OkTpmjoVBg+Gfv1g9mywqvpXEpF0NESaSB7NnQvHHQdPPw1t24a89qyzsnzoNh2q\nTcqSkuDS1dgIm28OCxfCu+/CtttGHZGIlAINkSaSR1tuGeoOzz0X1qyBs8+G730PPv88zQtSDdUm\nInnVps26USJUEiEizaEkWKQZ2rWDP/whDKfWtWsoi9h33/Az7HpaOFSbiDSfhkoTkZZQOYRIC739\nNhx5JLz3Hmy2WUiM99+f9J3g1DmubKkcorTNmBHG995kE/j001CuJCLVTTXBIgXW0ADHHguPPALt\n28PfLo9x/FsZEl0lwmVJSXDpGzgwlOC//HIY21tEqptqgkUKbJNNwuRvZ50Fq1bBv388mQva19HY\nLX9DtYlIdhoqTUSaSy3BInkyYQKceWboNPed78DNN4eJN6T8qSW49E2cGEZvOfjg8MuMiFQ3lUOI\nFNljj8HRR8PixTBkCNx3H/TqFXVU0lpKgkvfggXh/1rHjqFMqWPHqCMSkSipHEKkyA46CJ59NgwI\n8eKLMGxYmHZZRApr881h111h5Up47rmooxGRcqAkWCRh0qTsQ5jFYmG7DHbaCZ5/PrQEz5oFw4fD\nU0/lL0wRSU11wSLSHEqCRRKGD888lm9iZIfhw7PuqlcvePJJOOqo8LJDDoF//zvP8YrIepQEi0hz\nqCZYJFmex/hdswbOOQf+/Ofw+OKL4Ve/yjLVspQc1QSXh2XLwogtjY1hGmWNQihSvVQTLNJciSHM\nkluEWzG2b9u2cMUVcPnlIfH9zW/gjDNg9eoCxC5S5bp2hb33DklwfX3U0YhIqVMSLNJUciI8a1Ze\nJrc46yy4447QY33ChDDT3LJl+QtZRAJNoSwiuVISLJJKTQ2MGROmoRozJi+/q37nO+GDedNN4cEH\nwxTL8+blIdZCy1OHQZFiUF2wiORKSbBIKrEYjB8PM2eG+2xJYI722ScM3zRoEEydGoZQe/31vOy6\ncPLYYVCk0IYMCWUR06fD3LlRRyMipUxJsEhTyTXAAwZsWCPcSttuGxLhffaBOXNC7vjQQ3nZdWGk\nqpNOiMXg+OPh3HMzt5arpViKpH17GDEiLKskQkQyURIskixVJ7jkJPC221pfGjBpEj3bxXj8cTjm\nGFi6FA4/HP7yF1jbGb/UksZMHQavuQb++MdWtRQvWhRKRK64AkaPhlNOgfPOg6uvhjffLMD5SEVT\nXbCI5MTdi3ILhxIpYQ0N7qNHh/t06085JdwybZNpH022WbPG/fzz3UP66/6jH7l/8UkO+4hKIvaZ\nM9ePMd15Z3g/li1zv+km90MPdW/Xbt17kOq2777ut9zivmpV4U8xlfj1q2jXy1K4lfM1+7XXwt/N\nFlu4NzZGHY2IRCGX67bGCRZJmDQptFZm+1n/v/+F//2vdWMJN9n21lvh5JOh4xcx/tZnLPv9r46e\ng0p0kNNZs0KHwZkzQ7lIQtPzT/N+fPYZXHUVXHllaAGGMJTcPvvALrvAdtuFUTQ+/hhmzIC77w6t\n5RA6E95+e5iMpJg0TnB5cYfevWHBAnj7bdhhh6gjEpFiy+m6nS1LzteNMm5VENlA0xbOXFqAs+zj\nlccb/G+dR3t3Grx/f/dXXsnw2gceyH6shoawXT6lawnOYf3Spe7jxrl36bKuhXfYMPfrr3f/9NP0\nh1yyxH3CBPc+fcJr+vZ1f/HF/J5WNqgluOwcd1z4e7nyyqgjEZEo5HLd1gVVpKWyJYQt2Mf8txt8\n6NDwP7NDB/cbbkjzc24upRv5LqnINfGfOTOcwMyZ7u6+enU4j9691yW/3/iG+1NPNTm3LIn9vHnu\nXx/a4IfxgHfo4P7kk/k7tWwqMQkG+gFPAm8CbwA/bbI+f29gBG68MfytHXlk1JGISBSUBIsUWpOE\nLx/7WLnS/fTT1yWMJ50U6mc30II63A3k2qI8cWJux2qS1L/0aIPvuaf7YTzg3QkJ/tNPpznOrFnu\nI0dmTOxXjxrtZx7f4OC+2Wbuc+ZkP8V8qNAkuDewe3y5K/AOsGPS+vy9gRGYNSv8/+nePbpachGJ\njpJgkUIqQEvw2n088ID/+5oG79Qp/C/dfnv3KVNSvLZpctrcFuB8dgacNWvtvj791P0nJzT4VYTy\njh37NPi7B4/2xkW57yNdnKtWuR90UHhfhgwJXxoKrRKT4KY34B7gwKTHeXr3ojNoUPg7ef75qCMR\nkWJTEixSKAWoCV7vcXz5jWcafOedw//UjTZyHz8+lBek2rbFyXi2FuWJE7PvM96Ku2Zhg19/vXuP\nHiHmnu0a/NnBo33pnIbsx0n3XqZ43aefuvfvH45x2mnNO92WqPQkGBgAzAa6Jj2Xr7cvMqNGhb+R\nurqoIxGRYlMSLFII+ShDyGUf8eXl8xp89GhfWx5x8F4Nvui4Jq9NVZbRnM5zrUnq49u+/FiD77XX\nujgPPND97bdTn1POQ6tlSOxfecW9Y8dwrPr67GG2RiUnwfFSiJeBbzV5Pl9vX2TuuMPXDrG3Zk3U\n0YhIMSkJFsm3fHRIa84+kpYfeMB9h96hxGDzjRr8oovcV6zw9Aljc2NtYYvywpsfWFunC2EUh4kT\nm3R6Sx6pItfj5FBvfeGFYZO99irseLCVmgQD7YH/AmelWOfjxo1be3uymD0R8+Szz9zbt1/3N1KG\npyAiOXryySfXu2YpCRbJt3wMTdbcfSQljStPGb1ewrnrVg3+waFJtbbNbWnNMrJDJosXu593nnvn\nzuEl7du7//KXYUizrLIdJ8dEeenSdaNO3HZbDsdtoUpMggEDbgYuT7M+X29fpP7zn3XD64H74Ye7\nv/FG1FGJSKEpCRapFE2SxiefdN9np3Udz/be2/2RR+KtoS2osV3v+SyJZywWaiw33XRdYvGtb7m/\n806O55LtOM0szZgwIcSwzTbuX95dmPGTKzQJ3hdoBF4FpsZvhyatb9Z7VMqWLXO/6CL3rl3D30qb\nNu6nnuo+d27UkYlIoSgJFqkEqZLGhgZfc/pon/CHBu/Zc10yus8+oUX0ywXNLHXIIfH84IPQ0tu9\n+/rHe+aZFpxLSxP0FAnuqlXuO+wQ4rnu93koV0mhEpPgbLdKvGZ/8on7mWeum6a7U6fwa8bixVFH\nJiL5piRYpNylSxKTRmxYutT9kkvWjciQqMu98OcN/u7lD6yrlU1XgpAh8Vzxw9H+72sa/OCD1+0b\n3Gtr3R97rJl1uNkS3HTDo2V7vbvfc8+6897gC0AOr89GSXBleecd9+98Z93f82abuV91lfuXX0Yd\nmYjki5JgkXLWzFbRJUvcr77afaed1k9Yt9rKffT3GvyNEaP9zUkz/YtTU7e8rlrlPnu2+733uo8d\n6z5ihHuPNutKLjp0cD/hBPdnn83juSSvP+ywkAhn20+KUobGxjCWMrjffXeK47UiAXZXElypJk8O\nv2Yk/q9su637nXcWtpOliBRHq5NgoCPwQrxm7C3gkjTb/QV4D5gG7JFmm2Kdt0j5a8UoFI2NYciw\n0aPdt9jCvTvrElkIj6/faLQP3rrBf7TlA77noAbv0yfUSSYnzxB+Nv5WbYM/cMYDvnBhK86nCB0K\nL7ssnNtvBzdzFIocKAmuXI2N7nfd5b7dduv+7vfeu5llPiJScnK5blvYLj0z6+zuy82sHfAM8At3\nfyZp/WHAj939MDMbCvzZ3Yel2I9nO5aIxE2aBMOHQ01N+m1iMZg8GUaOTLtJ46IYn44ay12D6/jf\nazVMmQIffggdVsaoYyxjqWMx4RhmsPnmsNNOMHQoDBkCX/ta5hCKKhaDsWOhri5lUItmxLh927H8\nurGOqTNrGDAAmDULBg6EmTMJT7SMmeHu1uIdlKFqu2avWgV//StccAEsWBCeO+oouOQS2H77SEMT\nkRbI5bqdNQlO2lln4CngB+7+VtLz1wFPuvtt8cfTgRHu/kmT11fVBVUkcmmSRndYtAgWfhBjsyvG\n8ulZdXTsXUOfPtC+fYTx5iJdIhx//kef1nHDHTWMHQu/+0V82zFjYPz4tMlzLpQEV4+lS+HSS8Nt\n+XJo2xZGjYLzz4devaKOTkRylZck2MzaAFOAbYBr3f3cJuvvJ5RJPBt//BjwS3d/pcl2VXlBFYlM\nnlqTS07TRDjp8f9eq2HECNi+V4w3jxpL20s23KYlibCS4Oozfz6MGwc33giNjdC1K5x7Lpx9NnTp\nEnV0IpJNLtftNtl24u6N7r470BfY38xqUx2r6ctyjlJECmPkyOwJX01NeSXAEGKuqwtJ7axZ6yW3\n++0HQ7aL8ZNPxvLQ8KSEN/k1sVik4Ut56NMHrr8eXn8djjgCli0LrcGDBsENN8Dq1VFHKCKtlXM5\nBICZ/RZY4e6XJj13HVDv7hPjj9OWQ4wbN27t49raWmpra1sXvYhUr1T1vrEYrx4+ltrJdRxwVA13\n3dXkNTm2CNfX11NfX7/28YUXXqiW4Cr31FOhsuall8LjHXeEP/wBDj881NOLSGlpdTmEmfUEVrt7\nzMw6EeaYv9DdH0/aJrlj3DDgCnWME5GCiqWp9500iY/6D6ffLjV06QKffQYdO6Z4bTNLQFQOIRDq\n6e+4A379a5gxIzw3YgT88Y+hI6mIlI58JMG7AP8glE20Af7p7uPNbBSAu0+Ib3cVcCjwOXCyu09J\nsS9dUEWk9TLUBCdad7/6VZgyJZRFH3ZY6w+pJFiSffEFXHcdXHQRLFwYnjvmmPAnuM020cYmIkFe\nR4fIQzC6oIpI62QZHSLx/IUXhqGuRo0KyUprKQmWVGKxUBJxxRWwcmUYXWX0aDjvPOjZM+roRKqb\nkmARqRzZ6nmT1k+dWcPgwbDFFjBnDrTJ2gU4MyXBksmcOaHT3D/+EUomunULJRM/+xl06hR1dCLV\nSVL6ifgAAA9PSURBVEmwiFSOZgz55oeNZKut4KOPQkemPfds3aGVBEsuXnsNfvlLePjh8Lhv31Ay\nccIJYbxhESmevAyRJiJSEpox5JsZfPOb4an77it8aCIAu+4KDz0Ejz4Ku+8evoSdfDIMHgz//W9o\nJRaR0qEkWEQq0pFHhvt77402Dqk+Bx0Er7wC//wnbLVVaCE+9FA45BCYOjXq6EQkQeUQIlKRvvgC\nNtssTIObPJRwS6gcQlpq5Uq46qpQyh6LhTGFv//9UCbRv3/U0YlULpVDiEjV6tABvv71sPzQQ9HG\nItWrY0f4xS/ggw/ClMvt24cW4u23D9MwNzREHaFI9VISLCIV64ADwv3TT0cbh0iPHnDZZTB9Onzv\ne+GXivHjw7jCf/pTeCwixaUkWEQq1v77h/v//U+dkqQ0DBwI//oXvPwyfO1roSX4nHNghx3g3/+G\nxsaoIxSpHqoJFpGK1dgY6oIXLQrT3A4c2LL9qCZYCsE9DKd27rnwxhvhucGDQwtx4lcMEWkZ1QSL\nSFVr0wb23TcsqyRCSo0ZfOMb8OqrcNNNYXKXKVPgwAPDdN+JxFhECkNJsIhUtOSSCJFS1LZtGE/4\nvffCKBIbbxw6c+62G5xyCsydG3WEIpVJSbCIVLT99gv3agmWUte5M/zmN2EkiZ/8JPyScdNNsO22\nYUbwJUuijlCksqgmWEQq2qpVYSK55cth/nzo3bv5+1BNsETh/fdDUnzHHeFxz55h5vAuXcKta9fU\ny5keb7RRKMMQqXS5XLeVBItIxTv4YHjssZBMHH1081+vJFii9PzzMGYMPPNM6/fVtm3zkuZcH3fp\nEsZAFikVuVy32xUrGBGRqOy3X0iCn366ZUmwSJSGDQs17S++CPPmwbJl8Pnn627NebxqVSirKERp\nxUYb5Z40d+4cyj1gXct0qd3num3i1qZN7vfN2TZf97lss8kmYaKhaqEkWEQqnjrHSbkzg6FDW7+f\nL79cP0FubhKd6fGXX4abZsErX2aw5Zaw9dapb5tvXlnlNCqHEJGKt2IFdO8Oq1eHD+ju3Zv3epVD\niGTmHma9yzVpXrEivCbxJ1Zq983ZtrFx3bkkljPd57JNa+9b+pqFC2HNmvT/zp07h2R4m202TJAH\nDAjThJcK1QSLiMQNHRp+Tn788eZPRKAkWESqwapVMGdOmFwo1S1bK3+mVuRevYrbiqyaYBGRuD32\nCEnw1KmajUtEJJX27dclrak0NMDMmakT5Nmzw5jWc+emHpKyU6f0CfLAgWF9sSkJFpGqsMce4X7q\n1GjjEBEpV5tsEm6DB2+4bvVq+OijdUnxBx+snyQvWgRvvhluqWyxRfokuXfvwrQiqxxCRKrCSy/B\nkCGw007pL8LpqBxCRKR1YrH0rcizZoUkOp1OnUJrcbpW5M6dN3yNaoJFROJWrgzDNLnD0qWpL5rp\nKAkWESmcNWvWb0Vuevvss8yv7917w+T4pJOUBIuIrLXLLvDGG2HygeYMN6UkWEQkOkuWpG9Fnjkz\ndOjbkDrGiYistcceIQmeMiU/Y66KiEjhdesGu+0Wbk2tWRMmkUlOjD/4AG69Nft+lQSLSNUYPBj+\n+U91jhMRqRRt20K/fuE2YsS653NJgtsULiwRkdKiESJERCRBNcEiUjUWL4aaGujQIXSOa98+t9ep\nJlhEpLzkct1WS7CIVI3u3UOv4S++gOnTo45GRESipCRYRKpKYpB3lUSIiFQ3JcEiUlUSdcFTpkQb\nh4iIREtJsIhUFXWOExERUBIsIlUmkQRPmxZmjxMRkeqkJFhEqkqvXrDJJmGkiE8+iToaERGJipJg\nEakqZrDDDmFZI0SIiFQvJcEiUnWUBIuIiJJgEak6SoJFRERJsIhUHSXBIiKiJFhEqo6SYBERsWLN\nDa956EWkVKxaBV26hPvPP4fOnTNvn8sc9JVG12wRKWe5XLfVEiwiVad9exg0KCy/8060sYiISDSU\nBItIVVJJhIhIdVMSLCJVaccdw72SYBGR6qQkWESqklqCRUSqm5JgEalKSoJFRKqbRocQkaq0ZAl0\n7w4dO8KyZdC2bfptNTqEiEh50egQIiJpdOsGW2wBK1fC7NlRRyMiIsX2/9s7+xirjjIOPz9YsEAr\nSGqgLWzKH5AAwULRLYFSatMPiqatQVOqrRUSY7QG01ZKIU0kxg9sY1qNqTVKjbUGTaht0KLFRtBq\nkI/C8v0hASJgoU2oIGKEuq9/zNxyWPcue+/evXPO3vdJTu6cmTm7v5k77znvnTMf7gQ7jtOw+JAI\nx3GcxuWiTrCkkZLWSNopaYek+R3kuVHSSUlb4vFYz8h1HMepHY3sBEuaKWmPpL9KWphaj+M4Tr3p\nSk/wOeBBMxsPTAEekDS2g3x/MLNJ8fhaTVXWmbVr16aWUBFF0lskrVAsvUXSCvnQ26hOsKS+wPeA\nmcA44J4y9/VeQR7aWq3oLWXpLeUAL0uRuagTbGbHzKw1hk8Du4ErO8jaayaNFK0RFElvkbRCsfQW\nSSvkQ2/JCT5wIK2OBLQA+83skJmdA34O3JlYU4+Rh7ZWK3pLWXpLOcDLUmQqGhMs6WpgErC+XZIB\nUyVtlbRK0rjayHMcx+k5pk0Lk+JWr06tpO5cBRzOnB+JcY7jOA1DU1czSroUWAF8KfYIZ9kMjDSz\nM5JuB14CxtROpuM4Tu0ZOBCam1OrSIKvfeY4TsPTpXWCJfUDfg38xsye6kL+g8BkMzuRifObruM4\nhaU3rRMsaQqwxMxmxvNFQJuZfSuTx+/ZjuMUmovdty/aEyxJwDJgVzkHWNIw4E0zM0ktBOf6RDZP\nb3qAOI7jFJxNwOg4xO3vwN3APdkMfs92HKe305XhENOAe4FtkrbEuMVAM4CZ/QD4OPB5Se8AZ4A5\nPaDVcRzHqQFm9o6kLwKvAH2BZWa2O7Esx3GculK3bZMdx3Ecx3EcJy9UvWOcpGclHZe0PRPXImlD\n3DBjo6QPxfhLJC2XtE3SLkmPZq6ZLGl7XLD9O90rTsV6r5G0LupaKemyTNqiqGmPpFvrqbcSrZJu\nkbQpxm+S9OF6aq1Ubya9WdJpSQ/XU28V7eADMW1HTO9fL62V6k1tZyqzsY6koZJ+J2mfpNWShmSu\nSWJnlWrNg53VE/WSjTQ6sp8iUq69FpF4n1ovqTXep76ZWlN3kdRXwe/5VWot3UHSoXiP2yJpQ2o9\n1SJpiKQVknbHNjalbGYzq+oAphOWS9ueiVsL3BbDtwNrYvgzwPIYHgAcBJrj+QagJYZXATOr1VSF\n3o3A9BieC3w1hscBrUA/4GpgP+d7zXtcb4VaJwLDY3g8cCRzTe7qNpO+AvgF8HA99VZYt03AVmBC\nPH8f0CevdZvazoDhwMQYvhTYC4wFHgceifELgaUxnMzOqtCa3M7qdRCGR+yP30m/+B2NTa2ryrL8\nn/0U8SjXXlPr6kZ5BsbPJuAvwPWpNXWzPA8BPwNWptbSzXIcBIam1lGDcvwEmBfDTcDgcnmr7gk2\ns9eAt9tFvwEMjuEhwNFM/CCFXYoGAWeBU5KuAC4zs9IvjueAu6rVVIXe0TEe4FVgdgzfSXAmzpnZ\nIcID4bp66a1Eq5m1mtmxGL8LGCCpX47rFkl3AQei3lJc7uoWuBXYZmbb47Vvm1lbjus2qZ1Zxxvr\nXAXcQbgpET9L/zuZnVWqNQ92Vkd6zUYaZeyncJRprx1tWlUIzOxMDPYn/Og60Un2XCNpBDAL+BG9\nY9OwQpdB0mBCJ9GzEOY/mNnJcvmrdoLL8CjwbUl/A54gTKDDzF4BThEe0oeAJ8zsH4SHzpHM9Uep\n74LtOyWVbu6fAEbG8JXtdJUWkm8fX0+95bRmmQ28Hh9cuaxbhfWmHwGWtMufUm+5uh0DmKTfSnpd\n0oIYn8u6zZOd6cKNdYaZ2fGYdBwYFsO5sLMuas2SJzvrCXwjjRyj8ptWFQZJfSS1EmxsjZntutg1\nOeZJYAHQllpIDTDg1Tjk67OpxVTJKOAtST+WtFnSDyUNLJe51k7wMmC+mTUDD8ZzJN1LeD17RRT4\nZUmjavy/q2Ee8AVJmwivmM4m1tMZnWqVNB5YCnwugbaOKKd3CfBk7AnIyy/OclqbgOuBT8bPj0m6\nifQbDXSoNy92Fn/ovEDYWOef2TQL76dS19+7VKo1h3bWE+Tm+3EuRJ1vWlUYzKzNzCYCI4AbJN2Y\nWFJVSPooYXnYLeTnedYdppnZJMJw1gckTU8tqAqagGuBp83sWuBfhA7asplrSYuZ3RzDKwivBwCm\nAi+a2X8JHvqfgcnAnwhGUGIE54dQ9Dhmthe4DUDSGOAjMekoF/a0jiD0hhwlkd5OtJZex/wSuM/M\nDsboZFqhQ72zYlILMFvS44QhM22S/k3Qn7e6PQz80eKa15JWEYzr+VRay+gt1W1yO1PYWOcF4Kdm\n9lKMPi5puJkdi8MH3ozxSe2sQq25tLMeov33MpILe7udBGTa6/OZ9lpozOykpJeBDxLmFBWNqcAd\nkmYBlwDvlfScmX06sa6qMLM34udbkl4kPK9f6/yq3HGEMGdjYzxfQSdOcK17gvdLmhHDNwH7YnhP\nPEfSIGAKsCeOsTsl6TpJAu4jbLlcFyS9P372AR4Dvh+TVgJzJPWPPWmjgQ0p9ZbTqjB7/WVgoZmt\nK+WPjTlPdftM1HWDmY0ys1HAU8DXzezpPNYtYQ3VCZIGSGoCZgA7c9hun4lJSe0s/u2ONtZZCdwf\nw/dn/ncyO6tUa17trId4dyMNhdVQ7ibUi5OITtpr4ZB0uc6vujIAuAXY0vlV+cTMFpvZyPg8mwP8\nvqgOsKSBOr/S0CDCnJjCraoSnx+HYwcRwM3Azs4uqHb23XLCTkNnCT1mcwm/5tYTZhOvAybFvO8h\n9J5tj2KyKwJMjvH7ge9Wq6cKvfOA+YRZtnuBb7TLvzhq2kNc8aJeeivRSnCCThNuIqXj8jzXbea6\nrwAP5bVuY/5PATuirqV5brep7YwwZKQt2n+pLc4EhhIm8O0DVgNDMtcksbNKtebBzup5EF6H7o1l\nWpRaTzfKUbKf/0T7mZtaU5Xl6LC9ptZVZVkmAJtjWbYBC1JrqlG5ZlDg1SEIQ+ha47Gj4HZ/DWEV\npa2Et3dlV4fwzTIcx3Ecx3GchqPWwyEcx3Ecx3EcJ/e4E+w4juM4juM0HO4EO47jOI7jOA2HO8GO\n4ziO4zhOw+FOsOM4juM4jtNwuBPsOI7jOI7jNBzuBDuO4ziO4zgNhzvBjuM4juM4TsPxPwfI3PQG\nZ3jfAAAAAElFTkSuQmCC\n", "text": [ "" ] } ], "prompt_number": 33 }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 33 } ], "metadata": {} } ] }