{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Exercise 3.2 - Neural Networks\n", "\n", "This notebook covers my solutions to Andrew Ng's Machine Learning course Exercise 3 - Neural Networks.\n", "\n", "In this exercise, we implement a neural network to fit a non-linear hypothesis to the MNIST dataset. The parameters (weights) of the model have already been provided and the goal of this exercise is to implement the feed-forward algorithm to make a prediction using the provided weights.\n", "\n", "First, import necessary packages:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import os\n", "\n", "from scipy.io import loadmat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's load and inspect the training data" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(-0.5, 399.5, 19.5, -0.5)" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2oAAABMCAYAAAAcNkS9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsfflXm1mS5RVo3/cVLSDE7i2dmd2Vp0//4zPnzPzUM9UzlWU7DWYVIAmEQCtaQPv84LnhJyW2WQSmqhXn+LjSBdL73veWiBs3bmiGwyGmNrWpTW1qU5va1KY2talNbWrPx2Z+9ACmNrWpTW1qU5va1KY2talNbWqjNg3Upja1qU1talOb2tSmNrWpTe2Z2TRQm9rUpja1qU1talOb2tSmNrVnZtNAbWpTm9rUpja1qU1talOb2tSemU0DtalNbWpTm9rUpja1qU1talN7ZjYN1KY2talNbWpTm9rUpja1qU3tmdk0UJva1KY2talNbWpTm9rUpja1Z2bTQG1qU5va1KY2talNbWpTm9rUnplNA7WpTW1qU5va1KY2talNbWpTe2amfcovCwaDw6f8vqlNbWr/fDYcDjEYDNDr9dDv9zEzM4PZ2VnMzs5iZmaKPU1talOb2tSmNrXnbWdnZ5rb/NyTBmpTm9pDbTj8eqyv0dxqzU/tFqbO83OZ1+FwCI1Gg9nZWRgMBuj1euh0OvR6PfnT7XZ/9DCfnXHe/tFsfK//Iz7D1L5t3zrPVZu++/8a9o96Vk3tn89uczY91VqdBmpT+4ew4XAof75mGo1G/kztbsZ5Hf8bwMic/qi5ZRaNQZrT6UQ8Hofb7Ua1WkW1WsX5+TkuLi6ebIzP3alQ98w/2t4Y3+8ajWbkOab2j223Oc9VU8+f5/z+x5/puY/3Odj4nE33+dR+pKnr8Xvn01Pdq9NA7ZHsOWYkvmfPecwajQYzMzPfHNddLv6pjRrnldTB8Xm+7cH1GKZm0kwmE/x+P6LRKNbX1xEOh3FycoJcLod2u43z8/Mbx/8Y9tz2yLip73LcEXqONn7+zM7O/un//5FjH//u2wTqaqCp/v1f2ejY6HQ6yYprtVpotV/ckV6vJ/RmZsp7vZ4ANs91Hm+iXj+3/XbTOgb+vDafao6NRiOMRiMGgwEAoNvtot1uP7t5o90UVAJPM1/fO4Oe6764yW66k9Tx/6izc2ZmBlqtFgaDATqdDgaDAbOzsxgMBhgOh2i32386n/r9/qOO6Z8uUBt3KJ8ameFFoi5CBhjfcn5/5EXOcQwGA7kE1TGPO3pPbdw4vNT5bzT1gO90OrKJnnvG40fb+B7R6XSYnZ2FXq//k9PU6XTQ7/fR7XafdC3QOddqtTCbzQgGg1hdXcXq6ipWVlbg9XphNpvR6/WQy+UefTxcT6yHGwwGckg/J8didnYWWq0WOp1O6KF0dnu93r0+czw7N0lT9+vMzIw4b/zvfr+Pfr+PdruNTqcj59RTOUcc313QVtVmZmbkTP2vXEfJvWwwGOB2uxEKheD3++H1emGxWKDRaNDv99FqtdBsNlGr1VCpVFAsFlEqlXB9fY3r6+vvgnZPbcPhEDMzMzAYDHJ2DodDdDqdZ0PHVtevup5pqg+irtfHtmAwiEQiAY1Gg06ng/Pzc+RyOVxdXY2M60fbTXPH967O2WMZv1f1L1UfR/XZnusZo/qafBb132+y8ed6rPXAs0mr1cLr9WJubk7OJ4vFIudSqVRCuVxGvV5HvV5HsVhEpVKRz3iM8f3TBWrjmQHgaR0oIsE3IQPj//1cUFZ+v+qc034kis1D0GQywWw2w+l0wuFw/MlB4xgbjQZqtRoajQZardajoxz/qKauPwZmRqMRZrMZJpMJJpNJnGSNRiMHUr1ex+Xl5ZOuB15+NpsNPp8Pi4uLePnyJV6+fIl4PA6r1YpKpYJsNgutVovBYPCohznnjIFEp9NBq9W6d/AzaVP3jNVqhc1mg9PpFHoox3qfC+UxAyNewqw9dLvdcLlc0Gq1mJ2dRafTQbvdRqVSQbVaRbvdfjIwZvxMv8v38WfvG+A9hv1I5oRGo4HJZILH40E8HkcqlUIikUAkEoHNZsPs7Cx6vR7q9Tqq1SqKxSLOzs6QyWRgMBhQLBafTeCjmkajgV6vh8vlgt1uF7S9Wq2iVqs9i2ywCrx+j5mi/v3Y5vf7sbGxAa1Wi16vh729PTQaDQwGA3S73WeTQeW8MdN/kw/ymKYyDG6aj+dyvnzLVKDzLvfJUz2bXq+H3W5HLBbDxsYGFhcXEYvFYLfb5Vw6PT3F2dkZyuUyzs/PodVqcX19LYD2Y9g/VaCm0WgkIjYajZiZmUGz2cTV1dWjbnQuHqrP0Tki2t5sNnF5eTlywdAx0el0mJmZwXA4FNS43+8/GVI8HA4lU+VwOGCz2XB9fY1GoyHo5Y86KDUaDQwGA2KxGObn57GwsIBEInFjtm8wGOD09BRHR0dIp9M4PDzE5eUl+v3+oztzKrp200Fy2wvysU097Lj27HY75ubmEA6H4fV64fF44HQ6YbfbJUg6ODiQP81m88nWJ/CZFuPxeBCLxbC2toalpSXMz88jFApBp9Ph8vJSnLlms/loFyY/k9mAaDSKWCyGfD6PdDqNq6srWWs/0obDIWZnZxGJRJBMJrGwsICFhQVsbW1ha2sL2WwWzWZT6v1u+5kzMzMwm83Q6/WSueZZ9dDx0nm3Wq0IhUIIhUKIx+NIJBLQ6/XQ6/VoNptoNBo4ODjAp0+fkM/nUavVJGCb9N5Sz3SDwSDjs1gsIyDG10ylnWo0GjQaDVxeXqLRaKDRaDx5YD+OZKtgzdcYH5P8bqLVOp0O0WgUL1++xOLiIpLJJHw+H6xWKwwGAzQaDQaDgQTmV1dXuLy8xOnpKTKZDD58+IDNzU25m370fgO+zKHb7cbPP/+MZDKJXq+HZrOJ9+/fY2trS7LZ/PnHNPWcV/+3Xq+H2WyGzWaDw+GAyWT6U7BB2mG1WsXl5eVIBn7S+4v3i9PpRDQahdPphNlsFqf48PAQ+XxeMms/ytQzSgU0jUYj9Ho9rq+vR4BMdb4mNWc6nQ5GoxEWiwU2mw1Go1F8Rwa09XodjUYDV1dX4rc9hyBXZUqQ6WE0GuF0OmGxWKDVar+ZAby+vkar1UKj0UC9XpfgfZJ+HX1Ir9eLtbU1rK2tSWmF3W6HXq/HYDCAy+WCy+VCPB5HvV5HpVKBx+OByWRCPp/H2dmZfN4k7Z8mUKODwo1ktVqFMtNqtQA83gHJA4coeygUQjgclgvn4uICAHB1dTVCLeTBSSSJC/IpggsaDyCHwyHjrlarODs7Q6lUEofsqTc7n99gMCASieDVq1d48+YNXr169VV0JZ1OY2trC7OzsyiXy2i1Wn+ioT6GqVSDr83Tj0ZUVeoAs2gmkwmBQEBohOFwGMFgUAI2St7/53/+J2ZnZ1GtVpFOp58k8FUvcaJbv/76K5aXl2G326HT6dBsNiVIOzs7e5Jsn8FggMPhwNzcHNbW1jAzM4PT01O02+0fnsHle9HpdLJnXr16hZcvX8JoNKJWq6FarcrP3vY98jOZLajVaqjVakIzfuha0Gq1cDgcCAaDWF5extLSElZXV7G8vAyTyQSDwSAZFp/PJ+uDe7vf7wuFZpJGurXdbofH44HP54PH44HD4RhpBfG1Nac6KBcXF8jlcsjn87i+vka3233SM1XNBqiZwXEa1WPZcDiETqeD2WxGPB7Hzz//jJWVFUSjUZjNZnEq+bPqedXr9VAsFhGPx+V/l0oltNvtRx/3bZ6LoJfL5cKLFy/w66+/otfroVwuo1ar4fDwUGiQT0FJU9+v+sdqtcLtdiMQCGBubg52u31kPdAHqdfryOVyyOVyaDQaE1+rKgii0+lgs9kQCoUQiUTg8/mg1+vFD7q8vPzh9WosvXA4HHC73XC73QJm2mw2VCoVnJ+fI5/Pi785Sb9jOByKgFYgEEAoFJJgTfUdKaZVLpdRrVYF7PjRxgSKwWCA0WiEyWSC0+lEOByGx+ORoHP8HOK6rVarqFQqKBQKyOfzaDQaj3bfOhwOpFIpAYWZcOGYjEYj3G43NBoNer0eLi8vodFoBIg5Pz9/FKbHP02gBnxOW9LRNBgM6PV6kkJ/DDoUAxiPxyOc1mg0ikAggEAgIIfzxcUF8vk86vU6Op2O1NsQmQGARqOBUqmEvb09ZLPZkczapMdNJ431KwsLC5KpSCQSODo6wh9//IHBYIBarfZo4/jW+FRaHh0lUmOurq7QarVQqVRQqVTgcrmkxiGRSKBUKuH4+BjNZhOVSgW9Xm/i759zSHDAarXC5/PBbrfLzzDbx3qLcrmMSqUi6M1TZdl40FutVvj9fszNzcHtdsuho1KPeAGowXm73R45HO/i5N9nrIPBQKiuS0tL+Omnn7CysoJYLAaj0Yjr62tcXFzg8PAQBwcH+PjxI7LZLFqt1qP1UmPg6PV6EY/H4fV6hSr8FMXE3zOuJ+4Xn8+HUCgEADg9PcXp6SkKhQLq9TqA26G9XN9msxlerxcvX75ENBrFx48fsbW1NZJBvOta4GezVmljYwNra2uIx+OIxWJwu93Q6/UAPteezs7Owmq1Yn5+HhqNBqFQCOl0GicnJzg/P0etVsPV1dWDnEr1XNTr9fB4PAiHw5ibm8Pc3BwCgQBcLhesVuuNgRozEr1eT7JBrA0kffPq6grVahXX19f3GuN9nolZQYvFgkAgAL/fD7PZDKPRKCgwKaWPCcK4XC7EYjEkk0lRbCXrhdkAZuy5Nmw2G0wmk9ztKysruLq6wubmpgQRZKr8CDCR69hms8HtdsNut8Nutwu4wQxxoVCQjP9j3ec8o0gX5lgcDgfsdjvcbje8Xq84+xwn76l+vy+BGhkU+/v7SKfTEwc8uc+YmVLraQOBAF68eIFut4tqtYp+v/8oAeNtxshMqdvtxsrKClKpFJxOp2T/rFYrLi8vUSqVcHJygmw2K9Q47vOHUHV5H7pcLqyvryOZTCIWi8HpdI7USbfb7ZF6zkKhgHQ6jaOjowfVJT9k3NwbpLLH43EEg0G4XC4BvpxOp4h2jIPZfNfMVJ6eniKbzeLo6AjHx8fCUpvknrq+vkaxWMTBwQHq9TpmZ2dRr9fR6/Wg1Wphs9kQiUQQDAZhNpthsVjg9XoRiUSQy+UwOzs7ch9Malz/VIGa0WhEMBhENBoFALRaLZycnDxK+ld1Vt1uN5LJpKDXPp8PbrcbwGcno1gsClWn1WrJwiXdYzAYoFwu4/j4GFdXV7i4uBDO62M5w6RCWK1WLCws4F/+5V8EzX737p0USarI9fhYHrvWgVlHq9UKp9MJk8mE2dlZOZSOj4+RTqcxPz8vgW88HkexWITf7xfH9LGcIh5EVqsV4XAYS0tL4hxz/MPhEBcXFzKX3PT8/acI0gaDgay5VCqFn3/+GbFYTII3r9c7UvtHdJv7hg5mp9N5dGST42Vmenl5Gb/88guSyaQU9F5cXCCTyeD9+/f48OGDXI6PFZADkAvH7/cjlUrBYrHIoayK1/wIUx01o9EIl8sFn8+HQCCAXq8n80M0Evj+fuV7IOIdDofx+vVrrK2todVq4ejoSOg1953z2dlZOBwOyf799ttvCAQC8Pl8AlRxftVALRAIIBwOw+/3Y29vDzs7OwAwsT56pARHIhG8ePECy8vLSCaTCIfDEjiowgGcKzq619fXqFarqNfraDabaDab6Ha7AsZptVpxjh/T1LVrsVjg9/uxtraG5eVleDweWK1WfPz4ER8/fgQAqaV6DNNoNHA6nUJhn5ubg8PhEFT67OxMsgHX19fQ6/UjgaXL5ZK9x+COmaofWbOm3gFOpxM2m03udZPJhGAwiGAwiHq9/ujve3Z2VihyZEsQaIhEIvD7/fD5fPB6vfD7/bBarbJvuYbb7Tbq9Tr8fj8cDgfa7Tay2exEqZsqFdZsNotAGMFZAgnX19fIZDK4vLzE9fU12u32kwmcqGeq2+3G4uIi3r59i19//VUCYAaZFL05OTnB0dERdnZ2sLm5CQAoFovodDoPBo8cDgcWFxfx6tUrJJNJOJ1O+RnWQ15dXaHZbEr2fjgcolAoCIAFPD2YwSx6KBTCy5cvkUqlEAwG4fP54HK5YLPZRPVVpWcDX0RECHRnMhlkMhlYLBbU6/UR9cVJPJtGo8HV1RXOz8/R6/WQyWTQarVwfn6OTqcjPsnbt2+FPadmqW0226Otz3+4QO0m55YpdKfTKdFuLpfD2dkZWq3WxCePDqzVaoXD4RDUn04EAKENMgthtVphNBrF6eBhyoPKbrfDYrGg3W7DYDDg6OgI2WxW0JBJjp/iEclkEslkEm/evBFKGXm3pVJJalnG54+UFPXf1Q0zKRsOP0uhnp2dYXd3V/6t0+mg0+ng9PQUHz58AACEQiH4fD5BOex2uyDfkzY100AVwqWlJSQSCfj9fvk5OmP1eh21Wg27u7sIBoMolUqoVCojCLJK95mk8fJzOp2S0k8mkwiFQnKxm83mPwng0AaDgQR4lUoFp6enqNVqj6L+yItRq9UiGAzixYsXWF1dhc/nw2AwQCaTwenpKfb393FwcCD/zbqAxwJkNBqNZHWXl5fx9u1bKSRm1uRH1QJwfGoGkpdhq9XC/v6+zFexWLyzktrMzAwsFssI5Yfg0n2MtXEmkwlerxcvXrwYec8Gg2GkBoFnO/cG12swGJQz3+fzSQYgl8sJOHOX/cSss8lkQiwWk+BsYWEBwWBQMuXcy1SfZEB5fX2Nq6srFItFFItFXF5ejjiYl5eXIo7xFNk0Po/L5UIgEEAymUQkEpE65MFggFarhUAgAKvVKsqp5XJZKMRfOxPua+q77PV6UlOcyWSQzWZRLBbRbDZH7kin0wm/34/l5WXMz89Dq9UimUzi+PgY8XhcqKQ/shZnMBhIgM41AUDq5UkhByZL0+Tet1qtsFqtiEQiwpZwuVyS+VGDR61Wi06nI7U0zFIxw8qaULfbjWg0ip2dHdnvk8rKqD5Dq9XC8fEx/vrXv0pNOcfq8/mwsLAgQkjNZnMi3/89UynkDJAIFrpcLnQ6HRQKBXkOvV4vzACdTgeLxQKXywW3243ff/9d6mjvY/S1arUa0uk0rq+vsbOzI5RHCkOxhpiAAZU0q9WqBDhPxd4ZDoewWCzij1NjgH4H16Ner/8m0KkyjgwGA/x+P3Q6nbB9dnd3cXBwgGq1KvNwX+O8XF5e4vDwEGdnZ5iZmRE20czMjGQBVQ2MTqeDWq02kkEd/8xJ2D9UoHZTSpGLVK/Xy8IIBAJIp9MoFApotVoTXaAq0mK1WhEMBrG0tIS3b98KBTKfz+Pi4gKtVgvtdhs2mw0ej0eKpQFIlkMt6iWdymw2A/gc7E0SDeG8sc5maWkJv/32G1KpFFKpFFqtlgRqxWIR9Xr9TxegSktkYSj/fVK1bOocXV9fI5/PC4pNSoxer8fJyQk+fPgAm82GtbU1OJ3OkYJp1v9N0og8MhsVi8Xw66+/4s2bN4hEIpJJVY00qGAwCLfbLan709NTcfTvSyH73lgZULtcLgnUFhcX4fF4RrLC44E4g0zSLpaWlpDP57G/v49ut4vLy8uJOkfjVIlAIICNjQ2srKzA5/Oh2Wwim83i3bt3+Otf/4qdnR1xjB6TQso5pOgKA7X3799LBpw1Uo8BCnzN1Mwng5dAIICXL1/izZs3AD4zCra2tvDf//t/R7vdHglebjNXarZADdT0ev29gBmuJ61WC4vFgnA4jJ9//hm//fYb3G63ZFjUDIl6xgAQ4SNmDCKRCKLRKHw+n1ys7XYbzWZTqEHfu8DVPU1Q4t/+7d+wvLyMQCAAk8kkWWWKglBsgeILl5eXqNVqEngwo0ZarHp3PUUmjYFaMBjE2toafv31VywsLEgASfo4M1a9Xg+lUgndbhflclnOhUmCnARhWNORz+fxf//v/8WHDx+QTqdRKpVGHDatVit9Ey8vLzE7O4v5+XnEYjGk02lEo1E0m00UCoUfls3mOdntduW+J7CqBmmPdTYBED/k1atXePv2rciKq7U/vEspZlMsFkUdGYBk2sj0cbvdAtLp9foRNsUkMhfAl5q4o6MjOUftdjui0aiUEszPz+P8/BwHBwdP9o5VujABzp9//hkejwculwunp6cC0nU6HSkn8Hg8kkGNRCLQ6/U4Pj7G2dnZSLnDXUyl/6XTaeRyOaFWt1otaVViNpsxNzeHZDKJX375BYuLi5ifn5cz6+jo6EnqI3n2WK1WRKNRbGxs4KeffpJyAVU8RGVM3JSAUc8fvV4vtfMUdtLr9QKGUc34IWtTo9EIzZLW7/fR6XRgtVqlzQ3vHwZq1HSgGvH4nTUJ+4cI1NQMDql4dBToIJGHzfQlL1MW8E7ioFTpRXa7XdRhiAYTJSQli4uH/HWDwSALtNfrwWKxIBqNCo3HaDQKXYm9Gk5OTgRJmkSgRt58MpnE6uoqFhcXYbPZ0Gw2cXBwgN3dXbx7906cDT4zLyR+Bg90yinz4JpkLxFmz87OziQ4yGQyUkt1cXEBm80m2UitVotutyuH0yRl08fpGuFwGMvLy8IbZzayWq3i6upKaBoEEbi5U6kUfD4fUqkUMpkMjo6OhPZDihSzmJMwfq/b7UYwGITH45HLmxclnUoe+pTo59xyvZP2RVR80tTN4XAoSO7S0hLC4TCMRiOq1Sqy2Szev3+PP/74A4VCAdfX109SF0bnkrV8zNr2+31BeW/rQNyEGN7X1Jo0BtIrKytYWlqCz+cTWjCpL/eZKzqbBD46nQ7K5bLUi9xHbVFFRxOJBEKhEFwul2TSzs7OUCgUJGCjmAjwef7IQuA6JNK5sLAglyQz8bdR0GUwSiXPV69eYWNjA4lEQmTWC4UCMpmMSDIzSKMCWafTEdBAbR/ATM9N6+MxnE41QCOI9Pr1aywsLACAqOFms1mZnzdv3gj9+aeffoLf70c4HJb7p91uTyx7TmENAKhWqzg/P8fm5iZOTk4kqFW/h99bLpdxcHAgGYNoNCoCWGazWXyCH2G8H3lekp5PGlqpVBLAFZgsEMf9SfYBBRBMJhMGgwEuLi7kjKLytPqHCoFGo1HAWgIzqpP8GA4+gaZer4dKpYJ2uy0smOFwKPcO6/0dDgcuLi6eTFl3dnYWLpdLBLY8Hg+63e6I8ijv+UgkIoFRPB6X9/I9NcPbmkajkdop3rsM1tRgh5lH+h5msxkulwtGo/FJ5oz+0ezsLEKhEDY2NrC6uopoNCqiIQRp1PGoPqMKQFK9krWL9KPsdjvi8ThKpRLS6bQAZpOkd95EG19eXsba2hpCoRAMBoOU4FCR9vLy8r829ZGZFHJZVZSSi4PO+3igBkzucFRpRjwc//KXv4haEZWSfv/9d/y3//bfcHV1heFwCJvNBpfLBb1eL7SPdrsNt9uN9fV1rK2tQavVihCJxWIRZatWq4VcLvfgDa+ilOFwGC9fvsTKygqSyaQc4ru7u/if//N/Yn9/H7lcbiTYUD9ndnYWbrcbPp9PLqVJZ1j4XRRjoUCI2WyW7x4Oh0LvYNDY6XTkeSbZS03NBtAJ+uWXX7CxsYFwOAydToeTkxNRyyQCrNPpBKm02WxIJpOymdPpNLa3t/Hp0yfJSN7XAf6ascaIPGrSM0ibYKYql8tJoOj1ekUchxxyBsMs/J60cS+7XK4RFUq9Xi+I6rt377C5uYlWqyWUncdGCXnpejweJBKJEaEiOkG3cRK5ftTPVf++q6nZR4/Hg7W1Nfz888+Ix+Ow2WzY2trC7u6uBLX3AVD47KSqdDodybTTSbjP+FlLzOJyh8MhVLxcLoetrS0JdEhl5v4glc/lcsHhcAj6z4BudnYWFxcXIt7Q7Xa/enmq94fJZEIikcBf/vIXLC8vw+/3Y2ZmBq1WC/l8Hn//+9/x6dMnnJ+fS7BKmt44GPSjsjvAF6YG6Y6vX79GJBJBJpPB/v4+/va3v+Hjx4+yf0wmExYWFuRMnZubQzwex+7uLra3t1EqlQRYfOh5VKlUcHBwgEqlgqOjI1SrVRwfH/8pSBtnVPT7fRwcHGAwGGBxcREajQZGo1HYKE+RKfiW0SkeD9RKpRIuLi4EhJv03cj9yfofBmr1eh3lchmnp6dIp9M4OztDsVgUMSu1frLdbsPhcEjtvNfrndgYv2d0yFlqQVEJh8OBZDIp2VQGaiaT6cGCQd8zldlBhpbf74fb7cbx8TGOj4/x+++/43/8j/8h4MzCwoLUNVG1ctIOOynU6tmittoYDAYSgDM5QTo8A7XHPJfUs9RgMCAcDmNjYwNLS0sivKH6vv1+f0SBWp0vrgvW9BKco4/H+6hUKiEcDks2a1LAscrWI9DNEpeNjQ3JVpfLZVxcXOD09FTETSaZqFDtWQdqahqVvT8cDgdKpRIymQza7bYETqTAWCwWkcG/yZF4yIvkIllcXJSi7Lm5OVgsFvR6PRQKBWxvb+Pk5ERkZTUaDfr9vqC9AOTiazQa6Pf7EtD1+33JrIXDYayurqJYLOL4+PhBiAEDDErwr6ysYHV1FV6vF8PhECcnJ9jb28P79++xu7uLYrH4p/5EaqDndDqxuLiIcDiMcrmMs7OzEbWbSRvfJeeRDnosFkM0GpXGx61WC8ViUfqv1Gq1iTgYKlIUiUSwvLyMjY0NpFIpWK1WobqyRuby8lKoV5x3h8MBi8UidY1OpxPD4RDxeBxGoxFer1cybKVSSbIW9xkrDxi9Xo9wOIxUKiVZX6JapAwdHx9ja2tL+O/9fh8ej0cyu263G8PhUOT8WWs5aaeDc7K+vo43b94gFotBq9WiWCxia2sLm5ubIobB4Fs9mNWM7yRpkKT58tLT6XSStWXA+LULQnU8WXND9F9d03cdq/qOqTjFoIfvdW9vD+l0GuVy+V5zodYGsN6CGS9K898FBKHzQ+drcXERqVQKDocD3W4XhUJB2Ajv37+XTBVpznRK7Ha7BGk2m00CPv53LBbDwsICSqUScrncd+vBVIofa6JsNpsE491uV1QJSW+s1WrSn5PvUq3ZfWgQfh9TKbDMDC4uLmJ2dhanp6fY3t7G1tYWcrmcMA0IbPV6PXg8Hniz9VB4AAAgAElEQVQ8HqnvrdfrE6NLcR5Yz8HG5Spl6Vu0Oqr+MfApFouYnZ1FOBzGwcGB1K08ZYCslhF4PB4kk0ksLy9LoFMsFrG/v4/T09OJM3toXGuNRgOnp6fCKqnX66IgvbW1JeCK2huVAQ/njSUDVFptNBq4uLgQP2ZSgg03GQONer0uKqTlclmyME6nEx6PB3a7XajGTyUqAnwRtaCvRuYMs83FYhG5XA7JZBLdblfmcNKmZufVM4tCO4lEAolEAm63W9bE3t4eSqXSo88Xx8Oep4uLi4hEIkKVb7VaqFarKJfLIrDCejqfzweLxSLnK3tmsu0O/SbStNkmRafTPTq1mPRxzm0oFILJZJIkAtfDY6uSPutADfhCdwkGgwiFQggGgzg6OsLJyQn6/T50Op1kOOLxuHQJVznBDw3WSG/QarWw2+1YWVnBr7/+isXFReH3X11d4fT0FJubmyN9lWZmZuRwHP9MysdXKhVBx3Q6nTQgBoDj42N8+vRJqGn3depmZ2dF3pgZC61Wi6urKxweHuI//uM/sLW1hcPDQ0GhVVUzzgH7xSwtLSEej2Nrawvn5+d3GtNdTeW003GiA/nmzRskEgnYbDYcHx9jd3cXu7u7InrBd/CQzAVRaovFgng8jr/85S9YW1vD3NycNOF9//49tre3BUAgys7CU2aiDAaDtHFYXl7G6uoq5ubmsLKygt3dXZjNZuzu7kqX+/u8b7UOKBqN4ueff5bAXL2ImaV69+4d9vb2xGHiwW+325FKpUT4hoEaKbyTMK4tj8eDVColNVZ2u11orx8+fMDHjx9xcXExMifqugQw4jRPql5MvRDZd4iUDEovf43apl6qpBCS4ke63H0zvmqgRsDC4/FgZ2cHW1tb+PTpE9Lp9IOUMFmn4Xa74fF4UKlUJFAjunuXejdm54LBoARqVqsVV1dXyGQy+OOPP/C3v/0Nf//73wW4AiAgENVAqfBqtVqRTCbx9u1bLC8vIxwOS6NvimKwh+W3jO+HYIRerxfmBgWhmOHrdDqigMcgm7VwkwQI7mKcG1LSE4kEfvrpJ4TDYRQKBRwdHeHDhw/49OmTBJd8buDzvmGzcQKi2WxWGCCTMFKeqH6pjl0dy/jv8N/ZhJl9E2dmZhCNRqWk4K7AwUNMnW+j0Qifz4fV1VWsr6+LmFShUBi5hx4DQOKaK5fL2N3dlVrJRqOBarWKT58+4cOHD3IPqhl9zjszBgTKSDNm0MSA5LEUqNVnocIe+4GpwmDsX/gUvTJpaokN6YUMFPmn3++jXq9LixAVRJz0OFXQnGcWs49LS0t48eIFotEo+v2+1Mt++PABFxcXj5bpAb6AcKQ8kq0VDAaF5dZoNJDJZJBOpwUUDoVCiEajMrZsNovDw0MRZCJDjWy5ZDKJVCqFpaUlKXN6TBsOh6L0yNrYYDAotWkM1OgLPGYLhGcdqHGxm81mKdSMRqPo9XqYm5uD0WgEAHg8HrjdblgsFhHEoCOhKi8x9XrXCeXGUNGLUCgEh8MBrVYrxbn5fH6ktmv8OcY37mAwEETm6OgILpdLClFNJpPI/DudTnQ6nZGx3GXcLIqdn5/HxsaGUNroMKpOVCAQkNoKHkwMeOlkLS0tIRaLSYaGz/VYBygvJIvFIsFHLBbD2tqaBGnX19c4PT2VYJPKhMDDUECV+haPx6Wuz+VyodFoSCC9vb2NbDaLcrkslyIP1larJXz1mZkZXF1diQx1p9PB3NycOK8MCvluiHre9hmYxfP7/YjH41hbW0MqlZJ0vZp1/PDhA7a2tiQDzEuG7/74+Bj7+/uYm5sTilksFsPFxQWOj48BPKzFAIMsSiAvLCwICtfr9XB2dobj42Nks1mpgeTvsC8UFev0er3UDvHgfMh75/ewPQSpYRqNRmhE481Y1fWvBmder1fk8m02G1qtFmq1mqhWqtLXtxmXRvOlNo1y53q9HqVSCfv7+/jw4QNOT0/vrHyoGn+Pgkdut1tEKOgU35Z6qvL96XSxxpU1Kul0Gpubm8hmszdKmfO/r6+v0Ww2JWOs0WiEmuT3+2EymeDxeOD3+6Xvopppvek5qdpYKpVweHgo+4fACoVt7Hb7SFPZSqUiFzUdN37mU5pKUY3FYtJnqdvtIpvNYmtrC6enp6jX67LHCbo4nU44HA7o9Xr0+32ZA5UVMqlzXb0H+ZnfAxHUYJK0KVJimVk1GAxShzkp+tP3jPdRNBpFKpUSdVAGTsfHx5LNeMx+qHxnw+EQ1WpVShbYmohKdOOBIrMFTqdT+gQGg0EYDAY0m03k83ns7e2hUChI7TKf+zGM+5BlC7VaDdfX1yIKRrrb9/bzJMYBQBpX8z5R/VBm3VlCEAgE5I5looB3BH0p9bMnYaybmp+fx9raGmKxmGTPz8/PJYtOFspjBmkqDd/pdIq4k8lkwvX1tVCdP378iHQ6jdPTU3Q6HanfbDQa8Hq90rOP779cLktNMH19glHJZPLJgDECwOPgHFk2Pp8PwWBQki6PYc86UKORGka+crfbFRGMbrcLn88Hh8MBnU4njYXZHZ50GwZsan3LbUxF7cfVzyhbS3nOfD6P09NTQQzVS+hr1CgWTWazWdhsNqyurqLX68FgMIw0q2S93W1Nzaww6EulUnjx4gWCwaAIb3S7XelzwcuH9B7SfYjKO51Oof0xUNbpdI/SiV01jeazVC4L3V+/fo2XL19Kf5jr62vUajVks1l8/PgRx8fHaLVaDx6TihRRVWtjYwPRaBTD4RCHh4dCy9vb2xMFx3EHlhQ3PsvJyQkKhQIqlQpyuRz+5V/+BT6fT+SzDQbDSC3MbfvXMMvCrN3r16/x4sULJBIJOBwOzMzMoFQqYXt7Gx8+fMC7d+9weHgoghgcM4tkM5kMNjc3YTQaRbUplUrh9PRULvSHzC3nV6fTSSDs9/uh1+vlcFdprAQreEgy8+zz+WC1WqWGgH0IJ7EmTSYTnE4nvF4vQqGQ1D8Vi0VpsjwOyqhoeyAQkPYNqVQKXq9XVKL+4z/+QxBwVb3yNnNmMBjg9XqxvLyMxcVFDIdDobG+f/9+JGtxV1PpgKwZcbvdODk5GakPvsvcEgxi4106XUSj0+k0tra2pMHt+FyoDiazXKTFn56eyjngdDrlOywWizhTN41ZzdRfXV2Jiiw/n4E1ad61Wg2FQkEAOQIIpOOpSpPq5z+mqWwPj8eDxcVFxGIx6HQ6lMtlbG9vY3NzE7VabaQ3osVigc/nEyqRVqvF5eUl9vf38b//9//G9va2tBKY5Nk+nhW47e/Q6DDRJ6DoBGXnnyrbotFoRLp9fX0d8/PzcLlc4guk02ns7Ow8aB9+z5jlIWikNtwdDAZCax0PhhngabVa+Hw+oXXNzc1JwMcAnz2kbgvK3GXs48bSBoLtrK01Go2w2WwjvSsf21gjRTAGwEizY6/XK6qLCwsL+Nd//VcsLS3BYDBIfSDbQ006u6bRaITZ9fLlS6yvr8NqtWJzcxOfPn3C4eEhjo6OpI72scGL8UAtEAjA6XRK6cL+/r6wJeibDQYDnJ+fiy9kt9tFMZcJAiZVeMcaDAaYzWYkEomJKYx/ywgeEFjlXW2z2aDX62G32xEIBBCNRpFMJqX+9jHsHyJQU1FtOnIsQm232/B4PAiFQjCbzahWq9Ios91uw+VywWq1yqJ9SGaK1ESqzwyHw5FO5mywfNvaIl5a5ODzUJiU1DezgKTWsZcOlbKIGLNJM50RqgdVq1XJTM3MzCAcDmN9fR2pVAp2u13QZDVAmaRxHtjHiQqby8vLSKVSgraN1yxNAmnh4cqsCNGrQCAAnU6H8/Nz7O/vC2o1ngUYvxjVv3n4aDSf1ZyYqeJlH4lEEIlEUCgU0O12b5Ud4njJ6WctXTQahd1uR7/fR7lclswF61XowKljZuPTcrmMTCaD+fl5dDodyYiwncQk5peBgNfrRTgchsVikZYMOzs7SKfTqNfrUtxttVpFUCKRSCASiYjsr9/vRzAYxP7+vmS6G43GvS5JOsBOpxOxWEwaxLJ+gnWIao0aM3DMvJMBsLi4iHg8jnA4LOI3vJiYPWw0GreeU9V58fl88ll7e3vI5XIol8sTEaThM5EWSKn7+84nQSNmQGdmZmQvqP3J1O+/6XOAL+AHM2yqwivPNZWS+K2Mmurw7u7uCgpMtTdezBqNRlQH2RD7/PwchUIB2WxWslAUVrpJPGaSpgbuRqMRbrdbGkmXy2UpEaBankajEcooqaeJREIK8y8uLvDp0yd8+vRJWoc8Vn8yNfD+3jMCX1BtUoZ5dkxKWe82xvmmqibp68lkUmotT05OsL29jVwu91XQYdLj4T5QM03qd47fRWTZuFwuJBIJpFKpkXstm83i+PgYJycnqNVqEx83gD8JlPE5SHE9PT1FIBBAMBiUs/ixapHGTaPRiJCNSrflnMViMbx69UraMayvr2NlZQVOp1OUqXd3d3F8fHwnVeDvmXqGMRvq8/mE2cSsOFWvydj5XsLgoWMiJZ3MEeosqOVAm5ubOD4+RqlUknOae7ndbsNoNEoWcjxjSk2HVqslz/QUmXONRiP1tKTmt9tthEIheL1eoQwnk0nJnDKLylKnSY3xHyJQY52P3W6XQkK32y0BgtFohMfjEeUwItREC202G3Z3d7G3tyfOxn0mUKVt0Jnt9/vI5XLY3d3F+fn5CFXwNqaiuvxdooZcyHzOuxgPaqvVOtKQ2+12y3cy4DQajYJIX19fS3EknQ4GYXRanU4nAEhjV2Y7JnkhcY7poC8sLODt27f47bffRJyDBzczDA6HA4FAQCg7Dykw5/fr9XrYbDYJJBigkma5t7eHarUq7/wmp2H8kOT6a7fbIjHrcrkwGAzw6tUrOBwORCIR5PN5lMvlW12WaqBGgYlEIiFoOSlNm5ub+PDhA/b29kSx8Kb3NhwOpaC3UqmM9CmZ1Dtm/xfSCgnCsEnn5uYm0uk0ms2mZH3n5uYwPz8vQVogEBDHPxqNolQqwe12Q6vV4ujoaOQZb2vq2iOthfPY7XZRrVbljFGzney3FgwGsbCwIIEyszsA0Gw2Ybfb4ff7pe6iXq/L994mICdoZDQaJTOVz+exvb0tdXyTKh7/WnBzX6PUMoEoiqqM79OvfQf/Xe2PRuRVrVG6675nJqHdboswEINqNg92u92i4JpMJmE0GlGr1VCpVKSx+M7OjgR741m8x3As1HpUt9st0tHM6FD1k845qUm//PILfvnlFwQCARiNRhQKBfztb3/D5uamoPGPWfP1tYCf/zYeXPBOpAAJz/fxgPgxTXVK1RY38XgcBoMBpVIJOzs7ePfuHfL5/MTvxG8ZAYdxiiPHrY6fdf0sY1hZWYHL5UK325V77ejoSPyZSdYjA1/k2MeDdVJbS6USjo6OBIxT/aKnMI6r0+mg0Wjg5OQE+/v7iMfjmJubw9LSktQqt9ttJBIJxGIxNJtNnJycYHd3F5ubmzg8PBQAd5J+Ed8vEwcEprj/qexZKpWEefBY5xDfi8lkQigUQiAQgNfrhUajQaVSweHhId69e4f9/X2p4eP4ySbjfaX6F+o5wHNIbdH1FEEa8LlFCO9UqtSyDQ79q2QyKSUNp6en0kN5knv/2QZqdDrZoZ78aXKXy+UyAEhPHTaJZld2UjmMRqNscBb83XfyGBQQxeNGJtWBfHT+7F2MQQcPxU6ng1arhVarda9AjZ9pNpvh8/kQDodhtVoBfKa2MbPIhW8wGEaEL9hSgMgAVaHYCJMqbVSUeohowfeMBxLH1u/3paiYB5BOp4PP58PS0pJkodjA9b6oMLMAFotFiuxZD1Gr1aTo9a70IDWTyh50Ozs7cDgcIogTDAYRDodxdHR0p/EaDAYpvqWkMZ3Qw8ND7O3tIZvNolKpfNOZHQ4/96JiPzq1LmmSNCg2ambQQkc5n8+L5HEgEJCCXhb1skaU2Q5ma9xutzhy3W5Xatvue8kzo6QGCGxXEAqFRM2VZ5DP50MoFBJhC5PJJE3kSTnjz1KO+j5UMLVlyezsrKDRXO8A7pWVV/cUqSbMJt33vavZaXUfAZDWHgQC7gNy0VlQHdL7ADREa1nHSdESm80mynN8r/F4XMQ3zGbzSLaZvf/YU5Hn96QDH9WZ4flItkSr1UKz2RTBAbXGdH5+Hi9fvsTS0pLUihweHuKPP/5AJpMZUcud5Hk+HkjcJMCiBhwMLNSMiwqS1uv1kbPpMSlxXL88m1dWVqSFiMFgEIrX9vY29vf3Jav9mE6l+v7Zy40ZfdaOq04uwV9SskmbDofD0Gq14ohub2/j7Oxsor3fNJrPrSAsFouUc/AZ+IelFuylptLfeK49Be1RBc9Zh/7hwwfMzMzInbO8vCw15Ha7HVqtVqjGnz59QiaTQblcvpdy8/fGBXw+q87Pz6WvrNfrldZLFAeigiaZJXephb6tqeAF36vJZBL67OHhIXK5nPjGzI7yd7km+e/jtHFmzvV6vQSiLpfrSXon0kdjuRTrJ3neMHlE+rXf74fD4RDxoEnaswzUuCEdDgfC4TCSyaSkF5lCPTw8hNFoxMLCgtDg2KCXk1itVlEoFHBycoJ8Pi+OO3C3w4eXi16vh8FgkKCGMq1nZ2fIZrP34qNzgVIcgU1cKWdKGejb1impn0nesBrg8HJj8fV4RoUbj2qJ7E9EyWrWPmQyGezs7Iiq1SSRNxrHRPENZi7HFYzoHEciEXHaisWi9FG7T/Cs0or4zulUs98Z1bAegpgNh0OhSwQCAeTzeanFjMViePfu3Z0+j+uUYyUSTdpjJpNBo9G4FTWOYjdcJ5M8fPjdbBnh9XphNptFbYwUCfYIW15exvLyMiKRCCwWi9CfKTPN90RK0nD4pVEuf+6270lF/Or1OgqFghzQPJAp/MKLhLRsFpprNJ9prZlMRpBVUtPcbjfMZrPss7uaukdZa0KRjmw2+6AsMp07cvBdLpf0hnrI/iaIwNYPBI0uLy9lfsdbgjzUVCf/e6YGfcyUkvpE6icV6Nj8NJlMjjA82H8wlUqJYlkul0M+nx/p6/gYVD3V+WHgRklr9kSMxWJIJpOYn58XsZXDw0McHh5KJvC2Z8N9jGuLYOdNTYEJLtA5VyXkgS/sGjqqBG7vcj/eZ9zM/judTlEaXV1dhcPhQKPRwPb2Nt69e4dPnz7h5OTkm/37JmXq+cMeaF6vV9Yj1zLweV7b7TaazSYsFotIqCeTSaHsqbV1FCgBHj6nDHK9Xq9QwRcWFkZozIPBAPl8HtlsVgSByBbgc1DU5CkylBzX1dUVjo6O0Ov1ZE+RAQJ8OS9qtRoODg7wf/7P/xERmW+1bnmIEXjd3NxEvV5HvV7HwsKClE2wjv/4+Fjun6OjI9EeGJf5n8R4uC/pe7Avr9on8SbGkfpvN7F6hsOhBPixWAzr6+uYm5sTbQQGa48VwHMMDMoZP/T7fald531OoOQx6iifZaDGF8cMSTQahdPpxMHBgSjsHRwcSFBG2XSj0YjLy0tRkqFYw9HR0b3TkeqCZjBFNJyXLwO2uzrtXARqw0z2qGLjZmYH7ouE9Ho9XF5e4vT0FM1mUxB+bh7VURxPjXMhMgOg0+kkU3VwcCB0T9XBuot9a67U98Tmofv7+yNBF/+sr6+LQEE8HkehUBipK3kI1ZV1RzyAiORw7tQeQPf5fB4CpJlSoIOZh7v2ZCHKowopUGCHafnbZgD57Kpi5SSNKKvL5ZLxklrY6XSkBmtjYwOrq6sSpDFwpyKXms2i0E+/34fX65X+encJ1Dg2XohU0qpWqyIsA0AkgtkE3eFwQKPRSGB8cnKCdDqNvb09tFotxONxmM1m6QPEdXWXzBfHxWw+aZg2mw1zc3PI5/MoFAri6H5rbY5nMuhYskk66agWi+VPNSV3NYIezE4bDAY550h/uSvbYZxKxb1JyjgzdHy+2342g2A10JuZmUG5XJYG7GytQsdI7e9DMRO1xi2Xy4kS52OowKngkV6vh8/nk56Iw+EQoVBIqMMU3yJDZX9/H7lcDhcXF4+aBWJPPNbJkTHDtUXhJq7vdrs9AlKqAiIARu5GNZibdBYQgGRT5ufnsbS0hKWlJXi9XlxfXyObzeLTp08j6poPBTZuYzybXS4XXC4XotGovF/2v1SzFRTqMJvNkhkOBAIwmUwCRNHPsVqt6Pf78h4eYmQczM3NYW1tDWtra1hZWRnJmvb7feTzeTlb+a7VNhkM1J4iqwZAsnoXFxfo9XqIRqM4Pz8XBUruE4pL5XI5HBwciJIv99KkjSBgPp8XCvDl5SUS/78cgIJ3Go1GRFgoukQVSFUldZJGMKZer+Ps7AwXFxcjrVy+9jzjnwF8OQ8CgQDC4bBoE3g8HmnzwbvjsdcEQVEVEIpGo0I15zy73W6cn59Dq9VOdEzPKlDjg7GOIRKJ4MWLF/D5fGi32zg7O8P29jYODw+lLuj8/Fxqpfr9Pt69eyfqYeTqMqC67wHODWexWAQxt1gsUvgI3O9y4CHF2hZmFdQie/Zqukual2Pp9/uoVqv4+PEjyuUyrFYr9Hq9KNjQmbtp/BqNBolEAi9evJA6MWaqSI/Y3d2V93CfZx+nwowb61BIi2q1Wjg8PBwZI3+PTW+Z9aQSGB3b+xqpb7x4VSdGRXPuE6CriBBRe6JS9+Hk82AOh8PweDzSs6tSqaBYLKJarYri0vfGSloixXiMRqOgs5MyBgcqqs5L22KxIPH/uf/r6+sIhULSMiCTySCfz0vTXErfr62twWazAYCsA5PJBJ1OB+BuAQYzapVKRdQkM5mM0N/Yr4qOA2kR+XxeCvFPTk5wfn6OUqkkTezVYIgIPdVjb/NOgC9Ib7lcxv7+PoxGIywWC9bW1lCv13F1dSVN2L+WUVIBDGbfWevAmt9YLIZIJCLZL5V+dh9Tg0Eaac2qY3vbz1ezLPwMZkFZV3kf6hHHdxMy2uv1UK/Xpd/P3t6eCAzMzc1JU9ZgMIhYLCYO8R9//IFOp4Pz83MJ1h4aEKmZX2ZLqtUqnE4nVldXkUqlRKmM6/T8/BytVgsLCwtSz3J0dDSiVvYYQZpGoxHRr2AwiGAwKDRg9WzlnLNeWq1H0+v1WF1dRTAYlLIAdayTDtRUVgUpj+wR5fP50Ov1cHBwgK2tLWxtbYl4xFNk0ki19Xq9WFhYkExpIpEQIJk/y3khwMiSCLPZLLLner0eHo8HGxsbMJvNyGazyGaz0nQYuPu8co2zfoqU23g8jmAwOAKEDIdDqZ8kxZ6BfLfbFV+IjvJTUiC73a6AfeP9snj2MyghxRDAowbq9B+YuaJybiAQQCQSEYogwcpUKoV0Oo3Dw0O5yyiu95CsH88gUgM5T/zzPbDwa881HH5uh+BwOLCysoLXr19jZWUFsVgMGo0Gl5eXKJfLKJVKIzoKt/l8dTy3BfDUPX11dSV+sNfrHVG6nJ+fR6lUkn66k6JnPttAjcWuyWQSAAShzmQyIhnLPg1q81k2H6YIA9G2SSBcavGmWgw//uJv+6wajUYUu9j7h4EanaKvFdvfxgaDAZrNJg4PD3F+fi6XIsVCxnvKcRGTKkn0n2gtmxbu7e0hnU4jn8/L899lo/O5VedcPbDHjaiemrZXKR8vXrzA9fU17Ha7BPmTUgMbR9cBCM3OYrHAaDT+CdG9zfPTKQEgGz0QCMBut2NmZkYybLcNjjj/lN1m0NpqtaS/ByWPv0VtomNCOl8sFoPH44HRaBQFxUldklzjzKbQAaGCnc1mQyKRQDQahdlsRi6Xw/HxMd6/fy/Fyb1eT1QpI5HIvQ7hm4yBTLPZRL/fl1YMi4uLsFgs0Gq1cDqdgvqzkHxvb09U387Pz2WfOZ1OueS5X5i1ymQyt3bu1Iwa+4+ZzWasrKyI09btdkeC63FhHX4P6+24jwgUMDvJWkDVWXpI8MNzRK13ZB0cVRrV/f21+VCzHNyHVCJVGyM3m02pnb3rWTDOLuD3Ug2Oar9skk31x/n5eWnmGgwGEQqFpOamXq9LRm5SiDbnlXXbuVwOZrNZss8cM8FMqti63W64XC5UKhXJAj0WTYvU8Wg0irW1NUSjUQEAqP7JZyESzXXLukE6/dFoFC6XS0A4to4AMCLgMklHnnVViUQCq6uriMViMBgMyOfz2N3dxcePHyVwf+y6NAYwZrMZ8XgciUQCKysrwjiIRCI3Ah4qA4Xzoyqnck+zrxpb3lAL4K6mnjWkDYdCISQSCVHpbTabcr5SFC4UCkm9PsdIQS+32y11lzxD1NKGx7Dx9gBGo1Fqgvl8KvuJVN2nELwYDody95RKJSnBOTs7GxHdUmvYbDYbTCYTgM9N2cvl8oMBWGZrqRJOsPAupSHqemGATp9obW0NP/30kwSf5XJZesWxFddty1vG74G7nhXD4ZfafdbSUwzM4XAgFovh6OhItDG+l028rT3LQI0S4y6XCx6PB7lcDvv7+8hms6Lixvopr9crBzdRRVKiGIQ8VNZVddq4EJvNphTvq93r+fPfekY10DCZTPD7/XLIsr6F6PZdAo5x52I4HAr17fr6+k9Nv8eDTB56zBawYNrv96PdbiObzeL333/H+/fv7y2cwgCFFCjOKy/km5ARFeEmgjwYDOQCdTgccoB+K+C7izGY6na7Ih1eq9XEsfR4PJibm0OpVEI+n5es57cuDV6UdI554S4uLuLFixdYWVkRtJjCHzwEbms6nU4uFKoUqg1Ev0XVJPpD+uT8/Dx+/vlnoRUz0zuJAmk6kLVaDfl8XuiO5KLzIrfZbLK3z8/PJWDa2dlBt9sVsSE1+CWowD8PaURM+svBwQF6vZ4Ei6zdpDNzdnYmUuhEVhkkMCBnxoOKkXa7HbFYDIeHh5JVu01mje+v0Wjg8PBQgANmSXw+H1ZWVkTueFwyGICATarojEpxZYsTr9cLAFLvq/aJvKv1ej1Uq1W5XEnJIepL6ivn7Y9ZxDAAACAASURBVKbn5rohnYq9bFwuF4bDofQ6YxH9JEGFrz0TmwvXajWcnp7i06dPUjNLZ3pjYwMulwu7u7si2lEqlR6kDqZm9tvtNvL5PP7zP/8TmUwGXq9XgBquu36/j1gshoWFBVxeXooYUrlcFuGISRrflcvlQjgcxqtXr/Drr7/C7/cLIMU1rzp0pETflIG1Wq1CmzUYDML6yOVy0h6HQfBDx869GwqFsLq6ilevXiGZTAqb5uDgQHpocn6Bx8lIckw6nU56N7GvJx1ag8EwQgVVa9h4P9KXIeBAAY/Ly0tUKhUBHCgU9pB1oQIqamsjZr0PDg6wt7cn5zT7Ufl8PvHphsMhgsEg3rx5A6/XK/1qc7kczs7OhCXyGAAD8Pk+jcfj0rNseXkZFosFnU5HaiyZWCAQ/tgBmmrqHqEyITP3zPiw32ggEMDGxoYAw1tbW/j48SMqlcq96/94TrdaLZyfn0v/1/HerF8z1Vej3xuJRDA3N4dE4nN/v8XFRUQiERiNxhFmXTqdHtEi+NbYea6o7CgAQqm9rc+oZvzGfU22bGHN2iRow7RnFagBo4WJWq0Wg8FA+tuoHH8iHVThmZ2dlQJwFYWbRJDG4ID9fijwoSoRqS/5exE0ubdWq1UaTrLPktVq/dPFfZv0qYqScxHysOacqRtCRTFVasRwOITdbkcymUQqlUI8Hpf+KplMBltbWyOUx7sg1czWkH8ejUbR6/WkN5UaRHwrWKOpND8qr3HjEdl6iBGhp1NzcXEBs9kswg2JREJqyogqqTz68XfCebdarSJ843Q6sba2hrdv38Lv92N2dlZqAClCcRejc8HvIyqpjuOmeVGzhXa7HeFwGIuLi1hbW5PaK4orPERFUf2+4fBLC4BSqYRmswm9Xo9wOCw9EdnInlSnarUqFwLpbqQRMivDOko1KALu70D1ej2cnp6KI3NyciLBMMWESqWSXBoEiNQgjYAEM/3X19ewWq0jBzszCLcZK9HUfD6P4XAoNX6k4Pn9fnQ6HXHGifiOZ9a4lun8Uj1Sr9fD5XLBbDZL3SCdpPsEagSIKBtNB5C8fpfLBZvNJu/ra2eoGkjyYqRAC6mqhUIBhUIBrVbrzuMcH/N4oMAzFfgChFFWnHWgOp1O6H29Xg/hcFiyaw6HQ0Ay9bx4yBhZl3dxcSGtLQhwsF6v1WpJfz+tVit0rVKpJOvxMZzdwWAggMTS0hJevHghrVUYKFC5Ud03BDHtdruAl1yfdChnZ2cRiUSwtrY2UsvLbMtDxs1MICnYr1+/xtraGiKRiOy7k5MTHB0dIZfLyTt9TLobfZ5IJIL19XW8efMGr1+/FuCPzePH+zgx6zhe/86zlNlWZqKLxaJkWyah/Khm8FQVRzrcxWIRpVIJ9XpdVFOdTqfcX2w67/F4cHFxgaOjI+k9q4Ixk6S80qm32+1YXFzE27dvZQykvhFsZqbKbDZDq9VOdCzfM9VXJFvh8vISAJDNZmE2m7GwsID5+XkYjUYsLS2J/zUYDES0q1Kp3KuJNM9I3rmVSgWXl5fodruyLvkeb/psFWBlu5mFhQWsra0JpZf3Q6vVQqlUQjabxc7ODo6OjqSlyvfALr5PKlPy5xuNhvQ+vi1Nk2cTBUT4zumHkCUwyTXwrAI1Tl6n00G9Xsfe3h6MRqMU5LNxJyeBThqRl3FHf1ITxc9hUaua3lZFJm5SUQRGa5iIYvv9fkGQyC9nRo0OLA8vft9t0roMAtXvVgMz9Xn4v1XKo8lkkkaeRDEuLy+RzWaltwo34l3mWP0Oj8eD9fV1/Ou//iuazSb+9re/YXd3V2gQ48E13yk/gwhdOBzGy5cvkUgk4HA4BFHiYaHS/O5qfFfdbleax/7++++4vr6W5pavX78WihjpQ5SzZwaTyD8zUjMzM9LI0+l0SkbQZrOhWCwKnWZzcxO5XA5XV1e32vScm2azifPzc4RCIVxfXwtFplgsIpPJCBVS/Vz1YrJYLFhcXMTGxoYognW7XRQKBezv7+Pvf/87jo+PHxz8qAcls1HZbBaBQAAOh0MCYgoQ9Ho9+Hw+xONxLC0tyWHp9Xrx6tUrLC8vw+12o9/vo1Ao4OPHj8hms7JWHzJW4EvmhFQHFvLTCWbjThVFVAP1fr8v3HVSNlSH3uv1SqaJa+9bcwd8oWEUi0W8f/8ehUJBaGW83EwmE2KxmPwuMyykw1KURaU2NhoNOBwOvH79Wn6XmeOzszMRQ7ntfPJdM+Djcw6HQ6FZxuNxpFKpkSaxanZdzcSzzyCR7tXVVTidTjQaDZyfn+Ps7AzFYvFB/eQIGDIgpzBPpVJBs9m8MYgDIIEwVd8o2rGwsCC0vRcvXki2ezgciuLlQ+8sqrQyi8wMOAFLKqImk0lUq1Xs7+9L5vExskFqwONwOARM4Lql000UnkGB2rsymUyOlBzQ0Wd/UYPBgPn5+RF0m3fmQ7Op7JH08uVLrK+vIxwOQ6/Xy/yOt8N4LOdcpYRTVfTVq1dyP5fLZVQqFakroyiV1WqFy+VCMpmUjM9w+FkgaWdnB/v7+zL/BIl4lhEYu287I+4PAkrlcllUHWOxmASPkUhEfL6ZmRlhRNXrdWg0GvEJmO3nGmdQMkmKK03NXMbjcWxsbGBjYwP9fh8HBwe4uLjA+fm59MokDd5msz0J5fFrxvNDzbRT7KZer8s9RBGZhYUFUfccr727q6mZJo1GI4wMiqyoyQzVJ9Xr9bBarSOg2+LiIpaWloRVNzs7Kz5oOp3Gx48fsbm5KWIq3zo3+V0MotbW1rC+vi7Z0LOzMwFhmY382prinUC/Y2NjA2tra6IAyj6Kp6encqbzvTzUnl2gBkACtYODA3Hk6LSqqBUpP1qtVuofblIxnISRAkWHjPK7jKzNZvM3+51pNBqpAfH7/UilUlhbW8PGxgYSiQTsdrtIdtORyufz0kz6tqgRgxgAI5mU7x1oROVVpykQCGB2dha1Wk1aIhCdHw/+bjN/g8EABoMBLpcLi4uL+O233wTJY2HoTWg9n5l/M/CJxWLY2NgYocqRenabeqxvmYpSkfbJtDmFFubn5xEMBhEIBASB5LM0m01Zn8ye0dkMh8OYm5uTlhIMnk5OTvD7779jZ2cHmUwG1Wp1ZCy3sWaziUKhgEqlgna7DZPJhHA4jFKphPn5eal748XId0MAwefzYXl5GW/fvpV1mc/nkc/nsb+/jz/++AO5XE4Uwu5rKhJ3cXGBXC6Hw8NDyeQQreQhq9Pp4PV6kUgkUK/X4XQ6RSqZjaVZR3dyciIKbONnxn2NKPDV1RUuLi5ufJ7xi5LGC7JarUKn00mPuLm5OXi9XhFDYd3TbedvOPxSy0MwIZvNijO+sLAAn88Hn88nVE2i5o1GA81mU6hiVK6l+BLXtdvtRrfblYxysViUS+4uxkCtXC7j4uICtVpNMncejwfxeFz6DjG7rvbyYg2d3+9HKBTC0tIS1tfXJQAimEARFe6d+/aSo3NMQR2bzSb3y010FpWVQOpZo9GATqeTTILdbofD4cDS0hJmZmakByWd5IfeV6Req1QeOp10enj2FAoFHBwcSOZRzf5O2ujcEGlmEF6tViUrxUytRqORGtVmswmXyyXUZmb1WT9Vq9UAQEQTarUaLi4uHvQc6j3r8/mwuroqwixku3AN3KQsOmlTwUm2W0gmk1hZWRFQ9+LiAru7u9jZ2cH29raAhcFgEIlEAk6nUzKWnU4HZ2dn+PDhA/7+979LsD7uH3zrPLut8YwiE+P09BRHR0ciGsK/gc/nq8ViwWAwQKvVEkf34uICMzMzcDqd0teMZxdrEidtGs1nWm0oFJLmxvPz89je3sbOzg6y2SxyuRxmZmYQi8Uk88ba5R9l434SAZGrqyucnZ1JKYdWq0UymUQkEhGxoVwud2f2jmo8M/n91JigNoL62TxrGKRR1ZFzHYvFEI1GZe1xr+/v7+P333/H9va2KCl/LzBWQWiKqvz7v/+71M0eHx8jnU7j5OQEZ2dnI3WufBY1wGTtfigUwsbGBpaWloTNwV6KBHO/RuG/jz3LQI0KMuw/w3qw8QPRYDBIgz02IVYpiZMcFxc9Zf+r1ao0jlxZWcHl5SXS6TTS6bTUeqlOhsFgQDgclgJPdrlnsTlT1qTu/PHHH4Ia3PYi4AXI/81D7FtBGp10nU6HaDSKFy9eYG1tDaFQCDMzM3Khb21t4fDwUC7Th84n/7ZarUgkEhJAkJajZmyIqLKmJRwOIxwOY2NjA6lUCg6HA9fX1zg9PcXW1hbS6TSq1epE+pgwoGg0Gshms9KPhg1vnU4nzGYzkskkotEorq+v5SKhQ8uWDqroCGuzSIcj8sRWEldXV3ce93A4FHpAvV6X7KTJZEIoFMLLly+lWTO/mz0HbTYbYrEYEokE1tbWMD8/D4fDgX6/j7OzM/zxxx/Y29uTzOFD+9nwd+lYZrNZ/PWvf5VgQaW00nmjqpLVasXl5aXsLzacZc+Wd+/e4fj4+F6B7vfGqx7cX/v/v/UZ3W4XJycn2N3dlXknhZeqkXcR6FF/hlk7Zq6Ojo6kXQEFgji/pVJJ1hmDUFVUgL1rCIBMYg9RgTSTyWB7extms1kAi4WFBeh0OjgcDgQCAamb4R4iTZg9BlkD5nA45JI8ODjAwcHBvWjZqqnOsdlsRjAYRCqVgk6nw/7+PjKZjIjzjNc4qDSvwWCAQqEgqO/s7CxSqRQSiQTC4TCWlpaEknrfur+vzTfX6WAwEJGGRCIBo9EovQHT6TRqtdpE2Sc3jaNWq+Ho6Eh6olL0iUqAPHsI/lB8wuVywe/3SwsLZsszmQyy2SwymYx8BwEAzvd9Mi1qYMssCc94smaAL5mKcXGgxzDuHbPZjGg0iqWlJYRCIZjNZgFO3r9/LwJqpVIJXq9XeuYtLS2JwAiD4u3tbamT5H0+Pv7bnGe3HT9B7mw2K+UqwWAQJpMJy8vLsr+oXtvv9+Wc+PTpkwiF9Xo9ob2fnZ0JE2BSzjB9IYPBAL/fL3WJZrMZ+XweHz9+xP/6X/9LgvTb0tR/pKljKxaL2NnZgc/nw9zcnNSDnZycwGQy3QrQ/973cL2SzUHRK6PRKPRK3uWhUEho4fTpSA/X6/UivX9wcICdnR35c35+LkHQXeZeLZdicoXnUDKZFKBQpberxowahW3YfkWr1YrflcvlRupVJ2XPKlCj0XFoNBooFAojaJ+K8pBSwUCNhaXjhckPMX4G6wAajYYU6LM2Ynl5WQ6kQqEglzcXLWuSUqkU/vKXvyCVSiEYDMqCZJBGae90Oo3NzU1sb2+PyKl/DzkAvgiCjP/7t36H0vDRaBRv374VARHWpRwcHGB7exuZTObBlDd+L50I1gCwx0u/3/8TIk6n0ev1wu/3Sx+Wubk5RCIREZHJ5XLY3NxEOp2+Fz3za6bRaMSZYn+Q+fl5LC8vY2lp6f+1d6ZPTV1vHP+GRUECBJKwiNCMWC2o0+lMO75oZ/q39021LkUMEiCQhWw3CdkQErbA74W/7+NJDMhyk1zs85lh0FbIueeee+55tu8j95N926h4xHVL7w3TSpjmt729jVgsBsuykMvlUCqVRMnwurWVjCqa108FJaaO0UBgjxO+OCloMj09jYmJCTGiMpkMgsGgeF9bpaded14ZBWAEkU3eDw4OxMHBz2NE4ocffpCIBRsmZzIZxGIxxONxbG1tYXt725aUx+bxXvf3cR3SUBseHsb09DRmZ2floJrP5+XldpV+QabRy3RGRoAZzWVEyvRIU5jJvC6mlPLQxKLom3rYzYhaf38/1tfXZWyTk5MIBAKYmJiA1+vFzMyMHMg4bqYL+/1++Hw+UWE7Pj5GpVJBLpfD5uYmwuEwdnd3b5wSZUqL+/1+LC4uYnx8XGqmEokEUqmUqCU2Z3PwRc/WGACkPx0P2w8fPpQDKZ8rOxxLJjyc0FAzm8pHo1FxeLQDvv/K5TLi8TgCgQCKxaKkM3s8HpG9N/tOma0i2F6DjtJUKoVgMIjV1VV8/PhRDmtmhOumKVymMiYNNTNSQrn2Zrl4u+fRTBGjU+PHH3/E9PQ0BgcHUSgUsLy8jH/++QevX7+WeZudncVPP/2Ep0+fYnFxEf39/ajVakgmk3j79q3s5VR0bJehbnJ0dIRkMomDgwPMzMzg0aNH0rqC654HZLY3CYVCePnyJUqlEgYHB0V8iucEuxU2zcglm9o/f/5cggahUAivXr2SPaA57a4daZg3wdzX6cyoVCqy91DVd3JyskHN8ibXwfU6OjqK6elpySQhR0dHEtlfXFwUoZCpqSnpT8ezIXsAb21t4eXLlxL5umozefM5Yo9AflGpkem+psiema4JfN7TmUJJDY2enh7ppczMIBpqdj5TjjTUCG9G88PASeek9fT0oFQqNXhZAHsPaAzhU+iBktD08HIjHx0dldRIRqoGBwcxODiIQOCzkqLX65UHZ39/Hzs7O4jFYlK/wtQoM0f8Mtdy1ZcFF6LP58PMzAyePHmC+fl5SceLxWJ48+YNVlZWxFN/3RcSr4EyrplMBpFIRLziT548gdvtxqNHj6Q3hrlx8qHyeDySlkWvCxUSP3z4gGg0ilKpdKV5u+z4KRhSKpWkFiKRSGBtbU2K9ClqY0ZUmY5KLxyFJ2jwN6sTXsdDyM86ODgQye14PI56vS7F+5SYHhgYwMOHD0WZcHBwUKI6ExMTGBgYkAhlOp3GysoKNjc3pdeS3S92HuJ3d3cRj8dxcHAg4jXmJs6NkYZzNBqVdh35fF7qrhhNtTOqbgcul0u8wr29vRgbGxND+tmzZxgcHMTU1BRSqZT0iWNListG1+gVpRFsii+xZolpUKbwEPDloMJ1zK/zoohXhS9gpqcy5ZHeXV4/04729vZkD6VRxxROOs2oBLq2toa1tTURU7iJl53RLwp0cL8PBAIYHx/HH3/8IUqwNCiz2aw0xm0+vPf19UkNMvcJAFLfTKl5u+aZn80akLGxMQT+n8VxdnYmje+5T7ZbAOPg4ADZbBbhcFjUZGdnZ6UQn55t4IujkQ4cNnWnwMXy8jKCwaC0vjCvt5WS8VXg/eJBkgdZOoIplsAWHIlEArVare0RNaoKMu2XfQ0pFMO0dd7jhYUFLC4uwuPxoFarSX35+vp6Q+2z+RnthL+fNeThcBhutxu//PKLCDsAEKn5aDTa8M7hu5HGOu+v3UYa050Z7WFGB6P7LtdnYRaKc1ABlKmdbAfiFLi3sDcr6yqpnDg6Ogq/3y8lBEzFpoP3KlC9tVqtolgsilP1wYMHqNfruH//PvL5vLSOGRsbE4cwgy2mYjPPVolEAqurq+LIvWopi+kgpUrymzdv4PF44PF4ZA82BdgANKiuN38eayV51mBZ0IcPH5DNZqX+304ca6hddDN4ILlz506DoZbNZqXhYLOoxk3HwqJYGmp9fX0IBALSOZ2Sw7OzsxJRY02aWWNz//59AJBDeiqVapD5ZWNictXN6CovKdNQW1xcFEOtXq9Lusrbt28RDoclLHyTzZGGGg2JSCSCwcFBkRWn8cDNmQdMpofRw8+Hy8xd/uuvv7C5uYlkMmlLYf554+fLsVgsIh6PSxif+enNh1/gi3edXl9GM1rVVHJtX3d8rHOyLAvxeFzWHmXYR0dHRW3TrGFkzjjV2Pb396Vn2crKCiKRiERT23Gw47zu7+8jkUiI8T0/P4+FhQXx+vGQWy6X8fr1awSDQeTzeRSLRXlZcC6clpJCQ401YgMDAzg7O8OLFy/w888/S57+8vIylpaWRGjiKp7jZsOLhx/+neuLxoj5b83/bxpp5v+/CTxMV6tVpNNpDA8P4+nTp6hWq9Lfhy0BeFDn88Tx0NPK/X59fR3v3r3D2tqapLu2egavgmmo5XK5BnXfP//8E8+ePRNhC0bFaSjyIMT5YjSdzcOZRndyciLvB1MowQ7MQycL3+fm5jA3N4dUKiWGWrlcvla94WXh2mE2wsbGhjQMr9frmJ6eFsXTkZER2Xvo3WbT3FKphI2NDelZtrq62vJZv0nE24SG7dzcHKamptDT0yNKj3QKhEIhJJPJGyuLXoQphDA0NCRtfBh1YM1sT08PRkZGsLCwgN9//12iJcyuiEajePXqlWTFmOmOndwjKcS2vr4u6Yzz8/OSVrq7u4tCoYBoNIpgMChOVzPF0HTe24mZ7sxI6vj4OIaGhmTNulwueDwezM/P47fffkMgEJDUcLM3bTsN98teCx1EbrdbjBI6MtnKg2ubacZut1vSDYHLP0c8H9frdenpNj09jTt37mBmZgY+n09KQur1Ou7evSutiag2zM86OTlBpVKBZVlYX1+XVjzMkLnqvTd/797eHmKxmGRJ+Hw+3Lt3T5yAo6OjshdT8bW5RRYz61iywmDG0tKSpCLzvWUnjjXUWsGHiapQZgEnayza7dGg16BcLou0MBv0UaGGh2AzwsL6tmg0Kos5kUggHA4jHo+Lh7MTec98kN1ut8i3Pnv2DFNTUwCAXC4nPU7oubyuKAcx07PMNAx2mKe3+e7du5JixHEyVYY1Lvv7+5IWm8lkpH6OHmLz89oBfzcVqKgwed5LhAdcU5q41YZux/xyQ4pGoyJY8OTJEzx48ABTU1PSxoCHXvNnaShtb28jHo9LcTrTDW5al/at8ZviJtVqFZZl4fj4GIVCoWVELZFIIJ/PS61d81w4Ed7jer2OXC6H9fV19Pf3S5SLqSJ2CCPxZ8wo27fWmGmYmY2U2aT5pnNLg2Rvbw/pdBrBYFBk1ll7MzQ0JMYW93UabtwvWM8ZjUYblOvsjvQyk4J74uTkJPr7++H3+6Vu7tGjR5K+aj7bAMQQGh8fx8TEBHw+n/S7ogIn66/tTp1iZHJ2dhZerxe9vb3I5/MSqeikQp3L5RJRqsPDQ2SzWTkssY6SbTjMuj32LrUsC5ZlIZ/PN0RUzN9vfr/uGF0ul4hqsSG3ZVmSVhsOh0V4gAfPdhkOhFFuGv38u9/vx+PHjzE1NSXCBqz1ZCo4RUY2NjbE6dANIw34Mr/ValXq8M1oRqVSQalUQiQSgWVZDT1zzZ9v17hpqPX390url76+Pvh8PpycnODevXsIBAISuWTdFaMpzAbpNqahNjAwgEAggMePH6NWq+Hg4AAej0cUN1mWw951zBa6qpHGs1Amk8HS0pLcN2bDUPGVDiSW2/BdSAd+oVDA5uYmIpGIRNTYSP4mzzgdfOx5y/cMv7PdC50Gbre7oe0T1yDLQT59+oRUKoVkMomtrS1kMhnp39iOFNhbZagRFp0zkkFPtRkWb8fDbL64zVoPs/nq8PBww793uVwol8sNqVksfI5EInj//j3S6XTDeNulvkU4R6aK1PPnzzEyMiIF8CsrK9jY2GhoenlTLwEfSjYOZh+Xcrks6lozMzPyUHOs/LlqtSqiGyyMjsViSKfTcqi3Y5zfugbzXtH4YtTiJr/LjrFRBZAvu52dHVEyam7KadZ9cn6pCPbvv/8iHo8jHo83KPC1i+a5oDedBdCtPtuM8tjlTe8ELtfntEQamZTYHxkZwcjIiPSGsUNN7rprzPTYM0J72VYRF42FTemp5ru8vIxqtYrFxUXU63URGTBTNSkVXqvVkEqlsL29Ld5WjosH0HY8+2y2DkBStdgiYnh4GJOTk2KkmepnvGYerJmK1NPTI2nv7Btn9wve5XI11KaNjY2hp6cHuVxOGjSbB5B2wjXId6ZlWVhdXYXH4xFVUq/Xi/7+fpkTqtHxOw+ZN3UYfmucjDCwDrKvrw/5fB5LS0tYXl5GKBT6Sh2v3XsOI8psGE0HMNUw6chkxoRlWUgkEvjw4QP+/vtvJBIJFItFcWR04p63wjTUTk5OEAwGUSwWZTy83+VyWVRhKfDRyTHSiDANNabsnpycSO0kDYtIJILl5WVEo1HbGhzfFNZaMmX7xYsXsndzPTGKxn6+Y2NjUg96Ffg+Oz09RSqVQqVSwdnZmRg5zJZgBgl/hj93fHwsqY5bW1tYWlpCMBgUBWLz399kPti7z7Is+e+M7NFg47M1OjoKn88Ht9v9VUSN/R9TqRSy2SwqlYpEXNvFrTTUAEiYlSppnDA7VYBaQUOtWCwiGAwCAHw+H7xer9SicVHx0MwmuWwuyjA5RR/MepFOHDS56U9OTuLx48eYmJiQVEJKm3/8+LGtOfh8ONPpNACICqLP52tIceOhh4XENHoZZjbzqruVcnCdA1a7x0klulQqhb6+Prm3rFczoXw8PZobGxvY3NxEuVzumrJVs4peqzk2U9xug4FGOFaq26bTaVSrVWmgSal8NnPtxLWZUWLWZgWDQaTTaSSTSSSTSezu7tq2R3EfzWazspen02mJsHAPZb8stkXZ2dlBLpf7qo63XQ46OueYkrO1tSURoVgsBq/XK82EmS5z7949uUbziy1karUaYrEYwuEwtre3RUXVjvGbdYZerxeBQECa3bLeq1AotKXB9bdg1NDsP8aa2qGhIVHUZSNw3vPDw8Ov6n7bSS6Xw+rqKj59+iTiRCsrK/Kcmopz7cwwAD7fT9Z2xeNxrK2tSVre6emp1Pex4XWhUMDGxgZCoRDW19eRTCblXNRtzKyao6MjcSICkPpvlge0q0faRWPje5AR+0wmA7/f31Cn63K5UCqVkEwmJfMkFApJ82W7RSSuey3A5zNApVJBNBqVNiPj4+NSSsJ9ns3bWd5y3ewNPtv1el2aUg8MDEh2FPClhQjr+ngeLpVKiMfjMq90LJh9k+2ADjHzi30ZaWz19vaiUCiImJV5zmBk7vDwUGoXr9I+67rcSkONVjhTKRhyZcpQuwweLsajoyMUCgWJOkxPT8vm6fV65cbyRjJNh15UGhWsZWPBYrsx03GYFrOwsACv1wsAUpjPhoKUU7UbelLYJ6VSqWB7exsrKysNtV4mfKnXajXxzPLg0+6H5Ft0e2M24eGBHiQ2tly3KwAABf9JREFUKI/FYnj//j0GBgYAfJ02xM2T6qmsjeqW8Wsegs57YTtp3q8DjSKKoXDfYlTGrBvrBOb+lsvl8PbtW1GKrFarDb337Pis4+NjqZVKpVJwu90Nhg5TH/l1cnIiNXfcO9sZYTFhf8tIJCI9Bf1+vwgbeb1e+Hw++TvwdcSXUSWm96ytrSEWizXUId4Es85mYGAAXq9X+umZxe/mgbKTz5D5eZxPimm1upZmOhEJOjs7k55Sq6urGBoaQqVSkcwSuxRvrzIepoWFw2ER0uFeQYfmycmJNO9dWlrCu3fvsLOzI47gbkbSmmH0heuRdNIwawUj+GzEPjs7C5/PJw54Pl9sDxEKhRAKhaRGupvOYhN+PjMN6GTgXjA7Oyvqr4eHh4jFYpKBQ7Xq6xprNGKy2SzW1tYwNDQkrZ6AL0Z6sVjE1taWRHsLhQKy2Sx2dnYa6nztftaaa7gByPul1fVcRHN6cju5VYYaJ46pZqlUSv5uWZbItrY7j5kHrEKhgP39fezu7iKXy0nqIxc6DxXsWcQDhplv2+mDGD+TG9LGxobULeTzeViWJbLNdsnbX4RZt7W/vy8ejIv+LYuSeS2drLW4TfDQTY9RpVJpKRzAuiFzfs32Et3me7633E8YUTFrybqxrjkeZg3QWDKjVnZiPv/ValWiA+Z4qAJortFuKHpSkZAvdgofWZYlaasejwder7dlFJjvBMrjU/zkul7s86BXmMZQLpeTmuhsNmu7KvJ1aPZqN0cU+QyYX50aF2vduV8yukenZaff17yP29vbIkYUDoe/UqrL5/MoFArS8qW5VZHTMPc94Ot73knMaB/T2t6/fy/9cu/cuQPg8/qg0itrlOjEcto8M6DBzJjT01PJYopEIuIsTyQS0nDeDse8y+WSbLePHz+iXq9jbGysYUx7e3vIZDLI5/OSYfbp0yfs7+83OCjbOZ/meZh7tfn9vEwe/mwns+BcnfRiTE1N2fJhnCDm/QOQ9IiLJtduuJCY19qsoMUwLw8i7SgYvw5cZPQUcexMSWH6QTvFI84bV6s/k+aH6KJ/q3yNeQhqRfP86rx2jlaH+m7zrYhmuz7TvPbmz3bC/gk01p4xA4B/5qEOQMNBlKlV9CqbKYB2Xdfp6ak0jJ6bm8Ovv/4Kv98vaVqxWAypVKorh+FWXHTdTsqQ6Pa6o8o1hS7MBtzEdMqZNa5Oxmn7nnmmoyphs4osHcXNCq9OhdfE9cPvwJcIV61Ws+38zEhYb2+v1IA1r1XTIcxzMr93ew0Al3/ebzpWy7Iu9QtupaFGWqWWdJJWlngrzjsgO2VBnifK0MnxXWYuuzW22855Bm4zOr+KSac8m/ws8zNJqz2z22uzlff1qpjXYef1nJ2dibHo9XoxNzeHoaEh6fNGcY5uz6HTMd+L3YryXDSmi2jX2vqv8L29L791tmrnXnSd87GT59Ju/hOGmlO4zMbpdJwUofpea5KcgBM92Ipi4jQv+7e4rqHWLsyoACMCZvuH2xBpcQpOzCz4Hs4bt4Xv7X15nqHW6c/s9BicymUNtVtVo+ZUvodF5qRrcNJYvjd0bhWnc9vWqNPGy5ov9ng0U5GYxuXkVC0n4bR7CzhzTN8r39tcd+N6vrc57AYdjagpiqIoiqIoiqIo36b7km6KoiiKoiiKoihKA2qoKYqiKIqiKIqiOAw11BRFURRFURRFURyGGmqKoiiKoiiKoigOQw01RVEURVEURVEUh6GGmqIoiqIoiqIoisNQQ01RFEVRFEVRFMVhqKGmKIqiKIqiKIriMNRQUxRFURRFURRFcRhqqCmKoiiKoiiKojgMNdQURVEURVEURVEchhpqiqIoiqIoiqIoDkMNNUVRFEVRFEVRFIehhpqiKIqiKIqiKIrDUENNURRFURRFURTFYaihpiiKoiiKoiiK4jDUUFMURVEURVEURXEYaqgpiqIoiqIoiqI4DDXUFEVRFEVRFEVRHIYaaoqiKIqiKIqiKA5DDTVFURRFURRFURSHoYaaoiiKoiiKoiiKw1BDTVEURVEURVEUxWH8DwQfXPkBtHEmAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "path = os.path.join(os.getcwd(), 'data', 'ex3data1.mat')\n", "data = loadmat(path)\n", "y = data['y']\n", "X = data['X']\n", "\n", "# Choose a random sample of data to display\n", "sample = X[np.random.choice(X.shape[0], 20)]\n", "plt.figure(figsize=(15,15))\n", "plt.imshow(sample.reshape(-1, 20).T, interpolation='bilinear', cmap='gray')\n", "plt.axis('off')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The parameters/weights of the neural network have already been provided, let's load these weights and inspect them" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(25, 401) (10, 26)\n" ] } ], "source": [ "path2 = os.path.join(os.getcwd(), 'data', 'ex3weights.mat')\n", "weights = loadmat(path2)\n", "Theta1 = weights['Theta1']\n", "Theta2 = weights['Theta2']\n", "print(Theta1.shape, Theta2.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, we define our activation function $g(z)$ which is just the sigmoid function from logistic regression" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "def sigmoid(z):\n", " return 1 / (1 + np.exp(-z))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's define the predict function which uses the pre-trained weights to predict on the training data using the vectorised implementation of a Neural Network with 2 hidden layers." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "def predict(Theta1, Theta2, X):\n", " m = X.shape[0]\n", " n = X.shape[1]\n", " X = np.insert(X, 0, 1, axis=1)\n", " a2 = sigmoid(X.dot(Theta1.T))\n", " a2 = np.insert(a2, 0, 1, axis=1)\n", " a3 = sigmoid(a2.dot(Theta2.T))\n", " h = np.argmax(a3, axis=1) + 1\n", " return h" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, predict on the training data and assess the results" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "accuracy = 97.52 %\n" ] } ], "source": [ "predictions = predict(Theta1, Theta2, X)\n", "correct = [1 if a == b else 0 for (a, b) in zip(predictions, y)] \n", "accuracy = sum(correct)*100 / len(correct) \n", "print('accuracy = ', accuracy, '%')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the next exercise, we implement backpropagation to train the weights of the neural network." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" } }, "nbformat": 4, "nbformat_minor": 2 }