{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "

# Neural Networks Demystified

\n", "

## Part 2: Forward Propagation

\n", "\n", "\n", "

#### @stephencwelch

\n", "\n", "|Code Symbol | Math Symbol | Definition | Dimensions\n", "| :-: | :-: | :-: | :-: |\n", "|X|$$X$$|Input Data, each row in an example| (numExamples, inputLayerSize)|\n", "|y |$$y$$|target data|(numExamples, outputLayerSize)|\n", "|W1 | $$W^{(1)}$$ | Layer 1 weights | (inputLayerSize, hiddenLayerSize) |\n", "|W2 | $$W^{(2)}$$ | Layer 2 weights | (hiddenLayerSize, outputLayerSize) |\n", "|z2 | $$z^{(2)}$$ | Layer 2 activation | (numExamples, hiddenLayerSize) |\n", "|a2 | $$a^{(2)}$$ | Layer 2 activity | (numExamples, hiddenLayerSize) |\n", "|z3 | $$z^{(3)}$$ | Layer 3 activation | (numExamples, outputLayerSize) |" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Last time, we setup our neural network on paper. This time, we’ll implement it in the programming language python. We’ll build our network as a python class and our init method will take care of instantiating important constants and variables. We’ll make these values accessible to the whole class by placing a self dot in front of each variable name." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our network has 2 inputs, 3 hidden units, and 1 output. These are examples of hyperparameters. Hyperparameters are constants that establish the structure and behavior of a neural network, but are not updated as we train the network. Our learning algorithm is not capable of, for example, deciding that it needs another hidden unit, this is something that WE must decide on before training. What a neural network does learn are parameters, specifically the weights on the synapses." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We’ll take care of moving data through our network in a method called forward. Rather than pass inputs through the network one at a time, we’re going to use matrices to pass through multiple inputs at once. Doing this allows for big computational speedups, especially when using tools like MATLAB or Numpy. Our input data matrix, X, is of dimension 3 by 2, because we have 3, 2-dimensional examples. Our corresponding output data, y, is of dimension 3 by 1." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Populating the interactive namespace from numpy and matplotlib\n" ] } ], "source": [ "#Import code from last time\n", "%pylab inline\n", "from partOne import *" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(3, 2) (3, 1)\n" ] } ], "source": [ "print(X.shape, y.shape)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "class Neural_Network(object):\n", " def __init__(self): \n", " #Define Hyperparameters\n", " self.inputLayerSize = 2\n", " self.outputLayerSize = 1\n", " self.hiddenLayerSize = 3\n", " \n", " def forward(self, X):\n", " #Propagate inputs though network\n", " pass" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Each input value, or element in matrix X, needs to be multiplied by a corresponding weight and then added together with all the other results for each neuron. This is a complex operation, but if we take the three outputs we're looking for as a single row of a matrix, and place all our individual weights into a matrix of weights, we can create the exact behavior we need by multiplying our input data matrix by our weight matrix. Using matrix multiplication allows us to pass multiple inputs through at once by simply adding rows to the matrix X. From here on out, we'll refer to these matrics as X, W one, and z two, where z two the activity of our second layer. Notice that each entry in z is a sum of weighted inputs to each hidden neuron. Z is of size 3 by 3, one row for each example, and one column for each hidden unit. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We now have our first official formula, $z^{(2)} = XW^{(1)}$. Matrix notation is really nice here, because it allows us to express the complex underlying process in a single line!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$$\n", "z^{(2)} = XW^{(1)} \\tag{1}\\\\\n", "$$\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have the activities for our second layer, z two, we need to apply the activation function. We'll independently apply the function to each entry in matrix z using a python method for this called sigmoid, because we’re using a sigmoid as our activation function. Using numpy is really nice here, because we can pass in a scalar, vector, or matrix, Numpy will apply the activation function element-wise, and return a result of the same dimension as it was given." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "def sigmoid(z):\n", " #Apply sigmoid activation function to scalar, vector, or matrix\n", " return 1/(1+np.exp(-z))" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3deXxU9b3/8dcnk42QhLAGZN8REFQWW60VXHGptta2elt7a/Xnj161q7X2Wn+17e2ubb1eb71e9Wo3qbe1ipZKrYp1Q0EFZRMiIASQsIbsk5n5/P6YkQYMZEgmnFnez8djHpkz58zk/c3yzsmZs5i7IyIimS8v6AAiIpIaKnQRkSyhQhcRyRIqdBGRLKFCFxHJEvlBfeJ+/fr5iBEjOvXchoYGevbsmdpAAdFY0lO2jCVbxgEay3teffXVne7ev715gRX6iBEjWLp0aaeeu2jRImbNmpXaQAHRWNJTtowlW8YBGst7zOydQ83TJhcRkSyhQhcRyRIqdBGRLKFCFxHJEip0EZEs0WGhm9l9ZlZjZisOMd/M7N/NrMrM3jCzE1MfU0REOpLMGvr9wJzDzD8XGJu4XQ38suuxRETkSHW4H7q7/93MRhxmkYuAX3n8PLyLzazCzAa5+7YUZRQROaqiMachHKGhJUJza4ymcJTmSJTm1igtrTGaWuP3m1tjNLdGaWqN0tIapSUSIxyN0RqNEY7EaI16fDoSf+y96ZJIC92xS30qDiwaDGxuM12deOx9hW5mVxNfi6eyspJFixZ16hPW19d3+rnpRmNJT9kylmwZBxz5WCIxpz7s1LWS+OjUheOP1bc6zRFoinjixv6PzRGnOdp94wAYXurd8n1JRaFbO4+1e9UMd78buBtg+vTp3tkjpXTEWHrSWNJPtowD/jGWWMzZXtfMttpmttc28+6++G17beKxfc3sqg9T1xLp9Ocyg9LCfEqKQvQoCFFcEKKoIERxfh7FBe89Fr8fn5dHj4IQRfkhCvPzKAwZBaG8+O2g6cL8PNauWNYt35dUFHo1MLTN9BBgawpeV0RyVCzmbNnbRNWOejbtamTjrgZeW9vMv732LJt2NxKOxDp8jVCe0bukgN4lhfTpGb/17llIn5JCKkoKKCvOp6y4gNKifEqL8ylLfCwtyqdnYT55ee2tq6ZG86ZQt7xuKgp9PnCtmc0DTgJqtf1cRJJV29TKyi21rHm3jrXb61jzbh3rttfREG5vu0c9AP1KCxnUqweV5cUM7FX0j/uJ6X6lRZQXF3RrKaejDgvdzB4EZgH9zKwa+DZQAODudwELgPOAKqARuKK7wopIZmuNxli9bR/LNu9l2aa9LKvey/odDe0u26+0iHGVpQzv25PhfUuo37aB8z48k2F9SygtCuy8gmktmb1cLutgvgPXpCyRiGSNaMxZubWWF9/exYtv72LJht00tR645l2Yn8fEQeUcO6ic8ZWljB9YzrjKUvqWFh2w3KJFm5l4TPnRjJ9x9GdORFJqb2OYZ96q4W+ranhu3Q72NR/45uTIfj05YWgFxw+r4PihFUwYWE5hvg5aTwUVuoh02fZ9zTz+xjaeXPUuSzbuIRr7x45uQ/v04ORR/Th5TF8+OKovA8qLA0ya3VToItIpdc2tLFy5nUde38ILb+/EEx2en2ecMqYvZx5byRkTKhnWtyTYoDlEhS4iSXN3llfX8quXNrLgzW00t8Z3HywM5TF7Qn/On3IMp43rT68eBcEGzVEqdBHpUHNrlPnLt/Lrl97hzS21+x+fObIPHzthMOdNHkSvEpV40FToInJIdc2t/GbxJu59fj0768MAVJQU8MnpQ/n0ScMY3jc7LtqcLVToIvI+exvD3Pv8Bu5/cSN1ib1UJh1TzhWnjOSCKYMoLuieIx2la1ToIrJfc2uU+1/cyH8+U7V/d8OTRvbhmtljOHVsP8xy68jLTKNCFxFiMeePr1XzsyfXsq22GYBTxvTlK2eOY/qIPgGnk2Sp0EVy3IottXzrkRUs27wXgGMHlfPNcydojTwDqdBFclRdcys/e3ItD7y4kZjDgLIibjx3Ah89fnDOndQqW6jQRXLQ8+t28vU/LGdbbTN5Bp8/ZSRfOWssZcXa9TCTqdBFckhTOMqP/rKaB156B4CpQyv4wccmM+mYXgEnk1RQoYvkiBVbavnig6+zfmcD+XnGl88cy9zTRpMf0omxsoUKXSTLuTsPvrKZWx5bSTgSY3xlGbd9ciqTB2utPNuo0EWyWEvE+dpDy3n49S0AfPqkYdx8wUQdGJSlVOgiWap6TyPfW9xEdX0jPQpCfP9jk7n4xCFBx5JupEIXyULLNu/lqgeWsrPeGdWvJ7/8zDTGDywLOpZ0MxW6SJZZ8OY2vvL7ZbREYhzbJ495/3KKzoSYI1ToIlnk/hc2cMtjqwC4dMZQzuy9S2WeQ7S/kkgWcHf+/al1+8v8xnMn8MOLjyNfR3zmFK2hi2Q4d+ff/ryae5/fQJ7BDy8+jk/NGBZ0LAmACl0kg7k733pkBb99eRMFIeP2S0/gvOMGBR1LAqJCF8lQ7s53HlvFb1/eRFF+Hv91+TRmjR8QdCwJkLahi2Qgd+cHC1Zz/4sbKQypzCVOhS6SgW7761r++7kNFISMX37mRJW5ACp0kYxz/wsb+I9nqgjlGXdcdiJnHFsZdCRJEyp0kQzylze38Z3H47sm/vjjU5gzeWDAiSSdqNBFMsSSjbv50u+X4Q5fP2c8l0zTeVnkQCp0kQxQVVPPVQ8sJRyJ8ZkPDONfZo0OOpKkoaQK3czmmNlbZlZlZje2M7+XmT1mZsvNbKWZXZH6qCK5qbaxlaseWEJtUytnTazkOxdO1sWbpV0dFrqZhYA7gXOBicBlZjbxoMWuAVa5+1RgFnCbmRWmOKtIzolEY1z74Gts3NXIxEHl3H7p8YR0OL8cQjJr6DOBKndf7+5hYB5w0UHLOFBm8dWGUmA3EElpUpEc9OMn1vDcup307VnI3Z+dRkmhjgWUQzN3P/wCZpcAc9z9qsT05cBJ7n5tm2XKgPnABKAM+JS7/7md17oauBqgsrJy2rx58zoVur6+ntLS0k49N91oLOkpHcbywpZW/vvNMCGDG2YUM77PkV9lKB3GkSoaS9zs2bNfdffp7c5098PegE8A97SZvhy446BlLgF+DhgwBtgAlB/udadNm+ad9cwzz3T6uelGY0lPQY9l5ZZaH3vTAh/+jcf91y9t7PTrBD2OVNJY4oClfoheTWaTSzUwtM30EGDrQctcATyc+HxViUKfkNSfGxE5QH1LhGt+9xrhSIxPTR/KZz4wPOhIkiGSKfQlwFgzG5l4o/NS4ptX2toEnAFgZpXAeGB9KoOK5AJ3518ffpMNOxuYMLCM71w0KehIkkE6fIfF3SNmdi2wEAgB97n7SjObm5h/F/A94H4ze5P4ZpdvuPvObswtkpUefGUz85dvpaQwxJ2fPpHigiPfbi65K6m3zN19AbDgoMfuanN/K3B2aqOJ5JZVW/dxy2MrAfjBx45jdP/seANQjh4dKSqSBppbo3xp3uuEIzEunTGUj54wOOhIkoFU6CJp4KcL32JdTT2j+vfk2x/RdnPpHBW6SMBerNrJvc9vIJRn/PyTx9OjUNvNpXNU6CIBqm1q5fr/XQ7AF08fy9ShFQEnkkymQhcJ0C3zV7K1tpmpQyu4ZrbOoChdo0IXCcgTK7bxp9e30KMgxM8/OZX8kH4dpWv0EyQSgNrGVm5+NL6L4jfPm8Ao7aIoKaBCFwnA9xesYkddCzNG9OYzJ+nQfkkNFbrIUfZC1U4eWlpNYSiPH148hTyd31xSRIUuchQ1hiN88+E3AfjSmWMZM0CbWiR1VOgiR9HP/rqWTbsbOXZQOVd/eFTQcSTLqNBFjpIVW2q574UN5Bn8+OPHUaC9WiTF9BMlchTEYs63HllBzOGKU0YyZYgOIJLUU6GLHAUPLd3Mss17qSwv4itnjQs6jmQpFbpIN9vTEObHT6wB4KbzJ1JapAs9S/dQoYt0s58sfIs9ja2cPLovH5kyKOg4ksVU6CLdaNnmvcxbson8POO7F03CTPucS/dRoYt0k2jMufmRFbjDlaeOZMyAsqAjSZZToYt0k3lLNvHmlloG9Srmi6ePDTqO5AAVukg32Nfcys/+uhaAm84/lp56I1SOAhW6SDe48+kqdjWEmTGiN+cfpzdC5ehQoYuk2Du7GrjvhQ0A3HzBRL0RKkeNCl0kxX64YA2tUefiEwfriFA5qlToIim0eP0unlj5Lj0KQtxwzoSg40iOUaGLpEg05nzv8VUAzD1tNAN7FQecSHKNCl0kRf74WjUrt+5jUK9inRpXAqFCF0mBxnCEWxe+BcANc8bTozAUcCLJRSp0kRR44MV3qKlrYcqQXlw0dXDQcSRHqdBFuqi2qZW7nn0bgOvPHq9rhEpgVOgiXXTvc+upbWrlpJF9OHVsv6DjSA5ToYt0wa76Fu59Pn4Q0dfPGa+DiCRQSRW6mc0xs7fMrMrMbjzEMrPMbJmZrTSzZ1MbUyQ9/XLR2zSEo8we35/pI/oEHUdyXIdnDDKzEHAncBZQDSwxs/nuvqrNMhXAfwJz3H2TmQ3orsAi6WJbbRO/WvwOAF87e3zAaUSSW0OfCVS5+3p3DwPzgIsOWuafgIfdfROAu9ekNqZI+rnj6SrCkRjnHTeQyYN7BR1HBHP3wy9gdgnxNe+rEtOXAye5+7VtlvkFUABMAsqA2939V+281tXA1QCVlZXT5s2b16nQ9fX1lJaWduq56UZjSU8djaWmMcY3n2si5vD9D/XgmNL0fDsql74nmaQrY5k9e/ar7j69vXnJnKS5vXd5Dv4rkA9MA84AegAvmdlid197wJPc7wbuBpg+fbrPmjUriU//fosWLaKzz003Gkt66mgsX/39MqK+hY+fOIR/umDq0Qt2hHLpe5JJumssyRR6NTC0zfQQYGs7y+x09wagwcz+DkwF1iKSZdZur+NPy7ZQEDK+fKauRCTpI5n/E5cAY81spJkVApcC8w9a5lHgVDPLN7MS4CRgdWqjiqSHn/11Le5w6YxhDO1TEnQckf06XEN394iZXQssBELAfe6+0szmJubf5e6rzewJ4A0gBtzj7iu6M7hIEN6o3ssTK9+lKD+Pa08fE3QckQMkdaFDd18ALDjosbsOmv4p8NPURRNJP7cmrhP6uZNHUFmu0+NKeknPt+ZF0tDL63fx97U7KC3KZ+5po4OOI/I+KnSRJLg7t/41fnrcq04dSe+ehQEnEnk/FbpIEp5du4MlG/fQu6SAKz80Mug4Iu1SoYt0oO3a+dzTRlNWXBBwIpH2qdBFOvDEindZsWUfA8qK+OwHRwQdR+SQVOgihxGNObc9Gd+z5brTx+jScpLWVOgih/HI61uoqqlnSO8efGrGsKDjiByWCl3kEMKRGL94Kr52/uUzx1GYr18XSW/6CRU5hIeWbmbz7iZG9+/Jx07QhZ8l/anQRdoRjjp3PL0OiF+8IqQLP0sGSOrQf5Fc89SmCNv3hZl0TDlzJg0MOo5IUrSGLnKQuuZW/rw+DMD154wnT2vnkiFU6CIHue/5jdS3wvThvZk1rn/QcUSSpkIXaWNPQ5h7nlsPxNfOzbR2LplDhS7Sxl1/f5u6lgiT+4b4wKi+QccROSIqdJGEmn3NPPDiRgAuHqfztUjmUaGLJNz5TBXNrTHOmVTJqF46xF8yjwpdBNi8u5HfvbIJs/h+5yKZSIUuAvz7U+tojToXTT2GcZVlQccR6RQVuuS8qpp6/vhaNfl5xpfPHBd0HJFOU6FLzvv539YSc/jE9KGM6Ncz6DginaZCl5y2cmstf35jG4X5eXzxjDFBxxHpEhW65LTb/ho/Pe7lHxjOoF49Ak4j0jUqdMlZr76zm6fX1FBSGOILs0YHHUeky1TokpPcnZ8ujF/4+coPjaRfaVHAiUS6ToUuOemFql0sXr+b8uJ8rjp1VNBxRFJChS45J752vgaAubNG06uHDvOX7KBCl5yzcOW7LK+upX9ZEVecPDLoOCIpo0KXnBKJxvZvO//i6WPoUahztkj2UKFLTnn49S28vaOBoX168KkZw4KOI5JSKnTJGS2RKLf/LX7h56+eNY7CfP34S3ZJ6ifazOaY2VtmVmVmNx5muRlmFjWzS1IXUSQ1frt4E1v2NjG+sowLpw4OOo5IynVY6GYWAu4EzgUmApeZ2cRDLPdjYGGqQ4p0VX1LhDufqQLg6+eMJ6QLP0sWSmYNfSZQ5e7r3T0MzAMuame564A/AjUpzCeSEvc9v4FdDWFOHFbBGccOCDqOSLcwdz/8AvHNJ3Pc/arE9OXASe5+bZtlBgO/A04H7gUed/c/tPNaVwNXA1RWVk6bN29ep0LX19dTWlraqeemG42l+9WFnRv+3khTBG6cWcyEPh3v2ZKuYzlS2TIO0FjeM3v27FfdfXp78/KTeH57/5se/FfgF8A33D16uKuku/vdwN0A06dP91mzZiXx6d9v0aJFdPa56UZj6X7f//MqmiIb+PC4/sy9eGZSz0nXsRypbBkHaCzJSKbQq4GhbaaHAFsPWmY6MC9R5v2A88ws4u6PpCSlSCdtq23igZfeAeCGc3RpOcluyRT6EmCsmY0EtgCXAv/UdgF333+4nZndT3yTi8pcAnfrwrWEIzHOnzKIyYN7BR1HpFt1WOjuHjGza4nvvRIC7nP3lWY2NzH/rm7OKNIpK7fW8vDr1RSEjG+cMyHoOCLdLpk1dNx9AbDgoMfaLXJ3/1zXY4l0jbvzgwWrcYfPfnAEw/qWBB1JpNvpUDnJSovW7uCFql2UF+dz3em6tJzkBhW6ZJ1INMYPF6wG4LrTx1JRUhhwIpGjQ4UuWecPr1azdns9Q3r34LMnDw86jshRo0KXrNLQEuG2J+MXfr5hzgSK8nV6XMkdKnTJKv/93Hp21LUwdWgFH5kyKOg4IkeVCl2yxrbaJv7r2fUA3HTesRzuqGWRbKRCl6zxo7+soak1ypxJA5k5sk/QcUSOOhW6ZIUlG3fz6LKtFOXncdP5xwYdRyQQKnTJeNGY8+1HVwIw97TRDO2jg4gkN6nQJePNW7KJVdv2MbiiB3NPGx10HJHAqNAlo+1tDHPrwrcAuOn8Y+lRqN0UJXep0CWj/ezJtexpbOWDo/py7uSBQccRCZQKXTLWqq37+M3idwjlGd++cKJ2U5Scp0KXjBSLOTc98iYxh8s/MJwJA8uDjiQSOBW6ZKTfvbKJ1zftZUBZEV89e1zQcUTSggpdMk5NXTM/fmINALdcOIny4oKAE4mkBxW6ZJzvPb6auuYIs8f31xuhIm2o0CWjPLt2B48t30pxQR7fvWiy3ggVaUOFLhmjKRzl5kdWAPDlM8fpiFCRg6jQJWP8ZOEaNu1uZMLAMq780Mig44ikHRW6ZIRXNuzm/hc3EsozfnrJVApC+tEVOZh+KyTtNYWj3PCH5bjDF04bzXFDegUdSSQtqdAl7f1k4Ro27mpkfGUZ150xJug4ImlLhS5pre2mlls/MVXXCBU5DBW6pK36lghfT2xq+ZdZ2tQi0hEVuqStW+av5J1d8b1arjt9bNBxRNKeCl3S0vzlW/nDq9UU5edxx2UnUJivH1WRjui3RNLO5t2N3PSnNwG4+YKJjK0sCziRSGZQoUtaiURjfOX3y6hrjnDWxEo+fdKwoCOJZAwVuqSV259ax9J39lBZXsSPPz5F52oROQIqdEkbT63ezh1PV5Fn8PNPHk+fnoVBRxLJKEkVupnNMbO3zKzKzG5sZ/6nzeyNxO1FM5ua+qiSzTbtauQrv18GwNfOHs/JY/oFnEgk83RY6GYWAu4EzgUmApeZ2cSDFtsAnObuU4DvAXenOqhkr+bWKHN/8yr7miOceWwlXzhtdNCRRDJSMmvoM4Eqd1/v7mFgHnBR2wXc/UV335OYXAwMSW1MyVbuzrceWcGqbfsY3reE2z45lbw8bTcX6Qxz98MvYHYJMMfdr0pMXw6c5O7XHmL564EJ7y1/0LyrgasBKisrp82bN69Toevr6yktLe3Uc9NNro9l4cZWHlwTpjAPbv5gD4aWpcfbOtnyfcmWcYDG8p7Zs2e/6u7T253p7oe9AZ8A7mkzfTlwxyGWnQ2sBvp29LrTpk3zznrmmWc6/dx0k8tjeXLluz7ixsd9+Dce9/nLtnRPqE7Klu9LtozDXWN5D7DUD9Gr+Un8QagGhraZHgJsPXghM5sC3AOc6+67kv1rI7lp1dZ9fHHe67jDV88ax0emHhN0JJGMl8z/t0uAsWY20swKgUuB+W0XMLNhwMPA5e6+NvUxJZvU7GvmqgeW0BiOctHxx3Dd6TolrkgqdLiG7u4RM7sWWAiEgPvcfaWZzU3Mvwv4f0Bf4D8TB4JE/FDbeCSn1Ta18tn7XmFrbTMnDKvQwUMiKZTMJhfcfQGw4KDH7mpz/yrgfW+CirTV3Brl/zywlDXv1jGqf0/u+ex0igt0fnORVEmPXQok60WiMa793Wu8snE3A8uL+fWVJ9G3tCjoWCJZRYUu3S4ac67/3+X8bXUNFSUF/PrKmQyu6BF0LJGso0KXbhWNOV97aBmPLNtKz8IQ931uhk6HK9JNktqGLtIZ0Zjz1YeW8WiizO///ExOHNY76FgiWUuFLt2iNRrjaw8tZ/7yeJk/8PmZTB/RJ+hYIllNhS4p1xiO8IXfvMaza3eozEWOIhW6pNSehjBX3L+EZZv30qdnIfdfMYMpQyqCjiWSE1TokjI1jTEuuetF3t7RwOCKHvz6ypmM6p8dJ1MSyQQqdEmJl97exXdfaqK+FcZXlvGrK2dSWV4cdCyRnKJCly77zeJ3uGX+SiIxmD2+P7dfdgLlxQVBxxLJOSp06bSmcJTvPLaSeUs2AzBnRAF3/vMMQrpAhUggVOjSKeu213HN715j7fZ6CvPz+MHHjqNfXZXKXCRAOlJUjoi789DSzVz4Hy+wdns9o/r35NFrTuGSabrqoEjQtIYuSdu+r5mb/vQmf1tdA8DFJwzmex+dTM8i/RiJpAP9JkqH3J2HX9vCdx5byb7mCGXF+Xz7I5P4+ImDdS5zkTSiQpfDqqqp5zuPreS5dTuB+F4sP7j4OAb10tkSRdKNCl3aVd8S4Y6n1nHv8xuIxJzy4nxuvmAil0wborVykTSlQpcDhCMxfr90M3c8tY6auhbM4LKZQ7n+7PG6IIVImlOhCxA/1e2jy7bw87+tZfPuJgCOH1rBdy+apHOxiGQIFXqOa4lEefT1rdz93HqqauoBGN2/J9efPZ45kwdq84pIBlGh56japlZ+9/Im/ueFDdTUtQAwuKIHXz5zLB87YTD5IR2iIJJpVOg5xN1ZXl3Lgy9vYv7yrTS1RgGYMLCMqz88io9MPYYCFblIxlKh54Cd9S38+Y1tzFuymdXb9u1//OTRffm/p43mw2P7adOKSBZQoWepvY1hnljxLo+/sY0X395JzOOP9+lZyCXThvCpGUMZrXOVi2QVFXqWcHfe3lHPM2t28PSaGpZs3E0k0eIFIWP22P589ITBnD2pkqL8UMBpRaQ7qNAzWE1dM0s27GHx+l0sWluzf3dDgDyDU8f24yNTjuGcSQPpVaLzk4tkOxV6hojGnA0763mjupYlG3fz8vrdrN/ZcMAyfXoWctq4/syeMIAPj+1HRUlhQGlFJAgq9DTUEomyYWcDq7ft443qWlZsqWXV1n00hKMHLFdSGGLa8N7MGNGHD43tx9QhFTofuUgOU6EHJBZzdtS3sG5PlB1LN1O1o563a+qpqqln0+7G/W9itjWoVzGTB/di+vDezBzZh8mDe2k3QxHZT4XeDWIxZ09jmB31Leyoi9+21TZTvaeJ6j2NbNnTRPXeJsKRWPwJL79xwPPzDEb268nYAaUcN7gXk4f04rjBveinc6mIyGEkVehmNge4HQgB97j7jw6ab4n55wGNwOfc/bUUZw1EazRGbVPrAbd9Ta3sbTzwsd0N4f3lvbO+Zf8eJofTt2ch5fkRJo8YyJj+pYwZEL+N6FeiPVFE5Ih1WOhmFgLuBM4CqoElZjbf3Ve1WexcYGzidhLwy8THlGsMR6htcbYk1nDDkRit0Rgtbe6HIzHCB39M3G+NxGhqjdIYjtIYjtAQjtIUjtLQEqGpNfExHN3/eDga61TOipIC+pcW0b8sfhtQVsTQPiUM6d2DIb1LGFzRg55F+SxatIhZs05I8VdJRHJRMmvoM4Eqd18PYGbzgIuAtoV+EfArd3dgsZlVmNkgd9+W6sDfe3w1D77SCM88neqXblcozygvzqdXj4L4raQwcf8fj1X0KKSipIAB5cX0LyuiX2mh1rBF5KhLptAHA5vbTFfz/rXv9pYZDBxQ6GZ2NXA1QGVlJYsWLTrCuFC7I0xpgVMYyiM/j/jNjPw8KHhvOq/t9D/uh8woyIPCEBSFjKL8xMcQFCc+FuX/Y7owFH/ePw6LjyRubcSAhvht7w7YC6w7gvHU19d36uuQjjSW9JMt4wCNJRnJFHp7+8EdvIE4mWVw97uBuwGmT5/us2bNSuLTH2jWLBKbKY78uelIY0lP2TKWbBkHaCzJSGaft2pgaJvpIcDWTiwjIiLdKJlCXwKMNbORZlYIXArMP2iZ+cBnLe4DQG13bD8XEZFD63CTi7tHzOxaYCHx3Rbvc/eVZjY3Mf8uYAHxXRariO+2eEX3RRYRkfYktR+6uy8gXtptH7urzX0HrkltNBERORI6blxEJEuo0EVEsoQKXUQkS6jQRUSyhMXfzwzgE5vtAN7p5NP7ATtTGCdIGkt6ypaxZMs4QGN5z3B379/ejMAKvSvMbKm7Tw86RypoLOkpW8aSLeMAjSUZ2uQiIpIlVOgiIlkiUwv97qADpJDGkp6yZSzZMg7QWDqUkdvQRUTk/TJ1DV1ERA6iQhcRyRIZXehmdp2ZvWVmK83sJ0Hn6Sozu97M3Mz6BZ2ls8zsp2a2xszeMLM/mVlF0JmOhJnNSfxMVZnZjUHn6SwzG2pmz5jZ6sTvx5eCztRVZhYys9fN7PGgs3RF4hKdf0j8nqw2sw+m6rUzttDNbDbxa5lOcfdJwK0BR+oSMxtK/ELcm4LO0kVPApPdfTJvO+kAAAKiSURBVAqwFvhmwHmS1uaC6OcCE4HLzGxisKk6LQJ8zd2PBT4AXJPBY3nPl4DVQYdIgduBJ9x9AjCVFI4pYwsd+ALwI3dvAXD3moDzdNXPgRto59J9mcTd/+ru7114dTHxq1dliv0XRHf3MPDeBdEzjrtvc/fXEvfriJfG4GBTdZ6ZDQHOB+4JOktXmFk58GHgXgB3D7v73lS9fiYX+jjgVDN72cyeNbMZQQfqLDO7ENji7suDzpJinwf+EnSII3Coi51nNDMbAZwAvBxski75BfEVnljQQbpoFLAD+J/E5qN7zKxnql48qQtcBMXM/gYMbGfWTcSz9yb+7+QM4CEzG+Vpuh9mB2P5V+Dso5uo8w43Fnd/NLHMTcT/7f/t0czWRUld7DyTmFkp8Efgy+6+L+g8nWFmFwA17v6qmc0KOk8X5QMnAte5+8tmdjtwI3Bzql48bbn7mYeaZ2ZfAB5OFPgrZhYjfsKbHUcr35E41FjM7DhgJLDczCC+ieI1M5vp7u8exYhJO9z3BcDM/hm4ADgjXf/AHkJWXezczAqIl/lv3f3hoPN0wSnAhWZ2HlAMlJvZb9z9MwHn6oxqoNrd3/tv6Q/ECz0lMnmTyyPA6QBmNg4oJAPPxObub7r7AHcf4e4jiH/DT0zXMu+Imc0BvgFc6O6NQec5QslcED0jWHzt4F5gtbv/LOg8XeHu33T3IYnfj0uBpzO0zEn8Xm82s/GJh84AVqXq9dN6Db0D9wH3mdkKIAz8c4atDWar/wCKgCcT/3Esdve5wUZKzqEuiB5wrM46BbgceNPMliUe+9fE9YElWNcBv02sNKwHrkjVC+vQfxGRLJHJm1xERKQNFbqISJZQoYuIZAkVuohIllChi4hkCRW6iEiWUKGLiGSJ/w/NaA+kjCBEYAAAAABJRU5ErkJggg==\n", "text/plain": [ "