\n",
"\n",
"|Code Symbol | Math Symbol | Definition | Dimensions\n",
"| :-: | :-: | :-: | :-: |\n",
"|X|$$X$$|Input Data, each row in an example| (numExamples, inputLayerSize)|\n",
"|y |$$y$$|target data|(numExamples, outputLayerSize)|\n",
"|W1 | $$W^{(1)}$$ | Layer 1 weights | (inputLayerSize, hiddenLayerSize) |\n",
"|W2 | $$W^{(2)}$$ | Layer 2 weights | (hiddenLayerSize, outputLayerSize) |\n",
"|z2 | $$z^{(2)}$$ | Layer 2 activation | (numExamples, hiddenLayerSize) |\n",
"|a2 | $$a^{(2)}$$ | Layer 2 activity | (numExamples, hiddenLayerSize) |\n",
"|z3 | $$z^{(3)}$$ | Layer 3 activation | (numExamples, outputLayerSize) |\n",
"|J | $$J$$ | Cost | (1, outputLayerSize) |\n",
"|dJdz3 | $$\\frac{\\partial J}{\\partial z^{(3)} } = \\delta^{(3)}$$ | Partial derivative of cost with respect to $z^{(3)}$ | (numExamples,outputLayerSize)|\n",
"|dJdW2|$$\\frac{\\partial J}{\\partial W^{(2)}}$$|Partial derivative of cost with respect to $W^{(2)}$|(hiddenLayerSize, outputLayerSize)|\n",
"|dz3dz2|$$\\frac{\\partial z^{(3)}}{\\partial z^{(2)}}$$|Partial derivative of $z^{(3)}$ with respect to $z^{(2)}$|(numExamples, hiddenLayerSize)|\n",
"|dJdW1|$$\\frac{\\partial J}{\\partial W^{(1)}}$$|Partial derivative of cost with respect to $W^{(1)}$|(inputLayerSize, hiddenLayerSize)|\n",
"|delta2|$$\\delta^{(2)}$$|Backpropagating Error 2|(numExamples,hiddenLayerSize)|\n",
"|delta3|$$\\delta^{(3)}$$|Backpropagating Error 1|(numExamples,outputLayerSize)|"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Last time, we decided to use gradient descent to train our Neural Network, so it could make better predictions of your score on a test based on how many hours you slept, and how many hours you studied the night before. To perform gradient descent, we need an equation and some code for our gradient, dJ/dW. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our weights, W, are spread across two matrices, W1 and W2. We’ll separate our dJ/dW computation in the same way, by computing dJdW1 and dJdW2 independently. We should have just as many gradient values as weight values, so when we’re done, our matrices dJdW1 and dJdW2 will be the same size as W1 and W2."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial J}{\\partial W^{(2)}} = \\frac{\\partial \\sum \\frac{1}{2}(y-\\hat{y})^2}{\\partial W^{(2)}}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let’s work on dJdW2 first. The sum in our cost function adds the error from each example to create our overall cost. We’ll take advantage of the sum rule in differentiation, which says that the derivative of the sums equals the sum of the derivatives. We can move our sigma outside and just worry about the derivative of the inside expression first. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial J}{\\partial W^{(2)}} = \\sum \\frac{\\partial \\frac{1}{2}(y-\\hat{y})^2}{\\partial W^{(2)}}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To keep things simple, we’ll temporarily forget about our summation. Once we’ve computed dJdW for a single example, we’ll add all our individual derivative terms together. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can now evaluate our derivative. The power rule tells us to bring down our exponent, 2, and multiply. To finish our derivative, we’ll need to apply the chain rule. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The chain rule tells us how to take the derivative of a function inside of a function, and generally says we take the derivative of the outside function and then multiply it by the derivative of the inside function. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"One way to express the chain rule is as the product of derivatives, this will come in very handy as we progress through backpropagation. In fact, a better name for backpropagation might be \"don’t stop doing the chain rule. ever.\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We’ve taken the derivative of the outside of our cost function - now we need to multiply it by the derivative of the inside."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Y is just our test scores, which won’t change, so the derivative of y, a constant, with respect to W two is 0! yHat, on the other hand, does change with respect to W two, so we’ll apply the chain rule and multiply our results by minus dYhat/dW2. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial J}{\\partial W^{(2)}} = -(y-\\hat{y}) \\frac{\\partial \\hat{y}}{\\partial W^{(2)}}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We now need to think about the derivative of yHat with respect to W2. Equation 4 tells us that yHat is our activation function of z3, so we it will be helpful to apply the chain rule again to break dyHat/dW2 into dyHat/dz3 times dz3/dW2. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial J}{\\partial W^{(2)}} = \n",
"-(y-\\hat{y})\n",
"\\frac{\\partial \\hat{y}}{\\partial z^{(3)}} \n",
"\\frac{\\partial z^{(3)}}{\\partial W^{(2)}}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To find the rate of change of yHat with respect to z3, we need to differentiate our sigmoid activation function with respect to z. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"f(z) = \\frac{1}{1+e^{-z}}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"f^\\prime(z) = \\frac{e^{-z}}{(1+e^{-z})^2}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now is a good time to add a new python method for the derivative of our sigmoid function, sigmoid Prime. Our derivative should be the largest where our sigmoid function is the steepest, at the value z equals zero."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Populating the interactive namespace from numpy and matplotlib\n"
]
}
],
"source": [
"%pylab inline\n",
"#Import code from last time\n",
"from partTwo import *"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"def sigmoid(z):\n",
" #Apply sigmoid activation function to scalar, vector, or matrix\n",
" return 1/(1+np.exp(-z))"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"def sigmoidPrime(z):\n",
" #Derivative of sigmoid function\n",
" return np.exp(-z)/((1+np.exp(-z))**2)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3dd3xUVf7/8deZ9B6SQCgJBJBOCCUkoojBRUBQEL8oFlyx44pfXcviWlldlrV8Lbu2RXQVC/izg6KiYkB6byH0GgKE9Ez6zJzfHzfEEAMMMMmdmXyej8d9JDP3zsznMOGdmzPnnqO01gghhPB8FrMLEEII4RoS6EII4SUk0IUQwktIoAshhJeQQBdCCC/ha9YLx8TE6ISEBLNe/pyVlpYSEhJidhlNStrcPEibPcO6detytdYtG9pnWqAnJCSwdu1as17+nKWnp5OWlmZ2GU1K2tw8SJs9g1LqwKn2SZeLEEJ4CQl0IYTwEhLoQgjhJUzrQ29IdXU1WVlZVFRUmF3KKUVERJCZmWl2GU0qIiKCffv2ERcXh5+fn9nlCCFO4YyBrpR6F7gSyNFa925gvwJeBUYBZcAkrfX6cykmKyuLsLAwEhISMJ7W/ZSUlBAWFmZ2GU2quLiYqqoqsrKy6Nixo9nlCCFOwZkul/eAkafZfwXQpWa7C3jzXIupqKggOjrabcO8uVJKER0d7dZ/OQkhnAh0rfUSIP80h4wFZmvDSiBSKdXmXAuSMHdP8r4I4f5c0YfeDjhU53ZWzX1H6h+olLoL4yye2NhY0tPTT9ofERFBSUmJC0pqPHa73e1rdLUTba6oqPjde+atrFZrs2nrCdLms+fQmrJqKLNpyqo1pdVQWvN9uQ0qbJoKu6bCBpV2475Ku6Z1iIVJvQJc15Aargj0hk7dGpxkXWs9E5gJkJycrOsP6M/MzHTL/uk77riDBx98kJ49ezZaH/qoUaP4+OOPiYyMPOn+adOmERoaysMPP+zy13TWiTYHBgbSr18/0+poSp54wcn5kjZDRbWdXGsludYq8qyVtd/nWivJq/laUFZNcbmxlVTazul1/YPDSEsb7KJW/MYVgZ4FxNe5HQdku+B53casWbMa/TUWLFjQ6K8hRHNmszs4XFjO4cJyjhRWcKSonLWZlby/bzVHiirILiynuOLsAzos0JeIID/CA/2Mr0HG7dAAP0IDfAgJ8CUkwJfQAF+C/X0IDfAlKtS/EVromkCfB0xRSs0FUoEirfXvuls8RWlpKddddx1ZWVnY7XaefPJJ3nzzTV588UWSk5OZPXs2r776Km3btqVLly4EBATw2muvMWnSJIKCgti+fTsHDhzgv//9L++//z4rVqwgNTWV9957D4A5c+bwj3/8A601o0eP5rnnngN+mwohJiaG6dOnM3v2bOLj42nZsiUDBgww8V9ECM/hcGiyCsrZm2tlf24p+/PK2J9Xyv7cUrIKyrE5Guo8OF77nb+PhehQf2JCA4gO9Sc6JICYMH9iar5GhwTQItifiCAjvEMDffGxuM/nS84MW5wDpAExSqks4GnAD0Br/RawAGPI4m6MYYu3uqKwhEe/dcXT/M7+f44+7f7vv/+etm3b8u23xusXFRXx5pvGwJ3s7Gyef/55NmzYQFhYGJdddhlJSUm1jy0oKGDRokXMmzePq666imXLljFr1iwGDhzIxo0badWqFVOnTmXdunW0aNGC4cOH89VXX3H11VfXPse6deuYO3cuGzZswGaz0b9/fwl0IRqQa61kx9ESth8tYcfRYnYcLWHnMSvl1fZTPqZNRCDtIoNoGxlEm8hASnOyGJKcaNyOCCQqxN+jBwCcMdC11jecYb8G7nVZRSZLTEzk4YcfZurUqVx55ZVccskltftWr17NxRdfTFRUFADXXnstO3furN1/1VVXoZQiMTGR2NhYEhMTAejVqxf79+/nwIEDpKWl0bKlMVHaTTfdxJIlS04K9F9//ZVx48YRHBwMwJgxYxq9zUK4O2uljS1ZRWw8VMimQ4VsPFTI0eKGh9G2Cgugc8tQEmKCSYgOISEmhI4xIbSPCibQz+ekY9PTj5HWq3VTNKFJuNWVonWd6Uy6sXTt2pV169axYMEC/vrXvzJ8+PDafWdaUDsgwPjU2mKx1H5/4rbNZsPX17l/bk8+QxDCFXKtlazcm8eKPXms2Z/Prhwr9f/7hfj70K11GN1ah9O9dRjdWofRvXUYkcGN0z/tCdw20M2SnZ1NVFQUEydOJDQ0tLbvGyAlJYUHHniAgoICwsLC+Pzzz2vPwp2RmprK/fffT25uLi1atGDOnDncd999Jx0zZMgQJk2axKOPPorNZmP+/PncfffdrmqeEG7JWmlj6a5clu/JZcWePHblWE/a72tR9GgbTt/4SJLiI+kbH0mnmBAsbtR/7Q4k0OvZsmULjzzyCBaLBT8/P958883aIYPt2rXjoYceIjU1lbZt29KzZ08iIiKcfu42bdowY8YMhg4ditaaUaNGMXbs2JOO6d+/PxMmTKBv37506NDhpC4fIbzJgbxSFm3PYdH2HFbuzaPa/tspeKCfheQOUQzqHE1qxyh6t4v4XXeJ+D11pm6ExpKcnKzrL3CRmZlJjx49TKnHWUeOHKFNmzbYbDbGjRvHbbfdxrhx48wuq1GdGIfuCe+Pq8iY7MaxO6eE+ZuO8O2WI+yucxZuUdCvfQsu7dqSQZ2j6RMXQYBv4we4J77PSql1WuvkhvbJGfpZmjFjBkuWLKGiooLhw4ef9IGmEOL3DuWXMW9TNvM3ZbP96G9XWYcF+nJp15b8oUcrLu3aiqiQ5tv37SoS6Gdp+vTpbnk1qxDupNJm54eMY8xdfZDle/Jq7w8P9GVk79ZcldSWCztF4+cjSzK4kgS6EMJl9h638tGqg3yxPouCsmoAAnwtjOjVmjFJbbmka0yTdKU0VxLoQojzorVm5d583lm6l58yc2rv79kmnBtS4hnTtx0RQbIwSlOQQBdCnBO7Q/PN5mxmLtlLRnYxAP6+Fsb1bcdNF7YnsV2EXFPRxCTQhRBn5USQv/rzLvYeLwUgJtSfmy9M4KYL2xMT6vppYYVz5BMJJ9xxxx1s27atUV9j1KhRFBYW/u7+adOm8eKLLwIwadIkOnbsSN++fenfvz8rVqxo8Lmeeuopfvrpp0atVzQ/Dodm3qZsRryyhPvnbmTv8VLio4KYcU0iS6dexv3DukiYm0zO0J3gTtPnvvDCC4wfP56FCxdy9913s3nz5pP22+12nnnmmcYoUTRjq/fl8/dvt7E5qwiAuBZB3HfZBVzTP05GqrgReSfqKS0tZfTo0SQlJdG7d28++eQT0tLSOHER1OzZs+natStpaWnceeedTJkyBTDOnu+55x6GDh1Kp06dWLx4Mbfddhs9evRg0qRJtc8/Z84cEhMT6d27N1OnTq29PyEhgdzcXMAYGtmtWzeGDRvGjh07GqxzyJAh7N69u/axzzzzDIMHD+bTTz9l0qRJfPbZZ7X7HnvsMQYNGkRycjLr169nxIgRdO7cmbfeeqv2+V544QUGDhxInz59ePrpp133Dyo82oG8Uu75cB3X/WcFm7OKaBUWwIxrEln0UBoTBraXMHcz7nuGPs35S+rP7nmLTrvbU6bPnT9//knzyAQGBrJ06dLaNtQVHx/PihUr+POf/8ykSZNYtmwZFRUV9OrVi8mTJ7Nw4UJ27drF6tWr0VozZswYlixZwpAhQ87yH1d4i4pqO68t2s1/luyh2q4J8vPhriGduPvSTgT7u29sNHfyztTj7tPnPvLII/z973+nZcuWvPPOO7X3T5gw4ZRtOvEciYmJWK1WwsLCapeUKywsZOHChSxcuLB2eTmr1cquXbsk0JuppbtyeeKrLezPKwNg/IA4Hh7ejdYRgSZXJs7EfQP9DGfSjcXdp8890YdeX0hIyDnXpbXmr3/9q8zq2MyVVGn+/MlGvtxwGIBusWH845pEBnRoYXJlwlnSAVZPdnY2wcHBTJw4kYcffpj169fX7ktJSWHZsmUUFBRgs9n4/PPPz+q5U1NTWbx4Mbm5udjtdubMmcOll1560jFDhgzhyy+/pLy8nJKSEubPn++Sdp3OiBEjePfdd7FajcmSDh8+TE5OzhkeJbzJLztyeGJZOV9uOEyAr4VHRnRj/n2DJcw9jPueoZukOU6fO3z4cDIzMxk0aBAAoaGhfPjhh7Rq1arRX1uYq7TSxvQFmXy86iAAKQlRPD++Dwkxp/6LT7gvmT73LMn0ue79/riKJ06rera2ZBVx35z17M8rw9/HwtUX+DDjlsvdatHjxuaJ77NMn+tCMn2u8HRaaz5cdZBn52+jyu6ge+swXp7Ql2M71jerMPdGEuhnSabPFZ6stNLGY19u4euN2QDcfGEHnriyBwG+Phxr+JIH4UHcLtC11jKhjxsyq2tOuM7e41bu+mAdu3OsBPv7MOOaRMb2bWd2WcKF3CrQAwMDycvLIzo6WkLdjWitycvLIzBQxiF7qmW7c7nnw3UUV9jo0iqUNyf254JW8pemt3GrQI+LiyMrK4vjx4+bXcopVVRUNLtgq6ioIDIykri4OLNLEefgo1UHeOrrDOwOzfCesbw8oS8hAW71X1+4iFu9q35+fnTs2NHsMk4rPT299orK5qI5ttkb2B2aZ7/ZxnvL9wMw+dLO/GVENyzywafXcqtAF0K4RkW1nfvnbuCHjGP4+ShmXNOH8QPkLyxvJ4EuhJcpqajmztlrWbk3n/BAX2bdMpCUjlFmlyWagAS6EF7keEklk/67mozsYlqFBTD79hS6tw43uyzRRCTQhfASWQVlTJy1iv15ZSREB/PB7anERwWbXZZoQhLoQniBQ/ll3PD2SrIKyunVNpz3bk2hZZgsB9fcSKAL4eHqhnlSfCQf3J5CeKCf2WUJEzg1fa5SaqRSaodSardS6tEG9kcopeYrpTYppTKUUre6vlQhRH2H8su4fqYR5n0lzJu9Mwa6UsoHeB24AugJ3KCU6lnvsHuBbVrrJCAN+D+llL+LaxVC1HG4sJzrZ67kcGE5/dpHMlvCvNlz5gw9Bdittd6rta4C5gJj6x2jgTBlXK8fCuQDNpdWKoSolWut5OZZq2rD/P3bJMyFE/OhK6XGAyO11nfU3L4ZSNVaT6lzTBgwD+gOhAETtNbfNvBcdwF3AcTGxg6YO3euq9rRZKxWK6GhoWaX0aSkze6l3Kb55+oKDhQ7iA+z8GhKICF+53/1pzu3ubF4YpuHDh16XvOhN/STUv+3wAhgI3AZ0Bn4USn1q9a6+KQHaT0TmAnGAheeNrE8eOaE+OdL2uw+Kqrt3PLuag4Ul9EhOphPJw+iVZhr5hZy1zY3Jm9rszNdLllAfJ3bcUB2vWNuBb7Qht3APoyzdSGEi9jsDqZ8vJ5V+/KJDQ/gw9tTXRbmwjs4E+hrgC5KqY41H3Rej9G9UtdB4A8ASqlYoBuw15WFCtGcaa15al4GP2XmEBnsJxcNiQadsctFa21TSk0BfgB8gHe11hlKqck1+98CngXeU0ptweiimaq1zm3EuoVoVt7+dS8frzqIv6+Fd25JpmuszGUufs+pC4u01guABfXue6vO99nAcNeWJoQA+G7LEf6xYDsAL1/XlwEdZKIt0TCnLiwSQphj/cECHvhkIwBTR3ZndJ82Jlck3JkEuhBuKqugjDvfX0ulzcH1A+OZfGkns0sSbk4CXQg3VF5l5+4P1pFXWsXgC2J49uress6uOCMJdCHcjNaaR7/YTEZ2MR2ig3ntxn74+ch/VXFm8lMihJuZ9es+vt6YTbC/DzNvTiYyWKZFEs6RQBfCjSzdlcuM7zIBeOm6JLq1luGJwnkS6EK4iUP5ZUyZsx6HhilDL2BkbxnRIs6OBLoQbqDSZudPH62nsKyaod1a8ufLu5pdkvBAEuhCuIEZC7az5XARcS2CeGVCP3wsMqJFnD0JdCFMtmDLEd5bvh8/H8XrN/YnIljmNRfnRgJdCBMdyCtl6mebAXhsVA+S4iNNrkh4Mgl0IUxSabNz78frKam0MbJXayZdlGB2ScLDSaALYZJ/fJvJ1sPFxEcF8dz4PnIlqDhvEuhCmODHbcd4f8UB/H0sRr95kPSbi/MngS5EE8spqWDq50a/+V9GdqNPnPSbC9eQQBeiCWmtmfrZZvJrJt267eKOZpckvIgEuhBN6IOVB/hlx3Eigvx48dokLDLeXLiQBLoQTWR3TgnTvzXmaZlxTSKtI2SBZ+FaEuhCNIEqm4P7526k0uZg/IA4RiXKPC3C9STQhWgCL/24k4zsYtpHBTNtTC+zyxFeSgJdiEa27kA+/1myB4uClyckERrg1NrsQpw1CXQhGlFFtZ1HPt2M1nD3pZ0Z0CHK7JKEF5NAF6IR/d/CHezNLaVLq1AeGNbF7HKEl5NAF6KRrDuQz6yl+7AoePHaJAJ8fcwuSXg5CXQhGkH9rhaZRVE0BQl0IRqBdLUIM0igC+Fi0tUizCKBLoQLSVeLMJMEuhAu9K+fd0lXizCNBLoQLrLzWAkzl+xFKXhufB/pahFNTgJdCBdwODSPf7kFm0NzY0p7+rdvYXZJohlyKtCVUiOVUjuUUruVUo+e4pg0pdRGpVSGUmqxa8sUwr19tj6LNfsLiAn15y8juptdjmimzjiphFLKB3gduBzIAtYopeZprbfVOSYSeAMYqbU+qJRq1VgFC+Fu8kurmLHAmBb3idE9iQiW5eSEOZw5Q08Bdmut92qtq4C5wNh6x9wIfKG1Pgigtc5xbZlCuK8ZCzIpKKvm4guiGdu3rdnliGbMmWnf2gGH6tzOAlLrHdMV8FNKpQNhwKta69n1n0gpdRdwF0BsbCzp6ennULK5rFarR9Z9PqTNp7Yj386n6yrwVXBVmzIWL/bc3kZ5nz2fM4He0BpZuoHnGQD8AQgCViilVmqtd570IK1nAjMBkpOTdVpa2lkXbLb09HQ8se7zIW1uWJXNwd//9SsA917Whesv79oElTUeeZ89nzOBngXE17kdB2Q3cEyu1roUKFVKLQGSgJ0I4aXe/nUvu3OsdIwJ4Z60zmaXI4RTfehrgC5KqY5KKX/gemBevWO+Bi5RSvkqpYIxumQyXVuqEO7jYF4Z//p5FwDPju1NoJ+MORfmO+MZutbappSaAvwA+ADvaq0zlFKTa/a/pbXOVEp9D2wGHMAsrfXWxixcCLNorXlq3lYqbQ7G9m3L4C4xZpckBOBclwta6wXAgnr3vVXv9gvAC64rTQj39N3Wo6TvOE5YoC9PjO5pdjlC1JIrRYU4CyUV1fxtfgYAU0d2p2VYgMkVCfEbCXQhzsL/LdzJseJK+rWP5MaU9maXI8RJJNCFcNKWrCJmr9iPj0Ux/epELJaGRvQKYR4JdCGcYHdoHv9qCw4Nt16UQM+24WaXJMTvSKAL4YQPVx5gc1YRbSIC+bOHX0AkvJcEuhBncKy4ghd+2AHAtDG9CAlwanCYEE1OAl2IM3jmm21YK20M69GK4T1jzS5HiFOSQBfiNNJ35PDt5iME+fkwbUwvlJIPQoX7kkAX4hQqqu089bUx5vyBYV2IaxFsckVCnJ4EuhCn8Nqi3RzML6N76zBuG9zR7HKEOCP5dEeIBmRbHfxnxR4Apo/rjZ+PnPsI9yc/pULUo7Xm/YxKqu2aG1LiGdAhyuyShHCKBLoQ9Xy+/jA7ChxEh/gzdaQs+Cw8hwS6EHUUlFbxj5oFnx8f3YPIYH+TKxLCeRLoQtTxz++2k19aRfcoC+P6tTO7HCHOigS6EDXW7M/nk7WH8PNR/LFngIw5Fx5HAl0IoNru4PEvtwBwz6WdaRsq/zWE55GfWiGAWb/uY+cxKx2ig/nT0AvMLkeIcyKBLpq9Q/llvPrzTkAWfBaeTQJdNGtaa56el0FFtYOrktoypGtLs0sS4pxJoItm7YeMoyzankNYgC9Pju5hdjlCnBcJdNFsWSttTJu3DYBHRnajVXigyRUJcX4k0EWz9fKPOzlaXEFSXAQ3pXYwuxwhzpsEumiWth4u4r/L9mFRMH1cIj6y4LPwAhLootkxFnzeikPDLRcl0LtdhNklCeESEuii2flo1QE2HSokNjyAh4Z3M7scIVxGAl00K8eKK3j+e2PB57+N6U2oLPgsvIgEumhW/jY/o2bB51hG9JIFn4V3kUAXzcbPmcdYsOUowf4+/G2sLPgsvI8EumgWyqpstQs+P3h5V9pFBplckRCuJ4EumoWXf9zJ4cJyerUNZ9JFCWaXI0SjcCrQlVIjlVI7lFK7lVKPnua4gUopu1JqvOtKFOL8bD1cxLvL9mNRMOOaRHxlwWfhpc74k62U8gFeB64AegI3KKV6nuK454AfXF2kEOfK7tA8/uUW7A7NHwcl0Ccu0uyShGg0zpyqpAC7tdZ7tdZVwFxgbAPH3Qd8DuS4sD4hzsuHKw+wKauI1uGBPDxCxpwL7+bMINx2wKE6t7OA1LoHKKXaAeOAy4CBp3oipdRdwF0AsbGxpKenn2W55rNarR5Z9/nw1DYXVDiY8Ws5ANd21qxdsdTpx3pqm8+HtNnzORPoDY3t0vVuvwJM1VrbTzcUTGs9E5gJkJycrNPS0pws032kp6fjiXWfD09ss9aayR+uo8JezuU9Y3loQvJZPd4T23y+pM2ez5lAzwLi69yOA7LrHZMMzK0J8xhglFLKprX+yiVVCnGWFmw5yg8ZxwgN8OVvY3qZXY4QTcKZQF8DdFFKdQQOA9cDN9Y9QGvd8cT3Sqn3gG8kzIVZCkqreHreVgAevaI7bWXMuWgmzhjoWmubUmoKxugVH+BdrXWGUmpyzf63GrlGIc7KM99sI9daRWrHKG5MaW92OUI0GadmJtJaLwAW1LuvwSDXWk86/7KEODeLth/jyw2HCfSz8Nz/9MEi85yLZkSusBBeo7iimse+MLpaHrq8GwkxISZXJETTkkAXXmPGgu3GknLxkdw2uOOZHyCEl5FAF15h+Z5c5qw+iJ+P4oXxfWRJOdEsSaALj1dWZePRz7cAMGVoF7rGhplckRDmkEAXHm/Ggu0czC+je+sw7knrbHY5QphGAl14tCU7j/PBygP4+Sj+77ok/H3lR1o0X/LTLzxWUVk1f/lsMwAPDOtKr7YRJlckhLkk0IXHenreVo4WV9CvfSR3D+lkdjlCmE4CXXikBVuO8NXGbIL8fHjpur6yaIUQSKALD5RTUsHjXxqjWh4b1Z2OcgGREIAEuvAwWmse+2ILBWXVXNIlhokXdjC7JCHchgS68CgfrjzAT5k5hAf68vz4Ppxu/n0hmhsJdOExth8t5tlvMwGYcU0f2kTItLhC1CWBLjxCeZWd+z7eQJXNwQ0p8Yzu08bskoRwOxLowiM88802duVYuaBVKE9dKSsQCdEQCXTh9hZsOcKc1Qfx97Xwr+v7EeTvY3ZJQrglCXTh1rIKynj0c+Nq0MdH9aBn23CTKxLCfUmgC7dVZXMw5eMNFFfYGNajFX8cJEMUhTgdCXThtqZ/u42NhwppFxnE8+OTZIiiEGcggS7c0tcbD/P+igP4+1h446b+RIX4m12SEG5PAl24nZ3HSmoXrHjqqp4kxUeaXJEQnkECXbiVkopqJn+wjvJqO9f0a8dNqe3NLkkIjyGBLtyGw6H5y2eb2ZtbSvfWYUwflyj95kKcBV+zCxDihH8t2sV3W48SFuDLmxMHNN54c4cD8vdC7g7I2wP5e6A0F8oLoaIIHDaSy8phRyQER0FISwiNhZgu0LI7tOwGgbKYhnA/EujCLXy7+Qiv/LQLi4J/3djPtVPiag1HN8OuH+HQKji0GioKT/uQUIDS0xzQsgd0GAQdLobOlxnBL4TJJNCF6bYeLuKhTzcC8NioHgzt1ur8n9ThgEMrYcunsON7KMk+eX9oa2jdG6IvgKjOEBYLgZEQGA4WP9auWU1yvyQoz4fS41B02DijP35iyzS2te+C8oGEi6H7ldBrHIS6oH4hzoEEujBVTkkFd85eS0W1g/ED4rh9cMfze8LCg7DhQ9g0FwoP/HZ/WBvoMhw6XQpxKRARB6fpn7eG5ULcgIZ32iohewMcWA57f4H9y2DfEmP74THoOhL63gRdLgcfv/NrjxBnQQJdmKa8ys5ds9dxpKiCAR1aMH1c73P7EFRroxtl5euQOR+0w7g/vB0kXmucNbdJOm2AnxXfAGh/obFd8iCUF8DOhZDxhdGts/0bYwuPg9S7YcAt0ucumoQEujCFze7gvjnra68EfWviAAJ8z/JDUK1hx3ew5AXIXm/cZ/GFXtdA/5sh4RKwNMFEXkEtIGmCsZUchc2fwPrZkLcbfnwSFj8P/f8IF02B8LaNX49otiTQRZPTWvP0vAx+yswhIsiP928bSMuwgLN5Atj5PaTPgCObjPuCoiD5Nhh4B4SbOFd6WGu4+H4YdB/s/hGW/xv2/2r89bBmFgy8HQb/WfrZRaOQQBdN7o30PXy0ypgO951bkrmgVZjzD96bDj9NM/qwwRhOOPhBo1vDz41WMLJYoOsIY8veCEtfgm1fw8o3YO1/IeVOI9hldIxwIacuLFJKjVRK7VBK7VZKPdrA/puUUptrtuVKqSTXlyq8wWfrsnjhhx0oBa9O6EtygpOBdnwnfDwBZo81wjykFYyYAfdvggsnu1eY19e2L1w3G+7+FbqNAls5LP8X/Ls/rHwL7NVmVyi8xBnP0JVSPsDrwOVAFrBGKTVPa72tzmH7gEu11gVKqSuAmUBqYxQsPNd3W47wl8+MLpKnruzJFYlOdI2U5sHif8Kad0DbwT/M+CAydTL4BzdyxS7Wpg/cMAcOr4Mfnza6Yr6fanTFDP+7cTYvV8aK8+BMl0sKsFtrvRdAKTUXGAvUBrrWenmd41cCca4sUni+X7bn8L9zN+DQcN9lF3DrxWcYnuiwGyG+6O9QWQTKYvSRpz0GoS2bpujG0m4A3DIfdiyAhU9A3i6YMwE6DYUrnoeWXc2uUHgopbU+/QFKjQdGaq3vqLl9M5CqtZ5yiuMfBrqfOL7evruAuwBiY2MHzJ079zzLb3pWq5XQ0FCzy2hS59vmzDw7L62roNoBIzr4cn13/9MOTwwt2UPXnW8QXrIbgMPJ8MIAABIaSURBVPwW/dh9wW2UhTTdRF1N9T4rRzXtDi+gw4FP8LOV4lC+HIq/mgMdrsPhcxYfFLuA/Gx7hqFDh67TWic3tM+ZM/SG/uc1+FtAKTUUuB0Y3NB+rfVMjO4YkpOTdVpamhMv717S09PxxLrPx/m0ed2BAv69aBXVDrgxtT3Trz7NWPPKEvjlH7D+LWMseXgcjHqeqO6jSTn38s9J077Pl0PpE/DzNCzrZ9Ph4Gd0KF4NV7wA3UY2UQ3ys+0NnPlQNAuIr3M7Dsiuf5BSqg8wCxirtc5zTXnCk63el88f31lFWZUxFe7fx54izLU2Lgh6PdUYBYKCQVPg3lXQfXST122KkGgY82+4/UeITTSueJ0zAebeBIWHzK5OeAhnAn0N0EUp1VEp5Q9cD8yre4BSqj3wBXCz1nqn68sUnmbZ7lxueXc1pVV2xvZty/Pj+2CxNBDmhQdhzg3wyUQoPmz0L9+VDiOmQ4Bn/SnsEvEpNe2fAf6hxhWnr6fA0ldkNIw4ozMGutbaBkwBfgAygf+ntc5QSk1WSk2uOewpIBp4Qym1USm1ttEqFm7vl+053PreGsqr7VyXHMdL1/XF16fej5q9Gpa9apyV7/wOAsJh1IvGGWqbPuYU7i58fGHQn2DKGmPaguoy+Olp+M8QOLjK7OqEG3PqwiKt9QJgQb373qrz/R3A7z4EFc3Pd1uO8L9zN1Bt10y8sD3PjOn9+zPzQ2vgmwfg2Fbjdq9rYOQM4ypL8ZvwtnDte9BvInz7EORsg3eHQ/9bYNg0uShJ/I6sWCRc5r1l+/jTx+uptmvuGNyRZ8fWC/PyQvjmz/DO5UaYR3aAmz6Ha/8rYX46FwyDP62EIY+AxQ/Wvw+vDTRmlDzDKDXRvEigi/PmcGhmLMhk2vxtaA0PD+/K46N7/PYBqNaw5TMjhNa+a0yYNfhBI6S6DDO3eE/hFwSXPQH3LIMOg6EsF768G2aPgdxdZlcn3ITM5SLOS6XNziOfbmbepmx8LYrn/qcP/zOgznVleXvg2weNOVgA2g+CK1+GVj1MqdfjtewGk76BjR8bFyXtWwJvXmTMCzP4QfALNLtCYSI5Qxfn7HhJJRNnrWLepmxC/H14d9LA38K8ugLS/wlvDDLCPKiFMSxv0gIJ8/OlFPS7CaasNfrX7VWw+Dkj2Pf8YnZ1wkRyhi7OyeasQu7+wFiconV4ILNuSaZ3u5pFHPb8YnyIl7/HuN33Jrj8GQiJMa9gbxQSDWNfh6Qbjc8mcnfAB1dD4nXGsE+ZorfZkTN0cda+2nCYa99aUbvS0Lz7LjbCvOQYfHa7ESr5eyCmG0z6Fq5+Q8K8MSVcDJOXwh+eAt9A2PL/4LVk4/MKh8Ps6kQTkkAXTqu02Zk2L4MHPtlIpc3BDSnxfHxnKq2CfWH128aHnls/A98g+MPTRsgkNDgLhHA1X3+45CHjg+YLhkFFkXHW/u5wOLrV7OpEE5EuF+GUfbmlTPl4PRnZxfhaFE+P6cXE1Pao/Uvh+0d/G1PeZTiMegFaJJhab7MV1RFu+gy2fQXfPQpZa4wLkgb9CS59tHlefduMSKCLM1qebePeRb9SWmWnfVQw/76hH0mhRfDpLcYqPAAR7Y1+2x5XyZzeZlPKuMK082XG9MOr3zaWwtvymdEt0+d6Y0Ul4XXkXRWnVFBaxf/O2cDMzZWUVtm5sk8bvpncj6RdrxndK9u+Br9gGPoETFkNPcdImLuTwAjjr6U7F0HbflByBL66B95Og/1Lza5ONAI5QxcN+iHjKI9/uZVcayX+FnhmTA8m+P2KevsmKKmZbDPxOuMS9Ih2ZpYqzqRdf7hjkfFh6U9/MxbWfm80dL/SGH0U3dnsCoWLSKCLk+SXVvG3+Rl8vdEI7ZQOLfjfsHQGr50GuTUTabbpa6ys015WGfQYFgskXQ89xhjdL8teMWZy3PkDDLzDWNZPeDwJdAGA3aGZu+Ygz3+/g6LyagL9LLyUYuWKoy+jdq8xDmqRAJc9aUymJX2wnsk/GNKmQv8/wqJnjStOV70J69+nY5tRkJpkXAQmPJIEumDToUKe/Horm7OKALgt/iiPBH5N0LrFAFT5ReB/+ZPGLH++/maWKlwlvI1xfcCF98Ci6bDzOzoc/Axe+REummLcHxBmdpXiLEmgN2OHC8t5aeFOvtiQhdaaUaG7eTbyW6KPrzYO8A+Di+5jla0Pl6RcYW6xonG0ToQb58KhNeR/8RBRBZvgl+mw8k0j1FPulDN2DyKB3gwVllXxRvoe3lu+nyqbnT/4bmZa5PfEWzdBLhAQARdOhtTJEByFPT3d7JJFY4sfyOakZ0jr4GMMdTy00gj2Za9C8q1w4b3GWb1waxLozUhReTUfrNjPzCV7qawo4xqfpdwf/hNtqvaDFQiMNNbyTLkTgiJNrlaYouMlcNv3xiyOS1+Gvb8YH6Ku+g8k3QCD7jVmfBRuSQK9GcizVvLusn3MXn6AgMo8bvf9kVuDfybcUQRVQFhbSL3LGO0g/aZCKeh0qbEdXm+MiNk2z1hYY/370CkNUu6GriOMue2F25BA92L7c0t5f8V+Pll9gP72zczwWcTIwHX4YgMH0CYJBt0Hva4GHz+zyxXuqF1/uG62sYjGitdh8yfGdMh7040VpwbeYUzhK8vhuQUJdC/jcGgW7zrO+8v3k7FjF9f6LOZ7n0W09z9uHKAs0HW08adzh4vkyk7hnJgucNUrMOxp2PARrHkbCvbDj08awx+7XQF9JxrTDfhIrJhF/uW9xLHiCr7ccJh5q3fQvWAxt/os5+KArfiqmulTI+KNscd9b5IrO8W5C2rx27DGXT/Cmlmw52djGohtX0Noa+MCpqQboFV3s6ttdiTQPVhFtZ0fMo4yb+1efPYtYoxlOV9Y1hHoXw2AtvhC11EwYJJx5iT9ncJVLD7QbaSxFWfDpjnGmXv+HqPPfdkr0LK7MUlYz6sl3JuIBLqHKa+ys3jncZZs2oHeuZAhjlX8y7KZEL/K2mN0+0GoxGtRvcZJ36ZofOFtjbnYBz8Ih1bBxo+MD1GPb4f0GcbWsofxWU3XkdC6j1xp3Egk0D1AQWkVS3bmsGX9cnwPLGGIXsczlu1Gd0rNSbcttg++iddA7/9BRbY3t2DRPCkF7S80ttEvwd7FsO1LyPwGjmdCeqYR7qGx0OVyY+78TkMhMNzsyr2GBLobstkdbDhUyPotW6nY8TMdilZzsSWDsaqodsJju/KlPO5ignqPgW5X4BsZb27RQtTl4wddhhnb6JeNce2Z82D3T1B8GDZ8aGwWX4hLMca/Jww2vvcLNLt6jyWB7gYqqu1sOZTPnm1rqdy7nBZ5G0nS27nbkmMcUHMWXhrQEjoNJaTHcHy6DCNILskWnsDX/7dw1xqOZcCuhcZ2aBUcXG5si58DnwCIG2iEe3yKMWxSfs6dJoHexLTWHMorY/eubRTuXYsjexOxJRkkqV0MVOXGQcrYKi3BFLe+kIhew/DvOoyQmK4yzFB4NqWgdW9ju+RBKC+AA8uNBTf2/2qsf3pgqbGdEH0BtEuGuGQj4Fv1krP4U5BAb0QV1Xb2HT7Ksb1bKD60FZ+cDGJKd9Bd7+MyVfbbgTXdKIV+sVhjkwm94CIiu11CQKtetJQxvcKbBbWA7qONDaAs3wj4A8uM9VCPbIa83ca2ea5xjPIxQj62F8T2hNje0KonRLZv9ic8khbnyeHQHC8qIefQHvIO76Ly6A588ncRUbqfOPsheqh8etR/kIJiSyT54d1QbZKI6ZJKSOdBREa0Q2ZQEc1acBT0uNLYAGxVxgLkh9f9tuXthtwdxpbxxW+P9Q8zVl+K7gxRnY3Qj+4MUZ2azWgvCfQzqLbZyc3NofDYQUqOZ5GTuY5fMr/Cz5pFeMVhYmzHiCWfWKV//2AFVfiR4x9HWXhnfFr3IvqCZCI7JRMe1obwZn42IcQZ+fob3Szt+gN3GvdVlxtDIo9tM/rjczKMr6XH4chGY6svqIVxcV1EPETEGRfXRcQRXnQcirsaI2+84DqNZhfoWmvKysspyj+GNf845UU5VJbkYbPmYbMex2I9RkB5DsFVuUTY82ipC2ijqjkxcWhK/SdU4EBx3NKSooA2VIZ3xKdVV8Lie9MyIRH/mATivOAHRQi34RdkLHrdtt/J95fmQt4e4+KmvD3GmfyJ78sLjO3o5pMe0h9gw1SjGyckBkJaQWjLmq81W+19LY0ZSYMiwT/ULbt3nAp0pdRI4FWM8RaztNb/rLdf1ewfBZQBk7TW611cKwDlZaVYi/IoKymksrSIytIiqsuLsZUV46gsxlFhhcpiVJUVS7UV3+pSfKutBNuKCHEUE65LCFUVhDj7ggpKCaLAJwqrXwxFhBHQujt+MR0Ja92J6LguhMR0oKWvPy0bo8FCCOeExBhb/bVutQbrMSg6DEWHoCjL2IqzKM7KNGYdLT1uHGM9BseceC2L72/h/ruvEUbgB4TVfA01vtb9PjDcOM7FzhjoSikf4HXgciALWKOUmqe13lbnsCuALjVbKvBmzVeX2/r2nQws+Pbcn0CBTVsoVqFYLeGU+0ZQ6RdBdUAkOigaS3hr/CPbEBwdT0SrOCJbxRMSGFb7CyA9PZ3UtDRXNEUI0RSUgrDWxhY34KRd69PTSUtLA1tlTajn1PmaA9bjNV9zoCwPyguhohCqy6As19jORes+MPnX829bPc6coacAu7XWewGUUnOBsUDdQB8LzNZaa2ClUipSKdVGa33E1QXbg6LJLwinQgVTbgmmyieYKp8Q7H4h2P1CcfgZvwVVQBiWwDB8gyLwC4kgKLwloS1aER7VisDQFkQpRfP4mEQIcUa+ATV963HOHW+rMoK9vNDoyjnxfUUhVBRBZQlUWaHSanyt+32l1fjl0giUkcGnOUCp8cBIrfUdNbdvBlK11lPqHPMN8E+t9dKa2z8DU7XWa+s9113AXQCxsbED5s6d68q2NAmr1UpoaKjZZTQpaXPzIG32DEOHDl2ntU5uaJ8zZ+gN9fzX/y3gzDForWcCMwGSk5N1mgd2XaSf+BOtGZE2Nw/SZs/nzJRnWUDdiULigOxzOEYIIUQjcibQ1wBdlFIdlVL+wPXAvHrHzAP+qAwXAkWN0X8uhBDi1M7Y5aK1timlpgA/YAxbfFdrnaGUmlyz/y1gAcaQxd0YwxZvbbyShRBCNMSpceha6wUYoV33vrfqfK+Be11bmhBCiLMhy4YIIYSXkEAXQggvIYEuhBBe4owXFjXaCyt1HDhgyoufnxjgHK/39VjS5uZB2uwZOmitG5w6yrRA91RKqbWnukrLW0mbmwdps+eTLhchhPASEuhCCOElJNDP3kyzCzCBtLl5kDZ7OOlDF0IILyFn6EII4SUk0IUQwktIoJ8HpdTDSimtlIoxu5bGppR6QSm1XSm1WSn1pVIq0uyaGoNSaqRSaodSardS6lGz62lsSql4pdQvSqlMpVSGUup+s2tqKkopH6XUhpoFeryCBPo5UkrFY6yzetDsWprIj0BvrXUfYCfwV5Prcbk66+deAfQEblBK9TS3qkZnAx7SWvcALgTubQZtPuF+INPsIlxJAv3cvQz8hQZWZvJGWuuFWmtbzc2VGIuYeJva9XO11lXAifVzvZbW+ojWen3N9yUYAdfO3Koan1IqDhgNzDK7FleSQD8HSqkxwGGt9SazazHJbcB3ZhfRCNoBh+rczqIZhNsJSqkEoB+wytxKmsQrGCdkDrMLcSWn5kNvjpRSPwENLc39OPAYMLxpK2p8p2uz1vrrmmMex/gz/aOmrK2JOLU2rjdSSoUCnwMPaK2Lza6nMSmlrgRytNbrlFJpZtfjShLop6C1HtbQ/UqpRKAjsEkpBUbXw3qlVIrW+mgTluhyp2rzCUqpW4ArgT9o77yAoVmujauU8sMI84+01l+YXU8TuBgYo5QaBQQC4UqpD7XWE02u67zJhUXnSSm1H0jWWnvajG1nRSk1EngJuFRrfdzsehqDUsoX4wPfPwCHMdbTvVFrnWFqYY1IGWcl7wP5WusHzK6nqdWcoT+stb7S7FpcQfrQhbNeA8KAH5VSG5VSb53pAZ6m5kPfE+vnZgL/z5vDvMbFwM3AZTXv68aaM1fhgeQMXQghvIScoQshhJeQQBdCCC8hgS6EEF5CAl0IIbyEBLoQQngJCXQhhPASEuhCCOEl/j8YGO12M35I9wAAAABJRU5ErkJggg==\n",
"text/plain": [
"