{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Activation Functions\n",
"\n",
"In D.L., our **objective** is, almost always, to find a **set of weights** that **minimizes error.** All of these sets of weights are **linear operations** and hence, if performed alone, we would attain just a simple **multiple linear regression model.** \n",
"\n",
"##### What’s the Problem with Linear Models?\n",
"\n",
"If inputs are left untouched, they are not **flexible** as they can only model linear relationships while most data out there has a **non-linear patterns.** Hence, we need to find a way to force our model to be able to **learn non-linear patterns.** \n",
"\n",
"##### How do we do this?\n",
"\n",
"After a set of linear operations, we apply to the new **input** created by the linear operations ($Ax = \\hat{y}$) a **non-linear activation function**.\n",
"\n",
"Suppose we have a simple linear model $\\hat{y}=ax+b$. These $\\hat{y}$ form a linear operation such as below:\n",
"\n",
"\n",
"\n",
"Well, given our orange line ($\\hat{y}$), we then apply a **non-linear activation function** so as to **transform** our linear model into a **fixed non-linear model** such as below:\n",
"\n",
"\n",
"\n",
"**Why is this a fixed non-linear operation?**\n",
"\n",
"Because whatever formula we use for our non-linear operation, **we do NOT** have a set of weights on it that try to learn an optimal non-linear representation. It will always follow a **fixed, single transformation.**\n",
"\n",
"##### Well, isn’t our purpose to find an optimal non-linear operation?\n",
"\n",
"Yes and no. We find an optimal non-linear operation by letting our set of **linear weights** learn a **representation of the data** that, **once fed to the non-linear operation**, will **correctly identify the new pattern.** Hence, the objective of our linear weights now becomes to **find a representation of the data that, once fed to the non-linear activation, will correctly learn the non-linear patterns.**\n",
"\n",
"##### How do we backpropagate these non-linear activations?\n",
"\n",
"Given that these non-linear activations are in fact non-linear, we are unable to just take the input as the gradient as we can do with linear operations (3x => 3). Hence, we will need **two things**:\n",
"\n",
"1. The **input ($\\hat{y}$) that was fed to the non-linear activation** and\n",
"2. The **derivative equation of the non-linear function.**\n",
"\n",
"Given that we want to apply the non-linear operation to every input, we can classify these operations as element-wise. \n",
"\n",
"This has important implications on how we can calculate our gradient. \n",
"\n",
"**First**, as we learned on the \"Linear Layer\" tutorial, the dimension of the incoming gradient from our subsequent operation will equal the dimension of the output from our non-linear operation. \n",
"\n",
"Now, since the output of the non-linear operations equals the dimension of the input, we are able to calculate the corresponding chain-gradient with a simple Hadamard product (element-wise multiplication) between our incoming gradient and our current non-linear operation. In other words,\n",
"\n",
"```input.shape == output.shape == incoming_grad.shape```\n",
"\n",
"**Second**, given that there are no weight parameters to these operations holds two implications:\n",
"\n",
"i) from a backward perspective, these operations are only intermediate variables and \n",
"\n",
"ii) we can just apply the derivative of the equation to each input such as shown below\n",
"\n",
"$$z = \\sigma(y)=\\sigma(x_ow_0+x_1w_1+x_2w_2+x_3w_3) = \\sigma(x_ow_0)+\\sigma(x_1w_1)+\\sigma(x_2w_2)+\\sigma(x_3w_3)$$\n",
"\n",
"Hence\n",
"\n",
"$$\\frac{\\partial z}{\\partial y} = \\frac{\\partial z}{\\partial y}(x_ow_0+x_1w_1+x_2w_2+x_3w_3) = \\frac{\\partial z}{\\partial y}(x_ow_0)+\\frac{\\partial z}{\\partial y}(x_1w_1)+\\frac{\\partial z}{\\partial y}(x_2w_2)+\\frac{\\partial z}{\\partial y}(x_3w_3)$$\n",
"\n",
"Now that we generally understand how to implement non-linear operations, it begs to ask, **what are some common non-linear operations?**\n",
"\n",
"Some common activation functions are shown below:\n",
"\n",
"* ReLU\n",
"* Hyperbolic Tangent (tanh)\n",
"* Leaky ReLU\n",
"\n",
"Each has their own unique properties and usually, finding the best best corresponding non-linear activation to a model is left to trial and error."
]
},
{
"attachments": {
"image.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEKCAYAAAARnO4WAAAgAElEQVR4Ae29d7wcxZk13FcBzH77vvz57bfBXrPe9WIMSEKA/NpeB2zjtHKOGOcAJgtEkg0GvCBQTggwBpsoCQQWJkdhXoMFXEAoXQkJSSjrKks3TDzf71T101PTMz3dc+OEp++v7lO5uk9Vna55qrrK8/RSBBQBRUARUAQUAUVAEVAEFAFFQBFQBBQBRUARUAQUAUVAEVAEFAFFQBFQBBQBRUARUAQUAUVAEVAEFAFFQBFQBBQBRUARUAQUAUVAEVAEFAFFQBFQBBQBRUARUAQUAUVAEVAEFIFBRuDd//EejDh5FI4bMwpHf/g4HP2RD+LYMUfjOMccO+YYuOaYMcei2ByPY8bQWP9C3KNL8nLzDewnH4PjTi6UIfl84EPHgkbcIgv52zRBPs49l/M79qT/hGs+eOL7Yc3R+OCJR+OYk6w5+qRjQPP+k60Rt8gPnvwBuMbNs2f2D+DYkwp5fuDkD4LmP8dYI26Rbtm096xMFwstX/HX9if9SvqZ9L//HHMcXPOBk4+D93de+yBTd8+KP37kCZArCyCDLHK5FHKZLmTTnUhnaLoDk8p2I5VJh0wWqQyN75+V+JK+0+TF/Mobid8Nyb8rm4ZrgrxZRpC/pIvKt+DP56HJ51JAPm2elE8L8KnzIQPkUGwEo/IyHLtad2mupXdUGqfgU2154fiFnMSm5Re3CsGlvAzjWa27NFfFv7bwl/pgzR7MdcEb7rX2jHEHOdUJx59oOY9P4lz5fB7ZTBbd6RQyuQL5kR6TmHCTd7Iua5X4kjcp2TXiL1LiiyybaQ88JT++Clwj/pS8B9dNe28vyU+ez3122sVfpMQXqeX3DgHBUfBV/Ae2/9U6/tIeUj4veENqhfAP89q84ca0eofF39TIESda9mJLJ+p5oKujM+g9JP1yl7zxwrJc3J74hfMVd0/y6kkaaYBhybxcv57kXSmN5C3EE5YSLrJSXj0Jk3zD5YpbwkX2pIxKaSRfKS8sJVxkpbx6Eib5hssVt4SL7EkZldJIvlJeWEq4yEp59SRM8g2XK24JF9mTMiqlkXylvLCUcJGV8upJmOQr5QrRu5JxvGHx3DowY/lhRrd0ZNLCjjthpMWFT2EuUisdeSCfBTIpIJ0CMpnA5LM5FJkMkKfx/YO42RRAwzwqmkLeTCv55HI50IhbZJB/cE9x+YfCcxy75+zz8RnzfFZ5pVDGXHmmdQzz65XhTygnP7kVuQ1xB9KNy7Rafu8wUPy1/bl9ytfwSv9zpM+MNUX4bZ7nJSb8D44+3nkcMkoWyGaxdf16tC1dig2rV2Ft24pos2oV1rqmbQXWraJZltBI/EI+a9pWYU3bat/QXmzKlyf5xEtDDkWEX470A3YN6ff5cnAbR2/JVtK7ebJZhUxRmW5cSd9b6eYZKttgFQoP7qe35Ur6UP76/MVtIMA7hBNHWr1+4TOPUL41j7+MMss8P3XQNFn/RR7gk7JYhbHksxvj5yXhgoFLBT4b1I4Of6jX7rV4rcYM8caVHekf4Y3z/h/zk6T1X//9veax+BMmhRzSyGHt6vV0WCU2+76JkUHOaLVTyKMbyHcDOZq0Y3y/fCdQzpj4TJsumFwW+TxNvthI+wv7l7glvSuL80IuD9esXfNW4SXHyuzVVYYcpaFUI91GVe6ewuHirqaMSnElP5FhTMQ/LCvlWU1YSb6hGwiHi7uaMirFlfxEhoovmdcP4mn9lwxOKuEcFSZ4iozC3/hXwJxB5K5ul7+YSH4J58wdMMqBTrKeHb8hmzcDXWTJZ0I+pWM9kwD5mpq0Pcon+SM9r4UzyWPKkr7vefQHj0Y6m0dXPo9DSKEDWby1ZqMFjHgQQPPPTmHmYAk/zylVEncua01A4v7LAJwHsIYvCGvSYDpL8ELQeWThG2Jezkh4UhnKQ17YIt9as87WW8x/aVbhaOIvMhyubkVAEeg/BEjZUX/mxUw2pyHHC4Xxdkjk/nugw5fUzZNzOOAscFkCwk8wP1qJd/slbKjnzfS8iFG+X+Lxo0cYUNwRPgkxl/ZZ0xC6/NSxpC8j8srknEX5P4fgSeAhci7rTkr0Ei+UpxC9SCX8/uuMmrMi0N8IRJG9pXOf4R2yN/785UCSy+aNnoJ+fCeQ0bLIOVzFiP4lvzZcaYJqZ4RP3b3o748c7nmLPc8bW+ltcuyo4+TxkEPePPpbb61GjpOshuw5ind1fPYnkBBzJg8ExgfQvhYIJsEtNUFaUbWFCFrCAylEnlSG8hOiF7lm9drgmekX/FwPfK2FQaaxRPhHhYeiq1MRUAT6EIFKhC99MixNH6daH0B7Hnhp41YsePoZc1cpZGFN3ugvgj7vEr1rp0qnJkb47/KO8pcLtQ7xPE7eltfhO2+AY08c4VQFnyqLt1a3Id3JHz0+GToxaA2DWdnN6ilOw4+oeFGVk8nlSkb5nG+5+Xe/xz+/+z349ne/hwX334crrvqN+bArnc0inU4bI7fFn2NXXHEFHnvsMZtvDOGvbivo8PlOCxN++H6lHJWKgCIw+AhEEb4dqVtSlz7MQWM6C3RzsSEAstobeeCzE+bgR9fOwZYccAhAF/LGUGHNtAEnuEQv9pohfIfIk1pLCT+P1StWBrXKh+dPH2rjCQzNQQKXL2N8QAlqlGE+xI0fdPFi/sFI3ifq7nQGx40YhRWr2kycUaNPwM5d7ehMcSam+CL5p1IpvPPOOxg50i4xDecnI3uRq1atNpnQneXPE39SVyaOpbGILC5RXYqAIjCYCFQifAkz90eiIX+l7UtgD4AdAH5805P42MW34NnNKew1ZG+1ESR7aiTMJeQeln6gVxMj/KQs78QbcaK/Dt95yk1vvWMfWogdgIC1HQDNzh4aAtxBJqX6J6TSEYL9xre+Dc/z2kefNAYzZs3B6JNOAkf2vL71nW/jxhtvNPbZs2fjjDPOMHZ+IPbRj34U7e3thUlgXwUkRC9yZdsq+6JheKZ0hZDch0hTgP5TBBSBmkBASL1U+spks1TV19/kgIOpPA5QlQNg6mMv4NMXz8FNf9mE3RzIktDNgC9rBp7MIf6qFZWOQ+RJraWED7yzZpNZpZPKAtN/fx9+NesOjL/xHpx78wKcc/P9OGfuQpx344PWzFmE88SIXyAZT4yNP372vbjnz08aTLk6qKPLbt1A8heC7exO41/e86/Yf/CQUe2Mu+hCE7+juwu79+3FP/zDP2DJkiX4p3/6J+zatStQ8Zx22mm47777qiJ8vkhkZC9S7kNkfAPQGIqAIjBQCJQSvfiQrv0hOQV/vXOO0R+kLly9ER879wpcPf8F7KOmgnF4cTCZTZvPa6gdiL/qmPBHjR5pdVbmKS3rrl+5wbAvCe/uhxZj/PTbcO6sO3DGnLvw8znzcOac+3CWmNkLcVYi8yDOmv0gLpx5N+bedZ8pkyTPiyALuVJSpfP+o48xL4Nr/uc6XH311XbZlI2OBQsWmF8At912myX7biqKgMsnXIq5N80JCDxY7hma7HVH+C7hS3y/mOCexB1Iv01J2wpkEMFa3GfqT3uoWHUqAg2OQHFvErqndC/+ou9O5Yw6ekMWGHvNLRh71e/MyN5ftem/H/xlmG7i0LyjWyIT1a1KRwjfQmUfa8NbG83bjs+/P2fVN1sAbACwHsAmAHT3xOzyN0Vj3iR6GdnTLaDS/z3v/Tds2bYD02bMxKWXXhpUBVUwEyZMwL//+79j5syZxt+sKAJw5ZW/xsyZ05XwA7TUogg0IgLCFFaGCd8N5VCQqpzz5szD5y6bjZf22vnFYJBmRvSSohgr8Q3Luib80SeOCnZiFMpdtmxZ8K4kYJwq5SQsJ2tp6KbhW7Ja4+rISPZU35S7/uEf/xkbN23BM889j5/85CfBCL+1tRUnnHAC3n77bbz73e8G7zWf5f47GXzr29/AU08/EXzYVf47gCyWr15lnpkTNBzhy8hepNyPVLS4A6kj/AAKtSgCA4+A9EwrXcLnTF933vIS72tLZx5TnmzDJy++EQ8v24xdXBjCgHJ92H8QN3fGdQ3DGobw7U+iDJYuX2o+RODDEReStBhX/WJfED5KlUQIXBI99fdimGf4+n//v3/E7r37cbCjC8cee6wh/IMHD+LDH/4wXn31VbMy55lnnsGoUaOQTXeb7wb++Z//EdlsqirC785VJvzwfRl36HmCxhOK7Dac/rSHilWnItDgCBT3pjDhH0xng+/8H3hlHT56wWyc/bunzIqcYKVfuT7soya5u0QvdoY1HOGvXL3CrpHPZ5DLZQyBBvvfmIXrFhIBplpJyJiGxE+yl5eIj3eg2pF8v/vd7+LRRx81we7umTmusDEbJGVx79134qorfm0/GPNzcBuCa1+12i7LtPdgX3NSFmXsVa6xlHlpuXn2pz32fjWCItBQCBT3Jrdvk5ipdaBG4pUDwOcvuxk/nzLfrCrkUnKqf2VxhsgwNMW5i96jIBuM8HNY2bY6IGRDsGbfHLtHjtk4LRjvCzRhyApuieFKhtItZB9H+BzZP/fcc+ju7jbbJReI3s7CU53z2CMPY98eLrSS3N1mUGxXwvdhUqEI1CUCLpsU920qiPmtEJeNf/26efjClX/A0j3Wr7ODmzty76/Q5oohDIpzLxC9+DcY4QMrV60zS1N59knKLGviBwlpZNCNDDqRQ5f/uQJllBZflECUBahcbGVdPHXnrDaJV7BZHzcNP7QyN1dmRE3it5fNobgpFFxK+C6ialcE6g2BYoYo9GzLTJsB/OqeJ/F/zr4e9y3fjf0+s/ApU4c6lfAL+ikCScJfX0T4/ALN7jVhSZ+wJiN9eRkI+UtF2QZmCN9sd8w7kDjchlmo30r+9Kp82fBusx0E7TYHtyG4diX8ymhqqCJQ2wgUM0Q6lzLbJnD4yQ8772zdiDFnXINpjy81H4wekA/0jTqa9KAjfDMTTVIkWa5a9ZaZrSU+Rs8erGO3615c8iy22z3zuW8+t1EuNmVG+uRmw9UM48tBfjH4e1Rzz+pD1MbZSV6uz0+lOK/A++Rupzm7eie0NYIJrPAvTPgVomqQIqAI1BoCAW9YvurMpLDfbO8OLNkLfPqSWfjhtAexicvree8cT/Lcjjw5Rnio9KHkNSIh4g5LhtfOAShJP7H147nLMoXwV69aFRA+SV8mN4zk04euYkDkFVAYsRdG734NSIWZ2mDqUsKXIsoUJ0FG7t+/36p4nLc2XwTyUiiK7DuU8Muhon6KQB0hYDjEckwKGaO24XdC3518D747aQFWZGG2U+B5JoZeeNSqGYhGM4rwmKAg7rBkeAMRfgZvrVxhtCJyhqzo2kWa12YYBZfEAyIPR3L2T5Agk6kQvlUBcWmle/FXhlxcpsmLL5/OTvuFrdE/+YQv8SpJJfxK6GiYIlDbCAR0kO0CcgfN4UrUA1xw57P4r8tuwTPb8tgm++QU9NVmoE/+D9KHHlMoSbzFHZYMbyjCX7tymT3gO53ylz3mi45xLVGyhxERt7whoiTfHOb0c/98Sv/nFkfnc+bMwb+//z/NFsnjLhpv6oDr9ufe/Dv88Y9/lDqxMqTS4TLODRs2oKOjo2gbZUmkhC9IqFQE6gsBUgs5nJSCXJdR0+zPAvctWYGPjpuK3zzyGrb6pM5FJ4EGh9/++IpjJi13CW1JmLjDkuENRfhvr1hqiZhbGMdOmAo8SaX/U4BEb6rOfwWbl4LV6Xd1pvC+9/0H1m/chIUPLsLEGyaZzDnS/+Bx9tB1Udl0ddkRv5RO/6eeegokfS7jLHet9tfhM0zyKRdP/RQBRaC2ECBNGCI3a+o5pwg8tyVtJmnP//2fzAdXASOQqUXzIC8BX4ZJ3HUneeKGIvyNq1cAGR5EzjdoJ5DeA6R2At3brencCpQ1m4FOmnes6doAFJm3ge7NQIo76hBivlA4uvdVPX7lfPObdnvkE048CReOvxg33XKrqYOnn12M75x2urGTzLnFwuOPP27c1113ndlzRwj8Ax/4gJ3QLVN7SvhlQFEvRaDOENibA1bsB74zcR4+N+Ems207t0HmgYV2PF9M+ML+LrmXsyeBoaEIf/kbr1rAutqx9sk/Yu38a7H13kux5+5zcfCuM7Dvzl9gz10R5u6fYY8xP8Geu2l+VGQ2zL8Qa19+AOn0LmQzB5BN88XiVAztAN73b/+JPftYfXb3TMrxl1yGOXNvMqNyjuxXrFiB9773veajrJNPPtkQPF8E1O1zhP/KK6+Y9OF/SvhhRNStCNQXAvty9lyOM25+Gp+/dDaW7M6Dw8hOQyZU3tC4vEL1Mef8aEqXfrvEnwSJxiL8N5cC2W4guwdvP3sv2ub9Fpvvvhi77/gl9t3xE+z544+x646fljH0/zF23flD7Iwwa+edj6XP3Y1cbj/S3ZbQyxH+B44+Du27eeyKvbgk84c//inuvne+ObSEp1xx50zq+ltaWtq5P75c9P/Wt75ltlEOJnYlEIASvgOGWhWBWkNABoCh+5JxO4PXp4Cpf92FD427Bbe/8LZZf88vbLNGaU+yZ2yf8MnmeS43PGSNEr6FR5Zlrl271uw+ad6SuQ4guxtI7wDSW4DUJiD1jlXNUD3jmhTVNb5JbwbKmcxOINdp9udJZ7qL1S5+RacyWXMAyq49PKbArsFnnXHydubsG4M0XKlz+umnm10zeZ6tvASY5mtf+5pR95D8w5cSfhgRdSsCNYKA4YDS/el5TCHH5tytl9smvLQb+NBlc3DFoyuwdr8968QMHM3Inn2eGfmXyZP/5EVANrFKZXdkL3ZJFi3reD/8cuvw16xbG+xTX4DGrnk1095mPwq7J4UsiXS/XpNthrn9cDnDvXPMHK35yjZcMRbm9/3H+81HVyR/Xhzh//72P+Cqa35rRvb0W7hwoTnHdtWqVXjPe95jTr+iP0n+X/7lX8xxhyZx6J8SfggQdSoCNYMAGcdyjRBwmlvgANiXtWS/vBv48pWzMfaam7Asb/fJKay1NOxeTPjlVAhK+ATVQmwJX1xsCYVKKFrn5GLr8LZUFKm6rPF3yJTN0yS+SJb4z+9+D3bu2m1ePNIW31y+EsePPME49+3bZ0ie++LzmjVrFr75zW8a+969ezF69Ggz4jceoX9K+CFA1KkI1AgCZB2qZbiJC7mDnMATCDu77Xf4PIT8Fzc9gE9cMBFLd3eaVwO/2mG84itMTq67OKbwjsji0HKuhhzhDx7hc709R/Q879a9uM3C977/A3BET3VOuYs7a06aNAnTp083wWaztVBEJfwQIOpUBGoEAbIOyT7tr7URIu9M231ypi56CaeMn4JbX2wzu2JyHx3RPUhc+yguwYftxQ8rRC+yOLScqyEJ3741LYgCRUhG4BiK5f9uKOQXFy4jf5Eu5G+8uRzPP/98MHrnx1Xha+rUqeaFIC8FStco4YcRU7ciUEsIkCEcnUs6a07Yu+eN7fjYhdPw2/ufNZO0nOHjlzbUzJP0C1cEMRUiFNnCfFQUWNbR8ITPpw7DUm5fHItOmZjlUkf6CdGL9KvfZG78nIlYIXK3XrgWn3vsUJfP5Zt0u0YJ30VL7YpArSHgc4vodHiedgfwlatuxQ/nPGjO1eaumFyVw+EeP7QKCJ+//M2vf5f0Kz9fmK8qx2ZoHRP+CaNHBnp2UeKUTtryIV1YBMx4aMrFcHMqZxeidyXjhS8ZwZfzd1fnMF4U4UflEc5T3YqAItA/CAgH2NzpsntqkZi4MS53vfzxxLswdvwMvHnQrr/Z0bE/0N9zlB8QPpOTnoomaivft5QvsnJsm7d3mNda5T6VtRG9Xgif5M8K4UU1DgmcF3X25S53/b1L9rS7I3wl/HLoqZ8i0F8IcPQue2fZMoqJlpO1PH2DcewIftKfluGzF92Ch17fafe3Rx6ducK2KVaHb1i+QBJK+OVfMCT8AuB25E5y3fDOVry5vA1r1r5tJkk5Ubp61QprVq/C6gRm1ZpVKDGrVxvCJekOlnGbshK+i4baFYF+RoBknz1odrkkOwv3mOGb2V/LjtX5n+qaR1/bjM9ffCt+s+ANbM9Z9Y3V9JgUwSDQukID+wqKCLfcsD0egTpX6cgDF34GFR6ZG5al01mz62Q23Q1jsilwC+M4Y06i4Wk0IUN1y2AYGemT5IXoRRaeWG2KgCLQbwiQ1HMFwpdl26Y8/4CSg9kMOCH7SnsGX71kBn425UFs9HX1hfsS1qKMuCII301Zzh6Rm+Ndx4Q/atSI4C0rhO8uh3T3oi/o8QXJnkoHO7UqAopAEyFAiuX43Y7uyxH+7u4ubMgC37r2Lnz9mrux6iCwuxvIcHa2m1skEC6XqiPgE3oKBbspy9lD0cs465jwR448PoBOCJ8g8CLZp9J5ZDP+iNjsOy9T5+WgKvZjlZb787NXoQgoAk2IAFlCiJ7Un3Lm57jEkpugXX3PY/jkBdOwcPk+uxLnUDrYHqeY7IWtygCphF+qxx8x4riyhO9SdwBlAKCEBiG+RfzjZDiduhUBRaDREXBZgYRPshfC78wCh7IwRxXevmQjTrlgEuY89II5ptDgYs5a9Qf2SUf4/QZoTY7wW1q9od6iUoov9jn++GOV8PutYWjGioAiIAi4hO/6kb+5EHNLN/DSDuCzE27FOb97DLuYgFeem+n4G+qUDDolkh93QETtEf64Fu+wRZ7XkpjwrVbNoulWTBGcsW/WcMoo94DUihaiCCgCtYiAmbjlsku7dQpV8/sBcFes066bh69ecweWya4qPLeW+9jn0laTEzyPyy2B5wBZaovwj/K8lsWeN3yM5w1NTPgWvjxyoCnWkhVQjA6xccLhUe5CjmpTBBSBZkLAX4dvTrsj1efMlsfcFO3K+17FJy6cjSc2pLGRyvzgStv97AM3LS63FAUMgKOGCH+oZ9Q4YyzhD1tcrMApdX3w+GMMQIRPtjIOT8taBHsCsJvGtQ9AnWgRioAiUJMIcJm2XSCSxr6uDuwGcNdf1mDMz67BtKdWGTc/qyJjlLtcJomKUy5d3/nVDuGPPdzzZvq0PiZSpXOEN877O/NpcOt7jnpPgAMJ392lToAtfptWA7HkEJZBkWpRBBSBJkJA2KMr242OdKdZgfPc+gP42uWzcP7cB7DNX2/vMkYYHjdM8gvH6V93rRD+cG/mcM9rG+557cO9lvZhntfutVSeuB1xwogAG6pzoglf5tSrgThcNeIOilSLIqAINBUC5IAssv74nRugfeqyOfjSNbdjfbfd/dLOJxaGmWF4hEVEhsP7310rhO9obIZ73pjDrHrH8S21HnfC8QE+XDPPihAgRVrolfADoNSiCCgCPUSAO9faPex3ZoEJtz2KT148G49sJr/wIutEk70bVuAnk2QA/9Ug4XueN7YlAeEfO+o4BygLoQApsgB/wcdJVMEq8cOyQhINUgQUgYZGgFuyHAAw+y+rceqEm/HA0p1Gb28WAfpjfzJG1NVrNpGlnVEyquDAvzYJv3Q4X8YnGOETRR40G3rDFoCXsIJP8PyRFjeNa49MoAGKgCLQwAgIe/zlrR34yAWTMX7Bi+bLWnIvVcpU9whTRMEg4SKj4kX6RxG9+EcmlIBGIHyzNIdLpmyVCJjWxQct9ZHHj5ZuGtcenUJDFAFFoHER4ELMZfuAr10+w0zUbvA/uiLNy3YLcU/vMgntVV9C7FEyNsM6JvzjR4+wK6SU8GOrWSMoAopAAgSESAN9gT2GkGTPk6rOnn0fPnXutXh9F4xqx2rvRYcfn78SfhlVTVKvgPCJYkWVDisiDupweJQ7vlI1hiKgCNQhAtwITYy/SRr5n1/SbgYw5aGX8bVLJmH+iyuMKqe9C+D5G4ZasjmAhu6avup4hD/ixJEWYHKzT+hhmi5gHx1i44TDo9yFHNWmCCgCDYSAkHfOsgk/mOVnVlsBzF++Hx8/Zwomz3/SnFzFET+NEn7S4XkfxFPCb6DOpo+iCAw2AhzjUTtDFbHZ9jhnPq5asT+PT190I34waZE53GT3gTy6c8ChLj963hJ/fRxIVMcj/JEnjUo+wjc/tdxRe7h1uWGV7OF06lYEFIGGQIAcQcKnydvR/R4AZ0z9I74y4Ra8sMVug8xnzfOQchT0+xzt89cA/Wr7aibCLyL9cLVUInk3LJxO3YqAItAwCJDsO+3uZzzM5JJ5i3HSGb/Foyu2G0I3z+mwOmOS7PnVLaUJIs+IqTlglPD9KnFJvZK95mpQb0gRUAT6AgEh8qwd5D+8bCtOnTAXEx9fZiZuAzKXeM4IXwm/D3T0cVmoSqcvWrnmoQgoAgYB6u0PcogPLNsPfOHiGThr1j14x1fvWJSKh+5U98vkLqXzLrDRa+5/s4zwTVWwOsSEa0L842Q4nboVAUWg3hCQXs77zlMh71M16XxbB/DTSffi1AsnYW03zMRtZ9pX7Id0NcyHqblap5BLLaPRFIQv1UuZ9HLTuPak6TWeIqAI1BoCpifn7ahcRuTcZ7c7vQ/7u7uxE8B19z6Pz15wFZ7dsMvo5hmvQObFI/xae774+2kEwmcd+CN3l5ppF/+CjIfExgjnJO6k6TWeIqAI1BQC5Al2Y19FY9bR+/OrvE+S+m3Pr8AnzroKv3v6JaO3l5cCJY29DOGIo85kHRP+CVyWySuW8PlzjMa+ApLVkBB8WCZLrbEUAUWgxhDg1/iZlPkiluQtSynJDLS37klj7ITJOOfGeWbbhJT/ARb3yuGSS5qkl7BGOL74iwyH979bCT8CY6mSsIyIrt6KgCJQ2wgEhJ8xQz8SOMn+IIB1AM65+QF84+pZaOsGdh3keJ+j/hzSyBiyL4zw4x9TWCMcU/xFhsP7390UhC/wUia93DSuPWl6jacIKAI1hYCodKQ7+5oZnkt79q2PmdOrnnl7H3bmCuqbTqRBwyNUzXZdTOOaKh9QihZZZfI+iN5ghE9EBExKe5X6SEi0dNO49ugUGqIIKAI1jkCoK9N574ur8bHzbsDkJ4pPPzsAACAASURBVJZhaw7o9DmdY/s0UkgZwvf3zeHjKeHHrZjvn/BRo0c6rUtqUgnfAUWtikCTIFDo//aBrTvs644GOwA8vTGDz1w4A+fOmo8d/hezTGNZJAXkadLoQt68CEyYEn7/EHpcrkr4TdKX9TEVgVgEhNqLZbHL0jh18SR7LsH8+m/vxFev+iM25YG9wVp6MjpjcYKXRgk/josHJLyY8OW1G9UyJFyaQDie+MfJcDp1KwKKwGAjkIP8cUJWVuTRzy63lFU2XI3DSdq3Afx42kJ86dc34fnNaezlAh4TV3jCJ/xcGshlkUXehCd9TmGRpPEHLl4d6/CV8AeumWhJikAtI2AJVkhf6JbSjuq55oakvzsDtAO49vFVOPncSbj75bfNSJ80z8tuccx0pH9O1abNl7j0N5O2frw4IXcQF2/gw+uZ8EeNcPCSN7PjVWSV8KiqEP84WZSpOhQBRaAGEWAvtpd/ChXX1OeBXTngqfV5fPiCGRg/7xlzklWn+R0g0ckTlgO45ULB5P2XgeRbWQqLVI41GKFNQ/gEt1CZpVBLFcXJ0pTqowgoAjWCgIzr3NuhXwZI54F1aeCLF9+EH02+F9v8OLk8FT28uHuaS/KuXQl/QPT0lQoZVdUI36/TYJpe3CLjiF7CJb5KRUARqBUEpHdyFO8O2LOZrB3nAeZowrNufRYfOWcyVmztwqGOrH8Udh55dBulT0GF45I97dURfq3gUnofTTXC5+NL0whDIf5xMpxO3YqAIjDYCEivNZucBaN8+gIH08BeADc/vhwnnjUJf3zVH9vzoBN/fpeEn0Wno7NXwq802B6UsOpH+PL6tw2huJFKk4mTxanUpQgoArWBAHsuJ2btpgj+GnrA7Itz7xvt+K/zpuLX85/HJud2ORFrtzbmShyZoA2TfcHtJK1Ta9OM8OW1L4Qeri/xj5PhdOpWBBSBWkBACJ9rayz1203PWndm8OVrfo/Tpy/AWxn7AuDAnpche7P+Pg+z+LJoorZA9DJ56yerY9EUhC9kT5n0iiL+pOk1niKgCAwkAuyxHN2TxHOZFLpyMBOzZ0y7C6dedB1e2WfX4AvZy73Znm6XdBZUvo3a/xue8F2yV8KXRq5SEWh0BPiB1WW3P4xTL7oe97222nxda8ne5QSX7qNI3vWvd9SagvDNe99flpm0wtxKdu1J02s8RUARGHgEuM4G5vCSR5bvwGcumIrJj7ZiV7B4xyV72t2+ncQ+8E/UtyUq4UfgGVX5EdHVWxFQBAYUAVkqab+OLRR9AMBre4FvXn0rfjTpLqO3txO5fBVIvxbiF3dSWSinPm11Tfj+iVcB8qzE8EU/HeGHUVG3IlDvCJDouc7eJXxubbw+BZw2aR7++8rbsNXfO4cTsuC+OO4ifQNAUqKXeHWPGrzDvNZBWVYZKvRIb7jX6g2haWnzhnnjQuElzlGjlPDrvfnp/SsCPUUgPMLnsI6j+yvueRwfv2gm7nhtO/aR4nMZS/b+RmjuC6Iw4hdCj5M9vdtaSVdbI/wjfVY/0hD/cG9MCcs7Hkr4tdKI9D4UgYFBQOg4XBpJnGR/68sb8clf3Yipj79qPrYyk7Q+0XPbBG6oo4RfGyN8h8q9I70Wr9VLSPhsBPZSlY4goVIRaEQEhPA5mpd+T8kVOfyg6lMXTcHPb3zQ7IBpTq7KZ81+9natpiV7JfxaIvxhvJmWdm+IN9N9A5Szjxx5gmnTfItzJ4xyGjq7kUa1OnxpVuVkI3YjfSZFoD4QYI9kP+d2Z3b3G7sih/vbf+6i/8H3LrkOmzvsjglmWtZ04Xywn059PGV/3mVtqXSE16naWex53lHiEUjq9qnr97zW973vPwwyJHw2ABq7ztYFrCeTtuWIXvzcvNWuCCgCA4mAS/iH/OWX73C9/QMv4OQzfoUlazab2+k4xK0VYDZHq2Yf+4F8lsEpqzYJ3xvmeeO8IZUnbk86cYzBTBoB3/i0F19K+MV4qEsRqGcE7ClWHOWT8HlM4e1LVuIj516LiQ/9zYz++XSpbn8f/Hp+1H6599ohfI7qg0nb4Z7X6g31xgYj+zKW0aNPCiDhyJ5GCT+ARC2KQIMhQLJPmSMMqaQl4T+/rQNf/PUMjL/9EbONQke5abwGQ6F3j1M7hD9miOe1+qbN4wg/5hp1wsnBqioSPRtB6aUj/FJM1EcRqD8ESPeH8p2G8tnfNxwEvjPpTnzh6t9jiz9xWzrgq7/n7N87rh3Cj6H30mCX8M0GeWWRUsIvC4t6KgJ1hgAHdPwVz9U3+wGMv3EeTp0wF49vBdq5L7JeCRCoZ8IffaKdfedrnbxe9uecEn6CVqBRFIGaQIBdOWqULr/gOZq//rll+Nils3Hny5uxXQLSXJypV2UE6pjwR4waXSB8If2Sp1XCL4FEPRSBGkVACJ8cTiNu3i4nanly1ZPrtuMTE2bhovn/1znwhOs0qdXXqzICdUz4I0eOLIzszei+3NhACb9yA9BQRaB2EGAPlnX2VN108ODBDCdqrX/rbuDrE27Ez665GVu6in8NZMt1/9p5tBq5kzom/BEjRtiff+R0JfwaaVB6G4pAzxEQYpfvajpz3aZr78/B7Itz9uxHceo507G63XT4oCCmK/0GJwhWS4BAXRP+cc5PPlZ5uVe8jvCDulaLIlDjCOSQQRYZM8qnSqc7dciobdoBTH50KU49bxruf3W72UpBVPd8pGLClxFgOVnjAPT77SnhhyCWF0eUDEVXpyKgCPQRAiRoKnTSyCIfDN/25ICFq3bjI+fcgBseWAKSPz+yDF+F4V45ohe/cKpmczcF4UtlU8ZdUUQv/nHpNVwRUAR6hgD7Jwm/2xB+ZxrYnwHWHgS+eNksfH/qfPNlbYcfq/yqPCnZ7fOuXcKbVTY84bNiq6lwIfYo2awNRZ9bEehvBNhP7TaIVNdwkSVH8z+//nZ8+vxJ+Ms2u9+9UeVIl468JYkQlpEJmiRACT9U0VFEL/6h6OpUBBSB6hEgD5uL38+yb/HKAdkuIJ82E7A8ueryBX/FKRfNwiMrdhX09owuPO6nLBYSWE4Wx2w+lxJ+qM6F2KNkKLo6FQFFIBkCZflXCN/2t3Q3t06wX9LOe209xpx9A254cpVR5ZhCmEcuZ2dpaS+5yhXi+pUkaDKPBiR8l6oLten60h51heOF3VHp1F8RUASiEeDulSRq35g9i6VvcUGl3RSNGnyqcl7vBD5z8SycO/chrM9bvyBvSVvC45KffYmU+x/k0bQWJfxQ1RcajR1GhN2h6OpUBBSBBAj4/YhkLYRtVDnSv+xSzD0A1uWBL195O045fwZWHbLTuCyAMe0lafwtkM1I3/EzSqJydF/IQXJqPqmEH6rz4oZTSvqh6OpUBBSBWAQKvUqIWDYzL+xbz4+tuL/9FfNfxCfPmogn3zpgPrbqzOatyj5PZi/kVMkupYRl7I02fIRmIfySUUBUzcY1qKh06q8IKAJRCLBXcXUN9fNpMwIn4WeCydds3urtb1q8AaeMm43Zj76OXc6oPpvuBk0lknfDwkQv7qj7ax7/RiP8vFvtTjWKvi8YIThhxhpH9BIeTqduRUARSIIACV+2TOAXtST8g/u40TFwIA+8tgMYe/ntOPvmJ7HV7KNj+zLyfDFkgFwayDEX6YvRUgk+qkb6mvCPMAeXyMlVpZvY96HPiBFltlZQwo+qafVXBAYVAdIzR/h263pry2byZuT/Thb48cR5+MHkhVjjx6Eqh2kMwQvpJyB7xpe/QX3gmiy8rwl/uDfTa/HavBZvkedVPqKwt9xflvBD7/8A81iVTvRooXhEEeSoFkVAEagSAUvg7KQ8ZNxS/6Zu4JoH/oZPXDgTz23OGz0+Q+QFwZU7Aekr4VeJeDh6XxN+gcXHDPW8RUM8r22I5830PO+oQlDf2Koi/JKGEgZCCT+MiLoVgT5HIJh3tZO23PTy7iVt+Ng51+L6h1oN2XNX+/2dVP64vwjociZ6S/pzcf/VEX5UzfUv4c/kYeSG8FvMoeQk/j67khO+2xiigAj7u2lcezieuhUBRSApAuxJHV0kbntU4dO7gZPOmYgzJ95uji0kzYuWnjO60vPseL8c4SctWeNZBPqa8Id54zyS+zBvsTc0pNI5zOPh5H12JSN8aTIik1a8xA/LpOk1niKgCJRDgLTNj6vePAB89qrbMPa3d2Ez52N9NU7BJj6UTKWEXw7P6vz6g/Cj1Td9qtZJTvhuY0kKj9vYXHvS9BpPEVAEyiHAUfwOAGfd/BA+O34WntvIMX2BzisRvqhqRJbLX/0qIdDXhN9n4/f4jJTwK1WshikCtYnAbgBTnl2Jj104A/OXbDHr7TlJK+p9Jfz+rLemIPyejNDdNK69PytD81YEGguBVCqFHPfP4cbHWavKeX7DQXzmslvwm4eWglsppPJ2fb5V2Lh9rdQuI3uRsWjJ9zciYxM0eoSGJ3xWoNtwklaom8a1J02v8RQBRSCbseqafD5vdrp/Owd8/cpb8f3JD2KtP3HLrXVSyCOFrFk/X9xf3b4nNF+QsQgL0YuMTdDoEeqY8EeOPD6g8kIjKdjYVOzlNhrxi5NuGtcel07DFYHmRUB6iosAyZ7XAQDfn74QH75wLv6yCeDJVfZ1QJnz6V5yKC8LVG9tbjll7UL0IstGaiZPJfyI2i7f4CIiq7cioAg4v6Vl2JXO8khygGvr5z78N3zm0jm44829Rm9PwNjL+F8Jf6CaTx0T/qhRIxKO8KVpuSQeBtgNq2QPp1O3IqAICAK25/C/rIwD9gK4a+l+fOScqbh+3lNmZM9JWp5ZG+5pko/K/kJACd9HNtz0otz9VRGaryJQ3wiwx9iPpmizGnkS+6oM8MnLb8NP5zwGflkrV9b5sEp6m4Sp7C8E6pjwTxg9MhghFMYKBRsbUeGSJiWyEGJt4h8nw+nUrQgoAkSAPYf73lA9Q8Lnevu2HPDTWYvwqXMmYlUnsC9r4zF+Z4rbHes1sAg0C+GbkYVL5mGY3bBK9nA6dSsCigARYK8h4XNUT0lVzsX3vIAPnzcNj6/YjN0+2XNk39HdBUq9BhqBRiN83R55oFuQlqcIGARI+KK55+j+zyu24eMXzsasxWsN+cu6GleVo9ANNAJK+D7ilUb1bthAV5CWpwjUBwLsJZ3ZLDoBLNkNfP7S2fjJ9Pux0R/xy+YJ2psGsz5rhfDf5R3lHeEt9oZ7bd4wr8073BykUnF/hap0+KrSGcxWpmU3DAJC16UPxJDuXB7bAfz3NXfj1CvvNBuk8QWQ4YdV6Q7/YJLCPFtpLurTvwjUCuF73pHecG+Mz/BHekPMzpoVN1sTwpeVAfKTUZokZeFyfYtDbJxweJS7kKPaFIHmQ8DvFyH1O33thC1ww71P4KRfTsEdb+43WycQI/nAKtyrmg+/wX7i2iH84tE8T8wKb69cHMM7/vhjg4miLuSQMh9wFEYPbFz2cpuZ+MVJN41rj0un4YpAoyLAfuCfL0vCzwGdnfykitsi2Ena+1/djg99/3LMeuglQ/ZMwTQkfHkhaG8azPZRm4R/VItnRvilZ+Nyv/3hXqvnea1HHfWvAeHzxyING1Vxg3JdtvklgzucTtzJUmssRaDxEGAfILXT2H7Wmeo0ZM4dMJcdBE4dNwsXzH3UbKOQNS8F+4Ig4TOV/TVe6KONh1GtP1HtEf6Rh3te69AE5+GOPnGUQdc2phw4yi9tUGyksnaA9qSXEHxYJk2v8RSBRkOAfYHrbzqQRgop06/syVU8ePwb192F0yfeixWd1i9M+DrCr4X2UGOEP9TzFh/uxU/YUrvDrRV4sRmmzeheCb8WmpTeQ6MikEMub39Lk+4PIWPofyuACX9agk9cPheLt1uy50RtOp0FuBVmnrNrpaN7+uk10AjUEuEf5i3yhpgDz0Pa+vLO40cdH6DFJlVuc9XCj0c2r2qamMQPy6BItSgCTYZADuksdfZZs3ECP6zaBeCOv67Gf42fjqsefR0kf+6Cyb3vzT74PuETqHBPqqY3BkBTTeSaIEAtyRCoFcLnCp3hXrvRz1NHT3N46EzcEO8HhM8G4C/4CjcqC4LrmwyW8s2zR000aYEaTxGocQRySGU7cSDdgUPIG4XOI6vb8YXxU3DurY9itb8rpgzs7cNI3yt2FftW8dgu2Zt+X0VajWrelt5hZg40xKaD4yydpK1wHyNOGFF42/vjB2lIIkubWdJalxzCMml6jacI1B8Cpa1dfArP0pnOmFH82gPA2MtuxPcnLcAmam6oxqHiPufOpEm60nwkpCqphF8VXKWRa2WEX4HYo4KU8EurU30Ugd4gILQssviXbiFnHk141W1P4dRx0/DMO2ns4YwsyZhkn+ciiUIONlXYXcirKpsSflVwlUaud8LnE7ERVFTp2PBCI2TjC1/SIONkOJ26FYH6R6DQ6u3ni4UxuoRkDIUfygPtPMzkhR342LmzsPCVjdjSCWQYLSB54zCgSOoAISFs8RB3UinpVPYQgUYgfIfupYGJLKAiPiILIdYm/nEynE7dikD9I1Bo9QXCJ+lb4rd+XEdPsn9ucxYfPf9GXLGgFe902kPIzag+zxiMW7gk38BHiF08xJ1USjqVPUSgjgl/1OiRwUOzmZVfpSNRpOmJFH+R4h8nJb5KRaBxECht9XaJMzU1pHGuvufHVX/ZCXzu4jn48cS7sD5rz6llWuS6rPEJX/KLRSgp0Uu82Aw1QmUE6pjwuZeOVefYcUVFwmeDKfOTswCONNE4WUihNkWgURBwW719psKongsxSfZbAPzixkfxuYvn4rUdGRwMvrn1CT/PBZlW9SP5xeIjRJ5UxmaoESoj0EyEX0T6YVikicbJcDp1KwINhEBAvLYfcHTP9fY7AEx6/BV8avxs/HlNJ7YcpLLHqnyMheqcPH8HFKt0SpAJ8g+tp0/qX5KhelSHgBK+j1cc0Ut4dfBqbEWgrhCQZp6z25Hsz2XMx1UPrm3HZyfMwBXzXsBO54Ekuv31zDTW5kQptiYl9qh4xbmpq2oEmonwVaVTdfPQBE2GgDC4T/gcs7/cvg9jr7sJP7npQbwDGFWOnEfL6IUVPQWlaSRqYSKPjKgB/YNAsxB+EdmzmYavoKWHdP1h/3A6dSsCDYIAyZjN3Syjt5O2VOWcMf0P+OiFE/HiQbuVgh3H22dmdFnN4xJ/JCJK+JHQDExAUxC+S9pJYXXTuPak6TWeIlBnCBgytpudcV39rhww4c4X8F/nzcDCN3ca1U53nT2S3m4YgYYnfJesaU96hdOJO2l6jacI1BsCbONkfau2efjNnfj0BbMx7cm3DNlztY47uq+3p9P7JQJNQfhspuZ3ahV1LgQfllVkoVEVgTpDIJ/NgMS+Mg98atxcXD73IWzLALu6gQ5uk9PHzxPuXXHuPi6+CbNTwo+o9KimFxFdvRWBOkeALZ6TtNTbnz71YZx66W3YkEowEduL547qZVH+vShKkxoEmoLw3eaTtN7dNK49aXqNpwjUFwJcc88llzcs/BtOOX8Wnt4Ccy4tXwL5vOyCWXgBsFf09nJ7VhJ7b8vT9A1P+KxityklrXI3jWtPml7jKQL1gQDJPJvJYl8auPXpNnz6nOmYtug1Q/77/VE/dz3O5/kte896UxQS4fzi3FH5qH9SBJTwI5CKanoR0dVbEahzBNZsP4hPnTsd596y2OjxD+TtcYUc4dPwF4C79JI9pLdXVC+L8u9teZpeCT+iDWiTiwBGvesaAWnXdqTORzmYAda1d+IXV87Ed3/zByztAEj2vDL5wuZpJH0lfItL/f5vCsJn9UhDFxmuMvGPk+F06lYE6gUBtm27Yo073nAnTG6AxknaKxcswSd/eQOeX70L+zKW2PlUVOW4hjnoVc8IKOH7tRdH9BJez5Wt997cCJDmuZslTc6oaXhy1eTHluMzl96GOYvXYw+H8P4l54+7UsJU1isCSvh+zQmhx8l6rWi972ZHgC2b0652G/Gc0dM/uQ341GW34fxbHsGWnFXfuDi5ZE+7XvWOQLMQvtFJumQerjg3rJI9nE7dikB9IMBWzQE8J1+pj98F4HMTbsU3Jt6Ht7qBvVTfhB5FCT8ESN07m4nwi0g/XHOVSN4NC6dTtyJQHwiwFWeQxwEAmwFceMvD+MqvbsLf2oGtGd02oT5qsbd3qYTvI+iSeiV7bwHX9IrA4CDAVt2ZzYNr6yc/uhSnjJuJu//2tnkBcPKW4Xo1OgLNRPhFK3XCFVuJ5N2wcDp1KwL1gQDVNdwn5+XNXfjYeZMx/s7nsTELcAdM3RStPuqw93fZLIRfRPblxjIuqVey9x5yzUERGDAEclYrzxbNE2epyjntNzfhp9PvR1vGjur9JfdyPPSA3ZoWNBgINAXhuwSeFGQ3jWtPml7jKQKDgYDfVmW+KsspWvtxFXX3Z87+Mz4zbhZe3JIy6/AN5ZuzaPliYKI4MxjPpGX2HQINT/guWdOe9AqnE3fS9BpPERhsBHJArhs8zIR6+5mPLsOHfjkZD67pwg7ujcPbI9nnO8y6fPtbQAl/sGutf8tvCsLX/fD7txFp7rWAAIck/HqWxpK3vSv637/8ED501ixcfd/fDPlzkjaTZ7/gL4BMsFSTcW1q5lBu1C8DH5G18OR6D8kRUMKPwEoadFhGRFdvRWCQEWBLJdmTwl3iX7WHh5ncjJ/OfRobAewL7rOwzYJskMY8lPADgBrQ0hSE75J20jp007j2pOk1niIwOAhw75uuHLDXn6Q9a9Yj+PhZk7Fkl1XtkOa51bFdm8NtFgovCtqtskdH+INTe/1dasMTPgHsCWG7aVx7f1eI5q8I9AaBAmVzn5xL7nwRnzx/Lha+3m4OM+Hon2SfR9oYS/rcWUfW4YsOXwm/N7VQu2lrhfD/tzfT+3uv3ftfXpuX8Dph9Eh/5omNVfYIKUfv0pxtZ0hWGW4a154stcZqVgSkrQzG87Nse0gJV+Q8/OZ2/PclN+OGh940ZN+RsztfGsL3Sd8SPl8DYlxlkOrwB6MW+7fMWiH8I7wxHs3fK+H3b4Vr7v2LQIjwOWDOpYDMbiCzDchuAzI7emayOwBj/HyyWwCaHI3vx82Ou7dhx85NOPuSy/Gr66ejk7yd3g+ktwOZ7UB6K5DZ6sstQGYTkNloTXYzkKPZ6pvtQM4xvP84E/V8cekkXNNHt4802047kD0IZDjzwvZWzVUrhM9R/bu8o7zDvdaEA3zvmGOOTjjCJyDSEUWGQRL/OBlOp25FwEXAaT9m3SP3M2gHNj8OrL8T2LoA2HZ/9WbrA8C2B4Dt9wPbmcc9wLa7gD3zgI23Ahv/AGy/z5axYT7mXvBJzBn3eWDjE8DGh4HNDwJb7gc2LwC23mfL3878HgR2LLTG5H1/6B5ZlmMkTkXJfMsZPreTV6Q9Cp8F/vMTg0qmXNn0q/Pyt/L+/wRsfgTY8QrQyZdyd+G8Yf8jO7c1ltprifA97yjvsOQj/BEjjlPCL61R9RlUBEj4nBbNAdxq8lAnsGMNVk7/JjZOPwVvTP4sXp0ytgfmy3h1ypfx8hTKL+L1KZ/B65NOwcuTPo7nr/8Ynp76Obww++v4v5O/ir/N+C6enHE6/nbrmXhj5vexbNp30DrjG3hlxlfROvUraJ38Zbw66Ut45fqxRtIubvq9PPGLBXP95/GyY1654fOobL6IV26INi9f/1lUNsXluWXTXrlshkeXzbDKZfPeaqx8py5emfh5LL3u83jxf76Kv/7+10CKv7zqnfCHxYzw/5c3zjvCxGn9t397rxL+oJKbFl6KgEP4uRyQTgF73sLrk74MPHUZsGI+sOqxHpt82xN+2kXAqgeQa3sA3Wsfwe61T2HDymexaOFtuHz8mXj8gRux+63ncGDVszi08ml0tD2BrrbHkF75GHLLH0N+5ePGYBXzs0b8GF5kVjyCnG/yKx9FfuXDEYZhlY3N58/IrShnCuVIeWHZVOWH6gHLH0H2tfkG4x1LnwWyh4Bc3pxIBo7u63KE35J8hD9q1IjkhG9+Xjs/t0t6qhtWyV6SUD0UAQcBaTsc5GeAVAfyy5/FK9d/2apKDq4EOrf33HTs9NO+A3S9A7v4sgP7uvfgjU2b8JWzL8EVc27H/q59yKY5b7AfyOwFsrS3A+k91tDP+O8Dsr4RP4kTyF1A2jeci8hSh1zGmHkKllPBmHx2AulyxilHygvLpirfryuph9QuILUVyPMkgy7kMqng+EkSvyV8tr9KV42pdFq8mBG+o+AffeKo6gi/iPTDoEhHjZPhdOpWBAoIuK3HTNZ27kHqlYewYs7pAPXw2U3+dgbc0qA3ht/K0tjPpLgp2g+unYlv/WamOcyE87Q2LI0s98PMHwLyXLvDiT5ZkUMpq3JEumG0yydZIjs5KVHBcO/NSoZbuFUyUk6UrFQ2wyqVzbBKZTMsqlzxH+jynfoI2kvK8J45a7ho2rao9fktICxqhfAP8xZ5Q712b4jX7g3z2rzDvXEOt5e1KuGHK1Pdg41AUZfj6pwDW5F/9UG0TvkWsJ8/w9t7d4sctPgDF5I6KZ85nnPLAnz4l1fi0bW7zdYJ+1J2bX0X8khxi4R8t9H3cm6h6B5Dd+OGFexMY/9s4Sy5nJGbi5Y2FzlXNyyllGjZXOWXLjUp1ElpWKEqGSvqqhXCL0vplT2rJvweNPVKsEZBqv7Ni4BQlRlh57qAA5tw6Nk/4PUZpwPdL1s1SqVeWynM5ViuqfcVOg++thGnXjQTt7z4tllv34kc9nVm0JHn/vd5dJmPrEj4HMX7Lwzh5HBViX+UDMfva3dUueLf1+WF85NyomQ4fl+7o8oVf5ZHu1kUYAsPN5nKt1THhJ/8wysfICX8ym1BQ/sAAUv55vOnfAeQ2oQNCydi1e1nAZmlQOZg6Rgi3GPLuYXs/QVAnA+mAuLlTRl89eJZOHv2w9gOYE+qG9kcQ4BubrFgRvj+ypwfCAAAFzBJREFUV7Wc1CNZuPkLkRgSCb0M3DCx9wFCFbNgOe79uXa5x4oZ9DKwlssnFlIP5kR5Acc+s7goo686Jvzkk7bJoCgGyU3j2otjqUsRKCDAdlLYkAz5/UB+HZbNvwyr7rsYSC01E6TuweBMK62LnC6adEq6zcUEVA/lUsinuk18apLbDgFf+vUdOO2GB7C626a1aez+OMw3a3KRErhU1CmQ3tVe4fRO1gUyqjbTKuIHhCcj3SrS9kXUwS6/6BnkZoo8Yxx1TPgjRx7v/7xhp4naWsFtkdW08HA6ccfgqcFNjADbiEP43L4sswxtD1yO1+8dD2ANkNsH7mQjRlqVkL0zRecTPmNwks6f4EXelLADwFk3PYpTLr4Vf3orbfT2BJ6xC68Qyd2R7tvG2LkxvhBHAhlOz58aNOIfvL6cMov8EpRR9OYIx4/KN6l/OL9q3UnLiYpXbXnh+OHuJeFh/yh3UxC+7YTSHaKgKPaPqrDiWOpSBAoIsM04hJ/fDXS8gmV3n4+Vf/o1gA2xhF86wpc8+Spg3sDOLDDp6XX48Ljf4ea/rDOqnEM85cQnVrEV7su3CTeIZMRqL0lLKQWJlDA/T/F2ZbXFlcRnGaHLzT/OHkpavXOwy6/+jkMplPBDgIgzqulIuEpFIIwA20yI8LuW4KU5p2PT4qmW8PMHgtE9R/luK+Mo3zUSJszKsAN54Lk1u/CZS36HC+540azQsa8B3ou1STrKkktIWWRJhBgPSSdSChO3n1y8wzIm96qDw/nHuasuICZBXHnh8JjsBiC4KQjfhT0ppm4a1540vcZrTgRsW7GTtu1A6iW8MOWr2NP6eyC7Nlg7Xw4bt5WJPZVJBy+FHd3A8g7gSxfPxBkTb8fmtF01zhcB4+fzxS8Q+lW8SNK9uYTkRfp5yb2Xk70pTtKWyzepn+TRG5m0rHLxelNu36RteMInTC70SWFz07j2pOk1XrMiwNZCEkauHdj/NF6c+iXsf/MuIM/zpuwKmlJspI3ZXwjSZlM+KXOSlgeanDF1Ab5y6Y1YvTtnPhESspfUYVlaTh/7VEH2vLfeXuHnq9Zd7+X39v456ekdlvzj1soL4wc4NNmkLSFym0VSyNw0rj1peo3XjAhIS7GEvwPYuhCvTPtvdLUtALDV/5KzHDJMSbKXaVtL5SR8+pJXf//gM/jvc/8HD7+503xPSj/Gco2UL7JcSX3qp4RfxC6Ce5TsU+x7lJkSfgRstVtlETes3jWAgLQaQ/jcu/7tO/Da9C8ite5hANwDhYRe7pKUpHdL4fThtSsHPPPmenzpF5di6vynsNX/fophLtnTLrmItDn0438l/BLMBftysh9rImHWTUH4xCIMfxifcHiUO5xO3YpAAQG2GiFhc1DIitlYOvWLwK5Xwe9eZWK1kEJs0t4K7kPdKbN1wvKDwFfGT8eZN/zRLL/kK6McuUsOrpTcqpWSR7XpJL6kj5MSv6cyLv+48J6WK+ni8o8Ll3wGTirh+1jHVY2ED1zVaEn1hwBbSYHwtwCtk7B82heA/SuQNZ9VMUa5y7YvppVBM/fJWdkB/GzqvfjGr27Eq9uzRm8vLbFcTm5YufByJZfzk3zKhSXxk/RxMkleleLE5R8XXinvJGFx+ceFJymjb+Mo4ft4xlWNhPct/JpbYyHAViKjb2S2AC9cjZUzxwJdbyFntPGMUeYiy5tfoRl051Pm1bApD1y9aClOGTcTT6/YYvaolLyjWqP4iywtSUIooy+JFR2jcoikj5OVc4kPjcs/Ljy+hMox4vKPC6+ce3+ENgvhBx1KqiAMpvjHyXA6dSsCBQTYeoSUDeE/PQGrZn0JyG5wQwoJxOa3zzzSOJBJYzeAhSv24OMXzMK0x5cZZVB3uqsqxaRkXSzD7bs4VFwSS9zVSkkfJ6vNNxw/Lv+48HB+1brj8o8Lr7a83sdvJsIvIv0wdHFVI+HhdOpWBAoIsJW4hJ9+5CI7wse2QgjbYTljYmSxrTuHv+4FPjfhVpw5fT64iz3naVPpjmCT4sStMVxO7Cuj8Cxqa0QElPD9WpUuFCcbsRHoM/UVAm7r4Qi/c9F5WHfzNwBsiSV8puXFXS9Pm7QAX//tnVifhiH8jnQ3urpI+MXLD/wk0UIJPxqbpgxpJsIv6i7h2g53pSh3OJ26FYEIBNJbsG/+z7Htju8D2FyYtA2TsO/m6huqcq685zmccs7/4C8bD2E/w/xXRVe28NWttM6Ikgve4bKK+gBz0au5EGgWwo9t6NKF4mRzNQ992p4hwG0O0L0Bu+7+kSF95DcZdYzJTUjYz5oqIJI9v6S95YWNOOWCaZj95yV2ZN/BL3PzSGd4PF/x6D4RXUtZImP7gX9TKhoUgaYgfJfEk9ajm8a1J02v8ZoPAbIq2woPGuFGN+uwYe43kHnoPCC3yVfcF/T3jMnPrPZxB0wAz23NmxU5Z836M3byTGpml04DqUNAttMeU+i/NqRFxmIsRC9SCT8WssaO0PCEL11DZNLqlPhhmTS9xms+BMiqpHAeJ9gBdKzBuhlfBp64OCB8tqZgwpZHgWez2JQCXk8BX594D75z/T1YnbJHcefNmdwk/E4l/OZrTP30xE1B+OyENKa7JQQyTPTiTphcozUhAjKMTgHZg0BHG1ZM+hzwl6uBzGaDR7CCx5xg1WXaJPX2Z/7hGfyfi+bi2S1ZbMsAXVyWwyZrNtPJmp0wM6HtlBO1ZrklkTrCb8J26T6yEr6LhmMXgg9LJ4paFYEiBIRVu4DcAeDQSrxx7WeAJZOANFU6/pJNM/bgKVYp7E3nsPCVNfjI+VNx6xt7zP72fA0cPJCx4xOf+JmE/vLCkFZZVHw5h9ySSCX8cig1kZ8SfkRlS5cKy4jo6q0IUEWT6kQ+R8Lfh/y6Z7B2zjeB5TcCaW6NzLYEZMneJPUM8PzmNP7rrOtw1qx5ZqU+Sd1cJGhG9xmegtxP6ZK+RI+UQvQilfAjoWqOACX8iHoOE724I6KrtyJALUw+YydXs3uQWfMYVk7/OvDW7T7hW6bP5uyu+Os6gK9c8Xt87/q7sT4H7OFErUGR58z6hO83O2l9YRkLuhC9SCX8WMgaO0IdE/6oUSNsxzB9o9Ih5uxoNLY7JavQcNcSd7LUGqv5EGALIa+asXhqBzqXLcLrk8YCWxYAqQ1GhcMmeDAP8BDyC29/AV/71U14fsN+cyzKwWyqtIUGzY7t1+r8WULgHQezEL3IIGXiHOJK0PC6QkAJP6K6pEOEZUR09W56BNhS7MWVNTtw4LWFePHaLwK7HzFr8s1STf/jqhuefBsfG38z7luyEdu7ga581v8wy74yJKfCih6u1C8QPsML5QWxSy1C9CKV8EsxaiofJfyI6g4Tvbgjoqu3IsAd7zu5JDMNZHfjnWdvw4uTvg7gFbNhQgpZ7AHweFs7Pjp+Ji6/f4nZ796SOjXz/LjKEn4pmQeMrTgrAr1AQAk/Ajwh+LCMiK7eigBXUWayQJaEvwvrnrgFL838AZB62Rxezpa0pgv49lWz8P0pt2NFyioa7aDb/1jLIfxS0leIFYHeIlDHhH/C6JEyIDLfH2ZDXyHaDsP/qsPvbTPR9NUgkAK63sGqh6Zh6V0XA52vGwXMhhzww8m340uXTcGb2/fgIL+ilUsH8IKEyn5FQAk/Al6+KMqZiOjqrQgECKSA1HqsvP9aLJ//ayDVZg4vmf7ESzhl3HV4fG27jZkNFmEGKdWiCPQvAk1B+C5xJ4XTTePak6bXeE2LQL4bSK/Dirsvxxu3nQdgKx5e/jY+euENuH7h0+Yoc2Iju18anMwI3+rvmxY3ffABQKDhCZ8Y9oSw3TSufQDqRIuobwTSB4Cu5Xjj1vOxZfHvkO3ehrGXTcT3ZszDhqxda8NtcqjSobLRXNwpLc+JWyV9gURlfyBQS4R/uDfWG+K1tbR4bd4Qb5wXcyXT4RO0nhC2m8a190claJ4NgYBwdec2YM8LeGHGz7Hy9edx9m9vwOfH/xZ/22+/lu1GCh1dKRzqtPvbG5In4ZPv9VIE+hWB2iL8Ns/zjvI870iPpG/tkbRfPeFzPJUG8tSddgD5DiB3EMjtB3J7gayY/Y5d/FzJ8Lg45eJLujjppk1ij8svKjxJ3kniROUf558k7yRx4sqJCk+SdxVxMtwGbTewfymwdSGenfhtTJ8xFV8443I88sbb2J7n6VXdyJlNEuwWCxnZLIEvC44r9FIE+hWBWiH84d4Y713e4oDdh3njPJoK17HHHRP8As4h6ktbQY+9iUTPCbPNQP4dILcWyLYBmeVA+g0g/RrQ/SbQvdwxdDumaynQtcw3tCcxEr+M7FwB0AR5Mk6SPN04ZfItyi8q3M2jN/ao/OP8e1OmmzaunKhwN48+sHe2AgdfAPY9DLw1DSvnfg83TBiPeQ8+h0wOyAcqG/tTgC3SDurlp4G0VZWKQH8hUCuET3XO4d7MgN9J9q47CChYkm2tIMCxe3UAqXcwb/ZFeOiWS7Bo1ll4dObP8OT0H+DZ6afh+amn4fkpP8DzU34UaRZP/iEK5vtYPDmJcdP8EIsn/bi8CfJOkqcbp9r8Jb6bR2/skp8vY59P4vemTDet5DdY5dt7eX7yd9E6/TtY8ttTsXzSp/DyjNNwx7SJyPJ7qlQK4JbIIR09WyWNXorAwCBQK4Q/1BvrHeYtCuic7nKEf4Q3zjvMa/U8r/V97zsq6D+JRvhU5XRvxbxZl+Ohm3+NB2deiIdnnIPHpp+JJ6f9wpqpZ+LJqWfiiWlnlTWPT/0lCuYMPD41iXHT/BKPTzm7vAnyTpKnG6fa/CW+m0dv7JKfL2OfT+L3pkw3reQ3WOXbe3liyhl4YuLP8deZ5+HZyT/Gg5N/ifb1baYfc+MEs7HawPRqLUURiECgVgifKp1iwp8Zp9IZfeKo5ITPX815jrKor98D5HYC2e32YIr0BsA1mXesPw+tMIZu33Cb27DJrAfKmXA87oleZLYAaddIuF9GuTxdv9j8Jb+wDD2Dm2c19mYvP4xVaiPQtRXAfmDvOqCb7esgUmaID3Tm0jovG0FD6j1QCNQK4XNoP9Rrl4naIZ4ZxXMCN/IaOfL45IRPPI2qlD+g+YUjDX9r86zQQwVjPpGhP43EETcl5wFcw0OmKxk3Lu38We8Ys50uJ5PFT+JXytMNk/giJZ8oKfFEunn1xC75iIwqV/wlnsielOmmkXxESjlRUuKJdPPqpZ2LANJUG3YBGR5gwq2SO5FHNzg5m0JepmiDdWMD1c21HEXAIlBbhD/WG+a1+cRfccKWb4FEhF9mPixnjorLg7JwpIT5CWAwEb1qOclJNjWKQXEb4LobKhU5M5srrAI229p3G8LPIh9s0O22K6UhRWBgEaglwo8cy5cPSDRpGxA+u5m/LNMsjePyOFkix5E8u3CB9KMqwe2stosn+V/ggOL0Uf5J8nTjROUT5+/m0Rt7XDlR4b0p000blX+cv5tHb+wcv1vDXMzlVzR19/zjeyBoYs6XIVHtTP0Vgf5BoI4Jf8QJIwJMuHGabJ4W8LbwtyF99rgC4VuVjahthPDLkb6bCe1+Tw4k86xkwvGDW46wSPxKebphEt+X4dsNu4P7lnRuXj2xSz4R5Zc8ZSh+ReyS3E84v5ICQx7h+EnKiI+TQwbWMP9CM8nn8+YA8uBnIYNZJ725wnWq7sJYbaCxKFuPoTZW7T2F+2i16aPim3utY8I/9sQRQT/iR+r2Q3X+rPY7leDO4ZUY6lXLGieOGY5J4pC/5NMTGa7IKHelvHNZYDBNxXujOsMxjOurOtzxc+lL08E6ChPx7+2zSz79JQUfeaSwm/iIn8FHGmtSGUrv5tUM9t7Wf7Xps/wCOs+zK60RXV5Qv5bxpX3btu0PEGR+rlK9BPm49cq5H9e4YSG7pE8i2cSQhzfczI+WV5vUsu+xJ45yCJ96VAJNQBztTBwQ4QqU+G4etWTnJ/iDaaKwENxcybj+YDcMsw2p/r+MmHsqqy+xQoowFu6zR9nDnb9C9uWDojL2/cP3FHbHvejC8cPuwU4fS9hx/SNuwBRO76viwj/y2KAJuS8KtSI2SSBuX4bxDLsDfHuYPpxf2F3PhD9q9IkBuYcJRWAu6TRhACSi+PsJxLs/ZMk9VVGmDDgGS0bhETyT4EgZInupI+krUXlV8u/tc1fKuydhwXOLxX3+JHZJpzIRAuZ96avJSl76ocFv+N0auHuQPhhZuo3EP3Tebddu2459oHD7iE0QihBOH+c2yet4hG8InxXg7zsVBbzUUQiu4GUR/CLwI0j8cH7VuiWfsCy5jyrKlff+YMkoDOQZyz2bhLkyKp84/94+d1z+1YS7z0P7QFzhMpvNbV74Znkrl7iGjK99iR0UhNOJu0J6eVkEXOGTa73hT9KrW5WO55l1++arW355Wy9miOe1DR82vHVYS4sxw4cMaaWhu16eIcl9tvh1MsxKbobXd8/X4rV6PTF9eQ/J8jLPze9KbH1TWuN/a1IdJnHPHHdPA5C+paWlbfjw4dU9V9x913T4ED6rqeeWIV5rRdPS0tpSZIa1trTQ+P49Ti/5VJI+vxxueLOWtfUNd29sIM12NeMzs46b8bn1mZutd+vzVkCgXmfJKzxSbNCQpiQ+zzuiCZ/775rwmZuxT8d2eo1gEWjGDtGMxMfa/t9NSH5/34TP3Ix9Wvk8IQKHV97nP2Eu9RWNu5w249WMdR1zjkVDNoNmfOaGrEh9KEVAEVAEFAFFQBFQBBQBRUARUASSIjDOG2KWSB2ZNEFdxxvuzfSGe23eMLOkkofdNO5z8wAfu1SvzRvSNOqso4Z65phS1nGb13zqrFZvqHOIU113Vr35vkbgKM9712LPG8ozABqX+FzULAlaH55mdoRzpKUbrxHsQ8y6bJ7ncKR5wXlexbMdGuGR/XY8xn+WI83L/V1N8dyeeam3eIuU8BukJff9Yxy2yPPeddQQ713NQ/guiDzDuHEJf4zntSwOHnfIEE5YN9+ktSVA/tJp9Osob6j5ZTNGCb/Rq7pnzzd2qDfcHNje4h3GL/SaYfRXjBR/+jbuyoaxQ7whpn7tQxvCd9zFUDSoi226OQYzfLF5Hn/ZjPGGGeJv0CrVx4pG4AhvsXe41+od7rV5h3ltZv8K+zN/rP9JvVHjtDSaSudd3mLzrHxuGn6Q0uK1DfWMTlvwGufZTiLuRpNjh3lFulzWeTMR/pH+thqNP7qnmnJIULc6wm+0ntwHzzOmxfPah3gtbTQtXgtHQc0xErLgjfO8YVR3NPK8xZihxYQ/bkgTqXRabP02hwqLCxFaOF8ztJ19eRj7cmMPZvqAAps5i5bmIXuO8lvsypVGJnu25iMtCRhVndhlMrORW/uRQ7yhM4d4hzfTrxm3PsccVvyid8PUrgh43GulaUb3LRwJUa3FpZmHea0NPGnrefypb0Z+5pmbY7TreWOoomzxDm/zvGF2N1N3ZVbjd3iqsKjP10sRUAQUgaZAoNF/vTVFJepDKgKKgCKgCCgCioAioAgoAoqAIqAIKAKKgCKgCCgCioAioAgoAoqAIqAIKAKKgCKgCCgCioAioAgoAoqAIqAIKAKKgCKgCCgCioAiMCAIHGE20OJmeLyONB+ZNePGeD4AKhQBRUARaGwEhngzh3reTLNZWvMcfNLYdapPpwgoAopAFALDPa/VP/lJvzqNAkn9FQFFQBFoAASOOtzz2mgafEfQBqgqfQRFQBFQBHqDQIu3iLuCDuN2yIV90nuTo6ZVBBQBRUARqDkE/s4b6x9xZ1U5Q7xWb7iZyK25W9UbUgQUAUVAEVAEFAFFQBFQBBQBRUARUAQUAUVAEVAEFAFFQBFQBBQBRUARUAQUAUVAEVAEFAFFQBFQBBQBRUARUAQUAUVAEVAEFAFFQBFQBBQBRUARUAQUAUVAEVAEFIEeIPD/A0GKYNc9Xh75AAAAAElFTkSuQmCC"
}
},
"cell_type": "markdown",
"metadata": {},
"source": [
"# ReLU\n",
"In this tutorial, we will first focus on implementing the ReLU layer and towards the end, for comparison purposes, we will define alternative activation functions.\n",
"\n",
"ReLU is a piece-wise linear, vector valued function that adds non-linearity to our model. The effects that this simple piece-wise function has had on the DL sphere have been astonishing. \n",
"\n",
"The ReLU's forward and backward pass can both be seen as \"gates\" that either inhibit or advance the flow of either operations. \n",
"\n",
"During the forward pass, ReLU either retains the original content of the input if its greater than zero or else, turns it to zero.\n",
"\n",
"```python\n",
"[x if x > 0 else 0 for x in input]\n",
"```\n",
"\n",
"For the inputs that were \"cut\" to zero, its gradients are turned to zero while the rest of the values become 1. Hence, and given that ReLU is an intermediate operation, ReLU either restricts values of the incoming gradients or lets them \"flow\". This process is graphed below:\n",
"\n",
"![image.png](attachment:image.png)\n",
"\n",
"Such simple conditions make ReLU a \"lightweight\" operation as it does not take much to compute its forward and backward method\n",
"\n",
"Such properties, and its surprising effectiveness to model non-linearity, have made ReLU a very popular choice of option for most DL architectures.\n",
"\n",
"Let us model this process in PyTorch"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[ 1.6169, -0.8602],\n",
" [ 0.2214, -0.4084]], device='cuda:0')"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import torch\n",
"import torch.nn as nn\n",
"torch.randn((2,2)).cuda()"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# custom ReLU function \n",
"# Remember that:\n",
"# input.shape == out.shape == incoming_gradient.shape\n",
"\n",
"class ReLU_layer(torch.autograd.Function):\n",
" \n",
" @staticmethod\n",
" def forward(self, input):\n",
" # save input for backward() pass \n",
" self.save_for_backward(input) # wraps in a tuple structure\n",
" activated_input = torch.clamp(input, min = 0)\n",
" return activated_input\n",
"\n",
" @staticmethod\n",
" def backward(self, incoming_grad):\n",
" \"\"\"\n",
" In the backward pass we receive a Tensor containing the \n",
" gradient of the loss with respect to our f(x) output, \n",
" and we now need to compute the gradient of the loss\n",
" wrt the input.\n",
" \"\"\"\n",
" # keep in mind that the gradient of ReLU is binary = {0,1}\n",
" # hence, we will either keep the element of the output_grad_wrt_loss\n",
" # or turn it to zero\n",
" input, = self.saved_tensors\n",
" output_grad = incoming_grad.clone()\n",
" output_grad[input < 0] = 0\n",
" return output_grad "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# Wrap ReLU_layer function in nn.module\n",
"class ReLU(nn.Module):\n",
" def __init__(self):\n",
" super().__init__()\n",
"\n",
" \n",
" def forward(self, input):\n",
" output = ReLU_layer.apply(input)\n",
" return output\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[0.2316]], grad_fn=)"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# test function with linear + relu layer\n",
"dummy_input= torch.ones((1,2)) # input \n",
"\n",
"# forward pass\n",
"linear = nn.Linear(2,3)\n",
"relu = ReLU()\n",
"linear2 = nn.Linear(3,1)\n",
"\n",
"output1 = linear(dummy_input)\n",
"output2 = relu(output1)\n",
"output3 = linear2(output2)\n",
"output3"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"# backward pass\n",
"output3.backward()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[0.1558, 0.1558],\n",
" [0.0000, 0.0000],\n",
" [0.0000, 0.0000]])"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check computed gradients of 1st linear layaer\n",
"list(linear.parameters())[0].grad"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# MNIST"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we have validated our operation, let's us apply ReLU to the MNIST dataset by building a standard neural network with the following linear parameters:\n",
"\n",
"```[128, 64, 10]```"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"class NeuralNet(nn.Module):\n",
" def __init__(self, num_units = 128, activation = ReLU()):\n",
" super().__init__()\n",
" \n",
" # fully-connected layers\n",
" self.fc1 = nn.Linear(784,num_units)\n",
" self.fc2 = nn.Linear(num_units , num_units//2)\n",
" self.fc3 = nn.Linear(num_units // 2, 10)\n",
" \n",
" # init activation\n",
" self.activation = activation\n",
" \n",
" def forward(self,x):\n",
" \n",
" # 1st layer\n",
" output = self.activation(self.fc1(x))\n",
" \n",
" # 2nd layer\n",
" output = self.activation(self.fc2(output))\n",
" \n",
" # 3rd layer\n",
" output = self.fc3(output)\n",
" \n",
" # output.shape = (B, 10)\n",
" return output\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"NeuralNet(\n",
" (fc1): Linear(in_features=784, out_features=128, bias=True)\n",
" (fc2): Linear(in_features=128, out_features=64, bias=True)\n",
" (fc3): Linear(in_features=64, out_features=10, bias=True)\n",
" (activation): ReLU()\n",
")"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# instantiate model and feed it to GPU\n",
"device = torch.device('cuda')\n",
"model = NeuralNet().to(device)\n",
"model"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"# define optimizer\n",
"from torch import optim\n",
"optimizer = optim.SGD(model.parameters(), lr = .01)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"# define criterion\n",
"criterion = nn.CrossEntropyLoss()"
]
},
{
"cell_type": "code",
"execution_count": 201,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"torch.Size([60000, 28, 28])"
]
},
"execution_count": 201,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# import training MNIST dataset\n",
"import torchvision\n",
"from torchvision import transforms\n",
"import numpy as np\n",
"from torch.utils.data import DataLoader\n",
"from torchvision.utils import make_grid \n",
"import matplotlib.pyplot as plt\n",
"plt.style.use('ggplot')\n",
"\n",
"root = r'C:\\Users\\erick\\PycharmProjects\\untitled\\3D_2D_GAN\\MNIST_experimentation'\n",
"train_mnist = torchvision.datasets.MNIST(root = root, \n",
" train = True, \n",
" transform = transforms.ToTensor(),\n",
" download = False, \n",
" )\n",
"\n",
"train_mnist.data.shape"
]
},
{
"cell_type": "code",
"execution_count": 202,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"torch.Size([10000, 28, 28])"
]
},
"execution_count": 202,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# import evaluation MNIST dataset\n",
"\n",
"eval_mnist = torchvision.datasets.MNIST(root = root, \n",
" train = False,\n",
" transform = transforms.ToTensor(),\n",
" download = False, \n",
" )\n",
"eval_mnist.data.shape"
]
},
{
"cell_type": "code",
"execution_count": 203,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 203,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAcEAAAHSCAYAAACU+XDdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOydebxM9f/Hn3ZFSOQia/ZslbWyVJY2pYgWikKhVeSrXaEFpaQSqciWJSplrSQkRLskSwuVFqRknd8f83udM3fu3HHvNTPn3Dvv5+PhMe7MmZn3bOfzeb3XXIFAIIBhGIZhJCG5vTbAMAzDMLzCFkHDMAwjabFF0DAMw0habBE0DMMwkhZbBA3DMIykxRZBwzAMI2mxRdBIWjZs2ECuXLlYs2ZNpu6XkpLCiBEj4mSVYRiJJJfVCRp+JVeuXFFvr1ChAlu3bs3y4x8+fJidO3dSokQJ8ubNm+H77dy5k0KFCnH88cdn+bkzSkpKCv3796d///5xfy7DSEYy/ss3jASzY8cO5/+ffPIJl112GZ988gnlypUDIE+ePBHvd+DAAfLnz3/Ux8+TJw8pKSmZtqtkyZKZvo9hGP7E3KGGb0lJSXH+FS9eHAguQLpOi1FKSgqDBw+mV69eFC9enPPPPx+AESNGULduXQoVKkSZMmXo0qULv/32m/P44e5Q/T179mwuvPBCjj/+eKpUqcL06dPT2BXqDk1JSWHo0KH07duXYsWKkZKSwqBBgzhy5IhzzD///MMNN9xAkSJFKF68OLfddht33XUXtWvXzvR78vDDD9OjRw+KFClCSkoKL774Ivv27aN3794UK1aMcuXKMW7cuFT3O9p7AfDuu+9Sq1YtChYsyOmnn857771Hrly5mDlzpnPM9u3b6dKlCyVKlKBIkSI0a9aMFStWOLfv37+f2267jbJly1KgQAHKlCnD9ddfn6nXaBiJxBZBI0cwcuRIKlSowKpVq3jxxRcByJ07N6NGjeLLL79kxowZbNy4ka5dux71sQYOHEjPnj35/PPPadeuHddddx3btm076vNXrlyZ1atXM3z4cJ544olUi+edd97JggULmDZtGitWrCBfvnyMHz8+S6911KhR1KtXj08//ZSePXvSu3dvOnToQK1atVizZg033ngjvXv3ZtOmTc59jvZebN26lfbt29OyZUvWrVvH448/zp133pnqeffu3UuLFi04fPgwCxcuZO3atZx33nmcf/75fP/998778NZbbzF16lS+++475syZQ4MGDbL0Og0jIQQMIxuwbNmyABDYsmVLmttKlSoVuOiii476GCtWrAgAgd9//z0QCAQC33zzTQAIrF69OtXfY8aMce6zf//+QP78+QOvvPJKqucbPnx4qr+vvPLKVM/VokWLQLdu3QKBQCDw559/BvLmzRt47bXXUh1Tv379wGmnnRbV5kjP1blzZ+fvgwcPBgoUKBDo2LGjc92hQ4cCxx9/fGDcuHEZfi/69esXqFq1auDw4cPOMW+88UYACMyYMSMQCAQCzz//fKBSpUqpjgkEAoGmTZsGBg4cGAgEAoFevXoFLrjggsCRI0eivi7D8AumBI0cQaNGjdJct3jxYlq3bk25cuU44YQTaNWqFcBRVV39+vWd/+fPn58SJUrw66+/Zvg+AGXLlnXus3HjRg4dOkSTJk1SHRP+d0apV6+e8/+8efNy0kknUbduXee6PHnyUKJEiVTuzqO9F19//TWNGzcmd273lNC0adNUz7t69Wp++OEHihQpQuHChZ1/q1ev5rvvvgOgR48efPLJJ1SrVo0+ffrwxhtvcPDgwSy9TsNIBLYIGjmCQoUKpfp706ZNXHLJJVSvXp3p06ezZs0aZsyYAQQTZ6IRnlSTK1euVPG9rN7naNmuGSVfvnxpHjfSdXr+jL4XR7PvyJEj1K9fn/Xr16f698033/Dss88C0LBhQ7Zu3cpjjz1G7ty56du3Lw0aNOCff/455tdtGPHAskONHMmqVas4ePAgo0aNcsofli9f7okt1apVI2/evKxcuZLKlSs713/88ccJef6MvBe1atXi7bffJhAIOIthuH0NGjRg9uzZFC9e3ElUisQJJ5xAhw4d6NChA/3796dSpUqsWLGC1q1bx/iVGcaxY0rQyJFUq1aNI0eO8NRTT7FlyxZmzZrFo48+6oktJ554It27d2fgwIG8++67fPvttwwYMIAtW7bETB1GIyPvxa233srWrVu59dZb2bBhA4sWLeLBBx8EXIV4/fXXk5KSwsUXX8zixYvZunUrH3/8MUOGDGHevHkAPProo0ydOpWvv/6azZs38/LLL5MvXz6qVKkS99dpGFnBFkEjR9KwYUOefPJJnn76aWrVqsXo0aN56qmnPLPnqaeeonXr1nTq1IkmTZqwf/9+rrnmGgoWLBj3587Ie1GxYkXmzJnDe++9R7169bj77rsZMmQIgGNj4cKF+eijj6hduzZdu3alWrVqdOzYkfXr11O+fHnnmCeeeILGjRtTr1495s+fz5w5c6hUqVLcX6dhZAXrGGMYHnHWWWdRqVIlJk+e7LUpEVm4cCFt27Zl48aNVK1a1WtzDCMuWEzQMBLAunXr+Oqrr2jcuDH//fcfEyZMYOXKlQwdOtRr0xyeffZZGjRoQEpKCl9++SW33347LVq0sAXQyNHYImgYCeKZZ55hw4YNANSsWZN58+Zx7rnnemyVy+bNm3n88cf57bffKF26NBdccAGPPfaY12YZRlwxd6hhGIaRtFhijGEYhpG02CJoGIZhJC1RY4KJqGEyDMMwjHgSLepnStAwDMNIWmwRNAzDMJIWWwQNwzCMpMUWQcMwDCNpsUXQMAzDSFpsETQMwzCSFlsEDcMwjKTFFkHDMAwjabFF0DAMw0habBE0DMMwkhYbpZRAzjzzTABuueUWAK677joAJk6cCMDo0aP59NNPvTHOMAzPefrppwG47bbb+PLLLwG45JJLANi2bZtnduVkTAkahmEYSUvUeYKJaKCdJ08eAIoWLZrmNimm448/HoDq1asD0LdvXwBGjBjB1VdfDcB///0H4AwBHTx4cBytzjz169fnvffeA6BIkSIRj9m9ezcnnXRSIs1KGOeffz4AkydPBqBFixYAfPvtt57ZlFXuu+8+IPgdy507uI9s2bIlAEuXLvXKrKTnhBNOAKBw4cIAXHzxxQCcfPLJjBw5EoD9+/d7Y9xRqFixIgBr164FoFixYk7TZ72OBQsWeGLbsVCtWjUA8uXLR/PmzQF47rnnADhy5MhR7z937lwArrrqKgAOHDiQJTuiNdBOiDu0fPnyAOTPnx+As846i3POOQcIftgAHTp0OOrj/PTTT0BwQjfA5Zdfzt9//w3AZ599BvjvJNSoUSMAZs2a5Sz0+kBkuz7Yk046iaZNmwLujyGrH/rRaN68ubPgvvHGG3F5jlAaNmwIwJo1a+L+XPGiW7duAPzvf/8DUv+IbTa1N1SqVIm7774bwPnt1K5dO81xKSkpQNDN6Ed27twJwIcffgjApZde6qU5Wea0004D3N/KlVdeCUDu3LkpU6YM4P5uMvKb0fvwwgsvAHDHHXewZ8+emNps7lDDMAwjaYmrEjz99NMBWLJkCRDZ5ZkRtHOQG+qff/4BYMqUKWzfvh2Av/76C/DevSbX7RlnnAHAa6+9BkDp0qXTHPvdd98B8MQTTwAwbdo0PvroIwDuv/9+AIYNGxYXO1u2bEnVqlWB+CvB3LlzU6lSJcD1CmTHWZUVKlQAoECBAh5bknEaN24MQNeuXR13lHbron///gBs376dZs2aATBp0iQAVq1alShTM0SNGjWAoCIA6NKlCwULFgTc79SPP/4IuJ6WmjVr0qlTJ8B1xW3YsCFxRmcAndOye/LLo48+CsBFF10U08dVEuFLL73E8uXLY/rYpgQNwzCMpCWuSlC7mj/++APImBLUznPXrl2ce+65gBsX0+7Uz4wdOxbASdiJhtSiAvlLly51Eizq1KkTHwP/n+uuu46VK1fG9TlE6dKl6dmzJ+AqY7/txKPRqlUrAG699dZU12/YsMFJX//1118Tblc0OnfuDLgp9yVKlHCU0gcffABAyZIlARg+fLhzPx1TokQJwE1I8AqdMx5//HHAfV1KgglFnpW2bdsCbg7CN99847weXfoN5UbUq1fPY0uOjUWLFgFpleBvv/3GhAkTAPc7Fh4TbNq0qZMwl0hMCRqGYRhJS1yV4J9//gnAgAEDALfoc926dU6Gp1i/fj0ArVu3BoI+csUubr/99niaGRNUCK905vCY19KlS3n77bcBd+e9Y8cOIPh+QDCued5550W8f6xRan8iGD9+vPN/7dazA8pgfuWVV4C0nozhw4f7JoaTN2/wp6ws3HHjxgFujPrDDz/kkUceAXDizoptvv766wC0adPGeTy/ZPFefvnlAPTo0SPdY77//nvAPXcoJqiYd3ZAn5Ni5qHoM5X3xC/fuUg8//zzAMyZMyfV9QcPHuSXX36Jet8iRYo4DQKUSSr0ePH4XpoSNAzDMJKWhNQJahVXsfjff//t+L5vvPFGAKeYVVlSAF999RUAvXr1SoSZWaZ+/fqOL1yF8PJ3v/vuu0AwRih/t7JcpZBUI/TZZ585mbBSlIobxqqdWt26dQEoVapUTB4vI4QqKL1P2YHrr78eSJvZq5ia2t35gS5dugCpVTe473fnzp3T1FcpvhaqAFWL++qrr8bN1sygOrNwtm7dCsDq1asZOHAg4CpAoUzS7ICy3OV1eOihh5zb9P9du3YB8OyzzybStExx6NAhIO1nkRHatm3LiSeeGPE2fS/j0ewgob1DQ3+Eu3fvTnWb3B3Tpk0DMtZNwGvUDWHAgAHOif73338HXFenTiZ79+5l3rx5AM5lNI477jgA7rrrLgCuvfbamNisgLUeP55ooVV5BMDPP/8c9+eNBSVKlOCGG24A3O+iTkJDhw71zK5IDBkyhEGDBgHu5kulANpwRSowvvfee9Ncp2Jybcy8RglV2ggvXLgQgE2bNgHBhIv0SORGL1bIZR26COZ0lHzVs2fPdM9LDzzwQNye39yhhmEYRtLi2RQJ7XSUUCJXodLRtePzI0ooGDFiBBBUVyrMVVGnArjHqrgiBcqPBfVfBdfdHC/0/pQqVYqNGzcCbgGzX1EPx1mzZqW5bfTo0YDr1vca7Y4HDRrklBGpv6RchPv27XOOV1G53J/hjQuGDBni9Gr0C3ITZkUZqY1adiR37tzZwhuWFeTVkvfi1FNPBYL9RcNRwuTBgwfjZo8pQcMwDCNp8UwJKgFGPn8lfii1+/3333fU1JgxYwD/NClWskpoQehll10G+K+BdzRWr14dk8dRMtAFF1wAuEkaoQkXinUoruZX9BqUQARu2z8VnnuNCqv79OkDBH8XUoDt27ePeJ8qVao4EzzkfREzZ84E3PZ92YnbbruNQoUKAWmLsEMbTqxYsQIgYQ0ijpUjR4745nyXGeRJ6dq1K+B69kJR6VGk16fYtZrUv/POO0Bqj0asMSVoGIZhJC2eT5ZXoatGb7z88stAcCeh3YR2ekpJV+alV6icQzvPpUuXxkwBqohd8YB4Fs0XL1486u316tVz7NE8wFNOOQVwW1Jde+21zjHaran1ndKZ8+bN64yG8itSUJpHCW5RuUolwjOavULvfWgLMGV1nnzyyQB0794dcEfR1K5d22nPpx24LtXKLrQ8yW+omFwNNBQPDfXGhP92xI4dO5z34/Dhw3G3NRmR6lZMOau5DMuWLQPgxRdfjI1hGcCUoGEYhpG0eK4Ehcb5qP5n5MiRjvrQOCGNslGdVqJrztT2rX79+oC7k37zzTdj9hzhAyeVHRUrpNYCgYAzqPKee+6JeGzdunUdJaoi2H///ReAr7/+GoAJEyY4sVupYTWTVoHrcccd59uG2dGyQTdv3gz4rzm2MkFVy1eyZEm2bNkCpB833759uxNvUfG/alrfeuutuNqbVfLly+eMY9PnI9v1Pd6xY4cT71M8V6pR5MmThyuuuAJw47rxGlad7Oh8Ec2DlZ5iB/ccK4WvmGA88c0iKL744gsAOnXqRLt27QDXRXrTTTcBbk9A9QpMFCp3kDtKhbrTp08/psdVyUVoGrjS8BUgjhVKpti2bRtnnXVW1GN/+OEHx72hRe/jjz8+6nOosFlTCrSY+BGVEkT6QYa6Rv2Ekovkwn377bcd17bCC/rc1IHkzz//dBpRaCHR335Dv68LLriA2bNnp7pt8ODBgPv7WL58ufPadV34ZPmSJUs6c+5++OEHwO1iFY8OJLEgUomE5kH6sWOMztuagqPkOCVs/ffff+neV13Dwqe0JApzhxqGYRhJS65AlDxcv0z/Dk2wANc117ZtW6ePYyJQH8OpU6cCbn+80LZgmUEKUK2tBg0a5Lh4paa0k8pOSBnr/Ro+fLijuPyCXNpys4UH8ufOnUvHjh0Tble8aN68ueOulsLQdHY1AfAaFUs//PDDgDt9BmD+/PmAqzCkhkuWLOm4zFS6JFenSj5q167tlDCJxYsXpzrmr7/+cm7TVBcvOXz4cLqubZXvyDuT3VHLSc2dBTehK1bu0GjlJqYEDcMwjKTFdzFB7XI6duzozNGSAhTaAX344YeJNS6MrCbESIVop6tu/nPnzqVDhw6xMc5HhM8W8wNqyxfetV7lHSrZySkcd9xxaZKu/BITzJMnD+A2VOjfvz8QLNlQay15X6QAdW4YPXq0kzyjWZW9e/cGgg03INjMQfFvteyS0ghtz3isnp1Y8sILLzg5EOHISyQln91p27atp89vStAwDMNIWjxXgmrorMwgTZJOSUlJc6wKXVUsn+gGs+Hpv8rOy+jk+379+gFuDFC+cLWzUvNtI/6cdNJJQNrvkFr07d27N+E2xRM/x5albKQAVYZz0003OUqtSZMmgNsEQCn0BQsWdGKIyiIPn2W3Z88eJ6aoy6uvvhpIPaLszjvvjOXLOib8WlIUimK4bdq0cTJzM9PeTKPKRo0aFXvjMoEpQcMwDCNp8UQJpqSkcM011wDQt29fwC1ajoSKsVUkH8vi9MwQ3m5KavWZZ55hwoQJgJvhpJ2rWr/Vq1fPaTmmWiXtzjUANachxVy1alVfNS5++eWXnYLdcFR4ndPwOu4SjfCBqYoRDhgwwKmdrVKlSsT7PvTQQ04NYGZaoinGqEu/MXr0aMc7plFDQp6n0aNHO3WhiaRZs2aA22SjdevWThw12kR51XNKxav9ZGhzAynJeDbMDichi6AmPKvv3+jRo6lRo0bU+6xatYrhw4cDbuGv3+Zr6cfap08fJ6FFXTlU0B+KFgK5DuI5LdkPaLOQ3oKTaJSQ1Lp1a+e7pHR6uUH91h0mVoSfSP3EL7/8ArjNFVQ6VK9ePecYpcorGU7JVlu3bs2x/UA177Ny5cqprvf6PKiSmtCmBHfffTcQfV6ompuolCW8bOGDDz7g+eefB9ykpkTgj7OTYRiGYXhAXJSgZO/YsWMBdwcevqMJRW4oSeQFCxYkVBJnBCk5zeFTmja4rlGpXiH36LRp0zKcQJPTaNq0qdO+y0s0hy/0M1JzAiVl5FSWLVsWtWejl6gdmBLNpBR+++03J8ygYvZk6vmpSQpqH+lnVJaSGdR2Ur1rb7/99qjt1eKFKUHDMAwjaYmZEmzcuDEQDGY3atQIgLJly6Z7vFSeurprUoSfZ5ppKoI60quYVSUPoeh1aVKDCnmTCb+03TOCDY71HZRHRnFCTaPwCsWRJk2alOoy2VFTkG+++QaAmjVremmOg8pUbrnlFsCdtxmN77//3il90czAcePGAW7zba8wJWgYhmEkLTFroK2xM6FNb4V2MvL9Hj58mBEjRgBuGyQj56CWY4rnjBs3Lt0WUIlEcdvp06dzzjnnADhz+NJLwc9J6HMZP3484M5/VCp+TmnIbCQGZfF269aNIUOGAG4bQmXvLlq0CAhm+CsL2AusgbZhGIZhRCBbjFIyDOPYKVKkCACvv/46AK1atQJwBtd2797d1zF5w8gq0ZSgLYKGkWRoMVQHJqW3161b11yiRo7E3KGGYRiGEQFTgoZhGEaOxpSgYRiGYUTAFkHDMAwjabFF0DAMw0habBE0DMMwkhZbBA3DMIykxRZBwzAMI2mxRdAwDMNIWmwRNAzDMJKWuEyWNwzD8Ipq1aoBMH/+fADy5MlDhQoVvDTJ8DGmBA3DMIykxZSgYRg5gtGjRwPQuXNnAIoXLw7A22+/7ZlNhv8xJWgYhmEkLdZA2+csWbIEcD+L8847zxM7atWqBcAll1wCQM+ePQFYvXo169evT3XsqFGjADhw4EACLTSSkVKlSjnzEJs0aQK4zZK//PJLAM4//3z++OMPbww0fIE10DYMwzCMCHgeE8yXLx8AZ511FgDDhg0D4Oyzz/bMJj/w1FNPAe77MnHiRE/suOmmmwAYPnw4AIULF051+6mnnspVV12V6ro1a9YA8N577yXAwpxN4cKFnRjXf//9B8CZZ54JwAknnADAtddeywcffADAzz//nO5j/fLLLwDMnTsXcD+n7IgyQEeMGEHjxo1T3TZo0CDAfX3ZQQXK0zN16lQALrroIsf78tNPP3lmVzLguTu0RIkSAPz222+A+0M944wznP8nE4899hgAt99+OwAHDx4EoEePHgC8/vrrCbVHyQWaOH7yyScf9T67du0CcBbHhQsXxsm6nM8TTzxB//79Y/qYR44cAdzPdNq0aUDwBLxly5aYPle8aNq0KQDLli1zrtP5qkuXLoC7oGQHjj/+eAA2btwIQJkyZejVqxcA48eP98yunIK5Qw3DMAwjAp67Q8NJSUlxLpNRCSq4LzfxRx99BCReAYo///wTgIceeggIup/A3bn+8MMPlC9fPtV9ihUrBkDbtm2BnK0EVYR93HHHAXD11VcD0Lt3b+eYefPmAdC9e/dMP/4VV1yR7m1y833++efpHvPtt98CUL16dedzOf300wGoXbs2AEOGDAHgs88+870SlBt08uTJQGpvld4ruXuzE//++y+QWglmxOuSnbnrrrsAyJ8/PwA1a9bk2muvTXXMhg0bADjttNPiZocpQcMwDCNp8Z0SzCllGc2bNwfg3nvvBYIKQaoqPa6++mpnd/79998DxDwelFVeeOEFwE2UqVevHgB79uxJ9z5jxoyJv2Ee0KpVKyCoPKT8ihYtCkSOPUjdZ4W2bdtSvXp1wFV1Quphx44dGXosJdJ88cUXAGkU/KWXXuqoVr/StWtXwLX9nXfe4eabbwaiJwVlF/SbadmyJTVq1PDYmtjRokUL59zWokULAC6//HIg9Tk//PdTtWpVwI1fK1kolpgSNAzDMJIW32WHirPPPpuVK1fG/fnjhXzZ2sm0aNHCie+lx5dffun4vhXfeOONN+JoZebp2LEjAPfccw8A9evXT/dYvZZvvvkm/obFEWXn1alTB4CGDRumOebvv/8G3FiV0vOnTJnilDZ4zTXXXAPAa6+9lur6/fv3A0HvxerVqxNuV0ZYsWIF4H7ftm/fDsCFF17Id99955ldsaZcuXIAbNu2zWk2UalSJSDjit8LSpcuDbgZuZUrV051e9GiRSlUqBDgritr164FgpUAR0MqP6uN0KNlh/rOHSrOPPPMbL0IylWlN79gwYLpHqsfdvny5Z309WjHe8nMmTMBN2FnwYIFzuIQzsMPPwzAlVdemRjjYshJJ53Eo48+CsANN9wAuElC+vE+9thjTleSffv2AcFEIT+hpINnnnmG6667LuIxqkVdt25dwuzKKJdddhmAUwuo39OMGTMA933PaeTKlcv57C699FIAxo4d66VJ6dKqVSvGjRsHuIt4NOTS/P333wFXCJUpU4aXX34ZgFNOOSXVfeQOjQfmDjUMwzCSFs+V4KFDhwDYvXs34CYYnHrqqZ7ZdCw88sgjgOs6k1v0s88+S3Os3AMDBw4EgmUHH3/8MeAqLr+hFOa6desCbpp9JJYvX54Qm+LB/fffz4033gi40wmU5LR3717P7Moo6jGrwvFu3bo5t6kBw2233Qb4111drFgxmjVrFvG2v/76C4jeTUUNJ0LViV8SzY5GqPtOitCv3H333ekqQLnaBw4cyKpVq4C0CV4q9bn99tvTKMCtW7cCbkJUPDAlaBiGYSQtnitBtdhS+yNNKchuaCek6QpSuH379gVg586dae7z5JNPAm7MbPv27b7rmao0bXXqr1KlCgB58x79q/Pmm2/Gz7AYoaJ/qXHtOO+44w7ef/99IBj3BHyT4BKNRo0aAa7NefLkSXOMVMaPP/4IwOHDhxNkXeY4fPiw0yc1d+7gfl0x8w8//DDN8f369QPc13frrbcCqZMpVKAtxZETyiq8ok2bNkDkEiDFxvV7yohXKFwFgtv4QPHDeGBK0DAMw0haPFeCOYE6deo4SkmZToojLV26NM3xikuExmkAhg4dGkcrs0bNmjUBN007IwpQ3HHHHYAbe/Ij9913H+AqQbWnW7hwYbZQfuF06tQJiKwAhWJMmriuco633nqLOXPmAG5BvZe0aNHCiQlKAUphhE6GUHb1OeecA7jZlOKff/5xYodqPKCYu5q8b9u2LS6vIScjVS1vCrilLIMHDwaiK8ATTzwRCJa5gNtgJPRx3nnnnRhaHBlTgoZhGEbS4lsleNJJJ3ltQrpIDSnz7qWXXkoTs9CoFxWVjxw50hlLpBigikY1K9CPdUAq1pdS0qinjNQxqoDWz2j2nOJIKvbNjioQ3NitFLwK++WhiESDBg2cywcffBCAUaNGAcFRTpC2mUU8UXs3eR/ALRSfNGkSgFMgX61aNQYMGAC4NYWKHy1atAgI/vaKFCkCuDMulYXuV3LlyhW1wNsPvPjii0Dwu6XsfjVkyMjwA7W7U0Y9wFdffQW4Ho1EDFHw7SIY7tLwE3KhqJNIIBBwFr9NmzYBqU8sEHw9ZcuWBdzFQckyKsb2M8888wzgnnw0kQDcTYFcwDrhZAc++eQTwP2cnn32WSBYhK2TaHZCbqSLL74YcHtslihRglKlSgFuNyJ970I7Q2kzpyQTJaacf/75gLvJiydya2qwNLgnXDVg0GsZMWIEFxn5dHoAACAASURBVF10EeB27VEhvdx1VatWdXrf6hgthn51g/p9AQSYNWtWqsuM0q5dOwAeeOCBVNcfOnTIEQKJnCBk7lDDMAwjafG8d6i48847gaDrAoLTCULVhh/o3Lkz4PZeVBnErl27HDeAinj1OtQxHdz3U2+5LrXradmypTM9Ijuh1yVXmnZ4ei2tWrXyxY5brbfWrVvn9GWUi1rJO/fffz8QLIhX6rdfi8mPFTU+UCmByisi8b///Q9w3aPxRK730ESx8IQsJVzoMwVXrSoZLdL0ebl5/Vo0H9o7VJx77rlA5CS77IhKcsKXnj59+jiKP9bYZHnDMAzDiIBvYoLhjYfz5cvnFLn6QUWAO0tPtmqnOmHChDTHanetnU2kglIpKBVlZ0cVCG7KfbiPX+25vCrGVuxVpQCKj915552OmldTbMUCpQQLFy7spHDnVDTxYvr06QAsXrw4VZp6KGqSkAjkAcqVK1eaKfEqh6hYsaJzjGJ/UkqRps/rGCnB7ER2PS+EM2zYMCBt4wPhldI1JWgYhmEkLb5RgoqviVy5clGgQAGPrImMdqVKQ1fbqUgoJV0z9QBnCrnG74hoTYCzA6EpzqFIIXv1+j799FPAzVZVrCl8nh64hf1i8eLFaT6nnIp+e2vXrk1XCW7cuDGRJgHBOE56sRypiEAg4DRzl4dG5TtbtmwBoFmzZk4Kv+EN+fPn5/TTTwdSf3bgNjr3ai6kKUHDMAwjafFNdqjQ8MQaNWo4tT19+vRJuB1ZRUW4ihf27t0bCPr1FavwG2pMIOWmGNGUKVOOet/SpUs72ZPh9YGKI23evDlmtmYGFcKrNdpxxx2X5hjtPqtWrQq48ecOHTo4StLvlC5d2mncrtFdav+WEdRibcGCBc4IJiGVqMzL0EzLeBEpq1Pt0+rVqwe4TRsKFy7sHKPzlYrlu3fvDiSm9VasiJQdqu9mdowNqqValy5deO655wD3c9L5RfkTGqYQD7LVZPmFCxcCULZsWadgNzuhBVvdENRpI/zk4ieefvppwC1i1WL9888/O1321QRAxdM6ZsCAAWkWP5WHbN++Pc6WR0eT4ZWgI3dMq1atnGOU/KITpRIo9Hr9TEpKCgDz58935ldmJplHBef6nUX6jmqDk4jFT6h85d9//3VOoh999BEQ/WQWXiyfnRa/aKgZgJpRZAfU9UcT5zt27OjcpnI4JaMlogFDNMwdahiGYSQtvlOCIhAIODvC7IDKOXr06AG4O1aVSPg5+WXMmDGA26tR7qj333/fmewsN7XcUtrpgfta5Yp76KGHAP/03xwxYoTXJsQFpftLBYL7GWp69759+5zb5A6+++67AVcBhn6WclVJVXkxAWTt2rVAMJFMNrZs2TLisa+++qoz8WLdunVA9i4q//XXX4Hg761WrVoeW5N1NBswVAHKnasWjH7BlKBhGIaRtPhWCRYpUoT27dsDbkmCn1GzZSlCpeGrlZifWblyZapL2T5mzBinKFmXkVCruNByECP+LFmyBHA77oNbFiJVFFoaoKQtxUYjIQV4+eWXA96qqnnz5jFv3jzPnt8L5P0KVfCtW7cGskdMsEaNGgBp8jk2btzozA30G6YEDcMwjKTFd0pQu9r9+/c7cajswCuvvAK4o17efPNND63JGmoqrCYFoennalelgn+xe/du2rRpkyALjVAWL14MwLRp05zxXiKa2gtHZRCjRo1yxuKsWrUqRlYaWWH9+vVOJnbo79DvqO2ghg2IZ5991jftL8MxJWgYhmEkLb4rlp82bRoQnIytwbp+3UEYhh8oUKCAE8NTrZ/anIUOp1b2rtBgWWWSKo5oeE/FihWZOnUqEMyABZzmIX5E+QBqYqD4n7Ljn376aed75gXR6kt9twgahmEY2YvHH38ccJtNSLio0N/LBRBsnqBhGIZhRMSUoGEYhnFMqLfsggULgGDvXSDNPEivMCVoGIZhGBEwJWgYhmHkaEwJGoZhGEYEbBE0DMMwkhZbBA3DMIykxRZBwzAMI2mxRdAwDMNIWmwRNAzDMJIWWwQNwzCMpMUWQcMwDCNpsUXQMAzDSFpsETQMwzCSFlsEjZhRuXJlKleuzPTp05k+fToHDhzgwIED1KhRw2vTDMMwImKLoGEYhpG05PXaACP7c9ZZZwEwf/58AHbu3AnAmDFjAPj111+9MczIENWqVQOCk8uvvfZaAHbs2OGlSTGhZcuWACxZsoTcuXOnum7p0qUeWWX4Dc8Wwa5duwLQtm1bAOrVqwdA9erVnWM+/vhjANq1awfA7t27E2liwilUqBAffPABAGXKlAHg7LPPBmDr1q0eWRWdiy++mJkzZwLBkyjAvffeC8C///7rmV05hRNOOIHChQsD7vc/1u+rpn83b96cHj16APDoo48CcOjQoZg+VyLo1q0bALfeeisAR44ccW578sknAZg4cSLgbtSy4+vMiQwaNAiAoUOHAvDEE08A8L///S9uz2nuUMMwDCNpSeg8wRIlSgAwfvx4R93t2rULgJUrV6Y6tkWLFhQqVAiADRs2AFCrVq2Y2pNopO5KliyZ6vq//voLgHPPPZeXX34ZgG+//RaARo0aAfD3338nyswMUbVqVQDWr1/PsmXLAFdRhO68jWNjyJAhzi54wIABADz11FMxfY5mzZoB8P777zvXKZlp06ZNMX2ueCIFKC9T8+bNndvkDg3/blapUgWAbdu2JcDCrFGhQgUA7rzzTgD69OkDQN68eZk2bRoA11xzjTfGxZATTjjBOe+VKlUKgIMHDwLQt29fAF566aUsPbbNEzQMwzCMCCQ0JqjEiYoVKzq+3uHDhwPw559/pjq2Ro0afPLJJ4AbuH/ggQcAePjhhxNib1aoU6cO4MYjtIsD93WUL18+1X0ee+wxIKh0pb5//vlnAPLnzx9fgzNJwYIFARg3bhwAX3zxBZ06dQJyhgIsXrw4AJ07d+aee+4BXAUv7r//fgCGDRuWUNsefPBBADZv3gzA3LlzY/K42nVnJ4oVKwZA/fr1He+JPCwFChRIdeyGDRscJajfYHbghhtuAFzl/9133wFw0003AVCuXDnnO6Fzorxm2Ym8eYPLUO/evdN8F5VUF+4pjCWmBA3DMIykJSExwdatWwOuEnz99de5+uqrj3o/7W7uu+8+wPXbV6pUKSZ2xYPbbrsNiBy32b9/PwAzZswA4PzzzwegdOnSzjF6z6+77joAXnvttfgZmwWk3G+55RYgGBv86aefvDQpJjRt2hRwswcbNWoUNY4AMGnSJLp37x5Xu0JjgmLPnj0AtGnTBoA1a9Zk6bGVdbpw4ULAjT+D63VJtNo9Gu3btwegZ8+eQPA9SC/eJ2688UbnGHkwhN9igvL83HXXXc5noO+kfnvKozjjjDNYvXo14HqX5EHKTigmrcz4UJQ78s477xzTc1hM0DAMwzAikJCYYL58+QA300wZTUdD9WdSgopHFSlSBHB3xH7goYceAtwMPvHqq68CwQLyESNGOP+HYDwDYMGCBUAwe1a36bX7BcVZunTpAri7tuyuApWx/OKLLwJQs2ZNIPgZzZkzB3Bjb1LnV155JQBNmjRxdu4HDhyIi31btmxJc52+/4MHDwaCn4kyjDODMnxDFaBf0fdOv6dQpPLSI9SjdbRjvUaehSFDhnDHHXcAMHr06IjHtmnTht9++w3IngqwYsWKADz99NNpbluyZAmQOmM5XiRkEXzvvfcAOP3004GMF/vKfSgUNFU6sIqz/YDKOY477jjAda+ocDy0A4dcMEq8UED/33//dU5s//33XwKszjh333034LrQ9LqyO1rgtPjJNahyj1C0iWvVqhUAp5xyinO/zz77LC72vfLKK05ijpIghBpNdOjQgfHjx2f6sZV0oESbypUrO7e9/vrrWbI31mjxGzVqFOC6PPX7+PXXXznhhBMAN6lJ6Ji///7b2Tj4NXlLtj/yyCNAcBP8/PPPRzxWyXZqbJBdeeutt4DUpW8SNnL97tu3L+52+HtbZBiGYRhxJCFKMKuqRjvUr7/+GnB3DHLj+Am5Ly+88ELAVRYqf+jTpw9FixYF3ED3xRdfDLjlIUOHDuW5555LnNGZQEkYy5cvB+DTTz/10pyYEb7TzEzZwZ49e/j9999jbVIqDh8+zDPPPAPg9PWUJ0H07duXN954A4A//vgjw48tz0qoAvQT7du3d9yf4Qpu1apVQFCVq0g+POlFnpbZs2c7x/gNlQfodyX3Zu/evdNt5aZkucqVKzNy5MgEWBkfTjvtNCB10orOf4sWLUqYHaYEDcMwjKTF11Mk1DJHl35m/fr1gFvUKSWoMojWrVs7ZRPhxfKKA6YXAPcSpS83adIEgLp166Z7rDr0K7nnq6++iq9xMUBJE7pUgknBggU59dRTAbcd15lnngnAL7/8AgRj04lISFDjbKmFcCVYp04dypUrB6SvBPPnz+8UWQsl+PgNvd+KA4LrTZICVClSKIrLSj2GxtTkqVFphV+SgTp27Ai4RfznnXcekLZ5CLi5EPot7t2710m2y07IE6bfnJTgkiVLnJhoIjElaBiGYSQtvlaCSstXaYTwWzNpcDNZw8s2VAg/a9asNDsfNYNVKr4fURzqm2++Adw4rejWrZsTlzjxxBMB973o378/4I6r8SPhcYl+/foBwWJlKT9x1VVXAd6Vr8jLcP3116e5TcX+8khoxqMuCxcu7JQaRUNtt7JSchEr1JZOGdfgFu1rxFMoH330EQDvvvsuEHl+5d69e4G0Gedeo89SjaNXrFiR5piUlBTAbcChMo/Ro0dnq1mdOg+o4YF+c59//jkQPNd4kRVvStAwDMNIWnytBFVMGTpoF9z2a6Go6FnDebUznjFjhrPLSgQZab+kFkDy5//4449xtelYUBNfxSO0k1aR+IMPPujEmlT0rxo7NTb+/vvvI35mfkAxNNWaNWjQAAjGK7RTVV2rspS9QrWAir2Gth589tlnU12Gkzt37gzVyCmWrd16VkfXZAU1j9BnkTt3bvLkyXPU+2Vm3JO8MX4pmletp1qkheY/qLZx1qxZgHuOU330448/njA7j5VGjRo53ykpW6FGFcolSDS+WwTlAj3llFOcqerh6Euwdu1azjjjDMAtNlWCgFymVapUSUh6tH6sSiSJ1Hd13rx5gNsPz++cdtppTgp3eLq23vf58+encQ9Onz4dgHPOOQcITov26yIod6iSDU455RTAfQ0QTLEH7xdBIfez3LMZ4ciRI0fthRqK3o9ELIK1a9cG3JO93OqxLGxXkwdt3rwumlfCnAgvzWnbti1jx44F3EQ6LfYq/fBTx6yjccMNN6TqkQxuiCVW01Cyij+2Q4ZhGIbhAQlRgmoldvLJJwPBVPPGjRsDbkpw+LHRpshr967ic4AJEyYArtqSmytS78V4oH6oV1xxBRC5a3lmduJ+INRtEe5SVvlDtGQLpah/8cUXcbAutnz88ceAOw8yFL9NUsgKmzZtcr5/+o2o9EKuOK9QM4Dw0qFYolIEv5RGqCheiSDyPMgVXLJkSSf0IK+SEkv0uWUH1P/0xhtvTHP+03Sh7du3J9yuUEwJGoZhGElLXJSg1JwmKygGVqNGjXTvI/+2UpkPHTrkxKOEEgMUE/S6dZcaG3fv3p0OHToArtqTbSrg7d69u6OEsyPh0yIyUqaSHSdMKD6V0UQSv6Ei6x9++AFw44dTp05Nc6wa2nutBNNDTduPlRo1avDEE0+kum7r1q2Ad43q5R25+eabgaBSAvd8MXXqVCfJSfMiFSPMDig3Q02+c+fOzeHDhwG3vZ3XClCYEjQMwzCSlrgoQRV/y+cr37ZiEVu2bHEygnSbdmZSDxs2bHBaCalAW4XMUoteowyvhx9+2LlOMTLt4pQW3L17d99kF2aUXLlyRcxyzSgtWrQA/NncID3UUPvIkSPOzMR4zQrMKt9//z0AEydOBIKNlJVppwbExxqHVcN0ZWp6UTyfmWbgkZDnae7cuZx00kmAG4tTjNDrYnN9hrrU723UqFFOg3N5mfw2Xi0Saun35ptvAqnL21TsP3DgwMQbFgVTgoZhGEbSEhclqF2kMjO1k1m3bl36hvx//E8FoKeccoqza+vUqRPgHwWoYmVltQFceumlACxevBhwMytD4y1Su9mFQCCQpYzWfPnyAW68Y9KkSTG1Kx6oSFyxmZ07dzrZrX773BQ/VyODeFC2bFnArauLJ+kVsL/88suOQsoIqgXUfS677DLnNnmTLrnkEiBttrNfkPfklltuYejQoQCsXr3aS5MyhZRfeIMTcNWh34jLIqgT565du4Dorhn1BZ0xYwbgztjbv3+/UwzsdQJMOHLzqkRj6dKlvP3224C7AOjHpmNy5coV99lzsebrr79mx44dgDvhO71p1+C+dh2jjj+Rel36BX0+KubXyX/gwIGe9QhNFPp9aipGeCcPcMtD1BUovRl3x8KQIUMAt0wgtPTp/fffB9xzisIoWsTuvvtuZxHVgq0yCHX6GTZsmNPwwK+Ln5gyZQoQTBoJT+bJDqhpSTgffPCBb8NB5g41DMMwkpa4KMGNGzcCbi9A9YZTcPqzzz5z3BMDBgwAXPmseWF9+vSJ6j71Eu1KQy+lgpQI8/TTTwNuQsH48eN9OzU+PXbs2OEogfAJ1pMnTwbg1FNPdWYMqp2TAvhyi/tZAWu3LQWopgfZeWJ3RgkPV8yePdtJxhBS8ZrfFw8luGTJklR2qH1a0aJFad68OeC2OVNbwlDkRtUxS5cuBdImnfgZ9axVf9DbbrvNN+GfzJDePMDnn3/e08kk0TAlaBiGYSQtuQJRMh+OJT0e3F2B5spF6tyuYKka9fq10XIoKlpVIejMmTOdHXT4TlXK8K233kqghbGnb9++AAwfPhxwG52DWwKhRCHFePxWWhBKq1atADfGJBWh2KfXTX29oGHDhs73VIpEqBxIKiueSJX36tXLKTmK1rhACXTLli0D3PhldmgvppwIzRFUSUrt2rX5559/PLMrs6iVpcqKFBscPHgwEFwLvGwbGe25TQkahmEYSUtcG2hrQrQucwoqTBYdO3Z0VLPaVqnZrUomsjt6PX6eEp9RKlasmGpUErixr2RUgGL16tVOQwp5b9TgQq27EsHPP/8MBGdVppc7sGHDBiDomVDzgOXLlyfMxljRvXt3wJ2DqjyK7KQCwR29pQbgQs1Q/Dw8wJSgYRiGkbTENSaYU5HfvmfPnkBQ6WqnrBinWgQZ/kGN3YcPH+4U8isTsXPnzp7ZZSQvqp2TYmrYsCEQnyzcRLBt2zYAjj/+eMDNEPc60z+aErVF0Ega+vTpA8Do0aNZuXIl4CZ86CRkGIlEjQrUfzi7lVFlFywxxjAMwzAiYErQyPGojZZaZ7300kvOTLPsOPPQMIzMYUrQMAzDMCJgStAwDMPI0ZgSNAzDMIwI2CJoGIZhJC22CBqGYRhJiy2ChmEYRtJii6BhGIaRtNgiaBiGYSQttggahmEYSYstgoZhGEbSYougYRiGkbTYImgYhmEkLb5dBKdMmcLmzZvZvHkzjRs3pnHjxl6bZBiGYeQwfLsIGoZhGEa88W0D7eXLl9O0aVMANm3aBMBpp50GwMGDBz2zK6N06NABgIIFCwLQoEED7rjjDgDef/99IDjSB+Cbb74B4NNPP020mYZhGDmebDVZvly5ckBw4cuXL1+q2woVKgTAvn37Em5XNI477jhq1KgBwCOPPALAeeedB0CBAgWOev8tW7YA8N577zFw4EAA9uzZA8Dhw4djbq8R/MwA2rZtC8CDDz4IQP369YHIP5obb7wRgL/++su5Thu0L7/8Mn7G/j/t27cH4NZbbwXg3HPPBYK/0/R+xnPmzAHg3XffZeHChQCcdNJJAGzcuBGAvXv3xs9ow/h/atWqBeCIgTJlygBw8cUXAzB37lxWrFiR6j4vvvgiALt27Tqm57YpEoZhGIYRAd8pwTp16gDw2WefOddpN9uxY0cAjhw5knC7Qqlbty4AzZo1A4JqQruZWDF48GAA3njjDQC++OKLmD5+PClfvjwAK1euBILvTyKUUjjVq1cHcNR1KMcffzwAV1555TE9x1dffQW4302pq1giBThx4kTA9Yhklu+++w5wX/vvv/8OwIEDB5xj+vXrB5BmR24Yx8qTTz4JwO23357h+8jrcu+99wIwduzYLD23KUHDMAzDiEBerw0QefMGTYm0a586dSrgvQIUUoDPPPNMusf88MMPQPSYXunSpQE3eSYUxai0W/dKCVarVg2A//77D3BfVzSef/55wFUYf//9d5ysi86iRYsAKFu2bNyeQ8laq1evBuDVV18F4LbbbovZc5QsWRLIugIUVatWTfV3pPdl+vTpAFxxxRWA+7qMzKPPS7/vSy65BHDjzplF5xvlEGQ30vO6rFu3DoCff/45zW3KrbjqqquArCvBaJgSNAzDMJIW3yhB+YuvueYajy3JOIpVtm/fnl9++QWA8ePHAzB8+HAgeuad1MJTTz0VTzOzzOWXX+4oGynTaLaqpKVVq1YAPPbYYwBs27Ytnmami1SN4lyh7N69G3CzeXv16gW4yjezFC5cGICWLVsCrkJUzPBYeO655475MTKKMvY++ugjABYvXgxAly5dgNSZsX4lT548AFSuXDnNbfJk7N+/Py7Pfc0113D22WcDOJfKczhWLrroIsD1RP32228xeVyvUIxaryv09ZQqVQqANWvWAFCvXj0AunXrxrx58wDYuXNnTOwwJWgYhmEkLZ4rwZ49ewJuDVZ2YMqUKQBMmjQJCGYuKWa2devWDD9OtHjLP//8A8Rut5MVrr32WkftZkStXnbZZYAb3501a1b8jMsAUuWKUYZy6NAhwFUGs2fPBmDAgAFA0DPx+uuvA1ChQgUAihcvftTnlJJSHC8WSIVLjYmPP/4YgJdfftm5rkGDBoD7uxJVqlRxFFJG0Gd4wQUXAHDKKacAWVeCl156KQBvvvlmlu4fTpEiRQBo06YNN9xwAwD58+cHcOqLpZhCeeCBBwAYMmRITOwIZ/LkyU7ugi7DPSHLli1zftdqlBGJ2rVrA67HqEqVKkDwdwn+9SBlFJ3jQhXgiSeeCLjfX/2exEsvveScVzp16hQTOzxbBLt37w7A6NGjAfcLrK4pZ5xxhjeGZYDwE4EK24+GfpzDhg0DoqfnK0FoxowZWTExJpx99tnOQp8R9IX1stNQKFoQJk+efNRjdaK65ZZbnOs6d+4MwGuvvQZAo0aN0r2/Gjjox/vBBx9k3uB00PdFl9FYtmwZkPYE2bt3b6c0Qtxzzz0AFCtW7KiPqxKQrCZoKUkps5x88skAtG7dGnDLXlq0aAGkXuh07pg7dy7gJqQ0bNjQOUavOV6L4MaNGx1Xq55Dm6nMUK5cOZo3bx7xNq/CC7FGjVH0Or/88kveffddwP3thnPkyBFnYx4rzB1qGIZhJC1xUYJKElAwU7s37cg6derkyF6hAsp33nkHcIOmOYVzzz2XO++8EyBqYf3mzZsBt0jeC6ToChYsGLXINBz1S/33338Bt6zCKzKiAJXGLsWhZBpwXW7hpQWhKPHp5ptvBrz93KIRySWsllR6D0aNGsWFF14IuL9h0aZNG8B1zWaWrLY6lKJWW0J5GfS9zJUrl+PK7t27N+C615Ry37BhQ+d+ajgQL3SuyyqVKlUCguox3BsmhZtVVe031L5PvZSjoTDTE0884YSjYoUpQcMwDCNpiYsSlK9XUxLC0853797NuHHjABgxYgTgFoAqAJ9TUOxz7NixR01MePjhhx1/t0ouvGD79u1AMPYplaBG4NFSy9WUev369QB8//33zn3jlZKeVfS6lDyTlSD77t276dGjB+Am1mQn1MRACQrvvfdeul6KUaNGJcyuUKT4pCTVTnHo0KFAMI70448/Am4iipKbHnroIedx5GFRO0K/oDityoqkziMlVt1///2Ad80nEoliz0rM+uOPP4D4TBAyJWgYhmEkLXFRgkr7VUwwPKayZ8+eDLXfEsfaLiqRqLm2ygW0e4ukAhUzUxx04sSJvmqJNHPmTO666y7A3ZkOGjQIcHfWkVCRsgqthw0b5rs4hjIijyXNul+/ftlSAYaj2GekuKHKMDISt4kH+v5pZJUuI6EyDKk9ZYdu3brVUVo7duyIm61ZQWpVrzMaY8aMAVI34FAx+SuvvAJkrkQrkVSvXt3xFKXHkSNHnKYViq0f6wiljGBK0DAMw0ha4lonqDhQZsboyN/9yy+/kJKSArg7PO12/Ea+fPmcQlbV9elvcfjw4TT+bBXujhw5MgFWZp5HH33UeR2qaZRyUu3Tf//95yg/ZeBJWagw2m8qENwYk3bS6dUlRWPYsGFOjEpNgLMTyoiN9LtSfE23edWia/78+Uc9RjFA/Z6kABUrbNu2ra88LKGEnyeiEan4X9m8NWvWBNy2k14P41azBQ1+HjduXJqKAJ3rFyxYAATjvJ9//nkCrQziu3mCYvny5U4vSk0ijja1wUvuu+++dAPuKl6ePn16RHeT35EbV4XjKoNQKje4XR7UWUWuYBW+qjuLH9FJ6Mwzz0xzm/qmqslBJLSBufvuu+NgXWzRZ9mtWzfA7dLUuHFj5xhN/njiiSeArJdEJIp27doxc+ZMwP2c5Kq/9dZbAfd76Ec0bV3lApHQZqVr166A2yGoQoUKPP7444DbbERzIM8991xPfnc6L6hpRKSpQEKNKRJxXrR5goZhGIYRgWyhBKVCvGwhFooSdaQiZs2alUoZgZtIoN2b3wLysUSvUcpJcxJ//fVXz2yKBSoLefrpp4G0/TjBdTvJ7bN8+fIEWZd5pPyUhh+JDz/8EHBfj19RKcfUqVOdwn4l26nFWnZvuKEpFCoHue6664DUMz1VUK85e/q7Tp06fP311wmzVZ4UJYplpNStbdu2gJtAF09MCRqGvS5VbwAAIABJREFUYRhGBDyfIhENrd5+m5ulmEqkGKXaPF1++eVAchS2RprblhNQYpfaNCkpqGjRos4xirP5pWl4OEoa6du3LyVKlIh4jBrA16tXz3dNDcKRAlRTiTx58jhNGdTaza9JMJmhSZMmzjxOxZsjlZWpabhaBEoJLlq0iLJlyybCVOrXr+98HuFTH+Qpefvtt51cAb9hStAwDMNIWjxRglWrVk2TLquUdbXHeeqpp5zMJxVq61Kthh555BEnMyxWc8qioTTkSJmAS5YsAdyZbxlRgMqmVIxxyJAhznXhqEB20KBBTgaYHyhQoADt2rUD3FKYjI6Wyi5oBx6pZZNiMZ988klCbUqPs846C3Ab0msmnVoZhqJCeGWCZqaBRaLRd0yKJ7T5hG7LCQpQ9O/f3yku37Bhw1GP1/dP31GVlyWCyZMnp1GAagCitpj169c3JWgYhmEYfiOuSlC1K6eeeiqA02z4pptuSjPgU/VJUjyhU7xVmK1pzHrcokWLOo2m460E69ev79gRKfNJ7ZzUIi48jvnQQw+laZ2mCdHpqb9Q1IjbTyoQgjWCp59+OoCj3LM6Nsdv6LNUTDBSTE3fV31/vUbfTQ3BDUWNslWX9eijjwKJaU2VVaRgNaBWmaD6ffXt25dvv/3WG+PiSIkSJZzf1dSpUwF3qLIyeMFtYiGVFa2mNdbI61WtWjWn6YTa06klpJTpTTfdlDC7MktcFsFSpUoBbud5lThEQqUDSoL56quvALdb/NGI93wwsWnTJt577z3AXdRD0Yes5Ilwl2D58uWPKXnCr9M1QqcOzJo1y0NLgpx33nlOSUM4mvkXWrqxe/duwD15aHN21113OSncGdmk+IWHH3443ds0uWXevHmA9/Mej0aJEiWckhN9/+Wyvf766wFYunSpN8bFmfXr1zsdYlTyoZKJ33//3TlOyS/hG2yVw8QDzUzUdy137txOAkx6YaArrrjC+b82MH5JeDR3qGEYhpG0xEUJqn9degpw3rx5Trsp7fTiMScqluzdu9dJNlCKfKQJBEr4CU/8ySxqV/Xnn38C7mxGvxE6/Xrt2rUeWhKkSJEiTiuqcELdSELuZbk6w2dfRuOVV15x2uJ5jZSsvDCRUPtBXapT/7///pvm2AkTJgBu8ow8NYksoTjttNMcBagWYDq3+C0sEGsGDhzo9EBVkwZ5KcqXL5/u/aT2J02aFDfbdP4LnXmoRBih35EUu3qJgvu986JPaCRMCRqGYRhJS1zaplWsWBGAuXPnAu6kciWWqAFsduWSSy4B3B11ZltMqbv91VdfDRCxvZF86+rm7zc0K3LdunWOmo/U5T7RtG/fPm6xSSWQ6DNp1qxZhtLXE0H79u0BN4lCyWOxQin49957rxMbjzf16tVzvlsLFy4EUseWcjr6DJUMpJh2pGbb+nx0jo3WJuxYUYmYEqvA/W1Ioas0J1S1Kn6r72oiS6msbZphGIZhRMC3DbSzA9qhtWvXzlG/SuUWalYcGo/SqJdVq1YlwMr40LJlSyDYJEDt4+68804PLQpy+umn07dvX8AtQcmqKlLmsTLf+vfvD/i7pEAxGKkGxdTCi5mPhgrPFYeS96Jy5cqpYkHxRkpC73lOa8SQHZHKkzqPFocW+/bt46qrrgKCLdQSjSlBwzAMw4iAKUEjS6jVVrdu3Zy6Sb81C5dyUxF/NDQUef369c51qqfzekr3sdCgQQPAjeGCW9sZqY3V/fffD7g1n2paoSbV7dq1822mspFY6tSpAwQVoQb/hqOp8SNGjEhYLDkS0ZSgLYJGltAieNZZZ3HOOed4bI1hGEb6mDvUMAzDMCJgStAwDMPI0ZgSNAzDMIwI2CJoGIZhJC22CBqGYRhJiy2ChmEYRtJii6BhGIaRtNgiaBiGYSQttggahmEYSYstgoZhGEbSYougYRiGkbTYImgYhmEkLb5dBIcMGUIgECAQCLBt2za2bdtG4cKFnRl+hmEYhnGs+HYRNAzDMIx4k9drA0SePHkAd55Zv379mD9/PuBOYK9cuTIAn3/+uQcWGkb2pm3btgD873//A2DRokUArFmzBnAnhRtGMuGbRfCqq64C4IEHHgDgscce45577vHSJMPIUVxyySUANG/eHIAWLVoAsHTpUgCWL1/OP//8441xMeDCCy8E4O233wZg+/btAPTq1ctZ6Hfu3OmNcQa5cwcdjzrHP/jggwAMGjQICJ7zPbHLk2c1DMMwDD8QiAIQ93+NGjUKNGrUKLBz587Azp07A2vWrAmsWbMmUKBAgYQ8v/07tn8NGzZ0vi+HDx+O+O++++7z3M5k/3fRRRcF9uzZE9izZ4/zuRw5ciRw5MgR5+8yZcp4buex/LvwwgsDF154YeDgwYNp/s2ePTswe/Zsz21M1n+VKlUKTJw4MTBx4sQ054e33nor8NZbb8X1+aNhStAwDMNIWjyPCd58880AnHjiiQCMHDkSgP3793tmk5FxHnzwQY4cOQLgXEY6pmTJkgDMmjULgA8//DAxBsaZcuXKAXDHHXcA0LRp01SXK1eu5KyzzvLGOODUU08FYPLkyRQqVCjiMZMnTwbgt99+S5hdsaJYsWI899xzgBvrjIRigoY3jBw5kssuuyzVdQcPHgTgnXfe8cIkB1OChmEYRtLimRJUZlrXrl0BmD59OuBdhlC8qVOnjqMInn/++VS35cqVC4AFCxbw9NNPA/Duu+8m1sAMUrFiRSBoK0BKSkqG7nfLLbcAsHHjRiB7KsFOnToB0LhxYyCo9qT40uOnn36Ku13RuP322wEoUqRImtuWLFkCwCOPPALAoUOHEmdYjKhbty6NGjUCoHTp0kBkj8TgwYMB+OqrrwCYO3du3G0rWLAggOMFEWeffbbzm69atepRH0eK6bXXXgPcrNfDhw/HzNZ4Ua1aNSD4OYWjrNDw82GiMSVoGIZhJC2eKcHatWsDbpH8zz//7JUpceHKK68EoGPHjkCwRks7w2DirYv+bt26tbNjuvfeewF4+eWXE2JvRsmXLx/gNi7IqZQrV86J8+mzVPwvEitXrgRg1KhRALz++utxtjA6w4cPB6Bz587pHtO6detEmRM3TjzxxHRjnZEYO3Ys4KrFt956Ky52AbzyyiuA60E4VoYOHQrAm2++CcCNN97IH3/8EZPHjjWlSpUC3HhfpUqV0hzjlzitZ4vgxRdfDMCOHTsAmDBhglemxJRrr70WgCeffBKAEiVKADBp0iTHhZg/f34AHn/8cSC1u0RfnlNOOSUxBmcSuZWi0bNnTwAaNGgAwE033RRXm+LByJEjncVPzJgxA4CZM2c613m92KVH/fr1Aff7B+6Jf/To0Z7YFAsULpB7PRQVY0dDv68KFSrE1rAIFChQAIBPP/0UyHqyn4SC3L6XXnopAE2aNGHevHnHamZcKFq0KBB58VuxYgXghka8xtyhhmEYRtLiiRI88cQTOeOMMwB46aWXANiwYYMXpsScbt26Ae4OXH0an3nmGWcnKCVYs2ZNAPr06QNAoUKFnN3633//nTCboxHeiioSctOoHZJQMkbu3LmdXboSAvyKSnSaNGniKL+nnnoKcF2efua0004DoFatWmluk3uuX79+iTQppih0kF45DriKfdmyZUCwdOKKK65Idczll18OwLRp0wD4/fffY26rvEJKbNFlZsmbN3iaViJTs2bNAGjZsqVvlWD4+w2uAlSI6Ndff02oTelhStAwDMNIWjxRgv379+fkk08GvE8hjzUnnHBCqr+Vdn7w4EGngbF2syqZCA3sKw1aCRZ+IdrOO1wBiki79vCkIL+g5AWppJUrV3LXXXcB8OOPP3pmV2bp1asXELl0Zf369Yk255gpW7Ys4BbCd+nSJdXtu3btclTc2rVrAejbty8A+/btA3DONaHo8RS7iocS/Pfff2PyODo/SAEKv8ajwfVyic2bNzsx9vQUYLFixZw8CeWKKDFw27Zt8TLVlKBhGIaRvHiiBDt06OD8X+m+OYXwXY5U0l133cXevXuB9Atk58yZ4+xi/cLRskFVuBuKYp6hmYl+R+UQYuXKlU4hvDJ1s0NMsEmTJl6bEDPq1avneEYU4wz3SEyaNCndGKd+ZzaSLXEo+/a8885Ldf3YsWP55ZdfIt5HWa9jxoxxckWEcixOP/10AP76669YmguYEjQMwzCSmIQqweLFiwNBH/dHH30EkO7uIBrly5cH4IcffoidcTFCmZ7aETVs2BCI3LYqnAULFsQsjhArVNCqnVg4ikGFcuuttwLZYweuuEpow2sINm8YMWIEAB9//DHgxq/9GCOsXr06AGXKlAEiZ+GqOF7fRR2j13ngwIG425kZmjdvTo0aNbJ8f3kpxo4dm26tqjwd4bFGP9G+fXuvTcgwPXr0ANzvoc5nq1atSnOshibcf//9AGlUILgNKlRzGQ8SugjqC122bFkn8SNa/zv1ApS7o1ixYoDbBWP//v2MGTMGgIceeiguNmcWdb5R4F1dYnLlyuXYHd4rb+LEiYD/usOAW+ge7oaSG1vJCKFEWhj9RNOmTZ1SCC1+SoJRev2PP/7oJGXo+6dj/IjchTr5REpAateuXapLLYIq41m7dq2T2h4Pt1NGUSJIzZo1ndIaXX799dcAtGnTBnATKKIRWqIT/jgDBgyIoeXxQYuF0Ob/u+++c65T8xGdbyKhCS7xRL2FxZYtWwC3XCUUNT646KKLnOu0Cf3ss88Ad8pQPDF3qGEYhpG0eNY27WjF8R06dHB269qRq9h3ypQpQFBxKMnGL0pQyLWky5o1a6YJ4C9duhQIloxA1otp48U777yTphWVdp+hyU3hSGGE3lct46TcvSR0+oPUndrcJSNSXs2bN3dUvJpYxKN04GjIRdmzZ880HogXXngByJgClCoOfRwpQBWyZ+RxEolKCwoWLOiEj8LVqhK1QssGChcuDLi/vb1797Ju3ToAZs+eHV+jcZPg5GUQoWo1HIW1xC+//MI111wDBBsBgClBwzAMw4grninB9DryKyYxZcoUtm7dCsC5554L4CTTKPW5V69ezlRsv6Ldzh133OHYLcWnHZ7fOsFr1mP16tXTTI2PVuyudkjawYbu4r2eGRbKk08+mSr2lx7hDbT9jJJAdu3aBbjx81AUZ1GcRq3DQhk2bBgArVq1AnB25jt37oyxxekTKUFCqNm8pplk1nuilPvPP/88a8YdA1LcLVq0cObs6X2Wt0t5ExlJBAkEAk4OwsKFCwF3asPmzZv5/vvvY2h9dPR5hM9O1JzYUM4880zA/ZyVHNmxY0cn3nnjjTfGzdZwTAkahmEYSYtnSlAqT0g9SNlNmTLFmYq9Z88ewN0dqYAWImcn+gn5tnv06OGoKMUvN2/e7JFV0dFMw3CffXpoh6u2cGpFJXr27BnXuW1Z4WhlDv369XO8FYoX+rE0QigFfdOmTYA7xioUZeipIfhjjz0GwLhx4wCoU6eOc6yKnaUWX3zxxXiYHRE1hQ5vEwZwzjnnAJHbnSkzUXGp0GYN/9femQdMNfb//5U2SiKK9pSSrVKyfBN5SJHSIkUqEj15SNpIJU9SHkvaJFvoTou0iAjZIktUHlvpQYmkooVClu7fH/N7X2dm7rmne5mZc+aez+ufqTlnZq5zn+W63p9V0cz6u6TyuaFxqKt9tFrKDV1v0VYzWVwWLlyYqCEmDUWXP/300+7voNJoem5MmTIFCEWGKj5CCfR6Rv7+++9JG6MpQcMwDCNjSakSVEmxXbt2ubwWRQ2qdZBU3rx585wCVCsR2b21mhs+fLiLsAwaDRs2BODRRx8FQvZ7RbZpBZ6OxCpzpy7miriLJtUReCqGXZACw4rg7d+/v1uJB62YeSykkJQ0Hws1olV7LCWIx2tvddVVVwGpVYKx8kxVtKFXr15A7KhVRVbGivTV+PW8SSWKEN+1axcQUoJSQzqOrKwsIDJvWkpJuXPKm46Vc+c3KlYuS8QxxxwDeE0CWrZs6Z7xivEQUnmjR4920doqTKEcQvm6k0FKJ0E5ards2eJqZKrrshLF1ZW8XLly7mbV5CGzwLBhwwAYO3Zsikaed3ShamzqCr1q1aqY3bDTjeiH4ahRowqUUJ9M5IxXPVBNYrEmRaXhRKevzJ07N626SMhs9OmnnwJeEYBY1KlTB/BMqLE6hKjSh0ylqUTXix6k4JnHbr75ZsDrb9m2bVt3/SklJ/p4Jk6c6MvkJ7SY1/V42GGHucT1eGY+Bcxt3boV8J4tQUST1LJlywDv3KmO8IwZM3ItjKKKMeEolSpeikWiMHOoYRiGkbEUy44T856sLuAXX3yxC4CRqVOrHVGyZElXf3PPnj2AF1AyadIkIHi1DsFTtjKHKhz90ksvdduCTr9+/YBQAEX06lpKXSbpPn365Nhn1qxZgH/1GBVmLcuBlJxqgJ5++uk5gg1kclJpu3RQf7GQWnjmmWeA+F0lclNOAAsWLAC8IIxUolqmCoyD+GPNbR+ldXTp0iWQJsS8oqR3PVMUWBO01CrwgumkvJUKklfUdUfmYvVjLSzxUrtMCRqGYRgZiy8pEs8++6yb8ZWYHV1u56effnL+Jyk/hRgHjeLFi3PZZZcBXnqBHMVy0qeLCoTYHeH174ceeihi31j7jBw5MtlDjIvCrOWDkX9M6m/u3LkuWV6KKR16BeYFBSGp6ETfvn0BuOmmm1xprXisWLEC8Lqh+EG8ZPl4yGKkBHIlxsfqYJBOyKeYLMtcIpEVplWrVoCn5tu2bet80UL3oPyIs2fPdsFD8RorJBpTgoZhGEbG4otPsKjRuHFjl7iqJN7hw4cDnopNJ7p37w7A5MmTnXqI54tR1K+ieBXZFbSC4JlMuXLlXES22hDt3bvXbYNQxKWiQf30N8nnNXv2bNeSLJ5PUFHISqGKtlakO7JoyKoUZJ9gbhx99NHu/NSuXRvwComkwl9rPkHDMAzDiIEpwUKgYrdvvfWWK/umyDaVplJx2HSkT58+rqRRPCWo4rmGkUhq1qzpcj6bNm0K5IxS3rx5s8sr9KPtUyooCkoQvNxOWZdUmi/esyVRxFOCNgkWAJk8Va3mpJNOcv/u2bMnkL4h9tHIrKtgF3UrCK/q4WcismEUdbTAVkUcpfqkMngk3TFzqGEYhmHEwJRgPlAi6NKlSwGoVKkSANOnT3dh+UFM4DcMw8hkTAkahmEYRgxMCeaREiVKuI4Q6muoot8K/TUMwzCChylBwzAMw4iBKUHDMAyjSGNK0DAMwzBiYJOgYRiGkbHYJGgYhmFkLDYJGoZhGBmLTYKGYRhGxmKToGEYhpGx2CRoGIZhZCw2CRqGYRgZi02ChmEYRsZik6BhGIaRsdgkaBiGYWQsNgkGiEGDBrF79252797NgAEDGDBggN9DMgzDKNLYJGgYhmFkLCX8HkA0ixYtAqBt27Zcd911ADz44IN+DikhnHDCCZQoEfnn/uWXXwD4+uuv3XsHHXQQAFdccQUA48aNS9EIDcPwmwMOCOmSKVOm0KRJEwC2bt0KwNq1awGYN28eAGvWrGHHjh0+jLJoYUrQMAzDyFgCowRLlSoFQJkyZQDYt28fN954IwBPPfUUAD///LM/gysAhx9+OAATJkwAoGPHjq4jvXpbbd++HYDTTjstx+c3bNiQglEa0dSsWROA6dOn07x5c8A7X+qvuWbNGgBatGjBtm3bUj7GrKwsAL755hsAFixYkK/Pb9y4EcCXsRvx2bdvHwDjx49n9OjRALRp0wbAPT+6desGhFTjGWecAcBXX32V6qEmhdtvvz3i/yNHjtzvZ8455xwA3njjjQL9pilBwzAMI2MJTGf5Qw45BIClS5cCOHs4QJUqVQDYsmVLysZTUM4880zAW9G0aNHCbdPfM/pPfvLJJwMhxXvUUUcBcOyxxwIE1uavY6lcuTKdO3cG4JJLLgGgTp06AJx++umApzyCzHHHHQfgVt8XX3xxjvMV/f9XXnmFCy64INVD5YMPPgCgcePGbjz7G2uxYsXcv7/99lsAfvzxRwB69OgBeAo3HShbtiwA8+fP5/zzzwc8FSX0vJg2bZp775FHHgE8Fe03hx12GOCpGVmQjj/+eG644QYA9u7dC0C7du0Az0r02muvsWfPHrd/uqJnZV5UXzzizVfWWd4wDMMwYhAYn6BWcX///bd774477gCCq4bCufDCCwF4+umnATjwwAMjtj/44INupXLWWWcBcN555wEwZMgQAOrXr+9W50E75mrVqgEhhQTQtWtXAJo1a5Zj319//TXiNcgoCve+++4DPIWxevVqpxrmz58PwCmnnALA4sWLAf+UU9OmTQHo06cPELIayH+ZF3SMUpLyMer4goiipnXvzJ07FwjFEOiZ8cMPPwBQvHhxACpVqgTALbfc4r5HVorLLrssBaOOTcmSJZ3q6du3L+Apwljo2BcuXAhA9+7dAdi1axf16tUDvONKJ9+grGT5VYDy/b355psJGUdgJsEaNWoAcOqpp+Z4748//vBlTPlBD0ZN5h999BEArVu3BkJBCDrpv//+OwCTJk0CvIuhePHiLFmyJFVD3i8NGjRg6NChAHTo0AHwAphkkpk8ebJL/fjnP/8JwMsvvwx45rYgo8lPQSI6hgULFlCxYkUgFNQEXkCCJsUxY8akdKzRPPTQQwX6nFwN77//PgCff/55wsaUaLT4uvvuuwG49NJLI7bv2LGDm266CYAZM2YAnmtFZrZ+/fpF7O8306ZNc4tIpT3cdtttADzzzDMAlCtXzrkXNInr3qtQoQIQWmgrfSydJj/x+uuvR/z/jTfeyDGxacIraNBLXjBzqGEYhpGxBEYJpjtSgHLArly5EoDdu3e7fbSakXKKVn2LFy+mV69eyR5qrsg5r0CCI4880pl1ZRrUalvH9+uvv9KoUSPAU1GffPJJ6gZdQIYNGwbg1J7UXXi6gYJljjjiCABncpQCSwelG4sPP/wQ8K5ZmdmCxsEHH+zGpmtMaUU6X5MnT+bTTz+N+Fzt2rUB6NSpk3tP99qtt96a3EHHoWHDhkDIlSBz7uWXXx5z3y1btnDXXXcBMHXqVACqVq0KwMSJE4GQhUwKOajIytWiRYsc6Q+FTW1IFKYEDcMwjIwlMEpQdv105d577wVg4MCBAE7RKXT5+uuvd4ExTzzxRMRnN2/eDHh+Ab+Q4pE/c8+ePc5HoXJ20WHosfjtt9+SNMLE0b59eyB+6PSyZcsAnF9U+8qPk25I/eoc3nnnnYCnqoLCwQcfDIQUkBSgVLfScd56660cn5PPTAF1Uk6AU1U7d+5M0qj3j1IdsrOz3XHJfxmvEIhiIhREI4tE37593bMjaMjfF54iFq0E/VaAwpSgYRiGkbEERgmWLFkyx3tKBE0Hhg8fDoTSHMArdaSE8SVLljglqIRYreKUdiAF5hfyU+g1r0RHScqHEWSUIP7ZZ58BXui9xr5gwQKuvfZaAJeMrSRzlfFLJ0aPHu0U7apVqwDPtxQ06tatC3hpOAC9e/cGYivAE088EfD81kohCRqyINx///0MHjwYgC+//BKAESNGAPD4448DkRHx48ePB6BVq1aAF6Wc3/s0FYT7AMP597//nfrB5BFTgoZhGEbGEpiyafKTKREUvBVeOpVzKl++POBFoclHCN7fU61RVHLLbwVYWFavXg140W/KY/LT/5JX5POUjzC83Fh06TEdZ1CVRizkBxw1apQ7DpXmC2p0qxLZs7KyXAK8yhFGF5a/+uqrXam7cuXKAV7SfK1atYDQeVMkYni0tl8UL17cRR4rz09FJ5Rje+WVVzofoFrKyaoU5JzA6OlEfj/9/f0inu/fd3NodO08sX379rSoOBLNrl27AHjvvfeAyIWEeoXpJk33yS8ahd6rT2I6oIRkJcSrKED9+vVdZRXVcZW5LR1QoNmoUaOAkClXC8ygTn5CCxLwihhEPx9UV7NatWquoooWzTItahJ85513AjH5ib///tuldSjI5ZprrgHgnnvuAeD77793+6uyVJAnv9yQWfT11193ifCpSIDPD2YONQzDMDIW35WgemSpPJJYvnx5YCq95weVpFJJNMnwbdu2OVVYvXp1wFu5Rif7phPVqlVzph2lUYTXf00XlCYQni6g44hnSgkqUq8ae3Z2tjtPepWbQcowKG4H3TudOnXipJNOAnJX4R9++KHrrqBAOh2fVGQ6BGrp+BT8IssEeGlWr732WuoHlk8UABNdD7RFixY5aoVq3+jUiVRjStAwDMPIWHxXgkUF9QR89tlnAS/4QGkQbdu2dSWSBgwYAIQS6MErN5aOXHzxxS5JWSv4okDHjh2dcpdSevjhh/0cUr6YOXMm4HXyaN68eY5OGbF6DyoZ3c8E+q+//hqAsWPHOj9mtKVI6R0jRoxwClDqSUX45XMPisKNheIEpFblk+7du7frcKJj/euvvyL2DSJSddGvkFMd6v9++whNCRqGYRgZi+8pElJMmzZtinj/ueeei4gSCzrqYp1bIvyqVatcsVspQfkCFfkWhDYv+WXmzJmuvY0KF6dDJ/nckD9pxYoVlClTBvBSWRS+nq6okEPNmjUj/q8Ixfbt26dlGohSI6JTcpRkHp6mFDSU+K6ejiplN2LECHdc7777LuAdZ8+ePYHgRFfmlWifYHRCfTLnG+ssbxiGYRgxMJ9gApg2bZorPq0Vh/x9KlEVC63sorvQpxOVK1fm448/BtJbAQqppDJlyrhzl+4KUKhsl15feuklAL744gvA80elGy1btgS8e08J9UpEDyJ6XkT70cP9zsq3vfrqqwF49dVXAa9AeIcOHQKf8xmOlGt0aTW/FW1gJ8F4VdWDgpz2PXv2dA7uCRMmAF6ATDgKSJDsl+M+qJXgMwmZQZ988kkg9ED1u3N8qgjvqJGOxxzdfUUVZFSXM4ioHu2hhx4KeInwcquE8/777wPQv39/wJsou3XBx+vwAAAgAElEQVTr5p436YAmvbPPPtvfgURh5lDDMAwjY/FdCVaqVCnm++reHWQUMJGdne2qvr/yyisx9z3hhBNcKoTMhjKZpiPqg3baaafx9ttv+zyawqPwenWa37ZtW0SX+aKIumTodc2aNWl3zEcddZRLqNc9qO7zQUadMoSed+HdI6J57rnnIvZRnWK/uf322526UwJ8tImzRYsWrsdgNCqn5hemBA3DMIyMxXcleOWVV/o9hAIjPxJ41en//PNPwLP5q+RRuK9FYejp7Ats27YtEArqSeckeSk/9atTcEU6+sbyio5ZRZt1zPPmzfNtTAXlqquucv+Wj13l+9KJDz74YL/7VKlSBYASJXx/bANeInx4Enx02kM8pBatbJphGIZh+EQwlhRpyueffw7ASSed5KK8XnjhhVz3l91fPd7SGbUgAq/jejqiElQ1atQAvOjedIq6yy865saNGwNeibXoKMsgozZJPXv25LfffgNCBbfTBRVnl39PpeJiUa9ePQCGDBkCeKXW/LYkRZdByytBKZwtTAkahmEYGUvglODKlSsjXoPMuHHjAPjuu+8YNGhQxDY1mFXprXbt2uXoil0U+Pnnn13LmnRC0aDKkVOR5aLsC1RpLh2zimQr3zWdUBTrMccc4wpohzeiDTrKR1VzYFmQnnjiCbePEupvvvlmwCuGrjxIv5s8x2qbpPcULarIz6Covlj4Xju0S5cugGeS0cQyePDgpP+2UXA0aZQuXdrVDE0XypYty4oVKwAvcEkPFnUlT3fUWb5Zs2ZAqErM0KFDAc/Uqwk/HaqOHH300QDMmjUL8Lq2FC9e3AXXzZgxw5exFQYF0On5V6FChRz7aMLX+Vu3bl2KRld0sNqhhmEYhhED382hc+bMiXg1go0q8qsDQToUNYimfv36rvO6zDV+m5YSzbJlywDPlLZt27a07oZxyimnRLyKiRMnpqUCFDoXMn0aqceUoGEYhpGx+K4EjfRC/RLFM88849NICs7KlSsDk3CcLBRYpn6dRQ0FnhUVH67hH6YEDcMwjIzF9+hQwzAMw0gmFh1qGIZhGDGwSdAwDMPIWGwSNAzDMDIWmwQNwzCMjMUmQcMwDCNjsUnQMAzDyFhsEjQMwzAyFpsEDcMwjIzFJkHDMAwjY7FJ0DAMw8hYbBI0DMMwMpaiXUrfKDQHH3wwACeeeCIAl1xyCQA///wzEOrwXblyZQCmTp0KwPTp0wHYt29fSsdqGIaRX0wJGoZhGBmLL10kGjduzOOPPw5AgwYNAJg3bx4AV199NQC7du1Kym+nirPPPhuAJk2aADB8+HDKly8fc98DDgitRapVq8amTZtSM8A41KlTB4A77riD1q1bA3DooYcC8PvvvwPw119/AVC2bFn27t0LwIEHHghAy5YtAXj11VdTN+gEoeuxWbNmPPDAA3H3LVasGDt37gTg//7v/wBYs2ZNcgdoGEa+sS4ShmEYhhGDlCrBRo0aAbB06VIqVKgA4FREyZIlAWjXrh0AL7zwQkJ/O9n06tULgJEjRwI41VeuXDkg/kpEf+fNmzfz999/A/Dkk08CMGPGDADWrVuXhFHHZsmSJUDIp/fll18C8NNPPwHw7rvvArB27VogdJy//fYbAM8991zEtg4dOqRszAWlbt26gDfWf/7znwDUqlUr7jmL5vPPPwc8S8aKFSsSOUzDSHuqV68OwDXXXBPxfs+ePQGoUaOGe69fv34APPzwwwDccMMNQOj5+tlnnwFw1llnAfDHH3/s97dNCRqGYRhGDFKqBDWbT5gwgR07dgAh/yDAkCFDAOjWrRsADRs25Jtvvkno7+eHKlWqAPDjjz8C8Vcb5557rvNpKppS6G+YFyUYa59BgwYBMH78+LwOvdBoRbZx48Z8fW716tUAHHvssQAcddRRgBdJGiQOOeQQAF5++WUAmjZtGrG9WLFi+VKC4uabbwbg3nvvLeQI94+um+uvv36/+2RnZ3PEEUcAIf80wE033eS2ASxatMjXey5RHH/88QDceuutAHTt2pVzzz0XgDfffNO3cYVz8sknA3DRRRcBnvLROcrOznbnacyYMfv9Plmchg4dCoR823feeSfgWW/8QHECXbp0cefjmGOOKdR3/vnnnwAcfvjhAOzZs2e/nzElaBiGYRgx8C1P8NFHHwVwK88tW7YA3gr98ssvZ+zYsf4MDmjTpg3g+bl++OGHXPf94osv+M9//gPA7bffDsCvv/4KeMezc+dONmzYAHh+vrZt2wLQokWLHN/51VdfAbB48eJCHEXByK8CPO200wAvl3D+/PkA/PLLL4kdWCFRhOv48ePdCvywww7L8+flv5YVQ0o3VVSsWBGACy64AIBhw4YBXjRvLGJZGfTv+++/P+L/Q4YMISsrC/CuUfk6g8qRRx5J2bJlAfj6668Bzx9bpkwZAAYMGLBfBdi5c2fmzp2bxJF6ZGVl0aVLFwCKFy8esS08t3bUqFEALF++HIitYnVNv/jiiwCceuqpbpv290MJVq1aFfAsLfXr13fb9FxQhoCei8cdd1wOf2E0y5cvdwo5LwowL1iyfC488sgjed73u+++44knngBg69atAC6gRGaPZcuWOXNwPFOZLgg9pP/3v//la9yppmzZsu7YFTzzr3/9C4hvgvCDTp06AdC9e/cCfX79+vUAjBs3DvCc9qlC16SujURTuXJl55bo2rUr4KUuaXGXqAdPYSlRIvTomjVrllsE1KxZE8BNinoAP/bYY/v9Pk2YyeCUU04BPNfGJZdc4hYnCiLTolvul7p163LmmWcC8Pbbb+f63ffddx8QOflB6NgnTJiQqEPIM7lNfuvWrWPixImAF/QoAVS6dGkAtz0WCr4bO3Ysy5YtS+iYzRxqGIZhZCyBVYLxTDxBZPPmzYC36lSyvBLhly5dut9k+YceesiZn4KuAOXAf/rpp925UvCBVrNBQWXdFIodDwUhfP31165QgErF3XXXXQCUKlUqGcPcL8cdd1yu20aPHg0U7G8v85LOKXiqSsEzS5cuBeCll17K9/cnEv3tZa5t0aKFM/fJrSALhMyJfpvlFSzVsWNHAK677jqeeeYZwDOx7969O+Izq1evdoFmudGtWzcuv/zyiPe2b98OhFJ+8pI6kGgU/CIFKDdXmzZtnLk6GqU69O7dO9fvVcCkzL6JxJSgYRiGkbGkVAnKrguenfzSSy8FvNW2OP/8851ySocSagrX1WrvnnvuASKT5aN9ZB999BEA06ZNA0JKUMnyQUNBID169AA8n1GjRo3cilMJ5w0bNgRg5syZgLc69QsFF2lc4AUgaGxTpkwB4O677wZC1+qzzz4LwIgRIwAvWElpMK1bt3bnOxXo7zt48GAg0repcSjYKj+pDpMnTwZCfqigln076KCDAC8YSM+LVatWuWO/7LLLIj6Tn79BMq9RnRPxzDPPOP95flBg1OzZswE4/fTTnTJWIJ6Kdqi8od/omb9t27Yc26pVqwbggoRiocBEWSKSgSlBwzAMI2NJqRJUi51mzZo5/8w555wDwDvvvAPgwpQ7d+7sVnt5ie7ym0mTJgGeso3F888/D+BSP5SKIH9iEJFvUxGg8hWFo9Vo//79I95XuHO4AkslSt2oXbt2jm1a+R955JG5fl77RKsEhaXn5uNNFvIXX3vttYAXWditWzeXIC51qFSHwia/awXvl59X15asJVINKiPYr18/F2kpH+C3334L5PSzxUOKIxn897//BTzrV+fOnV1ZxHDrWDStWrUCvCL81113HeD5uMO57bbbgOQeR154+umnAe881apVCwg9+/XMV/EMnVPFEoQjP6+u52RGJZsSNAzDMDKWlCpB+Y769OnjlJOQGpIy7Ny5M1dccQXgKUiVywkieVEFWlVrpRpkBSi0AlOkms6FIr3kNwtHEWtKxh4xYgR33HFH0scajUpIyS8bjnyABeG8884DYq9gU4HuAxVomD17touWlHqQEswPffr0yfGeovFWrlxZoLEWlvPPPx/I6TdSQnzVqlVp37494PlqtU1trsJREX8VTlcOXjLvRRU3kG/6gQceYODAgYDXkiwWKl+o0mOxULSuVJXfKEFf0ZzKCWzXrp2zJimSOdZ9qXOmKO1URPb60k8wHnKAz5s3z4Wo61UJmEFEJorom1Xhv+qOAd6J1c0blHqGiUammTPPPDNflVkSxcKFC4GcgQljxoxxN1k8c1RuqFNErGT5VNYOTRQypT733HPO3P3WW28B3jXqV3CaKqqowlRe0ly+//57INI0KLO+6lYq2V7XRioqMykFZcKECS6FQJNyNMuWLXMTgjruKHlebNiwwZlKY034fqLnuApUjBs3zgUP5sbOnTud+TPRXYSsdqhhGIZhxCBwyfJamT/22GP84x//ALyu3UFWgjKnRHd70P9vu+02Z25SusFrr70GeCHvDz/8cL6c+UFHAU3RK9hkIzUWrr7BK2n33nvvFUgBCh1Xs2bNuPLKKyO2+WE9KSxSzApigJDJDvxPT1LKkPo8aqwKMqlatWqOc6D0lwsvvDDH98n8qSAjlTdMBQou6tatm1OFsQLNIBT0JFdEs2bNAHKUC5swYULgFKDQ/SUz/a5du9y5i0a1eHv27OlLH1lTgoZhGEbGEjglKObNm+dK8Fx88cUArj9WkANkcmPUqFEuyEBlqlQ0V4n1c+bMKVJKUJQsWdKtfFMRai/7f7QfYNGiRUDi/D/79u3L8RtBKxoeDxU+UMfv7Oxsp5BU1isoqLyYArH0OnLkSLePgp3i9VfMjVatWqW0JJzug7zcD9EpPira8NRTTyV+YAlGxcyji6GEo7Q4PzrmgClBwzAMI4MJrBIEr1SOfGZSE+mQWhCLDz74APCSyBV2ri72N9xwA7fccos/g0sCOl9//vln4IpqZzIqv6W2SSVLlgRCUZXRkbRBRUUOhg4d6u6rgihAIXUVJBSZGx1nIH9tQUqvpQq1plK6lNLdwItkVXqIWin5hSlBwzAMI2MJtBKMpkGDBkD6KkGhiLdo/5/KKxUVlIuXSho0aOAiCRONVqxqjhzexkb+Xq3Sg4yKUKs1k/yY06dPL3SZtVSh/NtSpUq5xr+FIZVRonlFTXhVpk+l4mbNmuXbmPKKClUorgO8yF5tU+F2NeL1i7SaBFXlwe+eZoVFHRh0PKJdu3a+XuBKIFbfvKFDh+YrCEmJzaoGJHNoKqvFfPzxx0ydOhXwarQmCk1+qtQSjqohFSb1ItkoyV/HoT6W69evB9Ij0EJVYbTQ2bRpk6tEUpQoV66cMymqQ4QC6JTqE0TUcUV1TsXgwYNdwQPVDg0KZg41DMMwMpa0UoJ+I+euKsBrJV2tWjU2bdoUsa/KNKmsUXZ2tluBRyfIqi5nssx4eUVJ7QMGDABC5jJ1FpcpJhYK4VYZMRU5+OSTTwBy1IlNN/r16wd4XQrCUQm8oJtBK1as6LpPyPyppHL14QtqL8FwdAyqMdy3b99AK6OC0r59e9d9Zfny5UBw6oPGQuZpPS9US1ml6x555BF3rygwS6UmVebOL0wJGoZhGBmLL0qwZcuWPPjgg4Bn35ZdP1ZR4iDQpEkTN+bolfRFF13kes6pAKxWRuGd5UV0QvXEiRMB/0tUffjhhwB89913QKjslIory5mtVAepxurVq9O5c2fA89cowEeFz4OSHqGE3ebNm7ugFnX0iKZBgwZO3SmFRYEx6tq9e/duVzD9jTfeSNq4E8EFF1zgrBJCxQOk2IPMiSeeCHg9K5UsL6tMUUHnaNy4ce69RAT+JJOyZcu6a0nPO1nG1E0ivB+gUj70bPErSV6YEjQMwzAyFl+U4HnnnefUkMJ/VfJI4bJZWVkxOyj7xbp165g9ezYAV111VcS2vPqDZBOXjV9Rk++//36ihlkolLKhcm5PPfWUW4HPnDkz5meKFSvmzuWrr74KeEnYW7ZsSep4c0Ph7j/88APgFSyXn6J8+fJO9eZG+HEJnT+lGATdDxiOxhyO+j0q+jDIyF+ua0qtqqTKiwrVqlUDQu2TVCpOxQCCysCBA50C1LWkSORwBajnZocOHQDPF6ioUb/wpZ9g7dq1nRnq2GOPBUKOU4DTTjsNCFUTUMi9TI3Ka/LLvCaz2GeffQbAIYccAkSaN5UDqBOs4JlRo0a5RrSvv/56agZcSI477jiXzqHgEF3kq1atAkINXWXO0CShv4Hf5NZPMC/EmgRvvPFGwMtvSgdUl7Fjx47uPZnX1FFhw4YNgGceDRK659auXQt47hLl0BU19Bzs1auXu6+iu6EEjbvuustV9dKY1SFHdOzYkVNPPRXwKhSpJvTzzz+f9DFaP0HDMAzDiEFgOstLMQ0cOBAIJV0q0EKrvnBnsZ8o/UHKoF27dkyYMAHw+pSp55zhH+re/e677wKecs8LxYoVIysrC8Al38sspZqH6YBUeazbXPe3+nUGxSwfjlwG9erVA7xE/6KYFgFegYchQ4bQqlUrwAu6Cup1F64E84KelfpMKixHpgQNwzAMIwaBUYKGYSSeeEpQyddKOwhagMzhhx/uLCvyTaeLP72gyE82f/58955q8KokWdAoX768i9uIRu9PmTLF9ahUTIVSzFKBKUHDMAzDiIGVTTOMDEMrcpUgCyqPPPIIo0ePBoq+AhSKgp0+fbpLkQh6fMGuXbtcJH86YkrQMAzDyFjMJ2gYhmEUacwnaBiGYRgxsEnQMAzDyFhsEjQMwzAyFpsEDcMwjIzFJkHDMAwjY7FJ0DAMw8hYbBI0DMMwMhabBA3DMIyMxSZBwzAMI2OxSdAwDMPIWGwSNAzDMDIWmwQNI4+ULl2a0qVLs3DhQhYuXMi+ffvYsGEDGzZs8HtoRhGmevXqVK9enXXr1rFu3Tqys7PJzs5m69atnHTSSZx00kl+DzGtsUnQMAzDyFh86yJx+umnA3DyyScDMHDgQABq164NwDnnnMObb76ZtN83Eo86X//73/8G4NNPPwVg7NixzJo1y7dxJYomTZoAsGLFCvfexo0bATj66KN9GZNo2rQpAB988EGhvueSSy4BcPfetm3bCjcwo8DUrFkTgJdeegmAunXr5tjnq6++AqBevXqpG1ghqVixIhDqFwnQtm1bt23kyJEAro9korAuEoZhGIYRg5QqwQoVKgAwefJkzj33XACOOOKImPvu2LGDTZs2xdw2fPhwAJYvX8727dsTOkbDW4XNnTuX//73v/vdv3Tp0gD8+OOPAJQpUyZi+7JlyzjnnHMSPMrUUaJECQBmzpwJQKdOndy2L774AoDjjz8+9QP7/7Rq1YoZM2YAsHXrVgAaNWoEwJ9//pnn7+natStPPPEEAM8++ywAXbp0SeBI/UOWpltvvRWApUuXAsE9vpYtW7JgwQIADjrooIht7777LgBnnHEG69evB+CYY45J7QALQN++fYHQ9QrQpk2bXPft168fAA8++GBCftuUoGEYhmHEoEQqf+wf//gHkLfVV4UKFTjssMNiblu4cCEACxYs4IorrgDg999/T9Aok8Nxxx3HjTfeCECHDh0Azza+Zs0aIORTmz9/fsrHpr9zu3btAG/F1rt3b6pXrw7kTVFEK8CigvwU4QoQ4Msvv3Tn0g9at24NwPTp052VRa8FseLs3buXffv2ASElAvDkk08C0LNnz0KPN9UccEBojT99+nS6du0KeEr5zjvv9G1c8ZBvecKECTkU4EcffQR452bWrFmceOKJqR1gPvm///s/AB5//HGOOuooAA4++GAAd63F4u677wbgwAMPBELWQ8ifZSOvpGQSPOOMMwB4+OGHE/q9HTp0cBPLf/7zn4R+d6IYNmwYALfccoubJCTN9XrssccCoQeO3pMpJBXo96dNm5ZjWzKDo9IFLQ6imTdvnjOHppKyZcsCXgDS4Ycf7h4oU6dOBeCvv/7K9/cuWLCA77//HvAexgpgK1euHAC//PJLIUaeWho3bgzA5Zdf7t7T3+zjjz/2ZUz7IysrC/DuSYAtW7YAcP311wPw22+/AXDVVVdRsmTJFI8wb5x99tkAzJ49G8jd7ZUbelZqMhT3339/AkYXiZlDDcMwjIwlqUrwkEMOAWDx4sUAlC9fPs+fXbx4sTPTSUnG4uabbwbggQceAGD37t0FGmthkWlTynTo0KGAp6TWrFnjQp2l8hRer5D7ihUrutVrKpWgERs57hWqLn7++WfAM9GkmkmTJgFwyimnuPemT58OwA033JDQ31LAhUzBCpzxm2uvvRYIPRsGDx4MeIFZMrfpbwKeWXfOnDmpHGaeUcBO+DmVAlTaigJixI4dO1I0uvwjN0F+FWBu6HybEjQMwzCMBJJUJXj44YcDeVOAcniOGzcOCK2M5ORXeL18ioceeqj7nL5bTnA/qFixIi+88ALg+SEU7NKjRw8A1q5dy6+//gqEgmQArrnmGsD7O82fP5+xY8embuBGXKSq5A9T8FX79u0BnP8sVejauuiiiyLe3759u7OEFHWU+nHfffcBoefGbbfdFrGPUlnq168PhFRg//79Adi1a1eqhpon5G/WMYT7+J5++mkA3nnnndQPrICoYEY8i0RentXR+0jdH3/88Xz++eeFGGGM30rotxmGYRhGGpFUJXjHHXfked+77roLgNtvv929p0T4efPmAd7K+6yzzgJCYduKotLqPNwPkCpGjx7tVulahXbv3j3X/Zs3bw54fgBFhL788stOLQYFJbY+99xzPo8ktYwbN84duyIv3377bYCUl/NTuTYleMvXLq677jpWrVqVkN/SMUaXgVNKgV8+QVl/pOgUIfvSSy/x7bffAl5ZsTPPPBPwVN9NN90UOAUodC6jozyXL1/uIsvTCT2/46U/iPzso/SK3r17M2DAgIIPMAamBA3DMIyMJSlKUNGcimqKhfJ1xo8fD5AnBaToKL2eeeaZTgnKX7hu3ToA3nvvvYIMvUDUr1/fqbl4CjB8f8iZL+hHojx4K+dYaNX1+uuvA170raK/atSokWseXToiK0OXLl3cedmzZw/gXaupRiUGoxXg6tWrAXj++ecT9lsqlKD2PPLBFS9ePGG/URCkMORj/+mnnwAi/IGvvfYa4KlGRWjv3LkzVcPMM9WqVQM8ZRvNqlWr3HWXDlStWjXu9p07d1KqVCnA8+8JxYNMmjTJPceVZ/jQQw9F7NujRw8XOf/WW28VfuAkaRIcMmQIkFPiK9F21apVLjFb4eaFRb+lcOno6h7JpHnz5jlOVixUk1NpFEqfUGi3XlNNdKBFOJoUtLj4+++/AS+YR3VD4xHU6hzhnHrqqQCu24XML+Cl4bz44oupHxje9SJ0ncicruTpRKDv+uOPPyLe1wOsVq1aKe2fqNB4JYpr8rv33nuBUNcMLcL0IFbd03vuuSdl48wvMrWri47QuZ0zZw69e/cG4Morr4z5Hdu2bXPJ5NHpE6mkYcOGrnZtNFqA9OnTx6WRTZkyJWIfpfzoPoPcu2KUL18+4ZWpzBxqGIZhZCxJUYLqCRjNsmXLgNzLUOWXOXPmcOGFF0a8V6dOnYR8d37Izs525sFatWoBOZPdO3TowPnnn+/2D2fMmDHJH2QMpPIU1BOPI488ssC/kyi1n0x69eoFQOXKld17SnORsvALjUnXjUz9L7/8ckK+X6o3/NgVeCKUirRkyRJnzk82nTp1cm4ThcxL8ciKcuedd+ZQyqNGjQK8oIpGjRq5Lh8KXPOb3NTd2rVrgVCnHNWGjYf6SCp4Sgn2qUDKe8aMGbleE//617+ASFfPN998A3jXc37MmosWLWLlypUFGm9umBI0DMMwMpakKEGt2qKLLye6GPOMGTM44YQTgFCB6mT8Rl5YtWqVWwnJ1i/VF142TZ2UlSSv0O6nnnoqpeMVUjoKi9fYE4X8AeHFDYKGlHu3bt2ASJUuv3Wqk+L3h5S7CivES5SXz0ldFGKhe0jBMPGoVKlSnsdZWC699NIcFgj5r2P5sXWvSVlIMZ944omBK7AfnYIiYgWpvfLKK4CXPK9gmhNOOMGpdxWqTmXfTlmSwntpRie5x1JtS5YsyfNvRH9f+/bt3XM0P98T9zcS8i2GYRiGkYYkRQnKFh/t+1LkXSLRb+T2m6mgadOmTglG+yekshYsWOBKwmmM8pH6FRW6bds2wAs7V5/H8Cit3NBqbNu2bW6VHR36rGjga6+9NmH+q0ShFaY6WEdHnE2bNs2V5vIb+ZdVEKJKlSqAF4Wt12Si1Jjzzjsv6b8lBg0a5MoRCil3WVOqVKnCDz/8AHiWjehnwJIlS1zx+nThl19+cSkfipDVc+LDDz8EvBQZSG1neaW+KaozVtK73uvTpw9Q8Gs01ncn+hlvStAwDMPIWFLaWV5+iURFaB1wwAGu87DfKKpLycbRZGVlOb+TFKAUmN9ohSnfUn6LMUsFRxfNVVf69evXF3aICUfJytHKXer40UcfTfmYckM5r7pvpNgThXxo4eXgFC0r1amE/ESVZ8sL3377rWuBJOS3DC+ILXWa6MLKfrJ8+XI6duwYc1us9kTyCSYTJbCrcXOsxgjynyviWPmd4BVIya0IwKGHHupyCWVpEuEJ9YlKkhcpnQQl2evUqcNXX31V6O+rVq1ajodYUAmvKuNXSkSykNkm0b3skkluQSCqz/n++++ncjh5omfPnoBnYtKCSybCcJYvXw54wVdNmjTJdWJXYvzevXvdewrs0iSo4Ay/UNCLJgaZ3ocMGVIkJr/o6y2WUFDqWXj91r/++gvwepImE7kMYk1+ixYtArwuEkr0FyVKlKBGjRqAZ7aOpnv37s5lJHeFzKGxEuoThZlDDcMwjIwlKUrw008/BUKhyeGozufzzz9P27ZtAfjyyy8L/DuxynHpt4OCSqU1btyYCRMmAIlLcg4KzZo183sI+aJ169Y89thjEe+98cYbQLDVrExCelXJrLwgE/z+kELWqkuhui0AAAQiSURBVD0oSAGOHDkS8JLmk9Fp3A+UCiKT43fffee2qTThddddB0TW6ZQKnjt3bkrGGYtFixY564RcK/np9KDjUQBQODK9ypSaDEwJGoZhGBlLUpSgVtNawaikmKhXr56zIc+ZMwfw+gmG+yVyQ+WEWrZs6d5TWZ6grOTVPV6V7LOzs3OUUisq5KWIdpCYOnWqc8ALOfR37Njhx5ACwyeffALAxo0bgchC4n5w0EEHAV4xAxVgGDhwoG9jSgTqzyn1I7+uCoOPGTPGBf2pEMjFF18c8R3r16/n0ksvTcVwI4hOYN+4cWO+0ryig19kFQwnlV1LTAkahmEYGUtSlKA6wl9xxRWAp9LCy3LJP6h+YCp8rTYbsfoBtmjRAvDCgcuVK+e2XX311YD/xZqlUhcvXgx4UW09evRIeGhvEKhUqZIvq9GCoPJg0cWhwVuJa5WqVIlMZevWrYCXmK2o2VQzaNAgwCsUoJY9qewXmgwmT54MeM0EpLg7d+4c8RoLRUwOGjSI//3vf8kcZtzfF2eddZbrtZlbf8RwpADbtGkT8X1vvvmmL9YyU4KGYRhGxpLUPMHff/8d8HyCyn3Rqi4c2fyVGPzXX3/lWHHI96TX7777zkWzqUSX34T7AMFLiC+q/sBSpUoVqs1SKtm1axfg5VaFowTevPikMwFF+5UoEXpEhEcrphI9D3RepDjSnc8++wzw4hpU1k0FsWM1AtAzRX7DhQsXJn2c4ege0X2kfMEGDRrQoEEDANfaLl5pM+WLq+2T8ga7du3qSwnJYtlxRpvojgwVKlQAQkEjcvLKlKbqIuG/ndvQNOH1798/InHUT4YNGwZ4KRGqviETblGlSpUqruu8ghiETGkjR45MWMX3RPDNN9+4ijFCZimZsY1goIdiVlYWADfddJOfw0k6CpQ544wz3MSvggebN28GvO4mfqGFicyZ3bt3d9uik9xjEV23N78VqgpCvEnZzKGGYRhGxpJSJRiLunXrAnDuuecCXn3Rs88+O8dqQiYDldB58cUXkz6+vNCxY0dX4/DXX38F4IILLgBSW2vRL1RiK/p8qHTS448/nvIxGenPQQcdxKZNmwAvlSq32rxG6tGzO7xsXTwlqEIHeibqc0rHSSamBA3DMAwjBiktoB0LhfjqVWVy0olWrVq54rLqH5gJClCoDFwqE1yNok/ZsmVdB5LwdCgjGOiZXbJkSZ9HUjhMCRqGYRgZi+9KsCiQnZ3twnzDI6UMwyg4P/74I02aNPF7GEYRx5SgYRiGkbH4Hh1qGIZhGMnEokMNwzAMIwY2CRqGYRgZi02ChmEYRsZik6BhGIaRscRNkYjnTDQMwzCMdMeUoGEYhpGx2CRoGIZhZCw2CRqGYRgZi02ChmEYRsZik6BhGIaRsdgkaBiGYWQs/w/rVmsjFp4MVgAAAABJRU5ErkJggg==\n",
"text/plain": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# visualize data\n",
"# visualize our data\n",
"\n",
"grid_images = np.transpose(make_grid(train_mnist.data[:64].unsqueeze(1)), (1,2,0))\n",
"plt.figure(figsize=(8,8))\n",
"plt.axis(\"off\")\n",
"plt.title(\"Training Images\")\n",
"plt.imshow(grid_images,cmap = 'gray')"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"# normalize data\n",
"train_mnist.data = (train_mnist.data.float() - train_mnist.data.float().mean()) / train_mnist.data.float().std()\n",
"eval_mnist.data = (eval_mnist.data.float() - eval_mnist.data.float().mean()) / eval_mnist.data.float().std()"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"batch_images.shape: torch.Size([64, 1, 28, 28])\n",
"--------------------------------------------------\n",
"batch_labels.shape: torch.Size([64])\n"
]
}
],
"source": [
"# parse data to batches of 128\n",
"\n",
"# pin_memory = True if you have CUDA. It will speed up I/O\n",
"\n",
"train_dl = DataLoader(train_mnist, batch_size = 64, \n",
" shuffle = True, pin_memory = True)\n",
"\n",
"eval_dl = DataLoader(eval_mnist, batch_size = 128, \n",
" shuffle = True, pin_memory = True)\n",
"\n",
"\n",
"batch_images, batch_labels = next(iter(train_dl))\n",
"print(f\"batch_images.shape: {batch_images.shape}\")\n",
"print('-'*50)\n",
"print(f\"batch_labels.shape: {batch_labels.shape}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Train Neural Net"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"# compute average accuracy of batch\n",
"\n",
"def accuracy(pred, labels):\n",
" # predictions.shape = (B, 10)\n",
" # labels.shape = (B)\n",
" \n",
" n_batch = labels.shape[0]\n",
" \n",
" # extract idx of max value from our batch predictions\n",
" # predicted.shape = (B)\n",
" _, preds = torch.max(pred, 1)\n",
" \n",
" \n",
" # compute average accuracy of our batch\n",
" compare = (preds == labels).sum()\n",
" return compare.item() / n_batch\n",
" \n",
" "
]
},
{
"cell_type": "code",
"execution_count": 119,
"metadata": {},
"outputs": [],
"source": [
"\n",
"def train(model, iterator, optimizer, criterion):\n",
" \n",
" # hold avg loss and acc sum of all batches\n",
" epoch_loss = 0\n",
" epoch_acc = 0\n",
" \n",
" \n",
" for batch in iterator:\n",
" \n",
" # zero-out all gradients (if any) from our model parameters\n",
" model.zero_grad()\n",
" \n",
" \n",
" # extract input and label\n",
" \n",
" # input.shape = (B, 784), \"flatten\" image\n",
" input = batch[0].view(-1,784).cuda().float() # shape: (B, 784), \"flatten\" image\n",
" # label.shape = (B)\n",
" label = batch[1].cuda()\n",
" \n",
" \n",
" # Start PyTorch's Dynamic Graph\n",
" \n",
" # predictions.shape = (B, 10)\n",
" predictions = model(input)\n",
" \n",
" # average batch loss \n",
" loss = criterion(predictions, label)\n",
" \n",
" # calculate grad(loss) / grad(parameters)\n",
" # \"clears\" PyTorch's dynamic graph\n",
" loss.backward()\n",
" \n",
" \n",
" # perform SGD \"step\" operation\n",
" optimizer.step()\n",
" \n",
" \n",
" # Given that PyTorch variables are \"contagious\" (they record all operations)\n",
" # we need to \".detach()\" to stop them from recording any performance\n",
" # statistics\n",
" \n",
" \n",
" # average batch accuracy\n",
" acc = accuracy(predictions.detach(), label)\n",
" \n",
" # record our stats\n",
" epoch_loss += loss.detach()\n",
" epoch_acc += acc\n",
" \n",
" # NOTE: tense.item() unpacks Tensor item to a regular python object \n",
" # tense.tensor([1]).item() == 1\n",
" \n",
" # return average loss and acc of epoch\n",
" return epoch_loss.item() / len(iterator), epoch_acc / len(iterator)\n"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [],
"source": [
"def evaluate(model, iterator, criterion):\n",
" \n",
" epoch_loss = 0\n",
" epoch_acc = 0\n",
" \n",
" # turn off grad tracking as we are only evaluation performance\n",
" with torch.no_grad():\n",
" \n",
" for batch in iterator:\n",
"\n",
" # extract input and label \n",
" input = batch[0].view(-1,784).cuda()\n",
" label = batch[1].cuda()\n",
"\n",
"\n",
" # predictions.shape = (B, 10)\n",
" predictions = model(input)\n",
"\n",
" # average batch loss \n",
" loss = criterion(predictions, label)\n",
"\n",
" # average batch accuracy\n",
" acc = accuracy(predictions, label)\n",
"\n",
" epoch_loss += loss\n",
" epoch_acc += acc\n",
" \n",
" return epoch_loss.item() / len(iterator), epoch_acc / len(iterator)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"import time\n",
"\n",
"# record time it takes to train and evaluate an epoch\n",
"def epoch_time(start_time, end_time):\n",
" elapsed_time = end_time - start_time # total time\n",
" elapsed_mins = int(elapsed_time / 60) # minutes\n",
" elapsed_secs = int(elapsed_time - (elapsed_mins * 60)) # seconds\n",
" return elapsed_mins, elapsed_secs"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"---------------------------------------------------------------------------\n",
"Epoch: 01 | Epoch Time: 0m 12s\n",
"\tTrain Loss: 2.257 | Train Acc: 19.67%\n",
"\t Val. Loss: 2.208 | Val. Acc: 11.47%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 02 | Epoch Time: 0m 15s\n",
"\tTrain Loss: 1.928 | Train Acc: 35.99%\n",
"\t Val. Loss: 2.658 | Val. Acc: 15.09%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 03 | Epoch Time: 0m 17s\n",
"\tTrain Loss: 1.684 | Train Acc: 45.90%\n",
"\t Val. Loss: 2.697 | Val. Acc: 18.07%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 04 | Epoch Time: 0m 17s\n",
"\tTrain Loss: 1.530 | Train Acc: 50.23%\n",
"\t Val. Loss: 3.533 | Val. Acc: 16.05%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 05 | Epoch Time: 0m 16s\n",
"\tTrain Loss: 1.444 | Train Acc: 51.98%\n",
"\t Val. Loss: 3.861 | Val. Acc: 16.42%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 06 | Epoch Time: 0m 17s\n",
"\tTrain Loss: 1.393 | Train Acc: 52.64%\n",
"\t Val. Loss: 4.699 | Val. Acc: 16.97%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 07 | Epoch Time: 0m 17s\n",
"\tTrain Loss: 1.359 | Train Acc: 53.11%\n",
"\t Val. Loss: 5.268 | Val. Acc: 16.34%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 08 | Epoch Time: 0m 16s\n",
"\tTrain Loss: 1.335 | Train Acc: 53.38%\n",
"\t Val. Loss: 4.952 | Val. Acc: 14.97%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 09 | Epoch Time: 0m 17s\n",
"\tTrain Loss: 1.319 | Train Acc: 53.84%\n",
"\t Val. Loss: 5.703 | Val. Acc: 15.13%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 10 | Epoch Time: 0m 17s\n",
"\tTrain Loss: 1.304 | Train Acc: 54.26%\n",
"\t Val. Loss: 5.436 | Val. Acc: 15.64%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 11 | Epoch Time: 0m 17s\n",
"\tTrain Loss: 1.292 | Train Acc: 54.25%\n",
"\t Val. Loss: 5.915 | Val. Acc: 15.70%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 12 | Epoch Time: 0m 17s\n",
"\tTrain Loss: 1.279 | Train Acc: 54.77%\n",
"\t Val. Loss: 5.348 | Val. Acc: 17.54%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 13 | Epoch Time: 0m 18s\n",
"\tTrain Loss: 1.269 | Train Acc: 55.03%\n",
"\t Val. Loss: 5.428 | Val. Acc: 14.49%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 14 | Epoch Time: 0m 17s\n",
"\tTrain Loss: 1.258 | Train Acc: 55.50%\n",
"\t Val. Loss: 5.109 | Val. Acc: 15.97%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 15 | Epoch Time: 0m 18s\n",
"\tTrain Loss: 1.250 | Train Acc: 55.75%\n",
"\t Val. Loss: 5.109 | Val. Acc: 15.11%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 16 | Epoch Time: 0m 18s\n",
"\tTrain Loss: 1.240 | Train Acc: 56.00%\n",
"\t Val. Loss: 5.531 | Val. Acc: 15.63%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 17 | Epoch Time: 0m 18s\n",
"\tTrain Loss: 1.232 | Train Acc: 56.31%\n",
"\t Val. Loss: 5.849 | Val. Acc: 14.88%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 18 | Epoch Time: 0m 18s\n",
"\tTrain Loss: 1.225 | Train Acc: 56.48%\n",
"\t Val. Loss: 5.722 | Val. Acc: 16.13%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 19 | Epoch Time: 0m 18s\n",
"\tTrain Loss: 1.218 | Train Acc: 56.77%\n",
"\t Val. Loss: 6.119 | Val. Acc: 17.38%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 20 | Epoch Time: 0m 20s\n",
"\tTrain Loss: 1.211 | Train Acc: 57.05%\n",
"\t Val. Loss: 5.578 | Val. Acc: 17.43%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 21 | Epoch Time: 0m 18s\n",
"\tTrain Loss: 1.205 | Train Acc: 57.24%\n",
"\t Val. Loss: 5.528 | Val. Acc: 15.08%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 22 | Epoch Time: 0m 18s\n",
"\tTrain Loss: 1.198 | Train Acc: 57.58%\n",
"\t Val. Loss: 5.489 | Val. Acc: 17.37%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 23 | Epoch Time: 0m 18s\n",
"\tTrain Loss: 1.193 | Train Acc: 57.58%\n",
"\t Val. Loss: 5.657 | Val. Acc: 16.93%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 24 | Epoch Time: 0m 18s\n",
"\tTrain Loss: 1.191 | Train Acc: 57.96%\n",
"\t Val. Loss: 5.609 | Val. Acc: 16.39%\n",
"---------------------------------------------------------------------------\n",
"Epoch: 25 | Epoch Time: 0m 18s\n",
"\tTrain Loss: 1.182 | Train Acc: 58.17%\n",
"\t Val. Loss: 5.460 | Val. Acc: 15.82%\n"
]
}
],
"source": [
"N_EPOCHS = 25\n",
"\n",
"# track statistics\n",
"track_stats = {'activation': [],\n",
" 'epoch': [],\n",
" 'train_loss': [],\n",
" 'train_acc': [],\n",
" 'valid_loss':[],\n",
" 'valid_acc':[]}\n",
"\n",
"\n",
"best_valid_loss = float('inf')\n",
"\n",
"for epoch in range(N_EPOCHS):\n",
"\n",
" start_time = time.time()\n",
" \n",
" train_loss, train_acc = train(model, train_dl, optimizer, criterion)\n",
" valid_loss, valid_acc = evaluate(model, eval_dl, criterion)\n",
" \n",
" end_time = time.time()\n",
" \n",
" # record operations\n",
" track_stats['activation'].append('ReLU')\n",
" track_stats['epoch'].append(epoch + 1)\n",
" track_stats['train_loss'].append(train_loss)\n",
" track_stats['train_acc'].append(train_acc)\n",
" track_stats['valid_loss'].append(valid_loss)\n",
" track_stats['valid_acc'].append(valid_acc)\n",
" \n",
" \n",
"\n",
" epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
" \n",
" # if this was our best performance, record model parameters\n",
" if valid_loss < best_valid_loss:\n",
" best_valid_loss = valid_loss\n",
" torch.save(model.state_dict(), 'best_linear_relu_params.pt')\n",
" \n",
" # print out stats\n",
" print('-'*75)\n",
" print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
" print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
" print(f'\\t Val. Loss: {valid_loss:.3f} | Val. Acc: {valid_acc*100:.2f}%')\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Visualization\n",
"\n",
"From the above, we can tell that our model is severely suffering from overfitting by the gap between training and validation accuracy. However, to attain a better understanding, we will graph our recorded statistics and use HiPlot, a new graphing library by Facebook, to understand the overall patterns of our model\n",
"\n",
"**NOTE**: If you do not have HiPlot installed, go to their [github repo](https://github.com/facebookresearch/hiplot) to find latest installation"
]
},
{
"cell_type": "code",
"execution_count": 140,
"metadata": {},
"outputs": [],
"source": [
"# save statistics\n",
"# track_stats = torch.load('ReLU_stats.pt')\n",
"#torch.save(track_stats, 'ReLU_stats.pt')"
]
},
{
"cell_type": "code",
"execution_count": 141,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"