{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Activation Functions\n", "\n", "In D.L., our **objective** is, almost always, to find a **set of weights** that **minimizes error.** All of these sets of weights are **linear operations** and hence, if performed alone, we would attain just a simple **multiple linear regression model.** \n", "\n", "##### What’s the Problem with Linear Models?\n", "\n", "If inputs are left untouched, they are not **flexible** as they can only model linear relationships while most data out there has a **non-linear patterns.** Hence, we need to find a way to force our model to be able to **learn non-linear patterns.** \n", "\n", "##### How do we do this?\n", "\n", "After a set of linear operations, we apply to the new **input** created by the linear operations ($Ax = \\hat{y}$) a **non-linear activation function**.\n", "\n", "Suppose we have a simple linear model $\\hat{y}=ax+b$. These $\\hat{y}$ form a linear operation such as below:\n", "\n", "\n", "\n", "Well, given our orange line ($\\hat{y}$), we then apply a **non-linear activation function** so as to **transform** our linear model into a **fixed non-linear model** such as below:\n", "\n", "\"The\n", "\n", "**Why is this a fixed non-linear operation?**\n", "\n", "Because whatever formula we use for our non-linear operation, **we do NOT** have a set of weights on it that try to learn an optimal non-linear representation. It will always follow a **fixed, single transformation.**\n", "\n", "##### Well, isn’t our purpose to find an optimal non-linear operation?\n", "\n", "Yes and no. We find an optimal non-linear operation by letting our set of **linear weights** learn a **representation of the data** that, **once fed to the non-linear operation**, will **correctly identify the new pattern.** Hence, the objective of our linear weights now becomes to **find a representation of the data that, once fed to the non-linear activation, will correctly learn the non-linear patterns.**\n", "\n", "##### How do we backpropagate these non-linear activations?\n", "\n", "Given that these non-linear activations are in fact non-linear, we are unable to just take the input as the gradient as we can do with linear operations (3x => 3). Hence, we will need **two things**:\n", "\n", "1. The **input ($\\hat{y}$) that was fed to the non-linear activation** and\n", "2. The **derivative equation of the non-linear function.**\n", "\n", "Given that we want to apply the non-linear operation to every input, we can classify these operations as element-wise. \n", "\n", "This has important implications on how we can calculate our gradient. \n", "\n", "**First**, as we learned on the \"Linear Layer\" tutorial, the dimension of the incoming gradient from our subsequent operation will equal the dimension of the output from our non-linear operation. \n", "\n", "Now, since the output of the non-linear operations equals the dimension of the input, we are able to calculate the corresponding chain-gradient with a simple Hadamard product (element-wise multiplication) between our incoming gradient and our current non-linear operation. In other words,\n", "\n", "```input.shape == output.shape == incoming_grad.shape```\n", "\n", "**Second**, given that there are no weight parameters to these operations holds two implications:\n", "\n", "i) from a backward perspective, these operations are only intermediate variables and \n", "\n", "ii) we can just apply the derivative of the equation to each input such as shown below\n", "\n", "$$z = \\sigma(y)=\\sigma(x_ow_0+x_1w_1+x_2w_2+x_3w_3) = \\sigma(x_ow_0)+\\sigma(x_1w_1)+\\sigma(x_2w_2)+\\sigma(x_3w_3)$$\n", "\n", "Hence\n", "\n", "$$\\frac{\\partial z}{\\partial y} = \\frac{\\partial z}{\\partial y}(x_ow_0+x_1w_1+x_2w_2+x_3w_3) = \\frac{\\partial z}{\\partial y}(x_ow_0)+\\frac{\\partial z}{\\partial y}(x_1w_1)+\\frac{\\partial z}{\\partial y}(x_2w_2)+\\frac{\\partial z}{\\partial y}(x_3w_3)$$\n", "\n", "Now that we generally understand how to implement non-linear operations, it begs to ask, **what are some common non-linear operations?**\n", "\n", "Some common activation functions are shown below:\n", "\n", "* ReLU\n", "* Hyperbolic Tangent (tanh)\n", "* Leaky ReLU\n", "\n", "Each has their own unique properties and usually, finding the best best corresponding non-linear activation to a model is left to trial and error." ] }, { "attachments": { "image.png": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEKCAYAAAARnO4WAAAgAElEQVR4Ae29d7wcxZk13FcBzH77vvz57bfBXrPe9WIMSEKA/NpeB2zjtHKOGOcAJgtEkg0GvCBQTggwBpsoCQQWJkdhXoMFXEAoXQkJSSjrKks3TDzf71T101PTMz3dc+OEp++v7lO5uk9Vna55qrrK8/RSBBQBRUARUAQUAUVAEVAEFAFFQBFQBBQBRUARUAQUAUVAEVAEFAFFQBFQBBQBRUARUAQUAUVAEVAEFAFFQBFQBBQBRUARUAQUAUVAEVAEFAFFQBFQBBQBRUARUAQUAUVAEVAEFIFBRuDd//EejDh5FI4bMwpHf/g4HP2RD+LYMUfjOMccO+YYuOaYMcei2ByPY8bQWP9C3KNL8nLzDewnH4PjTi6UIfl84EPHgkbcIgv52zRBPs49l/M79qT/hGs+eOL7Yc3R+OCJR+OYk6w5+qRjQPP+k60Rt8gPnvwBuMbNs2f2D+DYkwp5fuDkD4LmP8dYI26Rbtm096xMFwstX/HX9if9SvqZ9L//HHMcXPOBk4+D93de+yBTd8+KP37kCZArCyCDLHK5FHKZLmTTnUhnaLoDk8p2I5VJh0wWqQyN75+V+JK+0+TF/Mobid8Nyb8rm4ZrgrxZRpC/pIvKt+DP56HJ51JAPm2elE8L8KnzIQPkUGwEo/IyHLtad2mupXdUGqfgU2154fiFnMSm5Re3CsGlvAzjWa27NFfFv7bwl/pgzR7MdcEb7rX2jHEHOdUJx59oOY9P4lz5fB7ZTBbd6RQyuQL5kR6TmHCTd7Iua5X4kjcp2TXiL1LiiyybaQ88JT++Clwj/pS8B9dNe28vyU+ez3122sVfpMQXqeX3DgHBUfBV/Ae2/9U6/tIeUj4veENqhfAP89q84ca0eofF39TIESda9mJLJ+p5oKujM+g9JP1yl7zxwrJc3J74hfMVd0/y6kkaaYBhybxcv57kXSmN5C3EE5YSLrJSXj0Jk3zD5YpbwkX2pIxKaSRfKS8sJVxkpbx6Eib5hssVt4SL7EkZldJIvlJeWEq4yEp59SRM8g2XK24JF9mTMiqlkXylvLCUcJGV8upJmOQr5QrRu5JxvGHx3DowY/lhRrd0ZNLCjjthpMWFT2EuUisdeSCfBTIpIJ0CMpnA5LM5FJkMkKfx/YO42RRAwzwqmkLeTCv55HI50IhbZJB/cE9x+YfCcxy75+zz8RnzfFZ5pVDGXHmmdQzz65XhTygnP7kVuQ1xB9KNy7Rafu8wUPy1/bl9ytfwSv9zpM+MNUX4bZ7nJSb8D44+3nkcMkoWyGaxdf16tC1dig2rV2Ft24pos2oV1rqmbQXWraJZltBI/EI+a9pWYU3bat/QXmzKlyf5xEtDDkWEX470A3YN6ff5cnAbR2/JVtK7ebJZhUxRmW5cSd9b6eYZKttgFQoP7qe35Ur6UP76/MVtIMA7hBNHWr1+4TOPUL41j7+MMss8P3XQNFn/RR7gk7JYhbHksxvj5yXhgoFLBT4b1I4Of6jX7rV4rcYM8caVHekf4Y3z/h/zk6T1X//9veax+BMmhRzSyGHt6vV0WCU2+76JkUHOaLVTyKMbyHcDOZq0Y3y/fCdQzpj4TJsumFwW+TxNvthI+wv7l7glvSuL80IuD9esXfNW4SXHyuzVVYYcpaFUI91GVe6ewuHirqaMSnElP5FhTMQ/LCvlWU1YSb6hGwiHi7uaMirFlfxEhoovmdcP4mn9lwxOKuEcFSZ4iozC3/hXwJxB5K5ul7+YSH4J58wdMMqBTrKeHb8hmzcDXWTJZ0I+pWM9kwD5mpq0Pcon+SM9r4UzyWPKkr7vefQHj0Y6m0dXPo9DSKEDWby1ZqMFjHgQQPPPTmHmYAk/zylVEncua01A4v7LAJwHsIYvCGvSYDpL8ELQeWThG2Jezkh4UhnKQ17YIt9as87WW8x/aVbhaOIvMhyubkVAEeg/BEjZUX/mxUw2pyHHC4Xxdkjk/nugw5fUzZNzOOAscFkCwk8wP1qJd/slbKjnzfS8iFG+X+Lxo0cYUNwRPgkxl/ZZ0xC6/NSxpC8j8srknEX5P4fgSeAhci7rTkr0Ei+UpxC9SCX8/uuMmrMi0N8IRJG9pXOf4R2yN/785UCSy+aNnoJ+fCeQ0bLIOVzFiP4lvzZcaYJqZ4RP3b3o748c7nmLPc8bW+ltcuyo4+TxkEPePPpbb61GjpOshuw5ind1fPYnkBBzJg8ExgfQvhYIJsEtNUFaUbWFCFrCAylEnlSG8hOiF7lm9drgmekX/FwPfK2FQaaxRPhHhYeiq1MRUAT6EIFKhC99MixNH6daH0B7Hnhp41YsePoZc1cpZGFN3ugvgj7vEr1rp0qnJkb47/KO8pcLtQ7xPE7eltfhO2+AY08c4VQFnyqLt1a3Id3JHz0+GToxaA2DWdnN6ilOw4+oeFGVk8nlSkb5nG+5+Xe/xz+/+z349ne/hwX334crrvqN+bArnc0inU4bI7fFn2NXXHEFHnvsMZtvDOGvbivo8PlOCxN++H6lHJWKgCIw+AhEEb4dqVtSlz7MQWM6C3RzsSEAstobeeCzE+bgR9fOwZYccAhAF/LGUGHNtAEnuEQv9pohfIfIk1pLCT+P1StWBrXKh+dPH2rjCQzNQQKXL2N8QAlqlGE+xI0fdPFi/sFI3ifq7nQGx40YhRWr2kycUaNPwM5d7ehMcSam+CL5p1IpvPPOOxg50i4xDecnI3uRq1atNpnQneXPE39SVyaOpbGILC5RXYqAIjCYCFQifAkz90eiIX+l7UtgD4AdAH5805P42MW34NnNKew1ZG+1ESR7aiTMJeQeln6gVxMj/KQs78QbcaK/Dt95yk1vvWMfWogdgIC1HQDNzh4aAtxBJqX6J6TSEYL9xre+Dc/z2kefNAYzZs3B6JNOAkf2vL71nW/jxhtvNPbZs2fjjDPOMHZ+IPbRj34U7e3thUlgXwUkRC9yZdsq+6JheKZ0hZDch0hTgP5TBBSBmkBASL1U+spks1TV19/kgIOpPA5QlQNg6mMv4NMXz8FNf9mE3RzIktDNgC9rBp7MIf6qFZWOQ+RJraWED7yzZpNZpZPKAtN/fx9+NesOjL/xHpx78wKcc/P9OGfuQpx344PWzFmE88SIXyAZT4yNP372vbjnz08aTLk6qKPLbt1A8heC7exO41/e86/Yf/CQUe2Mu+hCE7+juwu79+3FP/zDP2DJkiX4p3/6J+zatStQ8Zx22mm47777qiJ8vkhkZC9S7kNkfAPQGIqAIjBQCJQSvfiQrv0hOQV/vXOO0R+kLly9ER879wpcPf8F7KOmgnF4cTCZTZvPa6gdiL/qmPBHjR5pdVbmKS3rrl+5wbAvCe/uhxZj/PTbcO6sO3DGnLvw8znzcOac+3CWmNkLcVYi8yDOmv0gLpx5N+bedZ8pkyTPiyALuVJSpfP+o48xL4Nr/uc6XH311XbZlI2OBQsWmF8At912myX7biqKgMsnXIq5N80JCDxY7hma7HVH+C7hS3y/mOCexB1Iv01J2wpkEMFa3GfqT3uoWHUqAg2OQHFvErqndC/+ou9O5Yw6ekMWGHvNLRh71e/MyN5ftem/H/xlmG7i0LyjWyIT1a1KRwjfQmUfa8NbG83bjs+/P2fVN1sAbACwHsAmAHT3xOzyN0Vj3iR6GdnTLaDS/z3v/Tds2bYD02bMxKWXXhpUBVUwEyZMwL//+79j5syZxt+sKAJw5ZW/xsyZ05XwA7TUogg0IgLCFFaGCd8N5VCQqpzz5szD5y6bjZf22vnFYJBmRvSSohgr8Q3Luib80SeOCnZiFMpdtmxZ8K4kYJwq5SQsJ2tp6KbhW7Ja4+rISPZU35S7/uEf/xkbN23BM889j5/85CfBCL+1tRUnnHAC3n77bbz73e8G7zWf5f47GXzr29/AU08/EXzYVf47gCyWr15lnpkTNBzhy8hepNyPVLS4A6kj/AAKtSgCA4+A9EwrXcLnTF933vIS72tLZx5TnmzDJy++EQ8v24xdXBjCgHJ92H8QN3fGdQ3DGobw7U+iDJYuX2o+RODDEReStBhX/WJfED5KlUQIXBI99fdimGf4+n//v3/E7r37cbCjC8cee6wh/IMHD+LDH/4wXn31VbMy55lnnsGoUaOQTXeb7wb++Z//EdlsqirC785VJvzwfRl36HmCxhOK7Dac/rSHilWnItDgCBT3pjDhH0xng+/8H3hlHT56wWyc/bunzIqcYKVfuT7soya5u0QvdoY1HOGvXL3CrpHPZ5DLZQyBBvvfmIXrFhIBplpJyJiGxE+yl5eIj3eg2pF8v/vd7+LRRx81we7umTmusDEbJGVx79134qorfm0/GPNzcBuCa1+12i7LtPdgX3NSFmXsVa6xlHlpuXn2pz32fjWCItBQCBT3Jrdvk5ipdaBG4pUDwOcvuxk/nzLfrCrkUnKqf2VxhsgwNMW5i96jIBuM8HNY2bY6IGRDsGbfHLtHjtk4LRjvCzRhyApuieFKhtItZB9H+BzZP/fcc+ju7jbbJReI3s7CU53z2CMPY98eLrSS3N1mUGxXwvdhUqEI1CUCLpsU920qiPmtEJeNf/26efjClX/A0j3Wr7ODmzty76/Q5oohDIpzLxC9+DcY4QMrV60zS1N59knKLGviBwlpZNCNDDqRQ5f/uQJllBZflECUBahcbGVdPHXnrDaJV7BZHzcNP7QyN1dmRE3it5fNobgpFFxK+C6ialcE6g2BYoYo9GzLTJsB/OqeJ/F/zr4e9y3fjf0+s/ApU4c6lfAL+ikCScJfX0T4/ALN7jVhSZ+wJiN9eRkI+UtF2QZmCN9sd8w7kDjchlmo30r+9Kp82fBusx0E7TYHtyG4diX8ymhqqCJQ2wgUM0Q6lzLbJnD4yQ8772zdiDFnXINpjy81H4wekA/0jTqa9KAjfDMTTVIkWa5a9ZaZrSU+Rs8erGO3615c8iy22z3zuW8+t1EuNmVG+uRmw9UM48tBfjH4e1Rzz+pD1MbZSV6uz0+lOK/A++Rupzm7eie0NYIJrPAvTPgVomqQIqAI1BoCAW9YvurMpLDfbO8OLNkLfPqSWfjhtAexicvree8cT/Lcjjw5Rnio9KHkNSIh4g5LhtfOAShJP7H147nLMoXwV69aFRA+SV8mN4zk04euYkDkFVAYsRdG734NSIWZ2mDqUsKXIsoUJ0FG7t+/36p4nLc2XwTyUiiK7DuU8Muhon6KQB0hYDjEckwKGaO24XdC3518D747aQFWZGG2U+B5JoZeeNSqGYhGM4rwmKAg7rBkeAMRfgZvrVxhtCJyhqzo2kWa12YYBZfEAyIPR3L2T5Agk6kQvlUBcWmle/FXhlxcpsmLL5/OTvuFrdE/+YQv8SpJJfxK6GiYIlDbCAR0kO0CcgfN4UrUA1xw57P4r8tuwTPb8tgm++QU9NVmoE/+D9KHHlMoSbzFHZYMbyjCX7tymT3gO53ylz3mi45xLVGyhxERt7whoiTfHOb0c/98Sv/nFkfnc+bMwb+//z/NFsnjLhpv6oDr9ufe/Dv88Y9/lDqxMqTS4TLODRs2oKOjo2gbZUmkhC9IqFQE6gsBUgs5nJSCXJdR0+zPAvctWYGPjpuK3zzyGrb6pM5FJ4EGh9/++IpjJi13CW1JmLjDkuENRfhvr1hqiZhbGMdOmAo8SaX/U4BEb6rOfwWbl4LV6Xd1pvC+9/0H1m/chIUPLsLEGyaZzDnS/+Bx9tB1Udl0ddkRv5RO/6eeegokfS7jLHet9tfhM0zyKRdP/RQBRaC2ECBNGCI3a+o5pwg8tyVtJmnP//2fzAdXASOQqUXzIC8BX4ZJ3HUneeKGIvyNq1cAGR5EzjdoJ5DeA6R2At3brencCpQ1m4FOmnes6doAFJm3ge7NQIo76hBivlA4uvdVPX7lfPObdnvkE048CReOvxg33XKrqYOnn12M75x2urGTzLnFwuOPP27c1113ndlzRwj8Ax/4gJ3QLVN7SvhlQFEvRaDOENibA1bsB74zcR4+N+Ems207t0HmgYV2PF9M+ML+LrmXsyeBoaEIf/kbr1rAutqx9sk/Yu38a7H13kux5+5zcfCuM7Dvzl9gz10R5u6fYY8xP8Geu2l+VGQ2zL8Qa19+AOn0LmQzB5BN88XiVAztAN73b/+JPftYfXb3TMrxl1yGOXNvMqNyjuxXrFiB9773veajrJNPPtkQPF8E1O1zhP/KK6+Y9OF/SvhhRNStCNQXAvty9lyOM25+Gp+/dDaW7M6Dw8hOQyZU3tC4vEL1Mef8aEqXfrvEnwSJxiL8N5cC2W4guwdvP3sv2ub9Fpvvvhi77/gl9t3xE+z544+x646fljH0/zF23flD7Iwwa+edj6XP3Y1cbj/S3ZbQyxH+B44+Du27eeyKvbgk84c//inuvne+ObSEp1xx50zq+ltaWtq5P75c9P/Wt75ltlEOJnYlEIASvgOGWhWBWkNABoCh+5JxO4PXp4Cpf92FD427Bbe/8LZZf88vbLNGaU+yZ2yf8MnmeS43PGSNEr6FR5Zlrl271uw+ad6SuQ4guxtI7wDSW4DUJiD1jlXNUD3jmhTVNb5JbwbKmcxOINdp9udJZ7qL1S5+RacyWXMAyq49PKbArsFnnXHydubsG4M0XKlz+umnm10zeZ6tvASY5mtf+5pR95D8w5cSfhgRdSsCNYKA4YDS/el5TCHH5tytl9smvLQb+NBlc3DFoyuwdr8968QMHM3Inn2eGfmXyZP/5EVANrFKZXdkL3ZJFi3reD/8cuvw16xbG+xTX4DGrnk1095mPwq7J4UsiXS/XpNthrn9cDnDvXPMHK35yjZcMRbm9/3H+81HVyR/Xhzh//72P+Cqa35rRvb0W7hwoTnHdtWqVXjPe95jTr+iP0n+X/7lX8xxhyZx6J8SfggQdSoCNYMAGcdyjRBwmlvgANiXtWS/vBv48pWzMfaam7Asb/fJKay1NOxeTPjlVAhK+ATVQmwJX1xsCYVKKFrn5GLr8LZUFKm6rPF3yJTN0yS+SJb4z+9+D3bu2m1ePNIW31y+EsePPME49+3bZ0ie++LzmjVrFr75zW8a+969ezF69Ggz4jceoX9K+CFA1KkI1AgCZB2qZbiJC7mDnMATCDu77Xf4PIT8Fzc9gE9cMBFLd3eaVwO/2mG84itMTq67OKbwjsji0HKuhhzhDx7hc709R/Q879a9uM3C977/A3BET3VOuYs7a06aNAnTp083wWaztVBEJfwQIOpUBGoEAbIOyT7tr7URIu9M231ypi56CaeMn4JbX2wzu2JyHx3RPUhc+yguwYftxQ8rRC+yOLScqyEJ3741LYgCRUhG4BiK5f9uKOQXFy4jf5Eu5G+8uRzPP/98MHrnx1Xha+rUqeaFIC8FStco4YcRU7ciUEsIkCEcnUs6a07Yu+eN7fjYhdPw2/ufNZO0nOHjlzbUzJP0C1cEMRUiFNnCfFQUWNbR8ITPpw7DUm5fHItOmZjlUkf6CdGL9KvfZG78nIlYIXK3XrgWn3vsUJfP5Zt0u0YJ30VL7YpArSHgc4vodHiedgfwlatuxQ/nPGjO1eaumFyVw+EeP7QKCJ+//M2vf5f0Kz9fmK8qx2ZoHRP+CaNHBnp2UeKUTtryIV1YBMx4aMrFcHMqZxeidyXjhS8ZwZfzd1fnMF4U4UflEc5T3YqAItA/CAgH2NzpsntqkZi4MS53vfzxxLswdvwMvHnQrr/Z0bE/0N9zlB8QPpOTnoomaivft5QvsnJsm7d3mNda5T6VtRG9Xgif5M8K4UU1DgmcF3X25S53/b1L9rS7I3wl/HLoqZ8i0F8IcPQue2fZMoqJlpO1PH2DcewIftKfluGzF92Ch17fafe3Rx6ducK2KVaHb1i+QBJK+OVfMCT8AuB25E5y3fDOVry5vA1r1r5tJkk5Ubp61QprVq/C6gRm1ZpVKDGrVxvCJekOlnGbshK+i4baFYF+RoBknz1odrkkOwv3mOGb2V/LjtX5n+qaR1/bjM9ffCt+s+ANbM9Z9Y3V9JgUwSDQukID+wqKCLfcsD0egTpX6cgDF34GFR6ZG5al01mz62Q23Q1jsilwC+M4Y06i4Wk0IUN1y2AYGemT5IXoRRaeWG2KgCLQbwiQ1HMFwpdl26Y8/4CSg9kMOCH7SnsGX71kBn425UFs9HX1hfsS1qKMuCII301Zzh6Rm+Ndx4Q/atSI4C0rhO8uh3T3oi/o8QXJnkoHO7UqAopAEyFAiuX43Y7uyxH+7u4ubMgC37r2Lnz9mrux6iCwuxvIcHa2m1skEC6XqiPgE3oKBbspy9lD0cs465jwR448PoBOCJ8g8CLZp9J5ZDP+iNjsOy9T5+WgKvZjlZb787NXoQgoAk2IAFlCiJ7Un3Lm57jEkpugXX3PY/jkBdOwcPk+uxLnUDrYHqeY7IWtygCphF+qxx8x4riyhO9SdwBlAKCEBiG+RfzjZDiduhUBRaDREXBZgYRPshfC78wCh7IwRxXevmQjTrlgEuY89II5ptDgYs5a9Qf2SUf4/QZoTY7wW1q9od6iUoov9jn++GOV8PutYWjGioAiIAi4hO/6kb+5EHNLN/DSDuCzE27FOb97DLuYgFeem+n4G+qUDDolkh93QETtEf64Fu+wRZ7XkpjwrVbNoulWTBGcsW/WcMoo94DUihaiCCgCtYiAmbjlsku7dQpV8/sBcFes066bh69ecweWya4qPLeW+9jn0laTEzyPyy2B5wBZaovwj/K8lsWeN3yM5w1NTPgWvjxyoCnWkhVQjA6xccLhUe5CjmpTBBSBZkLAX4dvTrsj1efMlsfcFO3K+17FJy6cjSc2pLGRyvzgStv97AM3LS63FAUMgKOGCH+oZ9Q4YyzhD1tcrMApdX3w+GMMQIRPtjIOT8taBHsCsJvGtQ9AnWgRioAiUJMIcJm2XSCSxr6uDuwGcNdf1mDMz67BtKdWGTc/qyJjlLtcJomKUy5d3/nVDuGPPdzzZvq0PiZSpXOEN877O/NpcOt7jnpPgAMJ392lToAtfptWA7HkEJZBkWpRBBSBJkJA2KMr242OdKdZgfPc+gP42uWzcP7cB7DNX2/vMkYYHjdM8gvH6V93rRD+cG/mcM9rG+557cO9lvZhntfutVSeuB1xwogAG6pzoglf5tSrgThcNeIOilSLIqAINBUC5IAssv74nRugfeqyOfjSNbdjfbfd/dLOJxaGmWF4hEVEhsP7310rhO9obIZ73pjDrHrH8S21HnfC8QE+XDPPihAgRVrolfADoNSiCCgCPUSAO9faPex3ZoEJtz2KT148G49sJr/wIutEk70bVuAnk2QA/9Ug4XueN7YlAeEfO+o4BygLoQApsgB/wcdJVMEq8cOyQhINUgQUgYZGgFuyHAAw+y+rceqEm/HA0p1Gb28WAfpjfzJG1NVrNpGlnVEyquDAvzYJv3Q4X8YnGOETRR40G3rDFoCXsIJP8PyRFjeNa49MoAGKgCLQwAgIe/zlrR34yAWTMX7Bi+bLWnIvVcpU9whTRMEg4SKj4kX6RxG9+EcmlIBGIHyzNIdLpmyVCJjWxQct9ZHHj5ZuGtcenUJDFAFFoHER4ELMZfuAr10+w0zUbvA/uiLNy3YLcU/vMgntVV9C7FEyNsM6JvzjR4+wK6SU8GOrWSMoAopAAgSESAN9gT2GkGTPk6rOnn0fPnXutXh9F4xqx2rvRYcfn78SfhlVTVKvgPCJYkWVDisiDupweJQ7vlI1hiKgCNQhAtwITYy/SRr5n1/SbgYw5aGX8bVLJmH+iyuMKqe9C+D5G4ZasjmAhu6avup4hD/ixJEWYHKzT+hhmi5gHx1i44TDo9yFHNWmCCgCDYSAkHfOsgk/mOVnVlsBzF++Hx8/Zwomz3/SnFzFET+NEn7S4XkfxFPCb6DOpo+iCAw2AhzjUTtDFbHZ9jhnPq5asT+PT190I34waZE53GT3gTy6c8ChLj963hJ/fRxIVMcj/JEnjUo+wjc/tdxRe7h1uWGV7OF06lYEFIGGQIAcQcKnydvR/R4AZ0z9I74y4Ra8sMVug8xnzfOQchT0+xzt89cA/Wr7aibCLyL9cLVUInk3LJxO3YqAItAwCJDsO+3uZzzM5JJ5i3HSGb/Foyu2G0I3z+mwOmOS7PnVLaUJIs+IqTlglPD9KnFJvZK95mpQb0gRUAT6AgEh8qwd5D+8bCtOnTAXEx9fZiZuAzKXeM4IXwm/D3T0cVmoSqcvWrnmoQgoAgYB6u0PcogPLNsPfOHiGThr1j14x1fvWJSKh+5U98vkLqXzLrDRa+5/s4zwTVWwOsSEa0L842Q4nboVAUWg3hCQXs77zlMh71M16XxbB/DTSffi1AsnYW03zMRtZ9pX7Id0NcyHqblap5BLLaPRFIQv1UuZ9HLTuPak6TWeIqAI1BoCpifn7ahcRuTcZ7c7vQ/7u7uxE8B19z6Pz15wFZ7dsMvo5hmvQObFI/xae774+2kEwmcd+CN3l5ppF/+CjIfExgjnJO6k6TWeIqAI1BQC5Al2Y19FY9bR+/OrvE+S+m3Pr8AnzroKv3v6JaO3l5cCJY29DOGIo85kHRP+CVyWySuW8PlzjMa+ApLVkBB8WCZLrbEUAUWgxhDg1/iZlPkiluQtSynJDLS37klj7ITJOOfGeWbbhJT/ARb3yuGSS5qkl7BGOL74iwyH979bCT8CY6mSsIyIrt6KgCJQ2wgEhJ8xQz8SOMn+IIB1AM65+QF84+pZaOsGdh3keJ+j/hzSyBiyL4zw4x9TWCMcU/xFhsP7390UhC/wUia93DSuPWl6jacIKAI1hYCodKQ7+5oZnkt79q2PmdOrnnl7H3bmCuqbTqRBwyNUzXZdTOOaKh9QihZZZfI+iN5ghE9EBExKe5X6SEi0dNO49ugUGqIIKAI1jkCoK9N574ur8bHzbsDkJ4pPPzsAACAASURBVJZhaw7o9DmdY/s0UkgZwvf3zeHjKeHHrZjvn/BRo0c6rUtqUgnfAUWtikCTIFDo//aBrTvs644GOwA8vTGDz1w4A+fOmo8d/hezTGNZJAXkadLoQt68CEyYEn7/EHpcrkr4TdKX9TEVgVgEhNqLZbHL0jh18SR7LsH8+m/vxFev+iM25YG9wVp6MjpjcYKXRgk/josHJLyY8OW1G9UyJFyaQDie+MfJcDp1KwKKwGAjkIP8cUJWVuTRzy63lFU2XI3DSdq3Afx42kJ86dc34fnNaezlAh4TV3jCJ/xcGshlkUXehCd9TmGRpPEHLl4d6/CV8AeumWhJikAtI2AJVkhf6JbSjuq55oakvzsDtAO49vFVOPncSbj75bfNSJ80z8tuccx0pH9O1abNl7j0N5O2frw4IXcQF2/gw+uZ8EeNcPCSN7PjVWSV8KiqEP84WZSpOhQBRaAGEWAvtpd/ChXX1OeBXTngqfV5fPiCGRg/7xlzklWn+R0g0ckTlgO45ULB5P2XgeRbWQqLVI41GKFNQ/gEt1CZpVBLFcXJ0pTqowgoAjWCgIzr3NuhXwZI54F1aeCLF9+EH02+F9v8OLk8FT28uHuaS/KuXQl/QPT0lQoZVdUI36/TYJpe3CLjiF7CJb5KRUARqBUEpHdyFO8O2LOZrB3nAeZowrNufRYfOWcyVmztwqGOrH8Udh55dBulT0GF45I97dURfq3gUnofTTXC5+NL0whDIf5xMpxO3YqAIjDYCEivNZucBaN8+gIH08BeADc/vhwnnjUJf3zVH9vzoBN/fpeEn0Wno7NXwq802B6UsOpH+PL6tw2huJFKk4mTxanUpQgoArWBAHsuJ2btpgj+GnrA7Itz7xvt+K/zpuLX85/HJud2ORFrtzbmShyZoA2TfcHtJK1Ta9OM8OW1L4Qeri/xj5PhdOpWBBSBWkBACJ9rayz1203PWndm8OVrfo/Tpy/AWxn7AuDAnpche7P+Pg+z+LJoorZA9DJ56yerY9EUhC9kT5n0iiL+pOk1niKgCAwkAuyxHN2TxHOZFLpyMBOzZ0y7C6dedB1e2WfX4AvZy73Znm6XdBZUvo3a/xue8F2yV8KXRq5SEWh0BPiB1WW3P4xTL7oe97222nxda8ne5QSX7qNI3vWvd9SagvDNe99flpm0wtxKdu1J02s8RUARGHgEuM4G5vCSR5bvwGcumIrJj7ZiV7B4xyV72t2+ncQ+8E/UtyUq4UfgGVX5EdHVWxFQBAYUAVkqab+OLRR9AMBre4FvXn0rfjTpLqO3txO5fBVIvxbiF3dSWSinPm11Tfj+iVcB8qzE8EU/HeGHUVG3IlDvCJDouc7eJXxubbw+BZw2aR7++8rbsNXfO4cTsuC+OO4ifQNAUqKXeHWPGrzDvNZBWVYZKvRIb7jX6g2haWnzhnnjQuElzlGjlPDrvfnp/SsCPUUgPMLnsI6j+yvueRwfv2gm7nhtO/aR4nMZS/b+RmjuC6Iw4hdCj5M9vdtaSVdbI/wjfVY/0hD/cG9MCcs7Hkr4tdKI9D4UgYFBQOg4XBpJnGR/68sb8clf3Yipj79qPrYyk7Q+0XPbBG6oo4RfGyN8h8q9I70Wr9VLSPhsBPZSlY4goVIRaEQEhPA5mpd+T8kVOfyg6lMXTcHPb3zQ7IBpTq7KZ81+9natpiV7JfxaIvxhvJmWdm+IN9N9A5Szjxx5gmnTfItzJ4xyGjq7kUa1OnxpVuVkI3YjfSZFoD4QYI9kP+d2Z3b3G7sih/vbf+6i/8H3LrkOmzvsjglmWtZ04Xywn059PGV/3mVtqXSE16naWex53lHiEUjq9qnr97zW973vPwwyJHw2ABq7ztYFrCeTtuWIXvzcvNWuCCgCA4mAS/iH/OWX73C9/QMv4OQzfoUlazab2+k4xK0VYDZHq2Yf+4F8lsEpqzYJ3xvmeeO8IZUnbk86cYzBTBoB3/i0F19K+MV4qEsRqGcE7ClWHOWT8HlM4e1LVuIj516LiQ/9zYz++XSpbn8f/Hp+1H6599ohfI7qg0nb4Z7X6g31xgYj+zKW0aNPCiDhyJ5GCT+ARC2KQIMhQLJPmSMMqaQl4T+/rQNf/PUMjL/9EbONQke5abwGQ6F3j1M7hD9miOe1+qbN4wg/5hp1wsnBqioSPRtB6aUj/FJM1EcRqD8ESPeH8p2G8tnfNxwEvjPpTnzh6t9jiz9xWzrgq7/n7N87rh3Cj6H30mCX8M0GeWWRUsIvC4t6KgJ1hgAHdPwVz9U3+wGMv3EeTp0wF49vBdq5L7JeCRCoZ8IffaKdfedrnbxe9uecEn6CVqBRFIGaQIBdOWqULr/gOZq//rll+Nils3Hny5uxXQLSXJypV2UE6pjwR4waXSB8If2Sp1XCL4FEPRSBGkVACJ8cTiNu3i4nanly1ZPrtuMTE2bhovn/1znwhOs0qdXXqzICdUz4I0eOLIzszei+3NhACb9yA9BQRaB2EGAPlnX2VN108ODBDCdqrX/rbuDrE27Ez665GVu6in8NZMt1/9p5tBq5kzom/BEjRtiff+R0JfwaaVB6G4pAzxEQYpfvajpz3aZr78/B7Itz9uxHceo507G63XT4oCCmK/0GJwhWS4BAXRP+cc5PPlZ5uVe8jvCDulaLIlDjCOSQQRYZM8qnSqc7dciobdoBTH50KU49bxruf3W72UpBVPd8pGLClxFgOVnjAPT77SnhhyCWF0eUDEVXpyKgCPQRAiRoKnTSyCIfDN/25ICFq3bjI+fcgBseWAKSPz+yDF+F4V45ohe/cKpmczcF4UtlU8ZdUUQv/nHpNVwRUAR6hgD7Jwm/2xB+ZxrYnwHWHgS+eNksfH/qfPNlbYcfq/yqPCnZ7fOuXcKbVTY84bNiq6lwIfYo2awNRZ9bEehvBNhP7TaIVNdwkSVH8z+//nZ8+vxJ+Ms2u9+9UeVIl468JYkQlpEJmiRACT9U0VFEL/6h6OpUBBSB6hEgD5uL38+yb/HKAdkuIJ82E7A8ueryBX/FKRfNwiMrdhX09owuPO6nLBYSWE4Wx2w+lxJ+qM6F2KNkKLo6FQFFIBkCZflXCN/2t3Q3t06wX9LOe209xpx9A254cpVR5ZhCmEcuZ2dpaS+5yhXi+pUkaDKPBiR8l6oLten60h51heOF3VHp1F8RUASiEeDulSRq35g9i6VvcUGl3RSNGnyqcl7vBD5z8SycO/chrM9bvyBvSVvC45KffYmU+x/k0bQWJfxQ1RcajR1GhN2h6OpUBBSBBAj4/YhkLYRtVDnSv+xSzD0A1uWBL195O045fwZWHbLTuCyAMe0lafwtkM1I3/EzSqJydF/IQXJqPqmEH6rz4oZTSvqh6OpUBBSBWAQKvUqIWDYzL+xbz4+tuL/9FfNfxCfPmogn3zpgPrbqzOatyj5PZi/kVMkupYRl7I02fIRmIfySUUBUzcY1qKh06q8IKAJRCLBXcXUN9fNpMwIn4WeCydds3urtb1q8AaeMm43Zj76OXc6oPpvuBk0lknfDwkQv7qj7ax7/RiP8vFvtTjWKvi8YIThhxhpH9BIeTqduRUARSIIACV+2TOAXtST8g/u40TFwIA+8tgMYe/ntOPvmJ7HV7KNj+zLyfDFkgFwayDEX6YvRUgk+qkb6mvCPMAeXyMlVpZvY96HPiBFltlZQwo+qafVXBAYVAdIzR/h263pry2byZuT/Thb48cR5+MHkhVjjx6Eqh2kMwQvpJyB7xpe/QX3gmiy8rwl/uDfTa/HavBZvkedVPqKwt9xflvBD7/8A81iVTvRooXhEEeSoFkVAEagSAUvg7KQ8ZNxS/6Zu4JoH/oZPXDgTz23OGz0+Q+QFwZU7Aekr4VeJeDh6XxN+gcXHDPW8RUM8r22I5830PO+oQlDf2Koi/JKGEgZCCT+MiLoVgT5HIJh3tZO23PTy7iVt+Ng51+L6h1oN2XNX+/2dVP64vwjociZ6S/pzcf/VEX5UzfUv4c/kYeSG8FvMoeQk/j67khO+2xiigAj7u2lcezieuhUBRSApAuxJHV0kbntU4dO7gZPOmYgzJ95uji0kzYuWnjO60vPseL8c4SctWeNZBPqa8Id54zyS+zBvsTc0pNI5zOPh5H12JSN8aTIik1a8xA/LpOk1niKgCJRDgLTNj6vePAB89qrbMPa3d2Ez52N9NU7BJj6UTKWEXw7P6vz6g/Cj1Td9qtZJTvhuY0kKj9vYXHvS9BpPEVAEyiHAUfwOAGfd/BA+O34WntvIMX2BzisRvqhqRJbLX/0qIdDXhN9n4/f4jJTwK1WshikCtYnAbgBTnl2Jj104A/OXbDHr7TlJK+p9Jfz+rLemIPyejNDdNK69PytD81YEGguBVCqFHPfP4cbHWavKeX7DQXzmslvwm4eWglsppPJ2fb5V2Lh9rdQuI3uRsWjJ9zciYxM0eoSGJ3xWoNtwklaom8a1J02v8RQBRSCbseqafD5vdrp/Owd8/cpb8f3JD2KtP3HLrXVSyCOFrFk/X9xf3b4nNF+QsQgL0YuMTdDoEeqY8EeOPD6g8kIjKdjYVOzlNhrxi5NuGtcel07DFYHmRUB6iosAyZ7XAQDfn74QH75wLv6yCeDJVfZ1QJnz6V5yKC8LVG9tbjll7UL0IstGaiZPJfyI2i7f4CIiq7cioAg4v6Vl2JXO8khygGvr5z78N3zm0jm44829Rm9PwNjL+F8Jf6CaTx0T/qhRIxKO8KVpuSQeBtgNq2QPp1O3IqAICAK25/C/rIwD9gK4a+l+fOScqbh+3lNmZM9JWp5ZG+5pko/K/kJACd9HNtz0otz9VRGaryJQ3wiwx9iPpmizGnkS+6oM8MnLb8NP5zwGflkrV9b5sEp6m4Sp7C8E6pjwTxg9MhghFMYKBRsbUeGSJiWyEGJt4h8nw+nUrQgoAkSAPYf73lA9Q8Lnevu2HPDTWYvwqXMmYlUnsC9r4zF+Z4rbHes1sAg0C+GbkYVL5mGY3bBK9nA6dSsCigARYK8h4XNUT0lVzsX3vIAPnzcNj6/YjN0+2XNk39HdBUq9BhqBRiN83R55oFuQlqcIGARI+KK55+j+zyu24eMXzsasxWsN+cu6GleVo9ANNAJK+D7ilUb1bthAV5CWpwjUBwLsJZ3ZLDoBLNkNfP7S2fjJ9Pux0R/xy+YJ2psGsz5rhfDf5R3lHeEt9oZ7bd4wr8073BykUnF/hap0+KrSGcxWpmU3DAJC16UPxJDuXB7bAfz3NXfj1CvvNBuk8QWQ4YdV6Q7/YJLCPFtpLurTvwjUCuF73pHecG+Mz/BHekPMzpoVN1sTwpeVAfKTUZokZeFyfYtDbJxweJS7kKPaFIHmQ8DvFyH1O33thC1ww71P4KRfTsEdb+43WycQI/nAKtyrmg+/wX7i2iH84tE8T8wKb69cHMM7/vhjg4miLuSQMh9wFEYPbFz2cpuZ+MVJN41rj0un4YpAoyLAfuCfL0vCzwGdnfykitsi2Ena+1/djg99/3LMeuglQ/ZMwTQkfHkhaG8azPZRm4R/VItnRvilZ+Nyv/3hXqvnea1HHfWvAeHzxyING1Vxg3JdtvklgzucTtzJUmssRaDxEGAfILXT2H7Wmeo0ZM4dMJcdBE4dNwsXzH3UbKOQNS8F+4Ig4TOV/TVe6KONh1GtP1HtEf6Rh3te69AE5+GOPnGUQdc2phw4yi9tUGyksnaA9qSXEHxYJk2v8RSBRkOAfYHrbzqQRgop06/syVU8ePwb192F0yfeixWd1i9M+DrCr4X2UGOEP9TzFh/uxU/YUrvDrRV4sRmmzeheCb8WmpTeQ6MikEMub39Lk+4PIWPofyuACX9agk9cPheLt1uy50RtOp0FuBVmnrNrpaN7+uk10AjUEuEf5i3yhpgDz0Pa+vLO40cdH6DFJlVuc9XCj0c2r2qamMQPy6BItSgCTYZADuksdfZZs3ECP6zaBeCOv67Gf42fjqsefR0kf+6Cyb3vzT74PuETqHBPqqY3BkBTTeSaIEAtyRCoFcLnCp3hXrvRz1NHT3N46EzcEO8HhM8G4C/4CjcqC4LrmwyW8s2zR000aYEaTxGocQRySGU7cSDdgUPIG4XOI6vb8YXxU3DurY9itb8rpgzs7cNI3yt2FftW8dgu2Zt+X0VajWrelt5hZg40xKaD4yydpK1wHyNOGFF42/vjB2lIIkubWdJalxzCMml6jacI1B8Cpa1dfArP0pnOmFH82gPA2MtuxPcnLcAmam6oxqHiPufOpEm60nwkpCqphF8VXKWRa2WEX4HYo4KU8EurU30Ugd4gILQssviXbiFnHk141W1P4dRx0/DMO2ns4YwsyZhkn+ciiUIONlXYXcirKpsSflVwlUaud8LnE7ERVFTp2PBCI2TjC1/SIONkOJ26FYH6R6DQ6u3ni4UxuoRkDIUfygPtPMzkhR342LmzsPCVjdjSCWQYLSB54zCgSOoAISFs8RB3UinpVPYQgUYgfIfupYGJLKAiPiILIdYm/nEynE7dikD9I1Bo9QXCJ+lb4rd+XEdPsn9ucxYfPf9GXLGgFe902kPIzag+zxiMW7gk38BHiF08xJ1USjqVPUSgjgl/1OiRwUOzmZVfpSNRpOmJFH+R4h8nJb5KRaBxECht9XaJMzU1pHGuvufHVX/ZCXzu4jn48cS7sD5rz6llWuS6rPEJX/KLRSgp0Uu82Aw1QmUE6pjwuZeOVefYcUVFwmeDKfOTswCONNE4WUihNkWgURBwW719psKongsxSfZbAPzixkfxuYvn4rUdGRwMvrn1CT/PBZlW9SP5xeIjRJ5UxmaoESoj0EyEX0T6YVikicbJcDp1KwINhEBAvLYfcHTP9fY7AEx6/BV8avxs/HlNJ7YcpLLHqnyMheqcPH8HFKt0SpAJ8g+tp0/qX5KhelSHgBK+j1cc0Ut4dfBqbEWgrhCQZp6z25Hsz2XMx1UPrm3HZyfMwBXzXsBO54Ekuv31zDTW5kQptiYl9qh4xbmpq2oEmonwVaVTdfPQBE2GgDC4T/gcs7/cvg9jr7sJP7npQbwDGFWOnEfL6IUVPQWlaSRqYSKPjKgB/YNAsxB+EdmzmYavoKWHdP1h/3A6dSsCDYIAyZjN3Syjt5O2VOWcMf0P+OiFE/HiQbuVgh3H22dmdFnN4xJ/JCJK+JHQDExAUxC+S9pJYXXTuPak6TWeIlBnCBgytpudcV39rhww4c4X8F/nzcDCN3ca1U53nT2S3m4YgYYnfJesaU96hdOJO2l6jacI1BsCbONkfau2efjNnfj0BbMx7cm3DNlztY47uq+3p9P7JQJNQfhspuZ3ahV1LgQfllVkoVEVgTpDIJ/NgMS+Mg98atxcXD73IWzLALu6gQ5uk9PHzxPuXXHuPi6+CbNTwo+o9KimFxFdvRWBOkeALZ6TtNTbnz71YZx66W3YkEowEduL547qZVH+vShKkxoEmoLw3eaTtN7dNK49aXqNpwjUFwJcc88llzcs/BtOOX8Wnt4Ccy4tXwL5vOyCWXgBsFf09nJ7VhJ7b8vT9A1P+KxityklrXI3jWtPml7jKQL1gQDJPJvJYl8auPXpNnz6nOmYtug1Q/77/VE/dz3O5/kte896UxQS4fzi3FH5qH9SBJTwI5CKanoR0dVbEahzBNZsP4hPnTsd596y2OjxD+TtcYUc4dPwF4C79JI9pLdXVC+L8u9teZpeCT+iDWiTiwBGvesaAWnXdqTORzmYAda1d+IXV87Ed3/zByztAEj2vDL5wuZpJH0lfItL/f5vCsJn9UhDFxmuMvGPk+F06lYE6gUBtm27Yo073nAnTG6AxknaKxcswSd/eQOeX70L+zKW2PlUVOW4hjnoVc8IKOH7tRdH9BJez5Wt997cCJDmuZslTc6oaXhy1eTHluMzl96GOYvXYw+H8P4l54+7UsJU1isCSvh+zQmhx8l6rWi972ZHgC2b0652G/Gc0dM/uQ341GW34fxbHsGWnFXfuDi5ZE+7XvWOQLMQvtFJumQerjg3rJI9nE7dikB9IMBWzQE8J1+pj98F4HMTbsU3Jt6Ht7qBvVTfhB5FCT8ESN07m4nwi0g/XHOVSN4NC6dTtyJQHwiwFWeQxwEAmwFceMvD+MqvbsLf2oGtGd02oT5qsbd3qYTvI+iSeiV7bwHX9IrA4CDAVt2ZzYNr6yc/uhSnjJuJu//2tnkBcPKW4Xo1OgLNRPhFK3XCFVuJ5N2wcDp1KwL1gQDVNdwn5+XNXfjYeZMx/s7nsTELcAdM3RStPuqw93fZLIRfRPblxjIuqVey9x5yzUERGDAEclYrzxbNE2epyjntNzfhp9PvR1vGjur9JfdyPPSA3ZoWNBgINAXhuwSeFGQ3jWtPml7jKQKDgYDfVmW+KsspWvtxFXX3Z87+Mz4zbhZe3JIy6/AN5ZuzaPliYKI4MxjPpGX2HQINT/guWdOe9AqnE3fS9BpPERhsBHJArhs8zIR6+5mPLsOHfjkZD67pwg7ujcPbI9nnO8y6fPtbQAl/sGutf8tvCsLX/fD7txFp7rWAAIck/HqWxpK3vSv637/8ED501ixcfd/fDPlzkjaTZ7/gL4BMsFSTcW1q5lBu1C8DH5G18OR6D8kRUMKPwEoadFhGRFdvRWCQEWBLJdmTwl3iX7WHh5ncjJ/OfRobAewL7rOwzYJskMY8lPADgBrQ0hSE75J20jp007j2pOk1niIwOAhw75uuHLDXn6Q9a9Yj+PhZk7Fkl1XtkOa51bFdm8NtFgovCtqtskdH+INTe/1dasMTPgHsCWG7aVx7f1eI5q8I9AaBAmVzn5xL7nwRnzx/Lha+3m4OM+Hon2SfR9oYS/rcWUfW4YsOXwm/N7VQu2lrhfD/tzfT+3uv3ftfXpuX8Dph9Eh/5omNVfYIKUfv0pxtZ0hWGW4a154stcZqVgSkrQzG87Nse0gJV+Q8/OZ2/PclN+OGh940ZN+RsztfGsL3Sd8SPl8DYlxlkOrwB6MW+7fMWiH8I7wxHs3fK+H3b4Vr7v2LQIjwOWDOpYDMbiCzDchuAzI7emayOwBj/HyyWwCaHI3vx82Ou7dhx85NOPuSy/Gr66ejk7yd3g+ktwOZ7UB6K5DZ6sstQGYTkNloTXYzkKPZ6pvtQM4xvP84E/V8cekkXNNHt4802047kD0IZDjzwvZWzVUrhM9R/bu8o7zDvdaEA3zvmGOOTjjCJyDSEUWGQRL/OBlOp25FwEXAaT9m3SP3M2gHNj8OrL8T2LoA2HZ/9WbrA8C2B4Dt9wPbmcc9wLa7gD3zgI23Ahv/AGy/z5axYT7mXvBJzBn3eWDjE8DGh4HNDwJb7gc2LwC23mfL3878HgR2LLTG5H1/6B5ZlmMkTkXJfMsZPreTV6Q9Cp8F/vMTg0qmXNn0q/Pyt/L+/wRsfgTY8QrQyZdyd+G8Yf8jO7c1ltprifA97yjvsOQj/BEjjlPCL61R9RlUBEj4nBbNAdxq8lAnsGMNVk7/JjZOPwVvTP4sXp0ytgfmy3h1ypfx8hTKL+L1KZ/B65NOwcuTPo7nr/8Ynp76Obww++v4v5O/ir/N+C6enHE6/nbrmXhj5vexbNp30DrjG3hlxlfROvUraJ38Zbw66Ut45fqxRtIubvq9PPGLBXP95/GyY1654fOobL6IV26INi9f/1lUNsXluWXTXrlshkeXzbDKZfPeaqx8py5emfh5LL3u83jxf76Kv/7+10CKv7zqnfCHxYzw/5c3zjvCxGn9t397rxL+oJKbFl6KgEP4uRyQTgF73sLrk74MPHUZsGI+sOqxHpt82xN+2kXAqgeQa3sA3Wsfwe61T2HDymexaOFtuHz8mXj8gRux+63ncGDVszi08ml0tD2BrrbHkF75GHLLH0N+5ePGYBXzs0b8GF5kVjyCnG/yKx9FfuXDEYZhlY3N58/IrShnCuVIeWHZVOWH6gHLH0H2tfkG4x1LnwWyh4Bc3pxIBo7u63KE35J8hD9q1IjkhG9+Xjs/t0t6qhtWyV6SUD0UAQcBaTsc5GeAVAfyy5/FK9d/2apKDq4EOrf33HTs9NO+A3S9A7v4sgP7uvfgjU2b8JWzL8EVc27H/q59yKY5b7AfyOwFsrS3A+k91tDP+O8Dsr4RP4kTyF1A2jeci8hSh1zGmHkKllPBmHx2AulyxilHygvLpirfryuph9QuILUVyPMkgy7kMqng+EkSvyV8tr9KV42pdFq8mBG+o+AffeKo6gi/iPTDoEhHjZPhdOpWBAoIuK3HTNZ27kHqlYewYs7pAPXw2U3+dgbc0qA3ht/K0tjPpLgp2g+unYlv/WamOcyE87Q2LI0s98PMHwLyXLvDiT5ZkUMpq3JEumG0yydZIjs5KVHBcO/NSoZbuFUyUk6UrFQ2wyqVzbBKZTMsqlzxH+jynfoI2kvK8J45a7ho2rao9fktICxqhfAP8xZ5Q712b4jX7g3z2rzDvXEOt5e1KuGHK1Pdg41AUZfj6pwDW5F/9UG0TvkWsJ8/w9t7d4sctPgDF5I6KZ85nnPLAnz4l1fi0bW7zdYJ+1J2bX0X8khxi4R8t9H3cm6h6B5Dd+OGFexMY/9s4Sy5nJGbi5Y2FzlXNyyllGjZXOWXLjUp1ElpWKEqGSvqqhXCL0vplT2rJvweNPVKsEZBqv7Ni4BQlRlh57qAA5tw6Nk/4PUZpwPdL1s1SqVeWynM5ViuqfcVOg++thGnXjQTt7z4tllv34kc9nVm0JHn/vd5dJmPrEj4HMX7Lwzh5HBViX+UDMfva3dUueLf1+WF85NyomQ4fl+7o8oVf5ZHu1kUYAsPN5nKt1THhJ/8wysfICX8ym1BQ/sAAUv55vOnfAeQ2oQNCydi1e1nAZmlQOZg6Rgi3GPLuYXs/QVAnA+mAuLlTRl89eJZOHv2w9gOYE+qG9kcQ4BubrFgRvj+ypwfCAAAFzBJREFUV7Wc1CNZuPkLkRgSCb0M3DCx9wFCFbNgOe79uXa5x4oZ9DKwlssnFlIP5kR5Acc+s7goo686Jvzkk7bJoCgGyU3j2otjqUsRKCDAdlLYkAz5/UB+HZbNvwyr7rsYSC01E6TuweBMK62LnC6adEq6zcUEVA/lUsinuk18apLbDgFf+vUdOO2GB7C626a1aez+OMw3a3KRErhU1CmQ3tVe4fRO1gUyqjbTKuIHhCcj3SrS9kXUwS6/6BnkZoo8Yxx1TPgjRx7v/7xhp4naWsFtkdW08HA6ccfgqcFNjADbiEP43L4sswxtD1yO1+8dD2ANkNsH7mQjRlqVkL0zRecTPmNwks6f4EXelLADwFk3PYpTLr4Vf3orbfT2BJ6xC68Qyd2R7tvG2LkxvhBHAhlOz58aNOIfvL6cMov8EpRR9OYIx4/KN6l/OL9q3UnLiYpXbXnh+OHuJeFh/yh3UxC+7YTSHaKgKPaPqrDiWOpSBAoIsM04hJ/fDXS8gmV3n4+Vf/o1gA2xhF86wpc8+Spg3sDOLDDp6XX48Ljf4ea/rDOqnEM85cQnVrEV7su3CTeIZMRqL0lLKQWJlDA/T/F2ZbXFlcRnGaHLzT/OHkpavXOwy6/+jkMplPBDgIgzqulIuEpFIIwA20yI8LuW4KU5p2PT4qmW8PMHgtE9R/luK+Mo3zUSJszKsAN54Lk1u/CZS36HC+540azQsa8B3ou1STrKkktIWWRJhBgPSSdSChO3n1y8wzIm96qDw/nHuasuICZBXHnh8JjsBiC4KQjfhT0ppm4a1540vcZrTgRsW7GTtu1A6iW8MOWr2NP6eyC7Nlg7Xw4bt5WJPZVJBy+FHd3A8g7gSxfPxBkTb8fmtF01zhcB4+fzxS8Q+lW8SNK9uYTkRfp5yb2Xk70pTtKWyzepn+TRG5m0rHLxelNu36RteMInTC70SWFz07j2pOk1XrMiwNZCEkauHdj/NF6c+iXsf/MuIM/zpuwKmlJspI3ZXwjSZlM+KXOSlgeanDF1Ab5y6Y1YvTtnPhESspfUYVlaTh/7VEH2vLfeXuHnq9Zd7+X39v456ekdlvzj1soL4wc4NNmkLSFym0VSyNw0rj1peo3XjAhIS7GEvwPYuhCvTPtvdLUtALDV/5KzHDJMSbKXaVtL5SR8+pJXf//gM/jvc/8HD7+503xPSj/Gco2UL7JcSX3qp4RfxC6Ce5TsU+x7lJkSfgRstVtlETes3jWAgLQaQ/jcu/7tO/Da9C8ite5hANwDhYRe7pKUpHdL4fThtSsHPPPmenzpF5di6vynsNX/fophLtnTLrmItDn0438l/BLMBftysh9rImHWTUH4xCIMfxifcHiUO5xO3YpAAQG2GiFhc1DIitlYOvWLwK5Xwe9eZWK1kEJs0t4K7kPdKbN1wvKDwFfGT8eZN/zRLL/kK6McuUsOrpTcqpWSR7XpJL6kj5MSv6cyLv+48J6WK+ni8o8Ll3wGTirh+1jHVY2ED1zVaEn1hwBbSYHwtwCtk7B82heA/SuQNZ9VMUa5y7YvppVBM/fJWdkB/GzqvfjGr27Eq9uzRm8vLbFcTm5YufByJZfzk3zKhSXxk/RxMkleleLE5R8XXinvJGFx+ceFJymjb+Mo4ft4xlWNhPct/JpbYyHAViKjb2S2AC9cjZUzxwJdbyFntPGMUeYiy5tfoRl051Pm1bApD1y9aClOGTcTT6/YYvaolLyjWqP4iywtSUIooy+JFR2jcoikj5OVc4kPjcs/Ljy+hMox4vKPC6+ce3+ENgvhBx1KqiAMpvjHyXA6dSsCBQTYeoSUDeE/PQGrZn0JyG5wQwoJxOa3zzzSOJBJYzeAhSv24OMXzMK0x5cZZVB3uqsqxaRkXSzD7bs4VFwSS9zVSkkfJ6vNNxw/Lv+48HB+1brj8o8Lr7a83sdvJsIvIv0wdHFVI+HhdOpWBAoIsJW4hJ9+5CI7wse2QgjbYTljYmSxrTuHv+4FPjfhVpw5fT64iz3naVPpjmCT4sStMVxO7Cuj8Cxqa0QElPD9WpUuFCcbsRHoM/UVAm7r4Qi/c9F5WHfzNwBsiSV8puXFXS9Pm7QAX//tnVifhiH8jnQ3urpI+MXLD/wk0UIJPxqbpgxpJsIv6i7h2g53pSh3OJ26FYEIBNJbsG/+z7Htju8D2FyYtA2TsO/m6huqcq685zmccs7/4C8bD2E/w/xXRVe28NWttM6Ikgve4bKK+gBz0au5EGgWwo9t6NKF4mRzNQ992p4hwG0O0L0Bu+7+kSF95DcZdYzJTUjYz5oqIJI9v6S95YWNOOWCaZj95yV2ZN/BL3PzSGd4PF/x6D4RXUtZImP7gX9TKhoUgaYgfJfEk9ajm8a1J02v8ZoPAbIq2woPGuFGN+uwYe43kHnoPCC3yVfcF/T3jMnPrPZxB0wAz23NmxU5Z836M3byTGpml04DqUNAttMeU+i/NqRFxmIsRC9SCT8WssaO0PCEL11DZNLqlPhhmTS9xms+BMiqpHAeJ9gBdKzBuhlfBp64OCB8tqZgwpZHgWez2JQCXk8BX594D75z/T1YnbJHcefNmdwk/E4l/OZrTP30xE1B+OyENKa7JQQyTPTiTphcozUhAjKMTgHZg0BHG1ZM+hzwl6uBzGaDR7CCx5xg1WXaJPX2Z/7hGfyfi+bi2S1ZbMsAXVyWwyZrNtPJmp0wM6HtlBO1ZrklkTrCb8J26T6yEr6LhmMXgg9LJ4paFYEiBIRVu4DcAeDQSrxx7WeAJZOANFU6/pJNM/bgKVYp7E3nsPCVNfjI+VNx6xt7zP72fA0cPJCx4xOf+JmE/vLCkFZZVHw5h9ySSCX8cig1kZ8SfkRlS5cKy4jo6q0IUEWT6kQ+R8Lfh/y6Z7B2zjeB5TcCaW6NzLYEZMneJPUM8PzmNP7rrOtw1qx5ZqU+Sd1cJGhG9xmegtxP6ZK+RI+UQvQilfAjoWqOACX8iHoOE724I6KrtyJALUw+YydXs3uQWfMYVk7/OvDW7T7hW6bP5uyu+Os6gK9c8Xt87/q7sT4H7OFErUGR58z6hO83O2l9YRkLuhC9SCX8WMgaO0IdE/6oUSNsxzB9o9Ih5uxoNLY7JavQcNcSd7LUGqv5EGALIa+asXhqBzqXLcLrk8YCWxYAqQ1GhcMmeDAP8BDyC29/AV/71U14fsN+cyzKwWyqtIUGzY7t1+r8WULgHQezEL3IIGXiHOJK0PC6QkAJP6K6pEOEZUR09W56BNhS7MWVNTtw4LWFePHaLwK7HzFr8s1STf/jqhuefBsfG38z7luyEdu7ga581v8wy74yJKfCih6u1C8QPsML5QWxSy1C9CKV8EsxaiofJfyI6g4Tvbgjoqu3IsAd7zu5JDMNZHfjnWdvw4uTvg7gFbNhQgpZ7AHweFs7Pjp+Ji6/f4nZ796SOjXz/LjKEn4pmQeMrTgrAr1AQAk/Ajwh+LCMiK7eigBXUWayQJaEvwvrnrgFL838AZB62Rxezpa0pgv49lWz8P0pt2NFyioa7aDb/1jLIfxS0leIFYHeIlDHhH/C6JEyIDLfH2ZDXyHaDsP/qsPvbTPR9NUgkAK63sGqh6Zh6V0XA52vGwXMhhzww8m340uXTcGb2/fgIL+ilUsH8IKEyn5FQAk/Al6+KMqZiOjqrQgECKSA1HqsvP9aLJ//ayDVZg4vmf7ESzhl3HV4fG27jZkNFmEGKdWiCPQvAk1B+C5xJ4XTTePak6bXeE2LQL4bSK/Dirsvxxu3nQdgKx5e/jY+euENuH7h0+Yoc2Iju18anMwI3+rvmxY3ffABQKDhCZ8Y9oSw3TSufQDqRIuobwTSB4Cu5Xjj1vOxZfHvkO3ehrGXTcT3ZszDhqxda8NtcqjSobLRXNwpLc+JWyV9gURlfyBQS4R/uDfWG+K1tbR4bd4Qb5wXcyXT4RO0nhC2m8a190claJ4NgYBwdec2YM8LeGHGz7Hy9edx9m9vwOfH/xZ/22+/lu1GCh1dKRzqtPvbG5In4ZPv9VIE+hWB2iL8Ns/zjvI870iPpG/tkbRfPeFzPJUG8tSddgD5DiB3EMjtB3J7gayY/Y5d/FzJ8Lg45eJLujjppk1ij8svKjxJ3kniROUf558k7yRx4sqJCk+SdxVxMtwGbTewfymwdSGenfhtTJ8xFV8443I88sbb2J7n6VXdyJlNEuwWCxnZLIEvC44r9FIE+hWBWiH84d4Y713e4oDdh3njPJoK17HHHRP8As4h6ktbQY+9iUTPCbPNQP4dILcWyLYBmeVA+g0g/RrQ/SbQvdwxdDumaynQtcw3tCcxEr+M7FwB0AR5Mk6SPN04ZfItyi8q3M2jN/ao/OP8e1OmmzaunKhwN48+sHe2AgdfAPY9DLw1DSvnfg83TBiPeQ8+h0wOyAcqG/tTgC3SDurlp4G0VZWKQH8hUCuET3XO4d7MgN9J9q47CChYkm2tIMCxe3UAqXcwb/ZFeOiWS7Bo1ll4dObP8OT0H+DZ6afh+amn4fkpP8DzU34UaRZP/iEK5vtYPDmJcdP8EIsn/bi8CfJOkqcbp9r8Jb6bR2/skp8vY59P4vemTDet5DdY5dt7eX7yd9E6/TtY8ttTsXzSp/DyjNNwx7SJyPJ7qlQK4JbIIR09WyWNXorAwCBQK4Q/1BvrHeYtCuic7nKEf4Q3zjvMa/U8r/V97zsq6D+JRvhU5XRvxbxZl+Ohm3+NB2deiIdnnIPHpp+JJ6f9wpqpZ+LJqWfiiWlnlTWPT/0lCuYMPD41iXHT/BKPTzm7vAnyTpKnG6fa/CW+m0dv7JKfL2OfT+L3pkw3reQ3WOXbe3liyhl4YuLP8deZ5+HZyT/Gg5N/ifb1baYfc+MEs7HawPRqLUURiECgVgifKp1iwp8Zp9IZfeKo5ITPX815jrKor98D5HYC2e32YIr0BsA1mXesPw+tMIZu33Cb27DJrAfKmXA87oleZLYAaddIuF9GuTxdv9j8Jb+wDD2Dm2c19mYvP4xVaiPQtRXAfmDvOqCb7esgUmaID3Tm0jovG0FD6j1QCNQK4XNoP9Rrl4naIZ4ZxXMCN/IaOfL45IRPPI2qlD+g+YUjDX9r86zQQwVjPpGhP43EETcl5wFcw0OmKxk3Lu38We8Ys50uJ5PFT+JXytMNk/giJZ8oKfFEunn1xC75iIwqV/wlnsielOmmkXxESjlRUuKJdPPqpZ2LANJUG3YBGR5gwq2SO5FHNzg5m0JepmiDdWMD1c21HEXAIlBbhD/WG+a1+cRfccKWb4FEhF9mPixnjorLg7JwpIT5CWAwEb1qOclJNjWKQXEb4LobKhU5M5srrAI229p3G8LPIh9s0O22K6UhRWBgEaglwo8cy5cPSDRpGxA+u5m/LNMsjePyOFkix5E8u3CB9KMqwe2stosn+V/ggOL0Uf5J8nTjROUT5+/m0Rt7XDlR4b0p000blX+cv5tHb+wcv1vDXMzlVzR19/zjeyBoYs6XIVHtTP0Vgf5BoI4Jf8QJIwJMuHGabJ4W8LbwtyF99rgC4VuVjahthPDLkb6bCe1+Tw4k86xkwvGDW46wSPxKebphEt+X4dsNu4P7lnRuXj2xSz4R5Zc8ZSh+ReyS3E84v5ICQx7h+EnKiI+TQwbWMP9CM8nn8+YA8uBnIYNZJ725wnWq7sJYbaCxKFuPoTZW7T2F+2i16aPim3utY8I/9sQRQT/iR+r2Q3X+rPY7leDO4ZUY6lXLGieOGY5J4pC/5NMTGa7IKHelvHNZYDBNxXujOsMxjOurOtzxc+lL08E6ChPx7+2zSz79JQUfeaSwm/iIn8FHGmtSGUrv5tUM9t7Wf7Xps/wCOs+zK60RXV5Qv5bxpX3btu0PEGR+rlK9BPm49cq5H9e4YSG7pE8i2cSQhzfczI+WV5vUsu+xJ45yCJ96VAJNQBztTBwQ4QqU+G4etWTnJ/iDaaKwENxcybj+YDcMsw2p/r+MmHsqqy+xQoowFu6zR9nDnb9C9uWDojL2/cP3FHbHvejC8cPuwU4fS9hx/SNuwBRO76viwj/y2KAJuS8KtSI2SSBuX4bxDLsDfHuYPpxf2F3PhD9q9IkBuYcJRWAu6TRhACSi+PsJxLs/ZMk9VVGmDDgGS0bhETyT4EgZInupI+krUXlV8u/tc1fKuydhwXOLxX3+JHZJpzIRAuZ96avJSl76ocFv+N0auHuQPhhZuo3EP3Tebddu2459oHD7iE0QihBOH+c2yet4hG8InxXg7zsVBbzUUQiu4GUR/CLwI0j8cH7VuiWfsCy5jyrKlff+YMkoDOQZyz2bhLkyKp84/94+d1z+1YS7z0P7QFzhMpvNbV74Znkrl7iGjK99iR0UhNOJu0J6eVkEXOGTa73hT9KrW5WO55l1++arW355Wy9miOe1DR82vHVYS4sxw4cMaaWhu16eIcl9tvh1MsxKbobXd8/X4rV6PTF9eQ/J8jLPze9KbH1TWuN/a1IdJnHPHHdPA5C+paWlbfjw4dU9V9x913T4ED6rqeeWIV5rRdPS0tpSZIa1trTQ+P49Ti/5VJI+vxxueLOWtfUNd29sIM12NeMzs46b8bn1mZutd+vzVkCgXmfJKzxSbNCQpiQ+zzuiCZ/775rwmZuxT8d2eo1gEWjGDtGMxMfa/t9NSH5/34TP3Ix9Wvk8IQKHV97nP2Eu9RWNu5w249WMdR1zjkVDNoNmfOaGrEh9KEVAEVAEFAFFQBFQBBQBRUARUASSIjDOG2KWSB2ZNEFdxxvuzfSGe23eMLOkkofdNO5z8wAfu1SvzRvSNOqso4Z65phS1nGb13zqrFZvqHOIU113Vr35vkbgKM9712LPG8ozABqX+FzULAlaH55mdoRzpKUbrxHsQ8y6bJ7ncKR5wXlexbMdGuGR/XY8xn+WI83L/V1N8dyeeam3eIuU8BukJff9Yxy2yPPeddQQ713NQ/guiDzDuHEJf4zntSwOHnfIEE5YN9+ktSVA/tJp9Osob6j5ZTNGCb/Rq7pnzzd2qDfcHNje4h3GL/SaYfRXjBR/+jbuyoaxQ7whpn7tQxvCd9zFUDSoi226OQYzfLF5Hn/ZjPGGGeJv0CrVx4pG4AhvsXe41+od7rV5h3ltZv8K+zN/rP9JvVHjtDSaSudd3mLzrHxuGn6Q0uK1DfWMTlvwGufZTiLuRpNjh3lFulzWeTMR/pH+thqNP7qnmnJIULc6wm+0ntwHzzOmxfPah3gtbTQtXgtHQc0xErLgjfO8YVR3NPK8xZihxYQ/bkgTqXRabP02hwqLCxFaOF8ztJ19eRj7cmMPZvqAAps5i5bmIXuO8lvsypVGJnu25iMtCRhVndhlMrORW/uRQ7yhM4d4hzfTrxm3PsccVvyid8PUrgh43GulaUb3LRwJUa3FpZmHea0NPGnrefypb0Z+5pmbY7TreWOoomzxDm/zvGF2N1N3ZVbjd3iqsKjP10sRUAQUgaZAoNF/vTVFJepDKgKKgCKgCCgCioAioAgoAoqAIqAIKAKKgCKgCCgCioAioAgoAoqAIqAIKAKKgCKgCCgCioAioAgoAoqAIqAIKAKKgCKgCCgCioAiMCAIHGE20OJmeLyONB+ZNePGeD4AKhQBRUARaGwEhngzh3reTLNZWvMcfNLYdapPpwgoAopAFALDPa/VP/lJvzqNAkn9FQFFQBFoAASOOtzz2mgafEfQBqgqfQRFQBFQBHqDQIu3iLuCDuN2yIV90nuTo6ZVBBQBRUARqDkE/s4b6x9xZ1U5Q7xWb7iZyK25W9UbUgQUAUVAEVAEFAFFQBFQBBQBRUARUAQUAUVAEVAEFAFFQBFQBBQBRUARUAQUAUVAEVAEFAFFQBFQBBQBRUARUAQUAUVAEVAEFAFFQBFQBBQBRUARUAQUAUVAEVAEFIEeIPD/A0GKYNc9Xh75AAAAAElFTkSuQmCC" } }, "cell_type": "markdown", "metadata": {}, "source": [ "# ReLU\n", "In this tutorial, we will first focus on implementing the ReLU layer and towards the end, for comparison purposes, we will define alternative activation functions.\n", "\n", "ReLU is a piece-wise linear, vector valued function that adds non-linearity to our model. The effects that this simple piece-wise function has had on the DL sphere have been astonishing. \n", "\n", "The ReLU's forward and backward pass can both be seen as \"gates\" that either inhibit or advance the flow of either operations. \n", "\n", "During the forward pass, ReLU either retains the original content of the input if its greater than zero or else, turns it to zero.\n", "\n", "```python\n", "[x if x > 0 else 0 for x in input]\n", "```\n", "\n", "For the inputs that were \"cut\" to zero, its gradients are turned to zero while the rest of the values become 1. Hence, and given that ReLU is an intermediate operation, ReLU either restricts values of the incoming gradients or lets them \"flow\". This process is graphed below:\n", "\n", "![image.png](attachment:image.png)\n", "\n", "Such simple conditions make ReLU a \"lightweight\" operation as it does not take much to compute its forward and backward method\n", "\n", "Such properties, and its surprising effectiveness to model non-linearity, have made ReLU a very popular choice of option for most DL architectures.\n", "\n", "Let us model this process in PyTorch" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[ 1.6169, -0.8602],\n", " [ 0.2214, -0.4084]], device='cuda:0')" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import torch\n", "import torch.nn as nn\n", "torch.randn((2,2)).cuda()" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# custom ReLU function \n", "# Remember that:\n", "# input.shape == out.shape == incoming_gradient.shape\n", "\n", "class ReLU_layer(torch.autograd.Function):\n", " \n", " @staticmethod\n", " def forward(self, input):\n", " # save input for backward() pass \n", " self.save_for_backward(input) # wraps in a tuple structure\n", " activated_input = torch.clamp(input, min = 0)\n", " return activated_input\n", "\n", " @staticmethod\n", " def backward(self, incoming_grad):\n", " \"\"\"\n", " In the backward pass we receive a Tensor containing the \n", " gradient of the loss with respect to our f(x) output, \n", " and we now need to compute the gradient of the loss\n", " wrt the input.\n", " \"\"\"\n", " # keep in mind that the gradient of ReLU is binary = {0,1}\n", " # hence, we will either keep the element of the output_grad_wrt_loss\n", " # or turn it to zero\n", " input, = self.saved_tensors\n", " output_grad = incoming_grad.clone()\n", " output_grad[input < 0] = 0\n", " return output_grad " ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# Wrap ReLU_layer function in nn.module\n", "class ReLU(nn.Module):\n", " def __init__(self):\n", " super().__init__()\n", "\n", " \n", " def forward(self, input):\n", " output = ReLU_layer.apply(input)\n", " return output\n", " " ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[0.2316]], grad_fn=)" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# test function with linear + relu layer\n", "dummy_input= torch.ones((1,2)) # input \n", "\n", "# forward pass\n", "linear = nn.Linear(2,3)\n", "relu = ReLU()\n", "linear2 = nn.Linear(3,1)\n", "\n", "output1 = linear(dummy_input)\n", "output2 = relu(output1)\n", "output3 = linear2(output2)\n", "output3" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# backward pass\n", "output3.backward()" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[0.1558, 0.1558],\n", " [0.0000, 0.0000],\n", " [0.0000, 0.0000]])" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# check computed gradients of 1st linear layaer\n", "list(linear.parameters())[0].grad" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# MNIST" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have validated our operation, let's us apply ReLU to the MNIST dataset by building a standard neural network with the following linear parameters:\n", "\n", "```[128, 64, 10]```" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "class NeuralNet(nn.Module):\n", " def __init__(self, num_units = 128, activation = ReLU()):\n", " super().__init__()\n", " \n", " # fully-connected layers\n", " self.fc1 = nn.Linear(784,num_units)\n", " self.fc2 = nn.Linear(num_units , num_units//2)\n", " self.fc3 = nn.Linear(num_units // 2, 10)\n", " \n", " # init activation\n", " self.activation = activation\n", " \n", " def forward(self,x):\n", " \n", " # 1st layer\n", " output = self.activation(self.fc1(x))\n", " \n", " # 2nd layer\n", " output = self.activation(self.fc2(output))\n", " \n", " # 3rd layer\n", " output = self.fc3(output)\n", " \n", " # output.shape = (B, 10)\n", " return output\n", " " ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "NeuralNet(\n", " (fc1): Linear(in_features=784, out_features=128, bias=True)\n", " (fc2): Linear(in_features=128, out_features=64, bias=True)\n", " (fc3): Linear(in_features=64, out_features=10, bias=True)\n", " (activation): ReLU()\n", ")" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# instantiate model and feed it to GPU\n", "device = torch.device('cuda')\n", "model = NeuralNet().to(device)\n", "model" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# define optimizer\n", "from torch import optim\n", "optimizer = optim.SGD(model.parameters(), lr = .01)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "# define criterion\n", "criterion = nn.CrossEntropyLoss()" ] }, { "cell_type": "code", "execution_count": 201, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "torch.Size([60000, 28, 28])" ] }, "execution_count": 201, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# import training MNIST dataset\n", "import torchvision\n", "from torchvision import transforms\n", "import numpy as np\n", "from torch.utils.data import DataLoader\n", "from torchvision.utils import make_grid \n", "import matplotlib.pyplot as plt\n", "plt.style.use('ggplot')\n", "\n", "root = r'C:\\Users\\erick\\PycharmProjects\\untitled\\3D_2D_GAN\\MNIST_experimentation'\n", "train_mnist = torchvision.datasets.MNIST(root = root, \n", " train = True, \n", " transform = transforms.ToTensor(),\n", " download = False, \n", " )\n", "\n", "train_mnist.data.shape" ] }, { "cell_type": "code", "execution_count": 202, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "torch.Size([10000, 28, 28])" ] }, "execution_count": 202, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# import evaluation MNIST dataset\n", "\n", "eval_mnist = torchvision.datasets.MNIST(root = root, \n", " train = False,\n", " transform = transforms.ToTensor(),\n", " download = False, \n", " )\n", "eval_mnist.data.shape" ] }, { "cell_type": "code", "execution_count": 203, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 203, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcEAAAHSCAYAAACU+XDdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOydebxM9f/Hn3ZFSOQia/ZslbWyVJY2pYgWikKhVeSrXaEFpaQSqciWJSplrSQkRLskSwuVFqRknd8f83udM3fu3HHvNTPn3Dvv5+PhMe7MmZn3bOfzeb3XXIFAIIBhGIZhJCG5vTbAMAzDMLzCFkHDMAwjabFF0DAMw0habBE0DMMwkhZbBA3DMIykxRZBwzAMI2mxRdBIWjZs2ECuXLlYs2ZNpu6XkpLCiBEj4mSVYRiJJJfVCRp+JVeuXFFvr1ChAlu3bs3y4x8+fJidO3dSokQJ8ubNm+H77dy5k0KFCnH88cdn+bkzSkpKCv3796d///5xfy7DSEYy/ss3jASzY8cO5/+ffPIJl112GZ988gnlypUDIE+ePBHvd+DAAfLnz3/Ux8+TJw8pKSmZtqtkyZKZvo9hGP7E3KGGb0lJSXH+FS9eHAguQLpOi1FKSgqDBw+mV69eFC9enPPPPx+AESNGULduXQoVKkSZMmXo0qULv/32m/P44e5Q/T179mwuvPBCjj/+eKpUqcL06dPT2BXqDk1JSWHo0KH07duXYsWKkZKSwqBBgzhy5IhzzD///MMNN9xAkSJFKF68OLfddht33XUXtWvXzvR78vDDD9OjRw+KFClCSkoKL774Ivv27aN3794UK1aMcuXKMW7cuFT3O9p7AfDuu+9Sq1YtChYsyOmnn857771Hrly5mDlzpnPM9u3b6dKlCyVKlKBIkSI0a9aMFStWOLfv37+f2267jbJly1KgQAHKlCnD9ddfn6nXaBiJxBZBI0cwcuRIKlSowKpVq3jxxRcByJ07N6NGjeLLL79kxowZbNy4ka5dux71sQYOHEjPnj35/PPPadeuHddddx3btm076vNXrlyZ1atXM3z4cJ544olUi+edd97JggULmDZtGitWrCBfvnyMHz8+S6911KhR1KtXj08//ZSePXvSu3dvOnToQK1atVizZg033ngjvXv3ZtOmTc59jvZebN26lfbt29OyZUvWrVvH448/zp133pnqeffu3UuLFi04fPgwCxcuZO3atZx33nmcf/75fP/998778NZbbzF16lS+++475syZQ4MGDbL0Og0jIQQMIxuwbNmyABDYsmVLmttKlSoVuOiii476GCtWrAgAgd9//z0QCAQC33zzTQAIrF69OtXfY8aMce6zf//+QP78+QOvvPJKqucbPnx4qr+vvPLKVM/VokWLQLdu3QKBQCDw559/BvLmzRt47bXXUh1Tv379wGmnnRbV5kjP1blzZ+fvgwcPBgoUKBDo2LGjc92hQ4cCxx9/fGDcuHEZfi/69esXqFq1auDw4cPOMW+88UYACMyYMSMQCAQCzz//fKBSpUqpjgkEAoGmTZsGBg4cGAgEAoFevXoFLrjggsCRI0eivi7D8AumBI0cQaNGjdJct3jxYlq3bk25cuU44YQTaNWqFcBRVV39+vWd/+fPn58SJUrw66+/Zvg+AGXLlnXus3HjRg4dOkSTJk1SHRP+d0apV6+e8/+8efNy0kknUbduXee6PHnyUKJEiVTuzqO9F19//TWNGzcmd273lNC0adNUz7t69Wp++OEHihQpQuHChZ1/q1ev5rvvvgOgR48efPLJJ1SrVo0+ffrwxhtvcPDgwSy9TsNIBLYIGjmCQoUKpfp706ZNXHLJJVSvXp3p06ezZs0aZsyYAQQTZ6IRnlSTK1euVPG9rN7naNmuGSVfvnxpHjfSdXr+jL4XR7PvyJEj1K9fn/Xr16f698033/Dss88C0LBhQ7Zu3cpjjz1G7ty56du3Lw0aNOCff/455tdtGPHAskONHMmqVas4ePAgo0aNcsofli9f7okt1apVI2/evKxcuZLKlSs713/88ccJef6MvBe1atXi7bffJhAIOIthuH0NGjRg9uzZFC9e3ElUisQJJ5xAhw4d6NChA/3796dSpUqsWLGC1q1bx/iVGcaxY0rQyJFUq1aNI0eO8NRTT7FlyxZmzZrFo48+6oktJ554It27d2fgwIG8++67fPvttwwYMIAtW7bETB1GIyPvxa233srWrVu59dZb2bBhA4sWLeLBBx8EXIV4/fXXk5KSwsUXX8zixYvZunUrH3/8MUOGDGHevHkAPProo0ydOpWvv/6azZs38/LLL5MvXz6qVKkS99dpGFnBFkEjR9KwYUOefPJJnn76aWrVqsXo0aN56qmnPLPnqaeeonXr1nTq1IkmTZqwf/9+rrnmGgoWLBj3587Ie1GxYkXmzJnDe++9R7169bj77rsZMmQIgGNj4cKF+eijj6hduzZdu3alWrVqdOzYkfXr11O+fHnnmCeeeILGjRtTr1495s+fz5w5c6hUqVLcX6dhZAXrGGMYHnHWWWdRqVIlJk+e7LUpEVm4cCFt27Zl48aNVK1a1WtzDCMuWEzQMBLAunXr+Oqrr2jcuDH//fcfEyZMYOXKlQwdOtRr0xyeffZZGjRoQEpKCl9++SW33347LVq0sAXQyNHYImgYCeKZZ55hw4YNANSsWZN58+Zx7rnnemyVy+bNm3n88cf57bffKF26NBdccAGPPfaY12YZRlwxd6hhGIaRtFhijGEYhpG02CJoGIZhJC1RY4KJqGEyDMMwjHgSLepnStAwDMNIWmwRNAzDMJIWWwQNwzCMpMUWQcMwDCNpsUXQMAzDSFpsETQMwzCSFlsEDcMwjKTFFkHDMAwjabFF0DAMw0habBE0DMMwkhYbpZRAzjzzTABuueUWAK677joAJk6cCMDo0aP59NNPvTHOMAzPefrppwG47bbb+PLLLwG45JJLANi2bZtnduVkTAkahmEYSUvUeYKJaKCdJ08eAIoWLZrmNimm448/HoDq1asD0LdvXwBGjBjB1VdfDcB///0H4AwBHTx4cBytzjz169fnvffeA6BIkSIRj9m9ezcnnXRSIs1KGOeffz4AkydPBqBFixYAfPvtt57ZlFXuu+8+IPgdy507uI9s2bIlAEuXLvXKrKTnhBNOAKBw4cIAXHzxxQCcfPLJjBw5EoD9+/d7Y9xRqFixIgBr164FoFixYk7TZ72OBQsWeGLbsVCtWjUA8uXLR/PmzQF47rnnADhy5MhR7z937lwArrrqKgAOHDiQJTuiNdBOiDu0fPnyAOTPnx+As846i3POOQcIftgAHTp0OOrj/PTTT0BwQjfA5Zdfzt9//w3AZ599BvjvJNSoUSMAZs2a5Sz0+kBkuz7Yk046iaZNmwLujyGrH/rRaN68ubPgvvHGG3F5jlAaNmwIwJo1a+L+XPGiW7duAPzvf/8DUv+IbTa1N1SqVIm7774bwPnt1K5dO81xKSkpQNDN6Ed27twJwIcffgjApZde6qU5Wea0004D3N/KlVdeCUDu3LkpU6YM4P5uMvKb0fvwwgsvAHDHHXewZ8+emNps7lDDMAwjaYmrEjz99NMBWLJkCRDZ5ZkRtHOQG+qff/4BYMqUKWzfvh2Av/76C/DevSbX7RlnnAHAa6+9BkDp0qXTHPvdd98B8MQTTwAwbdo0PvroIwDuv/9+AIYNGxYXO1u2bEnVqlWB+CvB3LlzU6lSJcD1CmTHWZUVKlQAoECBAh5bknEaN24MQNeuXR13lHbron///gBs376dZs2aATBp0iQAVq1alShTM0SNGjWAoCIA6NKlCwULFgTc79SPP/4IuJ6WmjVr0qlTJ8B1xW3YsCFxRmcAndOye/LLo48+CsBFF10U08dVEuFLL73E8uXLY/rYpgQNwzCMpCWuSlC7mj/++APImBLUznPXrl2ce+65gBsX0+7Uz4wdOxbASdiJhtSiAvlLly51Eizq1KkTHwP/n+uuu46VK1fG9TlE6dKl6dmzJ+AqY7/txKPRqlUrAG699dZU12/YsMFJX//1118Tblc0OnfuDLgp9yVKlHCU0gcffABAyZIlARg+fLhzPx1TokQJwE1I8AqdMx5//HHAfV1KgglFnpW2bdsCbg7CN99847weXfoN5UbUq1fPY0uOjUWLFgFpleBvv/3GhAkTAPc7Fh4TbNq0qZMwl0hMCRqGYRhJS1yV4J9//gnAgAEDALfoc926dU6Gp1i/fj0ArVu3BoI+csUubr/99niaGRNUCK905vCY19KlS3n77bcBd+e9Y8cOIPh+QDCued5550W8f6xRan8iGD9+vPN/7dazA8pgfuWVV4C0nozhw4f7JoaTN2/wp6ws3HHjxgFujPrDDz/kkUceAXDizoptvv766wC0adPGeTy/ZPFefvnlAPTo0SPdY77//nvAPXcoJqiYd3ZAn5Ni5qHoM5X3xC/fuUg8//zzAMyZMyfV9QcPHuSXX36Jet8iRYo4DQKUSSr0ePH4XpoSNAzDMJKWhNQJahVXsfjff//t+L5vvPFGAKeYVVlSAF999RUAvXr1SoSZWaZ+/fqOL1yF8PJ3v/vuu0AwRih/t7JcpZBUI/TZZ585mbBSlIobxqqdWt26dQEoVapUTB4vI4QqKL1P2YHrr78eSJvZq5ia2t35gS5dugCpVTe473fnzp3T1FcpvhaqAFWL++qrr8bN1sygOrNwtm7dCsDq1asZOHAg4CpAoUzS7ICy3OV1eOihh5zb9P9du3YB8OyzzybStExx6NAhIO1nkRHatm3LiSeeGPE2fS/j0ewgob1DQ3+Eu3fvTnWb3B3Tpk0DMtZNwGvUDWHAgAHOif73338HXFenTiZ79+5l3rx5AM5lNI477jgA7rrrLgCuvfbamNisgLUeP55ooVV5BMDPP/8c9+eNBSVKlOCGG24A3O+iTkJDhw71zK5IDBkyhEGDBgHu5kulANpwRSowvvfee9Ncp2Jybcy8RglV2ggvXLgQgE2bNgHBhIv0SORGL1bIZR26COZ0lHzVs2fPdM9LDzzwQNye39yhhmEYRtLi2RQJ7XSUUCJXodLRtePzI0ooGDFiBBBUVyrMVVGnArjHqrgiBcqPBfVfBdfdHC/0/pQqVYqNGzcCbgGzX1EPx1mzZqW5bfTo0YDr1vca7Y4HDRrklBGpv6RchPv27XOOV1G53J/hjQuGDBni9Gr0C3ITZkUZqY1adiR37tzZwhuWFeTVkvfi1FNPBYL9RcNRwuTBgwfjZo8pQcMwDCNp8UwJKgFGPn8lfii1+/3333fU1JgxYwD/NClWskpoQehll10G+K+BdzRWr14dk8dRMtAFF1wAuEkaoQkXinUoruZX9BqUQARu2z8VnnuNCqv79OkDBH8XUoDt27ePeJ8qVao4EzzkfREzZ84E3PZ92YnbbruNQoUKAWmLsEMbTqxYsQIgYQ0ijpUjR4745nyXGeRJ6dq1K+B69kJR6VGk16fYtZrUv/POO0Bqj0asMSVoGIZhJC2eT5ZXoatGb7z88stAcCeh3YR2ekpJV+alV6icQzvPpUuXxkwBqohd8YB4Fs0XL1486u316tVz7NE8wFNOOQVwW1Jde+21zjHaran1ndKZ8+bN64yG8itSUJpHCW5RuUolwjOavULvfWgLMGV1nnzyyQB0794dcEfR1K5d22nPpx24LtXKLrQ8yW+omFwNNBQPDfXGhP92xI4dO5z34/Dhw3G3NRmR6lZMOau5DMuWLQPgxRdfjI1hGcCUoGEYhpG0eK4Ehcb5qP5n5MiRjvrQOCGNslGdVqJrztT2rX79+oC7k37zzTdj9hzhAyeVHRUrpNYCgYAzqPKee+6JeGzdunUdJaoi2H///ReAr7/+GoAJEyY4sVupYTWTVoHrcccd59uG2dGyQTdv3gz4rzm2MkFVy1eyZEm2bNkCpB833759uxNvUfG/alrfeuutuNqbVfLly+eMY9PnI9v1Pd6xY4cT71M8V6pR5MmThyuuuAJw47rxGlad7Oh8Ec2DlZ5iB/ccK4WvmGA88c0iKL744gsAOnXqRLt27QDXRXrTTTcBbk9A9QpMFCp3kDtKhbrTp08/psdVyUVoGrjS8BUgjhVKpti2bRtnnXVW1GN/+OEHx72hRe/jjz8+6nOosFlTCrSY+BGVEkT6QYa6Rv2Ekovkwn377bcd17bCC/rc1IHkzz//dBpRaCHR335Dv68LLriA2bNnp7pt8ODBgPv7WL58ufPadV34ZPmSJUs6c+5++OEHwO1iFY8OJLEgUomE5kH6sWOMztuagqPkOCVs/ffff+neV13Dwqe0JApzhxqGYRhJS65AlDxcv0z/Dk2wANc117ZtW6ePYyJQH8OpU6cCbn+80LZgmUEKUK2tBg0a5Lh4paa0k8pOSBnr/Ro+fLijuPyCXNpys4UH8ufOnUvHjh0Tble8aN68ueOulsLQdHY1AfAaFUs//PDDgDt9BmD+/PmAqzCkhkuWLOm4zFS6JFenSj5q167tlDCJxYsXpzrmr7/+cm7TVBcvOXz4cLqubZXvyDuT3VHLSc2dBTehK1bu0GjlJqYEDcMwjKTFdzFB7XI6duzozNGSAhTaAX344YeJNS6MrCbESIVop6tu/nPnzqVDhw6xMc5HhM8W8wNqyxfetV7lHSrZySkcd9xxaZKu/BITzJMnD+A2VOjfvz8QLNlQay15X6QAdW4YPXq0kzyjWZW9e/cGgg03INjMQfFvteyS0ghtz3isnp1Y8sILLzg5EOHISyQln91p27atp89vStAwDMNIWjxXgmrorMwgTZJOSUlJc6wKXVUsn+gGs+Hpv8rOy+jk+379+gFuDFC+cLWzUvNtI/6cdNJJQNrvkFr07d27N+E2xRM/x5albKQAVYZz0003OUqtSZMmgNsEQCn0BQsWdGKIyiIPn2W3Z88eJ6aoy6uvvhpIPaLszjvvjOXLOib8WlIUimK4bdq0cTJzM9PeTKPKRo0aFXvjMoEpQcMwDCNp8UQJpqSkcM011wDQt29fwC1ajoSKsVUkH8vi9MwQ3m5KavWZZ55hwoQJgJvhpJ2rWr/Vq1fPaTmmWiXtzjUANachxVy1alVfNS5++eWXnYLdcFR4ndPwOu4SjfCBqYoRDhgwwKmdrVKlSsT7PvTQQ04NYGZaoinGqEu/MXr0aMc7plFDQp6n0aNHO3WhiaRZs2aA22SjdevWThw12kR51XNKxav9ZGhzAynJeDbMDichi6AmPKvv3+jRo6lRo0bU+6xatYrhw4cDbuGv3+Zr6cfap08fJ6FFXTlU0B+KFgK5DuI5LdkPaLOQ3oKTaJSQ1Lp1a+e7pHR6uUH91h0mVoSfSP3EL7/8ArjNFVQ6VK9ePecYpcorGU7JVlu3bs2x/UA177Ny5cqprvf6PKiSmtCmBHfffTcQfV6ompuolCW8bOGDDz7g+eefB9ykpkTgj7OTYRiGYXhAXJSgZO/YsWMBdwcevqMJRW4oSeQFCxYkVBJnBCk5zeFTmja4rlGpXiH36LRp0zKcQJPTaNq0qdO+y0s0hy/0M1JzAiVl5FSWLVsWtWejl6gdmBLNpBR+++03J8ygYvZk6vmpSQpqH+lnVJaSGdR2Ur1rb7/99qjt1eKFKUHDMAwjaYmZEmzcuDEQDGY3atQIgLJly6Z7vFSeurprUoSfZ5ppKoI60quYVSUPoeh1aVKDCnmTCb+03TOCDY71HZRHRnFCTaPwCsWRJk2alOoy2VFTkG+++QaAmjVremmOg8pUbrnlFsCdtxmN77//3il90czAcePGAW7zba8wJWgYhmEkLTFroK2xM6FNb4V2MvL9Hj58mBEjRgBuGyQj56CWY4rnjBs3Lt0WUIlEcdvp06dzzjnnADhz+NJLwc9J6HMZP3484M5/VCp+TmnIbCQGZfF269aNIUOGAG4bQmXvLlq0CAhm+CsL2AusgbZhGIZhRCBbjFIyDOPYKVKkCACvv/46AK1atQJwBtd2797d1zF5w8gq0ZSgLYKGkWRoMVQHJqW3161b11yiRo7E3KGGYRiGEQFTgoZhGEaOxpSgYRiGYUTAFkHDMAwjabFF0DAMw0habBE0DMMwkhZbBA3DMIykxRZBwzAMI2mxRdAwDMNIWmwRNAzDMJKWuEyWNwzD8Ipq1aoBMH/+fADy5MlDhQoVvDTJ8DGmBA3DMIykxZSgYRg5gtGjRwPQuXNnAIoXLw7A22+/7ZlNhv8xJWgYhmEkLdZA2+csWbIEcD+L8847zxM7atWqBcAll1wCQM+ePQFYvXo169evT3XsqFGjADhw4EACLTSSkVKlSjnzEJs0aQK4zZK//PJLAM4//3z++OMPbww0fIE10DYMwzCMCHgeE8yXLx8AZ511FgDDhg0D4Oyzz/bMJj/w1FNPAe77MnHiRE/suOmmmwAYPnw4AIULF051+6mnnspVV12V6ro1a9YA8N577yXAwpxN4cKFnRjXf//9B8CZZ54JwAknnADAtddeywcffADAzz//nO5j/fLLLwDMnTsXcD+n7IgyQEeMGEHjxo1T3TZo0CDAfX3ZQQXK0zN16lQALrroIsf78tNPP3lmVzLguTu0RIkSAPz222+A+0M944wznP8nE4899hgAt99+OwAHDx4EoEePHgC8/vrrCbVHyQWaOH7yyScf9T67du0CcBbHhQsXxsm6nM8TTzxB//79Y/qYR44cAdzPdNq0aUDwBLxly5aYPle8aNq0KQDLli1zrtP5qkuXLoC7oGQHjj/+eAA2btwIQJkyZejVqxcA48eP98yunIK5Qw3DMAwjAp67Q8NJSUlxLpNRCSq4LzfxRx99BCReAYo///wTgIceeggIup/A3bn+8MMPlC9fPtV9ihUrBkDbtm2BnK0EVYR93HHHAXD11VcD0Lt3b+eYefPmAdC9e/dMP/4VV1yR7m1y833++efpHvPtt98CUL16dedzOf300wGoXbs2AEOGDAHgs88+870SlBt08uTJQGpvld4ruXuzE//++y+QWglmxOuSnbnrrrsAyJ8/PwA1a9bk2muvTXXMhg0bADjttNPiZocpQcMwDCNp8Z0SzCllGc2bNwfg3nvvBYIKQaoqPa6++mpnd/79998DxDwelFVeeOEFwE2UqVevHgB79uxJ9z5jxoyJv2Ee0KpVKyCoPKT8ihYtCkSOPUjdZ4W2bdtSvXp1wFV1Quphx44dGXosJdJ88cUXAGkU/KWXXuqoVr/StWtXwLX9nXfe4eabbwaiJwVlF/SbadmyJTVq1PDYmtjRokUL59zWokULAC6//HIg9Tk//PdTtWpVwI1fK1kolpgSNAzDMJIW32WHirPPPpuVK1fG/fnjhXzZ2sm0aNHCie+lx5dffun4vhXfeOONN+JoZebp2LEjAPfccw8A9evXT/dYvZZvvvkm/obFEWXn1alTB4CGDRumOebvv/8G3FiV0vOnTJnilDZ4zTXXXAPAa6+9lur6/fv3A0HvxerVqxNuV0ZYsWIF4H7ftm/fDsCFF17Id99955ldsaZcuXIAbNu2zWk2UalSJSDjit8LSpcuDbgZuZUrV051e9GiRSlUqBDgritr164FgpUAR0MqP6uN0KNlh/rOHSrOPPPMbL0IylWlN79gwYLpHqsfdvny5Z309WjHe8nMmTMBN2FnwYIFzuIQzsMPPwzAlVdemRjjYshJJ53Eo48+CsANN9wAuElC+vE+9thjTleSffv2AcFEIT+hpINnnnmG6667LuIxqkVdt25dwuzKKJdddhmAUwuo39OMGTMA933PaeTKlcv57C699FIAxo4d66VJ6dKqVSvGjRsHuIt4NOTS/P333wFXCJUpU4aXX34ZgFNOOSXVfeQOjQfmDjUMwzCSFs+V4KFDhwDYvXs34CYYnHrqqZ7ZdCw88sgjgOs6k1v0s88+S3Os3AMDBw4EgmUHH3/8MeAqLr+hFOa6desCbpp9JJYvX54Qm+LB/fffz4033gi40wmU5LR3717P7Moo6jGrwvFu3bo5t6kBw2233Qb4111drFgxmjVrFvG2v/76C4jeTUUNJ0LViV8SzY5GqPtOitCv3H333ekqQLnaBw4cyKpVq4C0CV4q9bn99tvTKMCtW7cCbkJUPDAlaBiGYSQtnitBtdhS+yNNKchuaCek6QpSuH379gVg586dae7z5JNPAm7MbPv27b7rmao0bXXqr1KlCgB58x79q/Pmm2/Gz7AYoaJ/qXHtOO+44w7ef/99IBj3BHyT4BKNRo0aAa7NefLkSXOMVMaPP/4IwOHDhxNkXeY4fPiw0yc1d+7gfl0x8w8//DDN8f369QPc13frrbcCqZMpVKAtxZETyiq8ok2bNkDkEiDFxvV7yohXKFwFgtv4QPHDeGBK0DAMw0haPFeCOYE6deo4SkmZToojLV26NM3xikuExmkAhg4dGkcrs0bNmjUBN007IwpQ3HHHHYAbe/Ij9913H+AqQbWnW7hwYbZQfuF06tQJiKwAhWJMmriuco633nqLOXPmAG5BvZe0aNHCiQlKAUphhE6GUHb1OeecA7jZlOKff/5xYodqPKCYu5q8b9u2LS6vIScjVS1vCrilLIMHDwaiK8ATTzwRCJa5gNtgJPRx3nnnnRhaHBlTgoZhGEbS4lsleNJJJ3ltQrpIDSnz7qWXXkoTs9CoFxWVjxw50hlLpBigikY1K9CPdUAq1pdS0qinjNQxqoDWz2j2nOJIKvbNjioQ3NitFLwK++WhiESDBg2cywcffBCAUaNGAcFRTpC2mUU8UXs3eR/ALRSfNGkSgFMgX61aNQYMGAC4NYWKHy1atAgI/vaKFCkCuDMulYXuV3LlyhW1wNsPvPjii0Dwu6XsfjVkyMjwA7W7U0Y9wFdffQW4Ho1EDFHw7SIY7tLwE3KhqJNIIBBwFr9NmzYBqU8sEHw9ZcuWBdzFQckyKsb2M8888wzgnnw0kQDcTYFcwDrhZAc++eQTwP2cnn32WSBYhK2TaHZCbqSLL74YcHtslihRglKlSgFuNyJ970I7Q2kzpyQTJaacf/75gLvJiydya2qwNLgnXDVg0GsZMWIEFxn5dHoAACAASURBVF10EeB27VEhvdx1VatWdXrf6hgthn51g/p9AQSYNWtWqsuM0q5dOwAeeOCBVNcfOnTIEQKJnCBk7lDDMAwjafG8d6i48847gaDrAoLTCULVhh/o3Lkz4PZeVBnErl27HDeAinj1OtQxHdz3U2+5LrXradmypTM9Ijuh1yVXmnZ4ei2tWrXyxY5brbfWrVvn9GWUi1rJO/fffz8QLIhX6rdfi8mPFTU+UCmByisi8b///Q9w3aPxRK730ESx8IQsJVzoMwVXrSoZLdL0ebl5/Vo0H9o7VJx77rlA5CS77IhKcsKXnj59+jiKP9bYZHnDMAzDiIBvYoLhjYfz5cvnFLn6QUWAO0tPtmqnOmHChDTHanetnU2kglIpKBVlZ0cVCG7KfbiPX+25vCrGVuxVpQCKj915552OmldTbMUCpQQLFy7spHDnVDTxYvr06QAsXrw4VZp6KGqSkAjkAcqVK1eaKfEqh6hYsaJzjGJ/UkqRps/rGCnB7ER2PS+EM2zYMCBt4wPhldI1JWgYhmEkLb5RgoqviVy5clGgQAGPrImMdqVKQ1fbqUgoJV0z9QBnCrnG74hoTYCzA6EpzqFIIXv1+j799FPAzVZVrCl8nh64hf1i8eLFaT6nnIp+e2vXrk1XCW7cuDGRJgHBOE56sRypiEAg4DRzl4dG5TtbtmwBoFmzZk4Kv+EN+fPn5/TTTwdSf3bgNjr3ai6kKUHDMAwjafFNdqjQ8MQaNWo4tT19+vRJuB1ZRUW4ihf27t0bCPr1FavwG2pMIOWmGNGUKVOOet/SpUs72ZPh9YGKI23evDlmtmYGFcKrNdpxxx2X5hjtPqtWrQq48ecOHTo4StLvlC5d2mncrtFdav+WEdRibcGCBc4IJiGVqMzL0EzLeBEpq1Pt0+rVqwe4TRsKFy7sHKPzlYrlu3fvDiSm9VasiJQdqu9mdowNqqValy5deO655wD3c9L5RfkTGqYQD7LVZPmFCxcCULZsWadgNzuhBVvdENRpI/zk4ieefvppwC1i1WL9888/O1321QRAxdM6ZsCAAWkWP5WHbN++Pc6WR0eT4ZWgI3dMq1atnGOU/KITpRIo9Hr9TEpKCgDz58935ldmJplHBef6nUX6jmqDk4jFT6h85d9//3VOoh999BEQ/WQWXiyfnRa/aKgZgJpRZAfU9UcT5zt27OjcpnI4JaMlogFDNMwdahiGYSQtvlOCIhAIODvC7IDKOXr06AG4O1aVSPg5+WXMmDGA26tR7qj333/fmewsN7XcUtrpgfta5Yp76KGHAP/03xwxYoTXJsQFpftLBYL7GWp69759+5zb5A6+++67AVcBhn6WclVJVXkxAWTt2rVAMJFMNrZs2TLisa+++qoz8WLdunVA9i4q//XXX4Hg761WrVoeW5N1NBswVAHKnasWjH7BlKBhGIaRtPhWCRYpUoT27dsDbkmCn1GzZSlCpeGrlZifWblyZapL2T5mzBinKFmXkVCruNByECP+LFmyBHA77oNbFiJVFFoaoKQtxUYjIQV4+eWXA96qqnnz5jFv3jzPnt8L5P0KVfCtW7cGskdMsEaNGgBp8jk2btzozA30G6YEDcMwjKTFd0pQu9r9+/c7cajswCuvvAK4o17efPNND63JGmoqrCYFoennalelgn+xe/du2rRpkyALjVAWL14MwLRp05zxXiKa2gtHZRCjRo1yxuKsWrUqRlYaWWH9+vVOJnbo79DvqO2ghg2IZ5991jftL8MxJWgYhmEkLb4rlp82bRoQnIytwbp+3UEYhh8oUKCAE8NTrZ/anIUOp1b2rtBgWWWSKo5oeE/FihWZOnUqEMyABZzmIX5E+QBqYqD4n7Ljn376aed75gXR6kt9twgahmEY2YvHH38ccJtNSLio0N/LBRBsnqBhGIZhRMSUoGEYhnFMqLfsggULgGDvXSDNPEivMCVoGIZhGBEwJWgYhmHkaEwJGoZhGEYEbBE0DMMwkhZbBA3DMIykxRZBwzAMI2mxRdAwDMNIWmwRNAzDMJIWWwQNwzCMpMUWQcMwDCNpsUXQMAzDSFpsETQMwzCSFlsEjZhRuXJlKleuzPTp05k+fToHDhzgwIED1KhRw2vTDMMwImKLoGEYhpG05PXaACP7c9ZZZwEwf/58AHbu3AnAmDFjAPj111+9MczIENWqVQOCk8uvvfZaAHbs2OGlSTGhZcuWACxZsoTcuXOnum7p0qUeWWX4Dc8Wwa5duwLQtm1bAOrVqwdA9erVnWM+/vhjANq1awfA7t27E2liwilUqBAffPABAGXKlAHg7LPPBmDr1q0eWRWdiy++mJkzZwLBkyjAvffeC8C///7rmV05hRNOOIHChQsD7vc/1u+rpn83b96cHj16APDoo48CcOjQoZg+VyLo1q0bALfeeisAR44ccW578sknAZg4cSLgbtSy4+vMiQwaNAiAoUOHAvDEE08A8L///S9uz2nuUMMwDCNpSeg8wRIlSgAwfvx4R93t2rULgJUrV6Y6tkWLFhQqVAiADRs2AFCrVq2Y2pNopO5KliyZ6vq//voLgHPPPZeXX34ZgG+//RaARo0aAfD3338nyswMUbVqVQDWr1/PsmXLAFdRhO68jWNjyJAhzi54wIABADz11FMxfY5mzZoB8P777zvXKZlp06ZNMX2ueCIFKC9T8+bNndvkDg3/blapUgWAbdu2JcDCrFGhQgUA7rzzTgD69OkDQN68eZk2bRoA11xzjTfGxZATTjjBOe+VKlUKgIMHDwLQt29fAF566aUsPbbNEzQMwzCMCCQ0JqjEiYoVKzq+3uHDhwPw559/pjq2Ro0afPLJJ4AbuH/ggQcAePjhhxNib1aoU6cO4MYjtIsD93WUL18+1X0ee+wxIKh0pb5//vlnAPLnzx9fgzNJwYIFARg3bhwAX3zxBZ06dQJyhgIsXrw4AJ07d+aee+4BXAUv7r//fgCGDRuWUNsefPBBADZv3gzA3LlzY/K42nVnJ4oVKwZA/fr1He+JPCwFChRIdeyGDRscJajfYHbghhtuAFzl/9133wFw0003AVCuXDnnO6Fzorxm2Ym8eYPLUO/evdN8F5VUF+4pjCWmBA3DMIykJSExwdatWwOuEnz99de5+uqrj3o/7W7uu+8+wPXbV6pUKSZ2xYPbbrsNiBy32b9/PwAzZswA4PzzzwegdOnSzjF6z6+77joAXnvttfgZmwWk3G+55RYgGBv86aefvDQpJjRt2hRwswcbNWoUNY4AMGnSJLp37x5Xu0JjgmLPnj0AtGnTBoA1a9Zk6bGVdbpw4ULAjT+D63VJtNo9Gu3btwegZ8+eQPA9SC/eJ2688UbnGHkwhN9igvL83HXXXc5noO+kfnvKozjjjDNYvXo14HqX5EHKTigmrcz4UJQ78s477xzTc1hM0DAMwzAikJCYYL58+QA300wZTUdD9WdSgopHFSlSBHB3xH7goYceAtwMPvHqq68CwQLyESNGOP+HYDwDYMGCBUAwe1a36bX7BcVZunTpAri7tuyuApWx/OKLLwJQs2ZNIPgZzZkzB3Bjb1LnV155JQBNmjRxdu4HDhyIi31btmxJc52+/4MHDwaCn4kyjDODMnxDFaBf0fdOv6dQpPLSI9SjdbRjvUaehSFDhnDHHXcAMHr06IjHtmnTht9++w3IngqwYsWKADz99NNpbluyZAmQOmM5XiRkEXzvvfcAOP3004GMF/vKfSgUNFU6sIqz/YDKOY477jjAda+ocDy0A4dcMEq8UED/33//dU5s//33XwKszjh333034LrQ9LqyO1rgtPjJNahyj1C0iWvVqhUAp5xyinO/zz77LC72vfLKK05ijpIghBpNdOjQgfHjx2f6sZV0oESbypUrO7e9/vrrWbI31mjxGzVqFOC6PPX7+PXXXznhhBMAN6lJ6Ji///7b2Tj4NXlLtj/yyCNAcBP8/PPPRzxWyXZqbJBdeeutt4DUpW8SNnL97tu3L+52+HtbZBiGYRhxJCFKMKuqRjvUr7/+GnB3DHLj+Am5Ly+88ELAVRYqf+jTpw9FixYF3ED3xRdfDLjlIUOHDuW5555LnNGZQEkYy5cvB+DTTz/10pyYEb7TzEzZwZ49e/j9999jbVIqDh8+zDPPPAPg9PWUJ0H07duXN954A4A//vgjw48tz0qoAvQT7du3d9yf4Qpu1apVQFCVq0g+POlFnpbZs2c7x/gNlQfodyX3Zu/evdNt5aZkucqVKzNy5MgEWBkfTjvtNCB10orOf4sWLUqYHaYEDcMwjKTF11Mk1DJHl35m/fr1gFvUKSWoMojWrVs7ZRPhxfKKA6YXAPcSpS83adIEgLp166Z7rDr0K7nnq6++iq9xMUBJE7pUgknBggU59dRTAbcd15lnngnAL7/8AgRj04lISFDjbKmFcCVYp04dypUrB6SvBPPnz+8UWQsl+PgNvd+KA4LrTZICVClSKIrLSj2GxtTkqVFphV+SgTp27Ai4RfznnXcekLZ5CLi5EPot7t2710m2y07IE6bfnJTgkiVLnJhoIjElaBiGYSQtvlaCSstXaYTwWzNpcDNZw8s2VAg/a9asNDsfNYNVKr4fURzqm2++Adw4rejWrZsTlzjxxBMB973o378/4I6r8SPhcYl+/foBwWJlKT9x1VVXAd6Vr8jLcP3116e5TcX+8khoxqMuCxcu7JQaRUNtt7JSchEr1JZOGdfgFu1rxFMoH330EQDvvvsuEHl+5d69e4G0Gedeo89SjaNXrFiR5piUlBTAbcChMo/Ro0dnq1mdOg+o4YF+c59//jkQPNd4kRVvStAwDMNIWnytBFVMGTpoF9z2a6Go6FnDebUznjFjhrPLSgQZab+kFkDy5//4449xtelYUBNfxSO0k1aR+IMPPujEmlT0rxo7NTb+/vvvI35mfkAxNNWaNWjQAAjGK7RTVV2rspS9QrWAir2Gth589tlnU12Gkzt37gzVyCmWrd16VkfXZAU1j9BnkTt3bvLkyXPU+2Vm3JO8MX4pmletp1qkheY/qLZx1qxZgHuOU330448/njA7j5VGjRo53ykpW6FGFcolSDS+WwTlAj3llFOcqerh6Euwdu1azjjjDMAtNlWCgFymVapUSUh6tH6sSiSJ1Hd13rx5gNsPz++cdtppTgp3eLq23vf58+encQ9Onz4dgHPOOQcITov26yIod6iSDU455RTAfQ0QTLEH7xdBIfez3LMZ4ciRI0fthRqK3o9ELIK1a9cG3JO93OqxLGxXkwdt3rwumlfCnAgvzWnbti1jx44F3EQ6LfYq/fBTx6yjccMNN6TqkQxuiCVW01Cyij+2Q4ZhGIbhAQlRgmoldvLJJwPBVPPGjRsDbkpw+LHRpshr967ic4AJEyYArtqSmytS78V4oH6oV1xxBRC5a3lmduJ+INRtEe5SVvlDtGQLpah/8cUXcbAutnz88ceAOw8yFL9NUsgKmzZtcr5/+o2o9EKuOK9QM4Dw0qFYolIEv5RGqCheiSDyPMgVXLJkSSf0IK+SEkv0uWUH1P/0xhtvTHP+03Sh7du3J9yuUEwJGoZhGElLXJSg1JwmKygGVqNGjXTvI/+2UpkPHTrkxKOEEgMUE/S6dZcaG3fv3p0OHToArtqTbSrg7d69u6OEsyPh0yIyUqaSHSdMKD6V0UQSv6Ei6x9++AFw44dTp05Nc6wa2nutBNNDTduPlRo1avDEE0+kum7r1q2Ad43q5R25+eabgaBSAvd8MXXqVCfJSfMiFSPMDig3Q02+c+fOzeHDhwG3vZ3XClCYEjQMwzCSlrgoQRV/y+cr37ZiEVu2bHEygnSbdmZSDxs2bHBaCalAW4XMUoteowyvhx9+2LlOMTLt4pQW3L17d99kF2aUXLlyRcxyzSgtWrQA/NncID3UUPvIkSPOzMR4zQrMKt9//z0AEydOBIKNlJVppwbExxqHVcN0ZWp6UTyfmWbgkZDnae7cuZx00kmAG4tTjNDrYnN9hrrU723UqFFOg3N5mfw2Xi0Saun35ptvAqnL21TsP3DgwMQbFgVTgoZhGEbSEhclqF2kMjO1k1m3bl36hvx//E8FoKeccoqza+vUqRPgHwWoYmVltQFceumlACxevBhwMytD4y1Su9mFQCCQpYzWfPnyAW68Y9KkSTG1Kx6oSFyxmZ07dzrZrX773BQ/VyODeFC2bFnArauLJ+kVsL/88suOQsoIqgXUfS677DLnNnmTLrnkEiBttrNfkPfklltuYejQoQCsXr3aS5MyhZRfeIMTcNWh34jLIqgT565du4Dorhn1BZ0xYwbgztjbv3+/UwzsdQJMOHLzqkRj6dKlvP3224C7AOjHpmNy5coV99lzsebrr79mx44dgDvhO71p1+C+dh2jjj+Rel36BX0+KubXyX/gwIGe9QhNFPp9aipGeCcPcMtD1BUovRl3x8KQIUMAt0wgtPTp/fffB9xzisIoWsTuvvtuZxHVgq0yCHX6GTZsmNPwwK+Ln5gyZQoQTBoJT+bJDqhpSTgffPCBb8NB5g41DMMwkpa4KMGNGzcCbi9A9YZTcPqzzz5z3BMDBgwAXPmseWF9+vSJ6j71Eu1KQy+lgpQI8/TTTwNuQsH48eN9OzU+PXbs2OEogfAJ1pMnTwbg1FNPdWYMqp2TAvhyi/tZAWu3LQWopgfZeWJ3RgkPV8yePdtJxhBS8ZrfFw8luGTJklR2qH1a0aJFad68OeC2OVNbwlDkRtUxS5cuBdImnfgZ9axVf9DbbrvNN+GfzJDePMDnn3/e08kk0TAlaBiGYSQtuQJRMh+OJT0e3F2B5spF6tyuYKka9fq10XIoKlpVIejMmTOdHXT4TlXK8K233kqghbGnb9++AAwfPhxwG52DWwKhRCHFePxWWhBKq1atADfGJBWh2KfXTX29oGHDhs73VIpEqBxIKiueSJX36tXLKTmK1rhACXTLli0D3PhldmgvppwIzRFUSUrt2rX5559/PLMrs6iVpcqKFBscPHgwEFwLvGwbGe25TQkahmEYSUtcG2hrQrQucwoqTBYdO3Z0VLPaVqnZrUomsjt6PX6eEp9RKlasmGpUErixr2RUgGL16tVOQwp5b9TgQq27EsHPP/8MBGdVppc7sGHDBiDomVDzgOXLlyfMxljRvXt3wJ2DqjyK7KQCwR29pQbgQs1Q/Dw8wJSgYRiGkbTENSaYU5HfvmfPnkBQ6WqnrBinWgQZ/kGN3YcPH+4U8isTsXPnzp7ZZSQvqp2TYmrYsCEQnyzcRLBt2zYAjj/+eMDNEPc60z+aErVF0Ega+vTpA8Do0aNZuXIl4CZ86CRkGIlEjQrUfzi7lVFlFywxxjAMwzAiYErQyPGojZZaZ7300kvOTLPsOPPQMIzMYUrQMAzDMCJgStAwDMPI0ZgSNAzDMIwI2CJoGIZhJC22CBqGYRhJiy2ChmEYRtJii6BhGIaRtNgiaBiGYSQttggahmEYSYstgoZhGEbSYougYRiGkbTYImgYhmEkLb5dBKdMmcLmzZvZvHkzjRs3pnHjxl6bZBiGYeQwfLsIGoZhGEa88W0D7eXLl9O0aVMANm3aBMBpp50GwMGDBz2zK6N06NABgIIFCwLQoEED7rjjDgDef/99IDjSB+Cbb74B4NNPP020mYZhGDmebDVZvly5ckBw4cuXL1+q2woVKgTAvn37Em5XNI477jhq1KgBwCOPPALAeeedB0CBAgWOev8tW7YA8N577zFw4EAA9uzZA8Dhw4djbq8R/MwA2rZtC8CDDz4IQP369YHIP5obb7wRgL/++su5Thu0L7/8Mn7G/j/t27cH4NZbbwXg3HPPBYK/0/R+xnPmzAHg3XffZeHChQCcdNJJAGzcuBGAvXv3xs9ow/h/atWqBeCIgTJlygBw8cUXAzB37lxWrFiR6j4vvvgiALt27Tqm57YpEoZhGIYRAd8pwTp16gDw2WefOddpN9uxY0cAjhw5knC7Qqlbty4AzZo1A4JqQruZWDF48GAA3njjDQC++OKLmD5+PClfvjwAK1euBILvTyKUUjjVq1cHcNR1KMcffzwAV1555TE9x1dffQW4302pq1giBThx4kTA9Yhklu+++w5wX/vvv/8OwIEDB5xj+vXrB5BmR24Yx8qTTz4JwO23357h+8jrcu+99wIwduzYLD23KUHDMAzDiEBerw0QefMGTYm0a586dSrgvQIUUoDPPPNMusf88MMPQPSYXunSpQE3eSYUxai0W/dKCVarVg2A//77D3BfVzSef/55wFUYf//9d5ysi86iRYsAKFu2bNyeQ8laq1evBuDVV18F4LbbbovZc5QsWRLIugIUVatWTfV3pPdl+vTpAFxxxRWA+7qMzKPPS7/vSy65BHDjzplF5xvlEGQ30vO6rFu3DoCff/45zW3KrbjqqquArCvBaJgSNAzDMJIW3yhB+YuvueYajy3JOIpVtm/fnl9++QWA8ePHAzB8+HAgeuad1MJTTz0VTzOzzOWXX+4oGynTaLaqpKVVq1YAPPbYYwBs27Ytnmami1SN4lyh7N69G3CzeXv16gW4yjezFC5cGICWLVsCrkJUzPBYeO655475MTKKMvY++ugjABYvXgxAly5dgNSZsX4lT548AFSuXDnNbfJk7N+/Py7Pfc0113D22WcDOJfKczhWLrroIsD1RP32228xeVyvUIxaryv09ZQqVQqANWvWAFCvXj0AunXrxrx58wDYuXNnTOwwJWgYhmEkLZ4rwZ49ewJuDVZ2YMqUKQBMmjQJCGYuKWa2devWDD9OtHjLP//8A8Rut5MVrr32WkftZkStXnbZZYAb3501a1b8jMsAUuWKUYZy6NAhwFUGs2fPBmDAgAFA0DPx+uuvA1ChQgUAihcvftTnlJJSHC8WSIVLjYmPP/4YgJdfftm5rkGDBoD7uxJVqlRxFFJG0Gd4wQUXAHDKKacAWVeCl156KQBvvvlmlu4fTpEiRQBo06YNN9xwAwD58+cHcOqLpZhCeeCBBwAYMmRITOwIZ/LkyU7ugi7DPSHLli1zftdqlBGJ2rVrA67HqEqVKkDwdwn+9SBlFJ3jQhXgiSeeCLjfX/2exEsvveScVzp16hQTOzxbBLt37w7A6NGjAfcLrK4pZ5xxhjeGZYDwE4EK24+GfpzDhg0DoqfnK0FoxowZWTExJpx99tnOQp8R9IX1stNQKFoQJk+efNRjdaK65ZZbnOs6d+4MwGuvvQZAo0aN0r2/Gjjox/vBBx9k3uB00PdFl9FYtmwZkPYE2bt3b6c0Qtxzzz0AFCtW7KiPqxKQrCZoKUkps5x88skAtG7dGnDLXlq0aAGkXuh07pg7dy7gJqQ0bNjQOUavOV6L4MaNGx1Xq55Dm6nMUK5cOZo3bx7xNq/CC7FGjVH0Or/88kveffddwP3thnPkyBFnYx4rzB1qGIZhJC1xUYJKElAwU7s37cg6derkyF6hAsp33nkHcIOmOYVzzz2XO++8EyBqYf3mzZsBt0jeC6ToChYsGLXINBz1S/33338Bt6zCKzKiAJXGLsWhZBpwXW7hpQWhKPHp5ptvBrz93KIRySWsllR6D0aNGsWFF14IuL9h0aZNG8B1zWaWrLY6lKJWW0J5GfS9zJUrl+PK7t27N+C615Ry37BhQ+d+ajgQL3SuyyqVKlUCguox3BsmhZtVVe031L5PvZSjoTDTE0884YSjYoUpQcMwDCNpiYsSlK9XUxLC0853797NuHHjABgxYgTgFoAqAJ9TUOxz7NixR01MePjhhx1/t0ouvGD79u1AMPYplaBG4NFSy9WUev369QB8//33zn3jlZKeVfS6lDyTlSD77t276dGjB+Am1mQn1MRACQrvvfdeul6KUaNGJcyuUKT4pCTVTnHo0KFAMI70448/Am4iipKbHnroIedx5GFRO0K/oDityoqkziMlVt1///2Ad80nEoliz0rM+uOPP4D4TBAyJWgYhmEkLXFRgkr7VUwwPKayZ8+eDLXfEsfaLiqRqLm2ygW0e4ukAhUzUxx04sSJvmqJNHPmTO666y7A3ZkOGjQIcHfWkVCRsgqthw0b5rs4hjIijyXNul+/ftlSAYaj2GekuKHKMDISt4kH+v5pZJUuI6EyDKk9ZYdu3brVUVo7duyIm61ZQWpVrzMaY8aMAVI34FAx+SuvvAJkrkQrkVSvXt3xFKXHkSNHnKYViq0f6wiljGBK0DAMw0ha4lonqDhQZsboyN/9yy+/kJKSArg7PO12/Ea+fPmcQlbV9elvcfjw4TT+bBXujhw5MgFWZp5HH33UeR2qaZRyUu3Tf//95yg/ZeBJWagw2m8qENwYk3bS6dUlRWPYsGFOjEpNgLMTyoiN9LtSfE23edWia/78+Uc9RjFA/Z6kABUrbNu2ra88LKGEnyeiEan4X9m8NWvWBNy2k14P41azBQ1+HjduXJqKAJ3rFyxYAATjvJ9//nkCrQziu3mCYvny5U4vSk0ijja1wUvuu+++dAPuKl6ePn16RHeT35EbV4XjKoNQKje4XR7UWUWuYBW+qjuLH9FJ6Mwzz0xzm/qmqslBJLSBufvuu+NgXWzRZ9mtWzfA7dLUuHFj5xhN/njiiSeArJdEJIp27doxc+ZMwP2c5Kq/9dZbAfd76Ec0bV3lApHQZqVr166A2yGoQoUKPP7444DbbERzIM8991xPfnc6L6hpRKSpQEKNKRJxXrR5goZhGIYRgWyhBKVCvGwhFooSdaQiZs2alUoZgZtIoN2b3wLysUSvUcpJcxJ//fVXz2yKBSoLefrpp4G0/TjBdTvJ7bN8+fIEWZd5pPyUhh+JDz/8EHBfj19RKcfUqVOdwn4l26nFWnZvuKEpFCoHue6664DUMz1VUK85e/q7Tp06fP311wmzVZ4UJYplpNStbdu2gJtAF09MCRqGvS5VbwAAIABJREFUYRhGBDyfIhENrd5+m5ulmEqkGKXaPF1++eVAchS2RprblhNQYpfaNCkpqGjRos4xirP5pWl4OEoa6du3LyVKlIh4jBrA16tXz3dNDcKRAlRTiTx58jhNGdTaza9JMJmhSZMmzjxOxZsjlZWpabhaBEoJLlq0iLJlyybCVOrXr+98HuFTH+Qpefvtt51cAb9hStAwDMNIWjxRglWrVk2TLquUdbXHeeqpp5zMJxVq61Kthh555BEnMyxWc8qioTTkSJmAS5YsAdyZbxlRgMqmVIxxyJAhznXhqEB20KBBTgaYHyhQoADt2rUD3FKYjI6Wyi5oBx6pZZNiMZ988klCbUqPs846C3Ab0msmnVoZhqJCeGWCZqaBRaLRd0yKJ7T5hG7LCQpQ9O/f3yku37Bhw1GP1/dP31GVlyWCyZMnp1GAagCitpj169c3JWgYhmEYfiOuSlC1K6eeeiqA02z4pptuSjPgU/VJUjyhU7xVmK1pzHrcokWLOo2m460E69ev79gRKfNJ7ZzUIi48jvnQQw+laZ2mCdHpqb9Q1IjbTyoQgjWCp59+OoCj3LM6Nsdv6LNUTDBSTE3fV31/vUbfTQ3BDUWNslWX9eijjwKJaU2VVaRgNaBWmaD6ffXt25dvv/3WG+PiSIkSJZzf1dSpUwF3qLIyeMFtYiGVFa2mNdbI61WtWjWn6YTa06klpJTpTTfdlDC7MktcFsFSpUoBbud5lThEQqUDSoL56quvALdb/NGI93wwsWnTJt577z3AXdRD0Yes5Ilwl2D58uWPKXnCr9M1QqcOzJo1y0NLgpx33nlOSUM4mvkXWrqxe/duwD15aHN21113OSncGdmk+IWHH3443ds0uWXevHmA9/Mej0aJEiWckhN9/+Wyvf766wFYunSpN8bFmfXr1zsdYlTyoZKJ33//3TlOyS/hG2yVw8QDzUzUdy137txOAkx6YaArrrjC+b82MH5JeDR3qGEYhpG0xEUJqn9degpw3rx5Trsp7fTiMScqluzdu9dJNlCKfKQJBEr4CU/8ySxqV/Xnn38C7mxGvxE6/Xrt2rUeWhKkSJEiTiuqcELdSELuZbk6w2dfRuOVV15x2uJ5jZSsvDCRUPtBXapT/7///pvm2AkTJgBu8ow8NYksoTjttNMcBagWYDq3+C0sEGsGDhzo9EBVkwZ5KcqXL5/u/aT2J02aFDfbdP4LnXmoRBih35EUu3qJgvu986JPaCRMCRqGYRhJS1zaplWsWBGAuXPnAu6kciWWqAFsduWSSy4B3B11ZltMqbv91VdfDRCxvZF86+rm7zc0K3LdunWOmo/U5T7RtG/fPm6xSSWQ6DNp1qxZhtLXE0H79u0BN4lCyWOxQin49957rxMbjzf16tVzvlsLFy4EUseWcjr6DJUMpJh2pGbb+nx0jo3WJuxYUYmYEqvA/W1Ioas0J1S1Kn6r72oiS6msbZphGIZhRMC3DbSzA9qhtWvXzlG/SuUWalYcGo/SqJdVq1YlwMr40LJlSyDYJEDt4+68804PLQpy+umn07dvX8AtQcmqKlLmsTLf+vfvD/i7pEAxGKkGxdTCi5mPhgrPFYeS96Jy5cqpYkHxRkpC73lOa8SQHZHKkzqPFocW+/bt46qrrgKCLdQSjSlBwzAMw4iAKUEjS6jVVrdu3Zy6Sb81C5dyUxF/NDQUef369c51qqfzekr3sdCgQQPAjeGCW9sZqY3V/fffD7g1n2paoSbV7dq1822mspFY6tSpAwQVoQb/hqOp8SNGjEhYLDkS0ZSgLYJGltAieNZZZ3HOOed4bI1hGEb6mDvUMAzDMCJgStAwDMPI0ZgSNAzDMIwI2CJoGIZhJC22CBqGYRhJiy2ChmEYRtJii6BhGIaRtNgiaBiGYSQttggahmEYSYstgoZhGEbSYougYRiGkbTYImgYhmEkLb5dBIcMGUIgECAQCLBt2za2bdtG4cKFnRl+hmEYhnGs+HYRNAzDMIx4k9drA0SePHkAd55Zv379mD9/PuBOYK9cuTIAn3/+uQcWGkb2pm3btgD873//A2DRokUArFmzBnAnhRtGMuGbRfCqq64C4IEHHgDgscce45577vHSJMPIUVxyySUANG/eHIAWLVoAsHTpUgCWL1/OP//8441xMeDCCy8E4O233wZg+/btAPTq1ctZ6Hfu3OmNcQa5cwcdjzrHP/jggwAMGjQICJ7zPbHLk2c1DMMwDD8QiAIQ93+NGjUKNGrUKLBz587Azp07A2vWrAmsWbMmUKBAgYQ8v/07tn8NGzZ0vi+HDx+O+O++++7z3M5k/3fRRRcF9uzZE9izZ4/zuRw5ciRw5MgR5+8yZcp4buex/LvwwgsDF154YeDgwYNp/s2ePTswe/Zsz21M1n+VKlUKTJw4MTBx4sQ054e33nor8NZbb8X1+aNhStAwDMNIWjyPCd58880AnHjiiQCMHDkSgP3793tmk5FxHnzwQY4cOQLgXEY6pmTJkgDMmjULgA8//DAxBsaZcuXKAXDHHXcA0LRp01SXK1eu5KyzzvLGOODUU08FYPLkyRQqVCjiMZMnTwbgt99+S5hdsaJYsWI899xzgBvrjIRigoY3jBw5kssuuyzVdQcPHgTgnXfe8cIkB1OChmEYRtLimRJUZlrXrl0BmD59OuBdhlC8qVOnjqMInn/++VS35cqVC4AFCxbw9NNPA/Duu+8m1sAMUrFiRSBoK0BKSkqG7nfLLbcAsHHjRiB7KsFOnToB0LhxYyCo9qT40uOnn36Ku13RuP322wEoUqRImtuWLFkCwCOPPALAoUOHEmdYjKhbty6NGjUCoHTp0kBkj8TgwYMB+OqrrwCYO3du3G0rWLAggOMFEWeffbbzm69atepRH0eK6bXXXgPcrNfDhw/HzNZ4Ua1aNSD4OYWjrNDw82GiMSVoGIZhJC2eKcHatWsDbpH8zz//7JUpceHKK68EoGPHjkCwRks7w2DirYv+bt26tbNjuvfeewF4+eWXE2JvRsmXLx/gNi7IqZQrV86J8+mzVPwvEitXrgRg1KhRALz++utxtjA6w4cPB6Bz587pHtO6detEmRM3TjzxxHRjnZEYO3Ys4KrFt956Ky52AbzyyiuA60E4VoYOHQrAm2++CcCNN97IH3/8EZPHjjWlSpUC3HhfpUqV0hzjlzitZ4vgxRdfDMCOHTsAmDBhglemxJRrr70WgCeffBKAEiVKADBp0iTHhZg/f34AHn/8cSC1u0RfnlNOOSUxBmcSuZWi0bNnTwAaNGgAwE033RRXm+LByJEjncVPzJgxA4CZM2c613m92KVH/fr1Aff7B+6Jf/To0Z7YFAsULpB7PRQVY0dDv68KFSrE1rAIFChQAIBPP/0UyHqyn4SC3L6XXnopAE2aNGHevHnHamZcKFq0KBB58VuxYgXghka8xtyhhmEYRtLiiRI88cQTOeOMMwB46aWXANiwYYMXpsScbt26Ae4OXH0an3nmGWcnKCVYs2ZNAPr06QNAoUKFnN3633//nTCboxHeiioSctOoHZJQMkbu3LmdXboSAvyKSnSaNGniKL+nnnoKcF2efua0004DoFatWmluk3uuX79+iTQppih0kF45DriKfdmyZUCwdOKKK65Idczll18OwLRp0wD4/fffY26rvEJKbNFlZsmbN3iaViJTs2bNAGjZsqVvlWD4+w2uAlSI6Ndff02oTelhStAwDMNIWjxRgv379+fkk08GvE8hjzUnnHBCqr+Vdn7w4EGngbF2syqZCA3sKw1aCRZ+IdrOO1wBiki79vCkIL+g5AWppJUrV3LXXXcB8OOPP3pmV2bp1asXELl0Zf369Yk255gpW7Ys4BbCd+nSJdXtu3btclTc2rVrAejbty8A+/btA3DONaHo8RS7iocS/Pfff2PyODo/SAEKv8ajwfVyic2bNzsx9vQUYLFixZw8CeWKKDFw27Zt8TLVlKBhGIaRvHiiBDt06OD8X+m+OYXwXY5U0l133cXevXuB9Atk58yZ4+xi/cLRskFVuBuKYp6hmYl+R+UQYuXKlU4hvDJ1s0NMsEmTJl6bEDPq1avneEYU4wz3SEyaNCndGKd+ZzaSLXEo+/a8885Ldf3YsWP55ZdfIt5HWa9jxoxxckWEcixOP/10AP76669YmguYEjQMwzCSmIQqweLFiwNBH/dHH30EkO7uIBrly5cH4IcffoidcTFCmZ7aETVs2BCI3LYqnAULFsQsjhArVNCqnVg4ikGFcuuttwLZYweuuEpow2sINm8YMWIEAB9//DHgxq/9GCOsXr06AGXKlAEiZ+GqOF7fRR2j13ngwIG425kZmjdvTo0aNbJ8f3kpxo4dm26tqjwd4bFGP9G+fXuvTcgwPXr0ANzvoc5nq1atSnOshibcf//9AGlUILgNKlRzGQ8SugjqC122bFkn8SNa/zv1ApS7o1ixYoDbBWP//v2MGTMGgIceeiguNmcWdb5R4F1dYnLlyuXYHd4rb+LEiYD/usOAW+ge7oaSG1vJCKFEWhj9RNOmTZ1SCC1+SoJRev2PP/7oJGXo+6dj/IjchTr5REpAateuXapLLYIq41m7dq2T2h4Pt1NGUSJIzZo1ndIaXX799dcAtGnTBnATKKIRWqIT/jgDBgyIoeXxQYuF0Ob/u+++c65T8xGdbyKhCS7xRL2FxZYtWwC3XCUUNT646KKLnOu0Cf3ss88Ad8pQPDF3qGEYhpG0eNY27WjF8R06dHB269qRq9h3ypQpQFBxKMnGL0pQyLWky5o1a6YJ4C9duhQIloxA1otp48U777yTphWVdp+hyU3hSGGE3lct46TcvSR0+oPUndrcJSNSXs2bN3dUvJpYxKN04GjIRdmzZ880HogXXngByJgClCoOfRwpQBWyZ+RxEolKCwoWLOiEj8LVqhK1QssGChcuDLi/vb1797Ju3ToAZs+eHV+jcZPg5GUQoWo1HIW1xC+//MI111wDBBsBgClBwzAMw4grninB9DryKyYxZcoUtm7dCsC5554L4CTTKPW5V69ezlRsv6Ldzh133OHYLcWnHZ7fOsFr1mP16tXTTI2PVuyudkjawYbu4r2eGRbKk08+mSr2lx7hDbT9jJJAdu3aBbjx81AUZ1GcRq3DQhk2bBgArVq1AnB25jt37oyxxekTKUFCqNm8pplk1nuilPvPP/88a8YdA1LcLVq0cObs6X2Wt0t5ExlJBAkEAk4OwsKFCwF3asPmzZv5/vvvY2h9dPR5hM9O1JzYUM4880zA/ZyVHNmxY0cn3nnjjTfGzdZwTAkahmEYSYtnSlAqT0g9SNlNmTLFmYq9Z88ewN0dqYAWImcn+gn5tnv06OGoKMUvN2/e7JFV0dFMw3CffXpoh6u2cGpFJXr27BnXuW1Z4WhlDv369XO8FYoX+rE0QigFfdOmTYA7xioUZeipIfhjjz0GwLhx4wCoU6eOc6yKnaUWX3zxxXiYHRE1hQ5vEwZwzjnnAJHbnSkzUXGp0GYN/9femQdMNfb//5U2SiKK9pSSrVKyfBN5SJHSIkUqEj15SNpIJU9SHkvaJFvoTou0iAjZIktUHlvpQYmkooVClu7fH/N7X2dm7rmne5mZc+aez+ufqTlnZq5zn+W63p9V0cz6u6TyuaFxqKt9tFrKDV1v0VYzWVwWLlyYqCEmDUWXP/300+7voNJoem5MmTIFCEWGKj5CCfR6Rv7+++9JG6MpQcMwDCNjSakSVEmxXbt2ubwWRQ2qdZBU3rx585wCVCsR2b21mhs+fLiLsAwaDRs2BODRRx8FQvZ7RbZpBZ6OxCpzpy7miriLJtUReCqGXZACw4rg7d+/v1uJB62YeSykkJQ0Hws1olV7LCWIx2tvddVVVwGpVYKx8kxVtKFXr15A7KhVRVbGivTV+PW8SSWKEN+1axcQUoJSQzqOrKwsIDJvWkpJuXPKm46Vc+c3KlYuS8QxxxwDeE0CWrZs6Z7xivEQUnmjR4920doqTKEcQvm6k0FKJ0E5ards2eJqZKrrshLF1ZW8XLly7mbV5CGzwLBhwwAYO3Zsikaed3ShamzqCr1q1aqY3bDTjeiH4ahRowqUUJ9M5IxXPVBNYrEmRaXhRKevzJ07N626SMhs9OmnnwJeEYBY1KlTB/BMqLE6hKjSh0ylqUTXix6k4JnHbr75ZsDrb9m2bVt3/SklJ/p4Jk6c6MvkJ7SY1/V42GGHucT1eGY+Bcxt3boV8J4tQUST1LJlywDv3KmO8IwZM3ItjKKKMeEolSpeikWiMHOoYRiGkbEUy44T856sLuAXX3yxC4CRqVOrHVGyZElXf3PPnj2AF1AyadIkIHi1DsFTtjKHKhz90ksvdduCTr9+/YBQAEX06lpKXSbpPn365Nhn1qxZgH/1GBVmLcuBlJxqgJ5++uk5gg1kclJpu3RQf7GQWnjmmWeA+F0lclNOAAsWLAC8IIxUolqmCoyD+GPNbR+ldXTp0iWQJsS8oqR3PVMUWBO01CrwgumkvJUKklfUdUfmYvVjLSzxUrtMCRqGYRgZiy8pEs8++6yb8ZWYHV1u56effnL+Jyk/hRgHjeLFi3PZZZcBXnqBHMVy0qeLCoTYHeH174ceeihi31j7jBw5MtlDjIvCrOWDkX9M6m/u3LkuWV6KKR16BeYFBSGp6ETfvn0BuOmmm1xprXisWLEC8Lqh+EG8ZPl4yGKkBHIlxsfqYJBOyKeYLMtcIpEVplWrVoCn5tu2bet80UL3oPyIs2fPdsFD8RorJBpTgoZhGEbG4otPsKjRuHFjl7iqJN7hw4cDnopNJ7p37w7A5MmTnXqI54tR1K+ieBXZFbSC4JlMuXLlXES22hDt3bvXbYNQxKWiQf30N8nnNXv2bNeSLJ5PUFHISqGKtlakO7JoyKoUZJ9gbhx99NHu/NSuXRvwComkwl9rPkHDMAzDiIEpwUKgYrdvvfWWK/umyDaVplJx2HSkT58+rqRRPCWo4rmGkUhq1qzpcj6bNm0K5IxS3rx5s8sr9KPtUyooCkoQvNxOWZdUmi/esyVRxFOCNgkWAJk8Va3mpJNOcv/u2bMnkL4h9tHIrKtgF3UrCK/q4WcismEUdbTAVkUcpfqkMngk3TFzqGEYhmHEwJRgPlAi6NKlSwGoVKkSANOnT3dh+UFM4DcMw8hkTAkahmEYRgxMCeaREiVKuI4Q6muoot8K/TUMwzCChylBwzAMw4iBKUHDMAyjSGNK0DAMwzBiYJOgYRiGkbHYJGgYhmFkLDYJGoZhGBmLTYKGYRhGxmKToGEYhpGx2CRoGIZhZCw2CRqGYRgZi02ChmEYRsZik6BhGIaRsdgkaBiGYWQsNgkGiEGDBrF79252797NgAEDGDBggN9DMgzDKNLYJGgYhmFkLCX8HkA0ixYtAqBt27Zcd911ADz44IN+DikhnHDCCZQoEfnn/uWXXwD4+uuv3XsHHXQQAFdccQUA48aNS9EIDcPwmwMOCOmSKVOm0KRJEwC2bt0KwNq1awGYN28eAGvWrGHHjh0+jLJoYUrQMAzDyFgCowRLlSoFQJkyZQDYt28fN954IwBPPfUUAD///LM/gysAhx9+OAATJkwAoGPHjq4jvXpbbd++HYDTTjstx+c3bNiQglEa0dSsWROA6dOn07x5c8A7X+qvuWbNGgBatGjBtm3bUj7GrKwsAL755hsAFixYkK/Pb9y4EcCXsRvx2bdvHwDjx49n9OjRALRp0wbAPT+6desGhFTjGWecAcBXX32V6qEmhdtvvz3i/yNHjtzvZ8455xwA3njjjQL9pilBwzAMI2MJTGf5Qw45BIClS5cCOHs4QJUqVQDYsmVLysZTUM4880zAW9G0aNHCbdPfM/pPfvLJJwMhxXvUUUcBcOyxxwIE1uavY6lcuTKdO3cG4JJLLgGgTp06AJx++umApzyCzHHHHQfgVt8XX3xxjvMV/f9XXnmFCy64INVD5YMPPgCgcePGbjz7G2uxYsXcv7/99lsAfvzxRwB69OgBeAo3HShbtiwA8+fP5/zzzwc8FSX0vJg2bZp775FHHgE8Fe03hx12GOCpGVmQjj/+eG644QYA9u7dC0C7du0Az0r02muvsWfPHrd/uqJnZV5UXzzizVfWWd4wDMMwYhAYn6BWcX///bd774477gCCq4bCufDCCwF4+umnATjwwAMjtj/44INupXLWWWcBcN555wEwZMgQAOrXr+9W50E75mrVqgEhhQTQtWtXAJo1a5Zj319//TXiNcgoCve+++4DPIWxevVqpxrmz58PwCmnnALA4sWLAf+UU9OmTQHo06cPELIayH+ZF3SMUpLyMer4goiipnXvzJ07FwjFEOiZ8cMPPwBQvHhxACpVqgTALbfc4r5HVorLLrssBaOOTcmSJZ3q6du3L+Apwljo2BcuXAhA9+7dAdi1axf16tUDvONKJ9+grGT5VYDy/b355psJGUdgJsEaNWoAcOqpp+Z4748//vBlTPlBD0ZN5h999BEArVu3BkJBCDrpv//+OwCTJk0CvIuhePHiLFmyJFVD3i8NGjRg6NChAHTo0AHwAphkkpk8ebJL/fjnP/8JwMsvvwx45rYgo8lPQSI6hgULFlCxYkUgFNQEXkCCJsUxY8akdKzRPPTQQwX6nFwN77//PgCff/55wsaUaLT4uvvuuwG49NJLI7bv2LGDm266CYAZM2YAnmtFZrZ+/fpF7O8306ZNc4tIpT3cdtttADzzzDMAlCtXzrkXNInr3qtQoQIQWmgrfSydJj/x+uuvR/z/jTfeyDGxacIraNBLXjBzqGEYhpGxBEYJpjtSgHLArly5EoDdu3e7fbSakXKKVn2LFy+mV69eyR5qrsg5r0CCI4880pl1ZRrUalvH9+uvv9KoUSPAU1GffPJJ6gZdQIYNGwbg1J7UXXi6gYJljjjiCABncpQCSwelG4sPP/wQ8K5ZmdmCxsEHH+zGpmtMaUU6X5MnT+bTTz+N+Fzt2rUB6NSpk3tP99qtt96a3EHHoWHDhkDIlSBz7uWXXx5z3y1btnDXXXcBMHXqVACqVq0KwMSJE4GQhUwKOajIytWiRYsc6Q+FTW1IFKYEDcMwjIwlMEpQdv105d577wVg4MCBAE7RKXT5+uuvd4ExTzzxRMRnN2/eDHh+Ab+Q4pE/c8+ePc5HoXJ20WHosfjtt9+SNMLE0b59eyB+6PSyZcsAnF9U+8qPk25I/eoc3nnnnYCnqoLCwQcfDIQUkBSgVLfScd56660cn5PPTAF1Uk6AU1U7d+5M0qj3j1IdsrOz3XHJfxmvEIhiIhREI4tE37593bMjaMjfF54iFq0E/VaAwpSgYRiGkbEERgmWLFkyx3tKBE0Hhg8fDoTSHMArdaSE8SVLljglqIRYreKUdiAF5hfyU+g1r0RHScqHEWSUIP7ZZ58BXui9xr5gwQKuvfZaAJeMrSRzlfFLJ0aPHu0U7apVqwDPtxQ06tatC3hpOAC9e/cGYivAE088EfD81kohCRqyINx///0MHjwYgC+//BKAESNGAPD4448DkRHx48ePB6BVq1aAF6Wc3/s0FYT7AMP597//nfrB5BFTgoZhGEbGEpiyafKTKREUvBVeOpVzKl++POBFoclHCN7fU61RVHLLbwVYWFavXg140W/KY/LT/5JX5POUjzC83Fh06TEdZ1CVRizkBxw1apQ7DpXmC2p0qxLZs7KyXAK8yhFGF5a/+uqrXam7cuXKAV7SfK1atYDQeVMkYni0tl8UL17cRR4rz09FJ5Rje+WVVzofoFrKyaoU5JzA6OlEfj/9/f0inu/fd3NodO08sX379rSoOBLNrl27AHjvvfeAyIWEeoXpJk33yS8ahd6rT2I6oIRkJcSrKED9+vVdZRXVcZW5LR1QoNmoUaOAkClXC8ygTn5CCxLwihhEPx9UV7NatWquoooWzTItahJ85513AjH5ib///tuldSjI5ZprrgHgnnvuAeD77793+6uyVJAnv9yQWfT11193ifCpSIDPD2YONQzDMDIW35WgemSpPJJYvnx5YCq95weVpFJJNMnwbdu2OVVYvXp1wFu5Rif7phPVqlVzph2lUYTXf00XlCYQni6g44hnSgkqUq8ae3Z2tjtPepWbQcowKG4H3TudOnXipJNOAnJX4R9++KHrrqBAOh2fVGQ6BGrp+BT8IssEeGlWr732WuoHlk8UABNdD7RFixY5aoVq3+jUiVRjStAwDMPIWHxXgkUF9QR89tlnAS/4QGkQbdu2dSWSBgwYAIQS6MErN5aOXHzxxS5JWSv4okDHjh2dcpdSevjhh/0cUr6YOXMm4HXyaN68eY5OGbF6DyoZ3c8E+q+//hqAsWPHOj9mtKVI6R0jRoxwClDqSUX45XMPisKNheIEpFblk+7du7frcKJj/euvvyL2DSJSddGvkFMd6v9++whNCRqGYRgZi+8pElJMmzZtinj/ueeei4gSCzrqYp1bIvyqVatcsVspQfkCFfkWhDYv+WXmzJmuvY0KF6dDJ/nckD9pxYoVlClTBvBSWRS+nq6okEPNmjUj/q8Ixfbt26dlGohSI6JTcpRkHp6mFDSU+K6ejiplN2LECHdc7777LuAdZ8+ePYHgRFfmlWifYHRCfTLnG+ssbxiGYRgxMJ9gApg2bZorPq0Vh/x9KlEVC63sorvQpxOVK1fm448/BtJbAQqppDJlyrhzl+4KUKhsl15feuklAL744gvA80elGy1btgS8e08J9UpEDyJ6XkT70cP9zsq3vfrqqwF49dVXAa9AeIcOHQKf8xmOlGt0aTW/FW1gJ8F4VdWDgpz2PXv2dA7uCRMmAF6ATDgKSJDsl+M+qJXgMwmZQZ988kkg9ED1u3N8qgjvqJGOxxzdfUUVZFSXM4ioHu2hhx4KeInwcquE8/777wPQv39/wJsou3XBx+vwAAAgAElEQVTr5p436YAmvbPPPtvfgURh5lDDMAwjY/FdCVaqVCnm++reHWQUMJGdne2qvr/yyisx9z3hhBNcKoTMhjKZpiPqg3baaafx9ttv+zyawqPwenWa37ZtW0SX+aKIumTodc2aNWl3zEcddZRLqNc9qO7zQUadMoSed+HdI6J57rnnIvZRnWK/uf322526UwJ8tImzRYsWrsdgNCqn5hemBA3DMIyMxXcleOWVV/o9hAIjPxJ41en//PNPwLP5q+RRuK9FYejp7Ats27YtEArqSeckeSk/9atTcEU6+sbyio5ZRZt1zPPmzfNtTAXlqquucv+Wj13l+9KJDz74YL/7VKlSBYASJXx/bANeInx4Enx02kM8pBatbJphGIZh+EQwlhRpyueffw7ASSed5KK8XnjhhVz3l91fPd7SGbUgAq/jejqiElQ1atQAvOjedIq6yy865saNGwNeibXoKMsgozZJPXv25LfffgNCBbfTBRVnl39PpeJiUa9ePQCGDBkCeKXW/LYkRZdByytBKZwtTAkahmEYGUvglODKlSsjXoPMuHHjAPjuu+8YNGhQxDY1mFXprXbt2uXoil0U+Pnnn13LmnRC0aDKkVOR5aLsC1RpLh2zimQr3zWdUBTrMccc4wpohzeiDTrKR1VzYFmQnnjiCbePEupvvvlmwCuGrjxIv5s8x2qbpPcULarIz6Covlj4Xju0S5cugGeS0cQyePDgpP+2UXA0aZQuXdrVDE0XypYty4oVKwAvcEkPFnUlT3fUWb5Zs2ZAqErM0KFDAc/Uqwk/HaqOHH300QDMmjUL8Lq2FC9e3AXXzZgxw5exFQYF0On5V6FChRz7aMLX+Vu3bl2KRld0sNqhhmEYhhED382hc+bMiXg1go0q8qsDQToUNYimfv36rvO6zDV+m5YSzbJlywDPlLZt27a07oZxyimnRLyKiRMnpqUCFDoXMn0aqceUoGEYhpGx+K4EjfRC/RLFM88849NICs7KlSsDk3CcLBRYpn6dRQ0FnhUVH67hH6YEDcMwjIzF9+hQwzAMw0gmFh1qGIZhGDGwSdAwDMPIWGwSNAzDMDIWmwQNwzCMjMUmQcMwDCNjsUnQMAzDyFhsEjQMwzAyFpsEDcMwjIzFJkHDMAwjY7FJ0DAMw8hYbBI0DMMwMpaiXUrfKDQHH3wwACeeeCIAl1xyCQA///wzEOrwXblyZQCmTp0KwPTp0wHYt29fSsdqGIaRX0wJGoZhGBmLL10kGjduzOOPPw5AgwYNAJg3bx4AV199NQC7du1Kym+nirPPPhuAJk2aADB8+HDKly8fc98DDgitRapVq8amTZtSM8A41KlTB4A77riD1q1bA3DooYcC8PvvvwPw119/AVC2bFn27t0LwIEHHghAy5YtAXj11VdTN+gEoeuxWbNmPPDAA3H3LVasGDt37gTg//7v/wBYs2ZNcgdoGEa+sS4ShmEYhhGDlCrBRo0aAbB06VIqVKgA4FREyZIlAWjXrh0AL7zwQkJ/O9n06tULgJEjRwI41VeuXDkg/kpEf+fNmzfz999/A/Dkk08CMGPGDADWrVuXhFHHZsmSJUDIp/fll18C8NNPPwHw7rvvArB27VogdJy//fYbAM8991zEtg4dOqRszAWlbt26gDfWf/7znwDUqlUr7jmL5vPPPwc8S8aKFSsSOUzDSHuqV68OwDXXXBPxfs+ePQGoUaOGe69fv34APPzwwwDccMMNQOj5+tlnnwFw1llnAfDHH3/s97dNCRqGYRhGDFKqBDWbT5gwgR07dgAh/yDAkCFDAOjWrRsADRs25Jtvvkno7+eHKlWqAPDjjz8C8Vcb5557rvNpKppS6G+YFyUYa59BgwYBMH78+LwOvdBoRbZx48Z8fW716tUAHHvssQAcddRRgBdJGiQOOeQQAF5++WUAmjZtGrG9WLFi+VKC4uabbwbg3nvvLeQI94+um+uvv36/+2RnZ3PEEUcAIf80wE033eS2ASxatMjXey5RHH/88QDceuutAHTt2pVzzz0XgDfffNO3cYVz8sknA3DRRRcBnvLROcrOznbnacyYMfv9Plmchg4dCoR823feeSfgWW/8QHECXbp0cefjmGOOKdR3/vnnnwAcfvjhAOzZs2e/nzElaBiGYRgx8C1P8NFHHwVwK88tW7YA3gr98ssvZ+zYsf4MDmjTpg3g+bl++OGHXPf94osv+M9//gPA7bffDsCvv/4KeMezc+dONmzYAHh+vrZt2wLQokWLHN/51VdfAbB48eJCHEXByK8CPO200wAvl3D+/PkA/PLLL4kdWCFRhOv48ePdCvywww7L8+flv5YVQ0o3VVSsWBGACy64AIBhw4YBXjRvLGJZGfTv+++/P+L/Q4YMISsrC/CuUfk6g8qRRx5J2bJlAfj6668Bzx9bpkwZAAYMGLBfBdi5c2fmzp2bxJF6ZGVl0aVLFwCKFy8esS08t3bUqFEALF++HIitYnVNv/jiiwCceuqpbpv290MJVq1aFfAsLfXr13fb9FxQhoCei8cdd1wOf2E0y5cvdwo5LwowL1iyfC488sgjed73u+++44knngBg69atAC6gRGaPZcuWOXNwPFOZLgg9pP/3v//la9yppmzZsu7YFTzzr3/9C4hvgvCDTp06AdC9e/cCfX79+vUAjBs3DvCc9qlC16SujURTuXJl55bo2rUr4KUuaXGXqAdPYSlRIvTomjVrllsE1KxZE8BNinoAP/bYY/v9Pk2YyeCUU04BPNfGJZdc4hYnCiLTolvul7p163LmmWcC8Pbbb+f63ffddx8QOflB6NgnTJiQqEPIM7lNfuvWrWPixImAF/QoAVS6dGkAtz0WCr4bO3Ysy5YtS+iYzRxqGIZhZCyBVYLxTDxBZPPmzYC36lSyvBLhly5dut9k+YceesiZn4KuAOXAf/rpp925UvCBVrNBQWXdFIodDwUhfP31165QgErF3XXXXQCUKlUqGcPcL8cdd1yu20aPHg0U7G8v85LOKXiqSsEzS5cuBeCll17K9/cnEv3tZa5t0aKFM/fJrSALhMyJfpvlFSzVsWNHAK677jqeeeYZwDOx7969O+Izq1evdoFmudGtWzcuv/zyiPe2b98OhFJ+8pI6kGgU/CIFKDdXmzZtnLk6GqU69O7dO9fvVcCkzL6JxJSgYRiGkbGkVAnKrguenfzSSy8FvNW2OP/8851ySocSagrX1WrvnnvuASKT5aN9ZB999BEA06ZNA0JKUMnyQUNBID169AA8n1GjRo3cilMJ5w0bNgRg5syZgLc69QsFF2lc4AUgaGxTpkwB4O677wZC1+qzzz4LwIgRIwAvWElpMK1bt3bnOxXo7zt48GAg0repcSjYKj+pDpMnTwZCfqigln076KCDAC8YSM+LVatWuWO/7LLLIj6Tn79BMq9RnRPxzDPPOP95flBg1OzZswE4/fTTnTJWIJ6Kdqi8od/omb9t27Yc26pVqwbggoRiocBEWSKSgSlBwzAMI2NJqRJUi51mzZo5/8w555wDwDvvvAPgwpQ7d+7sVnt5ie7ym0mTJgGeso3F888/D+BSP5SKIH9iEJFvUxGg8hWFo9Vo//79I95XuHO4AkslSt2oXbt2jm1a+R955JG5fl77RKsEhaXn5uNNFvIXX3vttYAXWditWzeXIC51qFSHwia/awXvl59X15asJVINKiPYr18/F2kpH+C3334L5PSzxUOKIxn897//BTzrV+fOnV1ZxHDrWDStWrUCvCL81113HeD5uMO57bbbgOQeR154+umnAe881apVCwg9+/XMV/EMnVPFEoQjP6+u52RGJZsSNAzDMDKWlCpB+Y769OnjlJOQGpIy7Ny5M1dccQXgKUiVywkieVEFWlVrpRpkBSi0AlOkms6FIr3kNwtHEWtKxh4xYgR33HFH0scajUpIyS8bjnyABeG8884DYq9gU4HuAxVomD17touWlHqQEswPffr0yfGeovFWrlxZoLEWlvPPPx/I6TdSQnzVqlVp37494PlqtU1trsJREX8VTlcOXjLvRRU3kG/6gQceYODAgYDXkiwWKl+o0mOxULSuVJXfKEFf0ZzKCWzXrp2zJimSOdZ9qXOmKO1URPb60k8wHnKAz5s3z4Wo61UJmEFEJorom1Xhv+qOAd6J1c0blHqGiUammTPPPDNflVkSxcKFC4GcgQljxoxxN1k8c1RuqFNErGT5VNYOTRQypT733HPO3P3WW28B3jXqV3CaKqqowlRe0ly+//57INI0KLO+6lYq2V7XRioqMykFZcKECS6FQJNyNMuWLXMTgjruKHlebNiwwZlKY034fqLnuApUjBs3zgUP5sbOnTud+TPRXYSsdqhhGIZhxCBwyfJamT/22GP84x//ALyu3UFWgjKnRHd70P9vu+02Z25SusFrr70GeCHvDz/8cL6c+UFHAU3RK9hkIzUWrr7BK2n33nvvFUgBCh1Xs2bNuPLKKyO2+WE9KSxSzApigJDJDvxPT1LKkPo8aqwKMqlatWqOc6D0lwsvvDDH98n8qSAjlTdMBQou6tatm1OFsQLNIBT0JFdEs2bNAHKUC5swYULgFKDQ/SUz/a5du9y5i0a1eHv27OlLH1lTgoZhGEbGEjglKObNm+dK8Fx88cUArj9WkANkcmPUqFEuyEBlqlQ0V4n1c+bMKVJKUJQsWdKtfFMRai/7f7QfYNGiRUDi/D/79u3L8RtBKxoeDxU+UMfv7Oxsp5BU1isoqLyYArH0OnLkSLePgp3i9VfMjVatWqW0JJzug7zcD9EpPira8NRTTyV+YAlGxcyji6GEo7Q4PzrmgClBwzAMI4MJrBIEr1SOfGZSE+mQWhCLDz74APCSyBV2ri72N9xwA7fccos/g0sCOl9//vln4IpqZzIqv6W2SSVLlgRCUZXRkbRBRUUOhg4d6u6rgihAIXUVJBSZGx1nIH9tQUqvpQq1plK6lNLdwItkVXqIWin5hSlBwzAMI2MJtBKMpkGDBkD6KkGhiLdo/5/KKxUVlIuXSho0aOAiCRONVqxqjhzexkb+Xq3Sg4yKUKs1k/yY06dPL3SZtVSh/NtSpUq5xr+FIZVRonlFTXhVpk+l4mbNmuXbmPKKClUorgO8yF5tU+F2NeL1i7SaBFXlwe+eZoVFHRh0PKJdu3a+XuBKIFbfvKFDh+YrCEmJzaoGJHNoKqvFfPzxx0ydOhXwarQmCk1+qtQSjqohFSb1ItkoyV/HoT6W69evB9Ij0EJVYbTQ2bRpk6tEUpQoV66cMymqQ4QC6JTqE0TUcUV1TsXgwYNdwQPVDg0KZg41DMMwMpa0UoJ+I+euKsBrJV2tWjU2bdoUsa/KNKmsUXZ2tluBRyfIqi5nssx4eUVJ7QMGDABC5jJ1FpcpJhYK4VYZMRU5+OSTTwBy1IlNN/r16wd4XQrCUQm8oJtBK1as6LpPyPyppHL14QtqL8FwdAyqMdy3b99AK6OC0r59e9d9Zfny5UBw6oPGQuZpPS9US1ml6x555BF3rygwS6UmVebOL0wJGoZhGBmLL0qwZcuWPPjgg4Bn35ZdP1ZR4iDQpEkTN+bolfRFF13kes6pAKxWRuGd5UV0QvXEiRMB/0tUffjhhwB89913QKjslIory5mtVAepxurVq9O5c2fA89cowEeFz4OSHqGE3ebNm7ugFnX0iKZBgwZO3SmFRYEx6tq9e/duVzD9jTfeSNq4E8EFF1zgrBJCxQOk2IPMiSeeCHg9K5UsL6tMUUHnaNy4ce69RAT+JJOyZcu6a0nPO1nG1E0ivB+gUj70bPErSV6YEjQMwzAyFl+U4HnnnefUkMJ/VfJI4bJZWVkxOyj7xbp165g9ezYAV111VcS2vPqDZBOXjV9Rk++//36ihlkolLKhcm5PPfWUW4HPnDkz5meKFSvmzuWrr74KeEnYW7ZsSep4c0Ph7j/88APgFSyXn6J8+fJO9eZG+HEJnT+lGATdDxiOxhyO+j0q+jDIyF+ua0qtqqTKiwrVqlUDQu2TVCpOxQCCysCBA50C1LWkSORwBajnZocOHQDPF6ioUb/wpZ9g7dq1nRnq2GOPBUKOU4DTTjsNCFUTUMi9TI3Ka/LLvCaz2GeffQbAIYccAkSaN5UDqBOs4JlRo0a5RrSvv/56agZcSI477jiXzqHgEF3kq1atAkINXWXO0CShv4Hf5NZPMC/EmgRvvPFGwMtvSgdUl7Fjx47uPZnX1FFhw4YNgGceDRK659auXQt47hLl0BU19Bzs1auXu6+iu6EEjbvuustV9dKY1SFHdOzYkVNPPRXwKhSpJvTzzz+f9DFaP0HDMAzDiEFgOstLMQ0cOBAIJV0q0EKrvnBnsZ8o/UHKoF27dkyYMAHw+pSp55zhH+re/e677wKecs8LxYoVIysrC8Al38sspZqH6YBUeazbXPe3+nUGxSwfjlwG9erVA7xE/6KYFgFegYchQ4bQqlUrwAu6Cup1F64E84KelfpMKixHpgQNwzAMIwaBUYKGYSSeeEpQyddKOwhagMzhhx/uLCvyTaeLP72gyE82f/58955q8KokWdAoX768i9uIRu9PmTLF9ahUTIVSzFKBKUHDMAzDiIGVTTOMDEMrcpUgCyqPPPIIo0ePBoq+AhSKgp0+fbpLkQh6fMGuXbtcJH86YkrQMAzDyFjMJ2gYhmEUacwnaBiGYRgxsEnQMAzDyFhsEjQMwzAyFpsEDcMwjIzFJkHDMAwjY7FJ0DAMw8hYbBI0DMMwMhabBA3DMIyMxSZBwzAMI2OxSdAwDMPIWGwSNAzDMDIWmwQNI4+ULl2a0qVLs3DhQhYuXMi+ffvYsGEDGzZs8HtoRhGmevXqVK9enXXr1rFu3Tqys7PJzs5m69atnHTSSZx00kl+DzGtsUnQMAzDyFh86yJx+umnA3DyyScDMHDgQABq164NwDnnnMObb76ZtN83Eo86X//73/8G4NNPPwVg7NixzJo1y7dxJYomTZoAsGLFCvfexo0bATj66KN9GZNo2rQpAB988EGhvueSSy4BcPfetm3bCjcwo8DUrFkTgJdeegmAunXr5tjnq6++AqBevXqpG1ghqVixIhDqFwnQtm1bt23kyJEAro9korAuEoZhGIYRg5QqwQoVKgAwefJkzj33XACOOOKImPvu2LGDTZs2xdw2fPhwAJYvX8727dsTOkbDW4XNnTuX//73v/vdv3Tp0gD8+OOPAJQpUyZi+7JlyzjnnHMSPMrUUaJECQBmzpwJQKdOndy2L774AoDjjz8+9QP7/7Rq1YoZM2YAsHXrVgAaNWoEwJ9//pnn7+natStPPPEEAM8++ywAXbp0SeBI/UOWpltvvRWApUuXAsE9vpYtW7JgwQIADjrooIht7777LgBnnHEG69evB+CYY45J7QALQN++fYHQ9QrQpk2bXPft168fAA8++GBCftuUoGEYhmHEoEQqf+wf//gHkLfVV4UKFTjssMNiblu4cCEACxYs4IorrgDg999/T9Aok8Nxxx3HjTfeCECHDh0Azza+Zs0aIORTmz9/fsrHpr9zu3btAG/F1rt3b6pXrw7kTVFEK8CigvwU4QoQ4Msvv3Tn0g9at24NwPTp052VRa8FseLs3buXffv2ASElAvDkk08C0LNnz0KPN9UccEBojT99+nS6du0KeEr5zjvv9G1c8ZBvecKECTkU4EcffQR452bWrFmceOKJqR1gPvm///s/AB5//HGOOuooAA4++GAAd63F4u677wbgwAMPBELWQ8ifZSOvpGQSPOOMMwB4+OGHE/q9HTp0cBPLf/7zn4R+d6IYNmwYALfccoubJCTN9XrssccCoQeO3pMpJBXo96dNm5ZjWzKDo9IFLQ6imTdvnjOHppKyZcsCXgDS4Ycf7h4oU6dOBeCvv/7K9/cuWLCA77//HvAexgpgK1euHAC//PJLIUaeWho3bgzA5Zdf7t7T3+zjjz/2ZUz7IysrC/DuSYAtW7YAcP311wPw22+/AXDVVVdRsmTJFI8wb5x99tkAzJ49G8jd7ZUbelZqMhT3339/AkYXiZlDDcMwjIwlqUrwkEMOAWDx4sUAlC9fPs+fXbx4sTPTSUnG4uabbwbggQceAGD37t0FGmthkWlTynTo0KGAp6TWrFnjQp2l8hRer5D7ihUrutVrKpWgERs57hWqLn7++WfAM9GkmkmTJgFwyimnuPemT58OwA033JDQ31LAhUzBCpzxm2uvvRYIPRsGDx4MeIFZMrfpbwKeWXfOnDmpHGaeUcBO+DmVAlTaigJixI4dO1I0uvwjN0F+FWBu6HybEjQMwzCMBJJUJXj44YcDeVOAcniOGzcOCK2M5ORXeL18ioceeqj7nL5bTnA/qFixIi+88ALg+SEU7NKjRw8A1q5dy6+//gqEgmQArrnmGsD7O82fP5+xY8embuBGXKSq5A9T8FX79u0BnP8sVejauuiiiyLe3759u7OEFHWU+nHfffcBoefGbbfdFrGPUlnq168PhFRg//79Adi1a1eqhpon5G/WMYT7+J5++mkA3nnnndQPrICoYEY8i0RentXR+0jdH3/88Xz++eeFGGGM30rotxmGYRhGGpFUJXjHHXfked+77roLgNtvv929p0T4efPmAd7K+6yzzgJCYduKotLqPNwPkCpGjx7tVulahXbv3j3X/Zs3bw54fgBFhL788stOLQYFJbY+99xzPo8ktYwbN84duyIv3377bYCUl/NTuTYleMvXLq677jpWrVqVkN/SMUaXgVNKgV8+QVl/pOgUIfvSSy/x7bffAl5ZsTPPPBPwVN9NN90UOAUodC6jozyXL1/uIsvTCT2/46U/iPzso/SK3r17M2DAgIIPMAamBA3DMIyMJSlKUNGcimqKhfJ1xo8fD5AnBaToKL2eeeaZTgnKX7hu3ToA3nvvvYIMvUDUr1/fqbl4CjB8f8iZL+hHojx4K+dYaNX1+uuvA170raK/atSokWseXToiK0OXLl3cedmzZw/gXaupRiUGoxXg6tWrAXj++ecT9lsqlKD2PPLBFS9ePGG/URCkMORj/+mnnwAi/IGvvfYa4KlGRWjv3LkzVcPMM9WqVQM8ZRvNqlWr3HWXDlStWjXu9p07d1KqVCnA8+8JxYNMmjTJPceVZ/jQQw9F7NujRw8XOf/WW28VfuAkaRIcMmQIkFPiK9F21apVLjFb4eaFRb+lcOno6h7JpHnz5jlOVixUk1NpFEqfUGi3XlNNdKBFOJoUtLj4+++/AS+YR3VD4xHU6hzhnHrqqQCu24XML+Cl4bz44oupHxje9SJ0ncicruTpRKDv+uOPPyLe1wOsVq1aKe2fqNB4JYpr8rv33nuBUNcMLcL0IFbd03vuuSdl48wvMrWri47QuZ0zZw69e/cG4Morr4z5Hdu2bXPJ5NHpE6mkYcOGrnZtNFqA9OnTx6WRTZkyJWIfpfzoPoPcu2KUL18+4ZWpzBxqGIZhZCxJUYLqCRjNsmXLgNzLUOWXOXPmcOGFF0a8V6dOnYR8d37Izs525sFatWoBOZPdO3TowPnnn+/2D2fMmDHJH2QMpPIU1BOPI488ssC/kyi1n0x69eoFQOXKld17SnORsvALjUnXjUz9L7/8ckK+X6o3/NgVeCKUirRkyRJnzk82nTp1cm4ThcxL8ciKcuedd+ZQyqNGjQK8oIpGjRq5Lh8KXPOb3NTd2rVrgVCnHNWGjYf6SCp4Sgn2qUDKe8aMGbleE//617+ASFfPN998A3jXc37MmosWLWLlypUFGm9umBI0DMMwMpakKEGt2qKLLye6GPOMGTM44YQTgFCB6mT8Rl5YtWqVWwnJ1i/VF142TZ2UlSSv0O6nnnoqpeMVUjoKi9fYE4X8AeHFDYKGlHu3bt2ASJUuv3Wqk+L3h5S7CivES5SXz0ldFGKhe0jBMPGoVKlSnsdZWC699NIcFgj5r2P5sXWvSVlIMZ944omBK7AfnYIiYgWpvfLKK4CXPK9gmhNOOMGpdxWqTmXfTlmSwntpRie5x1JtS5YsyfNvRH9f+/bt3XM0P98T9zcS8i2GYRiGkYYkRQnKFh/t+1LkXSLRb+T2m6mgadOmTglG+yekshYsWOBKwmmM8pH6FRW6bds2wAs7V5/H8Cit3NBqbNu2bW6VHR36rGjga6+9NmH+q0ShFaY6WEdHnE2bNs2V5vIb+ZdVEKJKlSqAF4Wt12Si1Jjzzjsv6b8lBg0a5MoRCil3WVOqVKnCDz/8AHiWjehnwJIlS1zx+nThl19+cSkfipDVc+LDDz8EvBQZSG1neaW+KaozVtK73uvTpw9Q8Gs01ncn+hlvStAwDMPIWFLaWV5+iURFaB1wwAGu87DfKKpLycbRZGVlOb+TFKAUmN9ohSnfUn6LMUsFRxfNVVf69evXF3aICUfJytHKXer40UcfTfmYckM5r7pvpNgThXxo4eXgFC0r1amE/ESVZ8sL3377rWuBJOS3DC+ILXWa6MLKfrJ8+XI6duwYc1us9kTyCSYTJbCrcXOsxgjynyviWPmd4BVIya0IwKGHHupyCWVpEuEJ9YlKkhcpnQQl2evUqcNXX31V6O+rVq1ajodYUAmvKuNXSkSykNkm0b3skkluQSCqz/n++++ncjh5omfPnoBnYtKCSybCcJYvXw54wVdNmjTJdWJXYvzevXvdewrs0iSo4Ay/UNCLJgaZ3ocMGVIkJr/o6y2WUFDqWXj91r/++gvwepImE7kMYk1+ixYtArwuEkr0FyVKlKBGjRqAZ7aOpnv37s5lJHeFzKGxEuoThZlDDcMwjIwlKUrw008/BUKhyeGozufzzz9P27ZtAfjyyy8L/DuxynHpt4OCSqU1btyYCRMmAIlLcg4KzZo183sI+aJ169Y89thjEe+98cYbQLDVrExCelXJrLwgE/z+kELWqkuhui0AAAQiSURBVD0oSAGOHDkS8JLmk9Fp3A+UCiKT43fffee2qTThddddB0TW6ZQKnjt3bkrGGYtFixY564RcK/np9KDjUQBQODK9ypSaDEwJGoZhGBlLUpSgVtNawaikmKhXr56zIc+ZMwfw+gmG+yVyQ+WEWrZs6d5TWZ6grOTVPV6V7LOzs3OUUisq5KWIdpCYOnWqc8ALOfR37Njhx5ACwyeffALAxo0bgchC4n5w0EEHAV4xAxVgGDhwoG9jSgTqzyn1I7+uCoOPGTPGBf2pEMjFF18c8R3r16/n0ksvTcVwI4hOYN+4cWO+0ryig19kFQwnlV1LTAkahmEYGUtSlKA6wl9xxRWAp9LCy3LJP6h+YCp8rTYbsfoBtmjRAvDCgcuVK+e2XX311YD/xZqlUhcvXgx4UW09evRIeGhvEKhUqZIvq9GCoPJg0cWhwVuJa5WqVIlMZevWrYCXmK2o2VQzaNAgwCsUoJY9qewXmgwmT54MeM0EpLg7d+4c8RoLRUwOGjSI//3vf8kcZtzfF2eddZbrtZlbf8RwpADbtGkT8X1vvvmmL9YyU4KGYRhGxpLUPMHff/8d8HyCyn3Rqi4c2fyVGPzXX3/lWHHI96TX7777zkWzqUSX34T7AMFLiC+q/sBSpUoVqs1SKtm1axfg5VaFowTevPikMwFF+5UoEXpEhEcrphI9D3RepDjSnc8++wzw4hpU1k0FsWM1AtAzRX7DhQsXJn2c4ege0X2kfMEGDRrQoEEDANfaLl5pM+WLq+2T8ga7du3qSwnJYtlxRpvojgwVKlQAQkEjcvLKlKbqIuG/ndvQNOH1798/InHUT4YNGwZ4KRGqviETblGlSpUqruu8ghiETGkjR45MWMX3RPDNN9+4ijFCZimZsY1goIdiVlYWADfddJOfw0k6CpQ544wz3MSvggebN28GvO4mfqGFicyZ3bt3d9uik9xjEV23N78VqgpCvEnZzKGGYRhGxpJSJRiLunXrAnDuuecCXn3Rs88+O8dqQiYDldB58cUXkz6+vNCxY0dX4/DXX38F4IILLgBSW2vRL1RiK/p8qHTS448/nvIxGenPQQcdxKZNmwAvlSq32rxG6tGzO7xsXTwlqEIHeibqc0rHSSamBA3DMAwjBiktoB0LhfjqVWVy0olWrVq54rLqH5gJClCoDFwqE1yNok/ZsmVdB5LwdCgjGOiZXbJkSZ9HUjhMCRqGYRgZi+9KsCiQnZ3twnzDI6UMwyg4P/74I02aNPF7GEYRx5SgYRiGkbH4Hh1qGIZhGMnEokMNwzAMIwY2CRqGYRgZi02ChmEYRsZik6BhGIaRscRNkYjnTDQMwzCMdMeUoGEYhpGx2CRoGIZhZCw2CRqGYRgZi02ChmEYRsZik6BhGIaRsdgkaBiGYWQs/w/rVmsjFp4MVgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# visualize data\n", "# visualize our data\n", "\n", "grid_images = np.transpose(make_grid(train_mnist.data[:64].unsqueeze(1)), (1,2,0))\n", "plt.figure(figsize=(8,8))\n", "plt.axis(\"off\")\n", "plt.title(\"Training Images\")\n", "plt.imshow(grid_images,cmap = 'gray')" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "# normalize data\n", "train_mnist.data = (train_mnist.data.float() - train_mnist.data.float().mean()) / train_mnist.data.float().std()\n", "eval_mnist.data = (eval_mnist.data.float() - eval_mnist.data.float().mean()) / eval_mnist.data.float().std()" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "batch_images.shape: torch.Size([64, 1, 28, 28])\n", "--------------------------------------------------\n", "batch_labels.shape: torch.Size([64])\n" ] } ], "source": [ "# parse data to batches of 128\n", "\n", "# pin_memory = True if you have CUDA. It will speed up I/O\n", "\n", "train_dl = DataLoader(train_mnist, batch_size = 64, \n", " shuffle = True, pin_memory = True)\n", "\n", "eval_dl = DataLoader(eval_mnist, batch_size = 128, \n", " shuffle = True, pin_memory = True)\n", "\n", "\n", "batch_images, batch_labels = next(iter(train_dl))\n", "print(f\"batch_images.shape: {batch_images.shape}\")\n", "print('-'*50)\n", "print(f\"batch_labels.shape: {batch_labels.shape}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Train Neural Net" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "# compute average accuracy of batch\n", "\n", "def accuracy(pred, labels):\n", " # predictions.shape = (B, 10)\n", " # labels.shape = (B)\n", " \n", " n_batch = labels.shape[0]\n", " \n", " # extract idx of max value from our batch predictions\n", " # predicted.shape = (B)\n", " _, preds = torch.max(pred, 1)\n", " \n", " \n", " # compute average accuracy of our batch\n", " compare = (preds == labels).sum()\n", " return compare.item() / n_batch\n", " \n", " " ] }, { "cell_type": "code", "execution_count": 119, "metadata": {}, "outputs": [], "source": [ "\n", "def train(model, iterator, optimizer, criterion):\n", " \n", " # hold avg loss and acc sum of all batches\n", " epoch_loss = 0\n", " epoch_acc = 0\n", " \n", " \n", " for batch in iterator:\n", " \n", " # zero-out all gradients (if any) from our model parameters\n", " model.zero_grad()\n", " \n", " \n", " # extract input and label\n", " \n", " # input.shape = (B, 784), \"flatten\" image\n", " input = batch[0].view(-1,784).cuda().float() # shape: (B, 784), \"flatten\" image\n", " # label.shape = (B)\n", " label = batch[1].cuda()\n", " \n", " \n", " # Start PyTorch's Dynamic Graph\n", " \n", " # predictions.shape = (B, 10)\n", " predictions = model(input)\n", " \n", " # average batch loss \n", " loss = criterion(predictions, label)\n", " \n", " # calculate grad(loss) / grad(parameters)\n", " # \"clears\" PyTorch's dynamic graph\n", " loss.backward()\n", " \n", " \n", " # perform SGD \"step\" operation\n", " optimizer.step()\n", " \n", " \n", " # Given that PyTorch variables are \"contagious\" (they record all operations)\n", " # we need to \".detach()\" to stop them from recording any performance\n", " # statistics\n", " \n", " \n", " # average batch accuracy\n", " acc = accuracy(predictions.detach(), label)\n", " \n", " # record our stats\n", " epoch_loss += loss.detach()\n", " epoch_acc += acc\n", " \n", " # NOTE: tense.item() unpacks Tensor item to a regular python object \n", " # tense.tensor([1]).item() == 1\n", " \n", " # return average loss and acc of epoch\n", " return epoch_loss.item() / len(iterator), epoch_acc / len(iterator)\n" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [], "source": [ "def evaluate(model, iterator, criterion):\n", " \n", " epoch_loss = 0\n", " epoch_acc = 0\n", " \n", " # turn off grad tracking as we are only evaluation performance\n", " with torch.no_grad():\n", " \n", " for batch in iterator:\n", "\n", " # extract input and label \n", " input = batch[0].view(-1,784).cuda()\n", " label = batch[1].cuda()\n", "\n", "\n", " # predictions.shape = (B, 10)\n", " predictions = model(input)\n", "\n", " # average batch loss \n", " loss = criterion(predictions, label)\n", "\n", " # average batch accuracy\n", " acc = accuracy(predictions, label)\n", "\n", " epoch_loss += loss\n", " epoch_acc += acc\n", " \n", " return epoch_loss.item() / len(iterator), epoch_acc / len(iterator)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "import time\n", "\n", "# record time it takes to train and evaluate an epoch\n", "def epoch_time(start_time, end_time):\n", " elapsed_time = end_time - start_time # total time\n", " elapsed_mins = int(elapsed_time / 60) # minutes\n", " elapsed_secs = int(elapsed_time - (elapsed_mins * 60)) # seconds\n", " return elapsed_mins, elapsed_secs" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "---------------------------------------------------------------------------\n", "Epoch: 01 | Epoch Time: 0m 12s\n", "\tTrain Loss: 2.257 | Train Acc: 19.67%\n", "\t Val. Loss: 2.208 | Val. Acc: 11.47%\n", "---------------------------------------------------------------------------\n", "Epoch: 02 | Epoch Time: 0m 15s\n", "\tTrain Loss: 1.928 | Train Acc: 35.99%\n", "\t Val. Loss: 2.658 | Val. Acc: 15.09%\n", "---------------------------------------------------------------------------\n", "Epoch: 03 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.684 | Train Acc: 45.90%\n", "\t Val. Loss: 2.697 | Val. Acc: 18.07%\n", "---------------------------------------------------------------------------\n", "Epoch: 04 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.530 | Train Acc: 50.23%\n", "\t Val. Loss: 3.533 | Val. Acc: 16.05%\n", "---------------------------------------------------------------------------\n", "Epoch: 05 | Epoch Time: 0m 16s\n", "\tTrain Loss: 1.444 | Train Acc: 51.98%\n", "\t Val. Loss: 3.861 | Val. Acc: 16.42%\n", "---------------------------------------------------------------------------\n", "Epoch: 06 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.393 | Train Acc: 52.64%\n", "\t Val. Loss: 4.699 | Val. Acc: 16.97%\n", "---------------------------------------------------------------------------\n", "Epoch: 07 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.359 | Train Acc: 53.11%\n", "\t Val. Loss: 5.268 | Val. Acc: 16.34%\n", "---------------------------------------------------------------------------\n", "Epoch: 08 | Epoch Time: 0m 16s\n", "\tTrain Loss: 1.335 | Train Acc: 53.38%\n", "\t Val. Loss: 4.952 | Val. Acc: 14.97%\n", "---------------------------------------------------------------------------\n", "Epoch: 09 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.319 | Train Acc: 53.84%\n", "\t Val. Loss: 5.703 | Val. Acc: 15.13%\n", "---------------------------------------------------------------------------\n", "Epoch: 10 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.304 | Train Acc: 54.26%\n", "\t Val. Loss: 5.436 | Val. Acc: 15.64%\n", "---------------------------------------------------------------------------\n", "Epoch: 11 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.292 | Train Acc: 54.25%\n", "\t Val. Loss: 5.915 | Val. Acc: 15.70%\n", "---------------------------------------------------------------------------\n", "Epoch: 12 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.279 | Train Acc: 54.77%\n", "\t Val. Loss: 5.348 | Val. Acc: 17.54%\n", "---------------------------------------------------------------------------\n", "Epoch: 13 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.269 | Train Acc: 55.03%\n", "\t Val. Loss: 5.428 | Val. Acc: 14.49%\n", "---------------------------------------------------------------------------\n", "Epoch: 14 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.258 | Train Acc: 55.50%\n", "\t Val. Loss: 5.109 | Val. Acc: 15.97%\n", "---------------------------------------------------------------------------\n", "Epoch: 15 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.250 | Train Acc: 55.75%\n", "\t Val. Loss: 5.109 | Val. Acc: 15.11%\n", "---------------------------------------------------------------------------\n", "Epoch: 16 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.240 | Train Acc: 56.00%\n", "\t Val. Loss: 5.531 | Val. Acc: 15.63%\n", "---------------------------------------------------------------------------\n", "Epoch: 17 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.232 | Train Acc: 56.31%\n", "\t Val. Loss: 5.849 | Val. Acc: 14.88%\n", "---------------------------------------------------------------------------\n", "Epoch: 18 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.225 | Train Acc: 56.48%\n", "\t Val. Loss: 5.722 | Val. Acc: 16.13%\n", "---------------------------------------------------------------------------\n", "Epoch: 19 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.218 | Train Acc: 56.77%\n", "\t Val. Loss: 6.119 | Val. Acc: 17.38%\n", "---------------------------------------------------------------------------\n", "Epoch: 20 | Epoch Time: 0m 20s\n", "\tTrain Loss: 1.211 | Train Acc: 57.05%\n", "\t Val. Loss: 5.578 | Val. Acc: 17.43%\n", "---------------------------------------------------------------------------\n", "Epoch: 21 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.205 | Train Acc: 57.24%\n", "\t Val. Loss: 5.528 | Val. Acc: 15.08%\n", "---------------------------------------------------------------------------\n", "Epoch: 22 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.198 | Train Acc: 57.58%\n", "\t Val. Loss: 5.489 | Val. Acc: 17.37%\n", "---------------------------------------------------------------------------\n", "Epoch: 23 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.193 | Train Acc: 57.58%\n", "\t Val. Loss: 5.657 | Val. Acc: 16.93%\n", "---------------------------------------------------------------------------\n", "Epoch: 24 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.191 | Train Acc: 57.96%\n", "\t Val. Loss: 5.609 | Val. Acc: 16.39%\n", "---------------------------------------------------------------------------\n", "Epoch: 25 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.182 | Train Acc: 58.17%\n", "\t Val. Loss: 5.460 | Val. Acc: 15.82%\n" ] } ], "source": [ "N_EPOCHS = 25\n", "\n", "# track statistics\n", "track_stats = {'activation': [],\n", " 'epoch': [],\n", " 'train_loss': [],\n", " 'train_acc': [],\n", " 'valid_loss':[],\n", " 'valid_acc':[]}\n", "\n", "\n", "best_valid_loss = float('inf')\n", "\n", "for epoch in range(N_EPOCHS):\n", "\n", " start_time = time.time()\n", " \n", " train_loss, train_acc = train(model, train_dl, optimizer, criterion)\n", " valid_loss, valid_acc = evaluate(model, eval_dl, criterion)\n", " \n", " end_time = time.time()\n", " \n", " # record operations\n", " track_stats['activation'].append('ReLU')\n", " track_stats['epoch'].append(epoch + 1)\n", " track_stats['train_loss'].append(train_loss)\n", " track_stats['train_acc'].append(train_acc)\n", " track_stats['valid_loss'].append(valid_loss)\n", " track_stats['valid_acc'].append(valid_acc)\n", " \n", " \n", "\n", " epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n", " \n", " # if this was our best performance, record model parameters\n", " if valid_loss < best_valid_loss:\n", " best_valid_loss = valid_loss\n", " torch.save(model.state_dict(), 'best_linear_relu_params.pt')\n", " \n", " # print out stats\n", " print('-'*75)\n", " print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n", " print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n", " print(f'\\t Val. Loss: {valid_loss:.3f} | Val. Acc: {valid_acc*100:.2f}%')\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Visualization\n", "\n", "From the above, we can tell that our model is severely suffering from overfitting by the gap between training and validation accuracy. However, to attain a better understanding, we will graph our recorded statistics and use HiPlot, a new graphing library by Facebook, to understand the overall patterns of our model\n", "\n", "**NOTE**: If you do not have HiPlot installed, go to their [github repo](https://github.com/facebookresearch/hiplot) to find latest installation" ] }, { "cell_type": "code", "execution_count": 140, "metadata": {}, "outputs": [], "source": [ "# save statistics\n", "# track_stats = torch.load('ReLU_stats.pt')\n", "#torch.save(track_stats, 'ReLU_stats.pt')" ] }, { "cell_type": "code", "execution_count": 141, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
activationepochtrain_losstrain_accvalid_lossvalid_acc
0ReLU12.2567060.1966782.2078360.114715
1ReLU21.9282050.3599412.6582920.150910
2ReLU31.6841900.4590052.6972130.180676
3ReLU41.5303930.5022653.5326500.160502
4ReLU51.4438330.5197733.8613900.164161
5ReLU61.3931920.5263534.6988390.169699
6ReLU71.3585550.5311005.2681260.163370
7ReLU81.3351250.5337654.9521650.149723
8ReLU91.3187270.5384305.7034500.151305
9ReLU101.3039470.5426445.4363790.156448
10ReLU111.2915080.5424945.9153890.157041
11ReLU121.2787630.5477085.3475360.175435
12ReLU131.2690610.5503235.4276310.144877
13ReLU141.2579620.5549715.1086850.159711
14ReLU151.2495270.5575365.1088130.151108
15ReLU161.2399640.5599685.5308330.156349
16ReLU171.2324090.5630835.8491540.148833
17ReLU181.2248340.5647655.7223160.161294
18ReLU191.2181700.5676976.1194030.173754
19ReLU201.2108090.5704965.5784060.174347
20ReLU211.2048780.5723785.5277420.150811
21ReLU221.1980370.5757605.4889180.173655
22ReLU231.1932540.5757765.6571080.169304
23ReLU241.1905670.5796085.6088990.163865
24ReLU251.1823640.5816905.4597990.158228
\n", "
" ], "text/plain": [ " activation epoch train_loss train_acc valid_loss valid_acc\n", "0 ReLU 1 2.256706 0.196678 2.207836 0.114715\n", "1 ReLU 2 1.928205 0.359941 2.658292 0.150910\n", "2 ReLU 3 1.684190 0.459005 2.697213 0.180676\n", "3 ReLU 4 1.530393 0.502265 3.532650 0.160502\n", "4 ReLU 5 1.443833 0.519773 3.861390 0.164161\n", "5 ReLU 6 1.393192 0.526353 4.698839 0.169699\n", "6 ReLU 7 1.358555 0.531100 5.268126 0.163370\n", "7 ReLU 8 1.335125 0.533765 4.952165 0.149723\n", "8 ReLU 9 1.318727 0.538430 5.703450 0.151305\n", "9 ReLU 10 1.303947 0.542644 5.436379 0.156448\n", "10 ReLU 11 1.291508 0.542494 5.915389 0.157041\n", "11 ReLU 12 1.278763 0.547708 5.347536 0.175435\n", "12 ReLU 13 1.269061 0.550323 5.427631 0.144877\n", "13 ReLU 14 1.257962 0.554971 5.108685 0.159711\n", "14 ReLU 15 1.249527 0.557536 5.108813 0.151108\n", "15 ReLU 16 1.239964 0.559968 5.530833 0.156349\n", "16 ReLU 17 1.232409 0.563083 5.849154 0.148833\n", "17 ReLU 18 1.224834 0.564765 5.722316 0.161294\n", "18 ReLU 19 1.218170 0.567697 6.119403 0.173754\n", "19 ReLU 20 1.210809 0.570496 5.578406 0.174347\n", "20 ReLU 21 1.204878 0.572378 5.527742 0.150811\n", "21 ReLU 22 1.198037 0.575760 5.488918 0.173655\n", "22 ReLU 23 1.193254 0.575776 5.657108 0.169304\n", "23 ReLU 24 1.190567 0.579608 5.608899 0.163865\n", "24 ReLU 25 1.182364 0.581690 5.459799 0.158228" ] }, "execution_count": 141, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# format data \n", "import pandas as pd\n", "\n", "stats = pd.DataFrame(track_stats)\n", "stats" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Compare training vs validation statistics" ] }, { "cell_type": "code", "execution_count": 182, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1gAAAEYCAYAAABBWFftAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3yT1f7A8c+TpG26R7oolL1BqFA2CNoiICgOVK4LRBEVBypex0W5igxFxX1BRRTkInJB+CHiAJQNAlr2KFpmSzfdaZvk/P5ICYQWKN3j+3698kqeec5zCD35Ps8ZmlJKIYQQQgghhBCi3HTVnQEhhBBCCCGEqCskwBJCCCGEEEKICiIBlhBCCCGEEEJUEAmwhBBCCCGEEKKCSIAlhBBCCCGEEBVEAiwhhBBCCCGEqCASYIkKc+jQITRNY+fOnVd1XGhoKG+//XYl5arqVOV1jBgxgltvvdWxPHHiRCIiIi57zPfff4+maZw9e7bc6QcGBvLee++V+zxCCFEeUu9IvSNETSQBVj2iadplX02bNi3X+Vu1akVCQsIV/+BebO/evTz++OPlSrumW7RoETqdjr/++qvE7WPHjqVVq1aUdVq6yZMns3bt2vJksUQjR45k2LBhxdYfOHCARx55pMLTu1hFVs5CiKon9U71kXqn/AYMGIBer2fdunVVmq6o/STAqkcSEhIcrxUrVgDw+++/O9bt2LGjxOMKCgpKdX69Xk9oaCgGg+Gq8hUUFISHh8dVHVPb3HHHHZhMJj7//PNi27Kzs/nmm28YO3YsmqaV6fze3t6YTKbyZrPUgoOD6/y/mRCi/KTeqT5S75TP4cOH2bVrF08//TSffvpplaV7OaX9fyGqnwRY9UhoaKjjFRAQANgrmXPrgoKCHPu99tprPPLIIwQEBBAVFQXA22+/TadOnfD09CQsLIz77ruPpKQkx/kvbqpxbnnZsmUMGTIEDw8PWrZsyeLFi4vl68ImDqGhoUydOpXx48fj5+dHaGgoL730EjabzbFPTk4OY8aMwcfHh4CAAJ566imee+45OnbseNkyuNI1/Pjjj2iaxq+//kqfPn1wd3fnmmuu4ddff3U6z65du+jRowdubm60bduW5cuXXzZdV1dXRo8ezZdffonFYnHatmjRIvLz8xk9ejRg/6M+fPhwQkND8fT0JCIigiVLllz2/CU11Zg5cyZhYWF4eHhw8803k5CQ4LQ9MTGRkSNHEh4ejru7O+3atePjjz92OufixYtZtWqV427z//73P6B4U4309HQefPBBAgMDMRqN9OzZk/Xr1zu279u3D03TWLFiBYMGDcLDw4NWrVo5zlcec+bMoXXr1ri6utKkSROmTJni9F1Zu3YtPXr0wMvLCx8fH7p06cKGDRsAUErx6quv0rRpU9zc3AgODuamm25yOl4IUXZS70i9c6HaVO98+umn3H777TzxxBMsX76clJSUYvusWrWKXr164e7ujr+/P1FRUZw6dcqxff78+XTu3Bmj0UhgYCA333wzubm5AERGRjJhwgSn87344otO36dzzTLfeustGjdujNFoBGDlypX07dsXf39/R7oxMTFO58rIyGD8+PE0bNgQNzc3WrRowaxZsxxpP/vss077WywWwsLC6kTT2RpBiXpp48aNClBxcXHFtoWEhChvb2/1xhtvqCNHjqgDBw4opZR655131Nq1a9Xff/+tNm3apLp166ZuvPFGx3EHDx5UgNqxY4fTcsuWLdXSpUtVbGysmjBhgnJ1dVXHjh1zSm/mzJlOy/7+/urtt99WR44cUfPnz1c6nU7997//dewzduxYFRYWplatWqUOHjyonn32WeXj46M6dOhw2eu+0jWsXr1aAapLly7q559/VocPH1Z33323MplMKisrSymlVGZmpgoKClLDhw9Xu3fvVhs3blRdunRRbm5uTtdxsSNHjihN09SyZcuc1nfr1k3deeedjuUdO3aoOXPmqD179qjY2Fj11ltvKZ1Op7Zv3+7Y54477lDDhw93LD/33HOqc+fOjuWvv/5aubi4qI8//lgdPnxYffzxx8pkMilApaenK6WUiouLU2+//bb6448/1N9//63mzp2r3Nzc1Lfffuu4zuHDh6vo6GiVkJCgEhISlNlsVkopZTKZ1KxZsxzpDR48WLVs2VKtWbNG7d+/X40dO1YZjUbH92vv3r0KUK1bt1bfffedio2NVU8++aRyc3NTJ0+evGSZrVy50inPF/vmm2+UwWBQ7777ruO74uXlpWbMmKGUUio3N1d5enqql19+WcXGxqrDhw+rJUuWqK1btyqllPrqq69UQECAWr16tTp+/LjatWuXmjlzprJarZfMkxCibKTekXqnNtQ7SillNpuVyWRSa9euVUop1a9fv2LlvGLFCqXT6dQLL7yg9uzZo/bt26f+85//qL///lsppdQHH3ygXF1d1ZtvvqkOHjyoYmJi1MyZM1VmZqZSSqmuXbuqp59+2umcL7zwgtP36Y477lBeXl5q5MiRavfu3erPP/9UStnrvmXLlqkjR46oPXv2qHvvvVeFhoY6zm2xWFT37t1VmzZt1Pfff6/++usvtW7dOvXFF18opZT69NNPlclkcpStUkotX75cubq6qqSkpMuWjSgdCbDqqStVdDfddNMVz7FlyxYFqJSUFKXUpSu6jz/+2HFMfn6+cnV1VV9++aVTehdXdBf+4VdKqf79+6vRo0crpZRKS0tTBoNBff311077REREXLGiu9I1nKvoVq1a5dgnLi5OAeq3335TSin14YcfKl9fX8cfMqXslRNw2YpOKaWuv/56NWTIEMdyTEyMAtQvv/xy2eNuuOEGNWHCBMfylSq6zp07q0ceecTpHOPGjbtssKKUUmPGjFG33nqrY/nuu+9WQ4cOLbbfhRXduWtYv369Y7vValVt2rRRTz75pFLqfEU3Z84cxz55eXlKr9cX+3e80JUCrIiICPXggw86rXvjjTeUr6+vstls6sSJE07fyYu9/vrrqnPnzspisVwyD0KIiiH1TsnXIPVOzap3lLIHi+Hh4Y6bbXPnzlWtW7d22iciIkLdfffdJR5vtVqVyWRSL7zwwiXTKG2AFRwc7BQIlaSgoEAZjUa1fPlypZQ9WNI0Te3fv7/E/bOyspS3t7datGiRY92wYcPUXXfdddl0ROlJE0FRou7duxdbt2bNGgYOHEh4eDje3t5ER0cDcPz48cue68ImBK6urgQGBpKYmFjqYwAaNmzoOObIkSNYLBZ69uzptM/FyyUp7TVcmH7Dhg0BHOkfOHCAa665Bm9vb8c+kZGRuLu7XzH9cePG8dNPP3HixAkAPvvsM5o3b+5oDgOQmZnJc889R/v27fH398fLy4sNGzZcsZzPsdlsHDp0iN69ezut79u3r9OyxWLh9ddfp1OnTphMJry8vFiwYEGp0zln//796PV6p/R0Oh19+/Zl//79TvteWK5GoxGTyXTF78KlKKU4dOgQ1113ndP6/v37k5GRwalTpwgPD2fkyJH069ePoUOHMnPmTKcO3/fccw8pKSk0bdqUhx56iEWLFjmabwghqpbUO1LvlFZl1ztz5szhgQceQKez/0y+6667OH36NL/99hsAVquV3bt3c+ONN5Z4/LFjx0hNTb3k9qvRqVMn3NzcnNYdOXKEf/zjH7Ro0QIfHx/8/f3Jz893lOOuXbto1KgR7du3L/GcXl5e3HvvvXz22WcAnD59mtWrVzN27Nhy51fYSYAlSuTp6em0fPToUYYNG0abNm1YvHgxO3fudLTPvlKnS1dXV6dlTdOu2MelNMdcbcfcq7mGC9M/l8659JVSJaatSjES02233YbJZGLu3Lnk5eWxcOHCYp2Mn3zySb777jumTJnC+vXriYmJYcCAAVfdufVK5TNlyhTef/99Jk6cyNq1a4mJieGee+4pUyfaS5XHxevL8l24lHPlfXEaF69ftGgRW7duZcCAAfz888+0a9eOr7/+GoAWLVoQGxvL7Nmz8fPzY9KkSbRv377MQZ8Qouyk3pF6p7xpVUS9c+jQITZu3Mj06dMxGAwYDAb8/PzIyclxDHZxqfqnNHk8R6fTFfv3KywsLLbfxf8vlFIMGjSI1NRUZs+ezbZt24iJicHLy8upHK+Ut0cffZRff/2Vv/76i3nz5tGkSROnoFuUjwRYolS2b99OYWEh7733Hr1796ZNmzacOXOmWvLSunVrDAYDW7dudVq/bdu2yx5XUdfQoUMH9uzZQ3Z2tmPdrl27MJvNVzzW1dWVBx98kC+++IJFixaRk5PDgw8+6LTPhg0bePDBB7njjjvo1KkTTZo0ITY2ttT50+l0tG3bls2bNzutv3h5w4YN3HrrrTzwwANERETQsmXLYum4urpitVovm16HDh2wWCxs2bLFsc5ms7F582Y6dOhQ6nxfrXPXeWGnZrBfl6+vr+MOMNjvYD7//PP88ssv3HXXXY67dgDu7u4MHTqUd955hz179pCQkMAPP/xQafkWQpSO1DvnSb3jrDLrnTlz5hAZGcnu3buJiYlxvL766iuWLVtGamoqBoOBzp0789NPP5V4jiZNmmAymS65HeyjIsbHxzut++OPP66YvxMnTnDs2DFeeeUVBg4cSPv27bHZbGRlZTn26dq1KydPnuTgwYOXPE/nzp3p3r07n376KXPnzuXhhx8u84iSojgJsESptG7dGpvNxqxZs4iLi2Pp0qVMnz69WvLi7+/Pgw8+yAsvvMDq1as5fPgwzz//PHFxcZf941BR1zBq1ChcXFx44IEH2Lt3L5s3b+bRRx8t9gj/Uh555BFOnz7NxIkTueWWWwgJCXHa3qZNG5YuXcoff/zBvn37GD16NOnp6VeVx+eee44vv/yS2bNnExsby5w5c4qNnNSmTRt+/vlnNm3axKFDh3juuefYt2+f0z7NmjVj3759HDp0iJSUlBLvMnbu3JkhQ4bw8MMPs27dOg4cOMCjjz7K8ePHi41SVFZ79+51quhiYmIwm8289NJLLFiwgFmzZhEbG8vXX3/NjBkzeOmll9A0jb179zJp0iS2bNnCiRMn2LhxI9u2bXM0m/jkk0+YN28ee/fu5dixY3z11VcUFhbSrl27Csm3EKLspN45T+odZ5VV75jNZubPn88999xDx44dnV733nsvPj4+fPXVVwC8+uqrfPvtt7z44ouO/H722WccO3YMvV7PpEmTmDVrFm+99RaHDh1i3759zJo1yxEIRUdHs2rVKlasWEFsbCz//ve/+fPPP6+Yx9DQUHx8fJg9ezZHjx5l06ZNjBo1yum7MGzYMLp06cJtt93GqlWriIuLY8OGDY68nzNu3Djee+89Tp06VSzoFuUjAZYolW7duvHuu+/y/vvv0759ez788EPHcJ/VYdasWQwcOJC77rqLnj17kp+fzz333OMYwrQkFXUN3t7e/PDDD5w6dYrIyEhGjx7NSy+9hJ+fX6mOb9GiBVFRUaSnp5c4aeLHH39MQEAAffv2ZdCgQXTs2JEhQ4ZcVR7vv/9+XnvtNV577TU6d+7MihUrmDZtmtM+b7zxBl27dmXIkCH069cPpVSx9tePPfYY7dq1o3v37gQFBfF///d/Jaa3cOFC+vTpw5133sm1117Lnj17+PHHH2nSpMlV5ftSrrvuOq699lqn19GjR7nrrrv48MMP+c9//kOHDh14+eWX+ec//8nzzz8PgI+PD3v37uXOO++kVatW/OMf/2DQoEGOYWj9/PyYPXs2/fr1o3379syZM4cFCxaUql+FEKJySb1zntQ7xVVGvbNkyRLS09O58847i23T6/XccccdjhYQt912G8uXL2ft2rVERkbSq1cvFi9ejIuLCwATJkzgk08+cQzVfv311/Prr7+i1+sd1/nAAw/w8MMP06NHD7KyskrVB8rNzY1vv/2WmJgYOnbsyLhx45g0aZLTd0Gv1/PLL7/Qv39/xowZQ9u2bRkzZgxnz551OtfIkSMxGo3ccssthIaGlrncRHGaKk0DXiFqgd69e9OsWTMWLlxY3VkRQghRD0i9I2qz+Ph4mjRpwvfff8+gQYOqOzt1ytVNfS5EDfHnn3+yf/9+evTogdls5osvvmDr1q1MnTq1urMmhBCiDpJ6R9QVhYWFJCQkMGnSJNq2bVshox0KZxJgiVrrgw8+4NChQwC0a9eOVatWcf3111dzroQQQtRVUu+IuuCnn37illtuoWXLlvz3v/+VwS0qgTQRFEIIIYQQQogKIk+whBBC1EsxMTHMmzcPm81GVFQUt956a7F9tmzZwpIlS9A0jSZNmvD0009XQ06FEELUJhJgCSGEqHdsNhtz585l0qRJmEwmXnrpJSIjI2nUqJFjn4SEBJYvX86UKVPw8vIiIyOjGnMshBCitqjRAdbFE7DVB4GBgaSkpFR3NmodKbeykXIrGym3sgkLC6vuLDgcPXqU0NBQx3xAvXv3ZseOHU4B1tq1axk0aBBeXl4A+Pr6lurcUneJ0pJyKxspt7KRciubstRdNTrAEkIIISpDWloaJpPJsWwymYiNjXXa51yg9Morr2Cz2bjzzjuJiIio0nwKIYSofSTAEkIIUe+UNL7TxSNp2Ww2EhISmDx5Mmlpabz66qu88847eHp6Ou23Zs0a1qxZA8CMGTMIDAysvIzXUAaDoV5ed3lJuZWNlFvZSLlVHQmwhBBC1Dsmk4nU1FTHcmpqKv7+/k77BAQE0Lp1awwGA8HBwYSFhZGQkEDLli2d9ouOjiY6OtqxXB+b4EjTo7KRcisbKbeykXIrmzrfRFAphdlsxmaz1dkx+xMTE8nPz6/UNJRS6HQ6jEZjnS1HIYS4nBYtWpCQkEBSUhIBAQFs2bKFp556ymmf7t27s2nTJgYMGEBmZiYJCQmOPltXQ+qu8pE6SwhR29SqAMtsNuPi4oLBUKuyfVUMBgN6vb7S07FYLJjNZtzd3Ss9LSGEqGn0ej1jxoxh6tSp2Gw2rr/+esLDw1m8eDEtWrQgMjKSzp07s3v3bp555hl0Oh333Xcf3t7eV52W1F3lJ3WWEKI2qVV/7W02W52uoKqSwWCo9CdlQghRk3Xp0oUuXbo4rbv77rsdnzVNY9SoUYwaNapc6UjdVX5SZwkhahNddWfgakjTgIol5SmEEJVP/tZWDClHIURtIbfUhBD1yunMAmISchjS2g+d/GATQggh6iVlsYA5F/KKXuZcyMtD5eU4PmPOhfEvXPW5JcASQtQbJzPy+deaE2SYrRTabNzaznTlg4QQQghRoymlIC8HzqbB2TTU2dTznzPSIDfHOZAy50JBwZVPrNdLgFXZMjIy+O677xg9evRVHXf//ffz0Ucf4evre1XHTZgwgejoaIYNG3ZVxwkhijuVkc+kNSfQAZ1DPVgQk0yHYA9amaTTvKjbqrruEkKICymb1R7Y2IrPP1jKM4A576LAKbVo+fxnCkrop+nhCb4B4OkF3r5owQ3A3QOMHvb3opd24fKFn11cy5TjKguwcnJymD17NidPnkTTNB577DFat25dVclXiMzMTObPn1+skrJarZcdPWnBggWVnDMhxOWczixg0tqTAEyJboyf0cCEH+J4Z3M87w5piodL5Y/cKUR1kbpLCFHRlM0G2ZmQeRYyz6Iyz0Jm+vnlDPs7WWchMwOUrWIz4OIKfgHgF4DWpCV0tn/GNwDN31T02YTm5lax6ZZSlQVY8+bNIyIigueeew6LxVLu0YBs33yGOhlXQbmz08KboRs59pLbp02bxvHjxxk4cCAuLi54eHgQEhLC/v37+e233xgzZgzx8fHk5+fz0EMPcd999wHQo0cPVq9eTU5ODvfddx/du3dn586dhIaG8sUXX5Rq2NmNGzcyZcoUrFYrnTt3Zvr06bi5uTFt2jR+/vlnDAYD1113Ha+++iorV65k1qxZ6HQ6fHx8WLZsWYWVkRC1TXxmAZPWnMBmU7wxsDHhvvY/ts/2CWPSmhN8uiORCb2vfhJBIcqiPtRdCxcuZOHChRQUFNCsWTM++OAD3N3dSU5O5sUXX+T48eMATJ8+nW7durFkyRLmzJkDQLt27fjwww8rtHyEEM6UUlBYYH8qdOErPw91weeLt6vsTMgoCpqyMsBWQtBkcAEfP/D1B1MQWrNW4O0H3t6gK8fNTDcjml9R4ORnAg/PGj3wTZUEWLm5uRw8eJDx48fbEzUYauWQtS+//DKHDx/ml19+YcuWLTzwwAOsW7eOxo0bA/DOO+/g7+9PXl4eQ4cO5aabbiIgIMDpHHFxcXz88cfMnDmTcePG8cMPP3DHHXdcNl2z2cwzzzzjmJ/lqaeeYv78+YwYMYLVq1ezYcMGNE0jIyMDgPfee4+FCxfSoEEDxzoh6qOELHtwZbEp3ohuTGPf83eyOgR7cHfHQBbtTSGigScDmkkzKFE3VXXdNWTIEO69914A3nzzTRYtWsSYMWN45ZVX6NmzJ3PnzsVqtZKTk8Phw4f54IMPWLFiBQEBAaSnp1duYQhRhzj6HWVnQU4WZGfZg6CcTPu6bPu7KtqWbM7FlpttD5hKCo5K4uIKRnf7y9PbOWjy8UPztb/bX/725nY1OPCpKlUS5SQlJeHj48Mnn3zC8ePHad68OaNHj8ZoNDrtt2bNGtasWQPAjBkzCAwMdNqemJh4PjC777GqyLqTc00pzk2oeO2119K8eXPH9i+//JIffvgBgPj4eE6cOEFwcDCapqHX69Hr9TRu3JiIiAgAIiIiOH36dLFg89yyTqdDr9dz/PhxmjRpQps2bQAYOXIk8+bNY+zYsRiNRp5//nkGDhzIwIEDMRgMdO/enWeffZZbbrmFoUOHXjKYdXNzK1bGtZXBYKgz11KV6nK5nc4w8+q6PVgUfDSiEy0CPYvt8+gAEwdSC5i9I4mercJo5Fe6/lh1udxE5brck6aqEhER4QiuAL744gtWr14N2OuuuLi4YgFWeHg4HTt2BKBTp06cPHnykuc/fPgwb731FpmZmeTk5NC/f38ANm/ezPvvvw/Y61MfHx/+97//MXToUEd6/v7+FXehQtQiShX1M8rOdLxUVmbx5QuDp9xssFpLPqGms/c78vIGLx8wBeEaEEi+pjsfMLkZweiO5lh2t/c/umC7VgsfiNQEVVJqVquVuLg4xowZQ6tWrZg3bx7Lly9n5MiRTvtFR0cTHR3tWE5JSXHanp+fX6kzxV+JtehLbLFYsFqtuLu7Y7FYANiyZQvr16/n//7v/3B3d2fEiBHk5uZisVhQSmG1WrFarbi6ujqO0TSNwsJCxzLYf7idW7bZbFitVgoLC1FKOdZbrVb7f0Tg+++/Z9OmTaxYsYLPP/+cJUuWMH36dP744w/Wrl3LDTfcwM8//1yssgR7eV5cxrVVYGBgnbmWqlRXyy0xu4B//XICs8XGlOjG+JJHSkpeifs+2T2ICT9k86+V+5lxYxNc9Fe+81ZXy62yhYVJU8yawMPDw/F5y5YtbNy4kZUrVzrqrpKa8Ltd0I9Br9djNpsvef5nnnmGuXPn0qFDBxYvXszWrVsvua9SSu52i1pPKQWWQjCb7U3r8vOL3s2Qb0YVvZNvtj9xKgqcVLZzAMUFvwed6A32IOlcsBQWjuZZtOxpX6dd8BkvH/uTJJ3zdLe+UndVmSoJsEwmEyaTiVatWgHQs2dPli9fXhVJVyhPT0+ys7NL3JaVlYWvry/u7u4cPXqUP/74o8LSbdmyJSdPniQuLo5mzZqxdOlSevbsSU5ODnl5eURFRdGlSxf69u0LwLFjx+jSpQtdunThl19+IT4+vsQAS0Ch1cbWk9n0aOSFm6FWzbstLiEx294sMM9iY0pUY5r5Gy+7f5CnC0/0aMCMjadZuDuZ0V2CqyinQlSNqq67srOzCQkJobCwkO+++47Q0FAA+vbty/z58xk7dixWq5Xc3Fz69u3LQw89xNixYx1NBOUplqguymq1BzpZZyEr0z5wQ3amvb9RVgYqK8O+nG8uCqbMUFD0ubSDOGha0ZOlokAoMAStaSvw9nGs085t8/a1vxvd5UZELVMlAZafnx8mk4n4+HjCwsLYu3cvjRo1qoqkK1RAQADdunXjhhtuwGg0OjURGjBgAAsWLCA6OprmzZvTpUuXCkvXaDTy7rvvMm7cOMcgF/fffz9nz55lzJgx5Ofno5Ri8uTJALzxxhvExcWhlKJv37506NChwvJSlyil+OT3RNb9ncGgln483iO0urMkyik5p5BJa06SU2gPrpoHXD64OqdXY28Gt/Lju4NpdG7gybUNijcnFKK2quq66/nnn2fYsGE0atSItm3bOoK7119/nX/+859888036HQ6pk+fTmRkJE899RQjRoxAp9PRsWNH3nvvvXLnQdRvSin7HEe52UWvHMjNRuUULWdn2gOo7Az7CHfn3nNLvhGBprM/LfLxsz8l8g9Ec3U738zO1QjG8++am9G+3unlDm72Y7TyDPYgagVNnWtrVsmOHTvG7NmzsVgsBAcH8/jjj+Pl5XXZY+Lj452Wc3NznZo21EUXNhGsbHWpPMvSZOv/DqUxd1cS4b6unMwo4JUBjYhsePnvZF1Tl5q6JecU8q81J8jOt/J6VGNamkoXXJ2Tb7Ex8cdjZOZbef+mZvi5X/r+U10qt6pUX5oISt1VOepiOcrfkktzCpJysqAoOFI5WXiiyElOcgRQ6oIgipxsezO8y30fzwVM3r72uZG8fMDHF7x8wccXrWi9/QmSL3h6FWtuVxvJ961sylJ3VVnPtaZNmzJjxoyqSk6Iy/ozIYd5fyTRM9yLZ3uH8fxPx/lwWwIfDm2Gj1E6dFYmi02xaE8KR1PziGzoRc9wb4I8Xcp1zpTcQiatOUFWvpXXo8KvOrgCcDPomNi3IRN/PMb7WxN45fpG6KRJhhBCVAillL3pXUoSKjXZ/tQopyggysmyB0lFn8nNsb9bCks8VzaATmefRNbdEzy87EFQQJB9nYdX0TpPcPdC8zy37vy7PEUSlUl+SdYAL7/8Mjt27ADsA18opXj44Ye5++67qzlnddPpzAJmbjpNuK8bE3qF4WbQ8WzvBjz343E++T2RF/qFSVvnSpJptvDWpnj2JuYS4uXC51Zl0wAAACAASURBVLuS+HxXEq1MRnqFe9O7sTcNvK9u1vTUouAqM9/Kv28Ip5WpdCMBlqSJnxtjugQze0ciKw+lM7yd9F0U4lKk7hIXcg6gkiAlCVITL/icZJ976WJu7vY+SUVBEg3C0RzL3uDpiebpfX67pzem8Cak5uRKXS1qLAmwaoBp06Y5PldlE8H6KKfAytT1p9BrGv/q3xB3F/sj/6b+Ru7tFMhXMcn8FpfJ9c1lTqSKFpduZtr606TnWXi6VwNuaO7L6cwCtp7IYuvJLObHJDM/Jpkmfm70DvemZ7gXTfzcLluB2oOrk6TnWXnthnDaBJY9uDpncCs//kzIYX5MEh2CPcr0NEyI+kDqrvpBWSznm94VNdVTOVmQkX7lAMrLB0zB0LAxWqduEBiMZgq2r/P2tT9JMlx9CwadhydabskjwwpRE0iAJeoNq03x9qZ4zmQV8HpUY0K8nJ+UDG8XwI7T2Xy6M5GOIR7lbrYmztt8PJP3tybg6apn2sDGtC4KhBr6uDKio4kRHU0kZRey7VQWW09k8c3eFBbtTSHM24Ve4d70auxNywCjU7CVlmfhlbUnScuz8O/rG9E2qPzBFdjvxD/ZswFP/xDH25tP8+6Qpni4SFMSIUTdoPLNkJwAKYmo7CynwOl8U70L1psvE8hcMoAKsU9Ia6yYv8tC1DYSYIl6Y35MMn8k5PB491A6hhTvKK3XaUzo3YCnVtn74LweFS59cMrJphQLd6fwv/2ptAl058XrGhJwicEjgr1cuKVtALe0DSA9z8K2k/YnW98dTGPpgTQCPQyOYCvUy4VX154kNbeQydeH0y64Yju+e7vpebZ3GK+sPcFnOxN5ulf9GJxBCFE3qNxsSD6DSkqAopdKSrAHVhnpxQ/Q64ua43nbm+H5B6I1bHp+uehdu3DZ21cCKCEuQQIsUSPYlKrUYGbd3xksP5jGTa39GNTK75L7hXi5MjYymA+3nZE+OOWUU2Bl1pZ4dpzOYWALX8Z1C8FFX7pRmPzdDQxp7c+Q1v5k5Vv5/VQWW09m82PsWVYeTkengYtOY/L14bSv4ODqnI4hHtzZ0cTival0DvVkQDNpNiqEqBjKZoWMsxRmpaPS0+1zI2kAmv0zRcuOz1oJ+wCZZ1HJJQRR2VnOCfoFQHADtI5dIbgBBDVACwqxN9Pz9AY3o/RnEqICSYAlqt26vzOYsyOR+zoHMrSNf4UHWodT8vh4+xmuCfHgoa4hV9w/qrkv209lsyAmmWsbeNLYz61C81MfnMrMZ9r605zJKmBctxCGtPIrc+Xt7aYnqoUfUS38yC20sut0Dn8k5BDdwpcOlRRcnXN3x0D2nMll9u+JtAl0v+oBOIQQ9Y99sIcMSE+BtBRUWgqkJ9s/F60jIw2sVtIqKlFNg4AgexDVpbf9PahBUTAVap+XSQhRZSTAqkStWrUiNjaWM2fO8Morr/DZZ58V22fEiBG88sordO7cucRz9OjRg9WrVxMQUDefpORbbMyPSQYUn+9KYtupbJ7qGVqsf1RZpeYWMn39KUweBv7ZryEG3ZV/5GuaxvgeoTz1fRzvboln5qCmuOjlzl5p7TydzTub4zHoNF6Palxic8yy8nDR06+pD/2a+lTYOS9Hr9N4tncYE1bH8c7meKYPbFKrvwunMvMxaBqBni6l+r8g6qeKqLvqMlWQD2nJ50fLS7sweEqG9NTiw4sbXMDfBAFBaK07QkAg+Afi0zCcrKwsQIEClA377KQKVNELit7P7aMc2zUvX3sQFRiC5iL9hoWoKSTAqgKhoaElVlACVh1JJz3PwrToxiRkF/D5ziSeWnWMh7oGM7CFb7maLORbbExbf5o8i+L1qEb4uJV+oAI/o4HxPUKZtuE03+xN4f6IoDLno75QSrF0fxpf706mmb8bL13XiGCv2l/hB3u58ESPUN7cGM9/9yQz6trg6s7SVSm02th8IovvD6cTm2oG7K2MAjwMBHu6EOzpQpCnCyFe9vcgTwNBHi64GWr/pJqifOpr3aUKCyA12T4IRKp9qHFHMJWaVLwPk04HfiYICERr2gq69AL/ILSAQEcghXfJ9ZkxMJDsiyZ+lVsfQtR+tTbA+nxnInHp5go9ZzN/Iw9HXroJ2dSpU2nYsCGjR48G4J133kHTNLZt20ZGRgYWi4V//vOfDBo0yOm4kydPMmrUKNatW0deXh7PPvsssbGxtGzZErO59NcwZ84cFi9eDMA//vEPxo4dS25uLuPGjSMhIQGbzcbTTz/N8OHDmTZtGj///DMGg4HrrruOV1999eoLpJLlFFhZtj+VLg086RDiQYcQDzqFePLhtgQ+3n6GbSezGN8jFJPH1f9IV0rx0fYz/JVm5uX+DcvUzK9HuDfRLXxZdiCVyIaetAuq3OZotZnZYuPDbQlsOp5FvybePNmzQZ36gd67sQ+DWuay7EAanUI9GRgYWN1ZuqK0PAs/xqbzY+xZMsxWGvq48nDXYNxddCTlFJKUXUhyTiEHk/PYeDwTm3I+3s+odwRfwZ4uBHu50MjHlQ7BHujl6VeZ1Ze668UXX2T37t2YzWaGDh3KxIkTAYiJieHVV18lNzcXNzc3Fi9ejLu7O1OnTmX9+vVomsY999zDmDFjylcoF1E2G+TlFo2OZ385RtDLSIPUC+Zuyrio4Z5eb29+FxiCdk2kfdS8wGA0U4j9s5+/TForhHBSawOs6jB8+HAmT57sqKRWrlzJwoULGTt2LN7e3qSlpXHzzTdz4403XvLJy/z583F3d2fNmjUcOHCAwYMHlyrtPXv28O233/L999+jlGLYsGH06tWL48ePExoayoIFCwDIzMwkPT2d1atXs2HDBjRNIyMjo0Kuv6KtOJRGVoGNezuffzoU7OXCa1HhrD5yli//TOLJVXE8EhlC/6Y+V/U0a+mBNDYcy+T+zkF0b+Rd5jw+1DWYPWdyeW9LAu/d1Mwxb5Y4LzG7gOkbTnMsPZ9REUHc1j6gTnaWfqhrMAeSc3lvSzzXNm9Q3dkpkVKKI6lmvj+UzuYT9qCpa5gnQ9v4E9HA85L9G602RVqehaTsQpJy7IFXYtF7XLqZ7aeysRRFYCZ3A9c39+WG5r409JE+abVBddRdL7zwAv7+/litVu6++24OHDhAy5Yteeyxx/jPf/5DREQEWVlZGI1Gvv76a06ePMlPP/2EwWAgPb2EUe4uopSyP0lKSUSlJduDpuwLgqdzQ4xnZ0FuFuTkgLKVfDKdzh5AmYLROnaBQPsw41pRIIVfgARQQoirUmsDrMvdrassHTt2JCUlhTNnzpCamoqvry/BwcH8+9//Zvv27WiaxpkzZ0hOTiY4uORmRNu3b3fcmWvfvj3t2rUrVdq///47gwcPxsPD/hRlyJAhbN++nQEDBjBlyhSmTp1KdHQ0PXr0wGKx4ObmxsSJE4mKiiI6OrpiCqACZZgtrDiYTu/G3sUmctVpGkPb+HNtA0/e35rArC0JbDuZxaPdQ/EzXvkr+/upLL6OSaZfE2/u6FC+vmseLnom9G7Av345wbw/kni8R2i5zlfX7E3M4c2N8dhsilevb0SXMK/qzlKlcTPomNgnjIk/Hmf457/jZ9QT4uVCiKcrwV72Jnb2ZZcq7+NUaLWx8XgWqw6nczTNjIeLjpva+DO0tX+pBubQ67Si5oEudChhu00pzpqtHEzOZd1fGSw7kMr/9qfSLsidqOa+9GniLXOFlVJ9qbvOBXFWq5XExERiY2PRNI3g4GAiIiIA8Pa23/zatGkT999/PwaD/e+7v7+/PYCy2ex9mSyFYLFgO3kM69oVkJJob8J38aS2AO4e4OFln5/J08seJDkNNe59fqhxr6JhyT280PTy/RVCVJxaG2BVl6FDh7Jq1SqSkpIYPnw4y5YtIzU1ldWrV+Pi4kKPHj3Iz8+/7DnKcndfKVXi+hYtWrB69WrWrVvH9OnT6d+/P8888wyrVq1i06ZNrFixgnnz5rFkyZKrTrMyLd2fSoHVxj2dLt3UKszHlWkDG7PiUBoLd6dw4Ps4HuseSq/Gl34ideJsPu9sTqB5gJEnezaokCcpHYI9uLVdAN8dTKN7Iy8iG9bdIKK0lFL8cOQsn+9KJMzblX/1b0RYPXia0dTfyPQbG3MkA/5OOktSdiGHU/PYdMK5iZ1Og0APA8FeroR4OgdfwV4u+BkNFdLMLjW3kB9jz/LTUXszwEY+rozrFsL1zXwr9GmrTtMIcDfQp7EPfRr7kJpbyG9xmaz9O4OPtp/hs52J9G7sTVTRyI4yf1zNU5V114kTJ5gzZw6rVq3Cz8+PCRMmYDabUUo5nUMpBVYLtsJCyM1BpSaD1R5MYSm0B1gXSk2CrEwIa4LWqbu9mV5gCAQEg7ePPVAyyM8aIUT1k79EV2n48OE8//zzpKWlsXTpUlauXElgYCAuLi5s3ryZU6dOXfb4Hj168N1339GnTx8OHTrEwYMHS5Vuz549eeaZZ3jiiSdQSvHjjz/ywQcfcObMGfz8/Ljjjjvw9PTk22+/JScnh7y8PKKioujSpQt9+/atiEuvMCm5hfxw5CwDmvkS7nv5vlF6ncbt7U1Ehnnx3tZ4Zmw8Tf+mPjwSGYLXRYNWZOZbmbr+FO4GjZf7N6zQPkD3dg7kj4QcPtyWwIdDm+FTiidpddmS/aks3J1Ct4ZePNunQb16etHK5E6vNoGkpJyfYNNqU6TkFpJY1MQuMfv864+EHNLzLE7n0GngazQQ4K7H32ggwMOAv7vB/tn9/LKf0VDsSZhSikMpeaw6nM6WE1nYFEQ29GJYG386h3pUSfNMk4cLd3QwcXv7AA6nmFn791k2Hsvi17hMQrxcuKG5Lzc0860Tg5zUFVVZd2VlZeHu7o6Pjw/Jycn8+uuv9OrZkxaNGpKYkMCf638lonUrss+mY3Qx0L9zRxYsmE+v1s0xGN1Jz8nB3xRoH3nPYHC864Ibou8/6JLpCiFETVG/fyWWQZs2bcjJySE0NJSQkBBuv/12Ro0axZAhQ+jQoQMtW7a87PEPPPAAzz77LNHR0bRv397RVOJKrrnmGu68806GDh0K2Ae56NixI7/99htvvPEGmqbh4uLC9OnTyc7OZsyYMeTn56OUYvLkyeW+7or07d5UFIqR15hKfUxjPzfeGtSU/+1L5dt9KexNzOWJHqF0LXqaZLHamLnxNCm5FqYNbExgGQbGuBwXvY5nezfguR+P88nvZ3ihX8M62c+oNFYdTmfh7hQGNPPh6V4N5GkF9hsBIV6ul5xeIN9is/drKgrA0vIspOVZSC96P5pmJsNs5eLn1BrgY9QTUBR8+bsbOHY2n7/SzHi66BjWxj4Zc3XNz6VpGm2D3Gkb5M7DXUPYejKLtX9lsGhPCt/sSeGaUA+im/vSM9y7Tg16UhtVVd2llI32rVrSsW0brr/uOho3DCOyY3tITcI1LYlP/j2JV6ZOw1xQgNHozjdffsE/xo7j77NvM3DMOAwGA/feey8PPvhgZRSDEEJUCU1dqu1ZDRAfH++0nJub6+iDVFcZDAYsFsuVd6wA1VGeCVkFPL7yb4a08uORbmXrz/RXmpn3tsRzIqOAG1v68mCXYJYezuZ/uxN4ulcDbmjuW8G5Pm/Z/lS+ikmu9HSqSmBgICkXDRF8Ob/FZTBrSwLdG3nxYr+G9XY0uastt9Kw2hRnzecDr/Q8K2l5hUXv59f7uOkZ3MqPARXcDLAiJWYX8Ovf9iaESTmFeLro6NvEhzdu61LdWasS9anuUkpBXi66vBxs+WZ7v6hzPys0DVxc7S9Xt6J3V9AbynSDqi6WY2X8LakPpNzKRsqtbMLCwq76GHmCJarUf/ekYNBp3Nmx7MNctwgw8u6Qpvx3TwrfHUhj+8lsMvKt3NouoNKDnuHtAthxOpvPdiZyTYgHQZ7le1KWW2jF3aCrFU/Dfj+VxftbE7gmxIPn+4bV2+Cqsuh1GiYPlzJNS1DThHi5MrJTIHddY2JfYi5r/8rg17iaOZqpKBullH2I88x0KMjHpjfYgyh3D3BxswdSLi5oWs28CSCEEJVJAqwaYtiwYeTn56NpmmNAiw8++KDUowzWBsfSzWw8lsnt7QPwdy/fV89Fr2PUtcF0b+TFR9vO0KmhLw9UwWTAep3GhN4NeGrVMd7bmsCUqPBSN5HLNNubgh1NM3M01f5KzbPQraEnz/QOw9O15vZj2peYy8xN8bQIMPJy/4a46uVHk7gynabRKdSTTqGePFJgre7siAqgbDb70OeZ6WApZNgjj1NgtaHp9XW27hJCiKtVqwKsGtyasdy+//57oGqbCFZ1eS7ck4KHi47b2pe+79WVtAvy4KNhzQgMDCQ1NbXCzns5IV6ujI0M5sNtZ1h5KJ3h7YoPBZ9dYOXvNDOxqecDqqScQsf2hj6udAzxwNeoZ9XhdJ7/6Tgv929II5+rnxC5sh1NNfPGb6cI8XLh1QGN6tWAFqLi1OQbCJWtLtRdymqF7AzIPAtWK7gZISCQ73/6GU3TqqTuqgvlKISoH2pVgKXT6bBYLI65MkTZWSwWdLqqewpxOCWP309lc2/nQLzdKvaHlqZpVd7ELqq5L9tPZbMgJpm2Qe5YrOr8k6m0POKzzgdToV4utDIZuam1Hy1NRpr7G51+bPZo5M2bG0/z/I/Hea5PWI0aBv5URj6v/XoSbzcdr90QXu9HTxSiLGpz3aUsFsg6C1kZ9mHT3T3B1x/cjFX6d7eq6ywhhCiPWvXX3mg0YjabHU3p6iI3N7crzkVSXkopdDodRqPxyjtXkK9jkvF103Nzm/JN/FtTaJrG+B6hPPV9HP/86bhjfZCHgZYmI1HN7cFUiwDjFQPKjiEevDO4KdM2nOKN305xf0QQt7cPqPbveFJ2Ia+uO4lOg9ejGteJvkFCVIfaWHepvFxUwklIPmMftCIgCC0sHM3LB2wK8vKc9q/Muqs66iwhhCiPWhVgaZqGu7v7lXesxeriCC+7z+SwJzGXh7sG19hRz8rCz2jg5f6N2HMmh+YBRloGGPErY9+yYC8X3ryxCR9sS2B+TDJx6Wae7Nmg2oa2PptnYfK6E5gtNqZFN662YcCFqEwxMTHMmzcPm81GVFQUt956q9P23377jQULFhAQYL8xNHjwYKKioq46ndpUd6mTcagfl6J2bAK9Dq13FNqg29CCLz+KVl2su4QQoqxqVYAlah+lFF/HJBPoYWBQK7/qzk6FOzcHUEVwM+iY2CeMZv5pfB2TzOnMAl7u36jcIxVerewCK//+9SSpuRZeiwqnqb/cNRZ1j81mY+7cuUyaNAmTycRLL71EZGQkjRo1ctqvd+/ePPTQQ9WUy6qj/jqEbdW3sHcnuLmj3TgcLfoWNL+K6zMrhBD1Rd15nCDKZd3fGby/NZ7s/Iod6ev309kcSTUz8ppAGXmuFDRNY0QHE5MGNOJMdiHPrT7G/sTcKks/32Jj6m+nOJmRz4vXNaRdUN2ac0aIc44ePeqYdNdgMNC7d2927NhR3dmqFmr3DmxvvghxR9BuvQ/dm3PRjXhQgishhCgjeYIlOJZu5uPtZ7DYFAeS8vhX/0Y09iv/aHY2pVgYk0KYt0udmJS3KkU29GLm4CZM/e00r6w9wdjIEAa38qvU/huFVsWbG09zKCWPiX3C6BJWcwbbEKKipaWlYTKdDyBMJhOxsbHF9tu+fTsHDx6kQYMGjBo1isDA4nP4rVmzhjVr1gAwY8aMEvepqQr/Pkz6529jaN4a/ykfonP3LNN5DAZDrbrumkLKrWyk3MpGyq3qSIBVzxVabby3NQFPFx3je4Tyn9/P8PxPx5jQK4xejb3Lde6NxzI5npHPxD4yKW1ZNPJxY+bgJry7OZ7ZOxKJS89nbGQILvqKL0urTfHe1nh2xecwvkcofZr4VHgaQtQkJQ35ffENjK5du9KnTx9cXFz4+eef+fjjj5k8eXKx46Kjo4mOjnYs15a+SCotBdv0ieDhie3RF0nLyYOcvCsfWALpg1U2Um5lI+VWNlJuZRMWdvk+qCWRNlv13Dd7U4lLz2d8z1B6hHvzzpCmNPZ1Y8bG0yzcnYytjPOOWGyK/+5JoZm/G32alC9Qq8+8XPX8q38jRnQw8dPRs7yy9gRn8yp2rhmlFHN2JLLpeBajrg3ixpZ1r6+cEBczmUxOc+elpqbi7+/vtI+3tzcuLvY+kNHR0fz9999VmsfKpMy52D6cAuY8dE++iuZXN0Z4FUKImkACrHrsYHIuyw6kEt3Clx6N7EGQycOFaQMbE93Cl2/3pTJt/SlyCq6+X9bavzI4k13IfZ2D0NWSYYlrKr1O4/6IICb2CeOvNDPP/niM2NSy3WUuyYKYZH46epYRHUzcXoGTQAtRk7Vo0YKEhASSkpKwWCxs2bKFyMhIp33S09Mdn3fu3FlsAIzaSlmt2D59G+KPo3v0RbRGTas7S0IIUadIE8F6Kq/QxntbEgj0cOGhrsFO21z0Op7oEUqLACOf70xk4o/Hebl/Q8J9S9cvK99iY/HeFNoGutM1rGzt+UVx/Zr60NDHlWnrT/HyLycY3yOUAc3K17dt4c5TLD2QxuBWftzXWdpli/pDr9czZswYpk6dis1m4/rrryc8PJzFixfTokULIiMjWb16NTt37kSv1+Pl5cXjjz9e3dkuN6UU6pvPYO9OtPvHo3W4trqzJIQQdY6mSmqIXkPEx8dXdxaqXFW1j/1k+xl+PnqWqdGN6RBy6ZHi9ifm8uam0xRYFM/0aeB40nU5Kw6m8cUfSUyNbkzHy5y7ItWndsUZZgtvbjzN/qQ8olv4EupVtmHc081WVh1Op18Tb57pLf3krkZ9+r5VpLK0Y6+NanLdZftlBerbuWiDbkc3YnSFnVf+T5SNlFvZSLmVjZRb2ZSl7pInWPXQztPZ/HT0LLe2C7hscAXQIcSDdwY3ZcaG00xbf5p/XBPIXdeYLtnsL7fQypL9qUQ08Kyy4Kq+8TUaeD2qMXN3JbL6yFnKc4fkuhYmnuoWKMGVEPWA+nMbaskX0KU32u0PVHd2hBCizqqyAGv8+PEYjUZ0Oh16vZ4ZM2ZUVdLiAplmCx9tS6CJrxv3lrJJWJCnvV/W7B1nWLQ3hb/TzUzo3QAPF32xff/vUDpZ+VZpblbJDDqNcd1CeahrCOV5Bt0gJEjuZglRD6hjsdg+fxuatkL30DNoOumCLYQQlaVKn2BNnjwZHx8Z/rm6KKX4z45EsgqsTL4h/Kom/nUz6HiqZwOa+xv54o8knv/xOC/3b0RDH1fHPpn5VpYfSKNXuBetTO6VcQniIgZ58iSEuAKVmoTtozfA2w/dE/9Ccy3/PIdCCCEuTW5h1SPrj2Wy5UQW/+gURDN/41Ufr2kaN7cN4LUbwsnMt/L8j8fYeTrbsX3Z/lTMFhv3dA6qyGwLIYQoI5WbYx+OvaAA3VOvovn4X/kgIYQQ5VKlAdbUqVN54YUXHDPei6qTnFPIpzsSaRvozm3tyjffSadQT94Z3JQQLxfe+O0US/alkJpbyKoj6Qxo5kPjUo42KIQQovIoiwXbnDfhzCl0j72IFta4urMkhBD1QpU1EZwyZQoBAQFkZGTwxhtvEBYWRvv27Z32WbNmjSP4mjFjBoGB9a8fj8FgqPDrtinFlA37sAGvDW1PiF/5m+8FBsJnDYOZsfYoX+9O5ofYDKwKHuvfmkDfq386Vl6VUW71gZRb2Ui5iZpOKYX672w4EIM26km0dp2rO0tCCFFvVFmAFRBgf2ri6+tLt27dOHr0aLEAKzo6mujoaMdyfex8XxlDaK48lMbOkxk83j0UoyWHlJScCjv3+K4BNPKEr/5MZnArP9wKs0lJyb7ygRVMhh4tGym3spFyK5v6Mkx7TaB+/g618We0m+5E13dgdWdHCCHqlSoJsMxmM0op3N3dMZvN7NmzhxEjRlRF0vXeyYx85sckExnmyY0tyzcpbUk0TePWdib6NPYhwF1G/RdCiOqmdm1G/e9LtG790IbfW93ZEUKIeqdKfhFnZGTw9ttvA2C1Wunbty8RERFVkXS9ZrEpZm1JwM2g44meDdAuMXdVRQjyLNtkt0IIISqO+usQtrmzoEVbtAefluHYhRCiGlRJgBUSEsLMmTOrIilxgSX7UvgrzcwL/cLwl6dLQghRp6nkM9g+ngp+AejG/wvNxfXKBwkhhKhwcmurjjqSkse3+1IZ0MyH3o1l7jEhhKjLVE42tg9eB6vVPhy7d8U3CRdCCFE6EmDVQfkWG7O2JODvbmBsZEh1Z0cIIUQlU0vmQvIZdI+/hBbaqLqzI4QQ9ZoEWHXQV38mEZ9VwNO9GuDlqq/u7AghhKhEKi8XtWMjWp8otDbXVHd2hBCi3pMAq46JSchh1ZGz3NzGn86hntWdHSGEEJVM7dwEBQVovaOqOytCCCGQAKtOyc638sHWBBr5uHJ/RFB1Z0cIIUQVUFvWQWhDaN6murMihBACCbDqlDk7EzlrtvBM7zDcDPJPK4QQdZ1KioejB9B6R1fqVBxCCCFKT36F1xF7zuSw4Vgmd10TSEuTsbqzI4QQogqoLetA06H1HFDdWRFCCFFEAqw64n/7U/E36rm9fUB1Z0UIIUQVUDYbaus66BCB5m+q7uwIIYQoIgFWHRCbmsfuM7nc0i4AV738kwohRL1weC+kpcjgFkIIUcPIr/E6YNmBNDxddAxu5VfdWRFCCFFF1Ja14O6JFtGjurMihBDiAhJg1XKnMvPZeiKLIa398XCROa+EEKI+UHm5qD+2oHXvh+biWt3ZEUIIcQEJsGq57w6k4aLXuLmNf3VnRQghRBWRua+EEKLmkgCrFkvNLeS3uAyimvvi526o7uwIIYSoImrLWghtBM1aV3dWhBBCXEQCrFrs/w6lY1Nwm4wcKIQQ9YZKjIejB9F6R8ncV0IIUQNJgFVLZeVb+TH2LP2a+BDiJe3vhRCivnDMfdVrQHVnRQghHipx2wAAIABJREFURAkkwKqlfjiSjtlik3mvhBCiHlE2a9HcV9ei+cncV0IIURNJgFULmS02Vh5OJzLMk6b+xurOjhBCiKpyaC+ky9xXQghRk0mAVQut+essWflW7uggdy+FEKI+UVvWgocnWkT36s6KEEKIS5AAq5ax2BTLD6TRPsid9sEe1Z0dIYQQVUTl5qD+3IrW/TqZ+0oIIWowCbBqmY3HMknOtcjTKyGEqGfUrs0y95UQQtQCEmDVIjalWHYglSa+bnQN86zu7AghhKhCavMaaBAOTVtVd1aEEEJchgRYtciO09mcyCjg9g4BMveJEELUI+rMafjrEFrvG+TvvxBC1HASYNUSSimW7k8l2NOFfk18qjs7QgghqpDaWjT3Vc8B1Z0VIYQQVyABVi1xICmPwylmbmsfgF4ndy+FEPXbDz/8QGZmZrnOERMTw9NPP82TTz7J8uXLL7nftm3buOuuu/jrr7/KlV5Z2ee++lXmvhJCiFpCAqxaYumBVHzd9EQ1963urAghRLXbu3cv48ePZ8aMGWzZsoXCwsKrOt5mszF37lxefvllZs2axebNmzl16lSx/fLy8li9ejWtWlVjv6dDeyA9BV0fGdxCCCFqAwmwaoG4dDO74nO4ua0/bgb5JxNCiBdeeIFPPvmEiIgIVq1axSOPPMLs2bM5cOBAqY4/evQooaGhhISEYDAY6N27Nzt27Ci23+LFi7nllltwcXGp6EsoNbV5HXh4QWeZ+0oIIWoDQ3VnQFzZ0v2puBt0DGntX91ZEUKIGsPb25vBgwczePBgjh8/zkcffcSvv/5KYGAgUVFR3HTTTRiNxhKPTUtLw2Q639zOZDIRGxvrtE9cXBwpKSl07dqVlStXXjIfa9asYc2aNQDMmDGDwMDACrg6O1tONskxW3G/YRg+DcIq7LwVzWAwVOh11xdSbmUj5VY2Um5VRwKsGi4hq4DNJ7IY3jYAL1d9dWdHCCFqlL1797Jx40Z27NhBixYteOKJJwgMDOSHH35g2rRpvP7/7d15eFTl2T/w75ktkz2ZmeyQhCzsgo1RkEUEUmqVWkp9RS32BWtbt1fbWhWxVayi2IqoVau1gqC/tvCq9W21rRYXVhUQEIQkkJAQQhKSTJbJMpNZzvP7YyaTBBJITiazJN/Pdc2Vc86cmblzZ3LO3PM853l+85teHyeEOGdb99H5ZFnGxo0bcccdd1wwhoKCAhQUFHjX6+vrFfwmvZO3fwDY7ejIm+nT5/U1k8kU1PEFK+ZNGeZNGeZNmdTUgX+5xQIryL1b2ACVJOHaCYZAh0JEFDQ2bdqE3bt3IyIiAldccQXWrl0Lg6HrOJmbm4vly5f3+Xij0Qiz2exdN5vNiI/v6iVgs9lw6tQpPProowCApqYm/Pa3v8X999+P7OzsIfiNeid2f+SZ+yrHb69JRESDwwIriDVanfiotBnzs2JhCOefioiok8PhwC9/+Uvk5PReeGg0GqxZs6bPx2dnZ6O6uhq1tbUwGAzYvXs37r77bu/9EREReO2117zrq1atws033+zf4qpz7qvrlnHuKyKiEMJP7UHsH0UNcAmB701k6xURUXff+973oNPpemxrbW2F3W73tmSlpaX1+Xi1Wo1bbrkFq1evhizLmDt3LkaPHo3NmzcjOzsb+fn5Qxp/f4jdH7nnvpp2ZaBDISKiAfBrgSXLMlasWAGDwYAVK1b486VDTpvdhX8db8Llo6OREq278AOIiEaQ3/3ud7j99tsRFRXl3dbQ0ICXX34ZTzzxRL+eIy8vD3l5eT22LVmypNd9V61apThWJbxzX03OgxTHL9mIiEKJX8f8/uc//3nebxSpy7+ON6HdIeO6SZxUkojobFVVVUhPT++xLT09HadPnw5QRD5WeAhoMkM1Y16gIyEiogHyW4FlNpuxf/9+zJ/PiRIvpMMp4x9FDfhGSiSyDL0PMUxENJLFxMSgpqamx7aamhpER0cHKCLfErs/8sx9NS3QoRAR0QD5rYvg66+/jqVLl8Jqtfa5z1DOJRIqNBoN9tS60GRz4ZYZY2AyxQU6pJDAuR2UYd6UYd4Cb+7cuVi7di1uuOEGJCUloaamBps3b8a8eaHf4iPaWyEOfA5pVgGkAE5wTEREyvilwPryyy8RGxuLrKwsHDlypM/9hnIukVARZzDizb0VGGvUY3SYY0TmQAnO7aAM86YM86aMkrlE+rJo0SJoNBq88cYbMJvNMBqNmDdvHhYuXOiz1wgUsW8n4LBDmsEeH0REocgvBVZxcTH27duHAwcOwG63w2q14vnnn+8xJC65fXq8HjWtDizPS+SwvEREfVCpVLj22mtx7bXXBjoUnxO7PwZS04EMzn1FRBSK/FJg3XTTTbjpppsAAEeOHME//vEPFle9EELgjX2VGBWjw2Wjoi78ACKiEczpdKKqqgoWi6XH9smTJwcoosETNZWeua+W80s2IqIQxXmwgsj+qjaU1LfhnstToOKJlYioT0VFRXjmmWfgcDhgtVoRHh4Om80Go9GIF154IdDhKSZ2fwyoVJCmzQl0KEREpFC/C6z33nsPkydPRmZmJo4dO4Z169ZBrVbj7rvvxtixY/v9gpMmTcKkSZMUBTuc1bU5sH5/LZKiwjA7IybQ4RARBbWNGzfi2muvxcKFC7F8+XJs2LABb7311jmTD4cS79xXkzj3FRFRKOv3MO3vv/8+EhMTAQB/+ctfsHDhQixevBivv/76UMU2Yhyrt+K+f5fD3O7Eym/mQqtm6xUR0flUVVXh6quv7rFt0aJFeP/99wMUkQ8c/co999VMDm5BRBTK+l1gtbe3IyIiAlarFeXl5fj2t7+NefPmoaqqaijjG/a2l1vw0NYK6DQq/PZbGchP57DsREQX0nk+AoC4uDhUVlaitbUVNpstwJEp5537asplgQ6FiIgGod9dBI1GI4qLi3Hq1ClMmDABKpUK7e3tUKn8NlfxsCILgb8ersfmw2ZMTAjHg1ekIUbPS+KIiPpj2rRpOHDgAGbNmoV58+bh0UcfhVqtxuWXXx7o0BTxzn01+5uc+4qIKMT1+xP90qVL8cwzz0Cj0eDee+8FAOzfvx85ORxGdqA6nDKe+6wauypaMD8rFrdflsxugUREA7Bs2TLv8ne+8x3k5ubCarVi6tSpgQtqMAoPAU4HpMuuCHQkREQ0SP0usPLy8vDKK6/02DZ9+nRMnz7d50ENZ+Z2B57YdhqlDTYsz0vAd8cbOBQvEdEAyLKMe+65B8888wy0ntae8ePHBziqwRGlhYBGC2TmBjoUIiIapH4XWJWVlYiKikJcXBxsNhv+/ve/Q6VS4Tvf+Q40GnZt648Ssw2rt1Wi3SHjoTmjcCnnuiIiGjCVSgWVSgWHw+EtsEKdKCkEMnMhaYbH70NENJL1+wKq5557Du3t7QCATZs2obCwEMeOHcMf//jHIQtuONlVYcGD/zkJjQp4akE6iysiokG4+uqrsW7dOhw9ehQ1NTU4c+aM9xZqhL0DqDgBKTu0W+GIiMit301PdXV1SE1NhRACe/fuxdq1a6HT6XDXXXcNZXwhTwiB//3ajP93qB7jTeF4cE4a4jiYBRHRoKxfvx4AcOjQoXPu27x5s7/DGZyTpYDLCSlnQqAjISIiH+j3J32tVgur1YrKykoYjUbExMTA5XLB4XAMZXwhrcMp44XPa7D9pAVzx8TgzmnJ0Ko56iIR0WCFXBF1HqKk0L3AFiwiomGh3wXWzJkz8Zvf/AZWqxVXXXUVAKCsrMw7+TD11Gh14oltlThutuGHFydg8UQOZkFEROcSpYVAYiqk6NhAh0JERD7Q7wJr2bJl+Oqrr6BWqzF58mQAgCRJ+O///u8hCy5UnWhwD2bR0uHCiivSMH10dKBDIiIaVh5++OE+v7R69NFH/RyNckIIoLQI0pRLAx0KERH5yIAuBpo6dSrq6+tx7NgxGAwGZGdnD1VcIevzUy14ZlcVosLUWLMgA1kGfaBDIiIadubNm9djvampCZ988glmz54doIgUOlMFtFrYPZCIaBjpd4HV2NiIZ599FsePH0dUVBRaWlowduxY3HPPPTAYDEMZY8jYW9mKJ7efxlijHivnjEJ8OAezICIaCldeeeU526ZPn46XXnoJ1113nf8DUkiUFgEApGwOcEFENFz0e8SFV199FRkZGVi/fj3++Mc/YsOGDcjMzMSrr746lPGFDCEE/nyoDqnROjxekM7iiojIzwwGA06ePBnoMAamtBCIiARSRgU6EiIi8pF+VwHFxcX4xS9+4Z1UWK/XY+nSpbjtttuGLLhQcrCmHScaO/A/05MRpuFIgUREQ+njjz/usW632/HFF19g7NixAYpIGVFSCGSNh6TieYOIaLjod4EVGRmJyspKZGZmerdVVVUhIiJiKOIKOW8dMcMYrsGcTI4CRUQ01Hbs2NFjPSwsDOPGjcM111wToIgGTrS1AtWnIF12RaBDISIiH+p3gXXttdfisccew7x585CQkIC6ujp8+umnWLJkyVDGFxKK6634+kw7bslLhFbNodiJiIbaI488EugQBu9EMQBwgmEiomGm330SCgoK8POf/xwtLS348ssv0dLSgrvuugtms3ko4wsJbx8xI1qnwoKcuECHQkQ0Imzbtu2c663Ky8uxffv2AEU0cKKkEFCpgMzcQIdCREQ+NKBO35MnT8Ztt92GBx98ELfddhsmTpyId955Z6hiCwkVzR34orIV14yLR7iWfeiJiPxh8+bNMBqNPbaZTCb89a9/DVBEAydKC4FRYyDpwwMdChER+RArgkF654gZYWoJ14zjUPVERP5itVrPuQY4IiICbW1tAYpoYITLBZQdY/dAIqJhiAXWINS2OrC93IIFuXGICVMHOhwiohFj1KhR+Pzzz3ts27NnD0aNCpHhzivLAHsHJxgmIhqGLjjIxddff93nfU6n06fBhJp3ixogScB3x7P1iojIn37wgx/gySefxO7du5GcnIyamhocPnwYDz74YKBD6xdRwgmGiYiGqwsWWH/4wx/Oe7/JZPJZMKGk2ebEf0qaMCczFgmR2kCHQ0Q0oowfPx5r167Fzp07UV9fj5ycHCxbtix0zkmlhUC8CZIxIdCREBGRj12wwHrxxRf9EUfIea+4EQ6XwOKJbL0iIvI3h8OBuLg4LFq0yLvN6XTC4XBAqw3+L71EaREkdg8kIhqWeA2WAu0OF94/1ojpo6MwKjYs0OEQEY04jz/+OE6cONFj24kTJ7B69eoARdR/oqEeaKjj9VdERMNUvycapi4fHG9Cm13G9ycZL7wzERH5XEVFBXJze84flZOTc87cWOdz8OBBbNiwAbIsY/78+T1awwDgww8/xAcffACVSgW9Xo+f/vSnPhlEQ5R6rr/iCIJERMMSC6wBcrhk/F9RI6YkRyDXyLlLiIgCISIiAs3NzYiL65rgvbm5GWFh/etVIMsyXnvtNfzqV7+C0WjEgw8+iPz8/B4F1KxZs7BgwQIAwL59+7Bx40Y89NBDgw++tBDQ6YBRYwb/XEREFHTYRXCAPimzoNHqxPcnsvWKiChQpk2bhueeew4VFRXo6OhARUUFXnjhBUyfPr1fjy8pKUFycjKSkpKg0WgwY8YM7N27t8c+3efZstlskCTJJ7GLkkIgcywkDb/jJCIajnh0HwCXLPDOUTOyDXpMTY648AOIiGhI3HDDDdi0aRNWrlwJh8MBnU6HuXPn4oYbbujX4xsaGmA0dn1RZjQacfz48XP2+/e//433338fTqcTDz/8cK/PtXXrVmzduhUAsGbNmvOOZChsVtSeKkPk936AqFAZ8bAfNBpN6IzgGESYN2WYN2WYN/9hgTUAn51qQXWLAw/MTvDZN5lERDRwOp0Ot956K370ox+hpaUFjY2N2LZtG+655x688sorF3y8EOKcbb0d16+66ipcddVV2LlzJ95++23cdddd5+xTUFCAgoIC73p9fX3fr1v8NSC7YE3NgO08+4Uak8l03t+bese8KcO8KcO8KZOamjrgx7DA6ichBN4+YkZqtA7TRkUHOhwiohHPYrFg586d2LZtG8rLyzFhwgQsW7asX481Go0wm83edbPZjPj4+D73nzFjBl599dXBhgxRctS9wBEEiYiGLb8UWHa7HY888gicTidcLhemT5+O66+/3h8v7TMHqttworED/zM9GWoVW6+IiALB6XRi3759+PTTT/HVV18hOTkZM2fORG1tLX7+858jNja2X8+TnZ2N6upq1NbWwmAwYPfu3bj77rt77FNdXY2UlBQAwP79+73LgyFKi4CU0ZAi+UUdEdFw5ZcCS6vV4pFHHoFer/f2Y7/44osxduxYf7y8T7x9tAHGcA3mZPbv5E1ERL734x//GCqVCnPmzMH111+PrKwsAO4h1QdCrVbjlltuwerVqyHLMubOnYvRo0dj8+bNyM7ORn5+Pv7973/j8OHDUKvViIqKwp133jmo2IUsAyeKIX2jfwNxEBFRaPJLgSVJEvR6PQDA5XLB5XKF1DVMxfVWfH2mHbfkJUKrDp24iYiGm4yMDBQVFaGkpAQpKSlITExEVFSUoufKy8tDXl5ej21LlizxLi9fvnxQsZ7jzGmgrYXdA4mIhjm/XYMlyzIeeOAB1NTU4Fvf+tY5E0QCAxuJyZ+e/uwoYvQa3DgtGxE69ZC+Fkd4UYZ5U4Z5U4Z5C5xVq1ahrq4O27Ztwz/+8Q9s2LABU6ZMQUdHB1wuV6DDOy9RUgiAEwwTEQ13kuhtKKUh1NbWhqeffhrLly9Henr6efetqqryU1R9q2juwP+8V4YbLjLixikJQ/56HOFFGeZNGeZNGeZNGSUjMV1IUVERtm3bhs8++wxqtRpz587F0qVLff46A9HXuUt+/XmIg19Ate7NkOrF0R/8n1CGeVOGeVOGeVMmJEYRjIyMxMSJE3Hw4MELFljB4J0jZoSpJVwzzhDoUIiI6Czjx4/H+PHjsXz5cuzZswfbt28PdEh9EqWFQPb4YVdcERFRTyp/vIjFYkFbWxsA94iChw8fRlpamj9eelBqWx3YXm7Bgtw4xIQNbddAIiJSTqfTYdasWVi5cmWgQ+mVaLEANafZPZCIaATwSwtWY2MjXnzxRciyDCEELr/8clxyySX+eOlBebeoAZIEfHc8W6+IiGgQThQDACQOcEFENOz5pcDKyMjAb3/7W3+8lM8025z4T0kT5mTGIiFSG+hwiIgohInSo4BaDWScO8ATERENL37pIhiK3ituhMMlsHgiW6+IiGhwRGkRkJ4NKSws0KEQEdEQY4HVi3aHC+8fa8T00VEYFcuTIRERKSecTqDsOLsHEhGNECywevHB8Sa02WUsnmgMdChERBTqTpUBDjsLLCKiEYIF1lkcLhn/V9SIKUkRGGsKD3Q4REQU4kTpUfcCRxAkIhoRWGCd5ZMyCxqtTnx/EluviIjIB0qKAGMipDieV4iIRgK/TzQcaE5ZoNHq9N4aPLfO9eJ6K7INekxNjgh0qEREFOKEEBClhZDGTg50KERE5CfDrsBqd7jwVU07Gtp7Fk6dy5YOF8RZj1FJQGyYGvHhGoxPCMeNUxIgSVJA4iciomGkoQ5oamD3QCKiEWRYFVjtDhdW/qcCZY0dANyFU5xeA0O4BgmRWowzhcMQrkF8uAbx4WoYwrWID1cjTq+BWsWCioiIfEuUFALgBMNERCPJsCmwXLLA73ZU4WRTB+6dmYopSRGIDlOzcCIiosApLQLC9EBaZqAjISIiPxkWBZYQAi/vrcH+6jbcOS0ZV2TGBDokIiIiiNJCIGscJLU60KEQEZGfDItRBN8+0oAPS5px3SQjFuTEBTocIiIiCJsVOFXO7oFERCNMyBdY28qa8cZXdbgiMwZLp5oCHQ4REZFb2TFAyCywiIhGmJAusL4+047nP6/B5MRw3D09mSP/ERFR0BClhYAkAVnjAh0KERH5UcgWWKeaO/DE9kokR2nx4BWjoFWH7K9CRETDkCgtAlLTIUVEBToUIiLyo5CsShqtTvzmk1PQqiQ8PHcUosJ48TAREQUPIctAaTG7BxIRjUAhV2BZHTIe+7QSzTYXfnXlKCRF6QIdEhERUU/VpwBrG5DNCYaJiEaakCqwXLLA2l2nUdZow32z0pBrDA90SEREROcQpZ4JhnPYgkVENNKETIElhMCr+85g7+k2/CQ/CZeOYp92IiIKUiVFQHQskJAS6EiIiMjPQqbA+lthA/51vAmLJxrw7bHxgQ6HiIioT6K0EMiewNFtiYhGoJAosHaUW7DxQB1mZUTj5osTAh0OERFRn4SlCaitZvdAIqIRKugLrCO17Xj2s2pMTAjHPZenQMVvA4mIKJidKAIAjiBIRDRCBXWBVWnpwBPbKpEYqcWDc0ZBx7muiIgoyImSIkCjATJyAh0KEREFQFBXLL/5pBJqScIjc0chhnNdERFRCBClhUBGDiQtpxEhIhqJgrrAamx34FdXjkJyNE9SREQUIspL2D2QiGgEC+oC6xcn3kFue3WgwyAiIuo/pwMSJxgmIhqxgrrAuqzpOOSnHoA48HmgQyEiIuo/tmAREY1YQV1gqR5aC6RlQP7Dk5D/9TaEEIEOiYiI6PwSkiHFcr5GIqKRShPoAM5HijNA9cvVEK8/D/HORqC6Arj5LkhabaBDIyKiEHfw4EFs2LABsixj/vz5WLRoUY/733vvPXz00UdQq9WIiYnB7bffjoSEC8/FyO6BREQjW1C3YAGApAuD9ONfQrr2JojPPoG89iH3JI5EREQKybKM1157DStXrsS6deuwa9cuVFZW9tgnMzMTa9aswdNPP43p06fjzTff7N+Ts3sgEdGIFvQFFgBIkgTVd26A6qf3A6dOQH7ilxCVZYEOi4iIQlRJSQmSk5ORlJQEjUaDGTNmYO/evT32mTx5MsLCwgAAubm5aGho6NdzSzkssIiIRjK/dBGsr6/Hiy++iKamJkiShIKCAlx99dUDfh4pfxZUpiTIL66GvGYFVD++F9LUy4YgYiIiGs4aGhpgNBq960ajEcePH+9z/48//hgXX3xxr/dt3boVW7duBQCsWbMGpovyIKlH1tyNGo0GJpMp0GGEHOZNGeZNGebNf/xSYKnVatx8883IysqC1WrFihUrMGXKFIwaNWrAzyVl5kK1cq27yHpxNaTvL4O0YBEkSfJ94ERENCz1NmhSX+eR7du348SJE1i1alWv9xcUFKCgoMC7bm5s9EmMocRkMqG+vj7QYYQc5k0Z5k0Z5k2Z1NTUAT/GL10E4+PjkZWVBQAIDw9HWlpav7ta9EaKN0J135OQ8mZAvLUBYuPzEA6Hr8IlIqJhzmg0wmw2e9fNZjPi488d+e/QoUP429/+hvvvvx9aDrBERET94PdrsGpra1FWVoacnJxBPY8UFgbpJ/dB+s4NELs+grzu1xAtzT6KkoiIhrPs7GxUV1ejtrYWTqcTu3fvRn5+fo99ysrK8Oqrr+L+++9HbGxsgCIlIqJQIwk/Ti5ls9nwyCOPYPHixZg2bdo595/dj91ut/fveXf8B82/Xw21wYS4lb+FJj3Lp3H7k0ajgdPpDHQYIYd5U4Z5U4Z5U0an0wU6hB7279+PjRs3QpZlzJ07F4sXL8bmzZuRnZ2N/Px8PPbYY6ioqEBcXBwAd/eaBx544ILPW1VVNdShBx12PVKGeVOGeVOGeVNGSRdBvxVYTqcTTz31FKZOnYqFCxf26zEDOUmJsmOQX1wNdNiguvWXkKZeqjTUgOKbXxnmTRnmTRnmTRklJ6lQxAKL+ot5U4Z5U4Z5UyZor8ESQuDll19GWlpav4urgZLGjIVq5VogMQXyC49Bfm0d58siIiIiIiK/8ssogsXFxdi+fTvS09Nx3333AQBuvPFG5OXl+fR1JIMJqgeegnj/fyE+eAfi0F5I1y2DNLMAkiokpvwiIiIasCqLHfXtDlyUFMFRdYmIAswvBdb48eOxZcsWf7wUJF0YpO8thZg+B/KbL0FsegFi90dQLb0DUlqGX2IgIiLyl90VFjz3WTVsToFJieH4728kYpwpPNBhERGNWMO2WUdKGQ3VL5+AtOweoKYS8mM/g/z2RoiOjkCHRkRENGguWeCNg3V4akcVMuLC8KNLElFpseP+D07iqR2nUWXp30BRNPIIIXCmle8PoqHilxasQJEkCdLM+RBTLoV4ewPEv9+G2LsDqh/cBumi/As/ARERURBqtbvwzK4qfFnVhgU5sfhJfhK0ahUKsmPx98JG/K3QjC9OtWBBThxuuMiEuPBhfbqnfpKFwJ7KVmw+XI8TjR34dm4cbs1PgkbFbqXB7lRzB/adbkW7Q4ZLFnAJwCWEe1l2L8uia9klw7Pu2VcWSIoz47px0UiO9u+Irs02J3ZXtCBKp0ZCpBYJkRrE6TVQD+P33Yg44krRMZCW3QMxYz7kN/8A+fnfAJfMgOqGH0OKMwY6PCIion6raOrAE9srUdfmwO2XJeGq3K4JkiO0atwwxYSrcuPw18P1+LCkCZ+UNWPRBAO+O8GACK06gJFToMhC4PNTLdh82Izypg6kRGsxd0wM/nW8CacsdjwwKxUx+hHxkTCkVFo6sOtkC3adbMHJZncPLJUEqCQJGhWgliSoVBLUkntZrXLfp+7cppLc657lnScasK2kHjdeZMK1EwxDXljLQuA/Jc3YdLAWrXa5x31qCTBGuIuthAgtTJFdywmRWpgiNSF9vPLrPFgDNRRD3QqnA+KDv0G8vwVQqyEtuhnS3G9DUgXHH5FDaCrDvCnDvCnDvCnDYdoHr/N6q3CNCg/MTsOExIjzx2Kx442v6rC7ogWxejWWTDbhW7lxPv9g1d//CYdL4EybHVUWO6pa7KiyOHC6xY4wtYT02DCkx4UhPTYMo2N1CNMM26sYvIb6WOKSBT471YIth8042dyB1GgdllxkxOyMGKhVEj4ta8Y/ICWeAAAdcUlEQVQLn9fAEKHBQ3NGISMubMhi8aXhfAw+bbFj10kLdlW0oLypAxKACQnhmJkRjctHR8MYoVX83HJYNNZ8WIgvKlsxJj4Md05LRq5xaK7XPNFgw8t7a1Bcb8PkxHDccom7pbSuzYG6Ngfq253dlt3r8lkVSaRO5Sm4NEiM1CIlWoe0GB1SonVIjNT6rQUsqOfBUmIoT1Kithry/3sZOHoAyMiB6uY7IGXkDNnr9ddwPmgMJeZNGeZNGeZNGRZYyrlkgT8fqsdbR8wYZ9LjgdlpA/qgVVxvxcYDtThSa0VKtBY3T03AjPRon4042P1/QggBs9WJKosdpz2FVOfPM62OHh+iYsLUSInWwe6ScarZDqfnTglAcrTWXXR5Cy/3hyutevgUXkN1LHHJArsqWrD5cD0qLXaMitHh+slGzPIUVt0dq7fiie2nYXXI+MWMFEwbHe3zeHxtuB2Dqyx27KpwF1Vlje6WqgkJ4ZiZHo0Z6YMrqrrrzNtnp1rwyt4zaLY5cc24ePxgSgLCtb75v2p3uPDnQ/V4v7gR0To1lucl4soxMRc81rhkgUZbZ9HlRH2bA3XtnuV2B2paHLA6u1rBNCogKUqH1GgdUqN7Fl/GCA1UPhxNlQXWAAkhIPbthNj8J8DSDGneNZC++wNI4ef/RnAoDbeDhr8wb8owb8owb8qwwFKm+/VW38yOxU8vTVJUZAgh8GVVGzYeqEVFsx25Rj2WfSMRk5MGds5zygJNNicarU40WN0/24UOJWeaPK1SdnS4uj5ahKklpMa4PwileX52rkeHdfUecckC1a12VDR1oKLJjormDpxs6kBVi91blKkkIDVah/S4MGTEhmF0nA6ZcXqkRmtDcnh6Xx9LXLLAjpMWbPnajNMWO9Jjdbh+sgkz0qPP+22/ud2BJ7adRmmDDT+YasJ1k4xBnc/hcAyubrFj18kW7KyweIuq8SZ3S9WM9GiYfFRUddc9b212F944WId/H2+CMUKD2y9LRn5alOLnFkJg96kW/GlfLRqtTizIicMPL05AVJhveogJIdBsc+F0ix3Vni9tqj2t4NWtdti7HXN0agkpnsKr83iT5jn+RIepB/zeZoGlkGhvhfjbmxDb/gXExkO6+r8gTZsDKUL5G02p4XDQCATmTRnmTRnmTRkWWAPX/XqrH+f3vN5KKZcs8ElZM/58qB7mdifyUyNx88UJSI3RodHqRKPV5S2eOguoRqsTjTb3usXmwtkfHNQSkBSl7VE8pcW4lw3hg/s22eGScdpiR0Wzp/hqdt9qWhzeONJjdZg7JhZzxsT47Nt+f/DVscQlC2wrt+B/v65HVYsDGXFhWHKREZePju537jucMl74ogbbyy2YnRGN/5meErTdNEP1GFzTYseuihbsqrCgtMFdVI0z6TEzPQYz0qOREDm0793e8lZY146XvqhBRbMdM9Oj8eP8JMQPcFCc6hY7/rj3DPZXt2FMfBhuvyzZr1NFyELA3O70fsFT5SnCqlocqGmxo1vthSidCmkx7uNTWnQYUmO0SIsJQ0q0Fro+vrhigTVI4kQx5L/8ESg/Dmh1kC6ZCWn2AiB3ot++yQnVg0agMW/KMG/KMG/KsMAamM7rrfQaFVb043qrgepwynivuBFvHzGjzSH3uo9KAuL0GsSHa2AIV8MQrkV8uBrx4Z3b3D9z0pLQ1Njg0/j6E/+pZjuK663YVm5Bcb0VEoCpyRGYmxWL6aOjoQ/SAqHTYI8lTlng07Jm/O/XZtS0OjAmPgxLLjJh2qgoRUWtEALvHG3AGwfrkGXQY+WcNJ+3pJSYbdhVYUGcXoOJieEYE68f8DWBoXQMPtPa2VLVgtIGGwBgrFGPWRn+Kaq66ytvDpfA3wrN2HLYDJ1awrK8RBRkx17wPeRwyXjnaAPeOmKGWpLwg6kmXD02PqhGB3TJArVtDpz2dFk+bbHjtKcQa7A6vftJABKjtEjrbPHqdpuSM/B5dFlgnUUIAVSUQuz4EGLPdsDaDiSlQZr9TUiXz4UUM/hvD88nlA4awYR5U4Z5U4Z5U4YFVv90v95qrFGPFVcM7HqrgWrpcOGDkibIsuhROBnCNYgOU/frw1Iw/E9UWez4pKwZn5ZZUNvmgF4jYUZ6NOaOicXkpAifXpMxWE5ZoLbVAZcuEubGpp7Da/cxxLbsGZbbKQvIMtDhkrGt3IIzrQ5kG8KwZLIJl42K8skXwnsrW7F2VxXCNBIevGIUxicMrjWi3eHC9nILPixpQmlDB1QSvN0+w9QSck3hmGAKx8TEcIwzhSNSd/5uZUrfbw6XjDNtDkTp1IgbwlETz7R6WqpOtqDEU1TlGvXea6qSovw7THqnC+XttMWOl/bU4Osz7ZiYEI47piVjdGzvA58cqmnDH/acQVWLu+XrR5ckhlTrMeB+X1ZZHDht6fAWXZ3Xi9qcXeXR3vvmDfi5WWCdh+iwQXy5C2LHf4CSo4BaDUy9DKpZC4BJFw/JyIPBcJIKRcybMsybMsybMiywLsxX11v5WzD9T8hCoLDWik/KmrGrogXtDhmmCA2uHBOLK8fE9PmB0ddsThk1LXbUtDpQ7flZ02JHdat75LSzR0xTIteox5LJJuSnRfq8p01FUwdWb6tEfbsTd05Lxrys2AE9XgiBkgYbPjjehB0nLbA5BTLjwvCt3DjMyYyBzSmjsM7qvZU12iALd0tCelwYJiSEe2+JkT2vsTvf+63DKXtz7r65r9GpabGjrs3p7VZqjNAgx6BHjlGPHIMe2QY9YgdRdNW2OrwDVRw3u4uqHIMeMzOiMTOARVV3/fk/FULgoxPN2LC/FjanwHWTDLhuktF7HGq0OrF+fy22l1uQHKXFTy9NQl6q/y+pGUqdA/V0tnjdeuWkAT8HC6x+EtWVEDv/A/HZx0BLMxBvgjSzANLM+ZBMST57nWA6SYUS5k0Z5k0Z5k0ZFljn1/16q1svScJVuXFBPdBAd8H6P9HhlLGnshWflDXjQHUbZOEuSuaOicXsjGjFcz8JIWB3CVidMuraHKhu6SqeOn82dut+BADROhWSo3VIjnKPeJYcpUVWigntrRbvHEZqSYLKM2eRd1vnsgTPnEdd8x0N9TxGlg4XfrfjNA6daceiCQb88OKEC7ZottndrVUflDShrLEDYWoJszNj8K2cOOQa9X2+p60OGcfMXQVXcZ3VO2qcIVzTreCKwOTMZBRW1HQVUJ6cV7fYYW4/K+9haqR4cp4SrUVSlA4tHS6UmG0oabChqsXu3TcxUoNsQ7i38Mo26HsMxHK2ujZPUXWyBcc8RVW2QY9ZnpYqf0/oeyED+T9tsjnx2pfuQmpUjA63XZaEU812vHmwDh0uge9PMuD7E41Be52eL/EaLD8QTgfw1V7IOz8Ejhxwb5xwMVSzvwlcPA2SZnDNo8F6kgp2zJsyzJsyzJsyLLD69lVNG57Ydhp6jTQk11sNtVD4n2i0OrG93IJPyppR1tgBtQRckhaFSYnhsDsFbE4ZNpdAh1OGzSmjw9lz2ebqXHav9/bhyRCuQXKUFsmeD/PJUe6fKVG6XkdTC4W8OWWB9V+ewfvHmpCXEol7Z6Ui6qwufEIIHDPb8GFJE3aUW9DhEhgTH4Zv5cRhzpgYRRPGumSBk00dKKyzoqjOisK6dtSdVTx1itWrkRrdlevk6PPnvbs2uwulDTaUNrgLrhKzDTWtDu/9SVFad8Fl0CPbqIcxQoMvT7dhV4UFxfWdRVUYZqTHYGZ6NFKCrKjqTsn7bX9VK/6w5wxq29w5mZIcgdsuTUZaTPD+nr7GAsvPhLkOYtdWiF3/ARrqgagY9+iDYycDWeMgxRkG/JyhcLANRsybMsybMsybMiywere/qhVPbj+NlCgdHpk3KuSuYwBC73+ivNGGT8ss+LTc4m1p0qiAMI0KerXK/VMjIUzTtXz2dr3nZozQeFukBvptfijl7YPjTXhlbw2So3V4aM4opMXo0Gp3YVuZ+9qq8qYO6DUSZmfE4Fu5ccgx9N1apVRdmwOFdVZYoUOUZHfnPVqrqIA7n9YOF0obbd5WrhKzzVtgdBoTH4ZZ6TGYmRHcRVV3St9vNqeMvxc2ICVah1kZvps/L1SwwAoQIbuAo19B3vEhcGgP4PR8w2JMhJQ1zl1sZY0D0rMu2MIVSgfbYMK8KcO8KcO8KcMC61x7Klvw1I4qjI7V4TfzRivushZoofo/4ZLd3fz0GtWQd7frTajl7ciZdqzZcRouWeCStCh8fqoFdpdAVrz72qorMpW1Vg1UIPJm6XC3dJ1ptWNKUiRSQ7AFJ9Teb8FCybkrNI/kQUZSqYHJeVBPzoNw2IGKExAnioHSIojSQmDvDndXAo0WyMiGNGYcpGx34YV404j7JoCIiNzDsD+9swpZBj1WzR3tswk5qf/UKumc7m7Ut0lJEXj6qgw8uf009lS2Yu6YWCzIiUOOUR/o0IZcTJga30iJBBAZ6FAoBLDA8jFJqwOyx0PKHg9887sAANFoBk4UQ5wohjhRBLHtXxBb/8/9gDgDkDUekqeVS468JIDRExGRP2wvt2Dd7iqMNYbj4bmjLjgsNVGwSIrSYd23MyELBNV8R0TBhAWWH0jxRuCSGZAumQHAM1BGZTlEabG78Corhti/GwJAHQBERQPGJPfohJ6bd9mYCEkbev3ziYjI7eMTzfj959WYkBCOX105yi9dqoh8SfKMakhEvWOBFQCSRgtk5kLKzAXmLwQACEsjcOIYIiwNaDtZBlF/BuJUGfDVF4Cza94GSBIQa+hZdHmXE4GYeBZgRERB6sOSJrz0RQ0uSo7AQ3NGQT8ChjgmIhppWGAFCSkmHrh4GiJNJli7XYAoZBloagDqz0DUnwE8N2E+A3HsMPDFp4A4a7jY8EggJg6IjgVi4iDFxALRcV3LMXHedejDeQ0YEZEfvF/ciD/uO4O8lEisuCJtRMwfQ0Q0ErHACnKSSgUYTIDBBGnsuTNJC6cDaKjzFGC1gKXJPRGypQnC0gRUn4IoPgy0tbj3P/sJNNquYiw6FlJ4hLtAC49w3yKigPAISJ3bIiK77teHu+MjIqLz+r/CBqzfX4vLRkXh/lmp0Kp57CQiGq5YYIU4SaMFElOBxFScrx1KOJ1AazNg8RRfniIMLU1d6y3NELXVgLXNfXN2TejX61j+kgTow3sUXNCFAbowSGF6IEzvXUeYHgjzLOv07vt1YZ5tnn21Ove6VgdodWxZI6Jh4a0jZrxxsA6Xj47GvTNToeXFK0REwxoLrBFC0miAOKP7Bpy3GOskHHZ3odXeBljbPYVXO0R7m3cZ1nagvQ3C2gZ02AB7h7tQs3cAHR2A3ebeLss9n/uCAUvuQkun6yrStN2WdWGQum1riY2F7HABWq2nQOv6KWl1gMbzXJ3bNbre91XxYnMi8g0hBDZ/bcZfDtXjiowY/GxGCkddIyIaAVhgUZ8kT0sSYuJ7bh/g8wghAJfTXXB1FmF2m6cAc28Tdhtgt7vXHZ6f9o4e24R3WwfQaoGw2wGHe93qdEJ0dABCPvf1BxKsWtOt4Op+03YVfBqdeyARnec+jdZz03Q9Xq1xr3u3a91Fbm/7abXdnkPbta5WsxWPKEQJIfDmV/V464gZ87JicNc0FldERCMFCywacpIkdRUPkVG97zPI1+icnVy4XO6iy+FwF2pn/3R2LQvvffaz9u25Lrqvt7X2/linE3A6zhvjgAo9wN2K173gOrsA055V2KnVkNSaruJOrfYWajh7u2e5PTYOss0GqDXuolGt7VkYdj539/VuRSPUal6HR3QWIQReP1CHdwsbsCAnFrdflgwVvywhIhoxWGDRsCKp1YA6AujHpPK+/rgjhHB3hXQ63AWXywE4PD87CzCns2vZ5QQcDvdAJZ03h7OrCPRu6/lTdL/P6XC3CjodgMvlvtbO1XlzuX86uy2fpaV7/Ep/cUnlKdrU3Yq5Xn6q+r5f6r6u0QAqVd/P0+OnyvO87sdIGo17XdUtnr6WO5+j17hU7C5Kir36ZS3eL27E1WPj8OP8JBZXREQjDAssIh+RJKnrg3vYAB43dCH14O6q6eoqwJxOGGJj0VB7pqsg7F4AOrv2E92Lxh5FoqtnMdfjp8u7Lnq7397Rx/2eZdnVc39x4RJQcZHYG+/fU3NOMVav08EFdBWXnUWayrPey3bp7H1U3Z67s6A8uwA8pzj0FJ8qd2EpeZc9N0nVM4az71epAEntftNJKvfvqJIASJ5lzzZ4tnfu472perSCsgtr794vbsR3x8djeV4ic0RENAKxwCIaIdxdNT3d/DzURhMkceEPgMHwEVGcXXB13uQLLTvdLYveok2G6HwO+ezCT+65LvcsFDuXtVoN5PZ2T0yyez/Z5Xkdz3JnAep5DtF5n8vpvlawt5hdrl6vI+wzJ0OY737p0WrZ1VW1e7fVrq6pGmDd64GO2C+um2TE0qkmFldERCMUCywiCglSZ9c/rW7wzzXIx8d6rvkbCkKWvYWgtwCUXYDTdVYhJwPCs3zOrdt2cdZ9Qu7qzup+QUAWAIS7lVCWPa2Fwr1dCPc+Au7Xc7m6WjfPXu7WJVW4HF1FY2exOUKwuCIiGtlYYBERBRFvt78hPDrzo//QYnFFRDSycfgvIiIiIiIiH2ELFhERjUgHDx7Ehg0bIMsy5s+fj0WLFvW4/+jRo9i4cSNOnjyJn/3sZ5g+fXqAIiUiolDilxasl156Cbfeeivuvfdef7wcERHRecmyjNdeew0rV67EunXrsGvXLlRWVvbYx2Qy4Y477sCsWbMCFCUREYUivxRYV155JVauXOmPlyIiIrqgkpISJCcnIykpCRqNBjNmzMDevXt77JOYmIiMjAxeU0VERAPily6CEydORG1trT9eioiI6IIaGhpgNBq960ajEcePH1f0XFu3bsXWrVsBAGvWrIHJZPJJjKFEo9GMyN97sJg3ZZg3ZZg3/wmqa7B4kuKbXynmTRnmTRnmLfSJXiauVtpSVVBQgIKCAu/6UA3hH8xMQzh1wXDGvCnDvCnDvCmTmpo64McEVYHFkxTf/Eoxb8owb8owb8ooOUkNFaPRCLPZ7F03m82Ij48PYERERDRccJh2IiIacbKzs1FdXY3a2lo4nU7s3r0b+fn5gQ6LiIiGAUn01k9iCNTW1uKpp57C2rVr/fFyRERE57V//35s3LgRsixj7ty5WLx4MTZv3ozs7Gzk5+ejpKQETz/9NNra2qDVahEXF4dnnnkm0GETEVGQ80sL1rPPPotf/epXqKqqwm233YaPP/7YHy8bklasWBHoEEIS86YM86YM86ZMsOUtLy8Pzz33HH7/+99j8eLFAIAlS5Z4W7JycnLw8ssv44033sD69etZXJ1HsP1tQwXzpgzzpgzzpoySvPnlGqyf/exn/ngZIiIiIiKigOI1WERERERERD6iXrVq1apAB0E9ZWVlBTqEkMS8KcO8KcO8KcO8DV/82yrDvCnDvCnDvCkz0Lz5bZALIiIiIiKi4Y5dBImIiIiIiHyEBRYREREREZGP+GUUQeqfO++8E3q9HiqVCmq1GmvWrAl0SEHppZdewv79+xEbG+udV621tRXr1q1DXV0dEhIS8POf/xxRUVEBjjS49Ja3LVu24KOPPkJMTAwA4MYbb0ReXl4gwwwq9fX1ePHFF9HU1ARJklBQUICrr76a77cL6CtvfL8NTzx39Q/PXcrw3KUMz1/K+Or8xWuwgsidd96JJ5980vvHo94dPXoUer0eL774ovdg++abbyIqKgqLFi3Cu+++i9bWVixdujTAkQaX3vK2ZcsW6PV6XHvttQGOLjg1NjaisbERWVlZsFqtWLFiBe677z58+umnfL+dR1952717N99vwxDPXf3Dc5cyPHcpw/OXMr46f7GLIIWciRMnnvNty969ezFnzhwAwJw5c7B3795AhBbUessbnV98fLx35KDw8HCkpaWhoaGB77cL6CtvRCMZz13K8NylDM9fyvjq/MUugkFm9erVAIBvfvObKCgoCHA0oaO5uRnx8fEA3P8cFoslwBGFjg8++ADbt29HVlYWfvjDH/JE1ofa2lqUlZUhJyeH77cB6J63oqIivt+GKZ67lOGxRDkeS/qP5y9lBnP+YoEVRB577DEYDAY0Nzfj8ccfR2pqKiZOnBjosGgYW7BgAa677joAwObNm7Fp0ybccccdAY4q+NhsNqxduxbLli1DREREoMMJGWfnje+34YnnLvI3Hkv6j+cvZQZ7/mIXwSBiMBgAALGxsbj00ktRUlIS4IhCR2xsLBobGwG4+8/yWoD+iYuLg0qlgkqlwvz581FaWhrokIKO0+nE2rVrMXv2bEybNg0A32/90Vve+H4bnnjuUo7HEmV4LOkfnr+U8cX5iwVWkLDZbLBard7lQ4cOIT09PcBRhY78/Hxs27YNALBt2zZceumlAY4oNHQeZAFgz549GD16dACjCT5CCLz88stIS0vDwoULvdv5fju/vvLG99vww3PX4PBYogyPJRfG85cyvjp/cRTBIHHmzBk8/fTTAACXy4VZs2Zh8eLFAY4qOD377LM4evQoWlpaEBsbi+uvvx6XXnop1q1bh/r6ephMJvziF79gf+yz9Ja3I0eOoLy8HJIkISEhAT/5yU+8fbMJKCoqwsMPP4z09HRIkgTAPTRrbm4u32/n0Vfedu3axffbMMNzV//x3KUMz13K8PyljK/OXyywiIiIiIiIfIRdBImIiIiIiHyEBRYREREREZGPsMAiIiIiIiLyERZYREREREREPsICi4iIiIiIyEdYYBGFiNraWlx//fVwuVyBDoWIiKjfeP6ikYYFFhERERERkY+wwCIiIiIiIvIRTaADIAplDQ0NWL9+PQoLC6HX63HNNdfg6quvxpYtW3Dq1CmoVCocOHAAKSkpuP3225GZmQkAqKysxJ/+9CeUl5fDYDDgpptuQn5+PgDAbrfjr3/9Kz7//HO0tbUhPT0dv/71r72vuWPHDmzevBl2ux3XXHMNFi9eHIhfnYiIQhjPX0RDhy1YRArJsoynnnoKmZmZeOWVV/Dwww/jn//8Jw4ePAgA2LdvHy6//HKsX78eM2fOxO9+9zs4nU44nU489dRTmDJlCv70pz/hlltuwfPPP4+qqioAwKZNm3DixAk8/vjj2LBhA5YuXQpJkryvW1RUhOeeew6//vWv8dZbb6GysjIgvz8REYUmnr+IhhYLLCKFSktLYbFYcN1110Gj0SApKQnz58/H7t27AQBZWVmYPn06NBoNFi5cCIfDgePHj+P48eOw2WxYtGgRNBoNJk+ejLy8POzcuROyLOOTTz7BsmXLYDAYoFKpMG7cOGi1Wu/r/td//Rd0Oh0yMzORkZGBkydPBioFREQUgnj+Ihpa7CJIpFBdXR0aGxuxbNky7zZZljFhwgSYTCYYjUbvdpVKBaPRiMbGRgCAyWSCStX1/UZCQgIaGhrQ0tICh8OB5OTkPl83Li7OuxwWFgabzebD34qIiIY7nr+IhhYLLCKFTCYTEhMT8fzzz59z35YtW2A2m73rsizDbDYjPj4eAFBfXw9Zlr0nqfr6eqSkpCA6OhparRY1NTXe/u5ERES+xPMX0dBiF0EihXJychAeHo53330XdrsdsiyjoqICJSUlAIATJ07giy++gMvlwj//+U9otVrk5uYiNzcXer0ef//73+F0OnHkyBF8+eWXmDlzJlQqFebOnYtNmzahoaEBsizj2LFjcDgcAf5tiYhouOD5i2hoSUIIEeggiEJVQ0MDNm3ahCNHjsDpdCI1NRVLlixBUVFRj1GYkpOTcdtttyErKwsAcOrUqR6jMN1444247LLLALhHYfrzn/+Mzz77DDabDZmZmXjooYfQ1NSEu+66C3/5y1+gVqsBAKtWrcLs2bMxf/78gOWAiIhCD89fREOHBRbRENiyZQtqampw9913BzoUIiKifuP5i2jw2EWQiIiIiIjIR1hgERERERER+Qi7CBIREREREfkIW7CIiIiIiIh8hAUWERERERGRj7DAIiIiIiIi8hEWWERERERERD7CAouIiIiIiMhH/j8SXnb58ehDgQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig, axes = plt.subplots(nrows=1, ncols = 2,figsize = (12,4))\n", "stats[['epoch','train_loss','valid_loss']].plot(x = 'epoch',ax=axes[0])\n", "axes[0].title.set_text('Training and Validation Loss')\n", "axes[0].set_ylabel('Loss')\n", "stats[['epoch','train_acc','valid_acc']].plot(x = 'epoch',ax = axes[1])\n", "axes[1].title.set_text('Training and Validation Accuracy')\n", "axes[1].set_ylabel('Accuracy')\n", "plt.tight_layout()\n", "plt.legend(loc = 'upper left')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "From the above, we can confidently conclude that our model is severely suffering from overfitting due to the large gap in loss and accuracy statistics.\n", "\n", "Now, let us use HiPlot to attain a more complete picture of our model's performance" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'activation': 'ReLU',\n", " 'epoch': 1,\n", " 'train_loss': 2.2567057985740937,\n", " 'train_acc': 0.1966784381663113,\n", " 'valid_loss': 2.20783610283574,\n", " 'valid_acc': 0.11471518987341772},\n", " {'activation': 'ReLU',\n", " 'epoch': 2,\n", " 'train_loss': 1.9282052176339286,\n", " 'train_acc': 0.3599413646055437,\n", " 'valid_loss': 2.658291973645174,\n", " 'valid_acc': 0.15090981012658228},\n", " {'activation': 'ReLU',\n", " 'epoch': 3,\n", " 'train_loss': 1.6841899164195762,\n", " 'train_acc': 0.459005197228145,\n", " 'valid_loss': 2.697213281559039,\n", " 'valid_acc': 0.18067642405063292},\n", " {'activation': 'ReLU',\n", " 'epoch': 4,\n", " 'train_loss': 1.5303927748950559,\n", " 'train_acc': 0.5022654584221748,\n", " 'valid_loss': 3.5326503319076346,\n", " 'valid_acc': 0.16050237341772153},\n", " {'activation': 'ReLU',\n", " 'epoch': 5,\n", " 'train_loss': 1.4438330806902986,\n", " 'train_acc': 0.5197727878464818,\n", " 'valid_loss': 3.8613895464547072,\n", " 'valid_acc': 0.16416139240506328},\n", " {'activation': 'ReLU',\n", " 'epoch': 6,\n", " 'train_loss': 1.3931915998967217,\n", " 'train_acc': 0.5263526119402985,\n", " 'valid_loss': 4.698839404914953,\n", " 'valid_acc': 0.16969936708860758},\n", " {'activation': 'ReLU',\n", " 'epoch': 7,\n", " 'train_loss': 1.358555124766791,\n", " 'train_acc': 0.5311000799573561,\n", " 'valid_loss': 5.268125509913964,\n", " 'valid_acc': 0.16337025316455697},\n", " {'activation': 'ReLU',\n", " 'epoch': 8,\n", " 'train_loss': 1.3351253029634196,\n", " 'train_acc': 0.5337653251599147,\n", " 'valid_loss': 4.9521654346321204,\n", " 'valid_acc': 0.14972310126582278},\n", " {'activation': 'ReLU',\n", " 'epoch': 9,\n", " 'train_loss': 1.3187274078824627,\n", " 'train_acc': 0.5384295042643923,\n", " 'valid_loss': 5.703450263301028,\n", " 'valid_acc': 0.15130537974683544},\n", " {'activation': 'ReLU',\n", " 'epoch': 10,\n", " 'train_loss': 1.3039471396505198,\n", " 'train_acc': 0.5426439232409381,\n", " 'valid_loss': 5.4363789618769776,\n", " 'valid_acc': 0.15644778481012658},\n", " {'activation': 'ReLU',\n", " 'epoch': 11,\n", " 'train_loss': 1.2915082008345549,\n", " 'train_acc': 0.5424940031982942,\n", " 'valid_loss': 5.915389435200751,\n", " 'valid_acc': 0.15704113924050633},\n", " {'activation': 'ReLU',\n", " 'epoch': 12,\n", " 'train_loss': 1.2787634355427107,\n", " 'train_acc': 0.5477078891257996,\n", " 'valid_loss': 5.347536111179786,\n", " 'valid_acc': 0.17543512658227847},\n", " {'activation': 'ReLU',\n", " 'epoch': 13,\n", " 'train_loss': 1.269061058060701,\n", " 'train_acc': 0.5503231609808102,\n", " 'valid_loss': 5.427630847013449,\n", " 'valid_acc': 0.14487737341772153},\n", " {'activation': 'ReLU',\n", " 'epoch': 14,\n", " 'train_loss': 1.257962289903718,\n", " 'train_acc': 0.5549706823027718,\n", " 'valid_loss': 5.108685457253758,\n", " 'valid_acc': 0.1597112341772152},\n", " {'activation': 'ReLU',\n", " 'epoch': 15,\n", " 'train_loss': 1.2495273354211087,\n", " 'train_acc': 0.5575359808102346,\n", " 'valid_loss': 5.108813322043117,\n", " 'valid_acc': 0.15110759493670886},\n", " {'activation': 'ReLU',\n", " 'epoch': 16,\n", " 'train_loss': 1.2399636860341152,\n", " 'train_acc': 0.5599680170575693,\n", " 'valid_loss': 5.530833183964597,\n", " 'valid_acc': 0.15634889240506328},\n", " {'activation': 'ReLU',\n", " 'epoch': 17,\n", " 'train_loss': 1.2324086008295576,\n", " 'train_acc': 0.5630830223880597,\n", " 'valid_loss': 5.8491543154173256,\n", " 'valid_acc': 0.14883306962025317},\n", " {'activation': 'ReLU',\n", " 'epoch': 18,\n", " 'train_loss': 1.224833994786114,\n", " 'train_acc': 0.5647654584221748,\n", " 'valid_loss': 5.7223159210591374,\n", " 'valid_acc': 0.16129351265822786},\n", " {'activation': 'ReLU',\n", " 'epoch': 19,\n", " 'train_loss': 1.2181698406683101,\n", " 'train_acc': 0.5676972281449894,\n", " 'valid_loss': 6.119402535354035,\n", " 'valid_acc': 0.17375395569620253},\n", " {'activation': 'ReLU',\n", " 'epoch': 20,\n", " 'train_loss': 1.210808662463353,\n", " 'train_acc': 0.5704957356076759,\n", " 'valid_loss': 5.5784058389784414,\n", " 'valid_acc': 0.17434731012658228},\n", " {'activation': 'ReLU',\n", " 'epoch': 21,\n", " 'train_loss': 1.2048782316098081,\n", " 'train_acc': 0.572378065031983,\n", " 'valid_loss': 5.527741637410997,\n", " 'valid_acc': 0.150810917721519},\n", " {'activation': 'ReLU',\n", " 'epoch': 22,\n", " 'train_loss': 1.1980372186916977,\n", " 'train_acc': 0.5757595948827292,\n", " 'valid_loss': 5.488918256156052,\n", " 'valid_acc': 0.17365506329113925},\n", " {'activation': 'ReLU',\n", " 'epoch': 23,\n", " 'train_loss': 1.193253962470016,\n", " 'train_acc': 0.5757762526652452,\n", " 'valid_loss': 5.657107968873616,\n", " 'valid_acc': 0.16930379746835442},\n", " {'activation': 'ReLU',\n", " 'epoch': 24,\n", " 'train_loss': 1.190566984066831,\n", " 'train_acc': 0.5796075426439232,\n", " 'valid_loss': 5.608898694002176,\n", " 'valid_acc': 0.16386471518987342},\n", " {'activation': 'ReLU',\n", " 'epoch': 25,\n", " 'train_loss': 1.1823643275669642,\n", " 'train_acc': 0.5816897654584222,\n", " 'valid_loss': 5.459799464744858,\n", " 'valid_acc': 0.15822784810126583}]" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# organize data to hiplot format\n", "\n", "data = []\n", "for row in stats.iterrows():\n", " data.append(row[1].to_dict())\n", "data" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "application/javascript": [ "/*!\n", " * Copyright (c) Facebook, Inc. and its affiliates.\n", " * \n", " * This source code is licensed under the MIT license found in the\n", " * LICENSE file in the root directory of this source tree.\n", " */!function(t){var e={};function o(r){if(e[r])return e[r].exports;var n=e[r]={i:r,l:!1,exports:{}};return t[r].call(n.exports,n,n.exports,o),n.l=!0,n.exports}o.m=t,o.c=e,o.d=function(t,e,r){o.o(t,e)||Object.defineProperty(t,e,{enumerable:!0,get:r})},o.r=function(t){\"undefined\"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(t,Symbol.toStringTag,{value:\"Module\"}),Object.defineProperty(t,\"__esModule\",{value:!0})},o.t=function(t,e){if(1&e&&(t=o(t)),8&e)return t;if(4&e&&\"object\"==typeof t&&t&&t.__esModule)return t;var r=Object.create(null);if(o.r(r),Object.defineProperty(r,\"default\",{enumerable:!0,value:t}),2&e&&\"string\"!=typeof t)for(var n in t)o.d(r,n,function(e){return t[e]}.bind(null,n));return r},o.n=function(t){var e=t&&t.__esModule?function(){return t.default}:function(){return t};return o.d(e,\"a\",e),e},o.o=function(t,e){return Object.prototype.hasOwnProperty.call(t,e)},o.p=\"\",o(o.s=60)}([function(t,e,o){\"use strict\";t.exports=o(85)},function(t,e,o){\n", "/*!\n", " * jQuery JavaScript Library v3.4.1\n", " * https://jquery.com/\n", " *\n", " * Includes Sizzle.js\n", " * https://sizzlejs.com/\n", " *\n", " * Copyright JS Foundation and other contributors\n", " * Released under the MIT license\n", " * https://jquery.org/license\n", " *\n", " * Date: 2019-05-01T21:04Z\n", " */\n", "!function(e,o){\"use strict\";\"object\"==typeof t.exports?t.exports=e.document?o(e,!0):function(t){if(!t.document)throw new Error(\"jQuery requires a window with a document\");return o(t)}:o(e)}(\"undefined\"!=typeof window?window:this,(function(t,e){\"use strict\";var o=[],r=t.document,n=Object.getPrototypeOf,s=o.slice,a=o.concat,i=o.push,c=o.indexOf,l={},u=l.toString,p=l.hasOwnProperty,d=p.toString,f=d.call(Object),h={},m=function(t){return\"function\"==typeof t&&\"number\"!=typeof t.nodeType},b=function(t){return null!=t&&t===t.window},g={type:!0,src:!0,nonce:!0,noModule:!0};function _(t,e,o){var n,s,a=(o=o||r).createElement(\"script\");if(a.text=t,e)for(n in g)(s=e[n]||e.getAttribute&&e.getAttribute(n))&&a.setAttribute(n,s);o.head.appendChild(a).parentNode.removeChild(a)}function v(t){return null==t?t+\"\":\"object\"==typeof t||\"function\"==typeof t?l[u.call(t)]||\"object\":typeof t}var y=function(t,e){return new y.fn.init(t,e)},x=/^[\\s\\uFEFF\\xA0]+|[\\s\\uFEFF\\xA0]+$/g;function w(t){var e=!!t&&\"length\"in t&&t.length,o=v(t);return!m(t)&&!b(t)&&(\"array\"===o||0===e||\"number\"==typeof e&&e>0&&e-1 in t)}y.fn=y.prototype={jquery:\"3.4.1\",constructor:y,length:0,toArray:function(){return s.call(this)},get:function(t){return null==t?s.call(this):t<0?this[t+this.length]:this[t]},pushStack:function(t){var e=y.merge(this.constructor(),t);return e.prevObject=this,e},each:function(t){return y.each(this,t)},map:function(t){return this.pushStack(y.map(this,(function(e,o){return t.call(e,o,e)})))},slice:function(){return this.pushStack(s.apply(this,arguments))},first:function(){return this.eq(0)},last:function(){return this.eq(-1)},eq:function(t){var e=this.length,o=+t+(t<0?e:0);return this.pushStack(o>=0&&o+~]|\"+P+\")\"+P+\"*\"),$=new RegExp(P+\"|>\"),Y=new RegExp(O),V=new RegExp(\"^\"+R+\"$\"),X={ID:new RegExp(\"^#(\"+R+\")\"),CLASS:new RegExp(\"^\\\\.(\"+R+\")\"),TAG:new RegExp(\"^(\"+R+\"|[*])\"),ATTR:new RegExp(\"^\"+z),PSEUDO:new RegExp(\"^\"+O),CHILD:new RegExp(\"^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\\\(\"+P+\"*(even|odd|(([+-]|)(\\\\d*)n|)\"+P+\"*(?:([+-]|)\"+P+\"*(\\\\d+)|))\"+P+\"*\\\\)|)\",\"i\"),bool:new RegExp(\"^(?:\"+B+\")$\",\"i\"),needsContext:new RegExp(\"^\"+P+\"*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\\\(\"+P+\"*((?:-\\\\d)?\\\\d*)\"+P+\"*\\\\)|)(?=[^-]|$)\",\"i\")},G=/HTML$/i,Q=/^(?:input|select|textarea|button)$/i,K=/^h\\d$/i,J=/^[^{]+\\{\\s*\\[native \\w/,Z=/^(?:#([\\w-]+)|(\\w+)|\\.([\\w-]+))$/,tt=/[+~]/,et=new RegExp(\"\\\\\\\\([\\\\da-f]{1,6}\"+P+\"?|(\"+P+\")|.)\",\"ig\"),ot=function(t,e,o){var r=\"0x\"+e-65536;return r!=r||o?e:r<0?String.fromCharCode(r+65536):String.fromCharCode(r>>10|55296,1023&r|56320)},rt=/([\\0-\\x1f\\x7f]|^-?\\d)|^-$|[^\\0-\\x1f\\x7f-\\uFFFF\\w-]/g,nt=function(t,e){return e?\"\\0\"===t?\"�\":t.slice(0,-1)+\"\\\\\"+t.charCodeAt(t.length-1).toString(16)+\" \":\"\\\\\"+t},st=function(){d()},at=yt((function(t){return!0===t.disabled&&\"fieldset\"===t.nodeName.toLowerCase()}),{dir:\"parentNode\",next:\"legend\"});try{j.apply(F=L.call(x.childNodes),x.childNodes),F[x.childNodes.length].nodeType}catch(t){j={apply:F.length?function(t,e){N.apply(t,L.call(e))}:function(t,e){for(var o=t.length,r=0;t[o++]=e[r++];);t.length=o-1}}}function it(t,e,r,n){var s,i,l,u,p,h,g,_=e&&e.ownerDocument,w=e?e.nodeType:9;if(r=r||[],\"string\"!=typeof t||!t||1!==w&&9!==w&&11!==w)return r;if(!n&&((e?e.ownerDocument||e:x)!==f&&d(e),e=e||f,m)){if(11!==w&&(p=Z.exec(t)))if(s=p[1]){if(9===w){if(!(l=e.getElementById(s)))return r;if(l.id===s)return r.push(l),r}else if(_&&(l=_.getElementById(s))&&v(e,l)&&l.id===s)return r.push(l),r}else{if(p[2])return j.apply(r,e.getElementsByTagName(t)),r;if((s=p[3])&&o.getElementsByClassName&&e.getElementsByClassName)return j.apply(r,e.getElementsByClassName(s)),r}if(o.qsa&&!T[t+\" \"]&&(!b||!b.test(t))&&(1!==w||\"object\"!==e.nodeName.toLowerCase())){if(g=t,_=e,1===w&&$.test(t)){for((u=e.getAttribute(\"id\"))?u=u.replace(rt,nt):e.setAttribute(\"id\",u=y),i=(h=a(t)).length;i--;)h[i]=\"#\"+u+\" \"+vt(h[i]);g=h.join(\",\"),_=tt.test(t)&>(e.parentNode)||e}try{return j.apply(r,_.querySelectorAll(g)),r}catch(e){T(t,!0)}finally{u===y&&e.removeAttribute(\"id\")}}}return c(t.replace(U,\"$1\"),e,r,n)}function ct(){var t=[];return function e(o,n){return t.push(o+\" \")>r.cacheLength&&delete e[t.shift()],e[o+\" \"]=n}}function lt(t){return t[y]=!0,t}function ut(t){var e=f.createElement(\"fieldset\");try{return!!t(e)}catch(t){return!1}finally{e.parentNode&&e.parentNode.removeChild(e),e=null}}function pt(t,e){for(var o=t.split(\"|\"),n=o.length;n--;)r.attrHandle[o[n]]=e}function dt(t,e){var o=e&&t,r=o&&1===t.nodeType&&1===e.nodeType&&t.sourceIndex-e.sourceIndex;if(r)return r;if(o)for(;o=o.nextSibling;)if(o===e)return-1;return t?1:-1}function ft(t){return function(e){return\"input\"===e.nodeName.toLowerCase()&&e.type===t}}function ht(t){return function(e){var o=e.nodeName.toLowerCase();return(\"input\"===o||\"button\"===o)&&e.type===t}}function mt(t){return function(e){return\"form\"in e?e.parentNode&&!1===e.disabled?\"label\"in e?\"label\"in e.parentNode?e.parentNode.disabled===t:e.disabled===t:e.isDisabled===t||e.isDisabled!==!t&&at(e)===t:e.disabled===t:\"label\"in e&&e.disabled===t}}function bt(t){return lt((function(e){return e=+e,lt((function(o,r){for(var n,s=t([],o.length,e),a=s.length;a--;)o[n=s[a]]&&(o[n]=!(r[n]=o[n]))}))}))}function gt(t){return t&&void 0!==t.getElementsByTagName&&t}for(e in o=it.support={},s=it.isXML=function(t){var e=t.namespaceURI,o=(t.ownerDocument||t).documentElement;return!G.test(e||o&&o.nodeName||\"HTML\")},d=it.setDocument=function(t){var e,n,a=t?t.ownerDocument||t:x;return a!==f&&9===a.nodeType&&a.documentElement?(h=(f=a).documentElement,m=!s(f),x!==f&&(n=f.defaultView)&&n.top!==n&&(n.addEventListener?n.addEventListener(\"unload\",st,!1):n.attachEvent&&n.attachEvent(\"onunload\",st)),o.attributes=ut((function(t){return t.className=\"i\",!t.getAttribute(\"className\")})),o.getElementsByTagName=ut((function(t){return t.appendChild(f.createComment(\"\")),!t.getElementsByTagName(\"*\").length})),o.getElementsByClassName=J.test(f.getElementsByClassName),o.getById=ut((function(t){return h.appendChild(t).id=y,!f.getElementsByName||!f.getElementsByName(y).length})),o.getById?(r.filter.ID=function(t){var e=t.replace(et,ot);return function(t){return t.getAttribute(\"id\")===e}},r.find.ID=function(t,e){if(void 0!==e.getElementById&&m){var o=e.getElementById(t);return o?[o]:[]}}):(r.filter.ID=function(t){var e=t.replace(et,ot);return function(t){var o=void 0!==t.getAttributeNode&&t.getAttributeNode(\"id\");return o&&o.value===e}},r.find.ID=function(t,e){if(void 0!==e.getElementById&&m){var o,r,n,s=e.getElementById(t);if(s){if((o=s.getAttributeNode(\"id\"))&&o.value===t)return[s];for(n=e.getElementsByName(t),r=0;s=n[r++];)if((o=s.getAttributeNode(\"id\"))&&o.value===t)return[s]}return[]}}),r.find.TAG=o.getElementsByTagName?function(t,e){return void 0!==e.getElementsByTagName?e.getElementsByTagName(t):o.qsa?e.querySelectorAll(t):void 0}:function(t,e){var o,r=[],n=0,s=e.getElementsByTagName(t);if(\"*\"===t){for(;o=s[n++];)1===o.nodeType&&r.push(o);return r}return s},r.find.CLASS=o.getElementsByClassName&&function(t,e){if(void 0!==e.getElementsByClassName&&m)return e.getElementsByClassName(t)},g=[],b=[],(o.qsa=J.test(f.querySelectorAll))&&(ut((function(t){h.appendChild(t).innerHTML=\"\",t.querySelectorAll(\"[msallowcapture^='']\").length&&b.push(\"[*^$]=\"+P+\"*(?:''|\\\"\\\")\"),t.querySelectorAll(\"[selected]\").length||b.push(\"\\\\[\"+P+\"*(?:value|\"+B+\")\"),t.querySelectorAll(\"[id~=\"+y+\"-]\").length||b.push(\"~=\"),t.querySelectorAll(\":checked\").length||b.push(\":checked\"),t.querySelectorAll(\"a#\"+y+\"+*\").length||b.push(\".#.+[+~]\")})),ut((function(t){t.innerHTML=\"\";var e=f.createElement(\"input\");e.setAttribute(\"type\",\"hidden\"),t.appendChild(e).setAttribute(\"name\",\"D\"),t.querySelectorAll(\"[name=d]\").length&&b.push(\"name\"+P+\"*[*^$|!~]?=\"),2!==t.querySelectorAll(\":enabled\").length&&b.push(\":enabled\",\":disabled\"),h.appendChild(t).disabled=!0,2!==t.querySelectorAll(\":disabled\").length&&b.push(\":enabled\",\":disabled\"),t.querySelectorAll(\"*,:x\"),b.push(\",.*:\")}))),(o.matchesSelector=J.test(_=h.matches||h.webkitMatchesSelector||h.mozMatchesSelector||h.oMatchesSelector||h.msMatchesSelector))&&ut((function(t){o.disconnectedMatch=_.call(t,\"*\"),_.call(t,\"[s!='']:x\"),g.push(\"!=\",O)})),b=b.length&&new RegExp(b.join(\"|\")),g=g.length&&new RegExp(g.join(\"|\")),e=J.test(h.compareDocumentPosition),v=e||J.test(h.contains)?function(t,e){var o=9===t.nodeType?t.documentElement:t,r=e&&e.parentNode;return t===r||!(!r||1!==r.nodeType||!(o.contains?o.contains(r):t.compareDocumentPosition&&16&t.compareDocumentPosition(r)))}:function(t,e){if(e)for(;e=e.parentNode;)if(e===t)return!0;return!1},A=e?function(t,e){if(t===e)return p=!0,0;var r=!t.compareDocumentPosition-!e.compareDocumentPosition;return r||(1&(r=(t.ownerDocument||t)===(e.ownerDocument||e)?t.compareDocumentPosition(e):1)||!o.sortDetached&&e.compareDocumentPosition(t)===r?t===f||t.ownerDocument===x&&v(x,t)?-1:e===f||e.ownerDocument===x&&v(x,e)?1:u?I(u,t)-I(u,e):0:4&r?-1:1)}:function(t,e){if(t===e)return p=!0,0;var o,r=0,n=t.parentNode,s=e.parentNode,a=[t],i=[e];if(!n||!s)return t===f?-1:e===f?1:n?-1:s?1:u?I(u,t)-I(u,e):0;if(n===s)return dt(t,e);for(o=t;o=o.parentNode;)a.unshift(o);for(o=e;o=o.parentNode;)i.unshift(o);for(;a[r]===i[r];)r++;return r?dt(a[r],i[r]):a[r]===x?-1:i[r]===x?1:0},f):f},it.matches=function(t,e){return it(t,null,null,e)},it.matchesSelector=function(t,e){if((t.ownerDocument||t)!==f&&d(t),o.matchesSelector&&m&&!T[e+\" \"]&&(!g||!g.test(e))&&(!b||!b.test(e)))try{var r=_.call(t,e);if(r||o.disconnectedMatch||t.document&&11!==t.document.nodeType)return r}catch(t){T(e,!0)}return it(e,f,null,[t]).length>0},it.contains=function(t,e){return(t.ownerDocument||t)!==f&&d(t),v(t,e)},it.attr=function(t,e){(t.ownerDocument||t)!==f&&d(t);var n=r.attrHandle[e.toLowerCase()],s=n&&S.call(r.attrHandle,e.toLowerCase())?n(t,e,!m):void 0;return void 0!==s?s:o.attributes||!m?t.getAttribute(e):(s=t.getAttributeNode(e))&&s.specified?s.value:null},it.escape=function(t){return(t+\"\").replace(rt,nt)},it.error=function(t){throw new Error(\"Syntax error, unrecognized expression: \"+t)},it.uniqueSort=function(t){var e,r=[],n=0,s=0;if(p=!o.detectDuplicates,u=!o.sortStable&&t.slice(0),t.sort(A),p){for(;e=t[s++];)e===t[s]&&(n=r.push(s));for(;n--;)t.splice(r[n],1)}return u=null,t},n=it.getText=function(t){var e,o=\"\",r=0,s=t.nodeType;if(s){if(1===s||9===s||11===s){if(\"string\"==typeof t.textContent)return t.textContent;for(t=t.firstChild;t;t=t.nextSibling)o+=n(t)}else if(3===s||4===s)return t.nodeValue}else for(;e=t[r++];)o+=n(e);return o},(r=it.selectors={cacheLength:50,createPseudo:lt,match:X,attrHandle:{},find:{},relative:{\">\":{dir:\"parentNode\",first:!0},\" \":{dir:\"parentNode\"},\"+\":{dir:\"previousSibling\",first:!0},\"~\":{dir:\"previousSibling\"}},preFilter:{ATTR:function(t){return t[1]=t[1].replace(et,ot),t[3]=(t[3]||t[4]||t[5]||\"\").replace(et,ot),\"~=\"===t[2]&&(t[3]=\" \"+t[3]+\" \"),t.slice(0,4)},CHILD:function(t){return t[1]=t[1].toLowerCase(),\"nth\"===t[1].slice(0,3)?(t[3]||it.error(t[0]),t[4]=+(t[4]?t[5]+(t[6]||1):2*(\"even\"===t[3]||\"odd\"===t[3])),t[5]=+(t[7]+t[8]||\"odd\"===t[3])):t[3]&&it.error(t[0]),t},PSEUDO:function(t){var e,o=!t[6]&&t[2];return X.CHILD.test(t[0])?null:(t[3]?t[2]=t[4]||t[5]||\"\":o&&Y.test(o)&&(e=a(o,!0))&&(e=o.indexOf(\")\",o.length-e)-o.length)&&(t[0]=t[0].slice(0,e),t[2]=o.slice(0,e)),t.slice(0,3))}},filter:{TAG:function(t){var e=t.replace(et,ot).toLowerCase();return\"*\"===t?function(){return!0}:function(t){return t.nodeName&&t.nodeName.toLowerCase()===e}},CLASS:function(t){var e=C[t+\" \"];return e||(e=new RegExp(\"(^|\"+P+\")\"+t+\"(\"+P+\"|$)\"))&&C(t,(function(t){return e.test(\"string\"==typeof t.className&&t.className||void 0!==t.getAttribute&&t.getAttribute(\"class\")||\"\")}))},ATTR:function(t,e,o){return function(r){var n=it.attr(r,t);return null==n?\"!=\"===e:!e||(n+=\"\",\"=\"===e?n===o:\"!=\"===e?n!==o:\"^=\"===e?o&&0===n.indexOf(o):\"*=\"===e?o&&n.indexOf(o)>-1:\"$=\"===e?o&&n.slice(-o.length)===o:\"~=\"===e?(\" \"+n.replace(H,\" \")+\" \").indexOf(o)>-1:\"|=\"===e&&(n===o||n.slice(0,o.length+1)===o+\"-\"))}},CHILD:function(t,e,o,r,n){var s=\"nth\"!==t.slice(0,3),a=\"last\"!==t.slice(-4),i=\"of-type\"===e;return 1===r&&0===n?function(t){return!!t.parentNode}:function(e,o,c){var l,u,p,d,f,h,m=s!==a?\"nextSibling\":\"previousSibling\",b=e.parentNode,g=i&&e.nodeName.toLowerCase(),_=!c&&!i,v=!1;if(b){if(s){for(;m;){for(d=e;d=d[m];)if(i?d.nodeName.toLowerCase()===g:1===d.nodeType)return!1;h=m=\"only\"===t&&!h&&\"nextSibling\"}return!0}if(h=[a?b.firstChild:b.lastChild],a&&_){for(v=(f=(l=(u=(p=(d=b)[y]||(d[y]={}))[d.uniqueID]||(p[d.uniqueID]={}))[t]||[])[0]===w&&l[1])&&l[2],d=f&&b.childNodes[f];d=++f&&d&&d[m]||(v=f=0)||h.pop();)if(1===d.nodeType&&++v&&d===e){u[t]=[w,f,v];break}}else if(_&&(v=f=(l=(u=(p=(d=e)[y]||(d[y]={}))[d.uniqueID]||(p[d.uniqueID]={}))[t]||[])[0]===w&&l[1]),!1===v)for(;(d=++f&&d&&d[m]||(v=f=0)||h.pop())&&((i?d.nodeName.toLowerCase()!==g:1!==d.nodeType)||!++v||(_&&((u=(p=d[y]||(d[y]={}))[d.uniqueID]||(p[d.uniqueID]={}))[t]=[w,v]),d!==e)););return(v-=n)===r||v%r==0&&v/r>=0}}},PSEUDO:function(t,e){var o,n=r.pseudos[t]||r.setFilters[t.toLowerCase()]||it.error(\"unsupported pseudo: \"+t);return n[y]?n(e):n.length>1?(o=[t,t,\"\",e],r.setFilters.hasOwnProperty(t.toLowerCase())?lt((function(t,o){for(var r,s=n(t,e),a=s.length;a--;)t[r=I(t,s[a])]=!(o[r]=s[a])})):function(t){return n(t,0,o)}):n}},pseudos:{not:lt((function(t){var e=[],o=[],r=i(t.replace(U,\"$1\"));return r[y]?lt((function(t,e,o,n){for(var s,a=r(t,null,n,[]),i=t.length;i--;)(s=a[i])&&(t[i]=!(e[i]=s))})):function(t,n,s){return e[0]=t,r(e,null,s,o),e[0]=null,!o.pop()}})),has:lt((function(t){return function(e){return it(t,e).length>0}})),contains:lt((function(t){return t=t.replace(et,ot),function(e){return(e.textContent||n(e)).indexOf(t)>-1}})),lang:lt((function(t){return V.test(t||\"\")||it.error(\"unsupported lang: \"+t),t=t.replace(et,ot).toLowerCase(),function(e){var o;do{if(o=m?e.lang:e.getAttribute(\"xml:lang\")||e.getAttribute(\"lang\"))return(o=o.toLowerCase())===t||0===o.indexOf(t+\"-\")}while((e=e.parentNode)&&1===e.nodeType);return!1}})),target:function(e){var o=t.location&&t.location.hash;return o&&o.slice(1)===e.id},root:function(t){return t===h},focus:function(t){return t===f.activeElement&&(!f.hasFocus||f.hasFocus())&&!!(t.type||t.href||~t.tabIndex)},enabled:mt(!1),disabled:mt(!0),checked:function(t){var e=t.nodeName.toLowerCase();return\"input\"===e&&!!t.checked||\"option\"===e&&!!t.selected},selected:function(t){return t.parentNode&&t.parentNode.selectedIndex,!0===t.selected},empty:function(t){for(t=t.firstChild;t;t=t.nextSibling)if(t.nodeType<6)return!1;return!0},parent:function(t){return!r.pseudos.empty(t)},header:function(t){return K.test(t.nodeName)},input:function(t){return Q.test(t.nodeName)},button:function(t){var e=t.nodeName.toLowerCase();return\"input\"===e&&\"button\"===t.type||\"button\"===e},text:function(t){var e;return\"input\"===t.nodeName.toLowerCase()&&\"text\"===t.type&&(null==(e=t.getAttribute(\"type\"))||\"text\"===e.toLowerCase())},first:bt((function(){return[0]})),last:bt((function(t,e){return[e-1]})),eq:bt((function(t,e,o){return[o<0?o+e:o]})),even:bt((function(t,e){for(var o=0;oe?e:o;--r>=0;)t.push(r);return t})),gt:bt((function(t,e,o){for(var r=o<0?o+e:o;++r1?function(e,o,r){for(var n=t.length;n--;)if(!t[n](e,o,r))return!1;return!0}:t[0]}function wt(t,e,o,r,n){for(var s,a=[],i=0,c=t.length,l=null!=e;i-1&&(s[l]=!(a[l]=p))}}else g=wt(g===a?g.splice(h,g.length):g),n?n(null,a,g,c):j.apply(a,g)}))}function Ct(t){for(var e,o,n,s=t.length,a=r.relative[t[0].type],i=a||r.relative[\" \"],c=a?1:0,u=yt((function(t){return t===e}),i,!0),p=yt((function(t){return I(e,t)>-1}),i,!0),d=[function(t,o,r){var n=!a&&(r||o!==l)||((e=o).nodeType?u(t,o,r):p(t,o,r));return e=null,n}];c1&&xt(d),c>1&&vt(t.slice(0,c-1).concat({value:\" \"===t[c-2].type?\"*\":\"\"})).replace(U,\"$1\"),o,c0,n=t.length>0,s=function(s,a,i,c,u){var p,h,b,g=0,_=\"0\",v=s&&[],y=[],x=l,D=s||n&&r.find.TAG(\"*\",u),C=w+=null==x?1:Math.random()||.1,k=D.length;for(u&&(l=a===f||a||u);_!==k&&null!=(p=D[_]);_++){if(n&&p){for(h=0,a||p.ownerDocument===f||(d(p),i=!m);b=t[h++];)if(b(p,a||f,i)){c.push(p);break}u&&(w=C)}o&&((p=!b&&p)&&g--,s&&v.push(p))}if(g+=_,o&&_!==g){for(h=0;b=e[h++];)b(v,y,a,i);if(s){if(g>0)for(;_--;)v[_]||y[_]||(y[_]=M.call(c));y=wt(y)}j.apply(c,y),u&&!s&&y.length>0&&g+e.length>1&&it.uniqueSort(c)}return u&&(w=C,l=x),v};return o?lt(s):s}(s,n))).selector=t}return i},c=it.select=function(t,e,o,n){var s,c,l,u,p,d=\"function\"==typeof t&&t,f=!n&&a(t=d.selector||t);if(o=o||[],1===f.length){if((c=f[0]=f[0].slice(0)).length>2&&\"ID\"===(l=c[0]).type&&9===e.nodeType&&m&&r.relative[c[1].type]){if(!(e=(r.find.ID(l.matches[0].replace(et,ot),e)||[])[0]))return o;d&&(e=e.parentNode),t=t.slice(c.shift().value.length)}for(s=X.needsContext.test(t)?0:c.length;s--&&(l=c[s],!r.relative[u=l.type]);)if((p=r.find[u])&&(n=p(l.matches[0].replace(et,ot),tt.test(c[0].type)&>(e.parentNode)||e))){if(c.splice(s,1),!(t=n.length&&vt(c)))return j.apply(o,n),o;break}}return(d||i(t,f))(n,e,!m,o,!e||tt.test(t)&>(e.parentNode)||e),o},o.sortStable=y.split(\"\").sort(A).join(\"\")===y,o.detectDuplicates=!!p,d(),o.sortDetached=ut((function(t){return 1&t.compareDocumentPosition(f.createElement(\"fieldset\"))})),ut((function(t){return t.innerHTML=\"\",\"#\"===t.firstChild.getAttribute(\"href\")}))||pt(\"type|href|height|width\",(function(t,e,o){if(!o)return t.getAttribute(e,\"type\"===e.toLowerCase()?1:2)})),o.attributes&&ut((function(t){return t.innerHTML=\"\",t.firstChild.setAttribute(\"value\",\"\"),\"\"===t.firstChild.getAttribute(\"value\")}))||pt(\"value\",(function(t,e,o){if(!o&&\"input\"===t.nodeName.toLowerCase())return t.defaultValue})),ut((function(t){return null==t.getAttribute(\"disabled\")}))||pt(B,(function(t,e,o){var r;if(!o)return!0===t[e]?e.toLowerCase():(r=t.getAttributeNode(e))&&r.specified?r.value:null})),it}(t);y.find=D,y.expr=D.selectors,y.expr[\":\"]=y.expr.pseudos,y.uniqueSort=y.unique=D.uniqueSort,y.text=D.getText,y.isXMLDoc=D.isXML,y.contains=D.contains,y.escapeSelector=D.escape;var C=function(t,e,o){for(var r=[],n=void 0!==o;(t=t[e])&&9!==t.nodeType;)if(1===t.nodeType){if(n&&y(t).is(o))break;r.push(t)}return r},k=function(t,e){for(var o=[];t;t=t.nextSibling)1===t.nodeType&&t!==e&&o.push(t);return o},E=y.expr.match.needsContext;function T(t,e){return t.nodeName&&t.nodeName.toLowerCase()===e.toLowerCase()}var A=/^<([a-z][^\\/\\0>:\\x20\\t\\r\\n\\f]*)[\\x20\\t\\r\\n\\f]*\\/?>(?:<\\/\\1>|)$/i;function S(t,e,o){return m(e)?y.grep(t,(function(t,r){return!!e.call(t,r,t)!==o})):e.nodeType?y.grep(t,(function(t){return t===e!==o})):\"string\"!=typeof e?y.grep(t,(function(t){return c.call(e,t)>-1!==o})):y.filter(e,t,o)}y.filter=function(t,e,o){var r=e[0];return o&&(t=\":not(\"+t+\")\"),1===e.length&&1===r.nodeType?y.find.matchesSelector(r,t)?[r]:[]:y.find.matches(t,y.grep(e,(function(t){return 1===t.nodeType})))},y.fn.extend({find:function(t){var e,o,r=this.length,n=this;if(\"string\"!=typeof t)return this.pushStack(y(t).filter((function(){for(e=0;e1?y.uniqueSort(o):o},filter:function(t){return this.pushStack(S(this,t||[],!1))},not:function(t){return this.pushStack(S(this,t||[],!0))},is:function(t){return!!S(this,\"string\"==typeof t&&E.test(t)?y(t):t||[],!1).length}});var F,M=/^(?:\\s*(<[\\w\\W]+>)[^>]*|#([\\w-]+))$/;(y.fn.init=function(t,e,o){var n,s;if(!t)return this;if(o=o||F,\"string\"==typeof t){if(!(n=\"<\"===t[0]&&\">\"===t[t.length-1]&&t.length>=3?[null,t,null]:M.exec(t))||!n[1]&&e)return!e||e.jquery?(e||o).find(t):this.constructor(e).find(t);if(n[1]){if(e=e instanceof y?e[0]:e,y.merge(this,y.parseHTML(n[1],e&&e.nodeType?e.ownerDocument||e:r,!0)),A.test(n[1])&&y.isPlainObject(e))for(n in e)m(this[n])?this[n](e[n]):this.attr(n,e[n]);return this}return(s=r.getElementById(n[2]))&&(this[0]=s,this.length=1),this}return t.nodeType?(this[0]=t,this.length=1,this):m(t)?void 0!==o.ready?o.ready(t):t(y):y.makeArray(t,this)}).prototype=y.fn,F=y(r);var N=/^(?:parents|prev(?:Until|All))/,j={children:!0,contents:!0,next:!0,prev:!0};function L(t,e){for(;(t=t[e])&&1!==t.nodeType;);return t}y.fn.extend({has:function(t){var e=y(t,this),o=e.length;return this.filter((function(){for(var t=0;t-1:1===o.nodeType&&y.find.matchesSelector(o,t))){s.push(o);break}return this.pushStack(s.length>1?y.uniqueSort(s):s)},index:function(t){return t?\"string\"==typeof t?c.call(y(t),this[0]):c.call(this,t.jquery?t[0]:t):this[0]&&this[0].parentNode?this.first().prevAll().length:-1},add:function(t,e){return this.pushStack(y.uniqueSort(y.merge(this.get(),y(t,e))))},addBack:function(t){return this.add(null==t?this.prevObject:this.prevObject.filter(t))}}),y.each({parent:function(t){var e=t.parentNode;return e&&11!==e.nodeType?e:null},parents:function(t){return C(t,\"parentNode\")},parentsUntil:function(t,e,o){return C(t,\"parentNode\",o)},next:function(t){return L(t,\"nextSibling\")},prev:function(t){return L(t,\"previousSibling\")},nextAll:function(t){return C(t,\"nextSibling\")},prevAll:function(t){return C(t,\"previousSibling\")},nextUntil:function(t,e,o){return C(t,\"nextSibling\",o)},prevUntil:function(t,e,o){return C(t,\"previousSibling\",o)},siblings:function(t){return k((t.parentNode||{}).firstChild,t)},children:function(t){return k(t.firstChild)},contents:function(t){return void 0!==t.contentDocument?t.contentDocument:(T(t,\"template\")&&(t=t.content||t),y.merge([],t.childNodes))}},(function(t,e){y.fn[t]=function(o,r){var n=y.map(this,e,o);return\"Until\"!==t.slice(-5)&&(r=o),r&&\"string\"==typeof r&&(n=y.filter(r,n)),this.length>1&&(j[t]||y.uniqueSort(n),N.test(t)&&n.reverse()),this.pushStack(n)}}));var I=/[^\\x20\\t\\r\\n\\f]+/g;function B(t){return t}function P(t){throw t}function R(t,e,o,r){var n;try{t&&m(n=t.promise)?n.call(t).done(e).fail(o):t&&m(n=t.then)?n.call(t,e,o):e.apply(void 0,[t].slice(r))}catch(t){o.apply(void 0,[t])}}y.Callbacks=function(t){t=\"string\"==typeof t?function(t){var e={};return y.each(t.match(I)||[],(function(t,o){e[o]=!0})),e}(t):y.extend({},t);var e,o,r,n,s=[],a=[],i=-1,c=function(){for(n=n||t.once,r=e=!0;a.length;i=-1)for(o=a.shift();++i-1;)s.splice(o,1),o<=i&&i--})),this},has:function(t){return t?y.inArray(t,s)>-1:s.length>0},empty:function(){return s&&(s=[]),this},disable:function(){return n=a=[],s=o=\"\",this},disabled:function(){return!s},lock:function(){return n=a=[],o||e||(s=o=\"\"),this},locked:function(){return!!n},fireWith:function(t,o){return n||(o=[t,(o=o||[]).slice?o.slice():o],a.push(o),e||c()),this},fire:function(){return l.fireWith(this,arguments),this},fired:function(){return!!r}};return l},y.extend({Deferred:function(e){var o=[[\"notify\",\"progress\",y.Callbacks(\"memory\"),y.Callbacks(\"memory\"),2],[\"resolve\",\"done\",y.Callbacks(\"once memory\"),y.Callbacks(\"once memory\"),0,\"resolved\"],[\"reject\",\"fail\",y.Callbacks(\"once memory\"),y.Callbacks(\"once memory\"),1,\"rejected\"]],r=\"pending\",n={state:function(){return r},always:function(){return s.done(arguments).fail(arguments),this},catch:function(t){return n.then(null,t)},pipe:function(){var t=arguments;return y.Deferred((function(e){y.each(o,(function(o,r){var n=m(t[r[4]])&&t[r[4]];s[r[1]]((function(){var t=n&&n.apply(this,arguments);t&&m(t.promise)?t.promise().progress(e.notify).done(e.resolve).fail(e.reject):e[r[0]+\"With\"](this,n?[t]:arguments)}))})),t=null})).promise()},then:function(e,r,n){var s=0;function a(e,o,r,n){return function(){var i=this,c=arguments,l=function(){var t,l;if(!(e=s&&(r!==P&&(i=void 0,c=[t]),o.rejectWith(i,c))}};e?u():(y.Deferred.getStackHook&&(u.stackTrace=y.Deferred.getStackHook()),t.setTimeout(u))}}return y.Deferred((function(t){o[0][3].add(a(0,t,m(n)?n:B,t.notifyWith)),o[1][3].add(a(0,t,m(e)?e:B)),o[2][3].add(a(0,t,m(r)?r:P))})).promise()},promise:function(t){return null!=t?y.extend(t,n):n}},s={};return y.each(o,(function(t,e){var a=e[2],i=e[5];n[e[1]]=a.add,i&&a.add((function(){r=i}),o[3-t][2].disable,o[3-t][3].disable,o[0][2].lock,o[0][3].lock),a.add(e[3].fire),s[e[0]]=function(){return s[e[0]+\"With\"](this===s?void 0:this,arguments),this},s[e[0]+\"With\"]=a.fireWith})),n.promise(s),e&&e.call(s,s),s},when:function(t){var e=arguments.length,o=e,r=Array(o),n=s.call(arguments),a=y.Deferred(),i=function(t){return function(o){r[t]=this,n[t]=arguments.length>1?s.call(arguments):o,--e||a.resolveWith(r,n)}};if(e<=1&&(R(t,a.done(i(o)).resolve,a.reject,!e),\"pending\"===a.state()||m(n[o]&&n[o].then)))return a.then();for(;o--;)R(n[o],i(o),a.reject);return a.promise()}});var z=/^(Eval|Internal|Range|Reference|Syntax|Type|URI)Error$/;y.Deferred.exceptionHook=function(e,o){t.console&&t.console.warn&&e&&z.test(e.name)&&t.console.warn(\"jQuery.Deferred exception: \"+e.message,e.stack,o)},y.readyException=function(e){t.setTimeout((function(){throw e}))};var O=y.Deferred();function H(){r.removeEventListener(\"DOMContentLoaded\",H),t.removeEventListener(\"load\",H),y.ready()}y.fn.ready=function(t){return O.then(t).catch((function(t){y.readyException(t)})),this},y.extend({isReady:!1,readyWait:1,ready:function(t){(!0===t?--y.readyWait:y.isReady)||(y.isReady=!0,!0!==t&&--y.readyWait>0||O.resolveWith(r,[y]))}}),y.ready.then=O.then,\"complete\"===r.readyState||\"loading\"!==r.readyState&&!r.documentElement.doScroll?t.setTimeout(y.ready):(r.addEventListener(\"DOMContentLoaded\",H),t.addEventListener(\"load\",H));var U=function(t,e,o,r,n,s,a){var i=0,c=t.length,l=null==o;if(\"object\"===v(o))for(i in n=!0,o)U(t,e,i,o[i],!0,s,a);else if(void 0!==r&&(n=!0,m(r)||(a=!0),l&&(a?(e.call(t,r),e=null):(l=e,e=function(t,e,o){return l.call(y(t),o)})),e))for(;i1,null,!0)},removeData:function(t){return this.each((function(){Q.remove(this,t)}))}}),y.extend({queue:function(t,e,o){var r;if(t)return e=(e||\"fx\")+\"queue\",r=G.get(t,e),o&&(!r||Array.isArray(o)?r=G.access(t,e,y.makeArray(o)):r.push(o)),r||[]},dequeue:function(t,e){e=e||\"fx\";var o=y.queue(t,e),r=o.length,n=o.shift(),s=y._queueHooks(t,e);\"inprogress\"===n&&(n=o.shift(),r--),n&&(\"fx\"===e&&o.unshift(\"inprogress\"),delete s.stop,n.call(t,(function(){y.dequeue(t,e)}),s)),!r&&s&&s.empty.fire()},_queueHooks:function(t,e){var o=e+\"queueHooks\";return G.get(t,o)||G.access(t,o,{empty:y.Callbacks(\"once memory\").add((function(){G.remove(t,[e+\"queue\",o])}))})}}),y.fn.extend({queue:function(t,e){var o=2;return\"string\"!=typeof t&&(e=t,t=\"fx\",o--),arguments.length\\x20\\t\\r\\n\\f]*)/i,ht=/^$|^module$|\\/(?:java|ecma)script/i,mt={option:[1,\"\"],thead:[1,\"\",\"
\"],col:[2,\"\",\"
\"],tr:[2,\"\",\"
\"],td:[3,\"\",\"
\"],_default:[0,\"\",\"\"]};function bt(t,e){var o;return o=void 0!==t.getElementsByTagName?t.getElementsByTagName(e||\"*\"):void 0!==t.querySelectorAll?t.querySelectorAll(e||\"*\"):[],void 0===e||e&&T(t,e)?y.merge([t],o):o}function gt(t,e){for(var o=0,r=t.length;o-1)n&&n.push(s);else if(l=nt(s),a=bt(p.appendChild(s),\"script\"),l&>(a),o)for(u=0;s=a[u++];)ht.test(s.type||\"\")&&o.push(s);return p}_t=r.createDocumentFragment().appendChild(r.createElement(\"div\")),(vt=r.createElement(\"input\")).setAttribute(\"type\",\"radio\"),vt.setAttribute(\"checked\",\"checked\"),vt.setAttribute(\"name\",\"t\"),_t.appendChild(vt),h.checkClone=_t.cloneNode(!0).cloneNode(!0).lastChild.checked,_t.innerHTML=\"\",h.noCloneChecked=!!_t.cloneNode(!0).lastChild.defaultValue;var wt=/^key/,Dt=/^(?:mouse|pointer|contextmenu|drag|drop)|click/,Ct=/^([^.]*)(?:\\.(.+)|)/;function kt(){return!0}function Et(){return!1}function Tt(t,e){return t===function(){try{return r.activeElement}catch(t){}}()==(\"focus\"===e)}function At(t,e,o,r,n,s){var a,i;if(\"object\"==typeof e){for(i in\"string\"!=typeof o&&(r=r||o,o=void 0),e)At(t,i,o,r,e[i],s);return t}if(null==r&&null==n?(n=o,r=o=void 0):null==n&&(\"string\"==typeof o?(n=r,r=void 0):(n=r,r=o,o=void 0)),!1===n)n=Et;else if(!n)return t;return 1===s&&(a=n,(n=function(t){return y().off(t),a.apply(this,arguments)}).guid=a.guid||(a.guid=y.guid++)),t.each((function(){y.event.add(this,e,n,r,o)}))}function St(t,e,o){o?(G.set(t,e,!1),y.event.add(t,e,{namespace:!1,handler:function(t){var r,n,a=G.get(this,e);if(1&t.isTrigger&&this[e]){if(a.length)(y.event.special[e]||{}).delegateType&&t.stopPropagation();else if(a=s.call(arguments),G.set(this,e,a),r=o(this,e),this[e](),a!==(n=G.get(this,e))||r?G.set(this,e,!1):n={},a!==n)return t.stopImmediatePropagation(),t.preventDefault(),n.value}else a.length&&(G.set(this,e,{value:y.event.trigger(y.extend(a[0],y.Event.prototype),a.slice(1),this)}),t.stopImmediatePropagation())}})):void 0===G.get(t,e)&&y.event.add(t,e,kt)}y.event={global:{},add:function(t,e,o,r,n){var s,a,i,c,l,u,p,d,f,h,m,b=G.get(t);if(b)for(o.handler&&(o=(s=o).handler,n=s.selector),n&&y.find.matchesSelector(rt,n),o.guid||(o.guid=y.guid++),(c=b.events)||(c=b.events={}),(a=b.handle)||(a=b.handle=function(e){return void 0!==y&&y.event.triggered!==e.type?y.event.dispatch.apply(t,arguments):void 0}),l=(e=(e||\"\").match(I)||[\"\"]).length;l--;)f=m=(i=Ct.exec(e[l])||[])[1],h=(i[2]||\"\").split(\".\").sort(),f&&(p=y.event.special[f]||{},f=(n?p.delegateType:p.bindType)||f,p=y.event.special[f]||{},u=y.extend({type:f,origType:m,data:r,handler:o,guid:o.guid,selector:n,needsContext:n&&y.expr.match.needsContext.test(n),namespace:h.join(\".\")},s),(d=c[f])||((d=c[f]=[]).delegateCount=0,p.setup&&!1!==p.setup.call(t,r,h,a)||t.addEventListener&&t.addEventListener(f,a)),p.add&&(p.add.call(t,u),u.handler.guid||(u.handler.guid=o.guid)),n?d.splice(d.delegateCount++,0,u):d.push(u),y.event.global[f]=!0)},remove:function(t,e,o,r,n){var s,a,i,c,l,u,p,d,f,h,m,b=G.hasData(t)&&G.get(t);if(b&&(c=b.events)){for(l=(e=(e||\"\").match(I)||[\"\"]).length;l--;)if(f=m=(i=Ct.exec(e[l])||[])[1],h=(i[2]||\"\").split(\".\").sort(),f){for(p=y.event.special[f]||{},d=c[f=(r?p.delegateType:p.bindType)||f]||[],i=i[2]&&new RegExp(\"(^|\\\\.)\"+h.join(\"\\\\.(?:.*\\\\.|)\")+\"(\\\\.|$)\"),a=s=d.length;s--;)u=d[s],!n&&m!==u.origType||o&&o.guid!==u.guid||i&&!i.test(u.namespace)||r&&r!==u.selector&&(\"**\"!==r||!u.selector)||(d.splice(s,1),u.selector&&d.delegateCount--,p.remove&&p.remove.call(t,u));a&&!d.length&&(p.teardown&&!1!==p.teardown.call(t,h,b.handle)||y.removeEvent(t,f,b.handle),delete c[f])}else for(f in c)y.event.remove(t,f+e[l],o,r,!0);y.isEmptyObject(c)&&G.remove(t,\"handle events\")}},dispatch:function(t){var e,o,r,n,s,a,i=y.event.fix(t),c=new Array(arguments.length),l=(G.get(this,\"events\")||{})[i.type]||[],u=y.event.special[i.type]||{};for(c[0]=i,e=1;e=1))for(;l!==this;l=l.parentNode||this)if(1===l.nodeType&&(\"click\"!==t.type||!0!==l.disabled)){for(s=[],a={},o=0;o-1:y.find(n,this,null,[l]).length),a[n]&&s.push(r);s.length&&i.push({elem:l,handlers:s})}return l=this,c\\x20\\t\\r\\n\\f]*)[^>]*)\\/>/gi,Mt=/\\s*$/g;function Lt(t,e){return T(t,\"table\")&&T(11!==e.nodeType?e:e.firstChild,\"tr\")&&y(t).children(\"tbody\")[0]||t}function It(t){return t.type=(null!==t.getAttribute(\"type\"))+\"/\"+t.type,t}function Bt(t){return\"true/\"===(t.type||\"\").slice(0,5)?t.type=t.type.slice(5):t.removeAttribute(\"type\"),t}function Pt(t,e){var o,r,n,s,a,i,c,l;if(1===e.nodeType){if(G.hasData(t)&&(s=G.access(t),a=G.set(e,s),l=s.events))for(n in delete a.handle,a.events={},l)for(o=0,r=l[n].length;o1&&\"string\"==typeof b&&!h.checkClone&&Nt.test(b))return t.each((function(n){var s=t.eq(n);g&&(e[0]=b.call(this,n,s.html())),zt(s,e,o,r)}));if(d&&(s=(n=xt(e,t[0].ownerDocument,!1,t,r)).firstChild,1===n.childNodes.length&&(n=s),s||r)){for(c=(i=y.map(bt(n,\"script\"),It)).length;p\")},clone:function(t,e,o){var r,n,s,a,i=t.cloneNode(!0),c=nt(t);if(!(h.noCloneChecked||1!==t.nodeType&&11!==t.nodeType||y.isXMLDoc(t)))for(a=bt(i),r=0,n=(s=bt(t)).length;r0&>(a,!c&&bt(t,\"script\")),i},cleanData:function(t){for(var e,o,r,n=y.event.special,s=0;void 0!==(o=t[s]);s++)if(V(o)){if(e=o[G.expando]){if(e.events)for(r in e.events)n[r]?y.event.remove(o,r):y.removeEvent(o,r,e.handle);o[G.expando]=void 0}o[Q.expando]&&(o[Q.expando]=void 0)}}}),y.fn.extend({detach:function(t){return Ot(this,t,!0)},remove:function(t){return Ot(this,t)},text:function(t){return U(this,(function(t){return void 0===t?y.text(this):this.empty().each((function(){1!==this.nodeType&&11!==this.nodeType&&9!==this.nodeType||(this.textContent=t)}))}),null,t,arguments.length)},append:function(){return zt(this,arguments,(function(t){1!==this.nodeType&&11!==this.nodeType&&9!==this.nodeType||Lt(this,t).appendChild(t)}))},prepend:function(){return zt(this,arguments,(function(t){if(1===this.nodeType||11===this.nodeType||9===this.nodeType){var e=Lt(this,t);e.insertBefore(t,e.firstChild)}}))},before:function(){return zt(this,arguments,(function(t){this.parentNode&&this.parentNode.insertBefore(t,this)}))},after:function(){return zt(this,arguments,(function(t){this.parentNode&&this.parentNode.insertBefore(t,this.nextSibling)}))},empty:function(){for(var t,e=0;null!=(t=this[e]);e++)1===t.nodeType&&(y.cleanData(bt(t,!1)),t.textContent=\"\");return this},clone:function(t,e){return t=null!=t&&t,e=null==e?t:e,this.map((function(){return y.clone(this,t,e)}))},html:function(t){return U(this,(function(t){var e=this[0]||{},o=0,r=this.length;if(void 0===t&&1===e.nodeType)return e.innerHTML;if(\"string\"==typeof t&&!Mt.test(t)&&!mt[(ft.exec(t)||[\"\",\"\"])[1].toLowerCase()]){t=y.htmlPrefilter(t);try{for(;o=0&&(c+=Math.max(0,Math.ceil(t[\"offset\"+e[0].toUpperCase()+e.slice(1)]-s-c-i-.5))||0),c}function oe(t,e,o){var r=Ut(t),n=(!h.boxSizingReliable()||o)&&\"border-box\"===y.css(t,\"boxSizing\",!1,r),s=n,a=qt(t,e,r),i=\"offset\"+e[0].toUpperCase()+e.slice(1);if(Ht.test(a)){if(!o)return a;a=\"auto\"}return(!h.boxSizingReliable()&&n||\"auto\"===a||!parseFloat(a)&&\"inline\"===y.css(t,\"display\",!1,r))&&t.getClientRects().length&&(n=\"border-box\"===y.css(t,\"boxSizing\",!1,r),(s=i in t)&&(a=t[i])),(a=parseFloat(a)||0)+ee(t,e,o||(n?\"border\":\"content\"),s,r,a)+\"px\"}function re(t,e,o,r,n){return new re.prototype.init(t,e,o,r,n)}y.extend({cssHooks:{opacity:{get:function(t,e){if(e){var o=qt(t,\"opacity\");return\"\"===o?\"1\":o}}}},cssNumber:{animationIterationCount:!0,columnCount:!0,fillOpacity:!0,flexGrow:!0,flexShrink:!0,fontWeight:!0,gridArea:!0,gridColumn:!0,gridColumnEnd:!0,gridColumnStart:!0,gridRow:!0,gridRowEnd:!0,gridRowStart:!0,lineHeight:!0,opacity:!0,order:!0,orphans:!0,widows:!0,zIndex:!0,zoom:!0},cssProps:{},style:function(t,e,o,r){if(t&&3!==t.nodeType&&8!==t.nodeType&&t.style){var n,s,a,i=Y(e),c=Kt.test(e),l=t.style;if(c||(e=Gt(i)),a=y.cssHooks[e]||y.cssHooks[i],void 0===o)return a&&\"get\"in a&&void 0!==(n=a.get(t,!1,r))?n:l[e];\"string\"===(s=typeof o)&&(n=et.exec(o))&&n[1]&&(o=ct(t,e,n),s=\"number\"),null!=o&&o==o&&(\"number\"!==s||c||(o+=n&&n[3]||(y.cssNumber[i]?\"\":\"px\")),h.clearCloneStyle||\"\"!==o||0!==e.indexOf(\"background\")||(l[e]=\"inherit\"),a&&\"set\"in a&&void 0===(o=a.set(t,o,r))||(c?l.setProperty(e,o):l[e]=o))}},css:function(t,e,o,r){var n,s,a,i=Y(e);return Kt.test(e)||(e=Gt(i)),(a=y.cssHooks[e]||y.cssHooks[i])&&\"get\"in a&&(n=a.get(t,!0,o)),void 0===n&&(n=qt(t,e,r)),\"normal\"===n&&e in Zt&&(n=Zt[e]),\"\"===o||o?(s=parseFloat(n),!0===o||isFinite(s)?s||0:n):n}}),y.each([\"height\",\"width\"],(function(t,e){y.cssHooks[e]={get:function(t,o,r){if(o)return!Qt.test(y.css(t,\"display\"))||t.getClientRects().length&&t.getBoundingClientRect().width?oe(t,e,r):it(t,Jt,(function(){return oe(t,e,r)}))},set:function(t,o,r){var n,s=Ut(t),a=!h.scrollboxSize()&&\"absolute\"===s.position,i=(a||r)&&\"border-box\"===y.css(t,\"boxSizing\",!1,s),c=r?ee(t,e,r,i,s):0;return i&&a&&(c-=Math.ceil(t[\"offset\"+e[0].toUpperCase()+e.slice(1)]-parseFloat(s[e])-ee(t,e,\"border\",!1,s)-.5)),c&&(n=et.exec(o))&&\"px\"!==(n[3]||\"px\")&&(t.style[e]=o,o=y.css(t,e)),te(0,o,c)}}})),y.cssHooks.marginLeft=$t(h.reliableMarginLeft,(function(t,e){if(e)return(parseFloat(qt(t,\"marginLeft\"))||t.getBoundingClientRect().left-it(t,{marginLeft:0},(function(){return t.getBoundingClientRect().left})))+\"px\"})),y.each({margin:\"\",padding:\"\",border:\"Width\"},(function(t,e){y.cssHooks[t+e]={expand:function(o){for(var r=0,n={},s=\"string\"==typeof o?o.split(\" \"):[o];r<4;r++)n[t+ot[r]+e]=s[r]||s[r-2]||s[0];return n}},\"margin\"!==t&&(y.cssHooks[t+e].set=te)})),y.fn.extend({css:function(t,e){return U(this,(function(t,e,o){var r,n,s={},a=0;if(Array.isArray(e)){for(r=Ut(t),n=e.length;a1)}}),y.Tween=re,re.prototype={constructor:re,init:function(t,e,o,r,n,s){this.elem=t,this.prop=o,this.easing=n||y.easing._default,this.options=e,this.start=this.now=this.cur(),this.end=r,this.unit=s||(y.cssNumber[o]?\"\":\"px\")},cur:function(){var t=re.propHooks[this.prop];return t&&t.get?t.get(this):re.propHooks._default.get(this)},run:function(t){var e,o=re.propHooks[this.prop];return this.options.duration?this.pos=e=y.easing[this.easing](t,this.options.duration*t,0,1,this.options.duration):this.pos=e=t,this.now=(this.end-this.start)*e+this.start,this.options.step&&this.options.step.call(this.elem,this.now,this),o&&o.set?o.set(this):re.propHooks._default.set(this),this}},re.prototype.init.prototype=re.prototype,re.propHooks={_default:{get:function(t){var e;return 1!==t.elem.nodeType||null!=t.elem[t.prop]&&null==t.elem.style[t.prop]?t.elem[t.prop]:(e=y.css(t.elem,t.prop,\"\"))&&\"auto\"!==e?e:0},set:function(t){y.fx.step[t.prop]?y.fx.step[t.prop](t):1!==t.elem.nodeType||!y.cssHooks[t.prop]&&null==t.elem.style[Gt(t.prop)]?t.elem[t.prop]=t.now:y.style(t.elem,t.prop,t.now+t.unit)}}},re.propHooks.scrollTop=re.propHooks.scrollLeft={set:function(t){t.elem.nodeType&&t.elem.parentNode&&(t.elem[t.prop]=t.now)}},y.easing={linear:function(t){return t},swing:function(t){return.5-Math.cos(t*Math.PI)/2},_default:\"swing\"},y.fx=re.prototype.init,y.fx.step={};var ne,se,ae=/^(?:toggle|show|hide)$/,ie=/queueHooks$/;function ce(){se&&(!1===r.hidden&&t.requestAnimationFrame?t.requestAnimationFrame(ce):t.setTimeout(ce,y.fx.interval),y.fx.tick())}function le(){return t.setTimeout((function(){ne=void 0})),ne=Date.now()}function ue(t,e){var o,r=0,n={height:t};for(e=e?1:0;r<4;r+=2-e)n[\"margin\"+(o=ot[r])]=n[\"padding\"+o]=t;return e&&(n.opacity=n.width=t),n}function pe(t,e,o){for(var r,n=(de.tweeners[e]||[]).concat(de.tweeners[\"*\"]),s=0,a=n.length;s1)},removeAttr:function(t){return this.each((function(){y.removeAttr(this,t)}))}}),y.extend({attr:function(t,e,o){var r,n,s=t.nodeType;if(3!==s&&8!==s&&2!==s)return void 0===t.getAttribute?y.prop(t,e,o):(1===s&&y.isXMLDoc(t)||(n=y.attrHooks[e.toLowerCase()]||(y.expr.match.bool.test(e)?fe:void 0)),void 0!==o?null===o?void y.removeAttr(t,e):n&&\"set\"in n&&void 0!==(r=n.set(t,o,e))?r:(t.setAttribute(e,o+\"\"),o):n&&\"get\"in n&&null!==(r=n.get(t,e))?r:null==(r=y.find.attr(t,e))?void 0:r)},attrHooks:{type:{set:function(t,e){if(!h.radioValue&&\"radio\"===e&&T(t,\"input\")){var o=t.value;return t.setAttribute(\"type\",e),o&&(t.value=o),e}}}},removeAttr:function(t,e){var o,r=0,n=e&&e.match(I);if(n&&1===t.nodeType)for(;o=n[r++];)t.removeAttribute(o)}}),fe={set:function(t,e,o){return!1===e?y.removeAttr(t,o):t.setAttribute(o,o),o}},y.each(y.expr.match.bool.source.match(/\\w+/g),(function(t,e){var o=he[e]||y.find.attr;he[e]=function(t,e,r){var n,s,a=e.toLowerCase();return r||(s=he[a],he[a]=n,n=null!=o(t,e,r)?a:null,he[a]=s),n}}));var me=/^(?:input|select|textarea|button)$/i,be=/^(?:a|area)$/i;function ge(t){return(t.match(I)||[]).join(\" \")}function _e(t){return t.getAttribute&&t.getAttribute(\"class\")||\"\"}function ve(t){return Array.isArray(t)?t:\"string\"==typeof t&&t.match(I)||[]}y.fn.extend({prop:function(t,e){return U(this,y.prop,t,e,arguments.length>1)},removeProp:function(t){return this.each((function(){delete this[y.propFix[t]||t]}))}}),y.extend({prop:function(t,e,o){var r,n,s=t.nodeType;if(3!==s&&8!==s&&2!==s)return 1===s&&y.isXMLDoc(t)||(e=y.propFix[e]||e,n=y.propHooks[e]),void 0!==o?n&&\"set\"in n&&void 0!==(r=n.set(t,o,e))?r:t[e]=o:n&&\"get\"in n&&null!==(r=n.get(t,e))?r:t[e]},propHooks:{tabIndex:{get:function(t){var e=y.find.attr(t,\"tabindex\");return e?parseInt(e,10):me.test(t.nodeName)||be.test(t.nodeName)&&t.href?0:-1}}},propFix:{for:\"htmlFor\",class:\"className\"}}),h.optSelected||(y.propHooks.selected={get:function(t){var e=t.parentNode;return e&&e.parentNode&&e.parentNode.selectedIndex,null},set:function(t){var e=t.parentNode;e&&(e.selectedIndex,e.parentNode&&e.parentNode.selectedIndex)}}),y.each([\"tabIndex\",\"readOnly\",\"maxLength\",\"cellSpacing\",\"cellPadding\",\"rowSpan\",\"colSpan\",\"useMap\",\"frameBorder\",\"contentEditable\"],(function(){y.propFix[this.toLowerCase()]=this})),y.fn.extend({addClass:function(t){var e,o,r,n,s,a,i,c=0;if(m(t))return this.each((function(e){y(this).addClass(t.call(this,e,_e(this)))}));if((e=ve(t)).length)for(;o=this[c++];)if(n=_e(o),r=1===o.nodeType&&\" \"+ge(n)+\" \"){for(a=0;s=e[a++];)r.indexOf(\" \"+s+\" \")<0&&(r+=s+\" \");n!==(i=ge(r))&&o.setAttribute(\"class\",i)}return this},removeClass:function(t){var e,o,r,n,s,a,i,c=0;if(m(t))return this.each((function(e){y(this).removeClass(t.call(this,e,_e(this)))}));if(!arguments.length)return this.attr(\"class\",\"\");if((e=ve(t)).length)for(;o=this[c++];)if(n=_e(o),r=1===o.nodeType&&\" \"+ge(n)+\" \"){for(a=0;s=e[a++];)for(;r.indexOf(\" \"+s+\" \")>-1;)r=r.replace(\" \"+s+\" \",\" \");n!==(i=ge(r))&&o.setAttribute(\"class\",i)}return this},toggleClass:function(t,e){var o=typeof t,r=\"string\"===o||Array.isArray(t);return\"boolean\"==typeof e&&r?e?this.addClass(t):this.removeClass(t):m(t)?this.each((function(o){y(this).toggleClass(t.call(this,o,_e(this),e),e)})):this.each((function(){var e,n,s,a;if(r)for(n=0,s=y(this),a=ve(t);e=a[n++];)s.hasClass(e)?s.removeClass(e):s.addClass(e);else void 0!==t&&\"boolean\"!==o||((e=_e(this))&&G.set(this,\"__className__\",e),this.setAttribute&&this.setAttribute(\"class\",e||!1===t?\"\":G.get(this,\"__className__\")||\"\"))}))},hasClass:function(t){var e,o,r=0;for(e=\" \"+t+\" \";o=this[r++];)if(1===o.nodeType&&(\" \"+ge(_e(o))+\" \").indexOf(e)>-1)return!0;return!1}});var ye=/\\r/g;y.fn.extend({val:function(t){var e,o,r,n=this[0];return arguments.length?(r=m(t),this.each((function(o){var n;1===this.nodeType&&(null==(n=r?t.call(this,o,y(this).val()):t)?n=\"\":\"number\"==typeof n?n+=\"\":Array.isArray(n)&&(n=y.map(n,(function(t){return null==t?\"\":t+\"\"}))),(e=y.valHooks[this.type]||y.valHooks[this.nodeName.toLowerCase()])&&\"set\"in e&&void 0!==e.set(this,n,\"value\")||(this.value=n))}))):n?(e=y.valHooks[n.type]||y.valHooks[n.nodeName.toLowerCase()])&&\"get\"in e&&void 0!==(o=e.get(n,\"value\"))?o:\"string\"==typeof(o=n.value)?o.replace(ye,\"\"):null==o?\"\":o:void 0}}),y.extend({valHooks:{option:{get:function(t){var e=y.find.attr(t,\"value\");return null!=e?e:ge(y.text(t))}},select:{get:function(t){var e,o,r,n=t.options,s=t.selectedIndex,a=\"select-one\"===t.type,i=a?null:[],c=a?s+1:n.length;for(r=s<0?c:a?s:0;r-1)&&(o=!0);return o||(t.selectedIndex=-1),s}}}}),y.each([\"radio\",\"checkbox\"],(function(){y.valHooks[this]={set:function(t,e){if(Array.isArray(e))return t.checked=y.inArray(y(t).val(),e)>-1}},h.checkOn||(y.valHooks[this].get=function(t){return null===t.getAttribute(\"value\")?\"on\":t.value})})),h.focusin=\"onfocusin\"in t;var xe=/^(?:focusinfocus|focusoutblur)$/,we=function(t){t.stopPropagation()};y.extend(y.event,{trigger:function(e,o,n,s){var a,i,c,l,u,d,f,h,g=[n||r],_=p.call(e,\"type\")?e.type:e,v=p.call(e,\"namespace\")?e.namespace.split(\".\"):[];if(i=h=c=n=n||r,3!==n.nodeType&&8!==n.nodeType&&!xe.test(_+y.event.triggered)&&(_.indexOf(\".\")>-1&&(v=_.split(\".\"),_=v.shift(),v.sort()),u=_.indexOf(\":\")<0&&\"on\"+_,(e=e[y.expando]?e:new y.Event(_,\"object\"==typeof e&&e)).isTrigger=s?2:3,e.namespace=v.join(\".\"),e.rnamespace=e.namespace?new RegExp(\"(^|\\\\.)\"+v.join(\"\\\\.(?:.*\\\\.|)\")+\"(\\\\.|$)\"):null,e.result=void 0,e.target||(e.target=n),o=null==o?[e]:y.makeArray(o,[e]),f=y.event.special[_]||{},s||!f.trigger||!1!==f.trigger.apply(n,o))){if(!s&&!f.noBubble&&!b(n)){for(l=f.delegateType||_,xe.test(l+_)||(i=i.parentNode);i;i=i.parentNode)g.push(i),c=i;c===(n.ownerDocument||r)&&g.push(c.defaultView||c.parentWindow||t)}for(a=0;(i=g[a++])&&!e.isPropagationStopped();)h=i,e.type=a>1?l:f.bindType||_,(d=(G.get(i,\"events\")||{})[e.type]&&G.get(i,\"handle\"))&&d.apply(i,o),(d=u&&i[u])&&d.apply&&V(i)&&(e.result=d.apply(i,o),!1===e.result&&e.preventDefault());return e.type=_,s||e.isDefaultPrevented()||f._default&&!1!==f._default.apply(g.pop(),o)||!V(n)||u&&m(n[_])&&!b(n)&&((c=n[u])&&(n[u]=null),y.event.triggered=_,e.isPropagationStopped()&&h.addEventListener(_,we),n[_](),e.isPropagationStopped()&&h.removeEventListener(_,we),y.event.triggered=void 0,c&&(n[u]=c)),e.result}},simulate:function(t,e,o){var r=y.extend(new y.Event,o,{type:t,isSimulated:!0});y.event.trigger(r,null,e)}}),y.fn.extend({trigger:function(t,e){return this.each((function(){y.event.trigger(t,e,this)}))},triggerHandler:function(t,e){var o=this[0];if(o)return y.event.trigger(t,e,o,!0)}}),h.focusin||y.each({focus:\"focusin\",blur:\"focusout\"},(function(t,e){var o=function(t){y.event.simulate(e,t.target,y.event.fix(t))};y.event.special[e]={setup:function(){var r=this.ownerDocument||this,n=G.access(r,e);n||r.addEventListener(t,o,!0),G.access(r,e,(n||0)+1)},teardown:function(){var r=this.ownerDocument||this,n=G.access(r,e)-1;n?G.access(r,e,n):(r.removeEventListener(t,o,!0),G.remove(r,e))}}}));var De=t.location,Ce=Date.now(),ke=/\\?/;y.parseXML=function(e){var o;if(!e||\"string\"!=typeof e)return null;try{o=(new t.DOMParser).parseFromString(e,\"text/xml\")}catch(t){o=void 0}return o&&!o.getElementsByTagName(\"parsererror\").length||y.error(\"Invalid XML: \"+e),o};var Ee=/\\[\\]$/,Te=/\\r?\\n/g,Ae=/^(?:submit|button|image|reset|file)$/i,Se=/^(?:input|select|textarea|keygen)/i;function Fe(t,e,o,r){var n;if(Array.isArray(e))y.each(e,(function(e,n){o||Ee.test(t)?r(t,n):Fe(t+\"[\"+(\"object\"==typeof n&&null!=n?e:\"\")+\"]\",n,o,r)}));else if(o||\"object\"!==v(e))r(t,e);else for(n in e)Fe(t+\"[\"+n+\"]\",e[n],o,r)}y.param=function(t,e){var o,r=[],n=function(t,e){var o=m(e)?e():e;r[r.length]=encodeURIComponent(t)+\"=\"+encodeURIComponent(null==o?\"\":o)};if(null==t)return\"\";if(Array.isArray(t)||t.jquery&&!y.isPlainObject(t))y.each(t,(function(){n(this.name,this.value)}));else for(o in t)Fe(o,t[o],e,n);return r.join(\"&\")},y.fn.extend({serialize:function(){return y.param(this.serializeArray())},serializeArray:function(){return this.map((function(){var t=y.prop(this,\"elements\");return t?y.makeArray(t):this})).filter((function(){var t=this.type;return this.name&&!y(this).is(\":disabled\")&&Se.test(this.nodeName)&&!Ae.test(t)&&(this.checked||!dt.test(t))})).map((function(t,e){var o=y(this).val();return null==o?null:Array.isArray(o)?y.map(o,(function(t){return{name:e.name,value:t.replace(Te,\"\\r\\n\")}})):{name:e.name,value:o.replace(Te,\"\\r\\n\")}})).get()}});var Me=/%20/g,Ne=/#.*$/,je=/([?&])_=[^&]*/,Le=/^(.*?):[ \\t]*([^\\r\\n]*)$/gm,Ie=/^(?:GET|HEAD)$/,Be=/^\\/\\//,Pe={},Re={},ze=\"*/\".concat(\"*\"),Oe=r.createElement(\"a\");function He(t){return function(e,o){\"string\"!=typeof e&&(o=e,e=\"*\");var r,n=0,s=e.toLowerCase().match(I)||[];if(m(o))for(;r=s[n++];)\"+\"===r[0]?(r=r.slice(1)||\"*\",(t[r]=t[r]||[]).unshift(o)):(t[r]=t[r]||[]).push(o)}}function Ue(t,e,o,r){var n={},s=t===Re;function a(i){var c;return n[i]=!0,y.each(t[i]||[],(function(t,i){var l=i(e,o,r);return\"string\"!=typeof l||s||n[l]?s?!(c=l):void 0:(e.dataTypes.unshift(l),a(l),!1)})),c}return a(e.dataTypes[0])||!n[\"*\"]&&a(\"*\")}function We(t,e){var o,r,n=y.ajaxSettings.flatOptions||{};for(o in e)void 0!==e[o]&&((n[o]?t:r||(r={}))[o]=e[o]);return r&&y.extend(!0,t,r),t}Oe.href=De.href,y.extend({active:0,lastModified:{},etag:{},ajaxSettings:{url:De.href,type:\"GET\",isLocal:/^(?:about|app|app-storage|.+-extension|file|res|widget):$/.test(De.protocol),global:!0,processData:!0,async:!0,contentType:\"application/x-www-form-urlencoded; charset=UTF-8\",accepts:{\"*\":ze,text:\"text/plain\",html:\"text/html\",xml:\"application/xml, text/xml\",json:\"application/json, text/javascript\"},contents:{xml:/\\bxml\\b/,html:/\\bhtml/,json:/\\bjson\\b/},responseFields:{xml:\"responseXML\",text:\"responseText\",json:\"responseJSON\"},converters:{\"* text\":String,\"text html\":!0,\"text json\":JSON.parse,\"text xml\":y.parseXML},flatOptions:{url:!0,context:!0}},ajaxSetup:function(t,e){return e?We(We(t,y.ajaxSettings),e):We(y.ajaxSettings,t)},ajaxPrefilter:He(Pe),ajaxTransport:He(Re),ajax:function(e,o){\"object\"==typeof e&&(o=e,e=void 0),o=o||{};var n,s,a,i,c,l,u,p,d,f,h=y.ajaxSetup({},o),m=h.context||h,b=h.context&&(m.nodeType||m.jquery)?y(m):y.event,g=y.Deferred(),_=y.Callbacks(\"once memory\"),v=h.statusCode||{},x={},w={},D=\"canceled\",C={readyState:0,getResponseHeader:function(t){var e;if(u){if(!i)for(i={};e=Le.exec(a);)i[e[1].toLowerCase()+\" \"]=(i[e[1].toLowerCase()+\" \"]||[]).concat(e[2]);e=i[t.toLowerCase()+\" \"]}return null==e?null:e.join(\", \")},getAllResponseHeaders:function(){return u?a:null},setRequestHeader:function(t,e){return null==u&&(t=w[t.toLowerCase()]=w[t.toLowerCase()]||t,x[t]=e),this},overrideMimeType:function(t){return null==u&&(h.mimeType=t),this},statusCode:function(t){var e;if(t)if(u)C.always(t[C.status]);else for(e in t)v[e]=[v[e],t[e]];return this},abort:function(t){var e=t||D;return n&&n.abort(e),k(0,e),this}};if(g.promise(C),h.url=((e||h.url||De.href)+\"\").replace(Be,De.protocol+\"//\"),h.type=o.method||o.type||h.method||h.type,h.dataTypes=(h.dataType||\"*\").toLowerCase().match(I)||[\"\"],null==h.crossDomain){l=r.createElement(\"a\");try{l.href=h.url,l.href=l.href,h.crossDomain=Oe.protocol+\"//\"+Oe.host!=l.protocol+\"//\"+l.host}catch(t){h.crossDomain=!0}}if(h.data&&h.processData&&\"string\"!=typeof h.data&&(h.data=y.param(h.data,h.traditional)),Ue(Pe,h,o,C),u)return C;for(d in(p=y.event&&h.global)&&0==y.active++&&y.event.trigger(\"ajaxStart\"),h.type=h.type.toUpperCase(),h.hasContent=!Ie.test(h.type),s=h.url.replace(Ne,\"\"),h.hasContent?h.data&&h.processData&&0===(h.contentType||\"\").indexOf(\"application/x-www-form-urlencoded\")&&(h.data=h.data.replace(Me,\"+\")):(f=h.url.slice(s.length),h.data&&(h.processData||\"string\"==typeof h.data)&&(s+=(ke.test(s)?\"&\":\"?\")+h.data,delete h.data),!1===h.cache&&(s=s.replace(je,\"$1\"),f=(ke.test(s)?\"&\":\"?\")+\"_=\"+Ce+++f),h.url=s+f),h.ifModified&&(y.lastModified[s]&&C.setRequestHeader(\"If-Modified-Since\",y.lastModified[s]),y.etag[s]&&C.setRequestHeader(\"If-None-Match\",y.etag[s])),(h.data&&h.hasContent&&!1!==h.contentType||o.contentType)&&C.setRequestHeader(\"Content-Type\",h.contentType),C.setRequestHeader(\"Accept\",h.dataTypes[0]&&h.accepts[h.dataTypes[0]]?h.accepts[h.dataTypes[0]]+(\"*\"!==h.dataTypes[0]?\", \"+ze+\"; q=0.01\":\"\"):h.accepts[\"*\"]),h.headers)C.setRequestHeader(d,h.headers[d]);if(h.beforeSend&&(!1===h.beforeSend.call(m,C,h)||u))return C.abort();if(D=\"abort\",_.add(h.complete),C.done(h.success),C.fail(h.error),n=Ue(Re,h,o,C)){if(C.readyState=1,p&&b.trigger(\"ajaxSend\",[C,h]),u)return C;h.async&&h.timeout>0&&(c=t.setTimeout((function(){C.abort(\"timeout\")}),h.timeout));try{u=!1,n.send(x,k)}catch(t){if(u)throw t;k(-1,t)}}else k(-1,\"No Transport\");function k(e,o,r,i){var l,d,f,x,w,D=o;u||(u=!0,c&&t.clearTimeout(c),n=void 0,a=i||\"\",C.readyState=e>0?4:0,l=e>=200&&e<300||304===e,r&&(x=function(t,e,o){for(var r,n,s,a,i=t.contents,c=t.dataTypes;\"*\"===c[0];)c.shift(),void 0===r&&(r=t.mimeType||e.getResponseHeader(\"Content-Type\"));if(r)for(n in i)if(i[n]&&i[n].test(r)){c.unshift(n);break}if(c[0]in o)s=c[0];else{for(n in o){if(!c[0]||t.converters[n+\" \"+c[0]]){s=n;break}a||(a=n)}s=s||a}if(s)return s!==c[0]&&c.unshift(s),o[s]}(h,C,r)),x=function(t,e,o,r){var n,s,a,i,c,l={},u=t.dataTypes.slice();if(u[1])for(a in t.converters)l[a.toLowerCase()]=t.converters[a];for(s=u.shift();s;)if(t.responseFields[s]&&(o[t.responseFields[s]]=e),!c&&r&&t.dataFilter&&(e=t.dataFilter(e,t.dataType)),c=s,s=u.shift())if(\"*\"===s)s=c;else if(\"*\"!==c&&c!==s){if(!(a=l[c+\" \"+s]||l[\"* \"+s]))for(n in l)if((i=n.split(\" \"))[1]===s&&(a=l[c+\" \"+i[0]]||l[\"* \"+i[0]])){!0===a?a=l[n]:!0!==l[n]&&(s=i[0],u.unshift(i[1]));break}if(!0!==a)if(a&&t.throws)e=a(e);else try{e=a(e)}catch(t){return{state:\"parsererror\",error:a?t:\"No conversion from \"+c+\" to \"+s}}}return{state:\"success\",data:e}}(h,x,C,l),l?(h.ifModified&&((w=C.getResponseHeader(\"Last-Modified\"))&&(y.lastModified[s]=w),(w=C.getResponseHeader(\"etag\"))&&(y.etag[s]=w)),204===e||\"HEAD\"===h.type?D=\"nocontent\":304===e?D=\"notmodified\":(D=x.state,d=x.data,l=!(f=x.error))):(f=D,!e&&D||(D=\"error\",e<0&&(e=0))),C.status=e,C.statusText=(o||D)+\"\",l?g.resolveWith(m,[d,D,C]):g.rejectWith(m,[C,D,f]),C.statusCode(v),v=void 0,p&&b.trigger(l?\"ajaxSuccess\":\"ajaxError\",[C,h,l?d:f]),_.fireWith(m,[C,D]),p&&(b.trigger(\"ajaxComplete\",[C,h]),--y.active||y.event.trigger(\"ajaxStop\")))}return C},getJSON:function(t,e,o){return y.get(t,e,o,\"json\")},getScript:function(t,e){return y.get(t,void 0,e,\"script\")}}),y.each([\"get\",\"post\"],(function(t,e){y[e]=function(t,o,r,n){return m(o)&&(n=n||r,r=o,o=void 0),y.ajax(y.extend({url:t,type:e,dataType:n,data:o,success:r},y.isPlainObject(t)&&t))}})),y._evalUrl=function(t,e){return y.ajax({url:t,type:\"GET\",dataType:\"script\",cache:!0,async:!1,global:!1,converters:{\"text script\":function(){}},dataFilter:function(t){y.globalEval(t,e)}})},y.fn.extend({wrapAll:function(t){var e;return this[0]&&(m(t)&&(t=t.call(this[0])),e=y(t,this[0].ownerDocument).eq(0).clone(!0),this[0].parentNode&&e.insertBefore(this[0]),e.map((function(){for(var t=this;t.firstElementChild;)t=t.firstElementChild;return t})).append(this)),this},wrapInner:function(t){return m(t)?this.each((function(e){y(this).wrapInner(t.call(this,e))})):this.each((function(){var e=y(this),o=e.contents();o.length?o.wrapAll(t):e.append(t)}))},wrap:function(t){var e=m(t);return this.each((function(o){y(this).wrapAll(e?t.call(this,o):t)}))},unwrap:function(t){return this.parent(t).not(\"body\").each((function(){y(this).replaceWith(this.childNodes)})),this}}),y.expr.pseudos.hidden=function(t){return!y.expr.pseudos.visible(t)},y.expr.pseudos.visible=function(t){return!!(t.offsetWidth||t.offsetHeight||t.getClientRects().length)},y.ajaxSettings.xhr=function(){try{return new t.XMLHttpRequest}catch(t){}};var qe={0:200,1223:204},$e=y.ajaxSettings.xhr();h.cors=!!$e&&\"withCredentials\"in $e,h.ajax=$e=!!$e,y.ajaxTransport((function(e){var o,r;if(h.cors||$e&&!e.crossDomain)return{send:function(n,s){var a,i=e.xhr();if(i.open(e.type,e.url,e.async,e.username,e.password),e.xhrFields)for(a in e.xhrFields)i[a]=e.xhrFields[a];for(a in e.mimeType&&i.overrideMimeType&&i.overrideMimeType(e.mimeType),e.crossDomain||n[\"X-Requested-With\"]||(n[\"X-Requested-With\"]=\"XMLHttpRequest\"),n)i.setRequestHeader(a,n[a]);o=function(t){return function(){o&&(o=r=i.onload=i.onerror=i.onabort=i.ontimeout=i.onreadystatechange=null,\"abort\"===t?i.abort():\"error\"===t?\"number\"!=typeof i.status?s(0,\"error\"):s(i.status,i.statusText):s(qe[i.status]||i.status,i.statusText,\"text\"!==(i.responseType||\"text\")||\"string\"!=typeof i.responseText?{binary:i.response}:{text:i.responseText},i.getAllResponseHeaders()))}},i.onload=o(),r=i.onerror=i.ontimeout=o(\"error\"),void 0!==i.onabort?i.onabort=r:i.onreadystatechange=function(){4===i.readyState&&t.setTimeout((function(){o&&r()}))},o=o(\"abort\");try{i.send(e.hasContent&&e.data||null)}catch(t){if(o)throw t}},abort:function(){o&&o()}}})),y.ajaxPrefilter((function(t){t.crossDomain&&(t.contents.script=!1)})),y.ajaxSetup({accepts:{script:\"text/javascript, application/javascript, application/ecmascript, application/x-ecmascript\"},contents:{script:/\\b(?:java|ecma)script\\b/},converters:{\"text script\":function(t){return y.globalEval(t),t}}}),y.ajaxPrefilter(\"script\",(function(t){void 0===t.cache&&(t.cache=!1),t.crossDomain&&(t.type=\"GET\")})),y.ajaxTransport(\"script\",(function(t){var e,o;if(t.crossDomain||t.scriptAttrs)return{send:function(n,s){e=y(\"\n", " \n", " \n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import hiplot as hip\n", "hip.Experiment.from_iterable(data).display(force_full_width = False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are numerous insights that we can gather from the above graph:\n", "\n", "* As number of epochs increase, train loss correspondingly decreases while training accuracy increases\n", "* However, a higher training accuracy corresponded with a high validation loss, which means that on such instances, our model's prediction on the validation set was far from the truth. \n", "* Further, we reached our best validation accuracy of 18% on epoch 3! This gives us insight that our model was very quick to overfit the data and hence, a lower learning rate plus an added regularization term might produce better results.\n", "\n", "With this knowledge, we can make more informed decision on how we can \"tweak\" our architecture to possibly produce better results. We will perform these tweaks on our [lr and regularization]() tutorial.\n", "\n", "# Comparison\n", "\n", "Now, we will compare the performance of our ReLU activation function with two other activation functions:\n", "\n", "* Tanh and\n", "* Leaky ReLU\n", "\n", "As was previously alluded, distinct activation functions inherit distinct properties.\n", "\n", "As for the case of ReLU, we described its \"gate\" like properties. However, ReLU suffers from the \"dying ReLU\" problem: the input and gradient of negative inputs turn to zero, thereby making it impossible for weights that correspond to the negative input to update.\n", "\n", "In a similar fashion, Tanh suffers from the \"vanishing gradient\" problem. Let us look at the graph of tanh:\n", "\n", "$$\n", "f(x)=\\frac{ e^x-e^{-x}}{e^x+e^{-x}} \\\\f'(x) = 1-f(x)^2\n", "$$\n", "\n", "\n", "![Hyper-parameters in Action! Part I — Activation Functions](https://miro.medium.com/max/848/1*SeCBB7lfA7KPJ-T1Mi7GRg.png)\n", "\n", "\n", "\n", "As the input infinitely diverges from zero, its gradient infinitely approximates zero. In such case, while these neurons do not become \"dead,\" their chances of \"recovering\" towards a non-zero gradient to make meaningful weight changes are diminished as the input diverges from zero.\n", "\n", "Leaky ReLU attempts to solve the \"dying ReLU\" problem by giving a chance for the gradients of negative inputs to \"recover.\" Look at its below graph\n", "\n", "$$\n", "f(x) =\\begin{cases}.01x & x < 0 \\\\x & x \\ge 0\\end{cases} \\\\f'(x) =\\begin{cases}.01 & x < 0 \\\\1 & x \\ge 0\\end{cases}\n", "$$\n", "\n", "\n", "![Activation Functions | Code Odysseys](https://codeodysseys.com/posts/activation-functions/graph-activation-6.png)\n", "\n", "As we can see, negative inputs are no longer \"cut\" to zero. They are just \"down-weighted\" by a negative slope. Usually, this is left as a parameter for the user to choose, but it is standard to have a negative slope of .01. This gives negative inputs a chance to \"recover\" or still remain liable to make meaningful changes to our corresponding parameters.\n", "\n", "Figuring the optimal activation function for a model is often a matter of trial and error. Hence, we will apply the equivalent architecture as we did above but this time, we will use our newly defined activation functions." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "################## Tanh Layer #######################\n", "\n", "import torch.nn as nn\n", "import torch\n", "\n", "class tanh_layer(torch.autograd.Function):\n", " \n", " def __init__(self):\n", " ''\n", " \n", " def tanh(self,x):\n", " tanh = (x.exp() - (-x).exp()) / (x.exp() + (-x).exp())\n", " return tanh\n", " \n", " # forward pass\n", " def forward(self, input):\n", " # save input for backward() pass \n", " self.save_for_backward(input) \n", " activated_input = self.tanh(input)\n", " return activated_input\n", "\n", " # integrate backward pass with incoming_grad\n", " def backward(self, incoming_grad):\n", "\n", " input, = self.saved_tensors\n", " chained_grad = (1 - self.tanh(input)^2) * incoming_grad\n", " return chained_grad" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "class Tanh(nn.Module):\n", " def __init__(self):\n", " super().__init__()\n", " self.tanh = tanh_layer()\n", " \n", " def forward(self, input):\n", " output = self.tanh.forward(input)\n", " return output\n", " " ] }, { "cell_type": "code", "execution_count": 121, "metadata": {}, "outputs": [], "source": [ "################## Leaky ReLU Layer #######################\n", "\n", "class Leaky_ReLU_layer(torch.autograd.Function):\n", " \n", " @staticmethod\n", " def forward(self, input):\n", " # making operation in-place to tensors that hold \"history\" will mess up\n", " # the dynamic graph. Thus, clone it.\n", " activated_input = input.clone()\n", " self.save_for_backward(activated_input)\n", " activated_input[activated_input < 0] *= .01\n", " return activated_input\n", "\n", " @staticmethod\n", " def backward(self, incoming_grad):\n", "\n", " input, = self.saved_tensors\n", " output_grad = incoming_grad.clone()\n", " output_grad[input < 0] *= .01\n", " return output_grad " ] }, { "cell_type": "code", "execution_count": 122, "metadata": {}, "outputs": [], "source": [ "class LeakyReLU(nn.Module):\n", " def __init__(self):\n", " super().__init__()\n", " ''\n", " \n", " \n", " def forward(self, input):\n", " output = Leaky_ReLU_layer.apply(input)\n", " return output\n", " " ] }, { "cell_type": "code", "execution_count": 130, "metadata": {}, "outputs": [], "source": [ "# instantiate models\n", "tanh_model = NeuralNet(activation = Tanh()).to(device)\n", "leaky_model = NeuralNet(activation = LeakyReLU()).to(device)\n", "\n", "model = {'tanh': tanh_model,\n", " 'leaky_relu': leaky_model}\n", "\n", "\n", "optimizer = {'tanh': optim.SGD(tanh_model.parameters(), lr = .01),\n", " 'leaky_relu': optim.SGD(leaky_model.parameters(), lr = .01)}\n", "\n" ] }, { "cell_type": "code", "execution_count": 131, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 01 | Epoch Time: 0m 23s\n", "\tTrain Loss: 2.135 | Train Acc: 25.89%\n", "\t Val. Loss: 2.144 | Val. Acc: 16.62%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 02 | Epoch Time: 0m 38s\n", "\tTrain Loss: 1.789 | Train Acc: 43.35%\n", "\t Val. Loss: 2.489 | Val. Acc: 15.23%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 03 | Epoch Time: 0m 39s\n", "\tTrain Loss: 1.603 | Train Acc: 49.43%\n", "\t Val. Loss: 3.176 | Val. Acc: 16.60%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 04 | Epoch Time: 0m 39s\n", "\tTrain Loss: 1.510 | Train Acc: 51.15%\n", "\t Val. Loss: 3.610 | Val. Acc: 16.60%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 05 | Epoch Time: 0m 33s\n", "\tTrain Loss: 1.454 | Train Acc: 52.14%\n", "\t Val. Loss: 3.997 | Val. Acc: 17.10%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 06 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.411 | Train Acc: 52.73%\n", "\t Val. Loss: 4.143 | Val. Acc: 17.32%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 07 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.378 | Train Acc: 53.21%\n", "\t Val. Loss: 4.446 | Val. Acc: 16.69%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 08 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.351 | Train Acc: 53.54%\n", "\t Val. Loss: 4.687 | Val. Acc: 16.29%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 09 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.332 | Train Acc: 53.74%\n", "\t Val. Loss: 4.419 | Val. Acc: 16.83%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 10 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.313 | Train Acc: 54.02%\n", "\t Val. Loss: 4.609 | Val. Acc: 16.77%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 11 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.300 | Train Acc: 54.30%\n", "\t Val. Loss: 4.505 | Val. Acc: 16.28%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 12 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.288 | Train Acc: 54.45%\n", "\t Val. Loss: 4.570 | Val. Acc: 15.96%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 13 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.276 | Train Acc: 54.94%\n", "\t Val. Loss: 4.893 | Val. Acc: 15.96%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 14 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.266 | Train Acc: 54.99%\n", "\t Val. Loss: 4.865 | Val. Acc: 17.41%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 15 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.259 | Train Acc: 55.17%\n", "\t Val. Loss: 4.627 | Val. Acc: 11.21%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 16 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.249 | Train Acc: 55.60%\n", "\t Val. Loss: 4.820 | Val. Acc: 18.63%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 17 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.241 | Train Acc: 56.04%\n", "\t Val. Loss: 4.849 | Val. Acc: 16.69%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 18 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.233 | Train Acc: 56.27%\n", "\t Val. Loss: 4.656 | Val. Acc: 17.83%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 19 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.228 | Train Acc: 56.35%\n", "\t Val. Loss: 4.678 | Val. Acc: 17.43%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 20 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.219 | Train Acc: 56.68%\n", "\t Val. Loss: 4.487 | Val. Acc: 18.07%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 21 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.213 | Train Acc: 56.96%\n", "\t Val. Loss: 4.870 | Val. Acc: 17.18%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 22 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.207 | Train Acc: 57.20%\n", "\t Val. Loss: 4.697 | Val. Acc: 18.75%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 23 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.202 | Train Acc: 57.41%\n", "\t Val. Loss: 4.862 | Val. Acc: 16.95%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 24 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.194 | Train Acc: 57.54%\n", "\t Val. Loss: 4.773 | Val. Acc: 17.14%\n", "---------------------------------------------------------------------------\n", "Activation: tanh\n", "Epoch: 25 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.189 | Train Acc: 57.83%\n", "\t Val. Loss: 4.729 | Val. Acc: 17.24%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 01 | Epoch Time: 0m 17s\n", "\tTrain Loss: 2.229 | Train Acc: 19.80%\n", "\t Val. Loss: 2.163 | Val. Acc: 14.59%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 02 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.891 | Train Acc: 35.63%\n", "\t Val. Loss: 2.367 | Val. Acc: 17.35%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 03 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.675 | Train Acc: 45.36%\n", "\t Val. Loss: 2.842 | Val. Acc: 15.40%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 04 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.546 | Train Acc: 50.31%\n", "\t Val. Loss: 3.885 | Val. Acc: 17.01%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 05 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.461 | Train Acc: 51.76%\n", "\t Val. Loss: 3.837 | Val. Acc: 17.00%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 06 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.407 | Train Acc: 52.52%\n", "\t Val. Loss: 4.455 | Val. Acc: 14.65%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 07 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.371 | Train Acc: 53.03%\n", "\t Val. Loss: 5.038 | Val. Acc: 17.06%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 08 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.346 | Train Acc: 53.43%\n", "\t Val. Loss: 4.564 | Val. Acc: 16.89%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 09 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.324 | Train Acc: 53.94%\n", "\t Val. Loss: 4.592 | Val. Acc: 15.30%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 10 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.308 | Train Acc: 54.31%\n", "\t Val. Loss: 4.754 | Val. Acc: 13.86%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 11 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.294 | Train Acc: 54.36%\n", "\t Val. Loss: 4.803 | Val. Acc: 13.49%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 12 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.282 | Train Acc: 54.94%\n", "\t Val. Loss: 5.465 | Val. Acc: 15.55%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 13 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.270 | Train Acc: 55.12%\n", "\t Val. Loss: 5.038 | Val. Acc: 16.33%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 14 | Epoch Time: 0m 19s\n", "\tTrain Loss: 1.260 | Train Acc: 55.39%\n", "\t Val. Loss: 5.293 | Val. Acc: 16.86%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 15 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.251 | Train Acc: 55.74%\n", "\t Val. Loss: 4.618 | Val. Acc: 15.69%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 16 | Epoch Time: 0m 17s\n", "\tTrain Loss: 1.238 | Train Acc: 56.18%\n", "\t Val. Loss: 4.864 | Val. Acc: 17.60%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 17 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.228 | Train Acc: 56.52%\n", "\t Val. Loss: 5.425 | Val. Acc: 15.45%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 18 | Epoch Time: 0m 19s\n", "\tTrain Loss: 1.219 | Train Acc: 56.80%\n", "\t Val. Loss: 5.255 | Val. Acc: 16.88%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 19 | Epoch Time: 0m 19s\n", "\tTrain Loss: 1.211 | Train Acc: 57.14%\n", "\t Val. Loss: 5.215 | Val. Acc: 16.84%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 20 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.204 | Train Acc: 57.29%\n", "\t Val. Loss: 5.489 | Val. Acc: 16.86%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 21 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.197 | Train Acc: 57.66%\n", "\t Val. Loss: 5.389 | Val. Acc: 17.53%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 22 | Epoch Time: 0m 18s\n", "\tTrain Loss: 1.190 | Train Acc: 57.75%\n", "\t Val. Loss: 4.996 | Val. Acc: 16.31%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 23 | Epoch Time: 0m 19s\n", "\tTrain Loss: 1.185 | Train Acc: 58.14%\n", "\t Val. Loss: 5.293 | Val. Acc: 18.03%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 24 | Epoch Time: 0m 19s\n", "\tTrain Loss: 1.180 | Train Acc: 58.35%\n", "\t Val. Loss: 5.186 | Val. Acc: 17.22%\n", "---------------------------------------------------------------------------\n", "Activation: leaky_relu\n", "Epoch: 25 | Epoch Time: 0m 19s\n", "\tTrain Loss: 1.178 | Train Acc: 58.29%\n", "\t Val. Loss: 5.750 | Val. Acc: 18.29%\n" ] } ], "source": [ "N_EPOCHS = 25\n", "\n", "# track statistics\n", "track_stats = {'activation': [],\n", " 'epoch': [],\n", " 'train_loss': [],\n", " 'train_acc': [],\n", " 'valid_loss':[],\n", " 'valid_acc':[]}\n", "\n", "track_stats = {'tanh': track_stats.copy(),\n", " 'leaky_relu': track_stats.copy()} \n", "\n", "\n", "for activation in ['tanh','leaky_relu']:\n", " \n", " # re-initiate per activation\n", " best_valid_loss = float('inf')\n", "\n", "\n", " for epoch in range(N_EPOCHS):\n", "\n", " start_time = time.time()\n", "\n", " train_loss, train_acc = train(model[activation], train_dl, optimizer[activation], criterion)\n", " valid_loss, valid_acc = evaluate(model[activation], eval_dl, criterion)\n", "\n", " end_time = time.time()\n", "\n", " # record operations\n", " track_stats[activation]['activation'].append(activation)\n", " track_stats[activation]['epoch'].append(epoch + 1)\n", " track_stats[activation]['train_loss'].append(train_loss)\n", " track_stats[activation]['train_acc'].append(train_acc)\n", " track_stats[activation]['valid_loss'].append(valid_loss)\n", " track_stats[activation]['valid_acc'].append(valid_acc)\n", "\n", "\n", "\n", " epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n", "\n", " # if this was our best performance, record model parameters\n", " if valid_loss < best_valid_loss:\n", " best_valid_loss = valid_loss\n", " torch.save(model[activation].state_dict(), 'best_'+activation+'_params.pt')\n", "\n", " # print out stats\n", " print('-'*75)\n", " print(f'Activation: {activation}')\n", " print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n", " print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n", " print(f'\\t Val. Loss: {valid_loss:.3f} | Val. Acc: {valid_acc*100:.2f}%')\n", " \n", " # if this is the last loop, save statistics\n", " if epoch + 1 == N_EPOCHS:\n", " torch.save(track_stats[activation],activation + '_stats.pt')\n", " \n", "\n" ] }, { "cell_type": "code", "execution_count": 186, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
activationepochtrain_losstrain_accvalid_lossvalid_acc
0ReLU12.2567060.1966782.2078360.114715
1ReLU21.9282050.3599412.6582920.150910
2ReLU31.6841900.4590052.6972130.180676
3ReLU41.5303930.5022653.5326500.160502
4ReLU51.4438330.5197733.8613900.164161
\n", "
" ], "text/plain": [ " activation epoch train_loss train_acc valid_loss valid_acc\n", "0 ReLU 1 2.256706 0.196678 2.207836 0.114715\n", "1 ReLU 2 1.928205 0.359941 2.658292 0.150910\n", "2 ReLU 3 1.684190 0.459005 2.697213 0.180676\n", "3 ReLU 4 1.530393 0.502265 3.532650 0.160502\n", "4 ReLU 5 1.443833 0.519773 3.861390 0.164161" ] }, "execution_count": 186, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "\n", "relu_stats = pd.DataFrame(torch.load('ReLU_stats.pt'))\n", "tanh_stats = pd.DataFrame(track_stats['tanh'])\n", "leaky_stats = pd.DataFrame(track_stats['leaky_relu'])\n", "all_stats = pd.concat([relu_stats, tanh_stats, leaky_stats])\n", "all_stats.head()" ] }, { "cell_type": "code", "execution_count": 199, "metadata": {}, "outputs": [], "source": [ "data = []\n", "\n", "for row in all_stats.iterrows():\n", " data.append(row[1].to_dict()) " ] }, { "cell_type": "code", "execution_count": 204, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "\n", " \n", " \n", " HiPlot\n", " \n", " \n", " \n", "
Loading HiPlot...
\n", " \n", "
\n", "\n", " \n", " \n", " \n", "
\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 204, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import hiplot as hip\n", "hip.Experiment.from_iterable(data).display(force_full_width = True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are many insights our graph sheds light to:\n", "\n", "* tanh and leaky ReLU attained the best validation accuracies \n", "* ReLU reached its best accuracy early on the process at epoch 3 while tanh and leaky ReLU reached their best performance along epochs 16 to 25. \n", "\n", "Just the above two insights let us know that ReLU very quickly overfits our data while tanh and leaky ReLU take more time, which is a desirable property.\n", "\n", "Overall, if we had to choose one activation operation, we would go with tanh as it received the top two best validation accuracies.\n", "\n", "# Conclusion\n", "\n", "Activation functions are fundamental to any DL architecture and understanding their theory along with their implementation will make one more \"aware\" of the care and importance that goes on choosing the right activation function. Hopefully, it may also bring you to create your own, never before seen, activation function.\n", "\n", "That is it for this tutorial. Thank you guys!\n", "\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Where to Next?\n", "\n", "**Logistic Regression Tutorial:**\n", "https://nbviewer.jupyter.org/github/Erick7451/DL-with-PyTorch/blob/master/Jupyter_Notebooks/logistic_regression.ipynb" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" } }, "nbformat": 4, "nbformat_minor": 4 }