{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Important: This notebook will only work with fastai-0.7.x. Do not try to run any fastai-1.x code from this path in the repository because it will load fastai-0.7.x**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Table of Contents\n",
" <p><div class=\"lev1 toc-item\"><a href=\"#Linear-Regression-problem\" data-toc-modified-id=\"Linear-Regression-problem-1\"><span class=\"toc-item-num\">1 </span>Linear Regression problem</a></div><div class=\"lev1 toc-item\"><a href=\"#Gradient-Descent\" data-toc-modified-id=\"Gradient-Descent-2\"><span class=\"toc-item-num\">2 </span>Gradient Descent</a></div><div class=\"lev1 toc-item\"><a href=\"#Gradient-Descent---Classification\" data-toc-modified-id=\"Gradient-Descent---Classification-3\"><span class=\"toc-item-num\">3 </span>Gradient Descent - Classification</a></div><div class=\"lev1 toc-item\"><a href=\"#Gradient-descent-with-numpy\" data-toc-modified-id=\"Gradient-descent-with-numpy-4\"><span class=\"toc-item-num\">4 </span>Gradient descent with numpy</a></div>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
"from fastai.learner import *"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this part of the lecture we explain Stochastic Gradient Descent (SGD) which is an **optimization** method commonly used in neural networks. We will illustrate the concepts with concrete examples."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Linear Regression problem"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The goal of linear regression is to fit a line to a set of points."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Here we generate some fake data\n",
"def lin(a,b,x): return a*x+b\n",
"\n",
"def gen_fake_data(n, a, b):\n",
" x = s = np.random.uniform(0,1,n) \n",
" y = lin(a,b,x) + 0.1 * np.random.normal(0,3,n)\n",
" return x, y\n",
"\n",
"x, y = gen_fake_data(50, 3., 8.)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAF5dJREFUeJzt3X2QZXV95/H3x4EoikmAaRTRdiQhW1KkkrgNUeO6JBgl\nlAUVVyNmLcFSp3RFs2Y3G63dCpbZZDUPlexqEndUCt2NBKNRp9SNUj6EuAnWNKhxkHXBcRxGUFrH\nkJoSdQa++8e9uE1ze+b00Oece+59v6q6+j78bvf30MP93N/TOakqJEk6mof0XYAkaRgMDElSIwaG\nJKkRA0OS1IiBIUlqxMCQJDViYEiSGjEwJEmNGBiSpEaO67uAzbR169batm1b32VI0mDccMMN36yq\nhSZtZyowtm3bxvLyct9lSNJgJPlq07YOSUmSGmktMJJcmeTOJLtXPfa8JDcluTfJ0hFeuzfJF5J8\nLoldBkmaAm32MK4CLljz2G7gOcB1DV7/81X101W1brBIkrrT2hxGVV2XZNuax24GSNLWr5UktWRa\n5zAK+FiSG5Js77sYSdL0rpL6uaq6PcmpwLVJ/k9VTRzGGgfKdoDFxcUua5SkuTKVPYyqun38/U7g\n/cC5R2i7o6qWqmppYaHRUmJJ0jGYusBI8ogkj7zvNvBMRpPlkjRX9qwc5Jpd+9izcrDvUoAWh6SS\nXA2cB2xNsh+4AjgAvBlYAD6c5HNV9awkjwHeXlUXAo8C3j+eGD8OeHdV/XVbdUrSNNqzcpBnv/nT\nVEECH3rV0zhj4cRea2pzldQL1nnq/RPa3g5cOL69B/iptuqSpCHYtfcAVXD3oXs44fgt7Np7oPfA\nmLohKUkSnLPtZBI44fgtJKP7fZvWVVKSNNfOWDiRD73qaezae4Bztp3ce+8CDAxJmlpnLJw4FUFx\nH4ekJEmNGBiSpEYMDElSIwaGJKkRA0OS1IiBIUlqxMCQJDViYEiSGjEwJEmNGBiSpEYMDElSIwaG\nJKkRA0OS1IiBIUlqxMCQJDViYEiSGmktMJJcmeTOJLtXPfa8JDcluTfJ0hFee0GSLyW5Nclr26pR\nktRcmz2Mq4AL1jy2G3gOcN16L0qyBfgT4JeAs4AXJDmrpRolSQ21FhhVdR1wYM1jN1fVl47y0nOB\nW6tqT1V9H/gL4OKWypSkQdmzcpBrdu1jz8rBzn/3NF7T+3TgtlX39wM/u17jJNuB7QCLi4vtViZJ\nPdqzcpBnv/nTVEECH3rV0zq95vc0TnpnwmO1XuOq2lFVS1W1tLCw0GJZktSvXXsPUAV3H7qHqtH9\nLk1jYOwHHrfq/mOB23uqRZKmxjnbTiaBE47fQjK636VpHJLaBZyZ5AnA14BLgF/ttyRJ6t8ZCyfy\noVc9jV17D3DOtpM7HY6CFgMjydXAecDWJPuBKxhNgr8ZWAA+nORzVfWsJI8B3l5VF1bV4SSXAx8F\ntgBXVtVNbdUpScdqz8rBzt+8z1g4sfOguE+q1p0eGJylpaVaXl7uuwxJc6DvCejNkuSGqlp3X9xq\n0ziHIUlT70gT0H0ufW3TNM5hSNLUW28CelZ6HpMYGJLm0oOdf1hvAnp1z+OE47ewa+8BA0OShmqz\negGTJqD7XvraJgND0txpsxfQ99LXNhkYkuZO272APpe+tsnAkDR3ZrkX0CYDQ9JcmtVeQJvchyFJ\nA9X1fg97GJI0QH3s97CHIUkD1Mepzg0MSRqgPvZ7OCQlSQPUx0ovA0OSBqrrlV4OSUmSGjEwJEmN\nGBiSpEYMDEkzb1YvaNQ1J70lzbRZvqBR11rrYSS5MsmdSXaveuzkJNcmuWX8/aR1XntPks+Nv3a2\nVaOk2dfHBrdZ1eaQ1FXABWseey3w8ao6E/j4+P4kd1fVT4+/LmqxRkkzbpYvaNS11oakquq6JNvW\nPHwxcN749juBTwG/2VYNkuSpzDdP13MYj6qqOwCq6o4kp67T7mFJloHDwBur6gPr/cAk24HtAIuL\ni5tdr6QZ4KnMN8e0rpJarKol4FeBP07yY+s1rKodVbVUVUsLCwvdVShJc6brwPhGktMAxt/vnNSo\nqm4ff9/DaNjqZ7oqUJI0WdeBsRO4dHz7UuCDaxskOSnJQ8e3twI/B3yxswolSRO1uaz2auDvgX+W\nZH+SlwBvBH4xyS3AL47vk2QpydvHL30isJzk88AnGc1hGBiS1LM2V0m9YJ2nzp/Qdhl46fj23wE/\n2VZdkqRjM62T3pKkKWNgSJIaMTAkSY0YGJKkRgwMSVIjBoYkqREDQ9LM8sJJm8sLKEmaSV44afPZ\nw5DUmj4/4XvhpM1nD0NSK/r+hL/RCyftWTnoNTOOwsCQ1IrVn/BPOH4Lu/Ye6PSNeCMXTuo73IbC\nwJDUimm4NGrTCyf1HW5DYWBIasWQLo06DeE2BAaGpNYM5dKoQwq3PhkYksRwwq1PLquVJDViYEiS\nGjEwJEmNtBoYSa5McmeS3aseOznJtUluGX8/aZ3XXjpuc0uSS9usU1I7PJfTbGm7h3EVcMGax14L\nfLyqzgQ+Pr5/P0lOBq4AfhY4F7hivWCRNJ3u2wz3+p1f5Nlv/vQDQsMwGZ5WV0lV1XVJtq15+GLg\nvPHtdwKfAn5zTZtnAddW1QGAJNcyCp6rWypV0iZbbzPcnpWDfOQLd/CWT95KiDurB6SPZbWPqqo7\nAKrqjiSnTmhzOnDbqvv7x49JGohJm+Hu63UcuudeDt1TAO6sHpBp3YeRCY/VxIbJdmA7wOLiYps1\nSdqASZvhrtm1jyp+EBbHb4k7qwekj8D4RpLTxr2L04A7J7TZz/8ftgJ4LKOhqweoqh3ADoClpaWJ\noSKpH2s3w63udRTF5T//41z4k6fZuxiIPgJjJ3Ap8Mbx9w9OaPNR4HdXTXQ/E3hdN+VJm89TZ494\nCo5hazUwklzNqKewNcl+Riuf3gi8J8lLgH3A88Ztl4CXV9VLq+pAkt8Gdo1/1BvumwCXhsZTZ9+f\np+AYrrZXSb1gnafOn9B2GXjpqvtXAle2VJrUmSGeOtsekSaZ1klvaWYM7dTZ9oi0HgNDatnQxu2H\n2CNSNwwMqQOrx+2nfbhnaD0idcfAkDo0hOGeofWI1B0DQ+rQUIZ7XMmkSTy9udShroZ7PLGf2mAP\nQ+pQF8M9Qxj20jAZGFLH2h7uGcqwl4bHISlpBqwegmpz2KvNoS6H0aafPQxpCm1k6e2kIag2hr3a\nHOpyGG0YDAxpymz0zXPSENTzz1nc9DfcNoe6HEYbBoekpA1qe+hk9Ztn1ej+kXS18qrN3+NmwWE4\nag8jyeXAn1fVtzuoR5pqXQydbPTNs6uNdm3+HjcLDkOTIalHA7uS3Mjo7LEfrSovVKS51MXQybG8\neXa10a7N3+Nmwel31CGpqvpPwJnAO4DLgFuS/G6SH2u5NmnqdDV0csbCia3MQ0gPRqNJ76qqJF8H\nvg4cBk4C3pvk2qr6D20WKE0Th040z5rMYbya0aVUvwm8HfiNqjqU5CHALYCBobni0InmVZMexlbg\nOVX11dUPVtW9SZ7dTllSfzZ6+vH12k/7acyljTpqYFTVbx3huZs3txypXxtdBbVeezeiaRb1sg8j\nya8l2Z3kpiT/dsLz5yW5K8nnxl/rhpa0mTa6B2K99hv9OdIQdL7TO8nZwMuAc4HvA3+d5MNVdcua\npn9bVQ55qVMbXQW1XvtH//DDuOfee3nocQ9xI5pmRh+nBnkicH1VfQcgyd8Avwz8Xg+1SPez0VVQ\nk9rvWTnIK/78RpJQBX/2r5/kcJRmQh+BsRv4nSSnAHcDFwLLE9o9JcnngduBf19VN3VYo+bYRq+/\nvXbV1H3DUd87fC8nHL+Fr//TdzupW2pb54FRVTcneRNwLXAQ+DyjvR2r3Qg8vqoOJrkQ+ACjzYMP\nkGQ7sB1gcXGxtbo1f4514trzImlW9TLpXVXvqKonVdXTgQOM9nOsfv6fqurg+PZHgOOTbF3nZ+2o\nqqWqWlpYWGi9ds2PY524vm+Y6vUXneXqKM2UXk5vnuTUqrozySLwHOApa55/NPCN8Q7zcxkF27d6\nKFVz7MH0FNzcp1nU1/Uw3jeewzgEvLKqvp3k5QBV9VbgucArkhxmNM9xiSc81Fptb4zzNCDS/WWW\n3oeXlpZqeXnS/LlmjRvjpM2R5IaqWmrS1gsoaZDcGCd1z8DQILkSSeqe1/TWIDm/IHXPwNBguRJJ\n6pZDUpKkRgwMSVIjBoYGY8/KQa7ZtY89Kwf7LkWaS85haBDcdyH1zx6GBsF9F1L/DAwNgvsupP45\nJKVBcN/FZG2fT0tazcDQYLjv4v6c11HXHJKSWtTmyi7nddQ1exganKEMw7TdA3BeR10zMDQoQxqG\nWd0DOOH4Lezae2BTa3VeR10zMDQobb8Jb6YuegDO66hLBoYGZUjDMPYANGsMDA3K0N6E7QFolhgY\nGpy23oSHMpku9aWXwEjya8DLgABvq6o/XvN8gP8KXAh8B7isqm7svFDNjSFNpkt96XwfRpKzGYXF\nucBPAc9OcuaaZr8EnDn+2g78WadFau64p0E6uj427j0RuL6qvlNVh4G/AX55TZuLgXfVyPXAjyY5\nretCNT+GNJku9aWPIandwO8kOQW4m9Gw0/KaNqcDt626v3/82B1rf1iS7Yx6ISwuLrZRr+bA0CbT\npT50HhhVdXOSNwHXAgeBzwOH1zTLpJeu8/N2ADsAlpaWJraRmnBFk3RkvZxLqqreUVVPqqqnAweA\nW9Y02Q88btX9xwK3d1WfJOmBegmMJKeOvy8CzwGuXtNkJ/CijDwZuKuqHjAcJUnqTl/7MN43nsM4\nBLyyqr6d5OUAVfVW4COM5jZuZbSs9sU91SlJGuslMKrqX0x47K2rbhfwyk6LUiNubpPmlzu91Zib\n26T55gWU1Jib26T5ZmCoMTe3SfPNISkd0do5Cze3SfPLwNC6E9nrzVkYFNJ8MjDm3JEmsod0dTtJ\n7XMOY84daSLbOQtJq9nDmHNHCgXnLCStZmDMuaOFgnMWku5jYMhQkNSIcxiSpEYMDElSIwaGJKkR\nA0OS1IiBMUf2rBzkml372LNysO9SJA2Qq6TmxKQd3YB7LCQ1ZmAMwGZctGjtaT4+8oU7+NNPfdlr\nW0hqzMCYcpt10aK1O7qBTs8T5ZX6pOEzMKbcZp0AcO2OboA//dSXOzlPlFfqk2ZDL4GR5DXAS4EC\nvgC8uKq+u+r5y4DfB742fugtVfX2rut8MDbrE/VmngBw7Y7urs4T5VlvpdnQeWAkOR14NXBWVd2d\n5D3AJcBVa5peU1WXd13fZtjMT9RtngCwq1OCeNZbaTb0NSR1HHBCkkPAw4Hbe6qjFZv9iXro53ry\nrLfSbOg8MKrqa0n+ANgH3A18rKo+NqHpv0rydOD/Aq+pqtsm/bwk24HtAIuLiy1VvTF+on6goYee\nJEhVdfsLk5OA9wHPB/4R+EvgvVX1P1e1OQU4WFXfS/Jy4Feq6heO9rOXlpZqeXm5pco35kiXPfWT\ntqRpkeSGqlpq0raPIalnAF+pqhWAJH8FPBX4QWBU1bdWtX8b8KZOK9wEkz5Rb/ZqIcNHUpf6CIx9\nwJOTPJzRkNT5wP26BUlOq6o7xncvAm7utsR2bObcRtdLVQ0nSX3MYXwmyXuBG4HDwGeBHUneACxX\n1U7g1UkuGj9/ALis6zrb8GDmNta+YXe5VNV9FJKgp1VSVXUFcMWah39r1fOvA17XaVEdONbVQpPe\nsLucWHcfhSRwp3fnjmW10KQ37Oefs9jZUlVXfUkCA2MQ1nvD7mqpqvsoJIGBMQjT8IbtPgpJBsZA\n+IYtqW9ecU+S1IiBIUlqxMCQJDViYEiSGjEwJEmNGBhTZM/KQa7ZtY89Kwf7LkWSHsBltVPC8zVJ\nmnb2MKbE6tN/VI3uS9I0MTCmhOdrkjTtHJKaEtNw+g9JOhID4xi0dTEhT/8haZoZGBvk5LSkeeUc\nxljTJa1OTkuaV/Yw2Fiv4WiT0177WtKs6iUwkrwGeClQwBeAF1fVd1c9/1DgXcA/B74FPL+q9rZV\nz0YuQXqkyWmHqyTNss6HpJKcDrwaWKqqs4EtwCVrmr0E+HZV/TjwR8Cb2qxpo0taz1g4keefs/iA\nMHC4StIs62tI6jjghCSHgIcDt695/mLg9ePb7wXekiRVVW0Us1lLWt1LIWmWdR4YVfW1JH8A7APu\nBj5WVR9b0+x04LZx+8NJ7gJOAb7ZVl2bsaTVvRSSZlkfQ1InMepBPAF4DPCIJC9c22zCSyf2LpJs\nT7KcZHllZWVziz0G6w1XSdLQ9bGs9hnAV6pqpaoOAX8FPHVNm/3A4wCSHAf8CDBxQqCqdlTVUlUt\nLSwstFi2JM23PgJjH/DkJA9PEuB84OY1bXYCl45vPxf4RFvzF5KkZjoPjKr6DKOJ7BsZLal9CLAj\nyRuSXDRu9g7glCS3Ar8OvLbrOiVJ95dZ+uC+tLRUy8vLfZchSYOR5IaqWmrS1lODrOFV7yRpMk8N\nsoo7tSVpffYwVnGntiStz8BYxZ3akrQ+h6RWcae2JK3PwFjDq95J0mQOSUmSGjEwJEmNGBiSpEYM\nDElSIwaGJKkRA0OS1MhMnXwwyQrw1Q28ZCstXsVvSs3jMYPHPU/m8Zjh2I/78VXV6GJCMxUYG5Vk\nuelZGmfFPB4zeNx919GleTxm6Oa4HZKSJDViYEiSGpn3wNjRdwE9mMdjBo97nszjMUMHxz3XcxiS\npObmvYchSWpoLgIjyQVJvpTk1iSvnfD8Q5NcM37+M0m2dV/l5mpwzL+e5ItJ/iHJx5M8vo86N9vR\njntVu+cmqSSDX03T5JiT/Mr4731Tknd3XWMbGvwbX0zyySSfHf87v7CPOjdTkiuT3Jlk9zrPJ8l/\nG/83+YckT9rUAqpqpr+ALcCXgTOAHwI+D5y1ps2/Ad46vn0JcE3fdXdwzD8PPHx8+xVDP+amxz1u\n90jgOuB6YKnvujv4W58JfBY4aXz/1L7r7ui4dwCvGN8+C9jbd92bcNxPB54E7F7n+QuB/wUEeDLw\nmc38/fPQwzgXuLWq9lTV94G/AC5e0+Zi4J3j2+8Fzk+SDmvcbEc95qr6ZFV9Z3z3euCxHdfYhiZ/\na4DfBn4P+G6XxbWkyTG/DPiTqvo2QFXd2XGNbWhy3AX88Pj2jwC3d1hfK6rqOuBI146+GHhXjVwP\n/GiS0zbr989DYJwO3Lbq/v7xYxPbVNVh4C7glE6qa0eTY17tJYw+lQzdUY87yc8Aj6uqD3VZWIua\n/K1/AviJJP87yfVJLuisuvY0Oe7XAy9Msh/4CPCqbkrr1Ub/39+Qebji3qSewtqlYU3aDEnj40ny\nQmAJ+JetVtSNIx53kocAfwRc1lVBHWjytz6O0bDUeYx6kn+b5Oyq+seWa2tTk+N+AXBVVf1hkqcA\n/2N83Pe2X15vWn0vm4cexn7gcavuP5YHdk1/0CbJcYy6r0fq9k27JsdMkmcA/xG4qKq+11FtbTra\ncT8SOBv4VJK9jMZ4dw584rvpv+8PVtWhqvoK8CVGATJkTY77JcB7AKrq74GHMTrf0ixr9P/+sZqH\nwNgFnJnkCUl+iNGk9s41bXYCl45vPxf4RI1nkAbqqMc8Hpr574zCYhbGtOEox11Vd1XV1qraVlXb\nGM3dXFRVy/2Uuyma/Pv+AKNFDiTZymiIak+nVW6+Jse9DzgfIMkTGQXGSqdVdm8n8KLxaqknA3dV\n1R2b9cNnfkiqqg4nuRz4KKOVFVdW1U1J3gAsV9VO4B2Muqu3MupZXNJfxQ9ew2P+feBE4C/H8/v7\nquqi3oreBA2Pe6Y0POaPAs9M8kXgHuA3qupb/VX94DU87n8HvC3JaxgNy1w28A+CJLma0dDi1vHc\nzBXA8QBV9VZGczUXArcC3wFevKm/f+D//SRJHZmHISlJ0iYwMCRJjRgYkqRGDAxJUiMGhiSpEQND\nktSIgSFJasTAkFqS5JzxNQkeluQR42tRnN13XdKxcuOe1KIk/5nRKSlOAPZX1X/puSTpmBkYUovG\n5znaxejaG0+tqnt6Lkk6Zg5JSe06mdE5ux7JqKchDZY9DKlFSXYyuhrcE4DTqurynkuSjtnMn61W\n6kuSFwGHq+rdSbYAf5fkF6rqE33XJh0LexiSpEacw5AkNWJgSJIaMTAkSY0YGJKkRgwMSVIjBoYk\nqREDQ5LUiIEhSWrk/wGHmmYdLod/igAAAABJRU5ErkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7f8f91c22e10>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plt.scatter(x,y, s=8); plt.xlabel(\"x\"); plt.ylabel(\"y\"); "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You want to find **parameters** (weights) $a$ and $b$ such that you minimize the *error* between the points and the line $a\\cdot x + b$. Note that here $a$ and $b$ are unknown. For a regression problem the most common *error function* or *loss function* is the **mean squared error**. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def mse(y_hat, y): return ((y_hat - y) ** 2).mean()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Suppose we believe $a = 10$ and $b = 5$ then we can compute `y_hat` which is our *prediction* and then compute our error."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"4.1001300495563058"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y_hat = lin(10,5,x)\n",
"mse(y_hat, y)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def mse_loss(a, b, x, y): return mse(lin(a,b,x), y)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"4.1001300495563058"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mse_loss(10, 5, x, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So far we have specified the *model* (linear regression) and the *evaluation criteria* (or *loss function*). Now we need to handle *optimization*; that is, how do we find the best values for $a$ and $b$? How do we find the best *fitting* linear regression."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Gradient Descent"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For a fixed dataset $x$ and $y$ `mse_loss(a,b)` is a function of $a$ and $b$. We would like to find the values of $a$ and $b$ that minimize that function.\n",
"\n",
"**Gradient descent** is an algorithm that minimizes functions. Given a function defined by a set of parameters, gradient descent starts with an initial set of parameter values and iteratively moves toward a set of parameter values that minimize the function. This iterative minimization is achieved by taking steps in the negative direction of the function gradient.\n",
"\n",
"Here is gradient descent implemented in [PyTorch](http://pytorch.org/)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"((10000,), (10000,))"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# generate some more data\n",
"x, y = gen_fake_data(10000, 3., 8.)\n",
"x.shape, y.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"x,y = V(x),V(y)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(Variable containing:\n",
" 1.00000e-02 *\n",
" 2.9873\n",
" [torch.FloatTensor of size 1], Variable containing:\n",
" 0.1116\n",
" [torch.FloatTensor of size 1])"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Create random weights a and b, and wrap them in Variables.\n",
"a = V(np.random.randn(1), requires_grad=True)\n",
"b = V(np.random.randn(1), requires_grad=True)\n",
"a,b"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"89.19391632080078\n",
"0.6885505318641663\n",
"0.11982045322656631\n",
"0.11007291823625565\n",
"0.10528462380170822\n",
"0.10161882638931274\n",
"0.09879907965660095\n",
"0.09662991762161255\n",
"0.09496115148067474\n",
"0.09367774426937103\n"
]
}
],
"source": [
"learning_rate = 1e-3\n",
"for t in range(10000):\n",
" # Forward pass: compute predicted y using operations on Variables\n",
" loss = mse_loss(a,b,x,y)\n",
" if t % 1000 == 0: print(loss.data[0])\n",
" \n",
" # Computes the gradient of loss with respect to all Variables with requires_grad=True.\n",
" # After this call a.grad and b.grad will be Variables holding the gradient\n",
" # of the loss with respect to a and b respectively\n",
" loss.backward()\n",
" \n",
" # Update a and b using gradient descent; a.data and b.data are Tensors,\n",
" # a.grad and b.grad are Variables and a.grad.data and b.grad.data are Tensors\n",
" a.data -= learning_rate * a.grad.data\n",
" b.data -= learning_rate * b.grad.data\n",
" \n",
" # Zero the gradients\n",
" a.grad.data.zero_()\n",
" b.grad.data.zero_() "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Nearly all of deep learning is powered by one very important algorithm: **stochastic gradient descent (SGD)**. SGD can be seeing as an approximation of **gradient descent** (GD). In GD you have to run through *all* the samples in your training set to do a single itaration. In SGD you use *only one* or *a subset* of training samples to do the update for a parameter in a particular iteration. The subset use in every iteration is called a **batch** or **minibatch**."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Gradient Descent - Classification"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For a fixed dataset $x$ and $y$ `mse_loss(a,b)` is a function of $a$ and $b$. We would like to find the values of $a$ and $b$ that minimize that function.\n",
"\n",
"**Gradient descent** is an algorithm that minimizes functions. Given a function defined by a set of parameters, gradient descent starts with an initial set of parameter values and iteratively moves toward a set of parameter values that minimize the function. This iterative minimization is achieved by taking steps in the negative direction of the function gradient.\n",
"\n",
"Here is gradient descent implemented in [PyTorch](http://pytorch.org/)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def gen_fake_data2(n, a, b):\n",
" x = s = np.random.uniform(0,1,n) \n",
" y = lin(a,b,x) + 0.1 * np.random.normal(0,3,n)\n",
" return x, np.where(y>10, 1, 0).astype(np.float32)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"x,y = gen_fake_data2(10000, 3., 8.)\n",
"x,y = V(x),V(y)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def nll(y_hat, y):\n",
" y_hat = torch.clamp(y_hat, 1e-5, 1-1e-5)\n",
" return (y*y_hat.log() + (1-y)*(1-y_hat).log()).mean()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"a = V(np.random.randn(1), requires_grad=True)\n",
"b = V(np.random.randn(1), requires_grad=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"ename": "RuntimeError",
"evalue": "bool value of Variable objects containing non-empty torch.ByteTensor is ambiguous",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/numpy/core/fromnumeric.py\u001b[0m in \u001b[0;36m_wrapfunc\u001b[0;34m(obj, method, *args, **kwds)\u001b[0m\n\u001b[1;32m 56\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 57\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 58\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m 64\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 65\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__getattribute__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 66\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mAttributeError\u001b[0m: 'Variable' object has no attribute 'clip'",
"\nDuring handling of the above exception, another exception occurred:\n",
"\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-67-b1668cd3a12f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0mp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mlin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0my_hat\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnll\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_hat\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 6\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;36m1000\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 7\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mloss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mto_np\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m==\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mto_np\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_hat\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;36m0.5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m<ipython-input-64-fbf65a879f37>\u001b[0m in \u001b[0;36mnll\u001b[0;34m(y_hat, y)\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mnll\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_hat\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0my_hat\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_hat\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1e-5\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1e-5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0my_hat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlog\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0my_hat\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlog\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/numpy/core/fromnumeric.py\u001b[0m in \u001b[0;36mclip\u001b[0;34m(a, a_min, a_max, out)\u001b[0m\n\u001b[1;32m 1705\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1706\u001b[0m \"\"\"\n\u001b[0;32m-> 1707\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0m_wrapfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'clip'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0ma_min\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0ma_max\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1708\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1709\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/numpy/core/fromnumeric.py\u001b[0m in \u001b[0;36m_wrapfunc\u001b[0;34m(obj, method, *args, **kwds)\u001b[0m\n\u001b[1;32m 65\u001b[0m \u001b[0;31m# a downstream library like 'pandas'.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 66\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mAttributeError\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 67\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0m_wrapit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 68\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 69\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/numpy/core/fromnumeric.py\u001b[0m in \u001b[0;36m_wrapit\u001b[0;34m(obj, method, *args, **kwds)\u001b[0m\n\u001b[1;32m 45\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 46\u001b[0m \u001b[0mwrap\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 47\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 48\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mwrap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 49\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmu\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mndarray\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py\u001b[0m in \u001b[0;36m__bool__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 121\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 122\u001b[0m raise RuntimeError(\"bool value of Variable objects containing non-empty \" +\n\u001b[0;32m--> 123\u001b[0;31m torch.typename(self.data) + \" is ambiguous\")\n\u001b[0m\u001b[1;32m 124\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 125\u001b[0m \u001b[0m__nonzero__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m__bool__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mRuntimeError\u001b[0m: bool value of Variable objects containing non-empty torch.ByteTensor is ambiguous"
]
}
],
"source": [
"learning_rate = 1e-2\n",
"for t in range(3000):\n",
" p = (-lin(a,b,x)).exp()\n",
" y_hat = 1/(1+p)\n",
" loss = nll(y_hat,y)\n",
" if t % 1000 == 0:\n",
" print(loss.data[0], np.mean(to_np(y)==(to_np(y_hat)>0.5)))\n",
"# print(y_hat)\n",
" \n",
" loss.backward()\n",
" a.data -= learning_rate * a.grad.data\n",
" b.data -= learning_rate * b.grad.data\n",
" a.grad.data.zero_()\n",
" b.grad.data.zero_() "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Nearly all of deep learning is powered by one very important algorithm: **stochastic gradient descent (SGD)**. SGD can be seeing as an approximation of **gradient descent** (GD). In GD you have to run through *all* the samples in your training set to do a single itaration. In SGD you use *only one* or *a subset* of training samples to do the update for a parameter in a particular iteration. The subset use in every iteration is called a **batch** or **minibatch**."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Gradient descent with numpy"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from matplotlib import rcParams, animation, rc\n",
"from ipywidgets import interact, interactive, fixed\n",
"from ipywidgets.widgets import *\n",
"rc('animation', html='html5')\n",
"rcParams['figure.figsize'] = 3, 3"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"x, y = gen_fake_data(50, 3., 8.)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"65.167827371047636"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"a_guess,b_guess = -1., 1.\n",
"mse_loss(a_guess, b_guess, x, y)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lr=0.01\n",
"def upd():\n",
" global a_guess, b_guess\n",
" y_pred = lin(a_guess, b_guess, x)\n",
" dydb = 2 * (y_pred - y)\n",
" dyda = x*dydb\n",
" a_guess -= lr*dyda.mean()\n",
" b_guess -= lr*dydb.mean()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<video width=\"500\" height=\"400\" controls autoplay loop>\n",
" <source type=\"video/mp4\" src=\"data:video/mp4;base64,AAAAHGZ0eXBNNFYgAAACAGlzb21pc28yYXZjMQAAAAhmcmVlAABDlW1kYXQAAAKvBgX//6vcRem9\n",
"5tlIt5Ys2CDZI+7veDI2NCAtIGNvcmUgMTQ4IHIyNjk5IGE1ZTA2YjkgLSBILjI2NC9NUEVHLTQg\n",
"QVZDIGNvZGVjIC0gQ29weWxlZnQgMjAwMy0yMDE2IC0gaHR0cDovL3d3dy52aWRlb2xhbi5vcmcv\n",
"eDI2NC5odG1sIC0gb3B0aW9uczogY2FiYWM9MSByZWY9MyBkZWJsb2NrPTE6MDowIGFuYWx5c2U9\n",
"MHgzOjB4MTEzIG1lPWhleCBzdWJtZT03IHBzeT0xIHBzeV9yZD0xLjAwOjAuMDAgbWl4ZWRfcmVm\n",
"PTEgbWVfcmFuZ2U9MTYgY2hyb21hX21lPTEgdHJlbGxpcz0xIDh4OGRjdD0xIGNxbT0wIGRlYWR6\n",
"b25lPTIxLDExIGZhc3RfcHNraXA9MSBjaHJvbWFfcXBfb2Zmc2V0PS0yIHRocmVhZHM9MTIgbG9v\n",
"a2FoZWFkX3RocmVhZHM9MiBzbGljZWRfdGhyZWFkcz0wIG5yPTAgZGVjaW1hdGU9MSBpbnRlcmxh\n",
"Y2VkPTAgYmx1cmF5X2NvbXBhdD0wIGNvbnN0cmFpbmVkX2ludHJhPTAgYmZyYW1lcz0zIGJfcHly\n",
"YW1pZD0yIGJfYWRhcHQ9MSBiX2JpYXM9MCBkaXJlY3Q9MSB3ZWlnaHRiPTEgb3Blbl9nb3A9MCB3\n",
"ZWlnaHRwPTIga2V5aW50PTI1MCBrZXlpbnRfbWluPTEwIHNjZW5lY3V0PTQwIGludHJhX3JlZnJl\n",
"c2g9MCByY19sb29rYWhlYWQ9NDAgcmM9Y3JmIG1idHJlZT0xIGNyZj0yMy4wIHFjb21wPTAuNjAg\n",
"cXBtaW49MCBxcG1heD02OSBxcHN0ZXA9NCBpcF9yYXRpbz0xLjQwIGFxPTE6MS4wMACAAAAa+GWI\n",
"hAAQ//73gb8yy18iuslx+ed9LKzPPOQ8cl2JrrjQAAADAAADAnTFsu53XQ2mHYAAAqwv/wWSuLYB\n",
"NLykuinx4Xqj2A1HwP1Ur2xSYht5xG7EpLfR6V+a9Hh/D0gz4CnKQ5uZGKwMgD5yB0IV/0IF877X\n",
"rI9d2PFE/yUfFzDndS8/TY8dJNMmEQ0frii0ew9VKM1SYSQvEdnFi+IdySUJrV18KxgxM2OBWBsh\n",
"O9VSCZi/XOjSHuqrIkgUDXikUnYvHNWNO/DRMiBMelaB9G0tFJfSZB6T8sozsmt4qyOth6IzjNPg\n",
"bw8ZK2ukNqHqjO1Dx/yp/tl+8AaAs30ZX/w/7lZOq/iMyjIAAvwPj0zWqk11FQLPWMo3eQCPdYo2\n",
"JhCzeXTq3+LwW0nmfTSwuLuvRguSt25fqEn7nHCqnDUXI+leikvU7kFgTx+tZSxTP7e0kDyppCC7\n",
"gsJ8ZT7WWo9Nvt9bpHhr1GqGltAQ4eocNw6a7l3MwzFy2aZDWOHThxh7zHxzEW3kXTeBsV25PUvL\n",
"Fw8xJ80cGPQZf5lzbwtdw09ZL/vtnxNcj+Nghr7ckPKnnoERpnXz85LzoKlT07lsRMkAZA8CjKHy\n",
"tUZNgIx73sMXoxtQISpry7zkbb84CFlNPXyo35lOpFXJacfkp0qj+QBHQO3p1Kgja+42F5AomaaE\n",
"zVEZtPU+jUN9tFaL8tG7LhrAICr4BglztStyMdwhp2ojnebfpteYAcI/C215xH0ck253sLJnFgJY\n",
"6qYwogKdSDwwKPOq4eQ0F7gZzf1RJWZX7HocqNq4bTgpSMsm3X9SMh/gbhVhZQx+8PBbRjz8nZgr\n",
"59lhxZn7i8LJQAFmRPdk5zIYgIVuwhlITFsBj8oPctfGfkFdUdaI6tITHbc481igR3D8nPfpD++f\n",
"QEpAmm1I1wMWrPkRXaCalxECxF3uUsl+OQ1Llb5WQrTCvZT2PfiShGGRkXajvDIq9iYcVqgczpSy\n",
"RUEgLCr0cIkjREMS5+uPPk3E7hRaK72r72YKts10RKpLPqAGy0st2AwcjRxjqMbJAirBOmSjBOBB\n",
"fstbvzjsdzpeO9M2b0Xp7618kY2QWRLoTZacvBt6Cgzcy9+P9oZNZhT4/X566Tk/Pt6LfpK15KMe\n",
"piaJXshoij9mMYGZ9ZQ5jsaCnFz6Ups42lgkbItAM3KxXKAY34lIrHLxbzxfJGRfxxOCz+3WO6ej\n",
"R199nKYP/gYGi0B3QVJJyFLzh6sYhvhZh1xkykgeo6ZMXsJy12lC6HMRSUrwa9pQsipffvjyLM+9\n",
"KHQ1+cfRUWGYgAmdde/2HsPwlnmAIBBBUjZ2KBVRtDYVkUkIMWyL7DpWJORfgiBQlfQG4MTsex/V\n",
"nVqS0tegMzrpaxFxNh7RF+OH27CfYPnLQWbDWDM/hxtnEG9KcM2Rz4eG7rvXQU9blBoTyiRghKQr\n",
"gfe5pDuar/xLYoqA3yUTmfZORjEBnS7NmYGrUJvKC2UpLvdzCfLB39X6epvk+lMiW/k6jGIFcYdJ\n",
"jrEj4IB1QU6zSw1KNxTmj3cQfXx2q4ckLGAmm3bAIFl2Es1Brmp4poh12nS6Lvn4QuWQihU8jOkt\n",
"27HjePqthAOtu25EvD0kDkUltXZmyuLM1+MJrx5P6ZyK3+/BWSu1/oPvqAmeOhAAecQil2FOWl4P\n",
"q6d4gXiC0+ZbIjb+bFtHklzImEyArgRHLOkr6aaHsTlIRMzMxokuo/XZ9ceqIaHf0LYVeF6acGPm\n",
"wnbM5YatzRni+zGql9RVNUgsjunONK+HL68Hb0PqPDznb4+xv9f6waEqzqT9Vh11usR/37wFBybu\n",
"R2WL5vy1znCxs12JxxD1F4QD0eRCa6Ya3vXHV9xJfgS8a0bYXgtUJ1Ov5uLXymuRYjp0iO7dO1MV\n",
"lzghyxKCsbN03EV5bwv2VwCYv9QoTOEJmsV5fUPiGzHofH7F8F8geH01+7W7Mw4JOxFt9gCdZMVg\n",
"VniCmI8fpNGlBJbpx3dhtIiT3kW7bI1lePeuXYM30R3twKKTL/Pg/njQh8zs1ih4VEceZ5ljTLVm\n",
"uICuYupfa5Me7G/KLuD/m+nJ7G2iK1zAAAHgf7nupVjxpcmV26EtY3b+YG4GA4iIjFD87WkIPPEd\n",
"4/Xi8lUpT7ffhAMBkbpQjbEpKhyeQmzYowNZAuFx0zD3vMKijvcQnQ5gUzS52AgGSl7vHHQmQYz3\n",
"SMY2gsLzfQwwWMWhiZuOoC8PKhfkWnzrbBd2byqUfpuZ9FVTmSdEZGlDH/sDQzpKrYrhJ2KewVcn\n",
"JqCxU4jN+TdXZ1HuJFYaax/L/oApeAF2Un/yFiJ7ghCw9J9/kBmvlr6xqGX1SH/gy7O8QDb5SH9f\n",
"FlQIHrvXdzRrlooTykuG2PN5FtQtS5Fcv7ZG988KTfGRviWyjN/6Tnnk3bGd3Hw2XCH5dHBQI27g\n",
"GBZgxGs/EAi9Bnjdi+9gr5g3MhUbr1OGam1Fn3S5HXO9zMbf2NRoo+gcDAlw12xXV45iA01xnKQX\n",
"SkJmpzp1RManhoiOZrbnCVPCZah5L/ZP61n4G305W51WMnLISh1sNJ04OMdBm/xjA+yx+RpHKa7C\n",
"Ufo6zYOm0NqqnpnXLLfbQ52e+L7fC4Ky3veQnScFLaN8TZ0mYI3y/yLhU379O9zvLb7aIZ77O3Ln\n",
"ZXrEsiVsEwVztnZksBTa/R8GOwsIGZhXW7OtKHqbAustN606BhiVMZ/nZggwGGaTh38Koeu7Medz\n",
"pE+UWJdOor4AXiLV3tYpbBWDyiQL/f+yxS/8grBx/Yco8sz8MKFQolpQURQdTk32NYihzfksUW+1\n",
"4DTByUEFg1H4NSiHW/J0xYYmhM6iADjfmdg86gOh0IqnCWANILSpZ1HSqCYWnSauOD9ju127GQxu\n",
"T2P91jspfF+J520itf76M+befPov141fbTiCV2bG/giJUcjIM7P+wLepoi7gVxTInH3wdn2PEMri\n",
"yr4GABzd0n5lsOsilYfa9m3RXlTLmbBgOgjBs1+8NnTi1UpgRJt3qQkAEDF7KUH/++rr65R7C6GX\n",
"2PUI48nmqfSdyUmJqIytsTKYU4Zm9/xamgV5+oiestZ97O+A6fVNT49I8VcXqFF2Mnq+cTPm9ABb\n",
"1HAf2W/pG8K0e+sZ+c2JaVjvj5IPlxITZ/oRKgDb+1Z5cUybOjRRxauCQPjNMgtLfUVKqn4raFBY\n",
"nxT5PfZsp19HLkNOZAMvIE42yl3p2/3yKUDGfvz1sUh86a/tqjmvHLc+uTABqVFCuZm+UcRCR+wN\n",
"r8kThyIxU7IReNvX1XnO89Qsx8y829yDAIOygy36uFErW4fDSMkCLDvl/fYoFBlEI+Z0yiwwzzph\n",
"ZseolO587TMVt1WGKk9TUvol4x/pUgREIQ/eSzUlALNndePsbZ+EUkmln2xRlb6Sk+zbE2GV3v/I\n",
"CNkfH2f/+jqqcOUH+k8+K0aLN3UULBYf/vv+hFSluKLOkYLTtb+zhrCcT/kSfuthDfrkdFLVEALh\n",
"Iz5jOa8zihkPYt2bfqGT6PjkvM7/1gS/lJ6mWulMF33w7StfqHP9RSoafSb8wtvOaglbw21HOUhG\n",
"sIA4QHtJTXJH79idDQHiCjpFojzx63mrJvIltv4qLzhpyoUNU5LRdtKoinQcAwsa2fwgaU73ohaC\n",
"WImxJ7gR9QJVY0MK4fQFQIQEmCZBB/557Gchsnv2i+NMRJfnnY9FIcDcv8dqSqO/ohNAEjaA91sc\n",
"6uVQ2oZFyePU1nks74Q6qmMBmEn9OPk2ZPKWo+lO7y225F0C9FuM3huwM4R7DB38aM9DZ4FaOqK8\n",
"vF8YI53r9R6R5Sffb5a6Q0KaEYwVBOvErE3Ady8ew8LAXId/cyWDDwrPKw4/SdX6zpBBSRmpiOMU\n",
"XJ0uGcN/m6Hbd344eSAWNngpEHUMyQ/dRaAz6d2Hpesofvt0QQVCZPjo86G4Pue/q7jvpvagN7jx\n",
"4C0M5uomydsVNc7I11yIocz/4eoJHhI0YjOXK3tmrxfWqnPQLbLP5cDiHA7Ut3TBxEwD8ylBodwk\n",
"jAM+KLaKk68mXplKkOXGHbKo1eeSgqTCcNqf4UKnAfOPI25TK859nhqCuCpin3VUE4g5OFYfalZS\n",
"FmDrEt+5jmO6h8BKCu38MfGbnom61A5yqt+QsNPM1kX7qqd7dt0DFT0vXoFjP+IzxdKQWzZPR+NH\n",
"wToMIQaUhCycVY/5nHccJ634LPfb89LNb8FZ8IkvOejF6GiZbP+WzPLIJ57S2uxwRtPLAY+n7gnC\n",
"zX181OLDkisVk5OkG6REyDUb8Ouu+3BTsS14pl7tzR4jGKhZ2xwedRrYB5S9Uephn3SKUGKPBkSd\n",
"4cZgxDlcvFAnhqCNdgZl1B7jcqXmpjyY+MzKfKFTgTLoFcAXSwo8Q+bboGfvcudq6MIYdFKoqiJ9\n",
"qa+JFM9hlokxNIcyKRNbOAaRZ0fTBx43oy6RYMKj3PSYRu8vrA6KvcLbaDl7wOQzhfNXy8xWktv7\n",
"eqnoOZwq51IkEdAhCPuGz7cbBv0fenXpy4g1NlFfKzID5NDIKr3BoI4Vk4/cA+11wWbVkQ+EGhe0\n",
"tsWh/M07IBdzN/VK2ljaCeNs9xt5/YtTJJh2PHPdv39LcywIH4nu7HOoLdRZ1Rbx9DaBS/psLkzs\n",
"+DnZZOFhJohpr88t6t8tPGD+H6fkn3cLcwQ8PGna+VFQOisjDTaWHLwUNz/iU3lOmBy4sP3DCM7i\n",
"06+c/Z6FGI00r2IvusjYW75c3c1zRHarfXSXnui2y/JzmgXSHgOm7h0qI1VKI+1uyWOIWH09cfJi\n",
"QSkH41mdNauk9DIBXGzVdRzZEfbpzfsdcg/sa2GFaZBY1nixql9j9IQzZyY+XE7qRSdplPby70Vc\n",
"85ZGb0Fn9gfFwR5+0cRuoTJW69s+PvqEKp3k56wtjpz8JnxlMPCTboVSXLelS8GgRryq+cRvI2R5\n",
"1mGivOBbOILn+T9cUcomkBT4Tpq7ktrpgGTRjzSJpgS7FCmlm+8lABQqjru87vs7/2i1ITAfD5CH\n",
"2Qeu7DnnpW3+nbjBeykl/alXSAOV0PV6hleJIHvr/AIaj7aut26p4VOn7VsrfUHnwZta7e/bHn7d\n",
"rQic3VZXZcRSPi4428dXiLov6pLF68M+WjaZe9QJkBzJyt5KxH4DBqhUOjgFPWZXv1VZydbGAoag\n",
"w3j5f7WXGcKYX8AEklPobkPQz9K9cTq4J+Qol+oDiyNcAZsGFNwrnK/Yx7SzPgh4WcyaWvRe+hpM\n",
"C7c2+DB/x3pFOQg9+fJtrbWJ5+LoxVueTbFeGfOHQu3ntNEltaXqWIiHZ5qcJs79KpMQwUGXgnuZ\n",
"xT08DxWECOa2LU2gWxRetXYYo+/+DSwfyu1FQWIVwh51t4RU/8Oj9Llj8pCy35I2uOoS/Yc36MPY\n",
"4xAsadT8VsPytGPLwc6hgRpTigin+SU2p/zRSWTJiEMV9ZAMjHFZXsfhy5/UFU7uNZJDW13JIvYA\n",
"hPf4QvULVgm4LnyNIgqncYd3HIXyO8XUw5QFDO+A5tNbStA2tFEBriGkmoudT2eCFZQSjz840FQF\n",
"yP0jXiOQPf9BfqPUryJV0Jco4x2mzrJPT8QjUOF3Q0GT/9b2MJyTfRkz/SsdWzpP+XSguS12n7ke\n",
"jwP1Q2cIPPXrh0P+4B/M0e8aaY8GoohQo/w+vITcSMBm9YwSEpTP+JVACV/r/tuFis6mADdi0tOT\n",
"qtUhhMxGQ7Lr0/ViFJep3h4V0zYEzLSpVcJU/uNy4uxcZvDYUuwoa8jeHahMh0tXezi0TsjGRB0I\n",
"dDJ3dzeuzot8uBGSGt9yXLCP/CU5Wc42CxggaDbv246RR5cZf9JSHoCV6CN6QViCeLQMUevWx16A\n",
"Z5FLB43egqGaWiHRVv00M4RPLQNx2YBISox0q6bCGOjTm7rOVnIFYQwwF9vyh6lxWeS/anPs4+qh\n",
"E+wzWYAJ0YLBemnPo7rYHm5BY9DVPmjFSeqOvGeZ9EPYeyvGunPzI7hZ26SX0nOE85/xurQN+uGj\n",
"jtmlzaP0kYblfyThXV9gUJ7zP4v2m1UhAFx0YqbJx78/yO2ZBJDCAmzu+MnXZxW74PXwfd0mIbFG\n",
"xGv3F5VBbVEfwVfvtdXTg6Z0NWgMeZNTfRT0Oqyq1wM1m/5tOmyARE3YV7fIIwwOxDgZSEdK+kcJ\n",
"+b73evqmv40fedEmUV+hVte0owg265wcRXKGiDTr4fz1pMTcttDXN3Ny8u4SQpfuK69abxz2sXp/\n",
"xjZy+f7zTfYkAgi3cu+duDjE7D9cy2GMdQbRXxeI6qBlLihlfR/PBOwUt4n5KUA8gpVHYaWnwNGF\n",
"37RTBxXraas5RERRqzPP89MtxQ2ZOAcbAmOuPZtpT2PnQFSAy0/aZVy3t9a39Ndk2K0qDYa8BYbm\n",
"VEeKwmaaKmmy+R6LRsgDAjMKCUor/wOzctQV1yjS/22oyqlW2G083c6Rz69HcVVUghWgZdBgnSX2\n",
"tpyLGWeLhrbCP17QAG7lDZsvithLQtvOCnwtYYcQpTQMMFIhEFtZBYy+IPA3VALbV2s7OPXn4wdM\n",
"Zy+ISWLeeTVK9YRYR0YNHqUgZtnOvAa9oAt5vr1ucyVrbe4LTkBBdmsLhtCTWJqMRqwPfTbs6H35\n",
"HUBmoiKW86+JFaQWeI4NZgH1kMwGFUytgMQD4i7Fvbs7xn7Y2Yvz8Gl9kcApdgDVjm9cYcgAY1Ok\n",
"f6B8JaiIBg6xzdckITcUlg9pCby24vHa6GJJSo/XFDi+w8kWq4IIEVWZtFc7gmMy5NHkUrm1ZdWB\n",
"UlsY09crp4WZvIe6kvCN9DfDM379uWUWqg1QtCUSJ8BupEK7grhZUyESTRn8KldgvEddnbWpJ3cV\n",
"HVH97hKPJYtwSQ8XdWfpA/x3pRyIwpDXGeJYxuHz0+RumHxZ35VG9bY1/57cCrfn/1ovXAnIeood\n",
"/qQADSXiL/cvhOKKnAMFzYxqL2yR/uCiSyMm8Xz6w9XeJQOggnlzZigAoV/vh6osAC7wDK3J6yqC\n",
"i7tjJ/Sb3zQ6rUnOHMIDcuMKi6s+/xc7a8/n90HZmSpvQqSyafK9cQmfiGUOK/pIEMnL6SJ0fvNG\n",
"3n2+531ViFMIrWqFPi/s+1HQoK1RWO1UwqaKBEcAvWP+6rTtjg39xo3NLz7eRD3K2ZBR8Bma98X9\n",
"N4EKj494kLfUZJYRzvtDAAB7IM8KX3NEPOTOnBubL7MtejqUDbIXrN0pRBrrUCUWr3k3Ve2C789E\n",
"OhkovM9tUAZBe8tGgrsV/mXNfXeAeXHe0VPOIuroIQ5jXaKpsDY4V4T436ekSuiSuavVbSM+JC/j\n",
"feM+4rj1GtZnei9V4dxAU6n6nSJLeUxLzj6hh5RGsLveD1ZBLJAVGR754E14ykLnhuX3/8e1X7a5\n",
"mekFIqkyYiyozUgnhG3h/+1ojQfKZufPjH1one6gubALEaq+N88n5qX3A5jA12ks3PnDomFFii9x\n",
"+mkOL9n0IAJzKJmHYSQlgF/tfIgnpJmzpot2Ca7I9YrUEE2tp3Wd6A0J/SsUKXbAT+1wgqe8YlXY\n",
"hiC3cWfypqZv//9aiDs9qrmyneOoAuauHUnV8+fGQsLgOYZCNqj6HZC5SMTA/E2TMHZ/bCU8CDsi\n",
"aJz9XBwWTf8mnJkcujVnFUzEgKTK9lptXdXF2JssrdYFnAOiZpo1So3+lhCYf1G3msupbvhw4Eek\n",
"1O/tOiD9KmeefS2vCjEIc/maetV+SeMB9pw8fDdn1E6VkEq38gxFijpiXfAJ9Ykj1PKv7Vz0KJuS\n",
"l62yvJuG7XOVrx9usCnVmMP23dmT0QUELA6d1ezBziWiCVIuVTq9Vd/P5jWEcg7s7Puf0WI+bNRG\n",
"fHf9RJy6a7o3GYQ6AoWd6sXNr82Hrqu4z2t2ocXsmwqi0pl2H3JBNWqCJ3Whm8UXZsfl+8QBJD06\n",
"4lWT8OcA+AlTIZlOfC6ckLQcN0Gz/OqLTdOm2UC8WsMzgcAPvk4eR4Lqf01qHJV9hrYxQl9+lfUV\n",
"SJWyKvMc0Bev6XNUP+9CPjoQxuTpS7d6MwT5IEAghW/LFBwZC8FKX8jK4KDMoKjKLZjTDVC+NxNV\n",
"wj6bwrqxIhBikW/wX3TBMKnrMCYMcjXzH3Mv/ncB/+gavHN3auF8SiejOdi/yHnqc90/zDLUtPN0\n",
"ipJmAB52iJzB+Gy3HcORZezaxuGWdjDF3cztmHwKBTtnGfOh54rmoSDI/8T1vZyIqqd0D0N+snF7\n",
"/DwpiXrdyDtbWMr0poIMYoH7RrfSd5wi0y89jUKuaVgLjHqC9swspSU6FvWJPpp2lo9cCkzv21ri\n",
"xQSElXJqbpkKODGl2EFw5ILGZDdu0NjwB4Yc2biaxdpdLWaOa6+6Q2GVg/Ae/B26be62xuf37yJ+\n",
"C3EYFhnPGmia9KTi5I+Z5Ek4mxYvWQqUhSpiAvt6wWqo5EPHtin/apoQzj7UE3hPYVrzalRUBYzL\n",
"XlCuG4aLOdyPQ3BAZov5G4TmeLQIl80HQsg+porxbY4c59hj1ZdU3YyUwJNWRN6x53IOQh94+QeC\n",
"3g/FAEIrbp8yAtB2RpIlnNeg5/wPJoxsdpa5B5vTuv6V6xC2My11Ve88iG0K+71pKk+lTrunQSPm\n",
"4IXPG6oFT1gwmBVU2QmzC0yTkFqFmu6BLFmkYb9ajGf6YKhntJmYZoqHdwXEdCYSKXYAC+6ot0dl\n",
"DyYFkopU1zkxTqmLo0Dd7c++/HO9TAFaXvku15Tgs4/5v5EppECMw7BMS3avgu834ilB/h8oUAVy\n",
"LtjHTcvsQszKqZurAdwC02XsjZK13HJUORLpU8qWgF1EsELvDUMCbGjZ7DP84+Kra0shzOOuMQ+o\n",
"6bGViHoEK1jIm82mBsvW25D+iI5640aaZAh5QYak+dv02mWLvH1iCvEWxPWhjbJ43ehZTUPXGLqb\n",
"4ze8nLEfRS/ag9ITnt/rf469cjWNtsPOUhhGnqME0RcHpWQlaqyeh/xge0Po41fnwBbBCdu4ltrk\n",
"ajIX0lz8BTCMmpJK+Ymv8HEzehvX8gAVdFAtbs9UT+x2yLKvFGBZnN+LeH9shSUKKSYldImyBXdq\n",
"jVnlLyanxaWP1fXZxNlCWPcyR3eoZNBya45tkf4cKnPu1qUy/8IvhVMMpQe2dqo3PgNrQPn8vbi1\n",
"ng95qRuih9js3IbfveBj80FSg6FaLVjrfReIuPf+yT37reBYpAYZV3V1V+c3izNW+p5teGjsjlkW\n",
"AA9gAZcAAAS6QZohbEEP/qpVAApRY1pAA36AxfdIXzv/UzuMGIpMbcITKyYaI5FLbE8gzsTORbX8\n",
"Wuv0MluT0Q+xbOOiUljqPOf4HMdUfwTzhu64TPJNKojCyJFbnYNh603bvxNSMgl8SYGpOzm7KHJq\n",
"DGXsoGWVA8ilMjzt8KN0bhU2wrmPz0pIXYyz6k/osF9wwVQwQyRiiUjmnsYLwCFFx5dJiLStSmrG\n",
"j44DW8xJyvj2ayeIl9BPZckP/gGqtMTa4PzPiWUFR8N4c4W5t/qHqmR2GINwdg9oqfkteW7wN1gC\n",
"GxVsZxkEfeDYHbh2JsO72pdBEhTHLFNe3bzpXE7XP4uvxeigWQFoA5kd0/Wmgh6uymd+j0pI75l3\n",
"Xjb5x6i5u/E1673gj2meia26H53jLtcpM+z8j0kUWZJLbe0ZgmUv5/QeziATHb6BJIGD9+35xGqi\n",
"clu2wbrQvevQY53IUyfBRh9cvGL8EzxqZJysjCZPN91XOHHtKFvxYrSwTqSJKOzmh/cI3tzu2Mt2\n",
"urWGD2oxDgQzXTQA7WDGL8zTHYzogDkaLOHc238SyGTWtisfhon8GBqfzFN51779H5IUvdWg27Rv\n",
"RxoEl2yhD3p+jZl5Nv3W4j8oK5YD1CBZEEIuGfgyayDxuHlxwxVjGzYUltqqGe3+eU1wOnCUAGq0\n",
"GfxAL+6jLlIk8hK8jkH/eCawEzFF/TGB8dXkSdQRvFz9q6cWn6r1rHMVgNql2FxA6mVT1fyREaC+\n",
"6Eyx6On1Z1asrOWH9/wqfGvPisO8gx3++fIkHS3iopT57Xxh6LIcnQ9RbvQiJy1AWwb4i7Q+ACPH\n",
"cPTIoNqX/yiVXfIOK07q9hKHB/EfUpM/2qrVBZhVi4LxOegqDxeIoxaZb/j+d4k8+5z5AH0oThpZ\n",
"e7bWnMdvvFMsMc+HwWWm/EEPUL6vzvhRRHbEcgo8yfFoVnH7AMIXCwjFJcpluVmRWfuQ6NIPgoeq\n",
"4IYPaN1h369z/e89FDrlYCYJaZpubzEb23IWuudH77tZG43XExIiaBywt32Mk2E/iwc8G0KdNG/B\n",
"zeBDbPWOF/0lTEhoFkYPlJsy+tIkhgD2DDblneBjBxutqD7gxfaWLcegzSNWRs6xRdpIL0mBH1S+\n",
"4Axb8GUbqaI2EXcONXs3IYSUvANPIYNTjde3O1eHk4OqkB5xQSVt/uKTAQAAXNEgLc9SDlgmyjWo\n",
"bPV9127gOjewIHGWx+kfTformAOGvensUBDY9xj8RJ/maidh2UfEAWdhfFkJgK0oqigvEGsH1PYM\n",
"ro5q37G3IzSF2lL6vz8fjTdQPVFAxesJuKBOyvXe3Ui60COEahdiewtLPq+wlPEdivJA6lAvSZpR\n",
"+oDy1ysEmM+MeIgGD5pmWWtKMrD+NIDYlFUb76R0O/ru7W0sNhMBTR6KIwBLWaOkxB8pbyAO4OIr\n",
"u3zoKGZfM+A7UG12QrRkXAuDQj/nxhdy0bWkRgCOpWpFM8Zkz36Dj2wAKalimw/f3di7KrfM6Sed\n",
"j5rEnSWlMgn2phflCgZsgAEK1hAklZAAiVoALDy/W8QAqO33ZiNJWg8fVRjVqjpdw0sd+K9R0Cwn\n",
"PUXSFOCRsOVxT5fHiM+vIgca9CmoEAAABYlBmkU8IZMphD///qmWAAkB7bQAjE1dtlX0NIk+Q9cE\n",
"F4zJLzoEJf9+ro5E2gvK6eHCWFER9FRc5FRyLW9n88I2Upl0WrJyLLgsGAuq1x8Ug51LWD/gzxHc\n",
"iiTF3huaZ+27y8ueQBqZf22DlqqqrVEhre+hn2cEzWOAj6muhUCgLxc5HJ6BhnDkZUNFamnEVpfW\n",
"8l0481dJiB2czu7ZGRaFFw4zy6ZYgBixT39L9LqDVeSXgd9YJf4DWP7Xy6tKkzaYJboFY8OI0nMH\n",
"mHjnvy+mty4durNVU3gObm0DVRmDyW7YiA3m3w4C0A8urWPJynlo8u/jT7XM1oXk3D3yE41qdjLv\n",
"kRmQO3GjlDPZ12x+R9zohrullZIx7YT7ZHlYInaQOKKfPo2bQzWi9vj0Vo70SdOdkiR7iR+Mt/GV\n",
"8oQuBiBxJ11ciNL4Xa2flQy//l750vj/cjOZ7a6TpLcFgroa0hvU3hZb+uDEP/V7meDkMVXwEER9\n",
"JM6cPtVPa7NZ0tIZ+b75MqX4/tVbynAMJEYmZE3r7vlslVWBsjDvbLoE4I+SIzTc2OQTBsAn9k6C\n",
"Dfret4HhJVUyIfNzHukJab2UX9u+DFZqXG2IdczBRVyO/844m2J7+WASts38cZwx0vT6lelVzPYS\n",
"gYqF3J+eFHkFAp3GdgA5SbeJROnQf8Bm9xvzJ8/kYQm1Hn3qtzbGMpEJvvwnm1ruwzRwZyT8FtqJ\n",
"DukUMgVc8lEq3uweETt3vUqJpulVjIzjP5rfFsVUVVQuGdkYwJYX3oQ3EddEvSQxEjl2jdjiieIY\n",
"6fNGRzwIAeQVroMgHIUD6QWx0gNUXZKLiWw5a0P3tIAyU2aUpLd/wA18S3EdzTwbCs609f/pD19i\n",
"n4W5iTeaFaq96opa/0imgZb+aA/ursLqknI9ixGEdwC98bZ/YvUquJP04QqaU+cH3mbso6Zkgn0W\n",
"wm9Abm+xRF4JXMshRB6ChOYKZMyMAKI0s4x3z25WHVt2Xb8oRZ6kZgodO4oE10FRca56ly40z9ir\n",
"xE/VULR6Phab0979ifIZfuh1An2o84T6m1+0xtuoFVf/XNBKz+rmbvN0AKEo0GWeq9Na7nPMnxCw\n",
"5EBIOKEG3FRDSLawfuxfgZYiKtKpO5FIjI0xnI+HIvmbMSzvlDGOBgG1m303RNpqR3bmYHmby1qP\n",
"vjD9TGu4jN6iRFqDnBWD3Yta2Xod9B8mK1L64VKIbMHQ4jbGUiJ/XAyeBTnSV/IasKPAzduUvvkM\n",
"TXbhv2btPLrT+lFXG0pvvJCE3f35r8x7wOrLLD99QY4iNEoY4oeK+WGnX84YdGEq6jXVpfmyri2m\n",
"A/gSLttOlOmZbrP8iclP3qC8JggPrpXca1TawwRf9MQXk+TSfR359WVGeUjBZ23xEitqT3+kUhyO\n",
"L3HowxzP7j/Ry72IFkDaEXMR3K0Xi2yx76U3sU4XPu1lYK/85a/E2XBDtuEmLH7oA2sH402HiXWT\n",
"vE3gq6HGj5Myev5qzI3KwgZkroXgFQmMAFuxD6ucUUUjDK+6sva0MDKUQbLBWBbQ1eQzWa3JeOuO\n",
"rNgkluBkCpx/n48hki617nT2PIQby+pSZ9N9M92s2Vnx2RM9goyUEJmHDZD0WwDbUcfniGB3WMkV\n",
"zuOSBMVIxdP49K7PjdglkHyxgjZsK+wkqNnufuwX81phA94g0kVbRVyCj0gbXM1t08hzypXPSVdw\n",
"9NA76S/yAvNC1bjPEBzrU+isS7zjcyKZYekmbuyfJ0c//5+De4Zze7oCVBph14GL5Z1QMwfylwod\n",
"e1BrzkoiNn6GzHP9y2Qed/SE68zGm5kz4Z5kiKTmcFBNK7ge2JGpCSX4fKs4cm/h7249ZckF3ael\n",
"nPyAKPGc5gElpglviYYAABixAAAEHkGeY2pTw78ACe4wby9UOggSSpcmQAOzQEVoA5O49oFHnhXh\n",
"RjlAYB2VeLiLL4ZwOcPw8T7PB1GazFPlZrNZy+mXEzipMuByr9Bta6qAJdFYJUSPNxV261ltdYZe\n",
"R8dPTz1F70n73xi/f3cuxW6UJ+h9U+ZZGl7+a4ZVmYOj0Uhm06J6aQzNB3G2GoJ/y7+8QQYn48kQ\n",
"4ur+TbP83kyLXGZ4XdXb1AqVFzCtMV5pTeIPpbWS6fzZ+j/FNiYs1WhvTiZSkV3zLlgPE803tRiJ\n",
"H+KUkYdY8j6m3EjNII9Be+H+QgJA8EAxBS9j17+N/PaH0X83R07AVnps/rswJw2eotTDYkOXcUbs\n",
"cMKNwzhc41iyOjH2riYabdnmBqa+NP0Czn7z6sigDYkK4y1mmPaJSkX0vmnvSz3gv/J2Damhj42D\n",
"y2Aw121ibU5ncxvaCyEhrINF5f20Db/y1T9WrL1lcH+kFneIJdBC8A/tuMwSMjFuq0Xn/B4+FL9M\n",
"1dIn71XjlQ+1bc/TV4R2M6OesSKNW3Y1nTbWiQOUHmGAdbWsvg0Iuybq+4hvHsVSlw20rKYVFr/Q\n",
"jJEvlu90xpcokduAR0sycz2qJnh1lGnM9f9Gm5x9+DO56bNdq2FNgHU6YLLxRSQzvKqhWhadjLWH\n",
"W2luwWib37keBlgnKOTLIDmgv4s02f7YQohtQ02Ahr2WpVR4pC+NO80X6M6/g0ITZqgZl+uXodcw\n",
"npGMpVUE2fzFsmdxl5mkKbk7aHLD5WvtDSpoD4mjzKVL1JaQ5OAPwNZboMvOvz0OCk9db2YU6Xhj\n",
"xpNoTG0FEVxLBRaojgqMJq9YObRt9QxdhLp9MCPWYk8IW8VRuvXcxQ+aisy0VdimpoGdeY9CW9N7\n",
"qEdrLmE4jdO7lXZ+aeFH6fcUkY7fcFy1Dbj46pX8g2/cR1iSvjAMCc8PfqnFP0r4p/o3eUb8fErM\n",
"Jh6uKIbZ7IAe0IwZzOTVkyn8cOCy+qqUNqLbZQZ78oXSkht72Nnu/+V3kvuRPVrmXIO1s5xTlTXm\n",
"aQcC65tfIhTVpLuMMqrz2ySwqJ7IPuz39hXHz+uErKsZ+ywBYUzQW9ccnXVbyjeR+0KTT8BahhtN\n",
"XEfVCZi0vBIm2Tgw4PTEr/nKcKuOJ/d6bhdESG7ifsqorp3wAVWQjDgABBT8YjkVUYNYoNgPChGA\n",
"L9zjg0sHMlM5m6O8gyWa6t6EUmII5t4yKE32BY9v5vLHEf1xuZSD33uxwZA2i3sDbicJBNbzVi4/\n",
"ekM/h+gG+QA0zHJ2C3SB/kSo7xWzsu7ygEN/nbxaszOYK/tVD6H+sQ/SSopVaVTYyYLrwoF8ySbe\n",
"7bZ/qO05RnxxsLFhYf9S56ING9aTkdAckdKSa+CxQFZhzvSfUTaOOG0+ltvAAWkvu4AAAAR2AZ6C\n",
"dEN/AA4mnxx//SaIxs2kpUy4PRsAG1RLvheQc55FgCKUm3O6TmH+OdOoMeA5dXJkPI/0R4l+PJIO\n",
"rBMAKOOu/J+Nie32u//ol47/tob1oQ4TAAdqWf3Uj2c+OZQsVfvLuyL9WN13fSVXrZ6OttdqFA3A\n",
"wPZW1xx/1PXBclw6/gka+0UXhTItbFgZADbWjtYygn5SrPkv1GfYPRnUl2btmHMryzjMYV2EbfN0\n",
"6y5Fe7RlR6qGnEsEd/95SYWr2/TKlMRmFUiDA48c+Szhw1tTwTTT+0ZbYksXC6bKmZWQsFttVNEZ\n",
"v8bv2E21pCswqbOEs8ZYayRUTTWZEfWPI+/uq7NdTycvJA1czdSSp4uc1E8j0Kbo76gx8ARqLZH/\n",
"eaOn6K0wkmKz06E/lsnLjJePQwyg+dvmRIfX34c38pLsmtyVSkOzr3DD4i0WPS5ZkeCJlGfQOMTE\n",
"tV4uIAadwNSHRNZDjfsewbEB+Cwy70Mdo2ByVyDSMfEu+e0pFRqi6QAucam8WTY4wS+6HtqvDvsq\n",
"lztuRQaFQSmKjhIqfBYFkhpUbIlQmvhex8TC3XYRiyQmnIO08D1B/chWW9VJTUMMEZNuUZdqzxeW\n",
"RVtvI7iya+lILe4sZ7YosRge52TqQ0q3qgL9cYJPiAAJD/n7WhfTRgmyxSgJLUqlosWGZzgDO3fH\n",
"9Dgvy8WhAmUzDxKHN/Jd+YQ9XhU9PMnwF8mH3g9YNW+vbCfHcBIknrZ8sQ2tD2L4ug1gQveyFCjx\n",
"4icCI3GQmjZ2Y1AvVYytLy/WwqWYq/r9qFJM0YKhzFYU1pi8HCKoY1sHYRTNBv8R1pJVz7bRTnXY\n",
"rprmx97QX+MRcEp1JGXmtWV9j0SizR6bdcFj3KmY8mv4JTzHYZXb9CIexQmqkGjcE9Iq4GEJQ10C\n",
"n2NDP8Bkg1XcCXHoYV/BkZqhTDgwpy6jzh/KwXwJ/Kse52sY2NKUGnVElQGZ9S88dGwLyxSAcwIn\n",
"7H3t74RoO/5FRwW4BdnC0PO6o2AUZQuh9cHU2lPRoQ0LXg+vHKWl51Xg9zGeClVPQoclCiF5/eUg\n",
"CnK+wHijEwRWiXNHL89cx0eigQkHmRnIyp5glOMWiFBwJoWYBC9LLfAXUCpgepk2O2ja0d15T7SF\n",
"T9qPsJrdHlP5hmW+Hy81UHn9/3BuoAiqFKRso7cQfAVFbZVAxcztCNnkgF1PxynyYETS7CLIotE3\n",
"efa2QVM/DbpWV7pCnKHRkoTkAR3lSH0T3YR7zEhdvNexmT7bfiWuY4jiuRR9BKtqqV4ywbtOSTou\n",
"LxuDWEEVGz0OhrxsMNV1qzv0Pv9HecGGQu41FLsJw8G1ko5ZJC4pyroCh0F/oKGAjSRJQQ/wFSas\n",
"qYyWvUbnQQU2nDtvoEPS+HLP1V5LMM3oagm2jNKE0PzSwAOU7IUf0cd/8/7Ol8ZFV79pIDdgA4IA\n",
"4O2ZnLVr+ou4IjH9Sjeo60s9bUZaDeKohKZUOAZdkeTZZC55RlqBMFaRMBuiptmje1gwoGcACbkA\n",
"AAJyAZ6EakN/AA4pRRuc/dBrCvwAbRmjm3YYOZSVC3aa9FvLWD4+7Kdl3a/sK46vsLZPR6SAotUs\n",
"XENuDk3I4VHqeeRuR7e0VwA2K1/h69gLvzUsp6DdC8fMzQJBg0NUR3G7QAQ6U6sXZwHkm0EgOl8n\n",
"mFfPsgua5N+3j677ifzwwYoTI50rDj39kz/C2Adz0WvtQKG54bIs9meENQJdnNTa9screbeVoqRs\n",
"r7/E6SUDXq1YBxhYPsl8yLH6Rw6vNK7adJjxuhK6kEsvfGrVvWcP92662uJ7tiRRem2R6X4Dp/tu\n",
"lFrgu02Fullo8zYf1P6xbDGinIlqxgHarzVA98pq1o4qmoxhzEziuEmqwUUDjq3biPQKDT7Qsqnz\n",
"SjdiQ0FkKKqA0LlqFHze0J0SMiEsAktcv/Qpm3nxKueK+8SWD6WoOedMZ75+x81HSdIGTuDC7n5B\n",
"r7dHskgZwj9EPARGG+YFEhe3rZrSGk5iUzshC++f1Hsolf4J7Ho17KALsPZyS4jh2w0z5B+AJBwL\n",
"/1LdYAx6Bgc5nhkAT96mw5f1jkjB6/nRO5UnuMKN++gEIqIVxjqiiXNl5fIn1jbDAh2X4QTKxsQx\n",
"uXqX9yWMln2/IhbfdFgvLai4gIAFmH9abOiqYmkB3bj8IYsxhYTBx6lOjSLGwter5SpJkdvtDz2M\n",
"tQQhIwxwpXI6p156lHV1Cs2vMwDV4T4aheOLZ5KjCp9txYDDVREftOyIEm2W1ga8TtzkzP4s/x7s\n",
"B96J8fsbNRSLnS4n4hvl+XRI97lGaMOgX1sAhpanFnw8TWntKEV1fRSeJkGz22nb1pkdCfUkioMA\n",
"AkcAAAP0QZqGSahBaJlMCH///qmWAAlAPJUdxVeAQj5Eryp4DfTlnvXmSa21hg7JojsPTRYG+pkU\n",
"mH02IzOQFzotcVZaLTLzTjuswBJR+HdP9IxQEqbJ4JqwRkrSaz4cuistZ7WbYhPK+3zheor7rG8d\n",
"+SHXX60425cFT2d/+HBhgajRRYFxxoE5spvj6SsB5sLcRrlCu94GtK6V3hm51yyIwkI/9O4BOTXB\n",
"d6sT8ED6G1BGuRJuA/yW8lTJuHkIteA0dTT6h7+fIYPPnY94B+rmEYqrbdj2BZtmitxQs7kUoMqL\n",
"C6MeaW0wkU2gIqJpylNJsW84d/c3EBb5jfyGMlQM34OnJxuPhU79t+u49F9Lj72fJelk4hXcAEj0\n",
"nb/fB5alap/ZQ6f/ZFpvaLJUHgU5ralowD9ibr5CmhI1PfbQIR90mspRVdP01FYMJitXhgN+DzLp\n",
"0w2uGYxUz39MBWIGFO02bd/qgJSu1JcllBDlH6VNs8W7VG+KKnOU8e4f73xkoTIkljlzp4qGtCQq\n",
"T6+kWKfggg17JoLLi9JJivfrWUWhEGdAlsoTqwRCMNKMIgQNHn73NvrHiCcXoDRO+kvzNVTZm3Gx\n",
"ip6kTkg+4ceyLN5pikfxofejs50Ni3vojoLCq4go3FLOtvymdd0ZwoawEg9tAIn0nwFcfEGj3taZ\n",
"hZRAzwv1J6SIjVGWp1fCOfJuZnbCiii8iyXjRpDoDCzsTina9ycBOMLcm95t9OyohUO/sUiXt2HO\n",
"fS/4UTj129GZeukRLL6MMqBB6+KXk2UNksEcBuXrE76sO6a3dfXM49mDQyTXNL02iDEHxP/F26Tx\n",
"UI3piZkxcKDvzT2quMj1IWRexllCgKublUsZ2u6yqBUMVoVJklPqtvBM2qKr5PqyxZTqe88TKyJr\n",
"+etGyGiHM0pIyh5gNdU58C+O55jpT5xdV0ng4aT75yDMgeOhGIGzsqJloTqkVgWd64TuhvRgzEOH\n",
"dvXGNc/mCaEKiLMu1yVXprVxLvxwyodwTDDRtnCZBlYGD8hDw5pKBn699mfQNqymRJ7krs11GTsy\n",
"DxJpqUbYfuRKIl/0tdjq9yrnNUksAdVQSiUXDQzKig1KSBYQoL5RtJuK21MacPPIy1MqUpZauyk4\n",
"vCf9sQUcFKPgoO8E8d7MMcPpp1/+9VSnehqSsR998mF2ZK7xcVd7mjiNSg3i9z+nUWHgpXTXirPX\n",
"4nMdsbD2UzCaWvF4TEIRI56kw6jdwhcIRrxhEYnBj69EYB/XYDDp3NTuLoxu+6KqUg1EmO3f1WpO\n",
"fX2vq+0N2SDl9u1WVs9GUrPYlXlWM6FVKncMvY4JKDXk9y2DyzhhdzzDwTLOsAAZUQAAA5hBmqdJ\n",
"4QpSZTAgh//+qlUABMJu2AACOFlE2kCJdS94SW3I+D9nzdjjZ1TKpoHcVwdF1sUAvmQsJ0yIDy5r\n",
"HJLMVQGCznRXhtSp8rqFKsH1NEMeM36fiJZR/ph8e5qGs/ybJYbaydVwZ7Ewc1xUcS9an7Mp+tqF\n",
"PPgkN8NYX2OoaHFXDNEzmSNqc/jtKEcbeL8EZkqaxMlzPscW+9gS69oz0D4nLhnoFM1hzFOOm16g\n",
"eEMcRHFdAWCmqeiMD1x8MxSvk0EHCe/n3ya07it054dYSA1fOf9meQTAWqAIWY5xQOsDSvDjMmaw\n",
"+UcrrNgFGtLtYZvvSWASsqkw0C8nC2gpWucTjZa2wnkgLmHh/LN4sEsVq4athFNVHIG454xo6Rzv\n",
"IUGQButg0g1F6vXGaLyi3zENFwa3buz3iCY+Do0K4ZTxV8TOrJnShMy2wiZftqROxvpvbPN2Obz7\n",
"gLXX73SAG+MrDM4h7At/TRrBA+viVhiCfk8ituLLWCbARLBvH+1ofpQOI+KnpkLqvUSdRwRgmL9n\n",
"Tm1cPDRAb3Uxt3m1gHGY9qsk4iMBC3u9AzC50OTdYYInBEXqZ3yQ0ssPh6TvQPKN8SIi6KuXNkSg\n",
"qWJpSwT8UsPIsumVLsCvtNakgUNt8XY+TzWOl38XAw9NyEscH3Q8Njkgqu3UdlFx9wi4GD189llP\n",
"uox7dz4O78989mgift4AHxEIxwFALaQF0pWM7ycOma0Njzsg6Qdz2F9D9xUwNQDt/wqdrglh9zlZ\n",
"3+QXGhQCgoK6WrpjBez6hP5lnO2UTMkfdTMI0guF8zYuoy/pr6qKCvZ8gG34yL2j20/sBp0/3RoP\n",
"GI+qhnkPpAn2eIYOCOqaxl1Nca6h5kGKaHxZvFOv/z7q2hkWpD4dTy3MHOxqoikjw+bJO/pQLt0U\n",
"C9yPkGUEtB31H6/vt4zoYsVjVWkS5n5whOAB9uulXVfyEzz4RwttOEfQUymT1HJse1AapsK7i5DX\n",
"vFQXfY4tMZctKV30PjlcEeX5mO8O62p5aeEgH4OdrRLunXC62vQBW46ZIyCjlCcFy0pNTm1H9eSA\n",
"0zRDdJ69ACuw2LSDZC9xza/6FudpA9CMe1LoSrcf+jyvcDRmf7aCa8K+fVwhiQWF+CRDq8Ob2qsY\n",
"CzheT8Dh8UhW/9Y0QewQRwElP3oT3SJKny99pDeSJSdD+oAPQL6+/kshWZHaS0ScUNIVgPedLuNA\n",
"gACzgQAAAz1BmshJ4Q6JlMCCH/6qVQAEwm7H0ALBTwVoT48/r9SEkUQqsZQ2KSTf+f7z1JfezixI\n",
"q+imLiY3pdQ1TX97ZTv5LZGKJ9GhY/7n2NtOxCdRmCnz77MTGEx94erk6ktUVRPtvzDLAlY17WcE\n",
"+a9D/GlVdllU6Yy0hTKfTY0T5H7gSmMqPNPGDxiZ1S2YIs6Nt7z2IZCwGs1jhZiBBolkXc9HR4Rh\n",
"4Oz2m0ycebGxlGhFZ3D9dpqGTO0/d4eSbZAFONyNLwVFitwvAuwiDlK62PwVlpKV6VWUCuA5QsYo\n",
"B5b1URcFONiHxp6qUUJ+QDO4fIZ26wuXM5sDtNLG8uawHKhaStNzsrhkpY7/OewVar8EiEHYI6nz\n",
"HSEaOS/5hsfQMyVqxXxU2xZP3BvtRV2B0FOR/aUIDlnzCBltao2AXFJNjbilEU5xZmpslMdnJDxq\n",
"hnw/5TYCjhTivmx8t8ACRDq/7von1PZ1WCEVdqMr3LkTEuam8Gaf6BvAxEWkdLNTFxIZ6D/LRGUD\n",
"gl0fZ8TfegIhEsEiloTeG/coXepzL9hPnpSPg1rOi+KymYsfGMqWXNvsBKnZBKPsa9ePUP8i4OC0\n",
"E1idvMDd0b7NYFvserQs1HTEiXQcMic7v5g77kgGqWV0nupwSmZwfbf5RqNs3h6GdEcRkApGz2CW\n",
"aKgQQSyhkP3a1h7g+XFS0h3HjyEBFiVDpRBQpiMYXG51qGHsF3Fl/xlvj0XCV+5QB3X3GNPiBqEd\n",
"rKR10dD21vnu1AAB838kiLuFRDH+p/pwQRHRq3Sroa7vH5xxz+jwMmqtWKoBgz/PnZaiGRWQAC5O\n",
"enHsUv3qFWYFKbDzXB/gCEubFVkvfTUEMIye4pgOv3krWqPjVvUbM78sH/dv6H7iXBzH+tjkjEfM\n",
"6gpWx3Lam7JwdOKdhsoNCB+VvoDXFroyOZHEBqPVah9/TSPUcdGFLmEP7ygyDFC848RaRUrLW5R0\n",
"AbRXcBzrsQd9UHWiBDwgVuhJT/zUfLODcGGwuhi+Po8Xqa2TNA88xL/+xamvCnmAomAta5aW9XKc\n",
"p4KGwNdfo7mTYPAVFBj3fHQEIQ5kyVgts4TSNEffghhNpFv8AAsoAAABg0Ga6UnhDyZTAgj//rUq\n",
"gAJh/k1AA3PQ5J5u/xQloNo6410vvJ5Be0H91Dj8ebUlYK1Cy0RjVlLjUnHAv1xUDvtzgee5taJQ\n",
"TSqKhMiTZIT5YjVHtw1krWxMWapqEgaqtyfgN10Sj6zVc9AZsWY4cLsQ8vBmE1d9In0Qew866Dpi\n",
"z7CZ8//AFU7uwi4OWCyLQ1pJjj3iq6IW4ocnqI9C5uzTTUfsOgMqd0Eiqn2fcR5wJc/2f8Fy4pEZ\n",
"7Nngn85hu5Sc4QpkiD+QRaZMU6CN+64EhfekrD92sRM5kHrai/b01xCP5+7MSbUJgGGdf9FJb05W\n",
"7uljlvvX9TRxGpY0pwP8VI27ljPr0lBz/Y0J3A5GYLzwOUTk4pObW30ms5XAAiQudPk8qEA9dBdy\n",
"ohVyDOr5xt3Lv/JMSI/48/NJxynIMgV3NEZ9+feYwE3UryguKQQsT5TWHJKm6LX084eJ7g2w2e/G\n",
"1/odg6QzAkp0uJ2nEcn5Gpc2e+GE0vFZ35ruSACtgAAAAWdBmw1J4Q8mUwIIf/6qVQAEwm7YAAFr\n",
"OrbQCp9PIST5Q3AVFeDXdn5tMaztCI01i04cE2Ozkr+FzLJf/hxUeq6KeidDjDJHPHHBwvGHXMDp\n",
"4PRh/Nje3Gj6I5UkJh1l1ETS6b31wjQnF7hL0N1S419PZxx/SFH5aUvGQx2EvDcp7B2u4gyCzWp3\n",
"ecCddqTeSbDbjttUzqme+QtX0X62U75A3mGpTybMzujjafa0PVGCYUzdBEcZ8eOSyi0cln2prA5l\n",
"rp5XGCuCOwq8O78fSiPGT8gVWepTVy2X12gTJvnC5Km4dhFzyYyQSO5kDiDk0W0ZY1O1a98MvLT8\n",
"sSaNhDi2uvACFWD6qrgC5Lfi+8tYDJi1q73FqIRdz+a2D9HYVu4e7MDNpz6fekpCgVCP09LwggdO\n",
"Om+kkwMnIujHonzmweV/pJlwPyPagMt1BeSSEn5JQVpuTh0xFc5F92mEcVwkh+PdQAAuIQAAAFBB\n",
"nytFETw7/wAJ7jCDWjZFLC00ZbewAsGyXCc3vjx5932+4fHWtWI9u+9WBo5DrHqKmMECgglJLUK/\n",
"7WtGvnA1Y6K+ocYalIVTGdgBDds2gAAAACoBn0p0Q38ACq2XNF/aWAn4JpV0odiB3E5U0lr0r2Cj\n",
"rJPLOD5XLwAAI+AAAAAjAZ9MakN/AAqtlyd9lx23jmnbF1O/W5cXUlD/qj98kfgAAR8AAAEiQZtR\n",
"SahBaJlMCH///qmWABKEAJgBLV14W9ryp3B35/lgWC3JcZdBuANXXD7I0jVEzuWubUieVxh+SMAA\n",
"rWCiDefivg48BgT3ds6tv0fEqHH5pfh/bGgKkECmT7t6KqhiQ53ZDArqPgQCbhJYGiTTQCyP07n/\n",
"YrQSkfOY3OSN4nIFj3sJGQUZNuhBHRx369zS6cNedBQpzbVLTOi/5n9VDQ5a2No8oXoIOUAo/h/e\n",
"nGwxpWy5jyvm0C7IKgVhwvCTzgVYcICfulxdQDGOSWNDJsFb/BYFYkZQm5GshUIeULwe8so4Rksi\n",
"m+OR+s3ckdi1VdM0IY1nJ1cdN8aY648cty8AaRP+vlK1RZ0cN/HgWJKVEnJ12jACTEVg6l+a3pjA\n",
"AVsAAAAvQZ9vRREsO/8ACe4wby9T9AE/A5ZjGQ78prwsNGf9XyLKldt7B2LC0n50AAHaU2MAAAAi\n",
"AZ+OdEN/AA4mnyZGUjhNFuqcJWvT0HmV0fuGyNWmAAB7QAAAAB4Bn5BqQ38ADilFJ2XzNh9Ngojb\n",
"r8uNhVlYjd4AAe0AAAA7QZuTSahBbJlMFEw3//6nhAASWeqBLQ29EV/4DFpgBLj3GxI9xfWbcvtw\n",
"LK+ZAUbnBFCsQov8qNsgB00AAAAtAZ+yakN/AA4qleYncAJqyEeNn8yL7MPkf6I0Ns6ZvIBjBZpb\n",
"B66ngAAkIMqAAAAD/21vb3YAAABsbXZoZAAAAAAAAAAAAAAAAAAAA+gAAAfQAAEAAAEAAAAAAAAA\n",
"AAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAA\n",
"AAAAAAAAAAAAAAIAAAMpdHJhawAAAFx0a2hkAAAAAwAAAAAAAAAAAAAAAQAAAAAAAAfQAAAAAAAA\n",
"AAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAQAAAAAH0AAABkAAAAAAA\n",
"JGVkdHMAAAAcZWxzdAAAAAAAAAABAAAH0AAACAAAAQAAAAACoW1kaWEAAAAgbWRoZAAAAAAAAAAA\n",
"AAAAAAAAKAAAAFAAVcQAAAAAAC1oZGxyAAAAAAAAAAB2aWRlAAAAAAAAAAAAAAAAVmlkZW9IYW5k\n",
"bGVyAAAAAkxtaW5mAAAAFHZtaGQAAAABAAAAAAAAAAAAAAAkZGluZgAAABxkcmVmAAAAAAAAAAEA\n",
"AAAMdXJsIAAAAAEAAAIMc3RibAAAALRzdHNkAAAAAAAAAAEAAACkYXZjMQAAAAAAAAABAAAAAAAA\n",
"AAAAAAAAAAAAAAH0AZAASAAAAEgAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\n",
"AAAAABj//wAAADJhdmNDAWQAFv/hABlnZAAWrNlAgDPn4QAAAwABAAADABQPFi2WAQAGaOvjyyLA\n",
"AAAAHHV1aWRraEDyXyRPxbo5pRvPAyPzAAAAAAAAABhzdHRzAAAAAAAAAAEAAAAUAAAEAAAAABRz\n",
"dHNzAAAAAAAAAAEAAAABAAAAkGN0dHMAAAAAAAAAEAAAAAIAAAgAAAAAAQAAFAAAAAABAAAIAAAA\n",
"AAEAAAAAAAAAAQAABAAAAAAEAAAIAAAAAAEAABQAAAAAAQAACAAAAAABAAAAAAAAAAEAAAQAAAAA\n",
"AQAAFAAAAAABAAAIAAAAAAEAAAAAAAAAAQAABAAAAAABAAAMAAAAAAEAAAQAAAAAHHN0c2MAAAAA\n",
"AAAAAQAAAAEAAAAUAAAAAQAAAGRzdHN6AAAAAAAAAAAAAAAUAAAdrwAABL4AAAWNAAAEIgAABHoA\n",
"AAJ2AAAD+AAAA5wAAANBAAABhwAAAWsAAABUAAAALgAAACcAAAEmAAAAMwAAACYAAAAiAAAAPwAA\n",
"ADEAAAAUc3RjbwAAAAAAAAABAAAALAAAAGJ1ZHRhAAAAWm1ldGEAAAAAAAAAIWhkbHIAAAAAAAAA\n",
"AG1kaXJhcHBsAAAAAAAAAAAAAAAALWlsc3QAAAAlqXRvbwAAAB1kYXRhAAAAAQAAAABMYXZmNTcu\n",
"MjUuMTAw\n",
"\">\n",
" Your browser does not support the video tag.\n",
"</video>"
],
"text/plain": [
"<matplotlib.animation.FuncAnimation at 0x7f8f8e81ce48>"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fig = plt.figure(dpi=100, figsize=(5, 4))\n",
"plt.scatter(x,y)\n",
"line, = plt.plot(x,lin(a_guess,b_guess,x))\n",
"plt.close()\n",
"\n",
"def animate(i):\n",
" line.set_ydata(lin(a_guess,b_guess,x))\n",
" for i in range(30): upd()\n",
" return line,\n",
"\n",
"ani = animation.FuncAnimation(fig, animate, np.arange(0, 20), interval=100)\n",
"ani"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 1
}