{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"$$\n",
"\\newcommand{\\mat}[1]{\\boldsymbol {#1}}\n",
"\\newcommand{\\mattr}[1]{\\boldsymbol {#1}^\\top}\n",
"\\newcommand{\\matinv}[1]{\\boldsymbol {#1}^{-1}}\n",
"\\newcommand{\\vec}[1]{\\boldsymbol {#1}}\n",
"\\newcommand{\\vectr}[1]{\\boldsymbol {#1}^\\top}\n",
"\\newcommand{\\rvar}[1]{\\mathrm {#1}}\n",
"\\newcommand{\\rvec}[1]{\\boldsymbol{\\mathrm{#1}}}\n",
"\\newcommand{\\diag}{\\mathop{\\mathrm {diag}}}\n",
"\\newcommand{\\set}[1]{\\mathbb {#1}}\n",
"\\newcommand{\\norm}[1]{\\left\\lVert#1\\right\\rVert}\n",
"\\newcommand{\\pderiv}[2]{\\frac{\\partial #1}{\\partial #2}}\n",
"\\newcommand{\\bb}[1]{\\boldsymbol{#1}}\n",
"$$\n",
"# CS236605: Deep Learning\n",
"# Tutorial 3: Multilayer Perceptron"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Introduction\n",
"\n",
"In this tutorial, we will cover:\n",
"\n",
"* Linear (fully connected) layers\n",
"* Activation functions\n",
"* 2-Layer MLP implementation from scratch\n",
"* Back-propagation\n",
"* N-layer MLP with PyTorch's `autograd` and `optim` modules"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"# Setup\n",
"%matplotlib inline\n",
"import os\n",
"import numpy as np\n",
"import sklearn\n",
"import torch\n",
"import matplotlib.pyplot as plt\n",
"plt.rcParams['font.size'] = 20"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Reminder: The perceptron hypothesis class"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"The following hypothesis class\n",
"$$\n",
"\\mathcal{H} =\n",
"\\left\\{ h: \\mathcal{X}\\rightarrow\\mathcal{Y}\n",
"~\\vert~\n",
"h(\\vec{x}) = \\varphi(\\vectr{w}\\vec{x}+b); \\vec{w}\\in\\set{R}^D,~b\\in\\set{R}\\right\\}\n",
"$$\n",
"where $\\varphi(\\cdot)$ is some nonlinear function, is composed of functions representing the **perceptron** model."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Schematic of a single perceptron, and it's inspiration, a biological neuron.\n",
"\n",
"| . | . |\n",
"|---|---|\n",
"| ![](\"imgs/perceptron.png\")
| ![](\"imgs/neuron.png\")
|"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Last tutorial: we trained a **logistic regression** model by using\n",
"$$\\varphi(\\vec{z})=\\sigma(\\vec{z})=\\frac{1}{1+\\exp(-\\vec{z})}\\in[0,1].$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"**Limitation**: logistic regression is still a linear classifier. In what sense is it linear though?\n",
"\n",
"$$\\hat{y} = \\sigma(\\vectr{w}\\vec{x}+b)$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Generalized linear model: Output depends only on a linear combination of weights and inputs.\n",
"\n",
"Decision boundaries are therefore straight lines:\n",
"\n",
"![](\"imgs/perceptron_db.png\"/)
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Multilayer Perceptron (MLP)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"### Model\n",
"\n",
"![](\"imgs/mlp.png\")
\n",
"\n",
"Composed of $L$ hidden **layers**, each layer $l$ with $n_l$ **perceptron** (\"neuron\") units."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Each layer $l$ operates on the output of the previous layer ($\\vec{y}_{l-1}$) and calculates:\n",
"$$\n",
"\\vec{y}_l = \\varphi\\left( \\mat{W}_l \\vec{y}_{l-1} + \\vec{b}_l \\right),~\n",
"\\mat{W}_l\\in\\set{R}^{n_{l}\\times n_{l-1}},~ \\vec{b}_l\\in\\set{R}^{n_l}.\n",
"$$\n",
"\n",
"- Note that both input and output are **vectors**. We can think of the above equation as describing a layer of **multiple perceptrons**.\n",
"- We'll henceforth refer to such layers as **fully-connected** or FC layers.\n",
"- The first layer accepts the input of the model, i.e. $\\vec{y}_0=\\vec{x}\\in\\set{R}^d$."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Given an input sample $\\vec{x}^i$, the computed function of an $L$-layer MLP is:\n",
"$$\n",
"\\vec{y}_L^i= \\varphi \\left(\n",
"\\mat{W}_L \\varphi \\left( \\cdots\n",
"\\varphi \\left( \\mat{W}_1 \\vec{x}^i + \\vec{b}_1 \\right)\n",
"\\cdots \\right)\n",
"+ \\vec{b}_L \\right)\n",
"$$\n",
"\n",
"- Universal approximator theorem: an MLP with $L>1$, can approximate (almost) any function given enough parameters (Cybenko, 1989).\n",
"- This expression is fully differentiable w.r.t. parameters using the Chain Rule."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Given an input sample $\\vec{x}^i$, the computed function of an $L$-layer MLP is:\n",
"$$\n",
"\\vec{y}_L^i= \\varphi \\left(\n",
"\\mat{W}_L \\varphi \\left( \\cdots\n",
"\\varphi \\left( \\mat{W}_1 \\vec{x}^i + \\vec{b}_1 \\right)\n",
"\\cdots \\right)\n",
"+ \\vec{b}_L \\right)\n",
"$$\n",
"\n",
"- Universal approximator theorem: an MLP with $L>1$, can approximate (almost) any function given enough parameters (Cybenko, 1989).\n",
"- This expression is fully differentiable w.r.t. parameters using the Chain Rule.\n",
"\n",
"- Since it has a non-linear dependency on it's weights and inputs, non-linear decision boundaries are possible\n",
" - MLP with 1, 2 and 4 hidden layers, 3 neurons each\n",
" \n",
" ![\"overfit1\"](\"imgs/overfit_1-2-4HL_3N.jpg\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Activation functions "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"An **activation function** is the non-linear elementwise function $\\varphi(\\cdot)$ which operates on the affine part of the perceptron model.\n",
"\n",
"But why do we even need non-linearities?"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Without them, the MLP model would be equivalent to a single affine transform."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Common choices for the activation functions are:\n",
"\n",
"- The logistic function (sigmoid)\n",
" $$ \\varphi(t) = \\sigma(t) = \\frac{1}{1+e^{-t}} \\in [0,1] $$\n",
"- The hyperbolic tangent (a shifted and scaled sigmoid)\n",
" $$ \\varphi(t) = \\mathrm{tanh}(t) = \\frac{e^t - e^{-t}}{e^t +e^{-t}} \\in [-1,1]$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- ReLU, rectified linear unit\n",
" $$ \\varphi(t) = \\max\\{t,0\\} $$\n",
"Note that\n",
" $$ \\pderiv{\\varphi}{t} = \\begin{cases} 1, & t>0 \\\\ 0, & t<0 \\end{cases} $$"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# Plot some activation functions and their gradients\n",
"# Activation functions\n",
"relu = lambda x: np.maximum(0, x)\n",
"sigmoid = lambda x: 1 / (1 + np.exp(-x))\n",
"tanh = lambda x: (np.exp(x)-np.exp(-x)) / (np.exp(x)+np.exp(-x))\n",
"# Their gradients\n",
"g_relu = lambda x: np.array(relu(x) > 0, dtype=np.float)\n",
"g_sigmoid = lambda x: sigmoid(x) * (1-sigmoid(x))\n",
"g_tanh = lambda x: (1 - tanh(x) ** 2)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA7QAAAEyCAYAAADKsZxCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3XlcVPX+x/HXYVgFFAQXFNfcBcRd01DczRavtpha3azUXG+LrWbZptUvM027aVpprmlds8XSlMolNRQVV9RQQRRZRHaYme/vj0ECRWUZ5jDweT4e8xjmbPOecfnyOed7vl9NKYUQQgghhBBCCGFvHPQOIIQQQgghhBBClIYUtEIIIYQQQggh7JIUtEIIIYQQQggh7JIUtEIIIYQQQggh7JIUtEIIIYQQQggh7JIUtEIIIYQQQggh7JIUtEIIIYQQQggh7JIUtEIIIYQQQggh7JIUtEIIIYQQQggh7JKj3gFKw9fXVzVu3LjMx0lPT8fd3b3sgWzMXnOD/WaX3LZVMPeFK1lcSs2mtV91HB00nZPdXGX4vu2JNXOHh4cnKKVqWeVgVZS0zZLb1uw1u+S2LcltW7q0zUopu3t07NhRWcO2bduschxbs9fcStlvdsltW1dzG01m1eXtzWrM53v0DVRM9v592xtr5gb+UhWgfbPnh7TN2/SOUCr2mlsp+80uuW1LctuWHm2zdDkWQlRY208mcPFKNsM7+usdRQghhBBCVEBS0AohKqz14THUcHOib+vaekcRQgghhBAVkBS0QogK6UpWLj8fvsA97erh4mjQO44QQgghhKiA7HJQqKLk5uYSExNDVlZWsfepUaMGR48eLcdU5cMauV1dXfH398fJyclKqYSwrh8OxpFtNHOfdDcWwm5J22x90n4LIURhlaagjYmJwdPTk8aNG6NpxRsJNTU1FU9Pz3JOZn1lza2UIjExkZiYGJo0aWLFZEJYz7rwGJrV9iDIv4beUYQQpSRts3VJ+y2EENerNF2Os7Ky8PHxKXaDWZVpmoaPj0+JzpgLYUsX0s2En0nmvo7+8m9aCDsmbbN1SfsthBDXqzQFLSANZgnIdyUqsh2xRhw0+Ff7+npHEUKUkbQ31iXfpxBCFGaVglbTtKWapsVrmhZ5g/WapmnzNE07qWnaQU3TOhRY96imaVF5j0etkUcIYb/MZsWO80buaF6LOtVd9Y4jhN2StlkIIURVYK0rtF8Ag26yfjDQPO8xFvgEQNO0msBrQFegC/CapmneVsokhLBDu04nkpSlZDAoIcruC6RtFkIIUclZZVAopdTvmqY1vskm9wLLlFIK+FPTNC9N0/yA3sBmpVQSgKZpm7E0vquskUsIYX/Whcfg5gj929TRO4ooglIKozJiNBsxmU0YzcZ/XisTZmUGBWbMmJUZpRRmZcbMPz8rVJHLC627Zn9F4W1QcCTjCL3prfdXUmFJ23y90aNH061bNyZNmqR3FFHFxKXFEXU5ioPnY9l/9iz7fjNR27UxBs1+Rqs+fi6XuD1n9Y5RYpLbtk7H5tq8ZbbVKMf1gXMFXsfkLbvR8utomjYWyxlk6tSpQ1hYWKH1NWrUIDU1tUShTCZTife5FS8vL9q2bYvRaKRRo0YsWrQILy+vm+7j5+dHXFxc/uszZ87wwAMPsHv37vxl77zzDh4eHkyZMoW0tDQGDx7M999/j8FQ9PycOTk53HPPPXz//fc4Ohb9x5yVlXXd91je0tLSbP6e1iC5bSPTqPjhQAadayv+3PGH3nFKzJbfd67KJducTZY5i2yVbflZZRV6zlE5GJWRXJVLrsot8mejMpJlymLWylkYlREzZkzKhAlLcZr/rEyWIhOzTT5fcbhoLgSGBeodw55Vmbb5qrlz5/LCCy8QGRlJo0aNrHrs8sx9LWu23/bWThRU0bPnmHPYlbaLnWk7OZ97vtC6PdFfoUwu5KYGkpsYgjmntk4pS+jwIb0TlI7ktplqjooeNv53aTfT9iilFgGLADp16qR69+5daP3Ro0dLPFx+eQyx7+bmxsGDBwF49NFHWbZsGa+88sot9yuYw8PDAwcHh0LLXFxccHFxwdPTk0WLFnH//fffslAeMGAAP/74I6NGjSpyvaurK+3bty/Ox7KasLAwrv2zsweS2zbW7j1HjvkgoY3c7Cr3VaX9vk1mE8nZycRnxBOfEU9SVhKXsy9zOfsyKdkpXM4q8HP2Za7kXCHXnFvs4zs7OONicMHZ4IyroyvOjv+89jR4knY5jXq16+Hk4ISjgyMGzYCjg2P+4+rrgssLbaM5YnAwYNAMOGgOOGgOaJqGAwV+1hxwwAE0ilx+9WcN7Z/lea/zf1agmY1ouRloOekcPRhhl39PKhN7aZsBtmzZwty5czEYDGRmZl73HgaDgcDAQIxGI02aNGH58uW3bGc9PDxIS0vLz52YmMhdd91FZOQ/ty2//vrreHh48NxzzwGQmZnJoEGD2Lp1a5EnpXNycujXrx9bt2694Qlpa7bf9tZOFFSRs/969lfe3f0u8RnxBPkGMbLxSNrVasfE5VF4OabzSH9fdl/Yzh/nt5LjtY/BjYcytu0Uqjm56x39hnbt2kn37rfrHaPEJLdt7dq10+b/Lm1V0MYCDQq89s9bFguFrkr7A2FlfbOZGw9z5PyVW25nMplueIXzWm3qVee1u9uWKEf37t3zi9uvvvqKefPmkZOTQ9euXVm4cGGx3/taa9euZc2aNfmvQ0NDefnll+nfvz/Tp08nJSWF+fPnM3ToUF566aUbFrRCVDTr9sXQ1Ned27yU3lGsKtuUTWxaLDGpMZxLPce51HPEpcVxKfMS8RnxJGQmYFKm6/ZzdHDEy8Ur/9GkRhNquNSgukt13B3dcXe6/lHNqZrlZ0d33JzccHJwwkG7+XAJNvul0GyG7BTITC7wuPzPc/YVyE4t4lFgeYFCvpnBDYZOK//clZdN22a9rV27lg0bNvDII48QEBBw3Xo3NzciIiIAywnpBQsWFOuEdEktXbqUYcOG3fB3AGdnZ/r27cuaNWuk/bZDuaZcZu2ZxdcnvqaFdwtm3zGbznU75683mJKp7ujJg20G8WCbu0jOSmbRwUWsPLaSiITdfBT6ES1rttTxE9yYt6sDdWvY32CNktu2vF1tP4mOrQra74BJmqatxjLIRIpSKk7TtJ+BdwoMNjEAeMlGmcqVyWTi119/5fHHH+fo0aOsWbOGHTt24OTkxIQJE1ixYgWPPPJIiY+bk5NDdHQ0jRs3zl82c+ZMZsyYQXx8PPv37+e7774DICAggL1791rrIwlRrs4mZrDn7ySmDWyJpsXoHadUMs2ZRMRHcPLyScsj+STRV6KJz4i33PeZx83RjXru9ahdrTZN/JpQp1odalWrRW232tSqVgsfNx+8XLyo5litYk/RkZsJafGQfumf5/R4SLsEmUnXFK55RSs3OVnh6AYunnkPD3CpDl4NCiy7+qgOLp4cjTqDdDgukyrVNgcFBTF8+HAGDx6Mm5vbTbctrxPSACtWrGDlypWAnJCubFKyU5iydQr74vcxJmAMk9pPwsmh8D2yCgUF/lv3dvXmhS4vMLDxQJ777Tke/ulh3g95n14Netk4vRD2yyoFraZpq7CczfXVLL+JvgY4ASil/gv8CNwJnAQygMfy1iVpmvYmcLXqeuPqIBRlUdwrqeXRrSkzM5Pg4GBiY2Np3bo1/fv355NPPiE8PJzOnTvnb1O79o3vlbjRL7CappGQkECNGjUKLQ8JCUEpxZw5cwgLC8tvaA0GA87OzuXWfUsIa1q/LwYtb+7ZExEVv6DNNGZyJPEIhy4d4mDCQQ4nHOZ8+vn8Ow/dHN1o7tWcrn5d8ff0x9/DnwaeDfD39MfH1aeCF6pZkHoeruQ9UmIsz6lxBYrXBMi5wf2CLtWhWk1wqwlu3uDdxPLs5p233Pv6h2sNMJRscJTE5LCyf9ZKrKK1zXr3npo0aVKxBoMqrxPSYDkpffr06fyT0nJCuvJIyU5h7OaxRCVH8V7IewxuMrjI7ZQCjev//w+uHczKISuZvHUyU7dN5YNeH9C3Ud/yji1EpWCtUY4fusV6BUy8wbqlwFJr5KgIrnZZysjIYODAgSxYsABN03j00UeZNWtWsY7h4+NDcnJyoWVJSUk0adIENzc3srOzC607dOgQcXFx+Pj4XFe4Zmdn4+pqf90VRNViNivW74uhZzNf6nm5cULvQEVIz00n/GI4u87v4q+LfxGVHJXfTbi+R30CawXS0akjAzoMoJlXM+p51LtlV19dKGW5WpocDZfPQHI0zU/shvP/hSt5hWtG4vX7uXqBpx941IL6HcC9tuVn99rgXqvwz07yf05FIG1zyZT3CWmAhISEQvflygnpyiEjN4Nxm8cRlRzF3NC5hPiH3HBbdZNOKrWr1WbpwKWM3TyW535/jo9CP7rpsYQQFnYzKJS9qVatGvPmzWPo0KFs3LiR4cOH8/TTT1O7dm2SkpJITU294QiLHh4e+Pn5sXXrVvr06UNSUhKbNm1i6tSpeHt7YzKZyMrKwtXVlbi4OEaNGsWGDRuYMmUKmzZtYtAgy7SDiYmJ+Pr64uRkP0PCi6pp999JxCRn8tyAinPfkFKK48nH2XZ2G7vidnHo0iGMyoizgzPta7dnTMAYgmoFEegbiI+bD5B3L2qD3voGB8tvTFdiIeEEJP1tKV7zH2cs97EWUNvRE0yNoXo98O9sea5e/59nTz9LF2AhykCv3lPF6Q2hlCr3E9JgOemdlZWVv05OSNs/k9nEC7+/wNGko8wLnVesAvRmfyPdndz5pN8nPPnLkzz323MsG7yMVjVbWS+wEJWQFLTlqH379gQFBXHgwAHeeustBgwYgNlsxsnJiQULFuQXtBkZGfj7++fv98wzz7Bs2TImTpzIM888A8Brr73GbbfdBkCfPn3Yvn07t99+O8OGDeODDz6gdevWvPrqq7zwwgv5Be22bdsYMmSIjT+1ECW3fl8MHi6ODGxbV9ccZmVmf/x+fj37K1vPbiU2LRYNjQDfAP4d8G+6+nUluFYwro4V5JdMYw4knYaE45bi9dIJy3NCFOSm/7OdwRm8GoF3Y2jQ1fKc/2jEjl3hFXakUCHKSilVovlny+uENFDopHRycrKckK4E5oTPISwmjFe6vmK1+16rO1dnQd8FjPh+BJO3TmbVkFX4uvla5dhCVEZS0FrZ1eH7r9q4cWP+zw8++GCR+5jNRc/ruG3btiKXP/nkkyxatIh+/fqxa9eu/OUhISGFXq9cuZLZs2cXO7sQekjPNvLjoTjuDqqHm3PpB1opi7NXzrLh1AY2ntpIXHocTg5OdK/XnbFBY+nl3yv/CqxulILUC3AxEi4csjwuRkLiKSg4OnKNBuDbHDo8Ynn2bQ41b7NcYXWogN2fhbCRJUuWMHXqVPbu3cvChQt58803mTFjBp9++mmRRWNpTkgrpXj22WdvekIaLFPq/fLLL8yaNUtOSNu5X8/8yrIjy3io1UOMaDWiWPsopSjOEAq+br7M7zOfRzc9yvO/P8/i/osxOOjTRgpR0UlBa4eCg4MJDQ296cAZOTk5DB06lBYtWtg4nRAlsynyAhk5Ju7r5H/rja0o15zL5ujNrDm+hn3x+3DQHOju152pHabSu0Fv3PWaC1ApSP4bYsIhLuKfIrbgfa1eDaFOILS5F3xbWgpXn2bSLViIIhScfxagYcOGPPvssyxZsqRQMVvWE9IFu0rf6IQ0wMSJE/nwww/lhLSdi02L5dWdr9LWpy3TOhV/+rCSTErX2qc1L3V5iRk7Z7Akcgljg8aWPKgQVYAUtHZqzJgxN13v7Oxc6lEYhbCldeExNPKpRqdG3rfe2AqSs5L5+sTXrDm2hvjMeBp6NuQ/Hf7DXU3voo57HZtkKCQ9AWL3QexfEBtueWTm3YPn6Aq1W0PLO6FuoOVRp61lRGAhRLEUnH+2VatWzJs3D0dHRzw89DkB1KFDh5uelJYT0hWf0Wzk+d+fRynF+73ex6mEI7SXZIz7oc2GsituFwsjFtKlbheCaweXLKwQVYAUtEII3cQkZ7DrdCLP9G9R7tPYJGYm8uXhL1l9fDWZxky6+3Xntdtfo2f9nrYbjVgpyz2vZ3bAmZ2Wx+UzlnWaA9RqDa3vhvodLY9arcEg/00LURYF55+dOnUqb731FmvXrrUM4qbTveM3OyktJ6QrvuVHlnPw0kHeveNdGng2KNG+NxvluCiapjGj2wwOXTrEy9tfZv0963FzvPk8ykJUNfKbkhBCN9/siwUsc8+Wl8tZl1l6eCmrj60m25TN4CaDeSLgCZp5Nyu398xnNsOlo3nFa14Rm3bRsq6aDzTsDp0ftxSvfsHSZViIclBw/tlx48YBMG1a8buIClFQdEo0CyIW0KdBnxvONXsziuLdQ1uQh7MHb/R4gzE/j2HB/gU81/m5Er+vEJWZFLRCCF0oZZl7tntTHxrUrGb14+eac1lzbA0LDywkPTedwU0GMzZoLE1rNLX6exVyJQ5Ob4NTW+HUNshIsCyvXh+a9IJGt0OjHpb7Xsv5qrQQQgjrMSszr+18DWeDM9O7TS9Vz6KSXqG9qnPdzjzQ4gGWH13OwMYDCawVWLoDCVEJSUErhNDFX2eSOZOYwZQ+za1+7B2xO5i9ZzbRV6K5vd7tTOs0rfyuyOZmWq68ntpKp4MbISyvC7F7LbitDzTtDY17WgZykgJWCCHs1roT69gXv483e7xJrWq1bP7+T3d8mt9ifuO1Xa+x9q61ODrIr/FCgBS0QgidrPsrhmrOBgYFWG/u2eSsZN7b+x7fn/6extUbs6DvAu6of4f1789NT4ATP8PxHy1XYnMzwOBMrmcr6DfTUsjWCZCpcoQQopK4nHWZefvn0bluZ+697d5SH0dRskGhCvJw9uClLi/xn7D/sOb4Gka1HlXqHEJUJlLQCiFsLjPHxA+H4rgz0A93F+v8N7QpehOzds/iSvYVxrcbz5OBT+JscLbKsQFIOAnHf4DjP8G53aDMlm7EwSOhxSBo1IMDO/fQu2dv672nEEKICuHjiI9Jy0njxS4vlukkqVKlL2gB+jTsQ3e/7iyIWMDgJoOp6VqzDEcTonKQglYIYXM/H75AWraR4R3KPvdsRm4Gb+9+m+9OfUeATwCLByymhbeVprtIPAWR38DhbyD+iGVZ3UAImWaZSsevnXQjFkKISu540nG+PvE1I1qOsEL7UsqbaPNomsaLXV9k+IbhzNs3j9dvf72MeYSwf1LQ2khmZiaDBg1i69atRc47B5a55/r168fWrVtxdJQ/GlF5rd8Xg7+3G12blO3M8pHEIzz/+/OcSz3H+HbjGRc0ruz3FCWfgcPfQuR6uHDQsqxhdxj0LrS603IvrBBCiCpBKcWsPbOo7lydCcETrHPQMp4HbVqjKaNaj2LZkWXc3+J+2vq2tU4uIeyU3OBlI0uXLmXYsGE3LGbBMvdc3759WbNmjQ2TCWFb5y9nsv1kAsM6+OPgUPpW/duobxn942gyjZl8NuAzJgZPLH0xm5EEexbD4r7wURBseQ0MTjDgbXj6MIzZBN3GSzErhBBVzOYzmwm/GM6UDlOo4VKjzMcra5fjq8a3G4+3qzcfhH+AKu3QyUJUEpXzMuBPL8KFQ7fczM1kBEMxv4K6gTB49i03O3DgAJMnTyYhIYFjx46hlOLVV19ly5YtrFy5Mn+70NBQXn75Zfr378/06dNJSUlh/vz5DB06lJdeeolRo+RGf1E5fbs/FqVgeIfSzT1rNBv54K8P+OroV3T36857Ie/h5epV8gOZjJbpdfZ/ZRncyZQDtdtC39eg7b+gZpNS5RNCCFE55Jpzmbd/Hs28mjGs2TCrHLMsg0IV5OHswfh243ln9zv8EfsHIf4hVjiqEPapcha0OsnKyuLBBx9k2bJldOnShVdffZWsrCxeeeUVFi1aROPGjfO3nTlzJjNmzCA+Pp79+/fz3XffARAQEMDevXt1+gRClC+lFOvDY+jSuCaNfNxLvP+VnCs8F/Ycu+J2Mbr1aJ7t9GzJr8omRFmK2AOrIe0CuNWETmMsgzv5tStxJiGE/Rg9ejTdunVj0qRJekcRduDbqG85c+UM8/vMx+Bw4x52JWHNq6n3Nb+P5UeWM3ffXHrU62G1jELYm8pZ0BbjSipAZmoqnp6eVnvbLVu20KFDB7p06QJAUFAQmzZtIjExES+vwleQQkJCUEoxZ84cwsLC8rsiGwwGnJ2dSbVyNiEqgv3nLnM6IZ3xvW4r8b6XMi4xdvNYoq9E88btb/Cv5v8q/s6mXDj2A+z9DKL/AM0AzftD8ChoMRAcXUqcRwhhf5YsWcLUqVOJjo4udJJZiGtlGjP574H/0r52e3r597Luwa00lqCTwYkpHaYw7bdpfH/6e+5tVvrphISwZ3IPrRVFRkYSGBiY/3rfvn106NABNzc3srKyCm176NAh4uLicHZ2vq5wzc7OxtXV1SaZhbCldeExuDkZuDPIr0T7nbtyjod/epjYtFgW9l1Y/GI2JRa2vQMfBsDXj0JyNPR5FZ45CiPXQJt7pJgVoorYsmULw4cPJy4ujkuXLhV7v8zMTHr16oXJZLrhNjk5OQwaNAij0WiNqKICWHF0BZcyL/F0x6etOpe5tbocXzWw0UACfAL4OOJjsk3ZVjyyEPZDClor8vHx4eBBy6ioJ06c4JtvvmHEiBF4e3tjMpnyi9q4uDhGjRrFhg0b8PDwYNOmTfnHSExMxNfXFycnJ10+gxDlJSvXxMYD5xkUUBePEsw9ezzpOI9seoT03HSWDFhC93rdb76DUhC9A9aMhrmB8Nt7lnvgH1oDUw9AyHPgWaeMn0YIYW/Wrl2b3+4GBAQUe7/iDurYq1cvGdSxkkjJTmHpoaX09u9N+9rtrXpsa4/fpGkaT3d8mgvpF1h1dJV1Dy6EnZCC1ooeeugh0tLSCAgIYOzYsaxatQofHx8ABgwYwPbt28nIyGDYsGF88MEHtG7dmldffZWZM2fmH2Pbtm0MGTJEr48gRLnZfOQiqVlG7utY/LlnTySf4PFfHsdBc+DLQV8SWCvwxhubcuHQOlgcCl/caSlqb58EU/bD6HXQchDI/UVCVFlBQUEMHz6ckJAQ3Nzcrlt/4MABQkJCaNOmDQ4ODmiaxowZM1ixYgX33vtPV87Q0FA2b94MwPTp05k8eTIAd911FytWrLDNhxHl6rNDn5GWm8aUDlOsfuzyGJG4i18XetTrwZLIJaTnplv9+EJUdJXzHlqdeHh4sHHjxiLXTZw4kQ8//JB+/fqxa9eu/OUhISGFXq9cuZLZs4t3D7AQ9mRdeAz1arjSvalPsbY/dfkUT/7yJC4GF74Y+AUNqjcoesOsFAj/EnZ/CldiwKc53PUhtHsInK7/pVUIoRMdZyAAmDRp0g0Hg7LGoI5t2rSRQR0rgUsZl1h1bBV333Y3zb2bl8t7WLPL8VUTgycy8seRrDq2iicCnyiHdxCi4pIrtDbSoUMHQkNDb3kPztChQ2nRooUNkwlR/i5eyeKPqEvFnns2PjeeJ355AgfNgSUDlhRdzKZegJ9fgTltYfOrlml2HloDE/dYRi2WYlYIUUxFDeqYlJR0y0EdV69eXeSgjsJ+LY1citFsZHzQ+HI5vgKseEtuvsBagYT4h/DF4S9Iy0mz/hsIUYFZ5QqtpmmDgI8AA/CZUmr2Nes/BELzXlYDaiulvPLWmYCrp2zPKqXusUamimjMmDE3Xe/s7MwjjzxiozRC2M63+2MxKxhWjLlnEzITWHBxATjB0oFLaVyjceENLp+DHXNh33IwGyFgGHSfBPWCyye8EHaqwrXNOs1AUJwBfWbNmlXiQR19fHxkUMdK5lLGJb4+8TV3Nb3rxr2Cysr6PY7zTWg3gRE/jGDF0RWMazeu/N5IiAqmzFdoNU0zAAuAwUAb4CFN09oU3EYp9bRSKlgpFQzMB74psDrz6rrKXMwKUVVdnXu2YyNvmtbyuOm26bnpTNgygTRzGgv7LuQ2rwLT+ySegg2TYF6wpYtxuxEw+S8Y/pkUs0JcQ9rmfyilGDVqFPPnz0cpVeRDBnUUAJ8f/hyj2cjYoLF6RymVtr5t6d2gN18e+ZLUHOkpIKoOa3Q57gKcVEqdVkrlAKuBm02E9RAgw7AJUUUcjEkhKj6N4R1uPhhUrjmXZ8Oe5UTyCcbUGkNb37aWFZdOwPon4eNOcOhr6PQ4TI2Ae+ZBzaY2+ARC2CVpmwtYsmQJkZGR7N27l8cee4yYmBjGjBlDbm4uYJ1BHf/44w8Z1NGOJWQmsPb4WoY0HULD6g3L7X2sPW3PtSa0m0BqTipfHfmqHN9FiIpFK+toa5qm3QcMUko9kff6YaCrUuq6kRc0TWsE/An4K6VMecuMQARgBGYrpf53g/cZC4wFqFOnTsfVq1cXWl+jRg2aNWtWouwmk+mmw/BXVNbKffLkSVJSUqyQqPjS0tLw8Lj5VbqKSHKX3vIj2fweY2RuaDXcnW7cjK9JXMP2tO2MrDmSQC0QX0M6jaNXU+diGGYHZ2LrDyLGfyg5Lt42TF8yFeH7Lg3JDaGhoeFKqU5WOVgFIG3zP7Zt28bChQsxGAxMmzaNTZs2ERUVxccff1ysvz8REREsWLCAxYsX33S7kSNHMnPmTJo3L5+BhAqyZvttr//+wbrZv0n6ht9Sf2N6venUcqpllWMWZfzmdLrXUTwaVH7f+eL4xURlRfF6/depZqhmtePa698VyW1berTNth7leASw7mqDmaeRUipW07SmwFZN0w4ppU5du6NSahGwCKBTp06qd+/ehdYfPXq0xPfcpFr5Ph1bsVZuV1dX2re37vxqtxIWFsa1f3b2QHKXTrbRxNTffmVwYD2G9L/x37WvT3zN9jPbeSzgMZ5pMZLYVU9T/8Jm0Byg+wQMPZ+mobsv5XfO3Dr0/r5LS3JXeZW6bf7+++/54YcfeOSRR+jYsSNLlizB1dUVPz+/Yu1/xx13EBUVRbVq1W5YaOfk5HDXXXfRoUMHq+W+GWu23/b878ha2RMyE5i2fhp33XYX9/e8v+zBbsJh6yacnbVy/c79kvy4b+N9nK55mkntix7ZuzTs9e+uRnkxAAAgAElEQVSK5LYtPXJbo8txLFDwznn/vGVFGcE1XZqUUrF5z6eBMMC2FZYQotz8ejSelMxcht9k7tmI+Aje2f0OPep2YWpiMnwUjF/cz9B+tGUO2YFvg7uvDVMLUSlI25yn4PyzU6dO5a233iI4OJiwsLBiH2PMmDE3vWrs7OzMyJEjrZBW6OGLyC/IMefY7b2z12pZsyX9G/VnxdEVpGTbtieeEHqwxhXavUBzTdOaYGksRwDX/a+uaVorwBvYVWCZN5ChlMrWNM0X6AG8Z4VMQogKYH14DHWqu9CzWdEFaXxGPE9v+w/1HNx4d/9mDFlXIOgB9riF0m3wQzZOK0SlIm1znoLzz44bZxn5ddq0aXpGEhVIQmYCa46vYUiTITSq3qjc30+p8r2H9qpxQePYfGYzK46uYELwBBu8oxD6KfMVWqWUEZgE/AwcBdYqpQ5rmvaGpmkFR0YcAaxWhW/abQ38pWnaAWAblvt0jpQ1kxBCf5dSswk7cYl/tffHUMTcsyZjLi/8+BjpmQl8FH2CGo16wFM7YdgistyK1xVQCFE0aZuFKB5bX51V5T4slEXLmi3p27AvXx35iis5V8r9/YTQkzW6HKOU+lEp1UIpdZtS6u28ZTOUUt8V2OZ1pdSL1+y3UykVqJRql/e8xBp59HL58mUWLlxY6v2jo6MJCAgocl1mZia9evXCZDIVuR4s9/CEhIRgNBpLnUEIa9kQEYvJrLivYxFzz0ZvZ9EXPfgr/SzTc91pNup/8NAqqNPm+m2FEKUibbMQN5eYmcjaE2u5s8md1895Xk6UgmJMjWwV49uNJzU3lRVHV9jmDYXQiVUKWmFR1oL2ZpYuXcqwYcNueQ9P3759WbNmTblkEKK4lFKsC4+hXQMvmtUuMLhLwklYNZK9q4fxX8cM7vYO5J7Hd0HjnvqFFUIIUSUtP7KcLGMWTwY9qXeUctGqZitCG4Sy/MhymZdWVGq2HuXYJt7d8y7Hko7dcruSTA3QqmYrXujywk23efHFFzl16hTBwcH079+fqKgozp07R1ZWFlOnTmXsWEt3lujoaAYPHkzPnj3ZuXMn9evXZ8OGDfmZnnzyyULL3dzcWLFiBStXrsx/r9DQUF5++WX69+/P9OnTSUlJYf78+QwdOpSXXnqJUaNGFetzCVEeDp+/wrELqbw5NK/HQeZlCJsNexeT7OzGi/6NaVDNl+mDPwMHOa8mhBDCtlKyU1h1bBUDGw+kaQ3bzWlumw7H/xjfbjwPfv8gK46uYHy78TZ8ZyFsR36TtKLZs2dz2223ERERwfvvv8/SpUsJDw/nr7/+Yt68eSQmJuZvGxUVxcSJEzl8+DBeXl6sX7/+hstzcnI4ffo0jRs3zt9/5syZvP3226xYsYL9+/czd+5cAAICAti7d69NP7cQ11oXHoOzwYG7A+vA/hXwcSfY/V8IHsU7nf9Fksrl/V4fUM3JevPjCSGEEMX11dGvyDBm2P7qrLr1JtbUxqcNvf17s/zIctJy0mz75kLYSKW8QnurK6lXlfc8tPPmzePbb78F4Ny5c0RFReHj4wNAkyZNCA4OBqBjx45ER0fTs2fPIpcnJCTg5eVV6NghISEopZgzZw5hYWH5V5oNBgPOzs52O8eusH85RjPfHTjPmKaX8Vp1F8TsBf8uMGodP+dcZNNvzzG5/WRa+7TWO6oQQogqKDUnlRVHVtC3YV9aeLew6XsrlE2v0AKMDx7PiO9HsPLYykozNZEQBckV2nISFhbGli1b2LVrFwcOHKB9+/ZkZWXlr3dxccn/2WAw5A/kVNRyNze3QvsCHDp0iLi4OJydna8rXLOzs3F1dS2PjyXELW0/eJxnsz/hhXNPQXI0DP0ExvxMoncD3v7zbdr4tGFMwBi9YwohhKiiVh1bRWpuapUp7tr6tKWXfy+WHVlGem663nGEsDopaK3I09OT1FTLTfcpKSl4e3tTrVo1jh07xp9//lnq43p7e2MymfKL2gsXLjBq1Cg2bNiAh4cHmzZtyt82MTERX19fnJycyvZhhCgpswn2LqHL9/150HEbqutTMDkcgkeiNI23d79NWm4ab/d4G0eHStk5RAhRwY0ePZqPP/5Y7xhCRxm5GSw/spwQ/xDa+Nh+ZH1bjnJc0FPtnsq/b1iIykYKWivy8fGhR48eBAQEEBYWhtFopHXr1rz44ot069atTMceMGAA27dvJyMjg1GjRvHBBx/QunVrXn31VWbOnJm/3bZt2xgyZEhZP4oQJXM+Aj7rCz88wyFjA5YGfIXD4FngWgOAn6N/ZvOZzUwMnkgz72Y6hxVCVFVLliwhMjKS6OhovaMInaw9vpbL2ZcZFzROl/e38S20+dr6tuWO+nfw5eEvycjN0CmFEOVDLpNYWcGRiG+kcePGREZG5r9+7rnn8n++0fKJEyfy4Ycf0q9fP3799df8bsYhISHs2rWr0PvPnj27TJ9BiGLLToOwWfDnQnCvxbaA2Tz2VwN+7hmSv8mVnCvM3jObtj5t+Xfbf+uXVQhRpW3ZsoW5c+diMBi4dOlSoYEWL1++zMqVK5kwYUKpjh0dHc2dd97JkSNHrluXmZnJoEGD2Lp16w1nVsjJyaFfv35s3boVR0f51ay8ZBmz+OLwF3T3605QrSBdMihl+3tor3qq3VOM/HEkq46t4vHAx3VKIYT1yRVaO9GhQwdCQ0MxmUw33CYnJ4ehQ4fSooVtBzgQVdTxn2BBV9i1ADo+BhP38H/nAwis70XLuv/c1z1/33ySs5N5tfurGByKN02WEEJY29q1a/Nv1QkICCi0TuaRrxrWR60nMSuxytw7e63AWoH0qN9DrtKKSkcKWjsyZsyYWzaIjzzyiA0TiSrpynlY8zCsGgGu1eHxX+CuORy97MDh81cY3qF+/qaHEw6z5vgaRrQcQVuftjqGFkJUdUFBQQwfPpyQkBDc3NwKrSs4j/y0adMYOnQoHTt2pG3btixatCh/u+joaFq3bs2TTz5J27ZtGTBgAJmZmcA/88hfu3zFihXce++9+ccIDQ1l8+bNAEyfPp3JkycDMHToUFasWFGu30FVlmPKYWnkUjrW6Uinup10y6HAthPRXuOpdk+RnJ3MmuNy8kRUHtKvRQhRPGYT/LUUtswEcy70fQ1unwwGywBk68NjcDJo3BNsKWhNZhNv/PkGPm4+TGo/Sc/kQgjBpEmTmDSp6P+LZs+eTWRkJBEREQAkJSVRs2ZNMjMz6dy5M8OHD8+fdi8qKopVq1axePFiHnjgAdavX0/Pnj05deoUa9asKbT8gQceKHIe+RkzZhAfH8/+/fv57rvvAJlHvrz97+T/iM+I560eb+maQyld61na1WrH7fVu54vDX/BgywdlPnhRKVSqglYphabH0HF2SCm9hiUQdulCJGycCrF/wW19YMgcqNkkf3Wuycz/ImLp06o2Nd2dAVh7Yi1HEo/wXsh7eDrLnMhCVHXv7nmXY0nHbrmdyWS6aW+kglrVbFXsuedLojTzyDdq1Ejmka+gcs25LDm0hKBaQXTzK9sgndag92+qT7V7iod/epivT3zNo20f1TmNEGVXabocu7q6kpiYKIVaMSilSExMlLlqxa0Zc2DbLFjUCy6fgeFLYPQ3hYpZgN9PXCIhLYf7OjYAICU7hY/3f0xXv64MajxIj+RCCAGApmm3fBQk88hXPt+f+p7z6ecZFzROLnwAwbWD6ebXjaWRS8k0ZuodR4gyqzRXaP39/YmJieHSpUvF3icrK8suGw5r5HZ1dcXf399KiUSldD4CNkyEi5EQ9CAMmg3Vaha56brwGHzcnendshYAnxz4hLTcNJ7v/Lz88iCEACj2lVRrX6FUSjF69Gi6det2wy7HiYmJ5T6PvKurK3FxcfnzyE+ZMoVNmzYxaNCg/Awyj7z1Gc1GPjv0GW182nBH/Tt0zVKRLro81e4pHt30KGuPr5WrtMLuVZqC1snJiSZNmtx6wwLCwsJo3759OSUqP/aaW9gJYzb89h5s/xDca8FDq6Hl4Btunpyew5ajF3m4W2OcDA78nfI3a46tYVjzYbTwlhG3hRD6W7JkCVOnTmXv3r0sXLiQN998kxkzZvDpp5/i5ORUaB75/v37588j37JlS6vNI3/77bczbNiwQvPIv/DCC/kFrcwjXz42RW/ibOpZ5obO1f0E69V6tiKc5+1QpwNd63bl88jPeaDlA7g5ut16JyEqqEpT0AohrCB2H/xvAlw6CsGjYODb4OZ90102HjxPrklxX0fLFf854XNwcXRhYvBEWyQWQoibKjj/LEDDhg159tlnWbJkSaGroWWdR3737t1FLi84j3zBeeNlHvnyZzQb+fTAp7TwbkFog1C943D1+mwFqGcBGN9uPI/9/BjrTqzj4TYP6x1HiFKrNPfQCiHKIDcLtrwOn/WDrBQYtQ6GLrxlMQuW7sat/arTpl51dsftJuxcGE8EPoGvm2/55xZCiFsoOP9sq1atOH36NI6Ojnh4eNjk/WUeef389PdPRF+JZkLwBBw0+ZX3Wp3qdqJL3S4sjVxKljHr1jsIUUHJv24hqrqYv+DTEEsX4+CRMPFPaN6/WLueuJjKwZgU7uvoj1mZeX/v+9RzrydneoUQFUbB+WenTp3KW2+9RXBwMGFhYTbLIPPI257RbOSTA5/QumZr+jToo3ccoGLdQ3vV+HbjSchMYH3Uer2jCFFq0uVYiKoqNxO2vQO7PgbPejB6PTTrV6JDrA+PwdFB497gemz6exPHk48z645ZuBhcbr2zEELYQMH5Z8eNGwfAtGnT9IwkbGDjqY2cSz3H/D7zdb939qr8LscVIw4Anet2plOdTiw9tJT7Wtwn7bewS3KFVoiq6Oxu+G9P2DkPOjwCE3aVuJg1msx8uz+W3i1rU6OaAwsiFtDcuzl3NrmznEILIYQQt5ZrzuXTg5/S1qctvfx76R2nwnuq3VPEZ8az/oRcpRX2SQpaIaqSnAzY9DIsHWiZY/aRDXD3R+BavcSH+uNkAvGp2dzXsT7fnfyOs6lnmRQ8Se5TEkIIoasNJzcQmxbLxOCJFebqLPwzynFF07luZzrU7sCSyCXkmHL0jiNEiclvnkJUFWd2wn97wJ8LoPPjMGEnNO1d6sOtD4/Bu5oTPZp78cmBTwj0DawQo0gKIYSounJMOSw6uIigWkH0rN9T7ziFqLxOxxWnxLbQNI2ngp8iPiOeb6K+0TuOECVmlYJW07RBmqYd1zTtpKZpLxax/t+apl3SNC0i7/FEgXWPapoWlfeQmZ2FsLacdPjxefj8TjCb4NGNMOQDcPEs9SFTMnL55chF7g2uz4ZT67mYcZHJ7SdXqDPhQlR10jaLqujbqG+JS4+rcFdnocA8tPrGKFLXul1pX7s9nx36TK7SCrtT5oJW0zQDsAAYDLQBHtI0rU0Rm65RSgXnPT7L27cm8BrQFegCvKZp2q3nCRFCFItX8iH45HbY8yl0GWu5V7ZJSJmPu/HgeXKMZoYE1WTxocV0qduFbn7drJBYCGENFaltrogju9oz+T5vLNOYyaKDi+hQuwPd/brrHefGKmBFq2ka49uN52LGRb6N+lbvOEKUiDWu0HYBTiqlTiulcoDVwL3F3HcgsFkplaSUSgY2A4OskEmIqi07DX54luAD0wEN/v0j3PkeOLtb5fDr98XQso4nkWmbSMpKkquzQlQ8FaJtdnV1JTExUYowK1FKkZiYiKurq95RKqSVR1cSnxkvbVIpdffrTrta7fgsUq7SCvtijWl76gPnCryOwXJW91rDNU0LAU4ATyulzt1g3/pFvYmmaWOBsQB16tSxyvxxaWlpNp2HzlrsNTfYb3Z7yu2VfIBWxz7GJfsSf9cZTEyLxzBH50J0mFWOH5dmZv/ZTO5rCYv3L6aFawsuH7lM2BHrHB/s6/suSHLblr3mtpEK0TZrmoa7uzvnzp0rYu+iKaXsshixVW6TyUR6ejpnzpyxyvHs+d9RwezppnQ+jf2UALcA0o6lEXYsTNdsRck2WU7s5ObkVNjvvIfWg4XpC5n942xCPAv36LLXvyuS27b0yG2reWg3AquUUtmapo0DvgRKNMu1UmoRsAigU6dOqnfv3mUOFRYWhjWOY2v2mhvsN7td5M66AptnwIHPwacZjFrO2dNZVs/93qZjGBxO06J9Aj8fSOWjXh/RuW5nq76HXXzfRZDctmWvuSsQaZutSHLbXsHsc/6aQ1ZMFm/0f4Pm3s31DXYDGTlG2PwzLs7OFfY776V6sefnPfya8ivPDnoWd6d/enbZ698VyW1beuS2RpfjWKBBgdf+ecvyKaUSlVLZeS8/AzoWd18hRDGc/BUWdod9X8Ltk2H8dmho/XtaTWbFN/ti6dnci69PLqd97fZ0qtPJ6u8jhCgzaZtFlXEh/QIrjq7g7tvurrDFbCEVuAOCpmk80/EZkrKS+Dzyc73jCFEs1iho9wLNNU1rommaMzAC+K7gBpqm+RV4eQ9wNO/nn4EBmqZ55w04MSBvmRCiODIvw4ZJ8NUwcK4GY36BAW+Bk1u5vN3OUwlcuJJFk8bHuJB+gbFBY+2ya6AQVYC0zaLKWBCxAIBJwZN0TnJz9nIreWCtQAY1HsSyI8uIz4jXO44Qt1TmglYpZQQmYWnsjgJrlVKHNU17Q9O0e/I2m6Jp2mFN0w4AU4B/5+2bBLyJpeHdC7yRt0wIcSsnfrZclY1YAT3+A+P+gAbW7fp7rXXhMXi6auxOWkcbnzb0qNejXN9PCFE60jaLqiIqOYrvTn3HQ60ews/D79Y76OhqPatV5Eu0eaa0n0KuOZeFEQv1jiLELVnlHlql1I/Aj9csm1Hg55eAl26w71JgqTVyCFElZCTBphfh4Bqo3QZGrID6Hcr9ba9k5fLz4Qt0CzxLeNo55naeK1dnhajApG0WVcFH+z7C3dGdJwKfuPXGOrOn0b4bVG/AiJYjWHlsJaNbj6aZdzO9IwlxQ9bociyEsJWjG2FBV4hcD71egLFhNilmAX48GEdWrokLDj/SzKsZoQ1CbfK+QgghRFGOZh7lt5jfeDzwcbxcvfSOU2z2cip4XNA43B3dmbtvrt5RhLgpKWiFsAfpCfD1v2HNaPCsA09ug9CXwdHFZhHWhcfgX+8ssel/MyZgDA6a/PchhBBCH7nmXL5J/oYGng14uM3DescpFvu5Pmvh5erF44GP81vMb+yJ26N3HCFuSH4jFaIiU8pyNXZBFzj6PfSZbilm/YJsGiM6IZ2/ziTjWWcHtavVZlDjQTZ9fyGEEKKgtcfXciH3As91eg5ng7PecYrlao9je7pbZ1TrUdRzr8esPbMwKZPecYQokhS0QlRUqRctV2TXjQGvRjD+DwiZBgYnm0dZvy8GR9fzxGQdZFTrUTjpkEEIIYQASM5KZkHEAlq6trSv21/s7RIt4OroyvOdn+fk5ZP8kfqH3nGEKJIUtEJUNErBgdWWq7JRm6HfTHh8M9RurUscc97cs/Ua7aGaYzXua3GfLjmEEEIIsEzTk5GbwTDvYXY5OKG9Je7TsA/d/brz4+UfScxM1DuOENeRglaIiiTpb8ucst+Og1ot4akd0PM/YLDKgOSl8ufpRM6nXSDFYS/Dmg+junN13bIIIYSo2k4kn+DrE19zf4v7qedcT+84JaLs8RItoGkaL3Z9kWyVzUf7PtI7jhDXkYJWiIrAZIQdH1nmlT23Bwa/D4/9BL7N9U7Gun0xeNTahabB6Daj9Y4jhBCiijIrM2/9+Raezp5MDJ6od5wSy7+HVt8YpdK0RlNCq4fy7clvOXTpkN5xhChECloh9HZ+PyzuDZtnwG2hMHE3dB0LDga9k5GWbeSnyGgcvXYzoNEA6nvU1zuSEEKIKurbqG/ZH7+fZzs+a1fT9FyVf33WHitaYGCNgfi6+fLO7ncwmWWAKFFxSEErhF6y0+DnV2BxH0i7BA8sgxEroYa/3sny/XQoDqP7boxk8mjbR/WOI4QQoopKzExkTvgcOtbpyNBmQ/WOUyZ2Ws/i5uDGc52eIzIxktXHV+sdR4h8UtAKoYeozZbuxbs+ho7/tlyVbXNvhRvL/+vws1Tz3U37Wu1p69tW7zhCCCGqqP/76//IMGYwo9sMuxwICkAp+7yHtqA7m9xJj/o9+GjfR8SlxekdRwhAClohbCv1gmUanhX3gZMbPLYJ7voQ3Cpe16lzSRmEx+/G7HiJh1o/pHccIYQQVdSfcX/y/envGRMwhqZeTfWOU2r2X85aBoia0W0GAG/++WalKNKF/ZOCVghbMBnhz0/g485wdCP0ftkyr2yj7nonu6H1+2JwrrmTmi6+9GvYT+84QgghqqCM3Axm7pxJQ8+GPBn4pN5xyiR/UCj7vMCcr55HPSa3n8wfsX+wKXqT3nGEkIJWiHJ3Zhd8GgKbXoQGXWDCn9D7BXB00TvZDZnNirUH9uPocZwRrR7AyeCkdyQhhBBV0JzwOcSmxfJGjzdwdXTVO45V2Hk9C8DIViMJ8Alg9p7ZJGcl6x1HVHFS0ApRXtLi4dvx8PkgyL4CD34Fo9aBz216J7ulvdFJJGrbMGiO3N/yfr3jCCGEqIL+jPuTNcfXMLrNaDrW6ah3nDKz13loi2JwMDCzx0yu5FyRrsdCd1LQCmFtJiPsXgTzO8GhdXDHs5ZBn1rfbTf9jFb/dRInr3D6NuyHr5uv3nGEEEJUMWk5aczYMYPG1Rszpf0UveNYRyWr+Vp4t2By+8lsPrOZjac36h1HVGGOegcQolKJ3m7pWnzhEDTtDXf+H/g21ztViWTkGPnl7I841Mri4Taj9I4jhBCiCnr/r/e5mHGR5YOXV5quxlfZx6nt4nm0zaP8du433tn9Dh3rdJT56oUu5AqtENaQ9DeseRi+GAIZyXD/F/Dw/+yumAXL3LPKcwcNPVrQrlY7veMIIYSoYn6O/plvor5hTMAYgmoF6R3Haq5eoLWTzlrFYnAw8M4d7wDwyvZXMJlNOicSVZEUtEKURdYV2DwDFnSBk1sgdDpM2gtt/2W3LdayiF8xuF7kiaCH7XauPyGEEPYpJjWG13e+TpBvEBOCJ+gdx6oq622m9T3q82KXFwm/GM7nhz/XO46ogqTLsRClYTbBvmWw9S3ISIB2I6HvDKjup3eyMolJzuB4xmY8angwuMkgveMIIYSoQnJNuTz/+/NoaLzX6z2cHCrXCPuVaVCoa917273siN3B/P3zaVerHZ3rdtY7kqhC5AqtECWhFERtsUzD8/1/LF2Kn9wG//rE7otZgJV7D+PoeZjBTe6udPcsCSGEqNjm7Z/HoYRDzOwxs1Lfi1kZ+z5pmsbrt79OQ8+GPP/78yRkJugdSVQhUtAKUVzn9sKXd8OK4ZZpeO7/Ah77Cep30DuZVSilWB/1PzTNxGOBI/SOI4QQogrZenYrXxz+ggdbPkj/Rv31jlMu8rscV8aKFnB3cuf/ev0fqTmpvPj7i3I/rbAZKWiFuJVLx2kbOQuW9IP4ozD4PZgUbtf3yRZlb3Qiac47aFgtgKZeTfWOI4QQooo4dfkUL/3xEgE+AUzrPE3vOOWm8nY4/kfLmi15pesr7L6wmwURC/SOI6oIqxS0mqYN0jTtuKZpJzVNe7GI9c9omnZE07SDmqb9qmlaowLrTJqmReQ9vrNGHiGsIiUGNkyChd3wTo6A3i/D1AjoOg4cnfVOZ3WL9/6Cg3MiY4Ie0juKEMIKpG0W9iAlO4UpW6fg5ujGh6Ef4mJw0TtSuVF5l2grz6nwov2r+b8Y3nw4iw8t5ofTP+gdR1QBZR4UStM0A7AA6A/EAHs1TftOKXWkwGb7gU5KqQxN054C3gMezFuXqZQKLmsOIazmynnYPhfCvwAUdBnHbkN3evS+V+9k5SYr18Sfl37AycODu5oN1DuOEKKMpG0W9sBoNvLCHy9wPv08nw/8nLrudfWOZBOVvaAFeKXrK0RfiWbGjhn4e/rLNICiXFljlOMuwEml1GkATdNWA/cC+Y2mUmpbge3/BEZb4X2FsK6UWNj+oWX0YrMRgh+CkOfBuxG5YWF6pytX6yOOotwjCak3rFKfHReiCpG2WVRoSine2f0OO2J38Fr31wiubWfnT8wmyEiEtHhIj4e0S5CVAjmpkJ0GOemQk2Z5mIygzPjk5PCl0yUaxALLPwFHF8vD4GLp+eVUDVxrgKsXuHmDW95zNR/w9AMXD70/dbE5GZz4sPeHjPxhJFO3TmXVkFX4edj/4JmiYtJUGSfF0jTtPmCQUuqJvNcPA12VUpNusP3HwAWl1Ft5r41ABGAEZiul/neD/cYCYwHq1KnTcfXq1WXKDZCWloaHh/3853CVveaGipndJesSDc+uxy9uM6C4ULcPZxveT5ZbnfxtKmLu4ihu7lcP/8Rljx952e8V/Jz1P0Ne2b/vikZyQ2hoaLhSqpNVDlYBSNtse5K7ZH5J+YWNlzfSv3p/7vG+p1THKM/sDqYcXLPicc26mP9wy7yIa1Y8LtkJOOWmomEucl+zZsBkcMt7uKI0R5SmkascOJuqqOnqgJcLOJiNOJhz0FQuDuZcDKZsHE0ZN8xkNLiT7VKTbBcfsl18yHKtTaZbPTLd/Mh088PoVL5/jqX5vi/kXmBO3By8Hb2ZWncq1RyqlVO6G5N/m7alR9ts03loNU0bDXQCehVY3EgpFatpWlNgq6Zph5RSp67dVym1CFgE0KlTJ9W7d+8y5wkLC8Max7E1e80NFSx70mnYOR/2fwXKDO1HwR3PUs+7EfWu2bRC5S6B4uQ+fzmDpKjX8XNuw0MDKsboxpX5+66IJHfVJm2zdUju4tt4aiMbz2xkSNMhvNPzHRy00g3pYpXsORmQGAXxx+DSMbh03PKc/Lfld4OrDC7g1RDqNoHqPcGjNrjXtjxf/dnNC5w9cHB0wUHTuHYW3bOJGQx9fxtPtHRm+qgbjORsNlmu9GYmQ9ZlyAZQQzoAACAASURBVLwM6QmQeh7HK5aHe2ocXDkCF34tvG81H6h5G/i2gDptoU4bqBMA7r5l+47ylPb7bnS+ERN+ncDq7NV82v9T3BzdrJKnuOTfpm3pkdsaBW0s0KDAa/+8ZYVomtYPeAXopZTKvrpcKRWb93xa07QwoD1wXaMphNXE7oOd8+DIBtAMlkK25zPg3ejW+1ZCH+/6CQfnRB5uO0XvKEII65G2WVRI285uY8aOGXSp24U3b3+z1MVsqWSnQtxBiIuA8xGW54Qo8scfdnAEn2ZQNwAC77MUh96NwKsReNQBB+tkvek9tA4GqFbT8riV3ExIjobEU5B0Ku/5NET9AhFf/bOdRx2o3cZS5NZrb5lu0LuJzWZq6F6vO+/e8S7Tfp/G02FPMz90Pk6Ga8t9IUrPGgXtXqC5pmlNsDSWI4CRBTfQNK098CmW7k/xBZZ7AxlKqWxN03yBHlgGpRDCupSCU7/Cjo/g79/BpTrcPgW6jofqVfeeDqUUv5zbgMHZnRFth+gdRwhhPdI2iwpne+x2nv3tWVr7tOaj0I/Kt6gxm+BiJJzZCbHhlgI28ST5xaunH/gFQ9thULs11GoFNZuW6ywGytoT9zi5WbLXbn39urR4uHgY4o9Yni8ehj2LwZR33srNG+p1sBS39Ttafvasc/1xrGRA4wGk5qTy+q7XeXn7y8y+YzYGB0O5vZ+oWspc0CqljJr2/+3deXxV9Z3/8dc3+75BSEJYEgRZREBWUVFAreu4jHtr1VbrjNVap9PWGZ3ftLW1Lp22Op2O1bGtVjuDVsUN6kZBREFk30OAsAcICWTf7r3f3x/nhkTWhBvuuSf3/eRxHme55977vgnJN59zvud7zH3A+0As8Adr7VpjzCPAEmvt28AvgDTgL8Y5GrTdWnsVMBx41hgTwLmF0OOHjcAoEprWJljzOiz6b6dhSy+Ai38K4+6ApAy307lu/uYymhJWcnavqzUYlEgPorZZIs3n5Z/zwNwHGJw1mGcueoa0hG6+NtDvgz2rYNunsPVTp5BtrnYeyyh0itdRN0LBaGf5FBZvxxLisDVd09YV+rRp7dv8rbBvfbDAX+b0WPvkl+1dqzMHwICzg9Nkp8jvprPSANedfh21LbX8cukvMcbw6HmPEh+jM7USum65htZaOxuYfdi2f++wfNExnvcZcGZ3ZBD5koM7YMnvYemL0FgFucPh6v+GM2/okfeQPVnPLPkLxgS4f+LX3I4iIt1MbbNEivk75/O9ed+jf3p/nr34WTITM0N/Ub+P9JoSWLDCKWK3L4LmGuexnNPgjKuhaAoMPBcyC0N/v27QVs+aMHX1PUJsPBSMcia+4WxrqXe6Ye9aCju/gLKPYfWrzmNJWV8qcE2gNeQId4y8A7/189Syp2j1t/Lk+U+q+7GELKyDQomcUtbC1k9g8XOwIXgj76GXw8S7ofj8sF0r4hWNLT7W1s4hI3EQo/KGuh1HRER6oNlbZvPwgoc5Ped0nrnoGbKTsk/uhfytsHu5085v/RR2fM64ljrnsd6nw8jroOg8p4CN4kuJuiwhFQZOdiZw/pY6UOYcINi+0JlvfA+AKSYetk6A4inOwYJ+EyA+qctveeeZd5IYm8gTXzzBA/Me4FdTf6VeYhISFbTiffWVsOoV5/6xFeshOQfO/S6M/6YzIqEc1Z+WfQoJ5Vxe/IDbUUREpAd6ZcMrPPr5o4zLG8dvpv+ma92Mfc1Ol9htC2DrAtixGFqDt7TJHQajb2ZtfTZnXPYtV7oPn4y2W2VG9OF1Y5xriXMGwZjgZff1+2H7InZ9+ir9fdth/i/g4ycgLgn6T4Si850it+/YTveCu3XErSTEJvDTRT/l7g/u5ulpT5OVlHUKP5j0ZCpoxZsCAdgyF5a/5JyN9bdA4Xi4+rfOUdr48A4J70WvbHgDiOPb4693O4qIiPQg/oDTpfSFtS8wtd9UfnHBL0iKO8GZvNYm2LUkeP1rsID1NTmP9TkDzrq1/Qxs8DY0FfPmeaaYBbp7SKjwSe0Nw69k8940+k+d6txKaPtCKPsEts6HuY/CXAvxKU735KIpTs+4gjEQe+xS48ahN5KRmMHDnzzMrX+9lf++8L8ZkKETEdJ1KmjFWw5shZUznHvHVu9wRumbcBec9XXnfmvSKTsP1LAvsJDilElkJ3fDtUwiIiJAXUsdD37yIPN3zufmoTfzw4k/PPrAP62NzjWbWxc4RezOL4Ij8BrntjnjvhEsYM/p3C1sPKBtUKiIPkPbGclZMPQyZwJoqAp+Hz9xitw5P3G2J6Q7XZmLpjhncPNHObcl6uDSokvJS8nj/r/dz9dmf42npj3FuLxxYf5A4nUqaCXy1VXA2pmw+i+wczFgnFH7Ln4Ehl0BcbruoqueXvQmJraR20fd4HYUERHpIbbVbOOBuQ9QVl3Gv036N24adlP7gy31zlnXbZ86xc+upU7vKhPjFDoTv+WcfR042TlY3ZN5vqI9TEoOjLjKmcD5u23rJ+0FbukHzvakTOd7XHSeM+WdCTExnNXnLP58+Z/59pxvc9f7d/G98d/j1uG3ujd4lniOClqJTM21sP5dp4jdMg+sH/JGwkU/droU69rYk2atZe7OWcTFZXPtsKluxxERkR5g1pZZPLLwEeJj4/ndxb/j7JyRsOmjYBfiT50CNuADE+vcOmfSPzpFzYCznUInKnjgGtrukJYLI//emQBqytvP4G5dACXBwdeTsg4VuAOKp/C/l73M//vsRzz5xZMs37ecR855pPtv7yQ9kgpaiRwNVbDxfdjwrtMI+pqcwvW8B2Dk9epS3E0+3ryJpvj1nNP7Bt3UXEREQtLoa+TxxY/zRukbnJVexJMpQ8l/92FnRGLrh5g46HsWTL7P6Xo6YBIkprsd2xVhvQ9tJMkogFE3OBNA9S7nIEfZ/GCB69yZIiM5m6cGnMMLuZN5evscSqpK+PmUnzM6d7SL4cULVNCKu2rKnQJ2/TvOLzXrh/S+MPY2p4jtP1G32+lmv1v6F4yx3D/xq25HERERr6qrYMna/+VHpf/LDl8D36qu4dtl84mLWQiF45yD0QPPhf6TIFFn2aDDfWhdTREBMgth1I3OBFC989AZXLN1Ad8o2cqZSYk81MfHbbNv5c5e47hn3PeIzxsJMTHuZpeIpIJWwssGnG5HpR8511TsWuJs7zUEzr0fhv2dcyRXv7BOiaZWH2trPyIj4XRG9jnN7TgiIuIV1Tth22ew7VPqt33KU3Y/MzLSKfT5eT6uLxNHf80ZwKlwPCSkuJ02skV9RXuYzH4w+mZnAji4nfFbP+WNsrk8WbGQ/6lcysfv3MCP6vyM6jvJOUgy4GxnFOVO3iZIejYVtHLq1e+HzX+D0g85Z8N78HENYJzCdfq/wfCrIHeo2ymjwgtLP4b4Cq4svt3tKCIiEqn8Pti3zhl5eMdi2P4ZHNyOBd7L7M0vczLYRzq39ruI75z7Y1Ki5hrY0PSYUY5PtawBMGYAaWNu4RFg+vpX+emyX3NrYh1/X7OaB+bMJisQgNhEKBzbXuD2n9RjRsSWrlFBK92vuQ52LHK6j2z52LmOBgspvanKGUv+uV+D06Yfuo+chM9fSmZCIIF7J17ndhQREYkUdRVO8bpzMexcAruWQWu981hqLgyYzLoxN/JE9UqWHSxheM4Q/mPSQ4zpM8bd3B5jvXsnWldNHX4jEwZfwTMrnuHl9S/z0ZDh3JN/ATc0WxJ2fAELfwufPuXs3Pv09gK3cLyzLj2eCloJXUs9bA8WsFsXwO5lzkiGMXHOdTTTHoLBF0HBGDbMn0/+qKluJ45KOw5Us9e/iEGpZ5Oh65lERKJTSwPsXescbN65mEmln8C8Pc5jMXGQfyacdSv0mwD9J7AZH8+s+h3vb32V7MRsfjT5R1w7+FoNKihhlRqfyvcnfJ+rB1/N44sf5/Ht7/JSWiH3Tn+Ay/tNI7Z8pXMyZfvnzrgsy19ynpiQzuiUgdAy3Tmb23escwZY47P0KCpopWusheodTheknUuco7nlK9sL2L5j4Zz724fiT0h1O7EEPb3wdUxsM98480a3o4iISDi01MOe1bB7hdNWl6+AihJnAEaA9ALq0opJnnKvMwhjwWiITwZgS/UWnl35DH8t+ytJcUl868xvccfIO8hIyHDxA3mbuhyHbkj2EJ7/yvMs3L2Qp5Y9xUMLHuIPWYP55shvcum59xM/JR4CAdi/0TnBsmspsRs+hs9/59z3GCCld3txWzjW+X+flqci18NU0MrxNdc6jWHbdTQ7v4C6vc5jccnOL4LJ90HxFOh/tkYyjGDzds8mLrYXVw+f4nYUERHpbnX7nDOv+9Y5xevuFc4f9W3dXFP7QN8xMOxK5w/4vmMgo5C1H3/M1HOnAs59yhftXshL617ik12fkByXzB0j7+AbZ3yD7KRs1z5aT3GooFXdFBJjDOcUnsPZfc/mg20f8OzKZ3lowUP85/L/5Nbht3LdkOtI6zMM+gyDMV9lWeo8pp43OdgzYZnTpX7XMij9kEM/Hym9IX8k5I10einkjXTGd4mNd/WzSueooJV2dRWwZyWUr4I9q5x51RYO/bBnF8OgqU43pH4TIO8M/aB7xLzNJTTFbeS83rcQYzSCtIiIZzXXwr4NTuG6b12wiF0PDfvb90kvcEaAPeNap3AtGAPp+cespKqbq5m1ZRavlb5G6YFScpJy+PaYb3PT0JvISdIgO91F19B2rxgTw6VFl3LJwEv4ZNcn/HHNH/mPJf/Bb1f8lsuLL+e6IdcxsvdIZ+e44ABShWNhQvAFmuucgz97VsPe1bBnDSz+H/A3B98g3imK8850it3codB7qDMqs45KRBQVtNGouc45altRAhXrnYZxz2qo3d2+T9YAyB8Fo2+BglHOtbAaxMmznln6CsZYvjvpFrejiIjIiVjr9Iaq3AT7S9vnFRvg4Lb2/eJTnT+4h17mHGTuM8KZ0nJP+BatgVYW7V7EHyr+wJpX19AaaGV4znB+eu5Pubz4chJidTsU8QZjDOf3O5/z+53Pmv1reLXkVWaXzeb10tcZmj2UEYxgWP0w8lPzv/zExDQoOteZ2vh9UFnqFLdtRe7mObDyf9v3SUiD3kOc4jb3dMgd5ixnF0GsSis36KveU1kL9RVQVRYsXjcEC9gSqN7evl9sAvQa7HQZzh/lFK/5Z0Kyuhb1FE2tPtbV/I3MhOEMzy1yO46IiIDTTjcecArUqi1QuTlYvJY6y8017fvGJUHOac7B5bFfby9cswZ26b7tzf5mFu5eyIfbPmTejnnUtNSQGpPKTUNv4prB1zA0R7fQO5V0De2pN7L3SEb2HskPJ/yQ2WWzeW3ja8ysmsnM12Yyts9YLim6hOkDph9Z3LaJjYM+w52JG9q311XA/uDf0W1/V5fNh1UzOjw3AXIGOVN2MeQEp+xi50SRejWeMipovSwQgJpdTkN4oMyZV5U504EyaKlr3zc20Rm6fMAkyL3NOZqUO8z5IdPRpB7tj0vnQHwlVxbf5XYUEZHo0lIPB7fDgW1O4Xr4vGPRCpDRD3oPhlE3OWeAeg125hn9ulS4trHWsq1mG4vKF7Fw90IWlS+iwddAenw6U/tP5cKBFxLYHODiiRd30weWzlBv1VMvLSGNG4feyI1Db+SVD1/hYJ+DvLf1PR5b/BiPLX6MwVmDmdJvClMKpzCmzxjiY05QbKblOlPReV/e3lTjHIg6VOyWOn+Db54Lvsb2/UwsZPVvL3Szi52uy5n9nXla3kn9jItDlUykshaaqp2CtXqnMwWXx2xbCyvroGZ3+4ht4PT1zy5yflCKzm3/oek12NmuIfaj0l9K3oRAIt+eeK3bUUREeo7mWqgpd9rmmt0M2PYpvPvWl7Z96bpWcAZTzBoA2QOdOwFkD3TOsuYUO2dgE1JCihSwAcqqy1hVsYrl+5azqHwR5fXlABSmFXLFoCu4aMBFTMifQHzwbNG8LfNCek/pPKtLaF2RF5/HTaNv4h9G/wNbDm5h/s75LNi1gJfWvcQf1/yR5LhkRuWOYlyfcZyVdxajeo8iJb6TP4tJGdBvnDN11HbZwKGTTW0nn8pgzRvQdPDL+8fEQ0bfYJHrTAV7G2BjC2QUQFo+pPRS0XsMKmjDLeCHhkpnNMK6ve3z+org+l6o3es0hh3PsIJzdCejEEhzbhY9orBDl4ZBzmMqWqWD/U1N7Ass5rTk88hI1C2URESOq7XJaY/r90H9fqeNrq9on+r2Qe0eqC0/4uzqIIA92U5bnNEX+p7lnJHJKmovXNP6dNvpOX/Az47aHZQeLKWkqoRVFatYs38Nta21AGQkZDAxfyJ3nXkXkwsm0y+9H0anBiXKDcoaxKCsQdwx8g7qW+tZVL6Iz8s/Z/m+5Tyz8hksljgTx+DswQzPGc6wnGGM6DWC07NP73yRC87PeXq+Mw0858jHGw92OGm1IzgPrm9bCDW7GGr9sPGZDq8Z6/wOSctzpvS8Dsv5zjw11yl8E9OjqiuACtpQBPzOf8jGKuc6mIaqw5YPOOsNwak+2DDawJGvFZ/S/p8093Q4bZpzhCajMNgdoTDYHSGWFfPmMXXq1LB/XPGet/YuxcS08I3RuvesiESRQMApOBsPOGdCGg90mA62z5sOOoVrWwF7eBfgNvGpTnfD1FynC/Cgqc5Zk7biNb2A+Ss2cf6Fl3T7R6lvrWdn7U521O5gR+0ONh3cROmBUrZUb6E5OBprjInh9OzTuaz4MkbljmJU7igGZgzUqPYRpm2U4+gpMyJbanwqFw64kAsHXAhAbUstK/atYNm+ZayrXMe8HfOYuWkmAAbDgIwBFGcUU5RZRFFG0aF5TlJO1w8WJWc5U94ZR3884GfhBzOZPKK/cwCtbh/U7Wk/8VW727mv9LHqiph4p7BNyTlsHpyS29azISkLEjMgKRPivDkYXPQWtL5m4lsOtg+80FzbPjXVHLnt0FTToZGsPvbrmxhnYKXkbOc/TWYhFJ7VfiQlNTe4HCxidf9WOQXWNH9OXFwfrh52lKODIiKRJhAg1tfgdNtta3db2trfusPWj7Ktqbq9UD3aH3lt4lM6tNHZztnU1Nz2Ka1Ph/XekHDiHi6B2B1d/rgt/hYqGiuoaKigorGCfQ37qGioYE/DHnbU7mBn7U6qmqq+9Jzc5FyGZA/hpvybGJI9hCHZQzgt8zSS4pK6/P4SXupyHNnSE9Kd62r7TQGca9D3NuxlQ9UG1letp/RAKWXVZXy2+zNaAu2X/KXGp1KQWkB+aj4FqQX0TetLfmo++Sn59E7uTU5yDunx6V0remNiaU7q7Yx9czx+n3NpQ1uhW1/h9ARtqOxwUq0yeGuvSmf9eLePikt2ulEnZbYXuUmZR25LTHd+LyakOVNi2qF1E2jt/OfsJt1S0BpjLgWeBmKB5621jx/2eCLwJ2AcUAncZK3dGnzsX4E7AT9wv7X2/e7IdEILnuLcz34Onx1nn5h45xuYmB6cMp2jsYlD2wvV5GznqMeh5WDjmJipfu7iqrmb1+FLLOO8nK+rm5lIFPJk2zzv50xZ8AtYcIL9YuLa2+aE4Dyll3MZTkqwPU7K+nLRemjKcu5J2U38AT+NvkaqfdVsq9lGfWs9NS01VDdXU91c/aXl6uZqqluceUVjBdXNRx4Yj4uJo09yH/qn92da/2n0S+9H//T+9EvvR7+0fmQmZnZbdgkv1bPeYoxxCtPUfKb2n3pouz/gp7y+nK01W9lavZVddbsory9nd91u1u5fy4HmA0e8VnxMPDlJOeQk5dAruRc5STlkJ2aTnpBOWkIaGQkZznJ8GukJ6WQkZNAYaCRgA8fvaREb1961uaATHyrgdw78tRW3DZXOibqmGmd7c7Uzb1tvOugMYNe23naP3uM4NzYFppd3Ikz3CbmgNcbEAr8FLgZ2Al8YY9621q7rsNudwAFr7WBjzM3AE8BNxpgRwM3AGUBf4CNjzOnWWn+ouU5oyEVs3FXJ6SPHdShY04MFbLCI7cYGTyTcfrf0Faw1PDD5q25HEZEw82rbHBg0jY079lA0fBSB+FT8CSkEElLxJyQTiE/BF5dMICEZf0wcAQL4rZ9AIDi3AQK2fblt7gv4nPWWfbQ27aJ1fystgRZa/c68xd9Ca6D1S+uHtgVaafG30OxvptHXSIOvgYbWBhp9jc56awNN/qb2DzDz6J8rMTaRzMRMZ0rIZED6AMbljSM3OZc+KX3ITcklNzmX3JRcshKz1FW4h9MxZm+LjYl1DjCl9+O8wvOOeLzR18ie+j2U15dT2VhJVVMVVU1Vh5YrmyrZdHATB5sOfvn3x1H88E8/JDE2keS4ZJLikpx5rDNv29a2PTE2kfiY+PYp1pnHxcQdse3QlJlDnOlDjIkhxsQQa2KJiQnOzVHmfh8xrfXEtjRiWhuJ9TUS09pEbEs9Ma0NxLY0UrZpE+G+AVh3nKGdCGyy1m4BMMbMAK4GOjaaVwM/Di6/BvyXcU4ZXQ3MsNY2A2XGmE3B11vYDbmO66UdvfmvzVNJKW+7wNsPHAxOka2hoYGUJfPcjnFSvJrda7kD1s/+7Dmk2dMZ2ruf23FEJPw82Tb/w6p5LOJjWP/xqX6ro7OxQCzGxmFsHNBxnhicUjA2+9B6ik0Am0hrsyExPh1jE4ixKZhAKsamEBNIwZBAADgQnI7kA8qDU3h5rX3ryGvZm1qdY0KqZ3u25LhkijOLKc4sPuG+rf5WaltrqWupo7allpqWGupaneXl65eTPyCfZl8zDb4GmnxNNPmbaPI10ehrpLalln2N+w6tdzwQ5wv4wvBJjy4lJoXPw/ye3VHQFgIdLxzZCRze4fvQPtZanzGmGugV3L7osOcWHu1NjDF3A3cD5OXlMW/evJBC79vro19qgPi44x8ZiUStHs0N3s3utdx1cRupiq/mosTLQ/5ZcUNdXZ1yh5Fy90iebJsz63rTq/4rxMbEYogBDMbGADHB9RjMoW3mS9sg5rDtwbnt+FynWI1pK1qJw9hYZ07be56cVusjPtDhz6q2l4oNAJHbfnitfevIc9njYUh6HHlxTZ783eXV37leyx1PPDnkMClmEmkHjzLGTmxwOk5H0oAN4MeP3zqTz/rwE5xbPz58+KwzBWwAiyVA4IjlAMH1tmUbfOw42wMtgbB/vT0zKJS19jngOYDx48fbUEf5nQqM8+howfM8mhu8m91ruR+cP5eqXelMzZvgqdxtvPb1bqPc4eXV3D1J97fNUz37fVXu8PNqduUOL+UOLzdyd8dFGruA/h3W+wW3HXUfY0wckIkzAEVnnisiXVDTUsOc7XO4vPhy4k2823FExB1qm0VEJCp0R0H7BTDEGFNsjEnAGUji7cP2eRu4Pbh8PfA3a60Nbr/ZGJNojCkGhgCLuyGTSNR6r+w9mv3NXDv4WrejiIh71DaLiEhUCLnLcfC6m/uA93F6dP/BWrvWGPMIsMRa+zbwe+Cl4MASVTgNK8H9XsUZpMIH3BuWEY5FerC3Nr/F4KzBjOg1go9xaWAVEXGV2mYREYkW3XINrbV2NjD7sG3/3mG5CbjhGM99FHi0O3KIRLstB7ewqmIV3x//fd17ViTKqW0WEZFooBudifQgb25+k1gTyxWDrnA7ioiIiIjIKaeCVqSH8AV8vLP5HaYUTqF3cm+344iIiIiInHIqaEV6iM92f8b+xv1cM+Qat6OIiIiIiISFClqRHuLNTW+Sk5TD+f3OdzuKiIiIiEhYqKAV6QEONB1g7o65XDHoCuJjdO9ZEREREYkOKmhFeoBZW2bhC/i4ZrC6G4uIiIhI9FBBK9IDvLnpTc7odQanZ5/udhQRERERkbBRQSvicesr11NyoERnZ0VEREQk6qigFfG4Nze9SUJMApcVX+Z2FBERERGRsFJBK+JhLf4WZpXNYvqA6WQmZrodR0REREQkrFTQinjY3B1zqW6uVndjEREREYlKKmhFPOzNTW+Sl5LH2QVnux1FRERERCTsVNCKeNTe+r18tvszrjrtKmJjYt2OIyIiIiISdipoRTzqnS3vELABdTcWERERkailglbEg6y1vLXpLcb2GcuAjAFuxxERERERcYUKWhEPWlGxgq01W3V2VkRERESimgpaEQ96c9ObJMclc0nRJW5HERERERFxjQpaEY9paG3gvbL3uKToElLiU9yOIyIiIiLiGhW0Ih7z17K/0uBr4Loh17kdRURERETEVSpoRTzmtY2vMThrMKNzR7sdRURERETEVSpoRTykpKqENZVruP706zHGuB1HRERERMRVIRW0xpgcY8yHxpjS4Dz7KPuMMcYsNMasNcasMsbc1OGxF4wxZcaYFcFpTCh5RHq61za+RkJMAlcOutLtKCISodQ2i4hINAn1DO2/AHOstUOAOcH1wzUAt1lrzwAuBZ4yxmR1ePwH1toxwWlFiHlEeqxGXyOztszi4qKLyUzMdDuOiEQutc0iIhI1Qi1orwZeDC6/CBxxU0xr7UZrbWlweTewD8gN8X1Fos4HWz+gtrVWg0GJyImobRYRkagRakGbZ60tDy7vAfKOt7MxZiKQAGzusPnRYHenXxtjEkPMI9JjvV76OkUZRYzPG+92FBGJbGqbRUQkahhr7fF3MOYjIP8oDz0MvGitzeqw7wFr7RHX6gQfKwDmAbdbaxd12LYHpyF9DthsrX3kGM+/G7gbIC8vb9yMGTOO/8k6oa6ujrS0tJBfJ9y8mhu8m93t3OUt5fy8/Odck3UNF2Ze2OnnuZ37ZCl3eCk3TJs2bam11jNHi9Q2Rx7lDj+vZlfu8FLu8HKlbbbWnvQElAAFweUCoOQY+2UAy4Drj/NaU4F3O/O+48aNs91h7ty53fI64ebV3NZ6N7vbuR///HE75k9jbGVjZZee53buk6Xc4aXc1gJLbAjtYSRNapvdodzh59Xsyh1eyh1ebrTNoXY5fhu4md+5zgAAF2lJREFUPbh8O/DW4TsYYxKAmcCfrLWvHfZYQXBucK7xWRNiHpEep9nfzDtb3uHCAReSk5TjdhwRiXxqm0VEJGqEWtA+DlxsjCkFLgquY4wZb4x5PrjPjcD5wB1HuQXAn40xq4HVQG/gZyHmEelxPtr2EdXN1RoMSkQ6S22ziIhEjbhQnmytrQSOuKDPWrsEuCu4/DLw8jGePz2U9xeJBq+WvEq/tH5MKpjkdhQR8QC1zSIiEk1CPUMrIqdQSVUJy/Yt4+ZhNxNj9OMqIiIiItKR/kIWiWAzSmaQGJvINYOPuI2kiIiIiEjUU0ErEqFqW2qZtWUWlxVfRmZipttxREREREQijgpakQj19ua3afQ1cvOwm92OIiIiIiISkVTQikQgay0zNsxgVO9RnNHrDLfjiIiIiIhEJBW0IhFoUfkittZs1dlZEREREZHjUEErEoFmbJhBdmI2Xyn6ittRREREREQilgpakQhTXlfOvJ3zuHbItSTGJrodR0REREQkYqmgFYkwr258FWstNw690e0oIiIiIiIRTQWtSARp9DXyl41/YfqA6RSmFbodR0REREQkoqmgFYkg72x+h+rmar4+4utuRxERERERiXgqaEUiRMAGeGndS4zoNYKxfca6HUdEREREJOKpoBWJEAt2LWBrzVZuG3Ebxhi344iIiIiIRDwVtCIR4k/r/kSflD66VY+IiIiISCepoBWJACVVJXxe/jm3DLuF+Jh4t+OIiIiIiHiCClqRCPDy+pdJjkvmhtNvcDuKiIiIiIhnqKAVcdn+xv3M2jKLq067iszETLfjiIiIiIh4hgpaEZf9ad2f8Fs/t424ze0oIiIiIiKeooJWxEU1LTW8WvIqlwy8hAEZA9yOIyIiIiLiKSpoRVw0Y8MM6lvrufPMO92OIiIiIiLiOSpoRVzS6Gvk5XUvM6VwCkNzhrodR0RERETEc1TQirjkjdI3ONB8gLvOvMvtKCIiIiIinhRSQWuMyTHGfGiMKQ3Os4+xn98YsyI4vd1he7Ex5nNjzCZjzCvGmIRQ8oh4Rau/lRfWvsDYPmMZmzfW7Tgi0oOobRYRkWgS6hnafwHmWGuHAHOC60fTaK0dE5yu6rD9CeDX1trBwAFAFxJKVJhVNos99Xt0dlZETgW1zSIiEjVCLWivBl4MLr8IXNPZJxpjDDAdeO1kni/iVb6Aj+dXP8+wnGGcV3ie23FEpOdR2ywiIlEj1II2z1pbHlzeA+QdY78kY8wSY8wiY0xbw9gLOGit9QXXdwKFIeYRiXiztsxiW8027hl9D87fjiIi3Upts4iIRA1jrT3+DsZ8BOQf5aGHgRettVkd9j1grT3iWh1jTKG1dpcxZhDwN+BCoBpYFOzShDGmP/BXa+3IY+S4G7gbIC8vb9yMGTM68/mOq66ujrS0tJBfJ9y8mhu8m727cvutn5/t/hnJMcn8IP8Hp7ygjfavd7gpd3h1Z+5p06YttdaO75YXCwO1zZFHucPPq9mVO7yUO7xcaZuttSc9ASVAQXC5ACjpxHNeAK4HDLAfiAtunwy835n3HTdunO0Oc+fO7ZbXCTev5rbWu9m7K/drJa/ZkS+MtB/v+LhbXu9Eov3rHW7KHV7dmRtYYkNoDyNpUtvsDuUOP69mV+7wUu7wcqNtDrXL8dvA7cHl24G3Dt/BGJNtjEkMLvcGzgXWBUPODTagx3y+SE/R6m/l2VXPcmbvM5lSOMXtOCLSc6ltFhGRqBFqQfs4cLExphS4KLiOMWa8Meb54D7DgSXGmJU4jeTj1tp1wcceBL5njNmEc93O70PMIxKxZm6aSXl9OfeOuVfXzorIqaS2WUREokZcKE+21lbiXHNz+PYlwF3B5c+AM4/x/C3AxFAyiHhBk6+JZ1c9y5jcMZzT9xy344hID6a2WUREokmoZ2hFpBNeXv8y+xr2cf/Y+3V2VkRERESkm6igFTnFqpqq+P3q33NBvwuYkD/B7TgiIiIiIj2GClqRU+y5Vc/R4Gvgn8b9k9tRRERERER6FBW0IqfQ9prtvLLhFa4dfC2nZZ3mdhwRERERkR5FBa3IKfT0sqeJj43n3jH3uh1FRERERKTHUUErcoos37ecD7Z9wB1n3EFuSq7bcUREREREehwVtCKngD/g5+ef/5y8lDzuOOMOt+OIiIiIiPRIKmhFToFXN77KhqoNfH/C90mJT3E7joiIiIhIj6SCVqSbVTVV8Zvlv2FS/iQuGXiJ23FERERERHosFbQi3ezpZU/T2NrIQ5MewhjjdhwRERERkR5LBa1IN1pVsYo3St/g1hG3MihrkNtxRERERER6NBW0It2k1d/Kjxf+mD7JffjH0f/odhwRERERkR4vzu0AIj3F79f8ntIDpfxm+m9IjU91O46IiIiISI+nM7Qi3WDzwc08u+pZLiu6jKn9p7odR0REREQkKqigFQmRP+DnR5/9iLT4NB6c+KDbcUREREREooYKWpEQvbTuJVZWrOTBiQ/SK7mX23FERERERKKGClqREGyo2sDTy59mev/pXFF8hdtxRERERESiigpakZPU5GviwfkPkp2YzY/P+bHuOSsiIiIiEmYa5VjkJP1q6a/YUr2FZy96luykbLfjiIiIiIhEHZ2hFTkJ83bM4/82/B+3Dr+VcwrPcTuOiIiIiEhUUkEr0kU7anfw0CcPMTxnOA+Me8DtOCIiIiIiUUsFrUgXNPub+ed5/wwGfjX1VyTGJrodSUREREQkaoVU0BpjcowxHxpjSoPzIy4kNMZMM8as6DA1GWOuCT72gjGmrMNjY0LJI3KqPfb5Y6yvWs9j5z1Gv/R+bscRETmC2mYREYkmoZ6h/RdgjrV2CDAnuP4l1tq51tox1toxwHSgAfigwy4/aHvcWrsixDwip8yMDTN4vfR17jrzLi7of4HbcUREjkVts4iIRI1QC9qrgReDyy8C15xg/+uBv1prG0J8X5GwWt+4nscXP84F/S7gvjH3uR1HROR41DaLiEjUCLWgzbPWlgeX9wB5J9j/ZuD/Dtv2qDFmlTHm18YYXZAoEWfzwc38oeIPnJZ1Gk+c/wSxMbFuRxIROR61zSIiEjWMtfb4OxjzEZB/lIceBl601mZ12PeAtfaoN+Q0xhQAq4C+1trWDtv2AAnAc8Bma+0jx3j+3cDdAHl5eeNmzJhxgo92YnV1daSlpYX8OuHm1dzgvezVvmp+vffXNPub+UHfH5ATl+N2pC7x2te7jXKHl3LDtGnTllprx3fLi4WB2ubIo9zh59Xsyh1eyh1errTN1tqTnoASoCC4XACUHGff7wLPHefxqcC7nXnfcePG2e4wd+7cbnmdcPNqbmu9lf1g00F7zZvX2AkvT7B/fO+Pbsc5KV76enek3OGl3NYCS2wI7WEkTWqb3aHc4efV7ModXsodXm60zaF2OX4buD24fDvw1nH2vYXDujQFjwJjjDE41/isCTGPSLdoaG3g3jn3sq1mG09Pe5qixCK3I4mIdJbaZhERiRqhFrSPAxcbY0qBi4LrGGPGG2Oeb9vJGFME9Ac+Puz5fzbGrAZWA72Bn4WYRyRkTb4m/mneP7F6/2qePP9JJved7HYkEZGuUNssIiJRIy6UJ1trK4ELj7J9CXBXh/WtQOFR9pseyvuLdLeG1gbun3s/i8sX85NzfsJFAy9yO5KISJeobRYRkWgSUkEr0pPUt9Zz75x7Wb5vOY+e9yh/d9rfuR1JRERERESOQwWtCFDVVMV35nyHtZVreey8x7h80OVuRxIRERERkRNQQStRb3vNdu756B72Nuzllxf8kgsHHtFTT0REREREIpAKWolqKytW8p0538Fief4rzzOmzxi3I4mIiIiISCeFOsqxiGfNLJ3Jne/fSVpCGi9f/rKKWRERERERj9EZWok6zf5mHl/8OK9tfI1J+ZN48oInyUnKcTuWiIiIiIh0kQpaiSo7anbww/k/ZE3lGu4ceSf3nXUfcTH6MRARERER8SL9JS9RwVrL66Wv8+QXTxJn4nhq6lMa/ElERERExONU0EqPV9FQwU8W/oSPd37MpPxJ/Oy8n5Gfmu92LBERERERCZEKWumx/AE/r5S8wm+W/4bWQCsPTniQrw7/KjFGY6GJiIiIiPQEKmilR1pdsZqfff4z1lWuY3LBZB4++2EGZgx0O5aIiIiIiHQjFbTSo2yt3sp/Lv9PPtz2IbnJufzi/F9wSdElGGPcjiYiIiIiIt1MBa30CLvrdvM/q/+HmaUzSYhN4J7R93D7GbeTGp/qdjQRERERETlFVNCKp5UeKOWPa/7I7LLZGAw3Dr2Ru0fdTe/k3m5HExERERGRU0wFrXiOP+Bnwa4FvFLyCp/s+oTkuGRuGXYLt424jYK0ArfjiYiIiIhImKigFc/YU7+HmaUzeb30dfY27KV3cm++Pfrb3DLsFrKSstyOJyIiIiIiYaaCViJaZWMlH277kPe2vseyvcuwWM7pew4PTnyQqf2nEh8T73ZEERERERFxiQpaiSjWWspqyvhk5yd8svMTvtj7BQEbYFDmIO4Zcw9XDrqS/un93Y4pIiIiIiIRQAWtuG5/436W7V3G4j2LWbBrAbvqdgEwOGswd468k0uLL2VI1hDdekdERERERL5EBa2Eld/6KakqYV3lOlZUrGDp3qVsq9kGQHJcMpPyJ/HNkd/kvMLz6JvW1+W0IiIiIiISyVTQyilhraWyqZKy6jLKqstYX7We9ZXrKakswbfdB0B6Qjrj+ozj+iHXMzZvLMN7Ddc1sSIiIiIi0mkqaOWk+QI+KhoqKK8vp7y+nF11u9havZWtNVvZWr2V2tbaQ/umJ6QzImcEF2RcwFdGf4VhvYZRlFFEjIlx8ROIiIiIiIiXhVTQGmNuAH4MDAcmWmuXHGO/S4GngVjgeWvt48HtxcAMoBewFPi6tbYllEwSGmstDb4GqhqrqGyqpKqpigNNB6hqqqKqqYrKxkr2NOyhvL6cfQ37CNjAl57fJ6UPxRnFXD7ocooziynKKKI4s5iC1AKMMcybN4+pg6a68tlERKKB2mYREYkmoZ6hXQP8PfDssXYwxsQCvwUuBnYCXxhj3rbWrgOeAH5trZ1hjPkdcCfwTIiZooa1ltZAK03+Jpp9zTT5mpxlfzONvkaa/c3Odn8TTb4m6lvrqWuto7allrrWOupa6qhtraWupa59e0sdLYGj/92SGp9KTlIOBakFTMyfSH5qPgWpBYem/NR8UuJTwvxVEBGRw6htFhGRqBFSQWutXQ+caPTZicAma+2W4L4zgKuNMeuB6cBXg/u9iHNEOSyN5oaqDXxW+xkVGyuw1hKwAQI2gMUeWrd8eXvABpzHCBz9OUfZ3vacjq/lt358Ad+XplbbesS2Q5MN7hNw9mlqacL+2dLka8Jiu/zZU+JSSEtIIz0+nbSENLKSsuif3v/QtpykHLKTsslJyiEnOYdeSb3ITsomMTbxFHwnRESkO3m5bRYREekqY23XC6IjXsSYecD3j9atyRhzPXCptfau4PrXgUk4DeQia+3g4Pb+wF+ttSOP8R53A3cD5OXljZsxY0ZImd+vfp93D74b0muY4D+AGGIwxvClf4evB7fFEEOsiSWW2EPzGHPktljjTIfvH/AFSIpPIsEkEB8TT7yJd5Y7zNumhJj29SSTRFJMkqvXrdbV1ZGWluba+58s5Q4v5Q4v5YZp06YttdaO75YXixBebJtB/x/Dzau5wbvZlTu8lDu83GibT3iG1hjzEZB/lIcetta+dTLhToa19jngOYDx48fbqVOnhvR641rGMXH+RM4951xiTAwGc2hujLPccfvhj7Utu2HevHmE+vnd4tXsyh1eyh1eyu09PbVtBu9+X5U7/LyaXbnDS7nDy43cJyxorbUXhfgeu4D+Hdb7BbdVAlnGmDhrra/D9rBIT0gnOy6bPil9wvWWIiIi3aKnts0iIiJdFY6+p18AQ4wxxcaYBOBm4G3r9HWeC1wf3O92IGxHlUVERKKY2mYREekRQipojTHXGmN2ApOBWcaY94Pb+xpjZgMEj/DeB7wPrAdetdauDb7Eg8D3jDGbcG4P8PtQ8oiIiEQ7tc0iIhJNQh3leCYw8yjbdwOXd1ifDcw+yn5bcEZaFBERkW6gtllERKKJe8PdioiIiIiIiIRABa2IiIiIiIh4kgpaERERERER8SQVtCIiIiIiIuJJKmhFRERERETEk1TQioiIiIiIiCepoBURERERERFPMtZatzN0mTGmAtjWDS/VG9jfDa8Tbl7NDd7Nrtzhpdzhpdww0Fqb202vFZXUNiu3C7yaXbnDS7nDK+xtsycL2u5ijFlirR3vdo6u8mpu8G525Q4v5Q4v5ZZI4tXvq3KHn1ezK3d4KXd4uZFbXY5FRERERETEk1TQioiIiIiIiCdFe0H7nNsBTpJXc4N3syt3eCl3eCm3RBKvfl+VO/y8ml25w0u5wyvsuaP6GloRERERERHxrmg/QysiIiIiIiIepYJWREREREREPEkFbZAx5p+NMdYY09vtLJ1hjPmpMWaVMWaFMeYDY0xftzN1hjHmF8aYDcHsM40xWW5n6gxjzA3GmLXGmIAxJuKHUDfGXGqMKTHGbDLG/IvbeTrLGPMHY8w+Y8wat7N0hTGmvzFmrjFmXfD/yXfdztQZxpgkY8xiY8zKYO6fuJ2pK4wxscaY5caYd93OIqeG2ubwUNscHmqbw0ttc/i51S6roMX5Dw98BdjudpYu+IW1dpS1dgzwLvDvbgfqpA+BkdbaUcBG4F9dztNZa4C/B+a7HeREjDGxwG+By4ARwC3GmBHupuq0F4BL3Q5xEnzAP1trRwBnA/d65GveDEy31o4GxgCXGmPOdjlTV3wXWO92CDk11DaHldrmU0xtsyvUNoefK+2yClrHr4EfAp4ZIctaW9NhNRWPZLfWfmCt9QVXFwH93MzTWdba9dbaErdzdNJEYJO1dou1tgWYAVztcqZOsdbOB6rcztFV1tpya+2y4HItzi/zQndTnZh11AVX44OTJ36XGGP6AVcAz7udRU4Ztc1horY5LNQ2h5na5vBys12O+oLWGHM1sMtau9LtLF1ljHnUGLMD+BreOQrc0TeBv7odogcqBHZ0WN+JB36B9xTGmCLgLOBzd5N0TrB70ApgH/ChtdYTuYGncIqdgNtBpPupbXaV2uZTQ22zi9Q2h4Vr7XJcuN/QDcaYj4D8ozz0MPAQTpemiHO83Nbat6y1DwMPG2P+FbgP+FFYAx7DiXIH93kYpyvIn8OZ7Xg6k1vkeIwxacDrwAOHnamJWNZaPzAmeM3cTGPMSGttRF8nZYy5EthnrV1qjJnqdh45OWqbw0tts0Qrtc2nntvtclQUtNbai4623RhzJlAMrDTGgNPFZpkxZqK1dk8YIx7VsXIfxZ+B2URIo3mi3MaYO4ArgQttBN0IuQtf70i3C+jfYb1fcJucQsaYeJwG88/W2jfcztNV1tqDxpi5ONdJRWyjGXQucJUx5nIgCcgwxrxsrb3V5VzSBWqbw0tts+vUNrtAbXPYuNouR3WXY2vtamttH2ttkbW2CKf7x9hIaDBPxBgzpMPq1cAGt7J0hTHmUpzuCFdZaxvcztNDfQEMMcYUG2MSgJuBt13O1KMZ56/u3wPrrbW/cjtPZxljcttGMzXGJAMX44HfJdbaf7XW9gv+3r4Z+JuK2Z5DbXP4qW0OC7XNYaa2OXzcbpejuqD1uMeNMWuMMatwumV5Yihy4L+AdODD4G0Nfud2oM4wxlxrjNkJTAZmGWPedzvTsQQH9rgPeB9nAIRXrbVr3U3VOcaY/wMWAkONMTuNMXe6namTzgW+DkwP/r9eETxKGekKgLnB3yNf4Fyno1vgiJw8tc1hpLY5PNQ2h53a5i4yEdSrRERERERERKTTdIZWREREREREPEkFrYiIiIiIiHiSCloRERERERHxJBW0IiIiIiIi4kkqaEVERERERMSTVNCKiIiIiIiIJ6mgFREREREREU/6/8OQt4m2Cn7GAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"x = np.linspace(-4, 4, num=1024)\n",
"_, axes = plt.subplots(nrows=1, ncols=2, figsize=(16,5))\n",
"axes[0].plot(x, relu(x), x, sigmoid(x), x, tanh(x))\n",
"axes[1].plot(x, g_relu(x), x, g_sigmoid(x), x, g_tanh(x))\n",
"legend_entries = (r'\\mathrm{ReLU}(x)', r'\\sigma(x)', r'\\mathrm{tanh}(x)')\n",
"for ax, legend_prefix in zip(axes, ('', r'\\frac{\\partial}{\\partial x}')):\n",
" ax.grid(True)\n",
" ax.legend(tuple(f'${legend_prefix}{legend_entry}$' for legend_entry in legend_entries))\n",
" ax.set_ylim((-1.1,1.1))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Some reasons for using ReLU are:\n",
"\n",
"- Mitigates vanishing gradients due to many layers (even though they can still be zero).\n",
"- Promotes sparse weight vectors: \"dead neurons\" arguably cause sparsity in the next layer.\n",
"- Much faster to compute than sigmoid and tanh."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## 2-Layer MLP from scratch"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Let's solve a simple **regression** problem with a 2-layer MLP (one hidden layer, one output layer)."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"We're trying to learn a continuous and perhaps non-deterministic function $y=f(\\vec{x})$.\n",
"\n",
"- Domain: $\\vec{x}^i \\in \\set{R}^{D_{\\text{in}}}$\n",
"- Target: $y^i \\in \\set{R}^{D_{\\text{out}}}$\n",
"- Model: $\\hat{y} = \\varphi(\\mat{X}\\mat{W}_1 + \\vec{b}_1)\\mat{W}_2 + \\vec{b}_2$ (2-layer MLP), where:\n",
" - $\\mat{X}$ is the $(N,D_{\\text{in}})$ matrix with samples in it's rows\n",
" - $\\mat{W}_1\\in\\set{R}^{D_{\\text{in}}\\times H},\\ \\vec{b}_1\\in\\set{R}^{H}$\n",
" - $\\mat{W}_2\\in\\set{R}^{H\\times D_{\\text{out}}},\\ \\vec{b}_2\\in\\set{R}^{D_{\\text{out}}}$\n",
" - $\\varphi(\\cdot) = \\mathrm{ReLU}(\\cdot) = \\max\\{\\cdot,0\\}$\n",
" - $H$ is the hidden dimension\n",
" - We'll set $D_{\\text{out}}=1$ so output is a vector\n",
"- MSE loss with L2 regularization:\n",
" $$\n",
" L_{\\mathcal{D}}(h) =\n",
" \\frac{1}{N}\\norm{\\hat{\\vec{y}} - \\vec{y}}_2^2 + \\frac{\\lambda}{2}\\left(\\norm{\\mat{W}_1}_F^2 + \\norm{\\mat{W}_2}_F^2 \\right)\n",
" $$\n",
"- Optimization scheme: Vanilla SGD"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Computing the loss gradients with backpropagation"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Ignoring regularization, we define for brevity, $\\delta X \\triangleq \\pderiv{L(h)}{X}$."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"We can now apply the chain rule and write\n",
"$$\n",
"\\begin{align}\n",
"\\delta \\hat{\\vec{y}} &= \\pderiv{L}{\\hat{\\vec{y}}} = \\frac{2}{N}\\left(\\hat{\\vec{y}} - \\vec{y}\\right) \\\\\n",
"\\delta \\mat{W}_2 &= \\delta\\hat{\\vec{y}} \\pderiv{\\hat{\\vec{y}}}{\\mat{W}_2} = \\mattr{A}_b \\delta\\hat{\\vec{y}}\\\\\n",
"\\delta\\mat{A}_b &= \\delta\\hat{\\vec{y}} \\pderiv{\\hat{\\vec{y}}}{\\mat{A}_b} = \\delta\\hat{\\vec{y}} \\mattr{W}_2 \\\\\n",
"\\delta\\mat{Z} &= \\delta\\mat{A}\\pderiv{\\mat{A}}{\\mat{Z}} = \\delta\\mat{A}\\odot\\mathbb{1}(\\mat{Z}>0) \\\\\n",
"\\delta\\mat{W}_1 &= \\delta\\mat{Z}\\pderiv{\\mat{Z}}{\\mat{W}_1} = \\mattr{X}_b \\delta\\mat{Z}\n",
"\\end{align}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"The final gradients for weight update, including regularization will be $\\delta\\mat{W}_i + \\lambda\\mat{W}_i$."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# A simple MLP with one hidden layer\n",
"\n",
"# N: batch size\n",
"# D_in: number of features\n",
"N, D_in = 64, 10\n",
"# H: hidden-layer\n",
"# D_out: output dimension\n",
"H, D_out = 100, 1\n",
"\n",
"# Random input data\n",
"X = np.random.randn(N, D_in)\n",
"y = np.random.randn(N, D_out)\n",
"\n",
"# Model weights (note: bias included)\n",
"W1 = np.random.randn(D_in+1, H)\n",
"W2 = np.random.randn(H+1, D_out)\n",
"\n",
"reg_lambda = 0.5\n",
"learning_rate = 1e-3"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
".........................................................................................................................................................................................................................................................."
]
}
],
"source": [
"losses = []\n",
"for epoch in range(250):\n",
" # Forward pass, hidden layer: A = relu(Xb W1), Shape: (N, H+1)\n",
" Xb = np.hstack((np.ones((N,1)), X))\n",
" Z = Xb.dot(W1)\n",
" A = np.maximum(Z, 0)\n",
" # Forward pass, output layer: Y_hat = A W2, Shape: (N, D_out)\n",
" Ab = np.hstack((np.ones((N,1)), A))\n",
" Y_hat = Ab.dot(W2)\n",
" \n",
" # Loss calculation (MSE)\n",
" loss = np.mean((Y_hat - y) ** 2); losses.append(loss)\n",
" \n",
" # Backward pass: Output layer\n",
" d_Y_hat = (2./N) * (Y_hat - y)\n",
" d_W2 = Ab.transpose().dot(d_Y_hat)\n",
" d_Ab = d_Y_hat.dot(W2.transpose())\n",
" # Backward pass: Hidden layer\n",
" d_A = d_Ab[:,1:] # remove bias col\n",
" d_Z = d_A * np.array(Z > 0, dtype=np.float)\n",
" d_W1 = Xb.transpose().dot(d_Z)\n",
" # Backward pass: Regularization term\n",
" d_W2 += reg_lambda * W2\n",
" d_W1 += reg_lambda * W1\n",
" \n",
" # Gradient descent step\n",
" W2 -= d_W2 * learning_rate\n",
" W1 -= d_W1 * learning_rate\n",
" \n",
" print('.', end='')"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"scrolled": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAFACAYAAAD589sCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XuUXWd55/nvcy5VJamqJLlUkmVJtnwRENs0Mq0Ye0iyuITEdiZtMpMmJiG4iacN3SYDk6x0CGv16mSt0E1PDyFNJ2GNaRMMTbhMEoI7MQ6MgdBMACNjYyQ7DsKWbUm2JMu6X0p1eeaPs0t1JJWqSlLtc46qvp+Vs87e795nn6e0c5xf3r3fd0dmIkmSpM5QaXcBkiRJmmA4kyRJ6iCGM0mSpA5iOJMkSeoghjNJkqQOYjiTJEnqIIYzSZKkDmI4kyRJ6iCGM0mSpA5Sa3cB52PZsmW5du3adpchSZI0rYcffvjFzBycbr8LOpytXbuWjRs3trsMSZKkaUXEMzPZz8uakiRJHaT0cBYR1Yh4JCL+uli/PCK+ExFbIuJzEdFVtHcX61uK7WvLrk2SJKnTtKLn7D3AE03r/xH4cGZeBewF7ija7wD2Fu0fLvaTJEmaV0oNZxGxGvg54L8W6wG8AfjzYpd7gTcXy7cW6xTb31jsL0mSNG+U3XP2h8C/AcaK9QFgX2aOFOvbgFXF8irgOYBi+/5i/5NExJ0RsTEiNu7evbvM2iVJklqutHAWEf8zsCszH57N42bm3Zm5ITM3DA5OOxpVkiTpglLmVBqvBf5ZRNwC9AD9wH8GlkREregdWw1sL/bfDqwBtkVEDVgM7CmxPkmSpI5TWs9ZZv5OZq7OzLXAbcBXM/NXgK8Bv1jsdjvwxWL5vmKdYvtXMzPLqk+SJKkTtWOes98GfiMittC4p+yeov0eYKBo/w3gfW2oTZIkqa1a8oSAzPw68PVi+Sng+kn2OQb881bUM1MvHT7OA5te4CfXLWPNRQvbXY4kSZoHfELAFJ7ff5T3f+EHbN6xv92lSJKkecJwNoV6tfHPMzzqrW+SJKk1DGdTqFUac+COjI1Ns6ckSdLsMJxNwZ4zSZLUaoazKdSqRc+Z4UySJLWI4WwKtUrjn8fLmpIkqVUMZ1OoFz1nXtaUJEmtYjibQq2452xk1J4zSZLUGoazKUyM1rTnTJIktYbhbAr1Ez1nhjNJktQahrMpVCtBhAMCJElS6xjOplGvVBwQIEmSWsZwNo1aNRwQIEmSWsZwNo1aJRwQIEmSWsZwNo16tcKwPWeSJKlFDGfTaFzWtOdMkiS1huFsGrVKhWFHa0qSpBYxnE2jbs+ZJElqIcPZNGrVivOcSZKkljGcTaNWCec5kyRJLWM4m0a9WnGeM0mS1DKGs2nUqs5zJkmSWsdwNo3G45vsOZMkSa1RWjiLiJ6IeCgivh8RmyPi94r2T0TE0xHxaPFaX7RHRHwkIrZExGMR8eqyajsbznMmSZJaqVbisYeAN2TmoYioA9+MiC8V234rM//8lP1vBtYVr9cAHy3e26pWrXD4+Gi7y5AkSfNEaT1n2XCoWK0Xr6m6oG4FPll87tvAkohYWVZ9M1Wv+OBzSZLUOqXecxYR1Yh4FNgFfCUzv1Ns+kBx6fLDEdFdtK0Cnmv6+Lai7dRj3hkRGyNi4+7du8ssH/CypiRJaq1Sw1lmjmbmemA1cH1EXAv8DvAK4MeBi4DfPstj3p2ZGzJzw+Dg4KzXfKpa1cc3SZKk1mnJaM3M3Ad8DbgpM58vLl0OAX8KXF/sth1Y0/Sx1UVbWzUua9pzJkmSWqPM0ZqDEbGkWF4AvAn4h/H7yCIigDcDm4qP3Ae8vRi1eQOwPzOfL6u+mao5Ca0kSWqhMkdrrgTujYgqjRD4+cz864j4akQMAgE8Cryr2P9+4BZgC3AEeEeJtc1YvRoMOwmtJElqkdLCWWY+Blw3SfsbzrB/AneVVc+5qlXsOZMkSa3jEwKm4WhNSZLUSoazadQdrSlJklrIcDaNmqM1JUlSCxnOplGrVhgZSxq3xEmSJJXLcDaNeiUAGHHEpiRJagHD2TRq1cY/kZc2JUlSKxjOplGvNnrOHBQgSZJawXA2jdr4ZU17ziRJUgsYzqYxcVnTnjNJklQ+w9k0Ji5r2nMmSZLKZzibRq1iz5kkSWodw9k0auM9Z95zJkmSWsBwNo36+D1njtaUJEktYDibhqM1JUlSKxnOpjHeczbsPWeSJKkFDGfTGL/nzMc3SZKkVjCcTWN8tKY9Z5IkqRUMZ9MYn+fMe84kSVIrGM6mUXO0piRJaiHD2TTGR2s6z5kkSWoFw9k0TsxzZjiTJEktYDibxsRoTS9rSpKk8pUWziKiJyIeiojvR8TmiPi9ov3yiPhORGyJiM9FRFfR3l2sbym2ry2rtrNRPzFa054zSZJUvjJ7zoaAN2Tmq4D1wE0RcQPwH4EPZ+ZVwF7gjmL/O4C9RfuHi/3a7kTPmVNpSJKkFigtnGXDoWK1XrwSeAPw50X7vcCbi+Vbi3WK7W+MiCirvpk68eBzJ6GVJEktUOo9ZxFRjYhHgV3AV4AfAfsyc6TYZRuwqlheBTwHUGzfDwyUWd9MjF/WtOdMkiS1QqnhLDNHM3M9sBq4HnjF+R4zIu6MiI0RsXH37t3nXeN0ak5CK0mSWqglozUzcx/wNeBGYElE1IpNq4HtxfJ2YA1AsX0xsGeSY92dmRsyc8Pg4GDptZ948LmjNSVJUguUOVpzMCKWFMsLgDcBT9AIab9Y7HY78MVi+b5inWL7VzOz7d1V45PQ2nMmSZJaoTb9LudsJXBvRFRphMDPZ+ZfR8TjwGcj4veBR4B7iv3vAT4VEVuAl4DbSqxtxqoVR2tKkqTWKS2cZeZjwHWTtD9F4/6zU9uPAf+8rHrOVURQr4ajNSVJUkv4hIAZqFUq9pxJkqSWMJzNQK0aPiFAkiS1hOFsBurVis/WlCRJLWE4m4FaJRytKUmSWsJwNgP1asXLmpIkqSUMZzNQq4aXNSVJUksYzmbAy5qSJKlVDGcz0Lisac+ZJEkqn+FsBhqXNe05kyRJ5TOczUCtYs+ZJElqDcPZDNSr3nMmSZJaw3A2A7WKk9BKkqTWMJzNgI9vkiRJrWI4mwEf3yRJklrFcDYDznMmSZJaxXA2A85zJkmSWsVwNgPOcyZJklrFcDYDtUrFy5qSJKklDGczUK+GlzUlSVJLGM5mwMuakiSpVQxnM+DjmyRJUqsYzmbAxzdJkqRWMZzNQM1JaCVJUouUFs4iYk1EfC0iHo+IzRHxnqL9dyNie0Q8WrxuafrM70TEloh4MiJ+tqzazla90nh8U6a9Z5IkqVy1Eo89AvxmZn4vIvqAhyPiK8W2D2fm/9W8c0RcDdwGXANcAvy/EfGyzBwtscYZqVUbGXZ0LKlVo83VSJKkuay0nrPMfD4zv1csHwSeAFZN8ZFbgc9m5lBmPg1sAa4vq76zMR7IHLEpSZLK1pJ7ziJiLXAd8J2i6d0R8VhEfDwilhZtq4Dnmj62jUnCXETcGREbI2Lj7t27S6x6Qr3S+GdyxKYkSSpb6eEsInqBvwDem5kHgI8CVwLrgeeBD53N8TLz7szckJkbBgcHZ73eyZzoOXPEpiRJKlmp4Swi6jSC2acz8y8BMnNnZo5m5hjwMSYuXW4H1jR9fHXR1nbj95wNO2JTkiSVrMzRmgHcAzyRmX/Q1L6yabdfADYVy/cBt0VEd0RcDqwDHiqrvrNRr9hzJkmSWqPM0ZqvBX4V+EFEPFq0vR94a0SsBxLYCrwTIDM3R8TngcdpjPS8qxNGasJEz5nhTJIkla20cJaZ3wQmm3fi/ik+8wHgA2XVdK7qxT1nXtaUJEll8wkBM1Cr2HMmSZJaw3A2A+OjNZ1KQ5Iklc1wNgN1J6GVJEktYjibgYnLmvacSZKkchnOZmDisqY9Z5IkqVyGsxmoj0+l4WhNSZJUMsPZDNSchFaSJLWI4WwGxnvOHK0pSZLKdlbhLCIqEdFfVjGdquZoTUmS1CLThrOI+LOI6I+IRTSeg/l4RPxW+aV1jvHRmvacSZKkss2k5+zqzDwAvBn4EnA5jWdmzhuLuqsAHDneEY/6lCRJc9hMwlk9Iuo0wtl9mTlM46Hl80ZfTx2Ag8eG21yJJEma62YSzv5vYCuwCPhGRFwGHCizqE6zsF4lAg4dG2l3KZIkaY6rTbdDZn4E+EhT0zMR8frySuo8lUrQ213jgOFMkiSVbCYDAt5TDAiIiLgnIr4HvKEFtXWUvu4aBw1nkiSpZDO5rPlrxYCAnwGW0hgM8MFSq+pAfT11Dg15z5kkSSrXTMJZFO+3AJ/KzM1NbfNGb489Z5IkqXwzCWcPR8SXaYSzv42IPmDeTfjV11Pj0JDhTJIklWvaAQHAHcB64KnMPBIRA8A7yi2r8/R213hmz5F2lyFJkua4mYzWHIuI1cAvRwTA32Xmfy+9sg7T11P3sqYkSSrdTEZrfhB4D/B48frfI+Lfl11Yp+nrqTkJrSRJKt1MLmveAqzPzDGAiLgXeAR4f5mFdZq+7hpDI2McHxmjq3ZWz4uXJEmasZmmjCVNy4vLKKTT9fY0cqyDAiRJUplmEs7+A/BIRHyi6DV7GPjAdB+KiDUR8bWIeDwiNkfEe4r2iyLiKxHxw+J9adEeEfGRiNgSEY9FxKvP5w+bbePP1/QRTpIkqUzThrPM/AxwA/CXwF8AN2bm52Zw7BHgNzPz6uLzd0XE1cD7gAczcx3wYLEOcDOwrnjdCXz0LP+WUvUVPWcHvO9MkiSV6Iz3nE3Sc7WteL8kIi7JzO9NdeDMfB54vlg+GBFPAKuAW4HXFbvdC3wd+O2i/ZOZmcC3I2JJRKwsjtN2fd1e1pQkSeWbakDAh6bYlpzF8zUjYi1wHfAdYEVT4HoBWFEsrwKea/rYtqLtpHAWEXfS6Fnj0ksvnWkJ5238sqbTaUiSpDKdMZxl5utn4wsiopfG5dD3ZuaBYq608e/IiMizOV5m3g3cDbBhw4az+uz5mBgQ4GVNSZJUnlLnhIiIOo1g9unM/MuieWdErCy2rwR2Fe3bgTVNH19dtHWE8XvO7DmTJEllKi2cRaOL7B7gicz8g6ZN9wG3F8u3A19san97MWrzBmB/p9xvBo3HN4HhTJIklWsmk9Ceq9cCvwr8ICIeLdreD3wQ+HxE3AE8A7yl2HY/jQlvtwBH6LDnd/bUq3RVK4YzSZJUqqlGa74tM/9bsfzazPz/mra9OzP/aKoDZ+Y3gTjD5jdOsn8Cd82o6jbp9RFOkiSpZFNd1vyNpuX/csq2Xyuhlo7X11NzKg1JklSqqcJZnGF5svV5obe75mVNSZJUqqnCWZ5hebL1eaGvp+bjmyRJUqmmGhDwioh4jEYv2ZXFMsX6FaVX1oF6u+ts23uk3WVIkqQ5bKpw9mMtq+IC0e89Z5IkqWRTPSHgmeb1iBgAfgp4NjMfLruwTtTX4z1nkiSpXGe85ywi/joiri2WVwKbaIzS/FREvLdF9XWU3qLnrDHrhyRJ0uybakDA5Zm5qVh+B/CVzPx54DXM26k06oyOJUeHR9tdiiRJmqOmCmfNs62+kcYM/mTmQWCszKI61fgjnByxKUmSyjLVgIDnIuLXgW3Aq4EHACJiAVBvQW0dZ/zh5weOjbC8v83FSJKkOWmqnrM7gGuAfwH8UmbuK9pvAP605Lo60ng4c8SmJEkqy1SjNXcB75qk/WvA18osqlP19TQ6DH2+piRJKstUDz6/b6oPZuY/m/1yOtv4PWdOpyFJksoy1T1nNwLPAZ8BvsM8fZ5ms/4FjZ6zA0ftOZMkSeWYKpxdDLwJeCvwy8DfAJ/JzM2tKKwTDSzqAuDFQ0NtrkSSJM1VZxwQkJmjmflAZt5OYxDAFuDrEfHullXXYXrqVRYvqLPzgOFMkiSVY6qeMyKiG/g5Gr1na4GPAF8ov6zOtaK/m10Hj7W7DEmSNEdNNSDgk8C1NCaf/b2mpwXMayv6e+w5kyRJpZlqnrO3AeuA9wB/HxEHitfBiDjQmvI6z2BfN7sO2HMmSZLKMdU8Z1MFt3lrRX8Puw4OMTaWVCrzfgCrJEmaZQaws7Sir5uRsWTvkePtLkWSJM1BhrOztLy/B8D7ziRJUikMZ2dpRX83ADsdsSlJkkpQWjiLiI9HxK6I2NTU9rsRsT0iHi1etzRt+52I2BIRT0bEz5ZV1/la3tfoOdttz5kkSSpBmT1nnwBumqT9w5m5vnjdDxARVwO3AdcUn/mTiKiWWNs5G+wres4csSlJkkpQWjjLzG8AL81w91uBz2bmUGY+TeNpBNeXVdv56KlXWbKw7mVNSZJUinbcc/buiHisuOy5tGhbReMh6+O2FW2niYg7I2JjRGzcvXt32bVOanlfN7u8rClJkkrQ6nD2UeBKYD3wPPChsz1AZt6dmRsyc8Pg4OBs1zcjK/p72HnQcCZJkmZfS8NZZu4sHqg+BnyMiUuX24E1TbuuLto60vK+Hp8SIEmSStHScBYRK5tWfwEYH8l5H3BbRHRHxOU0Hhv1UCtrOxvL+7vZXTwlQJIkaTad8fFN5ysiPgO8DlgWEduAfwe8LiLWAwlsBd4JkJmbI+LzwOPACHBXZo6WVdv5Gn9KwEtHjrOst7vd5UiSpDmktHCWmW+dpPmeKfb/APCBsuqZTStOPCXgmOFMkiTNKp8QcA6WF08J2OWgAEmSNMsMZ+dg/CkBO/c7KECSJM0uw9k5uHhxD9VKsG3v0XaXIkmS5hjD2TmoVyusXrqAp/ccbncpkiRpjjGcnaPLBhax9UXDmSRJml2Gs3N0+cBCntlzhEznOpMkSbPHcHaO1i5bxKGhEV48dLzdpUiSpDnEcHaO1i5bBMBW7zuTJEmzyHB2jtYOFOHM+84kSdIsMpydo9VLF1CthD1nkiRpVhnOzlG9WmHN0gVsffFIu0uRJElziOHsPKxdtoinvawpSZJmkeHsPKwdWMQzew47nYYkSZo1hrPzsHZgIYePj7L7kA9AlyRJs8Nwdh5OTKfhfWeSJGmWGM7Ow+XOdSZJkmaZ4ew8rFqygK5ahR/uPNjuUiRJ0hxhODsPtWqFV1zcx+YdB9pdiiRJmiMMZ+fpmkv62bzjgCM2JUnSrDCcnaerL1nM/qPDbN93tN2lSJKkOcBwdp6uuaQfwEubkiRpVhjOztOPXdxPJQxnkiRpdpQWziLi4xGxKyI2NbVdFBFfiYgfFu9Li/aIiI9ExJaIeCwiXl1WXbNtQVeVKwZ7eXzH/naXIkmS5oAye84+Adx0Stv7gAczcx3wYLEOcDOwrnjdCXy0xLpm3figAEmSpPNVWjjLzG8AL53SfCtwb7F8L/DmpvZPZsO3gSURsbKs2mbbNZf08/z+Y7x0+Hi7S5EkSRe4Vt9ztiIzny+WXwBWFMurgOea9ttWtF0QrrlkMQCbvbQpSZLOU9sGBGRjYrCznhwsIu6MiI0RsXH37t0lVHb2xkdsbtrupU1JknR+Wh3Odo5frizedxXt24E1TfutLtpOk5l3Z+aGzNwwODhYarEztWRhF1cMLuKhp/e0uxRJknSBa3U4uw+4vVi+HfhiU/vbi1GbNwD7my5/XhBuvGKA727dy8joWLtLkSRJF7Ayp9L4DPAt4OURsS0i7gA+CLwpIn4I/HSxDnA/8BSwBfgY8K/LqqssN145wKGhETY5alOSJJ2HWlkHzsy3nmHTGyfZN4G7yqqlFV5z+QAA3/rRHtavWdLmaiRJ0oXKJwTMksG+btYt7+VbT3nfmSRJOneGs1l045UDbNz6EsPedyZJks6R4WwW3XDFAEeOj/LYNuc7kyRJ58ZwNotuuGL8vrMX21yJJEm6UBnOZtFFi7p41ZolfPnxne0uRZIkXaAMZ7Pslmsv5rFt+3nupSPtLkWSJF2ADGez7OZrG89rf2DTC22uRJIkXYgMZ7Ps0oGFXHNJP1/adEE94ECSJHUIw1kJbnnlSr737D6e33+03aVIkqQLjOGsBDdfezEAX/qBlzYlSdLZMZyV4IrBXq5d1c/nvvscjSdTSZIkzYzhrCRvv2EtT+48yENPv9TuUiRJ0gXEcFaSn3/VJSxeUOeT33qm3aVIkqQLiOGsJAu6qvzSj6/hbze/wAv7j7W7HEmSdIEwnJXoba+5jNFM/tu37T2TJEkzYzgr0aUDC7npmov5xN9vZc+hoXaXI0mSLgCGs5L95s+8jCPHR/iTr/+o3aVIkqQLgOGsZFct7+MX/+lqPvXtZ9i+z0lpJUnS1AxnLfCen34ZJPynB/6h3aVIkqQOZzhrgVVLFvCu113JXz26gwef2NnuciRJUgcznLXIu19/Fa+4uI/3f+EH7D863O5yJElShzKctUhXrcJ/+sVX8eKh4/zbv9rkY50kSdKkDGct9MrVi3nvG9dx3/d3cM83n253OZIkqQPV2vGlEbEVOAiMAiOZuSEiLgI+B6wFtgJvycy97aivTHe9/io27zjAv7//CV5+cR8/uW6w3SVJkqQO0s6es9dn5vrM3FCsvw94MDPXAQ8W63NOpRJ86C2v4mUr+njXpx7m4WfmXP6UJEnnoZMua94K3Fss3wu8uY21lGpRd41P/tr1LO/v4V98/CEefW5fu0uSJEkdol3hLIEvR8TDEXFn0bYiM58vll8AVrSntNZY3t/Dn/3L17B0URe/8rFv87Und7W7JEmS1AHaFc5+IjNfDdwM3BURP9W8MRtDGScdzhgRd0bExojYuHv37haUWp6Vixfw/7zrRtYuW8T/du9GPvWtrY7ilCRpnmtLOMvM7cX7LuALwPXAzohYCVC8T9qVlJl3Z+aGzNwwOHjh30y/or+Hz7/zRl73skH+7Rc38+ufeYSDx5wHTZKk+arl4SwiFkVE3/gy8DPAJuA+4PZit9uBL7a6tnZZ1F3jY2/fwL+56eV8adML3PSH/4Nv/OOF3SsoSZLOTTt6zlYA34yI7wMPAX+TmQ8AHwTeFBE/BH66WJ83KpXgX7/uKj7/zhvorld4+8cf4v/43KPsPHCs3aVJkqQWigv5HqcNGzbkxo0b213GrDs2PMoffXULd3/jKaqV4F/+1BXc8drLWbyw3u7SJEnSOYqIh5umEDvzfoazzvXsniN88IEnuP8HL9DbXeNtN1zGHT9xOYN93e0uTZIknSXD2RzyxPMH+JOv/4i/eWwH9WqFt2xYw9tuuIyXX9zX7tIkSdIMGc7moKdfPMxHv76Fv3pkB8dHx/inly3ll6+/lJ/7JyvpqVfbXZ4kSZqC4WwOe+nwcf7i4W185qFneerFw/T11Lj52ov5+Vddwo1XDFCrdtKDHyRJEhjO5oXM5NtPvcSfP7yNL29+gYNDIyzr7eLma1fypqtX8JorLqK7Zo+aJEmdwHA2zxwbHuXrT+7mv39/Bw/+w06ODY+xsKvKT1y1jDf+2HJe//LlLO/vaXeZkiTNWzMNZ7VWFKPy9dSr3HTtxdx07cUcGx7lWz/aw4P/sJOvPrGLLz++E4BrV/Xz2iuXccOVA/z42ovo7fb0S5LUaew5m+Mykyd3HuTBJ3bx9Sd38ehz+xgeTaqV4JWrFnPjlQPccMUA1126hP4e51GTJKksXtbUpI4cH+F7z+zjW0+9yLefeonvP7ePkbEkAtYt7+W6NUu57tIlXHfpUq5a3ku1Eu0uWZKkOcFwphk5PDTC957dyyPP7uORZ/fyyHP72Hek8eD13u4ar1qzmOvWLOXaVYu55pJ+Vi9dQISBTZKks+U9Z5qRRd01fnLdID+5bhBoXAbduudII6g9u49HntvLR//uR4yONUJ8f0+Nqy/p55pLFnP1yn6uWdXPlYO91J2+Q5KkWWE400kigsuXLeLyZYv4X169GoCjx0d5cudBNu/Yz+YdB3h8xwE+/Z1nODY8BkBXrcJVg72sW9HLuuW9XLW8l6uW93HZwEJDmyRJZ8lwpmkt6Kqyfs0S1q9ZcqJtZHSMrXsOs3nHATbvOMA/7jzIxq17+eKjO07sU682gt5Vy3u5crCXtQOLWLtsIZcNLGJgUZeXRyVJmoThTOekVq1w1fI+rlrex63rV51oPzw0wlO7D/PDXQf54a5DbNl1iMd3HOCBTS8w1nR7Y293jcsGFrJ22SLWDjQC29qBRVw2sJDB3m4qDkSQJM1ThjPNqkXdNV65ejGvXL34pPbjI2Ns33eUrXsOs/XFwzyz50ij5237fh7Y9MKJe9oAuqoVVi7pYdWSBY3X0gVcsmQBq4vllYsX0FXzcqkkaW4ynKklumqVE/ey8fKTtw2PjrFj31G27jnCs3sOs23fUbbvPcr2fUf5u3/cza6DQyftHwHL+7q5uL+H5f09rOifWL64v4cVxXv/gpqXTiVJFxzDmdquXq1w2cAiLhtYBAyetn1oZJQX9h9j+96jJ4Lbjn1H2XlwiGf3HOG7W186Mf1Hs556hRX9Pazo62HF4h6W9XaxrLebgUVdDPR2T6z3drGwy5+CJKkz+H+R1PG6a9Wm8Da5Y8Oj7DowxAsHjrGz6fXCgSF2HjjGY9v2sefQcQ4NjUz6+QX1KgO9jdA22NvFwKJuli7qYsnCOksW1FmysFheWGfJgsZyT92HykuSZp/hTHNCT73KpQMLuXRg4ZT7HRseZc/h47x4cIg9h4d48dBx9hw6zp5DQ432Q0Ns33eMx7btZ++R4wyPnnmS5p565URQaw5tixfU6eup0dtdo6+nsTzx3lju7a5535wkaVKGM80rPfXqiYEG08lMjg6PsvfIMPuOHGf/kWH2HR1m35Fh9h45zv6jjfZ9RxptT714iH1Hhtl/dJihkbFpj99dq9DXU6e/p0bveHDrLoJdEeIWdVVZ2F1rvHdVWdhVY1F38d5VY0FXlUXdVRbUq95fJ0lzhOFMOoOIYGFXjYVdtRmFuWbHR8Y4NDTCwWPDHDw2woFjwxw6NsLBY422xrYRDpyyvvvgoYn9znAJdvJaYWG9EeROhLgi2DXaqyzqqrGwu8rCemOfnnqF7nqVnnqVnlqFBV3jy41tPePbimUnFJak1jCcSSXoqlW4qNbFRYu6zvkYY2PJsZFRDg+NcuT4yIn3I8dPXj98fLScsQV9AAAJyElEQVTRNjS+PLHtwNFhXth/9KR9j8+gV28y1UrQUzs9tI0vL6hX6a5V6apV6KpWGu/jr2qF7nrxflJ79aR9umpN2yc5Rle14hx4kuY8w5nUoSqViZ476J614w6PjnFseJRjw433oZFRjh4f49jI6EntR4dHGWpaP3bKfkMntY+y9/Awx0Ya4e/4yBjHR8dOLI+MnfnevbNVrwZd1Qr1WoVapUK9GtSrFWrVoF4p3quN9lqx3lVsr1Ur1Cvj+0/sU69N/tmJY0/WNrFeq1aoRlCtBLVq8V4JKtG8XqFamdg2/u7laEmn6rhwFhE3Af8ZqAL/NTM/2OaSpDmlET4q9PW07jvHxpLjo2MMTRLcGuujE9tO3V4sn/rZ4dExhkeTkdFieayxPDLa+K6R0WRkbIyjw8lwsT48VryPf3asef8xZjFDzlglaAptlYlgd0qIq554VU5qP32/CtUKpx2rGkGlApVohMZqER7Hvz8iqFagGuPLjW0nPhuN44zvP3Gcxi0Azcdv3n5i/6KtGk3HHf+OU2s6pY7GMifVTNPnKgFBEMX3B8V7nPxeCQzDuiB0VDiLiCrwx8CbgG3AdyPivsx8vL2VSToflUrQU6l2/PQjY2ONADcR+hoBbnjk1GDX6A0cD3ljY8nIWDJavEbGxor3k7c13scYHYPRsbGT2k/eb6zpWE3vo8W+OXGskdHk+MgYo1nsNzpxjLGkcazRZCxhNBvfM1bsm+NtmYyNTSxnG0JqK50U1jg9xE0W6qA5SDZC3mT7jwfDE+sx8bnT1mlun+JznH6cSvFl49sax5pYpulvi+JvPrHe9PfEafvFiX+j0z5/4m88w3GLDza3n/iOqY5brNP0d574u85w3NPqovl98uNOVhdNx1pz0UJ+bGV/Of9Ld5Y6KpwB1wNbMvMpgIj4LHArYDiTVLpKJeiuVOnutP8ytlhmEeaKIDce5sayEWCbA93EtonPZBb7nLa96bhjTYGxCI0nvuOk75wIjpnJ6Bgkjf2yCJLjx21eTzgRNMfGJtbHEsjm72kc79T9ThynaT+Y+JtO+twk+5/2uen2G4NRxk6qq7nO0z5X1Enjfxp/e1Pt4wF7vD2b6h1fp/nfcXyfYvnU4570+dOOlU3HvHC97YZL+f03v7LdZQCdF85WAc81rW8DXtO8Q0TcCdwJcOmll7auMkmaJxqXKRuXEaWzNR6Sx4Nbc5BtbJ8i3OUp4btpf5qC5WmfnzY0nhxMswjazXUtPY8BXLOt08LZtDLzbuBugA0bNlzgOV2SpLll/HJksdbOUi5YnTZx0XZgTdP66qJNkiRpXui0cPZdYF1EXB4RXcBtwH1trkmSJKllOuqyZmaORMS7gb+lMZXGxzNzc5vLkiRJapmOCmcAmXk/cH+765AkSWqHTrusKUmSNK8ZziRJkjqI4UySJKmDGM4kSZI6iOFMkiSpgxjOJEmSOkjkBfyk0ojYDTzTgq9aBrzYgu/RzHlOOpPnpTN5XjqT56UzlXleLsvMwel2uqDDWatExMbM3NDuOjTBc9KZPC+dyfPSmTwvnakTzouXNSVJkjqI4UySJKmDGM5m5u52F6DTeE46k+elM3leOpPnpTO1/bx4z5kkSVIHsedMkiSpgxjOJEmSOojhbAoRcVNEPBkRWyLife2uZz6LiK0R8YOIeDQiNhZtF0XEVyLih8X70nbXOddFxMcjYldEbGpqm/Q8RMNHit/PYxHx6vZVPred4bz8bkRsL34zj0bELU3bfqc4L09GxM+2p+q5LSLWRMTXIuLxiNgcEe8p2v29tNEU56Wjfi+GszOIiCrwx8DNwNXAWyPi6vZWNe+9PjPXN80/8z7gwcxcBzxYrKtcnwBuOqXtTOfhZmBd8boT+GiLapyPPsHp5wXgw8VvZn1m3g9Q/HfsNuCa4jN/Uvz3TrNrBPjNzLwauAG4q/i39/fSXmc6L9BBvxfD2ZldD2zJzKcy8zjwWeDWNtekk90K3Fss3wu8uY21zAuZ+Q3gpVOaz3QebgU+mQ3fBpZExMrWVDq/nOG8nMmtwGczcygznwa20PjvnWZRZj6fmd8rlg8CTwCr8PfSVlOclzNpy+/FcHZmq4Dnmta3MfUJVLkS+HJEPBwRdxZtKzLz+WL5BWBFe0qb9850HvwNtd+7i0tkH2+67O95abGIWAtcB3wHfy8d45TzAh30ezGc6ULxE5n5ahpd/3dFxE81b8zGnDDOC9NmnoeO8lHgSmA98DzwofaWMz9FRC/wF8B7M/NA8zZ/L+0zyXnpqN+L4ezMtgNrmtZXF21qg8zcXrzvAr5Ao1t553i3f/G+q30VzmtnOg/+htooM3dm5mhmjgEfY+JSjOelRSKiTiMAfDoz/7Jo9vfSZpOdl077vRjOzuy7wLqIuDwiumjcEHhfm2ualyJiUUT0jS8DPwNsonE+bi92ux34YnsqnPfOdB7uA95ejEK7AdjfdDlHJTvlfqVfoPGbgcZ5uS0iuiPicho3oD/U6vrmuogI4B7gicz8g6ZN/l7a6EznpdN+L7Wyv+BClZkjEfFu4G+BKvDxzNzc5rLmqxXAFxq/KWrAn2XmAxHxXeDzEXEH8AzwljbWOC9ExGeA1wHLImIb8O+ADzL5ebgfuIXGDbRHgHe0vOB54gzn5XURsZ7GZbOtwDsBMnNzRHweeJzGyLW7MnO0HXXPca8FfhX4QUQ8WrS9H38v7Xam8/LWTvq9+PgmSZKkDuJlTUmSpA5iOJMkSeoghjNJkqQOYjiTJEnqIIYzSZKkDmI4kzRnRcRoRDza9Hrf9J+a8bHXRsSm6feUpLPjPGeS5rKjmbm+3UVI0tmw50zSvBMRWyPi/4yIH0TEQxFxVdG+NiK+Wjz8+MGIuLRoXxERX4iI7xev/6k4VDUiPhYRmyPiyxGxoG1/lKQ5w3AmaS5bcMplzV9q2rY/M18J/BHwh0XbfwHuzcx/Anwa+EjR/hHg7zLzVcCrgfGnhawD/jgzrwH2Af9ryX+PpHnAJwRImrMi4lBm9k7SvhV4Q2Y+VTwE+YXMHIiIF4GVmTlctD+fmcsiYjewOjOHmo6xFvhKZq4r1n8bqGfm75f/l0may+w5kzRf5RmWz8ZQ0/Io3scraRYYziTNV7/U9P6tYvnvgduK5V8B/kex/CDwrwAiohoRi1tVpKT5x/8vT9JctiAiHm1afyAzx6fTWBoRj9Ho/Xpr0fbrwJ9GxG8Bu4F3FO3vAe6OiDto9JD9K+D50quXNC95z5mkeae452xDZr7Y7lok6VRe1pQkSeog9pxJkiR1EHvOJEmSOojhTJIkqYMYziRJkjqI4UySJKmDGM4kSZI6yP8PBQuzf6CZfPAAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"_, ax = plt.subplots(figsize=(10,5))\n",
"ax.plot(losses)\n",
"ax.set_ylabel('MSE loss'); ax.set_xlabel('Epoch');"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Note that this implementation is not ideal, as it's:"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- Non modular (hard to switch components)\n",
"- Hard to extend (e.g. to add layers)\n",
"- Error prone (hard-coded manual calculations)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"But it works!\n",
"- In HW2, you'll implement a from scratch MLP that addresses these concerns.\n",
"- And now, we'll see how to address these issues using PyTorch's API."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## N-Layer MLP using PyTorch"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Let's create all our usual components:\n",
"- Dataset\n",
"- Model\n",
"- Loss function\n",
"- Optimizer\n",
"\n",
"But this time we'll create a modular implementation where each of these components is separate and can be changed independently of the others."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Dataset"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"As in the previous tutorial we'll tackle an image classification task, the MNIST database of handwritten digits."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"x0: torch.Size([1, 28, 28]), y0: 5\n"
]
}
],
"source": [
"import torch\n",
"import torch.utils.data\n",
"import torchvision\n",
"import torchvision.transforms as tvtf\n",
"\n",
"# Define the transforms that should be applied to each image in the dataset before returning it\n",
"tf_ds = tvtf.Compose([\n",
" tvtf.ToTensor(), # Convert PIL image to pytorch Tensor\n",
" tvtf.Normalize(mean=(0.1307,), std=(0.3081,)) # normalize to zero mean and unit std\n",
"])\n",
"\n",
"batch_size = 512\n",
"train_size = batch_size * 10\n",
"test_size = batch_size * 2\n",
"\n",
"# Datasets and loaders\n",
"root_dir = os.path.expanduser('~/.pytorch-datasets/mnist/')\n",
"ds_train = torchvision.datasets.MNIST(root=root_dir, download=True, train=True, transform=tf_ds)\n",
"dl_train = torch.utils.data.DataLoader(ds_train, batch_size,\n",
" sampler=torch.utils.data.SubsetRandomSampler(range(0,train_size)))\n",
"ds_test = torchvision.datasets.MNIST(root=root_dir, download=True, train=False, transform=tf_ds)\n",
"dl_test = torch.utils.data.DataLoader(ds_test, batch_size,\n",
" sampler=torch.utils.data.SubsetRandomSampler(range(0,test_size)))\n",
"\n",
"x0, y0 = ds_train[0]\n",
"n_features = torch.numel(x0)\n",
"n_classes = 10\n",
"\n",
"print(f'x0: {x0.shape}, y0: {y0}')"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Model Implementation"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- The `torch.nn` module contains building blocks such as neural network layers,\n",
" loss functions, activations and more.\n",
"\n",
"- We'll implement our model as a subclass of `nn.Module`, which means:\n",
" - Any tensors we set as properties will be registered as model parameters.\n",
" - We can nest `nn.Modules` and get all model parameters from the top-level `nn.Module`.\n",
" - Can be used as a function if we implement the `forward()` method."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"import torch.nn as nn\n",
"\n",
"class MLP(torch.nn.Module):\n",
" def __init__(self, D_in: int, hidden_dims: list, D_out: int):\n",
" super().__init__()\n",
" \n",
" all_dims = [D_in, *hidden_dims, D_out]\n",
" layers = []\n",
" \n",
" for in_dim, out_dim in zip(all_dims[:-1], all_dims[1:]):\n",
" layers += [\n",
" nn.Linear(in_dim, out_dim, bias=True),\n",
" nn.ReLU()\n",
" ]\n",
" \n",
" self.fc = nn.Sequential(*layers)\n",
" self.log_softmax = nn.LogSoftmax(dim=1)\n",
"\n",
" def forward(self, x):\n",
" x = torch.reshape(x, (x.shape[0], -1))\n",
" z = self.fc(x)\n",
" y_pred = self.log_softmax(z)\n",
" return y_pred"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"scrolled": true,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"MLP(\n",
" (fc): Sequential(\n",
" (0): Linear(in_features=784, out_features=32, bias=True)\n",
" (1): ReLU()\n",
" (2): Linear(in_features=32, out_features=64, bias=True)\n",
" (3): ReLU()\n",
" (4): Linear(in_features=64, out_features=128, bias=True)\n",
" (5): ReLU()\n",
" (6): Linear(in_features=128, out_features=64, bias=True)\n",
" (7): ReLU()\n",
" (8): Linear(in_features=64, out_features=10, bias=True)\n",
" (9): ReLU()\n",
" )\n",
" (log_softmax): LogSoftmax()\n",
")\n"
]
}
],
"source": [
"# Create an instance of the model (5-layer MLP)\n",
"mlp5 = MLP(D_in=n_features, hidden_dims=[32, 64, 128, 64], D_out=n_classes)\n",
"print(mlp5)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"scrolled": true,
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"number of parameter tensors: 10\n"
]
}
],
"source": [
"print(f'number of parameter tensors: {len(list(mlp5.parameters()))}')"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"number of parameters: 44458\n"
]
}
],
"source": [
"print(f'number of parameters: {np.sum([torch.numel(p) for p in mlp5.parameters()])}')"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"y_hat0=tensor([[-2.3167, -2.3167, -2.2767, -2.2198, -2.3167, -2.3167, -2.3167, -2.3167,\n",
" -2.3167, -2.3167]], grad_fn=),\n",
"shape=torch.Size([1, 10])\n"
]
}
],
"source": [
"# Test a forward pass\n",
"y_hat0 = mlp5(x0)\n",
"\n",
"print(f'y_hat0={y_hat0},\\nshape={y_hat0.shape}')"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Loss and Optimizer"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- For the loss, we'll use PyTorch's built in cross-entropy loss.\n",
"- We won't need to calculate the loss gradient this time, as we'll use `autograd` for automatic differentiation.\n",
"- As for the optimization scheme, we'll use a built in SGD optimizer from the `torch.optim` module."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"import torch.optim\n",
"\n",
"# Model\n",
"model = MLP(D_in=n_features, hidden_dims=[32, 32, 32], D_out=n_classes)\n",
"\n",
"# Loss:\n",
"# Note: NLLLoss assumes log-probabilities (given by our LogSoftmax layer)\n",
"loss_fn = nn.NLLLoss()\n",
"\n",
"# Optimizer\n",
"optimizer = torch.optim.SGD(params=model.parameters(), lr=1e-2, weight_decay=0.1, momentum=0.9)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Training loop"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"This time we'll train over lazy-loaded batches from our data loader.\n",
"\n",
"Notice that except from our model's `__init__()` and `__forward()__`, we're using PyTorch facilities for the entire training implementation."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch #1: Avg. loss=2.3054928541183473\n",
"Epoch #2: Avg. loss=2.3014203310012817\n",
"Epoch #3: Avg. loss=2.2982772827148437\n",
"Epoch #4: Avg. loss=2.2965104579925537\n",
"Epoch #5: Avg. loss=2.2955925703048705\n",
"Epoch #6: Avg. loss=2.2952797651290893\n",
"Epoch #7: Avg. loss=2.2953588247299193\n",
"Epoch #8: Avg. loss=2.2957545042037966\n",
"Epoch #9: Avg. loss=2.29633150100708\n",
"Epoch #10: Avg. loss=2.296979832649231\n"
]
}
],
"source": [
"num_epochs = 10\n",
"for epoch_idx in range(num_epochs):\n",
" total_loss = 0\n",
" for batch_idx, (X, y) in enumerate(dl_train):\n",
" # Forward pass\n",
" y_pred = model(X)\n",
"\n",
" # Compute loss\n",
" loss = loss_fn(y_pred, y)\n",
" total_loss += loss.item()\n",
"\n",
" # Backward pass\n",
" optimizer.zero_grad() # Zero gradients of all parameters\n",
" loss.backward()\n",
" \n",
" # Weight update\n",
" optimizer.step()\n",
" \n",
" print(f'Epoch #{epoch_idx+1}: Avg. loss={total_loss/len(dl_train)}')"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"source": [
"## Image credits\n",
"- MartinThoma [CC0], via Wikimedia Commons https://commons.wikimedia.org/wiki/File:Perceptron-unit.svg\n",
"- Sebastian Raschka https://sebastianraschka.com/Articles/2015_singlelayer_neurons.html\n",
"- Favio Vázquez https://towardsdatascience.com/a-conversation-about-deep-learning-9a915983107\n",
"- Fundamentals of Deep Learning, Nikhil Buduma, Oreilly 2017"
]
}
],
"metadata": {
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}