{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 578,
   "id": "2ac47cb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from preamble import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 579,
   "id": "448b459d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b0a1a96",
   "metadata": {},
   "source": [
    "# Reproducibility\n",
    "- We set the random seeds so that the training results are always the same\n",
    "- Feel free to change the seed number to see the effects of the random initialization of the network weights on the training results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 580,
   "id": "ad3d6c46",
   "metadata": {},
   "outputs": [],
   "source": [
    "SEED = 10\n",
    "torch.manual_seed(SEED)\n",
    "torch.backends.openmp.deterministic = True\n",
    "np.random.seed(SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69778674",
   "metadata": {},
   "source": [
    "# Conventions for this notebook\n",
    "\n",
    "## Jargon\n",
    "- Unit = activation = neuron\n",
    "- Model = neural network\n",
    "- Feature = dimension of input vector = number of independent variables\n",
    "- Hypothesis = prediction = output of the model\n",
    "\n",
    "\n",
    "## Indices\n",
    "- **Data points:** $i = 1,..., n$ \n",
    "- **Features:** $k = 1,..., p$ \n",
    "- **Layers:** $j = 1,..., l$ \n",
    "- **Activation unit label:** $s$ \n",
    "\n",
    "## Scalars\n",
    "- $u^j$ = number of units in layer $j$\n",
    "- $a_s^j$ is the activation unit $s$ in layer $j$\n",
    "\n",
    "## Vectors and matrices\n",
    "Check slide 13 of the [lecture notes](https://github.com/ansantam/2022-MT-ARD-ST3-ML-workshop/blob/main/slides/1-neural-networks.pdf) for a visualization of the dimensions.\n",
    "\n",
    "- $\\pmb{X}$: input vector of dimension $[n \\times (p \\times 1)]$\n",
    "- $a^j$: activation vector of layer $j$ of dimension $[(u^j + 1) \\times 1]$\n",
    "- $\\pmb{\\theta}^j$: weight matrix from layer $j$ to $j+1$, of dimension $[u^{j+1} \\times (u^j + 1)]$\n",
    "\n",
    "\n",
    "<span style='color:Blue'> where the $+1$ accounts for the bias unit </span>\n",
    "\n",
    "\n",
    "\n",
    "$$\n",
    "\\pmb{X} =\n",
    "\\begin{bmatrix}\n",
    "x_0 \\\\\n",
    "x_1 \\\\\n",
    "\\vdots \\\\\n",
    "x_p\n",
    "\\end{bmatrix} \\ \\ ; \\ \\\n",
    "\\pmb{\\theta}^j =\n",
    "\\begin{bmatrix}\n",
    "\\theta_{10} & \\dots & \\theta_{1(u^j + 1)}\\\\\n",
    "\\theta_{20} & \\ddots\\\\\n",
    "\\vdots \\\\\n",
    "\\theta_{(u^{j+1}) 0} & & \\theta_{(u^{j+1})(u^j + 1)}\\\\\n",
    "\\end{bmatrix} \n",
    "$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa2320f4",
   "metadata": {},
   "source": [
    "# 0. Introduction\n",
    "In this notebook we will train a neural network to fit an arbitrary function.\n",
    "\n",
    "## 0.1. Universal Approximation Theorem\n",
    "- When the activation function is non-linear, then a two-layer neural network can be proven to be a **universal function approximator**.\n",
    "- <span style='color:red'> This is where the power of neural networks comes from! </span>\n",
    "\n",
    "## 0.2. Create a function to fit\n",
    "Let's create a simple non-linear function to fit with our neural network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 581,
   "id": "e58cebe3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Function to be fitted')"
      ]
     },
     "execution_count": 581,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEWCAYAAABv+EDhAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAtDklEQVR4nO3dd1yV5/3/8dfnsEGmbBBBRQG34oxmiYlJTG22SWuSJma0TTP660rT3SZtv/12pM0eJjZ7TzMcWSZOnFGcoAxRQBRkyDzX749z0i81LuAc7jM+z8fjPCIHuO/3TeC8z33d933dYoxBKaWU/7FZHUAppZQ1tACUUspPaQEopZSf0gJQSik/pQWglFJ+SgtAKaX8lBaA8lkiMl1Edlid41gicraIVLhweX8QkYMickBEMkSkUUQCXLX846zvaRH5g7uWr/qOFoByGxHZKyJHnS9IXz1S3bg+IyJDvvrYGLPcGDPMDevJdK4r0NXL7kGWAcD/A/KMMcnGmDJjTD9jTKfz85+IyPxjvue/fk7Kf1n+C6x83sXGmKVWh/BhA4FaY0y11UGU99E9ANXnnHsGBV0+/o2IPOv891fvrq8TkTLn0MY9Xb42QER+LiLFItIgIutEZICIfOb8kk3OPY2rjh1qEZFc5zviOhHZKiLf6PK5p0XkQRFZ5FzuahEZfIJN+Gpddc51TRERm4j8QkRKRaRaRP4tItGn+Dn83Ll9e0XkW12eDxGR/3Vuf5WIPCIiYcf5/gJgCZDqzPF0170TEbkXmA484Pz8A8f7OTmXNVtENjp/NitEZFSX9YwVkfXOn8tLQOjJtkt5EWOMPvThlgewFyg41fPAb4Bnnf/OBAzwOBAGjAZagVzn538MfAkMA8T5+f7OzxlgSJflng1UOP8dBOwGfg4EA+cCDcAw5+efBg4BE3HsGT8HvHiC7foqY2CX525wLn8Q0A94HXjmBN9/NtAB/A0IAc4Cmrpk+QfwNhAHRALvAH88ybIqTpQN+ASYf8z3HPtzGgdUA5OAAOA65/+jEOfPqhS4y/kzvBxoB/5g9e+XPnr/0D0A5W5vOt9V1onIm934vt8aY44aYzYBm3C80APMB35hjNlhHDYZY2pPY3mTcbww/8kY02aM+Qh4F7i6y9e8boxZY4zpwFEAY7qR91vA34wxJcaYRuBuYO4pjhP80hjTaoz5FFgEXCkiAtwE3GWMOWSMaQDuA+Z2I0t33QQ8aoxZbYzpNMYsxFG6k52PIOAfxph2Y8yrwFo3ZlF9SI8BKHf7punZMYADXf7djOPFG2AAUNyD5aUC5cYYe5fnSoG001jn6S6/9JhlBwJJwL7jfP1hY0zTMV+fCiQA4cA6RxcAjj0dt53Vg+M4wnUi8oMuzwU78xhgnzGm66yRXbdTeTHdA1BWaMLxIveV5G58bzlworH5k6kEBohI19/5DI7/4nwqx5tCtxLHC2nXZXcAVSdYRqyIRBzz9ZXAQeAoMNwYE+N8RBtjulNGp8p6rHLg3i7rizHGhBtjXgD2A2nSpY2cWZUP0AJQVtiIY3gkSETycYwrn64ngN+LSLY4jBKR/s7PVeEYgz+e1TiK5yfO9Z4NXAy82IP8NYD9mHW9ANwlIlki0g/HsM1LzuGkE/mtiASLyHRgNvCKcw/lceDvIpIIICJpInJ+D3LC8X8mxz73OHCriExy/kwjROQiEYkEVuIostudB5YvxXGcRPkALQBlhV/ieBd/GPgt8Hw3vvdvwMvAYuAI8CSOg8XgOJi80Hm84cqu32SMaQO+AVyA4132Q8C1xpjt3Q1vjGkG7gW+cK5rMrAAeAbHGUJ7gBbgBydeCgdwbH8ljuMNt3bJ8lMcB5RXicgRYCmOg949cT9wuYgcFpF/Op/7DV1+TsaYQhzHAR5wZtoNXO/c1jbgUufHh4GrcBzgVj5A/ntoTymllL/QPQCllPJTWgBKKeWntACUUspPaQEopZSf8qoLweLj401mZqbVMZRSyqusW7fuoDEm4djnvaoAMjMzKSwstDqGUkp5FRE57tXbOgSklFJ+SgtAKaX8lBaAUkr5KS0ApZTyU1oASinlp7QAlFLKT2kBKKWUn/Kq6wCUUj1TWXeUL/fVU1rbREu7nYiQQAbEhpGfGUdcRLDV8ZRFtACU8lEt7Z28tLacV9aVs2XfkRN+3cTMOK6ZlMHsUSkEBuiggD/RAlDKxxhjeLmwnL98uIODjW2MTIvm7gtymJgVx+DEfoQHBdDQ0kFxTSMrimt5fX0Fd760kYc+2c0vZ+cxPftrMwYoH+VVN4TJz883OhWEUidWWXeUH768kVUlh5iQGcuPzhvGpEH9T/o9drthcdEB/vT+dvbWNnP91Ex+dkEOoUHuvA+96ksiss4Yk3/s87oHoJSPWFVSy/efW09Leyd/vHQkV+UPwGaTU36fzSbMGpHC2cMS+fMH23nqi71s2VfP49fmE6vHB3yaDvgp5QPe3lTJt59YTXR4EG/ddgZXT8w4rRf/rkKDAvj1xcN58JpxbN5Xz2UPr+BAfYubEitPoAWglJd7eW05d7y4gXEDY3nz+2cwJDGyV8u7aFQKz8+fRHVDK9c8sYqahlYXJVWeRgtAKS/2xoYKfvLaZqZnJ7DwOxOJCg1yyXLzM+NYcP0EKuuOcu2CNTS2drhkucqzaAEo5aU+21nDj1/ZzJRB/Xn82vGEBbv2oO3ErDgenZfPjgNH+OFLG7HbveeEEXV6tACU8kJFlUf47rPrGJLYj0evHU9IoHvO2DlraAL3XJTH4qIq/rFsl1vWoayjBaCUl6lvbufWZ9fRLzSQhTe4btjnRG44I5PLx6fzr492saL4oFvXpfqWFoBSXsRuN/zw5Y3srz/KQ98aT1JUqNvXKSL8bs5wsvpH8MOXNlHX3Ob2daq+oQWglBd5+NNilm2v5pez8xg/MLbP1hseHMj9c8dysLGVe97c0mfrVe6lBaCUl/iyop6/L9nJxaNTmTd5YJ+vf2R6NHfNHMqizftZUlTV5+tXrqcFoJQXaGnv5M6XNhDfL4Q/zBmBSPcu8nKVm88cxLCkSH791haa9NRQr6cFoJQX+PMH2ymuaeIvV4wiOty9B31PJijAxn2XjqCyvoW/LdlpWQ7lGloASnm41SW1PPXFXq6bMtAjZuocP9AxffRTX+xhZ1WD1XFUL2gBKOXB2jrs3PPmFtJiwvjpBTlWx/mPH503jIiQQO57b5vVUVQvaAEo5cEeX17C7upGfv/N4YQHe87kvXERwfzg3CF8sqOG5btqrI6jekgLQCkPVVrbxD+X7eKCEcmcm5NkdZyvuXZKJumxYdy7aBudOk2EV9ICUMoDGWP41VtbCQqw8euLh1sd57hCgwL46awcth9o4K2N+6yOo3pAC0ApD/Txjmo+3VnDnQXZJEe7/2rfnrpoZAq5KVH866PddHTarY6jukkLQCkP095p5w+LtjEoPoLrpmZaHeekbDbhjhnZ7DnYxNubKq2Oo7pJC0ApD/P86jJKapq4+8JcggI8/0/0vLwkclOi+OeyXboX4GU8/7dLKT9S39zO35fuZOrg/hTkJlod57TYbMKdBdnsrW3mzY26F+BNtACU8iD/+mgX9UfbueeiXMume+iJ8/KSyEmO5JFPi/XGMV5EC0ApD1F+qJmFK/dyxfh0hqdGWx2nW0SEW84axO7qRj7ZWW11HHWatACU8hD3L9uFiHDXzKFWR+mR2aNSSY0O5dFPS6yOok6TFoBSHmB3dSOvr69g3uSBpESHWR2nR4ICbNwwLYvVew6xqbzO6jjqNGgBKOUB/r50J6FBAXz37MFWR+mVuRMziAwN5LHluhfgDbQAlLLY1sp6Fm3ezw1nZBHfL8TqOL3SLySQayZl8P6X+yk/1Gx1HHUKWgBKWexvi3cSFRrITWcOsjqKS1w/NRMR4bnVZVZHUaegBaCUhdaXHWbZ9mpuOWsw0WHW3ejFlVKiwyjITeSltWW0tHdaHUedhBaAUhb657JdxEUEc72HT/nQXddOyeRwczuLNu+3Ooo6CS0ApSzyZUU9n+yoYf70LCJCPGeuf1eYOrg/gxIi+PeqUqujqJOwtABEZIGIVIvIFitzKGWFBz7eRVRoIPMmD7Q6isuJCPMmD2RTeR2bK+qsjqNOwOo9gKeBWRZnUKrP7TjQwIdbq7j+jCwiQ31j7P9Yl41PJzw4gGdW6l6Ap7K0AIwxnwGHrMyglBUe/Hg3EcEBfMfHxv67igoNYs6YNN7eVMmRlnar46jjsHoP4JRE5GYRKRSRwpoavfeo8n4lNY28u7mSb08ZSGxEsNVx3GruhAG0dth5W2cJ9UgeXwDGmMeMMfnGmPyEhASr4yjVaw9/UkxQgI3503zjvP+TGZUeTU5yJK8UllsdRR2HxxeAUr6k/FAzb2zYx9UTM0iI9O6rfk+HiHBF/gA2VdSz/cARq+OoY2gBKNWHHl9eggjccpbvv/v/yiVj0wgKEF5eW2F1FHUMq08DfQFYCQwTkQoRudHKPEq506GmNl4uLOeSsWleO+NnT8RFBDMzL4k3NlTQ1qG3jPQkVp8FdLUxJsUYE2SMSTfGPGllHqXc6d8r99LSbudmH5nzpzuuzB/A4eZ2lm6rsjqK6kKHgJTqA0fbOvn3ylJm5CQyJDHS6jh9bnp2AinRobysB4M9ihaAUn3g1fUVHGpq88t3/wABNuHScWl8trOGmoZWq+MoJy0Apdys0254YnkJowfEMDErzuo4lvnmmDTsBt7ZpNcEeAotAKXcbEnRAUprm7nlzEGIiNVxLJOdFMnw1Cje2rjP6ijKSQtAKTcyxvDoZyVkxIVz/vBkq+NY7pKxaWyqqKe4ptHqKAotAKXcqrD0MBvK6pg/PYsAm/+++//KxaNTEYG3NuhegCfQAlDKjR79tITY8CCuGD/A6igeISkqlDMGx/PmxkqMMVbH8XtaAEq5ye7qRpZuq2LelEzCggOsjuMx5oxJpexQM+vL6qyO4ve0AJRykyeWlxASaOO6Kb53w5femDUimZBAmx4M9gBaAEq5QXVDC6+v38fl49Pp38/3J33rjsjQIAryknhnUyXtnTo1hJW0AJRyg4Ur9tJutzN/un9e+HUqc0ancri5nRXFtVZH8WtaAEq5WHNbB8+uKuO8vCSy4iOsjuORzhyaQGRIIIs260VhVtICUMrFXl1XQf3Rdr+d9uF0hAYFMDMviQ+3VukMoRbSAlDKhTrthgWf72HMgBjGZcRaHcejXTgyhfqj7XxRfNDqKH5LC0ApF1q6rYq9tc3cNN2/p304HdOHxhMZEsh7m/dbHcVvaQEo5UJPLt9DWkwY5w9PsjqKxwsJDGDm8CQ+3HpAh4EsogWglItsKq9jzd5D3DAti8AA/dM6HbNHpXCkpYMvduswkBX0t1QpF3l8eQmRIYFcmZ9udRSvMW1IApGhgbyrw0CW0AJQygUqDjfz/pYDXD0pg8jQIKvjeI3gQBvn5SWzuOgArR2dVsfxO1oASrnA01/sBeD6qZmW5vBGs0el0KDDQJbQAlCqlxpa2nlxbTkXjUwhNSbM6jhe54wh8USFBvLuJh0G6mtaAEr10ktry2ls7WD+9Cyro3il4EAbM/OSWbqtSucG6mNaAEr1Qkennae+2MvErDhGpcdYHcdrnT88iSMtHawq0bmB+pIWgFK98P6WA+yrO8pNOulbr5w5NIGwoAA+3HrA6ih+RQtAqR4yxvDE8hKy4iOYkZNodRyvFhoUwNnDEli8tQq7Xe8U1le0AJTqocLSw2yqqOeGaVnY9H6/vXb+8GSqG1rZUF5ndRS/oQWgVA89/lkJMeFBXD5OL/xyhXNyEgm0CYt1GKjPaAEo1QN7DzaxZFsV3540UO/36yLRYUFMGdyfD7ce0BvG9xEtAKV6YMEXewiy2bh2qt7v15XOH57M3tpmdlY1Wh3FL2gBKNVNdc1tvFJYwZwxqSRGhlodx6ecl5eECHo2UB/RAlCqm55dVcrR9k5u1Au/XC4xKpRxGbF8sEULoC9oASjVDS3tnTz1xV7OHpZATnKU1XF80vnDkyjaf4TyQ81WR/F5gVYH8DYt7Z3srm5k+4EGdlY1UFl3lJqGVmqb2mjrsNNpN9hsEBMWTEx4EGkxYQxJ7MeQxH6MzYglOkxnivRmr66roLapjVvPGmx1FJ91/vBk7ntvOx9uPcB8vcDOrbQATqHTblhXepgvdh9kVUktG8rr/nP3ouBAG+kxYcRHhjA0qR/BATZsNsFuN9QdbedwcztFRVW8uLYcABHIS4lienYCs0elMDw1Sm8b6EU67YbHl5cwZkAMk7LirI7jswb2jyAnOZLFW6u0ANxMC+A42jvtrCiu5YMt+1m8tYrapjZsAsNTo7luykDGZsQyLDmSgXHhp3Xnp8NNbWw7cIQ1ew6xqqSWJ5aX8MinxQzsH87cCRnMnTCA2IjgPtgy1Rvvb9lPaW0zd1+Qo8XtZucPT+afH+3iYGMr8f1CrI7js7QAuiiuaeTlwnJeW7ePg42tRAQHcE5OIheMSGFadnyPh29iI4KZOjieqYPjAUchLC46wOvr9/HnD7bzj6U7uWRsGredO4T02HBXbpJyEWMMj3xazKD4CGbmJVsdx+fNzEvi/mW7+Gh7NVfmD7A6js/y+wJo67Cz6MtKnl9dxtq9hwmwCefmJHLF+HTOHJpAaJDrL/KJjQjmqgkZXDUhgx0HGli4ci+vrqvgtfUVXD0xg9vOHaKnF3qYFcW1bNl3hD9eOpIAnfbB7YanRpESHcrSoiotADfy2wKob27nuTWlLFyxl6ojrWTFR/DTWTlcNj6tT198hyVHct8lI7ntnCE88PFunl9dxuvr9/HDmUO5dspAvbm4h3jk02ISIkO4ZGya1VH8gohQkJvEq+sqaGnvdMsbMWXxaaAiMktEdojIbhH5WV+ss7S2iV+/tYXJf1zG/3ywg+zESJ76zgSW/fAsvnv2YMveeafGhHHfJSNZ8sOzGDcwlt+9W8TFD3zB5oo6S/Ko/7NlXz3Ldx3khjOy9IWoDxXkJXG0vZMVxXqrSHexbA9ARAKAB4GZQAWwVkTeNsYUuXpdxjjO5Hl8eQmLi6oItAnfGJ3GjdOyyEv1rHO5s+IjWPidCby/5QC/fWcrlz60gjtmZPPdswfr3oBFHv2shH4hgVwzKcPqKH5l8qA4IoIDWFJUzbk5SVbH8UlWDgFNBHYbY0oARORFYA7g8gL4xZtbeG51GdFhQXzv7MFcOyWTpCjPHWMXES4cmcIZg+P5xVtb+OuSnXy8o5r7545lQJweJO5Lew82sWhzJTdNH6TXcPSxkMAAzhqWwLJtVdjtI3TKbTew8i1lGlDe5eMK53P/RURuFpFCESmsqanp0YpmjUjm93OGs/Luc/nx+Tke/eLfVXR4EP+6eiz3zx3DrupGLn7gcz7b2bOfgeqZhz7ZTVCAjRun6bQPVijITaK6oZUtlfVWR/FJVhbA8er8a3PAGmMeM8bkG2PyExISerSi6dkJzJuSSXiwdx7znjMmjXdum0ZSZCjXPbWGBz/erdPl9oHyQ828vn4fV0/MINFL3jT4mnOGJWITWFpUZXUUn2RlAVQAXc/vSgcqLcri8TLjI3jj+1OZPSqVv3y4g9te2EBLe6fVsXzaI58WYxPhlrP0alSrxEYEk58Zx5Jt1VZH8UlWFsBaIFtEskQkGJgLvG1hHo8XHhzIP+eO4WcX5LBo837mPbmaw01tVsfySfvrj/JKYQVX5KeTEh1mdRy/NjM3iW37j1BxWCeHczXLCsAY0wHcBnwIbANeNsZstSqPtxARbj1rMA9cM5ZN5fVc9sgKymr1D8PVHv20BLsxfPdsnfTNajNyEwFYpnsBLmfpeYXGmPeMMUONMYONMfdamcXbzB6VyrPzJ1Hb2MalD69g+4EjVkfyGdUNLbywpoxLx6Xp1BweYFBCPwYlRLB0mx4HcDU9sdyLTcyK47XvTiXQJlz16Co2lddZHcknPP5ZCR12w/fPGWJ1FOU0MzeJVSW1HGlptzqKT9EC8HJDEvvxyq1TiAoL5FtPrGbNnkNWR/Jq1Q0tPLuqjDmjUxnYP8LqOMqpIC+J9k6jp0G7mBaADxgQF87Lt0whMSqEaxesZvku/SPpqYc+Lqat087tM7KtjqK6GJcRS2x4kB4HcDEtAB+REh3Gy7dMIbN/BPMXFrJit86f0l0Vh5t5bnUpV+YPIDNe3/17EscsvUl8tL2ajk671XF8hhaAD4nvF8LzN00ms38ENy4s1OGgbvrnsl2ICLfP0LF/TzQzL5H6o+0Ulh62OorP0ALwMXERwTw7fxKpMaF856k1rNM/ltNSXNPIq+sqmDd5oJ7376GmZycQHGDTq4JdSAvAByVEOvYEEiJDuH7BGj076DT8bclOwoIC+J6e9++xIkICmTqkP0u2VelUKC5ywgIQkfdEJLMPsygXSooK5fmbJhMTEcS8J1dTVKnXCZzI1sp6Fm3ez43Tsuiv95/1aAW5SZTWNrO7utHqKD7hZHsATwOLReQeEdF5cL1QakwYz8+fTERIINcuWMPeg01WR/I4xhjuXbSN2PAgbpyuc/54uq+uCl6qZwO5xAkLwBjzMjAWiAIKReRHIvLDrx59llD1yoC4cJ65cSKddjvffnI1B+pbrI7kUT7aXs2K4lruLBiq8/17gZToMEakRbFMrwp2iVMdA2gHmoAQIPKYh/ISQxIjWXjDRA43tekEcl20d9q5971tDEqI0Lt9eZGC3CTWlR2mtrHV6ihe72THAGYBG4FwYJwx5tfGmN9+9eirgMo1RqXH8Ph1+ZQeauY7T6+lqbXD6kiWe351GSU1TdxzYS5BertNr1GQm4Qxjr031Tsn+62/B7jCGPMzY4xON+kDpg6O54Grx/LlvnpueWYdrR3+ez+B+uZ2/rF0J2cM6c+5OYlWx1HdMDw1ipToUJ0czgVOdgxguk7P7HvOG57Mny8bxee7D3LnixvptPvn6XT3L9tF3dF27rkwDxG916w3ERFm5Cby2c6DelOkXtL9Xj90+fh0fjU7j/e3HODnr3/pd+dUb62s5+kVe7hmYgZ5qVFWx1E9UJCbxNH2TlYW11odxatpAfipG6Zlcfu5Q3ipsJz73tvmNyVgtxt+8eYW4iKC+cn5OVbHUT00ZXB/IoIDdBiol7QA/NhdM4dy/dRMHl++h399tNvqOH3ipcJyNpTV8fMLc4kO19M+vVVIYABnDk1gqV4V3CtaAH5MRPjV7DwuG5fO35bsZMHne6yO5Fa1ja386f3tTMqK45KxaVbHUb1UkJtE1ZFWtuzTq9x7KtDqAMpaNpvw58tG0tjazu/eLaJfaCBX5g+wOpZb/OadIprbOvjDN0fogV8fcE5OIjaBJduqGJkebXUcr6R7AIrAABv/vHos07Pj+dlrm3nvy/1WR3K597/czzubKrn93Gyyk/Q6Rl8QFxHM+IGxOjtoL2gBKMAxpvrovPGMzYjljhc38MkO37nIpraxlV+8uYURaVHcqrN9+pSC3CSK9h+hsu6o1VG8khaA+o/w4EAWXD+B7MRIbn12nU/cUMYYw6/e3sqRlnb+esUYveLXxxTkJQHo3EA9pH8N6r9EhwXx7xsnkhoTxo1Pr2VzRZ3VkXrllXUVLNq8nzsLhjIsWYd+fM3ghH5kxUewRGcH7REtAPU18f1CePbGSUSFBfGtJ1az0UtvKLOrqoFfv7WVqYP7c+tZOvTjqwpyE1lZfJCGlnaro3gdLQB1XKkxYbx0y2RiwoOY98Rq1pd5160lW9o7ue35DYQHB/CPq8YQYNOzfnxVQW4S7Z2G5bsOWh3F62gBqBNKjw3npZunENcvmGufXMO6Uu84JmCM4Z43trCjqoG/XjmaxKhQqyMpNxo/MJaY8CC9KrgHtADUSaXGhPHSzVNIiAzh2ifXsLrE8+deeWL5Hl5bX8HtM7I5e5jO9OnrAgNsnDsskY+3V9PRabc6jlfRAlCnlBwdyos3TyY5OpR5C9bwwZYDVkc6oY+3V3Pf+9u4cGQyd87ItjqO6iMzcpM43NzO+rI6q6N4FS0AdVqSokJ55dap5KVE8b3n1vHsqlKrI33N5oo6fvDCBoanRvHXK8Zg03F/v3Hm0HiCAkSHgbpJC0CdtriIYJ6/aRLnDEvkF29u4a+Ld2D3kPsJ7Kpq4LoFa4gJD+KJaycQFhxgdSTVhyJDg5g8qL9eFdxNWgCqW8KDA3l03niuyh/Avz7azfeeW0+jxbeX3HOwiXlPriEwwMZz8yeRHK0Hff3RzLwkSg42UVzTaHUUr6EFoLotMMDGny4byS8uymVx0QEuefAL9hxssiTL1sp6rnhkBe2ddp65cSID+0dYkkNZb0auXhXcXVoAqkdEhPnTB/HsjZM42NjKNx74nLc27uvTDCuKDzL3sVUEB9h4+dYp5CTr3b38WVpMGLkpUSwt0quCT5cWgOqVqUPiefu2aWQn9uOOFzfygxc2UN/s3isyjTEs+HwP855c4zg4/d2pDE7o59Z1Ku8wMzeRwtJDHGpqszqKV9ACUL02IC6cl2+Zwo/OG8r7X+6n4O+f8saGCrfcqam2sZXvP7+e371bxIycRN743lTSYsJcvh7lnQrykrAbx+nA6tS0AJRLBAbYuO3cbN743hmkRody10ubuOKRlRTudc3Vw3a74fX1FZz3989YWlTNT2fl8Mi3xxMZqrd1VP9nRGo0SVEhLNuuxwFOh94RTLnUyPRo3vjeGbyyrpz/+WAHlz+ykimD+nPzmYM4c2hCt+fk6ei0s3RbNfcv28W2/UcYnR7N/1w+Wmf2VMdlswkzcpN4a8M+Wjs6CQnU04FPRgtAuZzNJlw1IYOLR6fy/OoyHvushO88vZakqBAuHpXKmUMTmJgVR2jQ8f842zvtbK6oZ9m2Kt7aWMm+uqMMiAvj/rljuHhUql7gpU6qIDeR51eXsarkEGcNTbA6jkezpABE5ArgN0AuMNEYU2hFDuVe4cGBzJ8+iHlTBvLx9mpeKaxg4cq9PPH5HgJsQkZcOBlx4fQLDSTIJjS0dHDgSAu7qhpp67QTYBOmDu7PL2fnUZCbSKDezEWdhqmD4wkLCmBpUZUWwClYtQewBbgUeNSi9as+FBIYwKwRKcwakUJzWwdr9hxiXelhimsaKT90lPLDzbR32okKDSK+XwjThsQzMj2a6dkJRIfpGL/qntCgAKZnx7NsWxW/mzMcEd1jPBFLCsAYsw3Q/zF+KDw4kLOHJeosncqtCvKSWFxURdH+IwxPjbY6jsfy+H1qEblZRApFpLCmpsbqOEopL3BuTiIisHirng10Mm4rABFZKiJbjvOY053lGGMeM8bkG2PyExJ0PE8pdWrx/UKYMDDOo6cu9wRuGwIyxhS4a9lKKXUqs0Yk87t3iyipaWSQXil+XB4/BKSUUj0xa0QyAB9s1b2AE7GkAETkEhGpAKYAi0TkQytyKKV8V2pMGKPTo/lQh4FOyJICMMa8YYxJN8aEGGOSjDHnW5FDKeXbZo1IYVNFPfvqjlodxSPpEJBSymf9ZxhI9wKOSwtAKeWzsuIjyEmO1GGgE9ACUEr5tFkjkllbeojqhharo3gcLQCllE+bNSIZY/SisOPRAlBK+bRhSZFkxUfwoZ4O+jVaAEopnyYizBqRzMriWuqa9VaRXWkBKKV83qzhyXTYDUu36a0iu9ICUEr5vFHp0aTFhPH+l/utjuJRtACUUj5PRLhgRDKf7aqhvrnd6jgeQwtAKeUXLh6dSnun4cMiPRj8FS0ApZRfGJUeTUZcOO9sqrQ6isfQAlBK+QUR4eLRKaworuVgY6vVcTyCFoBSym9cPDqVTrvhfZ0aAtACUEr5kWFJkWQn9uOdjToMBFoASik/4hgGSmXN3kPsr9cporUAlFJ+5eLRqQAs2qzXBGgBKKX8SlZ8BCPTovVsILQAlFJ+6OLRjjuFldY2WR3FUloASim/M3uUYxjozQ3+vRegBaCU8jupMWFMGdSf1zdUYIyxOo5ltACUUn7p8vHplNY2U1h62OooltECUEr5pVkjkgkPDuC1dRVWR7GMFoBSyi9FhARywYgUFm3eT0t7p9VxLKEFoJTyW5eNT6OhtcNvbxepBaCU8luTs/qTFhPGq346DKQFoJTyWzabcOm4NL7YfZAD9S1Wx+lzWgBKKb926bh07AZe3+B/ewFaAEopv5YVH8GkrDheXFOO3e5f1wRoASil/N41kzIoO9TM57sPWh2lT2kBKKX83qwRycRFBPP86jKro/QpLQCllN8LCQzgivHpLNlWRdUR/zkYrAWglFLA1RMz6LQbXl5bbnWUPqMFoJRSQGZ8BNOGxPPCmjI6/eRgsBaAUko5XTMpg8r6Fj7eXm11lD6hBaCUUk4z85JIiQ5lwRd7rI7SJ7QAlFLKKSjAxnVTM1lRXMvWynqr47idFoBSSnVx9YQMwoMDePJz398LsKQAROQvIrJdRDaLyBsiEmNFDqWUOlZ0eBBX5g/gnU2VVPv4KaFW7QEsAUYYY0YBO4G7LcqhlFJf850zMumwG/69stTqKG5lSQEYYxYbYzqcH64C0q3IoZRSxzOwfwTn5SXxzKpSGlrarY7jNp5wDOAG4H2rQyilVFffP2cI9UfbeWaV7+4FuK0ARGSpiGw5zmNOl6+5B+gAnjvJcm4WkUIRKaypqXFXXKWU+i+j0mM4Z1gCTyzfQ1Nrx6m/wQu5rQCMMQXGmBHHebwFICLXAbOBbxljTnjZnTHmMWNMvjEmPyEhwV1xlVLqa34wI5tDTW0866N7AVadBTQL+CnwDWNMsxUZlFLqVMZlxDI9O57HPiuhuc339gKsOgbwABAJLBGRjSLyiEU5lFLqpO4syKa2qY0FPnhdgFVnAQ0xxgwwxoxxPm61IodSSp3K+IFxnD88iYc/KaamobXP119/tJ2b/13IzqoGly/bE84CUkopj/bTWTm0dti5f9nOPl/3Q5/sZsm2Kto67C5fthaAUkqdwqCEfnxrUgYvrClnd7Xr34mfSPmhZp76Yi+XjE1jRFq0y5evBaCUUqfh9hnZhAcH8Ms3t3KSExdd6n8X70CAH503zC3L1wJQSqnT0L9fCD+7IIeVJbW8uq7C7etbWVzLWxsrmT89i9SYMLesQwtAKaVO09UTMsgfGMu9722jttF9B4RbOzq5540vGRAXxm3nZLttPVoASil1mmw24Y+XjqSptYOfv/Gl24aCHvy4mJKDTdz7zZGEBQe4ZR2gBaCUUt2SnRTJT87P4cOtVTy7uszly19XepgHP97NJWPTOHOoe2c/0AJQSqluunFaFmcOTeD37xZRVHnEZcutP9rO7S9sIDUmlN/OGe6y5Z6IFoBSSnWTzSb89YrRxIYHMX/hWpfcOKaj086dL27gwJEW7p87lqjQIBckPTktAKWU6oGEyBCevG4CdUfbuXFhIY29mDHUGMPv3y3i4x01/G7OcMZlxLow6YlpASilVA+NSIvmX1ePpWj/Ea59cjVHenDzGGMMf/5gBwtXlnLT9Cy+NWmgG5IenxaAUkr1wozcJB68Zhxf7qvnqkdXUVZ7+hMct3XYuefNLTzyaTHfnpzB3RfkujHp12kBKKVUL80akcyT101g3+FmZv9rOa+uq8BuP/kpojurGrjy0ZU8v7qM7549mN/PGYHNJn2U2EH66pJmV8jPzzeFhYVWx1BKqeMqq23mjpc2sKGsjlHp0Vw/NZNzcxKJCQ8GHAd6N1XU8eKact7YsI9+oYHc+82RXDQqxa25RGSdMSb/a89rASillOvY7YbX1lfw8CeOi7kAkqJCCA60UX2kldYOO2FBAVyZn84dBUOJiwh2e6YTFUCg29eslFJ+xGYTrsgfwGXj0tlQfphVJYcorW2ircNOQmQIYwbEMi07nugw95/meSpaAEop5QY2mzB+YBzjB8ZZHeWE9CCwUkr5KS0ApZTyU1oASinlp7QAlFLKT2kBKKWUn9ICUEopP6UFoJRSfkoLQCml/JRXTQUhIjVAaQ+/PR446MI43kC32T/oNvuH3mzzQGPM1+4v6VUF0BsiUni8uTB8mW6zf9Bt9g/u2GYdAlJKKT+lBaCUUn7KnwrgMasDWEC32T/oNvsHl2+z3xwDUEop9d/8aQ9AKaVUF1oASinlp/yiAERklojsEJHdIvIzq/O4mogMEJGPRWSbiGwVkTucz8eJyBIR2eX8b6zVWV1NRAJEZIOIvOv82Ke3WURiRORVEdnu/P89xQ+2+S7n7/UWEXlBREJ9bZtFZIGIVIvIli7PnXAbReRu5+vZDhE5v6fr9fkCEJEA4EHgAiAPuFpE8qxN5XIdwP8zxuQCk4HvO7fxZ8AyY0w2sMz5sa+5A9jW5WNf3+b7gQ+MMTnAaBzb7rPbLCJpwO1AvjFmBBAAzMX3tvlpYNYxzx13G51/23OB4c7vecj5OtdtPl8AwERgtzGmxBjTBrwIzLE4k0sZY/YbY9Y7/92A40UhDcd2LnR+2ULgm5YEdBMRSQcuAp7o8rTPbrOIRAFnAk8CGGPajDF1+PA2OwUCYSISCIQDlfjYNhtjPgMOHfP0ibZxDvCiMabVGLMH2I3jda7b/KEA0oDyLh9XOJ/zSSKSCYwFVgNJxpj94CgJINHCaO7wD+AngL3Lc768zYOAGuAp57DXEyISgQ9vszFmH/C/QBmwH6g3xizGh7e5ixNto8te0/yhAOQ4z/nkua8i0g94DbjTGHPE6jzuJCKzgWpjzDqrs/ShQGAc8LAxZizQhPcPfZyUc9x7DpAFpAIRIvJta1NZzmWvaf5QABXAgC4fp+PYhfQpIhKE48X/OWPM686nq0Qkxfn5FKDaqnxucAbwDRHZi2NY71wReRbf3uYKoMIYs9r58as4CsGXt7kA2GOMqTHGtAOvA1Px7W3+yom20WWvaf5QAGuBbBHJEpFgHAdP3rY4k0uJiOAYF95mjPlbl0+9DVzn/Pd1wFt9nc1djDF3G2PSjTGZOP6ffmSM+Ta+vc0HgHIRGeZ8agZQhA9vM46hn8kiEu78PZ+B4xiXL2/zV060jW8Dc0UkRESygGxgTY/WYIzx+QdwIbATKAbusTqPG7ZvGo5dwM3ARufjQqA/jrMHdjn/G2d1Vjdt/9nAu85/+/Q2A2OAQuf/6zeBWD/Y5t8C24EtwDNAiK9tM/ACjmMc7Tje4d94sm0E7nG+nu0ALujpenUqCKWU8lP+MASklFLqOLQAlFLKT2kBKKWUn9ICUEopP6UFoJRSfkoLQKkecs7CukdE4pwfxzo/Hmh1NqVOhxaAUj1kjCkHHgb+5HzqT8BjxphS61Ipdfr0OgClesE5Bcc6YAFwEzDWOGadVcrjBVodQClvZoxpF5EfAx8A5+mLv/ImOgSkVO9dgOMy/hFWB1GqO7QAlOoFERkDzMRxJ7a7vpq9USlvoAWgVA85Z6d8GMf9F8qAv+C4eYlSXkELQKmeuwkoM8YscX78EJAjImdZmEmp06ZnASmllJ/SPQCllPJTWgBKKeWntACUUspPaQEopZSf0gJQSik/pQWglFJ+SgtAKaX81P8HF6rwCgQDdzEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sample_points = 3e3\n",
    "x_lim = 100\n",
    "x = np.linspace(0, x_lim, int(sample_points))\n",
    "y = np.sin(x * x_lim * 1e-4) * np.cos(x * x_lim * 1e-3) * 3\n",
    "plt.plot(x, y)\n",
    "plt.xlabel('X')\n",
    "plt.ylabel('Y')\n",
    "plt.title('Function to be fitted')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a73959ae",
   "metadata": {},
   "source": [
    "## 0.3. Data shape\n",
    "- Our data is 1D, meaning it has only one feature.\n",
    "- We want a model that for a given $x$ it returns the correspondent $y$ value.\n",
    "- This means that a model with one neuron input and a one neuron output suffices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 582,
   "id": "13d603ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_input = 1\n",
    "n_out = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ba45463",
   "metadata": {},
   "source": [
    "In order for the model to take each point of the data one by one we need to do some additional re-shaping, where we introduce an additional dimension for each entry:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 583,
   "id": "5726813c",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_reshape = x.reshape((int(len(x) / n_input), n_input))\n",
    "y_reshape = y.reshape((int(len(y) / n_out), n_out))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 584,
   "id": "e448eb81",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Uncomment to check the shape change\n",
    "# print(x.shape, y.shape)\n",
    "# print(x_reshape.shape, y_reshape.shape)\n",
    "# print(x[10], x_reshape[10])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4335bb62",
   "metadata": {},
   "source": [
    "## 0.4. Data type\n",
    "The data that we will input to the model needs to be of the type `torch.float32`\n",
    "\n",
    "_Side Remark_: The default dtype of torch tensors (also the layer parameters) is `torch.float32`, which is related to the GPU performance optimization. If one wants to use `torch.float64`/`torch.double` instead, one can set the tensors to double precision via `v = v.double()` or set the global precision via `torch.set_default_dtype(torch.float64)`. Just keep in mind, the NN parameters and the input tensors should have the same precision.\n",
    "\n",
    "Before starting, let's convert our data numpy arrays to torch tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 585,
   "id": "b413e6fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_torch = torch.from_numpy(x_reshape)\n",
    "y_torch = torch.from_numpy(y_reshape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 586,
   "id": "57ce4393",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "float64 float64\n",
      "torch.float64 torch.float64\n"
     ]
    }
   ],
   "source": [
    "# Type checking:\n",
    "print(x.dtype, y.dtype)\n",
    "print(x_torch.dtype, y_torch.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "794384e7",
   "metadata": {},
   "source": [
    "The type is still not correct, but we can easily convert it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 587,
   "id": "acce223d",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_torch = x_torch.to(dtype=torch.float32)\n",
    "y_torch = y_torch.to(dtype=torch.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 588,
   "id": "6597c669",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "float64 float64\n",
      "torch.float32 torch.float32\n"
     ]
    }
   ],
   "source": [
    "# Type checking:\n",
    "print(x.dtype, y.dtype)\n",
    "print(x_torch.dtype, y_torch.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b139da4",
   "metadata": {},
   "source": [
    "## 0.4. Data normalization\n",
    "We will also need to normalize the data to make sure we are in the non-linear region of the activation functions:\n",
    "<img src=\"img/activation-functions.png\" width=\"85%\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 589,
   "id": "6cb2f611",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_norm = torch.nn.functional.normalize(x_torch, p=5, dim=0)\n",
    "y_norm = torch.nn.functional.normalize(y_torch, p=5, dim=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29c1044a",
   "metadata": {},
   "source": [
    "The [`torch.nn.functional.normalize`](https://pytorch.org/docs/stable/generated/torch.nn.functional.normalize.html) function performs $L_p$ normalization, where the $L_p$ norm is:\n",
    "\n",
    "$$\n",
    "||x||_p = (\\sum_{i=1}^n |x_i|^p)^{1/p} \\ \\ \\ p>0\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 590,
   "id": "03c39d09",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Normalized function')"
      ]
     },
     "execution_count": 590,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEICAYAAABfz4NwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAvHklEQVR4nO3dd3yV5f3/8dfnZJGEhISQPQhhJqwAAQEVHEFBURwoWmerInbYqbW732qr1v7a2tYNLmzdWxAFFWRDUPbMICSsDCCL7Fy/P3Jo0zRAknNy7jM+z8fjPHLOucf1uXPgvHOP67rFGINSSills7oApZRS7kEDQSmlFKCBoJRSyk4DQSmlFKCBoJRSyk4DQSmlFKCBoHyEiCwXkTvtz28SkU+dvP5UETEi4n+a6UNF5GsRqRKRe53Z9lnqShGRahHxc1WbynNpICinEJH9InJURELbvHeniCy3sKwOGWP+aYy5xMXN3g8sN8aEGWP+1lON2D+H7FOvjTEHjDG9jTHNPdWm8h4aCMqZ/IHvO7oSaeVt/zb7AzusLkKpM/G2/3TKWo8BPxGRiI4mishkEdkoIhX2n5PbTFsuIr8XkdXASSDNfgjm2yKyz36o5UERGSgia0WkUkTeEJFA+/KRIvKRiJSKyHH786TT1HG7iKyyP7/ffkjl1KNRRF60T+sjIgtE5LCIHBSRh04dehERPxH5k4iUiUg+cPnpfiki8jlwIfAPextD2h7Cal+T/bURkXn2bT8uIk+IiLSZfpeI7LL/XnaKyFgRWQikAB/a27m//aEsEUkQkQ9E5JiI5IrIXW3W+Vv77/Rl+3p3iEjW6bZLeR8NBOVMOcBy4CftJ4hIX2AR8DcgCvgzsEhEotrMdgswFwgDCu3vTQfGARNpPezyLHATkAyMAG60z2cDXqD1L/EUoBb4x9kKNsb80X5IpTeQDpQCb9gnvwQ0AYOAMcAlwKkv8buAmfb3s4DZZ2jjImAl8F17W3vPVpfdTGA8MBq4HrgUQESuA34L3AqEA1cC5caYW4ADwBX2dv7YwTpfBYqBBHvNfxCRi9tMvxJ4DYgAPqATv0PlPTQQlLP9GvieiES3e/9yYJ8xZqExpskY8yqwG7iizTwvGmN22Kc32t971BhTaYzZAWwHPjXG5BtjKoCPaf1CxhhTbox52xhz0hhTBfwemNrZokUkGHgPeNwYs1hEYoEZwA+MMTXGmBLgL8AN9kWuB/5qjCkyxhwDHu5sW13wiDHmhDHmAPAFkGl//07gj8aYjaZVrjGm8LRrsRORZOA84KfGmDpjzGZgPq1BfMoqY8xi+zmHhbSGkfIRHV4RoVR3GWO2i8hHwAPArjaTEvjPX/2nFAKJbV4XdbDKo22e13bwOg5AREJo/cKeDkTap4eJiF8nT6guAPYYYx61v+4PBACH2xypsbWpMaFdvWf9Qu6GI22enwR6258nA3ndWF8CcMwemKcU0rqHc7o2e4mIvzGmqRvtKQ+jewiqJ/yG1kMqbb/sD9H6JdtWCnCwzWtHht79MTAUOMcYEw5Msb8vp1/EPoPIA/Zl72jzdhFQD/QzxkTYH+HGmOH26Ydp/WI+JaWL9dYAIW1ex3Vh2SJg4Gmmnel3eAjoKyJhbd5r/xkoH6aBoJzOGJMLvA60vd5+MTBERL4hIv4iMgfIAD5yUrNhtO4xnLCfr/hNZxYSkRn2Oq8yxtS22YbDwKfA/xORcBGx2U9onzoM9QZwr4gkiUgkrXtEXbEZuEZEQkRkEP8dRmczn9aT9+PsV2QNEpFTYXsUSOtoIWNMEbAGeFhEeonIKHu7/+xi7cpLaSConvI74N99Eowx5bSeJP0xUE7rCeKZxpgyJ7X3VyAYKAPWAUs6udwcIBrY1eZKo6ft024FAoGdwHHgLSDePu054BNgC/AV8E4X6/0L0EDrF/hLdOFL2RjzJq3nSP4FVNF67qOvffLDwC9F5ISI/M/JfVpPwqfSurfwLvAbY8zSLtauvJToDXKUUkqB7iEopZSy00BQSikFaCAopZSy00BQSikFuHnHtH79+pnU1FSry1BKKY+xadOmMmNM+5ECOsWtAyE1NZWcnByry1BKKY8hIt3uNa+HjJRSSgEaCEoppew0EJRSSgEaCEoppew0EJRSSgEaCEoppew0EJRSSgFu3g9BKdV1Tc0t7D5Sxc5DlZTV1NPSYugbGsTQuDBGJvYh0F//DlQd00BQykvsL6vhhdUFfLT1MOU1DR3OExrox+Wj4rl98gAyEsJdXKFydxoISnm4YzUNPPLxLt7+6iB+NmFaRiyXZMQyKimC+D69sIlQWl3PtuIKPt99lEVbD/PmpmJmjU7gF5dnEB0WZPUmKDfh1jfIycrKMjp0hVKnt2T7EX7x7jYq6xq5dVIqd09NIyas1xmXqaht5JkVecxfVUBYkD+PXDuKaRmxLqpY9TQR2WSMyerOsnowUSkP1NxieHTJbua9somEiGA+/N55/GpmxlnDAKBPcAD3Tx/Gou+dR3xEL+56OYcnl+fizn8cKtfQQFDKw9Q3NfPtf27iqeV53DghmbfvmcywuK6fDxgcG8Zb8yZzxegE/rhkD7/9YIeGgo/TcwhKeZC6xmbmvbKJ5XtK+fXMDL55bioi0u319Qrw4/E5mcSGBTF/VQF+Nhu/mpnu0DqV59JAUMpDNDa3cPfCTXy5r5RHrhnJDRNSnLJem034xeXpNLUYnl9dQGRIAN+7eLBT1q08i1MOGYnIdBHZIyK5IvJAB9NvEpGt9scaERntjHaV8hXGGH72zjZW7C3l4audFwaniAi/uSKDq8ck8v+W7uXjbYedun7lGRwOBBHxA54AZgAZwI0iktFutgJgqjFmFPAg8Kyj7SrlS/6ybB9vbSrmh9lDnB4Gp4gID18zkrEpEfzwjc3sPlLZI+0o9+WMPYQJQK4xJt8Y0wC8BsxqO4MxZo0x5rj95TogyQntKuUTPtlxhL99to/rxiVx78WDerStXgF+PHNLFr2DArj31a+pbWju0faUe3FGICQCRW1eF9vfO507gI9PN1FE5opIjojklJaWOqE8pTzX/rIafvLGFkYl9eGhq0e45GRvdFgQf75+NHuPVvPQop093p5yH84IhI7+hXZ47ZqIXEhrIPz0dCszxjxrjMkyxmRFR3frPtFKeYVTVxT5+QlP3jSWIH8/l7U9ZUg0d09J45/rD/DFnhKXtaus5YxAKAaS27xOAg61n0lERgHzgVnGmHIntKuUV3t0yW52H6niL3MySYoMcXn7P7pkCINievPLd7dTU9/k8vaV6zkjEDYCg0VkgIgEAjcAH7SdQURSgHeAW4wxe53QplJebU1eGS+s3s9tk/pz4dAYS2oI8vfj0WtHcqiilj99useSGpRrORwIxpgm4LvAJ8Au4A1jzA4RmSci8+yz/RqIAp4Ukc0iogMUKXUaVXWN3PfmVgb0C+WBGemW1jKuf19umdifF9fsZ1txhaW1qJ6ng9sp5WZ++tZW3txUxFv3TGZsSqTV5VBZ18iFjy0nLTqUN+6epL2Y3ZwObqeUl1ifX87rOUXcNSXNLcIAILxXAD++ZCgb9x9n8bYjVpejepAGglJuoqGphV++t52kyGB+cPEQq8v5L3PGJzMsLow/LN5FXaP2TfBWGghKuYn5q/LZV1LN72YNJzjQdZeYdoafTfj1zAwOnqjlpTX7rS5H9RANBKXcQNGxk/zts31cOjyWi4a5581qJg/qx5Qh0Ty9Io9qvQzVK2kgKOUG/rB4FzYRfnPFcKtLOaMfTRvC8ZONvLi6wOpSVA/QQFDKYhsKjvHx9iPcM3UgCRHBVpdzRpnJEWSnx/Dsl/lU1DZaXY5yMg0EpSzU0mJ4aNFO4vv04s7z06wup1N+OG0IlXVNLFiZb3Upysk0EJSy0PtbDrK1uIL7Lh3qdieST2d4Qh+mD4/jxTX79VyCl9FAUMoitQ3N/HHJHkYm9uGqzDMNEOx+7rlgIJV1Tby6/oDVpSgn0kBQyiLPry7gcEUdv7w8HZvNs3r/jk6OYFJaFAtWFdDQ1GJ1OcpJNBCUskDFyUaeXpFHdnoM56RFWV1Ot9w9NY0jlXW8t/mg1aUoJ9FAUMoCz63Mp6quiR9NG2p1Kd02dUg06fHhPPtlPi0t7jsmmuo8DQSlXKysup7nVxcwc1Q8GQnhVpfTbSLCvKlp5JZU6010vIQGglIu9vTyPOoam/lBtnuNV9Qdl42MJzY8iBd1OAuvoIGglAsdqajj5XWFXD0miUExva0ux2EBfjZuOqc/K/eVkVdabXU5ykEaCEq50D++2EdLi+EH2YOtLsVpbpyQQqCfjYVrC60uRTlIA0EpFzl4opbXNxZx/fhkkvu6/h7JPSU6LIjLR8Xz1qZi7ajm4TQQlHKRZ1fkYQx858JBVpfidLdNTqW6vol3viq2uhTlAA0EpVygpKqOVzcWcc3YRBLdfAC77shMjmB0cgQvrdmPO9+WV52ZBoJSLrBgZQFNzS3cc4H37R2ccvM5KeSV1pBTeNzqUlQ3aSAo1cOO1zTwyrpCrhidwIB+oVaX02MuHxVP7yB/XttQZHUpqps0EJTqYS+s2U9NQzPf9uK9A4CQQH+uzExg0bZDVNbpvRI8kQaCUj2oqq717mKXDo9laFyY1eX0uBvGJ1PX2ML7mw9ZXYrqBg0EpXrQwnWFVNY18d0LvaffwZmMTOxDenw4r2/UYbE9kQaCUj2ktqGZBSsLmDokmpFJfawuxyVEhBvGJ7P9YCXbD1ZYXY7qIg0EpXrIm5uKKK9p4LsXefe5g/auykwk0N/G6xv15LKn0UBQqgc0txjmryxgbEoE41P7Wl2OS/UJCWDGiDje23yQusZmq8tRXaCBoFQP+GTHEQ4cO8ncKQOtLsUS14xNoqquiS9267DYnsQpgSAi00Vkj4jkisgDHUwfJiJrRaReRH7ijDaVclfGGJ75Mp/UqBCmZcRaXY4lzh0YRXRYEO98rXdT8yQOB4KI+AFPADOADOBGEcloN9sx4F7gT462p5S727j/OFuKTnDH+Wn4edi9kp3F38/GrNEJLN9TwvGaBqvLUZ3kjD2ECUCuMSbfGNMAvAbMajuDMabEGLMR0N4qyus9+2UefUMDmT02yepSLHX12EQamw0fbdU+CZ7CGYGQCLS9nKDY/l63iMhcEckRkZzS0lKHi1PKlXJLqlm2q4RbJvYnONDP6nIslREfztDYMD1s5EGcEQgd7RN3e7hDY8yzxpgsY0xWdHS0A2Up5XrzV+YT5G/j1kn9rS7FciLC1WMT+frACQrKaqwuR3WCMwKhGEhu8zoJ0H1E5XNKqup456uDzB6XRFTvIKvLcQuzMhMQgXd1L8EjOCMQNgKDRWSAiAQCNwAfOGG9SnmUl9cU0tjSwp3np1ldituI7xPMpLQo3vv6oN4nwQM4HAjGmCbgu8AnwC7gDWPMDhGZJyLzAEQkTkSKgR8BvxSRYhEJd7RtpdzFyYYmFq4r5JKMWK8e4ro7rhqTyIFjJ9lSrENZuDt/Z6zEGLMYWNzuvafbPD9C66EkpbzSGxuLqKht9NmOaGdyaUYcv/DbxkdbDpGZHGF1OeoMtKeyUg5qam5h/qoCxvWPZFz/SKvLcTt9QgKYMjiaRdsO09Kih43cmQaCUg5asuMIxcdrmTtFzx2czszR8RyuqOOrA3p7TXemgaCUA4wxPPdlPgP6hZKd7pvDVHRGdnosgf42Ptp62OpS1BloICjlgI37j7OluII7zhvgs8NUdEZYrwAuGBLN4m2HadbDRm5LA0EpBzy3Mp/IkACu9fFhKjpj5ugESqrq2bj/mNWlqNPQQFCqmwrKali26yg36zAVnXLxsBh6Bdh0bCM3poGgVDc9v6qAAJuNW3SYik4JDfLn4mGxfLztCE3NLVaXozqggaBUNxyvaeDNTUXMykwgJqyX1eV4jJmj4imvaWBdvh42ckcaCEp1w782HKCuUYep6KoLh8UQGujHom162MgdaSAo1UX1Tc28uGY/U4ZEMzQuzOpyPEqvAD8uGBbD0p1H9WojN6SBoFQXfbD5EKVV9dx53gCrS/FIM0bEUVbdwKZC7aTmbjQQlOoCYwwLVhUwLC6M8wf3s7ocj3TB0BgC/W18vF07qbkbDQSlumBVbhm7j1Rxx3kDENGOaN3RO8ifKYP78cn2IzoktpvRQFCqC55bWUB0WBBXZiZYXYpHu3R4HIcq6th2UIfEdicaCEp10p4jVXy5t5TbJvUnyF87ojkiOz0WP5uwZPsRq0tRbWggKNVJC1bl0yvAxk3naEc0R0WGBjIpLYoletjIrWggKNUJpVX1vPf1IWaPSyIyNNDqcrzCpSPiyC+rIbek2upSlJ0GglKdsHDtfhpbWvjWuXqpqbNcmhGLCHrYyI1oICh1FrUNzSxcV8jFw2JJi+5tdTleIya8F2NTIlmyQwPBXWggKHUW73xdzPGTjdx1vu4dONv04XHsOFRJ0bGTVpei0EBQ6oxaWgwLVhYwKqkPEwb0tbocrzN9RBwAn+heglvQQFDqDD7deZT8shruOj9NO6L1gOS+IWTEh+t5BDfhb3UB7q68up7dR6rYc6SK3NJqSirrKa2up7K2kaaWFlpaICjARmRIIH1DA0mNCmFQTG8y4vuQHh+Gv59mrqcyxvD0ijxS+oYww/6XrHK+aRmx/O3zfZRX1xPVO8jqcnyaBkI7FScbWbGvlHX55azLLye/tObf0yJCAojvE0x0WBApfUPwtwk2EeoamzlR28CB8pN8ubeU+qbWm3/0DvInKzWS7PRYpo+Io5/+Y/co6wuOsbnoBA9eNUKDvQdNy4jl8c/28dnuEq7PSra6HJ+mgQCUVdfz6Y6jfLz9MGvzymlqMYQF+TN+QF/mZCUzIrEPQ2LDiA47+xd6c4vh4PFaNhefYH1+OWvyyvnle9v59fvbOXdQP26dlMpFw2L0huwe4OkVeUSFBnLdOL1fck8anhBOQp9eLNt5VAPBYj4bCE3NLXy5r5TXNhTx+e4SmloM/aNCuOP8AVw6PI5RiX269Vehn01IiQohJSqEK0cnYIxh95EqFm09zFubirnr5RySIoOZOyWNOeOTdQgEN7XrcCXL95Ty42lD6BWgn1FPEhGyM2J5M6eYusZm/X1byOcC4UhFHa+sK+TNTUUcraynX+9AvnXeAK4ek8iwuDCnnzgUEdLjw0mPD+f72YNZuvMoz68q4Nfv7+CZFfnce/EgZo9L1j0GN/PMijxCAv30fskukp0ey8trC1mdW8bF6bFWl+OzfCYQdhyqYP7KAj7ccogWY5g6JJr/uzKFi9NjCHDR8eEAPxuXjYxnxog4VuWW8adP9/LTt7excF0hD84awZiUSJfUoc6s6NhJPtx6mNsnpxIRosNUuMLEtCh6B/mzdOdRDQQLOSUQRGQ68DjgB8w3xjzSbrrYp18GnARuN8Z85Yy2z6SlxbB8bwnPfVnA2vzyf//F983JA0iJCunp5k9LRDh/cDTnDerHR1sP8+BHO7nmqTXcOCGFn80YRlivAMtqU7BgVQEC3KF3RHOZQH8bU4dGs2xXCS0tBpvuMVvC4UAQET/gCWAaUAxsFJEPjDE728w2Axhsf5wDPGX/2SPqGpt556uDLFiVT15pDXHhvfjZjGHcMCGFPsHu82UrIlwxOoELhkbz12X7eGF1AV/uLeUvczIZn6qdoKxwvKaB1zcWMSszkYSIYKvL8SmXZMSyaOthNhefYKzuLVvCGcdKJgC5xph8Y0wD8Bowq908s4CXTat1QISIxDuh7f9RVdfIeY9+zs/f3UZwoB+P35DJyp9eyN1TB7pVGLQV1iuAX83M4I27J2ETYc4za3nsk900NbdYXZrPeXHNfmobm5k3Nc3qUnzOBUNar75btvOo1aX4LGcEQiJQ1OZ1sf29rs4DgIjMFZEcEckpLS3tcjFhvQL45rkDeG3uRD787nnMykx02TkCR2Wl9mXx989n9rgknvgij1sWbKC8ut7qsnxGVV0jL6wuYFpGLINjw6wux+f0CQngnAF9WaqBYBlnfFN2dLCv/R0vOjNP65vGPGuMyTLGZEVHR3eroO9cOIiJaVEeOdRA7yB//jh7NH+6bjRfHTjOFX9fxZaiE1aX5RNeXltIZV0T91402OpSfFZ2eiz7SqrZX1Zz9pmV0zkjEIqBtr1JkoBD3ZhHtTF7XBJv3zMZEeG6p9fywRb9dfWkmvom5q/M58Kh0YxM6mN1OT5rWkbrFUbLduleghWcEQgbgcEiMkBEAoEbgA/azfMBcKu0mghUGGMOO6FtrzYisQ8ffe88MlMiuPfVr3lqeZ7ebrCHvLKukOMnG/nexbp3YKXkviEMiwvjUz1sZAmHA8EY0wR8F/gE2AW8YYzZISLzRGSefbbFQD6QCzwHfNvRdn1FZGggC++YwBWjE3h0yW5+9f52PdnsZLUNzTy3Mp/zB/fTq1vcwLSMWHL2H+N4TYPVpfgcp/RDMMYspvVLv+17T7d5boDvOKMtXxTk78fjczJJjAjm6RV5lFTW8/dvjNFhL5zkXxsOUFbdwL26d+AWstNj+fvnuXy+u4RrdRwpl/KMy28UNpvwwIxh/PaKDD7deZS7Xt5EbUOz1WV5vLrGZp5ZkcfEtL7a98NNjEzsQ0xYkJ5HsIAGgoe5/dwB/PHaUazcV8ptL2ygqq7R6pI82usbiyipqte9Azdis7UOdrdibyl1jfpHjytpIHig68cn8/gNY/iq8Dg3z19PxUkNhe442dDE3z/P5ZwBfZmUFmV1OaqNaemxnGxoZm1+udWl+BQNBA915egEnrp5HLsOV3Hr8+up1D2FLntpTSFl1fXcP32oR/ZZ8WaTBkYRHOCnvZZdTAPBg03LiOXJm8ay41Al33xhI9X1TVaX5DEqaht5ekUeFw+LYVx/PXfgbnoF+DFlSD8+21Wil1q7kAaCh8vOiOUf3xjD5qITfOvFjZxs0FDojPkr86mobeTHlwy1uhR1GtnpsRyprGPHoUqrS/EZGgheYPqIeP4yJ5Oc/ce486UcPRF3FmXV9SxYVcDMUfFkJIRbXY46jYuGxSCCjm3kQhoIXuLK0Qn86brRrM0v5+6Fm2ho0s5rp/PkF3nUN7Xwo2lDrC5FnUFU7yDGpUTq5acupIHgRa4Zm8TDV49kxd5SfvjGZppb9Nhre4XlNbyyrpDZY5NIi+5tdTnqLLIzYtlxqJJDJ2qtLsUnaCB4mRsmpPDzy4axaOthfvX+dj0h186jS3bj7yf86BLdO/AE2fbbaX6mewkuoYHgheZOGci3LxjIv9Yf4LFP9lhdjtvYUHCMxduOMG/qQGLDe1ldjuqEgdGhDOgXytJdJVaX4hOcMpaRcj/3XTqUE7WNPLk8j4iQAOZOGWh1SZZqaTE8tGgnceG9uOt8vRuapxARstNjeHHNfqrqGvV+4z1M9xC8lIjw4KwRzBwVzx8W7+b1jQesLslS7285yNbiCu6fPpTgQB0U0JNkp8fS2GxYua/M6lK8ngaCF/OzCX++PpOpQ6L52TvbWLLdN29BUVPfxKMf72FUUh+uyuzwzq3KjY3rH0lESID2WnYBDQQvF+hv46mbxzImJZJ7X93MKh/8K+uvy/ZypLKO31wxHJtNh6jwNP5+Ni4aGsPne0r0XiA9TAPBB4QE+vP8beNJiw5l7sIcNhUet7okl9l9pJLnV+/nxgnJjOuvN7/xVNkZsZw42ehT/3atoIHgI/qEBPDyHROICQvimy9sYKcPDAfQ0mL45bvb6RMcwP2XDrO6HOWA8wf3I8BPtJNaD9NA8CExYb145c5zCA3y59bn15NfWm11ST3qrU3F5BQe54EZw4gMDbS6HOWAsF4BTEyLYunOo9q3pgdpIPiYpMgQFt5xDsbAzfPXc9BLe4CWVNbx+8W7GJ8ayeyxehtGbzAtI5b95SfJK62xuhSvpYHggwbF9Oalb02gqr6Jm+evp7Sq3uqSnMoYw8/f3UZdYzOPXjtKTyR7iYu113KP00DwUSMS+/DC7eM5UlHHLQu8665r73x1kGW7Srjv0qE6XpEXSYwIJiM+XM8j9CANBB+WldqXZ24ZR35pDbe/uIEaL7jBzpGKOv7vwx2MT43km+cOsLoc5WTZGbFsKjxOebV37dW6Cw0EHzdlSDR/uzGTLUUnmLvQs++l0Nxi+OHrm2lsNjw2ezR+eqjI60xLj6XFwBd7Sq0uxStpICimj4jnsdmjWZNXzp0v5VDb4Jmh8I/Pc1mbX87vZg0ntV+o1eWoHjAiMZzY8CDttdxDNBAUANeOS+Kx2aNZnVfmkbfiXJdfzuOf7eXqMYnMHqdXFXmr1sHuYvlyX6lH7826Kw0E9W+zxyXx5+tHs76gnG++sNFjzikcqajj3le/pn9UKA9eNQIRPVTkzbIzYjnZ0Mza/HKrS/E6Ggjqv1w9Jom/zMlk4/5j3P7CBqrq3Pvqo7rGZuYuzKGmvomnbh5L7yAd0d3bTUqLIiTQTw8b9QCHAkFE+orIUhHZZ//Z4WAxIvK8iJSIyHZH2lOuMSszkb/dOIavD5zg+mfWUVJZZ3VJHTLGcN9bW9l2sIK/3jCGYXHhVpekXKBXgB9TBkezbJf2WnY2R/cQHgA+M8YMBj6zv+7Ii8B0B9tSLjRzVAILbh9PYXkN1zy1hjw3HObisU/28OGWQ9x/6TCmZcRaXY5yoeyMWI5W1rP9oPePyeVKjgbCLOAl+/OXgKs6mskY8yVwzMG2lItNHRLNa3MnUtvQzOyn1vDVAfcZafKZFXk8uTyPGyekMG+q3gHN11w4NBqbwFLtpOZUjgZCrDHmMID9Z4zjJSl3MiopgrfvmUx4cAA3PLOONzYWWV0SC9fu5+GPdzNzVDwP6UlknxTVO4ixKZF6HsHJzhoIIrJMRLZ38JjVEwWJyFwRyRGRnNJS7XziDlL7hfLet89l/IBI7n97K796bzsNTa6/UYkxhie+yOVX7+8gOz2GP1+fqZ3PfFh2Riw7D1d67QCNVjhrIBhjso0xIzp4vA8cFZF4APvPEkcLMsY8a4zJMsZkRUdHO7o65SSRoYG89M0JzJ2SxsJ1hcx5di2F5a4bdbK5xfDQol089skerspM4KmbxxHorxfJ+bJsHezO6Rz9H/UBcJv9+W3A+w6uT7kxfz8bP78snX98Ywy5JdXMeHwlr2040ONXepw42cDtL2xgwaoCbp+cyp+vzyTAT8PA1w2MDmVAv1CW6mEjp3H0f9UjwDQR2QdMs79GRBJEZPGpmUTkVWAtMFREikXkDgfbVRaaOSqBT34whczkCB54Zxu3LNhAbklVj7S1Jq+MmX9fxfr8YzxyzUh+e6XeF1m1au21HMO6/HK37y/jKcSdr+PNysoyOTk5VpehTqOlxfDK+kL+9MkeTjY0c9vkVO65YCD9egc5vO7y6nr+vHQv/1x/gNSoEP4yJ5MxKXpPZPXf1ueXM+fZdTx501guGxlvdTluQUQ2GWOyurOsdutU3WazCbdOSuXykfH8cckenl9dwD/XF3LjhBRunZTKgG4MMFdeXc8r6w7w3Mp8TjY08a1zB3DfpUMJDvTrgS1Qnm5c/0giQgJYtvOoBoITaCAoh0X1DuLR2aOYOzWNp5bn8fLaQl5YvZ+s/pFMHxHH+YOjGRLb+7SXh1acbGRlbimf7jjKku1HaGhuYVpGLD+dPpRBMWEu3hrlSfz9bFw0NIbP95TQ1NyCv55bcogeMlJOd7Syjne/Psjbm4rZV9Law7l3kD9p0aHEhvciJNCP5hbDiZON7C+vofh462WDkSEBzByVwK2T+jM4VoNAdc7ibYf59j+/4vW5EzknLcrqciynh4yUW4kN78W8qQOZN3UgB0/Usjq3jJ2HKskrrabo2ElqG5uxiRAREkBmcgTfOCeF8al9GZsSqf0KVJdNGRJNoJ+NZbuOaiA4SANB9ajEiGCuz0q2ugzlxXoH+TNxYBRLdx7l55ela891B+gBN6WUx5uWHsP+8pPklbqus6Q30kBQSnm8i+29lpdpr2WHaCAopTxeQkQwwxPCdbA7B2kgKKW8wsXpsWw6cJyy6nqrS/FYGghKKa8wfXgcxsCnO3Qvobs0EJRSXiE9PozUqBA+3n7Y6lI8lgaCUsoriAjTR8SzJq+c4zUNVpfjkTQQlFJe47KRcTS3GL21ZjdpICilvMbIxD4kRgTz8TY9bNQdGghKKa8hIlw2Mo5VuWVU1Oo9ErpKA0Ep5VWmj4insdnw+W49bNRVGghKKa8yJjmCuPBeLN52xOpSPI4GglLKq9hswvQRcazYW0p1fZPV5XgUDQSllNe5bGQ8DU0tfLG7xOpSPIoGglLK64zrH0m/3kHaSa2LNBCUUl7HzyZMHxHLF7tLqW1otrocj6GBoJTySpeNiKe2sZnP9bBRp2kgKKW80jlpUUSHBfHBloNWl+IxNBCUUl7JzybMHBXPF7tLtZNaJ2kgKKW81qzMRBqaW/hkh/ZJ6AwNBKWU1xqd1If+USF8uOWQ1aV4BA0EpZTXEhGuHJ3A6twySqrqrC7H7WkgKKW82pWjE2gxsHir9kk4G4cCQUT6ishSEdln/xnZwTzJIvKFiOwSkR0i8n1H2lRKqa4YHBtGenw47+tho7NydA/hAeAzY8xg4DP76/aagB8bY9KBicB3RCTDwXaVUqrTZmUm8PWBExwoP2l1KW7N0UCYBbxkf/4ScFX7GYwxh40xX9mfVwG7gEQH21VKqU67YnQCAB9u1b2EM3E0EGKNMYeh9YsfiDnTzCKSCowB1jvYrlJKdVpiRDDjUyN556tijDFWl+O2zhoIIrJMRLZ38JjVlYZEpDfwNvADY0zlGeabKyI5IpJTWlralSaUUuq0Zo9LIq+0hq+LTlhdits6ayAYY7KNMSM6eLwPHBWReAD7zw4HDRGRAFrD4J/GmHfO0t6zxpgsY0xWdHR017dIKaU6cPmoBIID/HhrU7HVpbgtRw8ZfQDcZn9+G/B++xlERIAFwC5jzJ8dbE8ppbqld5A/M0bG8eGWQ9Q16gioHXE0EB4BponIPmCa/TUikiAii+3znAvcAlwkIpvtj8scbFcppbps9rgkquqadCiL0/B3ZGFjTDlwcQfvHwIusz9fBYgj7SillDNMHBBFUmQwb+YUMytTL3ZsT3sqK6V8hs0mXDs2idV5ZRw8UWt1OW5HA0Ep5VNmj0vCGHhHTy7/Dw0EpZRPSe4bwuSBUbyeU0Rzi/ZJaEsDQSnlc246pz/Fx2v5cq/2dWpLA0Ep5XMuGR5LdFgQC9cVWl2KW9FAUEr5nAA/GzeOT+aLPSUUHdMB707RQFBK+aQbJqQgwKsbDlhditvQQFBK+aSEiGCy02N5fWMR9U3acxk0EJRSPuzmif0pr2lgyXbtuQwaCEopH3beoH6kRYcyf2WBDouNBoJSyofZbMKd56Wx7WAF6/KPWV2O5TQQlFI+7ZqxiUSFBvLcynyrS7GcBoJSyqf1CvDj1kmpfL67hH1Hq6wux1IaCEopn3fLpP4E+dt8fi9BA0Ep5fP6hgZyXVYS7319iEM+PAqqBoJSSgF3TxlIizE8uTzX6lIso4GglFK0joJ6/fhkXt9Y5LP3StBAUEopu+9cOAiAJ77wzb0EDQSllLJLjAhmzvhk3swp8slB7zQQlFKqje9cOAibCH/6dI8l7X+09RB/WLyLukbXj6+kgaCUUm3E9wlm7pQ03t98iK8OHHdp23WNzTy8eDdr8soI9HP917MGglJKtTNv6kBiwoL43Yc7XTrG0Ytr9nPwRC0/n5GOzSYua/cUDQSllGonNMifn1w6lM1FJ3j364MuafNYTQNPfJHLRcNimDyon0vabE8DQSmlOjB7bBKZyRE8tGgX5dX1Pd7eHxbvorahmZ/NGNbjbZ2OBoJSSnXAZhMevXYUVXWN/O6jnT3a1tq8ct7aVMzcKWkMjg3r0bbORANBKaVOY2hcGN+5cBDvbz7Esp1He6SNusZmfvHeNlL6hvC9iwb3SBudpYGglFJn8O0LBpEeH859b23hcIXzezD/ftEu8ktr+MPVIwkO9HP6+rtCA0Eppc4g0N/GP74xhoamFr73r69pam5x2rqX7jzKwnWF3HX+AM4bbM2J5LYcCgQR6SsiS0Vkn/1nZAfz9BKRDSKyRUR2iMj/OdKmUkq52sDo3vzhmpHkFB7n1x/scMqlqLklVfzojc2MSAznvkutO5HclqN7CA8AnxljBgOf2V+3Vw9cZIwZDWQC00VkooPtKqWUS83KTOSeCwbyr/UHeHJ5nkPrKq+u51sv5hDk78fTN48j0N89Dtb4O7j8LOAC+/OXgOXAT9vOYFqjtNr+MsD+0LtZK6U8zn2XDOXwiVoe+2QPNhHuuWBgl9dRVl3PzfPXc7SyjtfmTiQpMqQHKu0eRwMh1hhzGMAYc1hEYjqaSUT8gE3AIOAJY8z6061QROYCcwFSUlIcLE8ppZzHZhMeu240zQYeXbKbsup6HpgxjIBODjORV1rNXS/ncOhELc/fPp4xKf9zlN1SZw0EEVkGxHUw6RedbcQY0wxkikgE8K6IjDDGbD/NvM8CzwJkZWXpnoRSyq0E+Nn465xMokIDWbCqgC1FJ3j4mpFn7D/Q0mJ466tiHvxwJwH+Nl7+1jlMGNDXhVV3zlkDwRiTfbppInJUROLtewfxQMlZ1nVCRJYD04EOA0Eppdydn0347ZXDGZMSwS/f2870x1dy9ZhEbhifzOjkiH/vMVScbOSLPSU8v7qArcUVjE+N5K83jCExItjiLeiYo4eMPgBuAx6x/3y//QwiEg002sMgGMgGHnWwXaWUstyszETOHxzN48v28uamYt7aVEyQv43osCAam1soqarHGBjQL5THZo/i2rFJlgxa11niyOVTIhIFvAGkAAeA64wxx0QkAZhvjLlMREbResLZj9armt4wxvyuM+vPysoyOTk53a5PKaVcpbq+ieV7SthSdIKy6gb8bUJy3xAmD4xibEqky4JARDYZY7K6tawrh3btKg0EpZTqGkcCwT0uflVKKWU5DQSllFKABoJSSik7DQSllFKABoJSSik7DQSllFKABoJSSik7DQSllFKAm3dME5FSoLCbi/cDypxYjjvwtm3ytu0B3SZP4W3b1HZ7+htjoruzErcOBEeISE53e+u5K2/bJm/bHtBt8hTetk3O2h49ZKSUUgrQQFBKKWXnzYHwrNUF9ABv2yZv2x7QbfIU3rZNTtkerz2HoJRSqmu8eQ9BKaVUF2ggKKWUAjwwEERkuojsEZFcEXmgg+kiIn+zT98qImM7u6xVHNym/SKyTUQ2i4jb3E2oE9s0TETWiki9iPykK8tawcHt8dTP6Cb7v7etIrJGREZ3dlmrOLhNnvo5zbJvz2YRyRGR8zq77P8wxnjMg9bbcOYBaUAgsAXIaDfPZcDHgAATgfWdXdbTtsk+bT/Qz+rt6MY2xQDjgd8DP+nKsp60PR7+GU0GIu3PZ3jJ/6UOt8nDP6fe/Od88Chgd3c/J0/bQ5gA5Bpj8o0xDcBrwKx288wCXjat1gERIhLfyWWt4Mg2uauzbpMxpsQYsxFo7OqyFnBke9xVZ7ZpjTHmuP3lOiCps8taxJFtcled2aZqY08AIBQwnV22PU8LhESgqM3rYvt7nZmnM8tawZFtgtYP/1MR2SQic3usyq5x5Hftjp+TozV5w2d0B617qd1Z1lUc2Sbw4M9JRK4Wkd3AIuBbXVm2LX+HSnU96eC99tfNnm6ezixrBUe2CeBcY8whEYkBlorIbmPMl06tsOsc+V274+fkaE0e/RmJyIW0fnmeOjbtjp8ROLZN4MGfkzHmXeBdEZkCPAhkd3bZtjxtD6EYSG7zOgk41Ml5OrOsFRzZJowxp36WAO/SuptoNUd+1+74OTlUkyd/RiIyCpgPzDLGlHdlWQs4sk0e/TmdYg+wgSLSr6vLnlqBxzxo3aPJBwbwn5Mkw9vNczn/fQJ2Q2eX9cBtCgXC2jxfA0z3hG1qM+9v+e+Tym73OTm4PR77GQEpQC4wubu/Dw/aJk/+nAbxn5PKY4GD9u+KLn9Olm5sN39BlwF7aT17/gv7e/OAefbnAjxhn74NyDrTsu7w6O420Xr1wBb7Y4eHbVMcrX/BVAIn7M/D3fVz6u72ePhnNB84Dmy2P3LOtKw7PLq7TR7+Of3UXvNmYC1wXnc/Jx26QimlFOB55xCUUkr1EA0EpZRSgAaCUkopOw0EpZRSgAaCUkopOw0EpZRSgAaCUkopu/8PiuaIriIwbOYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x_norm.detach().numpy(), y_norm.detach().numpy())\n",
    "plt.title('Normalized function')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4f44199",
   "metadata": {},
   "source": [
    "# 2. Build your model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3eedf40a",
   "metadata": {},
   "source": [
    "- In PyTorch [`Sequential`](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html#torch.nn.Sequential) stands for *sequential container*, where modules can be added sequentially and are connected in a cascading way. The output for each module is forwarded sequentially to the next.\n",
    "- Now we will build a simple model with one hidden layer with `Sequential`\n",
    "- Remember that every layer in a neural network is followed by an **activation layer** that performs some additional operations on the neurons.\n",
    "\n",
    "## 2.1 Activation functions\n",
    "\n",
    "## 2.2. Model architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 591,
   "id": "56a81cf8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequential(\n",
      "  (0): Linear(in_features=1, out_features=5, bias=True)\n",
      "  (1): Tanh()\n",
      "  (2): Linear(in_features=5, out_features=1, bias=True)\n",
      "  (3): Tanh()\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "n_hidden_01 = 5\n",
    "\n",
    "model0 = nn.Sequential(nn.Linear(n_input, n_hidden_01),\n",
    "                      nn.Tanh(),\n",
    "                      nn.Linear(n_hidden_01, n_out),\n",
    "                      nn.Tanh()\n",
    "                      )\n",
    "print(model0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 592,
   "id": "ffe7db45",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequential(\n",
      "  (0): Linear(in_features=1, out_features=10, bias=True)\n",
      "  (1): Tanh()\n",
      "  (2): Linear(in_features=10, out_features=1, bias=True)\n",
      "  (3): Tanh()\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "n_hidden_11 = 10\n",
    "\n",
    "model1 = nn.Sequential(nn.Linear(n_input, n_hidden_11),\n",
    "                      nn.Tanh(),\n",
    "                      nn.Linear(n_hidden_11, n_out),\n",
    "                      nn.Tanh()\n",
    "                      )\n",
    "print(model1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 593,
   "id": "8f7cb08d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequential(\n",
      "  (0): Linear(in_features=1, out_features=5, bias=True)\n",
      "  (1): Tanh()\n",
      "  (2): Linear(in_features=5, out_features=5, bias=True)\n",
      "  (3): Tanh()\n",
      "  (4): Linear(in_features=5, out_features=1, bias=True)\n",
      "  (5): Tanh()\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "n_hidden_21 = 5\n",
    "n_hidden_22 = 5\n",
    "model2 = nn.Sequential(nn.Linear(n_input, n_hidden_21),\n",
    "                      nn.Tanh(),\n",
    "                      nn.Linear(n_hidden_21, n_hidden_22),\n",
    "                      nn.Tanh(),\n",
    "                      nn.Linear(n_hidden_22, n_out),\n",
    "                      nn.Tanh()\n",
    "                      )\n",
    "print(model2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd82d8db",
   "metadata": {},
   "source": [
    "<span style='font-family:\"Helvetica\"; background:yellow; font-size:1.4em' >\n",
    " How much do you think each hyperparameter will affect the quality of the model? \n",
    "</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c905a1b",
   "metadata": {},
   "source": [
    "You can uncomment and execute the next line to explore the methods of the `model` object you created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 594,
   "id": "2d4584d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dir(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6d3419d",
   "metadata": {},
   "source": [
    "## 2.1 - Understanding the PyTorch model\n",
    "Try the `parameters` method (needs to be instantiated)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 595,
   "id": "a0d0892c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<generator object Module.parameters at 0x16923d350>"
      ]
     },
     "execution_count": 595,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model0.parameters()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c834e0b8",
   "metadata": {},
   "source": [
    "The `parameters` method gives back a *generator*, which means it needs to be iterated over to give back an output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 596,
   "id": "08caf645",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([5, 1])\n",
      "torch.Size([5])\n",
      "torch.Size([1, 5])\n",
      "torch.Size([1])\n"
     ]
    }
   ],
   "source": [
    "for element in model0.parameters():\n",
    "    print(element.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e44a30e9",
   "metadata": {},
   "source": [
    "<span style='font-family:\"Helvetica\"; background:yellow; font-size:1.4em' >\n",
    "     Without taking into account any bias unit: can you identify the elements of the model by their dimensions?\n",
    "</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a750495a",
   "metadata": {},
   "source": [
    "- The first element corresponds to the weight matrix $\\theta^0$ from layer 0 to layer 1, of dimensions $u^{j+1} \\times u^j = u^2 \\times u^1$ (so, without bias)\n",
    "- The second element corresponds to the values of the activation units in layer 1\n",
    "- The third element corresponds to the weight matrix $\\theta^1$ from layer 1 to layer 2, of dimensions $u^{j+1} \\times u^j = u^3 \\times u^3 $ (without bias)\n",
    "- The fourth element is the output of the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35a13a21",
   "metadata": {},
   "source": [
    "Let's have a look at what the contents of those tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 597,
   "id": "383b806d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameter containing:\n",
      "tensor([[-0.0838],\n",
      "        [-0.0343],\n",
      "        [-0.3750],\n",
      "        [ 0.2300],\n",
      "        [-0.5721]], requires_grad=True)\n",
      "Parameter containing:\n",
      "tensor([-0.1763,  0.3876,  0.9386,  0.2356, -0.3393], requires_grad=True)\n",
      "Parameter containing:\n",
      "tensor([[ 0.0429, -0.0501,  0.1825,  0.0512,  0.1752]], requires_grad=True)\n",
      "Parameter containing:\n",
      "tensor([0.4337], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "for element in model0.parameters():\n",
    "    print(element)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6da98e7",
   "metadata": {},
   "source": [
    "<span style='font-family:\"Helvetica\"; background:yellow; font-size:1.4em' >\n",
    "     What are these values?\n",
    "</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f11f48b6",
   "metadata": {},
   "source": [
    "# 3 - Define the loss function\n",
    "- Reminder: the **loss function** measures how distant the predictions made by the model are from the actual values\n",
    "- `torch.nn` provides many different types of [loss functions](https://pytorch.org/docs/stable/nn.html#loss-functions). One of the most popular ones in the [Mean Squared Error (MSE)](https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html#torch.nn.MSELoss) since it can be applied to a wide variety of cases.\n",
    "- In general cost functions are chosen depending on desirable properties, such as convexity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 598,
   "id": "2ebd648b",
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_function = nn.MSELoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b443be59",
   "metadata": {},
   "source": [
    "# 4 - Define the optimizer\n",
    "[`torch.optim`](https://pytorch.org/docs/stable/optim.html) provides implementations of various optimization algorithms. The optimizer object will hold the current state and will update the parameters of the model based on computer gradients. It takes as an input an iterable containing the model parameters, that we explored before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 599,
   "id": "9d116553",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 200 # how many points to pass to the model at a time\n",
    "learning_rate = 0.015"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 600,
   "id": "8921c861",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer0 = torch.optim.Adam(model0.parameters(), lr=learning_rate)\n",
    "optimizer1 = torch.optim.Adam(model1.parameters(), lr=learning_rate)\n",
    "optimizer2 = torch.optim.Adam(model2.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 601,
   "id": "b89395a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# optimizer0 = torch.optim.SGD(model0.parameters(), lr=learning_rate)\n",
    "# optimizer1 = torch.optim.SGD(model1.parameters(), lr=learning_rate)\n",
    "# optimizer2 = torch.optim.SGD(model2.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fbca570",
   "metadata": {},
   "source": [
    "# 5 - Train the model on a loop\n",
    "The model learns iteratively in a loop of a given number of epochs. Each loop consists of:\n",
    "- A **forward propagation**: compute $y$ given the input $x$ and current weights and calculate the loss\n",
    "- A **backward propagation**: compute the gradient of the loss function (error of the loss at each unit)\n",
    "- Gradient descent: update model weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 602,
   "id": "94274de5",
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = 1000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 603,
   "id": "77062514",
   "metadata": {},
   "outputs": [],
   "source": [
    "losses0 = []\n",
    "for epoch in range(epochs):\n",
    "    pred_y0 = model0(x_norm)\n",
    "    optimizer0.zero_grad()\n",
    "    loss0 = loss_function(pred_y0, y_norm)\n",
    "    losses0.append(loss0.item())\n",
    "    loss0.backward()\n",
    "    optimizer0.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 604,
   "id": "628a5f23",
   "metadata": {},
   "outputs": [],
   "source": [
    "losses1 = []\n",
    "for epoch in range(epochs):\n",
    "    pred_y1 = model1(x_norm)\n",
    "    optimizer1.zero_grad()\n",
    "    loss1 = loss_function(pred_y1, y_norm)\n",
    "    losses1.append(loss1.item())\n",
    "    loss1.backward()\n",
    "    optimizer1.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 605,
   "id": "f360c837",
   "metadata": {},
   "outputs": [],
   "source": [
    "losses2 = []\n",
    "for epoch in range(epochs):\n",
    "    pred_y2 = model2(x_norm)\n",
    "    optimizer2.zero_grad()\n",
    "    loss2 = loss_function(pred_y2, y_norm)\n",
    "    losses2.append(loss2.item())\n",
    "    loss2.backward()\n",
    "    optimizer2.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 606,
   "id": "bd335e65",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAqM0lEQVR4nO3de5hcVZnv8e9b1dXppLtzTyBJJySBQAhDgEyMMKCAKEMQDR6dGaIDijoBHAc5HkYz6nAbPeo5joPMIJjDMIIi6JGLcQiKgyB3TaIQSUjIhUA693SS7iR9rap3/ti7undVqrurL5VKun+f56mnaq+91q61dqDfWnutvba5OyIiIrlipa6AiIgcnRQgREQkLwUIERHJSwFCRETyUoAQEZG8FCBERCQvBQgZcMzsXWa2rtT1EDnWKUBIvzKzzWb23lLWwd2fc/dTSlmHDDO7wMxq+3iMi8xsrZk1mtnTZnZCF3lHm9mjZnbIzN4ys49G9pWb2U/DfyM3swtyyt5iZm1mdjDymh7ZPzX8/sawPu/NKf/R8DsPmdljZjY6sm+Imd1rZg1mtsPMPt+XcyJHhgKEHHPMLF7qOgBYoKj/D5nZWOAR4B+B0cAK4MddFLkTaAWOAz4G3GVmp0X2Pw/8NbCjk/I/dveqyGtTZN+DwB+AMcCXgZ+a2biwnqcB3wOuDL+7EfhupOwtwAzgBOBC4AtmdknXrZeSc3e99Oq3F7AZeG+e9BiwGNgI1AE/AUZH9v9/gj9a9cCzwGmRfd8H7gKWAYeA94bfcyOwKizzY6AizH8BUJtTp7x5w/1fALYD24BPAw6c1En7ngG+BrwANAEnAVcDrwMHgE3ANWHeyjBPGjgYviZ2dy5yvm8R8GJkO3PMmXnyVhIEh5MjaT8AvpEnby1wQU7aLcAPO6nHyUALUB1Jew64Nvz8v4EfRfadGNalOtzeClwc2f9PwEOl/u9Vr65f6kHIkXI9cDlwPsEfyX0Ev3YzniD4hTke+D3wQE75jxL8Ya4m+BUM8JfAJcA0YDbwiS6+P2/e8Ffs5wmCzklh/bpzJcEf7mrgLWAXcBkwnCBY/IuZzXH3Q8B8YJt3/CLfVsC5iDoNeDWzER5zY5ie62Qg5e5vRNJe7SRvZz5gZnvNbLWZXZdTj03ufqCTY+fWcyNhsDKzUQTtfLWTsnKUUoCQI+Ua4MvuXuvuLQS/Vj9iZmUA7n6vux+I7DvDzEZEyv/M3V9w97S7N4dpd7j7NnffC/wcOLOL7+8s718C/+Huq929Ebi1gLZ8P8yfdPc2d3/c3Td64DfAk8C7ensuclQR9Hqi6gmCU1/y5vMT4FRgHPA3wE1mtrDAY3e1vyqy3Zt6SYkoQMiRcgLwqJntN7P9BJdkUsBxZhY3s2+Y2UYzayC4JAQwNlJ+S55jRq+jN9LxhyifzvJOzDl2vu/JlZXHzOab2cvhL+/9wKVk1z1Xp+ciT96DBD2TqOEEl7P6kvcw7r4mDKIpd38R+A7wkQKP3dX+g5HtHtdLSkcBQo6ULcB8dx8ZeVW4+1aCy0cLCC7zjACmhmUsUr5Yyw5vB2oi25MLKNNeFzMbAjwMfAs4zt1HEoyVWG7eiK7ORa7VwBmR76skuL6/Ok/eN4AyM5sRSTujk7yFcDrasRqYbmbRX/3RY+fWczowBHjD3fcRnOczOikrRykFCCmGhJlVRF5lwN3A1zJTNM1snJktCPNXEwyA1gHDCAY8j5SfAFeb2almNgy4qYflywn+EO4GkmY2H7g4sn8nMCbncllX5yLXo8CfmNmHzawirN8qd1+bmzEcn3gEuM3MKs3sXILA+4NMnnC6aUWm7uG/j4X7FpjZqHB21jyCsZKfhcd+A3gFuDks8yGCsZyHw2M9QDB+8a4wiN0GPBIZs7gf+Ep4/JkEl7C+30mb5SihACHFsIxgpk3mdQvB5YqlwJNmdgB4GXhnmP9+gsHercCacN8R4e5PAHcATwMbgJfCXS0Flj9A8If0JwSDzR8laGdm/1qC6aGbwktKE+n6XOQefzfwYYIB+n1hvisy+83sS2b2RKTIZ4ChBAPnDwLXuXv0l/o6gn+TScAvw8+Z+yquCM/BAYJ/k2+6+32RslcAc8N6fAP4SFg/wu+4liBQ7CII+p+JlL2ZYHD9LeA3wP9191/ka7McPcxdDwwSyTCzU4HXgCHunix1fURKST0IGfTM7EPhXcajgG8CP1dwEFGAEIFg2ulugksgKeC6rrOLDA66xCQiInmpByEiInnlu3PzmDV27FifOnVqqashInLMWLly5R53H5dvX1EDRLjOzXeAOHCPu38jZ//HgC+GmwcJpuS9Gu7bTDDdLgUk3X1ud983depUVqxY0X8NEBEZ4Mzsrc72FS1AhEsy3wm8j2DlyOVmttTd10SyvQmc7+77whuMlpA9H/xCd99TrDqKiEjnijkGMQ/Y4O6b3L0VeIjgrs527v5ieBs+BDcL1SAiIkeFYgaISWQvalYbpnXmUwRLPmc4wZ2mK81sUWeFzGyRma0wsxW7d+/uU4VFRKRDMccgLE9a3jm1ZnYhQYA4L5J8rrtvM7PxwK/MbK27P3vYAd2XEFyaYu7cuZqzKzLItLW1UVtbS3Nzc/eZB7GKigpqampIJBIFlylmgKgle2XMGoIndmUxs9nAPQSrW9Zl0sMHq+Duu8zsUYJLVocFCBEZ3Gpra6murmbq1KmE6w5KDnenrq6O2tpapk2bVnC5Yl5iWg7MMLNpZlZOsNDX0mgGM5tCsPrkldGnYIUrUVZnPhOsjvlaEesqIseo5uZmxowZo+DQBTNjzJgxPe5lFa0H4e5JM/sswYqRceBed19tZteG++8mWLp4DPDd8B83M531OIIHqmTq+COt/CginVFw6F5vzlFR74Nw92UESz9H0+6OfP40wUPic8ttIvvhIkV1x1PrOWPySM4/Oe+9IiIig5KW2gDuemYjz6/XDCgR6R0z48orr2zfTiaTjBs3jssuu6xHx5k6dSp79nR961dneVauXMnpp5/OSSedxPXXX09/rLOnAAHEY0YqXepaiMixqrKyktdee42mpiYAfvWrXzFpUlez+vvfddddx5IlS1i/fj3r16/nF7/o+1V5BQggZpDWqrYi0gfz58/n8ccfB+DBBx9k4cKF7fv27t3L5ZdfzuzZszn77LNZtWoVAHV1dVx88cWcddZZXHPNNVm/+n/4wx8yb948zjzzTK655hpSqVSn3719+3YaGho455xzMDOuuuoqHnvssT63aUAt1tdbsZgpQIgMALf+fDVrtjX06zFnTRzOzR84rdt8V1xxBbfddhuXXXYZq1at4pOf/CTPPfccADfffDNnnXUWjz32GL/+9a+56qqreOWVV7j11ls577zzuOmmm3j88cdZsmQJAK+//jo//vGPeeGFF0gkEnzmM5/hgQce4Kqrrsr73Vu3bqWmpmMhipqaGrZu3drntitAAHEzUmkFCBHpvdmzZ7N582YefPBBLr300qx9zz//PA8//DAA73nPe6irq6O+vp5nn32WRx55BID3v//9jBo1CoCnnnqKlStX8o53vAOApqYmxo8f3+l35xtv6I+ZXQoQqAchMlAU8ku/mD74wQ9y44038swzz1BX137fb5d/wPP9IXd3Pv7xj/P1r3+9oO+tqamhtra2fbu2tpaJEyf2tPqH0RgE6kGISP/45Cc/yU033cTpp5+elf7ud7+bBx54AIBnnnmGsWPHMnz48Kz0J554gn37grVLL7roIn7605+ya9cuIBjDeOutTlflZsKECVRXV/Pyyy/j7tx///0sWLCg0/yFUg+CYBaT4oOI9FVNTQ2f+9znDku/5ZZbuPrqq5k9ezbDhg3jvvvuA4KxiYULFzJnzhzOP/98pkyZAsCsWbP46le/ysUXX0w6nSaRSHDnnXdywgkndPrdd911F5/4xCdoampi/vz5zJ8/v8/tGVDPpJ47d6735oFB533z18ybOppv/9WZ/V8pESmq119/nVNPPbXU1Tgm5DtXZrayswey6RIT4X0QAyhQioj0BwUINAYhIpKPAgSaxSQiko8CBOpBiIjkowABmKFZTCIiORQgCKe5KkKIiGRRgECzmESkb46G5b6//OUvM3nyZKqqqnr0nV1RgABiGoMQkT44Gpb7/sAHPsDvfve7fj2mAgSZO6kVIESk90q53DfA2WefzYQJE/q1TVpqg2AWU1oPDBI59j2xGHb8sX+PefzpMP8b3WYr5XLfxaIAQTCLSWMQItIXpVzuu1gUIAguMbUm1YUQOeYV8Eu/mEq13HexaAwCzWISkf5RquW+i0UBgmAWk+6DEJG+6mq57xUrVjB79mwWL16ctdz3s88+y5w5c3jyySfzLvc9e/Zs3ve+97F9+/Yuv/sLX/gCNTU1NDY2UlNTwy233NLn9mi5b+CT31/OrgPN/OffvasItRKRYtJy34XTct+9EDM0i0lEJIcCBOElpgHUkxIR6Q8KEISD1BqDEBHJogBB8DwIzWISEcmmAEHmTmoFCBGRKAUIMmsxlboWIiJHFwUIwqU2FCFEpJdKvdx3Y2Mj73//+5k5cyannXYaixcv7tH3dqaoAcLMLjGzdWa2wcwOq7GZfczMVoWvF83sjELL9qe4ZjGJSB8cDct933jjjaxdu5Y//OEPvPDCCzzxxBN9PmbRAoSZxYE7gfnALGChmc3KyfYmcL67zwb+CVjSg7L9RrOYRKSvSrnc97Bhw7jwwgsBKC8vZ86cOdTW1va5TcVcrG8esMHdNwGY2UPAAmBNJoO7vxjJ/zJQU2jZ/hTT8yBEBoRv/u6brN27tl+POXP0TL4474vd5jtalvvev38/P//5z/Mu+dFTxQwQk4Atke1a4J1d5P8UkOkT9bRsnwSXmIp1dBEZDI6G5b6TySQLFy7k+uuvZ/r06X1uUzEDxOFr2ELeP8NmdiFBgDivF2UXAYuA9oWueiqmQWqRAaGQX/rFVOrlvhctWsSMGTO44YYbelbxThRzkLoWmBzZrgG25WYys9nAPcACd6/rSVkAd1/i7nPdfe64ceN6VdFYTPdBiEjflXK576985SvU19dz++2391t7ihkglgMzzGyamZUDVwBLoxnMbArwCHClu7/Rk7L9KW66k1pE+q5Uy33X1tbyta99jTVr1jBnzhzOPPNM7rnnnj63p6jLfZvZpcDtQBy4192/ZmbXArj73WZ2D/BhIBMak5llZ/OV7e77ervc99eXvc73X9zMuq/O73FZESktLfdduJ4u913UR466+zJgWU7a3ZHPnwY+XWjZYtEsJhGRw+lOasLnQSg+iIhkUYAgHINQhBA5Zg2kJ2MWS2/OkQIEwSUmQDOZRI5BFRUV1NXVKUh0wd2pq6ujoqKiR+WKOgZxrIiH85BT7sTy3oIhIkermpoaamtr2b17d6mrclSrqKigpqam+4wRChB09CBSaScRL3FlRKRHEokE06ZNK3U1BiRdYiJYrA9APVQRkQ4KEASzmADdLCciEqEAAcSs4xKTiIgEFCDouMSkWUwiIh0UIOgIELrEJCLSQQGCjktMWm5DRKSDAgSRAJEucUVERI4iChBAPDwLusQkItJBAYJoD0IBQkQkQwGCyCC1AoSISDsFCDSLSUQkHwUIOi4xaTVIEZEOChBE76QucUVERI4iChBEZjFpDEJEpJ0CBLpRTkQkHwUINItJRCQfBQgijxxVD0JEpJ0CBLrEJCKSjwIEkWdSaxaTiEg7BQggpllMIiKHUYCgowehS0wiIh0UINAsJhGRfBQgAFMPQkTkMAoQRJ5JrQAhItJOAQLNYhIRyUcBAs1iEhHJRwECXWISEcmnqAHCzC4xs3VmtsHMFufZP9PMXjKzFjO7MWffZjP7o5m9YmYrillPTXMVETlcWbEObGZx4E7gfUAtsNzMlrr7mki2vcD1wOWdHOZCd99TrDpmmGmaq4hIrmL2IOYBG9x9k7u3Ag8BC6IZ3H2Xuy8H2opYj27pEpOIyOGKGSAmAVsi27VhWqEceNLMVprZos4ymdkiM1thZit2797dq4pqFpOIyOGKGSAsT1pPfqKf6+5zgPnA35rZu/Nlcvcl7j7X3eeOGzeuN/Vsn8WU1iUmEZF2xQwQtcDkyHYNsK3Qwu6+LXzfBTxKcMmqKHSJSUTkcMUMEMuBGWY2zczKgSuApYUUNLNKM6vOfAYuBl4rVkUzz4NIKUCIiLQr2iwmd0+a2WeBXwJx4F53X21m14b77zaz44EVwHAgbWY3ALOAscCj4eyiMuBH7v6LYtW1/YFBusQkItKuaAECwN2XActy0u6OfN5BcOkpVwNwRjHrFqXVXEVEDqc7qYnMYlJ8EBFppwCBZjGJiOSjAIFmMYmI5KMAgWYxiYjkowCBZjGJiOSjAEF0FlOJKyIichRRgADC+KBLTCIiEQoQBMt9xwxcAUJEpJ0CRChmphvlREQiFCBCsZjpEpOISERBASJcPC8Wfj7ZzD5oZoniVu3IiptpFpOISEShPYhngQozmwQ8BVwNfL9YlSqFeMw0i0lEJKLQAGHu3gj8D+Bf3f1DBKuuDhgx053UIiJRBQcIMzsH+BjweJhW1JVgj7RYzBQgREQiCg0QNwD/ADwaPtNhOvB00WpVAnHNYhIRyVJQL8DdfwP8BiAcrN7j7tcXs2JHmnoQIiLZCp3F9CMzGx4+/nMNsM7M/r64VTuy1IMQEclW6CWmWe7eAFxO8IS4KcCVxapUKWgWk4hItkIDRCK87+Fy4Gfu3gYMqJ/bsZiW2hARiSo0QHwP2AxUAs+a2QkEz40eMGKmO6lFRKIKHaS+A7gjkvSWmV1YnCqVhsYgRESyFTpIPcLMvm1mK8LXPxP0JgYMzWISEclW6CWme4EDwF+GrwbgP4pVqVJQD0JEJFuhd0Of6O4fjmzfamavFKE+JRP0IEpdCxGRo0ehPYgmMzsvs2Fm5wJNxalSacRMz6QWEYkqtAdxLXC/mY0It/cBHy9OlUojrudBiIhkKXQW06vAGWY2PNxuMLMbgFVFrNsRpSfKiYhk69ET5dy9IbyjGuDzRahPycQ1i0lEJEtfHjlq/VaLo4BmMYmIZOtLgBhQf01jMTSLSUQkossxCDM7QP5AYMDQotSoROIxo61Nq/WJiGR0GSDcvfpIVaTU4rEYqXSq1NUQETlq9OUSU7fM7BIzW2dmG8xscZ79M83sJTNrMbMbe1K2v5XFNAYhIhJVtABhZnHgTmA+MAtYaGazcrLtBa4HvtWLsv0qHjOSChAiIu2K2YOYB2xw903u3go8BCyIZnD3Xe6+HGjradn+FvQgNAYhIpJRzAAxCdgS2a4N0/q1rJktyqwyu3v37l5VFNSDEBHJVcwAke8+iUL/Ahdc1t2XuPtcd587bty4giuXS2MQIiLZihkgaoHJke0aYNsRKNsr8ViMZEoBQkQko5gBYjkww8ymmVk5cAWw9AiU7RX1IEREshW6mmuPuXvSzD4L/BKIA/e6+2ozuzbcf7eZHQ+sAIYD6XABwFnhYoCHlS1WXQHicY1BiIhEFS1AALj7MmBZTtrdkc87CC4fFVS2mDSLSUQkW1FvlDuWaBaTiEg2BYiQxiBERLIpQITisZh6ECIiEQoQIfUgRESyKUCE4mGAcD1VTkQEUIBoVxYLbt5WL0JEJKAAEYrHgwChcQgRkYACREg9CBGRbAoQoXgsOBXqQYiIBBQgQupBiIhkU4AIxWOZMQgttyEiAgoQ7dSDEBHJpgARau9B6JkQIiKAAkS7srh6ECIiUQoQoZjpPggRkSgFiFBZOM1VPQgRkYACREizmEREsilAhDSLSUQkmwJEKK5BahGRLAoQIfUgRESyKUCEOsYgFCBEREABop1mMYmIZFOACKkHISKSTQEi1DEGoWmuIiKgANFOazGJiGRTgAhpLSYRkWwKEKEyjUGIiGRRgAjFNYtJRCSLAkQo04NoTWmQWkQEFCDalZcFp6JNAUJEBFCAaJeIhwEiqQAhIgIKEO0yPQhdYhIRCRQ1QJjZJWa2zsw2mNniPPvNzO4I968yszmRfZvN7I9m9oqZrShmPQHKMz0I3QchIgJAWbEObGZx4E7gfUAtsNzMlrr7mki2+cCM8PVO4K7wPeNCd99TrDpGJcL7IFp0iUlEBChuD2IesMHdN7l7K/AQsCAnzwLgfg+8DIw0swlFrFOnzIzyeIxWBQgREaC4AWISsCWyXRumFZrHgSfNbKWZLersS8xskZmtMLMVu3fv7lOFE3HTLCYRkVAxA4TlScu9wN9VnnPdfQ7BZai/NbN35/sSd1/i7nPdfe64ceN6X1uCgWr1IEREAsUMELXA5Mh2DbCt0DzunnnfBTxKcMmqqBLxmHoQIiKhYgaI5cAMM5tmZuXAFcDSnDxLgavC2UxnA/Xuvt3MKs2sGsDMKoGLgdeKWFdAPQgRkaiizWJy96SZfRb4JRAH7nX31WZ2bbj/bmAZcCmwAWgErg6LHwc8amaZOv7I3X9RrLpmlJfFaFEPQkQEKGKAAHD3ZQRBIJp2d+SzA3+bp9wm4Ixi1i2f8nhMd1KLiIR0J3VEeVmM1lSagy1JLv3Oc/zot2+XukoiIiWjABGRGaRe/uZe1mxv4EuP/rHUVRIRKRkFiIjMjXJv7jnUnqZZTSIyWClARASXmJy6Qy3tabsOtHRRQkRk4FKAiEiEPYi9h1rb03bUN5WwRiIipaMAETGkLEZrMkXdwY4Asb2+uYQ1EhEpHQWIiGAtJqfuUCunTRwOwA4FCBEZpBQgIjJ3Uu891Mq0sZUMTcTVgxCRQauoN8odazLTXBsPJhlbNYQJIyrUgxCRQUsBIqK8LMah1iTNbWlGV5Zz/IgKtmuQWkQGKV1iiigvi9HcFtz3MKYqCBDqQYjIYKUeBEB9LTQ3MDTRcTrGVJYzccRQdh5oIZlKUxZXLBWRwUV/9QD+5TS46xyqhnQEiNGVQ5g8eiiptGugWkQGJfUggH8fUc3biQSzKiI9iKpykqlhAGzZ28jk0cNKVT0RkZIY9D0Id+f20aN4pLqKKXXPtqePqSxvDwpb9jWyfucBPnLXi3z7V2+UqqoiIkfUoO9BtKRaKHMnacY9m2/F+BZOGcMrElQNKSMeM97e28jSV7ex4q19rHhrH7MmDOeSPzm+1FUXESmqQd+DqCir4LHa7QD8bmgFf37C/2JB4kli+zZRFo8xY3wVP3jpLV7YUMcXLjmFUycM57afr6axNcnBliTPr9+jqbAiMiBZ8FC3gWHu3Lm+YsWKHpfzW0Zw69jRPFxd1Z42PpnkOEuQSg8l3VRGWbqad0yfyIE2ePnNA1QNidPS5iRTAMaoYUMYmojTljZwKIvFSZTFSVicWMyAGHGMmMUAI0YMMws+Zb0bZrHgnThgxM3AYu3bZvFw2zArw4hhFgPiYdkYWBwsfti+oFwZZnEsFqYTC45JHIvHgBixWAUeH4Jb0MkMnv4aMCwrLbMrmieaj07257JIhtyshx+787Ld5s/N2+13dX7snpbtZrNn56AP5/uw89Xdd/XhHOTm6Orfpsf16sH57va/me6+K5Kju/8eD69GH85Bt//NQHk8zuk1I3JrURAzW+nuc/PuU4AA9myAeBl//O2/cs/mF3m+4gBV6TQGVKfSHIjFqI/HcOh4dffXbgAYlk5TnU5TlXYqUzAyBePbjLFtMSa2xhnfUsEhr6LeK6mnknqvZD9V7PKRbPcxbPfR7GEEro6qSFGNrRrCiq+8t1dluwoQg34MAoCxJwFw+iX/zHcyae5BaG49BGVDoaUebzmEJZugrQm3OO4pUp6CdIq0p0h7GsdJp1O4p8M0D9LdSZMmnU6RJpOWJuVp0qTxdLo9PciXxkmH5YNjZ8qkPU0qncI9OJZ7ilQ6lbU/Hdaj43s6jpE5pruH3x/Us70+7rSl2ziYauJAqpkDqWYaUi1sTjfzYrqZZBgmh3oz81KtfKhxBx9q2E9Z26HDTq3HyvDqiaRHn0h6zMmkx5xEeszJpMacjFeOI3q6OzZyjpGTkPubJvcnTu6PnqxDd3Psbjazyve5Xl38Nuuunj35rq7akC9H98fuQb162I6+lD1S5+DwNnb+31v+Y3V+8J6WdQ/KJOLF+cGqANGZTA+hvDJ4HzoKGzqqY3f4Gmy/jVPpFDsbd/LantdYvmM5T739FL8p283Mqefyj+9YzOyqyXBgO9RvhYatWMNWbP8WYnXr4dUfQFtjx8GGT4KJZ2W/ho0uXeNEJIsuMUmfJNNJfrn5l9z++9vZ07iHG/70Bq6addVh11EBSKehYSvseQN2vQ7bX4Gtv4e9GzvyjD0FTvgzOOFcOOEcGFFzxNoiMhhpDEKKrqG1gZtfuJn/evu/+JvT/4a/O+vv8geJfJr2w/ZXYesKePvl4NXSEOwbOQVOOA+mnw/TL4BqTS8W6U8KEHJEpD3NbS/dxsPrH+YzZ36G6864rpcHSsHO1+CtF+GtF2Dz89C0L9g3fhZMvzAIFlPP7bgEKCK9okFqOSJiFuOmc24imU7y3Ve+SyKW4NOnf7oXB4rDhDOC19nXBZemdqyCTU/Dxqdh+T3w8p0QS8Dkd8KJFwRBY+JZQVkR6RfqQUi/S6VTfOn5L7HszWUsnLmQG+feSHm8vP++oK0J3n4pCBabngmCB0DFCJj27qB3Mf1CGD29+5svRAY5XWKSIy6ZTnL7ytu5b819HDfsOP7i5L/goikXceLIEwsfmyjUoT3w5m86Akb9liB95JTwctT5wThG9XH9+70iA4AChJTMy9tf5nuvfo8VO4N/l0lVkzh19KlMrp5MdXk1VeVVVCYqGVo2tMtXIpYoLLC4w95NsPHXQbB48zloqQ/2jT0Fpp4H094VBIyqcV0eSmQwUICQktvVuIun336a3+74LW/se4NtB7fRlm4ruHzc4lSUVWQFjeryakZXjGbUkFGMqghe44eNZ1LVJCZVTWLEkBGQSsKOV4OB7jefCy5NtR4MDjr2lHCsYzYcfzocP1v3YcigowAhR6XWVCsHWg/Q2NZIY7KRpmQTTckmmpPN7Z+bkk00p5ppbGvMSmtKNtHQ2sC+5n3sa97H/pb9h91lWp2oZvLwycwYOYMZo2Zw8qiTOXnEiYzZ9zZsfi6YTrvjj8G9GRkjJncEi+NPD4LHiMkay5ABSwFCBrxUOkV9az07D+1k68GtbD24lS0HtvB2w9us37+ePU172vOOHTqWU0afwsxRM5k5eiYzhx7HlIN1xHauhu2rgqBRtx48eD45FSOD6bUjamD4hOAO8OEToXpi8F41XrOn5JilACGDXl1THev3r+eNvW+wbt861u1dx8b9G0l6EoChZUM5edTJnDLqFCZWTWTCkNEc39LIcQ07GVG3kWF7NhBr2B4sI5JqzT64xYMb+IZPhOoJQcBIDA3W8EqEr7IKSAyDREWYHm6XVUT2D4V4eRiYPHh3B4vBkOEQG2wLu8iRULIAYWaXAN8B4sA97v6NnP0W7r8UaAQ+4e6/L6RsPgoQ0hOtqVY21W9i7d61rNu7jrV717J+/3rqM4PaETGLUZWoorq8mqGxcsqBIe6Up1Ik0m2UJ9soTzZT3tZMeVsTiVSK8nQbCXfK3Uk4JNyDbTq2y8O0hEOCju3y6P5MfmKUY8RjZe1Lt2MWvoJl5INLYdaxjzz7o+n58nb1TuatuzyFHI8C8nbzXf1+HOumfT05TjfnopDzlJWHnDSCHyexsuCG0blX0xsluVHOggcM3Am8D6gFlpvZUndfE8k2H5gRvt4J3AW8s8CyIn1SHi8PLjGNnpmVfqjtEDsO7WDHoR3sbNzJgdYDNLQ2cLD1IA2tDTQlm2hNtdKabqUt1cbBVEv755ZUC62pVtrSbcEr1UrSU/1abwPKiZGwGOXEKLMgcCQsRgIL9xnlGFVWxnArY4SVMZx4+DnOcOJUEmt/DcMYihFziOO4O4+n97Iz3ca5sWomkiABJICycKHjYKlRz3knT1pkX3Q7b57Ie27+rHcKyFOC43R7DrrJ01uV43sdILpSzDup5wEb3H0TgJk9BCwAon/kFwD3e9CNednMRprZBGBqAWVFiqIyUcmJI0/kxJEn9svx0p4Og0VbeyBpTbe2p2WCSSawRN+T6WT+feFxcstGj92aauWtMLg1tOyhOdXcq/p/J7Uja9sw4rF48G5xErFE1gQBC3/tBg+0ssPKdrUNdDGdOXiwVld5sx7q0+3DmnLKWudlu8qbr5aF1jH/93pW7g6ep16BkeUjuK/TGvVeMQPEJGBLZLuWoJfQXZ5JBZYFwMwWAYsApkyZ0rcaixRBzGIMiQ9hSHxISevRkmqhoaWB+pb6oEfUdpDGtkYOtR2iMdlIc7KZVPtzQlJMGzGN+pZ63j7wNuWxchLxBIlYgtZUK+lwAD+ZTpL0ZNYfLg97IIc/uyD3uQld/2LuLn93x+9J3uj+vtSzp3XM2p/zNT05VnV5dZd17K1iBoh8ITb3THeWp5CyQaL7EmAJBGMQPamgyGAyJD6EccPGMW6YbhCUwhQzQNQCkyPbNcC2AvOUF1BWRESKqJjz5pYDM8xsmpmVA1cAS3PyLAWussDZQL27by+wrIiIFFHRehDunjSzzwK/JJiqeq+7rzaza8P9dwPLCKa4biCY5np1V2WLVVcRETmcbpQTERnEuroPQrdmiohIXgoQIiKSlwKEiIjkpQAhIiJ5DahBajPbDbzVy+JjgT3d5hpY1ObBQW0e+PrS3hPcPe/dkwMqQPSFma3obCR/oFKbBwe1eeArVnt1iUlERPJSgBARkbwUIDosKXUFSkBtHhzU5oGvKO3VGISIiOSlHoSIiOSlACEiInkN+gBhZpeY2Toz22Bmi0tdn/5iZpPN7Gkze93MVpvZ58L00Wb2KzNbH76PipT5h/A8rDOzPy9d7fvGzOJm9gcz+89we0C3OXxU70/NbG34733OIGjz/wz/u37NzB40s4qB1mYzu9fMdpnZa5G0HrfRzP7UzP4Y7rvDunpeai53H7QvgqXENwLTCR5S9Cowq9T16qe2TQDmhJ+rgTeAWcD/ARaH6YuBb4afZ4XtHwJMC89LvNTt6GXbPw/8CPjPcHtAtxm4D/h0+LkcGDmQ20zwSOI3gaHh9k+ATwy0NgPvBuYAr0XSetxG4HfAOQRP6nwCmF9oHQZ7D2IesMHdN7l7K/AQsKDEdeoX7r7d3X8ffj4AvE7wP9YCaH+++X3A5eHnBcBD7t7i7m8SPKNj3hGtdD8wsxrg/cA9keQB22YzG07wh+TfAdy91d33M4DbHCoDhppZGTCM4ImTA6rN7v4ssDcnuUdtNLMJwHB3f8mDaHF/pEy3BnuAmARsiWzXhmkDiplNBc4Cfgsc58FT+wjfx4fZBsq5uB34ApCOpA3kNk8HdgP/EV5Wu8fMKhnAbXb3rcC3gLeB7QRPonySAdzmiJ62cVL4OTe9IIM9QOS7Fjeg5v2aWRXwMHCDuzd0lTVP2jF1LszsMmCXu68stEietGOqzQS/pOcAd7n7WcAhgksPnTnm2xxed19AcCllIlBpZn/dVZE8acdUmwvQWRv71PbBHiBqgcmR7RqCruqAYGYJguDwgLs/EibvDLudhO+7wvSBcC7OBT5oZpsJLhe+x8x+yMBucy1Q6+6/Dbd/ShAwBnKb3wu86e673b0NeAT4MwZ2mzN62sba8HNuekEGe4BYDswws2lmVg5cASwtcZ36RThT4d+B193925FdS4GPh58/Dvwskn6FmQ0xs2nADILBrWOGu/+Du9e4+1SCf8tfu/tfM7DbvAPYYmanhEkXAWsYwG0muLR0tpkNC/87v4hgjG0gtzmjR20ML0MdMLOzw3N1VaRM90o9Ul/qF3ApwQyfjcCXS12ffmzXeQRdyVXAK+HrUmAM8BSwPnwfHSnz5fA8rKMHMx2OxhdwAR2zmAZ0m4EzgRXhv/VjwKhB0OZbgbXAa8APCGbvDKg2Aw8SjLG0EfQEPtWbNgJzw/O0Efg3whU0CnlpqQ0REclrsF9iEhGRTihAiIhIXgoQIiKSlwKEiIjkpQAhIiJ5KUCI9ICZpczslcir31YANrOp0ZU7RUqtrNQVEDnGNLn7maWuhMiRoB6ESD8ws81m9k0z+134OilMP8HMnjKzVeH7lDD9ODN71MxeDV9/Fh4qbmb/L3zWwZNmNrRkjZJBTwFCpGeG5lxi+qvIvgZ3n0dwt+rtYdq/Afe7+2zgAeCOMP0O4DfufgbB2kmrw/QZwJ3ufhqwH/hwUVsj0gXdSS3SA2Z20N2r8qRvBt7j7pvCRRJ3uPsYM9sDTHD3tjB9u7uPNbPdQI27t0SOMRX4lbvPCLe/CCTc/atHoGkih1EPQqT/eCefO8uTT0vkcwqNE0oJKUCI9J+/iry/FH5+kWBlWYCPAc+Hn58CroP2Z2gPP1KVFCmUfp2I9MxQM3slsv0Ld89MdR1iZr8l+OG1MEy7HrjXzP6e4MlvV4fpnwOWmNmnCHoK1xGs3Cly1NAYhEg/CMcg5rr7nlLXRaS/6BKTiIjkpR6EiIjkpR6EiIjkpQAhIiJ5KUCIiEheChAiIpKXAoSIiOT138wbjeKVbfzlAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(losses0, label='Model 0')\n",
    "plt.plot(losses1, label='Model 1')\n",
    "plt.plot(losses2, label='Model 2')\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('Epoch')\n",
    "plt.title(\"Learning rate %f\"%(learning_rate))\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f72684a",
   "metadata": {},
   "source": [
    "# 6 - Test the trained model\n",
    "- Let's create some random points in the x-axis within the model's interval that will serve as test data.\n",
    "- We will do the same data manipulations as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 607,
   "id": "b953f650",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_points = 50\n",
    "x_test = np.random.uniform(0, np.max(x_norm.detach().numpy()), test_points)\n",
    "x_test_reshape = x_test.reshape((int(len(x_test) / n_input), n_input))\n",
    "x_test_torch = torch.from_numpy(x_test_reshape)\n",
    "x_test_torch = x_test_torch.to(dtype=torch.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cc5f187",
   "metadata": {},
   "source": [
    "Now we predict the y-value with our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 608,
   "id": "cfa4d71a",
   "metadata": {},
   "outputs": [],
   "source": [
    "y0_test_torch = model0(x_test_torch)\n",
    "y1_test_torch = model1(x_test_torch)\n",
    "y2_test_torch = model2(x_test_torch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 609,
   "id": "bfbc0ae0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x169346f70>"
      ]
     },
     "execution_count": 609,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD4CAYAAADsKpHdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABCRUlEQVR4nO3de3zT1f348ddJmrSElgIt5VZ64eIFLDe5KV6GgKiIsItTrMq8VWDzss3bVjd1Xzu3/XbBbYLrmA414mVTBt613lFAUK4CgtCWcqfQAk2hbXJ+f3ySNm2TNm3SJmnfz8ejjzSffD6fnE/S5p1z3ueitNYIIYQQpnAXQAghRGSQgCCEEAKQgCCEEMJNAoIQQghAAoIQQgi3mHAXoCnJyck6IyMj3MUQQoiosW7duiNa616tOTaiA0JGRgZr164NdzGEECJqKKWKWnusNBkJIYQAJCAIIYRwk4AghBACiPAcghCi86murqakpIRTp06FuygRLS4ujtTUVCwWS8jOKQFBCBFRSkpKSEhIICMjA6VUuIsTkbTWlJaWUlJSQmZmZsjOK01GQnQw9k12MhZkYHrERMaCDOyb7OEuUoucOnWKpKQkCQZNUEqRlJQU8lqU1BCE6ADsm+zkFuRSVF6EQqExZjEuKi8iZ0UOK4tX8saONyguLyYtMY28yXlkZ2WHudT+STBoXlu8RhIQhIhi9k127nrzLkorS2u3eYKBh6PawZNrn2wUJICIDgqi/UmTkRBRyL7JTvIfkrn+levrBQN/fAWJ3ILctipe1FNKccMNN9Ter6mpoVevXlx55ZUtOk9GRgZHjhxp1T7r1q0jKyuLwYMHc+edd9Iea9dIQBAiytg32clZkRNQIGhKcXlxiErU8XTt2pXNmzdTWVkJwLvvvkv//v3btQzz5s0jPz+fHTt2sGPHDt566602f04JCEJEmdyCXBzVjoD3V/hua05LTAtVkcLLboeMDDCZjFt7aJLol19+Oa+//joAS5cuZfbs2bWPHT16lFmzZjF8+HAmTJjAxo0bASgtLeXSSy9l1KhR3H777fW+1T/33HOMGzeOkSNHcvvtt+N0Ov0+9/79+zl+/DjnnXceSiluvPFGli1bFpLraooEBCGihKf3UFF5E1PVaLC4MulWPYukqrtId/2Wc+P+Re+ae+hWfTWxzrNAm7BZbORNzmu/wrcVux1ycqCoCLQ2bnNyQhIUrr32Wl544QVOnTrFxo0bGT9+fO1jDz30EKNGjWLjxo389re/5cYbbwTgkUce4YILLuCrr77iqquuorjYqIVt3bqVF198kZUrV7J+/XrMZjP2Jsq4d+9eUlNTa++npqayd+/eoK+pOZJUFiIKeJqJ/NUMlO5CfM00EpxXYtF9AEiOjyUlIZYYs6LqdDLlDvf3P9MxLhkSx5WDZ9aeO7cgN2p6INWTmwuOBq+Jw2Fszw7uGoYPH05hYSFLly7liiuuqPfYp59+yn//+18ALrnkEkpLSykvL+fjjz/mlVdeAWD69On06NEDgIKCAtatW8fYsWMBqKysJCUlxe9z+8oXtEfPKwkIQkQBv81EWhHvnEr36jmYSSQjpYZ5F2Yx6cwUUrrF1dv1WEUVK789wstrS3h/02Eu/vYDLhxazlPbb8dRUwFEYQ+kYj95EH/bW+iqq67innvu4cMPP6S01KsnVxMf2L4+uLXWzJkzh8ceeyyg501NTaWkpKT2fklJCf369Wtp8VtMmoyEiAK+EsBmnUTvqsdIqr6Tgb26suzHE/nwZzO5Zmxao2AA0KOrlSuH92PJzeN4/c4LOKdfIivW2oivuB+z7lm7X1T1QErzkwfxt72Fbr75Zn7961+TlZVVb/tFF11U2+Tz4YcfkpycTLdu3eptf/PNNzl27BgAkydP5j//+Q+HDh0CjBxEUZH/pr++ffuSkJDAqlWr0FrzzDPPMHPmzJBcU1MkIAgRBRomgOOcI+l76q/E6SH84QfD+fBnVzFyQPeAzzesXyLP3jKOUsvfiXUNpc+pP2Nx1U2BEDU9kPLywGarv81mM7aHQGpqKnfddVej7Q8//DBr165l+PDhPPDAAyxZsgQwcgsff/wxo0eP5p133iHNHZiGDh3Ko48+yqWXXsrw4cOZOnUq+/fvb/K5Fy1axK233srgwYMZNGgQl19+eUiuqSmqPfq2ttaYMWO0LJAjRP0cQteaS0iqvgunqYR7p/fgromzmz+BHxkLMth3TJFS9WtMOoFD1t9w2rwJgPTE9LDkE7Zu3crZZ58d+AF2u5EzKC42agZ5eUHnD6KFr9dKKbVOaz2mNeeTGoIQUSA7K5v8Gfmkx9xMcvXPUNZveOQHiUEFA4C8yXlYYg9xIPbn1KhDpFQ9RKxzGFCXT4j4uZCys6GwEFwu47aTBIO2EJKAoJS6TCm1XSm1Uyn1gI/Hs5VSG90/nymlRoTieYXoTGpOTIQT32P68L588+ufccvo4D/4PIEmtXs3DsbmuoPCw1hcA4EoyyeIoAUdEJRSZuAJ4HJgKDBbKTW0wW67gYu11sOB/wPyg31eITqTl9fu4aHlW5g6tDcLrhmJNSZ0lfvsrGwK7y5Eq3IOxebiUidIOf3r2kRz1OQTRNBC8Vc1Dtiptd6lta4CXgDqpcO11p9prY+5764CUhFCBOTTHUd44JVNXDgkmb9fNwqLuW1aetMS03CqYxyy/gYTNnqd/hVoS8cZ0SyaFYq/rP7AHq/7Je5t/twCvOnvQaVUjlJqrVJq7eHDh0NQPCGi167DJ5lvX8fgXvEsuv5cYmPMbfZceZPzsFlsVJsKOWL9I7F6CCnO2zvGiGYRkFAEBF/D53x2XVJKTcIICPf7O5nWOl9rPUZrPaZXr14hKJ4Q0am8sppbl6wlxmxi8ZwxxMe27TjS2sR1YjqnzF+A7T26VF9GTy5p0+cVkSMUAaEEGOB1PxXY13AnpdRwYDEwU2sd3DSNQnRwWmse+O9Gio86ePL6cxnQ09b8QSHgySe4HnLxzS//xIgB3bnvvxs5UN651jeOhOmvc3NzGTBgAPHx8S16zmCEIiB8AQxRSmUqpazAtcBy7x2UUmnAK8ANWutvQvCcQnRo9tXFvLn5APdMO5NxmT2bP6ANWGNM/PXakVQ7XeS+uqld5uOPFJEw/fWMGTNYs2ZNuz5n0AFBa10D/AR4G9gKvKS13qKUmquUmuve7ddAErBQKbVeKSWjzYTwY+v+4/zmta+56Ixe5Fw4MKxlSU/qyj2XnknBtkMs39Co4h8ZdtthWQY8bzJud0f/9NcAEyZMoG/fviG5lkCFpLuC1voNrfUZWutBWus897YntdZPun+/VWvdQ2s90v3TqlF0QnR0p2uc3Ln0K7p3sfDnH47AZAr/2sI3TcxkVFp3Hl6+hdKTp8NdnPp222FNDjiKAG3crskJSVAI5/TX4SIjlYWIIH8t2MGOQyf5/Q+GkxwfG+7iAGA2Kf7w/eGcOFXDH9+JsBbfDbngbDALrNNhbA9Sc9Nfe3IMDae/vv766wH/01+PHDmSgoICdu3aFXQZQ02mvxYiQmwqKefJj3Zx9bmpTDrT/1z5Pu22Gx+CjmKw9jT6+VUfBVsajHB3G/U87tmWGfhI5yG9E5hzfgZPrdxN9vg0zumf2LLytRWHn0Fz/ra3ULimvw4XqSEIEQGqalzc8/IGkuOtPHhlw4H+PtS2myuj7fzz6+uaTapKobqU2iaUVTfB6puDbla5a8oQkrpaeWj5lshJMNv8DJrzt72FwjX9dbhIQBAiAjzxwU62HzzBY9/LIrGLxfdO3kHg8xvcH/DgZ9hPHV0Nrqr621rRrNItzsJ9085iXdGxyEkwj8gDc4MuuWZbXa0oSOGc/vq+++4jNTUVh8NBamoqDz/8cEiuqSky/bUQYVZ4pIJL//Ixl53Th7/OHuV7J0/ytGF7eVAUXOeq39zUTHOSy6WZ/rdPqThdw3s/uzikcyp5tHj66xaUv6OR6a+F6EC01jy8YgvWGBMPTm/iQ9BX8jRYtrQW99IxmRT3TjuD4qMOXly7x+c+7S4zG2YVGsFtVmGnCQZtQQKCEGH0ztcH+XD7YX469Qyfy17WCiZJqixgstbf5mlW8ddL5/Pr/fbp31f1Hlh28otlH5HxlyGRv16CCJgEBCHCxFFVw29WfM1ZfRKYc1560zu3JEkaEw+WJECBLR0mPA3jnzJ+92wbl298k24q0PioLdg32bn9tRwOqHxiSOJY6YjoWERHBES6nQoRJk98sJO9ZZW8dPt5xDQ3pfWIPB85BAVoUGbQTuODvqn2c1/bbWleyWkfPMln97G5Bbk4qh1g/hqH6Qu61fyAvVVvkFuQ2+5LbYrQk4AgRBiUHHPwz092891R/QObq8jzYR7q5KnPQNOAVy3Ce7GccstS+p7+Mwk1l1Fcviy4coiIIAFBiDD449vbUcC9084M/KDM7NAnTOsFGj81Ba/mqrTENIrKjf2qTN9QaVpPQs136ZG0MbTlEmEhOQQh2tnGkjKWrd/HrRdm0q97l3AXp66XznnPNdun37OIjkd5zIvE0JNTJ0Z3qDxCuKe/djgcTJ8+nbPOOothw4bxwAONlqpvExIQhGhHWmvyXt9KUlcrcy8eFO7i1JeZbSSbfSWf3TyL6CR1SQLgtGkTp0xbiam8jJzl8zpMUIiE6a/vuecetm3bxldffcXKlSt5802/C02GjAQEIdqBfZOdjAUZxD90Hqt3H2Xi0DIS4vyMSA6nAPr0Z2dlE291L9qi4HjMi8To3qhTY8ktCH5SuZbyvLamR0xkLMgIWVAK5/TXNpuNSZMmAWC1Whk9ejQlJSUhua6mSEAQoo3ZN9nJWZFDUVkJ3at/RLXaw7+23RLV36a9k8uVprVUqSK61cyiuCw0k8oFqva1LS9CoykqLwpZN9hImf66rKyMFStWMHny5KCvqTkSEIRoY56umvHOqVj0AI5ZnsZRcyIs36ZDYredNIvXR4eCEzH/w6oHktal7T+0vNV2g/XiqHaE5LWNhOmva2pqmD17NnfeeScDB7b9YknSy0iINlZUXgTaQmL1tZwyfU2lyVgW0ftbdtRwT3WR19NJziFwuFtEKswf0qPmRwy1/bhdi+PvNQzVaxvu6a9zcnIYMmQId999d8sK3kpSQxCiDdk32VEoEpyXE0MyZTHPGuPJMLpwRh33VBfZ3SA/BdJjjMtJs1QxOeMg2/dZ2H2kot2K4+81DNVrG87prx988EHKy8tZsGBBSK4lEBIQhGhDuQW5oGNJrL6aStMGTps3AcaHaF7WFU0fHIm8Bqlld4PCTHANMW5/3/UPxCjN0yt3t1txGnaDBbBZbORNju7pr0tKSsjLy+Prr79m9OjRjBw5ksWLF4fkmpoi018L0YZMj5hIqP4+PWp+xH7rPVSZt9U+pkekGz15osmyjCanuvjZvgd5s3wiq3452f+6Ds1o6fTX9k12cgtyKS4vJi0xjbzJeZ1mGo1QT38tOQQh2lBal1Scld/HYVpTLxikxxCyZR7bVTNTXdzc/QVeOTKBV78s4UcTM9ulSNlZ2Z0mALQ1aTISog1NYhpmEiizPFe7zaYgL4mQLfPYrjyD15TZ58PnJFczIjWR51YXR84ymyJgEhCEaCNHK6pYU3Yp59g+pZ91FwqjZpCfYrS/h2qZx3aXmQ0Tlvid5iJ7Qjo7D51kze6jrX4KCSbNa4vXSAKCEG3kHx99S6UrjgWpz9VLvmZ3w1ivIJpX9mpimosZw/vRLS4G++rWNYnFxcVRWloqQaEJWmtKS0uJi2tiUaVWkByCEG3gaEUVz64q4qrBNQzuehS8Zykw22DM42ErW8j4mX21i9XM989N5blVRRw5OZTk+NgWnTY1NZWSkhIOHz4cqpJ2SHFxcaSmpob0nCEJCEqpy4DHATOwWGv9uwaPnwU8DYwGcrXWfwzF8woRqZ76dDeV1U5+ctUUqMjvdIvAZ49P4+mVhby8toR532nZJH4Wi4XMzPZJSIv6gg4ISikz8AQwFSgBvlBKLddaf+2121HgTmBWsM8XkN32TvcPKCKA+++u/EQpS7Y9zRWZZganJABtsI5BhBucksCEgT15fk0Rt180EJOp8ehdEXlCkUMYB+zUWu/SWlcBLwAzvXfQWh/SWn8BVIfg+ZrmHlpv9JXWPteFFSLkvP7u/n3kSk44u/BjS26n/rvLHp/OnqOVrPy26fUAROQIRUDoD+zxul/i3tYqSqkcpdRapdTaVrUhuofW1+NZF1aItuL+uzvp7MJTR2YypdsqhsZu7dR/d1OH9iaxi4WX17b9tM0iNEIREHzVBVvdPUBrna+1HqO1HtOrV6+Wn8DfYJ9oHAQkoof77+vZ0isodyZwR8qL9bZ3RnEWMzMHnuStjUWUP5NgjHLuxDWmaBCKgFACDPC6nwrsC8F5W8ffYJ9oHAQkIt+a+bA0BtBUumJZfPi7XBS/jhG2Hcbjnfnvbredq2t+TZW2sLz8Qmm+jQKhCAhfAEOUUplKKStwLbA8BOdtnRF5za4LK0RIrJkPOxeBNvqU/nzvNEqd3Xmp6gUydoP9pKVz/91tyOUc6xbOitvNf45ONbZJ821ECzogaK1rgJ8AbwNbgZe01luUUnOVUnMBlFJ9lFIlwM+AB5VSJUqpbsE+t0/+BsyAUWV93iRVVxEa3+bX/vrvMgsryr7PKdNGTpu3UlQDOYcU9pNhLF+4OYp5/gRs0++yofIMMnamYT9Ok5PjifDqHLOdenqAeCebzbZGC4gL0SLP16XPMrZfDqd/zEHrLzll3li7PT0xncK7C8NQuPCzP5VMTkkpp1zdSD31DMdjllNlfcqYumPac/K/10aCme20c0xdIT2PRFtwT/BW5YqhpupqTpm2csq0sd4uUbkqWojkHjVWVHOp41Sa1hBfMwmHy0xuKfK/F6E6R0CQnkeiLQzKAWBZ2SRidArlMS806nMXlauihUhxRd3kdidj3sNMD7q4xlBcg9FsJM22EadzBATpeSRCZbe9Lhe17w1qkqfwxKEf0s+6A5N5Xb1dQ7lyVzTyDoaVprU4OUbXmktI88yPID2OIk7nCAjS80iEgo9R8N9f56Soqi/reZFKFPHWeBSK9MR08mfkd+qFW/Im52EzW407ykWF+RNsrnFMjetqbJNm24jTOQJCE1P1ChGwBrmoeQcUXxy/miq1m0rTajSak1UnmTtmLoV3F3bqYADGSmZzRt1S24pWEfMBCguvlJ9v9DYCabaNMJ0jIIDx4T+rEK5zGbcSDERLNfjweq7sfKw6jfKYl0DV9dbLX5ff8MhO640db9ROW1CldlCt9mKp+Y6RWAZpto0wnScgCBEsrw8vrSGh+hqq1R4c5pX1dnNqZ8MjO616vawUVJg/JNaVxd7qJGm2jUASEDy8k4UycE344pWLeu/4OKx6IOUxL4Ny1dvN7Ge94c6oYS+rCvOHKEz015Mgc47RDCf/cxFDAgLIlNkiMO5clO6Szt8PXUsX80EqzB822i3n3Jx2L1qkypuch81S16GjxrSf02obzurvYF+/WP7nIowEBJCBayJwmdl8MuwLNlSewa9nTmHe2NtrawRmZWbemHksnL4wzIWMHNlZ2eTPyCepS1LttoqYDzG7Mpi3r19dchnkfy4CdI6pK5rzvAnfM3YrIwkthJcfPvk5e445+OjeSVhj5DtVIDIWZFBUbsxhZNKJ7qksXiGxyxIK662WKf9zwZKpKxqwb7KTsSAD0yMmMhZkYN/UTDVUBq6JAK3aVcqawqPMvXiQBIMW8E4uu1Q5p0xf0dV5McXVDYZ2y/9cWHW4v2j7Jjs5K3IoKi9CoykqLyJnRU7TQUEGrokA/f39nSTHx3LN2AHN7yxq+Uoux+gU0tTQuo3yPxd2HS4g5Bbk4qiunw9wVDu4a9lcMu6NwfSwIuPeGOyL5tftIAPXRAC+LD7GpzuPcEQtxfZbS2C1TwE0Ti47zKvQnGa8dSryPxc5YprfJboUl/uea73UeZLSeOP3ongnOXsXwSLInudOAH4G5ALFQBqQB2T6PJXopH7xv49wqUpKqp9Hq7raJ9DpRyU3x/P65BbkUlxeTFr33gyzKL49Nh3ntX/GbPK1Eq9obx2uhpB20k8f8AZ/bw4L5O5yjyi12yEnB4qKjBFHRUXGfbt8+xOGzXvL2b7XQrl5GVqdqt3uqHaQWyA9YwKRnZVN4d2FuB5yUXh3IfMvnMCRk6dZvbu0+YNFu+hwASHvbSe2qgYb/XSkKu7qHlGam4t9kIOMu8H0EGTcDfZBDsht5h99/nyIiQGljNv585veX0StJz7YiYuTnIh5rdFjnXnNg2BMOqsXXSxmXtu4P9xFEW4dLiBkH08nfwWklxnTy6SXQZLD975pFUZtwt6tiJwZUNQdtDJuc2YY2/2aPx8WLQKnO6g4ncZ9T1Cw2yEjA0wm41ZqG1Hrm4MneHPzAUxdP0Grxn9MnXnNg2C8uv1FKs2reXb1JjL+MlDyMRGgwwUE8vLI/tZG4QJwPQKFC+Dxd82Nag22asgbaLT/5k4z47DWf9xhNbb7le9nArP8/MCboCRoRIWFH+zEZjXzy8vOr5cYBVnzoLU8vQEPud7GTCIHjyU23xtQtLmOFxCys40P5fR0oyknPZ3se5aQnzqP9JNmo9Zw0kx+/3m1CeXieN+TkfnbDtTVDHxtz80FR4Nvko4GTVDNBQ27HZKTjWtQyvhdAka7232kguUb9nH9hHRyxl5H/ox80hPTZc2DIHl6A54yrcOFA5vzQsnHRAAZqUz9UZTePAuk1zhd7DlWye4jJzlQfpqyyirKf/MY1cqESWvM2kUPx3GSHcfoU1HGkCNFpJwobZjHNj7YXe5RmBkZRhBo9KTpkJcHN98MVQ2qNRYLPP20EfQ85s83AqDTCWazEVQWytQJoXLffzbwv/X7+OT+SaQkxIW7OB2G6RET2p3cS6r6OV2c51ISdwNKuXA9JCOVgxHMSOUO1+20NfIm55GzIqd2/EKMqw8JahRj42/lsgUfs/PQSWpc9QOndexMrFWncSlFjclMVUz9NqfulccZvn8H5xdt5PziDZxz4FtMaV5tzcV+EpHFxUZNomEwAKiuNh7zBARPHsPDk8eAuqBgtxvHFBdDWpoRbLLlG20g9hx18MqXe7l+QroEgxBLS0yr/RLmMH9CvHMSca6R9O55JMwl69wkIABXnz2bnftjeHrVGqorz8Si+wOwrSSGUWlxTDorhYHJXRnYqyv9unehh81KnMXM/P/rQb6rDCcQo+O4TqVx+/Ur+ea1D9j+7hrW9hnC7ybdBED/44e5cmACV+0rZ1i/ROPD2VcNIS3Nf7CA+o81lcdYuLCuWcrTfOVplgIJCgH4+/s7MZkUt188MNxF6XC8v4RVmr7ERQWJ+jvkTR4V7qJ1ap22ycjl0qwrPsayr/by+qb9lDmqscaYOH9QEpeclcJ5A5MY1Csek58BM/OfGsaiPV832j5vwFAW3ryl9pv5odLjfHLuFN6Yeh0fVVipcWnGpPfgR64Spj1wK5aTJ+oOttmMD/PcXN/BAowmpcJC43fVxGAerZtulvKcQ/hUVFrBJX/6iBsmpPPwVcPCXZwOyb7JXjdQTecS5xzPhocuJzZG1pMIRjBNRp0uIOwtq+SFNcW88uVe9pZVEmcxMW1YH64c3o+Jg5OwWQOrNMU8ovCVVjYDNQ/5fk3LHFW88uVelnxeSFGpg1SLk7s+fZ7vfvwfYgak1jXn2O2B5RBiYnwnt81mqKkxei/5en+9cxmtYbfDXXdBqdeAoqQkePzxDlPzuOflDazYsI+P75tE727SXNTWPth2iJv+/QWLbxzDlKG9w12cqBb2HIJS6jLgcYzPw8Va6981eFy5H78CcAA/0lp/GYrnDoTLpfl05xGe+byI97cdRAMXDE7mnmlncOnQPnSNbfnL4K//UVOLJ3a3Wbn5gkzmnJ/B+9sO8XjBN9w7/gaevHIu9192FlOH9jYS0Z4PVe8PXV8fuDk59XMI3tuh6Wap1rLb4aabjHyGt9JSY7uHp5ZjNhtBy5Msj4KAsftIBa98WcJNEzMlGLSTiYOTSexi4bWN+yQghFHQAUEpZQaeAKYCJcAXSqnlWmvv9pTLgSHun/HAIvdtmypzVPGfdSU8t6qIwlIHSV2tzL14ELPHpTGgp635EzTBjO8Pf09lt151ODGNvMl5td0TzSbF1KG9mXJ2Cm9vOcAf3/mGnGfXMenMXjxy1TmkJdmMD87mPjw9iWN/vYzy8urnEMBolsoLot98bm7jYOBRXW0EscrKuuf01GCiKH/xt4IdWGNMzL14ULiL0mlYY0xMG9abNzYd4HSNU5qNwiQU4xDGATu11ru01lXAC8DMBvvMBJ7RhlVAd6VU3xA8t08bS8q49+UNjP9tAY++vpXk+Fgev3Ykn/3iEu677KyggwFAzoChfrcHOgW3UorLzunLm3ddyIPTz2bN7qNM+ctHLPxwJ05XgE15CxcazUNaG7feXU59jMkgPz+4D+SmEt5g1BQajsHw8DUWw9fAvLYYsBfgmtk7D51k2fq93HheBr0SYoN/XhGwy7P6cvJ0DZ/ukJ5G4RJ0DkEp9QPgMq31re77NwDjtdY/8drnNeB3WutP3fcLgPu11o0SBEqpHCAHIC0t7dwif8lVP06ermHMo+9iUopZo/pz/fh0hvbr1trLa9L8p4aRv+drnBg1gxx3Qrm5cQ3+HCg/xSMrtvDm5gOMSe/Bn3840qgtRBJ/iepAefIXDXtAgVF7mTMHlixpvD2YQOZZM9t7mVSzzed0y3cu/Yr3th7kk/smkRQvAaE9VdW4OPfRd5k2rA9/vHpE3QO77cbSmo5iYwGdEXkyTXYTwppUVkpdDUxrEBDGaa3v8NrndeCxBgHhPq31uqbO3dqk8sqdR8hKTaRbnKXFx4aC96AbbwrV7KAbrTX/W7+PX/1vM06X5pGrhnH1mAhajMVfDgGMpHe3bvWTzQ15ejj5CyyenIO/41pjWYZ7MfcGbOkwq+6c3xw8wbQFHzP34kHcf9lZrXsuEZSfvrie97cdYu2DU7CYTS0K5sIQ7iU0SwDvT6xUYF8r9gmZiYOTwxYMwP9kZw23+1rqU7lrNm/ffREjUrtz73828sB/N3Kquql0dTvKzjZ6OiUl1d+elGRsf/xx4xu9L975C39NT/6mBGmuqcof+3yo8FOjcdQ/51/e/QabxcxtF8q4g3C5/Jw+lFdW8/m37i8VG3LrBwMw7m+QKS7aQigCwhfAEKVUplLKClwLLG+wz3LgRmWYAJRrrTvsnLd5WVdga7hUrDK2ezSXZ+jXvQvP3TqeH08axAtf7OEHT37GnqN+2ubbW3Y2HDli5C08P0eO1CXCPXkLML7xQ+P8hb+eTmY/ycTW9Iyyz4eqRY3Wwqile9b++lXxMd7cfIDbLhpIz65WPweItnbRGb2wWc28ufmAscHh54uAv+0iKEEHBK11DfAT4G1gK/CS1nqLUmquUmque7c3gF3ATuCfQIdeOCD7xBvkp0B6jPFZlB4D+SnGdg9/S316T+5lNinunXYWi28cQ1Gpgxl//5RVu6JgMZHsbKN5x5Po1tq4750DyMtrXJOw2Yy8gq/trekZVZYP/tIAp4GXjF+11vz+rW0kdbVyq9QOwirOYuaSs1J4Z8sBo2OFzc8XAX/bRVBCMtup1voNrfUZWutBWus897YntdZPun/XWusfux/P8pVM7lAcxWR3g8JMcA0xbrO7Ue9bjb9FVXxtnzK0N6/dcQFJXa3c8K/V/HddSVuVvP346wG1cGHoekZ199P8pIHFwIqjAHz0zWFW7TrKHZcMJr65MSkNe0BNmSKLJIXY5ef0pbSiijW7jxoJZLOPJsjqk357ionW63jTX0eCAL7VNJdnaJhf+HTfMl6ZP5GxGT35+csb+NM724nkUeYB8dQkXK76NQh/21uqzE/z0xGMNbTT0nC5NL9/azsDenbhuvHpTZ/P15TlBQWNF0lSSta3CMJ3zuxFnMXEm5v3G4njcflgbZCzqi41ks0SFEJKAkJb8PWtxmwztrvlTc7zu9iKr/zCDa/cwC/ev4slN4/jmjED+Nv7O/npi+updspUwX51zzGahrx5morczVDLN+xj6/7j3HPpmVhjmvl38LXOhT/e61vI2hYt0jU2hovP6MVbmw/gcmkjKMTEN95RksshJwGhLXi+1djSAWXcNugml52V7XexFV/5BY3mybVP8tLXS/nd97O4d9qZLFu/j5xn1lJZFSE9kCJN9kKwzoNjZnBh1AwWA3uNZqiqa2bzp3e3M7RvN2YM79f8+Vra08nhMEZu33RT/a64paXGXFUSFPy6Iqsvh06c5qs9x4wNklxuF51ucrto4G8cA9Qf3Pb86mJyl21iTHoPFs8ZS2KX8HW1jUaLP9nFo69v5d83jeU7Z6Y0f0Cwg/IaaunYik60tsWJU9Wc+3/vccN56fzqyqEBjyUR4R+HIEKsqUXbvZPO141P42+zR7F+TxnX5q/i8ImG7SPCn9KTp3m8YAcXndGLi8/oFdhBvnpGBaMlNQ5f+YsbbqhLZnvfdoD8RUKchQuGJPPW5gNGrsxfcrlGksuhJAEhnPzMr5M3OQ/lp/N8w2Bx5fB+LJ4zlsIjFVz95GfsL69s40J3DH9+9xscVU5+Nf1sVFPrSnjz1TNq8mT/YydstsYD+Ly1ZGyFr/yFp3bvndSG+sEiioPD5ef0YW9ZJRtLyuuaYS0NXs8qSS6HkgSEcPEMyXcUAdq4df9hZ2dlM3fM3EZBwZN0bujiM3rx3K3jKT1ZxTX/WMXeMgkKTdm6/zhL1xRzw4R0hvROaNnBDXtAvfde3ViL555r3F328ceNKT0aslpbNraipfkLT7DwJLfnzw/9hIFtbOrQ3sSYVN0gtcxssEhyuS1JQAiXZobkL5y+kGe/96zPpLO3+a/PJ+Y3MYz5d092cjf7ysu55h+fR86o5gijtebR178mIc7C3VOGhPbkvrrL+prqIykJnnqqZe3/waxh4XDAk0/Wb27y9ICC+mMrkpONnwgIHN1tVs4blMSbm/fXdbGW5HKbkqRyuDxvAp+JYwXXBdaVdP7r81m0tv4COVbXINKc/4+U+ESW3jYh8mZLDbN3thwg59l1PDxjKD+amBnu4gTO1+ywwfIsWtTUeYOdaTZIz68u5pevbuKNOy80Zi2W5HKzJKkcjUIwJD9/XX6jbVWmbymJuQ9HVQ3X5H9O4ZGK1paww6mscvKb175mSEo82ROaGYQWaRrOERVo3qMpxcXNj61ouIYF1NUo2iGRfemw3pgUvLXZPfVZAGN8ROtJQAiXEPxhO7Xv8Qen1E6ev20Cp2tcXJP/Od8ePhlMSTuMxwt2UHKskkdnnWNMrRxtvOeIevbZxhMIem4bBgt/wSMtLbDchPc+3r2doPGKeA2DQpCD8pLjYxmX2ZM3vPMIzYzxEa0Xhf8VHURzf9gBrPBlVr57t5iVmbP7dmPpbRNwujSz81ex81DnDgp//mQpT370DSfN73HN8nMbrV4XdXxNIOi59QQLT3J77lz/EwYGkpvw3qepGoWvFfF8Dcq7/nojMASY6L4iqy87D51kx8ETxobMbKN56DqX8QVqQ26zK+GJwEhACCfvP+xZhfWDQcMeSJ/fAGvqT5yWc26Oz9N6tp/ZJ4Hs7xzhUMVhJv15GRl/HB/9H4St8OxGO396ax9OTnLM8pTfJU07jIbJ7aYmDGxubEXDmWabq1F4P97U+tulpca8T/4S3V6mDesDUNfbyKOJnnqidSSpHIn8Jc5QcN6z9arH81+fT/66fJzaiVmZyTk3h4XTjXWVPXMiVZ3uSe/Tv0WhKO/6GxbNeqhRb6WOLOO318HxbI5YFlAR817t9uaWNO00vEdA93SvEXH0qO/R0M2N1vYefW0y1XV/DZSP0dv2RfPJLRxIlSkO66k7yCuA7K/N8Ccn+BpT2MkTzGFdQrMtddqA4LcHEi36Y/de2znG1d8dFMyYk/5O4b2fh6asEW7h6uf53atmqky7OWj9Jai61zWQJU1FA031dmrYI6k1U3141tz2PN2i+eTsXYRZzaRn9W3sjc3BWrOP/BWQfT9+Fj8KvKdeRyS9jDqapnoataC/tfc0FzWmvRyM/QUaJ87SO9h24HgwJYwKz2208+jy3UAMpZbH6wUDaHqKEOFHoCvigVG78DUorykNchq5u/JxWMBh+gwAm/N8HFbInYwxWaEvXivhAY3XsIiCQXnhIgEhEo3Iw++6jy3oltrwA6/GtI+Dsb/AbHIxO38VX+/r2EHhwdfewOocyTHL09SY6rc/+xv1LQIQyIp4nv18rb/tj4+V8Yq7Gr2YnKbDnFbbsTknGtsTMaYx9ze9uYevOaAa5iokYNSSgBCJMrNh8FwaBYUWdkv1teaCNbaMn11ppovFzHWLV7FlX3kIChxiAfSwas6eow5cx2dRadrASfMbjR73NepbtAHv9bcbTu0xb16zK+OlVdT1pHOYVxKrhxDj6k1aOcYiR4uBwxjTmx+m3kp4gO9eUd69oQIJGJ2IBIRINW6hkUAOor+1vzUX7jx/Ni/knEdXawzX/XM1m/dGUFAIQc+RaqeLn764HpNSlFoWNGoqSk9Ml2AQDr56PzWzMl7ewBxs7o5KDvNKABKrzievwL3DZ8DdwA3uW/dKeLX89YrybA8kYETgtB5tRZLK0W7NfPg2H7QTlBkG5RjBJAB7jjq4Nn8VJ05VY791AlmpiW1c2ACEYGqC37+1jUUffssPJ1bwxKab6i02ZLPYpHYQZeyL5pO7K5/irk7SHH9hQLmTT/51vzEoTqn6PZkCTWx7ejP56wmllDGeI4Kn9fBHksqd1Zr5sHOREQzAuN25qNF4BX8G9LTxQs4EunWxcN3iVWzYU9Z2ZQ1UkJOXffTNYRZ9+C3Xjh3AH2b80O+qdCJ6ZM9bSOH/q+HZ7z8H3bawp8dZZPw+C/vG5xoPwvOV2PY3KA/8D8xLS2vdtB4eUZqXkBpCNFsaUxcMvCkzzK4J+DQlxxzM/ucqyiqqWXLLOEan9QhhIVsoiBrC/vJKrvzrpyTHx7LsxxPpYvWzToGIOnVjarrT/3Q+Ry35OLu8F1iAb2qlOV/daD3f/G+4oflxFA26yTZ7znaoTUgNobPyM5eR3+1+pPaw8WLOefSMt5L9z9V8sP1QCAoXAF/J41bO8VRZ5eS2Z9ZyqtrJE9mjJBh0MJ51xmtM+6hSu7A5J+KodpBbEMA6CL6mJfd+zN8o7pZO61Fb2GbyEhFMAkI08zOXkd/tTejXvQsvzz2PzOSu3LZkLf9dV+J3X/smOxkLMjA9YiJjQUbrpoDwlzyGFk9eprXmnv9sYMu+4/x19igGp7Rw0RsR8bzH1DjMK4lzDcOsk+ptbzV/AaOl03rUFraZRHYEk4AQzQb5nsuo3vYWdOFMSYjjxdsnMC6zJz9/eQP5H3/baB9P1b2ovAiNpqi8iOtfuZ7438a3LDA0tUCQvzme/Fjw3g5e37ifBy47i8ln9w68DCJqeI+pqXD3NrI5z2vbwYUNaw9JScZPE91kjcI2kZeIcBIQotm4hTB4Xl2NQJmN+55eRq3owpkQZ+Hpm8YyfXhffvvGNh5ZsYUaZ10bqafq3lBFdQU3Lbsp8KAQopWvnv28kMcLdvCDc1PJuWhgi44V0cN7TE2NqYQqVYTNOZGTVSfbdpJC79rDkSPGTxPdZI3CNpPIjmBBBQSlVE+l1LtKqR3uW5/ZSKXUU0qpQ0qpzcE8n/Bh3EIjgXydNm69u5w2s0ynP7ExZv527ShumpjB0ysLuXnJWsorjc7gTVXRq13V5BbkBtakFIIFgv63fi+/Xr6FKWen8Nj3slChWDRGRCTPmJqkLsaoZ4d5JbGuYRxzOCNv5tqm8hIRLtgawgNAgdZ6CFDgvu/Lv4HLgnwu0VJBfAs3mRQPzRjGY9/L4vNvj/DdJ1by7eGTzVbRi8qLyPnfzY2alJIfNWMv8OoO20TyOJCA8trGffz8pQ2My+jJ368bHZ0L3ogWyc7KJt4aDxgBQWHC5jwv8ORye2oqkR3Bgv0vmgkscf++BJjlayet9cfAUV+PiTYUgm/hs8elYb91AuWV1cz6+0pmD/kDyt88S4BZmXA4qxptL3W6yFm5qC4oZGZjT55DRqEZ0w7IKDRjT56D/SSNchQNvwG+9MUe7lz6FaPSurN4zhjiLNKjqLPw1FCrVRHVak/d3EahSC6LoANCb631fgD3bUqwBVJK5Sil1iql1h4+fDjY03VuIVp/dlxmT/73k4kMTIln6SddmdglD6VjG+1nUWac2v+0ww4NuauNdaDtm+zkrF5CUbUTDRRVO8lZvYS73ryrUY7C8w1Qa80TH+zkvv9u5IIhvXjm5vEkxLVwNk0R1WprqMrT2ygLk+4mM9eGSLMBQSn1nlJqs4+fmW1RIK11vtZ6jNZ6TK9evla/EAEL4fqzqT1s/GfuecwbeYqSo+dwrvMv9NJn1D6epODp3pr0mKbPU1xtjJHwlZx2VDsorSz1dRh7yg5yx9Kv+H9vb+eqEf34543nyliDTsg7uVxhXonCTHcukplrQ6SZf1/QWk/x95hS6qBSqq/Wer9Sqi/QTiOaRMAys0O2ALnFbOL+uJ8zMbM795T8lK6n/sj8pNf4eZ9nSTBXYkw5qcg5qHH4GeCZ5m7eaUkV3+LKpJ/zAV7ftJ/7LzuLuRcPlARyJ+UZlZxbkEtxWSGYDzO6xy1kZ80Ka7k6imCbjJYDc9y/zwH+F+T5RKRzFHNBwgbePWMec5JeY0nplUze/iTPl06jWpvJTtDk97GQ5OPz2qYgb7wxRsJfFT+pS1LdlN3aQmL1tfQ9/RcSLH35903jmPedQRIMOrnsrGwK7y7E9bCLuRdMYPdBK2WOxnmrWiGYTr2zCDYg/A6YqpTaAUx130cp1U8pVTsJvVJqKfA5cKZSqkQpdUuQzyvCxZ2QTjBX8nD/fF4dfA9p1gP8cu8dTN2+iBdPXsv3Jz3NkeHpPNcb0mOMVR3SLWbyJ84je7LRLdbXWg02i43HL3+cRdPzSbf+gP6nn6R7zfUMT3fx8T3TuPgMaUIU9aku66lxaQY8Nt13j7QQTKfemcjkdqJldtth1U2gq2s3aQ3vnxjLHw/cyNZTmXS3Wbj63FSuyOrLiNTumEy+v9HbN9mNqn95MWmJadw74beYT4/n2c+L2FtWydl9u/GrK8/m/EHJ7XV1IorYN9nJWZ5Dj5N/o0oVcTj2N42nNw/BdOrRJpjJ7SQgiJbbbYe1d0G1VwLYkoQ+93FW6ctY8lkh7209SI1L07tbLGMyejIytTtpSTb6dIsjzmJGozl6soq9ZZVs2XecdUXH2OReqGdcRk9uvTCTKWf39htMhMhYkEFReRHdq2+mW80M9sRlo5WD9MR0Cu8uNHZ63gT4+oxTxtQoHVAwAaHZpLIQjfhJVCvgPOC8QUmUO6op2HaQ97cd4qviMl7fuN/v6WxWM1n9E7l32plcObwv6Uld267sosPwdExwmFeSWPM9bM5xVMR8WL/Dgi3NTw1Buqn6IgFBtIlEm4XvjU7le6NTAThaUcXeY5UcOH6KavfcSN1tFvp0iyOtp40YGWksWigtMY2i8iKq1DfUqMPYnBdQEfNh/Q4LI/KMnIH3FC6tGIvTWUhAEO2iZ1crPbtaySIClukUHULe5DxyVuTgqHbgMK0kwTmdrjHJ9cckeGqyG3KNKVtsaUYwCFFX7I5GAoIQIip5j0k4cOxjujlncduwhWRnXV1/xxCOxenopJ4uhIhanjEJlQ9vIz3JxpFSyQ0EQwKCECLqKaWYMbwfK3ce4fCJ0+EuTtSSgCCE6BBmjOiHS8Mbm/z3aBNNk4AghOgQzuyTwJm9E1i+YV+4ixK1JCAIITqMq0b2Y13RMUqONV7mVTRPAoIQosOYMbwfAK81MRBS+CcBQQjRYaQl2RgxoDvL10uzUWtIQBBCdChXjejH1/uPs/PQyXAXJepIQBBCdChXDu+LUrBCksstJgFBCNGh9O4Wx4TMJFZs2Eckz+YciSQgCCE6nBkj+rHrSAWb9x4Pd1GiigQEIUSHc0VWH6xmE//9siTcRYkqEhCEEB1Od5uVKUNTWL5hX+1066J5EhCEEB3S90encrSiikfeeZmMBRmYHjH5X3d5WYaxutqyjE693rJMfy2E6JAuOqMXXWNd/GvlFg5YjFXTisqLyFmRA7inz95tr7+AjqPIuA+dcspsqSEIIToki9lEhflDrDWjMen42u2Oage5BbnGnQ259VdTA+P+htz2K2gEkYAghOiw9juXobBgc15Ub3vtusuOYh9HNbG9g5OAIITosPr2cFKldhNfM7ne9tp1l21+FtTxt72Dk4AghOiw8ibnUWX9hFh9JjGuVABsFlvdussj8sBsq3+Q2WZs74QkIAghOqzsrGwenT4TcBLvvIT0xHTyZ+TXrsdMZjaMywdbOqCM23H5nTKhDKCCGdqtlOoJvAhkAIXAD7XWxxrsMwB4BugDuIB8rfXjgZx/zJgxeu3ata0unxBCANz09Bq27j/Bp/dPIsbcsb8HK6XWaa3HtObYYF+ZB4ACrfUQoMB9v6Ea4Oda67OBCcCPlVJDg3xeIYQI2DVj0zhw/BQffXM43EWJaMEGhJnAEvfvS4BZDXfQWu/XWn/p/v0EsBXoH+TzCiFEwCafnUKvhFiWrumcvYcCFWxA6K213g/GBz+Q0tTOSqkMYBSwuol9cpRSa5VSaw8flmguhAiexWzih2NSeX/bIfaVVYa7OBGr2YCglHpPKbXZx8/MljyRUioe+C9wt9ba7xSEWut8rfUYrfWYXr16teQphBDCr2vHpuHS8NLaPeEuSsRqduoKrfUUf48ppQ4qpfpqrfcrpfoCh/zsZ8EIBnat9SutLq0QQrTSgJ42LhySzItf7OGOS4ZgNqlwFyniBNtktByY4/59DvC/hjsopRTwL2Cr1vrPQT6fEEK02nXj0thffoqPvvH53bXTCzYg/A6YqpTaAUx130cp1U8p9YZ7n4nADcAlSqn17p8rgnxeIYRosSlDe5McH8vzqyW57EtQs51qrUuByT627wOucP/+KSB1MyFE2HmSy09+9C0lxxyk9rA1f1An0rFHaAghRAPZE9JRSvHs50XhLkrEkYAghOhU+nfvwrRhvVm6phhHVQ32TfamF9DpRCQgCCE6nZsnZnL8VA0PrHiVnBU5FJUXodG1C+h01qAgAUEI0emcm96DrP6JLP/yOI6q+gPV6i2g08lIQBBCdDpKKW6amAHOPsS5RjV6vHYBnU623rIEBCFEpzR9eF8wHadbzVWNHktLTKtbb9lRBOi69ZY7cFCQgCCE6JRiY8xMOcdCF9cYLK7M2u21C+h0wvWWJSAIITqtP82aSWyMpp+ag0LVX0CnE663HNTANCGEiGaJNgs3TRzMPz5W7PzZcQb2iq970Jbmbi5qoAOvtyw1BCFEp3bLBZlYzSb+8dGu+g90wvWWJSAIITq1XgmxXDt2AK98VVJ/rYROuN6yBAQhRKd320UDAXjig531H8jMhlmFcJ3LqBlsyO3QXVAlIAghOr3UHjZmj0vjhS/2sPtIReMdOkkXVAkIQggB/OSSwVjNJv787je122rnOXrmejJ2OrB7r/XYAbugSkAQQgggJSGOWy7IZMWGfWzZV459k91rniMoqoGcQ9QPCh2sC6oEBCGEcLvtooEkdrHw+7e2k/teLo7q+gPTHBpyS702dLAuqBIQhBDCLbGLhTsuGczH3xzm8LE+PvcpqoGM3WA/aWmTLqjljmqOn6oO+XkDIQFBCCG8zDk/gzN6x9PLOQ+lrT73MZqPFPaToX/+37+9jUv//DGVVc7Qn7wZEhCEEMKLxWzikavOAWcySa7ZfvdzOKtCPk325r3lvLCmmMuz+tDFag7puQMhAUEIIRo4b1ASM0b0I7HmB6R3Het3v9ppskPA6dL88tVN9Oway92TzwjZeVtCAoIQQvjwq+lnEx9nZWSXv5DeLdPnPmmJoUsqP/N5IRtLyvn1jKEk2iwhO29LSEAQQggfUrrF8ZuZ57B+TxmX9v0DNkv9eY1qp8kOgT1HHfzx7e1cdEYvZgzvG5JztoYEBCGE8GPG8L5Mz+rLB5ts/Oq8f5GemN54muwg1Thd3PXCV5iUIm/WOSilQlDy1pHpr4UQwg+lFHnfPYcNJWUsXxXLuju2kxQfG9Ln+Ov7O/myuIzHrx3JgJ625g9oQ1JDEEKIJnS3WXny+nMprajiJ89/RVWNK2TnfnvLAf72/g6+PzqVmSP7h+y8rRVUQFBK9VRKvauU2uG+7eFjnzil1Bql1Aal1Bal1CPBPKcQQrS3c/on8tj3svh8Vyk/f3kDTpcO+pyb95Zz9wvrGZ7anbzvnhOCUgYv2BrCA0CB1noIUOC+39Bp4BKt9QhgJHCZUmpCkM8rhBDt6nujU3ng8rNYsWEfDy7bHFRQ2HbgODc+tYYeNgv/vPFc4iztP+bAl2ADwkxgifv3JcCshjtog2c8n8X9E3x4FUKIdjb34kH8eNIglq4p5ifPf8mpat+jiWtnSX3ERMaCDOyb6qbJ/qr4GNf9czVWswn7bRNISYhrr+I3K9ikcm+t9X4ArfV+pVSKr52UUmZgHTAYeEJrvdrfCZVSOUAOQFpax5o4SggR/e6ddhY9bFby3thK0cLP+OvskQxOSah93DNLqmdivKLyInJW5KA1mCov5OHlW+idGMuzN48nI7lruC7DJ6V101/WlVLvAb5mecoFlmitu3vte0xr3SiP4PV4d+BV4A6t9ebmCjdmzBi9du3a5nYTQoh2V7D1IPf+ZyMVp2v40cQM5l40iB5drWQsyKCovKjevlbXmfTVOVB1JhcOSeav146iR1ff8yQFSym1Tms9pjXHNltD0FpPaeKJDyql+rprB32BQ82cq0wp9SFwGdBsQBBCiEg1+ezevHX3hTz2xjbyP97F058WctEZvTh2ZCxd1UDAjEX3p4tzNFY9CCfH+f13s7h27ABMpvCNNWhKsE1Gy4E5wO/ct/9ruINSqhdQ7Q4GXYApwO+DfF4hhAi7lIQ4/nLNSOZ9ZxAvfrGHd74+QI+aH9U+rnFy2rSNozH/oGeP7Vw3fnv4ChuAYAPC74CXlFK3AMXA1QBKqX7AYq31FUBfYIk7j2ACXtJavxbk8wohRMQ4o3cCv7pyKL+6ciiL1z7PT996kFPVDmrUEVBOY5qLqfnhLmazggoIWutSYLKP7fuAK9y/bwRGBfM8QggRLW4dcx1dYjW5BbkUl7tIS0wnb3JeSKa5aGvNJpXDSZLKQgjRMsEklWXqCiGEEIAEBCGEEG4SEIQQQgASEIQQQrhJQBBCiFDbbYdlGfC8ybjdbW/uiIggC+QIIUQo7bbDmhxwGnMZ4Sgy7gNkRnbXU6khCCFEKG3IrQsGHk6HsT3CSUAQQohQchS3bHsEkYAghBChZPMzbb+/7RFEAoIQQoTSiDww2+pvM9uM7RFOAoIQQoRSZjaMywdbOqCM23H5EZ9QBullJIQQoZeZHRUBoCGpIQghhAAkIAghhHCTgCCEEAKQgCCEEMJNAoIQQgggwldMU0odBopaeXgycCSExYkEHe2aOtr1gFxTtOho1+R9Pela616tOUlEB4RgKKXWtnYZuUjV0a6po10PyDVFi452TaG6HmkyEkIIAUhAEEII4daRA0J+uAvQBjraNXW06wG5pmjR0a4pJNfTYXMIQgghWqYj1xCEEEK0gAQEIYQQQBQGBKXUZUqp7UqpnUqpB3w8rpRSf3U/vlEpNTrQY8MlyGsqVEptUkqtV0qtbd+S+xfANZ2llPpcKXVaKXVPS44NhyCvJ1rfo2z339tGpdRnSqkRgR4bLkFeU7S+TzPd17NeKbVWKXVBoMc2orWOmh/ADHwLDASswAZgaIN9rgDeBBQwAVgd6LHRdk3uxwqB5HBfRyuuKQUYC+QB97Tk2Gi6nih/j84Herh/v7yD/C/5vKYof5/iqcsHDwe2tfZ9irYawjhgp9Z6l9a6CngBmNlgn5nAM9qwCuiulOob4LHhEMw1Rapmr0lrfUhr/QVQ3dJjwyCY64lUgVzTZ1rrY+67q4DUQI8Nk2CuKVIFck0ntTsCAF0BHeixDUVbQOgP7PG6X+LeFsg+gRwbDsFcExhv/jtKqXVKqZw2K2XLBPNaR+L7FGyZOsJ7dAtGLbU1x7aXYK4Jovh9Ukp9Vym1DXgduLklx3qLthXTlI9tDfvN+tsnkGPDIZhrApiotd6nlEoB3lVKbdNafxzSErZcMK91JL5PwZYpqt8jpdQkjA9PT9t0JL5HENw1QRS/T1rrV4FXlVIXAf8HTAn0WG/RVkMoAQZ43U8F9gW4TyDHhkMw14TW2nN7CHgVo5oYbsG81pH4PgVVpmh+j5RSw4HFwEytdWlLjg2DYK4pqt8nD3cAG6SUSm7psZ4TRM0PRo1mF5BJXZJkWIN9plM/Absm0GOj8Jq6Aglev38GXBYN1+S178PUTypH3PsU5PVE7XsEpAE7gfNb+3pE0TVF8/s0mLqk8mhgr/uzosXvU1gvtpUv0BXANxjZ81z3trnAXPfvCnjC/fgmYExTx0bCT2uvCaP3wAb3z5You6Y+GN9gjgNl7t+7Rer71NrrifL3aDFwDFjv/lnb1LGR8NPaa4ry9+l+d5nXA58DF7T2fZKpK4QQQgDRl0MQQgjRRiQgCCGEACQgCCGEcJOAIIQQApCAIIQQwk0CghBCCEACghBCCLf/Dyhib7Gc07alAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x_norm.detach().numpy(), y_norm.detach().numpy())\n",
    "plt.scatter(x_test_torch.detach().numpy(), y0_test_torch.detach().numpy(), color='red', label='Model 0')\n",
    "plt.scatter(x_test_torch.detach().numpy(), y1_test_torch.detach().numpy(), color='orange', label='Model 1')\n",
    "plt.scatter(x_test_torch.detach().numpy(), y2_test_torch.detach().numpy(), color='green', label='Model 2')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67c1e49a",
   "metadata": {},
   "source": [
    "<span style='font-family:\"Helvetica\"; background:yellow; font-size:1.4em' >\n",
    "     What do you think of these results, do they follow the universal approximation theorem?\n",
    "</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "059e1811",
   "metadata": {},
   "source": [
    "# 7 - Play with the notebook!\n",
    "Some ideas:\n",
    "- Change the number of epochs in `Section 5` to 5000 and re-train the models. What happens?\n",
    "- Change the random seed in the `Reproducibility` cell at the very top. How do the results change?\n",
    "- Change the optimizer in `Section 4` from `Adam` to `SGD` and re-train the models. What happens?\n",
    "- [**if time allows, takes several minutes**] Change the epochs in `Section 5` to 1000000. What happens?\n",
    "- Go back to 1000 epochs and the Adam optimizer. Change the learning rate in `Section 4` to 0.05. How do the results change? what does it tell us about our previous value?\n",
    "- Change the learning rate to 0.5. What happens now?\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 610,
   "id": "6455d58a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %reset -f"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}