{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-07T02:19:02.272939Z",
     "start_time": "2019-04-07T02:19:02.268259Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "***\n",
    "***\n",
    "\n",
    "# Introduction to Neural Network\n",
    "\n",
    "\n",
    "***\n",
    "***\n",
    "\n",
    "![image.png](img/chengjun.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl.png)\n",
    "\n",
    "Sung Kim <hunkim+ml@gmail.com> HKUST\n",
    "\n",
    "- Code: https://github.com/hunkim/PyTorchZeroToAll \n",
    "- Slides: http://bit.ly/PyTorchZeroAll \n",
    "- Videos: https://www.bilibili.com/video/av15823922/\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src = './img/neuralnetwork/synapse.jpg' width = 500 align = 'right'>\n",
    "\n",
    "**The Neuron: A Biological Information Processor**\n",
    "\n",
    "- dentrites -   the receivers\n",
    "- soma -   neuron cell body (sums input signals)\n",
    "- axon  -   the transmitter\n",
    "- synapse 突触 -   point of transmission\n",
    "\n",
    "Neuron activates after a certain threshold is met.\n",
    "\n",
    "Learning occurs via electro-chemical changes in effectiveness of synaptic junction.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<div><img src = ./img/neuralnetwork/layer.png  width=450 align = 'right'></div>\n",
    "\n",
    "**An Artificial Neuron: The Perceptron simulated on hardware or by software**. Learning occurs via changes in value of the connection weights. \n",
    "\n",
    "- input connections -   the receivers\n",
    "- node simulates neuron body\n",
    "- output connection -   the transmitter\n",
    "- **activation function** employs a threshold or bias\n",
    "- connection weights act as synaptic junctions (突触)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src = \"./img/neuralnetwork/layer.png\"  width=400 align = 'right'>\n",
    "\n",
    "Neural Networks consist of the following components\n",
    "- An **input layer**, **x**\n",
    "- An arbitrary amount of **hidden layers**\n",
    "- An **output layer**, **ŷ**\n",
    "- A set of **weights** and **biases** between each layer, **W and b**\n",
    "- A choice of **activation function** for each hidden layer, **σ**. \n",
    "    - e.g., Sigmoid activation function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Each iteration of the training process consists of the following steps:\n",
    "\n",
    "1. Calculating the predicted output **ŷ**, known as `feedforward`\n",
    "1. Updating the weights and biases, known as `backpropagation`\n",
    "\n",
    "![image.png](img/dl2.png)\n",
    "\n",
    "**activation function** for each hidden layer, **σ**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl3.png)\n",
    "\n",
    "https://blog.ttro.com/artificial-intelligence-will-shape-e-learning-for-good/ \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-07T05:41:09.971531Z",
     "start_time": "2019-04-07T05:41:09.967172Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl4.png)\n",
    "\n",
    "http://playground.tensorflow.org/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl5.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Batch, Iteration, & Epoch\n",
    "\n",
    "Batch Size is the total number of training examples present in a single batch.\n",
    "\n",
    "![image.png](img/dl6.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Note: The number of batches is equal to number of iterations for one epoch. Batch size and number of batches (iterations) are two different things.\n",
    "\n",
    "\n",
    "Let’s say we have 2000 training examples that we are going to use .\n",
    "\n",
    "We can divide the dataset of 2000 examples into batches of 500 then it will take 4 iterations to complete 1 epoch.\n",
    "\n",
    "Where Batch Size is 500 and Iterations is 4, for 1 complete epoch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Gradient Descent\n",
    "\n",
    "![gradient.gif](img/gradient.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl7.png)\n",
    "\n",
    "Let's represent parameters as $\\Theta$, learning rate as $\\alpha$, and gradient as $\\bigtriangledown J(\\Theta)$, "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Mannual Gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T01:54:08.840701Z",
     "start_time": "2024-08-03T01:54:06.356078Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD7CAYAAAB+B7/XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAsRklEQVR4nO3de1jUZcI+8HtgOJ/kMICiaVlrRiSeU0tj1zMqiqQmSupmaRZpLkZo9uq2u2365ubl9nvdXRFRtFAEqRQ1dS1D0+hgJJnpegIdZjgf5/j8/rBmI9BhkJnvMNyf69prY56ZuPv6ePv1YeZ5ZEIIASIichhOUgcgIqL2xWInInIwLHYiIgfDYicicjAsdiIiB8NiJyJyMCx2IiIHI5c6AABUVNTBaLT87fSBgd4oK6u1QqK7w1yWYS7L2Ws25rJMW3M5Ocng7+9123G7KHajUbSp2H9+rT1iLsswl+XsNRtzWcYaubgUQ0TkYFjsREQOhsVORORgWlXsR48eRWxsLCZMmIA33nij2XhRURFiY2Mxbtw4rFy5Enq9vt2DEhFR65gt9mvXruH111/Hu+++i9zcXJw7dw7Hjx9v8pykpCSsXr0aBw8ehBACmZmZVgtMRNTRVZ/Kx6UVy/HZ1DhcWrEc1afy2/Xfb7bYDx8+jIkTJyI0NBQuLi7YsGED+vXrZxovLi5GY2MjIiMjAQCxsbHIy8tr15BERI6i+lQ+lOlp0JeXAUJAX14GZXpau5a72WK/cuUKDAYDFi1ahJiYGOzcuRN+fn6m8dLSUigUCtPXCoUCSqWy3QISETkS9d4sCK22yWNCq4V6b1a7fQ+z72M3GAz44osvsH37dnh6emLx4sXIzs5GbGwsAMBoNEImk/03oBBNvm6NwEBvC2P/l0Lh0+bXWhNzWYa5LGev2Zjr9hpKSm7dqbdAX1HebhnNFntQUBCGDRuGgIAAAMDo0aNx9uxZU7GHhoZCpVKZnq9WqxEcHGxRiLKy2ja9SV+h8IFKVWPx66yNuSzDXJaz12zM1TJhMKDi40Moy9kLyGRACwfXyf0DWp3RyUl2xxtis0sxUVFROHHiBKqrq2EwGPDpp58iPDzcNB4WFgY3NzcUFBQAAPbt24eRI0e2KhwRkaPTXLuGq395A+rd78Mz/GEoZs2GzNW1yXNkrq4Iip3ebt/T7B17v3798Mwzz2D27NnQ6XQYMWIEpk+fjoULFyIxMRERERFYv349Vq1ahdraWoSHhyMhIaHdAhIRdURGnQ7lH32A8gMfwdnTC10XPQ/vgYMhk8ng7OUF9d4s6CvKIfcPQFDsdPg+OrzdvrfMHg6z5lKMbTCXZew1F2C/2ZjrloaLP0K5LRXakhL4DBuO4Jmz4ezdfOmkrbnMLcXYxSZgRESOwKjRQJ2dhcojhyH390fYSy/DK+IRm+dgsRMRtYP6onNQbtsKnVoFv6jfIij2STh7eEiShcVORHQXDPV1UGW+j+oTn8AlJATdV7wKz9/0kTQTi52IqI1qvyqAcsd2GGqq4T8hGoGTY+D0q3e8SIHFTkRkIX1VFUp37UDtF2fg1qMHwl5cCvdevaSOZcJiJyJqJSEEak7lo/S9nRAaDQKnTUfAuAmQye2rSu0rDRGRndKVlUG5fRvqC8/Cvff9CHl6Ady6dZM6VotY7EREdyCMRlQdPwbVnt0ABBRPxaNL1O8gc7Lfc4pY7EREt6G9eQPKbVvRcOEHeD4UjpCEeXAJUph/ocRY7EREvyIMBlQcPICy3BzIXF0RMv/38B3+mMU710qFxU5E9AuNV69AmZYKzdUr8B4wEMHxcyH36yJ1LIuw2ImIABh1WpR/kIvyvP1w9vZG18VL4DNwsNSx2oTFTkSdXsOPF6BMS4X25g34Dh8BxYynWty0q6NgsRNRp2VsbIR67x5UHjsCuX8AwpYuh9fDEVLHumssdiLqlOq+K4QyfSv05eXoEvVbBMXGwcldmk272huLnYg6FUNtLVSZ76E6/wRcQkPRY0UKPB54QOpY7apVxT537lyUl5dD/tPHZteuXYt+/fqZxjdt2oSsrCz4+voCAGbMmIH4+HgrxCUiaruagjMozdgOQ20tAiZOQsDkKXBykX7TrvZmttiFELh8+TKOHTtmKvZfKywsxNtvv43+/fu3e0AiorulrahAyf/7P9QWfAG3e3oibOlyuN/TU+pYVmO22C9dugQAWLBgASorKzFjxgzMmTOnyXMKCwuxefNmFBcXY/DgwXjllVfg5uZmncRERK0khEB1/glc2v0+DI2NCIqNg//Y8Xa3aVd7M3vm6VdffYVdu3bhtddeg06nQ0JCAl599VWMGDECAFBXV4elS5ciOTkZPXv2RHJyMsLCwrBs2TKb/AcQEbWkUVmKi+/+Hyq//ga+D/VF7yWL4dk9TOpYNmHxYdZpaWkoKSlBSkpKi+Pnzp1DSkoKcnJyWv3v5GHWtsFclrHXXID9ZrOHXMJoROWxI1Dv3QNABsX0ONz/ZAzUZXWS5mqJZIdZf/HFF9DpdBg2bBiAW3+1+eVae0lJCfLz8xEXF9fiOBGRrWhKSqDclorGiz/C8+EIhMx9Gi6BQXa9E6M1mP2vrampwVtvvQWNRoPa2lpkZ2djzJgxpnF3d3esW7cO165dgxACGRkZTcaJiKxN6PUo++gDXF27GtqbNxD6+4UIe+lluAQGSR1NEmZvraOiovDNN99g6tSpMBqNmD17Nvr374+FCxciMTERERERWLt2LRYvXgydTocBAwZg/vz5tshORITGK5dvbdp17Sq8Bw1G8FNzIPfzkzqWpCxeY7cGrrHbBnNZxl5zAfabzZa5jFotyj7Yh4qDB+Ds44Pg+AT4DBgoeS5LSLbGTkRkb+p/OA/ltq3QKW/C97HHoXhyFpy9vKSOZTdY7ETUYRgbG6DK2o2qY0chDwpC2MtJ8HooXOpYdofFTkQdQt23Z6HcngZ9RQW6jB6LoKmxcHJ3lzqWXWKxE5FdM9TWovT9nag5mQ/Xrt3QI3klPHrfL3Usu8ZiJyK7JIRAbcEZlGbsgKG+DgGTpiAgejKcXFykjmb3WOxEZHf0lZVQZqSj7qsv4dazF7q//Ae49bhH6lgdBoudiOyGEALVn30K1fu7IPR6BMXNgP+YcZA5O0sdrUNhsRORXdCqSlGanob6onPw+E0fhCTMh2toqNSxOiQWOxFJShiNqDxyGOrsLMicnBA8JwF+I5/odPu7tCcWOxFJRlNSDGVaKhovXYRXxCMInvs0XAICpY7V4bHYicjmhF6P8gMfofyjDyBzd0foM8/CZ+gwyGQyqaM5BBY7EdlU4+X/4GZaKrTXr8FnyFAoZsVD/tN5ydQ+WOxEZBNGjQZluTmoOJQHZz8/dHvhJXhH8pxka2CxE5HV1Z///tamXaVK+I0chaC4GXD25KZd1sJiJyKrMTQ0QL0nE1XHj8FFoUD35Svg2fchqWM5PBY7EVlF7dmvUbo9HfrKCviPGYfAqbFwcnOTOlan0Kpinzt3LsrLy01nma5duxb9+vUzjRcVFWHlypWoq6vDoEGDsGbNGp57StRJGWpqUPreTtR8fhKu3cLQY/ESeNzXW+pYnYrZ9hVC4PLlyzh27NhtyzopKQlvvPEGIiMjkZKSgszMTMyePbvdwxKR/RJCQPXJCVze/C8YGuoRMDkGgdGTIeNNns2Z/WjXpUuXAAALFizAlClTsGPHjibjxcXFaGxsRGRkJAAgNjYWeXl57Z+UiOyWrqICJZvewQ//uwHyoCD0fO1/EBQzjaUuEbNXvbq6GsOGDcNrr70GnU6HhIQE3HvvvRgxYgQAoLS0FAqFwvR8hUIBpVJpUYg7nd1njkLh0+bXWhNzWYa5LGcP2YQQUB7+GFe3pkPo9eg1/2l0mxxtl5t22cP1aok1cpkt9v79+6N///++1zQuLg7Hjx83FbvRaGzyaTEhhMWfHuNh1rbBXJax11yAfWTTlpZCmb4VDd8XwaPPgwhJmI+wh++XPFdL7OF6tUSyw6y/+OIL6HQ6DBs2DMCt4v7lWntoaChUKpXpa7VajeDgYIuDElHHIIxGVH58COqcvZA5OyM4YR78Hh/F7QDsiNk19pqaGrz11lvQaDSora1FdnY2xowZYxoPCwuDm5sbCgoKAAD79u3DyJEjrZeYiCSjKb6Oa395A6rM9+DZ9yH0XPtndBn5BEvdzpi9Y4+KisI333yDqVOnwmg0Yvbs2ejfvz8WLlyIxMREREREYP369Vi1ahVqa2sRHh6OhIQEW2QnIhsRej3KPvoA5fs/hLOHJ0KfXQSfwUNZ6HZKJoSwfHG7nXGN3TaYyzL2mguwbbaGS5egTNsCbUkxfIY+iuBZ8XD2afkHfvZ6zRwt112vsRNR52TUaFCWsxcVHx+CvEsXdHtxKbz7RUodi1qBxU5EzdR/XwTltlToVCr4jYpC0PQn4ezpKXUsaiUWOxGZGOrrod7zPqo+OQ6X4BB0T0qGZ58HpY5FFmKxExEAoPbrr6DcsQ2Gqir4j5uAwClTuWlXB8ViJ+rk9NXVUO3agZozp+Ea1h1hL7wE9173Sh2L7gKLnaiTEkKg5vOTKH1vJ4wNDQiMmYaACdHc38UB8FeQqBPSlZehdEc66s5+A/f77kPI07+HW1iY1LGonbDYiToRYTSi6pN/Q70nE8JohGLmU+jyuzGQOZn9EDp1ICx2ok5Cq7wJ5bataPjhPDz7PoTghHlwVXBfJ0fEYidycMJgQMXhgyjblw2ZXI6QeQvgO+JxbgfgwFjsRA5Mc+0qbqalQnPlMrwi+yNkTgLkXfyljkVWxmInckBGnQ7lH+Wi/MB+OHt6oeui5+E9cDDv0jsJFjuRg2m4+COUaanQ3iiBz7DhCJ45G87ebT+ljDoeFjuRgzBqNFBn70HlkY8h9/dH2EsvwyviEaljkQRY7EQOoO7cd1Cmb4VerYZf1G8RFPsknD08pI5FEml1sf/1r39FRUUF3nzzzSaPb9q0CVlZWfD19QUAzJgxA/Hx8e2bkohaZKirg2r3e6g+8SlcQkLQfcWr8PxNH6ljkcRaVewnT55EdnY2nnjiiWZjhYWFePvtt5sceE1E1lfzZQFKM9JhqKmB/4RoBE6OgZOrq9SxyA6YLfbKykps2LABixYtwvfff99svLCwEJs3b0ZxcTEGDx6MV155BW7cEY7IavRVVfh+62aUfXYSbj16IOzFZXDv1UvqWGRHzH6OePXq1Vi2bJlpqeWX6urq0LdvXyQlJSE7OxvV1dV49913rRKUqLMTQqA6/zNcfi0F5Z+fQeC06bhn5essdWrmjmee7t69Gz/++CNeffVV7N27F6dPn262xv5L586dQ0pKCnJycqyRlajT0qhU+PHdzaj88iv49OmD+198Hp49uksdi+zUHZdi9u/fD5VKhZiYGFRVVaG+vh5//vOfkZKSAgAoKSlBfn4+4uLiANy6o5C3YctPHmZtG8xlGXvIJYxGVP37KFRZewAIKJ6KR5eo38EzxE/ybC2xh2vWEkfLdVeHWW/dutX0zz/fsf9c6gDg7u6OdevWYejQoejevTsyMjIwZswYi0MSUXPamzdubdp14Qd4PhSOkIR5cAlSSB2LOoA2vY994cKFSExMREREBNauXYvFixdDp9NhwIABmD9/fntnJOpUhF6PikN5KMvNgczVFSHzfw/f4Y9xOwBqtTuusdsKl2Jsg7ksI0WuxqtXoExLhebqFXgPGIjg+LmQ+3Wxi2ytwVyWkWQphohsw6jTovyDXJTn7Yeztze6Ll4Cn4GDpY5FHRSLnUhiDRcu4Oa2LdDdvAnf4SOgmPEUN+2iu8JiJ5KIsbER6r17UHnsCOQBAQhbuhxeD0dIHYscAIudSAJ1hd9CmZ4GfUU5ukT9DkGxcXByd5c6FjkIFjuRDRlqa6HK3IXq/M/gEhqKHitS4PHAA1LHIgfDYieykZqCMyjN2A5DbS0CJk5CwOQpcHLhpl3U/ljsRFamr6xE6c7tqP2yAG739ETY0uVwv6en1LHIgbHYiazk1qZdJ6B6fxeEVoug2Dj4jx0PWRu23SCyBGcYkRXo1Coo09NQf+47eDzwG4Q8PR+uoV2ljkWdBIudqB0JoxGVR49Anb0HgAzB8XPhNyoKMiezO2QTtRsWO1E70ZSUQLktFY0Xf4TnwxEImfs0XAKDpI5FnRCLneguCb0e5Xn7Uf5hLmRubgj9/UL4PDqcm3aRZFjsRHeh8cplKNO2QHPtGrwHDUbwU3Mg9/OTOhZ1cix2ojYwarUoy81BxaE8OPv4oOvzL8JnwECpYxEBYLETWaz+h/NQbkuFTqmE72MjoXhyJpy9vKSORWTCYidqJUNDA9R7d6Pq2FHIg4IQ9nISvB4KlzoWUTOtLva//vWvqKioaHaYdVFREVauXIm6ujoMGjQIa9asadO5p0T2rO7bs1BuT4O+ogJdRo9F0LTpcHJzkzoWUYta1cAnT55EdnY2nnjiiWZjSUlJeOONNxAZGYmUlBRkZmZi9uzZ7Z2TyGaqT+VDvTcLP1SUQ96lC5wDg6D58QJcu3ZDj+SV8Oh9v9QRie7I7KcmKisrsWHDBixatKjZWHFxMRobGxEZGQkAiI2NRV5eXruHJLKV6lP5t7bTLS8DhIC+ogKaHy/AK3IA7lm9hqVOHYLZYl+9ejWWLVsGX1/fZmOlpaVQKP57arpCoYBSqWzfhEQ2pN6bBaHVNntcc/UKnFxcJEhEZLk7LsXs3r0bXbt2xbBhw7B3795m40ajscmHMIQQbfpQxp0OZTVHofBp82utibksYw+5hBD4obysxTF9RbldZPwle8vzM+ayjDVy3bHY9+/fD5VKhZiYGFRVVaG+vh5//vOfkZKSAgAIDQ2FSqUyPV+tViM4ONjiEGVltTAahcWvc7STx62NuW5PqyqFctvW247L/QMkz/hL9nDNWsJclmlrLicn2R1viO9Y7Fu3/nei7927F6dPnzaVOgCEhYXBzc0NBQUFGDhwIPbt24eRI0daHJJIKsJoROWRw1BnZ0Hm5ASfEY+h9szpJssxMldXBMVOlzAlkWXa9L7EhQsXIjExEREREVi/fj1WrVqF2tpahIeHIyEhob0zElmFprgYym1b0HjpErwiHkHw3KfhEhCI6r4PQb03C/qKcsj9AxAUOx2+jw6XOi5Rq8mEEJavgbQzLsXYBnPdIvR6lB/4CGUf5sLJwwPBs2bDZ+iwZj8fstfrBdhvNuayjCRLMUSOpvE/l3AzLRXa4uvwGTIUilnxkLfwji+ijozFTp2CUaNBWW42Kg4dhLOfH7q98BK8I/tLHYvIKljs5PDqvy+CMj0NulIl/EaOQlDcTDh7ekodi8hqWOzksAz19VBnZaLq+L/holCg+/IV8Oz7kNSxiKyOxU4Oqfbs1yjdvg36ykr4jxmHwKmx3LSLOg0WOzkUfU01VO/tRM3np+DaLQw9Fr8Aj/t6Sx2LyKZY7OQQhBCoOf05VLsyYGioR8DkGARGT4aMW0hTJ8RZTx2errwcpTu2oe7sN3DrdS+6z1sAt+49pI5FJBkWO3VYwmhE1aefQL3nfQiDAYoZs9Bl9FjInMxuWkrk0Fjs1CFplUoo07ei4fz38HiwL0IS5sO1DRvQETkiFjt1KMJoRMXhgyjblw2ZszOCE+bB7/FRbdoumshRsdipw9AUX8fNrVugufwfePWLRPCcp+Hi7y91LCK7w2Inu2fU6VC+/0OU7/8Qzh6eCH12EXwGD+VdOtFtsNjJrjVcughlWiq0JcXwGToMwbNmw9nHPk/CIbIXLHayS0aNBuqcvaj8+BDkXfzRLXEpvB+JlDoWUYfAYie7U190Dsr0rdCpVPAbFYWguBlw9vCQOhZRh9GqYn/nnXdw8OBByGQyxMXFYf78+U3GN23ahKysLPj+tK/1jBkzEB8f3/5pyaEZ6uug2v0+qj/9BC7BIeielAzPPg9KHYuowzFb7KdPn8apU6eQm5sLvV6PiRMnYtSoUbjvvvtMzyksLMTbb7+N/v25vzW1Te3XX0G5YxsMVVXwHzcBgVOmctMuojYyW+xDhgxBeno65HI5lEolDAYDPH+1l3VhYSE2b96M4uJiDB48GK+88grc+JuSWkFfXY3zaf+E+sRncA3rjrAXXoJ7r3uljkXUobX6zNONGzciNTUV48ePx1/+8hfTW83q6uqwdOlSJCcno2fPnkhOTkZYWBiWLVtm1eDUsQkhoDr+Cf7zr1QYGhrRY0YcwmKnwsnFRepoRB2eRYdZNzQ0YNGiRZg4cSJmzpzZ4nPOnTuHlJQU5OTktDoED7O2DXvJpSsvQ+n2baj79izc7+uNvsteRJ1HF6ljNWMv16sl9pqNuSwj2WHWFy9ehFarRd++feHh4YGxY8fi/PnzpvGSkhLk5+cjLi4OwK07MTm3SqUWCKMRVcf/DXVWJoTRCMWs2ejy29HwDPFDnR3+piPqqMxug3f9+nWsWrUKWq0WWq0WR44cwcCBA03j7u7uWLduHa5duwYhBDIyMjBmzBirhqaOR6u8ievr/4rSjHS433sfeq55A/7ciZHIKszeWo8aNQpnz57F1KlT4ezsjLFjxyI6OhoLFy5EYmIiIiIisHbtWixevBg6nQ4DBgxo9nZI6ryEwYCKQwdRlpsNmVyOkHkL4DvicW4HQGRFFq2xWwvX2G3D1rk0167iZloqNFcuwyuyP0LmJEDepfmmXbxelrPXbMxlGcnW2IksZdTpUP5RLsoP7Iezpxe6Lnoe3gMH8y6dyEZY7NSuGi7+eGvTrhsl8B02AoqZT8HZ+/Z3FkTU/ljs1C6MjY1Q52Sh8sjHkPv7I+yll+EV8YjUsYg6JRY73bW67wqh3J4GvVoNv6jfQTE9Dk7u3LSLSCosdmozQ10dVLvfQ/WJT+ESEoLuK16F52/6SB2LqNNjsVOb1HxZgNKMdBhqauA/IRqBk2Pg5OoqdSwiAoudLKSvqkTpzh2oLfgCbj3uQVjiMrj37CV1LCL6BRY7tYoQAjUn81H63k4IrQaB06YjYNwEyLh9BJHd4e9KMktXpoYyPQ313xXCvff9CJ23AK5du0kdi4hug8VOtyWMRlT9+yhUWXsACCieikeXqN9xfxciO8dipxZpb96ActtWNFz4AZ4PhSMkYR5cghRSxyKiVmCxUxNCr0fFoTyU5eZA5uqKkPm/h+/wx7gdAFEHwmInk8arV6BMS4Xm6hV4DxyE4NlzIPfrInUsIrIQi51g1GlR/kEuyvP2w9nbG10XL4HPwMFSxyKiNmKxd3INFy7g5rYt0N28Cd/hj0ExYxY37SLq4FpV7O+88w4OHjwImUyGuLi4ZgdpFBUVYeXKlairq8OgQYOwZs0aHo9n54yNDVDv3YPKY0chDwhA2LI/wCv8YaljEVE7MPu+tdOnT+PUqVPIzc1FVlYWtm/fjkuXLjV5TlJSElavXo2DBw9CCIHMzEyrBaa7V1f4LS6vXoXKY0fRJep36LXmTyx1IgdittiHDBmC9PR0yOVylJWVwWAwwNPT0zReXFyMxsZGREZGAgBiY2ORl5dntcDUdrqaGtxM/SeK//a/kLm6oMeKFATPngMnd3epoxFRO2rVeomLiws2btyI1NRUjB8/HiEhIaax0tJSKBT/fX+zQqGAUqls/6R0V2oKzuA/uzKgq65GQPRkBEyaDCcXbtpF5IhavRCemJiIhQsXYtGiRcjMzMTMmTMBAEajscl7nIUQFr/n+U5n95mjUPi0+bXWZC+5tOUVuPSPf6Ls5Ofwuu9ePPT6Knjfd6/UsZqxl+v1a/aaC7DfbMxlGWvkMlvsFy9ehFarRd++feHh4YGxY8fi/PnzpvHQ0FCoVCrT12q1GsHBwRaF4GHW7U8IgerPTkCVuQtCq0XQ9Cfxm/gnoS6vR4OdXTN7uF4tsddcgP1mYy7LWOswa7Nr7NevX8eqVaug1Wqh1Wpx5MgRDBw40DQeFhYGNzc3FBQUAAD27duHkSNHWhyU2o9OrULxhvVQpm2BW1h39PyfPyJgQjRkzs5SRyMiGzB7xz5q1CicPXsWU6dOhbOzM8aOHYvo6GgsXLgQiYmJiIiIwPr167Fq1SrU1tYiPDwcCQkJtshOvyKMRlQePQJ19h4AMgTHz4XfqChu2kXUyciEEJavgbQzLsXcPU1JCZTbUtF48Ud4PhyBkLlPwyUwSPJcrcFclrPXbMxlGWstxfBTRB2c0OtRnrcf5R/mQubmhtDfL4TPo8O5aRdRJ8Zi78AaL1/GzbQt0F6/Bu9BQxD8VDzkfn5SxyIiibHYOyCjVouy3BxUHMqDs48Pui15Ed79B5p/IRF1Ciz2Dqb+h/NQbkuFTqmE72MjoXhyJpy9vKSORUR2hMXeQRgaGqDeuxtVx45CHhSEsJeT4PVQuNSxiMgOsdg7gLpvz0K5PQ36igp0GT0WQdOmw8nNTepYRGSnWOx2zFBTg9L3d6Lm1Em4duuGHskr4dH7fqljEZGdY7HbISEEar84g9Kd22Gor0fApCkIiJ4MJxcXqaMRUQfAYrcz+soKKHeko+7rr+DWsxe6v7wCbj16SB2LiDoQFrudEEKg+sQnUGW+B6HXIyhuBvzHjOP+LkRkMRa7HdCqSqHcthUN3xfB4zd9EPL0fLiGhEodi4g6KBa7hITRiMojh6HOzoLMyQnBcxLgN/IJbtpFRHeFxS4RTXExlNu2oPHSJXg90g/BcxLgEhAodSwicgAsdhsTej3KD3yEsg9z4eThgdCFz8FnyKPctIuI2g2L3YYa/3MJN9NSoS2+Dp8hQ6F4Kh5yH1+pYxGRg2Gx24BRo0FZbjYqDh2Es58fur3wErwj+0sdi4gcVKuKfdOmTThw4ACAWycqrVixotl4VlYWfH1v3X3OmDED8fHx7Ry1Y6r/vgjK9DToSpXwGzkKQXEz4ezpKXUsInJgZos9Pz8fJ06cQHZ2NmQyGZ555hkcPnwYY8aMMT2nsLAQb7/9Nvr3513ozwz19VBnZaLq+L/holCg+/IV8Oz7kNSxiKgTMFvsCoUCycnJcHV1BQD07t0bJSUlTZ5TWFiIzZs3o7i4GIMHD8Yrr7wCt068SVX5mS9w5e//B31lJfzHjkdgzDRu2kVENmP2DdMPPPAAIiMjAQCXL1/GgQMHMGrUKNN4XV0d+vbti6SkJGRnZ6O6uhrvvvuu1QLbM31NNW784/9Q9MZf4OTphR6vvgbFjFksdSKyqVYfZn3hwgU899xzePHFFzFt2rTbPu/cuXNISUlBTk5Oe2W0e0IIqD89gUv/TIWhvh7dn5yO7tOncdMuIpJEq354WlBQgMTERKSkpCA6OrrJWElJCfLz8xEXFwfgVsnJ5Za92aasrBZGY6v+fGnCHk4e15WXo3THNtSd/Qbu996HsOUL0D2y70+5GiXN9mv2cL1awlyWs9dszGWZtuZycpIhMND7tuNmG/jGjRtYsmQJNmzYgGHDhjUbd3d3x7p16zB06FB0794dGRkZTX6w6qiE0YiqTz+Bes/7EAYDFDNmocvosdwOgIgkZ7bYt2zZAo1GgzfffNP02KxZs3D06FEkJiYiIiICa9euxeLFi6HT6TBgwADMnz/fqqGlplUqoUzfiobz38Pjwb4ISZgP1+BgqWMREQGwYI3dmjrKUowwGFDx8SGU7cuGzNkZQU/OhN/jo5ptB+Bof+2zNuaynL1mYy7LSLYUQ7dorl/DzbRUaC7/B179IhE852m4+PtLHYuIqBkWuxlGnQ7l+z9E+f4P4ezpia7PLob34CHctIuI7BaL/Q4aLl2EMi0V2pJi+AwdhuBZs+Hs4yN1LCKiO2Kxt8Co0UCdsxeVHx+CvIs/uiUuhfcjkVLHIiJqFRb7r9QXnYMyfSt0KhX8RkUhKG4GnD08pI5FRNRqLPafGOrroNr9Pqo//QQuwSHonpQMzz4PSh2LiMhiLHYAtV99CeWOdBiqq+A/fiICp0yF00+bnhERdTSdutj11dVQ7dqBmjOn4RrWHWEvvgT3XvdKHYuI6K50ymIXQqDm1EmUvpcBodEgcGosAsZPhMzCPW6IiOxRp2syXXkZSrdvQ923Z+F+X2+EzFsAt25hUsciImo3nabYhdGIquP/hjorE8JohGLWbHT57Whu2kVEDqdTFLv25s1bm3b9cB6efcMRkjAPLgqF1LGIiKzCoYtdGAyoOHQQZbnZkMnlCJm3AL4jHud2AETk0By22DXXruLm1i3QXL0Cr/4DEBI/F/Iu3LSLiByfwxW7UadD+Ue5KD+wH86eXui66Hl4DxzMu3Qi6jQcqtgbLv54a9OuGyXwHTYCiplPwdn79nsWExE5olYV+6ZNm3DgwAEAwKhRo7BixYom40VFRVi5ciXq6uowaNAgrFmzxuJzTy1RfSof6r1Z+KGiHHL/AARMngJt8XVUHvkYcv8AhC19GV4PP2K1709EZM/MvtcvPz8fJ06cQHZ2NnJycvDdd9/h8OHDTZ6TlJSE1atX4+DBgxBCIDMz02qBq0/lQ5meBn15GSAE9OVlKN22FZUfH4bfE79Fr7VvsNSJqFMzW+wKhQLJyclwdXWFi4sLevfujZKSEtN4cXExGhsbERkZCQCIjY1FXl6e1QKr92ZBaLXNHnf29UVI/Fw4uXMnRiLq3MyulzzwwAOmf758+TIOHDiAXbt2mR4rLS2F4hfvCVcoFFAqlRaFuNPZfb/2Q0V5i48bamqgUNjPIRj2lOWXmMsy9poLsN9szGUZa+Rq9UL4hQsX8Nxzz2HFihXo1auX6XGj0djkHSdCCIvfgWLJYdZy/4BbyzAtPG4vh9U62sG51sZclrPXbMxlGWsdZt2qz9MXFBRg3rx5WL58OaZNm9ZkLDQ0FCqVyvS1Wq1GcHCwxUFbKyh2OmS/2lJX5uqKoNjpVvueREQdidliv3HjBpYsWYL169cjOjq62XhYWBjc3NxQUFAAANi3bx9GjhzZ/kl/4vvocIQkzIM8IBCQySAPCERIwjz4Pjrcat+TiKgjMbsUs2XLFmg0Grz55pumx2bNmoWjR48iMTERERERWL9+PVatWoXa2lqEh4cjISHBqqF9Hx0O30eH2+1fr4iIpCQTQrRucduKLFlj/yV7LXbmsgxzWc5eszGXZSRdYycioo6DxU5E5GBY7EREDsYuNgFzcmr7zot381prYi7LMJfl7DUbc1mmLbnMvcYufnhKRETth0sxREQOhsVORORgWOxERA6GxU5E5GBY7EREDobFTkTkYFjsREQOhsVORORgWOxERA7Grou9trYWkyZNwvXr15uNFRUVITY2FuPGjcPKlSuh1+sBACUlJYiPj8f48eOxePFi1NXV2TTXxx9/jJiYGEyZMgXPP/88qqqqAADZ2dl47LHHEBMTg5iYGGzYsMGmuTZt2oSoqCjT98/IyAAg7fUqKioy5YmJicHjjz+OSZMmAbD+9dq0aROio6MRHR2Nt956q9m4VPPLXC6p5pe5XFLNrzvlknJ+vfPOO5g4cSKio6OxdevWZuNWn1/CTn399ddi0qRJIjw8XFy7dq3ZeHR0tPjqq6+EEEK8+uqrIiMjQwghxLPPPis+/PBDIYQQmzZtEm+99ZbNctXU1IgRI0aImzdvCiGE+Nvf/ib++Mc/CiGEWLt2rfjggw/aNUtrcwkhxHPPPSe+/PLLZo9Leb1+qb6+XkRHR4szZ84IIax7vT777DMxc+ZModFohFarFQkJCeLQoUNNniPF/DKXS6r51ZrrJcX8ak2un9lyfn3++edi1qxZQqfTiYaGBhEVFSUuXrzY5DnWnl92e8eemZmJ119/vcXzU4uLi9HY2IjIyEgAQGxsLPLy8qDT6XDmzBmMGzeuyeO2yqXT6fD6668jJCQEANCnTx/cuHEDAPDtt98iOzsbkydPxh/+8AfTnZYtcgFAYWEhNm/ejMmTJ2Pt2rXQaDSSX69f2rx5MwYPHoxBgwYBsO71UigUSE5OhqurK1xcXNC7d2+UlJSYxqWaX+ZySTW/zOUCpJlfrcn1M1vOryFDhiA9PR1yuRxlZWUwGAzw9PQ0jdtiftltsf/pT38y/SL8WmlpKRQKhelrhUIBpVKJiooKeHt7Qy6XN3ncVrn8/f0xZswYAEBjYyP+8Y9/YPTo0aYszz//PHJzc9G1a1esXbvWZrnq6urQt29fJCUlITs7G9XV1Xj33Xclv14/q6mpQWZmJl544QXTY9a8Xg888IDpN9Xly5dx4MABjBo1yjQu1fwyl0uq+WUul1Tzy1yun9l6fgGAi4sLNm7ciOjoaAwbNsz0hzFgm/llt8V+J0ajETLZf7etFEJAJpOZ/v+Xfv21LdTU1ODZZ5/Fgw8+iGnTpgEA/v73v2PgwIGQyWR45pln8Omnn9osj5eXF/75z3+id+/ekMvlWLBgAY4fP2431ys3NxejR49GYGCg6TFbXK8LFy5gwYIFWLFiBXr16mV6XOr5dbtcP5Nqft0ul9Tzy9z1kmp+JSYm4uTJk7hx4wYyMzNNj9tifnXIYg8NDYVKpTJ9rVarERwcjICAANTU1MBgMAAAVCqV2SWA9lZaWorZs2ejT58++NOf/gTg1m/EtLQ003OEEHB2drZZppKSEuzZs6fJ95fL5XZxvYBbPxCcOHGi6WtbXK+CggLMmzcPy5cvN5Xjz6ScX3fKBUg3v+6US8r5Ze56AbafXxcvXkRRUREAwMPDA2PHjsX58+dN47aYXx2y2MPCwuDm5oaCggIAwL59+zBy5Ei4uLhg0KBB2L9/PwAgJycHI0eOtFkug8GARYsWYcKECVi5cqXpT1tPT0/861//wjfffAMA2LFjh+mv1Lbg7u6OdevW4dq1axBCICMjA2PGjJH8egG3flN999136N+/v+kxa1+vGzduYMmSJVi/fj2io6ObjUs1v8zlkmp+mcsl1fwylwuQZn5dv34dq1atglarhVarxZEjRzBw4EDTuE3mV5t+5GpDUVFRpndTPPPMM+Ls2bNCCCGKiorE9OnTxbhx48TLL78sNBqNEEKI69evizlz5ogJEyaIBQsWiMrKSpvlOnTokOjTp4+YMmWK6X8pKSlCCCHOnDkjpk6dKsaPHy8WLVokqqurbZZLCCHy8vJEdHS0GDt2rEhOTraL6yWEEGq1WgwfPrzZ8615vf74xz+KyMjIJr9OO3fulHx+mcsl1fxqzfWSYn61JpcU80sIITZu3CgmTJggJk2aJDZu3CiEsG1/8QQlIiIH0yGXYoiI6PZY7EREDobFTkTkYFjsREQOhsVORORgWOxERA6GxU5E5GBY7EREDub/A3z4FygT2pv1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "sns.set()\n",
    "\n",
    "x_data = [1.0, 2.0, 3.0]\n",
    "y_data = [2.0, 4.0, 6.0]\n",
    "\n",
    "plt.plot(x_data, y_data, 'r-o');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T01:55:33.358919Z",
     "start_time": "2024-08-03T01:55:33.355426Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# our model for the forward pass\n",
    "def forward(x):\n",
    "    return x * w\n",
    "\n",
    "# Loss function\n",
    "def loss(y_pred, y_val):\n",
    "    return (y_pred - y_val) ** 2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T01:55:35.573657Z",
     "start_time": "2024-08-03T01:55:35.570014Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# List of weights/Mean square Error (Mse) for each input\n",
    "w_list = []\n",
    "mse_list = []\n",
    "\n",
    "for w in np.arange(0.0, 4.1, 0.1):\n",
    "    # Print the weights and initialize the lost\n",
    "    #print(\"w=\", w)\n",
    "    l_sum = 0\n",
    "    for x_val, y_val in zip(x_data, y_data):\n",
    "        # For each input and output, calculate y_hat\n",
    "        # Compute the total loss and add to the total error\n",
    "        y_pred = forward(x_val)\n",
    "        l = loss(y_pred, y_val)\n",
    "        l_sum += l\n",
    "        #print(\"\\t\", x_val, y_val, y_pred_val, l)\n",
    "    # Now compute the Mean squared error (mse) of each\n",
    "    # Aggregate the weight/mse from this run\n",
    "    #print(\"MSE=\", l_sum / 3)\n",
    "    w_list.append(w)\n",
    "    mse_list.append(l_sum / 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T01:55:40.298861Z",
     "start_time": "2024-08-03T01:55:40.112529Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEJCAYAAACKWmBmAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA1vElEQVR4nO3deVhTV94H8G829h3Cvi8Cyqq4KxSVRQFpsbUuU7XWdqZj69SZsWN1pra+te1b7ebb2jpt387b0S5Ui6hVqqJ1ww1UFkVUZAch7IQ1JPf9w5oR0UCQ5N6E3+d5+jwkJ7f5ckB+ueeeew6PYRgGhBBCyEPw2Q5ACCGE26hQEEIIUYkKBSGEEJWoUBBCCFGJCgUhhBCVqFAQQghRiQoFIYQQlYRsB9CEpqZ2KBTq3x5ia2uGhgapBhI9Gq7mAribjXKph3KpR99y8fk8WFubPrRdLwuFQsEMqVDcPZaLuJoL4G42yqUeyqWekZSLhp4IIYSoRIWCEEKISlQoCCGEqESFghBCiEpUKAghhKhEhYIQQohKVCh+UymR4oW3j6C+uZPtKIQQopZeuQIb/3UBJy9XaeT/T4XiN6ZGItQ2tuN4bjXbUQghRC2Xb9Sj9HYbjA01c2scFYrfWJsbYlygA07l1aBXrmA7DiGEDNqJ3GrYWBgi3N9eI/9/KhT3iJvogZb2HuQVN7AdhRBCBqW+uRNXShoxLdgJAj5PI+9BheIeEYEOsDIzwAkafiKE6IgTeTUAgOkhzhp7DyoU9xAI+JgW4oT8Ww1obO1iOw4hhKgkVyhwKq8aQd62sLU00tj7UKG4z/QQZzAMcPK3Kk0IIVyVX9yIZmkPIkM1dzYBUKHoR2xljDGe1jiZV83Z1SEJIQS4cxHbwtQAob62Gn0fKhQPEBnmgsbWbhSU0EVtQgg3NbV1I7e4HtOCnSAUaPZPuUb3o5BKpViwYAE+//xzFBcX44MPPlC21dbWIjQ0FNu3b+9zTFpaGt5//33Y2t6pkI899hhWr16tyZj9hPvZwdxEhOOXqxHiY6fV9yaEkME4lVcNhgEiQ500/l4aKxS5ubn4+9//jtLSUgBAVFQUoqKiAAASiQQLFy7Ea6+91u+4goICrF27FomJiZqKNiChgI+pwU44dL4CzdJuWJkZspaFEELup2AYnMyrQaCHNeytTTT+fho7X0lNTcWGDRtgb9//BpD33nsPCxYsgKenZ7+2/Px8pKWlISkpCX/961/R0tKiqYgqRYY6Q8EwOJ1PF7UJIdxytbQR9S1diArT7EXsuzRWKDZt2oSIiIh+z5eWluL8+fNYsmTJA48Ti8X44x//iL1798LJyQkbN27UVESVHG1M4O9mhRO51VAwdFGbEMIdJy5Xw8xYhHA/sVbeT+t7Zv/www9YtGgRDAwMHtj+6aefKr9esWIFYmJi1H4PW1uzIecTi82VXydO98b7317E7eZuhI7Szg/kYe7NxTVczUa51EO51MNWrqa2Lly6UY+k6d5wdrLs166JXFovFJmZmfjqq68e2NbW1obdu3dj2bJlAACGYSAQCNR+j4YG6ZCmtorF5pBI2pSPRzmbw9RIiL0nbsLZWnM3s6ibi0u4mo1yqYdyqYfNXAfPlkGuYBDhZ9cvw1Bz8fk8lR+wtTo9trGxEV1dXXBzc3tgu4mJCb788kvk5uYCAHbs2DGkM4rhIhIKMHmMIy5el6Cto4e1HIQQAtz58Hwitxp+rpZwtjPV2vtqtVBUVlbC0dGx3/Pr169HZmYmBAIBPvroI7zxxhuYPXs2rly5gjVr1mgzYj+RYc7olTPIKrjNag5CCCkqb0ZtU6fG78S+n8aHno4ePar8OiQkBKmpqf1es2nTJuXXERERSEtL03SsQXMVm8HH2QIncqsRO94NPJ5mVmckhJCBnMithrGhEBEBmllO/GHozuxBiAx1Rk1DB25UsjNVlxBCpJ0yZBdJMHmMAwxF6l+7fRRUKAZhQqADjAwEOH6Zlh8nhLDjTMFt9MoVWh92AqhQDIqhwZ2L2heu1UHaKWM7DiFkhGEYBscuVcHLyQLuDtqflkuFYpCix7qgV67AyTw6qyCEaFdhWRNuN3ZgxlgXVt6fCsUguYrNMMrNCscuVtGd2oQQrTp2sQpmxiJMCNTuRey7qFCoYcZYF9S3dKHgFi0/TgjRjsbWO3diTw9xgkio3YvYd1GhUMPYUWJYmhrg6MUqtqMQQkaI45erwTAMHgtnZ9gJoEKhFqGAj8hQZ+QXN0DS3Ml2HEKInuuVK3A8txrBPrYQWxmzloMKhZqiwpzB4/Hw6yU6qyCEaNbF6xK0tvdgxlhXVnNQoVCTjYURwv3scDKvBrJeOdtxCCF67GhOJcRWRgjytmE1BxWKIZgx1gXSThnOF9axHYUQoqcq66S4XtmC6HBX8FleOogKxRAEeFjDydaELmoTQjTm6KUqiIR8TAvR/J7YA6FCMQQ8Hg/R4S4oqWlFSU0r23EIIXqms7sXZwpuY0KgPcyMRWzHoUIxVFOCnGAoEuAYnVUQQoZZVsFtdMvkrF/EvosKxRCZGAkxeYwDzhXW0vpPhJBhwzAMjl6shJeTObycLNiOA4AKxSOJHusKWa8Cp/Jq2I5CCNET18qbUdPQgehwbpxNAFQoHombvRn8XC3x6yVa/4kQMjyOXayEqZGQtXWdHoQKxSOKHuuCuuZOXClpZDsKIUTHNbV14+L1ekwPcYaBljcnUkWjhUIqlSIxMRGVlZUAgNdeew2xsbFITk5GcnIyDh8+3O+Y6upqLF68GPHx8XjxxRfR3t6uyYiPLMLfHhYmIrqoTQh5ZMcvV/22rpP2NydSRWOFIjc3FwsXLkRpaanyuYKCAuzYsQPp6elIT09HTExMv+PefPNNLFq0CBkZGQgKCsK2bds0FXFYCAV8RIY5I/dmPepp/SdCyBDdXdcpyNsW9tYmbMfpQ2OFIjU1FRs2bIC9/Z1xts7OTlRXV2PdunVISkrC1q1boVAo+hwjk8lw4cIFxMXFAQBSUlKQkZGhqYjD5rEwF/B4PLoBjxAyZDlFErRIe1jbnEgVjRWKTZs2ISIiQvm4vr4ekyZNwttvv43U1FRkZ2dj165dfY5pamqCmZkZhEIhAEAsFqO2tlZTEYeNjYURxvmLcTy3Gl09vWzHIYToGIZhcOhCBRysjRHsY8t2nH6E2nojNzc3fPrpp8rHzzzzDPbs2YP58+crn2MYBrz71jS5//Fg2NqaDTmnWDy0/Wjnx/jjwv+cRG5JExKneQ/5/R9mqLm0gavZKJd6KJd6hjNXYUkjSmpa8YcnguFg/2j3Tmiiv7RWKIqKilBaWqocVmIYRnnmcJeNjQ3a2togl8shEAggkUiUQ1fqaGiQQqFQf7qqWGwOiaRN7eMAwNZUBG9nC6T9ehPjR9kN6yJej5JL07iajXKph3KpZ7hzpR4pgomhECFe1o/0/x1qLj6fp/IDttamxzIMg7fffhstLS2QyWT44Ycf+l3MFolEiIiIwIEDBwAAe/bsQWRkpLYiPrLY8W6oa+pE3k3aKpUQMjj1LZ3IKapDVJgzjAy09tldLVorFAEBAXjhhRewcOFCJCQkIDAwEImJiQCA9evXIzMzEwCwYcMGpKamYs6cOcjOzsYrr7yirYiPbOwoMazNDXHoQjnbUQghOiIzpxI88DBzHHfuxL6fxsvX0aNHlV8vXrwYixcv7veaTZs2Kb92cXHBv//9b03H0gihgI9Z41zx46/FKK9tg7sDN8dWCSHc0NndixO5NYgIEMPGwojtOA9Fd2YPs8gwZxiI+DicXcF2FEIIx53Or0Fndy9ixruxHUUlKhTDzNRIhKnBTjh3tRYt0m624xBCOEqhYHAkuxI+zhbwcbZkO45KVCg0ICbCDb1yBscu0Q14hJAHyy2uR11zJ+fPJgAqFBrhaGOCUB9bHLtUBVmvnO04hBAOOnyhAjYWhhjnL2Y7yoCoUGhIzHg3tHXIcPYK9+8sJ4RoV3ltG66VN2PmOFcI+Nz/M8z9hDoq0MMarmIzHM6uAEN7VRBC7nH4QgUMRQJEhXJrldiHoUKhITweDzHjXVEpaUdhWRPbcQghHNEi7ca5wlpMC3aCiZGI7TiDQoVCgyaNdoCFiQiHLtBUWULIHUcvVkEuZzArgrs32N2PCoUGiYQCPBbugrziBtxu7GA7DiGEZbJeOY5dqkKorx0cbLi154QqVCg0LHqsK4QCHt2ARwjBmSu1kHbKdGJK7L2oUGiYpakBJo52wOn8Gkg7ZWzHIYSwRMEwOHyhAq5iMwS4W7EdRy1UKLQgfoI7emQKHM2pZDsKIYQlecUNqKpvx+yJ7kPaZ4dNVCi0wEVshjBfOxzJqUR3D92AR8hIdOBsGWwtjDA+UP09dthGhUJLZk9yh7RThpN51WxHIYRo2fWKZtysbEHcBDcIBbr3Z1f3EusoP1cr+Lla4pfz5eiVK9iOQwjRooNny2BmLMJ0HbnB7n5UKLRo9iQPNLR240JhHdtRCCFaUimRIre4AbPGucJQJGA7zpBQodCiEB9buNiZ4sC5MlrWg5AR4uDZchiI+JjB4R3sBqLRQiGVSpGYmIjKyjuzfX744QckJiYiKSkJr732Gnp6evodk5aWhmnTpiE5ORnJycn48MMPNRlRq/g8HmZPckeVpB15xbSvNiH6rqGlC+cLaxEV6gIzY91YruNBNFYocnNzsXDhQpSWlgIASkpK8NVXX+H777/H3r17oVAo8O233/Y7rqCgAGvXrkV6ejrS09OxevVqTUVkxYRAB9haGOLg2TK2oxBCNOyXC+UAgLgJunWD3f00VihSU1OxYcMG2NvfmQpmYGCADRs2wMzMDDweD6NGjUJ1df8ZQPn5+UhLS0NSUhL++te/oqWlRVMRWSEU8BE7wR3XK1tws1K/vjdCyH9IO2U4kVuNSaMdOL0f9mBorFBs2rQJERERyscuLi6YOnUqAKCxsRE7d+7EzJkz+x0nFovxxz/+EXv37oWTkxM2btyoqYisiQxxhpmxCAforIIQvZWZU4kemQLxkzzYjvLIhNp+w9raWqxYsQLz5s3DxIkT+7V/+umnyq9XrFiBmJgYtd/D1tZsyPnEYvMhH6uOudO98e2hInTIGXg4Wgz4em3lGgquZqNc6qFc6lGVq6u7F0cvVmLiGEeEBTpqMZVm+kurhaK4uBgrVqzAM888g+XLl/drb2trw+7du7Fs2TIAAMMwEAjUn07W0CCFQqH+rCKx2BwSSZvaxw3FpEB77Dp2A98dLMRziaM5k0tdXM1GudRDudQzUK7D2RVo65BhZriLVvMPtb/4fJ7KD9hamx4rlUrx3HPP4U9/+tMDiwQAmJiY4Msvv0Rubi4AYMeOHUM6o9AFZsYiRIY64+zVWjS0dLEdhxAyTHrlChw6X45RrpbwdbVkO86w0Fqh2LVrF+rr6/H1118rp75+/PHHAID169cjMzMTAoEAH330Ed544w3Mnj0bV65cwZo1a7QVUevixrsDAG1sRIgeOV9Yi4bWbszWg2sTd/EYPbzzSxeGnu76cv9VZBfVYcsfpz50njVXT78B7majXOqhXOp5WC4Fw2DDV+cBHrBx+QStrxKr80NP5MFmT7yzBHkmLUFOiM7Lu3lnKfE5Ez10bilxVahQsMxFbIZwPzscvlCBjq5etuMQQoaIYRjsPV0CO0vdXEpcFSoUHDB3qhc6unuRmUPXKgjRVfm3GlB6uw2JUzx1cilxVfTru9FRHo7mCPO1w6ELFejsprMKQnQNwzBIP1UKO0sjTAnS7n0T2kCFgiPmTvNEe1cvXasgRAcVlDSipKYVCZM99O5sAqBCwRmejhYI8bHFL+fL6ayCEB3CMAz2niqBrYUhpgY7sR1HI6hQcEjyNC+0d9259Z8QohuulDaiuLoVCZP179rEXfr5XekoLycLBHvb4pfzFejqobMKQrjuztlEKWwsDDEtRD/PJgAqFJwzd5onpJ0yHLtYxXYUQsgArpY14WZVCxIm6ee1ibv09zvTUT7OlgjytsHBc+V0VkEIh92Z6VQCa3NDTAtxZjuORlGh4KDkqV53ziou0VkFIVxVWNaEm5UtSJjsAZFQv/+U6vd3p6N8XCwxxssGGefK0d0jZzsOIeQ+d2c6WZsbYrqen00AVCg4K3mqF9o66KyCEC7KL67H9coWzJmk/2cTABUKzvJ1tcRoT2tknCujaxWEcMx3h4pgZWaAyFD9nel0LyoUHDZ3qhdaO2TIOEN7axPCFUXlTSgobsDsSR4QCdXfgVMXUaHgsFFuVgj0sMZPx26gR0bXKgjhgrsznaJC9f/axF1UKDgueZoXmtq6cZTuqyCEdVdLG3GtvBlPzvCDgWhknE0AVCg4b5SbFcYF2OPnM6Xo6JKxHYeQEYthGOw+XgxbC0PET/ZkO45WabRQSKVSJCYmorLyztpFWVlZSEpKQmxsLD788MMHHlNdXY3FixcjPj4eL774Itrb2zUZUScsmTMa7V29yDhfznYUQkasi9clKKlpQ/I07xF1NgEMslDU19cjMzMTALB582YsXboU165dU3lMbm4uFi5ciNLSUgBAV1cX1q1bh23btuHAgQMoKCjA8ePH+x335ptvYtGiRcjIyEBQUBC2bdum5rekf7xdLDFxtAMOXahAi7Sb7TiEjDhyhQI/nbgFZztTvdxvYiCDKhRr165FRUUFzpw5g5MnTyI5ORlvvfWWymNSU1OxYcMG2Nvf2RIwLy8PHh4ecHNzg1AoRFJSEjIyMvocI5PJcOHCBcTFxQEAUlJS+r1mpHp8uhfkcgb7skrZjkLIiJOVfxs1DR1IifQGn68/e2EP1qAKRXNzM5YtW4YTJ04gMTERKSkp6OzsVHnMpk2bEBERoXxcV1cHsVisfGxvb4/a2to+xzQ1NcHMzAxCoRAAIBaL+71mpHKwNkFkqDOOX65GXbPqvieEDB9Zrxx7TpXA29kC4X52bMdhhXAwL5LJZJDJZDh58iTeffdddHZ2oqOjQ603UigU4PH+U4kZhunz+GHP3f94MGxtzdQ+5i6x2HzIx2qSWGyOZXODcLrgNjLOV+Avi8exHUmJy33GRZRLPWzn2nP8JpraurHmdxGwt7dQPs92rofRRK5BFYqZM2di8uTJCAwMRFBQEBITE5GYmKjWGzk6OkIikSgfSyQS5bDUXTY2Nmhra4NcLodAIHjgawajoUEKhYJR+zix2BwSSZvax2navbliIlxx4EwZHgt1grsD+7+outBnXEK51MN2ro6uXvxw+DqCvGzgaGmozMJ2rocZai4+n6fyA/aghp5WrVqF/fv345tvvgEAbNmyBStXrlQrSGhoKEpKSlBWVga5XI79+/cjMjKyz2tEIhEiIiJw4MABAMCePXv6vWaki5/oDmNDIX46cYvtKITovV/Ol0PaKcO8KB+2o7Bq0LOerly5Ah6Ph82bN+Odd94ZcNbT/QwNDfHuu+/i5Zdfxpw5c+Dt7Y34+HgAwPr165WzqjZs2IDU1FTMmTMH2dnZeOWVV9T7jvScqZEIcyZ7IK+4AdcrmtmOQ4jeamnvwaELFRgfYA8PR/bP3tnEYxhmwDGaFStWYNq0afD398c777yDZcuW4aeffsKOHTu0kVFt+jz0BADdMjnWbj8DsZUxXls8dkjXcTSVjSsol3ooV387D1/HsYtVeOv5iXC0MeFMLlVYHXoayqwnojmGIgGSp3rhZmULcosb2I5DiN6RNHfi10tVmB7q1K9IjESDKhT3znqaMmXKkGY9keE1LcQJ9tbG2H28eEhnT4SQh9tzsgR8Pg9zp3qxHYUTBlUo7s56sra2RlBQEJ566im1Zz2R4SUU8JES6Y0qSTvOXaV7TQgZLpV1Upy9chuzxrnC2tyQ7TicMKjpsatWrcL8+fPh6Hjn1vUtW7YgICBAo8HIwCIC7OF+tgw/nbiFiADxiFkbnxBN2nW8GEaGQsye5MF2FM4Y1BmFQqHAvn378Mwzz2DhwoU4cuQIentp1zW28Xk8PB3ti4bWLhy6UMF2HEJ0Xv6tBuQVNyBpiifMjEVsx+GMQRWK999/H2fPnsXSpUvx7LPP4tKlS3jvvfc0nY0MQqCnDcL97LD/TBmaacFAQoZMrlDg+8wbsLcyxsxxrmzH4ZRBFYqTJ0/i888/x6xZsxAbG4vPPvsMJ06c0HQ2Mkjzo33R26ugm/AIeQS/XqpGTUMH5s/whUhIW/Xca1C9wTAMRKL/nIYZGBj0eUzY5WBjglkRrjidV4Oy29yb200I17V3ybDn5C0EuFuN2IX/VBlUoQgICMDbb7+N8vJyVFRU4J133sGoUaM0nY2oIWmKJ0yNRfgu8wYGcQ8lIeQee0+VoqOrFwtm+rF6AytXDapQbNiwAa2trViwYAHmz5+PhoYGLFy4UNPZiBpMjER4ItIb1yuakVMkGfgAQggAoKahHUcvVmJ6qDMnFtrkokFNjzUzM8O7777b57mxY8fi4sWLGglFhiYy1AlHL1Yi9dhNhPra0nRZQgYh9ehNiIR8PBHpzXYUzhryFRsa3uAeAZ+PBTP8UN/ShcPZlWzHIYTzCkoakPvbdFhLUwO243DWkAsFjeNx0xgvG4T52mF/Vintr02ICnKFAj9k3oTYygizItzYjsNpNAdMD82f4QtZrwJpJ2m6LCEPc+JyNarq2zE/mqbDDkTlNYrw8PAHnjkwDIOuri6NhSKPxtHGBDPHueLwhQrMGOtKF+gIuU9HlwxpJ0vg72aFsaPEbMfhPJWFYv/+/drKQYZZ0lRPZBXcxndHbuDVRQ8u+ISMVHtPl6K9U0bTYQdJZaFwcXHRVg4yzEyNRHh8uhd2HLqOnCIJIgLU33ucEH1U09COzJxKTAtxGvE71w3WoKbHDqcff/yxz854lZWVSE5Oxuuvv6587pNPPsHu3bthYWEBAJg/fz4WL16s7ag6LyrMGb9eqsZ3mTcQ5G0DIwOt/7gJ4RSGYbDj0HUYiARIoemwg6b1vxxPPfUUnnrqKQDAjRs3sHLlSrz00kt9XlNQUIAPPvgA4eHh2o6nVwR8PpbE++Ptf+cg/VQJnp7hx3YkQlh17motCsua8EzsKFia0V4Tg8Xqpf433ngDq1evho2NTZ/nCwoKsH37diQlJWHjxo3o7qZpnkPl62KJyFBnHL5QifJaWgeKjFztXTJ8n3kDXk7miAqjYXV1sFYosrKy0NXVhdmzZ/d5vr29HYGBgVizZg3S0tLQ2tqKbdu2sZRSPzz5mA9MjIT496EiKOhGSTJC/XT8Fto6ZVgSFwA+ny5gq4PHsHSL9apVqxAbGzvglqpXr17FunXrsGfPHu0E01NHs8vx4XeX8NJToYib5Ml2HEK06np5E/669QSSpnnj+ceD2Y6jc1i5utnT04MLFy70Wz8KAKqrq5GVlYUnn3wSwJ2LT0KhejEbGqRQKNSvf2KxOSQS7g3PDEeuIHcr+LtZ4et9V+DraA6LYVquQJ/7TBMol3qGI5dcocDH31+EpakB4iJch+X71Lf+4vN5sLU1e3j7o4QaqqKiInh6esLExKRfm5GRETZv3oyKigowDIOdO3ciJiaGhZT6hcfj4Zk4f3T1yJF67CbbcQjRmqM5VSivlWLhrFEwNqSZf0PBSqGoqKiAo6Njn+eef/555Ofnw8bGBhs3bsSLL76I+Ph4MAyDZ599lo2YesfZzhTxE92RVXAb18qa2I5DiMY1tXUj7eQtBHnbIMKf7sAeKlbK65w5czBnzpw+z33xxRfKr+Pi4hAXF6ftWCNC4hRPnLtai38fKsKbyydAKKA1boj++i7zBuQKBr+LGUV3YD8C+isxwhiKBPhd7CjUNHQg41w523EI0Zj8Ww3IvlaHxMkesLfuP8xNBo8KxQgU4mOHcf5i7MsqRV1zJ9txCBl2PTI5dhwqgqONCeInerAdR+dRoRihFs70A5/Pw85D12kTKqJ39p8pg6S5C8/EjqIlxIcB9eAIZWNhhCemeyP/VgPOXqllOw4hw6a8tg0Hz5Zh8hgHBHraDHwAGRAVihFs1jhX+LpY4tsj19FMu+ERPdArV+CrnwthaizCwlmj2I6jN6hQjGB8Pg/LEwLR06vANxlFNARFdN7+rFJU1EmxNM4fZsYituPoDSoUI5yjjQnmRXrj8s16ZBXcZjsOIUNWdrsNP5+5M+QUTrvWDSsqFASzItzg62qJ747cQFMbDUER3XN3yMmMhpw0ggoFAZ/Pw3NzAtErV+D/Mq7REBTROftOl6JSIsXS+AAactIAKhQEAOBgY4J5UT7IK27A6XwagiK64+6Q05QgR4T52bEdRy9RoSBKMyNcMcrVEt9l3kBjaxfbcQgZkKxXgS9/vgoLUxEWzqIdHDWFCgVR4vN4eDYhEHKFAv+iISiiA/ZllaBK0o6l8QEwNaIhJ02hQkH6cLA2wZNRPii41YhTeTVsxyHkoUpqWnHgTDmmBjsi1JeGnDSJCgXpZ8Y4V/i7WeH7ozQERbhJ1qvA//5cCEszAyycSUNOmkaFgvRzdwhKoQC++rlwSLsFEqJJaSduoar+zpCTCQ05aRwVCvJA9lbGWDjLD4VlTTh4roztOIQo5d9qQMb5ckSHuyDEx5btOCMCFQryUNNDnDAh0B5pJ0pws7KF7TiEoFnajS/3X4Wr2BRPz/BlO86IQYWCPBSPx8OSuADYWBhi+94CtHfJ2I5ERjCFgsEX+66iWybHH5KDYCASsB1pxGClUDzzzDNISEhAcnIykpOTkZub26e9sLAQKSkpiIuLw/r169Hb28tGTALAxEiIPyQHoVnag38dpCmzhD0HzpahsKwJi2eNgrOdKdtxRhStFwqGYVBaWor09HTlf6GhoX1es2bNGrz++uv45ZdfwDAMUlNTtR2T3MPb2QIpUd7IKZLg18vVbMchI9DNyhbsOVmCCYH2mBbixHacEUfrheLWrVsAgOXLl2Pu3LnYsWNHn/aqqip0dXUhLCwMAJCSkoKMjAxtxyT3iZvgjiAvG3x35AYq66RsxyEjSHuXDNv3FsDW0hBL4gLA4/HYjjTiCLX9hq2trZg8eTL+8Y9/QCaTYcmSJfDy8sLUqVMBAHV1dRCL/7NEsFgsRm2tejuw2dqaDTmfWGw+5GM1iQu5/rZ0Ala9fwxf/HwVH/wpCkaGd359uJDtQSiXeriYi2EYfJt5E83SHrz38nR4uFmzHUmJi/0FaCaX1gtFeHg4wsPDlY+ffPJJHD9+XFkoFApFn08MDMOo/QmioUE6pLn/YrE5JJI2tY/TNC7lWp4QiA++v4z/+eEils0O5FS2e1Eu9XA114Ub9TiTX4P50b6wNhZyJiNX+2uoufh8nsoP2FofesrOzsaZM2eUjxmGgVD4n3rl6OgIiUSifFxfXw97e3utZiQPN8bTBnMme+BEbg3OF9Je20RzKuqk+DK9AEHeNoid4MZ2nBFN64Wira0N7733Hrq7uyGVSpGWloaYmBhlu4uLCwwNDZGTkwMASE9PR2RkpLZjEhWSp3nBx8UC/5dxDdX1dL2CDL/O7l58nl4AM2MRViSMBp+uS7BK64UiOjoaUVFRePzxxzFv3jzMmzcP4eHheP7555Gfnw8A2LJlC9555x3Ex8ejo6MDS5Ys0XZMooJQwMfv544Bn8fDW/97Hp3dNH2ZDB8Fw+Crnwtxu7EDf1k8DhamBmxHGvF4jB5OjKdrFNpxtbQRH6TmItTHFitTgjn1qY+rfUa5BpZ+qgTpp0qwYKYfFs8ZzZlc9+JSf91Lb65REP0x2tMGzyWNwaUb9dh7qoTtOEQPXLwuQfqpEkwNckRMhCvbcchvqFCQR5I03RtTgx2x93Qpcorq2I5DdFiVRIov9l+Fl5M5lsT70/0SHEKFgjySO+tB+cPLyQJf7i+km/HIkEg7Zfif3fkwEgnwUkoIREJax4lLqFCQRyYSCvBSSjCMDAX4n5/yIO2kxQPJ4MkVCmxPL0BjWxdWpgTD2tyQ7UjkPlQoyLCwNjfES08Eo6mtG5+nF0CuULAdieiIXb8W40ppE34X6w9fF0u245AHoEJBho2PiyWeifXH1dIm/HismO04RAdkFdTgl/MVmDHWBZGhzmzHIQ+h9SU8iH6bHuqM8jopDl2ogJu9GaYG00qf5MFKalrxr4NF8HezwgLa95rT6IyCDLunZ/giwN0K/zp4DVdLG9mOQziorrkTH+/Kg6WpAV58IghCAf0p4jL66ZBhJxTwsTIlGI42Jvjkp3yU3ebejUmEPS3tPfjg+8uQyxVYPT8UFiZ05zXXUaEgGmFqJMKfnw6DqZEQH6ZeRl1TB9uRCAd0dvfio9RcNEu78aenQmmnOh1BhYJojLW5If78dBjkCgYf/JCLlvYetiMRFvXKFfg0LR8VdVK8+HgQzXDSIVQoiEY52ZrilfmhaG7vxoepl2kBwRFKwTD4cv9VXC1twrNzAhDqa8d2JKIGKhRE43ycLfHHx4NRWdeOT37Kh6yX7rEYSRiGwfdHbuB8YR2eesyHZsLpICoURCtCfGzx7JwAFJY14cv9V6HQv0WLyUMcOFuGIzmViB3vhviJ7mzHIUNA91EQrZka7IS2DhlSj92EhakBFs3yo4Xf9NzJvGrsPn4Lk0Y7YP4MX/p56ygqFESr4ie6o6W9G7+cr4CJoRCPT/eiPx56KvtaHf7vYBHGeNlgeUIgp/YrIeqhQkG07qloX3R09WJfVinkCgbzorypWOiZs1du44v9V+HrYomVdEOdzmOlUHzyySc4ePAgACAqKgqvvvpqv/bdu3fDwsICADB//nwsXrxY6zmJZvB5PCydHQChgI8DZ8vQK1fgaRqW0Bun82vwvz8Xwt/dCqueDIGRAX0e1XVa/wlmZWXh1KlTSEtLA4/Hw4oVK3D48GHExMQoX1NQUIAPPvgA4eHh2o5HtITP4+F3saMgEPBw6EIFeuUKLIoZRcMTOu745Sp8k1GE0Z7WeGleCAxFtK+EPtB6oRCLxVi7di0MDO7ctu/j44Pq6uo+rykoKMD27dtRVVWF8ePH429/+xsMDWmNen3D4/GwcKYfhAI+Ms6Vo1euwJL4ACoWOiozpxI7D19HiI8tVj4RRJsP6RGtDxz6+fkhLCwMAFBaWoqDBw8iKipK2d7e3o7AwECsWbMGaWlpaG1txbZt27Qdk2gJj8fDU4/5IHGKJ07k1uDrnwuhUNDUWV3zy/ly7Dx8HeF+dlj5RDAVCT3DYxh2JrTfuHEDv//97/Hyyy/jiSeeeOjrrl69inXr1mHPnj3aC0dY8f3hIuzMuIbIcBf8eeFYCOgCqE74MfM6vjlQiKmhzvjr4nF04VoPsXKVKScnB6tWrcK6deuQkJDQp626uhpZWVl48sknAdy5q1MoVC9mQ4N0SJ9KxWJzSCTcW+mUq7mA4c02M8wZ3V0y7Pq1GO3tPXhh7ughfzLlap/pUy6GYZB+qgR7T5di0mgHLIsbhabGdtZzaYO+5eLzebC1NXt4+6OEGoqamhqsXLkSW7Zs6VckAMDIyAibN29GRUUFGIbBzp07+1zoJvptziQPLJjph5zrEmz+7jJaaSFBTpL1KvDl/kLsPV2KacFOWJE4GgI+nUnoK62fUXz11Vfo7u7Gu+++q3xuwYIFOHr0KFatWoXg4GBs3LgRL774ImQyGcaOHYtnn31W2zEJi2LHu8HG3BBf7L+Kt77JxqonQ+AqfvinHaJdrR09+OSnfNysbMHj072QNMWTpjbrOdauUWgSDT1pjyazldS0YuuuPHTL5Hjx8SAEe9tyItej0PVcVfXt+PjHO0vGP5cQiAmBDpzIpW36lotzQ0+EDJaXkwX+sTQCYitjfPRjLjJzKtmONKIVlDTg7X9no6dXgVcXhWu8SBDuoEJBOM3Gwgiv/W4sQn3ssPPwdew4VAS5gpYp17ajFyvxUWoebC2M8Y8lEfBxpk2HRhK6t55wnpGBEC+lBGPXr8XIOF+O2qZOvJgcBBMj+vXVNLlCge8zbyIzpxKhPrZ4Ye4YGBtSv480dEZBdAKfz8P8Gb5YNjsA18qasPFfF3CrupXtWHqtoaULm7+7jMzf9pJ4eV4IFYkRin7qRKdEhjrD0cYE/9x3Be/syEHyNC/MmeQBPp9m3Qyn84W1+CajCHKGwXMJgbQr3QhHhYLonFFuVnhz+QR8k1GEn07cwpWSRjyfNBo2FkZsR9N5nd29+PbIdZzOvw0vJwu8MHc0HKxN2I5FWEaFgugkUyMR/pA8BsHetth5+Dpe/+o8ls4OwPgAe7aj6axb1a34594rkDR3InGKB+ZO9aLlOAgAKhREh/F4PEwLcYKfmyX+ufcKPttTgPxgJyyK8WM7mk5RKBikHrmOb3+5BkszA7y6KBz+7tZsxyIcQoWC6DwHaxO89rtx2Hu6BD9nleF6ZTNWPR0OZysaihpIdX07vvmlCNcrmjE+wB5L4v1haiRiOxbhGCoURC8IBXykRPpgjKcNvvq5EH//PAvj/MV4OtoXdlbGbMfjnI4uGdJPleLoxUoYiAR4ZUE4gj2saCkO8kBUKIhe8Xe3xlsrJuLUlVqkHrmOvOIGzJ7ojtmTPGi3NQAKhsGpvBrsPl4MaYcMkWHOeCLSGz4etpxckoJwAxUKoncMRAI8HeOPMG8bpB67ib2nS3EqvwZPz/BDhL94xH5qvlnVgp2Hr6Psdht8XS3x5/mj4OFoznYsogOoUBC9ZWNhhD8kByE6vAnfHrmBz/YUIMDdCgtm+sHdYeT8gWxq68auX4tx5sptWJsb4oWk0Zg42mHEFkyiPioURO/5u1tjw7LxOJ5bjbQTt/DG1xcQ4mOLuAnuCHDX33H5KokUv5yvwJkrt8HjAQmTPZAw2QNGBvTPnqiHfmPIiMDn8xAd7oIJgfY4mlOJzJxKbP7uEjwczRE/wR0RAWK92HiHYRhcK29Gxrly5N9qgIGIj8fCXBA7wQ1iuqhPhogKBRlRTI1ESJrqhfiJ7sgquI2M8xXYvvcKdv1qhNjxbpge6qSTn7jlCgWyr0mQca4cZbVtsDAR4YnpXoge6wozY5ruSh6N7v2LIGQYiIQCRIW5YHqoM3Jv1iPjXDm+y7yB9FMlmDjaAeP8xfB3t+L0WQbDMCirbUNOkQRnr9SiobULjjYmWBrvjylBjkPeb5yQ+1GhICMan8dDuJ8Y4X5iFFe14HB2BU4X1ODYpSqYGYsQ5meHCH8xRnvacGI5CwXD4FZ1K3KK6pBTJEF9Sxf4PB4CPKywKMYPob524OvpNRfCHlYKxb59+/DZZ5+ht7cXS5cuxeLFi/u0FxYWYv369Whvb0dERATefPNNCIVU04hm+bhYwsfFEt0yOQpuNSCnSILsa3U4lVcDY0MBQn3tMNZPDG9nC1ibG2rtIri0U4bSmlbk3mxAzvU6NEt7IODzMMbLBklTPBHmZwdzEwOtZCEjk9b/+tbW1uLDDz/ETz/9BAMDAyxYsAATJ06Er6+v8jVr1qzBW2+9hbCwMKxbtw6pqalYtGiRtqOSEcpQJMA4f3uM87eHrFeBq6WNyCmS4NKNO0M8AGBuIoKHgzk8HM3h7mAODwcziK2MH7l4NEu7UXa7DWW1bSi73YbyWikaWrsAACIhH0FeNogIsEeojx1t3ES0Ruu/aVlZWZg0aRKsrKwAAHFxccjIyMBLL70EAKiqqkJXVxfCwsIAACkpKdi6dSsVCsIKkZCPUF87hPraoVfuj9Kae/+ItyHjXDnkCgYAYGwohL2VMUyMhDAxEsLUSAgTIxFMDO98bWQohIFhA+rqpWjv6kVHlwwd3b3Krxtbu9HS3qN8bwcbE/i4WGDGOBe4O5jDx9lCJy+0E92n9d+6uro6iMVi5WN7e3vk5eU9tF0sFqO2tlat97C1NRtyPrGYmzdicTUXwN1smsjl5GiJyfc87pHJUX67DcVVzbhZ2YL65k5IO3pQ19wJaYcM7Z0y9PT23+NbwOfBzEQEM2MRTI1FsLYwhpeL1Z3hL1creDlbwETLi/ONpJ/jcBhJubReKBQKRZ/Tc4Zh+jweqH0wGhqkUPz2KU8dYrE5J9e74WougLvZtJnL0kiAsT62GOtj+8B2Wa8cHV296OjuhbOjJbrau2Eg4qv8vW5v60J7W5emIvdDP0f16FsuPp+n8gO21qdxODo6QiKRKB9LJBLY29s/tL2+vr5POyG6RiQUwNLMEE62prCzMoahgUBv7wYn+knrhWLKlCk4c+YMGhsb0dnZiUOHDiEyMlLZ7uLiAkNDQ+Tk5AAA0tPT+7QTQgjRLq0XCgcHB6xevRpLlizB448/jsTERISEhOD5559Hfn4+AGDLli145513EB8fj46ODixZskTbMQkhhPyGxzCM+oP5HEfXKLSHq9kol3ool3r0LRfnrlEQQgjRLVQoCCGEqESFghBCiEp6eZsnnz/0qYePcqwmcTUXwN1slEs9lEs9+pRroGP08mI2IYSQ4UNDT4QQQlSiQkEIIUQlKhSEEEJUokJBCCFEJSoUhBBCVKJCQQghRCUqFIQQQlSiQkEIIUQlKhSEEEJUGpGFYt++fZgzZw5iY2Oxc+fOfu2FhYVISUlBXFwc1q9fj97eXk7k+uSTTxAdHY3k5GQkJyc/8DWaIpVKkZiYiMrKyn5tbPXXQLnY6q9PPvkECQkJSEhIwHvvvdevna3+GigXW/318ccfY86cOUhISMDXX3/dr53N36+BsrH5b/K///u/sXbt2n7Pa6S/mBHm9u3bTHR0NNPU1MS0t7czSUlJzI0bN/q8JiEhgbl06RLDMAzz2muvMTt37uRErt///vfMxYsXNZ7lfpcvX2YSExOZMWPGMBUVFf3a2eivweRio79Onz7NPP3000x3dzfT09PDLFmyhDl06FCf17DRX4PJxUZ/nTt3jlmwYAEjk8mYzs5OJjo6mikuLu7zGrZ+vwaTja1/k1lZWczEiROZv/3tb/3aNNFfI+6MIisrC5MmTYKVlRVMTEwQFxeHjIwMZXtVVRW6uroQFhYGAEhJSenTzlYuACgoKMD27duRlJSEjRs3oru7W+O5ACA1NRUbNmx44N7lbPXXQLkAdvpLLBZj7dq1MDAwgEgkgo+PD6qrq5XtbPXXQLkAdvprwoQJ+OabbyAUCtHQ0AC5XA4TExNlO5u/XwNlA9jps+bmZnz44Yf4wx/+0K9NU/014gpFXV0dxGKx8rG9vT1qa2sf2i4Wi/u0s5Wrvb0dgYGBWLNmDdLS0tDa2opt27ZpPBcAbNq0CREREQ9sY6u/BsrFVn/5+fkp/5GWlpbi4MGDiIqKUraz1V8D5WLz90skEmHr1q1ISEjA5MmT4eDgoGxj8/droGxs9dnrr7+O1atXw8LCol+bpvprxBUKhUIBHu8/S+oyDNPn8UDtbOUyNTXFF198AR8fHwiFQixfvhzHjx/XeK6BsNVfA2G7v27cuIHly5fj1Vdfhaenp/J5tvvrYbnY7q9Vq1bhzJkzqKmpQWpqqvJ5tvtLVTY2+uzHH3+Ek5MTJk+e/MB2TfXXiCsUjo6OkEgkyscSiaTP0MX97fX19Q8d2tBmrurqauzatUv5mGEYCIXsbyfCVn8NhM3+ysnJwbJly/CXv/wFTzzxRJ82NvtLVS62+qu4uBiFhYUAAGNjY8TGxqKoqEjZzmZ/DZSNjT47cOAATp8+jeTkZGzduhVHjx7F22+/rWzXVH+NuEIxZcoUnDlzBo2Njejs7MShQ4cQGRmpbHdxcYGhoSFycnIAAOnp6X3a2cplZGSEzZs3o6KiAgzDYOfOnYiJidF4roGw1V8DYau/ampqsHLlSmzZsgUJCQn92tnqr4FysdVflZWV+Pvf/46enh709PQgMzMT48aNU7az+fs1UDY2+uzrr7/G/v37kZ6ejlWrVmHGjBlYt26dsl1j/fXIl8N10N69e5mEhAQmNjaW+ec//8kwDMOsWLGCycvLYxiGYQoLC5l58+YxcXFxzJ///Gemu7ubE7kyMjKU7WvXrtVarruio6OVs4u40F8D5WKjv/7rv/6LCQsLY+bOnav879tvv2W9vwaTi63fr61btzKzZ89mEhMTma1btzIMw53fr4Gysflvcvfu3cpZT5ruL9rhjhBCiEojbuiJEEKIeqhQEEIIUYkKBSGEEJWoUBBCCFGJCgUhhBCVqFAQQghRiQoFIYQQlahQEKIhycnJOHPmDABg//79CA4ORldXFwBg/fr1+Pbbb9mMR8igUaEgRENiYmJw4sQJAMDJkydhaWmJ7OxsMAyD48ePc2IJFkIGgwoFIRpyt1AwDIPs7GwsW7YMp0+fxuXLl+Hu7t5nOWhCuIz95UcJ0VP+/v6QyWTIzMyEp6cnoqOjsXr1agiFQsTFxbEdj5BBozMKQjRo1qxZeP/99zF16lT4+PhAKpVi3759iI2NZTsaIYNGhYIQDYqJicGtW7cwZcoUAHeWkxeLxXBycmI5GSGDR6vHEkIIUYnOKAghhKhEhYIQQohKVCgIIYSoRIWCEEKISlQoCCGEqESFghBCiEpUKAghhKhEhYIQQohK/w9NU5r4ppDfbAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot it all\n",
    "plt.plot(w_list, mse_list)\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('w')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T01:57:05.575849Z",
     "start_time": "2024-08-03T01:57:05.553267Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 w= 3.48 loss= 19.68\r",
      "Epoch: 1 w= 3.09 loss= 10.76\r",
      "Epoch: 2 w= 2.81 loss= 5.88\r",
      "Epoch: 3 w= 2.6 loss= 3.21\r",
      "Epoch: 4 w= 2.44 loss= 1.76\r",
      "Epoch: 5 w= 2.33 loss= 0.96\r",
      "Epoch: 6 w= 2.24 loss= 0.52\r",
      "Epoch: 7 w= 2.18 loss= 0.29\r",
      "Epoch: 8 w= 2.13 loss= 0.16\r",
      "Epoch: 9 w= 2.1 loss= 0.09\r",
      "Epoch: 10 w= 2.07 loss= 0.05\r",
      "Epoch: 11 w= 2.05 loss= 0.03\r",
      "Epoch: 12 w= 2.04 loss= 0.01\r",
      "Epoch: 13 w= 2.03 loss= 0.01\r",
      "Epoch: 14 w= 2.02 loss= 0.0\r",
      "Epoch: 15 w= 2.02 loss= 0.0\r",
      "Epoch: 16 w= 2.01 loss= 0.0\r",
      "Epoch: 17 w= 2.01 loss= 0.0\r",
      "Epoch: 18 w= 2.01 loss= 0.0\r",
      "Epoch: 19 w= 2.0 loss= 0.0\r",
      "Epoch: 20 w= 2.0 loss= 0.0\r",
      "Epoch: 21 w= 2.0 loss= 0.0\r",
      "Epoch: 22 w= 2.0 loss= 0.0\r",
      "Epoch: 23 w= 2.0 loss= 0.0\r",
      "Epoch: 24 w= 2.0 loss= 0.0\r",
      "Epoch: 25 w= 2.0 loss= 0.0\r",
      "Epoch: 26 w= 2.0 loss= 0.0\r",
      "Epoch: 27 w= 2.0 loss= 0.0\r",
      "Epoch: 28 w= 2.0 loss= 0.0\r",
      "Epoch: 29 w= 2.0 loss= 0.0\r",
      "Epoch: 30 w= 2.0 loss= 0.0\r",
      "Epoch: 31 w= 2.0 loss= 0.0\r",
      "Epoch: 32 w= 2.0 loss= 0.0\r",
      "Epoch: 33 w= 2.0 loss= 0.0\r",
      "Epoch: 34 w= 2.0 loss= 0.0\r",
      "Epoch: 35 w= 2.0 loss= 0.0\r",
      "Epoch: 36 w= 2.0 loss= 0.0\r",
      "Epoch: 37 w= 2.0 loss= 0.0\r",
      "Epoch: 38 w= 2.0 loss= 0.0\r",
      "Epoch: 39 w= 2.0 loss= 0.0\r",
      "Epoch: 40 w= 2.0 loss= 0.0\r",
      "Epoch: 41 w= 2.0 loss= 0.0\r",
      "Epoch: 42 w= 2.0 loss= 0.0\r",
      "Epoch: 43 w= 2.0 loss= 0.0\r",
      "Epoch: 44 w= 2.0 loss= 0.0\r",
      "Epoch: 45 w= 2.0 loss= 0.0\r",
      "Epoch: 46 w= 2.0 loss= 0.0\r",
      "Epoch: 47 w= 2.0 loss= 0.0\r",
      "Epoch: 48 w= 2.0 loss= 0.0\r",
      "Epoch: 49 w= 2.0 loss= 0.0\r",
      "Epoch: 50 w= 2.0 loss= 0.0\r",
      "Epoch: 51 w= 2.0 loss= 0.0\r",
      "Epoch: 52 w= 2.0 loss= 0.0\r",
      "Epoch: 53 w= 2.0 loss= 0.0\r",
      "Epoch: 54 w= 2.0 loss= 0.0\r",
      "Epoch: 55 w= 2.0 loss= 0.0\r",
      "Epoch: 56 w= 2.0 loss= 0.0\r",
      "Epoch: 57 w= 2.0 loss= 0.0\r",
      "Epoch: 58 w= 2.0 loss= 0.0\r",
      "Epoch: 59 w= 2.0 loss= 0.0\r",
      "Epoch: 60 w= 2.0 loss= 0.0\r",
      "Epoch: 61 w= 2.0 loss= 0.0\r",
      "Epoch: 62 w= 2.0 loss= 0.0\r",
      "Epoch: 63 w= 2.0 loss= 0.0\r",
      "Epoch: 64 w= 2.0 loss= 0.0\r",
      "Epoch: 65 w= 2.0 loss= 0.0\r",
      "Epoch: 66 w= 2.0 loss= 0.0\r",
      "Epoch: 67 w= 2.0 loss= 0.0\r",
      "Epoch: 68 w= 2.0 loss= 0.0\r",
      "Epoch: 69 w= 2.0 loss= 0.0\r",
      "Epoch: 70 w= 2.0 loss= 0.0\r",
      "Epoch: 71 w= 2.0 loss= 0.0\r",
      "Epoch: 72 w= 2.0 loss= 0.0\r",
      "Epoch: 73 w= 2.0 loss= 0.0\r",
      "Epoch: 74 w= 2.0 loss= 0.0\r",
      "Epoch: 75 w= 2.0 loss= 0.0\r",
      "Epoch: 76 w= 2.0 loss= 0.0\r",
      "Epoch: 77 w= 2.0 loss= 0.0\r",
      "Epoch: 78 w= 2.0 loss= 0.0\r",
      "Epoch: 79 w= 2.0 loss= 0.0\r",
      "Epoch: 80 w= 2.0 loss= 0.0\r",
      "Epoch: 81 w= 2.0 loss= 0.0\r",
      "Epoch: 82 w= 2.0 loss= 0.0\r",
      "Epoch: 83 w= 2.0 loss= 0.0\r",
      "Epoch: 84 w= 2.0 loss= 0.0\r",
      "Epoch: 85 w= 2.0 loss= 0.0\r",
      "Epoch: 86 w= 2.0 loss= 0.0\r",
      "Epoch: 87 w= 2.0 loss= 0.0\r",
      "Epoch: 88 w= 2.0 loss= 0.0\r",
      "Epoch: 89 w= 2.0 loss= 0.0\r",
      "Epoch: 90 w= 2.0 loss= 0.0\r",
      "Epoch: 91 w= 2.0 loss= 0.0\r",
      "Epoch: 92 w= 2.0 loss= 0.0\r",
      "Epoch: 93 w= 2.0 loss= 0.0\r",
      "Epoch: 94 w= 2.0 loss= 0.0\r",
      "Epoch: 95 w= 2.0 loss= 0.0\r",
      "Epoch: 96 w= 2.0 loss= 0.0\r",
      "Epoch: 97 w= 2.0 loss= 0.0\r",
      "Epoch: 98 w= 2.0 loss= 0.0\r",
      "Epoch: 99 w= 2.0 loss= 0.0\r"
     ]
    }
   ],
   "source": [
    "# compute gradient\n",
    "def gradient(x, y):  # d_loss/d_w\n",
    "    return 2 * x * (x * w - y)\n",
    "\n",
    "# Training loop\n",
    "for epoch in range(100):\n",
    "    for x_val, y_val in zip(x_data, y_data):\n",
    "        # Compute derivative w.r.t to the learned weights\n",
    "        # Update the weights\n",
    "        # Compute the loss and print progress\n",
    "        grad = gradient(x_val, y_val)\n",
    "        w = w - 0.01 * grad\n",
    "        #print(\"\\tgrad: \", x_val, y_val, round(grad, 2))\n",
    "        y_pred = forward(x_val)\n",
    "        l = loss(y_pred, y_val)\n",
    "    print(\"Epoch:\", epoch, \"w=\", round(w, 2), \"loss=\", round(l, 2),  end='\\r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Auto Gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T01:58:16.765667Z",
     "start_time": "2024-08-03T01:58:14.531866Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "w = torch.tensor([1.0], requires_grad=True) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:02:04.835883Z",
     "start_time": "2024-08-03T02:02:04.735123Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 99 | Loss: 9.094947017729282e-13\n"
     ]
    }
   ],
   "source": [
    "# Training loop\n",
    "for epoch in range(100):\n",
    "    for x_val, y_val in zip(x_data, y_data):\n",
    "        y_pred = forward(x_val) # 1) Forward pass\n",
    "        l = loss(y_pred, y_val) # 2) Compute loss\n",
    "        l.backward() # 3) Back propagation to update weights\n",
    "        #print(\"\\tgrad: \", x_val, y_val, w.grad.item())\n",
    "        w.data = w.data - 0.01 * w.grad.item()\n",
    "        # Manually zero the gradients after updating weights\n",
    "        w.grad.data.zero_()\n",
    "\n",
    "print(f\"Epoch: {epoch} | Loss: {l.item()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:02:13.268063Z",
     "start_time": "2024-08-03T02:02:13.240284Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2.0000])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w.data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Back Propagation in Complicated network\n",
    "\n",
    "\n",
    "![image.png](img/dl8.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl9.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl10.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl11.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:11:11.483682Z",
     "start_time": "2024-08-03T02:11:11.480333Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "import torch\n",
    "from torch import tensor\n",
    "from torch import sigmoid\n",
    "\n",
    "x_data = tensor([[1.0], [2.0], [3.0]])\n",
    "y_data = tensor([[2.0], [4.0], [6.0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:11:59.347834Z",
     "start_time": "2024-08-03T02:11:59.342480Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class Model(nn.Module):\n",
    "    def __init__(self):\n",
    "        \"\"\"\n",
    "        In the constructor we instantiate two nn.Linear module\n",
    "        \"\"\"\n",
    "        super(Model, self).__init__()\n",
    "        self.linear = torch.nn.Linear(1, 1)  # One in and one out\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        In the forward function we accept a Variable of input data and we must return\n",
    "        a Variable of output data. We can use Modules defined in the constructor as\n",
    "        well as arbitrary operators on Variables.\n",
    "        \"\"\"\n",
    "        y_pred = self.linear(x)\n",
    "        return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:12:48.244056Z",
     "start_time": "2024-08-03T02:12:46.188311Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# our model\n",
    "model = Model()\n",
    "# Construct our loss function and an Optimizer. The call to model.parameters()\n",
    "# in the SGD constructor will contain the learnable parameters of the two\n",
    "# nn.Linear modules which are members of the model.\n",
    "criterion = torch.nn.MSELoss(reduction='sum')\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=0.01)\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:14:08.464646Z",
     "start_time": "2024-08-03T02:14:08.239161Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 | Loss: 89.232177734375 \n",
      "Epoch: 100 | Loss: 0.25201869010925293 \n",
      "Epoch: 200 | Loss: 0.05925912410020828 \n",
      "Epoch: 300 | Loss: 0.013934069313108921 \n",
      "Epoch: 400 | Loss: 0.0032764303032308817 \n"
     ]
    }
   ],
   "source": [
    "# Training loop\n",
    "for k, epoch in enumerate(range(500)):\n",
    "    # 1) Forward pass: Compute predicted y by passing x to the model\n",
    "    y_pred = model(x_data)\n",
    "\n",
    "    # 2) Compute and print loss\n",
    "    loss = criterion(y_pred, y_data)\n",
    "    if k%100==0:\n",
    "        print(f'Epoch: {epoch} | Loss: {loss.item()} ')\n",
    "\n",
    "    # Zero gradients, perform a backward pass, and update the weights.\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:14:48.461618Z",
     "start_time": "2024-08-03T02:14:48.455103Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction (after training) 4 7.967861175537109\n"
     ]
    }
   ],
   "source": [
    "# After training\n",
    "hour_var = tensor([[4.0]])\n",
    "y_pred = model(hour_var)\n",
    "print(\"Prediction (after training)\",  4, model(hour_var).data[0][0].item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Pytorch  Rhythm\n",
    "\n",
    "![image.png](img/dl12.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Regression\n",
    "\n",
    "Let’s start with a simple example of House Price. \n",
    "- Say you’re helping a friend who wants to buy a house.\n",
    "\n",
    "- She was quoted $400,000 for a 2000 sq ft house (185 meters). \n",
    "\n",
    "Is this a good price or not?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "So you ask your friends who have bought houses in that same neighborhoods, and you end up with three data points:\n",
    "\n",
    "\n",
    "\n",
    "| Area (sq ft) (x)  |  Price (y)   |  \n",
    "| -------------|:-------------:|\n",
    "|2,104|399,900|\n",
    "|1,600|329,900|\n",
    "|2,400|369,000|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "$$y = f(X) = W X$$\n",
    "\n",
    "- Calculating the prediction is simple multiplication.\n",
    "- But before that, we need to think about the weight we’ll be multiplying by. \n",
    "- “training” a neural network just means finding the weights we use to calculate the prediction.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A simple predictive model (“regression model”)\n",
    "- takes an input, \n",
    "- does a calculation, \n",
    "- and gives an output \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src= 'img/neuralnetwork/data_points_graph_animated.gif' width= \"700px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-07T03:15:14.317623Z",
     "start_time": "2019-04-07T03:15:14.313438Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Model Evaluation\n",
    "- If we apply our model to the three data points we have, how good of a job would it do?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "<img src= 'img/neuralnetwork/data_points_error_animated.gif' width= \"700px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "<img src= 'img/neuralnetwork/model_evaluation.png' width= \"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Loss Function**\n",
    "\n",
    "how bad our prediction is\n",
    "\n",
    "- For each point, the error is measured by the difference between the **actual value** and the **predicted value**, raised to the power of 2. \n",
    "- This is called **Mean Square Error**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "<img src= 'img/neuralnetwork/lines_and_errors_animated.gif' width= \"700px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- We can't improve much on the model by varying the weight any more. \n",
    "- But if we add a bias (intercept) we can find values that improve the model.\n",
    "\n",
    "<img src= 'img/neuralnetwork/NNs_bias_2.png' width= \"500px\">\n",
    "\n",
    "$$y = 0.1 X + 150$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Gradient Descent**\n",
    "\n",
    "- Automatically get the correct weight and bias values \n",
    "- minimize the loss function.\n",
    "\n",
    "<img src= 'img/neuralnetwork/gd.png' width= \"700px\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Regression\n",
    "\n",
    "<img src= 'img/neuralnetwork/NNs_2_variables.png' width= \"500px\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:20:28.046509Z",
     "start_time": "2024-08-03T02:20:28.039965Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "from torch import nn, optim\n",
    "from torch.autograd import Variable\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:20:32.908403Z",
     "start_time": "2024-08-03T02:20:32.726653Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD7CAYAAACPDORaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAXnklEQVR4nO3df5DddX3v8eduAslqEn+ExYABBCLvCErDFZAaLT/80XYuM2gnKa0Z1KkEow2dq3C53LkqCXqlCA0dOsPFhiBMowWreEdMcqedqEHAoAIJVzHvCzGmDWznZtL2QiCRLGfvH9/v2mUnyZ5v2JM9fHw+/mHP93w+uy92s6/z2c/3nPPtGRoaQpJUpt6JDiBJ6hxLXpIKZslLUsEseUkqmCUvSQWbPNEBRpgCnAUMAC9OcBZJeqWYBBwD/Bj41eg7u6nkzwJ+MNEhJOkV6t3A/aMPdlPJDwD8678+R6vV/Ln7M2dOY9eu3eMe6uUyVzPmaq5bs5mrmUPN1dvbw+te92qoO3S0bir5FwFaraFDKvnhud3IXM2Yq7luzWauZl5mrv1uc3viVZIKZslLUsEseUkqWNt78hFxI3BUZn40It4LrAD6gLsz8zP1mHnAbcAM4D5gSWYOjntqSVJb2lrJR8R7gI/UH/cBtwMXAW8BzoqI36+HrgaWZuYpQA+weNwTS5LaNmbJR8Trgf8OfLE+dDbwRGZuq1fpq4GFEXEC0JeZG+txdwALxz+ypJGe2ZL8y9rvsGfrkxMdRV2one2aLwP/DTiuvn0sL30+5gAw+yDHG5k5c1rTKb/W3z/9kOd2krmaMVf7ntmS/Oyzy2gNDtI7eTKnfX4ZM+bGRMf6tW78nsFvVq6DlnxEXAr8U2auj4iP1od7gZFP5uwBWgc53siuXbsP6bmi/f3T2bnz2cbzOs1czZirmX956BFag4PQatEaHGTgoUf41cxjJzoW0L3fs9Jy9fb2HHRxPNZK/mLgmIjYBLwemAacwEufdD8LeBrYQfX+CaOPS+qQvphL7+TJtAYH6Zk0mb6YO9GR1GUOWvKZ+b7hj+uV/HnAEuCJiJgDbAM+BNyemdsjYm9EzM/MB4BLgHWdCi4J+k6ew2mfX8bAQ4/QF3PpO3nOREdSl2n8tgaZubcu/G8CU4G1wDfquxcBKyNiBvAIcPM45ZR0ADPmRtds0aj7tF3ymXkH1TNmyMz1wG/tZ8xmqmffSJK6gK94laSCWfKSVDBLXpIKZslLUsEseUkqmCUvSQWz5CWpYJa8JBXMkpekglnyklQwS16SCmbJS1LBLHlJKpglL0kFs+QlqWCWvCQVzJKXpIJZ8pJUMEtekgrW1jVeI+JaYAEwBKzKzBX1xbyvAl4EvgtckZmDETEPuA2YAdwHLMnMwQ5klySNYcyVfEScC1wAnA6cCVweEQF8AXhPZr4NOAL4s3rKamBpZp4C9ACLOxFckjS2MUs+MzcA59er8aOpVv9nAz/MzIF62HeAD0TECUBfZm6sj98BLBz31JKktrS1J5+Z+yJiOfA4sB74EXBORBwXEZOotnJmAccCAyOmDgCzxzeyJKldbe3JA2TmNRFxPXAvcC5wNfBtYA/wdarVfS/Vvv2wHqDVJNDMmdOaDH+J/v7phzy3k8zVjLma69Zs5mqmE7nGLPmImAtMzcxNmfl8RNxDVejXZ+YZ9ZiFwFZgB3DMiOmzgKebBNq1azet1tDYA0fp75/Ozp3PNp7XaeZqxlzNdWs2czVzqLl6e3sOujhuZ7vmJGBlREyJiCOBi4DvAesjYnp97HLg7szcDuyNiPn13EuAdY1TS5LGRTsnXtcCa4BHgYeBBzPzq8ByYCPwU+C7mfm1esoi4KaI2AJMA27uRHBJ0tja2pPPzGXAslHHVgGr9jN2M9V2jiRpgvmKV0kqmCUvSQWz5CWpYJa8JBXMkpekglnyklQwS16SCmbJS1LBLHlJKpglL0kFs+QlqWCWvCQVzJKXpIJZ8pJUMEtekgpmyUtSwSx5SSqYJS9JBbPkJalgbV3jNSKuBRYAQ8CqzFwREe8HbgAmAY8Al2bmCxExD7gNmAHcByzJzMFOhJckHdyYK/mIOBe4ADgdOBO4PCKC6iLef5SZbwVeBXy4nrIaWJqZpwA9wOJOBJckjW3Mks/MDcD59Wr8aKrV/3NUK/gZETEJmArsiYgTgL7M3FhPvwNY2IngkqSxtbUnn5n7ImI58DiwHngK+CTwfeBp4CjgG8CxwMCIqQPA7HHMK0lqoGdoaKjtwRHxKuBeYAPwIeA/AtuAFcARwNeAP8/Md9fj3wzcm5lz2/j0b6o/lySpuROBX44+OOaJ14iYC0zNzE2Z+XxE3EO1iv9pZm6tx6wEvg58CThmxPRZVCv9tu3atZtWq/0HnmH9/dPZufPZxvM6zVzNmKu5bs1mrmYONVdvbw8zZ0478P1tfI6TgJURMSUijgQuojq5enZEvKEecxHw48zcDuyNiPn18UuAdY1TS5LGRTsnXtcCa4BHgYeBBzPzOuCzwPci4jGqZ91cWU9ZBNwUEVuAacDNnQguSRpbW8+Tz8xlwLJRx+4E7tzP2M3A2eOQTZL0MvmKV0kqmCUvSQWz5CWpYJa8JBXMkpekglnyklQwS16SCmbJS1LBLHlJKpglL0kFs+QlqWCWvCQVzJKXpIJZ8pJUMEtekgpmyUtSwSx5SSqYJS9JBbPkJalglrwkFaytC3lHxLXAAmAIWAVsAb44YsgbgYcy88KImAfcBswA7gOWZObgeIaWJLVnzJV8RJwLXACcDpwJXA5szcx5mTkP+D3gGeBT9ZTVwNLMPAXoARZ3ILckqQ1jlnxmbgDOr1fjR1Ot/p8bMeQG4NbMfCIiTgD6MnNjfd8dwMLxjSxJaldb2zWZuS8ilgNXAn8HPAUQEW8GzgMurYceCwyMmDoAzG4SaObMaU2Gv0R///RDnttJ5mrGXM11azZzNdOJXG2VPEBmXhMR1wP3Um3B/DVwGXBLZv6qHtZLtW8/rAdoNQm0a9duWq2hsQeO0t8/nZ07n208r9PM1Yy5muvWbOZq356tT9K7Yxut2SfSd/KcRnN7e3sOujges+QjYi4wNTM3ZebzEXEP1f48wAeA948YvgM4ZsTtWcDTjRJL0m+QPVufZMeNX2LoxUF6Jk1m9pVXNS76g2nnKZQnASsjYkpEHAlcBNwfEUdR7b9vGx6YmduBvRExvz50CbBu3NJKUmH25BaGBvdBq8XQi4PsyS3j+vnbOfG6FlgDPAo8DDyYmXdRlf+O/UxZBNwUEVuAacDN4xdXksrSF3PpmXwE9PbSM2kyfTF3XD9/uydelwHLRh37EXDOfsZuBs4eh2ySVLy+k+cw+8qrDnlPfixtn3iVJHVG38lz6D/njI6cEPZtDSSpYJa8JBXMkpekglnyklQwS16SCmbJS1LBLHlJKpglL0kFs+QlqWCWvCQVzJKXpIJZ8pJUMEtekgpmyUtSwSx5SSqYJS9JBbPkJalglrwkFayty/9FxLXAAmAIWJWZKyLit4GbgOnAY8BHMvOFiJgH3AbMAO4DlmTmYCfCS5IObsyVfEScC1wAnA6cCVweEb8F3ANclpmn1UM/Vv93NbA0M08BeoDF455aktSWMUs+MzcA59er8aOpVv/zgB9m5mP1sMuBb0XECUBfZm6sj98BLBzv0JKk9vQMDQ21NTAilgNXAn8H/Bw4DTgSmAs8AFwBnAHckJnvqufMAdbWq/qxvAnY1jC/JKlyIvDL0Qfb2pMHyMxrIuJ64F6qMv5d4BzgH4FVwNXAP1Dt2w/rAVpNUu7atZtWq70HnpH6+6ezc+ezjed1mrmaMVdz3ZrNXM0caq7e3h5mzpx24PvH+gQRMbc+mUpmPk+1F381sDEzt2Xmi8DXgbOBHcAxI6bPAp5unFqSNC7aeQrlScDKiJgSEUcCFwGXAW+PiOPqMRcCD2fmdmBvRMyvj18CrBvv0JKk9rRz4nUtsAZ4FHgYeDAz/wb4OHBvRGwBXg9cV09ZBNxUH58G3NyJ4JKksbW1J5+Zy4Blo46toSr/0WM3U23dSJImmK94laSCWfKSVDBLXpIKZslLUsEseUkqmCUvSQWz5CWpYJa8JBXMkpekglnyklQwS16SCmbJS1LBLHlJKpglL0kFs+QlqWCWvCQVzJKXpIJZ8pJUMEtekgrW1jVeI+JaYAEwBKzKzBUR8RXgXcBz9bDlmfmtiJgH3AbMAO4DlmTm4LgnlySNacySj4hzgQuA04EjgMcjYg1wJvA7mTkwaspq4NLM3BgRq4DFwP8Y39iSpHaMuV2TmRuA8+vV+NFUDwx7gOOB2yPisYhYHhG9EXEC0JeZG+vpdwALOxNdkjSWtvbkM3NfRCwHHgfWU63ovwv8CXAO8G7gY8CxwMiV/QAwezwDS5La19aePEBmXhMR1wP3Au/JzA8O3xcRfwV8mOpBYGjEtB6g1STQzJnTmgx/if7+6Yc8t5PM1Yy5muvWbOZqphO52tmTnwtMzcxNmfl8RNwDXBwRuzLzm/WwHmAfsAM4ZsT0WcDTTQLt2rWbVmto7IGj9PdPZ+fOZxvP6zRzNWOu5ro1m7maOdRcvb09B10ct7NdcxKwMiKmRMSRwEXABuAvI+J1EXEEcBnwrczcDuyNiPn13EuAdY1TS5LGRTsnXtcCa4BHgYeBBzPzWuA64AGqLZpNmfm39ZRFwE0RsQWYBtzcieCSpLG1tSefmcuAZaOO3QLcsp+xm4GzxyGbJOll8hWvklQwS16SCmbJS1LBLHlJKpglL0kFs+QlqWCWvCQVzJKXpIJZ8pJUMEtekgpmyUtSwSx5SSqYJS9JBbPkJalglrwkFcySl6SCWfKSVDBLXpIKZslLUsHausZrRFwLLACGgFWZuWLEfUuBBZl5Xn17HnAbMAO4D1iSmYPjG1uS1I4xV/IRcS5wAXA6cCZweUREfd+pwNWjpqwGlmbmKUAPsHhcE0uS2jZmyWfmBuD8ejV+NNXq/7mImAJ8Gfjc8NiIOAHoy8yN9aE7gIXjHVqS1J629uQzc19ELAceB9YDTwHXAbcDvxgx9FhgYMTtAWD2+ESVJDXV1p48QGZeExHXA/cClwHHZ+anI+K8EcN6qfbth/UArSaBZs6c1mT4S/T3Tz/kuZ1krmbM1Vy3ZjNXM53INWbJR8RcYGpmbsrM5yPiHuAdwGkRsQmYBsyKiLuBq4BjRkyfBTzdJNCuXbtptYbGHjhKf/90du58tvG8TjNXM+ZqrluzmauZQ83V29tz0MVxO9s1JwErI2JKRBwJXAT8fWa+JTPnAZcCP8nMizNzO7A3IubXcy8B1jVOLUkaF+2ceF0LrAEeBR4GHszMuw4yZRFwU0RsoVrl3zweQSVJzbW1J5+Zy4BlB7jv+8B5I25vBs5+2ckkSS+br3iVpIJZ8pJUMEtekgpmyUtSwSx5SSqYJS9JBbPkJalglrwkFcySl6SCWfKSVDBLXpIKZslLUsEseUkqmCUvSQWz5CWpYJa8JBXMkpekglnyklQwS16SCtbWNV4j4lpgATAErMrMFRHxCWAp0EN1oe+rMnMoIuYBtwEzgPuAJZk52InwkqSDG3MlHxHnAhcApwNnApdHRACfprpg99uAdwLvq6esBpZm5ilUDwCLO5BbktSGMUs+MzcA59er8aOpVv/PAadm5nPAa4HXAP8WEScAfZm5sZ5+B7CwA7lfYs/WJ9nxjXvYs/XJTn8pSXpFaWtPPjP3RcRy4HFgPfBUfWwx8AtgANgEHFt/PGwAmD2uiUfZs/VJdtz4JbZ/9W/ZceOXLHpJGqGtPXmAzLwmIq4H7qXagvnrzFwZEV8BvgIso9qbHxoxrQdoNQk0c+a0JsPZsWEbQy8OQqvFEIP07thG/zlnNPocndbfP32iI+yXuZrp1lzQvdnM1Uwnco1Z8hExF5iamZsy8/mIuAd4R0T8LDMfyMzBiLgL+ATwZeCYEdNnAU83CbRr125araGxB9Zas0+kZ9JkhhikZ9JkWrNPZOfOZ5t8yY7q75/eVXmGmauZbs0F3ZvNXM0caq7e3p6DLo7bWcmfBCyPiHdRrdIvonrWzFfrZ9L8P6pn3tyfmdsjYm9EzM/MB4BLgHWNUzfQd/IcZl95Fb07ttGafSJ9J8/p5JeTpFeUdk68rqXahnkUeBh4MDO/AFwHPAhsBp4H/qKesgi4KSK2ANOAmzuQ+yX6Tp7D7AV/YMFL0iht7cln5jKqPfeRx75MtT0zeuxmqqdWSpImmK94laSCWfKSVDBLXpIKZslLUsHafjHUYTAJqud8HqqXM7eTzNWMuZrr1mzmauZQco2YM2l/9/cMDbX/wqMOexfwg4kOIUmvUO8G7h99sJtKfgpwFtX73bw4wVkk6ZViEtU7DfwY+NXoO7up5CVJ48wTr5JUMEtekgpmyUtSwSx5SSqYJS9JBbPkJalglrwkFayb3tZgvyJiBtXFSS7MzF9GxG8DNwHTgceAj2TmC/VVqm4DZlBduWpJfWnC44HVwNFAAosyc/d45gJOBb444u43Ag9l5oUTmav+fr0fuIHqBROPAJdO9PerzvVR4CqqF759F7ii/vqHO9c1wB/WN9dk5lUR8V5gBdAH3J2Zn6nHHrZs+8tVHz8C+F/A5zPz+92QKyIuA/6M6spxPwE+frj/jR0g1yeApVTXml4DXJWZQxOda8R9S4EFmXlefbsjubp6JR8R76B6me4p9e0ZwD3AZZl5Wj3sY/V/VwNLM/MUqh/q4vr4LcAtmTmX6h/gZ8c7V2auzcx5mTkP+D3gGeBTE52rtgr4o8x8K/Aq4MMTnSsiAvgC8J7MfBtwBFVJHO5c7wXeD5wBzAPeHhF/DNxOdZnLtwBnRcTvH85sB8j1wfr79n3gnaOmTGSu/wL85zrT6VSd8qddkOtTwKepLmD0tjrf+7og1wfr+04Frh41pSO5urrkqf4n/5R/vxj4+4AfZuZj9e3LgW9FxAlAX2ZurI/fASysVz2/A3xj5PEO5BrpBuDWzHyiS3JNAmZExCRgKrCnC3KdTvVzHKhvfwf4wATkGqD6C+KFzNwH/JzqgeiJzNyWmYNUv3gLD3O2/eU6nmpBcwPw0PDALsg1FfhkZj6TmUPA/waO74JcLeDUzHwOeC3wGuDfuiDX8RExheqqep8bHtjJXF29XZOZlwJUCxgA5gC7I+IuYC7wAHAF1SPlwIipA8Bs4CjgmfqXdeTx8c5FffvNwHnApfWhY7sg1yepVn/PANuo/rH8hwnOtRlYERHHURX/AmAWh//79bPhj+uf3R8Cf3WADIct2wFyzc/MJ+pj/2nE8G7K1U+1PfLRLsm1LyIWAzcCPwI2AW+f6FxU18e+ner3cVjHvl/dvpIfbTLwu8B/pfphvZrqT55eqv3AYT1Uj+Sjj1Mf75TLqP6sGn6ToAnNFRGzgD8H3kr1BkYbqfaaJzRXZv4fqp/bt6neefQx4IWJyhURpwH/QLXt8Is2M3Q828hcw0W6H12RKyLeCKwHVtXnCroiV2auBGYC/0x1neoJzQW8CTg+M78yaljHcr3SSv6fgY31n9IvAl+n2nPbQVViw2ZRrRD/L/CaequCesz+tljGyweAu0bcnuhc7wZ+mplbM7MFrKT6S2NCc0XEVOBHmXlGZr4TeArYOhG5ImI+VTldnZl3HiTDYc22n1wHMuG5ImIu1Un1OzPz892QKyKOq49Rr4LvotomnOjv1x8Dp0XEJqqTrGdGxN2dzPVKK/m/pzp5cVx9+0Lg4czcDuwd/qEClwDr6n2wHwAX18c/DKzrRLCIOIpqT+3Xf4J1Qa6fAmdHxBvq2xcBP+6CXK8G1kfE9Ig4kurcyt2HO1f97+h/Ah/KzOEH54equ2JO/Yv1oTrDYct2gFz7NdG5ImI61e/lZzLzL7olF9Ue/Fcj4rUR0UO1JXj/ROfKzD/JzLfUT9K4FPhJZl7cyVxdvSc/Wmb+U0R8HLi3Xg1uAq6s714ErKyfgfMIcHN9/JPAnRHxGeAfqR5JO+Ekqkfj0SYsV2b+PCI+C3wvIgaBJ6m2lCY6166IWE61fXQE8LXM/NoE5LqS6sThihHnC26l2lP+Zn3fWv79pNfhyrbfXJl56wHGT2Suu4E3AFdExBX1sW9n5ucmONetVHvfDwKDVEU5/CD0G/Vz9P3kJalgr7TtGklSA5a8JBXMkpekglnyklQwS16SCmbJS1LBLHlJKpglL0kF+/8KP/6wkbz7pwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_train = np.array([[2104],[1600],[2400]], dtype=np.float32)\n",
    "y_train = np.array([[399.900], [329.900], [369.000]], dtype=np.float32)\n",
    "\n",
    "plt.plot(x_train, y_train, 'r.')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:20:38.777084Z",
     "start_time": "2024-08-03T02:20:38.772543Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "x_train = torch.from_numpy(x_train)\n",
    "y_train = torch.from_numpy(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-07T04:47:17.479477Z",
     "start_time": "2019-04-07T04:47:17.468259Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**nn.Linear**\n",
    "\n",
    "> help(nn.Linear)\n",
    "\n",
    "Applies a linear transformation to the incoming data: $y = xA^T + b$\n",
    "\n",
    "- **in_features**: size of each input sample\n",
    "- **out_features**: size of each output sample\n",
    "- **bias**: If set to False, the layer will not learn an additive bias. Default: ``True``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:21:01.098561Z",
     "start_time": "2024-08-03T02:21:01.093617Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Linear Regression Model\n",
    "class LinearRegression(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(LinearRegression, self).__init__()\n",
    "        self.linear = nn.Linear(1, 1)  # input and output is 1 dimension\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = self.linear(x)\n",
    "        return out\n",
    "\n",
    "model = LinearRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:21:12.445941Z",
     "start_time": "2024-08-03T02:21:12.442040Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Define Loss and Optimizatioin function\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=1e-9)#1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-20T13:07:42.012941Z",
     "start_time": "2019-06-20T13:07:42.003484Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "> help(nn.MSELoss)\n",
    "\n",
    "To measures the **mean squared error** (squared L2 norm) between each element in the input `x` and target `y`.\n",
    "\n",
    "> help(optim.SGD)\n",
    "\n",
    "Implements **stochastic gradient descent** (optionally with momentum).\n",
    "\n",
    "Momentum is a variation on stochastic gradient descent that takes previous updates into account as well and generally leads to faster training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:22:00.519176Z",
     "start_time": "2024-08-03T02:22:00.080266Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[100/1000], loss: 2008.706909\n",
      "Epoch[200/1000], loss: 2006.916992\n",
      "Epoch[300/1000], loss: 2006.592407\n",
      "Epoch[400/1000], loss: 2006.533691\n",
      "Epoch[500/1000], loss: 2006.524048\n",
      "Epoch[600/1000], loss: 2006.521362\n",
      "Epoch[700/1000], loss: 2006.521118\n",
      "Epoch[800/1000], loss: 2006.520508\n",
      "Epoch[900/1000], loss: 2006.521118\n",
      "Epoch[1000/1000], loss: 2006.521118\n"
     ]
    }
   ],
   "source": [
    "num_epochs = 1000\n",
    "for epoch in range(num_epochs):\n",
    "    inputs = Variable(x_train)\n",
    "    target = Variable(y_train) \n",
    "    # forward\n",
    "    out = model(inputs)\n",
    "    loss = criterion(out, target)\n",
    "    # backward\n",
    "    optimizer.zero_grad() # Clears the gradients of all optimized\n",
    "    loss.backward()\n",
    "    optimizer.step() # Performs a single optimization step.\n",
    "\n",
    "    if (epoch+1) % 100 == 0:\n",
    "        print('Epoch[{}/{}], loss: {:.6f}'\n",
    "              .format(epoch+1, num_epochs, loss.data.item()))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:22:09.561469Z",
     "start_time": "2024-08-03T02:22:09.342964Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAESCAYAAADTx4MfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA+20lEQVR4nO3dd3gU5drH8W8KaSShJIGELpA8NOkg0kEsIDa6IChVRVResXCUo2LFzgHL4dCbghgsKKgoJQLSe3vovSYBQ4DU3feP2SwJJJDdJDub5P5c17mOMzuz89uQ5M4z5X48rFYrQgghhKM8zQ4ghBCicJICIoQQwilSQIQQQjhFCogQQginSAERQgjhFG+zA7iIL9AMOA2km5xFCCEKCy8gAtgAJF//YnEpIM2Av8wOIYQQhVQbYNX1K4tLATkNcOHCZSwWx597CQkJJC4uMd9D5ZXkcozkcpy7ZpNcjnE2l6enB2XKlATb79DrFZcCkg5gsVidKiAZ+7ojyeUYyeU4d80muRyTx1zZnvqXi+hCCCGcIgVECCGEU6SACCGEcIoUECGEEE4pLhfRhRCiWBk5cRUJl1NuWB9c0ofxz7bOl2PICEQIIYqg7IrHzdY7QwqIEEIIp0gBEUKIIuZs/BWXHEeugQghRBGRbrGwdMMJvv/rkEuO5xYFRCn1MRCqtX5CKTUMeA6wAhuBJ7XWKUqphsAUIBiIAZ7SWqeZlVkIIdzJ8XOJTF+8hyNnLtEoMpQt+2ML/Jimn8JSSt0FPG777yjgJaAlUB8j3zO2TecAI7TWUYAHMNT1aYUQwr2kpln4PuYQb83YQHxCEk8/XI8R3W4nuKRPttvntN4Zpo5AlFJlgXeB94AGGO2Ch2utE2yv7wCqKKWqAv5a67W2XWcAY4GvXB5aCCHcxMGT/zB9yV5OxV7mzrrhPNopkkD/EgBZbtUNCwvi/PlL+X58s09hTQJeAyoDaK2PAkcBlFJhwAjgCaACWbtBngYquTKoEEK4i+SUdBbGHOKPjccpE+zLyJ4NqF8jxOU5TCsgSqkhwHGt9Z9KqSeue60isASYqrVeoZRqhXFNJIMHYHH0mCEhgU7nDQsLcnrfgiS5HCO5HOeu2Yprrm37zjNxwVbOxl+hS8tqPH5/HQL8SpiSy8wRSG8gQim1FSgLBCqlPsMYlfwGTNBaf2Lb9gTGrFgZwoFTjh4wLi7RqZbGBTX8yyvJ5RjJ5Th3zVYcc11JSmX+sgP8tf005cv480rfRqgqZbh8KYnLl5IKJJenp8dN//A2rYBore/O+G/bCKQ98DqwC3hNaz0707ZHlVJJSqlWWuvVQH+MEYoQQhR5m/edZ/bvmkuXU+nSoioPtqqGTwkvs2OZfg3kekOA8sAopdQo27qftNavA/2AyUqpYGAzMMGkjEII4RL/XE5h7tJ9bNx7jsrlAnm+R32qhQebHcvOLQqI1noGxp1VAJ/lsM02oLmLIgkhhGmsVit/7zrDN3/sJzk1nW5tq3PfHVXw9jL9yYss3KKACCGEMMT9k8Ss3zQ7DsVRo2IwAzvXpkJoSbNjZUsKiBBCuAGL1cqKLSdZsOIgWKFvp0g6Nq6Ep6eH2dFyJAVECCFMdib+CjMW72HfiX+oW60Mj99Xi9DS/mbHuiUpIEIIYZJ0i4Xf1h/nh78O4+PtyaAutWl1ezgeHu476shMCogQQpjg2NlLTF+8l6NnL9EkKozH7omiVKCv2bEcIgVECCFcKDUtnZ9WH2HJ2mMEBpRg+MP1aFqrnNmxnCIFRAghXOTAiX+YvmQPp+Ou0KpeOL3vutb8sDCSAiKEEAUsKSWN6JWHWLbpBGWDfXmhVwPqVXd988P8JgVECCEK0M7DccxcoolPSKJjk0p0a1sdf9+i8au3aHwKIYRwM4lXU5m/bD+rd5whvGwAox9rTGSl0mbHyldSQIQQIp9t3HuOOUv3kXgllfvvNJoflvA2v/lhfpMCIoQQ+eRCQhJffL+DTfo8VcoH8kKvBlQp757zluQHKSBCCJFHVquV1TvO8O3yAySlpNO9XXXube5+zQ/zmxQQIYTIg9iLV5n5m2bX4Xjq3FaWfp0iiQhxz+aH+U0KiBBCOMFitbJs0wmiVx4CD3jsnih63l2LuLhEs6O5jBQQIYRw0Om4y0xfspcDJ/6hXvWyDLhXEVrK36075xYEKSBCCJFLaekWfl13jJ9WH8a3hBdDutbmzrqFp/lhfpMCIoQQuXD0zCWmL97DsXOJNK1Vjn53R1GqpI/ZsUwlBUQIIW4iJdVofvjrumMEBZTgmUdup4kKMzuWW3CLAqKU+hgI1Vo/oZTqBHwK+APztdZjbNs0BKYAwUAM8JTWOs2kyEKIYmDf8YtMX7KXs/FXaF0/gt4da1LSr/A2P8xvpt+krJS6C3jc9t/+wDTgIaA20Ewp1dm26RxghNY6CvAAhpoQV4hiI2HtGjYOeZJ9Q57g0MujSFi7xuxILnM1OY05v2vGzd1MerqFUX0aMqhLbSke1zF1BKKUKgu8C7wHNACaA/u11odtr88BeiqldgP+Wuu1tl1nAGOBr1weWohiIGHtGs7OmoE1JQWAtPg4zs6aAUBwi5YmJit42w/GMeu3vVxISObuppXp1rY6vj5Frw1JfjD7FNYk4DWgsm25AnA60+ungUo3WS+EKACxC6PtxSODNSWF2IXRRbaAJF5N5Zs/9vP3rjNEhATwr/5NqFmxlNmx3JppBUQpNQQ4rrX+Uyn1hG21J2DNtJkHYLnJeoeEhAQ6FxYIC3PPfjaSyzGSK3f2XYjPdn3ahXi3yZpfOaxWK6u3n2LSwh1cupJC77uj6N0pyunmh+7y9bleQeQycwTSG4hQSm0FygKBQFUgPdM24cAp4AQQkc16h8TFJWKxWG+94XXCwoI4f/6Sw/sVNMnlGMmVe95lypIWH5ftenfIml9fs4uJycz+TbNlfyxVw4MY2bM+VcoHcfHCFVNz5Tdnc3l6etz0D2/TCojW+u6M/7aNQNoDTwH7lVI1gcNAX2Ca1vqoUipJKdVKa70a6A8scX1qIYqH0G7ds1wDAfDw8SG0W3cTU+Ufq9XKqu2nmbfsAGnpFnp2qME9zSrj5Wn6fUWFitnXQLLQWifZikk04AcsBr6zvdwPmKyUCgY2AxNMCSlEMZBxnePCjwtJPh+Ld9kQQrt1LxLXP85dvMqsX/ey+8gFoiqXZmDnWpQvG2B2rELJLQqI1noGxp1VaK3/xLgj6/pttmHcpSWEcIHgFi2p8cC9bnlKxhkWi5U/Np1gYcxBPD086H+vol3DCngW0zYk+cEtCogQQhSkk7GXmbF4DwdPJVC/RggD7lWUDfYzO1ahJwVECFFkpaVbWLz2KD+vOYKfjzdDH6hDizrli23zw/wmBUQIUSQdPp3A9MV7OXE+kea1y9G3UxTBxbz5YX6TAiKEKFJSUtP5YdVhflt/jFIlfXi2++00ipTmhwVBCogQosjQxy4wfclezl24StsGFejVoQYB0r+qwEgBEUIUeleT01iw4iArtpwkrLQfL/VpSO1qZc2OVeRJARFCFGrbDsQy6zfNxcRk7mlWmUfaVse3hDQ/dAUpIEKIQunSlRS++XM/a3edpWJoSYY/Uo8aFaT5oStJARFCFCpWq5WYLSf4Kno7V5PTeLBVNbq2rIa3l7QhcTUpIEKIQuPCJaP54dYDsdwWEcTAzrWpVM75Ltsib6SACCHcntVqJWbbKb5dfoD0dCuDH6zLnbXK4ekpDwSaSQqIEMKtnbtwhRlL9rL32EVqVSnNE51rUTeqfJHp0VWYSQERQrgli8XK0o3H+T7mEF5eHjx+n6JtgwrShsSNSAERQridE+cTmb54L4dPJ9CwZij971WUCfI1O5a4jhQQIYTbSEu38MvfRvNDf19vnnywLs1rl5NRh5uSAiKEcAuHTiUwfckeTp6/TIs65Xm0UyRBAdL80J1JARFCmCo5NZ3vYw6xdONxSgf68lyP+jSsGWp2LJELUkCEEKbZc/QCM5bs4fzFJNo3qkjP9jXw95VfS4WF/EsJIVzuSlIa3y4/QMy2U5Qr488rfRuhqpQxO5ZwkOkFRCn1FtADsAJTtdafKqXuAT4CvIDNwBCtdYpSqiEwBQgGYoCntNZp5iQXQjhj6/5YZv22l38up3DfHVV4qPVt0vywkDK1eYxSqh3QEagPNAWeVUopYCrQR2tdDwgABth2mQOM0FpHAR7AUNenFkI4I+FKCv/9cScTorcT6F+CMQOa0qtDTSkehZipBURrvRLoYBtFlMMYEV3GGHkEK6W8AD/gqlKqKuCvtV5r230G0NP1qYUQjrBarazddYYxk9exSZ/n4Ta38foTzbgtItjsaCKPTD+FpbVOVUqNBV4EFgAngeHACiABOAx8BzQGTmfa9TRQyaVhhRAOiU9IYtZvmu0H46heIZiBnWtRMUyaHxYVHlar1ewMACilAoBFwEqgL3A/RvH4FCgBfA2M01q3sW0fCSzSWtfKxdtXs72XEMIFLBYrv609wvSfd2OxWunfuTZdW1fHS5ofFla3AUeuX2nqCEQpVQvw01pv1VpfUUotxBh97NRaH7RtMxn4FvgQiMi0ezhwypHjxcUlYrE4XjDDwoLcsnGb5HKM5HKcM9nOxhvND/Xxi9SuWobHO9eiXGl/4uMSTc3lCkUtl6enByEhOY8YzZ6BpTowWSnlq5TyAR7CuFDeXClV3rbNQ8AGrfVRIEkp1cq2vj+wxOWJhRDZSrdYWLLuKK9PW8+xc4kM7FyLF/s0pFxpf7OjiQJi6ghEa71YKdUc2AKkA9Fa6/eVUqeA5UqpNOAAMMy2Sz+MghOMcXvvBDNyCyGyOn4ukemL93DkzCUaRYby2D3S/LA4cIeL6G8Cb163biYwM5tttwHNXRJMCHFLqWkWfl5zhMVrj1LSz5unH65HUxUmzQ+LCdMLiBCicDpw8h+mL97D6bgr3Fk3nEc7RRLoX8LsWMKFpIAIIRySnJJOdMxB/tx4gjLBvozs2YD6NULMjiVMIAVECJFru47EM3PJXmL/SaJj44p0byfND4sz+ZcXQtxg5MRVJFxOyfa18mX8Gd2vMVGVS7s2lHA7UkCEEDfIqXgAjB3UHB/pXyUw/zkQIUQhI8VDZJACIoTIwl3aGwn3J6ewhBB2+45fZNzczWbHEPkkYe0aYhdGs+9CPN5lyhLarTvBLVrm2/tLARFCkG6x8Ma0DZyKvWx2FJFPEtau4eysGVhTjOtZafFxnJ01AyDfioicwhKimNu6P5ahH66wF49X+jYiuKRPttvmtF64n9iF0fbikcGakkLswuh8O4aMQIQoplLT0vm/iau5kmzMCl2rSmleerQRHh4ejH+2NeC+3WXFraXFxzm03hlSQIQohlZtP820xXvsy28ObEaV8kEmJhL5zbtsSLbFwrts/nUNkAIiRDFyJSmNEeNj7Mt31CnPkw/WNTGRKCih3bpnuQYC4OHjQ2i37vl2DCkgQhQTS9YeZcGKg/blcU+2oFyZABMTiYKUcaE8dmE0aXIXlhDCGf8kJvN/n6+2L9/bvDK9O0aamEi4SnCLlgS3aFlg17KkgAhRhM1ftp/f1h+3L382ohWlAmWiJ5E/pIAIUQSdu3CF0ZPW2pd7dqhB5zuqmphIFEVSQIQoYib9tIt1u8/alz8f2ZYAP/lRF/lPvquEKCKOnrnE2Bkb7MuDutSmdf0IExOJos70AqKUegvoAViBqVrrT5VSdwKfAUHAduBxrXWKUqohMAUIBmKAp7TWaeYkF8I9WK1WPvx6C/r4RQBK+nnz6YhWlPCWrrmiYJnaykQp1Q7oCNQHmgLPKqUaAAuBYVrrjBvUB9v+fw4wQmsdBXgAQ10cWQi3svfoBQZ/sNxePJ7rXp+JI9tK8RAuYeoIRGu9UinVQWudppSqaMvTEPhba73dttmzgLdSqirgr7XOuDI4AxgLfOXi2EKYLt1iYczkdZy9cBWAiqEleXNQM7w8pb2dcB3TT2FprVOVUmOBF4EFQDiQqJSaB9QCVgOjgEbA6Uy7ngYquTiuEKbbpM/zxfc77Msyvawwi4e7TB6jlAoAFmFc2xgOtACOAVOBI8BSYJzWuo1t+0hgkda6Vi7evhpwOP9TC+E6SSlp9H/jV5JS0gFoGBXGW8PuxMPDw+Rkohi4DeP3cBamjkCUUrUAP631Vq31FaXUQuBj4Het9WHbNt8CI4DpQOZbSsKBU44cLy4uEYvF8YLprh1JJZdjCnOumG2nmLFkr3157KDmVC4XSGxsounZzCC5HONsLk9PD0JCAnN+PS+h8kF1YLJSylcp5QM8BAwDmiilKtu26Qps0lofBZKUUq1s6/sDS1yeWAgXupyUyqBxy+zFo2W9cKaN7kjlcjn/UAvhKmZfRF+slGoObAHSgWit9WylVDywSCnlB2zFuD4C0A+j4AQDm4EJJsQWwiV++fsI0SsP2Zc/eOpOwkr7m5hIiKzc4SL6m8Cb1637Bfglm223Ac1dEkwIk1y4lMyoL641P+zcogo929c0MZEQ2XOogCilZgGzgD+11u5x9V2IIuTrpfv4Y9MJ+/L4Z1vLNLLCbTk6AnkM4zTSGaXUXGBOpuc1hBBOOhN/hVf/d635YZ+ONbmneRUTEwlxa44WkGoYRaQvxnWJUUqpncBM4But9emb7CuEuI7VamXczA2s3n7thsIv/q8t/r6mn10W4pYc+i7VWh8D3gPes/Wlegzog3Hr7QdKqWUYp7i+11pfyeesQhQpR84k8NaMjfbloV3rcGe9cBMTCeEYp//M0VpvBbYqpV4CWmPcbtsHo4BcUUpFA5O11qtzfhchih+L1cq4OZs5cPIfAEoH+vLBU3dSwtvsu+qFcEx+jJOrA60wikhFjCaHx4BeQH+l1J9AP631+Xw4lhCF2u4j8Xw8b6t9eWTP+tzV4ja3fPhMiFtxqoAopcKB3hjXQppiFI1TwKfATK31LtuzGqNt/5sFdM6XxEIUQmnpFv41aS1xCUkAVCkXyOtPNMPTU9qQiMLL0dt4B2EUjXaAF3AVmIdxEf0PrbUlY1utdQLwqlKqF9Am3xILUchs2HuOr37YaV9+tX8TalYsZWIiIfKHoyOQKRgTP/2FUTQWaK1v1YznFLDjFtsIUeQkp6QzYnwM6bb+a/VrhPB8j/rS/FAUGY4WkNeB2ba+VLmitW7r4DGEKPSWbz7B7N/32ZffHtycimHSv0oULY7exvtOQQURoihIvJrKc//5y77ctkEET3SubWIiIQqOPK0kRD75adVhflh1bdqZj55uSUgpPxMTCVGwpIAIkUfxCUm8+OUa+3LXltXo1ra6iYmEcA0pIELkwezfNMu3nLQv/+e51gQFSPNDUTxIARHCCadiLzNmyjr7cr+7o7irSSUTEwnhelJAhHCA1Wrl84U72LI/FjCeoP3ihbb4+ciPkih+5LteiFw6dCqBd2Zda3745IN1uaNOeRMTCWEuKSBC3ILFauXdWRs5fNroV1UmyGh+6O0lzQ9F8SYFRIib2Hk4jk/nb7Mvv9C7AfVuCzExkRDuw/QCopR6C+iB0SJlqtb600yvjQB6aK3b25YbYrRTCQZigKe01mmuziyKvrR0Cy99tYZ/ElMAuC0iiNcGNMVT2pAIYWfqGFwp1Q7oCNTH6Or7rFJK2V6rg9HJN7M5wAitdRTG9cuhLowriol1u88y7KMV9uIxZkBT/v14MykeQlzH1AKitV4JdLCNIsphjIguK6V8gUkYvbcAUEpVBfy11hkTR88Aero2sSjKklLSGDRuGZN+2gVAo8hQpr7SgeoVgk1OJoR7Mv0UltY6VSk1FmOO9QXASeATYBpwONOmFYDMc66fBuTGe5Ev/tx0grlLrzU/fHfoHUSElDQxkRDuz/QCAqC1fkMp9QGwCBgGVNFav6CUap9pM0+M6yQZPAALDggJcb4balhYkNP7FiTJ5Zjrc/2TmMxjb/xqX+7cshrDuzdwdSy3/XqB+2aTXI4piFymFhClVC3AT2u9VWt9RSm1ELgDqKuU2goEAuFKqfnAy0BEpt3DMeYaybW4uEQsFuutN7xOWFiQW045Krkcc32uhTEH+XnNtZkJPh7ekrLBfi7P7q5fL3DfbJLLMc7m8vT0uOkf3maPQKoDY5VSrTFGFw8B07TWgwBsI5A3tda9bctJSqlWWuvVQH9giTmxRWEW+89VXv7qb/vyw61v48HWt5mYSIjCydQCorVerJRqDmwB0oForfW8m+zSD5hsm299MzDBBTFFETJ98R7+2n7tUtqE59sQ6F/CxERCFF5mj0DQWr8JvJnDayuA9pmWtwHNXRBLFDEnzycyaNwy+3L/exUdGlU0MZEQhZ/pBUSIgmS1Whm/YDs7DsUB4O3lycTn2+Dr42VyMiEKPykgosg6cPIf3pu9yb48ekAzoiq45x0yQhRGUkBEkWOxWBk7YwPHzyUCEFrKj/eGtSAivJRb3iEjRGElBUQUKdsPxjJ+wXb78ot9GlKnWlkTEwlRdEkBEUVCapqFUV+sJvFqKgA1K5VidL/G0r9KiAIkBUQUemt2nmbKz3vsy68/0ZRq4dK/SoiCJgXkJhLWriF2YTT7LsTjXaYsod26E9yipdmxhM3V5DSe+SzGvtysVjmeeqguHjLqEMIlpIDkIGHtGs7OmoE1xWjpnRYfx9lZMwCkiLiB39cfY96yA/bl94e1oHzZABMTCVH8SAHJQezCaHvxyGBNSSF2YbQUEBP9czmF/5u4yr7cqUkl+t4dZWIiIYovKSA5SIuPc2i9KHgLlh9gybpj9uVPnmlFmSBfExMJUbxJAcmBd9mQbIuFd1mZD9vVzl+8yiv/vdb8sHu76tx/Z7UCPWZaWiqXLyeQnHwViyW9QI917pwnFotDMxO4jLtmk1yOycjl6emFr68/JUsG4+2d9x5wUkByENqte5ZrIAAePj6EdutuYqriZ8rPu1mz84x9eeLINpT0K9jmh2lpqcTHnyUgIIiyZcPx8vIq0Avz3t6epKW53y8dcN9skssx3t6epKamk56eTlLSZeLjz1K2bPk8FxEpIDnIuM4RuzCaNLkLy+WOn0vkjWnr7ctPdK5F2wYVXHLsy5cTCAgIIjCwlEuOJ4QreHh44O3tbf++vnw5gVKl8nZGRQrITQS3aElwi5ZuO0lMUWS1Wvlk/lZ2H7kAgK+PF/95tjU+JVzX/DA5+Sply4a77HhCuJqfX0ni48/cesNbkAIi3Ma+4xcZN3ezfXlEt9tpHBXm8hwWSzpeXtKtVxRdXl5e+XJtTwqIMF26xcLrU9dzOu4KAOXLBvDOkOZ4eXqalkkeRhRFWX59f0sBEabasv88E6N32Jdf6dsIVaWMiYmEELklBUSYIiU1nf/7fDVXk9MAqFWlNC892kj+8heiEJECIlzur+2nmL54r335zYHNqFJeJnoSorAxvYAopd4CegBWYKrW+lOl1DDgOdu6jcCTWusUpVRDYAoQDMQAT2mt08xJLhx1JSmNEeOvNT9sUac8wx6sa2IikRtHjhxmwYJv2LRpA+fPn8Pf358KFSrRsWMnunZ9mMDAwFy/1+LFi3jvvbE899wL9OrV1+EsI0YMY+vWzSxduhJ//5IO759bp0+fomfPB2nTph3vv/+JU+/xn/98woIF3zBhwn9p3LipU++xfv1agoKCqF3bPX9OzLtKCSil2gEdgfpAU+BZpZQCXgJa2tZ7As/YdpkDjNBaRwEewFCXhxZOWbL2aJbiMe7JFlI8CoG5c2fx+ON9+PnnH6lYsTLdu/eifftOJCcn8/nn4+nXrwc7d+649RvZREZGMXDgUOrUud2pPF26PMDAgUPx8fFxav/C5Pvvv+OFF0Zw/vx5s6PkyNQRiNZ6pVKqg9Y6TSlV0ZYnCRiutU4AUErtAKoopaoC/lrrtbbdZwBjga9MiC5y6WJiMi98vtq+fG/zyvTuGGliIpFb0dHfMnHieKpVq87bb4/jttuqZ3l96dJfef/9t3nuuaeYMmUW1avXuOV7RkYqIiOV05m6dHkAcN8nvvNTfCHou2fqCARAa52qlBoL7Ab+BI5prZcCKKXCgBHAj0AF4HSmXU8DlVwcVzhg3p/7sxSPz0a0kuJRSMTGxvLFF+MpVao048d/cUPxALj77vt45ZXXSElJ5v33x5qQUpjN9GsgAFrrN5RSHwCLME5L/c82IlmCcV1khVKqFcY1kQwegEN/goSE5P5c7fXCwtzzIq875joVm8iT45bZlwd2rUO3Du5ROHLz9Tp3zhNv74L/2+rimjWci15AWlwc3iEhlOvek9It3aNVzqJFC0lJSeGxxx4nPLx8jtvdf39XvvlmNnv27ObAgb3UqlWHU6dO0a1bVwYNGsqlS5dYtOgHfH39ePnlf3H16lXeeedNRo4cRZ8+/ezvs3HjeqZOncy+fZoSJbxp374jPXv2oV+/XgwePIyhQ58C4Omnh7JlyyaWLl1JUFAQmzZt5JlnhjFmzJtYrRbmzfua48ePUapUae6++16GDXsKPz9/+3HS0lKJjl7A77//ypEjR0hOTiI0NJQ772zFsGHDKVPGuIXcy8v49zfaf9z6e2HRoh+YP/8bjh8/TlhYGP369SfjMSYvr6zfT6tWxfDdd9+yd+9uLl1KJCgokPr1GzJkyJNERaksnxPg1VdfBGDtWuMh2ytXrjBv3lyWL/+TkydPkJaWRrly5WnXrgNDhjyJv78/2bn+c3h6eub594epBUQpVQvw01pv1VpfUUotBOrb1v8GTNBaZ1zBOgFEZNo9HDjlyPHi4hKxWKy33vA67trKxB1zTfppF+t2n7Uvfz6yLQF+3m6RM7dfL4vFUuCnR26YsCwujtMzpmGxWNyi39r69esAuPPOlrf8WrRp054DB/azfPlyatasRXq6sf0PPywE4OGHe3Ds2BFq1arLpk0bALBYrPb3XblyGa+//i/8/QNo164Dfn5+/PHH7/YMmbe1Wq/9/KalWezHWrBgPgcP7qddu47cccedrFy5nK+/nk1CQgKjR//bvs+YMf9ixYpl1K/fkAcffISUlGTWr1/L999Hs2fPHqZMmQVgf1+r1XrLzz958lfMnDmV8PAIHnjgIWJjY/n44w8oXbq0/b0y3iM6ej6fffYRFStWolOneylRwoc9e3YRE7OCjRs38PXX0YSGhtK5c1esVitbt27mrrvupkqVaqSlWUhLS2PEiKfYs2cXzZvfSbNmLbhy5QqrV8cwd+4sTp48wTvvfHhDxuxO+Vksllv+PHh6etz0D2+zRyDVgbFKqdYYo4uHgNnA78BrWuvZGRtqrY8qpZKUUq201quB/hgjFOEGjp65xNgZG+zLg7rUpnX9iJvsUby5+4Rlx48fBaBKlWq33LZatdsAOHXqZJb1Fy7EM23aXCIjc57wKykpiU8++QB//wAmT55J5cpVAOjb93EGD+6X437XO3BgH198MZl69eoDMGDAYPr0eYTff/+V559/EX9/f3bu3MGKFcu4557OvP762/Z909LSGDz4Mfbu3c2xY0epUqVqro97/Pgx5syZQWRkFBMmTKJMmVKkpVlYvfovRo9+Icu2KSkp/O9/X1K5chWmTZubZaTw8cfj+OGH71i9OoaHHupGly4PcPr0KVsBuZe2bdsDsGLFn+zevZMBAwYxbNhw+/7Dhz9Lnz7d+OuvlSQlJeHn55frz5AXpl4D0VovBn4BtgCbgDVAKFAeGKWU2mr731u2XfoBnyml9gKBwAQTYotMLFYrH8zdbC8eJf28iR7XVYrHLbj7hGWJiYkAlCx561tlg4ODAfjnn4tZ1leqVPmmxQNg3bo1xMfH0aNHb3vxAAgPD6d379wXkIYNG9uLB0BgYCC3316flJRkzp0zRsTlypXjtdfeZPDgJ7Ps6+3tze23NwTgwoULuT4mwPLlf5Cens6AAYMICrp2OqhVqzY0b35nlm0tFgsvvzyG0aP/fcNppkaNmtiOH3/T40VF1WL06DH07p31FuiAgJJERSnS09NJSPjHoc+QF2aPQNBavwm8ed3qz3LYdhvQvIAjiVzae/QCH36zxb78XPf6NIwMdWnn3MLK3ScsCwoK5sKFeJKTk/H1zf6ceoarV5MAKF06awuaiIhbt9/fs2c3QLbPOdx+e4Pcxs1SfDKULGmcekmxjfTKlStP585dSUtLQ+u9HDt2hJMnT7B/v2bjRmPqAEcbDB44sB+AWrXqZJO/PuvWrbEv+/n5cddddwNw7NhRjhw5zMmTJzh8+GCm49/8dFmVKlWpUqUqycnJ7Nq1k+PHj3Ly5Am03mO/ZuLKCa1MLyCi8ElLtzBmyjrOXbgKQMXQkrw5qJmpzQ8LG3efsKxixUpcuBDP8eNHqVmz1k23PXLkEADly2dtge/re+vphjNGLSEhNxbO0NDcd2IuUeLG50KutcW5dt3khx+imTFjCrGxxrMVgYFB1K17O1Wr3sbu3TuzXGPJjUuXEgAICAi44bWgoOAb1m3dupkJEz5l3z6jE4OPjy81a0aiVG3OnTt7y+NbLBZmz57OvHlz7ccuU6Ys9erVJyKiAkeOHHb4M+SFFBDhkE36PF98f+3BsdH9GhNVubR5gQqpLBOWxcfhXTbErSYsa9OmHTt3bmflyhW3LCCrVhkPiGacp3dEQIBxiuzy5cs3vJbdurxYtuwPPv74fWrUiGTUqFeIiqplL3off/w+u3fvdPg9M4pEYmIipUqVzvLa1atXsiyfOXOaF198Dh8fX15++TXq129I5cpV8PLy4s8/f+evv1bc8njz5s1h8uSvaNSoCf36PU5kZBQhIaEAjBr1HEeOHHb4M+SFFBCRK8mp6Tw/4S9SUo3hcd3byvJCrwbS/DAPMiYsc8eH4u67735mz57OggXzuP/+hwgPz/6a1sqVy9i9eydRUcqpdhtK1QZg9+5dNGnSLMtrzvxCv5mlS38F4I033rnhoUdnf/EqVYtly5ayY8c2KlbM+lja3r17sizHxKwgKSmJp59+jgcffOSWx8/uZ2vp0l/x8vJi3LhP7KfowLhb7NixI/b/dhU55yBuaeXWkzz9yUp78XhrUHNG9W4oxaMICwkJ5dlnXyAxMZGRI4dz6NDBG7ZZuXI577zzBj4+Prz2mnMPErZp047g4FJ89928LHdxnTt3lq+/nuV0/uxktD+5/kL1kiU/s3Wr8YxFWppjrfXuuusefHx8mTFjKnFxsfb127ZtYdWqlbk6/oED+1mwYN4Nx/f2Nv6+T01NzfQevqSnp3Px4sUs7zFz5lROnz7l1GfICxmBiBxdTkrl2fF/2Zdb1QtncNcbLxaKoun++x8ErHz00fsMHNiXZs3uoEaNSFJTU9i2bSta7yEkJIQ33niXGjVqOnUMf39/XnjhZcaOHcPgwf1p164Dnp6exMQst2+TX7ND3ntvF/7883deffVFOnW6l5IlS7J79y62bt1MmTJluXAh/oY7yW4lPDyCZ555ns8++5BBg/rRvn1HLl1KZPnyPyhfPpyTJ0/Yt23Vqg3//e/nzJ49naNHj1CxYiVOnDjGmjWr7KOJzMcPCysHwKxZU9m/XzNw4FDuuaczu3bt4OmnB9OxYye8vUuwefNG9u3ba/8MxeouLOGeFq05wvcxh+zLHzx1J2Glb343jih6HnroERo2bEJ09Hw2blzPtm1bKVGiBBUrVmL48Ofp2vVBgoNL5ekYnTrdi5+fP7NmTWPp0l9tdyvdQ4MGjXnjjX/h65s/zzS0bNmasWPfY+7cmfz++xJ8ff2oUKEiL7zwCvXq3c6gQY+xdu1q7r77Pofet3v3XoSFlWP27On8/PMiSpUqxeDBT1KiRAkmTrx2Q2lYWDnGj/+SSZM+Z9OmDaxf/zfh4RH06NGbxx4byKOPPsK6dX9jtVrx8PCgY8e7+fvv1axZ8xfff7+ALl260q1bT8DK999Hs2jRDwQGBlG5chXefPNdfH19+de/XuTvv1dnuaW5IHm48nyZiaoBh+VJ9Fu7cCmZUV9c61/VpUVVerS/dZO8gs6VH3Kb68yZo4SH5/5hsrxyx2sgGQo62+XLiVy5coXQ0LAbTon+8stPvP/+W4wd+7799ldX5XJWYcqVm+/zTE+i3wYcueF98y+iKOzmLt3Hn5uuDbnHP9ua4JJFv222MM/x48cYMmQAXbo8wKuvvmFfn5ycxMKFC/Dy8qJBg4bmBRQ3JQVEcCb+Cq/+b619uc9dkdzTrLKJiURxERVVi9q167J48SJOnz5F7dp1SU5OYs2aVZw+fYphw4Y79DyIcC0pIMWY1Wrlyx92sklfm7Dmi/9ri7+vfFsI1/D09OSzz75g/vy5LF/+BwsXfou3dwlq1oxk+PDn6NChk9kRxU3Ib4pi6vDpBN6eudG+PLRrHe6sF36TPYQoGIGBgQwe/OQNPaqE+5MCUsxYrFben7OJgyeNNgjBASX4aHgrSrhg/gshRNEiBaQY2XUknk/mbbUvj+xZn/o1Qs0LJIQo1KSAFANp6Rb+NWktcQlG19Qq5QN5/fFmeHrKk+RCCOdJASniNuw9x1c/XOsp9Fr/JtSomLcHv4QQAqSAFFnJKek881kMFtuDovVrhPB8j/rSv0oIkW+kgBRByzefYPbv++zLbw+5g4qht55ZTgghHCEFpAhJvJrKc/+51vywbYMInuhc28REQoiiTApIEfHjqsP8uOranAIfPd2SkFL504ROCCGyY3oBUUq9BfTAmHdyqtb6U6VUJ+BTwB+Yr7UeY9u2ITAFCAZigKe01q5rfu+G4hOSePHLa/Mud21ZjW5tq5uYSAhRXJj69JhSqh3QEagPNAWeVUo1AKYBDwG1gWZKqc62XeYAI7TWUYAHMNT1qd3Hl9HbshSP/zzXWoqHyBeLFy+ideumtGjRmNatm+b4v5iYFVm2//bbr7O8z9Klv2aZEwMgNjaWX375Kcu6Hj0e4L772hfkR7qpqVMn0bp1UxYvXpTrbTM+e3Fm6ghEa71SKdVBa52mlKpoy1Ma2K+1PgyglJoD9FRK7Qb8tdYZXf9mAGOBr1yf3FynYi8zZso6+3K/u6O4q0mlm+whhHMaNWpCw4aNc3y9atVqAERGRjFw4FDq1Lnd/tqXX07g669nMX36XPu6Cxfi6du3O02aNLVNWGXo1etRkpNT8v8DFIBGjZoA1z57cWb6KSytdapSaizwIrAAqACczrTJaaDSTdYXG1arlYnRO9h6wJg608PDaH7o52P6P6Mooho3bsLAgcNuuV1kpCIyUmVZd/3UrQBJSUlcuXL5hvW9evV1PqSLNW7clMaNm5odwy24xW8erfUbSqkPgEVAFMb1kAwegAXjdFt263PNNjGKU8LCgpzeNz/oo/G8OOHaHVYvPdaEto3ct36a/fXKSW5ynTvnibeLe4O5+ni3krlLgbPZMp458vK69vX08vK0v5bXz5yfX7OMz+vp6V658tP1uTw9PfP8c2pqAVFK1QL8tNZbtdZXlFILMS6op2faLBw4BZwAIrJZn2uFcUZCi8XK27M2cvSMcfwyQb588NSdeNt+EAvzzH+ulttcFoulwGeVGzlxFQmXbzxlE1zSh/HPti7QY+dG5p+T3HwtFi9exHvvjeW5516gV6++9OjxAGfOGCcMBgx4lPDwCAYNGsZ7740FICZmBS1aNObVV9+gS5cH6NHjARITL/HrryuyvN/48V+yb5/mp58Wcu7cWUJDw7j//gd54olBWK3XilxaWhrffDObX35ZxLlzZ6lQoQJ9+jxGbOx5pkz5LwsW/ERERIVbfl6LxXrLzzt16iSmT5/Me+99TNu27QFo3bopnTt35ZFHuvPllxPReg9eXt40b96Cp59+9oZjnzhxnGnT/seGDetITLxEhQoVue+++3n00f54e+f/r+XsZiS0WCy3/HnINCNh9u+bL+mcVx0Yq5RqjTG6eAiYBHyklKoJHAb6AtO01keVUklKqVZa69VAf2CJWcFdYeehOD79dpt9+YXeDah3W4iJiUR+ya543Gx9YdOr16MsXvwzBw7s46GHulG1ajUiI6Po2fNRFiz4hipVqnLXXfcQGRl10/f56quJHDt2hA4dOhEUFMQff/zG5Mlf4eHhwYABg+zbvf76v4iJWU6NGpE88kgPTp06wbhxb1OhQsWC/qh2+/bt5ZlnhlG/fkMeeaQHu3fvYtmypezbp/n66+/w9DT+6NN6L88//xTJycm0bduB8PAItm/fwqRJX7B16xY+/PAzvLy8XJY7L8y+iL5YKdUc2IIx6ojWWs9TSp0HogE/YDHwnW2XfsBkpVQwsBmYYELsApeWbuGlr9bwT6Lxy6R6hWBe7d8ET2lDYrrVO06zavvpW2+YBx/M3Zyn/VvXj6DV7RG33jAXNm/ehMUyKdvXunR5IMe/6nv16sv+/fs4cGAfDz/c3X59pFevIBYs+IaqVavlav6PkyePM33611SqZMyQ2aNHHx59tBs//rjQXkBWrPiTmJjltGnTnrffHmf/Cz46+ls+++xDhz+zsw4ePMCIEc/Tp09/wLhmOWrUs6xfv5YtWzbRpEkzrFYr7777BikpqXz11TRq1br2oO/EiZ8yf/7X/PjjQrp16+my3Hlh9ggErfWbwJvXrfsTaJDNttuA5i4JZpK1u87wv0W77ctjBjSleoVgExOJ4mzLlk1s2bIp29caNWpy09NC+aFdu4724gEQEVGBatWqc+DAPpKTk/H19WXJkp8BGDFiZJbTP4880oPo6PkcO3a0QDNm8PX1pXfvazcDeHh40KJFS9avX8vx40dp0qQZu3bt5NChg3Tr1jNL8QAYMuRpFi5cwOLFi6SACMdcTU7jmc9i7MuNo8J45pF60vzQzbS6PX/+uh80blmOr73SL+fbZl1t8OBhuboLq6BUrlzlhnWBgcY5+dTUVHx9fdm7dzelSpWiYsWsN5V4enpSr159lxWQ8PAISpQokeVaQ8mSRtaUlFQAtN4DwMmTJ5k69caRXUBAAAcO7MNqtRaKn30pIG5g6cbjfPPHfvvyu0PvICJEmh8K4ePjk+NrVlun6YsXL1KlStVstwkJcd2EaSVK3Jj1WhEwsiYmGhet161bw7p1a27YPsPVq1cICHD/3wFSQEyUcCWFkRNW2Zc7NKpI/3vVTfYQRUVwSZ8c78ISjilZMpDLl298tgTI9pkTM/n7BwAwevS/6dr1IZPT5J0UEJMsjDnIz2uuDa0/Ht6SssHS/LC4yHyrbna3WBYF2Z2CKYjTMkrVYuPG9cTGxhIamnXEsXv3zhz2MkfNmpEA7N2754YCkpaWxldfTSQiIoIePfqYEc9h7vnESxEW+89VBo1bZi8eD7e5jWmjO0rxEEWOl5fx92lqaqp9XcZF7szr8qpLlwewWq18+eV/SE+/9gjZb78tZs+e3TfZ0/UaNGhERERFfvnlR3bu3J7ltTlzZjB//ly03mtSOsfJCMSFpi/ew1+ZbgGd8HwbAv1LmJhIiIITFhYGwOefj6dp0+YMGjSMUqVK4+Pjw+bNm5g48VPatu1IgwYN83ScTp3u5bffFvP770s4fPggjRs35cSJ46xZs4rSpUtz8eJF+zMYtzJnzowcGyp2796LDh065Smrl5cXY8aM5cUXn+WZZ4bSpk07KlSohNZ72LRpAxERFXnyyRF5OoYrSQFxgRPnE3l96nr7cv97FR0aue4BJyHM0K1bL3bs2Ma2bVs5cuQwffo8RkBAAC+88DJTpkzi+++/o2TJwDwXEA8PD95990NmzJjK778vYeHCBVSsWJl///stVq2KYdmypfj55W6Ef+zY0Rzv2sp46jyvGjRoyP/+N5OZM6eyadMG1qxZRVhYOXr06MOAAQMpW7bwPCzskXEnQxFXDTjs6lYmVquVzxZsY+cho6mct5cnE59vg69P/jxlWthbhrhabnOdOXOU8PDs7+opCO58DcRds2XOdfbsGQIDA+23zGY2YsQw9u7dzdKlf7nkttjC8PXKkJvv80ytTG4Djtzwev5FFJkdOPEPgz9Ybi8ewx+ux/9eap9vxUMIYZg7dyb33tv+hgced+7cwfbtW2nUqEmheKaiMJJTWPnMYrHy5vQNnDifCEBoKT/eG9bC3vxQCJG/7r//IRYt+oGXXx5Ju3YdCQsrx6lTJ/nrr5UEBATwzDMjzY5YZEkByUfbD8YyfsG1Oyte6tOQ2tXKmphIiKJPqVpMmjSd2bNnsHnzRi5ciKd06TLcddfdPPHEkBueUBf5RwpIPkhNS2fUF2tIvGrcmhhZqRSv9GsszQ+FcJGoqFq8/fY4s2MUO1JA8mj1jtNM/WWPffn1J5pSLVyaHwohij4pIE66vvlhs1rleOqhunKxTghRbEgBccJv648xf9kB+/L7w1pQvmyAiYlEfiss3VCFcEZ+Pb4hBSQHOU05mlmnppXo2+nmM6qJwsfT04v09PQCmVpUCHeQnp6Op2feHymQn5Ac3Kp4fDqiFaUDfV2URriSr68/SUmXCQwsZXYUIQpEUtJlfH398/w+UkCcMG10R7MjiAJUsmQw8fFnAfDzK4mXl5eczhKFntVqJT09naSky1y5comyZcvn+T2lgAhxHW/vEpQtW57LlxOIjz+DxZJ+653ywNPTE4vF/dpfgPtmk1yOycjl6emFr68/ZcuWx9s7741cTS8gSqk3gF62xV+01i8rpe4BPgK8gM3AEK11ilKqITAFCAZigKe01mkmxBZFnLd3CUqVck1TO3ftHQbum01yOaagcpnaX0Mp1Qm4B2gENASaKKUeAaYCfbTW9YAAYIBtlznACK11FOABDHV5aCGEEID5zRRPA6O01ila61RgD1AFY+QRrJTyAvyAq0qpqoC/1nqtbd8ZQM+CCpbT1KIy5agQQhhMPYWltd6V8d9KqUiMU1mtgKPACiABOAx8BzTGKDgZTgMF1uQm85Sj7josFUIIM5l+DQRAKVUX+AV4CbgEjAPqYRSPT23/+xrI/PSLB+DQ1SpbX3unhIUFOb1vQZJcjpFcjnPXbJLLMQWRy/QCopRqBUQDI7XW85RSPYGdWuuDttcnA98CHwIRmXYNB045cixXTyhV0CSXYySX49w1m+RyjLO5Mk0olf3reQmVV0qpysAPQF+t9Tzb6p1Ac6VUxk3KDwEbtNZHgSRbwQHoDyxxZV4hhBDXmD0CeRHjIvmnSqmMdf8F/g0sV0qlAQeAYbbX+gGTlVLBGLf3TnBtXCGEEBnMvoj+PPB8Di/PzGb7bUBzJw7lBcZwzFl52bcgSS7HSC7HuWs2yeUYZ3Jl2ifbxlke+dWV0c21Bv4yO4QQQhRSbYBV168sLgXEF2iGcetvwfalEEKIosML4+alDUDy9S8WlwIihBAin5n9JLoQQohCSgqIEEIIp0gBEUII4RQpIEIIIZwiBUQIIYRTpIAIIYRwihQQIYQQTjG7F5apbD211gBdtdZHlFJ3Ap8BQcB24PGbTaWrlKqCMUtiOUAD/bTWifmZC6gDvJfp5YrAOq11VzNz2b5eDk097MJcTwAvYzw0ugxj0rI0E3JlN11zJ4zpCfyB+VrrMbZtXZYtu1y29SWAX4G3tdYr3CGXUmoY8BzGVA4bgSdd/T2WQ66ngREY00r8ArystbaanSvTayOAHlrr9rblAslVbEcgSqk7MB7Nj7ItBwMLgWFa67q2zQbb/j+nqXS/BL7UWtfC+Ob+d37n0lov1lo31Fo3BO7DmGTr/8zOZePo1MMFnksZXTnfAe7SWt8OlMD4BeTqXNlN1/woMA2jw3RtoJlSqrMrs+U0jbTt67YCaHndLmbmegVjjqCWQH2M31fPuEGu/wNewOjLd7st391ukOsR22t1gNHX7VIguYptAcH4Aj7DtTlF7gb+1lpvty0/C3yf01S6tr/W2mLMlmhfXwC5MvsI+K/Wer+b5Mr11MMuzFUf498xY/bKn4GHTciV3XTNUcB+rfVhrXUaxg91Txdny2ka6cEY31/rMjZ0g1x+wHCtdYLW2grsAKq4QS4LUEdrfRkoDZQCLrpBripKKV9gEvB6xoYFmavYnsLSWg8ByNRGviaQqJSaB9QCVgOjMCp8dlPphgIJtl8Emdfndy5sy5FAe2CIbVUFN8g1nNxPPeyqXNswpgeojFFUemBMPubqr1d20zVPzCGDy7LlNI201nq/bd3ITJu7U64wjFNGT7hJrlSl1FDgY2A9sBVoYnYu4H2MUe7hTJsX2NerOI9ArucN3Av8C+MboSTGMNCT7KfSvX49ODjFroOGYQw1MxqamZpLKRXOtamHI4C1GOf2Tc2ltd6H8e/2E0YH5u1Ailm5bNM1L8U4FXMolxkKPFvmXBm/pLPhFrmUUhWBP4GptmszbpFLaz0ZCAHOAG+anQuoBlTRWk+/brMCyyUF5JozwFrb6YV0jGl0mwMnyH4q3XNAKdvpG2zbODTFroMeBuZlWjY7VxtsUw9rrS3AZIwRkqm5lFJ+wHqtdSOtdUvgJHDQjFy22TP/BEZrrWfeJINLs2WTKyem51JK1cK4QWKm1vptd8illKqcMTOq7a/3eRinTs3+ej0K1FVKbcW4YN5UKTW/IHNJAbnmd4wLUZVty12BTTqHqXRt5x3/Anrb1g+ggKbYVUqFYpzDtA9L3SCXQ1MPuzBXSeBPpVSQUsoH41rWfFfnUtlP17zOeEnVtP3Q9rVlcFm2HHJly+xcSqkgjJ/LMVrrT9wlF8Y1j7lKqdJKKQ+M06SrzM6ltR6kta5tu+FmCLBRa927IHMV22sg19NaH1dKPQkssv0VuxVjyl3IeSrd4cBMpdQY4BjGXwAFoTrGXxHXMy2X1nqPUsrRqYddkStOKTUW45RaCeBrrfXXJuTKabrmJ4Bo22uLuXYB01XZss2ltf5vDtubmWs+UB4YpZQaZVv3k9b6dZNz/RfjWsMaIA3jl3BGgStW/44yH4gQQginyCksIYQQTpECIoQQwilSQIQQQjhFCogQQginSAERQgjhFCkgQriIUipQKXVQKWVVSvW6yXaTbNtc3xBPCLciBUQIF9FGm+yBGO0ivlBKlbt+G6VUN4znaVYCH7o2oRCOkQIihAtprWOA/2A0svsq82u2p4unABeA/rYWMUK4LSkgQrjeq8BeoJtSqi+ArbXJ10AZjEmTjpuYT4hckSfRhTCBUqo5RiuMi0BdjBYn44AZWuuB5iUTIvdkBCKECbTW64EPMNqBz8FoB34Qo/mjEIWCjECEMImtW/AGjFbgaUBrrfW6m+8lhPuQEYgQJtFap2B0RgVj0qtYE+MI4TApIEKYRCn1IMa1j1ggAKOttvxMikJDvlmFMIFtSuApQBLQFtiCMaf1qJvtJ4Q7kQIihIvZZrGbDoQBr2mt92CMRFKAt5VS9UyMJ0SuSQERwvWeBe4DYoDxAFrr7cDbgC8wSylVwrR0QuSSFBAhXEgpVRfj9t1LwOPXPW0+DtgINAL+bUI8IRwit/EK4SJKKV9gPcZtu0O01lOz2aYOxp1ZXkBLrfUG16YUIvdkBCKE67yHUTx+zq54AGitdwOvA94Yp7L8XJhPCIfICEQIIYRTZAQihBDCKVJAhBBCOEUKiBBCCKdIARFCCOEUKSBCCCGcIgVECCGEU6SACCGEcIoUECGEEE6RAiKEEMIpUkCEEEI45f8BDMrbuJbtFy0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "predict = model(Variable(x_train))\n",
    "predict = predict.data.numpy()\n",
    "plt.plot(x_train.numpy(), y_train.numpy(), 'ro', label='Original data')\n",
    "plt.plot(x_train.numpy(), predict, 'b-s', label='Fitting Line')\n",
    "plt.xlabel('X', fontsize= 20)\n",
    "plt.ylabel('y', fontsize= 20)\n",
    "plt.legend( fontsize= 20)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Have a try"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:22:22.909977Z",
     "start_time": "2024-08-03T02:22:22.905309Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "x_train = np.array([[3.3], [4.4], [5.5], [6.71], [6.93], [4.168],\n",
    "                    [9.779], [6.182], [7.59], [2.167], [7.042],\n",
    "                    [10.791], [5.313], [7.997], [3.1]], dtype=np.float32)\n",
    "\n",
    "y_train = np.array([[1.7], [2.76], [2.09], [3.19], [1.694], [1.573],\n",
    "                    [3.366], [2.596], [2.53], [1.221], [2.827],\n",
    "                    [3.465], [1.65], [2.904], [1.3]], dtype=np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Classification\n",
    "\n",
    "<img src= 'img/neuralnetwork/softmax-regression-scalargraph.png' width= \"500px\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Activation Function\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:24:23.921276Z",
     "start_time": "2024-08-03T02:24:23.715009Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAESCAYAAAABl4lHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAofUlEQVR4nO3deZxcZZ3v8U9V753O0iSdhKyEhPxIgpBASILsElQWF2TRAR2XEeRe0Rkd9eVcHcVtFmdGHUeu4xVcRgYdBFwJKEhYQzYkCaTDjyQkhCzd6XTW3peq+8epDpWik67q7qpTXf19v155VZ1znqr6cujuX53nnPM8kXg8joiISDqiYQcQEZGhQ0VDRETSpqIhIiJpU9EQEZG0qWiIiEjaisMOkEVlwLnAHqA75CwiIkNFEXAysAZoT91YyEXjXOCpsEOIiAxRFwJPp64s5KKxB+DAgWZisczvRRk7torGxqZBDzVYlG9glG/g8j2j8vVPNBqhunoEJP6GpirkotENEIvF+1U0el6bz5RvYJRv4PI9o/INSK/d+joRLiIiaVPREBGRtKloiIhI2kI/p2Fmo4AVwNXuvj1l23zgTmAU8CRwq7t35TqjiIgEQj3SMLPFBJd0zT5Ok7uB29x9NhABbs5VNhEReaOwu6duBj4O7E7dYGbTgQp3X5lY9RPg+txFExEZOuLxOPF4nFjiX7aE2j3l7h8FMLPeNk/i2OuE9wBTchBLRApUV3eMto5u2tq7aOvoprUjeGzr6Ka1ved58NjVFaM7Hqe7O053LEZ3rOd5nO7uxHLS867E9lg8WBePQyweJx6HOInH+OuPRCLEYj1/6I/dFgeCv/txkv/+97xXsADHKw3FRVE+d+MCZk0ePej7MPRzGicQ5dh9EgFimb7J2LFV/Q5QUzOy36/NBeUbGOUbuHzI2NkVY++BFvbsaw7+NTZT39hCc1snrW1dtLR30treRUtbF51d6f0JKSmOUlIcpSgapbgoQlFR4rGX5bKyEoqKIhQXRSmKRoLn0SjRaIRIBCKR4DEaiRzznKPrjt0WiUSIAJFIkKVnfY+e7cEC9Cwl3hKA0pIi3jR7PFWVpYOxi4+Rz0VjJ8H4Jz0m0ks3Vl8aG5v6dQNNTc1IGhqOZPy6XFG+gVG+gctlxvbObhoOtLL3YCt7jz62sPdAK42H2475Nl5WWkTN6AqqR5UzqrKE8dXllJcWU1FaRHlpEeVlxZSXFlFRWkx5WRHlpanLRRRFs99zn+3919rcTmvzG4aO6lM0Gjnhl+28LRru/qqZtZnZ+e7+DPAB4KGwc4lIdtXvb+HPLzewp7ElKAwHWznY1HFMmxHlxYyvrmTW5NGcN28i46srmFBdyfjqCkZWlhCJRIZE4R2K8q5omNky4Evuvha4Cfhh4rLcPwPfDTWciGRFU2snqzfV8+yLdWzdfRiA0SNKGV9dwbwZJzG+upLxYyoYXx38G1FeEnLi4Ssvioa7n5L0/Mqk5+uBRWFkEpHs6uyKsX7LPp7dWMeGrY10x+JMqRnB9ZfOZMnciVSPLAs7ovQiL4qGiAwP8XiczTsP8ezGOtZs2ktLexejq0pZunAK582byLQJ4Z9YlxNT0RCRrKvf38KKF+t4dmMd+w61UVoS5ZzZNZx3xkTmTj+JaDTS95tIXlDREJGsONLSwepNe3l2Yx2v7D5MJAJzp1fz7gtncPbsGspL9ednKNL/NREZVAeOtPPfj7zM+i37Eucpqrjh0lksnjtB5ykKgIqGiAyaXQ1NfOve9bS0d3H5wqmcd8ZEpo7v/w22kn9UNERkUPiOA3z3/hcoLYnydzedrZPaBUpFQ0QGbPWmeu78fS01Yyr41A1nMW50RdiRJEtUNERkQP6wegf/89gWTpsymk9ceyZVFbrxrpCpaIhIv8RicX7+6GYeWfsa51gNt7xjLiXFRWHHkixT0RCRjHV2dfPNu9fyzPrdLD1nCu+77DTdazFMqGiISEaa2zr5j/tf4OXXDnLDpbN426KpRCIqGMOFioaIpK3xUBvfuncdDQdb+ez7z2HOlMGf5EfyW9jTvYrIELGj/ghf/9laDjZ18Okb5nPRAk2kORzpSENE+rRx+37ueOAFKsqK+bv3n82UGt2wN1ypaIjICT37Yh0/WraJk8dW8jfXn8VJo8rDjiQhUtEQkV7F43GWrXyV+594hdOnjeG295xJZbn+ZAx3+gkQkTeIxeL89yMvs/z5XSyeO4GPXDmHkmKdAhUVDRFJ0dUd4/u/fpHnN+/j7Yuncd0lM4nqklpJUNEQkWM8unYnz2/ex19cdhqXnzs17DiSZ3S8KSJHtbR18uCz2zljxkkqGNIrFQ0ROerBla/S0tbFdZfMDDuK5CkVDREBYP/hNh5du5Ml8yZoLgw5LhUNEQHg109vIx6Pc82Fp4YdRfKYioaIsKuhiWde2MNbzp7CuDGaQEmOT0VDRLj/iVcoLy3iqvOmhx1F8pyKhsgw9/JrB1m3ZR9XLpnOyMrSsONInlPREBnG4vE4v1y+hTFVpSxdqEtspW8qGiLD2J9f3sfW3Yd594WnUlaiqVqlbyoaIsNUdyzG/U9s5eSxlZz/polhx5EhQkVDZJh6av0e6va3cN3FMymK6k+BpEc/KSLDUHtHN795ehuzpoxm/mnjwo4jQ0joAxaa2Y3AF4ES4DvufkfK9rOBHwClwGvA+939YK5zihSSP67ZwaHmDj5+zZuIaARbyUCoRxpmNhn4BnABMB+4xczmpjT7d+BL7n4W4MBnchpSpMAcbungoVU7WHDaOGZNGR12HBliwu6eWgo85u773b0ZuA+4LqVNETAq8bwSaM1hPpGC8/tnttPe2c21F2tQQslc2N1Tk4A9Sct7gEUpbT4N/NHMvgM0A4sz+YCxY6v6Ha6mJr8HbVO+gRmO+eoam3l83S7eung6Z80Z+BVTw3EfDqZ8z9ebsItGFIgnLUeAWM+CmVUAdwFL3X21mX0a+C/gqnQ/oLGxiVgs3nfDFDU1I2loOJLx63JF+QZmuOa787cbiUYivPWcKQN+/+G6DwdLvuaLRiMn/LIddvfUTuDkpOWJwO6k5TOAVndfnVj+AXBJbqKJFJZX646wqraey8+dSvXIsrDjyBAVdtF4FLjMzGrMrBK4Fng4afsWYKqZWWL5XcCaHGcUKQi/fHwLVRUlXLFYgxJK/4VaNNx9F/AFYDmwDrgn0Q21zMwWuvsB4EPAvWa2AfgI8OGw8ooMVRu37ad2+wGufvMpVJaH3SstQ1noPz3ufg9wT8q6K5OePwQ8lOtcIoUiFo/zy8e3MG50OZcumBx2HBniwu6eEpEsW11bz476Jq656FRKivUrLwOjnyCRAtbZFeOBJ19h2vgqFs+dEHYcKQAqGiIF7PHnd7HvUBvXXTqTqIYLkUGgoiFSoFrauvjdiu3MPaWaM2aMDTuOFAgVDZEC9fDqV2lq7eS6SzRciAweFQ2RAnTgSDt/XP0ai+dO4JSJo/p+gUiaVDRECtBvn9lGdyzONRedGnYUKTAqGiIFpv5AC0+t38OlCyYzfkxF2HGkwKhoiBSYFS/UESfOFUs0XIgMPhUNkQISj8dZVVvPnOnVGpRQskJFQ6SAbK87wt6DrSyeoxv5JDtUNEQKyMqN9RQXRTjHasKOIgVKRUOkQMRicVZvqufMmeOoLC8JO44UKBUNkQLx0o4DHGruYInGmJIsUtEQKRCrauspLy3izJkaMkSyR0VDpAB0dsVY6w2cPbuG0pKisONIAVPRECkAL7zSSGt7l7qmJOtUNEQKwMraekZWljDnlOqwo0iBU9EQGeJa27tYv2Uf554+nqKofqUlu/QTJjLEPb+5gc6uGEvmTgw7igwDKhoiQ9zK2nrGjS5n5mQNgS7Zp6IhMoQdbu6gdtsBFs+dQETTuUoOFJ9oo5n9qJ/vG3f3v+rna0UkTWte2kssHmexrpqSHDlh0QA+1Mu6eOKxt6818cT6OKCiIZJlqzbVM7lmBFNqqsKOIsNEX91TC1L+XQbUA5uA9wHTgCpgIvAO4DlgCzAvS3lFJGHfwVa27DykezMkp054pOHu65OXzewuoAu42N0bkza1AA+a2VPA88BXgRsGOauIJFm1qR6ARRoGXXIo0xPh1wC/SSkYR7n7YeB3wNsGGkxETmxVbT2zJo+mRlO6Sg5lWjTiwJg+2kwB2vqVRkTSsrOhiZ0NzToBLjmXadF4GrjezC7ubaOZvQd4N/DHAeYSkRNYVVtPNBJh4enjw44iw0xfV0+l+iJwKfCImT1EcOL7CDAaOB94C8GJ8i8MZkgReV3PPOBzT6lm9IjSsOPIMJNR0XD3F8zsIuDfCa6WekfS5jjwMHCbu+9I9z3N7EaCYlQCfMfd70jZbsAPgGqgDnifux/IJLdIIdm6+zD7DrXxrgtmhB1FhqGM7wh393XufjEwGbgSuCnxOMndr3L3bem+l5lNBr4BXADMB24xs7lJ2yPAb4F/cvezCK7M+nymmUUKyaraekqKo5w9W/OAS+5l2j11lLvvAfYM8POXAo+5+34AM7sPuI7gkl2As4Fmd384sfwP9H0iXqRgdcdirNlUz1kzx1JR1u9fX5F+62sYkQeAX7j7vUnL6Yi7+7VptJvEsYVnD7AoaXkWUJe4P2QBwU2Fn0gzg0jB2fTqAQ63dLJYI9pKSPr6qvJuYF3KcjrifTcBgu6x5LYRIJa0XAxcAlzk7mvN7GvAt+h9eJNejR3b/+EVampG9vu1uaB8AzMU8617dDMjyot5y+LpeTGt61Dch/kk3/P1pq+iMQM4mLI8mHYCFyYtTwR2Jy3XAZvdfW1i+efAfZl8QGNjE7FYujXsdTU1I2loOJLx63JF+QZmKObr6OxmxYbdLDx9PIcOtoSU7HVDcR/mk3zNF41GTvhlu69hRF490fIgeBS43cxqgGbgWuCWpO0rgBozOysxpEnP+FYiw86GrY20dXTrhj4JVb/OpJnZKcBfAmcBlUAjsJHg/EfaV0+5+y4z+wKwHCgF7nT31Wa2DPhSokvqGuCHZjaC4MjkA/3JLDLUraqtZ9SIUuZM0zzgEp6Mi4aZ3Upwn0ZJL5u/bGZ/7e4/SPf93P0e4J6UdVcmPV/FsSfHRYadlrZO1m9t5JIFk4hGNdmShCej+zTMbClwB7APuI3g3orxgBF0K+0F7kjcACgig+S5lxvo6tY84BK+TI80PgccAs539+1J6/cBm83sTwTnHD4LPDkoCUWE1bX1jB9TwYyTh97VNlJYMr0jfBHB0Ojbe9uYOJ/xG2DJAHOJSMKhpnZqXz3AIs0DLnkg06JRCjT10aaJ4OS4iAyC1S/tJR5HM/RJXsi0aLwEvN3Mep31xcwqgSsAH2gwEQmsqq1n2vgqJo0bEXYUkYyLxg+BmcB9ZjY9eUNioMHfENwAeNfgxBMZ3vYebOWV3YdZPE9HGZIfMj0R/p8E82lcB7xiZrsIToxPIhhIMALcnzq8uYj0z6raxDzgp6toSH7I6EjD3ePAe4EPAo8TnLuwxObHgQ+6+/WDmE9k2IrH46zcWMfsKaMZO7o87DgiQD9u7ksUjp8l/olIlry2t4k9jS0sfZv13VgkR/o9IL+ZlQFlx9vu7of7+94iEnRNFUUjLDRNtiT5I6OiYWblBDPt3UhwJ/jxxDN9bxF5XSwWZ/WmeubNOImRlZoHXPJHpn/Yv0kwfMhhYDXQNuiJRIRN2/fTeLid91w8M+woIsfItGi8B3iRYBiR/BsIXqRAPPH8TkqLoyw4bVzYUUSOkel9GqOBZSoYItnT1R3j6XW7mX/aOMpL1csr+SXTovEMwVzdIpIltdsPcKSlQ5MtSV7KtGh8GjjXzP7ZzE50IlxE+mlVbR1VFSW86dSxYUcReYNMj31fAu4HPgN8xszagPZe2sXdXT/xIhlq7+zmzy/v45JzplBclOl3OpHsy7RofA34CMFwIfsI5vUWkUGyfss+2ju7uXjBlLCjiPQq06LxYeBV4O3u/nIW8ogMa6tq66keWcbcU8eyv7GvWQhEcq8/V0/dr4IhMvia2zrZsLWRc08fT5HmAZc8lWnR+DPB0OciMsie8wa6Y3GWaBh0yWOZFo0vAFeb2SfMTBeQiwyilRvrmHBSJdMnaB5wyV+Z/uF/P7AZ+A7wz2b2Gr2fDI+7+zkDzCYybBw40o7vOMg7L5ihecAlr2VaND6a9LwcOO047eL9iyMyPK3ZVE8cdEOf5L2Mioa768JxkSxYWVvP9IkjmXhSZdhRRE5IRUAkZHX7W9hed4QlOsqQISDT+TQuSqNZN9AC7HT3hn6lEhlGVtXWEwEWzVHRkPyX6TmNx8ngfIWZbQRuc/cnM/wckWEhHo+zqrYemzaG6pHHnQhTJG9k2j31IWAbwTAijwJfAT4GfJZgTKpuguFF/gW4G5gOPGxm8wcnrkhh2VHfRN3+Fp0AlyEj0yONCcBU4Cp3fyh1o5ldCvwB2O3unzezmcBzwOeB9w00rEihWVlbR1E0wjkaNFqGiEyPNG4F7u2tYAC4+3KCI45PJpa3Ag8A6ZwLERlWYrGga+pNp46lqqIk7Dgiacm0aEwk6H46kXpgctLyHmDM8Rqb2Y1mVmtmm83s4ydod5WZbcskrEg+e/m1gxxs6tCwITKkZFo0tgJXmVmvF5ObWQVwBbA9afUsgsLRW/vJwDeAC4D5wC1mNreXdhOAfyU4lyJSEFZtqqespIizZmkecBk6Mi0adxAUgT+a2cU9xcPMisxsEfDbxPYfJtb/JfBO4LHjvN9S4DF33+/uzcB9wHW9tLuT4KS7SEHo6o6x9qW9LJg9jrKSorDjiKQt0zvCf2BmpxOcs3gMwMyaCYYUKSI4EvhP4NtmVg38BDgA/ONx3nISxx6F7AEWJTcws08SjK67MpOsPcaOrerPywCoqcnvgeOUb2DCzLd6Yx3NbV287bwZx82R7/sP8j+j8g2+jEeqdfdPmdndwAeBhcA44DDBVVI/dveVAGZWSnAp7i/cfddx3i7Ksfd9RIBYz4KZnQFcC1wG9Gsqs8bGJmKxzIfCqqkZSUPDkf58ZE4o38CEne8Pz26jqqKEydXlveYIO1868j2j8vVPNBo54Zftfg1v7u7PERSJE7WpB/6tj7faCVyYtDwR2J20fD1wMrAWKAUmmdlT7p78GpEhpa2ji3Vb9nH+GSdrHnAZck5YNMzsTKDO3fcmLafF3Tek0exR4HYzqyEYYv1a4Jak9/gy8OXEZ58CPK6CIUPdus376OiM6YY+GZL6OtJYB9wOfDVpOd2+nj7P7rn7LjP7ArCc4EjiTndfbWbLgC+5+9o0P0tkyFhZW89Jo8qYNWV02FFEMtZX0fgpQaHo8V8M8lwZ7n4PcE/Kuit7abcdOGUwP1sk1460dLBx237eeu5UoppsSYagExYNd/9wyvKHemtnZkUEc4fXuXvToKUTKTBrE/OAq2tKhqqMz8KZ2UVm9otEoeg5z7ENcGCvmX15kDOKFIxVtfWcPLaSqeP7fym4SJgyKhpm9hbgTwRXNU1NrL6T4HLY5QR3gn/JzN4/iBlFCsL+w228/NpBlsydoHnAZcjK9Ejjc8ARYJG7bzezOQT3avzB3ZcSDAXyEnDcMaREhqtVm+oBzQMuQ1umReNcgpv1eu7RuJrgxPi9AO7eATwMzBu0hCIFYtXGek6dNIrx1ZoHXIauTItGOXAoafmKxOMjKe/ZNZBQIoVm975mduxtYrGmdJUhLtOisQVYDGBmU4DzgY3uvjOxrhS4imA0XBFJWFVbTyQC587RZEsytGVaNB4ALjGz5cATBJfs/hiC+S6AZ4GZJEa5FZHX5wGfM72aMVWaB1yGtkzHnvo6wfhQNxMMLvg/wHcT294MnAV8CxUNkaO27TnC3oOtXPXm6WFHERmwTIdG7wb+l5l9Doi6e/L5jR8C300MVCgiCStr6yguinLObHVNydDX31Fu3zCeb2KYDxFJEovFWbNpL2fOHEtleb9+3UTyisZlFsmil3Yc4FBzB0t0b4YUCBUNkSxaWVtPeWkRZ84cG3YUkUGhoiGSJZ1dMZ7zBs6ZXUOp5gGXAqGiIZIlL7zSSGt7l4YNkYKioiGSJStr6xlZWcKcU6rDjiIyaFQ0RLKgtb2L9Vv2sej0CRRF9WsmhUM/zSJZ8OeXG+jsirF4nrqmpLCoaIhkwaraesaNLmfmpFFhRxEZVCoaIoPscHMHtdsPsFiTLUkBUtEQGWSPrH2NWDzOknkTw44iMuhUNEQG0YEj7Tyy5jWWzJvA5HEjwo4jMuhUNEQG0W+efoVYPM41F54adhSRrFDREBkku/Y189SGPVy6YAo1YyrCjiOSFSoaIoPkgSe2Ul5axNWaN0MKmIqGyCDYvPMgz2/exxWLpzOysjTsOCJZo6IhMkDxeJxfLt/K6KpSLl84New4IlmloiEyQM9v3seWXYd49wUzKCvVaLZS2FQ0RAagOxbj/ie2cvLYSi448+Sw44hknYqGyAA8vWEPexpbuPbimRqYUIaF0CctNrMbgS8CJcB33P2OlO3vAr4CRIBtwIfd/UDOg4qkaO/s5tdPb2PW5NEsOG1c2HFEciLUr0ZmNhn4BnABMB+4xczmJm0fBXwfuMrdzwI2ALfnPqnIGz2y5jUONXVw3SUzNcaUDBthH08vBR5z9/3u3gzcB1yXtL0E+Li770osbwCm5TijyBscaengoVWvMn/WOGZPHRN2HJGcCbt7ahKwJ2l5D7CoZ8HdG4FfAZhZBfB54D9yGVCkN79f8SptHd1ce8nMsKOI5FTYRSMKxJOWI0AstZGZjSYoHuvd/aeZfMDYsVX9DldTM7Lfr80F5RuY/uara2xm+fM7WXruNObPyd5Itvm+/yD/Myrf4Au7aOwELkxangjsTm5gZicDfwAeAz6V6Qc0NjYRi8X7bpiipmYkDQ1HMn5drijfwAwk312/20gkEuHt507N2n9jvu8/yP+Mytc/0WjkhF+2wy4ajwK3m1kN0AxcC9zSs9HMioDfAfe6+9fDiSjyulfrjrByYz1XLplO9ciysOOI5FyoRcPdd5nZF4DlQClwp7uvNrNlwJeAqcDZQLGZ9ZwgX+vuHw0nsQx39z2xlRHlxVy5RNdjyPAU9pEG7n4PcE/KuisTT9cS/hVeIgBs3L6fjdv28963zKKyvCTsOCKh0B9kkTTE4nHuW76VsaPKecvZU8KOIxIaFQ2RNKzZtJdX64/wnotOpaRYvzYyfOmnX6QPXd0xHnhyK1PHV7F43oSw44iESkVDpA+PP7+LhoNtXH/JTKIaLkSGORUNkRNobe/it89sZ870aubNOCnsOCKhU9EQOYGHV+2gqbVTgxKKJKhoiBzHwaZ2/rBmB4vmjGfGyaPCjiOSF1Q0RI7jt89sp7s7znsuOjXsKCJ5Q0VDpBd7Gpt5ct1uLpk/mfHVlWHHEckbKhoivXjgyVcoKYnyjvNPCTuKSF5R0RBJsfalvTznDVyxaBqjRpSGHUckr4Q+9pRIPnl07Wv8/NHNzJw0irct0qCEIqlUNERIjC31+FYeXrWDBaeN45Z3zqOspCjsWCJ5R0VDhr3Orhg/WraJVbX1XHr2ZG5aOptoVPdkiPRGRUOGtZa2Tr73wAu8tOMg110ykysWT9NNfCInoKIhw9b+w218+5frqWts4ear53LeGdmb71ukUKhoyLD06p7DfONnz9Ha3sWnbjiLuadoXCmRdKhoyLCz6dUD/N9fvUBJcZTP33Q20yaMDDuSyJChoiHDyqraeu56sJaTx1Xxyfe8ibGjy8OOJDKkqGjIsBCPx/nD6te4d/kWZk8dw1duOY/W5vawY4kMOSoaUvBisTi/+NNmHn1uJwtPH8/NV8+hqrJURUOkH1Q0pKB1dHbzw9/X8pw3cPnCqbz3slmafU9kAFQ0pGA1tXby3fs3sGXnId73llm8VcOCiAyYioYUpH2HWvn2vetpONjKre+ax6I5E8KOJFIQVDSkoDS1drJ6Uz2/W7Gdzs4Yf/ve+di06rBjiRQMFQ0Z8jq7Yqzfso9nN9axYWsj3bE408ZXcfM75jK5pirseCIFRUVDhqR4PM7mnYd4dmMdazbtpaW9i9FVpSxdOIXz5k3UDXsiWaKiIUNK3f4WVrxYx8qNdew71EZpSZRzZtdw3hkTmTv9JI1OK5JlKhqS9460dLB6015WvFjHtj2HiURg7vRq3n3hDM6eXUN5qX6MRXJFv22Slzq7ulm/pZEVL9bxwivBeYopNVXccOksFs+dQPXIsrAjigxLKhoSqua2TvYeaKX+QAsNB1qD5wdb2dXQRGt7N6OrSrl84VTOO2MiU8frpLZI2EIvGmZ2I/BFoAT4jrvfkbJ9PnAnMAp4ErjV3btynVP6Jx6Pc7ilk70HWtibKAp7D7YeXW5uO/Z/ZfXIMmrGVLBozgQW2njmTK/WeQqRPBJq0TCzycA3gHOAdmCFmS1399qkZncDH3X3lWZ2F3Az8P3cpx2+4vE4Xd0xWtu7aevooq2jm9b24LGto5vWji7akra1dXTRGYOd9UfYe6CV9s7uo+8VicDYUeVMqA4KQ82YCiZUV1BTXUHNmArNyy2S58I+0lgKPObu+wHM7D7gOuCrieXpQIW7r0y0/wnwFbJcNLq6Yzz7wh72H2g+bpt4PGWZN6x4fU389e3Jr4v3rI8fbZa0LR5sjwevjMchlrRuxIgyjjS1HV2OJRr1tIvFoTsWo7s7Tncs+BdLWu6Kxenujh3dduzzOF2x2DGFoDuW8t93HGWlRZSXFjFqRBnVI8uwaWOYUF15tDiMHV1OcVE0rfcSkfwTdtGYBOxJWt4DLOpj+5RMPmDs2Mz7wVds2M0//nRNxq/LJ5EIFEWjFBdFKCqKUhSNHH1eHI1SVBShuCjxGI1SVBSltDRoV1QUpaQ4SkVZMRVlxVSWJx7Liqk4+rzk9eeJx/LS4iHVlVRTk9/3cuR7Psj/jMo3+MIuGlGO/YIdAWIZbO9TY2MTsTS/Jfc47eSR/ODzl7F3X9Mx61P/HPY1WGokEnn9NZGk1yetjxzdFnnDe0ajwdpIJEIkEjxGEy+oqalif2Pz0fXB4+ufGcnhSK6xji6aO7pIPi6rqRlJQ8ORnGXIlPINXL5nVL7+iUYjJ/yyHXbR2AlcmLQ8Edidsv3kE2zPmkk1VZSkdjnlkcryEppL1f8vIrkVdufyo8BlZlZjZpXAtcDDPRvd/VWgzczOT6z6APBQ7mOKiAiEXDTcfRfwBWA5sA64x91Xm9kyM1uYaHYT8G0zewmoAr4bSlgREQm9ewp3vwe4J2XdlUnP13PsyXEREQlJ2N1TIiIyhKhoiIhI2lQ0REQkbaGf08iiImBAN5vl+41qyjcwyjdw+Z5R+TKXlKnXa/oj8dTxMArHBcBTYYcQERmiLgSeTl1ZyEWjDDiXYOiR7j7aiohIoIjgpuo1BAPJHqOQi4aIiAwynQgXEZG0qWiIiEjaVDRERCRtKhoiIpI2FQ0REUmbioaIiKRNRUNERNJWyMOIpM3MvgZ0u/vtieUxwH8DpwINwA3uXpfymgjwL8DVBFPQ3uzuz2Qp33jgj0mrRgM17l6V0m468CKwNbGq3t3flo1MvWT8IPBPQH1i1YPu/oWUNqXAXcBCoBW40d1fylG+84FvA6VAI/CRxCRfyW1yvv/M7Ebgi0AJ8B13vyNl+3zgTmAU8CRwq7t3ZTNTyud/Gbghsfigu3+ul+0fAQ4kVv0w9b8hy/mWA+OBzsSqj7n7qqTt8wlp/5nZR4HbklbNAH7m7rcltQl1//XHsC4aZjYa+BbwF8A3kzZ9HXjK3a8ysw8A/w68N+Xl1wJzgLnALOBBM5uTjR9Id98LzE9kjgJ/Ipi8KtVCgomsPjbYGdKwEPi0u//8BG0+CTS7+xwzuwj4CbAkF+EIvgS80903mNlHCCbzeldKm5zuPzObDHwDOIfgztsVZrbc3WuTmt0NfNTdV5rZXcDNwPdzlG8p8FZgARAHHjaza9z9V0nNFgLvc/dnc5EpJV8EmA1MP8HvXWj7z93vJChYmNk84NfA7SnNQtt//TXcu6feBWwG/i1l/VUEf2QAfg5cYWYlvbT5hbvH3P1lYAfw5myGTfgw0JKYvCrVucAZZrbOzB4zszflIE/yZ3/QzF4ws7vNrLqXNkf3q7s/CdSY2bRsBzOzMuCL7r4hsWoD0Nvn5nr/LQUec/f97t4M3Adcl5R7OlDh7isTq34CXJ/lTMn2AH/r7h3u3gls4o37bSHwf8xsg5l9z8zKc5jPEo9/NLP1ZnbbMRvD33/Jvg/8H3ffl7I+zP3XL8O6aLj7f7n7P/HGsakmEfzCkPgGcxioOV6bhD3AlCxFBcDMigiOMD5/nCZtBN+szgb+Ffh1oksoF/YAXwPOBF4DvtdLm5zvMwB3b3f3u+HokdrtBN/6UuV6//W1P0LZXz3cfWPPH1wzO42gm2pZz3YzqwKeBz5LsM/GAH+fq3xANcFR9zXAZcCtZnZ50vZQ91+PxBFbhbv/MmV92PuvX4ZF95SZXU/Qn53sJXdfepyXpI5XHCE4b5EsSnDIfqI2Gesj69uBze7+Qm+v7Tknk7DMzP6RoAtt/UBzpZmvp803ef28QLKs7LN08yUKwE8Jfu7/IfW1udh/KfraH1nfX+lIdK08CHzW3Tf3rHf3JuDKpHb/BvyI3rtOB12iS+dot06i++lK4JHEqrzYf8DHCLrBjxH2/uuvYVE0EhX+l302fN0uYCKw08yKgZEEJ0+T7SQYCbLHRGD3QHJCn1nfDfzieK81s08Q9Mn3ZI3w+gnCQdFbPjMbbWafcveeP9YRoLc+5p591lNQBmWf9ZUvkbEK+C3B/8d3JbpbUttkff+l2Ekw/HSP1P2RlZ+xTCQuILgf+Bt3/0XKtmnAUnf/UWJVtvdXarYLgDJ3/9NxPj8f9l8pcDHwoV62hbr/+mtYd0+dwDLgLxPP30twUjz1f+Yy4CYzKzKzWQQn5NZkOdd5nHiOkIuBvwIws4sJhjjOxdVJTcDnzGxxYvk24Fe9tDu6XxO/8G3uviMH+SDodtoCvNfd3zDcc0Ku99+jwGVmVmNmlQQXVzzcszFxdVdb4g83wAeAh7KY5xhmNpWgG+/G1IKR0Ap808xmJE5Kf5ze/79nyxjgX8ys3MxGAh9M/vyw91/CmcDLiXNWqcLef/2iotG7vweWmNlG4H8T/M/EzN5pZncm2twHbCQ4qfob4K/cvTXLuU4l+PZ0lJndamZfTSz+NXC5mb1I0Cf/F+6e9cNxd+8m6O/+vpltIrga6HO95PsPoCyxX79L8EucdWa2gOCih/OBPydOdC/rJV9O95+77yLoilgOrCM4ylltZsvMbGGi2U3At83sJaCKYL/lymeAcuBbiX22LrG/lpnZQndvIOh6+R3gBN+UUy8qyRp3/z1Bt9nzwHPAj9z92Tzaf9D772xe7L/+0nwaIiKSNh1piIhI2lQ0REQkbSoaIiKSNhUNERFJm4qGiIikTUVDRETSpqIhIiJpU9EQEZG0qWiIiEjahsWAhSJhM7MLgScIhpSYmxjhtGdAu+cIJvO6xN1PNLaYSOh0pCGSA4li8D1gKvCVpE1fAc4A/lUFQ4YCjT0lkiOJkWxfIJj97hyCwQBXALXAQnfvCDGeSFpUNERyyMwuAR4DngbGEswvf27SVLQieU3dUyI55O6PE8wXfSHBeYy/V8GQoURFQyT37k96viK0FCL9oO4pkRwys3KCSYOmJ1btAOa7e1t4qUTSpyMNkdz6KnB64vEfAEs8FxkSdKQhkiNmtoigO2oTcDbB9J7rCOaXf7O7rw4vnUh6dKQhkgNmVgb8mOB37mPu3pm4xPZjiXU/TrQRyWsqGiK5cTvB1VL/z92PnvxO3ND3o8S220NJJpIBdU+JiEjadKQhIiJpU9EQEZG0qWiIiEjaVDRERCRtKhoiIpI2FQ0REUmbioaIiKRNRUNERNKmoiEiImlT0RARkbT9f6Y34Wm5r7t+AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def sigmoid(x):\n",
    "    return 1/(1 + np.exp(-x))\n",
    "\n",
    "plt.plot(range(-10, 10), [sigmoid(i) for i in range(-10, 10)])\n",
    "plt.xlabel('x', fontsize = 20)\n",
    "plt.ylabel('sigmoid', fontsize = 20);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:25:15.940350Z",
     "start_time": "2024-08-03T02:25:15.781120Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAESCAYAAAAfXrn0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAgt0lEQVR4nO3deXiU5aH38W92IAkEQpCdsN5AUBGQiqKioAgIqW2FWlu3IvCet+f19HQ5p9t1bM/pZnvavn17rgIqonWttRo2FUUUERE3RJJwI8hOAiEsWUlIZt4/ZqAxBpLJzOSe5fe5Li7JM8/M/LxD5pfnfmbuJ8Hr9SIiIvEt0XUAERFxT2UgIiIqAxERURmIiAgqAxERAZJdB2inNOByoARodJxFRCRaJAF9gHeBuqY3RGsZXA686TqEiEiUuhrY2HRDtJZBCcCJE9V4PIF/TiI7O4Py8qqQhwoV5QuO8gVH+YITyfkSExPo3j0d/K+hTUVrGTQCeDzedpXB2ftGMuULjvIFR/mCE+n5aGF6XSeQRUREZSAiIioDERFBZSAiIqgMREQElYGISFQJ12UHVAYiIlHivR1H+cGSzdSdCf3CCyoDEZEocPREDcvWFNM1PZWU5NC/dKsMREQiXEOjhyUrCklMSGDBnNEkJiSE/DlUBiIiEe75DZ+yp6SSu2aMpGe3zmF5DpWBiEgE276nnBff2c+UsX2ZMLJX2J5HZSAiEqFOVdfz0Kpi+vVMZ97U4WF9LpWBiEgE8ni9PLyqiNq6Bhbm55GWkhTW51MZiIhEoLVbDrB9z3G+OnU4/XMywv58KgMRkQizp6SC597YzfgROUwZ27dDnlNlICISQWrrGlhSUEi3jFTunDGShDC8jbQlKgMRkQjh9Xr5y1pL2alaFszOI6NzSoc9t8pARCRCbNpeyubCI+RPHsyIAVkd+twqAxGRCFB6vIbH1+7EDMji5km5Hf78KgMREcfONHhYXLCd5KQE7p09msTEjjlP0JTKQETEsb+9vpv9R6q4Z9YoenTt5CSDykBExKGPdh3jlfcOMHV8fy4bnuMsh8pARMSRE5V1PLy6mAG9Mph73VCnWVQGIiIOeDxeHlpVRH1DI4vy80hJDu9yE61RGYiIOLBm8z6K953g9mkj6JOd7joOya4DGGO+DvzA/+WL1trvuswjIhJuuw6d4oU39zBxVC8mX9LHdRzA8ZGBMaYL8EfgWuBS4GpjzDSXmUREwqnm9BmWFBTSo2sad0zvuOUmWuN6mijJnyEdSPH/qXWaSEQkTLxeL8tfspysqmNhfh5dOjmfnDnHaRlYayuBnwA7gIPAXmCTy0wiIuGy4aPDvLfjKLdcM4Shfbu5jvMZCV6v19mTG2MuAR4FpgOngMeBLdba37Ry11xgT3jTiYiEzv7SCr79hw2Mzu3BTxdMcvIp4yYG4/vl+xzXxyjTgXXW2qMAxpjlwD8BrZUBAOXlVXg8gZdZTk4mZWWVAd+voyhfcJQvOMoXnJby1Z9p5JePvUdaSiJ3TB9BeXmVk2yJiQlkZ7d8oRzXZfAR8IAxJh2oAWYD77qNJCISWs+s38XBsmq+PfdSsjLSXMdpketzBmuBp4D3gW34TiD/ymUmEZFQ+mBnGes/OMT0iQO4eEi26zjn5frIAGvtr4Ffu84hIhJqxytO88iaYgb1zuTL17pdbqI1rt9aKiISkxo9HpauKKTB42VRfh7JSZH9chvZ6UREotTKt/ay8+Ap7rjRcFH3Lq7jtEplICISYnb/CVZu2suVY3ozaUxv13HaRGUgIhJCFdX1LF1ZRE5WZ26/YYTrOG2mMhARCRGv18sfn/mQiup6FuXn0TnN+Xt02kxlICISIus/PMQ7haXcOmUoub27uo4TEJWBiEgIHDhaxdPrdjFh1EXccPkA13ECpjIQEQlSXX0jiwu2k94pmfvmXRYxy1IHQmUgIhKkp9btpLS8hntnjyYrMzKXm2iNykBEJAhbio+w4aMSZk4axOjcHq7jtJvKQESknY6drOXRlyxD+3Ylf/Jg13GCojIQEWmHhkYPS1YUAl4WzIn85SZaE93pRUQcKdi4h92HK7jzppHkZHV2HSdoKgMRkQAV7z3Omrf3cc2lfZg46iLXcUJCZSAiEoCKmnqWriqid3YXbpsaPctNtEZlICLSRl6vl2Wri6mubWBR/hjSUpNcRwoZlYGISBu98t5Btu0uZ971wxjQq+VrCUcrlYGISBvsK63k2fW7GDusJ9eP6+c6TsipDEREWlFb18Digu10TU/lnlmjonK5idaoDEREWvHkKzs5erKWBbNHk9E5xXWcsFAZiIhcwNuFpby1vZTZV+ZiBnZ3HSdsVAYiIudx5EQNj71sGdG/G7OvynUdJ6xUBiIiLWho9LCkoJDkxATunZ1HUmJsv1zG9v+diEg7/f2NT9lbWsldM0aR3a2T6zhhpzIQEWnm40/LeWnLfq67rB/jTY7rOB1CZSAi0sSpqjoeXlVEv5x05l0/zHWcDqMyEBHx83i9PLiqiNP1jSyak0dqSuwsN9EalYGIiN/L7+ynaO8JvjptOP1yYmu5idaoDEREgN2HT/H3DZ8yweRw7aV9XcfpcCoDEYl7NacbWFJQSFZGKnfOGBmTy020RmUgInHN6/Xy2Ms7OF5Rx8I5Y0jvFJvLTbRGZSAicW3jxyVsKT5K/tWDGda/m+s4zqgMRCRulZRX88QrOxk5MItZVwxyHccplYGIxKUzDY0sLigkNTmJe2fnkZgYf+cJmlIZiEhc+uv63Rw4WsU9s0bRPTPNdRznkl0HMMbMBv4DSAfWWmvvcxxJRGLc1k+Ose79g0wb35+xw3q6jhMRnB4ZGGOGAIuBLwKXAOOMMTNcZhKR2Haiso5la4oZ2CuDW6+Ln+UmWuP6yOAW4Blr7UEAY8w84LTbSCISqzweL0tXFFLf0MjC/DxSkjVTfpbrMhgG1BtjVgADgVXAT9xGEpFYtfrtvdgDJ7l75kj6ZKe7jhNRXJdBMnANMAWoAlYAdwLL23Ln7Oz2rx2Sk5PZ7vt2BOULjvIFJxbzFe0pp+CtvVxzWT9uuX5EWD9lHOnj1xLXZVAKvGqtLQMwxjwPTKSNZVBeXoXH4w34SXNyMikrqwz4fh1F+YKjfMGJxXzVp8/wwGPv0iMzjXlThnLsWFWY0kX2+CUmJpz3l2jXZbAKeNQYkwVUAjOAF1wGEpHY4vV6Wb5mByer6vnB18fTOc31y15kcnr2xFr7DvAAsBEoAvYBj7jMJCKx5Y2th3l/ZxlfumYIQ/p2dR0nYjmvSGvtMmCZ6xwiEnsOllXx1LpPyBvcg+lfGOg6TkTT+6pEJCbVnWlkSUEhnVOTmD9rFIlxuCx1IFQGIhKTnnltF4eOVTP/5tF0y9ByE61RGYhIzHlvx1Fe//AQN00cyJgh2a7jRAWVgYjElGOnaln+4g4G98nkS9cOcR0naqgMRCRmNHo8LF1ZhMfrZeGcPJKT9BLXVhopEYkZKzbuZdfBU3xjuqFX9y6u40QVlYGIxIQd+06watNerhrTm0l5vV3HiToqAxGJepU19SxdWUivHl24/cYRruNEJZWBiEQ1r9fLI2t2UFV7hkVz8uiU6vyztFFJZSAiUW3d+wfZuusYX5kyjEG9o2+10EihMhCRqLX/SCV/Xb+LS4Zmc8OE/q7jRDWVgYhEpbr6RhYXFJLeOYV7Zo0K6/UJ4oHKQESi0hOv7uTI8RoW3Dyarl1SXceJeioDEYk67xQdYeO2EmZOGsSo3B6u48QElYGIRJXS8moefWkHQ/t1JX/yYNdxYobegyUiUaOh0cNvn/yAhIQEFs7WchOhFFAZGGOOt3FXr7VWSwWKSEg9/+an7Nx/kv/1xTH0zOrsOk5MCfTIoAJo6Qr0XYBsfNNOHwOfBplLROQzCvcc58XN+5l+xSAuH9nLdZyYE1AZWGtzz3ebMaYb8CPgm8C84GKJiPxDRXU9D64qom/PdObnj6HyVK3rSDEnZBNu1tpT1trvA4X4LnIvIhI0j9fLQ6uLqDndoOUmwigcZ182AdeE4XFFJA6t3XKA7Z8e56tTh9G/V4brODErHGUwlpbPK4iIBGRPSQXPvbGbcSNyuO6yfq7jxLRA30005zw3JQAZwM3ADcDzQeYSkThXW9fAkhWFdE1P5a4ZI7XcRJgFOvn2Ahf+rT8BOAz8W3sDiYgAPL7WUnayln/72jgyOqe4jhPzAi2Dn9FyGXiBemAHsNpaeybYYCISvzZtL+HtwiPkTx7MiAFZruPEhUDfWnp/mHKIiABw5HgNf3l5JyMGZDH7ylzXceKGPsstIhHjTIOHxQWFJCclsGD2aBITdZ6go1zwyMAY80E7H9drrR3fzvuKSJx67o3d7DtSyT9/6WJ6dO3kOk5caW2aaGw7H1dvLRWRgGzbXc7adw9w/bh+XDYix3WcuHPBMrDWahpJRMLuZFUdD68uon9OOvOuH+Y6TlzSi72IOOXxenlwZRF19Y0syh9DSnKS60hxqV2LfBhjbgLuxjeN1N1a28sYczswFPittbYmdBFFJJa9uHkfxftOcNeMkfTtme46TtwK+MjAGLMYWA3cCgzBt3Q1wATgfmCdMUYLiIhIq3YfOsXzG/Zw+cheXH1JH9dx4lpAZWCMWQgsAJ4DhgM/b3Lzz4BlwBeA74QqoIjEpprTvuUmenRN486bjJabcCzQI4NFwDZr7Vxr7W6avGvIWnvCWjsfeBeYG8KMIhJjvF4vj728g+MVdSyYk0eXTlpuwrVAy8AAL7Wyz+tAbkAPasxvjTHLA8wiIlHqzW0lbCk+yi3XDGZYv26u4wiBl0EN0Nr15vr692sTY8xU4M4Ac4hIlDp8rJonX93JqEHdmXHFINdxxC/QMtgIfMkYM6ClG40xw4FbgLfa8mDGmB74zjv8IsAcIhKFzjQ0srigkNTkJObfPJpEnSeIGO1ZtXQ6sMUY8xt800YYY64FLse3dHUK8Ms2Pt4SfNdNbrFcRCS2/PW13Rwsq+Jfbr2E7plpruNIEwleb2ArR/g/Y/AY0LPJZi++axlUAPOttX9rw+PMB0Zba//VGHMXMMVae1cbY+QCewKILSKOvbO9hP96ZAtzrhnCvfkXu44T7wYDe5tuCPRKZ08DbwKDgHxgHJAFVAHbgOettafa+HDzgD7GmK1ADyDDGPN7a+2325qnvLwKjyfwZZBycjIpK6sM+H4dRfmCo3zBCUe+4xWn+f1THzDwogxmTRwY1OPH4/iFSmJiAtnZLX8MLNBpotnAMWttLfC0/0+7WGtvOPv3JkcGbS4CEYkOHo9vuYmGRq9/uQmtghOJAv2ulAFdwxFERGLTqk17sQdO8vUbR9C7RxfXceQ8Aj0yWAQ8Y4x5AN+nkPcCtS3taK2taOuDWmuXA8sDzCIiEW7ngZMUvLWHSXkXcdXFWm4ikgVaBn/Gd6L4O1x4yQlvOx5bRGJIVe0Zlq4sJKdbZ75+o3EdR1oR6Av2PpqdgRYRac7r9bL8xR2cqqrnh98YT+c0/W4Y6QL6Dllrp4Qph4jEkNc/PMQHO8uYe90wBvfRacZooNP6IhJSB49W8dS6XYwZ3IMbJ+rzpNFCZSAiIVN3ppHFKwrp0imZb2q5iaiiMhCRkHl63SccPlbNvTePplt6qus4EgCVgYiExLs7jvLG1sPMuGIgeYN7uI4jAVIZiEjQjp2qZfmLOxjcpyu3XD3EdRxpB5WBiASl0eNh6YoiwMvC/DySk/SyEo30XRORoBRs3MOuQ6f4xnRDr6zOruNIO6kMRKTdivedYPWmfUy+uA9XjO7tOo4EQWUgIu1SWVPPgysLuahHF26/YYTrOBIklYGIBMzr9bJsdTFVtWdYlJ9HWmqS60gSJJWBiATs1fcO8tHucuZeN4yBF2W6jiMhoDIQkYDsK63k2dd3MXZYT6aO7+86joSIykBE2ux0fQOLVxSS0TmFu2eOJEHLTcQMlYGItNkTr+zk6PEaFszOI7OLlpuIJSoDEWmTzYWlvPVxKbOuzGXkoO6u40iIqQxEpFVHT9Tw2MuWYf27kT8513UcCQOVgYhcUEOjhyUrCklMSGDB7NEkJeplIxbpuyoiF/T3DZ+yp6SSu2aMpGc3LTcRq1QGInJe2/eU89I7+5kyti8TRvZyHUfCSGUgIi06VV3PQ6uK6dczna9OHe46joSZykBEPsfj9fLQqiJq6xpYlJ9HaoqWm4h1KgMR+ZyXt+yncM9xbps6nH45Ga7jSAdQGYjIZ+zcf4K/v/Ep400O147t6zqOdBCVgYicU1vXwG8ef4+sjFTumqHlJuKJykBEAN+y1H952fqWm5iTR3qnFNeRpAOpDEQEgE3bS9lcdITbpo9keP8s13Gkg6kMRITS4zU8vnYnIwdmcetUXbUsHqkMROLcmQYPiwu2k5KcyL2z80hK1HmCeKQyEIlzz76+i/1Hqrhn5ii6Z6a5jiOOqAxE4tjWXcd49b2DTB3fn7HDe7qOIw6pDETi1InKOpatLmZArwzmXjfUdRxxTGUgEoc8Hi8PriykvqGRRfl5pCRruYl4l+w6gDHmP4C5/i9XW2u/7zKPSDxYs3kfO/af5O6ZI+mTne46jkQAp0cGxphpwI3AZcBYYLwx5haXmURi3a6Dp3jhzT1MHNWLyRf3cR1HIoTrI4MS4DvW2noAY0wxMNBtJJHYVXP6DEtWFNKjaxp3TNdyE/IPCV6v13UGAIwxw4G3gKustZ+0snsusCfsoURiiNfr5dd/eY/NH5fw629Nxgzq4TqSuDMY2Nt0g+sjAwCMMXnAauB7bSiCc8rLq/B4Ai+znJxMysoqA75fR1G+4Chfy97Yeoi3PjrMrVOG0qNLynkzaPyCE8n5EhMTyM5ueUly5+8mMsZcBawD/t1a+6jrPCKx6NCxap569RPycrsz/QuaiZXPc3pkYIwZALwAzLPWvuYyi0isqj/TyOKC7XRKTWL+zaNJ1HkCaYHraaLvAp2A3xljzm5bbK1d7C6SSGx5Zv0uDpVV8+25l9ItQ8tNSMucloG19j7gPpcZRGLZ+7aM9R8cYvrEAVw8JNt1HIlgzs8ZiEh4lJ86zfIXi8ntncmXr9VyE3JhKgORGNTo8fDgykIaPF4W5ueRnKQfdbkw/QsRiUEr39rLzoOnuGO64aLuXVzHkSigMhCJMXb/CVZu2suVY3ozKa+36zgSJVQGIjGkqvYMS1cW0SurM1+/UZevlLZTGYjECK/XyyNriqmormdR/hg6pbp+57hEE5WBSIx47YNDfPjJMW6dMpRBvTNdx5EoozIQiQH7j1TyzGu7uGRoNjdcPsB1HIlCKgORKFdX38iSFYWkd07mnlmjtCy1tIvKQCTKPbVuJ6XlNdx782i6dkl1HUeilMpAJIptKT7Cho9KmDlpEKNzdX0CaT+VgUiUKjtZy6Mv7WBo367kTx7sOo5EOZWBSBRqaPSwdEUhAAvnaLkJCZ7+BYlEoYKNe9h9uII7bxpJz6zOruNIDFAZiESZor3HWfP2Pq65tA8TR13kOo7ECJWBSBSpqKnnwZVF9M7uwm3TtNyEhI7KQCRKeL1elq0upvp0A4vyx5CWkuQ6ksQQlYFIlHjl3QNs213OvOuHMaBXhus4EmNUBiJRYG9pBc++vpvLhvfk+nH9XMeRGKQyEIlwtXUNLC4opGt6KnfP1HITEh4qA5EI98QrOyk7WcuC2aPJ6JziOo7EKJWBSAR7e3spm7aXMvvKXMzA7q7jSAxTGYhEqCMnanhsrWVE/27MvirXdRyJcSoDkQjU0OhhSUEhyYkJLJiTR1KiflQlvPQvTCQCPffGbvaWVnL3zFH06NrJdRyJAyoDkQizbXc5L285wHXj+jFuRI7rOBInVAYiEeRUVR0Pry6if046864b5jqOxBGVgUiE8Hi9PLiqiLr6RhbmjyFVy01IB1IZiESIl97ZT9HeE9w2bTj9eqa7jiNxRmUgEgF2Hz7F8xs+ZcLIXlxzaV/XcSQOqQxEHKs53cCSgkKyMtK46yaj5SbECZWBiENer5fHXt7B8Yo6Fubn0aWTlpsQN1QGIg5t3FbCluKjfPHqwQzr1811HIljKgMRRw4fq+aJV3cyalB3Zl4xyHUciXMqAxEHzjQ0smRFIanJScy/eTSJiTpPIG4luw5gjPka8GMgBfiDtfZ/HEcSCbu/rt/NgaNV3PeVS+iemeY6jojbIwNjTD/g58BkYCywwBgz2mUmkXB7Z3sJ694/yA0TBnDpsJ6u44gA7o8MpgGvWWuPAxhj/gZ8BfhZuJ6woqYe+9EhKipOh+spgtb1cIXyBSGS8zV6PDz16icMvCiDr0wZ6jqOyDmuy6AvUNLk6xJgYlvvnJ0d+EXB//78NlZt3BPw/URCpUunZH549xfomxO5F7XPycl0HeGClC/0XJdBIuBt8nUC4GnrncvLq/B4vK3v2MScSYO4aVIuJ45XB3S/jtS9R7ryBSHS8w3NzeZ0dR1lZZWuo7QoJyczYrOB8gUjMTHhvL9Euy6Dg8DVTb7uDRwO5xMmJyXSJyeTLkmR++6NHOULSqTny+ySyunqOtcxRD7DdRm8CtxvjMkBqoEvAwvcRhIRiT9O301krT0E/AhYD2wFnrTWbnGZSUQkHrk+MsBa+yTwpOscIiLxTJ9AFhERlYGIiKgMRESECDhn0E5JQFCLe0X6wmDKFxzlC47yBSdS8zXJ9bkLbCd4vYF9aCtCTAbedB1CRCRKXQ1sbLohWssgDbgc3/IVjY6ziIhEiySgD/Au8JlPPkZrGYiISAjpBLKIiKgMREREZSAiIqgMREQElYGIiKAyEBERVAYiIkL0LkfRZsaY/wQarbX3+7/OAp4AhgBlwFxrbWmz+yQAvwFuxncZznuttW+FKV8vYG2TTd2AHGttRrP9BgHbgd3+TUestdPDkamFjHcCvwKO+Detttb+qNk+qcDDwASgFviatXZHB+W7Cvg9kAqUA/dYa/c126fDx88Y8zXgx0AK8Adr7f80u30s8BDQFdgALLLWNoQzU7Pn/w9grv/L1dba77dw+z3ACf+mB5v/P4Q533qgF3DGv2mhtfadJrePxdH4GWPmA99qsmkw8Bdr7bea7ON0/AIVs2VgjOkG/A64DXigyU3/BbxprZ1ljPkG8H+Bec3u/mVgFDAaGAasNsaMCsc/NGvtUWCsP3MisA7fBX+am4Dv4j8LQ52hDSYA/2qtfeoC+/wfoNpaO8oYcw2wHLiiI8LhK/c51tptxph7gD8C+c326dDxM8b0A34OjMf3Sc9Nxpj11tqiJrs9Dsy31m42xjwM3Av8uYPyTQNuBC7Ddx3yl4wxt1hrn2+y2wTgq9batzsiU7N8CcAIYNAFfu6cjZ+19iF8RYQxJg94Abi/2W7Oxq89YnmaKB/4BPjvZttn4XvxAHgKmGGMSWlhn6ettR5r7U5gP3BlOMP63Q3U+C/409zlwBhjzFZjzGvGmIs7IE/T577TGPOxMeZxY0z3FvY5N67W2g1AjjFmYLiDGWPSgB9ba7f5N20DWnrejh6/acBr1trj1tpq4G/AV5rkHgR0ttZu9m9aDtwa5kxNlQDfsdbWW2vPAMV8ftwmAD80xmwzxvzJGNOpA/MZ/3/XGmM+MsZ86zM3uh+/pv4M/NBae6zZdpfjF7CYLQNr7WPW2l/x+bWL+uL7QcD/G0cFkHO+ffxKgP5higqAMSYJ3xHBv59nl9P4fhMaB/wWeME/NdMRSoD/BC4BDgB/amGfDh8zAGttnbX2cTh3ZHU/vt/Smuvo8WttPJyM11nW2sKzL6TGmOH4povWnL3dGJMBfAh8D9+YZQE/6ah8QHd8R8m3AFOBRcaYG5rc7nT8zvIfYXW21j7bbLvr8QtY1E8TGWNuxTdf3NQOa+2089yl+dqyCfjOCzSViO/Q+UL7BKyVrDcBn1hrP27pvmfPefitMcb8Et9U1kfB5mpjvrP7PMA/5t2bCsuYtTWf/4X9UXz/pn/R/L4dMX7NtDYeYR+vtvBPcawGvmet/eTsdmttFTCzyX7/DSyj5SnMkPNPrZybXvFPA80EXvFviojxAxbim47+DNfj1x5RXwb+Rn621R3/4RDQGzhojEkGMvGddGzqIL6V/c7qDRwOJie0mvWLwNPnu68x5p/xzXmfzZrAP06shURL+Ywx3Ywx37bWnn0RTgBamsM9O2ZniyIkY9ZaPn/GDGAFvu9jvn/ao/k+YR+/Zg7iWyb4rObjEZZ/Y4Hwn3h/DvgXa+3TzW4bCEyz1i7zbwr3eDXPNhlIs9auO8/zR8L4pQLXAne1cJvT8WuPmJ0muoA1wB3+v8/DdzK5+TdpDXC7MSbJGDMM34msd8OcaxIXvkbDtcA3AYwx1+JbirYj3q1TBXzfGPMF/9ffAp5vYb9z4+r/QT5trd3fAfnAN/2zC5hnra07zz4dPX6vAlONMTnGmC743pTw0tkb/e92Ou1/QQb4BvBiGPN8hjFmAL7ptK81LwK/WuABY8xg/8nc/03L3/dwyQJ+Y4zpZIzJBO5s+vyux8/vEmCn/5xQc67HL2DxWAY/Aa4wxhQC/4Tvm4QxZo4x5iH/Pn8DCvGdjCwAvmmtrQ1zriH4fts5xxizyBjzM/+X9wE3GGO245vzvs1aG/bDYmttI7755D8bY4rxvTvm+y3k+39Amn9c/4jvhzPsjDGX4XuzwFXAB/4TxGtayNeh42etPYRvSmA9sBXfUckWY8waY8wE/263A783xuwAMvCNW0f5LtAJ+J1/zLb6x2uNMWaCtbYM3xTISsDi+822+ZsxwsZauwrf9NWHwPvAMmvt2xE0ftDyz2xEjF976HoGIiISl0cGIiLSjMpARERUBiIiojIQERFUBiIigspARERQGYiICCoDERFBZSAiIsTAQnUiLhljrgbewLcswWj/apVnFzF7H98FkqZYay+07pSIczoyEAmC/0X+T8AA4KdNbvopMAb4rYpAooHWJhIJkn9V0o/xXSlsPL4F4DYBRcAEa229w3gibaIyEAkBY8wU4DVgI5CN79rZlze5HKdIRNM0kUgIWGtfx3ct3KvxnSf4iYpAoonKQCR0nmvy903OUoi0g6aJRELAGNMJ34VYBvk37QfGWmtPu0sl0nY6MhAJjZ8BI/3//QVg/H8XiQo6MhAJkjFmIr5poWJgHL5LHG7Fd+3sK621W9ylE2kbHRmIBMEYkwY8gu9naaG19oz/raQL/dse8e8jEtFUBiLBuR/fu4eWWmvPnTT2f9Bsmf+2+50kEwmApolERERHBiIiojIQERFUBiIigspARERQGYiICCoDERFBZSAiIqgMREQElYGIiKAyEBER4P8D/D9SvMEYqNEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Naive scalar relu implementation. \n",
    "# In the real world, most calculations are done on vectors\n",
    "def relu(x):\n",
    "    if x < 0:\n",
    "        return 0\n",
    "    else:\n",
    "        return x\n",
    "\n",
    "\n",
    "plt.plot(range(-10, 10), [relu(i) for i in range(-10, 10)])\n",
    "plt.xlabel('x', fontsize = 20)\n",
    "plt.ylabel('relu', fontsize = 20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Softmax\n",
    "\n",
    "The softmax function, also known as softargmax or normalized exponential function, is a function that takes as input a vector of K real numbers, and normalizes it into a probability distribution consisting of K probabilities. \n",
    "\n",
    "$$softmax = \\frac{e^x}{\\sum e^x}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:26:09.199623Z",
     "start_time": "2024-08-03T02:26:09.192743Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.02364054, 0.06426166, 0.1746813 , 0.474833  , 0.02364054,\n",
       "       0.06426166, 0.1746813 ])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def softmax(s):\n",
    "    return np.exp(s) / np.sum(np.exp(s), axis=0)\n",
    "\n",
    "softmax([1.0, 2.0, 3.0, 4.0, 1.0, 2.0, 3.0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:26:14.244841Z",
     "start_time": "2024-08-03T02:26:14.086963Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAESCAYAAAABl4lHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAj/0lEQVR4nO3de3RcZ3nv8e+MLpZ18U2W73bshOSJ5ZCrEyhJICQGmgQI1OGyQksTSh1azuk5vXFoAyRQQlktbVmUHKBJCXDAhDYt0IC5JLGThoRciUkiOY9NIvkq27JGvkiyrMvs88eesUZj2ZqRJe2Z0e+zlpdm7/3O3k/eBfObvd89744FQYCIiEgu4lEXICIixUOhISIiOVNoiIhIzhQaIiKSM4WGiIjkrDzqAibQNOBSoA0YjLgWEZFiUQYsBJ4BjmVvLOXQuBR4LOoiRESK1JXAL7JXlnJotAF0dnaTTOb/W5T6+lo6OrrGvahipf4YTv0xRH0xXLH3RzweY/bsGkh9hmYr5dAYBEgmgzGFRvq9MkT9MZz6Y4j6YrgS6Y8RL+trIFxERHKm0BARkZwpNEREJGcKDRERyZlCQ0REcqbQEBGRnCk0RERKSE9vP7fd/SRbdx6ckP0rNERESkhzaydtHT3EY7EJ2b9CQ0SkhDS1Jpg+rYwVi+omZP8KDRGREhEEAU0tCc5dNpuy+MR8vCs0RERKxP6DRzlwqJdVK+ZM2DEUGiIiJaK5JQHAquUKDRERGUVTayf1M6qYN3v6hB0j8lluzewm4BNABfBFd78ra7sBXwNmA3uB97t756QXKiJSwAaTSbZsT3DpufOITdCdUxDxmYaZLQbuBK4ALgTWmVljxvYY8F/A5939AuB54OMRlCoiUtBa2o5w9Nggq1bUT+hxor48tQbY6O4Jd+8G7gduzNh+MdDt7j9NLX8OuAsRERmmuSVBDFh5xuwJPU7Ul6cWMfzpUG3AZRnLrwH2mtm/AhcBW4D/OXnliYgUh6bWBGcsqKN2esWEHifq0IgDmY+4igHJjOVy4Crgje7+rJn9DfCPwM25HqC+vnbMxTU0TMyPY4qV+mM49ccQ9cVwk90fPb39vLLnMGvf/JoJP3bUobGL8OHlaQuAPRnLe4Ft7v5savm7hJewctbR0TWmRy82NNTR3n4k7/eVKvXHcOqPIeqL4aLoj+e3tZNMBqyYV3vax47HY6f8sh31mMZDwDVm1mBm1cBa4KcZ258AGszsgtTyO4DnJrlGEZGC1tzSSWVFnLMWz5zwY0UaGu6+G7gN2ARsBta7+9NmtsHMVrv7UeDdwN1m1gRcDfx5ZAWLiBSgptYEtnQ2FeUT/5Ee9eUp3H09sD5r3XUZr59i+OC4iIikdBzqZW+ih6suWjwpx4v68pSIiJyGptb01CETe6ttmkJDRKSINbcmmFVbyaK5NZNyPIWGiEiRSgYBza2drFo+Z0KnDsmk0BARKVI79h2h62g/jRM4FXo2hYaISJFqSk2F3jiBU6FnU2iIiBSp5tZOljTUMrOmctKOqdAQESlCx/oH2bbrIOdN4qUpUGiIiBSlrTsPMjAY0Lhicm61TVNoiIgUoaaWBOVlcc5ZMmtSj6vQEBEpQk2tCc5eMpPKirJJPa5CQ0SkyBzsOsbu9u5JH88AhYaISNFpbp38W23TFBoiIkWmqaWTuuoKls4f+0PmxkqhISJSRIIgoLk1QePyOcQnaeqQTAoNEZEisru9m0PdfTRO0qy22RQaIiJFZGgq9MkfzwCFhohIUWlqTbCwvpo5M6oiOb5CQ0SkSPQPDLJ1x8HIzjJAoSEiUjR+s+sQfQPJSZ0KPZtCQ0SkSDS1dlIWj2FLZ0VWg0JDRKRINLUkOGvRDKZPK4+sBoWGiEgRONLTx459R1gV4aUpgOjiKsXMbgI+AVQAX3T3u7K23w58COhMrbo7u42ISKnbsr2TACIdz4CIQ8PMFgN3ApcAx4AnzGyTuzdnNFsNvN/dfxlFjSIihaCpJUH1tHJWLJgRaR1RX55aA2x094S7dwP3AzdmtVkN/LWZvWBmXzazaG5OFhGJSBAENLUmWLl8NvH45E8dkinq0FgEtGUstwFL0gtmVgs8D/wlcDEwC/jkJNYnIhK5vYkeEoePRfr7jLSoxzTiQJCxHAOS6QV37wKuSy+b2T8AXwduy/UA9fVjnwWyoaFuzO8tReqP4dQfQ9QXw413fzzl7QBceclSGuprxnXf+Yo6NHYBV2YsLwD2pBfMbBmwxt2/nloVA/rzOUBHRxfJZDB6wywNDXW0tx/J+32lSv0xnPpjiPpiuInoj6debKNhVhVlyeSE93U8Hjvll+2oQ+Mh4A4zawC6gbXAuoztR4G/M7NNQCvwUeD7k12kiEhUBgaTvLyjk9evWhB1KUDEYxruvpvwUtMmYDOw3t2fNrMNZrba3duBW4EHACc80/iHqOoVEZlsr+45TG/fIKsimgo9W9RnGrj7emB91rrrMl7/B/Afk12XiEghaG5NEIvByjMKIzSivntKREROoaklwZkLZ1BdVRF1KYBCQ0SkYPX09vNq22EaC+BW2zSFhohIgdqy/SBBQOTzTWVSaIiIFKjm1gTTKss4c1G0U4dkUmiIiBSoppYEK5fNpryscD6qC6cSERE5rv3gUfYfPEpjgdxqm6bQEBEpQE2tCaCwxjNAoSEiUpCaWhLMrpvGgjnVUZcyjEJDRKTAJJMBL2/vZNWKOcRi0U6Fnk2hISJSYFr3HqG7d6AgpkLPptAQESkw6fGMlQU2CA4KDRGRgtPUkuCM+XXMqK6MupQTKDRERApIb98Ar+w+ROOKwjvLAIWGiEhB8R0HGUwGBTmeAQoNEZGC0tSSoKI8ztlLZkZdyogUGiIiBaSpNYEtnUVFeVnUpYxIoSEiUiASh3tp6+gpqKnQsyk0REQKRHNrJ1B4U4dkUmiIiBSIptYEM2oqWdJQE3UpJ6XQEBEpAMkgoLk1warlswtu6pBMeYWGmb0/hzazzew7Yy9JRGTq2bW/iyM9/QU9ngH5n2msN7PvmtmskTaa2buBZmDUcBERkSHpqUNKLTQeBN4HvGhmb0mvNLN6M7sPuB+YCXxq/EoUESl9TS0JFjfUMLtuWtSlnFJeoeHubwP+CJgF/NTM/tnMbiI8u3gvsBE4393vzHWfZnaTmTWb2TYz++gp2l1vZi351CsiUgz6+gfZuvNQwf4KPFPeA+Hu/jXgPGAT8MfA/wPKgd9197e4+29y3ZeZLQbuBK4ALgTWmVnjCO3mA18ACnd0SERkjLbtOsTAYLLgL03B2O+eWgksZehDfCZwuZnNyHM/a4CN7p5w927Cy1s3jtDuHuDTY6xVRKSgNbUkKC+LYUtnRV3KqMrzaWxmC4EvAb8DHAM+BnwfuJvwstW7zOx/u/u/57jLRUBbxnIbcFnWMf8E+BXwZD61ptXX147lbQA0NNSN+b2lSP0xnPpjiPpiuHz7w3cdpHFFPUsWz5qYgsZRXqEBbAFmAL8EbnH3ran1V5vZHwOfB+4zs5vd/foc9hcHgozlGJBML5jZecBa4BpgSZ61AtDR0UUyGYzeMEtDQx3t7UfGcsiSpP4YTv0xRH0xXL79cai7j5Y9h1n7pjMLoh/j8dgpv2zne3mqAvgz4IqMwADA3f8vQ2Mdv53j/nYBCzOWFwB7Mpbfk9r+LLABWGRmj+VZs4hIwdpSJLfapuV7pnHBqQa63X0HsMbM1uW4v4eAO8ysAegmPKs4/l53vx24HcDMlgOPuPuVedYsIlKwmloS1FSVc8b84rjEl+8tt7neGfVgjvvbDdxGeHayGVjv7k+b2QYzW51PbSIixSYIAppaEzQun0M8Xhw3h+Z7poGZXQfcBMwDyhi6gypGePmqHjg71327+3pgfda660Zo1wosz7deEZFCtaejh4NdfQU9q222fO+e+h3g3zn17yW6gf86naJERKaCppb0eEZhPg98JPkOhP8ZMED46+8FwPOEt9suAK4GniO8G+r/jGONIiIlqbk1wfw51cydOT3qUnKWb2i8FviBu9/v7vuBXxDeSbXf3R8B3kb4+43bxrdMEZHSMjCYxHccZFURnWVA/qFRBWQOhr8MnGNm0wDcPQH8AHj9uFQnIlKiXtl9iGP9g0Ux31SmfENjH9CQsfxKah+rMtYdYIw/xBMRmSpeakkQj8WwZaV9pvEosNbMzkkt/zr194aMNpcDidMtTESklDW3Jjhz8Qyqq/K+iTVS+YbG54HphM/TuNHd9wEPAH9tZt8zs0cIQyOn32mIiExFXUf7aW07UnSXpiD/H/c1AVcRPjfjUGr1/yAc23gP8EbgGeCvxq9EEZHS8vL2TgIoytDI+7zI3Z8Grs1Y3gW81szOB3qBbe6e/wyBIiJTxEstCaZPK2PFouKYOiTTuF1Mc/cXxmtfIiKlKggCmlsTnLtsNmXxsT7SKDpjmUbkAuD9hFN6nOxhtoG7rz2NukREStL+g0c5cKiX337dsqhLGZN8pxFZC9xHOOfUqejylIjICNJThxTjeAbkf6ZxO+G4xUeBp1KvRUQkR00tCebOrGLe7OKZOiRTvqFxFvB1d//WRBQjIlLKBpNJXt7RyaXnzicWK46p0LPlOwqznZOPY4iIyCm0tB3h6LHBopoKPVu+ofEF4P2pZ3eLiEgemloSxICVZxTX1CGZ8ro85e5fT/0e41dm9iDQSjirbbbA3f98HOoTESkZTa0Jli+so3Z6RdSljFm+d0+9FfhI6n3XnqJpACg0RERSjh4b4NXdh7n29cV5q21avgPhnye83fZvgccJn9InIiKjeHlHJ8kgKNpbbdPyDY1zgW+5ux6yJCKSh6aWBNMqyjhr8cyoSzkt+Q6E7wW6JqIQEZFS1tTaiS2bRUV58U0dkinf6v+F8O6pxRNRjIhIKTpw6Cj7Ej00FvmlKcj/8tQTwHuBl8zsPwkf/TriuIa7fymXHZrZTcAngArgi+5+V9b2dwOfJhxLeQZY5+59edYtIhKZ5tZOgKJ7HvhI8g2NRzJe33KKdgEwamikzljuBC4hvHX3CTPb5O7Nqe01wJeBi919n5ndB9xMeMYjIlIUmloSzKqtZNHcmqhLOW35hsaHGN/JCNcAG909AWBm9wM3Ap8BcPduM1vu7v1mVg3MAzrH8fgiIhMqGQRs2d7JBWfVF+3UIZny/XHfN3Jpl/qAz8UioC1juQ24LOuY/WZ2LfBtYDfw8xz3LSISuR37jtB1tJ/GIp46JFO+P+57Ffgnd//nU7T5FOEjYOflsMs4w89cYkAyu5G7/wSoN7PPAV8Bbsq15vr62lybnqChofieqjWR1B/DqT+GqC+Gy+yPR14IvxdfeclSZtdVRVXSuDllaJjZcmBGxqrlwMrUVCIjqSS85JTrhbtdwJUZywuAPRnHnwOsdvf02cV3gO/luG8AOjq6SCbzv6LW0FBHe/uRvN9XqtQfw6k/hqgvhsvuj6dfamPpvFoGevtp7+2PsLLcxOOxU37ZHu1M4/XAeobOBgLg1tS/k4mR+yWkh4A7zKyB8C6stcC6rH1928xWu/sO4D3AL3Lct4hIpI71DfKb3YdYc8nSqEsZN6cMDXe/z8wuIrzUFAM+CPwa2DxC8wDoJxx3uGuE7SPtf7eZ3QZsIjxLucfdnzazDcCn3P1ZM1sH/MjMAqCZcO4rEZGC5zsPMjAY0Lii+G+1TRvt8tSbgH9w9/0Zy/fm+huMXLj7esKzmcx112W8/gHwg/E6nojIZGluTVBeFuecJbOiLmXcjPaL8H8H/jRj+RHC6dBFRGQUTa0Jzlk6k8qKsqhLGTejhUYdwwfCfx+4YOLKEREpDQe7jrG7vbvoZ7XNNtpA+DbgFjOrBTpS6641s9Eu0OkhTCIypTW1JABKYr6pTKOFxscJL1H9Xmo5ILyj6vWjvE8PYRKRKa25NUFddQVL54/9t2KFaLS7pzak5oc6F6gCNgLfAL458aWJiBSnIAhobu2kcfkc4iUwdUimUX8R7u4HgScBzOxR4BF3f3SC6xIRKVq727s51N1HYwnMapst37mn3px+bWbLCAfFqwnHO5rdfc/J3isiMlW8lBrPKLVBcMh/ltv01CJ3A1dnbQrMbCNwq7u3jENtIiJFqbk1wcL6aubMKP65prLlO2HhAuBxYCHhA5EeJ5wrajbwJsJ5px41s4vd/cA41yoiUvD6+gfZuvMgb7xgUdSlTIh8zzRuJwyMP3L3r2VvNLMPEz4g6a+BPzv98kREisuWlgR9A8mSmQo9W77PCL8eeHCkwABw93uAB4EbTrcwEZFi9PzW/ZTFY5y7bFbUpUyIfENjPvDCKG1eJHy4kojIlLN5WztnLZ5JVWXeQ8ZFId/Q2Aec7FkaaecDGs8QkSnncE8fr+4+xKoSvNU2Ld/Q2ACsMbNbRtpoZrcC1wA/Pt3CRESKzZbWToKAkh3PgPwHwu8A3gXcY2YfBB4DDhFejroSuITwbOQz41eiiEhxaGpNUDO9ghULZozeuEjl++O+vWZ2OeEdUm8mvM020ybC32noR34iMqV09/bzwm8OcMHZc4nHS2vqkEx5j9S4+yvANak5qS4inDq9C0gAm929a3xLFBEpbEEQ8I0NL9PdO8DaN58ddTkTKt8xDczsjWZ2H7DX3X8ENAFfBh4F9pvZ7eNco4hIQXvk+d08t7WdtW86i3OWle4gOOQZGmZ2NfAw8B4g/aT0u4ElhJemWoFPmdnvjmONIiIFa+f+Lr778G947Zn1vPWypaO/ocjle6bxMeAIcJm7t5rZSmA18DN3XwNcCLwMfHRcqxQRKUDH+gb56g9fomZ6OX9w/cqSmwZ9JPmGxqXAfe7+XGr57YQPXPo3AHfvA34KrBq3CkVECtR3HtzK3o4e1r29kRk1lVGXMynyDY0qwlts065N/X0wa58Dp1OUiEih+2XTXn7xYhtvf8NyVpbgFOgnk+/dU78BXgdgZkuAy4Emd9+VWldJOD/VK7nu0MxuAj4BVABfdPe7srbfAHwaiAEtwC3u3pln3SIi42Zfoodv/cw5e8lM3nnF8qjLmVT5nmn8J3CVmW0ivFuqHLgXwMyuB34JnEU4OD6q1G27dwJXEI6HrDOzxoztM4CvANe7+wWE817dkWfNIiLjpn8gyVd/2ER5PMat71xFWTzvm1CLWr7/tZ8Fvkb46+/lwPeAL6W2vYHwSX7/SI6hQfj8jY3unnD3buB+4MaM7RXAR919d2r5BWBZnjWLiIyb+x95he37jvCh61eW5EOWRpPvL8IHgT8ys48BcXfPHN+4G/iSu+/LY5eLgLaM5TbgsozjdQDfBzCz6cDHgX/Op2YRkfGyedsBHnx2J2suWcJFZzdEXU4kxjR3r7sfGWFd6xh2FSe8+yotBiSzG5nZTMLw+LW7fzOfA9TX146hrFBDQ92Y31uK1B/DqT+GTIW+OHDwKPf+ZAtnLp7JH7/3QirKy07atpT7I+oJ33cRXupKW0D4+NjjzGwh8DNgI/Cn+R6go6OLZDIYvWGWhoY62ttPyMYpS/0xnPpjyFToi8Fkkr9f/zx9/Uk+fP1KDnb2nLRtsfdHPB475ZftqEPjIeAOM2sAuoG1wLr0RjMrAx4A/s3dPxtNiSIy1T3weCtbdx3iD9/eyII51VGXE6lIQ8Pdd5vZbYRTkFQC97j702a2AfgU4VQlFwPlZpYeIH/W3T8cTcUiMtVs2d7JA4+3cvl5C/it8xZEXU7koj7TwN3XA+uz1l2XevksY5hUUURkPBzu6ePuB5qYP6eaD7z1nKjLKQj6QBYRGUEyCPj6j7fQdXSAj9ywqmSf+Z0vhYaIyAgefGYnL7zSwfuufg3L5pfu3VD5UmiIiGRpaTvM/Y+8wsXnNHD1xYujLqegKDRERDL09A7w1R++xKzaSm657lxiU2C683woNEREUoIg4Fs/e5mOQ8dY985V1FRVRF1SwVFoiIikPPZCG09v2c+7rlzB2UtmRV1OQVJoiIgAuw90s/7BrTQun811v3VG1OUULIWGiEx5ff3hY1urKsv4w7c3TonHto6VQkNEprz7Ht7G7vZuPvz2RmbWTou6nIKm0BCRKe2Zl/fzyOY9XPu6ZZx3Zn3U5RQ8hYaITFntB4/yjZ9s4cxFM3j3G8+MupyioNAQkSlpYDB8bCuEj20tL9PHYS7USyIyJX3/v1+lpe0wt1x7Lg2zpkddTtFQaIjIlPPiqx385KkdXHXRYlafOy/qcoqKQkNEppSDXce450fNLG6o4f1XvybqcoqOQkNEpoxkMuDuB5o51jfIR244j8qKkz/nW0am0BCRKePHT25ny/ZOPvCWc1g8tybqcoqSQkNEpoStOw/yw8daeF3jfK44f2HU5RQthYaIlLyuo/38ywNNzJ1ZxQffZpru/DQoNESkpAVBwL0btnCoq49bb1jF9Gl6bOvpUGiISEnb+KvdPL/tAO+56ixWLJwRdTlFT6EhIiVrx74jfG/jNs4/q563XLo06nJKgkJDREpSb98AX/lhE7XTK/iD61dqHGOcRB4aZnaTmTWb2TYz++gp2n3LzG6exNJEpIh9++db2d/Zw63vXEVddWXU5ZSMSEPDzBYDdwJXABcC68ysMavNIjN7ALhx8isUkWL0+IttPPHSXt7xhuXYstlRl1NSoj7TWANsdPeEu3cD93NiOHwA+CHwb5NdnIgUn7aObr79862cs3QW77h8edTllJyo7z1bBLRlLLcBl2U2cPe/BzCzK8ZygPr62jEX19BQN+b3liL1x3DqjyGF0hd9/YN89lvPUVlRxl/dfBlzI5q9tlD6YyJEHRpxIMhYjgHJ8TxAR0cXyWQwesMsDQ11tLcfGc9Sipr6Yzj1x5BC6ovvPLiVV/cc4k9uPJ+gfyCSugqpP8YiHo+d8st21JendgGZv+dfAOyJqBYRKWK/2trOw8/t4i2rl3Lha+ZGXU7JivpM4yHgDjNrALqBtcC6aEsSkWLTcaiXezds4YwFddx41VlRl1PSIj3TcPfdwG3AJmAzsN7dnzazDWa2OsraRKTwDQwmeXTzbv72O88xkAz4yA2rqCiP+gJKaYv6TAN3Xw+sz1p33Qjtbp6smkSksA0MJnn8xTZ+9MR2Og73cuaiGdz6zlXMn10ddWklL/LQEBHJ1Uhh8cHfNs5bMUe/+J4kCg0RKXgKi8Kh0BCRgqWwKDwKDREpOAqLwqXQEJGCkR0WKxYqLAqNQkNEIjdSWPze24zXnqmwKDQKDRGJjMKi+Cg0RGTSKSyKl0JDRCaNwqL4KTREZMIpLEqHQkNEJozCovQoNERk3CksSpdCQ0TGjcKi9Ck0ROS0KSymDoWGiIxZb98AP3uyle/+zBUWU4RCQ0RGFQQBHYd72bm/a9i/9s6jBKCwmEIUGiIyTP/AILsPdLNzXxc7UuGwa38XPccGjreZN2s6S+fV8oZVC1h93kIWzpymsJgiFBoiU1QQBBzq7jvh7GFvRw/JIABgWkUZSxpquGzlPJbOq2XpvDoWN9QwfdrQR0dDQx3t7Uei+s+QSabQEJkCBgaT7O3oyQiHI+zY38WRnv7jbebMmMbShlouPmcuy+bVsXReLQ2zpxPXGYRkUGiIlJiuo/3s3Hdk2NnDno5uBgbDs4fysjiL59ZwwVlzU2cPtSyZV0vt9IqIK5dioNAQKVLJZMC+zp4TLi91Hjl2vM2MmkqWzqtlzYo5LEsFxPw51ZSXxSOsXIqZQkMkYskgoPfYAF29A/T09tPdO0D30X56egfoTi2fuH6AIz199A0kAYjHYiycW40tm3X87GHpvDpm1lRG/F8npSby0DCzm4BPABXAF939rqztFwL3ADOA/wY+4u4D2fsRiVIQBBzrH6T7aPhBP/wDP+vD/+jw9T3HBkiNO4+ovCxOzfRyaqoqqK4qZ86MKpbMK6euuoJFc2tYNq+ORXOrqSgvm7z/YJmyIg0NM1sM3AlcAhwDnjCzTe7enNHs28CH3f1JM/tX4A+Br0x+tVLogiBgMBkwMJhkYDD9N8ngYEB/6m963UAOr0/2vv7BJAPJgIOHe8Nv/6mQGEye/JM/HotRXVVOzfQKaqrKqa2uYMGcaqqryqmuqqA29bcm1aa6KgyJmqpyKisUBlI4oj7TWANsdPcEgJndD9wIfCa1fAYw3d2fTLX/BvBpJjg0BgaT/PLFNhKd3Sdsy/xGGDBsYaSXWeuHFk71zfKEYwRDuwmCjL2k16fWDXtful1Wm8z9Z+47CIbXNuw9AVRXV9LVfYxkEBAE4fX0IAgvraTXBUFAMvU3CAKSyfS6zLbp7el9hMdIv05m7CeZ2k/m8QaS4Qf34GByxA/08RaPxSgvi1FeFqe8LEZZWZyKsjgzaiuZVlnG7BlVQx/46bOBaUPhkP7wr6os0+8YpCREHRqLgLaM5TbgslG2L8nnAPX1tXkX9cQLe/jbbz6T9/umgngM4vEYsVj4rywOsViMeGo5HmfodUbbeGpb2CbdPnP70PvKyuJUZGxPt43FYlSUxykvix//W14Wo6K8LPxgLw/XV5TFw9dlGW1PeF+qbdb649tTf8vi+qDPRUNDXdQlFJRS7o+oQyPO8C/mMSCZx/ZRdXR0kTzFZYORnL2wjq99/Br2H+gaduDjr0/yOZL5TXJYk9iIL4ftKHuX2e+PETvefNhxYqm2sVjq79B70+2Ot8nYx9D62PH3DG3LPFbYZt68Og5k9EfJSSYJkkn6+qEvh+b6QdsQ9cVwxd4f8XjslF+2ow6NXcCVGcsLgD1Z2xeeYvuEWdRQSwXjf7mjWOnSiohA+E0+Sg8B15hZg5lVA2uBn6Y3uvt2oNfMLk+t+j3gJ5NfpoiIQMSh4e67gduATcBmYL27P21mG8xsdarZB4B/MrOXgVrgS5EUKyIikV+ewt3XA+uz1l2X8frXDB8cFxGRiER9eUpERIqIQkNERHKm0BARkZxFPqYxgcogvOd4rE7nvaVI/TGc+mOI+mK4Yu6PjNpHnL8mljl9RIm5Angs6iJERIrUlcAvsleWcmhMAy4lnHpkMOJaRESKRRnhj6qfIZxIdphSDg0RERlnGggXEZGcKTRERCRnCg0REcmZQkNERHKm0BARkZwpNEREJGcKDRERyVkpTyMyZmZ2E/AJoAL4orvfFXFJkTGz24H3phZ/7O4fi7KeQmFmXwDmuvvNUdcSJTN7B3A7UAP83N3/V8QlRcbMfhf4q9TiT9z9L6KsZ6LoTCOLmS0G7iSchuRCYJ2ZNUZaVETMbA3wVuAiwr64xMzeHWlRBcDMrgF+P+o6omZmZwJfBd4FnA9cbGbXRlpURFJPHv0S8CbgAuDK1P9/So5C40RrgI3unnD3buB+4MaIa4pKG/Dn7t7n7v3AFmBZxDVFyszmEH6p+FzUtRSAdwPfc/ddqf99vA94KuKaolJG+HlaQ3iFogI4GmlFE0SXp060iPDDMq2NKfrkQHdvSr82s7MJL1NdfvJ3TAlfI3xE8dKoCykArwH6zOy/CL9M/Aj4ZLQlRcPdj5jZJ4GXgR7gUeCJaKuaGDrTOFEcyJyQKwYkI6qlIJjZKuBB4C/dfVvU9UTFzD4M7HT3h6OupUCUE56Z/wHwW8DrmKKX7czsfOBDwBmEXzwHAY1pTBG7CGd4TFsA7ImolsiZ2eXAw8DH3f2bUdcTsfcBbzWzzcBngHea2T9FW1Kk9gIPuXu7ux8Fvs8UPSsH3gY87O773f0Y8A3gqkgrmiC6PHWih4A7zKwB6AbWAuuiLSkaZrYU+AHwPnffGHE5kXP3t6Rfm9nNwFXu/qfRVRS5HwHfNLNZwBHgWsL/vUxFvwb+zsxqCC9PvYNwavGSozONLO6+m/Ca9SZgM7De3Z+OtKjo/AVQBfyjmW1O/ftI1EVJYXD3p4C/I3xQTzOwHbg30qIi4u4/B74LPAe8QDgQ/vlIi5ogep6GiIjkTGcaIiKSM4WGiIjkTKEhIiI5U2iIiEjOFBoiIpIzhYaIiORMoSEiIjlTaIiISM4UGiIikjPNPSUyCczsSsLpsncBje7elVpfSTj1RCPhXFaPRVelyOh0piEyCVJh8GXC53B8OmPTp4HzgC8oMKQYaO4pkUmSeiToi4QPLLqEcDLIJwgn+1vt7n0RlieSE4WGyCQys6uAjYQzw9YTPv3uUnd/Icq6RHKly1Mik8jdHwG+AlxJOI7xSQWGFBOFhsjk+4+M1yX5HGkpXbo8JTKJzKwKeJ7wWdIAO4AL3b03uqpEcqczDZHJ9Rng3NTfzwGWei1SFHSmITJJzOwywstRW4CLgRjhI4XPAd4whR8rLEVEZxoik8DMphE+PzsO3Oru/albbG9Nrbs31UakoCk0RCbHHYR3S/2Lux8f/E79oO/rqW13RFKZSB50eUpERHKmMw0REcmZQkNERHKm0BARkZwpNEREJGcKDRERyZlCQ0REcqbQEBGRnCk0REQkZwoNERHJmUJDRERy9v8B3CcYqmit9h8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(10), softmax(range(10)))\n",
    "plt.xlabel('x', fontsize = 20)\n",
    "plt.ylabel('softmax', fontsize = 20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Softmax is often used in neural networks, to map the non-normalized output of a network to a probability distribution over predicted output classes.\n",
    "\n",
    "- Prior to applying softmax, some vector components could be negative, or greater than one; and might not sum to 1;\n",
    "- After applying softmax, each component will be in the interval (0,1), and the components will add up to 1, so that they can be interpreted as probabilities. Furthermore, the larger input components will correspond to larger probabilities. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Logistic Regression \n",
    "\n",
    "![image.png](img/dl13.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl14.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl15.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:28:15.191059Z",
     "start_time": "2024-08-03T02:28:15.185584Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from torch import tensor\n",
    "from torch import nn\n",
    "from torch import sigmoid\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "\n",
    "# Training data and ground truth\n",
    "x_data = tensor([[1.0], [2.0], [3.0], [4.0]])\n",
    "y_data = tensor([[0.], [0.], [1.], [1.]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:28:37.739679Z",
     "start_time": "2024-08-03T02:28:37.734799Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class Model(nn.Module):\n",
    "    def __init__(self):\n",
    "        \"\"\"\n",
    "        In the constructor we instantiate nn.Linear module\n",
    "        \"\"\"\n",
    "        super(Model, self).__init__()\n",
    "        self.linear = nn.Linear(1, 1)  # One in and one out\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        In the forward function we accept a Variable of input data and we must return\n",
    "        a Variable of output data.\n",
    "        \"\"\"\n",
    "        y_pred = sigmoid(self.linear(x))\n",
    "        return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:28:59.758902Z",
     "start_time": "2024-08-03T02:28:59.755265Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# our model\n",
    "model = Model()\n",
    "\n",
    "# Construct our loss function and an Optimizer. The call to model.parameters()\n",
    "# in the SGD constructor will contain the learnable parameters of the two\n",
    "# nn.Linear modules which are members of the model.\n",
    "criterion = nn.BCELoss(reduction='mean')\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:29:20.633660Z",
     "start_time": "2024-08-03T02:29:19.942525Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1000 | Loss: 1.3918\n",
      "Epoch 101/1000 | Loss: 0.6489\n",
      "Epoch 201/1000 | Loss: 0.6090\n",
      "Epoch 301/1000 | Loss: 0.5852\n",
      "Epoch 401/1000 | Loss: 0.5633\n",
      "Epoch 501/1000 | Loss: 0.5429\n",
      "Epoch 601/1000 | Loss: 0.5239\n",
      "Epoch 701/1000 | Loss: 0.5061\n",
      "Epoch 801/1000 | Loss: 0.4895\n",
      "Epoch 901/1000 | Loss: 0.4740\n"
     ]
    }
   ],
   "source": [
    "# Training loop\n",
    "for k, epoch in enumerate(range(1000)):\n",
    "    # Forward pass: Compute predicted y by passing x to the model\n",
    "    y_pred = model(x_data)\n",
    "\n",
    "    # Compute and print loss\n",
    "    loss = criterion(y_pred, y_data)\n",
    "    if k%100==0:\n",
    "        print(f'Epoch {epoch + 1}/1000 | Loss: {loss.item():.4f}')\n",
    "\n",
    "    # Zero gradients, perform a backward pass, and update the weights.\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:29:27.892634Z",
     "start_time": "2024-08-03T02:29:27.883786Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Let's predict the hours need to score above 50%\n",
      "==================================================\n",
      "Prediction for x = 1.0, y_pred = 0.3849 | Above 50%: False\n",
      "Prediction for x = 7.0, y_pred = 0.9694 | Above 50%: True\n"
     ]
    }
   ],
   "source": [
    "# After training\n",
    "print(f'Let\\'s predict the hours need to score above 50%\\n{\"=\" * 50}')\n",
    "y_pred = model(tensor([[1.0]]))\n",
    "print(f'Prediction for x = 1.0, y_pred = {y_pred.item():.4f} | Above 50%: {y_pred.item() > 0.5}')\n",
    "y_pred = model(tensor([[7.0]]))\n",
    "print(f'Prediction for x = 7.0, y_pred = {y_pred.item():.4f} | Above 50%: { y_pred.item() > 0.5}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "####  Diabetes Classification\n",
    "\n",
    "![image.png](img/dl16.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:33:52.443692Z",
     "start_time": "2024-08-03T02:33:52.421652Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X's shape: torch.Size([759, 8]) | Y's shape: torch.Size([759, 1])\n"
     ]
    }
   ],
   "source": [
    "from torch import nn, optim, from_numpy\n",
    "import numpy as np\n",
    "\n",
    "xy = np.loadtxt('../data/diabetes.csv.gz', delimiter=',', dtype=np.float32)\n",
    "x_data = from_numpy(xy[:, 0:-1])\n",
    "y_data = from_numpy(xy[:, [-1]])\n",
    "print(f'X\\'s shape: {x_data.shape} | Y\\'s shape: {y_data.shape}')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:34:09.243398Z",
     "start_time": "2024-08-03T02:34:09.236206Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class Model(nn.Module):\n",
    "    def __init__(self):\n",
    "        \"\"\"\n",
    "        In the constructor we instantiate two nn.Linear module\n",
    "        \"\"\"\n",
    "        super(Model, self).__init__()\n",
    "        self.l1 = nn.Linear(8, 6)\n",
    "        self.l2 = nn.Linear(6, 4)\n",
    "        self.l3 = nn.Linear(4, 1)\n",
    "        self.sigmoid = nn.Sigmoid()\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        In the forward function we accept a Variable of input data and we must return\n",
    "        a Variable of output data. We can use Modules defined in the constructor as\n",
    "        well as arbitrary operators on Variables.\n",
    "        \"\"\"\n",
    "        x = self.sigmoid(self.l1(x))\n",
    "        x = self.sigmoid(self.l2(x))\n",
    "        y_pred = self.sigmoid(self.l3(x))\n",
    "        return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:34:10.199551Z",
     "start_time": "2024-08-03T02:34:10.194234Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# our model\n",
    "model = Model()\n",
    "\n",
    "\n",
    "# Construct our loss function and an Optimizer. The call to model.parameters()\n",
    "# in the SGD constructor will contain the learnable parameters of the two\n",
    "# nn.Linear modules which are members of the model.\n",
    "criterion = nn.BCELoss(reduction='mean')\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T02:34:12.474847Z",
     "start_time": "2024-08-03T02:34:11.018698Z"
    },
    "code_folding": [],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/1000 | Loss: 0.7625\n",
      "Epoch: 201/1000 | Loss: 0.6445\n",
      "Epoch: 401/1000 | Loss: 0.6442\n",
      "Epoch: 601/1000 | Loss: 0.6439\n",
      "Epoch: 801/1000 | Loss: 0.6435\n",
      "Epoch: 1000/1000 | Loss: 0.6429\n"
     ]
    }
   ],
   "source": [
    "# Training loop\n",
    "for k, epoch in enumerate(range(1000)):\n",
    "    # Forward pass: Compute predicted y by passing x to the model\n",
    "    y_pred = model(x_data)\n",
    "\n",
    "    # Compute and print loss\n",
    "    loss = criterion(y_pred, y_data)\n",
    "    if k % 200 ==0:\n",
    "        print(f'Epoch: {epoch + 1}/1000 | Loss: {loss.item():.4f}')\n",
    "\n",
    "    # Zero gradients, perform a backward pass, and update the weights.\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "print(f'Epoch: {epoch + 1}/1000 | Loss: {loss.item():.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![image.png](img/dl17.png)\n",
    "\n",
    "The images in CIFAR-10 are of size 3x32x32, i.e. 3-channel color images of 32x32 pixels in size. http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-24T14:21:11.794120Z",
     "start_time": "2020-05-24T14:21:11.787466Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 6, 5) # in_channels = 3, out_channels = 6, kernel_size= 5\n",
    "        self.pool = nn.MaxPool2d(2, 2) # pool of square window of size = 2, stride = 2\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5) # in_channels = 6, out_channels = 16, kernel_size= 5\n",
    "        self.fc1 = nn.Linear(16 * 5 * 5, 120)  # in_features = 16*5*5, out_features = 120\n",
    "        self.fc2 = nn.Linear(120, 84) # in_features = 120, out_features = 84\n",
    "        self.fc3 = nn.Linear(84, 10)  # in_features = 84, out_features = 10\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.pool(F.relu(self.conv1(x)))\n",
    "        x = self.pool(F.relu(self.conv2(x)))\n",
    "        x = x.view(-1, 16 * 5 * 5) # Flatten the data (n, 16, 5, 5)-> (n, 400)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Run in Google Colab\n",
    "\n",
    "![image.png](img/dl18.png)\n",
    "\n",
    "https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/cifar10_tutorial.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl19.png)\n",
    "\n",
    "深度学习 Deep Learning 视频系列 https://space.bilibili.com/88461692/channel/detail?cid=26587"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "\n",
    "![image.png](img/chengjun2.png)\n",
    "\n",
    "<!--NAVIGATION-->\n",
    "< [In-Depth: Decision Trees and Random Forests](09.08-Random-Forests.ipynb) | [Contents](Index.ipynb) |[In-Depth: Neural Network Advanced](09.10.neural_network_advanced.ipynb)>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "幻灯片",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "170px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}