{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CS 20 : TensorFlow for Deep Learning Research\n",
    "## Lecture 04 : Eager execution\n",
    "### Custon training basics\n",
    "* Reference\n",
    "    + https://www.tensorflow.org/tutorials/eager/custom_training?hl=ko"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.12.0\n"
     ]
    }
   ],
   "source": [
    "from __future__ import absolute_import, division, print_function\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "tf.enable_eager_execution()\n",
    "\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variables\n",
    "Tensors in TensorFlow are immutable stateless objects. Machine learning models, however, need to have changing state: as your model trains, the same code to compute predictions should behave differently over time (hopefully with a lower loss!). To represent this state which needs to change over the course of your computation, you can choose to rely on the fact that Python is a stateful programming language:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[2. 2. 2. 2. 2. 2. 2. 2. 2. 2.]\n",
      " [2. 2. 2. 2. 2. 2. 2. 2. 2. 2.]\n",
      " [2. 2. 2. 2. 2. 2. 2. 2. 2. 2.]\n",
      " [2. 2. 2. 2. 2. 2. 2. 2. 2. 2.]\n",
      " [2. 2. 2. 2. 2. 2. 2. 2. 2. 2.]\n",
      " [2. 2. 2. 2. 2. 2. 2. 2. 2. 2.]\n",
      " [2. 2. 2. 2. 2. 2. 2. 2. 2. 2.]\n",
      " [2. 2. 2. 2. 2. 2. 2. 2. 2. 2.]\n",
      " [2. 2. 2. 2. 2. 2. 2. 2. 2. 2.]\n",
      " [2. 2. 2. 2. 2. 2. 2. 2. 2. 2.]], shape=(10, 10), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# Using python state\n",
    "x = tf.zeros([10, 10])\n",
    "x += 2  # This is equivalent to x = x + 2, which does not mutate the original\n",
    "        # value of x\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***TensorFlow, however, has stateful operations built in, and these are often more pleasant to use than low-level Python representations of your state.*** To represent weights in a model, for example, it's often convenient and efficient ***to use TensorFlow variables.*** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Computations using Variables are automatically traced when computing gradients***. For Variables representing embeddings TensorFlow will do sparse updates by default, which are more computation and memory efficient. Using Variables is also a way to quickly let a reader of your code know that this piece of state is mutable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=1.0>\n",
      "<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=3.0>\n",
      "<bound method ResourceVariable.numpy of <tf.Variable 'Variable:0' shape=() dtype=float32, numpy=9.0>>\n"
     ]
    }
   ],
   "source": [
    "v = tf.Variable(1.0)\n",
    "print(v)\n",
    "\n",
    "# Re-assign the value\n",
    "v.assign(3.0)\n",
    "print(v)\n",
    "\n",
    "# Use `v` in a TensorFlow operation like tf.square() and reassign\n",
    "v.assign(tf.square(v))\n",
    "print(v.numpy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Fitting a linear model\n",
    "1. Define the model\n",
    "2. Define a loss function\n",
    "3. Obtain training data\n",
    "4. Run through the training data and use \"optimizer\" to adjust the variables to fit the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### define model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Model():\n",
    "    def __init__(self):\n",
    "        self.w = tf.Variable(tf.random_normal(shape = []))\n",
    "        self.b = tf.Variable(0.)\n",
    "        \n",
    "    def __call__(self, x):\n",
    "        return self.w * x + self.b\n",
    "    \n",
    "model = Model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define a loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss_fn(predicted_y, desired_y):\n",
    "    return tf.reduce_mean(tf.square(predicted_y - desired_y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Obtain training data\n",
    "true_w = 3.0\n",
    "true_b = 2.0\n",
    "num_examples = 1000\n",
    "\n",
    "inputs  = tf.random_normal(shape=[num_examples])\n",
    "noise   = tf.random_normal(shape=[num_examples])\n",
    "outputs = inputs * true_w + true_b + noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3X+QHOV5J/Dvs6NdYFcitoYNRoBGVOBcJ1yJE7bk48rnwpZMsCoV7JRzh2/hINi10S7k5POlLrZVZeecUtVxOcen+CyBEsjJ2ol/1Nkc3Fk2P4SrHOeMzUIJLAn7rGBJIAhIK2ykyAFp97k/3u7bnp5+e96e7p7unvl+qrp2p6d35p3Vqp/u93nf5xVVBRERkW+o6AYQEVG5MDAQEVELBgYiImrBwEBERC0YGIiIqAUDAxERtWBgICKiFgwMRETUgoGBiIhaLCu6Ad246KKLdM2aNUU3g4ioUp588skTqjre6bhKBoY1a9Zgbm6u6GYQEVWKiBxxOY5dSURE1IKBgYiIWjAwEBFRCwYGIiJqwcBAREQtGBiIiHqo2QTWrAGGhszXZrPoFrWr5HBVIqIqajaBqSngzBnz+MgR8xgAJieLa1cY7xiIiHpky5aloOA7c8bsLxMGBiIaaL3s2jl6NNn+ojAwENHA8rt2jhwBVJe6dvIKDqtXJ9tflEwCg4jcJyKviMj+wL6VIvKIiPzE+/pmy8/e6h3zExG5NYv2EBG56HXXztatwOho677RUbO/TLK6Y/jvAG4I7fs4gL2qehWAvd7jFiKyEsCnAbwDwDoAn7YFECKirCXt2knb7TQ5CezcCTQagIj5unNnuRLPQEaBQVW/A+BkaPeNAHZ53+8C8P6IH/1NAI+o6klVfRXAI2gPMEREuUjStZNVt9PkJHD4MLC4aL6WLSgA+eYYLlbVl7zv/x7AxRHHXArg+cDjF7x9RES5S9K1U5URRVnoSfJZVRWApnkNEZkSkTkRmTt+/HhGLSOiQRPsDtqyBbj1VreunaqMKMpCnoHhZRG5BAC8r69EHHMMwOWBx5d5+9qo6k5VnVDVifHxjutMEBG1ieoO2rXL3CF06tqpyoiiLOQZGB4E4I8yuhXAAxHHPATgehF5s5d0vt7bR0SUuTTdQVUZUZSFrIarfgnA9wC8VUReEJEPA/hPAN4rIj8BsMF7DBGZEJG/BABVPQngTwA84W2f8fYRESXiMmIoTXdQVUYUZUJVK7ddc801SkTkm51VHR1VNR1EZhsdNfuDGo3WY/yt0SimzY2Gqoj5Gm5rp+e7AWBOHc6xnPlMRJXn2kUU1R0EAKdPJx92mmZOQ6ehr72ekR0mJohUy8TEhM7NzRXdDCIqCRH7/sXF1n3NJrB5MzA/37p/dNS9ayhcJTXpz69ZY072YY2GSYB3er5bIvKkqk50PI6BgYiqrNkEbrnFXFmH2U6kaU+8aX9+aCi6vX4g6/R8t1wDA7uSiKjStmyxn0RtI4bSzklI+/Odhr4WPTSWgYGISqHbPnvbyVg13ZyEuPakPXF3Gvpa+NBYlwx12TaOSiLqL66jiqJ0M9Io6v0A1XrdPNepPWnaG2xDWUclFX6S72ZjYCDqL2mGkXZ7kp6eNifd8HuOjqqOjXVuTx4n7ry5BgYmn4mocGmTrc2myTUcPWq6c7ZujR8dFDWqyEXa5G/RmHwmosqw9c0PDbnlGpKWso6a9+CiH+siRWFgIKLC2SaeLSzkM7Gr24qo/VgXKQoDAxEVzq9DVKu1P+dS5M4fQSQCLFtmvsaNbIq78rdNlqvX+7QuUgQGBiIqhclJe/993BV+sHwEYO4ygPgyErY7FMDsHxlp37dtW3z7+wkDAxGVRtL5Ac2mWWjHli+w3W0EK6UCrXcJ//APJhFerw9AFVULBgYiKo0kE7v8OwX/DsHGdrfhJ6wbjfYRUWfPAsuXl3td5jwxMBBRR2kqiSaRZM0D15FFqvFtHqQlO11xHgMRxUpbSTQvtiSxzcgIsGIFcPJk61yHvCqZlhHnMRBRJtIsh5mXZjN5YHjjDVNqW731DW65BZiZKUFdohLKNTCIyFtFZF9ge01EPho65joR+XngmE/l2SYiSsalq6VXXU0+W0XVJFSBu+823w/Mkp2uXOpmZLEBqAH4ewCN0P7rAPzvJK/FWklEvdOpjlFUrSK/BpGthlDaOkNRNY663YpY1rMoKOHSnusB/J2qRvTmEVEvJbnC79TVEtXV5F/NR80lcF22spuy190Y5CSzlUv0yGIDcB+AOyP2XwdgHsDTAL4J4GrLz08BmAMwt3r16lyiKdEg6KYaadwVvuvVe63W+jXqef/1169vf91w2eus7hp4x1BQdVURGQHwonfSfzn03IUAFlX1tIhsBLBNVa+Kez2OSiLqXtajcGyvlwe/jc0mcPPN6V+vDKOreqlso5LeB+CpcFAAAFV9TVVPe9/vATAsIhf1qF1EAyeLcfvB2kTPP59Js5wcPbrUFZUGk8zxehUYPgTgS1FPiMhbRMzAMxFZ57VpvkftIho4aZelDNcm6uX6BCtXdl8y2zc9Pbgzml3lHhhEZAzAewF8PbBvk4hs8h5+EMB+EXkawJ8DuEl70b9FNKDSjttPe2JO49Sp+G6ret3+XK1mgsL27dm3q99w5jPRAEq64llQ0ollWavVousjNRrm80Sd0qq+8lpWypZjIKIS8HMDt9xiHu/e7d6lMjNj1jroVlYBZWHBfseTtpuMDAYGogHhOn8gyswMsGNH50qmUep18367d7evc9CNWs2U2vZnKtfrwAUXmGB3+nT0e5w+nf9s7H7CwEDUR+ImhaWpebRzZ3ftCS5ws2WLqVeU1sICsGuXuUPYvRv4xS+WaiDNz5v3OP/81p+Zn1+qjUSdMcdA1Cc6VUEdGkrW/x7MQ3R7mhgbM+1ZudKcnIsmYoLJoI5GYo6BaMB0uiNI0v/ebAK33bbU7dQtfzW0MgQFwLSlyKqwVcHAQNQnOk1c27ixPQEcNUy12TTdLufOZd/GMmBtpM4YGIj6RNwdQbNp+uWDV/8iwLXXmitoPycxM2O6o8rSw+yvyZwljlDqjIGBqE/ETVyzVUB97LHWUUp3313c5LUoW7d2FxyGh4HzzmvfP+gL8LhiYCDqE3HrJdu6T8J3BmW5UwDMsFTAnMiHhzsfX68vDV8VAV5/vf151kZyw8BA1CfiZjOvXFls2zpZvrx938KCma/wt3/rNjlu2zYzumr58uhhscuXMyi4SjGPkYh6zXbyDw9V9Sev+V57rZj2ujp9Onr/woLp3nK5k/E/bxbVYwcdAwNRRcSd/DsNVT17tnftTGpoKL6OkWv3lv95V6+OLrTHpLM7diURVUTcyT/uKrnsV8pZFrc7ejR99VhiYCCqjLiTf9xQ1X68UrblHFavjk/CkxsGBqKKiDv5x10lb9yYf9t6qVYDNm2KvyuYnDRVY7kgT3cYGIgq4soro/dv3Nh6lQyYfvszZ8y6yPfc07s29sLiollsh3cF+enFCm6HReSHIrJPRNoq34nx5yJySESeEZHfyLtNRFXTbJrJaFH27DFfJyfNFXM4mVu1BWpGR4HZWfvENv/OiXcF+enVqKR3q+oJy3PvA3CVt70DwA7vKxF5tmyxj84J5h5+//erFwjCglf+UdVimUTOXxm6km4E8EU1HgfwJhG5pOhGEfVC3PoJQXEji1auND8rYqqZVp0fFJhELk4v7hgUwMMiogDuUdXwkh+XAng+8PgFb99LPWgbUWHi5iWET362sfmAKWldlrLWaYW7jyYnGQiK0Is7hneq6m/AdBndISLv6uZFRGRKROZEZO748ePZtpCox5pNU+7BdUU113pBVcclOMsh98Cgqse8r68AuB/AutAhxwBcHnh8mbcv/Do7VXVCVSfGx8fzai5R7vw7Bdv6yVHdRpOTwIUX5tuuNMbGsnmd+Xn3dagpP7kGBhEZE5EV/vcArgewP3TYgwD+jTc66Z8B+LmqshuJ+lbUDOagoaHonMPJk7k3rSujo2bd5U78SWlDQ+37glzXoab85H3HcDGA74rI0wB+AOAbqvotEdkkIpu8Y/YAeA7AIQB/AYDLdVPfiEoudypRsbCwtD5C8Oq5LDOYx8Zaq6FecIHbSCh/VNXi4tKQVJsjR3jXUCTRMhVgdzQxMaFzc21TIohKJZxcBoCREVPQLsl/u0bDjNNvNs2EtX7hJ5ptSfXRUY5CypqIPKmqE52OK8NwVaK+FNVl9MYbyRfDOXLE3G30U1AA7AXvfOxSKg7LbhPlJMuqprar6irzC94B9qBX9sqw/Yp3DEQ5KUtOoFdcVlnzhQvedSp/Qb3FwECUk40b3U6W9bq9O6UqRkdNxVP/BB/+3MPDS2sxR81g5hoK5cLAQJSDZhPYtatzPmFoCHj11fjhq1Wwc6epeHr4sPnMu3e3lrL4q78CTpywF7xj+Yty4agkohysWdOfeYEo/qgpKj+OSiIqSLPpFhSG+uB/n4jpMnMpBEjVwVFJRBny5y64qHp5bMB0G+3a5VYIkKqjD65ZiMqjU7mLflOruRcCpOpgYCBKIVzyol/zCiLt1V1HR90LAbquO0HlwMBA1CW/2+jIkaXaRknG8lfJ7t1mZFF41JDL/IOo3xMrqJYbRyURdamf7xCC6nUz1DRKVD2ocI0j2++Jo5l6j6OSiHLkOvKo37nMP7CVtWC5i/LiqCSihJpN4Pbbi25F73RaB6LT8pu2ZUlZ7qK8eMdAFNIpUbp5s6mSOijSnsBZ7qJ6GBhooIWDwMxMdKJ0ZmbpuPn5ghvdQ1mcwFnuonpySz6LyOUAvgizipsC2Kmq20LHXAfgAQA/9XZ9XVU/0+m1UyefZ2bMX+bCghmIPTVlCr3QQIlKnIpE1zey7e9ntZqZvMYTeP9wTT7nmWM4B+Dfq+pT3rrPT4rII6p6MHTc36jqb+XYjlYzM8COHUuPFxaWHjM4DJSoyWi2k/+gBQXAzMxmUBhMuXUlqepLqvqU9/0pAM8CuDSv93O2c2ey/dS3OComHpPDg6snOQYRWQPg1wF8P+Lpa0XkaRH5pohcnXtjbFM1bfupb7me+Pp10lpQ+DMyOTzYcg8MIrIcwNcAfFRVXws9/RSAhqr+GoDPA/ifMa8zJSJzIjJ3/Pjx7htUqyXb74pz/ivH5cQ3NNQfVVDjiCwtssPkMAEAVDW3DcAwgIcAfMzx+MMALup03DXXXKNdm55WNV3Grdv0dPevOTurOjra+nqjo2Y/lcrsrGqjoSpivo6NRf85DNpGgwHAnDqci3O7FhIRAXAvgGdV9c8sx7zFOw4isg7mDibfwYDbtwPT00t3CLWaeZwm8RyVxexUYnJmBli2zFyihbdly8zzFKnbm7Oomj1nz7YXhxs0tnpHNMBcokc3G4B3wgxTfQbAPm/bCGATgE3eMXcCOADgaQCPA/jnLq+d6o4hDyLRl2Ei0cfb7lrC29gY7zpCOt2che8Igr++RsP+a67Vir9qz3oLf6bhYdWREfvvjvofHO8YcgsMeW6lCwy2M06jEX18krPQ6KgJJMuX2//nh8+Afcz2q67VzK8pLmjY4ne/bvX6UpCs1802oH825GFg6KWkOYZeXCb26f/6uJO77Tk/PtuCSr9u/g0rU2Dkcw0MfT7eokeSzvlPOwIqjj/s9sgR4LbbTEd8OH8hUtmRU3FDTFWj9/vzFTZuzL49Zeb/rrpJgdFgY2DIyuSkKS6/uGi+xo31c10UOK1z59rPlsHAcfPN7YnvoaFSJ76jCrJ14n+ke+/Np01ldeWV5ivLXlNSDAxFCI+MitPr2VWqpkRIMFjUaqUJFv7Nme1XF/XrWlgA7r57sCqiAsBjj5mbQttdFmc2kw0DQ1G2b1+6og9us7OtXVKbNpnunyItLrYHiwKDxuSkKe4WVcr5Pe+J/hlbN1M/UzXdRSx7TYm5JCLKtpUu+Zy32dnoUUll3mq13LOb4aGp69f378ijtWu7+7lgAto2jJcGB5h87iOTk8CpU/a7C6B1wh4ALF9eTFt9CwvROQwRYMOGTN4imNbZuBHYu7c/7wzWrwcOHGifl7l+fefJaX53UZIUGBEDQ5X5/9tVl7ql/K+nTrnnMXpt797ogHF1fA1F24zncCX1fvO975nPuGePObE3GqYr7dFHl/75p6dZCI8y5HJbUbZt4LqSsjI9Xc0pvtPT1rH409P9230U7hIKf/ZwdxC7i6gTOHYl5baCW55Sr+BGrTZsMFfxJWX7C/0CpvHR2vaBrZjeaJg7BiJXriu4sSuJTJ+E7WJ1/fqiWwexbHdgB84uCBaxtC1gCJ9HOYbW5o3zECgvDAwUzxY0ShowhqC4AztagoW/7UP+60CltXZt+9BS21QWzkOgvDAwEIAuSllXLGAIgF/FwciAUaY7jE9+sr26yqZN0cFi0Ep8UA+5JCLKtjH5nK3ci6zNzpY66b1o2fZhbc+bEyzIG0wmRy0oxEJ4lBSYfCZXa9aY0klhuSc3Sz7O1PY/4yhWYQ2O5fKeImZIqr+oULj4XRgT0JQEk8/krLAia9u3R184z84CY2NQoG3rJVuX1Gq8GNkldRiXpn7PuIqoUZiApjzkHhhE5AYR+bGIHBKRj0c8f56IfMV7/vsisibvNlGroousteU3MInmPacxXlcMYWk7hQsig0W/BIzghDTXEz4T0JSHXAODiNQAfAHA+wCsBfAhEVkbOuzDAF5V1SsBfA7AXXm2idoVWWQtah3m3/s94PbbgfnQ6t+/hDMtgcLfnsHaSgaMvbUN1iU8XE74nNlMuXFJRHS7AbgWwEOBx58A8InQMQ8BuNb7fhmAE4DJfdg2Jp+z18tZs8H3GhrKL5H7IczqP6IWmVguOuHtJ70jn1u1KnJAwMiIWZ6TM5upWyhD8llEPgjgBlX9iPf4FgDvUNU7A8fs9455wXv8d94xJ2yvy+RzdbkmVfP0eczgDkQnvXu8+kWs8P/MV1etxcpjBwppC/WHvks+i8iUiMyJyNzx48eLbg4lEMwh3HprsUEBAP4A20vdJeULd0mtfPFgdPHBS9MnvYmC8g4MxwBcHnh8mbcv8hgRWQbglwCEepcBVd2pqhOqOjE+Pp5Tc6lbtglyzabJGfg5hDLXNXo7DkQGjC9gulQBo82LLzJgUKbyDgxPALhKRK4QkREANwF4MHTMgwBu9b7/IIDHNM/+LcpcVAJ5asrs37wZOHu26BamU5U7jDa2gFGSZVqpvHINDKp6DsCdMAnmZwF8VVUPiMhnROS3vcPuBVAXkUMAPgagbUgrlVvUmPszZ8z+8MiifhK8w6jJ0tfSBwzbMq0MGOTJPcegqntU9Z+o6q+o6lZv36dU9UHv+39U1d9V1StVdZ2qPpd3m6h7UV1GUbOmgcGafKVqho9u2gTc2DiAmiiuaCj+enYpaDyM9eUMFD5bwMhoxT2qDpbEIGdRI4pGRoA33og+vtEwdwynT/emfWUQVaKiY8mRkpcGsVq71qw5SpXRd6OSKL3EFVRDorqMbEFBxEy+Ou+8blpaXVF3SR0nENpKg5SgUm2sgxGjpC66KPkfFpWPy2SHsm2c4JZcFhVUky6h2c3PVH0LVkcN//4zm0A4O1v8B+1mW7UqxYemLMBxghvvGAZEXILYVZK6PI1G8p8ps0YDmJ62L5oDxJeomJw03UaLi+arX/qiK5OT1bzDsI2S4h1G6TAwDIgsKqhGdYmMjADDw637gifIqJ+poqNHTY/P7t1Li+jU62aLqnVUiBIvnhTr5pujA8bV5V9xr18xMAyIThVUZ2aAZcvM/8dly6JHLk5OmpNfvb60b8UK4CMfgbUYnP8z/h1EVfm/p+CV/4kTZsvkLiBPcWt6D5X4FBCVw+CQ2p4o8V8FZSkuAeoPivFnJS8smMe2/4O/+MXS9/PzwK5d5nX8EyTQmuT2378bwSDUCyMj7d1FfV3FdGGhPVjMzhbdKjvbkNqhIQaNLLkkIsq2MfncHVsC1LbqZq3W/hqNhj232GioTk+3J7n910qaqxQxr9er3KifjO9lpdlKWb++d/8YWW31Ov8BA1CG6qp54TyGbMUlVMN/HkND7fvCr5Xln1SjYZ9Al1Rc20RM/qC03UFltmEDsHdv0a1ws3696VobUJzHQM5qNftz4QEjnUYZZRkUhoayCwq1mpmVbFPB66PysOUwZmdN31yZ7N0b3RXlb7YE24BhYCBMTdmfCw9n7eUoo8XFbF5ndNTkQbZvj89ZJBm6Sw4mJ4HXX6/WSCk/wXb11elmg1YcAwNh+3b7c0eOtP6fKPMoo1pt6cTv3wWFR0lt22b/+fBnpRxF3WV0mijSSwcPtpYLvuUWcyeRtnxAVbgkIsq2MfmcvXrdnr8bHo7O35VxAm69bv+MflLZJQFNJVOWxPfwcOtjf2p/RUYpgDOfKStnz5p1FcLKmKidn4++iAuuGREn6Wxw6hFbHqPXdxnhxUXUS04FFyHpAwwMBAA4eTL++fAJ17+jLqOoE3tUSRCbQSoXXnnbt5tkVDBYzM4Cy5f3vi2uVxUV6I7icFUCYC8NHVSrmf+DK1cCr71W3pXZRNoT152G2QZFlc6mPuAvKZjn6lFRf3zhNoRr1wMmObZtW+634RyuWjFFX0Rs3dpe8yjMnyQ7P1/eoABED6l1LebnlwunPjQ5aWqYRHVJzc621nVZv769iyqqMFhYpz80263r/Hx7V1SRJwWXRETSDcCfAvgRgGcA3A/gTZbjDgP4IYB9cEyKqPZf8jmLkthZtSMuCV2FzfZ7i/odhzd/pjWRqkZPgQ+OYAjXlHf5T9upDr1ftz2nk4LreTavwHA9gGXe93cBuMty3GEAFyV9/X4LDLaRMrba/nlzOYmWafPLbXQaGBL+fz49zdIXlEI3tVM6DYsTiT8u5Umh0MDQ8gbABwA0Lc8xMKj9IsL/G7HJs6ZP8LWLPvG7/D/qxe+EKLVOV13+ib/bk0IHZQoM/wvAzZbnfgrgKQBPApjq8DpTAOYAzK1evTrVL6dsurk4SHOnmfTkWYbupbGxzr+jsnTJEcWy9dkG/1irescA4FEA+yO2GwPHbPFyDGJ5jUu9r78M4GkA73J57367Y+jmhNbt343tgiWuCOXsrOrISHFBYfny9nlFgGlTsM1l65IjihV3hdaPOQbz/rgNwPcAjDoe/8cA/tDl2H4LDKrJr+Jd7jSjXjOuizPu785l1nAe28iI/Y4lPMs5p7tvomLk0C9adPL5BgAHAYzHHDMGYEXg+/8D4AaX1+/HwNBJ+G/EdrLsNKih04k46uo6+N7drKuQZqvX3U/4vGMgiucaGPKax/DfAKwA8IiI7BORuwFARFaJyB7vmIsBfFdEngbwAwDfUNVv5dSeSguWc1A1X0+dil9rOWq4tMvM3/Akt/B7+6u89crJk52XJfXFrVJHRAm4RI+ybf1wx5DkLtF2JVyv21+j29FE/qptve46srXX/1yu3a0clURkh6JzDHluVQ8MSfNK3fSdpzmp25bnzHqr1VrnFMT9TnjCJ0rPNTCwVlIBbHWJbDV6kh4P2EuylEm4rEyzabrAjh413URbt5azgitRVbFWUonZqnfa9nfTdx5cUEckfvnOooRzBJOTJtAtLpqvwaBQdC0pokHCwFAA12SqL3ySD69KZhM80e7alf2SnGNj3f+sH9hcTvhRyfc+Kn1PVDoMDAXo9g7AdjXtIiq4zM4mbXmr8883r+EacPxilX5gA9xO+LYRVlxQhygnLomIsm1VTz6rJk+m5lUALm25iyRJ7nAbXecdcOIaUTbA5HP/cEkkj466dS+FzcwAO3Z03zYRc5p20Wi0JpRti+eEk9LdJN+JqB2Tz33EZVnKbrtW9uzpfIxNve6+AA7Q3lXEiWtE5cTAUAGuaxB3s1Zxp+U8bYaHzUqEUSftOMEA5nrC7zb5TkTdYWCoANer8iRX775uh7FeeKH5Gjxpu/IDWJITftrkOxG5Y2CoAJercpeulfDQ0JmZ+NpH09P2JW6DS9T6J23XEUorVy61Y8sW026e8IlKxCVDXbatH0YlJZV2VFLS5Tr9ktadaiaFRxAF21mvt6+jMDzcvrYDF9Mh6g2wVlK1pKkF5PKzSWsnBdc6mJ21H5d0+dFO5cKJKD+ugYHDVUsgajiq6/DTuJ8FlmoPJf1nDg4ZtQ0XBZIPGXUdokpE2eNw1QpJMrM3nCfYvDn6Zzdvbp1VnFQwkR032inpkNGk5UCIqPcYGErAtaheVM2g+fnon52f776yajiRbTtp1+vJk8Wck0BUfrkFBhH5YxE55q3gtk9ENlqOu0FEfiwih0Tk43m1p8xcr6JdJrqlFTVk1HYy37Yt+etzTgJR+S3L+fU/p6r/xfakiNQAfAHAewG8AOAJEXlQVQ/m3K5S2bo1Ok8QvoruZgJbErYuJ/+kndVaCZOTDAREZVZ0V9I6AIdU9TlVfQPAlwHcWHCbes71KjquSyetThPdOMGMaHDkHRjuFJFnROQ+EXlzxPOXAng+8PgFb18bEZkSkTkRmTt+/HgebS2Uy4k3rksnyczjKHET3YhosKQKDCLyqIjsj9huBLADwK8AeDuAlwB8Ns17qepOVZ1Q1Ynx8fE0L1VZcXcWtqDhKm1gIaL+kSowqOoGVX1bxPaAqr6sqguqugjgL2C6jcKOAbg88Pgyb9/A6rSime3OwhY0XE74HBVEREG5JZ9F5BJVfcl7+AEA+yMOewLAVSJyBUxAuAnAv86rTWUXnqzml6kG3Pr0bUndcGJ7ZARYsQI4eTJ9IpmI+k+eOYb/LCI/FJFnALwbwL8DABFZJSJ7AEBVzwG4E8BDAJ4F8FVVPZBjm0otjyUso+4k7rsPOHGCiWQiisaSGCXCchFElCeWxKigIspFdMppENHgYWAokV6Xi4gqsRFcepOIBhMDQ4n0ulxEHjkNIqo+BoaChbtygN7NMHYt3kdEg4WBoUBFd+WwBDYRRWFgKJBrV05eCWKWwCaiKAwMBXLpysnzroIlsIkoCucxFMi2ZGZwuUyXY4iIXHAeQwW4dOUwQUxEvcbAUCCXrhwmiImo1xgYCtZpHQYmiImo1xgYSo4JYiLqtbzXfKYMcI1kIuol3jFq1R3wAAAGmUlEQVRkJOlcAxavI6Ky4h1DBpIusJN2QR4iojxxHkMGks414NwEIipCofMYROQrIrLP2w6LyD7LcYe9Vd72iUjPzvRZd+MknWvAuQlEVGa5dCWp6r/yvxeRzwL4eczh71bVE3m0I0oe3TirV0ffAcTNQUhyPBFRL+WafBYRAfAvAXwpz/dJIo81CJLONeDcBCIqs7xHJf0LAC+r6k8szyuAh0XkSRGZyrktAPLpxkk614BzE4iozLpOPovIowDeEvHUFlV9wDtmB4BDqvpZy2tcqqrHROSXATwC4A9U9TuWY6cATAHA6tWrrzkS1RfjgIlfIhpUuSefVXWDqr4tYvODwjIAvwPgKzGvccz7+gqA+wGsizl2p6pOqOrE+Ph4t81mNw4RUQd5diVtAPAjVX0h6kkRGRORFf73AK4HsD/H9gBgNw4RUSd5TnC7CaGks4isAvCXqroRwMUA7jf5aSwD8Neq+q0c2/P/scQEEZFdboFBVW+L2PcigI3e988B+LW83p+IiLrDWklERNSCgYGIiFowMBARUQsGBiIiasHAQERELRgYiIioBQMDERG1GMjAwGU1iYjsBm5pTy6rSUQUb+DuGPJYj4GIqJ8MXGDgsppERPEGLjDELbdJREQDGBi4HgMRUbyBCwxcj4GIKN7AjUoCuB4DEVGcgbtjICKieKkCg4j8rogcEJFFEZkIPfcJETkkIj8Wkd+0/PwVIvJ977iviMhImvYQEVF6ae8Y9gP4HQDfCe4UkbUwS3teDeAGANtFpBbx83cB+JyqXgngVQAfTtkeIiJKKVVgUNVnVfXHEU/dCODLqvq6qv4UwCEA64IHiFns+T0A/oe3axeA96dpDxERpZdXjuFSAM8HHr/g7QuqA/iZqp6LOYaIiHqs46gkEXkUwFsintqiqg9k3yRrO6YAeFWNcFpEou5UsnYRgBM9eJ888TOUAz9DOfTDZwC6/xwNl4M6BgZV3dDFmx8DcHng8WXevqB5AG8SkWXeXUPUMcF27ASws4u2dE1E5lR1ovOR5cXPUA78DOXQD58ByP9z5NWV9CCAm0TkPBG5AsBVAH4QPEBVFcC3AXzQ23UrgJ7dgRARUbS0w1U/ICIvALgWwDdE5CEAUNUDAL4K4CCAbwG4Q1UXvJ/ZIyKrvJf4IwAfE5FDMDmHe9O0h4iI0ks181lV7wdwv+W5rQDaKhCp6sbA988hNFqpZHradZUTfoZy4Gcoh374DEDOn0NMjw4REZHBkhhERNSCgSGGiPyJiDwjIvtE5OFAbqRSRORPReRH3me5X0TeVHSbkoorv1J2InKDVxrmkIh8vOj2JCUi94nIKyKyv+i2dEtELheRb4vIQe/vaHPRbUpKRM4XkR+IyNPeZ/iPub0Xu5LsRORCVX3N+/7fAlirqpsKblZiInI9gMdU9ZyI3AUAqvpHBTcrERH5pwAWAdwD4A9Vda7gJjnxSsH8XwDvhZnE+QSAD6nqwUIbloCIvAvAaQBfVNW3Fd2ebojIJQAuUdWnRGQFgCcBvL9i/w4CYExVT4vIMIDvAtisqo9n/V68Y4jhBwXPGIBKRlFVfTgww/xxmDkjlRJTfqXs1gE4pKrPqeobAL4MUzKmMlT1OwBOFt2ONFT1JVV9yvv+FIBnUbFKC2qc9h4Oe1su5yQGhg5EZKuIPA9gEsCnim5PBm4H8M2iGzFAXMrDUA+JyBoAvw7g+8W2JDkRqYnIPgCvAHhEVXP5DAMfGETkURHZH7HdCACqukVVLwfQBHBnsa216/Q5vGO2ADgH81lKx+UzEKUhIssBfA3AR0M9ApWgqguq+naYu/51IpJL195AruAWlKDkRxPAHgCfzrE5Xev0OUTkNgC/BWC9ljSx1GX5lbJzKQ9DPeD1y38NQFNVv150e9JQ1Z+JyLdhljXIfFDAwN8xxBGRqwIPbwTwo6LakoaI3ADgPwD4bVU9U3R7BswTAK7yFqUagVmn5MGC2zRwvMTtvQCeVdU/K7o93RCRcX9EoYhcADOgIZdzEkclxRCRrwF4K8xomCMANqlq5a72vJIj58EULgSAx6s2ukpEPgDg8wDGAfwMwD5VjVwZsGxEZCOA/wqgBuA+rypAZYjIlwBcB1PR82UAn1bVSpWvEZF3AvgbAD+E+f8MAJ9U1T3FtSoZEflVmHVrajAX9V9V1c/k8l4MDEREFMSuJCIiasHAQERELRgYiIioBQMDERG1YGAgIqIWDAxERNSCgYGIiFowMBARUYv/ByMbmQQRy6jKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current loss: \n",
      "18.592113\n"
     ]
    }
   ],
   "source": [
    "plt.scatter(inputs, outputs, c='b')\n",
    "plt.scatter(inputs, model(inputs), c='r')\n",
    "plt.show()\n",
    "\n",
    "print('Current loss: '),\n",
    "print(loss_fn(model(inputs), outputs).numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run through the training data and use \"optimizer\" to adjust the variables to fit the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = 10\n",
    "batch_size = 64\n",
    "learning_rate = .1\n",
    "\n",
    "data = tf.data.Dataset.from_tensor_slices((inputs, outputs))\n",
    "data = data.shuffle(500)\n",
    "data = data.batch(batch_size = batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:  1, w: 2.79, b: 1.95, mse_loss: 4.200\n",
      "epoch:  2, w: 3.03, b: 1.97, mse_loss: 0.998\n",
      "epoch:  3, w: 2.95, b: 2.03, mse_loss: 0.982\n",
      "epoch:  4, w: 2.96, b: 2.01, mse_loss: 0.979\n",
      "epoch:  5, w: 3.00, b: 2.07, mse_loss: 0.987\n",
      "epoch:  6, w: 2.98, b: 2.04, mse_loss: 0.987\n",
      "epoch:  7, w: 2.99, b: 1.95, mse_loss: 0.988\n",
      "epoch:  8, w: 3.02, b: 2.00, mse_loss: 0.980\n",
      "epoch:  9, w: 3.02, b: 1.97, mse_loss: 0.985\n",
      "epoch: 10, w: 2.98, b: 2.04, mse_loss: 0.984\n"
     ]
    }
   ],
   "source": [
    "# When using tf.train, you read the document (https://www.tensorflow.org/guide/eager)  \n",
    "w_hist = []\n",
    "b_hist = []\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    avg_loss = 0\n",
    "    tr_step = 0\n",
    "    for mb_x, mb_y in data:\n",
    "        with tf.GradientTape() as tape:\n",
    "            mb_yhat = model(mb_x)\n",
    "            mb_loss = loss_fn(mb_yhat, mb_y)\n",
    "        dw, db = tape.gradient(target = mb_loss, sources = [model.w, model.b])\n",
    "        \n",
    "        model.w.assign_sub(learning_rate * dw)\n",
    "        model.b.assign_sub(learning_rate * db)\n",
    "        tr_step += 1\n",
    "        avg_loss += mb_loss\n",
    "    else:\n",
    "        w_hist.append(model.w.numpy())\n",
    "        b_hist.append(model.b.numpy())\n",
    "        avg_loss /= tr_step\n",
    "\n",
    "    print('epoch: {:2}, w: {:.2f}, b: {:.2f}, mse_loss: {:.3f}'.format(epoch + 1, w_hist[-1],\n",
    "                                                                       b_hist[-1], avg_loss))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XucVXW9//HXBxgdrmKAaFwEExFvDDKCiIMo2VE0SUXyF5LmKUwsxaPnlNo9K0vzqBXihZNomhpYmmleUhKFMCBUriZCiKLcJJDrXD6/Pz4zzp257WHNrHk/H4/1YO+91t77szdr3vu7vuu71jJ3R0RE0qVV0gWIiEjmKdxFRFJI4S4ikkIKdxGRFFK4i4ikkMJdRCSFFO4iIimkcBcRSSGFu4hICrVJ6o27du3qffr0SertRUSapQULFmx09241LZdYuPfp04f58+cn9fYiIs2Smf2rNsupW0ZEJIUU7iIiKaRwFxFJIYW7iEgKKdxFRFJI4S4ikkIKdxGRFFK419dbb8Gdd8KHHyZdiYhIJQr3+tiyBc44AyZNgp494YorYMWKpKsSEflYYkeoNltFRfDFL8K//gXTp8OsWXDvvTBlCoweDVdfDaNGgVnSlYoka/NmePvtmDZsAPeYoPR2dY/VdL8+zwE46CDIyYHjjoPOnRv/O0hQjeFuZtnAS8D+xcvPcPfvVlhmf+B+YDCwCfi8u6/OeLVNwY9/DH/8I/ziFxHyX/wi/OQncNddEfCnnw5HHw2TJ8P48dC2bdIVizSO/HxYs6Y0wCtOW7YkXWGpksZWScgDHHpoBP3AgTHl5ECfPtAqHR0a5mU/bFULmBnQ3t0/MrMs4GXgKnf/W5llJgHHuftXzexC4Fx3//zeXjc3N9eb3bllnnkGzjwTvvAFeOCByq3z3bvh4Yfhtttg0SLo2hW++tXovjnkkGRqFmmIsq3vitOaNVBYWLrsfvtB375w2GGl06c+Ff8edFCEplnp303J7Zru12XZqp5bwh3WrYPXXotp0aL49803Y4scoGPHaNWXhH5ODhxzTJNqpJnZAnfPrXG5msK9wou2I8L9cnefV+bxZ4DvuftcM2sDvA908728eLML99WrYfBg6NED5s6F9u2rX9YdXnopQv7xx6FNG/j856PL5vjj91nJIjWqa+v7oIOqDu/DDoNPfrJ5tnp37IDFi8uH/uuvw7ZtMb9VKzjiiPKBP3AgHHxwIt2vGQ13M2sNLAAOB37l7t+oMH8xcIa7ry2+vxIY6u4bq3vNBoX7yJGVHxs3LlrIO3ZE33dFl1wS08aNMHZs5fmXXx4B/M47MGFC+XlFRfDuu7BpEzz6aHTNVPStb8GnPx0rxuTJpY/v3BnP3bw5bg8cCAUF0KVL+RXjtttipXn+ebjxxsqvf9dd0L9/dAn9/OeV5z/wAPTqBY88EqN4KpoxI7Yk7rsvpoqeegratYuupUcfrTx/1qz495Zb4Mkny89r2xaefjpu//CH8Je/lJ/fpQvMnBm3r7sufhzL6tkTfvObuD15cnyHZR1xBNx9d9yeODFaWmUNGABjxsCcOTBtGuzZA9nZMe2/P5xwAtx8c9Qxdmz8P5Y1ahR8+9tx+8wz4/+prLPPhmuvjdv7et0DuOYa+OxnY6f9ZZdVnl/dulfim9+M//unn4Zf/xp27YrPuGtXTGWZxffWtm3pv//1X5CXB8uWxfpRUVrXvV27oiV/0kkR+n/5C3z0Uen8rKzYIh83Lv6uH344fhDK/l0PGxbdtgDnn1+67pV8pnqobbjXaoequxcCOWbWGfi9mR3j7ovrUdREYCJA79696/r05Pzzn/D++9EKP/TQuj23bVs4/HC4/npYsgR+9rN4rf33jxXr4IOjZS+14x7BtHUr/PvfMf31rzB1arSwOnWK7rHNm0s3tZcsiVBp1y6WycqK77/kB2DVqmil9uyZ6EerE/dodefnw/z58N57sHAhrFwZP24l0+7d8YNVVlZWfO5OnaB7d7j00vjRevfdaERUNGpUNCxWr94Xn6zpyM6Obqbvfz/uT5wIS5dGwG/fHv/u3Bn733bvjmXMYqu+Q4f4d82a2PpJYOdtnbplAMzsO8AOd7+lzGPp7Za59174ylfghhuqblHXVWEhPPFEtNRfeilWgksvhSuvjE1cKW/XrgitV16JlvmcObB+fczr1ClaRsOHR+tq6ND4PiHCb9OmGNW0Zk3pVPb+Bx+Ufy+zaIn17l06HXpo+fsHHth4m+LuEQTvvx/TBx+U3q742Pr1pT9eZbVvHw2GkumQQ8p3o/TtW/odSWYUFMRWVUkffknXTsl6CrHulN15O3RovRsTGeuWMbNuQL67bzGztsCzwE/d/ckyy1wBHFtmh+p57j5ub6/bLML973+Hk0+OVs1TT0Hr1pl9/QUL4PbbY3OuoADOOSc2DU85peUOpfzgg9IQf+WV+I727Il5hx8eIX7SSRHoRx3VsD7eXbuiK6S68F+zprRFVqJDh72Hf48e0TIu66OPKgd1dcFd8lnL2m+/COru3csHd1X3FdxNx/vvl99x+9prsHx5/Chfe210FdZDJsP9OGA60Jo46OlRd/+Bmf0AmO/uTxQPl3wAGARsBi5097f39rpNPtw3bowdqGYRMF26NN57rVsX/Y1Tp8b75uREyF94YXQfpFVhYWzmlgT5nDnRrQARaLm5pUE+bFiE177kHq2vvYX/hg3ln9OqVexY7N69tBW+fXvl127VKnZOVhXSFR/r3Lnl/tinzc6d0U3YuXM0VuqhUUbLZFKTDvfCwuinfOklePnlCJl9YedOeOih6LJZvDj++CdNiuGU+zrYGsO2bfDqq6VBPndu9J1DfNaSID/ppPhhbQ4/bDt2lG/9l4T/Bx9EF051od21a+a3BKVFULg3xA03xIiYe++F//zPff/+7rFn/rbb4E9/ilbs+PFw1VXRX9ccuEfQle1ief312CQ1iwO9SoJ8+PDoD1brVKRGCvf6evxx+Nzn4MtfhnvuSbqaGPZ3++0x2mPHDjj11Bgvf9ZZTWdMcckIlsWLy+/4fO+9mN++PZx4YmmQDx2a+kO/RRqLwr0+/vnP6ILp1y+6Y7Kzk66o1IcfxpbEL34R3QCHHx4jbC65JMbi7k1+fvwwbN9e/t9MPbZjR+XDust2sRx7rIZ7imSIwr2utm+P1uW6dbEDta7j2feVggJ47LHospk7Fw44IEb07NpVfRgXFNTtPcxiTHjJ1L595dtVPfapT0WY9+jROJ9dRDJ7EFPquccBCkuWxPljmmqwQ7SAx42Lad48uOOOGHHSvn204A8+uPrwrSmcS25nZ6v/W6SZU7gD/PKXMUrlRz+Kszo2F0OHwoMPJl2FiDRBTWSPXIJeeSXOnXHOOXEODhGRFGjZ4f7++3DBBXEO5+nTm87oExGRBmq53TL5+XEmvi1bop9dQ/NEJEVabrh/4xtxBOpvfhND9UREUqRl9kM88gj87//C178eR36KiKRMywv3JUvilALDh8fJ/0VEUqhlhfvWrXDeeXFa1EcfjXO2iIikUMvpc3ePQ/VXroQXXojTsoqIpFTLCfebb4bf/x5uvRVGjEi6GhGRRtUyumVeeCEujjtuXNUXEBYRSZn0h/s778QVjfr3h2nTdM4UEWkR0h3uu3fHEai7dsWZFHV9SRFpIdLd53711XHmxJkz4cgjk65GRGSfSW/Lffp0uPPOOBL1vPOSrkZEZJ9KZ7j/4x9xUenTToMbb0y6GhGRfS594b55M5x/flxd/re/1eXdRKRFSlfyFRXBRRfB2rUwezYcdFDSFYmIJCJd4f7DH8LTT0df+9ChSVcjIpKY9HTLPPUUfP/7cPHFcNllSVcjIpKodIT722/HqXsHDoxWuw5UEpEWrvmH+86dsQPVLMazt22bdEUiIolr3n3u7nD55fDaa/CnP8FhhyVdkYhIk1Bjy93MepnZi2a21MyWmNlVVSxzgJn90cxeK17mS41TbgV33RUHK333u3DmmfvkLUVEmoPatNwLgGvcfaGZdQQWmNlz7r60zDJXAEvd/bNm1g1YYWYPuvuexigaiNMKXHkljB4N3/52o72NiEhzVGPL3d3XufvC4tvbgGVAj4qLAR3NzIAOwGbiR6FxrF8PY8dCz57wwAPQqvnvOhARyaQ69bmbWR9gEDCvwqxfAk8A7wEdgc+7e1EVz58ITATo3bt33asFKCiIU/hu3Ahz5sAnPlG/1xERSbFaN3nNrAMwE5js7lsrzP4PYBHwSSAH+KWZdar4Gu5+t7vnuntut27d6lfxr38NL74IU6fCoEH1ew0RkZSrVcvdzLKIYH/Q3R+rYpEvATe5uwNvmdkq4Ejg1YxVWuLSS+O0AmPGZPylRUTSojajZQyYBixz91urWWwNMKp4+e5Af+DtTBVZTuvWCnYRkRrUpuU+HJgAvGFmi4ofux7oDeDuU4EfAveZ2RuAAd9w942NUK+IiNRCjeHu7i8Tgb23Zd4DPpOpokREpGE0hlBEJIUU7iIiKaRwFxFJIYW7iEgKKdxFRFJI4S4ikkIKdxGRFFK4i4ikkMJdRCSFFO4iIimkcBcRSSGFu4hICincRURSSOEuIpJCCncRkRRSuIuIpJDCXUQkhRTuIiIppHAXEUmh2lwgW0SkUeXn57N27Vp27dqVdClNRnZ2Nj179iQrK6tez1e4i0ji1q5dS8eOHenTpw9mlnQ5iXN3Nm3axNq1a+nbt2+9XkPdMiKSuF27dtGlSxcFezEzo0uXLg3aklG4i0iToGAvr6Hfh8JdRCSFFO4iIimkcBcRAW6++WbuuOMOAK6++mpOO+00AF544QXGjx+fZGn1otEyItK0TJ4MixZl9jVzcuC22/a6SF5eHj//+c+58sormT9/Prt37yY/P5/Zs2czYsSIzNazD9TYcjezXmb2opktNbMlZnZVNcuNNLNFxcv8NfOliog0nsGDB7NgwQK2bt3K/vvvz7Bhw5g/fz6zZ88mLy8v6fLqrDYt9wLgGndfaGYdgQVm9py7Ly1ZwMw6A1OAM9x9jZkd1Ej1ikja1dDCbixZWVn07duX++67j5NOOonjjjuOF198kbfeeosBAwYkUlND1Nhyd/d17r6w+PY2YBnQo8JiXwAec/c1xcutz3ShIiKNLS8vj1tuuYURI0aQl5fH1KlTGTRoULMcplmnHapm1gcYBMyrMOsI4EAzm2VmC8zsi5kpT0Rk38nLy2PdunUMGzaM7t27k52d3Sy7ZKAOO1TNrAMwE5js7lureJ3BwCigLTDXzP7m7m9WeI2JwESA3r17N6RuEZGMGzVqFPn5+R/ff/PNN/eydNNWq5a7mWURwf6guz9WxSJrgWfcfbu7bwReAgZWXMjd73b3XHfP7datW0PqFhGRvajNaBkDpgHL3P3WahZ7HDjZzNqYWTtgKNE3LyIiCahNt8xwYALwhpmVDD69HugN4O5T3X2Zmf0ZeB0oAu5198WNUbCIiNSsxnB395eBGncVu/vNwM2ZKEpERBpGpx8QEUkhhbuISAop3EVEgNWrV3PMMcckXUbGKNxFRFJI4S4iUqygoIDx48czYMAAxo4dy44dO5Iuqd50yl8RaVISOuMvACtWrGDatGkMHz6cSy+9lClTpnDttddmtph9RC13EZFivXr1Yvjw4QBcdNFFvPzyywlXVH9quYtIk5LQGX+Byhelbo5ngyyhlruISLE1a9Ywd+5cAB566CFOPvnkhCuqP4W7iEix/v3786tf/YoBAwbw4YcfcvnllyddUr2pW0ZEBOjTpw/Lly9PuoyMUctdRCSFFO4iIimkcBcRSSGFu4hICincRURSSOEuIpJCCncREWDLli1MmTIl6TIyRuEuIkL14V5QUJBANQ2ncBcRAb75zW+ycuVKcnJyOOGEE8jLy+Occ87hqKOOqnQhj1tuuYXvfe97AKxcuZIzzjiDwYMHk5eXV+2BUIWFhfTt2xd3Z8uWLbRu3ZqXXnoJgBEjRvDPf/4zo59HR6iKSNMzcmTlx8aNg0mTYMcOGD268vxLLolp40YYO7b8vFmzanzLm266icWLF7No0SJmzZrFWWedxeLFi+nbty+rV6+u9nkTJ05k6tSp9OvXj3nz5jFp0iReeOGFSsu1bt2a/v37s3TpUlatWsXxxx/P7NmzGTp0KO+88w79+vWrsca6ULiLiFRhyJAh9O3bd6/LfPTRR8yZM4cLLrjg48d2795d7fJ5eXm89NJLrFq1iuuuu4577rmHU045hRNOOCFjdZdQuItI07O3lna7dnuf37VrrVrqNWnfvv3Ht9u0aUNRUdHH93ft2gVAUVERnTt3ZlEtry4yYsQI7rzzTt577z1+8IMfcPPNNzNr1izy8vIaXG9F6nMXEQE6duzItm3bqpzXvXt31q9fz6ZNm9i9ezdPPvkkAJ06daJv37787ne/A8Ddee2116p9jyFDhjBnzhxatWpFdnY2OTk53HXXXYwYMSLjn0fhLiICdOnSheHDh3PMMcfw3//93+XmZWVl8Z3vfIchQ4Zw+umnc+SRR34878EHH2TatGkMHDiQo48+mscff7za99h///3p1asXJ554IhDdNNu2bePYY4/N+Ocxd8/4i9ZGbm6uz58/P5H3FpGmZdmyZQwYMCDpMpqcqr4XM1vg7rk1PVctdxGRFNIOVRGRDPvRj370cT98iQsuuIAbbrhhn9VQY7ibWS/gfqA74MDd7n57NcueAMwFLnT3GZksVESkubjhhhv2aZBXpTYt9wLgGndfaGYdgQVm9py7Ly27kJm1Bn4KPNsIdYqISB3U2Ofu7uvcfWHx7W3AMqBHFYt+HZgJrM9ohSIiUmd12qFqZn2AQcC8Co/3AM4F7qzh+RPNbL6Zzd+wYUPdKhURkVqrdbibWQeiZT7Z3bdWmH0b8A13L6r8zFLufre757p7brdu3eperYiI1EqtRsuYWRYR7A+6+2NVLJILPGxmAF2B0WZW4O5/yFilIiJSazW23C0SexqwzN1vrWoZd+/r7n3cvQ8wA5ikYBeR5mRfXKzjkksuYcaMfTOQsDbdMsOBCcBpZraoeBptZl81s682cn0iIvtE2i7WUWO3jLu/DFhtX9DdL2lIQSIiCZzOvdzFOrKyssjOzubAAw9k+fLlPPvss5x99tksXrwYiIt1fPTRR3zve99j5cqVXHHFFWzYsIF27dpxzz33lDv3TEXPP/88N910E1u3buXWW2/l7LPPrrm4etARqiIiNP7FOkqsXr2aV199lZUrV3Lqqafy1ltvkZ2dnfHPo3AXkSanCZzOvVEu1gEwbtw4WrVqRb9+/TjssMNYvnw5OTk5DS+4AoW7iEgVGuNiHQDFowqrvZ8pOiukiAj75mIdAL/73e8oKipi5cqVvP322/Tv3z+zH6SYWu4iIpS/WEfbtm3p3r37x/PKXqyjR48elS7Wcfnll3PjjTeSn5/PhRdeyMCBA6t9n969ezNkyBC2bt3K1KlTG6W/HXSxDhFpAnSxjqrpYh0iIlKOumVERDKsWVysQ0RkX3D3Rhs5sq9l4mIdDe0yV7eMiCQuOzubTZs2NTjQ0sLd2bRpU4N2tqrlLiKJ69mzJ2vXrkXXeSiVnZ1Nz5496/18hbuIJC4rK6vGo0GlbtQtIyKSQgp3EZEUUriLiKSQwl1EJIUU7iIiKaRwFxFJIYW7iEgKKdxFRFJI4S4ikkIKdxGRFFK4i4ikkMJdRCSFFO4iIimkcBcRSSGFu4hICtUY7mbWy8xeNLOlZrbEzK6qYpnxZva6mb1hZnPMbGDjlCsiIrVRm4t1FADXuPtCM+sILDCz59x9aZllVgGnuPuHZnYmcDcwtBHqFRGRWqgx3N19HbCu+PY2M1sG9ACWlllmTpmn/A2o/7WhRESkwerU525mfYBBwLy9LPafwNP1L0lERBqq1tdQNbMOwExgsrtvrWaZU4lwP7ma+ROBiQC9e/euc7EiIlI7tWq5m1kWEewPuvtj1SxzHHAvMMbdN1W1jLvf7e657p7brVu3+tYsIiI1qM1oGQOmAcvc/dZqlukNPAZMcPc3M1uiiIjUVW26ZYYDE4A3zGxR8WPXA70B3H0q8B2gCzAlfgsocPfczJcrIiK1UZvRMi8DVsMyXwa+nKmiRESkYXSEqohICincRURSSOEuIpJCCncRkRRSuIuIpJDCXUQkhRTuIiIppHAXEUkhhbuISAop3EVEUkjhLiKSQgp3EZEUUriLiKSQwl1EJIUU7iIiKaRwFxFJIYW7iEgK1eYyeyLNwjvvwPPPQ7ducPzxcMghYHu9hphIeincpdlyh9dfh8cfj2nhwvLzu3ePkC87HXqoAl9aBoW7NCv5+TB7dmmg/+tfEdYnngg33QSjR8O//x1BXzI9+ywUFsbzDzywNOgHDYp/+/WDVuqglJRRuEuTt20b/PnPEeZPPQUffgj77w+nnw7f+hacfTYcfHD555x8cuntnTvhjTfKB/7tt8OePTG/QwfIySnfwh8wANror0OaMa2+0iStWwdPPBGB/pe/RBB/4hPw2c/CmDHwmc9EKNdG27YwZEhMJfbsgaVLywf+PffEDwFAdjYcd1z5wD/mmPhREWkOzN0TeePc3FyfP39+Iu8tTY97hG1Jd8urr8bjhx0WYT5mDAwf3rit6cJCePPN8oG/cCFs3Rrz27SJgC8b+McdB+3bN15NIhWZ2QJ3z61xOYW7JKWwEF55pTTQV66Mx084oTTQjz462R2gRUWwalX5sF+wADZtivmtWsGRR5YP/JwcOOCA5GqWpm3Pnphqu+VZkcJdmqTt2+G55yLMn3wSNm6E/faD006LMP/sZ6FHj6Sr3Dt3WLu2cgv/vfdKlzniCJg4MaaOHZOrVZoGd5g/H6ZPh9/+FiZPhm9/u36vVdtwV597M/bmm/DYYzG+u2vX0qlLl/L327VLts716+GPf4xAf+452LUrWrZnnRWBfsYZ0KlTsjXWhRn06hXTmDGlj7//PvzjH6UjdK69Fn70I/j61+HKK+P/RVqW996D3/wmQn3p0thn87nPwSmnNP57q+XejLjDkiUwcybMmAGLF8fjBx4IW7bE/Kq0bVt16Ff3Y9C1a+xQbIgVK0q7W+bOjdp69y7tbhkxArKyGvYeTd28efCTn8R30K5dtOKvuQZ69ky6MmlMO3fCH/4Qgf7cc9G1d9JJcPHFMG4cdO7csNdXt0xKuEdrcObMmFasiJbjySfD+efDeedFC7KwMIYIbtxYedq0qerHt2yp/n3bt6/+x6CqH4QDD4RFi0oDffnyeJ1Bg0oDfeDAlnkA0ZIl8NOfwkMPRR/9hAnwjW9E142kg3vsP5o+HR59NHbC9+4NX/xiTP36Ze69MhbuZtYLuB/oDjhwt7vfXmEZA24HRgM7gEvcfWHF1ypL4V69oqIYLVIS6KtWQevWMHJkBPq551Ye110fBQWweXPtfww2bYoDhPamTZuoc8wYOOecWMElrF4NN98M//d/sHs3jB0L110XP4DSPK1eDfffH9PKldEoGjs2WumnnNI4B8dlMtwPAQ5x94Vm1hFYAHzO3ZeWWWY08HUi3IcCt7v70L29rsK9vMJCmDMnulseeyx22GVlwahRsbKMGRMt5KTt2VP5B6Hkx6Bv3zhCtKGbnWn3wQdw220wZUq08P7jPyLkR4xI55bN1q2xjvTpk47Pt21b/J1Onw5//Wt8plNPjUA/77z6j4KprYztUHX3dcC64tvbzGwZ0ANYWmaxMcD9Hr8UfzOzzmZ2SPFzM27kyMqPjRsHkybBjh0RMBVdcklMGzdGWFZ0+eXw+c/HzskJEyrPv+aaGMmxYgVcdlnl+d/6Fnz609E1MXly5fk//nH0u82ZA9dfH4+5R9fIxo2xk3Hz5gj0Tp1ieF2XLtF/98AD0Q3TtWvsmPz5zyu//gMPRPfMI4/AnXdWnj9jRjz/vvtiquipp6JfeMqU2KysaNas+PeOO2KUS1lt28LTT8ftH/4wDjoqq0uX2AKBCLG5c8vP79kzdjpBfHeLFpWff8QRcPfdcXvixNiRXFZOToQlwEUXxQ9jWcOGRd83xJZPyTDGEqNGlY5cOPPM0gOZSpx9duwchcyue8ceGzvc5s6N1z3++PjxrLjjtTHWvbJuuy2+w+efhxtvrDz/rrugf/+6rXu7d5f+8G/bFluJbdvGDsUOHWIEUceO8Vht171bbklu3SssjC3R+fPjcxUVxXsPGwYPPxxbqBddFFtlZVW37pV8psZUp9EyZtYHGATMqzCrB/BOmftrix8rF+5mNhGYCNC7hW6v5+dHiG/YECtJQUFsup12Gnz5y7HC3Hpr0lXKvtCmTYTC978frduf/ATefTeC7tBD4+yWzaWl6x7dh6tXx3q9fXs83rZt/CAdc0yE//z58RlLOgxat47RUkOGxN/Fzp3xnKZiy5b4QXzggWg0tG4dJ6Q7+OBohOXkNOGuR3ev1QR0ILpkzqti3pPAyWXu/wXI3dvrDR482FuKHTvc//AH9wkT3A84wB3cO3Z0/8IX3GfOdN++PekKpSnYs8f9/vvdjzoq1pHDDnO/8073nTuTrqxqu3a5//nP7pMmuffsGTWbuQ8f7v6zn7kvX17183bvdl+40P2ee9wvu8w9N9d9v/3i+eDeqZP7yJHu11zj/tBD7itWuBcW7rvPtWmT+5Qp7kOHRj2tW7uPHu3+yCNN4/8CmO+1yOxajZYxs6ziAH/G3Su1K83sLmCWu/+2+P4KYKTvpVsm7X3u27fH5ubMmfCnP8FHH8WIkjFjYvPs9NN1nhKpWlFRdIH8+MexY/3gg+Hqq+GrX03+eIDNm6Mb5PHH42Ru27bFlsZnPhPr9llnxRZHXe3ZE6OKFiwonV5/Pbp3ID73oEEweHBMublw+OGZ22GZnw/PPBP96E88EfUce2z0o48fn5kBDJmSyR2qBkwHNrt7FT16YGZnAV+jdIfqHe4+pKplS6Qx3LdujT7BGTNixd+5M1b0c8+NQD/11PSP7ZbMcYcXX4zumuefjx3VV1wBV11VvwCtr7ffLj2J2+zZ0f988MGxH+Ccc2KfRWN0peTnVw78114rDfyOHWM/RUngDx5c99MjrZpiAAAFmElEQVQ3v/ZaBPqDD8bBdt26wRe+EKGek9M0u8UyGe4nA7OBN4Ci4oevB3oDuPvU4h+AXwJnEEMhv+Tue03utIT75s2x4s+YEQcs7NkTVwA6//yY8vKin06kIf7+9zhf/e9/HweYfeUrsaO1Mfp7i4ri/Z54IqaSg+WOPrp0iOsJJyRzDvz8/DjSs2Lg79oV8zt2LN/CHzw4doqWrfWDD+KYg+nT47lZWfFDdfHFsUO9qTfAdBBThuzeHTs/16+vPC1aFC2rgoL4Ixs7NgL9xBN18QdpHMuWwc9+VjrC46KL4oCoI49s2Ovu3AkvvBCt8z/+MU6l0Lp1NE5KzvnzqU81vP7GkJ8f38uCBbHDtmLgd+hQGvhvvRXdSoWFsRP34otjlFxzOjWEwr0aRUXR2q4qrKuaqjtoZ7/9Sk9HO3ZsrDhNcRNO0mnNmhgaeO+9EWLnnhtD/XJr/JMvtWFD7A96/PE4F86OHRGEZ54ZrfPRo+Mc+s1RQUHlFv6iRRHiEybEUaMDBiRdZf20mHB3j52XtQ3rjRtLL7lWllmMAz/ooNpNHTsqzCV5GzbEVaV++ctoiJx+eoT8yJFVr59vvhlh/sQTMe69qCjGep9zTkwjR6Z3R39hYWxRN/e/29SG++zZcZ6OsoFd8aCTEh071j6su3RR37g0X1u3wtSpcYzEBx/A0KER8medFScwKwn0FSti+Zyc0v7zQYOaf+C1JKk95e/u3XFU30EHRT9jdWHdrVvTOhhCpDF16gT/8z9xauFf/zrOYfO5z0UrfPfuOGDq1FPha1/TOX9aimbXcheRmhUUxBGhs2dHqJ9xhq4OlRapbbmLSM3atImDb8aPT7oSSYoG7ImIpJDCXUQkhRTuIiIppHAXEUkhhbuISAop3EVEUkjhLiKSQgp3EZEUSuwIVTPbAPyrnk/vCmzMYDnNnb6P8vR9lNJ3UV4avo9D3b3Gy7UkFu4NYWbza3P4bUuh76M8fR+l9F2U15K+D3XLiIikkMJdRCSFmmu43510AU2Mvo/y9H2U0ndRXov5Pppln7uIiOxdc225i4jIXjS7cDezM8xshZm9ZWbfTLqeJJlZLzN70cyWmtkSM7sq6ZqSZmatzewfZvZk0rUkzcw6m9kMM1tuZsvMbFjSNSXFzK4u/htZbGa/NbPspGtqbM0q3M2sNfAr4EzgKOD/mdlRyVaVqALgGnc/CjgRuKKFfx8AVwHLki6iibgd+LO7HwkMpIV+L2bWA7gSyHX3Y4DWwIXJVtX4mlW4A0OAt9z9bXffAzwMjEm4psS4+zp3X1h8exvxx9sj2aqSY2Y9gbOAe5OuJWlmdgAwApgG4O573H1LslUlqg3Q1szaAO2A9xKup9E1t3DvAbxT5v5aWnCYlWVmfYBBwLxkK0nUbcD/AEVJF9IE9AU2AL8u7qa618zaJ11UEtz9XeAWYA2wDvi3uz+bbFWNr7mFu1TBzDoAM4HJ7r416XqSYGZnA+vdfUHStTQRbYDjgTvdfRCwHWiR+6jM7EBiC78v8EmgvZldlGxVja+5hfu7QK8y93sWP9ZimVkWEewPuvtjSdeToOHAOWa2muiuO83MfpNsSYlaC6x195ItuRlE2LdEnwZWufsGd88HHgNOSrimRtfcwv3vQD8z62tm+xE7RZ5IuKbEmJkRfarL3P3WpOtJkrtf5+493b0PsV684O6pb51Vx93fB94xs/7FD40CliZYUpLWACeaWbviv5lRtICdy22SLqAu3L3AzL4GPEPs8f4/d1+ScFlJGg5MAN4ws0XFj13v7k8lWJM0HV8HHixuCL0NfCnhehLh7vPMbAawkBhh9g9awJGqOkJVRCSFmlu3jIiI1ILCXUQkhRTuIiIppHAXEUkhhbuISAop3EVEUkjhLiKSQgp3EZEU+v/nVwTLCuDfsAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's plot it all\n",
    "plt.plot(range(epochs), w_hist, 'r',\n",
    "         range(epochs), b_hist, 'b')\n",
    "plt.plot([true_w] * len(range(epochs)), 'r--',\n",
    "         [true_b] * len(range(epochs)), 'b--')\n",
    "plt.legend(['w', 'b', 'true_w', 'true_b'])\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}