{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Descending into ML\n",
"\n",
"Author: Gaurav Vaidya "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Source\n",
"This content is based on the [Descending into ML](https://developers.google.com/machine-learning/crash-course/descending-into-ml/) section of Google's *Machine Learning Crash Course*.\n",
"\n",
"# Learning objectives\n",
"\n",
"* A model is a way to predict the label for a given set of features.\n",
"* Loss is a way of measuring how far the predicted label is from the actual label.\n",
"\n",
"# Linear regression for fun and profit\n",
"\n",
"[Linear regression](https://en.wikipedia.org/wiki/Linear_regression) is a method for finding the straight line or hyperplane that best fits a set of points. \n",
"\n",
"> If you remember this from previous mathematical training -- great! If not, just think of it as drawing a *line of best fit* on your data. And if you don't know what that is, don't worry, I'll show you!\n",
"\n",
"# Working with the Iris flower dataset\n",
"\n",
"Let's start by loading the [Iris flower data set](https://en.wikipedia.org/wiki/Iris_flower_data_set) introduced earlier."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"iris1 = iris_dataset.plot(\n",
" \"sepal_length\",\n",
" \"petal_length\",\n",
" kind=\"scatter\",\n",
" title=\"Petal and sepal length in three species of Iris\"\n",
")\n",
"iris1.set_xlabel(\"Sepal length (cm)\")\n",
"iris1.set_ylabel(\"Petal length (cm)\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Looks like the answer is... yes! If we draw a line across the plot, we can *predict* what the petal length might be for a plant given a particular sepal length.\n",
"\n",
"**That's all a model is!** -- something that can extrapolate from known data to predict what the value might be for a given input value."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Linear regression can define that model precisely\n",
"\n",
"Drawing a line by hand is fine, but we would like to determine exactly how the petal length varies as the sepal length varies. Luckily, `matplotlib` can run a linear regression for us easily."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 5.1\n",
"1 4.9\n",
"2 4.7\n",
"3 4.6\n",
"4 5.0\n",
"Name: sepal_length, dtype: float64"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_dataset.sepal_length.head()"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 1.4\n",
"1 1.4\n",
"2 1.3\n",
"3 1.5\n",
"4 1.4\n",
"Name: petal_length, dtype: float64"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_dataset.petal_length.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Scikit-learn's LinearRegression module needs the data as a two-dimensional array:\n",
"* It expects each row to contain multiple features.\n",
"* It expects as many rows as there are data points."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[5.1],\n",
" [4.9],\n",
" [4.7],\n",
" [4.6],\n",
" [5. ],\n",
" [5.4],\n",
" [4.6],\n",
" [5. ],\n",
" [4.4],\n",
" [4.9],\n",
" [5.4],\n",
" [4.8],\n",
" [4.8],\n",
" [4.3],\n",
" [5.8],\n",
" [5.7],\n",
" [5.4],\n",
" [5.1],\n",
" [5.7],\n",
" [5.1],\n",
" [5.4],\n",
" [5.1],\n",
" [4.6],\n",
" [5.1],\n",
" [4.8],\n",
" [5. ],\n",
" [5. ],\n",
" [5.2],\n",
" [5.2],\n",
" [4.7],\n",
" [4.8],\n",
" [5.4],\n",
" [5.2],\n",
" [5.5],\n",
" [4.9],\n",
" [5. ],\n",
" [5.5],\n",
" [4.9],\n",
" [4.4],\n",
" [5.1],\n",
" [5. ],\n",
" [4.5],\n",
" [4.4],\n",
" [5. ],\n",
" [5.1],\n",
" [4.8],\n",
" [5.1],\n",
" [4.6],\n",
" [5.3],\n",
" [5. ],\n",
" [7. ],\n",
" [6.4],\n",
" [6.9],\n",
" [5.5],\n",
" [6.5],\n",
" [5.7],\n",
" [6.3],\n",
" [4.9],\n",
" [6.6],\n",
" [5.2],\n",
" [5. ],\n",
" [5.9],\n",
" [6. ],\n",
" [6.1],\n",
" [5.6],\n",
" [6.7],\n",
" [5.6],\n",
" [5.8],\n",
" [6.2],\n",
" [5.6],\n",
" [5.9],\n",
" [6.1],\n",
" [6.3],\n",
" [6.1],\n",
" [6.4],\n",
" [6.6],\n",
" [6.8],\n",
" [6.7],\n",
" [6. ],\n",
" [5.7],\n",
" [5.5],\n",
" [5.5],\n",
" [5.8],\n",
" [6. ],\n",
" [5.4],\n",
" [6. ],\n",
" [6.7],\n",
" [6.3],\n",
" [5.6],\n",
" [5.5],\n",
" [5.5],\n",
" [6.1],\n",
" [5.8],\n",
" [5. ],\n",
" [5.6],\n",
" [5.7],\n",
" [5.7],\n",
" [6.2],\n",
" [5.1],\n",
" [5.7],\n",
" [6.3],\n",
" [5.8],\n",
" [7.1],\n",
" [6.3],\n",
" [6.5],\n",
" [7.6],\n",
" [4.9],\n",
" [7.3],\n",
" [6.7],\n",
" [7.2],\n",
" [6.5],\n",
" [6.4],\n",
" [6.8],\n",
" [5.7],\n",
" [5.8],\n",
" [6.4],\n",
" [6.5],\n",
" [7.7],\n",
" [7.7],\n",
" [6. ],\n",
" [6.9],\n",
" [5.6],\n",
" [7.7],\n",
" [6.3],\n",
" [6.7],\n",
" [7.2],\n",
" [6.2],\n",
" [6.1],\n",
" [6.4],\n",
" [7.2],\n",
" [7.4],\n",
" [7.9],\n",
" [6.4],\n",
" [6.3],\n",
" [6.1],\n",
" [7.7],\n",
" [6.3],\n",
" [6.4],\n",
" [6. ],\n",
" [6.9],\n",
" [6.7],\n",
" [6.9],\n",
" [5.8],\n",
" [6.8],\n",
" [6.7],\n",
" [6.7],\n",
" [6.3],\n",
" [6.5],\n",
" [6.2],\n",
" [5.9]])"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_dataset.sepal_length.values.reshape(-1, 1)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1.85750967] -7.0953814782793145\n"
]
}
],
"source": [
"from sklearn.linear_model import LinearRegression\n",
"\n",
"X = iris_dataset.sepal_length.values.reshape(-1, 1)\n",
"Y = iris_dataset.petal_length\n",
"\n",
"model = LinearRegression()\n",
"model.fit(X, Y)\n",
"slopes = model.coef_\n",
"intercept = model.intercept_\n",
"\n",
"print(slopes, intercept)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In other words, based on the available data, we can construct a model that predicts a petal length given a particular sepal length.\n",
"\n",
"$$petal\\_length = 1.8575 * sepal\\_length - 7.095$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The Equation of a Line\n",
"\n",
"You may be familiar with this as the [equation of a line](https://en.wikipedia.org/wiki/Linear_equation#One_variable):\n",
"\n",
"$$ y = mx + c $$\n",
"\n",
"See how easy it is to predict a petal value given a sepal value: you just plug it into the equation! For example:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2.1925\n"
]
}
],
"source": [
"# What is the predicted petal length when the sepal length is 5cm?\n",
"sepal_length = 5\n",
"petal_length = 1.8575 * sepal_length - 7.095\n",
"print(petal_length)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2.192166848827913\n"
]
}
],
"source": [
"# We could be more precise by plugging in the slope and intercept values directly.\n",
"petal_length = slopes[0] * sepal_length + intercept\n",
"print(petal_length)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Machine learning? For real?\n",
"\n",
"Yes! By convention, we write this equation like this when thinking about it in machine-learning terms:\n",
"\n",
"$$ y' = b + w_1x_1 $$\n",
"\n",
"Where:\n",
"* $y'$ is the *predicted label* (the desired output), which in our example is the petal length in centimeters.\n",
"* $b$ is the *bias* (the y-intercept, sometimes referred to as $w_0$), which in our example is 7.095 cm.\n",
"* $w_1$ is the *weight* of feature 1, which is the same concept as the \"slope\" in the traditional equation of the line. In our example, this is 1.8575.\n",
"* $x_1$ is a *feature* (a known input), which in our example is the sepal length.\n",
"\n",
"Writing it in this way makes it easy to extend our model when we are considering multiple features, such as sepal length and sepal width and many more. In that case, our equation would look like:\n",
"\n",
"$$ y' = b + w_1x_1 + w_2x_2 + w_3x_3 + \\ldots + w_nx_n $$\n",
" \n",
" \n",
"# What does this model actually look like?\n",
"\n",
"We can draw this model onto our plot from earlier as a *line of best fit*.\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEWCAYAAABliCz2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztvXmYVMX1uP+eGWaYYRlRGYgoOO6CRFFQg/sCmBi3qFFJjEERogY0PzVGJRoVlejXjcUNJeIGcUXcgoAfkEURQRZxABUdEBBnkH0ZGZjz++Pe6eluerk9vU+f93nu032r6ladW919uu6pU6dEVTEMwzAaP3npFsAwDMNIDabwDcMwcgRT+IZhGDmCKXzDMIwcwRS+YRhGjmAK3zAMI0cwhZ8GRKRMRFREmqSgrbtE5KVkt+MVEekjIjPC5KWsX2KRK0z5L0XktAS1fZqIrExEXZmOiHQQkS0ikp/CNg8TkXkisllErm/A9SeLyNJkyJZqTOH7ISIVIrLd/UL+KCLPiUgLD9fFpCyM9JKIPxZVPUJVpzawfRWRgxvadjajqitUtYWq7kphs7cAU1W1paoOC84UkakicnW4i1V1uqoellQJU4Qp/N05V1VbAMcAxwL/TLM8hhFAKkfHjYT9gS8bcmE6njaTiSn8MKjqKuB/QGcAEdlDREaJyA8iskpE7hWRfBHpCDwFdHefDDa45X/rPkZuEpHvReQur22LyK0issx9BC0Xkd/55fURkRki8pCIrBeR70TkN375B4jIR+61k4DWEdppLSLvisgGEVknItNFJM/Naycib4hIldvG9X7X3SUir4vIK247n4vIUV7kj4VwfR5DP0xzZZgsIo/7mbamua8b3M+su991IesLIVuFiPTw649XReQFt70vRaRbmOvq2l7gtn2pX95NIlLp3u+VfumjReRJEXlfRLYCp4tIU1fWFe7T6FMiUux3zTkiMt/9bD8WkSPDyCMi8qjb7kYRWSginf3afUpEJrn39ZGI7O937eFu3joRWSoil/jlFYvIwyKy3K13hpsW8HQV5TM+2G1zo4isFZFXInwe57n9vkGcEXtHN/3/gNOBEW5/HxquDrf8aSKyUkT+ISJrgOckyOTm5q1y+2SpiJwZqc6MQlXtcA+gAujhvm+PMyoY7J6/BTwNNAfaALOBv7h5fYAZQXWdBvwS50/1SOBH4AI3rwxQoEkYOX4PtHOvvRTYCuzj11YN0A/IB64FVgPi5n8CPAI0BU4BNgMvhWlnCM6fVYF7nAyI2+5c4E6gEDgQ+BY4y73uLleGi93rbga+Awo8yj8jjDwB/eKhz6P1w0Ou/CcBm+r6IVT/R6svynflLqAaONu9dggwK8L3TIGDg74rO4F73P48G9gG7OnmjwY2Aie6fVoEPAa8DewFtATeAYa45Y8BKoHjXXn+7MrbNIQsZ7mfdSv3s+/o91mNxvn+nILzfRpa99m5n8n3wJVAE7fNtcARbv7jwFRgX1eGE9w6YvmMxwKD/O75pDD9eSjOd6yn23+3AN8AhW7+VODqCJ+HL9/vs3jAlbfYTVvp5h/m3nc7v+/SQenWXZ51XLoFyKTD/VFsATYAy4En3A+8LfAzUOxXtjcwxX3fhzBKzK/8Y8Cjfl+SsAo/xLXzgfP92vrGL6+ZW9cvgA7ul7W5X/4Ywiv8e4Dx+CkfN/14YEVQ2m3Ac+77u/BTaO4P8gfgZI/yR1X4Hvs8Wj8088t/iegKP2R9Eb4r/gp/sl9eJ2B7hM8zlMLfHiRPJfAr9/1o4AW/PMFRcAf5pXUHvnPfP4k7UPHLXwqcGkKWM4CvgF8BeUF5o4H/+p23AHbhDIYuBaYHlX8a+Jf7fdgOHBXnZ/wCMBLYL8rv4w7g1aDv4yrgNPd8KrEp/B1AUdDnU6fwD3Y/mx64A5xsOhqVfSpBXKCqk/0TROSXOCOHH0SkLjkP558+JCJyPPBvHJNQIc5o4TUvAojIFcCNOD8OcH5o/qaZNXVvVHWbK1NdmfWqutWv7HKcH2go/h+Ospro1jFSVf+NY/NsJ655yiUfmO537rt3Va11H3nbeZTfC/sTvc8j9cM6Vd0WJG+4fohWnxfW+L3fBhSJSBNV3enx+p+Cym4Latv/vktx/pDm+vWN4HxG4PTdn0VkoN81hbifjz+q+n8iMgJnRN5BRMYBN6vqpuB2VXWLiKxz69kfOD7oO9IEeBGn/4uAZVHuOdpnfAswGJgtIuuBh1X1PyHqaYfzPa+Ts1ZEvsd5umgIVapaHSpDVb8Rkb/h/G6OEJEPgBtVdXUD20oppvC98T3OSKR1mB9wqJCjY4ARwG9UtVpEHsOD0nNtpM8AZwKfqOouEZmP84OOxg/AniLS3E/pdwgjH6q6GbgJuElEjgCmiMhnOPf7naoeEqEtn/IUx+6/H7A6Tvn9idbnkfgB2EtEmvkpfX9ln40hYv1lXoszgj5CnbmmYL4H7lPV+zxV7HiuDBORNsCrwN9xRs0Q+Dm3wDEhrXbb+EhVewbX534fqoGDgAURmo74GavqGhwTGyJyEjBZRKap6jdBRVfjmE/r2hdX7lB944WI3w9VHQOMEZESnKeaB4A/NbCtlGKTth5Q1R+AicDDIlIiInkicpCInOoW+RHYT0QK/S5riTPKrBaR44A/eGyuOc4XrgrAnbzr7FHO5cAc4G4RKXR/JOeGK+9O7B3s/kA24Tyu78Kxo25yJ6eKxZmc7iwix/pd3lVELnQn3/6G88OdFY/8QfcSrc+99MNdbj90D+qHKqAWZ24iHfwYT9uqWovzp/qoq6QRkX1F5Cy3yDPANSJyvDg0F8eJoGVwXSJyrFuuAMdMVI3zHajjbBE5yf1uDwY+VdXvgXeBQ0XkTyJS4B7HikhHV77/AI+IM/mfLyLdRaRp0H1E/IxF5Pcisp9bfD3O9yqUO+erwG9F5Ez3Pm7C+T5+HFvPRkccn/4z3HupxvnjTaWLaVyYwvfOFTiPxeU4X77XgX3cvP/DmeBdIyJr3bTrgHtEZDPO5OerXhpR1XLgYZxJxx9xRi4zY5DzDzg2+HU49tQXIpQ9BJiMM2/xCfCEqk5Vx0f6XKALzmTsWuBZYA+/a8fj2HHX44xuLlTVmgTI70+kPo/GH3Hs2j8B9wKv4CgB3FH/fcBM16vjVw2Ur6HcBTzvtn1JtMJh+AfOxOQsEdmE8zkeBqCqc3BGxiNw+u0bnDmKUJTg/EGsxzGL/IQz2V3HGJzv0TqgK06/1j0d9gIuwxlhr6F+ohOcifwvgM/cax8gtL6J9BkfC3wqIltwJqhvUNXvgitQ1aXA5cBwnO/quTju1TvC3HM8NMUx1a7Fuec2wO1JaCcp1Hk0GIZnxHExPVhVL0+3LF4Rx6Vviar+K92yZAsiMhpnstLWojQSbIRvNEpc88JBrpng18D5OC6AhpGz2KSt0Vj5BfAmsDewErhWVeelVyTDSC9m0jEMw8gRzKRjGIaRI2SUSad169ZaVlaWbjEMwzCyhrlz565V1VIvZTNK4ZeVlTFnzpx0i2EYhpE1iMjy6KUczKRjGIaRI5jCNwzDyBFM4RuGYeQIpvANwzByBFP4hmEYOYIpfMMwjBzBFL5hGEaOYArfMAwjTaxcuZLu3buzbt26lLRnCt8wDCMNDBgwgPbt2zNr1iw++OCDlLSZUSttDcMwGjtfffUVhx12mO98+PDh9O7dOyVtm8I3DMNIAarKJZdcwuuvv+5L27x5My1atIhwVWIxk45hGEaS+fzzz8nLy/Mp+5dffhlVTamyBxvhG4ZhJA1V5bTTTmPatGkAtGnThhUrVtC0adMoVyYHG+EbhmEkgY8++oi8vDyfsn/33Xf58ccf06bswUb4hmEYCWXnzp107tyZpUuXAtC5c2fmz59Pfn5+miWzEb5hGEbCePvttykoKPAp+2nTpvHFF19khLIHG+EbhmHETXV1Nfvssw8bNmwA4Mwzz2TSpEmISJolC8RG+IZhGHHwwgsvUFxc7FP28+fPZ/LkyRmn7MFG+IZhGA1i06ZN7LHHHr7z3r17M2bMmDRKFB0b4RuGYcTI0KFDA5T9119/nfHKHmyEbxiG4ZmqqiratGnjO7/++usZOnRoGiWKDRvhG4ZheOCOO+4IUParVq3KKmUPSVT4InKYiMz3OzaJyN+S1Z5hGEYyWLFiBSLCvffeC8DgwYNRVdq1a5dmyWInaSYdVV0KdAEQkXxgFTAuWe0ZhmEkmmuvvZannnrKd/7TTz+x1157pVGi+EiVSedMYJmqLk9Re4ZhGA1myZIliIhP2T/xxBOoalYre0jdpO1lwNhQGSLSH+gP0KFDhxSJYxiGsTuqyoUXXshbb70FQF5eHhs3bgwf1bKqCioqoKwMSktTJmdDSfoIX0QKgfOA10Llq+pIVe2mqt1Ks6DDDMNonMyZM4e8vDyfsh87diy7du0Kr+zHjoX994eePZ3XsSHHtBlFKkb4vwE+V9UfU9CWYRhGTNTW1nLyySfz8ccfA9CuXTu+++47CgsLw19UVQV9+8L27c4BznmPHhk90k+FDb83Ycw5hmEY6WTKlCnk5+f7lP3//vc/Vq1aFVnZg2PGCS5TUOCkZzBJHeGLSDOgJ/CXZLZjGIYRCzU1NXTs2JFly5YB0KVLF+bMmeM9qmVZGezYEVypk57BJHWEr6rbVHVvVd2YzHYMwzC88tZbb1FYWOhT9jNnzmTevHmxhTAuLYVRo6C4GEpKnNdRozLanAMWWsEwjBxh+/bttG3bls2bNwPQq1cvJkyY0PColr17Q4cOMHEi9OoFJ56YQGmTg4VWMAyj0fPcc8/RrFkzn7JfsGABH3zwQXwhjAcOhJNOgnvucV4HDkyQtMnDFL5hGI2WjRs3IiJcddVVAFx++eWoKkceeWR8FS9eDCNGBKaNGOGkZzCm8A3DaJQ88sgjtGrVyne+bNkyXnzxRe8VVFXBZ585r8HMnh36mnDpGYIpfMMwGhWVlZWICDfddBMAN954I6rKgQce6L2SaIuqjjsu9HXh0jMEU/iGYTQabrvtNtq2bes7X716NQ8//HBslfgvqtq40Xnt2zdwpN+xIwwYEHjdgAFOegZjCt8wjKxn+fLliAj//ve/Abj//vtRVfbZZ5/YK/O6qGr4cCgvh9Gjndfhwxsiekoxt0zDMLKafv368eyzz/rO161bx5577tnwCmNZVNWxY8aP6v2xEb5hGFlJeXk5IuJT9k8//TSqGp+yh6xdVOUFG+EbhpFVqCrnn38+77zzDgCFhYWsW7eO5s2bJ66R3r2dQGipCH2cwhDLNsI3DCNrmD17Nnl5eT5l/8orr/Dzzz8nVtnXUVoKxx6bXCWc4hDLNsI3DCPjqa2tpXv37sx2/dz3228/li1bFj2qZSaThhDLNsI3DCOjmTx5Mvn5+T5lP2HCBL7//HMKFywIvSgqW0hDiGVT+IZhZCQ1NTUccMAB9OzZE4CuXbuyc+dOzlq3Lut2mgpJGkIsm8I3DCPjePPNNyksLKTCHe1+8sknTrz6deuiL4rKFtLgDWQ2fMMwMoZt27bRunVrtrs27bPPPpt33323PqplnRmkzuYN9WaQbHSbTKU3EDbCNwwjQxg1ahTNmzf3KfsvvviC9957LzCEcZbuNBWRVHgDuZjCNwwjrWzYsAER4eqrrwagT58+qCqdO3fevXAjXhSVCkzhG4aRNh588MGAlbHffvstzz33XOTQxL17w/LlMHmy89q79+5lIl2fSFLVToIwhW8YRspZs2YNIsI//vEPAP7+97+jqhxwwAHeFiNFMoOkajFTihdNJQJR1XTL4KNbt246Z86cdIthGEYS+cc//sGDDz7oO//hhx/4xS9+4ZxUVTnK039StrjYGcl7MdvEe71XUtWOB0Rkrqp281I2qSN8EWklIq+LyBIRWSwi3ZPZnmFkLFn26J8MKioqEBGfsn/ggQdQ1Xpl7xSKbzFSqhYzpWHRVCJItklnKDBBVQ8HjgIye8NHw0gGWfjon2iuvPJKx1zjsn79em655ZbdC5aVwbZtgWnbt3v3wkmVF0+WegslTeGLSAlwCjAKQFV3qOqGZLVnGBmJl92TGjGLFi1CRBg9ejQAzz77LKoasNfsbvi7YYY6j0SqvHiy1FsomQuvDgSqgOdE5ChgLnCDqm71LyQi/YH+AB06dEiiOIaRBrJtoVCCQvWqKueccw7vv/8+AMXFxaxdu5ZmzZpFbqOiwlGe/qPnoqLY+itVi5lSvGgqESTTpNMEOAZ4UlWPBrYCtwYXUtWRqtpNVbuVZkGHGUZMZNOjf4JMT7NmzSIvL8+n7F9//XW2bdvmKPtobSSqv1K1mCmFi6YSQTIV/kpgpap+6p6/jvMHYBi5Q7Y8+ifA9LRr1y66detG9+6Ob0ZZWRk7duzgoosu8t5GtvRXlpI0ha+qa4DvReQwN+lMoDxZ7RlGxuJloZAXEuHpE66OOL1OJk6cSJMmTZg7dy4AkyZN4rvvvqOgoCD2Nnr3hrlzYdgw57Wh/WXsRrKDpw0EXhaRQuBb4Mokt2cYmUlpaXyj1LFjndFwYaFj8hg1KnZFGKmOBppSduzYwUEHHcTKlSsBOP744/n444/JywsxlvTaRiLu1QiNqmbM0bVrVzUMI4jKStXiYlWoP4qLnfRE1jFmjJNWUuK8jhkTscpXX31VAd/x6aefRpcjWhuJuNccA5ijHnWshUc2jEwnEZ4+Xurw4nVSVcXWxYvZs0cPampqADj33HMZP358YFTLcERrI9u8mrIMU/iGkekkwnOlrAw2bw5M27Jl9zoimZ7GjmXkn//MX1xFD/Dll1/SqVMn73JEayObvJqyEAueZhiZTiI8V9auhdrawLTaWifdA+u//hr5wx98yr4voMXFdLIFTVmFKXzDiEYivGMWL4bnn3deG5Ifr6ePuwG453Q/hgwZwl6HHuo7rwCeheTFjkmUV5OxO16N/ak4bNLWyDjqJhn32MPTRGZIBgwInIQcMCC2/ERQXh7YRt1RXh72ktWrVwdMyt7apIlNpmYgxDBpG3WELyJ5InK0iPxWRM4QkbbJ/hMyjIwgEXFwFi+GESMC00aMqB/JR8tPFK1bQ35+YFp+vpMegptvvpl27dr5ztesWcOQF14wU0uWE1bhi8hBIjIS+Ab4N9AbuA6YJCKzRORKETGTkNF4SUQI3GimlFhMLfGYlioqoEWLwLTmzXe7l28/+wwR4eGHHwbgoYceQlVp27atmVoaAZG8dO4FngT+4j42+BCRNsAfgD8BzydPPMNII4nwGDnuuMjp0fLriHcxkod7+dPJJ/PSjBm+8w3PPsseffsGXhPvAjIjvXi1/aTiMBu+kXHEuBgpJJFs9JWVqsG28SZNAm3jiVqMFOZeFi5cGGCr/4/Z6LMKErnwSkTygd8CZfg9EajqI8n5CzKMDMLjYqSI+cOHw3XXOWaa446Djh3r8yoqHNPKxo31ac2aBS40StRipKB70dat+fVZZzFx4kQAWgI/AsWR2khQ+GQjPXixwb8D9AH2xvlO1B2GkRskYsPsjh3hz38OVPbgzWyUyMVI7r18/PXX5OXl+ZT9uNGj2VRcXK/sQ7VhO3dlP9EeAYCFXh8X4j3MpGNkFUk2tcRcxgM7d+7Uo446yme+Ofjgg3XHjh3R27AYNxkLCY6l8z8R6aWqE5P5x2MYWUciTS0dOsDEidCrF5x4Yugyce6uNGHCBH7zm9/4zj/88EPOOOMMb21YjJtGgReFPwsY57pg1gACqKqWJFUyw8h0EmVqGTiw3hf/nntgwADH7h9MAz1kduzYwf7778+aNWsAOOGEE5g+fXroEMbh2rAYN40CLzb8h4HuQDNVLVHVlqbsDYPExH1J8sKr//73vzRt2tSn7D/77DNmzpwZWtlHwmLcNAq8jPC/Bha5tiLDMPyJ19QSaeFV8ARvDB4yW7dupaSkhFo3YNrvfvc73njjDWTtWmfxVkNkzcJNu41AvCj8H4CpIvI/4Oe6RDW3TMNwiGcxUhIWXj355JNcd911vvPFixdz+OGHJ2YnKVt4ldV4ea77DvgQKMTcMg0jsXTs6Njs/RkwIHB07zGmz7p16xARn7Lv378/quoo+0TEBTKynqgjfFW9OxWCGEaDmDkzsneLFzNIKhYTRWoj0sIs8OQhc++993LHHXf4spcvX06HDh1iqiPu+zAyn2h+m8AkoJXf+Z7AB179PmM5zA/fiImePQP9wnv1Csz3Eto4EeGPoxFvG5WVqoWFgfdaWKhaWamrVq0KCIswaNCg8HXE60efir4yYoYY/PC9KPz5IdLmearc2SvhC2C+F6FM4RsBVFaqzp4dWinNmBGovOqOGTPqr42m4FKxmChRG5AXFATWUVCgN/TvH6DsK6PVWaewmzePXWHbwquMJRaF78WGv0tEfM+GIrK/+wXzyumq2kVVu8VwjZHrRFvGPzHMOsC6dC+hjUOFOVZN7C5OiWijosKJr+PyDSA1NQwdORKARx99FFWl1IuJpc7ZLlanu0SEijbSjheFPwiYISIvisiLwDTgtuSKZeQ0XiYYe/UKfW1dupeFQi1aBNq0Aaqrd48bHw+JaMPvXv4IHOKXtXHjRv72t79Fr6OuT6urYetW5zWWSVtbeNUoiKrwVXUCcAzwCvAq0FVVP/BYvwITRWSuiPQPVUBE+ovIHBGZU2UeA7lFuA09vIwmTzxxd6XvP3HrZaHQli1Ouj/FxU66Fzm94LWNSJSWsuDOOxFgjJv0/DXXoKqUlHhcAxnvCN2/P5s3t4VX2Uo4Ww9QFskWhBNiYb8oZdq5r22ABcApkcqbDT+HiDQBGIu9eMYM1TvvrLfdBxNpHsBLO4mYcI0W7z4CtbW1esYZZ/js9K1attTtK1bEJkOdHImYtC0qcuYAiops0jZDIBGTtsBrwBvAFcARrtLuAJwBDAY+Bnp6bgjuAm6OVMYUfo4Qi6KNMzpkVJIdIbIBm4fXMX369IBJ2fHjxzfgBv2Ip09t0jZjiUXhh/XDV9Xfi0gnHLPhVcA+wDZgMfA+cJ+qVoe7XkSaA3mqutl93wu4J4aHD6Ox4sUnPFHL+KP5jffuDV26hN+cxIvveqQ2Ygmd4LJr1y66dOnCokWLADjssMNYtGgRTZp4WRgfgXg2c7FomY0Dr/8MsR7AgThmnAXAl8CgaNfYCD9HSNVoMV4//ESYfGIc4b/33nsBo/opU6bE3w9eSZSZzUgpJNIPP5WHKfwcItkmm0T54SfC5BNpT1uX6upqLS0t9Sn6k08+WXft2pXYPolEJpnZjJiIReHH+YxoGA0k2ZEXvZggvJqWwm1O4tXMMXw4XHZZ2BAQY8aM4Y9//KPvfO7cuRxzzDENv/dIxGOy8fqZWfiFzMXrP0MqDhvhGwkjUSP8SKNzryP8MKaSzZs3B5hvfv/732ttbW3y+iQVJhsLv5BySLRJB9gXOAE4pe7w2kAshyl8I6HEu1esF/t7tDbCKNLhQ4YEKPslS5Ykpw+iyJFQk43Z+dNCLAo/qklHRB4ALgXKgV11DwY4K24NI3PxYoKIVMaLh020NoJMJT8Brbdvh9ucxerXXnstTzzxhLf7iWYqiZSfCs8o8+TJeLzY8C8ADlPVn6OWNIxMw8uGHeHKeN2cJFIbfiEJ7gH+5Ze1YsUK2rdvH1m2OqJtXhIt32tohHg2OLHwCxmPl1g63wIFyRbEMDKO1q0h2Pe9SRMn3Sulpax88EGEemV/54UXoqrelX202EJeYg+lYk9a2/c24wk7wheR4Timm23AfBH5kMAtDq9Pvng5SLZ4OCRCznhMFIlk8eLwC6+aN3eUaB3NmsW08GrAgAE8/vjj9UUXL6b14YfvLkM85hivppRU7Elr+95mNuGM+8CfIxxXeJ0kiOXI+UnbbPFwSISc0epIVV/E64UTRs6lS5cGTMoOHTo0vAzR7jWaHDZZmtOQ4A1QbvCSlogjpxV+tvxoE7WhRyYosHi9cELIWVtUpJecf36Ast+0aVPD+8KLHF7yjUZLLArfiw3/zyHS+jT8mcIISbZsMOFVzkghhaPVkaq+iOSFU0fv3rB8OUye7Lz6T4QGyTkPyKuu5tXx4wF48cUXUVVatmwZXyjoaHLU5c+dC8OGOa/B+XXEE+rZyH7C/RMAvYF3gPXA237HFGCy13+UWA4b4TeSEX62mCjiiGTpL2ct6Kl+I/q999pLt2/fXl8uUxY8ZYvJ0IgJEhQeeX/gNOAT4FS/4xigidcGYjlyWuGrZs9jeSLiy2SKicJDnJtIfHTHHQHmm3duvjmwQKYseMqWAYURMwlR+Ok4cl7hq0besCOTCCfn7NnOCNJfsZSUOOle6/CanyjKy1VHj/Y+slfVmpoaPfzww32KvlO7dlqzcOHuBb32RwNkiKmNWD4XI6uIReF7WWm72f1S+7MRmAPcpKrfNtCaZIQinoUvqSScnLEsvol2r6nqi44dw8amD8U777zDeeed5zv/qLCQU7ZuheOPb9iCp2iLpqLhpQ1bFGXgbeHVI8DfceLp7AfcDDwD/Bf4T/JEM7KSRrz4prq6mr322sun7E8/6SRqi4o4ZceOhi948rJoKhpe+rwRfy6Gd8R5IohQQORTVT0+KG2Wqv5KRBao6lGJEqZbt246Z86cRFVnpBMvi6ailQm3ICrRcnjgxRdf5IorrvCdf/755xy9cyf07Bm4MKukxPGkOfZYb3J89pn3OqKRiD43sg4Rmauq3TwVjmbzwZm0vQTnaSDPfT/LzZvv1Xbk5TAbfg4RzWMkzslUT214YOPGjQGTspdeeml9CONUrEkwjCiQ4IVXB+K4Z64Fqtz3BwPFwEleG/JymMLPEaIpuXjdJb204YHHHnssQNl/9dVXuxdKhDdRtnhnGRlJLAo/6qStOpOy54bJnuHpMcIw/IkW+yWWjb+TsOn22rVrKfUrM3DgQIYNGxa6cCJix1j8GSNFePHSKQX6AWX+5VX1quSJZTRqysoCFTFAdXW9x4jXsMSRvFsa6JVy5513MnjwYN/5ypUr2XfffSNekxBvomzxzjKyGi9eOuOBPYDJwHt+h2E0nGBnAf9zL2GJo3m3xOiV8v333yMiPmV/9913o6rRlb1GAAyvAAAdTElEQVRhZBFeNkBppqr/aGgDIpKP47O/SlXPaWg9RpYRycOmosIJM+zvmVJcHBjuN1pY4gTu4HRtnz489fzzvvO1a9ey99571xcwzxajkeBlhP+uiJwdRxs3AIvjuN7INgYOhE6doE8f53XgwMD8aOYWrwuJtm0LLLN9e+gdnI49NqSiXrp0KSLiU/aPFxSgY8YEKvuxY2H//R3Xyf33d84NI0vxovBvwFH61SKySUQ2i8gmL5WLyH7Ab4Fn4xHSyCIWL4YRIwLTRoxw0uuIZm7xao4RiXweBlXloosu4nB3IxIBNgPX1dTEvpOUYWQRXrx0WsZR/2PALUDYOkSkP9AfoEOHDnE0ZXgmmSYKrx42vXtDly7hzT5eNgcvLg58EigqiuqFM3fuXLp1q1+jMqa4mN7x7iRlGFlC1BG+OFwuIne45+1FJIwbRcB15wCVqjo3UjlVHamq3VS1W6n9iJJPsk0UsXjYdO0KN9zgvIaSI4I5JqqnTxC1tbWcfPLJPmXftm1bqr//nt0i1sRqWjKMbCKaoz7wJPA4sNg93xP4zMN1Q4CVQAWwBmdv3JciXWMLr5JMqlZ1Rlslm6gVqgUFgXUUFISsY8qUKQELqN577736zEwJ02wYDYRELrwCjlfVY0RknvsHsV5ECqNdpKq3AbcBiMhpwM2qernnfyIj8aTKRDF8OFx2GUycCL16wYknJl6OaJ4+OKP6zp07s9idPzjyyCP5/PPPyc/Pr78mmunIFkUZjQgvk7Y1rmulgm8hVm1SpTKSQ6pMFGPHOiajoUOd12BzTSLkiFLHkiVLOPnkk33KfsaMGSxYsCBQ2dcRyXTkJd8wsgQvCn8YMA5oIyL34YRTuD+WRlR1qpoPfvpJRYhcL54tiZAjTB01rVoxZMgQunTpwpIlSxg9ejS1tbWcGPyUYRg5SNTwyAAicjhwJo4H24eqmhS/eguPnCKS6aUTS7jfRMjhV8f8Vau46qqrmDdvHhdffDEjRoygbdu2DbwRw8gOYgmPHNaGLyJ7+Z1WAmP981R1XcNFNNJKMuO2eF0QlSg5SkupbtmSe++9lwceeIC9996bN954gwsvvDC+eg2jERJp0nYujt2+bjVL3aOAuO8PTKJcRjbTwAVRDeGTTz7hqquuYsmSJfTp04eHH36YvfbaK/qFhpGDhFX4qnpAKgUxGgmxLIiKw6SzdetWBg0axLBhw2jfvj0TJkzgrLPOilN4w2jceJm0NQzvePXAiWMB2Icffsgvf/lLhg4dynXXXceiRYtM2RuGB0zhG4nFiwdOA2PUbNy4kX79+tGjRw+aNGnCtGnTGDFiBC1bxhP9wzByBy8Lr4xMIhtC9XqJg+Nl4ZXfvb4zaxbXXHMNa9as4ZZbbuGuu+6iuLg46bdiGI0Jr146u2FeOmkg0g5PmUYkDxwvZh/3XquaNOGGbdsYu2sXv/zlLxk/fnxA8DPDMLwT1g9fRL4j0EvHH1XVhHvpmB9+BKqqHFu3/6i4uBiWL8/ckX4k6v68CgocZe//51VVhXbowH+rq7ke2Aj8s0kTbq2ooNB2oDKMABLih29eOhlGYwvVG8Hss2r2bK7duZN3gOOA/wBHNGsGq1eDKXzDaDCebPgisidwCFBUl6aq05IllBGCxhiqN8jso6qMGjWKm2+6iR07d/Iwzu47+ZD992oYGYCXePhXA9OAD4C73de7kiuWsRupiIOTRr799lt69uxJv379OPqYY1j4yCPcWFxMfiO8V8NIF15G+DcAxwKzVPV0N67O3ckVK4eJ5IUTbZeoLGTXrl2MGDGC22+/nfz8fJ566in69etHXl4eXH555nskGUYW4UXhV6tqtYggIk1VdYmIHJZ0yXKRaF442eSl44HFixfTt29fPvnkE84++2yeeuop2rdvX18gmTF/DCMH8bLwaqWItALeAiaJyHhgdXLFykGiLUZqRBtq19TUcN9999GlSxeWLl3KSy+9xLvvvhuo7A3DSDheNjH/nfv2LhGZAuwB/C+pUuUi0bxwGomXzrx587jqqquYP38+l1xyCcOHD6dNmzbpFsswcgIvk7Yv1r1X1Y9U9W0cTzkjkUTzwslyL53q6mpuv/12jj32WNasWcO4ceN45ZVXTNkbRgrxYtI5wv/E3e6wa3LEyWGieeFksZfOxx9/zNFHH82QIUO44oorKC8v54ILLki3WIaRc0QKrXAbcDtQLCKbqF9xuwMYmQLZco9GtqH2li1bGDRoEMOHD6dDhw588MEH9OrVK91iGUbOEmml7RBgiIgMUdXbUihTbhPNMyVLPFcmT55Mv379WL58OQMGDOD++++nRYsW6RbLMHIaLyadQSJyuYjcASAi7UXkuGgXiUiRiMwWkQUi8qWImO9+DrBhwwb69u1Lz549adq0KdOmTWPYsGGm7A0jA/Ci8B8HugN/cM+3uGnR+Bk4Q1WPAroAvxaRXzVISiM2qqqczcRT7LI5fvx4OnXqxPPPP8+tt97K/PnzOemkk1Iqg2EY4fGi8I9X1b8C1QCquh4ojHaROmxxTwvcI3RoTiNxxLGTVEOprKzksssu44ILLqBNmzZ8+umnDBkyhKKiougXG4aRMrwo/BrXM0cBRKQUqPVSuYjki8h8oBKYpKqfNlhSIzopXpylqowZM4ZOnToxbtw4Bg8ezGeffUbXrubEZRiZiBeFPwwYB7QRkfuAGcD9XipX1V2q2gXYDzhORDoHlxGR/iIyR0TmVGXhqtGMom5xlj91i7MSzMqVKznvvPP44x//yCGHHMK8efP45z//SUFBQcLbMgwjMURV+Kr6MnALMAT4AbhAVV+LpRFV3QBMBX4dIm+kqnZT1W6lWeB9ktGkYHGWqjJy5EiOOOIIPvzwQx599FFmzJhBp06dEtaGYRjJIZIffhFwDXAw8AXwtKru9Fqxa/qpUdUNIlIM9AAeiFNeIxJ1i7OCd5JK0B/psmXL6NevH1OmTOGMM87gmWee4cADE77xmWEYSSJSLJ3ngRpgOvAboCPwtxjq3gd43rX/5wGvquq7DRXU8EgSFmft2rWLYcOGMWjQIAoKChg5ciRXX301IqF2vzQMI1OJpPA7qeovAURkFDA7lopVdSFwdByyGQ0lgYuzysvL6du3L7NmzeKcc87hySefZL/99ktI3YZhpJZINvyaujexmHKMxkFNTQ2DBw/m6KOP5uuvv+bll1/m7bffNmVvGFlMpBH+UW4MHXDi6PjH1FFVLUm6dEZamDt3LldddRULFy7ksssuY+jQoRbV0jAaAWFH+Kqar6ol7tFSVZv4vTdl3wjZvn07t956K8cffzxVVVWMHz+esWPHmrI3jEaCly0OjRxgxowZ9O3bl6+++oq+ffvy0EMP0apVq3SLZRhGAvGy8MpoxGzZsoWBAwdyyimnsGPHDiZNmsSzzz5ryt4wGiGm8HOYiRMn0rlzZx5//HGuv/56vvjiC3r06JFusQzDSBKm8HOQ9evXc+WVV3LWWWdRVFTEjBkzeOyxxyyEsWE0ckzh5xjjxo2jU6dOvPjii9x+++3Mnz+fE044Id1iGYaRAmzSNkf48ccfGThwIK+99hpdunTh/fff5+ijbV2cYeQSNsJv5KgqL730Ep06dWL8+PHcd999zJ4925S9YeQgNsJvxHz//fdcc801vP/++3Tv3p1Ro0bRsWPHdItlGEaasBF+I6S2tpann36aI444gqlTp/LYY48xffp0U/aGkePYCL+R8cEHH/DrXzvbDpx55pk888wzHHDAAWmWyjCMTMAUfiNhx44dHHDAAaxevRpwlP2kSZMshLFhGD7MpNMIeOWVV2jatKlP2c+ePZvJkyebsjcMIwAb4WcxW7dupVWrVuzc6USvvuCCC3jzzTdN0RuGERIb4WcpTz/9NC1atPAp+/LycsaNG2fK3jCMsNgIP8tYt24de++9t++8X79+jBw5Mo0SGYaRLdgIP4u4//77A5R9RUWFKXvDMDxjI/wsYPXq1ey7776+80GDBnHvvfemUSLDMLIRU/gZzo033sijjz7qO//xxx9tByrDMBpE0kw6ItJeRKaIyGIR+VJEbkhWW42RZcuWISI+Zf/II4+gqqbsDcNoMMkc4e8EblLVz0WkJTBXRCapankS22wUXH755bz88su+840bN1JSYtsIG4YRH0kb4avqD6r6uft+M7AY2DfyVbnNggULEBGfsh89ejSqasreMIyEkBIbvoiUAUcDn4bI6w/0B+jQoUMqxEkfVVVQUQFlZVBa6ktWVXr16sXkyZMB2GOPPfjhhx8oLi5OaDuGYeQ2SXfLFJEWwBvA31R1U3C+qo5U1W6q2q20MSunsWNh//2hZ0/ndexYAGbOnEleXp5P2b/11lts2LCh4co+TDuGYRiiqsmrXKQAeBf4QFUfiVa+W7duOmfOnKTJkzaqqhzlu327L2lXURHHHHggC8udKY1DDz2UL7/8kiZN4njoCtEOxcWwfLmN9A2jkSIic1W1m5eyyfTSEWAUsNiLsm/UVFRAYaHv9H2gSXW1T9lPmTKFpUuXxqfsQ7QDQEGBk24YRs6TTBv+icCfgC9EZL6bdruqvp/ENjOTsjLYsYOfgQ5ApZt80vHH89HHH5OXl6D/XbedAGpqnHTDMHKeZHrpzFBVUdUjVbWLe+SesgcoLWVs374UUa/s59x7L9NnzUqcsnfbYdQox4xTUuK8jhpl5hzDMABbaZt0tmzZQsuWLX3nF59xBq+OHYskawFV797QpQvMng3HHQfJ2tZw8eLkt2HeRoaRUCx4WhJ54oknApT9kiVLeO3DD5On7MHxyunaFW64wXlNhpfOwIHQqRP06eO8DhyY+DbM28gwEk5SvXRipbF46fz000+0bt3ad37ttdfyxBNPJL/hVHjpLF7sKPlgyssTN9I3byPD8ExGeOnkJFVVDL7mmgBlv2LFinplv3gxPP+889pQZs6Ef/3LeQ0mFV46s2fHlt4QzNvIMJKCKfwEserxx5E2bbjz6acBuON3v0NVad++vVMgEWaQXr3gpJPgnnuc17POCswvK4Nt2wLTtm9PrJfOccfFlt4QzNvIMJKCKfwEcH2/fuw3YIDvvAq4Z8IExzQBzoh+xIjAi0aMiG2kP3MmTJoUmDZx4u4j/eAtDhO95WHHjuB3r4BznsiJW/M2MoykYF46sRDkNfL1119z6KGH+rKHAtfXndSZIEpLI5tBghXlzJmOIu/VC048sT594sTQdUycWF+uosJRjv6j46Kiejnq8OJhE8lDZvhw5+nirbfgggvgnHNC1xEPqfI2MoxcQlUz5ujatatmLGPGqBYXq+6xh2pxsV7WvbsCvmNTUZEq1B/FxaqVlc615eWBeXVHeXlgGz17Bub36lWfN2NG6DpmzKgvU1mpmpcXmJ+XVy+HquqAAYH5AwZEvVcdMyYw30sd8RJNBsMwVFUVmKMedWzalbz/kbEKv7LSUTqgn/speUBfeOEFp0ydgiopaZiSjKbQKytD5/sr82h1ePnj8bvXuP68EtTfIWUwDMNHLAo/d2z4VVXw2Wf1dvVYqKhACwo4HTjGTdpbhO3Tp/OnP/3JSejd23EbnDzZee3dO7CO4cPhnXegb1/ndfjwwPxIJhtXBvbYIzCvpCTQcyVaHV48bKJ5yHj10omzv81LxzAST24o/DgX8Uxfs4a8TZuY6p6/A6wVoeiwwwILlpbCsceGnlwcOBDOPdeZfDz33N29dHr1Ct14XXpZGWzZEpi3dWug50q0Orx42ETzkPFSR7yLpsxLxzCSg9dHgVQcSTHpxGEeqKmp0U6dOvnMN51AaxpixvBqSolksvFSR7QylZWq+fmBefn5u/dFPOapRJljoslgGIaqmkknEK/mgSATxLvvvktBQQHlbgjjj4AvCXJr8mrG8GpKiWSy8VJHtDIVFdCiRWBe8+a794UX89SMGXDnnc6rv3kqUeaYaDIYhhE7Xv8ZUnEkbYQfzXPFzyOkuqhI927RwjeqP+2007T2yy+jj64jeZWkarLUywg/kaPvUPdaWalaUBDYRkGBTbgaRpLAvHT8iEEJvhjkgTNv3rz6euI1Y8TiDhnKjOHVHBOtnXhNJdHutbJStbAwML+w0BS+YSQJU/j+jB4dWuGPHu3kz56tm0pKAhT9ZU2aqM6evXtd5eXOdcG2+9mzndGuf/0lJbvXEe56fyorneuCFaTXNry0E64NL0STIxY5E0E892IYjYBYFH7jX2kbxatk+OTJXL+pfm/1r4BDCgpCe4R07Bh6xadXr5Jw1/tTWhray6esLDB6JEB1dWxyRmvDC9HuNZUeNmPHOm6uhYVOm6NGma3fMCLQ+Cdtw8R+WVtaiohw/e23O0n5+WhJCYc0JG5LqmK/qEY+TwXR7jVVfVFV5Sj77dth40bntW/fhvn9G0aO0DhG+NF2Rho+HK67zheX5a5XXuFuv3IrV65k38LC+HZX6t0bevRI3g5NFRXQrJmj3OooLt49Tk4qiHavye4LqPcG8n/q8Y9fZBjGbmS/wvf6WN+xI9+3aEGHDh18SXfffTd33nlnfZl4FUU8ppJoZNpipGj3msy+gMzrD8PIArLbpBPDY/1f//rXAGW/du3aQGWf6VjI4ECsPwwjZpI2wheR/wDnAJWq2jkpjXh4rF+6dCmHH364L3vEiBH89a9/bVh76d5UOxWmkmzC+sMwYiKZJp3RwAjghaS1EOGxXlW55JJLeP31131ZmzdvpkXwSlOvZIpHSLJNJdmG9YdheCZpJh1VnQasS1b9QNjH+rkrVpCXl+dT9i+//DKq2nBlbx4hhmE0AtI+aSsi/YH+QICN3TN+j/W1HTpw6sUXM2PGDADatm3L8uXLadq0aXxCmkeIYRiNgLRP2qrqSFXtpqrdShuqPEtLmbp1K/m/+IVP2b/33nusWbMmfmUP5hFiGEajIO0KPxEsWrSI008/HYAjjzySnTt3cvbZZyeuAfMIMQyjEdAoFP6ee+7JqaeeyvTp01mwYAH5+fmJb8TC9RqGkeUk0y1zLHAa0FpEVgL/UtVRyWhr3333ZerUqcmoOhDzCDEMI4tJmsJXVRsCG4ZhZBCNwqRjGIZhRMcUvmEYRo5gCt8wDCNHMIVvGIaRI5jCNwzDyBFM4RuGYeQIpvANwzByBNF07IsaBhGpApanUYTWwNo0tu8VkzPxZIusJmdiaQxy7q+qnlaEZpTCTzciMkdVu6VbjmiYnIknW2Q1ORNLrslpJh3DMIwcwRS+YRhGjmAKP5CR6RbAIyZn4skWWU3OxJJTcpoN3zAMI0ewEb5hGEaOYArfMAwjR8hZhS8i+SIyT0TeDZHXR0SqRGS+e1ydJhkrROQLV4Y5IfJFRIaJyDcislBEjslQOU8TkY1+/XlnmuRsJSKvi8gSEVksIt2D8jOlP6PJmSn9eZifDPNFZJOI/C2oTNr71KOcmdKn/5+IfCkii0RkrIgUBeU3FZFX3P78VETKYqk/aRugZAE3AIuBkjD5r6jqgBTKE47TVTXcgovfAIe4x/HAk+5rOogkJ8B0VT0nZdKEZigwQVUvFpFCoFlQfqb0ZzQ5IQP6U1WXAl3AGUABq4BxQcXS3qce5YQ096mI7AtcD3RS1e0i8ipwGTDar1hfYL2qHiwilwEPAJd6bSMnR/gish/wW+DZdMsSJ+cDL6jDLKCViOyTbqEyEREpAU4BRgGo6g5V3RBULO396VHOTORMYJmqBq+UT3ufBhFOzkyhCVAsIk1w/uhXB+WfDzzvvn8dOFNExGvlOanwgceAW4DaCGUuch9BXxeR9imSKxgFJorIXBHpHyJ/X+B7v/OVblqqiSYnQHcRWSAi/xORI1IpnMuBQBXwnGvKe1ZEmgeVyYT+9CInpL8/g7kMGBsiPRP61J9wckKa+1RVVwEPASuAH4CNqjoxqJivP1V1J7AR2NtrGzmn8EXkHKBSVedGKPYOUKaqRwKTqf9HTTUnquoxOI/FfxWRU4LyQ/2zp8PPNpqcn+PE+zgKGA68lWoBcUZOxwBPqurRwFbg1qAymdCfXuTMhP704ZqdzgNeC5UdIi0tvuBR5Ex7n4rInjgj+AOAdkBzEbk8uFiISz33Z84pfOBE4DwRqQD+C5whIi/5F1DVn1T1Z/f0GaBrakX0ybHafa3EsTkeF1RkJeD/9LEfuz8CJp1ocqrqJlXd4r5/HygQkdYpFnMlsFJVP3XPX8dRrMFl0t2fUeXMkP705zfA56r6Y4i8TOjTOsLKmSF92gP4TlWrVLUGeBM4IaiMrz9ds88ewDqvDeScwlfV21R1P1Utw3m8+z9VDfgXDbIxnoczuZtSRKS5iLSsew/0AhYFFXsbuML1hPgVziPgD5kmp4j8os7OKCLH4XzvfkqlnKq6BvheRA5zk84EyoOKpb0/vciZCf0ZRG/Cm0nS3qd+hJUzQ/p0BfArEWnmynImu+uet4E/u+8vxtFfnkf4ueylE4CI3APMUdW3getF5DxgJ86/Z580iNQWGOd+B5sAY1R1gohcA6CqTwHvA2cD3wDbgCszVM6LgWtFZCewHbgsli9pAhkIvOw+2n8LXJmB/elFzkzpT0SkGdAT+ItfWsb1qQc5096nqvqpiLyOY17aCcwDRgbpplHAiyLyDY5uuiyWNiy0gmEYRo6QcyYdwzCMXMUUvmEYRo5gCt8wDCNHMIVvGIaRI5jCNwzDyBFM4RtpQ0QGuZEBF7oRChMaVEucCIihoqGGTE9AexeISCe/86kiEnXjaRHZJxHyiEipiEyItx6j8WIK30gL4oT8PQc4xg1h0YPAmCvZyAVAp6ildudGnBXdcaGqVcAPInJivHUZjRNT+Ea62AdYWxfCQlXX1oVoEJGuIvKRG4ztg7qVz+6I+TER+ViceOHHuenHuWnz3NfDwrYahLtS+D8i8pl7/flueh8ReVNEJojI1yLyoN81fUXkK1eeZ0RkhIicgLMq+/+5TysHucV/LyKz3fInhxHjImCCW3e+iDwkzv4CC0VkoJteISL3i8gnIjJHRI5x+2ZZ3QIil7eAP3q9fyPHUFU77Ej5AbQA5gNfAU8Ap7rpBcDHQKl7finwH/f9VOAZ9/0pwCL3fQnQxH3fA3jDfX8a8G6Itn3pwP3A5e77Vq48zXFWV3+LE6ukCFiOE8OkHVAB7OXKOh0Y4V4/GrjYr52pwMPu+7OBySFkOQCY63d+LfCG3/3s5b5WANe67x8FFgItgVKcYIB11+8LfJHuz9eOzDwstIKRFlR1i4h0BU4GTgdeEZFbgTlAZ2CSG64hHydUbB1j3euniUiJiLTCUXzPi8ghOJEDC2IQpRdOML2b3fMioIP7/kNV3QggIuXA/kBr4CNVXeemvwYcGqH+N93XuUBZiPx9cMIh19EDeEqd0LfUtePytvv6BdBCVTcDm0WkWkRaqRM3vxLnT8kwdsMUvpE2VHUXzih4qoh8gRMUai7wpap2D3dZiPPBwBRV/Z04W75NjUEMAS5SZ1ek+kRnAvlnv6RdOL8Xz5tNuNTVUXd9MNtx/mT85QkX76Surtog2Wr96i5y6zSM3TAbvpEWxNln9BC/pC44ZpOlQKk7qYuIFEjgZhSXuukn4URe3Ihjdlnl5veJUZQPgIF+kRKPjlJ+NnCqiOwpTnjai/zyNuM8bcTCVwSO/CcC17h1IyJ7xVjfoeweVdUwAFP4RvpogWOGKReRhTjeLXep6g6cyIUPiMgCHDu/f0zw9SLyMfAUzv6eAA8CQ0RkJo4JKBYG45iAForIIvc8LOrsSnQ/8CnO5jjlOLsOgbO/wt/dyd+DwlQRXN9WYJmIHOwmPYsTJnehe/9/iPF+Tgfei/EaI0ewaJlG1iAiU4GbVXVOmuVo4c5BNMHZ8OU/qhpqU2yv9f0O6Kqq/0yAbNOA81V1fbx1GY0PG+EbRuzcJSLzcUwn3xHndnjun0VFvEKJSCnwiCl7Ixw2wjcMw8gRbIRvGIaRI5jCNwzDyBFM4RuGYeQIpvANwzByBFP4hmEYOcL/D66ArbGEUki7AAAAAElFTkSuQmCC\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"iris1 = iris_dataset.plot(\"sepal_length\", \"petal_length\", kind=\"scatter\", title=\"Petal and sepal length in three species of Iris\", color=\"red\")\n",
"iris1.set_xlabel(\"Sepal length (cm)\")\n",
"iris1.set_ylabel(\"Petal length (cm)\")\n",
"\n",
"iris1.plot(iris_dataset.sepal_length, iris_dataset.sepal_length * slopes[0] + intercept, 'black')\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Err...\n",
"\n",
"You might have noticed that this is *not* a very good model:\n",
"\n",
"* There's a lot of points at the bottom of the figure that don't follow the predicted relationship.\n",
"* Petal sizes above 3cm seem to be increasing less quickly as sepal size increases than the predicted relationship.\n",
"\n",
"Let's calculate the predicted petal length for each flower -- what does our model predict as compared to the actual petal length we see?"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"predicted1 = data.plot('petal_length', 'predicted_petal_length', kind='scatter', color='red')\n",
"predicted1.plot(data.petal_length, data.petal_length, 'black')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Loss\n",
"\n",
"Loss is a number indicating how bad the model's prediction was on one particular data point. If the model's prediction is perfect, the loss is zero; otherwise, the loss is greater. **The goal of training a model is to find a set of *weights* and *biases* that have low loss, on average, across all examples.**\n",
"\n",
"In this example, we have **150 data points** that provide a feature (the sepal length) as well as the label (the petal length). We can use these to determine how much total loss our model has over this dataset by calculating predicted labels and comparing them to the actual labels.\n",
"\n",
"There are many different measures of loss. One common measure of loss that is particularly useful in linear regressions is *squared loss* (or $L_2$ loss). This is defined as *the square of the difference between the label and the prediction*. In other words, it is equal to:\n",
"$$ = (predicted\\ label - actual\\ label)^2 $$\n",
"$$ = (observation - prediction(x))^2 $$\n",
"$$ = (y - y')^2 $$\n",
"\n",
"We can use this equation to find the loss for a single data point. What does this look like in Python?"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
sepal_length
\n",
"
petal_length
\n",
"
predicted_petal_length
\n",
"
squared_error
\n",
"
\n",
" \n",
" \n",
"
\n",
"
0
\n",
"
5.1
\n",
"
1.4
\n",
"
2.377918
\n",
"
0.956323
\n",
"
\n",
"
\n",
"
1
\n",
"
4.9
\n",
"
1.4
\n",
"
2.006416
\n",
"
0.367740
\n",
"
\n",
"
\n",
"
2
\n",
"
4.7
\n",
"
1.3
\n",
"
1.634914
\n",
"
0.112167
\n",
"
\n",
"
\n",
"
3
\n",
"
4.6
\n",
"
1.5
\n",
"
1.449163
\n",
"
0.002584
\n",
"
\n",
"
\n",
"
4
\n",
"
5.0
\n",
"
1.4
\n",
"
2.192167
\n",
"
0.627528
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" sepal_length petal_length predicted_petal_length squared_error\n",
"0 5.1 1.4 2.377918 0.956323\n",
"1 4.9 1.4 2.006416 0.367740\n",
"2 4.7 1.3 1.634914 0.112167\n",
"3 4.6 1.5 1.449163 0.002584\n",
"4 5.0 1.4 2.192167 0.627528"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data['squared_error'] = (data.petal_length - data.predicted_petal_length)**2\n",
"data.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"But how can we measure our total loss across all our 150 data points?\n",
"\n",
"## Mean Square Error (MSE)\n",
"\n",
"The Mean Square Error (MSE) can be calculated as the arithmetic mean of all squared losses in a particular dataset $D$. We can calculate this as the total squared loss divided by the number of data points, i.e.:\n",
"\n",
"$$ MSE = \\frac{1}{N} \\sum_{(x, y)\\ \\in\\ D}{(y - y')^2} $$\n",
"\n",
"What does this look like in Python?"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.7423201713947026"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data.squared_error.mean()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Tada!\n",
"\n",
"We now have:\n",
"* A model that allows us to *predict* (or *infer*) the petal length from the sepal length within this dataset.\n",
"* A measure for determining how much the prediction varied from the actual *label*, i.e. its *loss*.\n",
"* A measure for determining how well the model performed on a particular dataset."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercises\n",
"\n",
"Here are a few exercises to test your understanding of this material."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise 1\n",
"\n",
"In the [Iris flower dataset](https://en.wikipedia.org/wiki/Iris_flower_data_set), we looked at whether we could predict the petal length based on sepal length.\n",
"\n",
"For this exercise, try using the sepal width to predict petal width, find the equation of the line of best fit, and plot that line on the same graph."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas\n",
"import numpy\n",
"import matplotlib.pyplot as plt\n",
"\n",
"# Import Iris dataset.\n",
"iris_dataset = # How can we load our dataset?\n",
"\n",
"# Plot sepal widths against petal widths.\n",
"iris_dataset.plot(\n",
" # How do we plot this dataset?\n",
")\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Hmm, this is NOT looking good. Oh well, let's see how awful it is!\n",
"# Construct our model.\n",
"slope, intercept = # How do we calculate the slope (weight) and intercept (bias).\n",
"print(slope, intercept)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"iris1 = iris_dataset.plot(\n",
" # How do you plot a pretty graph?\n",
")\n",
"iris1.set_xlabel(\"#TODO\")\n",
"iris1.set_ylabel(\"#TODO\")\n",
"iris1.plot(\n",
" # How can you plot the line of best fit?\n",
")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise 2\n",
"\n",
"For the model that predicts petal length from sepal length in Exercise 1, calculate the Mean Square Error (MSE)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Calculate predicted petal widths.\n",
"iris_dataset['predicted_petal_width'] = # How?\n",
"iris_dataset['squared_error'] = # How??\n",
"print(\"The mean squared error is: \",\n",
" # How???\n",
")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}