{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Logistic Regression With Linear Boundary Demo\n",
"\n",
"_Source: 🤖[Homemade Machine Learning](https://github.com/trekhleb/homemade-machine-learning) repository_\n",
"\n",
"> ☝Before moving on with this demo you might want to take a look at:\n",
"> - 📗[Math behind the Logistic Regression](https://github.com/trekhleb/homemade-machine-learning/tree/master/homemade/logistic_regression)\n",
"> - ⚙️[Logistic Regression Source Code](https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/logistic_regression/logistic_regression.py)\n",
"\n",
"**Logistic regression** is the appropriate regression analysis to conduct when the dependent variable is dichotomous (binary). Like all regression analyses, the logistic regression is a predictive analysis. Logistic regression is used to describe data and to explain the relationship between one dependent binary variable and one or more nominal, ordinal, interval or ratio-level independent variables.\n",
"\n",
"Logistic Regression is used when the dependent variable (target) is categorical.\n",
"\n",
"For example:\n",
"\n",
"- To predict whether an email is spam (`1`) or (`0`).\n",
"- Whether online transaction is fraudulent (`1`) or not (`0`).\n",
"- Whether the tumor is malignant (`1`) or not (`0`).\n",
"\n",
"> **Demo Project:** In this example we will try to classify Iris flowers into tree categories (`Iris setosa`, `Iris virginica` and `Iris versicolor`) based on `petal_length` and `petal_width` parameters."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# To make debugging of logistic_regression module easier we enable imported modules autoreloading feature.\n",
"# By doing this you may change the code of logistic_regression library and all these changes will be available here.\n",
"%load_ext autoreload\n",
"%autoreload 2\n",
"\n",
"# Add project root folder to module loading paths.\n",
"import sys\n",
"sys.path.append('../..')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Import Dependencies\n",
"\n",
"- [pandas](https://pandas.pydata.org/) - library that we will use for loading and displaying the data in a table\n",
"- [numpy](http://www.numpy.org/) - library that we will use for linear algebra operations\n",
"- [matplotlib](https://matplotlib.org/) - library that we will use for plotting the data\n",
"- [logistic_regression](https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/logistic_regression/logistic_regression.py) - custom implementation of logistic regression"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# Import 3rd party dependencies.\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"# Import custom logistic regression implementation.\n",
"from homemade.logistic_regression import LogisticRegression"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Load the Data\n",
"\n",
"In this demo we will use [Iris data set](http://archive.ics.uci.edu/ml/datasets/Iris).\n",
"\n",
"The data set consists of several samples from each of three species of Iris (`Iris setosa`, `Iris virginica` and `Iris versicolor`). Four features were measured from each sample: the length and the width of the sepals and petals, in centimeters. Based on the combination of these four features, [Ronald Fisher](https://en.wikipedia.org/wiki/Iris_flower_data_set) developed a linear discriminant model to distinguish the species from each other."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
sepal_length
\n",
"
sepal_width
\n",
"
petal_length
\n",
"
petal_width
\n",
"
class
\n",
"
\n",
" \n",
" \n",
"
\n",
"
0
\n",
"
5.1
\n",
"
3.5
\n",
"
1.4
\n",
"
0.2
\n",
"
SETOSA
\n",
"
\n",
"
\n",
"
1
\n",
"
4.9
\n",
"
3.0
\n",
"
1.4
\n",
"
0.2
\n",
"
SETOSA
\n",
"
\n",
"
\n",
"
2
\n",
"
4.7
\n",
"
3.2
\n",
"
1.3
\n",
"
0.2
\n",
"
SETOSA
\n",
"
\n",
"
\n",
"
3
\n",
"
4.6
\n",
"
3.1
\n",
"
1.5
\n",
"
0.2
\n",
"
SETOSA
\n",
"
\n",
"
\n",
"
4
\n",
"
5.0
\n",
"
3.6
\n",
"
1.4
\n",
"
0.2
\n",
"
SETOSA
\n",
"
\n",
"
\n",
"
5
\n",
"
5.4
\n",
"
3.9
\n",
"
1.7
\n",
"
0.4
\n",
"
SETOSA
\n",
"
\n",
"
\n",
"
6
\n",
"
4.6
\n",
"
3.4
\n",
"
1.4
\n",
"
0.3
\n",
"
SETOSA
\n",
"
\n",
"
\n",
"
7
\n",
"
5.0
\n",
"
3.4
\n",
"
1.5
\n",
"
0.2
\n",
"
SETOSA
\n",
"
\n",
"
\n",
"
8
\n",
"
4.4
\n",
"
2.9
\n",
"
1.4
\n",
"
0.2
\n",
"
SETOSA
\n",
"
\n",
"
\n",
"
9
\n",
"
4.9
\n",
"
3.1
\n",
"
1.5
\n",
"
0.1
\n",
"
SETOSA
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" sepal_length sepal_width petal_length petal_width class\n",
"0 5.1 3.5 1.4 0.2 SETOSA\n",
"1 4.9 3.0 1.4 0.2 SETOSA\n",
"2 4.7 3.2 1.3 0.2 SETOSA\n",
"3 4.6 3.1 1.5 0.2 SETOSA\n",
"4 5.0 3.6 1.4 0.2 SETOSA\n",
"5 5.4 3.9 1.7 0.4 SETOSA\n",
"6 4.6 3.4 1.4 0.3 SETOSA\n",
"7 5.0 3.4 1.5 0.2 SETOSA\n",
"8 4.4 2.9 1.4 0.2 SETOSA\n",
"9 4.9 3.1 1.5 0.1 SETOSA"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Load the data.\n",
"data = pd.read_csv('../../data/iris.csv')\n",
"\n",
"# Print the data table.\n",
"data.head(10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Plot the Data\n",
"\n",
"Let's take two parameters `petal_length` and `petal_width` for each flower into consideration and plot the dependency of the Iris class on these two parameters."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEXCAYAAACpuuMDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl4VeW1x/HvIiBBBKkEC4JIBAEZNGjqhANK61AoWK+oaK1oW3rrtdqrpYPSihW1DtfZW0sdkCsXFScsVKwXaYtW0DAosyMKSBVxYBQhWfePsxNzTk6SfXLmnN/nefIk593v3nudoGdl7/2u9zV3R0REpFqLbAcgIiK5RYlBRESiKDGIiEgUJQYREYmixCAiIlGUGEREJIoSgxQ8MzvPzP6a7ThEcoUSgzR7ZrbGzL5Z33Z3n+ruJyd4zOVmtjX4qjSzL2q9vjL5qEWyp2W2AxDJJjNr6e67E93P3fvXOsbfgIfd/b5UxiaSLbpikIJiZmPM7CUzu83MNgETgrYXg+0WbPvIzDab2VIzG5DgOYrN7DMzO7hWWxcz225mHc3sm8FVzG/NbJOZvWtm58Tsf6uZrTWzD83sv82sONi2r5n9JTj+J2b2jxT9akRqKDFIIToSeAf4OnBdzLaTgeOB3sDewFnApkQO7u5fAI8B36vVfC7wnLtXH6sb0A7YD/gB8ICZ9Qq23QyUAocABwE9gKuCbeOC2DsBnYHxicQmEoYSgxSiD9z9Lnff7e47YrbtIvKB3Rcwd1/p7huacI6HgHPNzILX5wP/U2t7FXC1u+909xeA2cAoM2sB/Aj4mbt/6u6bgRuA6iuKXUSSSXd3/9LddcUgKafEIIVobX0bgg/pu4F7gI/MbJKZtU/0BO7+ErAbODa4FdUdmFWryyZ3317r9XtEPvA7A62B14LbRZ8BM4F9g36/D/rOMbO3zWxcorGJNEaJQQpRg1MKu/ud7n440I/ILaWmfvhOIXI76XzgMXffWWtbRzNrU+t1d+AD4EPgS6CPu3cIvvZ2972D2Da7+3+6ew/gdOCXZnZCE+MTiUuJQaQWM/uGmR1pZq2AbcAXRG77NMX/AGcSeb4wJWZbCyIPvvcwsyHAacDj7l4J3Afcbmadgofh3czs5CC+75hZz+AW1edAZRLxicSlxCASrT3wJ+BTIrdsNhF5GJwwd18DLAV2uvs/YzavI5J4NhB5HvFDd38z2HZFcO5XiHz4/5XIQ2iAPsALwFbgJeAOd5/XlPhE6mNaqEckfcxsCvCOu0+o1fZN4L7gdpBIzlGBm0iamNmBwEhgYLZjEUmEbiWJpIGZ3QC8Blzv7u9nOx6RROhWkoiIRNEVg4iIRMnLZwwlJSXeo0ePbIchIpJXFi5c+LG7d2qsX14mhh49elBRUZHtMERE8oqZvRemn24liYhIFCUGERGJosQgIiJR8vIZQzy7du1i3bp1fPHFF9kOpeAVFxfTrVs3WrVqle1QRKQJmk1iWLduHe3ataNHjx58NQW+ZJq7s2nTJtatW0dpaWm2wxGRJmg2t5K++OILOnbsqKSQZWZGx44ddeUmkseaTWIAlBRyhP4dRPJbWm8lmdn+ROah/zqRxVEmufsdMX2GADOAd4OmJ939d+mMS0Tyz6x3ZnHHojv417Z/0bltZy477DKGHTgsqf2BpI7ZXKX7GcNu4Ap3X2Rm7YCFZva8u6+I6TfP3YenOZaMuO666/jf//1fioqKaNGiBX/84x/55S9/yYYNG2jTJrJgV69evRg0aBDTp08HYOnSpQwcGJmA86KLLuLSSy9l0qRJ3HrrrQC0b9+eW2+9lWOPPRaAmTNn8pvf/Iaqqip27drFZZddxo9//OOaGMrKyujbty+PPPJIJt+6SNrMemcWE/45gS8qI7coN2zbwIR/TgAI9UEeb//xL47HzNhVtatJx2zO0poYgkXUNwQ/bzGzlUBXIDYxNAsvv/wyM2fOZNGiRbRu3ZqPP/6YL7/8EoCpU6dSXl4e1f+qq64CYK+99mLJkiU17TNnzuSPf/wjL774IiUlJSxatIjTTz+dV155hY4dOzJ27FheeeUVunXrxs6dO1mzZk3NvitXrqSyspJ58+axbds22rZtm/43LpJmdyy6o+ZDvdoXlV9wx6I7Qn2Ix9t/t++us8hrIsdszjL2jMHMegCDgAVxNh9tZq+Z2bNm1r+e/ceaWYWZVWzcuDHpeJ5evJ7Bv3+B0l/NYvDvX+DpxeuTPuaGDRsoKSmhdevWAJSUlLDffvslfJwbb7yRm2++mZKSEgAOO+wwLrjgAu655x62bNnC7t276dixIwCtW7emT58+NftOmzaN888/n5NPPpkZM2Yk/Z5EcsG/tv0rofam9ku0b3OVkcRgZnsBTwA/c/fNMZsXAQe4+6HAXcDT8Y7h7pPcvdzdyzt1anQOqAY9vXg9v35yKes/24ED6z/bwa+fXJp0cjj55JNZu3YtvXv35uKLL+bvf/97zbbzzjuPsrIyysrKGDeu4bXlly9fzuGHHx7VVl5ezvLly9lnn30YMWIEBxxwAKNHj2bq1KlUVX215O+jjz7KOeecw+jRo5k2bVpS70ckV3Ru2zmh9qb2S7Rvc5X2xBAsqv4EMNXdn4zd7u6b3X1r8PNfgFZmVpLOmG5+bjU7dlVGte3YVcnNz61O6rh77bUXCxcuZNKkSXTq1Imzzz6byZMnA5FbSUuWLGHJkiXcfHOTlhCucd999zFnzhyOOOIIbrnlFi666CIAKioqKCkpoXv37gwdOpTFixfzySefJHUukVxw2WGXUVxUHNVWXFRc8wC5Kfu3tJa0ahFdhJnIMZuzdI9KMuB+YKW731pPn87Ah+7uZnYEkWS1KZ1xffDZjoTaE1FUVMSQIUMYMmQIAwcO5KGHHkr4GP369WPhwoWcdNJJNW0LFy6kf/+v7rINHDiQgQMHcv7551NaWsrkyZOZNm0aq1atonpK8s2bN/PEE0/wox/9KOn3JZJN1ff8mzqCqL79kzlmc5buUUmDgfOBpWZW/XT1SqA7gLvfC5wJ/MTMdgM7gHM8zcvK7dehDevjJIH9OrRJ6rirV6+mRYsWHHTQQQAsWbKEAw44gGXLliV0nF/84hf88pe/ZPbs2XTs2JElS5YwefJkFixYwNatW6moqGDIkCFR56iqquKxxx5j6dKlNc815s6dy7XXXqvEIM3CsAOHJfWhXd/+SgR1pXtU0otAg9VO7n43cHc644g17pQ+/PrJpVG3k9q0KmLcKX0a2KtxW7du5ac//SmfffYZLVu2pFevXkyaNIkzzzyT8847r2a4aklJCf/3f/9X73FGjBjB+vXrOeaYYzAz2rVrx8MPP0yXLl3YsmULN910Ez/+8Y9p06YNbdu2ZfLkycybN4+uXbtGPew+/vjjWbFiBRs2bKBLly5JvTcRKRx5ueZzeXm5xy7Us3LlSg4++ODQx3h68Xpufm41H3y2g/06tGHcKX04fVDXVIdasBL995D8kWyhWRgT509k+hvTqfIqWlgLRvUexfijxqf0HIXIzBa6e3lj/ZrNJHqJOn1QVyUCkQQlW2gWxsT5E3l09aM1r6u8qua1kkNmNKu5kkQkvRoqNEuV6W9MT6hdUk+JQURCS7bQLIwqr0qoXVJPiUFEQku20CyMFhb/Y6m+dkk9/aZFJLRkC83CGNV7VELtknoF+/BZRBKXbKFZGNUPmDUqKYvcPe++Dj/8cI+1YsWKOm2ZNGTIEJ89e3ZU22233eannnqqFxcX+6GHHlrz9dBDD7m7+wEHHOADBgzwgQMH+vHHH+9r1qyp2XfixIner18/HzhwoB966KE+f/58d3c/4YQT/NVXX3V39y1btvjYsWP9wAMP9MMOO8xPOOGEmn5r1671ESNGeK9evfzAAw/0Sy+91Hfu3Onu7nPnzvVhw4bVeQ87d+70yy67zHv27Om9evXyESNG+Nq1a2u2t2jRwg899FDv37+/Dx8+3D/99NN6fx/Z/vcQkbqACg/xGasrhhQZPXo0jzzyCKecckpN2yOPPMJNN93E2rVro6bVrm3u3LmUlJRw9dVXM3HiRP70pz81OH13bT/84Q8pLS3lzTffpEWLFrz77rusWLECd+eMM87gJz/5CTNmzKCyspKxY8dy1VVXNThP05VXXsmWLVtYvXo1RUVFPPjgg5xxxhksWLAAM6NNmzY176N6ttfqqcOlcIRd8CZsW9irjbD1E4nUWWSiJiNZ2YixcBPD64/BnN/B5+tg724w9LdwyFlNPtyZZ57J+PHj+fLLL9ljjz1Ys2YNH3zwAfvvv3+o/Y8++mjuvPNOIP703bHefvttFixYwNSpU2nRIvKoqLS0lNLSUubMmUNxcTEXXnghEJm/6bbbbqO0tJRrrrkm7vm3b9/Ogw8+yLvvvktRUREAF154IQ888AAvvPACQ4cOrRPv66+/Huq9SfMRdsGb37z0G9w9suZBA/3C1kCErZ9IpM4iEzUZycpWjIX58Pn1x+DPl8LnawGPfP/zpZH2Jtpnn3044ogjePbZZ4HI1cJZZ52FmfH222/XTLldVlbGvHnz6uw/e/ZsTj/9dKDh6burLV++nLKyspoP8dhtsdN2t2/fnu7du/PWW2/Fjf+tt96ie/futG/fPqq9errv2iorK5kzZw4jRoxo4DcizVF9C95Uf9hX21W1qyYpNNQvbA1E2PqJROosMlGTkaxsxViYiWHO72BXzCR6u3ZE2pNQfTsJIolh9OjRAPTs2bNmyu0lS5Zw3HHH1exz4okn0rVrV5599tma/g1N351NO3bsoKysjM6dO/Phhx/yrW99K9shSYalYxGbMMcMWz+RSJ1FJmoykpWtGAszMXy+LrH2kEaOHMmcOXNYtGgR27dvr/NXezxz587lvffeo6ysjKuvvrqmvXr67muuuYa7776bJ554Imq//v3789prr1FZWRl7yJppu2vbvHkz77//Pr169YobR8+ePXn//ffZsmVLVHvt6b6rnzG89957uDv33HNPo+9Pmpd0LGIT5phh6ycSqbPIRE1GsrIVY2Emhr27JdYe0l577cWJJ57IRRddVPPXfxgtW7bk9ttvZ8qUKXzyySesXr2aN998s2Z79dTatfXs2ZPy8nKuvvpqPJgIcc2aNcyaNYuhQ4eyfft2pkyZAkRu/VxxxRWMGTOGPffcM24Mbdu25YILLuDyyy+vSTZTpkxh+/btUetCAOy5557ceeed/Nd//Re7d++OdzhppsIueNOqRStaWstG+4WtgQhbP5FInUUmajKSla0YCzMxDP0ttIpZe6FVm0h7kkaPHs1rr70WlRhinzFUP2SurUuXLowePZp77rmHrVu3csEFF9CvXz8OOeQQVqxYwYQJE+rsc9999/Hhhx/Sq1cvBgwYwJgxY9h3330xM5566immT5/OQQcdRO/evSkuLub666+v2XfOnDl069at5uvll1/mhhtuoLi4mN69e3PQQQcxffp0nnrqKSLrLUUbNGgQhxxyiJYPLTDDDhzGhGMm0KVtFwyjS9suTDx2ItcOvjaq7drB1zLx2ImN9ptwzIRQD1HjnTfevmH7Jdo3W7IVY8FOu53qUUkSTdNui+QeTbvdmEPOUiIQyVHpqFnIJbked+EmBhHJSemoWcgl+RB3YT5jEJGclY6ahVySD3ErMYhITklHzUIuyYe4lRhEJKeko2Yhl+RD3EoMIpJT0lGzkEvyIW4lhhQ58cQTee6556Labr/9dk477TQGDBgAwN/+9jf23ntvysrK6Nu3Lz//+c+j+s+ePZsjjjiCvn37UlZWxtlnn837778PwJgxY3j88ccBGDJkCOXlX404q6ioYMiQITXnGD58eM22Z599lvLycvr168egQYO44ooros5ZVlbGOeeck5pfgkgKpKNmIZfkQ9walZQiDU27ffHFF9e0HXfcccycOZMdO3YwaNAgvvvd7zJ48GCWLVvGT3/6U5555pma8f/PPPMMa9asoXv37nXO99FHH/Hss89y2mmn1RvTsmXLuOSSS5g1axZ9+/alsrKSSZMm1WxfuXIllZWVzJs3j23bttG2bdtU/CpEkjbswGGhC99y6QM1rFyPu2ATQ6rHESc67XabNm0oKytj/fr1ANx4441ceeWVUUVhDc1eOm7cOK677roGE8NNN93EVVddRd++fYHI/Es/+clParZPmzaN888/n5UrVzJjxgzOPffchN6zSKIysZZDrtUI5Fo8YRRkYkjHOOLa026PHDkyatrteD799FPefPNNjj/+eCAyVXbsraWGHH300Tz11FPMnTuXdu3axe2zbNmyOreOanv00Ud5/vnnWbVqFXfddZcSg6RVvP/vwq7bEK9fPtQ25Fo8YRXkM4Z0jSOub9rt2ubNm8ehhx5K165dOeWUU+jcue5IhE2bNlFWVkbv3r255ZZb6j3f+PHjmThxYpNiraiooKSkhO7duzN06FAWL17MJ5980qRjiYQR7/+7sOs2xOuXD7UNuRZPWAWZGNI1jjjMtNvHHXccr732GsuXL+f++++vWSqzf//+LFq0CICOHTuyZMkSxo4dy9atW+s930knncSOHTuYP39+3O39+/evM/12tWnTprFq1Sp69OhBz5492bx5c52pvUVSKRNrOeRajUCuxRNWQSaGdI0jTmTa7dLSUn71q19x4403AvCLX/yC6667jpUrV9b02b59e6PnHD9+PDfddFPcbePGjeP666/njTfeAKCqqop7772XqqoqHnvsMZYuXcqaNWtYs2YNM2bM0EypklaZWMsh12oEci2esAoyMaRzHHG8abfr8+///u/84x//YM2aNQwcOJA77riD73//+/Tp04fBgwezcuXKRu/7f/vb36ZTp05xtx1yyCHcfvvtjB49moMPPpgBAwbwzjvvMG/ePLp27cp+++1X0/f4449nxYoVbNiwIbE3LBJSvP/vwq7bEK9fPtQ25Fo8YRXstNv5OFIgn2jabYlHo5KyG0/YabcLNjFIeunfQyT35MR6DGa2PzAF+DrgwCR3vyOmjwF3AN8GtgNj3H1ROuMSkWhh/5IfduCwhPo2Z8lcCeTSVUQ8ab1iMLMuQBd3X2Rm7YCFwOnuvqJWn28DPyWSGI4E7nD3Ixs6bn1XDH379q23bkAyx91ZtWqVrhjyROxYe4jc069dNwCRe+Mje41kxlszovq2tJZRdQfVfXNtmodUivc7C/uek9k3WWGvGNL68NndN1T/9e/uW4CVQNeYbiOBKR4xH+gQJJSEFBcXs2nTJvLx1lhz4u5s2rSJ4uLixjtLTghbX/BF5RdMf2N6nb7x6g7yYax+MpKpT8iH2oaMVT6bWQ9gELAgZlNXYG2t1+uCtqjhMWY2FhgLxJ07qFu3bqxbt46NGzemLGZpmuLiYrp165btMCSkRMbUV3lVWo6bb5KpT8iH2oaMJAYz2wt4AviZu29uyjHcfRIwCSK3kmK3t2rVitLS0qTiFClEndt2ZsO2cMOUW1iL0Mkh18fqJ6O+31mY95zMvpmS9joGM2tFJClMdfcn43RZD9Seaa5b0CYiGRC2vqC4qJhRvUfV6Ruv7iAfxuonI5n6hHyobUj3qCQD7gdWuvut9XR7BrjEzB4h8vD5c3dXlZVIhlQ/8Aw70mjQvoMKflRSfb+zsFOFN3XfTEn3qKRjgXnAUqD6+vNKoDuAu98bJI+7gVOJDFe90N0r4hyuRrxRSSIi0rCcqGNw9xeBBsePeiQz/Uc64xCRxNU31n7i/IlMf2M6VV5FC2vBqN6jGH/U+ND755J8iDEbmk3ls4ikTn1j7cs6lTH/X3Vn8z27z9lRySGbY/XDyocYUy0n6hhEJD/VN9Y+XlIAmP7G9FD759JY/XyIMVuUGESkjkTH1McOYc2Hsfr5EGO2KDGISB2JjqlvYdEfJfmwDkE+xJgtSgwiUkd9Y+2P6nxU3P6jeo8KtX8ujdXPhxizJWNTYohI/mhorH2YUUn5MFY/H2LMFo1KEhEpEDlRxyAiuWfizDFM/7iCKiL3kkeVlENJr1C1CZD6sf/xrkDiVVcnu1pbc14/IdV0xSBSQCbOHMOjH1dA7XVL3KNfB2JrEyD1Y/8nzp/Io6sfrdNuGM5Xn03xzpFILPm6fkKqqY5BROqYHpsUIG5SgLq1CZD6sf/xzgFEJYX6zpFILM19/YRUSygxmFlbMytKVzAikl7hV1OIv/ZCqsf+J7O+QyKxNPf1E1KtwcRgZi3M7Fwzm2VmHwGrgA1mtsLMbjazXpkJU0RSIZG/BGNrEyD1Y//jnaM+sedIJJZk4i7EeofG/lXmAj2BXwOd3X1/d98XOBaYD9xoZt9Lc4wikiKjSsojzxRqq+c5Y2xtAqR+7H+8c0DkGUNj50gklua+fkKqNTYq6Zvuviu20d0/IbL4zhPBQjwikgfGD58MSYxKSvXY/+pzNGVUUiKxNPf1E1It9KgkM/sakZXWapKJuy9KU1wN0qgkEZHEpbSOwcyuBcYAb0PNcAEHTmpqgCLSdBkZV//6YzDnd/D5Oti7Gwz9LRxyVmrPITkpbIHbWUBPd/8yncGISONix9Vv2LaBCf+cAJC65PD6Y/DnS2HXjsjrz9dGXoOSQwEIOyRgGdAhnYGISDgZGVc/53dfJYVqu3ZE2qXZC3vFcAOw2MyWATurG919RFqiEpF6ZWRc/efrEmuXZiVsYngIuBFYSmI1MiKSYp3bdmbDtg1x21Nm726R20fx2qXZC3srabu73+nuc93979VfaY1MROLKyLj6ob+FVm2i21q1ibRLsxf2imGemd0APEP0raSsDFcVKWQZGVdf/YBZo5IKUqg6BjObG6fZ3T0rw1VVxyAikriU1jG4+4nJhyQiKRO2xiAdtQg5VN9QaOskZEqoZwxmdr2Zdaj1+mtmNjF9YYlIvaprDD5fC/hXNQavP9a0fuk4dwZU13Ns2LYBx2vqOWa9MyvjsTQ3YR8+n+bun1W/cPdPgW+nJyQRaVDYGoN01CLkUH1DIa6TkClhE0ORmbWufmFmbYDWDfQXkXQJW2OQjlqEHKpvKMR1EjIlbGKYCswxsx+Y2Q+A54nUNohIptVXSxDbHrZfOs6dAYW4TkKmhEoM7n4jMBE4OPi61t1vSmdgIlKPsDUG6ahFyKH6hkJcJyFTGhyVZGbmwXhWd58NzG6oj4hkQNgag3TUIuRQfUMhrpOQKQ3WMZjZ34gsyDPD3d+v1b4HkVXcLgDmuvvk9IYZTXUMIiKJS1Udw6nARcA0MysFPgPaELkF9Vfgdndf3EAQDwDDgY/cfUCc7UOAGcC7QdOT7q7pG6X5S7YW4Ja+sLXWfEl7dYGTf1f3mBDuPDMvh4WTwSvBiuDwMczqNzTuX+OqHWj+ElnBrRVQAuyoPXS1kX2OB7YCUxpIDD939+GhI0ZXDJLnYtc6gMh9+u/cGS45xCaF+rRoBWZQWWsZlXjnmXk5VNwfteustnsyYd99+aLWnJnFRcWM7DWSGW/NiBomWlxUzIRjJig55IGwVwxhRyXh7rvcfUPYpBDs8w/gk7D9RQpCsrUAYZICQNWu6KRQ33kWTq6z6x1f6xCVFCBSIzD9jemqHSgAoRNDGh1tZq+Z2bNm1r++TmY21swqzKxi48aNmYxPJLWyXQsQex6vrNPlXy2L4u5a5fFn3VftQPOS7cSwCDjA3Q8F7gKerq+ju09y93J3L+/UqVPGAhRJuWzXAsSex+omgc676yYLgBYW/yNDtQPNS1YTg7tvdvetwc9/AVqZWUk2YxJJu2RrAfbqEq5fi1ZQtEfj5zl8TJ1dL/v0M4pjPh6Ki4oZ1XuUagcKQNhJ9M4wszfN7HMz22xmW8xsc7InN7POZmbBz0cE8WxK9rgiOe2QsyIPgPfeH7DI97APngF+vqpuctirC5zxp+hjnv7fMPKexs8z/FYo/8FXVw5WxLCDRzPhuOvp0rYLhtGlbRcmHDOB8UeNZ8IxE+q068Fz8xJ2PYa3gO+4+8qEDm42DRhCZDTTh8DVQCsAd7/XzC4BfgLsBnYAl7v7Pxs7rkYliYgkLqXrMQAfJpoUANx9dCPb7wbuTvS4IiKSPo1NiXFG8GOFmT1K5OFw7aU9n0xjbCK5LVML1sQpPmP4reHieX9+3X27H5UTU1o0REV02dXYlBgPNrCvu/tFqQ+pcbqVJFmXbJFaWHGKz4DIM4HaySFePC2KoCrO6KLY9nTEnYTqBXhURJd6YW8lhX3GMNjdX2qsLVOUGCTrbhsQrGIWY+/94T+Xpe481+wTt84AK4Kra9WO1hdPWKmOOwknP34yG7bVLeLr0rYLfz3zr1mIqPlIdeXzXSHbRApDporU4iWFeO3JnjcLC+3URwvwZF9jzxiOBo4BOpnZ5bU2tQfil0aKFIK9u9VzxZDiIjUrqv+KIUw8YWVhoZ36dG7bOe4Vg4roMqexK4Y9gL2IJJB2tb42A2emNzSRHJapBWviFJ/FbY8XT4t6/naLbc/SQjv10QI82dfgFYO7/x34u5lNdvf3MhSTSO7L1II11Q+YGxuVVF88eTgqSQvwZF9jo5L+DNTbwd1HpCOoxujhs4hI4lJV4HZL8P0MoDPwcPB6NJFKZpH8lupahIdGwLt//+p16QnQsVfdv9oh/lVAvJqFeH/hQ9MX5RFpRNjhqhWxWSZeW6boikFSItW1CLFJIVElfeHjVXXbYx9AF+0B7pH1FqqFXZRHClqqh6u2NbMDax28FGjb1OBEckKyC+bESiYpQPykAHVHJVV+GZ0UIPyiPCIhhJ0r6T+Bv5nZO4ABBwA/TltUIpmQ7QVzMqE5vRfJmFCJwd1nm9lBQN+gaZW772xoH5Gcl6lahGxqTu9FMqbBW0lmdlLw/QxgGNAz+BpWa4I9kfyU6lqE0hOSi6ekb/z22GK2oj0izxRqC7soj0gIjT1jqP4v/TtxvoanMS6R9Et2wZxYFzxTNzmUnlBnERzKfxC/7ZIF8du/e290jCPviSzC05RFeURCCDUqKddoVJKISOJSulCPmb0NzAfmAfPcfXmS8Ynkv3g1EBCuliCR+olkai0ytWaENCth6xhaA0cCxwGDgT7A6+7+3fSGF5+uGCTr4q5/ELIOm7DQAAAR6ElEQVSWIJH6iWRqLTK1ZoTkjVTXMVQCu4LvVcBHwZdIYYpXAxG2liCR+olkai1SXachBSNsHcNmYClwK/And9+UvpBE8kAi9QGxfROpn0im1qIQ6jQkLcJeMYwG/gFcDDxiZteY2dD0hSWS4xKpD4jtW9++8doT6ZvKfaWghUoM7j7D3ccRqXb+CzAGmJnGuERyW9z1D0LWEiRSP5FMrUWm1oyQZidUYjCzJ8zsLeAOYE/g+8DX0hmYSE6LVwMRtpYgkfqJZGotUl2nIQUj7KikcmCxe/wFaM3sW+7+fKqDq49GJYmIJC6ldQzu3tin8I1AxhKDNDP5MNY+mZoFkTwTdlRSYyxFx5FCEzvW/vO1kdeQOx+y8WKc8R/RayLkYtwiTRR2VFJj8m9eDckN+TDWPl6M8dZEyLW4RZooVYlBpGnyYax9MjULInkoVYlhTYqOI4UmH8baJ1OzIJKHGnzG0NiaC+7+ZPBdazNI0wz9bfz5fHJprH28GOOtu5xrcYs0UWMPn7/TwDYHnkxhLFKIqh/U5vLonvpijNeWS3GLNJHWYxARKRAprWMIDjgM6A8UV7e5e4NDMMzsASIrvX3k7gPibDci1dTfBrYDY9x9UdiYRJh5OSycDF4ZWfHs8DEw/Nbk+qZ6/QPQlYXklbAL9dxLZCqME4H7gDOBV0LsOhm4G5hSz/bTgIOCryOBPwTfRRo383KouP+r11751evYD/ywfZOpq4i379MXR6/RoHoHyQNhRyUd4+7fBz5192uAo4Heje3k7v8APmmgy0hgikfMBzqYWZeQMUmhWzg5fHvYvqle/yDsGg0iOSRsYqj+r327me1HZNGeVHyAdwXW1nq9Lmirw8zGmlmFmVVs3LgxBaeWvBd/6q747WH7pmP9g2T7imRY2MQw08w6ADcDi4jULUxLV1DxuPskdy939/JOnTpl8tSSq6wofHvYvulY/yDZviIZFjYx3OTun7n7E8ABQF9gYgrOvx7Yv9brbkGbSOMOHxO+PWzfVK9/EHaNBpEcEjYxvFz9g7vvdPfPa7cl4Rng+xZxFPC5u29IwXGlEAy/Fcp/8NVf/VYUeR1vpFHYvqle/yDsGg0iOaTBOgYz60zknv/DwLl8NYtqe+Bed+/b4MHNpgFDgBLgQ+BqoBWAu98bDFe9GziVyHDVC0NM8a06BhGRJkhVHcMpRJbx7AbU/tNqM3BlYwd399GNbHfgPxo7joiIZE6DicHdHwIeMrN/C54viIhIMxf2GcNLZna/mT0LYGb9zOwHaYxLRESyJGxieBB4DtgveP0G8LO0RCQiIlkVNjGUuPtjQBWAu+8G6qkYEhGRfBY2MWwzs44ES3hWDy1NW1QiIpI1YWdXvZxIzUFPM3sJ6ERkIj0REWlmQiUGd19kZicAfYjUMqx2912N7CYiInko7LTbxcDFwLFEbifNM7N73f2LdAYnIiKZF/ZW0hRgC3BX8Ppc4H+AUekISkREsidsYhjg7v1qvZ5rZivSEZCIiGRX2FFJi4KRSACY2ZGAJisSEWmGwl4xHA7808zeD153B1ab2VIiUx4dkpboREQk48ImhlPTGoWIiOSMsMNV30t3ICIikhvCPmMQEZECocQgIiJRlBhERCSKEoOIiERRYhARkShKDCIiEkWJQUREoigxiIhIFCUGERGJEnZKDEnC04vXc/Nzq/ngsx3s16EN407pw+mDumY7LBGRuJQY0uzpxev59ZNL2bGrEoD1n+3g108uBVByEJGcpFtJaXbzc6trkkK1Hbsqufm51VmKSESkYUoMafbBZzsSahcRyTYlhjTbr0ObhNpFRLJNiSHNxp3ShzatiqLa2rQqYtwpfbIUkYhIw/TwOc2qHzBrVJKI5Aslhgw4fVBXJQIRyRtpv5VkZqea2Woze8vMfhVn+xgz22hmS4KvH6Y7plzx9OL1DP79C5T+ahaDf/8CTy9en+2QRETSe8VgZkXAPcC3gHXAq2b2jLuviOn6qLtfks5Yco3qG0QkV6X7iuEI4C13f8fdvwQeAUam+Zx5QfUNIpKr0p0YugJra71eF7TF+jcze93MHjez/eMdyMzGmlmFmVVs3LgxHbFmlOobRCRX5cJw1T8DPdz9EOB54KF4ndx9kruXu3t5p06dMhpgOqi+QURyVboTw3qg9hVAt6Cthrtvcvedwcv7gMPTHFNOUH2DiOSqdCeGV4GDzKzUzPYAzgGeqd3BzLrUejkCWJnmmHLC6YO6csMZA+naoQ0GdO3QhhvOGKgHzyKSdWkdleTuu83sEuA5oAh4wN2Xm9nvgAp3fwa41MxGALuBT4Ax6Ywpl6i+QURykbl7tmNIWHl5uVdUVGQ7DBGRvGJmC929vLF+qnwOKexiO+f96WVeevuTmteDe+7DqPLucfcNe0wt9CMimaQrhhBii9Eg8qA49plAbFKoZkDt33KbVkX82+FdeWLh+kaPGfbcIiKNCXvFkAvDVXNe2GK0eEkBopNC9b7TFqwNdUwVwolIpikxhJCOYrTKeq7UYo+pQjgRyTQlhhDSUYxWZBbqmCqEE5FMU2IIIWwx2uCe+8TdPzYFtGlVxOgj9w91TBXCiUimKTGEELYYbeqPjq6THAb33Ifbzi6rs+/E0weGOqYK4UQk0zQqSUSkQKiOIcXGP72UaQvWUulOkRmjj9yfdzduDV2zEI/qE0QkF+mKIYTxTy/l4fnvh+obr2Yh3q0f1SeISKapjiGFpi1Y23inQLyahXg1B6pPEJFcpcQQQn01B2HFqzlQfYKI5ColhhDqqzkIK17NgeoTRCRXKTGEMPrIuKuNxhWvZiFezYHqE0QkVykxhDDx9IF876juNVcORWZ876juoWsW4j1MVn2CiOQqjUoSESkQqmOII2zdQLyahQXvbOLNj7bV9Dlo37a8u3Ebu2vl1ZYGe+5RxOadX402at+6iDZ7FPHhli9r2r7ebg8WXPUtrccgIjmpYK4YwtYNJFKzkIz2rYvYVYXWYxCRjFEdQ4ywdQOJ1CwkY/POSq3HICI5qWASQ9i6gWRrFpKl9RhEJNsKJjGErRtItmYhWVqPQUSyrWASQ9i6gURqFpLRvnWR1mMQkZxUMIkhbN1AfTULB+3bNqrfQfu2pWXMxUVLi3zg19a+dRFfb7dHVNvX2+3B69ecqvUYRCQnFcyoJBGRQqc6hpCSqRGIt+89c9+sU+/w/OVD0hS9iEjqFfQVQzI1AvH2rY+Sg4jkAtUxhJBMjUC8fetT+wpCRCTXFXRiSKZGQHUEItJcFXRiSKZGQHUEItJcFXRiSKZGIN6+9Ykd6ioikssKOjEkUyMQb9/bzy6LW++gB88ikk8KelSSiEghyZlRSWZ2qpmtNrO3zOxXcba3NrNHg+0LzKxHumMSEZH6pTUxmFkRcA9wGtAPGG1m/WK6/QD41N17AbcBN6YzJhERaVi6rxiOAN5y93fc/UvgEWBkTJ+RwEPBz48DQ82yPMWpiEgBS3di6ArUXvlmXdAWt4+77wY+BzrGHsjMxppZhZlVbNy4MU3hiohI3oxKcvdJ7l7u7uWdOnXKdjgiIs1WuhPDeqD2Agfdgra4fcysJbA3sCnNcYmISD3SPbvqq8BBZlZKJAGcA5wb0+cZ4ALgZeBM4AVvZAztwoULPzaz95KIqwT4OIn9c4neS27Se8ldzen9JPpeDgjTKa2Jwd13m9klwHNAEfCAuy83s98BFe7+DHA/8D9m9hbwCZHk0dhxk7qXZGYVYcby5gO9l9yk95K7mtP7Sdd7Sft6DO7+F+AvMW2/rfXzF8CodMchIiLh5M3DZxERyYxCTQyTsh1ACum95Ca9l9zVnN5PWt5LXs6VJCIi6VOoVwwiIlIPJQYREYlSUInBzB4ws4/MbFm2Y0mWme1vZnPNbIWZLTezy7IdU1OZWbGZvWJmrwXv5Zpsx5QsMysys8VmNjPbsSTDzNaY2VIzW2JmeT3XvZl1MLPHzWyVma00s6OzHVNTmFmf4N+j+muzmf0specopGcMZnY8sBWY4u4Dsh1PMsysC9DF3ReZWTtgIXC6u6/IcmgJCyZNbOvuW82sFfAicJm7z89yaE1mZpcD5UB7dx+e7XiayszWAOXunvcFYWb2EDDP3e8zsz2APd39s2zHlYxgBuv1wJHunkzRb5SCumJw938QKaLLe+6+wd0XBT9vAVZSd4LCvOARW4OXrYKvvP2Lxcy6AcOA+7Idi0SY2d7A8UQKanH3L/M9KQSGAm+nMilAgSWG5ipY3GgQsCC7kTRdcOtlCfAR8Ly75+17AW4HfgFUZTuQFHDgr2a20MzGZjuYJJQCG4EHg1t895lZc1iM/RxgWqoPqsSQ58xsL+AJ4Gfuvjnb8TSVu1e6exmRiRaPMLO8vNVnZsOBj9x9YbZjSZFj3f0wIott/UdwOzYftQQOA/7g7oOAbUCdFSXzSXA7bAQwPdXHVmLIY8H9+CeAqe7+ZLbjSYXg8n4ucGq2Y2miwcCI4N78I8BJZvZwdkNqOndfH3z/CHiKyOJb+WgdsK7WlejjRBJFPjsNWOTuH6b6wEoMeSp4YHs/sNLdb812PMkws05m1iH4uQ3wLWBVdqNqGnf/tbt3c/ceRC7zX3D372U5rCYxs7bBwAaC2y4nA3k5os/d/wWsNbM+QdNQIO8GasQYTRpuI0EGJtHLJWY2DRgClJjZOuBqd78/u1E12WDgfGBpcG8e4Mpg0sJ80wV4KBhh0QJ4zN3zephnM/F14Klgpd2WwP+6++zshpSUnwJTg1sw7wAXZjmeJgsS9beAH6fl+IU0XFVERBqnW0kiIhJFiUFERKIoMYiISBQlBhERiaLEICIiUZQYREQkihKDNDtmNsbM9gvRb7KZndnA9r+ZWXmKY+tgZhfXej0k7NTcZnZ7KqakMLNbzOykZI8jzZcSgzRHY4BGE0OWdAAubrRXDDPrCBwVzBCcrLvI83mCJL2UGCSnmVmPYGGVqcHiKo+b2Z7BtsPN7O/BzJ/PmVmX4AqgnEiF6xIza2NmvzWzV81smZlNCqYTSTSOk83sZTNbZGbTg8kLqxeyuSZoX2pmfYP2Tmb2fLDw0H1m9p6ZlQC/B3oGsd0cHH6vWgvITK0nvn8DaqqOzewbZvZPiyxu9IqZtQuulJ4OzrvGzC4xs8uD2UTnm9k+AMEUzR3NrHOivwcpDEoMkg/6AP/t7gcDm4GLgwkE7wLOdPfDgQeA69z9caACOM/dy9x9B3C3u38jWJypDZDQwjnBB/p44JvBTKMVwOW1unwctP8B+HnQdjWReZL6E5mwrXvQ/isi8+eXufu4oG0Q8DOgH3AgkelOYg0mshhT9ayajxJZzOhQ4JvAjqDfAOAM4BvAdcD2YDbRl4Hv1zreonrOI1JYcyVJ3lrr7i8FPz8MXErkr+cBwPPBH9hFwIZ69j/RzH4B7AnsAywH/pzA+Y8i8qH9UnCuPYh80Farntl2IZEPZYBjge8CuPtsM/u0geO/4u7rAIJ5r3oQWcWuti5E1hOASKLc4O6vBsffHOwLMDdYuGmLmX1e630uBQ6pdbyPyN3bbZJlSgySD2In9HLAgOXu3uC6vWZWDPw3keUp15rZBKA4wfMbkcWDRtezfWfwvZKm/T+1s9bP9R1jB+Hirn2sqlqvq2KOW8xXVxkiUXQrSfJBd/tq4fZzifw1vRroVN1uZq3MrH/QZwvQLvi5+sP04+C5QL2jkBowHxhsZr2Cc7U1s96N7PMScFbQ/2Tga3FiS8RKoFfw82qgi5l9Izh+OzNLNCH1Jk+n0Jb0U2KQfLCayOphK4l8wP7B3b8k8iF/o5m9BiwBjgn6TwbuDW7L7AT+RORD8Dng1URP7u4biYx0mmZmrxO5jdS3kd2uAU42s2XAKOBfwBZ330TkltSyWg+fw5hFZMp4gvd+NnBX8N6fJ4GroOD5TC8iz0pE6tC025LTLLKe9czgwXHeMLPWQKW77w6uav4QLF2azDFfBIYnu4i9mX0XOMzdf5PMcaT50jMGkfToDjxmZi2AL4EfpeCYVwTHTSoxEPn//r+SD0eaK10xSMEzs6eA0pjmX7r7c9mIRyTblBhERCSKHj6LiEgUJQYREYmixCAiIlGUGEREJMr/A3XldMGr8dFdAAAAAElFTkSuQmCC\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# List of suppported Iris classes.\n",
"iris_types = ['SETOSA', 'VERSICOLOR', 'VIRGINICA']\n",
"\n",
"# Pick the Iris parameters for consideration.\n",
"x_axis = 'petal_length'\n",
"y_axis = 'petal_width'\n",
"\n",
"# Plot the scatter for every type of Iris.\n",
"for iris_type in iris_types:\n",
" plt.scatter(\n",
" data[x_axis][data['class'] == iris_type],\n",
" data[y_axis][data['class'] == iris_type],\n",
" label=iris_type\n",
" )\n",
"\n",
"# Plot the data. \n",
"plt.xlabel(x_axis + ' (cm)')\n",
"plt.ylabel(y_axis + ' (cm)')\n",
"plt.title('Iris Types')\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prepara the Data for Training\n",
"\n",
"Let's extract `petal_length` and `petal_width` data and form a training feature set and let's also form out training labels set."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"# Get total number of Iris examples.\n",
"num_examples = data.shape[0]\n",
"\n",
"# Get features.\n",
"x_train = data[[x_axis, y_axis]].values.reshape((num_examples, 2))\n",
"# Get labels.\n",
"y_train = data['class'].values.reshape((num_examples, 1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Init and Train Logistic Regression Model\n",
"\n",
"> ☝🏻This is the place where you might want to play with model configuration.\n",
"\n",
"- `polynomial_degree` - this parameter will allow you to add additional polynomial features of certain degree. More features - more curved the line will be.\n",
"- `max_iterations` - this is the maximum number of iterations that gradient descent algorithm will use to find the minimum of a cost function. Low numbers may prevent gradient descent from reaching the minimum. High numbers will make the algorithm work longer without improving its accuracy.\n",
"- `regularization_param` - parameter that will fight overfitting. The higher the parameter, the simplier is the model will be.\n",
"- `polynomial_degree` - the degree of additional polynomial features (`x1^2 * x2, x1^2 * x2^2, ...`). This will allow you to curve the predictions.\n",
"- `sinusoid_degree` - the degree of sinusoid parameter multipliers of additional features (`sin(x), sin(2*x), ...`). This will allow you to curve the predictions by adding sinusoidal component to the prediction curve."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
Theta 1
\n",
"
Theta 2
\n",
"
Theta 3
\n",
"
\n",
" \n",
" \n",
"
\n",
"
SETOSA
\n",
"
30.291853
\n",
"
-8.404206
\n",
"
-12.313125
\n",
"
\n",
"
\n",
"
VERSICOLOR
\n",
"
-2.858904
\n",
"
1.546523
\n",
"
-3.107221
\n",
"
\n",
"
\n",
"
VIRGINICA
\n",
"
-45.349540
\n",
"
5.766294
\n",
"
10.458614
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Theta 1 Theta 2 Theta 3\n",
"SETOSA 30.291853 -8.404206 -12.313125\n",
"VERSICOLOR -2.858904 1.546523 -3.107221\n",
"VIRGINICA -45.349540 5.766294 10.458614"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Set up linear regression parameters.\n",
"max_iterations = 1000 # Max number of gradient descent iterations.\n",
"regularization_param = 0 # Helps to fight model overfitting.\n",
"polynomial_degree = 0 # The degree of additional polynomial features.\n",
"sinusoid_degree = 0 # The degree of sinusoid parameter multipliers of additional features.\n",
"\n",
"# Init logistic regression instance.\n",
"logistic_regression = LogisticRegression(x_train, y_train, polynomial_degree, sinusoid_degree)\n",
"\n",
"# Train logistic regression.\n",
"(thetas, costs) = logistic_regression.train(regularization_param, max_iterations)\n",
"\n",
"# Print model parameters that have been learned.\n",
"pd.DataFrame(thetas, columns=['Theta 1', 'Theta 2', 'Theta 3'], index=['SETOSA', 'VERSICOLOR', 'VIRGINICA'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Analyze Gradient Descent Progress\n",
"\n",
"The plot below illustrates how the cost function value changes over each iteration. You should see it decreasing. \n",
"\n",
"In case if cost function value increases it may mean that gradient descent missed the cost function minimum and with each step it goes further away from it.\n",
"\n",
"From this plot you may also get an understanding of how many iterations you need to get an optimal value of the cost function. "
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAELCAYAAAA2mZrgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl8VOX1+PHPmUnCZCNhCSQhLBHCEhaDRlqrIi5QtRbUUgVtK9WKtl+sW92ta21darULv1ZcqlYLtbZW6oJFpQrWLWzKIrLIEvadhJBt5vz+mJtxEiYLITeTZM779ZpX5j73zr3nOpiT+9z7nEdUFWOMMQbAE+0AjDHGtB2WFIwxxoRYUjDGGBNiScEYY0yIJQVjjDEhlhSMMcaEuJoUROQsEVklImtE5JYI6x8VkSXO6wsR2edmPMYYYxombo1TEBEv8AUwFigGPgEmq+qKera/Ghipqpe5EpAxxphGuXmlMApYo6rrVLUSmAVMaGD7ycBMF+MxxhjTCDeTQi9gU9hysdN2GBHpC+QC77gYjzHGmEbERTsAxyTgJVX1R1opIlOBqQDJycnHDx48uDVjM8aYdm/hwoW7VDWjse3cTAqbgd5hyzlOWySTgP+rb0eqOgOYAVBYWKhFRUUtFaMxxsQEEdnQlO3c7D76BMgTkVwRSSD4i3923Y1EZDDQBfjAxViMMcY0gWtJQVWrgWnAm8BK4EVVXS4i94rI+LBNJwGz1Mq1GmNM1Ll6T0FVXwder9N2Z53lu92MwRhjTNO1lRvNxph2rKqqiuLiYsrLy6MdSszz+Xzk5OQQHx/frM9bUjDGHLXi4mJSU1Pp168fIhLtcGKWqrJ7926Ki4vJzc1t1j6s9pEx5qiVl5fTrVs3SwhRJiJ069btqK7YLCkYY1qEJYS24Wi/h9hJCju/gLfuBnvIyZgO6/7772fo0KGMGDGCgoICPvroI8aMGcOgQYMoKCigoKCAiRMncv/994eWvV5v6P3vfvc7AGbMmMHgwYMZPHgwo0aNYsGCBaFjvPrqq4wcOZJjjz2W/Px8Hn/88VoxFBQUMGnSpFY975YUO/cU1syFBY9CWg6c8KNoR2OMaWEffPABr776KosWLaJTp07s2rWLyspKAF544QUKCwtrbX/77bcDkJKSwpIlS0Ltr776Ko8//jgLFiyge/fuLFq0iPPOO4+PP/6Ybt26MXXqVD7++GNycnKoqKhg/fr1oc+uXLkSv9/P/PnzOXjwIMnJye6feAuLnSuFr/0YBpwJc26D7cujHY0xpoVt3bqV7t2706lTJwC6d+9Odnb2Ee/nwQcf5OGHH6Z79+4AHHfccVx66aVMnz6dkpISqqur6datGwCdOnVi0KBBoc/OnDmT73//+4wbN45XXnmlBc6q9cVOUvB44Lw/gi8NXroMKsuiHZExpgWNGzeOTZs2MXDgQH7yk5/w7rvvhtZdcskloS6iG2+8scH9LF++nOOPP75WW2FhIcuXL6dr166MHz+evn37MnnyZF544QUCgUBou7/97W9MmjSJyZMnM3Nm+yz6HDvdRwApPeCCx+Ev58Obt8K3fxvtiIzpcO7593JWbDnQovvMz+7MXd8e2uA2KSkpLFy4kPnz5zNv3jwuuugiHnjgASBy91FzPfnkk3z22We89dZb/PrXv2bu3Lk888wzFBUV0b17d/r06UOvXr247LLL2LNnD127dm2R47aW2LlSqNH/dDjpWlj4DCx/OdrRGGNakNfrZcyYMdxzzz384Q9/4B//+McR7yM/P5+FCxfWalu4cCFDh36VlIYPH851113H3LlzQ8eYOXMmn3/+Of369aN///4cOHCgWcePtti6Uqhx+h2wfj7Mvgayj4MufaMdkTEdRmN/0btl1apVeDwe8vLyAFiyZAl9+/Zl2bJlR7Sfm266iZtvvpk5c+bQrVs3lixZwjPPPMNHH31EaWkpRUVFjBkzptYxAoEAL774Ip999lnoPsa8efO47777uOKKK1r0PN0Wm0nBGw/feQoeHw3/+BH88A3wxuZ/CmM6itLSUq6++mr27dtHXFwcAwYMYMaMGUycOJFLLrmExMREIHgD+q233qp3P+PHj2fz5s184xvfQERITU3l+eefJysri5KSEh566CGuvPJKEhMTSU5O5plnnmH+/Pn06tWr1o3t0aNHs2LFCrZu3UpWVpbr599SXJuj2S0tOp/CZy/BPy6HU34GZ/y8ZfZpTAxauXIlQ4YMiXYYxhHp+xCRhara6I2V2LunEG74RBj5PZj/CKx7t/HtjTGmg4vtpABw9kPQPQ/+ORUO7op2NMYYE1WWFBKSYeLTcGgP/OsnVgbDGBPTLCkAZA6Hcb+A1W/Ch3+MdjTGGBM1MZMU1u1fx7/W/Kv+DUZNhUHnwNw7YcuS+rczxpgOLGaSwnub3uPn7/+cksqSyBuIwITpkJwRLIOxey3sL4YDW6BkG5TugIO7oWwPHNoH5QegohT81a17IsYY46KYSQqZyZkAbDu4rf6NkrrCd56APevg98fBo0PhN0PgkUHw6zx4+Bh4KBce7AsP9IZf9Qq2r3m7lc7CGBPJaaedxptvvlmr7bHHHuPss88mMTExVPeooKCA5557DoB+/foxfPhwRowYwamnnsqGDRtCn41UghtgzJgx1DwSX1paypVXXkn//v05/vjjGTNmTGi74uJiJkyYQF5eHv379+eaa64JVWz973//y7nnnnvYOVRWVnLttdcyYMAA8vLymDBhAsXFxaH1NSW+hw0bxre//W327dvXgv8FvxIzI7ZqksL2su3kdcmrf8N+J8Pl/4Gdq0ADdV56eNvSWfDCRPjmL+FrVwWvOIwxrWry5MnMmjWLb37zm6G2WbNm8dBDD7Fp06ZapbHDzZs3j+7du3PXXXfxi1/8gieeeKLBEtzhfvSjH5Gbm8vq1avxeDx8+eWXrFixAlXlggsu4Mc//jGvvPIKfr+fqVOncvvtt/Pwww/Xew633XYbJSUlrFq1Cq/Xy5///GcuuOACPvroI0SExMTE0HnUVG2tKf/dklxNCiJyFvBbwAs8qaoPRNjmQuBuQIGlqnqxG7H0TOoJNHKlUKP3qOCrKQovg5evhDm3BEtyf+sRiOt0FJEaY47UxIkTueOOO6isrCQhIYH169ezZcsWevfu3aTPn3jiiaEJdiKV4K5r7dq1fPTRR7zwwgt4PMEOl9zcXHJzc3n77bfx+Xz88Ic/BIJ/4T/66KPk5uZyzz33RDx+WVkZf/7zn/nyyy/xer0A/PCHP+Tpp5/mnXfe4Ywzzjgs3k8//bRJ53akXOs+EhEvMB04G8gHJotIfp1t8oBbgZNUdShwrVvxZCRl4BFP05LCkeiUAhf+BUbfCIv/As+Oh9KdLXsMY0yDunbtyqhRo3jjjTeA4FXChRdeiIiwdu3aWt1H8+fPP+zzc+bM4bzzzgMaLsFdY/ny5aFZ2yKtq1t6u3PnzvTp04c1a9ZEjH/NmjX06dOHzp0712qvKdkdzu/38/bbbzN+/PgG/os0n5tXCqOANaq6DkBEZgETgBVh21wBTFfVvQCqusOtYOI8cXRP7N7ySQGCczWcfgf0GBIc6/DEaTB5ZvBRV2NizRu3wLbPWnafmcPh7MM6Gmqp6UKaMGECs2bN4qmnngKgf//+9XYfnXbaaezZs4eUlBTuu+8+oP4S3FOmTGnRUzpShw4doqCggM2bNzNkyBDGjh3rynHcvNHcC9gUtlzstIUbCAwUkfdF5EOnu8k1mcmZbC/b7t4Bhn0HLpsDAT88NQ5WzHbvWMaYWiZMmMDbb7/NokWLKCsrO+yv9UjmzZvHhg0bKCgo4K677gq1N1aCe+jQoSxduhS/33/YPiOV3j5w4AAbN25kwIABEePo378/GzdupKSk9tOR4SW7a+4pbNiwAVVl+vTpjZ5fc0T7RnMckAeMAXKA90RkuKrWuq0uIlOBqQB9+vRp9sEykzL5Yu8Xzf58k2SPhKnzYNYl8OL3YcxtcOpNdgPaxI5G/qJ3S0pKCqeddhqXXXYZkydPbvLn4uLieOyxxxg+fDh33HEHO3fujFiCO1z//v0pLCzkrrvu4r777kNEWL9+PcuXL+ecc87hlltu4bnnnuMHP/gBfr+fG264gSlTppCUlBQxhuTkZC699FKuv/56/vSnP+H1ennuuecoKyvj9NNPr7VtUlISv/vd7zjvvPP4yU9+Qlxcy/4ad/NKYTMQfpcnx2kLVwzMVtUqVf0S+IJgkqhFVWeoaqGqFmZkZDQ7oJ7JPdleth3XK8OmZsKU12DEJPjvL+HvU2z6T2NaweTJk1m6dGmtpFD3nkLNDeVwWVlZTJ48menTp1NaWsqll15Kfn4+I0aMYMWKFdx9992HfebJJ59k+/btDBgwgGHDhjFlyhR69OiBiPDyyy/z97//nby8PAYOHIjP5+OXv/xl6LNvv/02OTk5odcHH3zAr371K3w+HwMHDiQvL4+///3vvPzyy0iEPyhHjhzJiBEjXJny07XS2SISR/CX/BkEk8EnwMWqujxsm7OAyap6qYh0BxYDBaq6u779Hk3p7OeWP8fDRQ+zYNIC0jqlNWsfR0QV/vc7mHuXU0rjPuh1PHRKdf/YxrQiK53dthxN6WzXuo9UtVpEpgFvEnwk9WlVXS4i9wJFqjrbWTdORFYAfuDGhhLC0QofwNYqSUEETroGMoYE5214bgIgkDEYco4PJohehdAj3yb5Mca0Ca7+JlLV14HX67TdGfZegeudl+vCB7AN6jqoNQ4ZNHAcXLcMij+B4oWwuQg+fx0WPx9cH58EWQXQ6zjIKYTMEcHR1Z3Sgk82GWNMK4mpP0+bVOrCLb40GHBm8AXBrqW962HzQiguCiaKj5+AD/4Q9iEJfi6xCySmB3/60msvx/mC04t64p2fccGf3oSv3nuc5fhEiPcFk1Cc89MbbzfBjTEhMZUUuvm6ESdx0UkKdYlA19zga/jEYFt1JWxfFiyxcWhv8FW+76v3h/bB3g1ftWmgBeLwOskiEeKcpOE5kn8WEpZUBKS+9rqJJ2y5oXUtpcmJr50kyBMuh2MnRTsK0wHFVFLwerxkJGW0jaQQSVxCsAup13GNbxsIQGUJVFeAvwoCVcGKrYEq8FeGvXfWVVdC9SGoKoeqMqh2flaVQ9UhZ92hYFtTk034QwqqBCuVNPA+0ucaXNdSmrjP9jTBkjc+2hGYDiqmkgIEu5C2lbXRpHAkPJ5g15IxxrSgmLuLmZmUyfaDLo5qNsa0uoZKZw8bNgwIlqxOS0ujoKCAwYMH87Of/azW9nPmzGHUqFEMHjyYgoICLrroIjZu3AjAlClTeOmll4Bg+ezCwq+e7CwqKmLMmDGhY4SXxX7jjTcoLCwkPz+fkSNHcsMNN9Q6ZkFBAZMmta1uwJhLCq02gM0Y02pq6h6FmzVrFrfeemuttlNOOYUlS5awePFiXn31Vd5//30Ali1bxtVXX82zzz7L559/zpIlS7jkkktYv359xOPt2LEjVHyvPsuWLWPatGk8//zzrFixgqKiolplLlauXInf72f+/PkcPHiwGWftjphLCpnJmVT4K9hbsTfaoRhjWsjEiRN57bXXQvMeNFY6u2binc2bg0UWHnzwQW677bZaA77Gjx/P6NGjI37+xhtv5P77728wpoceeojbb7+dwYMHA8F6Sj/+8Y9D62fOnMn3v/99xo0bxyuvvNL0k3VZ7CWFpCg+lmqMcUVDpbMj2bt3L6tXrw790l++fDnHHdeEBzwcJ554IgkJCcybN6/ebZYtW9ZgUb6//e1vTJo0icmTJ7tSrqK5YvJGM8D2g9vJ75bfyNbGmCP14McP8vmez1t0n4O7DubmUTc3uE19pbPDzZ8/n2OPPZbVq1dz7bXXkpmZedg2u3fv5owzzqCsrIypU6cedu+hxh133MEvfvELHnzwwSM+n6KiIrp3706fPn3o1asXl112GXv27KFr165HvK+WFnNXCj2TnRnYOsITSMaYkKaUzj7llFNYunQpy5cv56mnngrNszB06FAWLVoEQLdu3ViyZAlTp06ltLS03uOdfvrpHDp0iA8//DDi+qFDhx5WQrvGzJkz+fzzz+nXrx/9+/fnwIEDh5XnjpaYu1Lo6utKnKeNDGAzpgNq7C96txxJ6ezc3FxuueUWHnzwQWbOnMlNN93E+eefz9e//vXQfYWyssYrG99xxx1cddVVHHPMMYetu/HGG7ngggs4+eSTGThwIIFAgBkzZjB16lRefPFFPvvsM7Kzs4HgvA733XcfV1xxRTPOvGXF3JWCRzz0TOppScGYDihS6ez6XHXVVbz33nusX7+e4cOH89vf/pYf/OAHDBo0iJNOOomVK1dy8cUNTxl/zjnnUF85/xEjRvDYY48xefJkhgwZwrBhw1i3bh3z58+nV69eoYQAMHr0aFasWMHWrVuP7IRd4FrpbLccTensGlPmTEFVefbsZ1soKmNim5XObluOpnR2zF0pQCtMy2mMMe1UTCaFnknBAWyBligoZ4wxHUhMJoXM5EyqA9XsKd8T7VCMMaZNic2kYAPYjGlx7e3+ZEd1tN9DbCaFsAFsxpij5/P52L17tyWGKFNVdu/ejc/na/Y+Ym6cAoTNwGYD2IxpETk5ORQXF7Nz585ohxLzfD4fOTk5zf58TCaF9E7pdPJ2su4jY1pIfHw8ubm50Q7DtICY7D4SERvAZowxEbiaFETkLBFZJSJrROSWCOuniMhOEVnivH7kZjzhMpMzLSkYY0wdriUFEfEC04GzgXxgsohEKkv6N1UtcF5PuhVPXTaAzRhjDufmlcIoYI2qrlPVSmAWMMHF4x2Rnkk92VG2A3/AH+1QjDGmzXAzKfQCNoUtFzttdX1HRD4VkZdEJPI0SS7ITM7Er352HdrVWoc0xpg2L9o3mv8N9FPVEcBcIGKFOhGZKiJFIlLUUo+82WOpxhhzODeTwmYg/C//HKctRFV3q2qFs/gkEHHuOlWdoaqFqlpYX5naI9UzyZlsx242G2NMiJtJ4RMgT0RyRSQBmATMDt9ARLLCFscDK12MpxYb1WyMMYdzbfCaqlaLyDTgTcALPK2qy0XkXqBIVWcDPxWR8UA1sAeY4lY8dXVO6ExiXKJ1HxljTBhXRzSr6uvA63Xa7gx7fytwq5sx1McGsBljzOGifaM5qjKTM637yBhjwsR8UrDuI2OM+UrMJ4Vdh3ZRHaiOdijGGNMmxHRS6JnUk4AG2Flm5X6NMQZiPCnYADZjjKkttpOCTctpjDG1xHZSsAFsxhhTS8wkhZkfb+SkB96h2h8ItaUkpJAcn2zdR8YY44iZpACwed8htpdU1GrLTLLJdowxpkbMJIWsNB8AW/cdqtVuM7AZY8xXYiYpZKcnArBlf3mtdpuBzRhjvhIzSaG+K4WeyT3ZfWg3Vf6qaIRljDFtSswkhVRfPKm+OLbU7T5KykRRu1owxhhiKCkAZKclHtZ91DPZJtsxxpgaMZUUstJ9bN1/+I1msFHNxhgDsZYU0hLZuq/OjeYkG8BmjDE1Yiop9Er3sftgJeVV/lBbUnwSqQmp1n1kjDHEWFLISgs+lro1wmOp1n1kjDGxlhTS6xnAlmQzsBljDMRYUshOq38Am3UfGWOMy0lBRM4SkVUiskZEbmlgu++IiIpIoZvxZDZQ6mJvxV4q/BWRPmaMMTHDtaQgIl5gOnA2kA9MFpH8CNulAtcAH7kVSw1fvJfuKQmHj1VICo5VsC4kY0ysc/NKYRSwRlXXqWolMAuYEGG7+4AHgfII61pcVlri4aOak22yHWOMAXeTQi9gU9hysdMWIiLHAb1V9TUX46glK80GsBljTH2idqNZRDzAb4AbmrDtVBEpEpGinTt3HtVxs9MPH8Bm3UfGGBPkZlLYDPQOW85x2mqkAsOA/4rIeuDrwOxIN5tVdYaqFqpqYUZGxlEFlZXmo6SimpLyr6qi+uJ8pHdKt+4jY0zMczMpfALkiUiuiCQAk4DZNStVdb+qdlfVfqraD/gQGK+qRS7GFJpXwQawGWPM4VxLCqpaDUwD3gRWAi+q6nIRuVdExrt13MZkOwPYNkcYwGZXCsaYWBfn5s5V9XXg9Tptd9az7Rg3Y6kRKnVR975Cck8W7VjUGiEYY0ybFVMjmgF6pHbCI0R8AulA5QHKqsqiFJkxxkRfzCWFOK+Hnp19bKnvCSSbgc0YE8NiLilAI2MV7L6CMSaGxWRSyE63Uc3GGBNJzCaFrfvLUdVQm3UfGWNMjCaFrDQfFdUB9hysDLUleBPo5utmVwrGmJjWpKQgIn9pSlt7Ud8MbD2Te9oANmNMTGvqlcLQ8AWnLPbxLR9O66gZwHbYfQWbgc0YE+MaTAoicquIlAAjROSA8yoBdgCvtEqELqgpdRHpZrN1HxljYlmDSUFVf6WqqcDDqtrZeaWqajdVvbWVYmxx3ZITSIjzRKx/VFpVSmllaZQiM8aY6Gpq99GrIpIMICLfE5HfiEhfF+NylYiQlearfwY2ewLJGBOjmpoU/giUicixBOc/WAs851pUrSArzRdxrmawsQrGmNjV1KRQrcGH+icAf1DV6QTnQ2i3stMSI3YfgSUFY0zsamqV1BIRuRX4PnCKM2tavHthuS87PZFtB8rxBxSvRwDISMpAEHss1RgTs5p6pXARUAFcpqrbCM6i9rBrUbWCrHQf/oCyo+Srq4V4TzwZiRn2WKoxJmY1KSk4ieAFIE1EzgXKVbVd31PITqt5LDXCADbrPjLGxKimjmi+EPgY+C5wIfCRiEx0MzC3ZTkD2CJVS7XuI2NMrGrqPYXbgRNUdQeAiGQAbwEvuRWY2+qdgS2pJws2L0BVEZFohGaMMVHT1HsKnpqE4Nh9BJ9tkzr74kjpFHf4XM3JmRyqPkRJVUmUIjPGmOhp6pXCHBF5E5jpLF9EnbmX25uaAWx1u496JgcHsG07uI3OCZ2jEZoxxkRNg0lBRAYAPVX1RhG5ADjZWfUBwRvP7VpWeoSxCklfjVUY2GVgNMIyxpioaawL6DHgAICq/lNVr1fV64GXnXUNEpGzRGSViKwRkVsirL9KRD4TkSUiskBE8ptzEs2VnXb4XM02gM0YE8saSwo9VfWzuo1OW7+GPuiU154OnA3kA5Mj/NL/q6oOV9UC4CHgN00NvCVkpSWyq7SCimp/qC0jMYM4iePTnZ+2ZijGGNMmNJYU0htYl9jIZ0cBa1R1napWArMIlskIUdUDYYvJgNKKauZV2BbWheT1eJk0eBKvrH2F19a91prhGGNM1DWWFIpE5Iq6jSLyI2BhI5/tBWwKWy522uru6/9EZC3BK4WfNrLPFvXVvAq1u5CuL7ye43ocx93/u5tVe1a1ZkjGGBNVjSWFa4Efish/ReQR5/UucDlwTUsEoKrTVbU/cDNwR6RtRGSqiBSJSNHOnTtb4rBAsFIqHD6ALd4TzyNjHqFzQmeumXcN+yv2t9gxjTGmLWtskp3tqvoN4B5gvfO6R1VPdEpfNGQz0DtsOcdpq88s4Lx64pihqoWqWpiRkdHIYZuuvrmaAbondueRMY+wvWw7N793M/6A/7BtjDGmo2lq7aN5qvp75/VOE/f9CZAnIrkikgBMAmaHbyAieWGL3wJWN3HfLSIxwUuXpPjDpuWsUdCjgFtH3cr7W95n+pLprRmaMcZERVMHrx0xVa0WkWnAm4AXeFpVl4vIvUCRqs4GponImUAVsBe41K146pOdnlhvUgD47sDvsnz3cp747AmGdhvKGX3PaMXojDGmdbmWFABU9XXqjHxW1TvD3rfIfYmjkZWWSPHesnrXiwi3fe02vtjzBbctuI2ZaTM5Jv2YVozQGGNaT7uuX9QSstN9DV4pAHTyduLR0x7FF+fjmnnXUFpZ2krRGWNM64r5pJCVlsiB8moOVlQ3uF1mcia/PvXXbCrZxG0LbiOggVaK0BhjWk/MJ4XseuZViOSEzBO4ofAG5m2axxOfPuF2aMYY0+piPinUPJa6ed/hj6VG8r0h3+Oc3HOYvmQ684vnuxmaMca0uphPCqErhUbuK9QQEe7+xt0M7DKQm+ffzMYDG90MzxhjWlXMJ4WenX2IwJYIA9jqkxiXyKOnPYogTHtnGst2LXMxQmOMaT0xnxTivR56pHZq8pVCjd6pvXl0zKPsr9jP5Ncmc+v8W63ctjGm3Yv5pADB+wqRSl00ZlTWKF47/zWuGH4F/1n/H859+Vx+v/j3lFXVP+7BGGPaMksKNG2sQn1SElL46XE/5d/n/5sz+pzBjE9n8K2Xv8U/V//T6iUZY9odSwpAdloiW/YfQrX50zlkp2Tz4OgHef6c5+mV0ou7/ncXF756IR9s+aAFIzXGGHdZUiA4V3N5VYB9ZVVHva9jM47lL2f/hYdPfZiDVQeZOncq096exrr961ogUmOMcZclBYJzNQNsacIAtqYQEc7qdxavnPcK1x1/HQu3L+SCVy5gzvo5LbJ/Y4xxiyUFglcKAFubOICtqTp5O3HZsMt47YLX6NO5D8+veL5F92+MMS3NkgItf6VQV1dfV8495lyW7lxqj60aY9o0SwpA95ROxHvlsLmaW9LYvmMBeHvj264dwxhjjpYlBcDjETLTfE0qitdcuWm5DEgfwH/W/8e1YxhjzNGypODISkts8XsKdY3rN47FOxazs2ynq8cxxpjmsqTgyE7zuXZPoca4vuNQlLc2vuXqcYwxprksKTiy0hPZtr8cf6D5A9ga0z+9P8ekHcPcDXNdO4YxxhwNSwqO7PREqgPKrtIKV48ztu9YFm5fyK5Du1w9jjHGNIclBUfosdRm1kBqqrF9xxLQAO9sfMfV4xhjTHO4mhRE5CwRWSUia0TklgjrrxeRFSLyqYi8LSJ93YynITUzsDWnWuqRGNhlIP069+M/G+wpJGNM2+NaUhARLzAdOBvIByaLSH6dzRYDhao6AngJeMiteBpTMwOb21cKIsLYvmMp2lbEnvI9rh7LGGOOlJtXCqOANaq6TlUrgVnAhPANVHWeqtZMPvAhkONiPA1KS4wnMd7r+pUCBLuQ/Opn3sZ5rh/LGGOOhJtJoRewKWy52Gmrz+XAGy7G0yAROapnsfiCAAAV6UlEQVR5FY7E4K6DyUnJsaeQjDFtTpu40Swi3wMKgYfrWT9VRIpEpGjnTvcGfmWnJx7RXM3NJSKM6zeOj7Z+xP6K/a4fzxhjmsrNpLAZ6B22nOO01SIiZwK3A+NVNeLzoKo6Q1ULVbUwIyPDlWABstJ8RzxXc3ON6zuOaq22p5CMMW2Km0nhEyBPRHJFJAGYBMwO30BERgKPE0wIO1yMpUmy0hLZWVpBZXXA9WPld8snOznbupCMMW2Ka0lBVauBacCbwErgRVVdLiL3ish4Z7OHgRTg7yKyRERm17O7VpGd7kMVth9onS6ksX3H8sHWDzhQecD14xljTFO4ek9BVV9X1YGq2l9V73fa7lTV2c77M1W1p6oWOK/xDe/RXTVjFVrjZjMEC+RVB6p5d9O7rXI8Y4xpTJu40dxWZKe3zgC2GsO7DyczOdPKaRtj2gxLCmFCA9hcrpZaQ0Q4s8+ZvL/lfUorS1vlmMYY0xBLCmGSEuJIS4x3fV6FcOP6jaMqUMW7xdaFZIyJPksKdWS5PANbXcdmHEuPxB72FJIxpk2wpFBHdnoim1vxSsEjHs7seyYLNi+grKqs8Q8YY4yLLCnUkZ3eulcKEKyFVOGv4L3i91r1uMYYU5clhTqy0hLZV1bFoUp/qx1zZI+RdPN1s3Laxpios6RQR2s/gQTg9XitC8kY0yZYUqgjNNlOK95XgGAtpEPVh3h/y/utelxjjAlnSaGO7FYe1VzjuJ7H0dXX1QayGWOiypJCHZlpPkRg097W7caJ88Rxep/Tebf4XcqrW/cqxRhjalhSqCMhzkNh3y68vHgz1X73q6WGG9t3rHUhGWOiypJCBFNH96d47yFe+2xrqx73hMwTSO+UbgPZjDFRExftANqiMwb3YECPFP707jrGH5uNiLTKceM98Zze53ReXv0y84vn4/P6SPAm4Itzfnpr/0yMS+TiIReT3y2/VeIzxnR8lhQi8HiEqaOP4aaXPmX+6l2MHujebG91TR0xlfRO6RyqPkSlv5Jyf3nwZ3Xw56HqQ+yr2EeFv4KdZTtZsHkBL377RXok9Wi1GI0xHZeoarRjOCKFhYVaVFTk+nEqqwOc8tA79M9I4a9XfN314zXHmr1ruPj1ixncdTBPffMp4j3x0Q7JGNNGichCVS1sbDu7p1CPhDgPl5+cy//W7ubT4n3RDieiAV0GcPeJd7N4x2IeXfhotMMxxnQAlhQaMHlUH1J9cTz+7rpoh1Kvc445h4sHX8xfVvyFOevnRDscY0w7Z0mhAam+eL739b68sWwr63cdjHY49fpZ4c84NuNY7nr/Ltbta7sJzBjT9llSaMQPv9GPOI+HJ+a33V+28d54fn3qr/HF+bjuv9dZ/SRjTLO5mhRE5CwRWSUia0TklgjrR4vIIhGpFpGJbsbSXD06+/jO8b34+8JidpZURDucemUmZ/LQ6IdYf2A9d/3vLtrbAwTGmLbBtaQgIl5gOnA2kA9MFpG6D9RvBKYAf3UrjpZwxSnHUOUP8Oz/1kc7lAZ9LetrXD3yauasn8MLK1+IdjjGmHbIzSuFUcAaVV2nqpXALGBC+Aaqul5VPwVat57EETomI4Vv5mfy3AfrOVhRHe1wGnT5sMs5rfdpPFL0CIt3LI52OMaYdsbNpNAL2BS2XOy0tUtXnnoMB8qrmfnxxmiH0iAR4f6T7yc7JZsb/nsDuw7tinZIxph2pF3caBaRqSJSJCJFO3fujEoMI/t04Wu5XXlqwZdUtXKhvCOVmpDKb8b8hpLKEm567yaqA2376sYY03a4mRQ2A73DlnOctiOmqjNUtVBVCzMyWq/kRF1XndqfrfvLmb1kS9RiaKpBXQdx54l38sm2T/jd4t9FOxxjTDvhZlL4BMgTkVwRSQAmAbNdPJ7rxgzKYFDPVB5/b227eLrn2/2/zYUDL+TPy/7MG1++wb7yfRyoPMDBqoOUV5dT5a/CH/C3i3MxxrQO1wriqWq1iEwD3gS8wNOqulxE7gWKVHW2iJwAvAx0Ab4tIveo6lC3YjpaIsKVpx7D9S8uZd6qHZw+uGe0Q2rUzaNuZsXuFdz03k0NbucVLx7x4BUv8d74UCXWTt5OdPJ2qvW+Zjk1IZX+6f0ZkD6AAekD6OLr0kpnZYxxixXEO0JV/gCnPjSPnK5JvHjliVGL40jsK9/H3I1zqfRXEtAA/oAfv/oJaIBqra7V5g/4qQpUUeGvoNJfWetn+KvSX8ne8r2UVJWEjtPV15W89LxgougSTBT90/vTOaFzFM/eGANNL4hnpbOPULzXw+WnHMN9r65g0ca9HNen7f91nO5L57sDv9vi+1VVdh7ayZq9a1izL/hau28t/1rzL8qqvxpV3SOpB3ld8hiYPjD4s8tActNySfAmtHhMxpijY1cKzXCwoppvPPAOXz+mK49/v9HEG3MCGmDbwW2hRLFm7xpW71vN2n1rqQpUARAncfRL6xdKEgO7DCQvPY/M5MxWm9TImFhiVwouSu4Uxw9O7Msf5q1h7c5S+mekRDukNsUjHrJTsslOyWZ0zuhQe1Wgio0HNrJ672q+2PsFX+z9gqU7lvLGl2+EtonzxBEncaH7GyISut9R86pp90jTn5MQDk80kZJPUlwSKQkppManBn8mpJKakEpKfAqdEzqH2hLjEiPus6nHjsTj8ZAUl0RyfDLJ8ckkxiUe0Tka0xLsSqGZdpVWcNID7/CtEVn84rxhJCVYfm2uksoS1uxbwxd7vmDLwS3BexzOPY+al1+DT0nVtPvV3+T9R/o3rkRoU+VQ9SFKKksoqSqhtLKUksoSSqtKj+r8mksQEuMSQ0kiKd5JGHHJeD3eJu3DIx7iPfGHPSwQ742v9dBAgieBeE88Xo+XOIkjzhMXfO+JwyveULL2erx4pWnHdkssX0n2SOxBui+9WZ+1KwWXdU/pxIWFvfnLhxv456LNdPbFkZnmIzMtkczOnchMSyQrzUdmZx+ZaT6y0nykJ1kfeiSpCamM7DGSkT1GRjuUiPwBPwerD4aSREllCYeqDzXps5GST32qA9WUVZdRVlXGwaqDoVdZ9VfLZVVlbCvb1uSkGAgEqAwEHxSo8n/1AEG12oDG9ujnX/85Fw660NVjWFI4Cj8/N5/Cfl3YvO8Q2/eXs3V/OdsPlPP51gPsLK2g7h+oJ/Trwg3jBvH1Y7pFJ2DTLF6Pl84JnTvUU1TVgWoq/ZXBV6CSiuoKqrWa6kB1rafQ/OoPtgX8ofUBjd6I/iNJsh3R4K6DXT+GJYWjkBDnYUJB5HJOVf4AO0sqQoli3c5SnvtgA5NmfMhJA7px/dhBHN+37T+5ZDqmOE+wiygpPinaoZg2xu4ptKLyKj/Pf7iBP727ll2llYwZlMENYwcxPCct2qEZYzq4pt5TsKQQBWWV1Tz7vw08/t5a9pVVMS6/J9eNHciQrI7TPWGMaVssKbQDJeVVPL1gPU/OX0dJRTXfGpHFdWfmMaBHarRDM8Z0MJYU2pF9ZZU8MX8df35/PeVVfs4fmcPt3xpC12R7WskY0zKamhRsZEwbkJ6UwI3fHMz8m07j8pNzmb10M2N/8y5vfLY12qEZY2KMJYU2pFtKJ27/Vj6zp51MZpqPH7+wiKtnLmbPwcpoh2aMiRGWFNqgIVmd+df/ncT1YwcyZ9lWxj36LnOW2VWDMcZ9lhTaqHivh5+ekcfsaSfTs7OPq563qwZjjPssKbRxka8atkU7LGNMB2VJoR0Iv2rokerjqucXcvXMxey1qwZjTAuzMhftyJCszrwy7ST+37y1/P6d1Xywdhfnjsgmp0sivbsmkdMlkZwuSaQlxkc7VGNMO2VJoZ2J93q45sw8xub35J5/L+fvRZs4WFm7YmaqL46cLsEk0dv52b9HCl/L7YovPrplj40xbZslhXYqP7szf7vyRFSV/Yeq2LTnEMV7yyje+9XPDbsP8v6aXZQ5SSMpwcvovAzG5vfk9ME96GKD44wxdVhSaOdEhPSkBNKTEiIW1lNV9pZV8WnxPt5auZ25K7YzZ/k2vB7hhH5dGJufybj8nvTuatUyjTEul7kQkbOA3wJe4ElVfaDO+k7Ac8DxwG7gIlVd39A+O2KZi9YUCCifbd7P3BXb+c+KbXyxPTir2ODMVMbm92Rsfk+GZafh8cTu7FbGdERRr30kIl7gC2AsUAx8AkxW1RVh2/wEGKGqV4nIJOB8Vb2oof1aUmhZG3YfdBLEdorW7yGgwW6mQZmpDMnqzJCszuRnpTIoszMpnezC0pj2qi0khROBu1X1m87yrQCq+quwbd50tvlAROKAbUCGNhCUJQX37DlYyX9X7eDT4v2s3HqAlVsPcKD8q2kb+3ZLYkhmZydZpJLbPbnWFUXda4vwuXTjPILXI2E/PXi9weWatliee9cYt7WFOZp7AZvClouBr9W3japWi8h+oBuwy8W4TD26JidwwXE5XHBcDhC8H7FlfzkrthwIJYmVWw/w5opth0012hK8TnLwiuAR8IQtiwheD8F1HsHjbFNXfYklYmukz9cTW939xnmEOdeObvB8jGmP2kV/gIhMBaYC9OnTJ8rRxA4RoVd6Ir3SExmb3zPUfrCims+3lVC8t6zez4YnDUXxB8AfCFAdUPwBpcqvXy37leqAUu0sq4Lf2S6gwZc/ELwf4neWA4HDZ+utL1FFao50MVpvnouwwu65mI7KzaSwGegdtpzjtEXaptjpPkojeMO5FlWdAcyAYPeRK9GaJkvuFMfxfbvYHNPGdEBulrn4BMgTkVwRSQAmAbPrbDMbuNR5PxF4p6H7CcYYY9zl2pWCc49gGvAmwUdSn1bV5SJyL1CkqrOBp4C/iMgaYA/BxGGMMSZKXL2noKqvA6/Xabsz7H058F03YzDGGNN0ViXVGGNMiCUFY4wxIZYUjDHGhFhSMMYYE2JJwRhjTIirVVLdICI7gQ3N/Hh3Ol4JjY52Th3tfKDjnVNHOx/oeOcU6Xz6qmpGYx9sd0nhaIhIUVMKQrUnHe2cOtr5QMc7p452PtDxzulozse6j4wxxoRYUjDGGBMSa0lhRrQDcEFHO6eOdj7Q8c6po50PdLxzavb5xNQ9BWOMMQ2LtSsFY4wxDYiZpCAiZ4nIKhFZIyK3RDueoyUi60XkMxFZIiLtcn5SEXlaRHaIyLKwtq4iMldEVjs/282kDfWcz90istn5npaIyDnRjPFIiUhvEZknIitEZLmIXOO0t8vvqYHzabffk4j4RORjEVnqnNM9TnuuiHzk/M77mzOFQeP7i4XuIxHxAl8AYwlOC/oJMFlVV0Q1sKMgIuuBQlVtt89Wi8hooBR4TlWHOW0PAXtU9QEneXdR1ZujGWdT1XM+dwOlqvrraMbWXCKSBWSp6iIRSQUWAucBU2iH31MD53Mh7fR7kuBcscmqWioi8cAC4BrgeuCfqjpLRP4ELFXVPza2v1i5UhgFrFHVdapaCcwCJkQ5ppinqu8RnEcj3ATgWef9swT/h20X6jmfdk1Vt6rqIud9CbCS4Nzq7fJ7auB82i0NKnUW452XAqcDLzntTf6OYiUp9AI2hS0X087/IRD80v8jIgudOaw7ip6qutV5vw3o2dDG7cQ0EfnU6V5qF90skYhIP2Ak8BEd4Huqcz7Qjr8nEfGKyBJgBzAXWAvsU9VqZ5Mm/86LlaTQEZ2sqscBZwP/53RddCjO1KztvX/zj0B/oADYCjwS3XCaR0RSgH8A16rqgfB17fF7inA+7fp7UlW/qhYAOQR7RgY3d1+xkhQ2A73DlnOctnZLVTc7P3cALxP8h9ARbHf6fWv6f3dEOZ6joqrbnf9hA8ATtMPvyemn/gfwgqr+02lut99TpPPpCN8TgKruA+YBJwLpIlIzu2aTf+fFSlL4BMhz7sYnEJwLenaUY2o2EUl2bpIhIsnAOGBZw59qN2YDlzrvLwVeiWIsR63mF6fjfNrZ9+TcxHwKWKmqvwlb1S6/p/rOpz1/TyKSISLpzvtEgg/UrCSYHCY6mzX5O4qJp48AnEfMHgO8wNOqen+UQ2o2ETmG4NUBBOfZ/mt7PB8RmQmMIVjRcTtwF/Av4EWgD8FquBeqaru4eVvP+Ywh2CWhwHrgyrC++DZPRE4G5gOfAQGn+TaC/fDt7ntq4Hwm006/JxEZQfBGspfgH/ovquq9zu+JWUBXYDHwPVWtaHR/sZIUjDHGNC5Wuo+MMcY0gSUFY4wxIZYUjDHGhFhSMMYYE2JJwRhjTIglBdNuiUhPEfmriKxzyn18ICLnH+U+7xaRnznv7xWRM5u5n4L6Km2KSJKIvOBUuV0mIgtEJEVE0kXkJ0cTvzFHy5KCaZecQUj/At5T1WNU9XiCgxJzImwbV7etKVT1TlV9q5khFgD1lV++BtiuqsOdaqqXA1VAOmBJwUSVJQXTXp0OVKrqn2oaVHWDqv4eQESmiMhsEXkHeNv5S/xtEVnk/IUeqpIrIreLyBcisgAYFNb+jIhMdN4fLyLvOlckb4aVePiviDzo1LP/QkROcUbN3wtc5NTmv6hO7FmElRxQ1VXOoKIHgP7OZx529n+jiHziFGqrqZPfT0Q+d642VorISyKS5Kx7QIJzBXwqIu2uDLSJvmb9BWVMGzAUWNTINscBI1R1j3O1cL6qHhCR7sCHIjLb2WYSwb/s45x9LgzfiVMr5/fABFXd6fySvx+4zNkkTlVHOd1Fd6nqmSJyJ8H5LqZFiOtpghVuJwJvA8+q6mrgFmCYU9gMERkH5BGswyPAbKfw4UaCyetyVX1fRJ4GfiIifyZYomGwqmpN6QNjjoQlBdMhiMh04GSCVw8nOM1zw0ovCPBL55dqgGAZ4Z7AKcDLqlrm7CdSTaxBwDBgbrDXCi/BSpo1aorELQT6NRarqi5xShCMA84EPhGRE4FDdTYd57wWO8spBJPERmCTqr7vtD8P/JRgGZdy4CkReRV4tbFYjKnLkoJpr5YD36lZUNX/c64AwqcmPRj2/hIgAzheVaskOHOdr4nHEmC5qp5Yz/qaejJ+mvj/lDMpyj+Bf4pIgOD9h39EOO6vVPXxWo3BeQDq1qdRVa0WkVHAGQQLoU0j2M1mTJPZPQXTXr0D+ETkx2FtSQ1snwbscBLCaUBfp/094DwRSXQqz347wmdXARnOX/OISLyIDG0kvhIgNdIKETlJnElcnPsP+QSLytX9zJvAZRKs/Y+I9BKRHs66PjXxABcDC5zt0lT1deA64NhGYjTmMJYUTLvkTOxyHnCqiHwpIh8TrBRZ3zzBLwCFIvIZ8APgc2c/i4C/AUuBNwiWWa97rEqCf3k/KCJLgSXANxoJcR6QX8+N5v7Au04siwle3fxDVXcD7zuPqT6sqv8B/gp84Gz7El8ljVUEJ1daCXQhOElMKvCqiHxKcJ7e6xuJ0ZjDWJVUY9oZp/voVedxVmNalF0pGGOMCbErBWOMMSF2pWCMMSbEkoIxxpgQSwrGGGNCLCkYY4wJsaRgjDEmxJKCMcaYkP8PkkkFTWmoSgUAAAAASUVORK5CYII=\n",
"text/plain": [
"