{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Multivariate Linear Regression Demo\n", "\n", "_Source: 🤖[Homemade Machine Learning](https://github.com/trekhleb/homemade-machine-learning) repository_\n", "\n", "> ☝Before moving on with this demo you might want to take a look at:\n", "> - 📗[Math behind the Linear Regression](https://github.com/trekhleb/homemade-machine-learning/tree/master/homemade/linear_regression)\n", "> - ⚙️[Linear Regression Source Code](https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/linear_regression/linear_regression.py)\n", "\n", "**Linear regression** is a linear model, e.g. a model that assumes a linear relationship between the input variables `(x)` and the single output variable `(y)`. More specifically, that output variable `(y)` can be calculated from a linear combination of the input variables `(x)`.\n", "\n", "**Multivariate Linear Regression** is a linear regression that has _more than one_ input parameter and one output label.\n", "\n", "> **Demo Project:** In this demo we will build a model that will predict `Happiness.Score` for the countries based on `Economy.GDP.per.Capita` and `Freedom` parameters." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# To make debugging of linear_regression module easier we enable imported modules autoreloading feature.\n", "# By doing this you may change the code of linear_regression library and all these changes will be available here.\n", "%load_ext autoreload\n", "%autoreload 2\n", "\n", "# Add project root folder to module loading paths.\n", "import sys\n", "sys.path.append('../..')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Import Dependencies\n", "\n", "- [pandas](https://pandas.pydata.org/) - library that we will use for loading and displaying the data in a table\n", "- [numpy](http://www.numpy.org/) - library that we will use for linear algebra operations\n", "- [matplotlib](https://matplotlib.org/) - library that we will use for plotting the data\n", "- [plotly](https://plot.ly/python/) - library that we will use for plotting interactive 3D scatters\n", "- [linear_regression](https://github.com/trekhleb/homemade-machine-learning/blob/master/src/linear_regression/linear_regression.py) - custom implementation of linear regression" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/vnd.plotly.v1+html": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Import 3rd party dependencies.\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import plotly\n", "import plotly.graph_objs as go\n", "\n", "# Configure Plotly to be rendered inline in the notebook.\n", "plotly.offline.init_notebook_mode()\n", "\n", "# Import custom linear regression implementation.\n", "from homemade.linear_regression import LinearRegression" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load the Data\n", "\n", "In this demo we will use [World Happindes Dataset](https://www.kaggle.com/unsdsn/world-happiness#2017.csv) for 2017." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CountryHappiness.RankHappiness.ScoreWhisker.highWhisker.lowEconomy..GDP.per.Capita.FamilyHealth..Life.Expectancy.FreedomGenerosityTrust..Government.Corruption.Dystopia.Residual
0Norway17.5377.5944457.4795561.6164631.5335240.7966670.6354230.3620120.3159642.277027
1Denmark27.5227.5817287.4622721.4823831.5511220.7925660.6260070.3552800.4007702.313707
2Iceland37.5047.6220307.3859701.4806331.6105740.8335520.6271630.4755400.1535272.322715
3Switzerland47.4947.5617727.4262271.5649801.5169120.8581310.6200710.2905490.3670072.276716
4Finland57.4697.5275427.4104581.4435721.5402470.8091580.6179510.2454830.3826122.430182
5Netherlands67.3777.4274267.3265741.5039451.4289390.8106960.5853840.4704900.2826622.294804
6Canada77.3167.3844037.2475971.4792041.4813490.8345580.6111010.4355400.2873722.187264
7New Zealand87.3147.3795107.2484901.4057061.5481950.8167600.6140620.5000050.3828172.046456
8Sweden97.2847.3440957.2239051.4943871.4781620.8308750.6129240.3853990.3843992.097538
9Australia107.2847.3566517.2113491.4844151.5100420.8438870.6016070.4776990.3011842.065211
\n", "
" ], "text/plain": [ " Country Happiness.Rank Happiness.Score Whisker.high Whisker.low \\\n", "0 Norway 1 7.537 7.594445 7.479556 \n", "1 Denmark 2 7.522 7.581728 7.462272 \n", "2 Iceland 3 7.504 7.622030 7.385970 \n", "3 Switzerland 4 7.494 7.561772 7.426227 \n", "4 Finland 5 7.469 7.527542 7.410458 \n", "5 Netherlands 6 7.377 7.427426 7.326574 \n", "6 Canada 7 7.316 7.384403 7.247597 \n", "7 New Zealand 8 7.314 7.379510 7.248490 \n", "8 Sweden 9 7.284 7.344095 7.223905 \n", "9 Australia 10 7.284 7.356651 7.211349 \n", "\n", " Economy..GDP.per.Capita. Family Health..Life.Expectancy. Freedom \\\n", "0 1.616463 1.533524 0.796667 0.635423 \n", "1 1.482383 1.551122 0.792566 0.626007 \n", "2 1.480633 1.610574 0.833552 0.627163 \n", "3 1.564980 1.516912 0.858131 0.620071 \n", "4 1.443572 1.540247 0.809158 0.617951 \n", "5 1.503945 1.428939 0.810696 0.585384 \n", "6 1.479204 1.481349 0.834558 0.611101 \n", "7 1.405706 1.548195 0.816760 0.614062 \n", "8 1.494387 1.478162 0.830875 0.612924 \n", "9 1.484415 1.510042 0.843887 0.601607 \n", "\n", " Generosity Trust..Government.Corruption. Dystopia.Residual \n", "0 0.362012 0.315964 2.277027 \n", "1 0.355280 0.400770 2.313707 \n", "2 0.475540 0.153527 2.322715 \n", "3 0.290549 0.367007 2.276716 \n", "4 0.245483 0.382612 2.430182 \n", "5 0.470490 0.282662 2.294804 \n", "6 0.435540 0.287372 2.187264 \n", "7 0.500005 0.382817 2.046456 \n", "8 0.385399 0.384399 2.097538 \n", "9 0.477699 0.301184 2.065211 " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Load the data.\n", "data = pd.read_csv('../../data/world-happiness-report-2017.csv')\n", "\n", "# Print the data table.\n", "data.head(10)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl8AAAJOCAYAAACJNWIjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzs3XmYHFW9//H3hyQssoWQiCEhDIob4GUxIly3yKLIYnAHEYgX4aLyEwQVxA0ENXjdcDcCEgRZZBEEF9bIRSGQYBAheFkMEghZgECiiCzf3x/ntKk03TM9M93V3TOf1/PMM9W19beq61SdOufUKUUEZmZmZlaONdodgJmZmdlw4syXmZmZWYmc+TIzMzMrkTNfZmZmZiVy5svMzMysRM58mZmZmZXIma+SSTpe0mntjqNI0h2SptSZNkXSwiZ9zyxJH2rGusyGM0nXS3pXu+OwoUHSXyS9IQ+fLOnMNoc05A2pzJekBZKelLRC0nJJf5B0uKQBb6ekHkkhaWQzYoyIL0fEgDIghe1bKelhSWdKWq8JMW0dEbMGux4bvKrfuPL33XbH1WqSNpV0uqSH8jbfJ+kMSS/P07fM6XBl4fj/paRdq9azsCqNnCFp3TZt0xhJ38sxrZR0t6SvSho92HVHxBsj4qL8PUdI+s3gI7ZOVOecsGkzvyMiXh4R/9vMdVrvhlTmK9snItYHNgemA8cCp7c3pKbaJyLWA7YDtgc+3eZ4rPn2iYj1Cn9HtDugVpI0DrgJWBN4PbA+8Grg98BuxXkr+4R07F8LXCbpA1WrfFue5zXAzrQ4jdS6MZP0AuB3wCRgF9I2vQF4mpR2zfqj+pzwULsDssEZipkvACLi8Yi4DHgfcLCkN0haLGlEZR5J75R0Wx7eUdIcSU/k+b6RZ7s+/1+e7zh2lrSGpM9Kul/SEklnSdowr6dSUnZYvotfJOkThe88QdLZhc8/z3foj+eqhK0b3L6Hgd9SOJFLWkvS1yT9LW/DDyWtk6eNlXR5LhF8VNL/VkoE853Vbnl4nVyi9pikO0kXMArfEZK2LHw+U9LJeXij/B1L8/KXS5rYyPZY3yQdKml+Ltm9U9IOefwrlap0lytVIb+9sMyZufTlirzcbEkvKUz/T0m35OPvFkn/WZg2S6kK4g/52P+lpI0lnZPTyS2SevK835P09ap4L5P08QY27RhgKXBQRNwXyWMRcXpEfK/WAhGxKCK+CZwEfFWSaszzAPAbYJsa+3LtfCwfkY//pZK+VFyPpP9Wqo55NO+/CVXLfljSvcCfa4R4GLAB8O6I+L+8TQ9HxGcqpcySTszfvULS7ZLeVvjuIyRdJem0vK/vkPT6wvQ5kvaT9Brg68Bu+TdamKe/S9Kf8rrvl3RsH7+BdZF8DbowXzuW57T6ysL0syV9R9Jv83FxvaRN8rjl+TyybWH+harR9CQv/+GqcXdK2qelGzgMDNnMV0VE3AwsJJ2AHwHeUph8IHBWHj4VODUiNgBeAlyQx78x/x+d7zhuBKblvzcDLwbWA6qrht4MvDR/37GVzE0Nv87zvRC4FTinke3KmZq3AfcURk8HXkbKkG0JTAA+n6cdQ9oP44BNgOOBWu+W+gJp+18CvBU4uJF4sjWAn5BKHScBT/L8/WIDIOk9wAnAQaSL+tuBRySNAn4JXEk6hv4fcI5ydV22H3AisBHpePlSXucY4Arg28DGwDeAKyRtXLXsgaRj6SXAjaTfeAwwn3S8AMwE9i9k6MeSSq1+1sDm7QZcEgN719nFwHjS8b4aSZNIaeSPvSy/Dym97AjsDxyQl30fcFSevklex9lVy+5NKqHbvsZ6dwMuj4inevnu+cBOwGjSvj8v/yYVbwZuJv02XwcuUVUzg4i4hZS2r87np8rNzhOkG88NgXcBx/VyDrLudDnp2vEi0g3AT6umvw84DhhLOtffREq/GwOXAl9r4DtmAv8uWZb06ry+Xw8ydouIIfMHLAB2qzH+JuAzpCrIc/K4McA/gPH58/WkC9TYqmV7SAfuyMK4a4CPFD6/nFSdMLIw/ysK078KnJ6HTwDOrhP/6Lzshr1s30pgRZ7vGlKmEEDA34GXFObfGfhrHv4iKcFt2dt+A+4D9ihMOwxYWPgcxXUAZwIn14l3O+CxwudZwIfafZx08l/hN15e+DuUVMp5ZI353wA8DKxRGHcucELh9zmtMG1P4K48fCBwc9X6bgSmFX6vzxSmfR34deHzPsC8wuf5wO55+AjgV/3Y5g8VPr8zb/eKyjpImauosex6+Zh8bf68sLD/7idl/teusdzaebkphXFHA1fk4euAAwrTRpHS+CaFZf+zl226EfhsP3/7e4BdC/vvnqrpdwLvyMNzgP0K8/6mj3WfBpzU7uPbf/3/q3FO+EWNeSoZrHXz57OBHxSmfxy4vfB5e2BZ4fPCSloATgbOzMPr5O/cIn/+FvDtdu+TofA35Eu+sgnAo6QDch+lBrjvBf43IhbleQ4hlRrdlatT9u5lfZuSTuwV95MyXpsUxj1QNf15DSQljZA0XdK9kp4gJTJICamefSO1aZsCvKIw7zjgBcDcXKy8nFTlMi5P/x/Syf1KpcbMx/WybdWxN0TSCyT9KFdzPEHK0I5WoarXGrJvRIwu/P0Y2Ay4t8a8mwIPRMRzhXH3k475iocLw/8gZVgqy1b/vtXLLi4MP1njc7EkpniX/AGefydezyOk0isAIuLiiBgNfJLUDqw3lVgfLYzbO++3zSPiiIj4Zy/L10unmwM/LKSlpcAzwMQ6y1ZbbZtqUWqacHvhO3pYPe1Xr7/meaTOut+Yq5qWSXqc9Hv0dl6xzlY8J+ybrx1fzefyJ1hVA1L8jfuTdmuKiCeBC4EP5PP4fjSerq0XQz7zldtETABuiIgHSXek7yTd9f/7IIqIuyNif1LVzSnAhTmTVqsq5CHSybliEunEXDy4N6uaXquB5PuBqaQqig1JJ19IpVi9iojfkUo1KkXHy0gJautCIt0wUsNjImJFRBwTES8mVVkdraonxbJFNWIv+gcpk1fxosLwMaRSwNdGqr6tVNn2uT3WpwdI1X7VHgI20+pP9E4CHmxgndXHcX+WreVsYGpuS/JK4BcNLncN8I5a7bYa8A5S5vKevmaso146fYBUAljMBK8TEXML8/dWTXo1sLektWpNlLQVqarxEGBMzmwuYPW0Ut1est55pFYcPyf9HhMiYsM87HQ4dBxEKsXehXTtqFS7t+I3nkmqjn8LqSbjlhZ8x7AzZDNfkjbIpVfnkar5bs+TzgI+BbyK1F6kMv8HJI3LJQjL8+jnSHe8z5HadlWcC3xc0ha5DcaXgfMj4pnCPJ/LJUFbAx8Ezq8R5vrAU6S75Bfk9fTHt4DdJW2b4/4x8E1JL8zbNEHSW/Pw3kqP6wt4HHg2b1e1C4BPKzWen0hqQ1Q0D3h/vvPaA3hT1fY8SXo4YQyr2gPZ4J0GfELSq5VsKWlzYDYpQ/wpSaNyo9l9SMd9X34FvEzS+yWNzO2ctiK1Jem3iFgI3EK6qbko3zU34mukm56zJL04b98GwLb1FsiNhz8GfBY4NnKdyAAcK2lDpQcHjmBVOv0h8Fmt6upiI/WvX60ZpGrT8yvpTtILJX0x/0brser8soakI1h181WxhaQP5d/mYFLJ+tU1vmsxMEn5qcucEV+XdF75l1JDffcJNrRUXzu+1MLvuoFUAn0KLvVqmqGY+fqlpBWkO9fPkO4uP1iYfgnpbv+SiPhHYfwewB2SVpIa3+8XEU/meb4E/D5XD+wEnEE6CK8H/gr8k+dnUn5Huhu/BvhaRFxZI9azSFUJD5Lac9xUnKjUIWvdho0RsTSvo9Ko/tj8nTflouirSSVRkBpmXk1qO3Aj8P2IuK7Gak/MMf2V1Ii7OrEdSbq4LyfdDRVLN75FaiOwLG+L+x4amF9q9T59LomIn5OOw5+RLuq/IJWY/Iv0e7yNtN+/T3pq8K6+viQiHiE1Gj+GdBL/FKnKbtkgYp9JurFZ7biRdKWkT+XhEXm7ds5xLCE1PH8G+EPevltJbas+WrWelTmN/on0QMg7I+IsGqD05Oe3qkZfAdxGakNVKS0iIs4ltRe7OKelecDufax/gaR98/L/IJX8PkQ6F6wgdZ2xJvDHSA8CnU5qyP8QqTpxXtUqryPtl0dJafudEbGixlf/inQOWSrp/nwjdjjpPPY4qS3bhYU41837sdaDAtYdfkI6bh4C7iClm5bINzZnkR5aa+iBMOubBn7D2L2UHg//74iodRc52HX3kDIuo6pKwsyGPElvJGVgNh9EaVTLSVqbVEq7WS6x6yi5JGzviNij3bGYSfov0k3dlHbHMlQMxZKvXuWqgyB10GhmTaLU7cWRpKcrOzbjZWaNy22fP0KqSrcmGVaZL0mzgB8AH616OszMBkGpg8flpCf8qqv2zKwLSdoLWAL8jdrtlm2AhmW1o5mZmVm7DKuSL7NWUXrlzM2SblN6FcyJefwWSq/0uUfS+ZL66rfKzMyGuFJLvsaOHRs9PT2lfZ8NX3Pnzl0WEeP6nrM5chce60bEytz26QZS+6ejgYsj4jxJPwRui4gf1FuP04iVpew00kxOJ1aWVqWTkc1eYW96enqYM2dOmV9pw5Skhnvmb4bcwHxl/jgq/wWpE8T35/EzSa+Xqpv5chqxspSdRprJ6cTK0qp00nC1Y+6b54+SLs+fXZ1iVpDTyDxSA9WrSK8DWl7ocmQhq7+6p7LcYZLmSJqzdOnS8gI2M7O26E+bryNJL86tOAX4ZkRsCTxGek2G2bAVEc9GxHak18LsSHr3ZiPLzYiIyRExedy4rqwFMjOzfmgo85VfM7MX6RUnlfYtu7Cq1+SZwL6tCNCs20TEclLv5DuTXixeqd6fyMDfm2hmZkNEo22+vkV69cj6+fPGNFCdAqlKBTgMYNKk6nc0Dx89x10xoOUWTN+ryZFYK0gaBzwdEcslrUN6Fc0ppEzYu0nvWjwYuLR9UVqzOD2b9c3ppL4+S76UXk69JCLmDuQLXKViw8R44DpJfyK9XPqqiLic9E6+oyXdQ7ppOb2NMZqZWQdopOTrdcDbJe1JetHtBqQXto6WNDKXfrk6xYa1iPgT8LwXFUfEfaT2X2ZmZkADJV8R8emImBgRPcB+wLURcQCrqlPA1SlmZmZmDRlMD/euTjEzMzPrp351shoRs4BZedjVKWZmZmb95Hc7mpmZmZXImS8zMzOzEpX6bkfrP/eTYtY3pxMz6yYu+TIzMzMrkTNfZmZmZiVy5svMzMysRM58mZmZmZXImS8zMzOzEjnzZWZmZlYiZ77MzKw0kkZI+qOky/PnLSTNlnSPpPMlrdnuGM1azZkvMzMr05HA/MLnU4BvRsSWwGPAIW2JyqxEznyZmVkpJE0E9gJOy58F7AJcmGeZCezbnujMyuPMl5mZleVbwKeA5/LnjYHlEfFM/rwQmFBrQUmHSZojac7SpUtbH6lZC/n1QmY2bA30tUTWf5L2BpZExFxJU/q7fETMAGYATJ48OZocnlmpnPkyM7MyvA54u6Q9gbWBDYBTgdGSRubSr4nAg22M0awUrnY0M7OWi4hPR8TEiOgB9gOujYgDgOuAd+fZDgYubVOIZqVx5svMzNrpWOBoSfeQ2oCd3uZ4zFrO1Y5mZlaqiJgFzMrD9wE7tjMes7L1WfIlaW1JN0u6TdIdkk7M490xnpmZmVk/NVLt+BSwS0RsC2wH7CFpJ9wxnpmZmVm/9Zn5imRl/jgq/wXuGM/MzMys3xpqcJ/fxTUPWAJcBdyLO8YzMzMz67eGMl8R8WxEbEfqg2VH4BWNfkFEzIiIyRExedy4cQMM08zMzGxo6FdXExGxnNQny87kjvHyJHeMZ2ZmZtaARp52HCdpdB5eB9id9EZ6d4xnBkjaTNJ1ku7MTwQfmcePkXSVpLvz/43aHauZmbVfI/18jQdmShpByqxdEBGXS7oTOE/SycAfccd4Nnw9AxwTEbdKWh+YK+kqYBpwTURMl3QccBypQ0kzM6tjoO9cXTB9ryZH0jp9Zr4i4k/A9jXGu2M8MyAiFgGL8vAKSfNJD6BMBabk2WaSOpV05svMbJjz64XMmkhSD+lmZTawSc6YATwMbFJnGT8RbGY2jDjzZdYkktYDLgKOiognitMiIkj94z2Pnwg2Mxte/G7HIWqgdebQXfXmnULSKFLG65yIuDiPXixpfEQskjSe1E+emZkNcy75MhskSSI9cDI/Ir5RmHQZ6Ulg8BPBZmaWueTLbPBeBxwI3J7fBAFwPDAduEDSIcD9wHvbFJ+ZmXUQZ77MBikibgBUZ/KuZcZiZmadz9WOZmZmZiVy5svMzMysRK52NLOOMZindM3MuoVLvszMzMxK5MyXmZmZWYmc+TIzMzMrkTNfZmZmZiVy5svMzMysRM58mZmZmZXImS8zMzOzErmfLzMzM+t6A+0ncMH0vZocSd/6LPmStJmk6yTdKekOSUfm8WMkXSXp7vx/o9aHa2ZmZtbdGin5egY4JiJulbQ+MFfSVcA04JqImC7pOOA44NjWhWpm3cI91ZuZ1ddnyVdELIqIW/PwCmA+MAGYCszMs80E9m1VkGZmZmZDRb8a3EvqAbYHZgObRMSiPOlhYJM6yxwmaY6kOUuXLh1EqGZmZmbdr+HMl6T1gIuAoyLiieK0iAggai0XETMiYnJETB43btyggjUzMzPrdg1lviSNImW8zomIi/PoxZLG5+njgSWtCdHMzMxs6GjkaUcBpwPzI+IbhUmXAQfn4YOBS5sfnpmZmdnQ0sjTjq8DDgRulzQvjzsemA5cIOkQ4H7gva0J0czMzGzo6DPzFRE3AKozedfmhmNmZmY2tLmHezOry/11WbNI2gw4i/RkfAAzIuJUSWOA84EeYAHw3oh4rF1xmpXB73Y0M7MyVDrs3grYCfiopK1IHXRfExEvBa7Jn82GNGe+zMys5dxht9kqznyZmVmp3GG3DXfOfJmZWWncYbeZM19mZlYSd9htljjzZWZmLecOu81WcVcTZkOcu4uwDuEOu80yZ77MmkDSGcDewJKI2CaPc/9FZpk77DZbxdWOZs1xJrBH1Tj3X2RmZs/jki+zJoiI6/Pj80VTgSl5eCYwCzi2tKDMzAbJzRZawyVfZq3j/ovMzOx5nPkyK4H7LzIzswpXO9rzDLSYecH0vZocSddbLGl8RCxy/0VmZlbhzJdZ61T6L5qO+y8yfGNjZomrHc2aQNK5wI3AyyUtzH0WTQd2l3Q3sFv+bGZmw5xLvsyaICL2rzPJ/ReZmdlqXPJlZmZmVqI+S77cc/fq3OeJmZmZDUYjJV9n4p67zczMzJqiz8xXRFwPPFo1eiqpx27y/32bHJeZmZnZkDTQBvcN9dwNqfdu4DCASZMmDfDrrBv4MXozs87kJjOdZdAN7nvruTtPd+/dZmZmZtlAM1+Lc4/duOduMzMzs8YNNPNV6bkb3HO3mZmZWcMa6WriXGAKMFbSQuALpJ66L8i9eN8PvLeVQZqZ22yYmQ0VfWa+hmrP3b6QmZmZWTu4h3szMzOzEjnzZWZmZlYiZ77MzMzMSjTQTlbNzMysZG6vPDS45MvMzMysRM58mZmZmZXImS8zMzOzEnVMmy+/lNnMzIYLt90a3lzyZWZmZlaijin5MjOz2lwzYDa0uOTLzMzMrETOfJmZmZmVyNWO1nauUjEzs+HEJV9mZmZmJXLJl5mZ2QC5ywgbCJd8mZmZmZXImS8zMzOzEg2q2lHSHsCpwAjgtIiY3pSo+sFFvtbpOiGdmHUypxEbbgac+ZI0AvgesDuwELhF0mURcWezgjPrdk4nZr1rdhrx09PWDQZT7bgjcE9E3BcR/wLOA6Y2JyyzIcPpxKx3TiM27Aym2nEC8EDh80LgtdUzSToMOCx/XCnpL4P4zlYaCyxrdxBNMpS2Bepsj07pdZnNWxVMP/WZTvqRRjr1d3VcjSs1pqGSRqD16aSPfdVKnXic1tNNsUKD8bYjnbS8q4mImAHMaPX3DJakORExud1xNMNQ2hYYettTrdE00qn7wXE1rhNj6hbdnk7q6aZ4uylW6Ox4B1Pt+CCwWeHzxDzOzFZxOjHrndOIDTuDyXzdArxU0haS1gT2Ay5rTlhmQ4bTiVnvnEZs2BlwtWNEPCPpCOC3pMeDz4iIO5oWWfk6vmq0H4bStkAXb0+T00mn7gfH1bhOjKmtWnAt6bZ93E3xdlOs0MHxKiLaHYOZmZnZsOEe7s3MzMxK5MyXmZmZWYmGfeZL0hmSlkj6c7tjGSxJm0m6TtKdku6QdGS7YxooSWtLulnSbXlbTmx3TGWRtIekv0i6R9JxNaavJen8PH22pJ4OiWuapKWS5uW/D5UQU6/pV8m3c8x/krRDB8Q0RdLjhf30+VbHNBR1ajqppRPTTm86MV3V07XpLSKG9R/wRmAH4M/tjqUJ2zIe2CEPrw/8H7BVu+Ma4LYIWC8PjwJmAzu1O64StnsEcC/wYmBN4Lbq3xD4CPDDPLwfcH6HxDUN+G7J+6vX9AvsCfw6H087AbM7IKYpwOXtPta6+a9T08kgYi097fQRc8elq0HE2pHpbdiXfEXE9cCj7Y6jGSJiUUTcmodXAPNJvUd3nUhW5o+j8t9weDqkkVetTAVm5uELgV0lqQPiKl0D6XcqcFY+nm4CRksa3+aYbPA6NZ3U0pFppzedmK7q6db0NuwzX0NVLmLfnlRi1JUkjZA0D1gCXBURXbst/VDrVSvVGeh/zxMRzwCPAxt3QFwA78rVEBdK2qzG9LI1GnfZds5V6r+WtHW7g+lCnZpOaunWtNObTk1X9XRcenPmawiStB5wEXBURDzR7ngGKiKejYjtSD1e7yhpm3bHZL36JdATEf8BXMWqUgdb3a3A5hGxLfAd4Bdtjsfaz2mndToyvTnzNcRIGkXKeJ0TERe3O55miIjlwHXAHu2OpQSNvGrl3/NIGglsCDzS7rgi4pGIeCp/PA14dYtjakTHvbomIp6oVKlHxK+AUZLGtjOmLtSp6aSWbk07vem4dFVPp6Y3Z76GkNye4XRgfkR8o93xDIakcZJG5+F1gN2Bu9obVSkaedXKZcDBefjdwLWRW5a2M66qNh9vJ7U5bLfLgIPy01k7AY9HxKJ2BiTpRZW2R5J2JJ2H25Ep6Gadmk5q6da005uOS1f1dGp6G/DrhYYKSeeSnoYYK2kh8IWIOL29UQ3Y64ADgdtzWymA43Nuv9uMB2ZKGkFKLBdExOVtjqnlos6rViR9EZgTEZeRMtg/lXQPqaHpfh0S18ckvR14Jsc1rdVx1Uq/pIcziIgfAr8iPZl1D/AP4IMdENO7gQ9LegZ4EtivTZmCrtWp6WQQsZaednrTiemqnm5Nb369kJmZmVmJXO1oZmZmViJnvrqMpB5JkRuQmlkTSfqhpM+1Ow6zIkmTJK3MzTCGNEkLJO3W7jhazZmvQcgHyZM5UVT+Nm13XGbtImk/pVe5/D2/8mO2pI+0qXPLfouIwyPiJPj3a0kWtjsm6xy1MgZKrwa6oZXfGxF/i4j1IuLZVn5PI3K6eC5f71YovTapbW2+upUzX4O3T04Ulb+HihNdQmXDhaRjgFOB/wFeBGwCHE56EGTNEuNwmjNrrYciYj1gA+DjwI8lvbzNMXUVZ76arFAteIikvwHX5vE7SfqDpOW5p90phWU2lHS6pEWSHpR0cqV4WamX969JWibpPmCvqu/bVNJlkh5VesnpoYVpJ0j6uaSz8x3K7ZJeJunTuVTiAUlvKWXH2JAmaUPgi8BHIuLCiFiRXz3yx4g4ICKeUnrR8dck/U3S4lzFt05efoqkhZKOycfmouLddIPLHivpYeAnefyhOU08mtPIpnm8JH0zf88TOV1sk6edmdPfuqR3121aLNWW9A9JGxfi2kHphcijStrV1sEkHSfp3ny+vVPSOwrTpkn6vaTvKr3o+S5Juxamz5L0FUk35+PyUklj8rTVmpvkeU/K61sh6UoV+q7q43ozTdJ9ebm/Sjogj99S0u9ybMsknd/X9uY0/ivSE5r/UfiOU/P15QlJcyW9oTDtBEkXSDorx3CHpMl19ucrc4z7N/QDdBFnvlrnTcArgbdKmgBcAZwMjAE+AVwkaVye90zSI8Zbkl4J9Bag8lb7Q4G98/jJpMdmi84jvdph0zzty5J2KUzfB/gpsBHwR9LjzmuQXgXxReBHTdlaG+52BtYCLu1lnunAy4DtSMf6BODzhekvInWEOQE4BPiepI36sewYYHPgsJwGvgK8l9Rtyf2ktAIpfb0xr2/DPM9q/f5ExN+Bt5Hv8Aul2rPy/BUHAudFxNO9bLcNH/cCbyAdVycCZ2v1Prxem+cZS+oS4eJKBis7CPgv0jH7DPDtXr7r/aQuHl5IKln+BEBv15t8U/Ft4G0RsT7wn0ClW6KTgCtJ14qJpN7geyVpDaUuMsaSup2ouIWUVscAPwN+LmntwvS3k9LjaFKfYd+tse4dSNer/xcR5/YVS9dp9A3c/qv5tvQFwEpgef77BdBDegH0iwvzHQv8tGrZ35I6ANwEeApYpzBtf+C6PHwtcHhh2lvy+keSehh+Fli/MP0rwJl5+ATSOxEr0/bJ8Y7In9fP6xrd7n3pv+7+Az4APFw17g85XTxJuhn5O/CSwvSdgb/m4Sl5vpGF6UuAnQA1sOy/gLUL008Hvlr4vB7wdE6fuwD/l9e9RlXMZwInF9a7sGr6+4Df5+ERwMPAju3e//4r56/GOX85qZ+rG+rMPw+YmoenAQ+Ru3jK424GDszDs4DphWlb5eN6ROG6MrIw72cL834E+E0e7u16s26O+V0Urjl5nrOAGcDEPvbBFOC5vJ6nSNego/pY5jFg2zx8AnB11XY+WbWPTyQVKkxp92/eqj+XfA3evhExOv/tWxhffOno5sB7chHwcknLgdeT7m42J3UIt6gw7UekuxlIJVrFdd1fGN4UeDQiVlRNL77gdHFh+ElgWaxqtPlk/r9eoxtrVscjpE4O/93eKiL+MyJG52mbAC8A5haO898A44rriPQC5Ip/kI7NcQ0suzQi/ln4vCmFtBLp9SKPABMi4lrSnfb3gCWSZkjaoMFhwPqpAAAgAElEQVTtvBTYStIWpLcuPB4RNze4rA0NxXP+aFLGBwBJB0maVzhOtyGVClU8GDmHkd1POlYrqs/1o6qWL3q4MFxJK9DL9SZSie77SG0xF0m6QtIr8nKfIt3o3JyrAv+rl33wUN72DUglacXaFiR9QtL8XIW5nFQSWNyO6tjX1uptNQ8H/hARs3qJoas589U6xQT2AOlOZHThb92ImJ6nPQWMLUzbICIqb15fxOrv0JpUGH4IGCNp/arpHfmOLRvSbiQdx1PrTF9GyuxvXTjON4zUaLcvjSxb3Vv0Q6SLEAC5umVjctqIiG9HxKtJd90vAz5Z43uf1wN1zuBdQCrpO5BUpW+GpM2BHwNHABvnzMmfSRmaignSak/+TiIdqxXV5/qnScd/f/R2vSEifhsRu5Nu/u/KMRMRD0fEoRGxKfDfwPclbdnbF0V6H+WxwKsk7QuQ23d9ilQ9v1HeD4+z+n7oy+HAJEnf7McyXcWZr3KcDewj6a1KDejXVmokPDHS+7CuBL4uaYNch/4SSW/Ky15AevXExNz+5bjKSiPiAVLVzlfyOv+D1Fbm7HI3z4a7SC8/P5F0wn63pPXzsbwdqarjOdJJ/puSXgipbYqktzaw7oEsey7wQUnbSVoL+DIwOyIWSHqNpNcqNZL/O/DPHF+1xcDGSg8TFJ1FqkJ6O8582SrrkjLsSwGUHhjZpmqeF5LO56MkvYfULrj4+rcPSNpK0gtIbXIvjP53L1H3eiNpE0lT883IU6Qq1OdyvO+RNDGv47G8LbXSxWoi4l/A11nVBnN9Unu1pcBISZ8nlZD1xwpgD+CNkqb3c9mu4MxXCXImaSpwPOmAfIB0p13Z/weRGkzeSTroLyTdlUC66PwWuA24Fbi4avX7k9oDPARcQno35dUt2hSzuiLiq8DRpLvexfnvR6Q74z/k//cAN0l6ArgaaPTx9H4tm9PA54CLSKXHL2HVu/02IKWrx0hVO4+QuseoXsddpEzcfbn6ZtM8/veki9KtEXF/9XI2PEXEnaRMyI2kY/9VwO+rZpsNvJRUmvUl4N0RUXzY46ekdocPA2sDHxtAHL1db9YgpdGHSE8ovgn4cF70NcBsSStJjeCPjIj7AHI15AG9fO0ZpJKqfUjXq9+Q2lXeT7q5eaCXZettx3JS1f7bJJ3U3+U7nd/taGbWT5KuBX4WEae1OxbrDpKmAR+KiNfXmT4LONvH1PDgzgjNzPpB0muAHajfvs3MrFeudjQza5CkmaQqz6OqnjI2M2uYqx3NzMzMSuSSLzMzM7MSldrma+zYsdHT01PmV9owNXfu3GURMa7vOTuL04iVpVvTCDidWHlalU5KzXz19PQwZ86cMr/ShilJXdkFgNOIlaVb0wg4nVh5WpVOXO1oZmZmVqI+M1+5d9ybJd2WO1o7MY/fQtJsSfdIOl/Smq0P18zMzKy7NVLy9RSwS0RsC2wH7CFpJ+AU4JsRsSWpp+hDWhemmZmZ2dDQZ+YrkpX546j8F6S3mF+Yx88E9m1JhGZmZmZDSEMN7iWNAOYCWwLfA+4FlkfEM3mWhcCEOsseBhwGMGnSpMHGayXoOe6KAS23YPpeTY7E2s3HgnWbso/Zbkkj7YjTv0V9DTW4j4hnI2I7YCKwI/CKRr8gImZExOSImDxuXFc+1WxmZmbWNP162jG/Zfw6YGdgtKRKydlE4MEmx2ZmZmY25DTytOM4SaPz8DrA7sB8Uibs3Xm2g4FLWxWkmZl1N0lnSFoi6c+FcSdIelDSvPy3ZztjNCtLIyVf44HrJP0JuAW4KiIuB44FjpZ0D7AxcHrrwjQzsy53JrBHjfHfjIjt8t+vSo7JrC36bHAfEX8Ctq8x/j5S+y8zM7NeRcT1knraHYdZJyj19UJmZmZVjpB0EDAHOCYiHqs1U6ufnB/ok3Ld8n0D1Y44u2XfDIYzX0PUcDh4zazr/QA4idR35EnA14H/qjVjRMwAZgBMnjw5ygrQrBX8bkczM2uLiFicuzJ6Dvgxbspiw4QzX2Zm1haSxhc+vgP4c715zYYSVzuamVnLSToXmAKMlbQQ+AIwRdJ2pGrHBcB/ty1AsxI582VmZi0XEfvXGO0uimxYcrWjmZmZWYmc+TIzMzMrkTNfZk0iaYSkP0q6PH/eQtJsSfdIOl/Smu2O0czM2s+ZL7PmOZL03tOKU0ivTtkSeAw4pC1RmZlZR3Hmy6wJJE0E9gJOy58F7AJcmGeZCezbnujMzKyTOPNl1hzfAj4FPJc/bwwsj4hn8ueFwIRaC0o6TNIcSXOWLl3a+kjNzKytnPkyGyRJewNLImLuQJaPiBkRMTkiJo8bN67J0ZmZWadxP19mg/c64O2S9gTWBjYATgVGSxqZS78mAg+2MUYzM+sQLvkyG6SI+HRETIyIHmA/4NqIOAC4Dnh3nu1g4NI2hWhmZh3EmS+z1jkWOFrSPaQ2YO7N28zMXO1o1kwRMQuYlYfvA3ZsZzxmZtZ5XPJlZmZmViJnvszMzMxK5MyXmZmZWYmc+TIzMzMrkTNfZmZmZiVy5svMzMysRM58mZmZmZXImS8zMzOzErmTVWuanuOuGNByC6bv1eRIzMzMOlefJV+SNpN0naQ7Jd0h6cg8foykqyTdnf9v1PpwzczMzLpbI9WOzwDHRMRWwE7ARyVtBRwHXBMRLwWuyZ/NzMzMrBd9Zr4iYlFE3JqHVwDzgQnAVGBmnm0msG+rgjQzMzMbKvrV4F5SD7A9MBvYJCIW5UkPA5vUWeYwSXMkzVm6dOkgQjUzMzPrfg1nviStB1wEHBURTxSnRUQAUWu5iJgREZMjYvK4ceMGFayZmXUnSWdIWiLpz4Vxbjtsw1JDTztKGkXKeJ0TERfn0YsljY+IRZLGA0taFaSZdRc/+Wo1nAl8FzirMK7Sdni6pOPy52PbEJtZqRp52lHA6cD8iPhGYdJlwMF5+GDg0uaHZ2ZmQ0FEXA88WjXabYdtWGqk2vF1wIHALpLm5b89genA7pLuBnbLn83MzBrVUNthcPthG1r6rHaMiBsA1Zm8a3PDMTOz4SgiQlLNtsN5+gxgBsDkyZPrzmfWDfx6ITMza5fFuc0wbjtsw4kzX2Zm1i5uO2zDkjNfZmbWcpLOBW4EXi5poaRDcNthG6b8Ym0zM2u5iNi/ziS3HbZhxyVfZmZmZiVyyZfZEDfQDk/NzKw1XPJlZmZmViJnvszMzMxK5GrHkvhdd2ZmZgYu+TIzMzMrlTNfZmZmZiVy5svMzMysRG7z1U9lP7Y/HLoJcHs4MzMbTlzyZWZmZlYiZ77MzMzMSuTMl5mZmVmJnPkyMzMzK5EzX2aDJGkzSddJulPSHZKOzOPHSLpK0t35/0btjtXMzNrPmS+zwXsGOCYitgJ2Aj4qaSvgOOCaiHgpcE3+bGZmw5wzX2aDFBGLIuLWPLwCmA9MAKYCM/NsM4F92xOhmZl1Eme+zJpIUg+wPTAb2CQiFuVJDwOb1FnmMElzJM1ZunRpKXGamVn7OPNl1iSS1gMuAo6KiCeK0yIigKi1XETMiIjJETF53LhxJURqZmbt5MyXWRNIGkXKeJ0TERfn0Ysljc/TxwNL2hWfmZl1Dme+zAZJkoDTgfkR8Y3CpMuAg/PwwcClZcdmZmadx+92NBu81wEHArdLmpfHHQ9MBy6QdAhwP/DeNsVnZmYdxJkvs0GKiBsA1Zm8a5mxmJlZ5+uz2lHSGZKWSPpzYZw7jzQzMzMbgEbafJ0J7FE1zp1HmpmZmQ1An5mviLgeeLRqtDuPNDOzppC0QNLtkuZJmtPueMxabaBtvhrqPBJSB5LAYQCTJk0a4NeZWc9xV7Q7hJYb6DYumL5XkyOxNnhzRCxrdxBmZRh0VxO9dR6Zp7sDSTMzM7NsoJkvdx5pZmbNEsCVkubm2pLn8Wu4bCgZaObLnUeamVmzvD4idgDeBnxU0hurZ3Atig0ljXQ1cS5wI/BySQtzh5HTgd0l3Q3slj+bmZn1W0Q8mP8vAS4BdmxvRGat1WeD+4jYv84kdx5pZmaDImldYI2IWJGH3wJ8sc1hmbWUe7g3M7N22gS4JL0ilZHAzyLiN+0Nyay1nPkys67nLiq6V0TcB2zb7jjMyjToribMzMzMrHHOfJmZmZmVyJkvMzMzsxI582VmZmZWIme+zMzMzErkzJeZmZlZiYZtVxMDfTTdzMzMbDBc8mVmZmZWIme+zMzMzErkzJeZmZlZibq+zZfbbpmZmVk3ccmXmZmZWYmc+TIzMzMrUcdUO7r60MzMzIYDl3yZmZmZlciZLzMzM7MSOfNlZmZmViJnvszMzMxK5MyXmZmZWYmc+TIzMzMrkTNfZmZmZiVy5svMzMysRM58mZmZmZWoY3q4NzMr20DfrLFg+l5NjsTMhpNBlXxJ2kPSXyTdI+m4ZgVlNpQ4nZj1zmnEhpsBl3xJGgF8D9gdWAjcIumyiLizWcGZdTunk6Gp7BKzoVxC5zRiw9FgSr52BO6JiPsi4l/AecDU5oRlNmQ4nZj1zmnEhp3BtPmaADxQ+LwQeG31TJIOAw7LH1dK+kud9Y0Flg0injJ1S6zdEicMIFad0uvkzQcTTBP1mU66MI04jgHG0Mcx2/Q4hkoagYbTSSccEwPRjXF3Y8xQI+52pJOWN7iPiBnAjL7mkzQnIia3Op5m6JZYuyVO6K5Ym63b0ojj6KwYOimOVmoknXTrfujGuLsxZuicuAdT7fggsFnh88Q8zsxWcTox653TiA07g8l83QK8VNIWktYE9gMua05YZkOG04lZ75xGbNgZcLVjRDwj6Qjgt8AI4IyIuGMQsfRZ7dJBuiXWbokTuivWhjU5nXTKPnIcq3RCDNA5cfTbEE0j/dWNcXdjzNAhcSsi2h2DmZmZ2bDh1wuZmZmZlciZLzMzM7MSlZ756us1EpLWknR+nj5bUk/ZMeY4+orzaEl3SvqTpGskta3PnEZfzSHpXZJCUtses20kVknvzfv2Dkk/KzvGduuUNNIJaaBTju1OOW4b+E0mSbpO0h/z77JnK+LoNN30eiJJCyTdLmmepDl53BhJV0m6O//fqAPiPEPSEkl/LoyrGaeSb+f9/ydJO3RY3CdIejDv83nFdCHp0znuv0h6a2mBRkRpf6TGlPcCLwbWBG4Dtqqa5yPAD/PwfsD5ZcbYjzjfDLwgD3+4HXE2Gmueb33geuAmYHKnxgq8FPgjsFH+/MJ2xNquv05JI52QBjrl2O6U47bBOGYAH87DWwEL2nEcl/nX6HHSKX/AAmBs1bivAsfl4eOAUzogzjcCOwB/7itOYE/g14CAnYDZHRb3CcAnasy7VT5e1gK2yMfRiDLiLLvkq5HXSEwFZubhC4FdJanEGKGBOCPiuoj4R/54E6lvmnZo9NUcJwGnAP8sM7gqjcR6KPC9iHgMICKWlBxju3VKGumENNApx3anHLeNxBHABnl4Q+ChFsTRaYbC64mKaXomsG8bYwEgIq4HHq0aXS/OqcBZkdwEjJY0vpxIV1cn7nqmAudFxFMR8VfgHtLx1HJlZ75qvUZiQr15IuIZ4HFg41KiqxFDVivOokNIuf526DPWXAS8WUQM7O28zdPIfn0Z8DJJv5d0k6Q9SouuM3RKGumENNApx3anHLeNxHEC8AFJC4FfAf+vBXF0mv4eq+0WwJWS5iq9Mglgk4hYlIcfBjZpT2h9qhdnN/wGR+Qq0TMK1bpti7vlrxca6iR9AJgMvKndsdQiaQ3gG8C0NofSqJGkKpwppJKU6yW9KiKWtzUqq6tdaaDDju1OOW73B86MiK9L2hn4qaRtIuK5kuOw+l4fEQ9KeiFwlaS7ihMjIiR1fB9Q3RJn9gNSCXnk/18H/qudAZVd8tXIayT+PY+kkaSi80dKia5GDFnN111I2g34DPD2iHiqpNiq9RXr+sA2wCxJC0j18Ze1qdF9I/t1IXBZRDydi4H/j3RRGy46JY10QhrolGO7U47bRuI4BLgAICJuBNYmvUh4KOuq1xNFxIP5/xLgElI11+JKNV3+36nNLerF2dG/QUQsjohn803Ij1lVtdi+uEtuCDcSuI/UsK3SMHLrqnk+yuqNiS8oM8Z+xLk9qXHeS8uOr7+xVs0/i/Y1uG9kv+4BzMzDY0lFwhu3cx934D5qeRrphDTQKcd2pxy3Dcbxa2BaHn4lqc2XyjyGy/7r73HS5ljXBdYvDP8hHzv/w+oN2b/a7lhzLD2s3nC9ZpzAXqze4P7mDot7fGH446R2XgBbs3qD+/soqcF9O3bKnqS7wnuBz+RxXyTdOUO6U/s5qeHbzcCL2/Tj9RXn1cBiYF7+u6yNB1qvsVbN25ILVBP3q0hVSXcCtwP7tSvWDt5HpaSRTkgDnXJsd8px20AcWwG/zxeUecBb2n08l/FXa7904h/piczb8t8dhd9wY+Aa4O6crsZ0QKznAouAp0klu4fUizMf/9/L+//2Nl9jasX90xzXn0jvDS1mxj6T4/4L8Lay4vTrhczMzMxK5B7uzczMzErkzJeZmZlZiZz5aoH8io+Vkka0OxazvuTX8mzZy/QF+anGZn7namlE0iaSrpe0QtLXm/ldZmZ9kXSlpAPK+r4hnfmqddGQNE3SDa383oj4W0SsFxHPtvJ7GiXpeEl/zRe7hZLOb3dM1j9lHcuSzpR0chPXN0vSh6rH10gjhwHLgA0i4ph+fsc0Sc/m47v4t2kTNqHfJE3JnZxaL6p+q+ckPVn43NSLoKSTJZ3Zz2U2lPQtSfdL+rukv0n6uaTXNDO2bpOvIVP6mGeCpJ9IeljSE5LmS/qCpHVKCrNXtY6HiHhLRJxTVgxDOvNlIOlg4EBgt4hYj9QZ5jVN/g531muDtTlwZwz8CaAbc2au+DccXq3TtYq/FfA3YJ/CuOddBMs8z0haG7gOeAXpScoNSF13XAC8raw4cixddX6VNBa4kdQFyGsjYgNSdxrjSE979mdda+TOlHsd15Xa/Thrix85XUDKdBTHTQNuyMPHkR4xXUF6RPwdVfP9Hvgu6fUtdwG7FqbPAr5CetT/CeBSVj1220PqSXdkYd6T8vpWAFdSeLEqqV+UPwDLSY8gT6mK47683F+BA/L4LYHf5diWUeelxjn+b/Wyj8YAPyH1B/QY8IvCtENJ3Rk8Sno8d9PCtCD1N3U38Nc87hXAVXn+vwDvbfcxMFT+GjiWNwUuApbm4+Rjhfl2JJ0Ml5Mewf4usGbVb7klqfTpaeBfwErgl4Xv/gTpMe3HgfOBtRuMexbwoRrj/51GgDOrvnc30o1hJX0+Qrro1Xz8vrgfakx7ST4edyjsp6WVNEYv6ThP7y1tPi/tkPpuehJ4Lm/LyvydjfwGh+f0tJz02L4K0w8F5rPqXLUD8Engoqrt/TZwaruP1yYd3yfnY+3cvN3TgLOBEwrz7Ebh5eHA8fn3eIJ0zp4C7J2Prafz7zG3gXgOJ3W2uU4f870emJPTxc2kzAbAAcBNVfN+Erg4D69N6prkAVJ3Ld8np6nKNuVteTgfY5Vxn8rH70PAQYV1nw18B/ht3sbrSa/++U4+nuYD2xbmn0jq4LVyvvho1X4/N69zBfBnVqWfc/Ox/WT+nqNr7JPppJfM1+1frt5+y9NuIF0vb8zf01Nn3EJWT48nk97uAOl8FqR081D++3ieVvN4yN8xLQ+vAXweuJ/UkeyZpFL54roPyjEsJfd91q9jvt2Jrg0JehqrLljvIZ0Y1wDeB/yd3P9Hnu8ZUodso/L0x1mVwZpFSpzbkE64FwFn52k9PD/zdS/p/W/r5M/T87QJpIvLnjmO3fPncXm9TwAvz/OOJ3cemBPBZ/Iya5NeWVFrH3yAdPH5JKnUa0TV9CtIJ7iN8na+KY/fhZSp24HUAd13gOsLywUpozUmb9O6pBPJB0kX1O3z8lu1+zgYCn+9Hcv5GJibTxZrku4u7wPemud7NSkTMTIfm/OBo6p+yy3z8JnAyTW+++acVsbk5Q9vMO5Z9JH5qvW9wJGseln3WsCPgHPrfMc06mS+8vRDSRmWF5AuTl+riq9eOq6bNvtIO1OAhVUxNPIbXA6MBiaRTuh75GnvyTG+htSf0pakksLxpHPW6DzfSNKF4tXtPl6bdHyfTLpI7pP3/zr0kvkidZh5P/Ci/HkLch94FC7MDcZzIXBaH/OMJV0T9s/7/sB8fGwErJd/mxcX5v8j8O48/B1S5mcjUqnar4CTCtv0DPBlUnpepzDuC/lYe3tefyVDcHb+7bcnXQ9+R8pUvR8YQcoQXZXnXYPUB9zxef1b5v2/a2FfPQm8NS/7PxTSF1WZnhr7ZQ7wuYHstzz9hhzPK/O2jqwzrpHM109J6X7b/B1T6h0PrJ75OozUb9wWpDdpXAr8pGrdP8z7egfgKfrZ2XTbE10JCXolKedf+fsH9e+S5wFT8/A0qnqHJl2ADszDs8gZqPx5K9KJYgS1M1+fLcz7EeA3efhY4KdVcfwWOJh0MVgOvIuqOzDgLGAGMLGB/XAAqUO8v+cD8Ng8fjzpLmajGsucTqGXZdLJ5GmgJ38OYJfC9PcB/1u1jh8BX2j3cTAU/no7loHXAn+rmv/TlZNFjXUdBVxS+NxI5usDhc9fJfew30DcsxhY5ms+q5c0j8/H38ga65pGujAV9829VfNcxqpOFteqiq9eOu4tbfaWdqZQlflq8Dd4feHzBazqSfy3wJF11vNr4NA8vDep6rbtx+sAj+9ama9rq8b1lvl6OakUadfq44T+Z75mVR2Pk/Nx9QRwRx73QeAPVcvdUkkrwHnA8Xn4FaQMx9qkzM8/gc0Ly70BuLuwTf9k9ZLR3Ujpf0Rh3KPkzkzzfvlBYdrHgdsLn7cHluXh1wH3VcX9OeDHhX31m8K0/wBWFj73lfn6KzXSfGF6X/vtBuDzVdNrjWsk87VlYfo3gB/VOx5YPfP1O+CwwrStSRmsNQrrflFh+q3kjHWjf91fb9q3fSNidOWPlPEBQNJBkuZJWi5pOenut/getAcj79nsftLdf8UDVdNGUf89ag8Xhv9BysxAuoN9TyWGHMfrSSVwfydlag4HFkm6QtIr8nKfIt0F3yzpDkl1XxIaEedExG6ku+rDgZMkvZX0TqtHI+KxGottmrepso6VpIxb8Y3vxe3fHHht1XYcALyoXlzWb/WO5c2BTav2/fGkagckvUzS5ZXGr6Q76v6+76/e8dsqmwOXFLZnPvAseZtquKm4byLiJVXTf0xK39+J57+Dsl46rps26T3tPE+Dv0G9fbwZqeS8lpmk0m3y/582Ek8XeaDvWZKI+AtwDKnH/yWSzpU00PPPI6TfubLuOTnNvZdUEgtV58jsfladI39GKt2BdC68OCL+STonrgXcVjiuLgdeWFjP4oj4V9W6l8XqD3FVp8PFheEna3wuXnMmVR3Xn2L1c3X1sbgujVtt39XQ136D2r97w8dCnWWqr9+9qY7xflIp4bjKiIgY1DlxOGS+apK0OemEfATpHWyjSXXbKsw2QVLx8yRSaVjFZlXTniZVtfXHA6S76+KFY92ImA4QEb+NiN1JB/NdOWYi4uGIODQiNgX+G/h+b90F5GWejoifk+7+t8nfPUbS6BqzP0RKpABIWpf0aoniS0eLGdMHgN9Vbcd6EfHhfu0NG4gHSO3uivt+/YjYM0//AenYeWmkxq/Hs/pxXhR1xpftAdKrPorbtHbklxL3h6T1gG+RSnNPkDSmapZ66bi3tNlb2qm1D/vzG1R7gNR2rZZfAP8haRtSyVdpT2uVpHpf/p1UjVSxWuYqIs6OiNeRqotGkNrz1VpPX64B9pD0gl7mWe0cmU1i1TnyN6RryKtImbCf5fGLSaWrLy8cVxtGxIbFTelnvP3xAKmUrfp8sU+Dy/cV29XAO6qunUV97bd639GvYyGrTtuV63df21Ad4yTSb7a0j+UaNmwzX6ScfJB3pqQPkjIkRS8EPiZplKT3kOqbf1WY/gFJW+UE+kXgwuh/9xJnA/tIequkEZLWzo+qT1Tq+2hqzvg8RSp2fi7H+x5JE/M6Hsvb8lz1yvNj+HtJWj8/JfI2UhHq7IhYRKq2+L6kjfJ2vjEvei7wQUnbSVqLdKc+OyIW1NmOy4GXSTowr2eUpNdIemU/94f1383ACknHSlonH0fbFB6JX59UXbIyl5z2liFeTD+fSGrAyHxcV/5GNbDMD4Ev5ZskJI2TNHWA338qMCciPkRqp/XDqun10nHdtNlH2lkMbCypeDHtz29Q7TTgE5JerWTLyn7JJSkXki7sN0fE3/qx3m40D9gr7/PxwMcqEyS9UtKb8/nqSVY9+ADpN+npJUNQ7SekDPjFkrbOv/86pOrHisuBrSW9T9JISe8nVUldAZBLri4iVXetC1ybxz9L+k2/lY9r5fP9WwawPwbiRuBfko7Jx/QISa+S9OoGl+/rHPE1UqnuTyRNAsjbd6qkreljv/XDPGC/vI4dgXfWmOdz+Zz4KlJzgUo3S30dD+cCR0vqkbQ+8CVSm9PnXWMHathmviLiTuDrpANxMfAq0tOIRbOBl5IS4ZdIdbqPFKb/lNRW5WFSXf7H6KeIeACYSroTXkq6K/kk6bdZAzialAt/FHgTq07arwFmS1pJas9yZETcB6BUDVnpJ+eJvO6/kdosfBX4cERU+oc6kHSnfxepweZROa6rSe0ALiI9nfUSYL9etmMF8JY8z0N5n5zCqiJ6a5F8Mt8b2I7U3mIZ6eReufh/gtTwdgWp5LS3ft5OB7bK1RG/6Ou7taqz1MpJ9gBJd1TN9gNWXQyfJF3Y+nIq6bi+UtIKUuP71xa+d6WkNxTm31nP7+frNTnDtger0s3RwA5avR+pmum4j7QJ9dPOXaST9315P25K/36D1eTS6i+RMlgrSKVdxdK7maTz12pVjpJOk/TdRr+nS5xJqoK+n1SydF5h2lqk89sy0m+5EemhJEj7e03gUUk3w3ltskEAABxFSURBVPP3j6S/SHofQEQ8STrf/oWUya48Pbkt+TwYEUtJDd+PJVW1fRzYu6oq+mek9loXVN2YH5O34WZSW7ArSdealouIZ0gPkexIamu3jNQ+d4MGV/Fl4MR8bB+VM28rJe2c178M2DnPe0tOv1fl77mvwf3WiM+Q2tItJ12rflZjnhtIDx9dCXwlIq7N4593PFSppNH/ZVVvA0c2EpSkgyXd1ud8qzdpsgpJ00iNBl9fZ/os0lNRp5UZl5k1z1BIxznjexepAfAT7Y7HrN2UmuDcHRGNlnSWbtiWfJmZdTulziaPBs5zxsuse3RVz7lmZpYotQVdTKq+2qPN4ZhZP7ja0czMzKxErnY0MzMzK1Gp1Y5jx46Nnp6eMr/Shqm5c+cui4hxfc/ZWZxGrCzdmkbA6cTK06p0Umrmq6enhzlz5pT5lTZMSaruQbkrOI1YWbo1jYDTiZWnVenE1Y5mZmZmJXLmy8zMzKxEznyZmZmZlcj9fHW4nuP6+7qrZMH0vZocidnQ4/RlFT4WrEwu+TIzMzMrkTNfZmZmZiVy5svMzMysRM58mZmZmZXImS8zMzOzEjnzZWZmZlYiZ77MzMzMSuTMl5mZmVmJnPkyawJJa0u6WdJtku6QdGIev4Wk2ZLukXS+pDXbHauZmbWXM19mzfEUsEtEbAtsB+zx/9u7/2DJyvrO4++PiKjxB4xM2BEYh6wEM3FTohPFsBpWNLJiAamkKNxojSmSqdpVo9FdHd2tdcu4W2O55a/S0kwAGZUFEU1BMIlhR35INrAO6qowISCCjs4wo0BAjRrku3+cc6XnMvfennu7T/96v6q67ulzTnd/T8/9Tn/v8zz9PElOAt4FvLeqng7cC5w7whglSWPA4ksagGr8oL17aHsr4EXAZe3+bcBZIwhPkjRGXNtxSi13nTJwrbLlSnIIcBPwdOBDwDeA+6rqwfaUXcDRB3jcJmATwNq1a7sJVpI0MrZ8SQNSVT+rqmcBxwDPBZ7R5+O2VtWGqtqwevXqocYoSRo9iy9pwKrqPuBq4PnA4UnmWpiPAb4zssAkSWPB4ksagCSrkxzebj8OeAmwk6YI+932tI3A5aOJUJI0LhzzJQ3GGmBbO+7rUcClVXVlkluAS5K8E/gycP4og5QkjZ7FlzQAVfVV4MQD7L+DZvyXJEmA3Y6SJEmdsviSJEnqkMWXJElShyy+JEmdSXJIki8nubK97/qnmjkOuJekjix35YkpW3Xi9TTTsDypvT+3/uklST5Cs/7ph0cVnNQFW74kSZ1IcgxwOnBeez+4/qlm0JLFV5Jjk1yd5JYkNyd5fbt/VZKrktzW/jxi+OFKkibY+4A3Aw+1959CH+ufQrMGapIdSXbs27dv+JFKQ9RPy9eDwJuqaj1wEvCaJOuBzcD2qjoe2N7elyTpEZK8HNhbVTct5/GugappsuSYr6raDexutx9IspPmL5MzgVPa07YB1wBvGUqUkqRJdzJwRpKXAY+lGfP1ftr1T9vWL9c/1Uw4qAH3SdbRzOJ9I3BUW5gB7AGOWuAxm4BNAGvXrl1unGPDAbOSdPCq6q3AWwGSnAL8x6r6vSSfoln/9BJc/1Qzou8B90meAHwaeENV3d97rKoKqAM9zqZiSdIi3gK8McntNGPAXP9UU6+vlq8kh9IUXhdV1Wfa3XcnWVNVu5OsAfYOK0hJ0vSoqmtohqq4/qlmUj/fdgzNXyI7q+o9PYeuoGkiBpuKJUmS+tJPy9fJwKuAryX5SrvvbcAW4NIk5wJ3AWcPJ0RJkqTp0c+3Ha8HssDhUwcbjiRJ0nRzhntJkqQOWXxJkiR1yOJLkiSpQxZfkiRJHbL4kiRJ6pDFlyRJUocsviRJkjp0UAtrS3qkJMcCH6NZXL6ArVX1/iSrgE8C64A7gbOr6t5RxanBWbf5s6MOQdIEs+VLWrkHgTdV1XrgJOA1SdYDm4HtVXU8sL29L0macRZf0gpV1e6q+lK7/QCwEzgaOBPY1p62DThrNBFKksaJ3Y4dmaRuiuXGeueW0wccyeRJsg44EbgROKqqdreH9tB0Sx7oMZuATQBr164dfpCSpJGy5UsakCRPAD4NvKGq7u89VlVFMx7sEapqa1VtqKoNq1ev7iBSSdIo2fIlDUCSQ2kKr4uq6jPt7ruTrKmq3UnWAHtHF6E0Gyapl0Gzy5YvaYWSBDgf2FlV7+k5dAWwsd3eCFzedWySpPFjy5e0cicDrwK+luQr7b63AVuAS5OcC9wFnD2i+CQNiWNktRwWX9IKVdX1QBY4fGqXsUiSxp/djpIkSR2y5UvSxHOQtaRJYvGlgXHsgyRJS7PbUZIkqUMWX5IkSR2y+JIkSerQzI75coCuJEkaBVu+JEmSOmTxJUmS1CGLL0mSpA7N7JgvjQ/nB5MkzRJbviRJkjpk8SVJktQhux0lDZxdyZovybHAx4CjgAK2VtX7k6wCPgmsA+4Ezq6qe0cVp9SFJYuvJBcALwf2VtUz231jkyzO1yVJE+FB4E1V9aUkTwRuSnIV8Gpge1VtSbIZ2Ay8ZYRxSkPXT7fjhcBp8/ZtpkmW44Ht7X1Jkg6oqnZX1Zfa7QeAncDRwJnAtva0bcBZo4lQ6s6SxVdVXQfcM2+3ySJJWpYk64ATgRuBo6pqd3toD0235IEesynJjiQ79u3b10mc0rAsd8B9X8kCJowk6WFJngB8GnhDVd3fe6yqimY82CNU1daq2lBVG1avXt1BpNLwrPjbjoslS3vchJEkkeRQmsLroqr6TLv77iRr2uNrgL2jik/qynKLL5NFktS3JAHOB3ZW1Xt6Dl0BbGy3NwKXdx2b1LXlFl8miyTpYJwMvAp4UZKvtLeXAVuAlyS5DXhxe1+aav1MNXExcApwZJJdwNtpkuPSJOcCdwFnDzNISdJkq6rrgSxw+NQuY5FGbcniq6pescAhk0Wacs6jJ0mD5/JC0gAkuSDJ3iRf79m3KslVSW5rfx4xyhglSePB5YWkwbgQ+CDN8ilz5iYjduZuSftxCa7ZZsuXNABORixJ6pfFlzQ8ztwtSXoEiy+pA87cLUmaY/ElDY+TEUuSHsEB99LwzE1GvAUnI+6LU1tImgW2fEkD0E5G/HfACUl2tRMQO3O3JOkRbPmSBsDJiCVJ/bL4kqacXXmSNF7sdpQkSeqQxZckSVKH7HaUJoTdh5I0HWz5kiRJ6pDFlyRJUofsdpSkMbfcLuc7t5w+4Eg0a1Yy3MHfv4XZ8iVJktQhW74kSZpyfmFnvNjyJUmS1CGLL0mSpA5ZfEmSJHXI4kuSJKlDFl+SJEkdsviSJEnqkMWXJElSh5znS5IkjY1ZWNFhbIovJ4CTJEmzYGyKL0mStDgbKhY2SS1mjvmSJEnqkMWXJElShyy+JEmSOrSi4ivJaUluTXJ7ks2DCkqaJuaJtDhzRLNm2QPukxwCfAh4CbAL+GKSK6rqlkEFJ00680RanDkyvfxywMJW0vL1XOD2qrqjqn4KXAKcOZiwpKlhnkiLM0c0c1Yy1cTRwLd77u8Cnjf/pCSbgE3t3R8kuXUFrzkKRwLfG3UQHZi468y7Fj38tI7CWMqSeXIQOTKO/0bjGBOMZ1ydx7RUjiTZVFVbOwpnIYP+LBnHf/thm7VrHuj1juKzZOjzfLWJPerkXrYkO6pqw6jjGLZZuc5x1G+OjOO/0TjGBOMZ17jGxIT8/zzJeTJss3bN03C9K+l2/A5wbM/9Y9p9kh5mnkiLM0c0c1ZSfH0ROD7JcUkeA5wDXDGYsKSpYZ5IizNHNHOW3e1YVQ8meS3wOeAQ4IKqunlgkY2PiWiSH4BZuc5ODThPxvHfaBxjgvGMy5gOYAifJSO/phGYtWue+OtNVY06BkmSpJnhDPeSJEkdsviSJEnqkMXXIpIckuTLSa4cdSzDlOTwJJcl+fskO5M8f9QxzZqllldJcliST7bHb0yyrufYW9v9tyZ5aYcxvTHJLUm+mmR7kqf1HPtZkq+0t4ENnu4jplcn2dfz2n/Qc2xjktva28YOY3pvTzz/kOS+nmPDep8uSLI3ydcXOJ4kH2hj/mqSZ/ccG8r7NGgryZlJtJJ8nFRLXXPPeb+TpJJMzvQTVeVtgRvwRuB/AVeOOpYhX+c24A/a7ccAh486plm60Qwy/gbwS+37//+A9fPO+Q/AR9rtc4BPttvr2/MPA45rn+eQjmL6N8Dj2+1/PxdTe/8HI3qfXg188ACPXQXc0f48ot0+oouY5p3/OpoB5UN7n9rnfSHwbODrCxx/GfBXQICTgBuH+T6N6HfhgDkzibeV5uMk3vrNLeCJwHXADcCGUcfd782WrwUkOQY4HThv1LEMU5In0/xHfT5AVf20qu5b/FEasH6WVzmTpkgGuAw4NUna/ZdU1U+q6pvA7e3zDT2mqrq6qn7U3r2BZn6mYVrJMjQvBa6qqnuq6l7gKuC0EcT0CuDiAbzuoqrqOuCeRU45E/hYNW4ADk+yhuG9T4O2kpyZROOYj8PWb279CfAu4MddBrdSFl8Lex/wZuChUQcyZMcB+4CPtl2s5yX5hVEHNWMOtLzK0QudU1UPAv8IPKXPxw4rpl7n0rSkzHlskh1Jbkhy1gDiOZiYfqfterksydzknSN/n9puoOOAz/fsHsb71I+F4h7W+zRoK8mZSbTSfJxES15z211+bFVN3AreFl8HkOTlwN6qumnUsXTg0TTdEx+uqhOBHwIL9q1L8yV5JbABeHfP7qdVs/zHvwPel+RfdhTOXwDrqurXaFptti1xfpfOAS6rqp/17BvV+6QptUA+Tp0kjwLeA7xp1LEsh8XXgZ0MnJHkTpqmzhcl+cRoQxqaXcCuqrqxvX8ZTTGm7vSzvMrPz0nyaODJwPf7fOywYiLJi4H/DJxRVT+Z219V32l/3gFcA5zYRUxV9f2eOM4DntPvY4cVU49zmNflOKT3qR8LxT0pS/2sJGcm0YrycUItdc1PBJ4JXNN+Vp8EXDExg+5HPehs3G/AKUz/gPsvACe02/8NePeoY5qlG03r4x00XVJzA0t/dd45r2H/wcOXttu/yv4D7u9gMAPu+4npRJoBscfP238EcFi7fSRwG4sMQh9wTGt6tn8buKHdXgV8s43tiHZ7VRcxtec9A7iTdmLrYb5PPc+/joUH3J/O/gPu/+8w36dB31aSM5N4W0k+Tuqt39zqOf8aJmjA/bKXF9JUeR1wUZp11e4Afn/E8cyUWmB5lSTvAHZU1RU0X4j4eJLbaQZSn9M+9uYklwK3AA8Cr6n9u7WGGdO7gScAn2rHMX+rqs4AfgX40yQP0bSub6mqWzqK6Y+SnEHzXtxD8+1HquqeJH9Cs44gwDuqarEB6YOMCZp/r0uq/ZRoDeV9AkhyMc0fjkcm2QW8HTi0jfkjwF/SfOPxduBHtDk/rPdp0FaSM5Nohfk4kQ4ityaSywtJkiR1yDFfkiRJHbL4WoZ2Jt2nL3Ds95L8TR/PcWGSdw4+OkjyziQXLnL81iQv6PO5diU5ZVCxaTZNes5I0iBZfPHz5Vn+at6+2xbYt+i4gaq6qKp+axhxDkpVnVBVXxh1HJpcs5YzkjRIFl+N64DfSHIIQDvT86HAifP2Pb09d2y0X6GWumbOSNIyWXw1vkjzwfGs9v4LgKuBW+ft+0ZVfbe9/+L2r/r7knxobtmKNAv7Xt9uJ82iunuT3J/ka0meOf/FkzwxydVpFrpNmgVh/2eSbyW5O8lHkjyuPfeUtivwLUn2AB9d4JoOS/KJJA8k+Xr2Xzj3512JSR7fnndfmkVZN7dzpvR6dhv7Pya5OMlhB/PmaipNY870Pv9vJ7m5jfXzSU5o9/9hkj/vOe+bab5ZOHd/94HilaReFl806xkCN9KscUj78wvA9fP29f4F/3Lg14FfA86mWRNtvt9qH/fLNBP8nc28Sf6SPAXYDvxtVf1R+1X0Le1jnkXTcnA08F97HvYvaObjeRqwaYHLOgv4OHA4zXw+H1jgvHcAT6WZE+ilwCsPcM7ZwEtoFjh9DvCqBZ5LM2JKc2bu+X+FJndeB6wG/jfN5I2HAtcCL2wLvmNp5sk6uX3cL9MUpDcv9vySZPH1sGt5+EPjBTQfJF+Yt+/anvO3VNV9VfUtmr/4n8Uj/TPNLLzPoJnWY2dV7e45/tT2OT9VVf8Fmr/8aT4c/riaxW0fAP4H+89R8xDw9moWU/6nha6nqj7Xzvn08QXig+bD7b+31/Jt4IMHOOd9VbWnqr4PXLnIc2m2TFvOzDkHuKKqPl9V/0xT2D0ZeF5V/QPwU+BfAb9JM1/W99J8meA3gevmzeUlSY9g8fWw64B/nWQVsLqqbgP+D824llU0yxj0/hW/p2f7RzST2+2nqj5PU8x8CNibZGuSJ/WccjrwOOAjPftWA48Hbmq7PO4D/rrdP2dfVS21gvv8+BZaLHsN+y9e+u0DnLPktWomTVvOzHkqcFdPTA+x/6K+19JMYPrCdvsamsLrN9m/2JSkA7L4etjf0fx1+4fA3wJU1f3Ad9t9362qbx7sk1bVB6rqOcB6mm6R/9Rz+M9oPiT+MslccfQ94J9ollE4vL09uap6P6gG+Zf1Hpo1s+Ycu9CJ0jzTmjPfpemeBH6+gG/vunJzxddcy961WHxJOggWX622K2IH8EaarpM517f7DvobW0l+Pcnz2rEiPwR+TNP90eu1NIOU/yLJ49q/sv8MeG+SX2yf5+gkBxofMwiXAm9LcniSY2jWQ5OWNMU5cylwRjtQ/1Ca4u8BmjFu0BRYL6bpFt1Dc51n0LTkfXWZrylphlh87e9a4BdpPjzmfKHdt5yvyz+J5kPhXppujO/TrL/1c+34kE003RqXJ3ks8BaaNdduSHI/zYDfExZ6kSS/lOQHSZ66jBjfDtxNs+jv39B88PxkGc+j2TR1OVNVNwMbgQ8D+4DTgDPa8V+06y/+uL1Oqupemvy5vi0EJWlRru2o/SR5HXBWVZ066lgkSZpGtnzNuLZ75jeSPKr9iv0fA3++1OMkSdLyONOzDqPp5llH09VzMfCnowxIkqRpZrejJElSh+x2lCRJ6lCn3Y5HHnlkrVu3rsuX1Iy66aabvldVq5c+c7yYI+rKpOaINA06Lb7WrVvHjh07unxJzagkdy191vgxR9SVSc0RaRrY7ShJktQhiy9JkqQOWXxJkiR1yHm+xty6zZ9d1uPu3HL6gCORxpd5ImmS2PIlSZLUIYsvSZKkDll8SZIkdcjiS5IkqUMWX5IkSR2y+JIkSeqQxZckSVKHnOdL0sxyfjBJo2DLlyRJUocsviRJkjpkt+OUWm53CtilIknSMNnyJUmS1CGLL0mSpA5ZfEmSJHXI4kuSJKlDFl+SJEkdsviSJEnqkMWXJElShyy+JEmSOmTxJUmS1CGLL0mSpA5ZfEmSJHXI4kuSJKlDFl+SJEkdsviSJEnqkMWXJElSh5YsvpIcm+TqJLckuTnJ69v9q5JcleS29ucRww9XGj/miCTpYPTT8vUg8KaqWg+cBLwmyXpgM7C9qo4Htrf3pVlkjkiS+rZk8VVVu6vqS+32A8BO4GjgTGBbe9o24KxhBSmNM3NEknQwHn0wJydZB5wI3AgcVVW720N7gKMWeMwmYBPA2rVrlxunNBHMkca6zZ9d1uPu3HL6gCORpPHT94D7JE8APg28oaru7z1WVQXUgR5XVVurakNVbVi9evWKgpXGmTkiSepHX8VXkkNpPlQuqqrPtLvvTrKmPb4G2DucEKXxZ45IkvrVz7cdA5wP7Kyq9/QcugLY2G5vBC4ffHjS+DNHJEkHo58xXycDrwK+luQr7b63AVuAS5OcC9wFnD2cEKWxZ45Ikvq2ZPFVVdcDWeDwqYMNR5o85ogk6WA4w70kSVKHLL4kSZI6dFDzfGn5ljvvkSRJmi62fEmSJHXI4kuSJKlDFl+SJEkdcszXQXLsliRJWglbviRJkjpk8SVJktQhux0ljQ279SXNAlu+JEmSOmTxJUmS1CGLL0mSpA7N7Jgvx5ZIWq7l/v9x55bTBxyJpElky5ckSVKHLL4kSZI6ZPElSZLUoZkd86WFOZ5FcxwbKUmDZ8uXJElShyy+JEmSOmTxJUmS1CHHfGlgHCsmLc4ckQS2fEmSJHXK4kuSJKlDFl+SJEkdcsyXRs5xMNLizBFputjyJUmS1CGLL0mSpA5NfLejy59IkqRJsmTLV5ILkuxN8vWefauSXJXktvbnEcMNUxpv5okkqV/9dDteCJw2b99mYHtVHQ9sb+9Ls+xCzBNJUh+WLL6q6jrgnnm7zwS2tdvbgLMGHJc0UcwTSVK/ljvg/qiq2t1u7wGOWujEJJuS7EiyY9++fct8OWki9ZUn5ogkzZYVf9uxqgqoRY5vraoNVbVh9erVK305aSItlifmiCTNluUWX3cnWQPQ/tw7uJCkqWGeSJIeYbnF1xXAxnZ7I3D5YMKRpop5Ikl6hCXn+UpyMXAKcGSSXcDbgS3ApUnOBe4Czh5mkNK4G+c8cS48SRovSxZfVfWKBQ6dOuBYpIllnkiS+uXyQpIkSR2y+JIkSeqQxZckSVKHLL4kSZI6ZPElSZLUIYsvSZKkDll8SZIkdcjiS5IkqUMWX5IkSR2y+JIkSeqQxZckSVKHLL4kSZI6ZPElSZLUIYsvSZKkDll8SZIkdcjiS5IkqUMWX5IkSR2y+JIkSeqQxZckSVKHLL4kSZI6ZPElSZLUIYsvSZKkDll8SZIkdcjiS5IkqUOPHnUAc9Zt/uyoQ5DGmjkiSdPBli9JkqQOWXxJkiR1yOJLkiSpQxZfkiRJHVpR8ZXktCS3Jrk9yeZBBSVNE/NEktRr2cVXkkOADwH/FlgPvCLJ+kEFJk0D80SSNN9KWr6eC9xeVXdU1U+BS4AzBxOWNDXME0nSflYyz9fRwLd77u8Cnjf/pCSbgE3t3R8kuXUFrzlsRwLfG3UQHZro6827Fj38tI7CWMqSeTJhOdKPif696tNEXOOE5Ig0c4Y+yWpVbQW2Dvt1BiHJjqraMOo4ujJr1zuuJilH+jELv1ezcI2Shmcl3Y7fAY7tuX9Mu0/Sw8wTSdJ+VlJ8fRE4PslxSR4DnANcMZiwpKlhnkiS9rPsbseqejDJa4HPAYcAF1TVzQOLbDSmpuunT7N2vZ2b0jxZyiz8Xs3CNUoaklTVqGOQJEmaGc5wL0mS1CGLL0mSpA5ZfPVIckiSLye5ctSxDFOSw5NcluTvk+xM8vxRx6TpMe15ZP5IWqmhz/M1YV4P7ASeNOpAhuz9wF9X1e+238B7/KgD0lSZ9jwyfyStiC1frSTHAKcD5406lmFK8mTghcD5AFX106q6b7RRaVpMex6ZP5IGweLrYe8D3gw8NOpAhuw4YB/w0bZr6LwkvzDqoDQ1pj2PzB9JK2bxBSR5ObC3qm4adSwdeDTwbODDVXUi8ENg82hD0jSYkTwyfyStmMVX42TgjCR3ApcAL0ryidGGNDS7gF1VdWN7/zKaDxNppWYhj8wfSStm8QVU1Vur6piqWkez/Mvnq+qVIw5rKKpqD/DtJCe0u04FbhlhSJoSs5BH5o+kQfDbjrPpdcBF7Te17gB+f8TxSJPE/JG0Ii4vJEmS1CG7HSVJkjpk8SVJktQhiy9JkqQOWXxJkiR1yOJLkiSpQxZfkiRJHbL4kiRJ6tD/B6Jp1EKouXHcAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Print histograms for each feature to see how they vary.\n", "histohrams = data.hist(grid=False, figsize=(10, 10))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Split the Data Into Training and Test Sets\n", "\n", "In this step we will split our dataset into _training_ and _testing_ subsets (in proportion 80/20%).\n", "\n", "Training data set will be used for training of our linear model. Testing dataset will be used for validating of the model. All data from testing dataset will be new to model and we may check how accurate are model predictions." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "# Split data set on training and test sets with proportions 80/20.\n", "# Function sample() returns a random sample of items.\n", "train_data = data.sample(frac=0.8)\n", "test_data = data.drop(train_data.index)\n", "\n", "# Decide what fields we want to process.\n", "input_param_name_1 = 'Economy..GDP.per.Capita.'\n", "input_param_name_2 = 'Freedom'\n", "output_param_name = 'Happiness.Score'\n", "\n", "# Split training set input and output.\n", "x_train = train_data[[input_param_name_1, input_param_name_2]].values\n", "y_train = train_data[[output_param_name]].values\n", "\n", "# Split test set input and output.\n", "x_test = test_data[[input_param_name_1, input_param_name_2]].values\n", "y_test = test_data[[output_param_name]].values" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's visualize the training and test datasets to see the shape of the data." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "data": [ { "marker": { "line": { "color": "rgb(255, 255, 255)", "width": 1 }, "opacity": 1, "size": 10 }, "mode": "markers", "name": "Training Set", "type": "scatter3d", "uid": "b2ccc8e3-f5b5-4efb-af5c-e982a479b71b", "x": [ 1.1027104854583702, 1.3208793401718102, 0.7922212481498722, 0.39724862575531, 0.9097844958305359, 1.1893955469131499, 0.401477217674255, 1.06457793712616, 0.479820191860199, 0.368745893239975, 1.00985014438629, 0.381430715322495, 1.07062232494354, 1.12786877155304, 0.6364067792892459, 0.564305365085602, 0.932537317276001, 1.28948748111725, 1.29178786277771, 1.00726580619812, 0.8584281802177429, 0.30544471740722695, 1.1018030643463101, 1.63295245170593, 0.7288706302642819, 1.49438726902008, 1.8707656860351598, 1.3469113111496, 0, 1.05469870567322, 0.951484382152557, 0.5916834473609921, 0.982409417629242, 1.15687310695648, 1.4637807607650801, 1.48238301277161, 1.69227766990662, 0.43108540773391707, 1.3753824234008798, 0.0921023488044739, 1.12112903594971, 0.786441087722778, 1.00082039833069, 0.24454993009567302, 0.36711055040359497, 0.511135876178741, 1.41691517829895, 1.4870972633361799, 1.1531838178634601, 0.7268835306167599, 1.1073532104492199, 0.11904179304838199, 1.2860119342803997, 1.15655755996704, 0.716249227523804, 1.22255623340607, 0.667224824428558, 0.368610262870789, 1.0352252721786501, 0.995538592338562, 1.74194359779358, 1.6263433694839498, 1.12209415435791, 0.6595166921615601, 1.5649795532226598, 1.4844149351120002, 0.925579309463501, 1.2175596952438401, 1.36135590076447, 1.23374843597412, 1.12843120098114, 1.28177809715271, 1.29121541976929, 1.07498753070831, 0.85769921541214, 1.480633020401, 1.1536017656326298, 0.6484572887420649, 0.89465194940567, 0.872001945972443, 0.950612664222717, 1.43092346191406, 1.1614590883255, 1.10970628261566, 1.21768391132355, 0.8089642524719242, 0.9910123944282528, 0.305808693170547, 1.3412059545516999, 0.37584653496742204, 0.339233845472336, 1.40570604801178, 1.0272358655929599, 0.900596737861633, 1.1982743740081798, 1.40167844295502, 1.3559380769729599, 0.560479462146759, 1.07937383651733, 1.48792338371277, 1.3145823478698702, 0.996192753314972, 1.3950666189193701, 0.964434325695038, 1.25278460979462, 1.55167484283447, 1.50394463539124, 1.44163393974304, 1.18529546260834, 1.54625928401947, 1.5357066392898602, 1.1307767629623402, 0.6017650961875921, 0.524713635444641, 0.7885475754737851, 0.0226431842893362, 1.44357192516327, 1.53062355518341, 0.730573117733002, 1.1982102394104, 0.737299203872681, 1.08116579055786, 1.2845562696456898, 0.233442038297653 ], "y": [ 0.288555532693863, 0.479131430387497, 0.469987004995346, 0.147062435746193, 0.432452529668808, 0.491247326135635, 0.106179520487785, 0.325905978679657, 0.44030594825744607, 0.5818438529968261, 0.561213254928589, 0.443185955286026, 0.47748741507530196, 0.580200731754303, 0.461603492498398, 0.430388748645782, 0.473507791757584, 0.0957312509417534, 0.520342111587524, 0.289680689573288, 0, 0.38042613863945, 0.465733230113983, 0.49633759260177596, 0.24072904884815197, 0.612924098968506, 0.604130983352661, 0.47120362520217896, 0.270842045545578, 0.479246735572815, 0.260287940502167, 0.24946372210979503, 0.204403176903725, 0.24932260811328896, 0.5397707223892211, 0.626006722450256, 0.549840569496155, 0.42596277594566295, 0.40598860383033797, 0.235961347818375, 0.194989055395126, 0.6582486629486078, 0.4551981985569, 0.348587512969971, 0.514492034912109, 0.390017777681351, 0.505625545978546, 0.567766189575195, 0.412730008363724, 0.23521526157856, 0.437453746795654, 0.33288118243217496, 0.17586351931095098, 0.295400261878967, 0.25471106171608, 0.255772292613983, 0.423026293516159, 0.0303698573261499, 0.45000287890434293, 0.443323463201523, 0.59662789106369, 0.60834527015686, 0.505196332931519, 0.0149958552792668, 0.620070576667786, 0.601607382297516, 0.474307239055634, 0.5793922543525699, 0.518630743026733, 0.550026834011078, 0.15399712324142498, 0.373783111572266, 0.40226498246192893, 0.28851598501205394, 0.585214674472809, 0.6271626353263849, 0.39815583825111406, 0.0960980430245399, 0.12297477573156401, 0.5313106179237371, 0.309410035610199, 0.470222115516663, 0.28923171758651695, 0.580131649971008, 0.457003742456436, 0.4350258708000179, 0.418421149253845, 0.18919676542282102, 0.572575807571411, 0.336384207010269, 0.408842742443085, 0.6140621304512021, 0.39414396882057207, 0.198303267359734, 0.300740599632263, 0.257921665906906, 0.35511153936386103, 0.45276376605033897, 0.55258983373642, 0.562511384487152, 0.234231784939766, 0.381498634815216, 0.256450712680817, 0.520303547382355, 0.376895278692245, 0.490968644618988, 0.5853844881057739, 0.508190035820007, 0.49451920390129106, 0.505740523338318, 0.5731103420257571, 0.41827192902565, 0.633375823497772, 0.47156670689582797, 0.571055591106415, 0.602126955986023, 0.6179508566856379, 0.449750572443008, 0.348079860210419, 0.31232857704162603, 0.447551846504211, 0.47278770804405196, 0.43745428323745705, 0.466914653778076 ], "z": [ 4.49700021743774, 5.61100006103516, 4.31500005722046, 3.5910000801086404, 6.002999782562259, 5.62900018692017, 3.7939999103546103, 5.175000190734861, 4.961999893188481, 3.47099995613098, 4.44000005722046, 4.08099985122681, 6.35699987411499, 6.4239997863769505, 4.513999938964839, 4.69500017166138, 5.493000030517581, 5.2270002365112305, 5.97300004959106, 4.80499982833862, 3.79500007629395, 3.4949998855590803, 5.525000095367429, 6.10500001907349, 5.837999820709231, 7.28399991989136, 6.375, 5.80999994277954, 2.69300007820129, 4.8289999961853, 5.27899980545044, 3.59299993515015, 5.18200016021729, 4.69199991226196, 6.89099979400635, 7.52199983596802, 6.57200002670288, 3.65700006484985, 7.212999820709231, 4.2800002098083505, 5.23699998855591, 5.97100019454956, 6.00799989700317, 3.50699996948242, 4.54500007629395, 3.34899997711182, 5.92000007629395, 7.00600004196167, 6.57800006866455, 5.26900005340576, 6.6350002288818395, 3.5329999923706095, 5.32399988174438, 5.5689997673034695, 4.77500009536743, 5.2930002212524405, 4.11999988555908, 3.6029999256133998, 5.71500015258789, 5.26200008392334, 6.86299991607666, 6.6479997634887695, 3.7660000324249303, 4.138999938964839, 7.49399995803833, 7.28399991989136, 5.31099987030029, 6.4539999961853, 6.1680002212524405, 6.4520001411438, 5.25, 5.962999820709231, 6.08400011062622, 5.22499990463257, 5.42999982833862, 7.50400018692017, 5.234000205993651, 4.29199981689453, 4.09600019454956, 6.4539999961853, 4.28599977493286, 6.44199991226196, 4.71400022506714, 7.0789999961853, 5.824999809265139, 4.29099988937378, 5.33599996566772, 3.64400005340576, 5.75799989700317, 3.875, 4.46000003814697, 7.31400012969971, 4.95499992370605, 4.37599992752075, 5.5, 5.837999820709231, 5.62099981307983, 4.55299997329712, 5.230000019073491, 6.9510002136230495, 5.90199995040894, 4.64400005340576, 5.96400022506714, 4.57399988174438, 6.65199995040894, 5.47200012207031, 7.3769998550415, 6.71400022506714, 6.59899997711182, 6.993000030517581, 6.9770002365112305, 5.82200002670288, 4.1680002212524405, 5.04099988937378, 5.07399988174438, 5.151000022888179, 7.468999862670901, 6.343999862670901, 5.1810002326965305, 4.46500015258789, 6.07100009918213, 5.2729997634887695, 5.8189997673034695, 3.97000002861023 ] }, { "marker": { "line": { "color": "rgb(255, 255, 255)", "width": 1 }, "opacity": 1, "size": 10 }, "mode": "markers", "name": "Test Set", "type": "scatter3d", "uid": "297d492d-6698-40a5-843b-cf6b403a4442", "x": [ 1.6164631843566901, 1.47920441627502, 1.35268235206604, 1.34327983856201, 1.4336265325546298, 1.38439786434174, 1.32539355754852, 1.4884122610092199, 0.907975316047668, 1.09186446666718, 1.26074862480164, 0.833756566047668, 1.06931757926941, 0.8781145811080929, 1.3151752948761002, 0.783756256103516, 0.885416388511658, 0.5962200760841371, 0.989701807498932, 0.36842092871666, 0.586682975292206, 0.23430564999580397, 0.479309022426605, 0.476180493831635, 0.6030489206314089, 0.35022771358490007, 0.16192533075809498, 0.43801298737525896, 0.521021246910095, 0.777153134346008, 0.09162256866693501 ], "y": [ 0.635422587394714, 0.611100912094116, 0.49094617366790794, 0.588767051696777, 0.361466586589813, 0.408781230449677, 0.295817464590073, 0.536746919155121, 0.5475093722343439, 0.233335807919502, 0.32570791244506797, 0.5587329268455511, 0.20871552824974102, 0.408158332109451, 0.498465299606323, 0.394952565431595, 0.5015376806259161, 0.454943388700485, 0.28211015462875394, 0.318697690963745, 0.478356659412384, 0.48079109191894503, 0.37792226672172496, 0.306613743305206, 0.4477061927318571, 0.32436785101890603, 0.36365869641304, 0.16234202682972002, 0.390661299228668, 0.0815394446253777, 0.0599007532000542 ], "z": [ 7.537000179290769, 7.31599998474121, 6.60900020599365, 6.52699995040894, 6.42199993133545, 6.40299987792969, 6.09800004959106, 6.08699989318848, 5.955999851226809, 5.872000217437741, 5.849999904632571, 5.82299995422363, 5.39499998092651, 5.2350001335144, 5.19500017166138, 5.07399988174438, 5.01100015640259, 5.0040001869201705, 4.7350001335144, 4.70900011062622, 4.60799980163574, 4.550000190734861, 4.53499984741211, 4.19000005722046, 4.17999982833862, 4.03200006484985, 4.02799987792969, 3.9360001087188703, 3.80800008773804, 3.46199989318848, 2.90499997138977 ] } ], "layout": { "margin": { "b": 0, "l": 0, "r": 0, "t": 0 }, "scene": { "xaxis": { "title": "Economy..GDP.per.Capita." }, "yaxis": { "title": "Freedom" }, "zaxis": { "title": "Happiness.Score" } }, "title": "Date Sets" } }, "text/html": [ "
" ], "text/vnd.plotly.v1+html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Configure the plot with training dataset.\n", "plot_training_trace = go.Scatter3d(\n", " x=x_train[:, 0].flatten(),\n", " y=x_train[:, 1].flatten(),\n", " z=y_train.flatten(),\n", " name='Training Set',\n", " mode='markers',\n", " marker={\n", " 'size': 10,\n", " 'opacity': 1,\n", " 'line': {\n", " 'color': 'rgb(255, 255, 255)',\n", " 'width': 1\n", " },\n", " }\n", ")\n", "\n", "# Configure the plot with test dataset.\n", "plot_test_trace = go.Scatter3d(\n", " x=x_test[:, 0].flatten(),\n", " y=x_test[:, 1].flatten(),\n", " z=y_test.flatten(),\n", " name='Test Set',\n", " mode='markers',\n", " marker={\n", " 'size': 10,\n", " 'opacity': 1,\n", " 'line': {\n", " 'color': 'rgb(255, 255, 255)',\n", " 'width': 1\n", " },\n", " }\n", ")\n", "\n", "# Configure the layout.\n", "plot_layout = go.Layout(\n", " title='Date Sets',\n", " scene={\n", " 'xaxis': {'title': input_param_name_1},\n", " 'yaxis': {'title': input_param_name_2},\n", " 'zaxis': {'title': output_param_name} \n", " },\n", " margin={'l': 0, 'r': 0, 'b': 0, 't': 0}\n", ")\n", "\n", "plot_data = [plot_training_trace, plot_test_trace]\n", "\n", "plot_figure = go.Figure(data=plot_data, layout=plot_layout)\n", "\n", "# Render 3D scatter plot.\n", "plotly.offline.iplot(plot_figure)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Init and Train Linear Regression Model\n", "\n", "> ☝🏻This is the place where you might want to play with model configuration.\n", "\n", "- `polynomial_degree` - this parameter will allow you to add additional polynomial features of certain degree. More features - more curved the line will be.\n", "- `num_iterations` - this is the number of iterations that gradient descent algorithm will use to find the minimum of a cost function. Low numbers may prevent gradient descent from reaching the minimum. High numbers will make the algorithm work longer without improving its accuracy.\n", "- `learning_rate` - this is the size of the gradient descent step. Small learning step will make algorithm work longer and will probably require more iterations to reach the minimum of the cost function. Big learning steps may couse missing the minimum and growth of the cost function value with new iterations.\n", "- `regularization_param` - parameter that will fight overfitting. The higher the parameter, the simplier is the model will be.\n", "- `polynomial_degree` - the degree of additional polynomial features (`x1^2 * x2, x1^2 * x2^2, ...`). This will allow you to curve the predictions.\n", "- `sinusoid_degree` - the degree of sinusoid parameter multipliers of additional features (`sin(x), sin(2*x), ...`). This will allow you to curve the predictions by adding sinusoidal component to the prediction curve." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Initial cost: 224369.59\n", "Optimized cost: 2761.58\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Model Parameters
05.393174
10.765539
20.357765
\n", "
" ], "text/plain": [ " Model Parameters\n", "0 5.393174\n", "1 0.765539\n", "2 0.357765" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Set up linear regression parameters.\n", "num_iterations = 500 # Number of gradient descent iterations.\n", "regularization_param = 0 # Helps to fight model overfitting.\n", "learning_rate = 0.01 # The size of the gradient descent step.\n", "polynomial_degree = 0 # The degree of additional polynomial features.\n", "sinusoid_degree = 0 # The degree of sinusoid parameter multipliers of additional features.\n", "\n", "# Init linear regression instance.\n", "linear_regression = LinearRegression(x_train, y_train, polynomial_degree, sinusoid_degree)\n", "\n", "# Train linear regression.\n", "(theta, cost_history) = linear_regression.train(\n", " learning_rate,\n", " regularization_param,\n", " num_iterations\n", ")\n", "\n", "# Print training results.\n", "print('Initial cost: {:.2f}'.format(cost_history[0]))\n", "print('Optimized cost: {:.2f}'.format(cost_history[-1]))\n", "\n", "# Print model parameters\n", "theta_table = pd.DataFrame({'Model Parameters': theta.flatten()})\n", "theta_table.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Analyze Gradient Descent Progress\n", "\n", "The plot below illustrates how the cost function value changes over each iteration. You should see it decreasing. \n", "\n", "In case if cost function value increases it may mean that gradient descent missed the cost function minimum and with each step it goes further away from it. In this case you might want to reduce the learning rate parameter (the size of the gradient step).\n", "\n", "From this plot you may also get an understanding of how many iterations you need to get an optimal value of the cost function. In current example you may see that there is no much sense to increase the number of gradient descent iterations over 500 since it will not reduce cost function significantly. " ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEWCAYAAACwtjr+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xt8XVWd/vHPkzS3pknbNGlpS6FFilgQKhQoigyoYGFQ0OHmOFoVRUd0dMa5gHMBdXSYcRRlVBSFARwH9IcXOopC5SKK3MqdgtACxd7v90toku/vj71STkOapu052cnJ8369ziv7rL323mulaZ7stdfZWxGBmZlZKVXk3QAzMyt/DhszMys5h42ZmZWcw8bMzErOYWNmZiXnsDEzs5Jz2FhZkbRA0tvS8mclfS/vNpmZw8b6kKTzJT0gabOkFWn545JUiuNFxJci4sP7uh9JEyWFpCE91LlM0nZJG9PrOUnfkDR2X49fKqlPB/ew/gOS2iVtkrRB0mOSzujLNlr5cNhYn5D0GeDrwJeB/YAxwMeANwHVu9imss8aWBw/jIgGoAl4F1k/H+7PgdML90XEMGAEcA3wI0kju1bqKYj3RrH3Z/lz2FjJSRoOfB74eETcHBEbI/NoRLw3IlpTveskXSXpVkmbgZMl/amkR9Nf1gslXdZl3++T9JKk1ZL+scu6yyT9T8H76ZJ+L2mdpMclnVSw7m5JX5B0bzozuV1Sc1p9T/q6Lv2Vf3xP/Y2I7RExFzgPWAl8puA4Z6QzhHWpLUcUrPsHSYvT8Z+V9NZUXpmGBJ9P6x6WNCGtO1TSbElr0jbnFuzvOknflPSLtN0Dkl6T1nX26fHUp/N206cO4FqgDniNpJMkLUptXgb8d9rvRyTNT+2ZJWlcQXtOTW1cL+lbkn4j6cNp3QfS9/4KSauBy1L5hyQ9I2mtpNskHZjKlequSD8bT0o6PK07XdLTqc+LJf1tT32zPhIRfvlV0hcwA2gDhuym3nXAerKznQqgFjgJeH16fwSwHDgr1Z8CbAJOBGqAr6bjvC2tvwz4n7Q8HlgNnJ72dUp635LW3w08DxxC9gv1buDytG4iED21v/BYXco/DzyQlt8ArACOAyqBmcCC1PbXAguBcQXHfE1a/jvgyVRHwJHAKKA+bfNBYEja/ypgSsH3czVwbFr/A+CmgrYFcHAPffoA8Lu0PAT4FLARGJ7+XdqAf0/trwPeko5/VCr7L+CetH0zsAF4d8G+tgMfLjhWG/DJtL4OOBOYD7wulf0T8PtU/+3Aw2RnXEp1xqZ1S4E3p+WRwFF5/x/wK3xmY32iGVgVEW2dBQVnGFslnVhQ95aIuDciOiJiW0TcHRFPpvdPADcCf5Lqng38PCLuiezs6J+Bjl204S+AWyPi1rSv2cAcsvDp9N8R8VxEbAV+BEwtQt+XkA2rAVwIfCciHoiI9oi4HmgFpgPtZL+gp0iqiogFEfF82u7DwD9FxLOReTwiVgNnAAsi4r8joi0iHgV+DJxTcPyfRsSD6Xv/g73o03RJ64BlwHuAd0XE+rSuA7g0IlrT9+y9wLUR8Uj697gEOF7SRLLv89yI+Elqy5Vpnzt9ryLiv1JftpINs/5bRDyTtvkSMDWd3WwHGoBDAaU6S9N+tqfvY2NErI2IR/awz1YCDhvrC6uB5sJx+Ih4Y0SMSOsKfw4XFm4o6ThJd0laKWk92S+gzuGtcYX1I2Jz2l93DgTOSQG3Lv0CPQEovJ5S+MtvCzBsTzq5C+OBNQVt+EyXNkwgO5uZD3ya7AxphaSbCoagJpCddXXXp+O67O+9ZNeKitWn+yNiREQ0R8T0iPh1wbqVEbGt4P044KXONxGxiezfYzyv/rcKYFGXYy3s8v5A4OsFfVtDdhYzPiLuBL4BfJPs+3W1pMa03Z+RhdtLaaiux2FP6xsOG+sL95H9BX9mL+p2vQ35/wKzgAkRMRz4NtkvHMiGSyZ0VpQ0lGx4qTsLge+nX5ydr/qIuHwv2tQrkiqAdwC/LWjDF7u0YWhE3AgQEf8bESeQ/ZINsiGqzu1es4s+/abL/oZFxF/uTXv3QtfvyxKytgMgqZ7s32Mx2b/V/gXrVPh+F/tbCHy0S//qIuL3ABFxZUQcTTacegjZcCMR8VBEnAmMBn5GdpZqOXPYWMlFxDrgc8C3JJ0tqUFShaSpZNcdetIArImIbZKOBf68YN3NwBmSTpBUTXZ9ZFc/0/8DvEPS29MF99p0kbvrL7zurCQbMjqoF3WRNETS68iG/PYju5YE8F3gY+lsTZLqlU2AaJD0WklvkVQDbAO28sqQ4PeAL0ianLY7QtIo4OfAIcomSVSl1zHp2L2xvLd96qUbgQ9Kmpr68SWy61ULgF8Ar5d0VjrDvYidz8C6823gEkmHQTbRRNI5afmY9H2sAjaTfc86JFVLeq+k4RGxnew60a6GVq0POWysT0TEfwB/A/w92S+55cB3gH8Aft/Dph8HPi9pI/AvFPyVGtmMr4vIzn6WAmt59dBMZ92FZGdWnyULj4Vkfwnv9v9ARGwBvgjcm4Z0pu+i6nmSNpFNcphFNoR0dEQsSfuZA3yEbPhnLdnF7w+kbWuAy8kusC8j+6v8krTuq6nft5P98rwGqIuIjcCpwPlkZxXLeOWCfW9cBlyf+nTu7irvThpi+2ey60ZLyc7Gzk/rVpFdS/oPsu/LFLJrZq097O+nZP25SdIG4CngtLS6kSy815IN3a0mm1YP8D5gQdrmY2RDi5YzZUOnZmZ9Jw0xLgLeGxF35d0eKz2f2ZhZn0hDmCPSENtnya693Z9zs6yPOGzMrK8cTzarbhXZxImz0hRnGwQ8jGZmZiXnMxszMys53+wuaW5ujokTJ+bdDDOzAeXhhx9eFREtu6vnsEkmTpzInDlz8m6GmdmAIuml3dfyMJqZmfUBh42ZmZWcw8bMzErOYWNmZiXnsDEzs5Jz2JiZWck5bMzMrOQcNvvozj8s51t3z8+7GWZm/ZrDZh/d89wqrrqruyf2mplZJ4fNPhpVX83G1jZa29rzboqZWb/lsNlHTcOqAVi7eXvOLTEz678cNvtoVH0WNqs37/LptmZmg57DZh811WePe1+z+eWcW2Jm1n85bPZRUzqzcdiYme2aw2Yf7RhG2+SwMTPbFYfNPhpeV0VlhXxmY2bWA4fNPqqoECOHVrHaYWNmtksOmyIYObSaNZ6NZma2Sw6bImiqr/YwmplZDxw2RTBqWLWH0czMeuCwKQKf2ZiZ9cxhUwRN9TWs27KdtvaOvJtiZtYvOWyKoPOzNmu3+P5oZmbdcdgUge8iYGbWM4dNEfhmnGZmPXPYFEHnYwZ8ZmNm1j2HTRF4GM3MrGclCxtJEyTdJelpSXMlfSqVN0maLWle+joylUvSlZLmS3pC0lEF+5qZ6s+TNLOg/GhJT6ZtrpSkno5RKiOH+macZmY9KeWZTRvwmYiYAkwHLpI0BbgYuCMiJgN3pPcApwGT0+tC4CrIggO4FDgOOBa4tCA8rgI+UrDdjFS+q2OURFVlBcPrqnxmY2a2CyULm4hYGhGPpOWNwDPAeOBM4PpU7XrgrLR8JnBDZO4HRkgaC7wdmB0RayJiLTAbmJHWNUbE/RERwA1d9tXdMUpmlD/YaWa2S31yzUbSROANwAPAmIhYmlYtA8ak5fHAwoLNFqWynsoXdVNOD8fo2q4LJc2RNGflypV73rECTfXVno1mZrYLJQ8bScOAHwOfjogNhevSGUmU8vg9HSMiro6IaRExraWlZZ+O09JQwypfszEz61ZJw0ZSFVnQ/CAifpKKl6chMNLXFal8MTChYPP9U1lP5ft3U97TMUqmeVgNKzf6zMbMrDulnI0m4BrgmYj4asGqWUDnjLKZwC0F5e9Ps9KmA+vTUNhtwKmSRqaJAacCt6V1GyRNT8d6f5d9dXeMkmlpqGH91u20trWX+lBmZgPOkBLu+03A+4AnJT2Wyj4LXA78SNIFwEvAuWndrcDpwHxgC/BBgIhYI+kLwEOp3ucjYk1a/jhwHVAH/DK96OEYJdPSUANk05/Hjagr9eHMzAaUkoVNRPwO0C5Wv7Wb+gFctIt9XQtc2035HODwbspXd3eMUmoeloXNyo2tDhszsy58B4Ei6TyzWbXJ123MzLpy2BRJZ9h4koCZ2as5bIqk887PDhszs1dz2BRJbVUljbVDPIxmZtYNh00RtTTUsNJhY2b2Kg6bIvIHO83MuuewKSLfssbMrHsOmyJqafCZjZlZdxw2RdQ8rIZNrW1sfdm3rDEzK+SwKSJ/sNPMrHsOmyLqDJsVHkozM9uJw6aIWob5zMbMrDsOmyLyLWvMzLrnsCmipvpqJIeNmVlXDpsiqqqsoGlota/ZmJl14bApstGNtazYsC3vZpiZ9SsOmyLbr7GG5RsdNmZmhRw2Rbbf8FqWrfcwmplZIYdNkY1uqGX15la2t3fk3RQzs37DYVNk+w2vJcIf7DQzK+SwKbL9GmsBWO5JAmZmOzhsimx0Y/bBzuXrHTZmZp0cNkXWeWazzGc2ZmY7OGyKrKm+murKCpZv8DUbM7NODpsik8ToxhpfszEzK+CwKYExjbUs8zUbM7MdHDYlsF9jre8iYGZWwGFTAmMaaz0bzcysgMOmBMY01rD55XY2btued1PMzPoFh00J7De884OdnpFmZgYOm5IY47sImJntxGFTAp1h4xlpZmYZh00JjB3uuwiYmRVy2JRAbVUlo+qrWbxua95NMTPrFxw2JTJuRB1LHDZmZkAJw0bStZJWSHqqoOwySYslPZZepxesu0TSfEnPSnp7QfmMVDZf0sUF5ZMkPZDKfyipOpXXpPfz0/qJpepjT8aNqHXYmJklpTyzuQ6Y0U35FRExNb1uBZA0BTgfOCxt8y1JlZIqgW8CpwFTgPekugD/nvZ1MLAWuCCVXwCsTeVXpHp9btyIOhav3UpE5HF4M7N+pWRhExH3AGt6Wf1M4KaIaI2IF4H5wLHpNT8iXoiIl4GbgDMlCXgLcHPa/nrgrIJ9XZ+Wbwbemur3qfEj6tj8cjsbtrb19aHNzPqdPK7ZfELSE2mYbWQqGw8sLKizKJXtqnwUsC4i2rqU77SvtH59qt+nxo2oA/AkATMz+j5srgJeA0wFlgJf6ePj70TShZLmSJqzcuXKou67M2x83cbMrI/DJiKWR0R7RHQA3yUbJgNYDEwoqLp/KttV+WpghKQhXcp32ldaPzzV7649V0fEtIiY1tLSsq/d28m4EdlnbZasd9iYmfVp2EgaW/D2XUDnTLVZwPlpJtkkYDLwIPAQMDnNPKsmm0QwK7Kr7ncBZ6ftZwK3FOxrZlo+G7gzcrhK31xfQ/WQCg+jmZkBQ3ZfZe9IuhE4CWiWtAi4FDhJ0lQggAXARwEiYq6kHwFPA23ARRHRnvbzCeA2oBK4NiLmpkP8A3CTpH8FHgWuSeXXAN+XNJ9sgsL5pepjTyoqxLjhtSxZ57sImJmVLGwi4j3dFF/TTVln/S8CX+ym/Fbg1m7KX+CVYbjC8m3AOXvU2BLxBzvNzDK+g0AJOWzMzDIOmxIaN6KO5Ru2sb29I++mmJnlymFTQuNH1NIRfq6NmZnDpoQ6P2uzaK2H0sxscHPYlNCEkUMBWLhmS84tMTPLl8OmhMaNqKNCDhszM4dNCVUPqWDs8Dr+6LAxs0HOYVNiBzQNddiY2aDnsCmxLGw8QcDMBjeHTYkdMGooqza1svXl9rybYmaWG4dNiU1oSjPS1noozcwGL4dNiR2QwuaPqx02ZjZ4OWxKbEfYeJKAmQ1iDpsSGzm0imE1Qxw2ZjaoOWxKTBITmob6g51mNqg5bPrAAU3+YKeZDW4Omz7Q+cHOHJ5ObWbWLzhs+sABTUNpbetg+YbWvJtiZpYLh00fmNQ8DIAXVm3KuSVmZvnoVdhI+n5vyqx7B7XUA/Diqs05t8TMLB+9PbM5rPCNpErg6OI3pzzt11hLbVUFL6502JjZ4NRj2Ei6RNJG4AhJG9JrI7ACuKVPWlgGKirExFH1PrMxs0Grx7CJiH+LiAbgyxHRmF4NETEqIi7pozaWhYNa6nnBYWNmg1Rvh9F+LqkeQNJfSPqqpANL2K6yc1DzMP64Zgvb2zvyboqZWZ/rbdhcBWyRdCTwGeB54IaStaoMTWqup70jfCcBMxuUehs2bZF9IvFM4BsR8U2goXTNKj+T0oy0FzxJwMwGoSG9rLdR0iXA+4A3S6oAqkrXrPJzULOnP5vZ4NXbM5vzgFbgQxGxDNgf+HLJWlWGRgytpqm+2pMEzGxQ6lXYpID5ATBc0hnAtojwNZs9NKm5nhd9FwEzG4R6eweBc4EHgXOAc4EHJJ1dyoaVo0nN9TzvazZmNgj19prNPwLHRMQKAEktwK+Bm0vVsHI0efQwbn54Eeu2vMyIodV5N8fMrM/09ppNRWfQJKv3YFtLDtkvm8D33HIPpZnZ4NLbwPiVpNskfUDSB4BfALeWrlnl6ZAxnWGzMeeWmJn1rR6H0SQdDIyJiL+T9G7ghLTqPrIJA7YHxg2vZVjNEOY5bMxskNndNZuvAZcARMRPgJ8ASHp9WveOkrauzEji4NHDPIxmZoPO7obRxkTEk10LU9nEkrSozB0yZhjzVvjMxswGl92FzYge1tX1tKGkayWtkPRUQVmTpNmS5qWvI1O5JF0pab6kJyQdVbDNzFR/nqSZBeVHS3oybXOlJPV0jP7ikDENrNr0Mqs3+RHRZjZ47C5s5kj6SNdCSR8GHt7NttcBM7qUXQzcERGTgTvSe4DTgMnpdSHZjT+R1ARcChwHHAtcWhAeVwEfKdhuxm6O0S+8MknAQ2lmNnjsLmw+DXxQ0t2SvpJevwEuAD7V04YRcQ+wpkvxmcD1afl64KyC8hsicz8wQtJY4O3A7IhYExFrgdnAjLSuMSLuTzcIvaHLvro7Rr/QGTYeSjOzwaTHCQIRsRx4o6STgcNT8S8i4s69PN6YiFialpcBY9LyeGBhQb1Fqayn8kXdlPd0jFeRdCHZmRQHHHDAnvZlr4xprKGhdoinP5vZoNKrOwhExF3AXcU8cESEpCjmPvf0GBFxNXA1wLRp00ralk6SeO2YBp5d5rAxs8Gjr+8CsDwNgZG+dt6VYDEwoaDe/qmsp/L9uynv6Rj9xpRxjTyzdCMdHX2Sb2ZmuevrsJkFdM4omwncUlD+/jQrbTqwPg2F3QacKmlkmhhwKnBbWrdB0vQ0C+39XfbV3TH6jSljG9nU2sYf/dROMxskensjzj0m6UbgJKBZ0iKyWWWXAz+SdAHwEtkdpCG79c3pwHxgC/BBgIhYI+kLwEOp3ucjonPSwcfJZrzVAb9ML3o4Rr9x2LjhAMxdsoGJ6aFqZmblrGRhExHv2cWqt3ZTN4CLdrGfa4FruymfwyuTFgrLV3d3jP5k8phhDKkQTy9dz58eMTbv5piZlZzv3JyD2qpKDh49jLlLNuTdFDOzPuGwycmUcY0OGzMbNBw2OTls3HBWbmxlxcZteTfFzKzkHDY5mTK2EYCnfXZjZoOAwyYnU8ZlYeOhNDMbDBw2ORleV8WBo4byxKJ1eTfFzKzkHDY5OnL/ETy+cH3ezTAzKzmHTY6mThjBsg3bWLbekwTMrLw5bHI09YDs2XSPLVybc0vMzErLYZOjKWMbqaoUjy70dRszK28OmxzVVlUyZWwjj/3RYWNm5c1hk7OpE0bw5OL1tPtxA2ZWxhw2OZt6wAi2vNzux0SbWVlz2OTsDRNGAvDISx5KM7Py5bDJ2YGjhtI8rIY5C9bsvrKZ2QDlsMmZJI6dNJIHXnTYmFn5ctj0A8dObGLxuq0sWuvHRJtZeXLY9APHThoFwEMeSjOzMuWw6Qdeu18DjbVDeNBDaWZWphw2/UBlhThmYpOv25hZ2XLY9BPHTGrihZWbWbmxNe+mmJkVncOmn5h+UHbd5r4XVufcEjOz4nPY9BOvHz+c4XVV/Pa5lXk3xcys6Bw2/URlhXjTwaP43fxVRPg+aWZWXhw2/cibJ7ewdP02nl+5Ke+mmJkVlcOmHznh4GYA7nluVc4tMTMrLodNPzKhaSiTmuv57TxftzGz8uKw6WfePLmZ+19YQ2tbe95NMTMrGodNP3Pya0ezdXs79z3vKdBmVj4cNv3M8a8ZxdDqSmY/vTzvppiZFY3Dpp+prarkxMkt/PqZ5XT4UdFmViYcNv3QKVPGsHxDK08uXp93U8zMisJh0w+95dDRVFaIXz/joTQzKw8Om35oZH010w4cye1zHTZmVh4cNv3UaYfvx7PLNzJv+ca8m2Jmts8cNv3U6UeMpUIw6/EleTfFzGyf5RI2khZIelLSY5LmpLImSbMlzUtfR6ZySbpS0nxJT0g6qmA/M1P9eZJmFpQfnfY/P22rvu/lvhndUMvxrxnF/z2+xDfmNLMBL88zm5MjYmpETEvvLwbuiIjJwB3pPcBpwOT0uhC4CrJwAi4FjgOOBS7tDKhU5yMF280ofXeK751HjmPB6i2elWZmA15/GkY7E7g+LV8PnFVQfkNk7gdGSBoLvB2YHRFrImItMBuYkdY1RsT9kZ0S3FCwrwFlxmFjqaoUsx7zUJqZDWx5hU0At0t6WNKFqWxMRCxNy8uAMWl5PLCwYNtFqayn8kXdlL+KpAslzZE0Z+XK/nfzy+FDqzj5taP52WNL2N7ekXdzzMz2Wl5hc0JEHEU2RHaRpBMLV6YzkpJfqIiIqyNiWkRMa2lpKfXh9sp5x0xg1aZW7vzDirybYma213IJm4hYnL6uAH5Kds1leRoCI33t/O26GJhQsPn+qayn8v27KR+Q/uSQFsY01vDDhxbuvrKZWT/V52EjqV5SQ+cycCrwFDAL6JxRNhO4JS3PAt6fZqVNB9an4bbbgFMljUwTA04FbkvrNkianmahvb9gXwPOkMoKzjl6Anc/u4Kl67fm3Rwzs72Sx5nNGOB3kh4HHgR+ERG/Ai4HTpE0D3hbeg9wK/ACMB/4LvBxgIhYA3wBeCi9Pp/KSHW+l7Z5HvhlH/SrZM6dNoGOwGc3ZjZgyZ/hyEybNi3mzJmTdzN2aea1DzJ3yQbuvfhkaoZU5t0cMzMAJD1c8BGWXepPU5+tBxecMIlVm1r5v8eX7r6ymVk/47AZIN48uZnJo4dx7e9e9B0FzGzAcdgMEJL40AmTeHrpBu57wY+MNrOBxWEzgLzrDeNpaajhG3fOz7spZmZ7xGEzgNRWVfLREw/i98+v5qEFa3a/gZlZP+GwGWDee9yBNA+r5uu/npd3U8zMes1hM8DUVVdy4YkH8bv5q7jf127MbIBw2AxA75s+kbHDa/niL56ho8Mz08ys/3PYDEB11ZX83dtfy5OL13PL4wP2tm9mNog4bAaos6aO5/Xjh/Mfv3qWza1teTfHzKxHDpsBqqJCXPqOKSxdv40rZj+Xd3PMzHrksBnApk1s4j3HTuDae1/kKT862sz6MYfNAHfxjNfRVF/DP/z4CV5u89M8zax/ctgMcMOHVvGvZx3O3CUbuOLXHk4zs/7JYVMGZhy+H+cfM4Fv/+Z5f/bGzPolh02Z+OczpjBpVD2fvPFRlq3flndzzMx24rApE/U1Q/j2+45mS2sbF35/Dtu2t+fdJDOzHRw2ZeSQMQ187fw38MSi9Vzykyf93Bsz6zccNmXmlClj+NtTD+Gnjy7m3375BweOmfULQ/JugBXfRScfzMqNrVx9zwvUVVXy16cckneTzGyQc9iUIUlc+o7D2PJyO1+/Yx6VFeKTbzkYSXk3zcwGKYdNmaqoEJf/2RG0R/DV2c+xYuM2PvfOw6mscOCYWd9z2JSxygrxn2cfyeiGWr79m+dZtr6VK847kobaqrybZmaDjCcIlLmKCnHxaYfyuXcexl3PruCd37iXZ5ZuyLtZZjbIOGwGiZlvnMj/fvg4Nre2cdY37+W6e1/0g9fMrM84bAaR4w4axS/+6s1MP2gUl/3f05z7nfuYv2JT3s0ys0HAYTPItDTUcN0Hj+Er5xzJvBWbmPG1e7hs1lzWbH4576aZWRnzBIFBSBJ/dvT+nHhIC1+d/Rw33LeAHz+yiA+8cSIz3ziR5mE1eTfRzMqM/AnzzLRp02LOnDl5NyMXzy3fyFduf5bbn15OdWUF506bwPuOP5BDxjTk3TQz6+ckPRwR03Zbz2GTGcxh02n+ik18954X+Mmji9jeHhw5YQTnHL0/p79+LE311Xk3z8z6IYfNHnLYvGLVplZ+9uhifjRnIc8t30SFYNqBTZwyZQxvfd1oJjXX+24EZgY4bPaYw+bVIoK5SzZw+9xl3P70cv6wbCMAYxprOHbSKI6d1MRRB4xg8ugGqod4ronZYOSw2UMOm91buGYLv3luJQ++uIYHX1zDsg3ZQ9qqKsXk0Q1MGdfIofs1MKm5ngNH1TOhqY6aIZU5t9rMSslhs4ccNnsmIli4ZiuPL1rH00s3MHfJBp5esp5Vm16ZQi3BuOF1TGiqY3RDLWMaaxjdUMvo9LWloZrGuiqG11U5lMwGqN6Gjac+216RxAGjhnLAqKG848hxO8rXbH6ZBas389LqzSxYtYWXVm9m0dqtPLZwHSs2bmPb9o5u91dbVUFjbRY8w+uqaKyroq66krqq9KqupLZzuapix/vqygqGVFYwpFLZcoUYUllBVaUYUpG+pvdVaX2F0quCHctS5zI73vu6lFnxlG3YSJoBfB2oBL4XEZfn3KRBoam+mqb6ao46YOSr1kUEG7a1sXLjNlZsaGXlplY2bN3Ohm1trN+6nfVbtrNh23bWb93Oio3b2PpyO9u2d7B1eztbX25nax8/6rowgFQQRF3DqTCUVLAtXUoLy7rWEwX70M51srJXB9+Oejvtt6djvbqdueoHjegHTQDy/8PmS+96PcdOairpMcoybCRVAt8ETgEWAQ9JmhURT+fbssFN0o4zl4NH7/lneCLilfBJAbS9vYO29mB7Rwfb2zpo64gdZW0dHWzv/NqW1WlrDzoi6Ihsf53LHRFEQEdH4ftXlndVv73g/nJBpHYWtHlH23fqyU5lO9fv3T4669Ftveim7NX18tQfhu/zb0HSDxpSX1P6YeyyDBvgWGB+RLwAIOkm4EzAYTPV1sFHAAAGo0lEQVSAScqG1qp9fcdsoCnX+arjgYUF7xelsp1IulDSHElzVq5c2WeNMzMbbMo1bHolIq6OiGkRMa2lpSXv5piZla1yDZvFwISC9/unMjMzy0G5hs1DwGRJkyRVA+cDs3Juk5nZoFWWEwQiok3SJ4DbyKY+XxsRc3NulpnZoFWWYQMQEbcCt+bdDjMzK99hNDMz60ccNmZmVnK+EWciaSXw0l5u3gysKmJzBgL3eXBwnweHfenzgRGx28+OOGyKQNKc3tz1tJy4z4OD+zw49EWfPYxmZmYl57AxM7OSc9gUx9V5NyAH7vPg4D4PDiXvs6/ZmJlZyfnMxszMSs5hY2ZmJeew2UeSZkh6VtJ8SRfn3Z5ikXStpBWSniooa5I0W9K89HVkKpekK9P34AlJR+XX8r0jaYKkuyQ9LWmupE+l8rLtM4CkWkkPSno89ftzqXySpAdS/36YbmiLpJr0fn5aPzHP9u8tSZWSHpX08/S+rPsLIGmBpCclPSZpTirrs59vh80+KHj89GnAFOA9kqbk26qiuQ6Y0aXsYuCOiJgM3JHeQ9b/yel1IXBVH7WxmNqAz0TEFGA6cFH6tyznPgO0Am+JiCOBqcAMSdOBfweuiIiDgbXABan+BcDaVH5FqjcQfQp4puB9ufe308kRMbXgMzV99/Md6Vnrfu35CzgeuK3g/SXAJXm3q4j9mwg8VfD+WWBsWh4LPJuWvwO8p7t6A/UF3AKcMsj6PBR4BDiO7NPkQ1L5jp9zsjupH5+Wh6R6yrvte9jP/dMv1rcAPwdUzv0t6PcCoLlLWZ/9fPvMZt/06vHTZWRMRCxNy8uAMWm5rL4PaajkDcADDII+pyGlx4AVwGzgeWBdRLSlKoV929HvtH49MKpvW7zPvgb8PdCR3o+ivPvbKYDbJT0s6cJU1mc/32X7iAErrYgISWU3b17SMODHwKcjYoOkHevKtc8R0Q5MlTQC+ClwaM5NKhlJZwArIuJhSSfl3Z4+dkJELJY0Gpgt6Q+FK0v98+0zm30z2B4/vVzSWID0dUUqL4vvg6QqsqD5QUT8JBWXdZ8LRcQ64C6yYaQRkjr/GC3s245+p/XDgdV93NR98SbgnZIWADeRDaV9nfLt7w4RsTh9XUH2R8Wx9OHPt8Nm3wy2x0/PAmam5Zlk1zU6y9+fZrBMB9YXnJoPCMpOYa4BnomIrxasKts+A0hqSWc0SKoju071DFnonJ2qde135/fjbODOSIP6A0FEXBIR+0fERLL/r3dGxHsp0/52klQvqaFzGTgVeIq+/PnO+6LVQH8BpwPPkY1z/2Pe7Sliv24ElgLbycZrLyAbq74DmAf8GmhKdUU2K+954ElgWt7t34v+nkA2pv0E8Fh6nV7OfU79OAJ4NPX7KeBfUvlBwIPAfOD/ATWpvDa9n5/WH5R3H/ah7ycBPx8M/U39ezy95nb+rurLn2/frsbMzErOw2hmZlZyDhszMys5h42ZmZWcw8bMzErOYWNmZiXnsDErEkmb0teJkv68yPv+bJf3vy/m/s1KzWFjVnwTgT0Km4JPr+/KTmETEW/cwzaZ5cphY1Z8lwNvTs8N+et0o8svS3ooPRvkowCSTpL0W0mzgKdT2c/SjRLndt4sUdLlQF3a3w9SWedZlNK+n0rPKjmvYN93S7pZ0h8k/SDdJQFJlyt7bs8Tkv6zz787Nij5RpxmxXcx8LcRcQZACo31EXGMpBrgXkm3p7pHAYdHxIvp/YciYk26dcxDkn4cERdL+kRETO3mWO8mew7NkUBz2uaetO4NwGHAEuBe4E2SngHeBRwaEdF5qxqzUvOZjVnpnUp2n6nHyB5bMIrsoVQADxYEDcBfSXocuJ/sRoiT6dkJwI0R0R4Ry4HfAMcU7HtRRHSQ3X5nItkt8rcB10h6N7Bln3tn1gsOG7PSE/DJyJ6QODUiJkVE55nN5h2Vslvev43sYV1Hkt2zrHYfjttasNxO9nCwNrK7/d4MnAH8ah/2b9ZrDhuz4tsINBS8vw34y/QIAyQdku6829VwskcQb5F0KNnjqTtt79y+i98C56XrQi3AiWQ3jOxWel7P8Ii4FfhrsuE3s5LzNRuz4nsCaE/DYdeRPS9lIvBIuki/Ejirm+1+BXwsXVd5lmwordPVwBOSHonslvidfkr2/JnHye5a/fcRsSyFVXcagFsk1ZKdcf3N3nXRbM/4rs9mZlZyHkYzM7OSc9iYmVnJOWzMzKzkHDZmZlZyDhszMys5h42ZmZWcw8bMzEru/wPNleTIfvYoJAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Plot gradient descent progress.\n", "plt.plot(range(num_iterations), cost_history)\n", "plt.xlabel('Iterations')\n", "plt.ylabel('Cost')\n", "plt.title('Gradient Descent Progress')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Plot the Model Predictions\n", "\n", "Since our model is trained now we may plot its predictions over the training and test datasets to see how well it fits the data." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "data": [ { "marker": { "line": { "color": "rgb(255, 255, 255)", "width": 1 }, "opacity": 1, "size": 10 }, "mode": "markers", "name": "Training Set", "type": "scatter3d", "uid": "5121ac59-e6e4-4c30-8d0c-c3482182891a", "x": [ 1.1027104854583702, 1.3208793401718102, 0.7922212481498722, 0.39724862575531, 0.9097844958305359, 1.1893955469131499, 0.401477217674255, 1.06457793712616, 0.479820191860199, 0.368745893239975, 1.00985014438629, 0.381430715322495, 1.07062232494354, 1.12786877155304, 0.6364067792892459, 0.564305365085602, 0.932537317276001, 1.28948748111725, 1.29178786277771, 1.00726580619812, 0.8584281802177429, 0.30544471740722695, 1.1018030643463101, 1.63295245170593, 0.7288706302642819, 1.49438726902008, 1.8707656860351598, 1.3469113111496, 0, 1.05469870567322, 0.951484382152557, 0.5916834473609921, 0.982409417629242, 1.15687310695648, 1.4637807607650801, 1.48238301277161, 1.69227766990662, 0.43108540773391707, 1.3753824234008798, 0.0921023488044739, 1.12112903594971, 0.786441087722778, 1.00082039833069, 0.24454993009567302, 0.36711055040359497, 0.511135876178741, 1.41691517829895, 1.4870972633361799, 1.1531838178634601, 0.7268835306167599, 1.1073532104492199, 0.11904179304838199, 1.2860119342803997, 1.15655755996704, 0.716249227523804, 1.22255623340607, 0.667224824428558, 0.368610262870789, 1.0352252721786501, 0.995538592338562, 1.74194359779358, 1.6263433694839498, 1.12209415435791, 0.6595166921615601, 1.5649795532226598, 1.4844149351120002, 0.925579309463501, 1.2175596952438401, 1.36135590076447, 1.23374843597412, 1.12843120098114, 1.28177809715271, 1.29121541976929, 1.07498753070831, 0.85769921541214, 1.480633020401, 1.1536017656326298, 0.6484572887420649, 0.89465194940567, 0.872001945972443, 0.950612664222717, 1.43092346191406, 1.1614590883255, 1.10970628261566, 1.21768391132355, 0.8089642524719242, 0.9910123944282528, 0.305808693170547, 1.3412059545516999, 0.37584653496742204, 0.339233845472336, 1.40570604801178, 1.0272358655929599, 0.900596737861633, 1.1982743740081798, 1.40167844295502, 1.3559380769729599, 0.560479462146759, 1.07937383651733, 1.48792338371277, 1.3145823478698702, 0.996192753314972, 1.3950666189193701, 0.964434325695038, 1.25278460979462, 1.55167484283447, 1.50394463539124, 1.44163393974304, 1.18529546260834, 1.54625928401947, 1.5357066392898602, 1.1307767629623402, 0.6017650961875921, 0.524713635444641, 0.7885475754737851, 0.0226431842893362, 1.44357192516327, 1.53062355518341, 0.730573117733002, 1.1982102394104, 0.737299203872681, 1.08116579055786, 1.2845562696456898, 0.233442038297653 ], "y": [ 0.288555532693863, 0.479131430387497, 0.469987004995346, 0.147062435746193, 0.432452529668808, 0.491247326135635, 0.106179520487785, 0.325905978679657, 0.44030594825744607, 0.5818438529968261, 0.561213254928589, 0.443185955286026, 0.47748741507530196, 0.580200731754303, 0.461603492498398, 0.430388748645782, 0.473507791757584, 0.0957312509417534, 0.520342111587524, 0.289680689573288, 0, 0.38042613863945, 0.465733230113983, 0.49633759260177596, 0.24072904884815197, 0.612924098968506, 0.604130983352661, 0.47120362520217896, 0.270842045545578, 0.479246735572815, 0.260287940502167, 0.24946372210979503, 0.204403176903725, 0.24932260811328896, 0.5397707223892211, 0.626006722450256, 0.549840569496155, 0.42596277594566295, 0.40598860383033797, 0.235961347818375, 0.194989055395126, 0.6582486629486078, 0.4551981985569, 0.348587512969971, 0.514492034912109, 0.390017777681351, 0.505625545978546, 0.567766189575195, 0.412730008363724, 0.23521526157856, 0.437453746795654, 0.33288118243217496, 0.17586351931095098, 0.295400261878967, 0.25471106171608, 0.255772292613983, 0.423026293516159, 0.0303698573261499, 0.45000287890434293, 0.443323463201523, 0.59662789106369, 0.60834527015686, 0.505196332931519, 0.0149958552792668, 0.620070576667786, 0.601607382297516, 0.474307239055634, 0.5793922543525699, 0.518630743026733, 0.550026834011078, 0.15399712324142498, 0.373783111572266, 0.40226498246192893, 0.28851598501205394, 0.585214674472809, 0.6271626353263849, 0.39815583825111406, 0.0960980430245399, 0.12297477573156401, 0.5313106179237371, 0.309410035610199, 0.470222115516663, 0.28923171758651695, 0.580131649971008, 0.457003742456436, 0.4350258708000179, 0.418421149253845, 0.18919676542282102, 0.572575807571411, 0.336384207010269, 0.408842742443085, 0.6140621304512021, 0.39414396882057207, 0.198303267359734, 0.300740599632263, 0.257921665906906, 0.35511153936386103, 0.45276376605033897, 0.55258983373642, 0.562511384487152, 0.234231784939766, 0.381498634815216, 0.256450712680817, 0.520303547382355, 0.376895278692245, 0.490968644618988, 0.5853844881057739, 0.508190035820007, 0.49451920390129106, 0.505740523338318, 0.5731103420257571, 0.41827192902565, 0.633375823497772, 0.47156670689582797, 0.571055591106415, 0.602126955986023, 0.6179508566856379, 0.449750572443008, 0.348079860210419, 0.31232857704162603, 0.447551846504211, 0.47278770804405196, 0.43745428323745705, 0.466914653778076 ], "z": [ 4.49700021743774, 5.61100006103516, 4.31500005722046, 3.5910000801086404, 6.002999782562259, 5.62900018692017, 3.7939999103546103, 5.175000190734861, 4.961999893188481, 3.47099995613098, 4.44000005722046, 4.08099985122681, 6.35699987411499, 6.4239997863769505, 4.513999938964839, 4.69500017166138, 5.493000030517581, 5.2270002365112305, 5.97300004959106, 4.80499982833862, 3.79500007629395, 3.4949998855590803, 5.525000095367429, 6.10500001907349, 5.837999820709231, 7.28399991989136, 6.375, 5.80999994277954, 2.69300007820129, 4.8289999961853, 5.27899980545044, 3.59299993515015, 5.18200016021729, 4.69199991226196, 6.89099979400635, 7.52199983596802, 6.57200002670288, 3.65700006484985, 7.212999820709231, 4.2800002098083505, 5.23699998855591, 5.97100019454956, 6.00799989700317, 3.50699996948242, 4.54500007629395, 3.34899997711182, 5.92000007629395, 7.00600004196167, 6.57800006866455, 5.26900005340576, 6.6350002288818395, 3.5329999923706095, 5.32399988174438, 5.5689997673034695, 4.77500009536743, 5.2930002212524405, 4.11999988555908, 3.6029999256133998, 5.71500015258789, 5.26200008392334, 6.86299991607666, 6.6479997634887695, 3.7660000324249303, 4.138999938964839, 7.49399995803833, 7.28399991989136, 5.31099987030029, 6.4539999961853, 6.1680002212524405, 6.4520001411438, 5.25, 5.962999820709231, 6.08400011062622, 5.22499990463257, 5.42999982833862, 7.50400018692017, 5.234000205993651, 4.29199981689453, 4.09600019454956, 6.4539999961853, 4.28599977493286, 6.44199991226196, 4.71400022506714, 7.0789999961853, 5.824999809265139, 4.29099988937378, 5.33599996566772, 3.64400005340576, 5.75799989700317, 3.875, 4.46000003814697, 7.31400012969971, 4.95499992370605, 4.37599992752075, 5.5, 5.837999820709231, 5.62099981307983, 4.55299997329712, 5.230000019073491, 6.9510002136230495, 5.90199995040894, 4.64400005340576, 5.96400022506714, 4.57399988174438, 6.65199995040894, 5.47200012207031, 7.3769998550415, 6.71400022506714, 6.59899997711182, 6.993000030517581, 6.9770002365112305, 5.82200002670288, 4.1680002212524405, 5.04099988937378, 5.07399988174438, 5.151000022888179, 7.468999862670901, 6.343999862670901, 5.1810002326965305, 4.46500015258789, 6.07100009918213, 5.2729997634887695, 5.8189997673034695, 3.97000002861023 ] }, { "marker": { "line": { "color": "rgb(255, 255, 255)", "width": 1 }, "opacity": 1, "size": 10 }, "mode": "markers", "name": "Test Set", "type": "scatter3d", "uid": "973829dd-d770-41f6-b456-046962af6151", "x": [ 1.6164631843566901, 1.47920441627502, 1.35268235206604, 1.34327983856201, 1.4336265325546298, 1.38439786434174, 1.32539355754852, 1.4884122610092199, 0.907975316047668, 1.09186446666718, 1.26074862480164, 0.833756566047668, 1.06931757926941, 0.8781145811080929, 1.3151752948761002, 0.783756256103516, 0.885416388511658, 0.5962200760841371, 0.989701807498932, 0.36842092871666, 0.586682975292206, 0.23430564999580397, 0.479309022426605, 0.476180493831635, 0.6030489206314089, 0.35022771358490007, 0.16192533075809498, 0.43801298737525896, 0.521021246910095, 0.777153134346008, 0.09162256866693501 ], "y": [ 0.635422587394714, 0.611100912094116, 0.49094617366790794, 0.588767051696777, 0.361466586589813, 0.408781230449677, 0.295817464590073, 0.536746919155121, 0.5475093722343439, 0.233335807919502, 0.32570791244506797, 0.5587329268455511, 0.20871552824974102, 0.408158332109451, 0.498465299606323, 0.394952565431595, 0.5015376806259161, 0.454943388700485, 0.28211015462875394, 0.318697690963745, 0.478356659412384, 0.48079109191894503, 0.37792226672172496, 0.306613743305206, 0.4477061927318571, 0.32436785101890603, 0.36365869641304, 0.16234202682972002, 0.390661299228668, 0.0815394446253777, 0.0599007532000542 ], "z": [ 7.537000179290769, 7.31599998474121, 6.60900020599365, 6.52699995040894, 6.42199993133545, 6.40299987792969, 6.09800004959106, 6.08699989318848, 5.955999851226809, 5.872000217437741, 5.849999904632571, 5.82299995422363, 5.39499998092651, 5.2350001335144, 5.19500017166138, 5.07399988174438, 5.01100015640259, 5.0040001869201705, 4.7350001335144, 4.70900011062622, 4.60799980163574, 4.550000190734861, 4.53499984741211, 4.19000005722046, 4.17999982833862, 4.03200006484985, 4.02799987792969, 3.9360001087188703, 3.80800008773804, 3.46199989318848, 2.90499997138977 ] }, { "marker": { "size": 1 }, "mode": "markers", "name": "Prediction Plane", "opacity": 0.8, "surfaceaxis": 2, "type": "scatter3d", "uid": "b9286cc7-12bf-4571-b516-f8bda6c3c119", "x": [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.20786285400390664, 0.20786285400390664, 0.20786285400390664, 0.20786285400390664, 0.20786285400390664, 0.20786285400390664, 0.20786285400390664, 0.20786285400390664, 0.20786285400390664, 0.20786285400390664, 0.4157257080078133, 0.4157257080078133, 0.4157257080078133, 0.4157257080078133, 0.4157257080078133, 0.4157257080078133, 0.4157257080078133, 0.4157257080078133, 0.4157257080078133, 0.4157257080078133, 0.6235885620117199, 0.6235885620117199, 0.6235885620117199, 0.6235885620117199, 0.6235885620117199, 0.6235885620117199, 0.6235885620117199, 0.6235885620117199, 0.6235885620117199, 0.6235885620117199, 0.8314514160156266, 0.8314514160156266, 0.8314514160156266, 0.8314514160156266, 0.8314514160156266, 0.8314514160156266, 0.8314514160156266, 0.8314514160156266, 0.8314514160156266, 0.8314514160156266, 1.0393142700195332, 1.0393142700195332, 1.0393142700195332, 1.0393142700195332, 1.0393142700195332, 1.0393142700195332, 1.0393142700195332, 1.0393142700195332, 1.0393142700195332, 1.0393142700195332, 1.2471771240234397, 1.2471771240234397, 1.2471771240234397, 1.2471771240234397, 1.2471771240234397, 1.2471771240234397, 1.2471771240234397, 1.2471771240234397, 1.2471771240234397, 1.2471771240234397, 1.4550399780273464, 1.4550399780273464, 1.4550399780273464, 1.4550399780273464, 1.4550399780273464, 1.4550399780273464, 1.4550399780273464, 1.4550399780273464, 1.4550399780273464, 1.4550399780273464, 1.662902832031253, 1.662902832031253, 1.662902832031253, 1.662902832031253, 1.662902832031253, 1.662902832031253, 1.662902832031253, 1.662902832031253, 1.662902832031253, 1.662902832031253, 1.8707656860351598, 1.8707656860351598, 1.8707656860351598, 1.8707656860351598, 1.8707656860351598, 1.8707656860351598, 1.8707656860351598, 1.8707656860351598, 1.8707656860351598, 1.8707656860351598 ], "y": [ 0, 0.07313874032762309, 0.14627748065524618, 0.21941622098286928, 0.29255496131049236, 0.36569370163811543, 0.43883244196573856, 0.5119711822933616, 0.5851099226209847, 0.6582486629486078, 0, 0.07313874032762309, 0.14627748065524618, 0.21941622098286928, 0.29255496131049236, 0.36569370163811543, 0.43883244196573856, 0.5119711822933616, 0.5851099226209847, 0.6582486629486078, 0, 0.07313874032762309, 0.14627748065524618, 0.21941622098286928, 0.29255496131049236, 0.36569370163811543, 0.43883244196573856, 0.5119711822933616, 0.5851099226209847, 0.6582486629486078, 0, 0.07313874032762309, 0.14627748065524618, 0.21941622098286928, 0.29255496131049236, 0.36569370163811543, 0.43883244196573856, 0.5119711822933616, 0.5851099226209847, 0.6582486629486078, 0, 0.07313874032762309, 0.14627748065524618, 0.21941622098286928, 0.29255496131049236, 0.36569370163811543, 0.43883244196573856, 0.5119711822933616, 0.5851099226209847, 0.6582486629486078, 0, 0.07313874032762309, 0.14627748065524618, 0.21941622098286928, 0.29255496131049236, 0.36569370163811543, 0.43883244196573856, 0.5119711822933616, 0.5851099226209847, 0.6582486629486078, 0, 0.07313874032762309, 0.14627748065524618, 0.21941622098286928, 0.29255496131049236, 0.36569370163811543, 0.43883244196573856, 0.5119711822933616, 0.5851099226209847, 0.6582486629486078, 0, 0.07313874032762309, 0.14627748065524618, 0.21941622098286928, 0.29255496131049236, 0.36569370163811543, 0.43883244196573856, 0.5119711822933616, 0.5851099226209847, 0.6582486629486078, 0, 0.07313874032762309, 0.14627748065524618, 0.21941622098286928, 0.29255496131049236, 0.36569370163811543, 0.43883244196573856, 0.5119711822933616, 0.5851099226209847, 0.6582486629486078, 0, 0.07313874032762309, 0.14627748065524618, 0.21941622098286928, 0.29255496131049236, 0.36569370163811543, 0.43883244196573856, 0.5119711822933616, 0.5851099226209847, 0.6582486629486078 ], "z": [ 3.6332939796459245, 3.757851868566506, 3.8824097574870877, 4.00696764640767, 4.131525535328251, 4.256083424248833, 4.380641313169415, 4.5051992020899965, 4.629757091010578, 4.75431497993116, 3.8998205174683562, 4.024378406388938, 4.14893629530952, 4.273494184230102, 4.398052073150683, 4.5226099620712645, 4.647167850991846, 4.771725739912428, 4.89628362883301, 5.020841517753592, 4.166347055290788, 4.29090494421137, 4.415462833131952, 4.5400207220525335, 4.664578610973114, 4.789136499893696, 4.913694388814278, 5.03825227773486, 5.162810166655442, 5.287368055576024, 4.432873593113219, 4.557431482033801, 4.6819893709543825, 4.806547259874964, 4.931105148795545, 5.055663037716127, 5.180220926636709, 5.304778815557291, 5.429336704477873, 5.5538945933984545, 4.699400130935651, 4.823958019856232, 4.948515908776814, 5.073073797697396, 5.197631686617977, 5.322189575538559, 5.446747464459141, 5.5713053533797225, 5.695863242300304, 5.820421131220886, 4.965926668758082, 5.090484557678664, 5.215042446599246, 5.339600335519828, 5.464158224440409, 5.588716113360991, 5.7132740022815725, 5.837831891202154, 5.962389780122736, 6.086947669043318, 5.232453206580514, 5.357011095501096, 5.481568984421678, 5.60612687334226, 5.7306847622628405, 5.855242651183422, 5.979800540104004, 6.104358429024586, 6.228916317945168, 6.35347420686575, 5.498979744402945, 5.623537633323527, 5.748095522244109, 5.87265341116469, 5.997211300085271, 6.121769189005853, 6.246327077926435, 6.370884966847017, 6.495442855767599, 6.620000744688181, 5.765506282225377, 5.8900641711459585, 6.01462206006654, 6.139179948987122, 6.263737837907703, 6.388295726828285, 6.512853615748867, 6.637411504669449, 6.7619693935900305, 6.886527282510612, 6.032032820047808, 6.15659070896839, 6.281148597888972, 6.405706486809554, 6.530264375730135, 6.654822264650717, 6.7793801535712985, 6.90393804249188, 7.028495931412462, 7.153053820333044 ] } ], "layout": { "margin": { "b": 0, "l": 0, "r": 0, "t": 0 }, "scene": { "xaxis": { "title": "Economy..GDP.per.Capita." }, "yaxis": { "title": "Freedom" }, "zaxis": { "title": "Happiness.Score" } }, "title": "Date Sets" } }, "text/html": [ "
" ], "text/vnd.plotly.v1+html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Generate different combinations of X and Y sets to build a predictions plane.\n", "predictions_num = 10\n", "\n", "# Find min and max values along X and Y axes.\n", "x_min = x_train[:, 0].min();\n", "x_max = x_train[:, 0].max();\n", "\n", "y_min = x_train[:, 1].min();\n", "y_max = x_train[:, 1].max();\n", "\n", "# Generate predefined numbe of values for eaxh axis betwing correspondent min and max values.\n", "x_axis = np.linspace(x_min, x_max, predictions_num)\n", "y_axis = np.linspace(y_min, y_max, predictions_num)\n", "\n", "# Create empty vectors for X and Y axes predictions\n", "# We're going to find cartesian product of all possible X and Y values.\n", "x_predictions = np.zeros((predictions_num * predictions_num, 1))\n", "y_predictions = np.zeros((predictions_num * predictions_num, 1))\n", "\n", "# Find cartesian product of all X and Y values.\n", "x_y_index = 0\n", "for x_index, x_value in enumerate(x_axis):\n", " for y_index, y_value in enumerate(y_axis):\n", " x_predictions[x_y_index] = x_value\n", " y_predictions[x_y_index] = y_value\n", " x_y_index += 1\n", "\n", "# Predict Z value for all X and Y pairs. \n", "z_predictions = linear_regression.predict(np.hstack((x_predictions, y_predictions)))\n", "\n", "# Plot training data with predictions.\n", "\n", "# Configure the plot with test dataset.\n", "plot_predictions_trace = go.Scatter3d(\n", " x=x_predictions.flatten(),\n", " y=y_predictions.flatten(),\n", " z=z_predictions.flatten(),\n", " name='Prediction Plane',\n", " mode='markers',\n", " marker={\n", " 'size': 1,\n", " },\n", " opacity=0.8,\n", " surfaceaxis=2, \n", ")\n", "\n", "plot_data = [plot_training_trace, plot_test_trace, plot_predictions_trace]\n", "plot_figure = go.Figure(data=plot_data, layout=plot_layout)\n", "plotly.offline.iplot(plot_figure)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Calculate the value of cost function for the training and test data set. The less this value is, the better." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train cost: 2761.58\n", "Test cost: 105.85\n" ] } ], "source": [ "train_cost = linear_regression.get_cost(x_train, y_train, regularization_param)\n", "test_cost = linear_regression.get_cost(x_test, y_test, regularization_param)\n", "\n", "print('Train cost: {:.2f}'.format(train_cost))\n", "print('Test cost: {:.2f}'.format(test_cost))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's now render the table of prediction values that our trained model does for unknown data (for test dataset). You should see that predicted happiness score should be quite similar to the known happiness score fron the test dataset." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Economy GDP per CapitaFreedomTest Happiness ScorePredicted Happiness ScorePrediction Diff
01.6164630.6354237.5377.3020780.234922
11.4792040.6111017.3167.0006830.315317
21.3526820.4909466.6096.4791390.129861
31.3432800.5887676.5276.706585-0.179585
41.4336270.3614676.4226.2982260.123774
51.3843980.4087816.4036.3298710.073129
61.3253940.2958176.0985.9446810.153319
71.4884120.5367476.0876.831415-0.744415
80.9079750.5475095.9565.8402010.115799
91.0918640.2333365.8725.3792730.492727
\n", "
" ], "text/plain": [ " Economy GDP per Capita Freedom Test Happiness Score \\\n", "0 1.616463 0.635423 7.537 \n", "1 1.479204 0.611101 7.316 \n", "2 1.352682 0.490946 6.609 \n", "3 1.343280 0.588767 6.527 \n", "4 1.433627 0.361467 6.422 \n", "5 1.384398 0.408781 6.403 \n", "6 1.325394 0.295817 6.098 \n", "7 1.488412 0.536747 6.087 \n", "8 0.907975 0.547509 5.956 \n", "9 1.091864 0.233336 5.872 \n", "\n", " Predicted Happiness Score Prediction Diff \n", "0 7.302078 0.234922 \n", "1 7.000683 0.315317 \n", "2 6.479139 0.129861 \n", "3 6.706585 -0.179585 \n", "4 6.298226 0.123774 \n", "5 6.329871 0.073129 \n", "6 5.944681 0.153319 \n", "7 6.831415 -0.744415 \n", "8 5.840201 0.115799 \n", "9 5.379273 0.492727 " ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test_predictions = linear_regression.predict(x_test)\n", "\n", "test_predictions_table = pd.DataFrame({\n", " 'Economy GDP per Capita': x_test[:, 0].flatten(),\n", " 'Freedom': x_test[:, 1].flatten(),\n", " 'Test Happiness Score': y_test.flatten(),\n", " 'Predicted Happiness Score': test_predictions.flatten(),\n", " 'Prediction Diff': (y_test - test_predictions).flatten()\n", "})\n", "\n", "test_predictions_table.head(10)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.0" } }, "nbformat": 4, "nbformat_minor": 2 }