{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Linear Regression"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Environment setup"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Python version: 3.7.5\n",
"NumPy version: 1.18.1\n"
]
}
],
"source": [
"import platform\n",
"\n",
"print(f\"Python version: {platform.python_version()}\")\n",
"assert platform.python_version_tuple() >= (\"3\", \"6\")\n",
"\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"import plotly\n",
"import plotly.graph_objs as go\n",
"import pandas as pd\n",
"\n",
"print(f\"NumPy version: {np.__version__}\")"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Setup plots\n",
"%matplotlib inline\n",
"plt.rcParams[\"figure.figsize\"] = 10, 8\n",
"%config InlineBackend.figure_format = \"retina\"\n",
"sns.set()\n",
"\n",
"# Configure Plotly to be rendered inline in the notebook.\n",
"plotly.offline.init_notebook_mode()"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"scikit-learn version: 0.22.1\n"
]
}
],
"source": [
"import sklearn\n",
"\n",
"print(f\"scikit-learn version: {sklearn.__version__}\")\n",
"assert sklearn.__version__ >= \"0.20\"\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.linear_model import LinearRegression, SGDRegressor, Ridge\n",
"from sklearn.metrics import mean_squared_error\n",
"from sklearn.preprocessing import PolynomialFeatures, StandardScaler\n",
"from sklearn.pipeline import Pipeline"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Introduction"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Linear regression in a nutshell\n",
"\n",
"A linear regression model searches for a linear relationship between inputs (features) and output (target).\n",
"\n",
"[](https://en.wikipedia.org/wiki/Linear_regression)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Problem formulation\n",
"\n",
"A linear model makes a prediction by computing a weighted sum of the input features, plus a constant term called *bias* (or sometimes *intercept*).\n",
"\n",
"$$y' = \\theta_0 + \\theta_1x_1 + \\dotsc + \\theta_nx_n$$\n",
"\n",
"- $y'$ is the model's output (prediction).\n",
"- $n$ is the number of features for a data sample.\n",
"- $x_i, i \\in [1..n]$ is the value of the $i$th feature.\n",
"- $\\theta_i, i \\in [0..n]$ is $i$th model parameter. $\\theta_0$ is the bias term.\n",
"\n",
"The goal is to find the best set of parameters."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Example: using a linear model to predict country happiness\n",
"\n",
"(Inspired by [Homemade Machine Learning](https://github.com/trekhleb/homemade-machine-learning) by Oleksii Trekhleb)\n",
"\n",
"The [World Happiness Report](https://www.kaggle.com/unsdsn/world-happiness#2017.csv) ranks 155 countries by their happiness levels. Several economic and social indicators (GDP, degree of freedom, level of corruption...) are recorded for each country.\n",
"\n",
"Can a linear model accurately predict country happiness based on these indicators ?"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Data loading and analysis"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Dataset shape: (155, 12)\n"
]
}
],
"source": [
"# Load World Happiness Report for 2017\n",
"dataset_url = \"https://raw.githubusercontent.com/bpesquet/mlhandbook/master/_datasets/world-happiness-report-2017.csv\"\n",
"df_happiness = pd.read_csv(dataset_url)\n",
"\n",
"# Print dataset shape (rows and columns)\n",
"print(f\"Dataset shape: {df_happiness.shape}\")"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"RangeIndex: 155 entries, 0 to 154\n",
"Data columns (total 12 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 Country 155 non-null object \n",
" 1 Happiness.Rank 155 non-null int64 \n",
" 2 Happiness.Score 155 non-null float64\n",
" 3 Whisker.high 155 non-null float64\n",
" 4 Whisker.low 155 non-null float64\n",
" 5 Economy..GDP.per.Capita. 155 non-null float64\n",
" 6 Family 155 non-null float64\n",
" 7 Health..Life.Expectancy. 155 non-null float64\n",
" 8 Freedom 155 non-null float64\n",
" 9 Generosity 155 non-null float64\n",
" 10 Trust..Government.Corruption. 155 non-null float64\n",
" 11 Dystopia.Residual 155 non-null float64\n",
"dtypes: float64(10), int64(1), object(1)\n",
"memory usage: 14.7+ KB\n"
]
}
],
"source": [
"# Print a concise summary of the dataset\n",
"df_happiness.info()"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Country | \n",
" Happiness.Rank | \n",
" Happiness.Score | \n",
" Whisker.high | \n",
" Whisker.low | \n",
" Economy..GDP.per.Capita. | \n",
" Family | \n",
" Health..Life.Expectancy. | \n",
" Freedom | \n",
" Generosity | \n",
" Trust..Government.Corruption. | \n",
" Dystopia.Residual | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" Norway | \n",
" 1 | \n",
" 7.537 | \n",
" 7.594445 | \n",
" 7.479556 | \n",
" 1.616463 | \n",
" 1.533524 | \n",
" 0.796667 | \n",
" 0.635423 | \n",
" 0.362012 | \n",
" 0.315964 | \n",
" 2.277027 | \n",
"
\n",
" \n",
" | 1 | \n",
" Denmark | \n",
" 2 | \n",
" 7.522 | \n",
" 7.581728 | \n",
" 7.462272 | \n",
" 1.482383 | \n",
" 1.551122 | \n",
" 0.792566 | \n",
" 0.626007 | \n",
" 0.355280 | \n",
" 0.400770 | \n",
" 2.313707 | \n",
"
\n",
" \n",
" | 2 | \n",
" Iceland | \n",
" 3 | \n",
" 7.504 | \n",
" 7.622030 | \n",
" 7.385970 | \n",
" 1.480633 | \n",
" 1.610574 | \n",
" 0.833552 | \n",
" 0.627163 | \n",
" 0.475540 | \n",
" 0.153527 | \n",
" 2.322715 | \n",
"
\n",
" \n",
" | 3 | \n",
" Switzerland | \n",
" 4 | \n",
" 7.494 | \n",
" 7.561772 | \n",
" 7.426227 | \n",
" 1.564980 | \n",
" 1.516912 | \n",
" 0.858131 | \n",
" 0.620071 | \n",
" 0.290549 | \n",
" 0.367007 | \n",
" 2.276716 | \n",
"
\n",
" \n",
" | 4 | \n",
" Finland | \n",
" 5 | \n",
" 7.469 | \n",
" 7.527542 | \n",
" 7.410458 | \n",
" 1.443572 | \n",
" 1.540247 | \n",
" 0.809158 | \n",
" 0.617951 | \n",
" 0.245483 | \n",
" 0.382612 | \n",
" 2.430182 | \n",
"
\n",
" \n",
" | 5 | \n",
" Netherlands | \n",
" 6 | \n",
" 7.377 | \n",
" 7.427426 | \n",
" 7.326574 | \n",
" 1.503945 | \n",
" 1.428939 | \n",
" 0.810696 | \n",
" 0.585384 | \n",
" 0.470490 | \n",
" 0.282662 | \n",
" 2.294804 | \n",
"
\n",
" \n",
" | 6 | \n",
" Canada | \n",
" 7 | \n",
" 7.316 | \n",
" 7.384403 | \n",
" 7.247597 | \n",
" 1.479204 | \n",
" 1.481349 | \n",
" 0.834558 | \n",
" 0.611101 | \n",
" 0.435540 | \n",
" 0.287372 | \n",
" 2.187264 | \n",
"
\n",
" \n",
" | 7 | \n",
" New Zealand | \n",
" 8 | \n",
" 7.314 | \n",
" 7.379510 | \n",
" 7.248490 | \n",
" 1.405706 | \n",
" 1.548195 | \n",
" 0.816760 | \n",
" 0.614062 | \n",
" 0.500005 | \n",
" 0.382817 | \n",
" 2.046456 | \n",
"
\n",
" \n",
" | 8 | \n",
" Sweden | \n",
" 9 | \n",
" 7.284 | \n",
" 7.344095 | \n",
" 7.223905 | \n",
" 1.494387 | \n",
" 1.478162 | \n",
" 0.830875 | \n",
" 0.612924 | \n",
" 0.385399 | \n",
" 0.384399 | \n",
" 2.097538 | \n",
"
\n",
" \n",
" | 9 | \n",
" Australia | \n",
" 10 | \n",
" 7.284 | \n",
" 7.356651 | \n",
" 7.211349 | \n",
" 1.484415 | \n",
" 1.510042 | \n",
" 0.843887 | \n",
" 0.601607 | \n",
" 0.477699 | \n",
" 0.301184 | \n",
" 2.065211 | \n",
"
\n",
" \n",
"
\n",
"
"
]
},
"metadata": {
"image/png": {
"height": 701,
"width": 928
}
},
"output_type": "display_data"
}
],
"source": [
"# Plot histograms for all numerical attributes\n",
"df_happiness.hist(bins=20, figsize=(16, 12))\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Univariate regression\n",
"\n",
"Only one feature is used by the model, which has two parameters.\n",
"\n",
"$$y' = \\theta_0 + \\theta_1x$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Example: predict country happiness using only GDP"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"outputs": [],
"source": [
"def filter_dataset(df_data, input_features, target_feature):\n",
" \"\"\"Return a dataset containing only the selected input and output features\"\"\"\n",
" \n",
" feature_list = input_features + [target_feature]\n",
" return df_data[feature_list]"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [],
"source": [
"# Define GDP as sole input feature\n",
"input_features_uni = [\"Economy..GDP.per.Capita.\"]\n",
"# Define country happiness as target\n",
"target_feature = \"Happiness.Score\"\n",
"\n",
"df_happiness_uni = filter_dataset(df_happiness, input_features_uni, target_feature)"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Economy..GDP.per.Capita. | \n",
" Happiness.Score | \n",
"
\n",
" \n",
" \n",
" \n",
" | 89 | \n",
" 0.982409 | \n",
" 5.182 | \n",
"
\n",
" \n",
" | 146 | \n",
" 0.397249 | \n",
" 3.591 | \n",
"
\n",
" \n",
" | 121 | \n",
" 0.792221 | \n",
" 4.315 | \n",
"
\n",
" \n",
" | 67 | \n",
" 1.101803 | \n",
" 5.525 | \n",
"
\n",
" \n",
" | 97 | \n",
" 0.596220 | \n",
" 5.004 | \n",
"
\n",
" \n",
" | 123 | \n",
" 0.808964 | \n",
" 4.291 | \n",
"
\n",
" \n",
" | 32 | \n",
" 1.433627 | \n",
" 6.422 | \n",
"
\n",
" \n",
" | 93 | \n",
" 0.788548 | \n",
" 5.074 | \n",
"
\n",
" \n",
" | 26 | \n",
" 1.343280 | \n",
" 6.527 | \n",
"
\n",
" \n",
" | 78 | \n",
" 1.081166 | \n",
" 5.273 | \n",
"
\n",
" \n",
"
\n",
"