{ "cells": [ { "cell_type": "markdown", "id": "8d8584aa", "metadata": {}, "source": [ "# Lesson 19: Regression demonstration\n", "\n", "Supervised machine learning is the task of learning a mapping from inputs to outputs using labeled training examples. Each example (or observation) pairs a set of input features with a known target value. We will use linear regression to explore the core concepts of model training, evaluation, and regularization, implemented with [scikit-learn](https://scikit-learn.org/stable/); the standard Python library for machine learning.\n", "\n", "1. **What is regression**\n", " - Regression problems\n", " - Model training\n", "\n", "2. **Regression metrics**\n", " - Residuals: RSS, MSE, RMSE and MAE\n", " - R-squared\n", "\n", "3. **Evaluating regression model performance**\n", " - Train-test split\n", " - Overfitting\n", "\n", "4. **Dealing with overfitting**\n", " - Ridge (L2) penalty\n", " - Lasso (L1) penalty\n", "\n", "5. **Optimizing regression models**\n", " - Hyperparameter tuning\n", " - Grid search/Randomized search\n", "\n", "6. **Scikit-learn pipelines example**" ] }, { "cell_type": "markdown", "id": "49f87fdc", "metadata": {}, "source": [ "## Notebook set up\n", "### Import libraries" ] }, { "cell_type": "code", "execution_count": 1, "id": "d839e96b", "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pandas as pd" ] }, { "cell_type": "markdown", "id": "a3b87cd6", "metadata": {}, "source": [ "## 1. What is regression?\n", "\n", "A regression model predicts the value of a continuous output variable (label) from one or more input variables (features).\n", "\n", "### 1.1. Regression problem: circumference of a circle" ] }, { "cell_type": "code", "execution_count": 2, "id": "9742c621", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
| \n", " | diameter | \n", "circumference | \n", "
|---|---|---|
| 0 | \n", "3.0 | \n", "9.1 | \n", "
| 1 | \n", "3.2 | \n", "10.2 | \n", "
| 2 | \n", "3.6 | \n", "11.0 | \n", "
| 3 | \n", "4.8 | \n", "14.8 | \n", "
| 4 | \n", "6.1 | \n", "19.4 | \n", "
| 5 | \n", "7.2 | \n", "22.1 | \n", "
| 6 | \n", "8.7 | \n", "27.2 | \n", "
| 7 | \n", "10.0 | \n", "31.2 | \n", "
| 8 | \n", "11.3 | \n", "35.3 | \n", "
| 9 | \n", "11.9 | \n", "37.3 | \n", "
| \n", " | area | \n", "temperature | \n", "humidity | \n", "wind_speed | \n", "rainfall | \n", "quality | \n", "area_log | \n", "area_sqrt | \n", "area_power | \n", "area_quantile_uniform | \n", "... | \n", "hash_6 hash_7 | \n", "hash_6 hash_8 | \n", "hash_6 hash_9 | \n", "hash_7^2 | \n", "hash_7 hash_8 | \n", "hash_7 hash_9 | \n", "hash_8^2 | \n", "hash_8 hash_9 | \n", "hash_9^2 | \n", "distance_to_water | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "1.049357 | \n", "2.109251 | \n", "0.809476 | \n", "0.744979 | \n", "-1.446606 | \n", "-1.90156 | \n", "1.021565 | \n", "1.073922 | \n", "1.061607 | \n", "1.226256 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "-0.318691 | \n", "0.0 | \n", "0.0 | \n", "3.000000 | \n", "0.0 | \n", "-0.327001 | \n", "95.888293 | \n", "
| 1 | \n", "2.145387 | \n", "-1.237192 | \n", "0.751604 | \n", "0.854545 | \n", "-0.601307 | \n", "0.32744 | \n", "1.492859 | \n", "1.836667 | \n", "1.688007 | \n", "1.589323 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "-0.318691 | \n", "0.0 | \n", "0.0 | \n", "-0.318311 | \n", "0.0 | \n", "-0.327001 | \n", "32.734564 | \n", "
| 2 | \n", "-0.468882 | \n", "0.115644 | \n", "-0.840557 | \n", "-0.302526 | \n", "-1.364800 | \n", "0.32744 | \n", "-0.174431 | \n", "-0.349037 | \n", "-0.276483 | \n", "-0.382535 | \n", "... | \n", "-0.0 | \n", "0.0 | \n", "0.0 | \n", "3.000000 | \n", "-0.0 | \n", "-0.0 | \n", "-0.318311 | \n", "0.0 | \n", "-0.327001 | \n", "13.430377 | \n", "
| 3 | \n", "0.214393 | \n", "-1.053487 | \n", "1.214892 | \n", "1.284614 | \n", "-0.209732 | \n", "0.32744 | \n", "0.498002 | \n", "0.371657 | \n", "0.434467 | \n", "0.560494 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "-0.318691 | \n", "0.0 | \n", "0.0 | \n", "-0.318311 | \n", "0.0 | \n", "-0.327001 | \n", "37.928220 | \n", "
| 4 | \n", "0.348350 | \n", "0.665536 | \n", "1.499702 | \n", "-1.150032 | \n", "-1.313338 | \n", "0.32744 | \n", "0.597342 | \n", "0.494455 | \n", "0.548238 | \n", "0.692516 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "-0.318691 | \n", "0.0 | \n", "0.0 | \n", "-0.318311 | \n", "0.0 | \n", "-0.327001 | \n", "577.643954 | \n", "
5 rows × 465 columns
\n", "| \n", " | area | \n", "temperature | \n", "humidity | \n", "wind_speed | \n", "rainfall | \n", "quality | \n", "area_log | \n", "area_sqrt | \n", "area_power | \n", "area_quantile_uniform | \n", "... | \n", "hash_6 hash_7 | \n", "hash_6 hash_8 | \n", "hash_6 hash_9 | \n", "hash_7^2 | \n", "hash_7 hash_8 | \n", "hash_7 hash_9 | \n", "hash_8^2 | \n", "hash_8 hash_9 | \n", "hash_9^2 | \n", "distance_to_water | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2855 | \n", "2.254697 | \n", "-0.597214 | \n", "0.801857 | \n", "-1.307860 | \n", "0.094637 | \n", "0.32744 | \n", "1.531775 | \n", "1.905947 | \n", "1.742546 | \n", "1.603443 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "-0.318691 | \n", "0.0 | \n", "0.0 | \n", "-0.318311 | \n", "0.0 | \n", "-0.327001 | \n", "181.512517 | \n", "
| 1178 | \n", "-0.068005 | \n", "1.826612 | \n", "0.727853 | \n", "-1.247204 | \n", "1.347683 | \n", "0.32744 | \n", "0.260186 | \n", "0.095706 | \n", "0.171522 | \n", "0.223567 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "-0.318691 | \n", "0.0 | \n", "0.0 | \n", "-0.318311 | \n", "0.0 | \n", "-0.327001 | \n", "130.971723 | \n", "
| 4433 | \n", "0.084802 | \n", "-1.436764 | \n", "-1.755434 | \n", "1.862404 | \n", "0.015015 | \n", "1.44194 | \n", "0.394179 | \n", "0.248139 | \n", "0.318063 | \n", "0.414276 | \n", "... | \n", "-0.0 | \n", "-0.0 | \n", "-0.0 | \n", "-0.318691 | \n", "0.0 | \n", "0.0 | \n", "-0.318311 | \n", "0.0 | \n", "-0.327001 | \n", "67.776354 | \n", "
| 195 | \n", "1.208175 | \n", "2.158445 | \n", "0.056115 | \n", "-0.215953 | \n", "1.492728 | \n", "-0.78706 | \n", "1.101242 | \n", "1.193507 | \n", "1.163215 | \n", "1.307456 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "-0.318691 | \n", "0.0 | \n", "0.0 | \n", "-0.318311 | \n", "0.0 | \n", "-0.327001 | \n", "105.835494 | \n", "
| 2372 | \n", "-0.628039 | \n", "0.755211 | \n", "-1.415713 | \n", "-0.379636 | \n", "0.205195 | \n", "-0.78706 | \n", "-0.396722 | \n", "-0.549977 | \n", "-0.490346 | \n", "-0.632159 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "-0.0 | \n", "-0.318691 | \n", "0.0 | \n", "-0.0 | \n", "-0.318311 | \n", "-0.0 | \n", "3.000000 | \n", "164.903483 | \n", "
5 rows × 465 columns
\n", "Pipeline(steps=[('Preprocessor',\n",
" ColumnTransformer(transformers=[('Numerical',\n",
" Pipeline(steps=[('scaler',\n",
" StandardScaler())]),\n",
" ['area', 'temperature',\n",
" 'humidity', 'wind_speed',\n",
" 'rainfall']),\n",
" ('Categorical',\n",
" Pipeline(steps=[('encoder',\n",
" OneHotEncoder(drop='first',\n",
" sparse_output=False))]),\n",
" ['quality', 'region'])])),\n",
" ('Model',\n",
" Lasso(alpha=np.float64(4.2427939582759056), max_iter=20000))])In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. ['area', 'temperature', 'humidity', 'wind_speed', 'rainfall']
| \n", " | \n",
" \n",
" copy\n",
" \n",
" copy: bool, default=True If False, try to avoid a copy and do inplace scaling instead. This is not guaranteed to always work inplace; e.g. if the data is not a NumPy array or scipy.sparse CSR matrix, a copy may still be returned.\n", " \n", " | \n",
" True | \n", "
| \n", " | \n",
" \n",
" with_mean\n",
" \n",
" with_mean: bool, default=True If True, center the data before scaling. This does not work (and will raise an exception) when attempted on sparse matrices, because centering them entails building a dense matrix which in common use cases is likely to be too large to fit in memory.\n", " \n", " | \n",
" True | \n", "
| \n", " | \n",
" \n",
" with_std\n",
" \n",
" with_std: bool, default=True If True, scale the data to unit variance (or equivalently, unit standard deviation).\n", " \n", " | \n",
" True | \n", "
['quality', 'region']