{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multiple linear regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now investigate **multiple linear regression** (MLR), where several output depends on more than one input variable:\n",
    "\n",
    "$$\n",
    "\\begin{cases}\n",
    "y_1 = w_1 \\, x_1 + w_2 \\, x_2 + b_1\\\\\n",
    "\\\\\n",
    "y_2 = w_3 \\, x_1 + w_4 \\, x_2 + b_2\\\\\n",
    "\\end{cases}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The California Housing Dataset consists of price of houses in various places in California. Alongside with the price of over 20000 houses, the dataset provides 8 features:\n",
    "\n",
    "- `MedInc`       : median income in block group\n",
    "- `HouseAge`     : median house age in block group\n",
    "- `AveRooms`     : average number of rooms per household\n",
    "- `AveBedrms`    : average number of bedrooms per household\n",
    "- `Population`   : block group population\n",
    "- `AveOccup`     : average number of household members\n",
    "- `Latitude`     : block group latitude\n",
    "- `Longitude`    : block group longitude"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The California housing dataset can be directly downloaded from scikit-learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.datasets import fetch_california_housing\n",
    "\n",
    "dataset = fetch_california_housing()\n",
    "\n",
    "X = dataset.data\n",
    "t = dataset.target\n",
    "\n",
    "print(X.shape)\n",
    "print(t.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 20640 samples with 8 input features and one output (the price). The following cell decribes the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(dataset.DESCR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell allows to visualize how each feature influences the price individually:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12, 15))\n",
    "\n",
    "for i in range(8):\n",
    "    plt.subplot(4, 2 , i+1)\n",
    "    plt.scatter(X[:, i], t)\n",
    "    plt.title(dataset.feature_names[i])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear regression\n",
    "\n",
    "**Q:** Apply MLR on the California data using the same `LinearRegression` method of `scikit-learn` as last time. Print the mse, plot how the predictions predict the price for each feature,  and plot the prediction $y$ against the true value $t$ for each sample as in the last exercise. Does it work well?\n",
    "\n",
    "You will also plot the weights of the model (`reg.coef_`) and conclude on the relative importance of the different features: which feature has the stronger weight and why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A good practice in machine learning is to **normalize** the inputs, i.e. to make sure that the features have a mean of 0 and a standard deviation of 1. The formula is:\n",
    "\n",
    "$$X^\\text{normalized} = \\dfrac{X - \\mathbb{E}[X]}{\\text{std}(X)}$$\n",
    "\n",
    "i.e. you compute the mean and standard deviation of each column of `X` and apply the formula on each column. \n",
    "\n",
    "**Q:** Normalize the dataset. Make sure that the new mean and std is correct. \n",
    "\n",
    "*Tip:* `X.mean(axis=0)` and `X.std(axis=0)` should be useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q:** Apply MLR again on $X^\\text{normalized}$, print the mse and visualize the weights. What has changed?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularized regression\n",
    "\n",
    "Now is time to investigate **regularization**:\n",
    "1. MLR with L2 regularization is called **Ridge regression**\n",
    "2. MLR with L1 regularization is called **Lasso regression** \n",
    "\n",
    "Fortunately, `scikit-learn` provides these methods with a similar interface to `LinearRegression`. The `Ridge` and `Lasso` objects take an additional argument `alpha` which represents the regularization parameter:\n",
    "\n",
    "```python\n",
    "reg = Ridge(alpha=1.0)\n",
    "reg = Lasso(alpha=1.0)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge, Lasso"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q:** Apply Ridge and Lasso regression on the scaled data, vary the regularization parameter to understand its function and comment on the results. In particular, vary the regularization parameter for LASSO and identify the features which are the most predictive of the price. Does it make sense?  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.13 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "3d24234067c217f49dc985cbc60012ce72928059d528f330ba9cb23ce737906d"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}