{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "74db09f1",
   "metadata": {},
   "source": [
    "# Regression models comparison\n",
    "\n",
    "## Overview\n",
    "\n",
    "In this activity, you will apply six different regression models to the same dataset and compare their performance. The goal is to build intuition about how different model families handle non-linear data.\n",
    "\n",
    "### Dataset\n",
    "\n",
    "The dataset is generated with [`make_friedman1`](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_friedman1.html). The target is a mix of non-linear and linear relationships, making it a useful benchmark for comparing model expressiveness.\n",
    "\n",
    "### Tasks\n",
    "\n",
    "1. Train and evaluate a [linear regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) model\n",
    "2. Train and evaluate a [decision tree](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html) model\n",
    "3. Train and evaluate a [random forest](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html) model\n",
    "4. Train and evaluate a [k-nearest neighbors (KNN)](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsRegressor.html) model\n",
    "5. Train and evaluate a [support vector machine (SVR)](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html) model\n",
    "6. Train and evaluate a [Gaussian process regression (GPR)](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html) model\n",
    "7. Compare model performance\n",
    "8. Investigate why the models perform differently\n",
    "\n",
    "### Suggested project organization\n",
    "\n",
    "Because this activity spans eight tasks, you may want to treat it as a small project: create a Git repository and give each task its own notebook. Benefits of this approach:\n",
    "\n",
    "- **Version control**: commit after each task so you can review progress and roll back mistakes without affecting other tasks.\n",
    "- **Isolated execution**: a crash or error in one notebook does not corrupt the kernel state of another.\n",
    "- **Cleaner history**: commits are scoped to individual tasks, making it easy to see what changed and when.\n",
    "- **Easier collaboration**: team members can work on different tasks in parallel without merge conflicts on a single large file.\n",
    "\n",
    "### Collecting results\n",
    "\n",
    "As you complete each task, store the training and testing RMSE in a shared data structure so everything is ready for the model comparison in task 7. A list of dictionaries is a convenient starting point:\n",
    "\n",
    "```python\n",
    "results = []\n",
    "\n",
    "# After training and evaluating each model, append a row:\n",
    "results.append({'model': 'Linear regression', 'train_rmse': ..., 'test_rmse': ...})\n",
    "```\n",
    "\n",
    "By task 7 you can convert this to a DataFrame with `pd.DataFrame(results)`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4be303ee",
   "metadata": {},
   "source": [
    "## Notebook set up\n",
    "\n",
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f194e0fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.datasets import make_friedman1\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.svm import SVR\n",
    "from sklearn.gaussian_process import GaussianProcessRegressor\n",
    "from sklearn.gaussian_process.kernels import RBF, WhiteKernel\n",
    "from sklearn.model_selection import train_test_split, cross_val_score\n",
    "from sklearn.metrics import mean_squared_error, root_mean_squared_error, r2_score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5db5d65b",
   "metadata": {},
   "source": [
    "### Create dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0d6bb02",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate the Friedman1 dataset\n",
    "X, y = make_friedman1(n_samples=5000, n_features=5, random_state=315)\n",
    "\n",
    "# Convert to DataFrame\n",
    "X_df = pd.DataFrame(X, columns=[f'feature_{i}' for i in range(5)])\n",
    "y_series = pd.Series(y, name='label')\n",
    "\n",
    "# Combine into a single DataFrame for easy exploration\n",
    "df = X_df.copy()\n",
    "df['label'] = y_series\n",
    "\n",
    "# Display the first few rows\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87753c18",
   "metadata": {},
   "source": [
    "## Task 1: Train and evaluate a linear regression model\n",
    "\n",
    "**Tasks**:\n",
    "\n",
    "1. Split the data into training and testing sets using an 80-20 split. Use `random_state=315`.\n",
    "\n",
    "2. Train a `LinearRegression` model on the training data.\n",
    "\n",
    "3. Calculate and print the following metrics:\n",
    "   - Training RMSE\n",
    "   - Testing RMSE\n",
    "\n",
    "4. Create a scatter plot of true vs. predicted values for the test set:\n",
    "   - Add a diagonal reference line (y=x) to show perfect predictions\n",
    "   - Label the axes appropriately\n",
    "   - Add a title\n",
    "\n",
    "**Hints**:\n",
    "\n",
    "- Use `train_test_split()` with `test_size=0.2`\n",
    "  - Example: `X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=315)`\n",
    "\n",
    "- To calculate metrics:\n",
    "  - First make predictions: `y_pred = model.predict(X_test)`\n",
    "  - Then calculate: `rmse = root_mean_squared_error(y_test, y_pred)`\n",
    "\n",
    "- To add a reference line to a plot:\n",
    "  - `plt.plot([min, max], [min, max], 'k--', alpha=0.3)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c17f022",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52e629e6",
   "metadata": {},
   "source": [
    "## Task 2: Train and evaluate a decision tree model\n",
    "\n",
    "**Tasks:**\n",
    "\n",
    "1. Train a `DecisionTreeRegressor` on the training data from task 1.\n",
    "\n",
    "2. Calculate and print the following metrics:\n",
    "   - Training RMSE\n",
    "   - Testing RMSE\n",
    "\n",
    "3. Create a scatter plot of true vs. predicted values for the test set:\n",
    "   - Add a diagonal reference line (y=x) to show perfect predictions\n",
    "   - Label the axes appropriately\n",
    "   - Add a title\n",
    "\n",
    "**Hints:**\n",
    "\n",
    "- Suggested hyperparameters: `DecisionTreeRegressor(max_depth=8, min_samples_split=30, min_samples_leaf=15, random_state=315)`\n",
    "- Use the same `X_train`, `X_test`, `y_train`, `y_test` from task 1.\n",
    "- You are encouraged to experiment with the hyperparameter values and observe how they affect training and testing RMSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "caa58443",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15690a37",
   "metadata": {},
   "source": [
    "## Task 3: Train and evaluate a random forest model\n",
    "\n",
    "**Tasks:**\n",
    "\n",
    "1. Train a `RandomForestRegressor` on the training data from task 1.\n",
    "\n",
    "2. Calculate and print the following metrics:\n",
    "   - Training RMSE\n",
    "   - Testing RMSE\n",
    "\n",
    "3. Create a scatter plot of true vs. predicted values for the test set:\n",
    "   - Add a diagonal reference line (y=x) to show perfect predictions\n",
    "   - Label the axes appropriately\n",
    "   - Add a title\n",
    "\n",
    "**Hints:**\n",
    "\n",
    "- A random forest trains an ensemble of decision trees and averages their predictions. This typically reduces the variance (overfitting) seen in a single tree.\n",
    "- Suggested hyperparameters: `RandomForestRegressor(n_estimators=100, random_state=315)`\n",
    "- Use the same `X_train`, `X_test`, `y_train`, `y_test` from task 1.\n",
    "- You are encouraged to experiment with the hyperparameter values and observe how they affect training and testing RMSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93c1d7b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63129406",
   "metadata": {},
   "source": [
    "## Task 4: Train and evaluate a k-nearest neighbors (KNN) model\n",
    "\n",
    "**Tasks:**\n",
    "\n",
    "1. Train a `KNeighborsRegressor` on the training data from task 1.\n",
    "\n",
    "2. Calculate and print the following metrics:\n",
    "   - Training RMSE\n",
    "   - Testing RMSE\n",
    "\n",
    "3. Create a scatter plot of true vs. predicted values for the test set.\n",
    "\n",
    "**Hints:**\n",
    "\n",
    "- KNN regression predicts the mean target value of the k nearest training points. There is no explicit training step: the training data is stored and queried at prediction time.\n",
    "- Suggested hyperparameters: `KNeighborsRegressor(n_neighbors=10)`\n",
    "- Use the same `X_train`, `X_test`, `y_train`, `y_test` from task 1.\n",
    "- You are encouraged to experiment with different values of `n_neighbors` and observe how they affect training and testing RMSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be386702",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57a6bffb",
   "metadata": {},
   "source": [
    "## Task 5: Train and evaluate a support vector machine (SVR) model\n",
    "\n",
    "**Tasks:**\n",
    "\n",
    "1. Train an `SVR` model on the training data from task 1.\n",
    "\n",
    "2. Calculate and print the following metrics:\n",
    "   - Training RMSE\n",
    "   - Testing RMSE\n",
    "\n",
    "3. Create a scatter plot of true vs. predicted values for the test set.\n",
    "\n",
    "**Hints:**\n",
    "\n",
    "- SVR fits a tube of width epsilon around the training data and finds a hyperplane that maximizes the margin. The `C` parameter controls the trade-off between margin width and training error.\n",
    "- Suggested hyperparameters: `SVR(kernel='rbf', C=10, epsilon=0.1)`\n",
    "- Use the same `X_train`, `X_test`, `y_train`, `y_test` from task 1.\n",
    "- You are encouraged to experiment with `C`, `epsilon`, and `kernel` and observe how they affect training and testing RMSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9af4058",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd2f3559",
   "metadata": {},
   "source": [
    "## Task 6: Train and evaluate a Gaussian process regression (GPR) model\n",
    "\n",
    "**Tasks:**\n",
    "\n",
    "1. Train a `GaussianProcessRegressor` on a subset of the training data.\n",
    "\n",
    "2. Calculate and print the following metrics:\n",
    "   - Training RMSE (on the training subset)\n",
    "   - Testing RMSE (on the full test set)\n",
    "\n",
    "3. Create a scatter plot of true vs. predicted values for the test set.\n",
    "\n",
    "**Notes:**\n",
    "\n",
    "- GPR scales as O(n³) due to matrix inversion, making it impractical on large datasets. Train on a 500-sample subset to keep runtime manageable.\n",
    "- Unlike the other models, GPR also returns a standard deviation per prediction, which lets you quantify prediction uncertainty.\n",
    "\n",
    "**Hints:**\n",
    "\n",
    "- Use a subset of the training data: `X_train_sub, y_train_sub = X_train[:500], y_train[:500]`\n",
    "- Suggested kernel: `kernel = 1.0 * RBF(length_scale=1.0) + WhiteKernel(noise_level=1.0)`\n",
    "- Suggested model: `GaussianProcessRegressor(kernel=kernel, n_restarts_optimizer=2, random_state=315)`\n",
    "- Predictions with uncertainty: `y_pred, y_std = gpr_model.predict(X_test, return_std=True)`\n",
    "- Evaluate on the same `X_test`, `y_test` from task 1.\n",
    "- You are encouraged to experiment with different kernels and `length_scale` values and observe how they affect the fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9686b7a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31905b9e",
   "metadata": {},
   "source": [
    "## Task 7: Compare model performance\n",
    "\n",
    "**Tasks:**\n",
    "\n",
    "1. Using the results you collected during tasks 1–6, create a horizontal bar chart of test RMSE for all six models, sorted from lowest to highest error.\n",
    "\n",
    "2. Print a summary table with training RMSE and testing RMSE for all models.\n",
    "\n",
    "3. Based on the results:\n",
    "   - Which model achieves the lowest test RMSE?\n",
    "   - Which models show the largest gap between training and testing RMSE? What does that indicate?\n",
    "   - Is the best-performing model always the right choice? Consider factors like interpretability and training cost.\n",
    "\n",
    "**Notes:**\n",
    "\n",
    "- The GPR model was trained on only 500 samples, so its metrics are not directly comparable to the other models. Note this caveat when interpreting the results.\n",
    "\n",
    "**Hints:**\n",
    "\n",
    "- To create a sorted bar chart: sort the test RMSE values (and model names) together before plotting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a76208f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07e0a701",
   "metadata": {},
   "source": [
    "## Task 8: Investigate why the models perform differently\n",
    "\n",
    "**Tasks:**\n",
    "\n",
    "1. Create visualizations to understand the relationship between features and label:\n",
    "   - For features 0 and 1: create a 2D scatter plot colored by the label value (use a colormap)\n",
    "   - For features 2, 3, and 4: create individual scatter plots vs. label\n",
    "\n",
    "2. Based on your plots:\n",
    "   - Identify which relationships are linear\n",
    "   - Identify which relationships are non-linear\n",
    "   - Explain how this affects each model's performance\n",
    "\n",
    "3. (Optional) Try to improve the linear regression model by adding polynomial features for the non-linear relationships. Does this improve performance?\n",
    "\n",
    "**Hints:**\n",
    "\n",
    "- For a 2D scatter plot with color mapping:\n",
    "  - `plt.scatter(X[:, 0], X[:, 1], c=y, cmap='viridis')`\n",
    "  - `plt.colorbar(label='Label')`\n",
    "\n",
    "- Tree-based models and KNN can capture non-linear relationships, while linear regression assumes linear structure. GPR can model arbitrary non-linear patterns via the kernel function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74694d23",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edef9a87",
   "metadata": {},
   "source": [
    "## Reflection\n",
    "\n",
    "Based on your analysis, answer the following questions:\n",
    "\n",
    "1. **Model performance**: Which model performed better and why?\n",
    "\n",
    "2. **Linear assumptions**: What happens when you apply linear regression to non-linear data?\n",
    "\n",
    "3. **Model complexity**: What are the trade-offs between simpler models (linear regression) and more complex models (decision trees)?\n",
    "\n",
    "4. **Real-world implications**: In what situations would you prefer:\n",
    "   - A linear regression model?\n",
    "   - A decision tree model?\n",
    "   - Something else?\n",
    "   - Consider factors like interpretability, performance, and data characteristics."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}