{ "cells": [ { "cell_type": "markdown", "id": "9ca5e171", "metadata": {}, "source": [ "# Random forests intro and regression\n", "\n", "Random Forests are powerful machine learning algorithms used for supervised classification and regression. Random forests works by averaging the predictions of the multiple and randomized decision trees. Decision trees tends to overfit and so by combining multiple decision trees, the effect of overfitting can be minimized. \n", "\n", "Random Forests are type of ensemble models. More about ensembles models in the next notebook. \n", "\n", "Different to other learning algorithms, random forests provide a way to find the importance of each feature and this is implemented in Sklearn. " ] }, { "cell_type": "markdown", "id": "99058d12", "metadata": { "tags": [] }, "source": [ "## Imports" ] }, { "cell_type": "code", "execution_count": null, "id": "df278c5f", "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "import numpy as np\n", "import pandas as pd\n", "import seaborn as sns\n", "import sklearn\n", "import matplotlib.pyplot as plt\n", "import pytest\n", "import ipytest\n", "import unittest\n", "\n", "ipytest.autoconfig()" ] }, { "cell_type": "markdown", "id": "35699c38", "metadata": { "tags": [] }, "source": [ "## Loading the data\n", "\n", "In this regression task with random forests, we will use the Machine CPU (Central Processing Unit) dataset which is available at [OpenML](https://www.openml.org/t/5492).\n", "\n", "If you are reading this, it's very likely that you know CPU or you have once(or many times) thought about it when you were buying your computer. In this notebook, we will predict the relative performance of the CPU given the following data: \n", "\n", "* MYCT: machine cycle time in nanoseconds (integer)\n", "* MMIN: minimum main memory in kilobytes (integer)\n", "* MMAX: maximum main memory in kilobytes (integer)\n", "* CACH: cache memory in kilobytes (integer)\n", "* CHMIN: minimum channels in units (integer)\n", "* CHMAX: maximum channels in units (integer)\n", "* PRP: published relative performance (integer) (target variable)" ] }, { "cell_type": "code", "execution_count": null, "id": "67905ee6", "metadata": {}, "outputs": [], "source": [ "# Let's hide warnings\n", "\n", "import warnings\n", "\n", "warnings.filterwarnings(\"ignore\")" ] }, { "cell_type": "code", "execution_count": null, "id": "d3083946", "metadata": {}, "outputs": [], "source": [ "machine_data = pd.read_csv(\"../../../assets/data/machine_cup.csv\")" ] }, { "cell_type": "code", "execution_count": null, "id": "46c09852", "metadata": {}, "outputs": [], "source": [ "type(machine_data)" ] }, { "cell_type": "code", "execution_count": null, "id": "da794464-2da8-4114-a3a9-c20c6db713f7", "metadata": {}, "outputs": [], "source": [ "machine_data.shape" ] }, { "cell_type": "code", "execution_count": null, "id": "e216742c-5603-47e1-879d-9bcfeb2b44e9", "metadata": {}, "outputs": [], "source": [ "machine_data.head()" ] }, { "cell_type": "markdown", "id": "acda9039-bd82-4ebd-97e8-e32b9510638f", "metadata": { "tags": [] }, "source": [ "## Tasks and roles" ] }, { "cell_type": "markdown", "id": "c492fd93", "metadata": { "tags": [] }, "source": [ "### Task 1: Exploratory analysis" ] }, { "cell_type": "markdown", "id": "e9906083", "metadata": {}, "source": [ "Before doing exploratory analysis, let's get the training and test data. " ] }, { "cell_type": "code", "execution_count": null, "id": "98d74144", "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "train_data, test_data = train_test_split(machine_data, test_size=0.2, random_state=20)\n", "print(\n", " \"The size of training data is: {} \\nThe size of testing data is: {}\".format(\n", " len(train_data), len(test_data)\n", " )\n", ")" ] }, { "cell_type": "markdown", "id": "c3fc640e", "metadata": { "tags": [] }, "source": [ "#### Part 1: The histogram" ] }, { "cell_type": "code", "execution_count": null, "id": "61118a4c-dea4-4ce2-bdc3-3a52fae5e1e8", "metadata": {}, "outputs": [], "source": [ "def df_hist(df):\n", " if df is not None and not df.empty:\n", " df.hist(bins = 50, figsize = (15, 10))" ] }, { "cell_type": "code", "execution_count": null, "id": "04856278", "metadata": {}, "outputs": [], "source": [ "df_hist(train_data)\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "d76527cf-1527-4bc5-a9bc-f30619ec3cc3", "metadata": { "tags": [] }, "source": [ "
Check result by executing below... 📝
" ] }, { "cell_type": "code", "execution_count": null, "id": "5214a698-0905-4d12-baa8-bbfcac819dc6", "metadata": {}, "outputs": [], "source": [ "%%ipytest -qq\n", "\n", "from unittest.mock import Mock, patch\n", "\n", "class TestDFHist(unittest.TestCase):\n", " \n", " def test_df_hist_happy_case(self):\n", " # assign\n", " test_df = Mock(return_value=pd.DataFrame(\n", " {\n", " 'c1': [1, 2, 3, 4, 5], \n", " }\n", " ))\n", " test_df.empty = False\n", " \n", " with patch.object(test_df, 'hist') as mock_df_hist:\n", " # act\n", " actual_result = df_hist(test_df)\n", "\n", " # assert\n", " mock_df_hist.assert_called_once()\n", "\n", " def test_df_hist_with_empty_df(self):\n", " # assign\n", " test_df = Mock(return_value=pd.DataFrame())\n", " \n", " with patch.object(test_df, 'hist') as mock_df_hist:\n", " # act\n", " actual_result = df_hist(test_df)\n", "\n", " # assert\n", " mock_df_hist.assert_not_called()\n", " \n", " def test_df_hist_with_none_df(self):\n", " # assign\n", " test_df = Mock(return_value=None)\n", " \n", " with patch.object(test_df, 'hist') as mock_df_hist:\n", " # act\n", " actual_result = df_hist(test_df)\n", "\n", " # assert\n", " mock_df_hist.assert_not_called()" ] }, { "cell_type": "markdown", "id": "79f839cb-498c-427f-8598-38b782cff541", "metadata": { "tags": [ "hide-input" ] }, "source": [ "
\n", " \n", "
👩‍💻 Hint\n", " \n", "You can consider to use pandas.DataFrame.hist.\n", "\n", "
\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "5eca7747", "metadata": { "tags": [] }, "source": [ "#### Part 2: The pairplot" ] }, { "cell_type": "code", "execution_count": null, "id": "f895d99a-dea8-48d8-98a8-8ecd97ef7e50", "metadata": {}, "outputs": [], "source": [ "def df_pairplot(df):\n", " if df is not None and not df.empty:\n", " sns.pairplot(df)" ] }, { "cell_type": "code", "execution_count": null, "id": "e865d6c1-d4b0-4c08-9bc3-cfebae56dc48", "metadata": {}, "outputs": [], "source": [ "df_pairplot(train_data)" ] }, { "cell_type": "markdown", "id": "cbe5fd5f-55f0-402f-84df-116aa1db2815", "metadata": { "tags": [] }, "source": [ "
Check result by executing below... 📝
" ] }, { "cell_type": "code", "execution_count": null, "id": "b9a2c240-e166-40c5-ac69-65bc04b43d09", "metadata": {}, "outputs": [], "source": [ "%%ipytest -qq\n", "\n", "from unittest.mock import Mock, patch\n", "\n", "class TestDFHist(unittest.TestCase):\n", " \n", " def test_df_pairplot_happy_case(self):\n", " # assign\n", " test_df = Mock(return_value=pd.DataFrame(\n", " {\n", " 'c1': [1, 2, 3, 4, 5], \n", " 'c2': [2, 4, 6, 8, 10], \n", " }\n", " ))\n", " test_df.empty = False\n", " \n", " with patch.object(sns, 'pairplot') as mock_pairplot:\n", " # act\n", " actual_result = df_pairplot(test_df)\n", "\n", " # assert\n", " mock_pairplot.assert_called_once_with(test_df)\n", "\n", " def test_df_pairplot_with_empty_df(self):\n", " # assign\n", " test_df = Mock(return_value=pd.DataFrame())\n", " \n", " with patch.object(sns, 'pairplot') as mock_df_pairplot:\n", " # act\n", " actual_result = df_pairplot(test_df)\n", "\n", " # assert\n", " mock_df_pairplot.assert_not_called()\n", " \n", " def test_df_pairplot_with_none_df(self):\n", " # assign\n", " test_df = Mock(return_value=None)\n", " \n", " with patch.object(sns, 'pairplot') as mock_df_pairplot:\n", " # act\n", " actual_result = df_pairplot(test_df)\n", "\n", " # assert\n", " mock_df_pairplot.assert_not_called()" ] }, { "cell_type": "markdown", "id": "06b1e265-45d8-442c-b0e1-388fedb7f336", "metadata": { "tags": [ "hide-input" ] }, "source": [ "
\n", " \n", "
👩‍💻 Hint\n", " \n", "You can consider to use seaborn.pairplot.\n", "\n", "
\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "6d6c4eb8-05e2-4a58-ac30-5116a0c8c6d5", "metadata": { "tags": [] }, "source": [ "#### Part 3: Check the train data" ] }, { "cell_type": "code", "execution_count": null, "id": "6f42ce9f-7355-4f34-b1cc-53336fa32cd0", "metadata": {}, "outputs": [], "source": [ "def df_desc(df):\n", " if df is None:\n", " raise Exception('df cannot be None.')\n", " return df.describe()" ] }, { "cell_type": "code", "execution_count": null, "id": "753bb539", "metadata": {}, "outputs": [], "source": [ "# Let's check the summary stats\n", "df_desc(train_data)" ] }, { "cell_type": "code", "execution_count": null, "id": "ad45c875-abdd-4f26-8f6d-6d191b24970b", "metadata": {}, "outputs": [], "source": [ "def df_null(df):\n", " if df is None:\n", " raise Exception('df cannot be None.')\n", " return df.isnull().sum()" ] }, { "cell_type": "code", "execution_count": null, "id": "f0e4fb67", "metadata": {}, "outputs": [], "source": [ "# Let's check the missing values\n", "df_null(train_data)" ] }, { "cell_type": "markdown", "id": "4977a1a1", "metadata": {}, "source": [ "Great! We don't have any missing values. " ] }, { "cell_type": "markdown", "id": "198bf061-fd22-4f0f-974b-cb0927f1bc26", "metadata": { "tags": [] }, "source": [ "#### Part 4: Look the correlation" ] }, { "cell_type": "code", "execution_count": null, "id": "f00e0828-0b13-4f97-aad6-242aa2bf8dc3", "metadata": {}, "outputs": [], "source": [ "def df_corr(df):\n", " if df is not None and not df.empty:\n", " corr = df.corr()\n", " return corr" ] }, { "cell_type": "code", "execution_count": null, "id": "368d545e-04c0-4597-96bc-8a3516d186e7", "metadata": {}, "outputs": [], "source": [ "corr = df_corr(train_data)\n", "corr[\"class\"]" ] }, { "cell_type": "code", "execution_count": null, "id": "8b6e8411-8a8c-48c8-9a99-35602d52794f", "metadata": {}, "outputs": [], "source": [ "def df_heat(correlation):\n", " if correlation is not None:\n", " return sns.heatmap(correlation, annot=True, cmap=\"crest\")" ] }, { "cell_type": "code", "execution_count": null, "id": "0e24c4e6", "metadata": {}, "outputs": [], "source": [ "## Visualizing correlation\n", "\n", "plt.figure(figsize=(12, 7))\n", "df_heat(corr)" ] }, { "cell_type": "markdown", "id": "e6733727", "metadata": { "tags": [] }, "source": [ "### Task 2: Data preprocessing\n", "\n", "It is here that we prepare the data to be in the proper format for the machine learning model. \n", "Let's set up a pipeline to scale features but before that, let's take training input data and labels." ] }, { "cell_type": "code", "execution_count": null, "id": "acb7a81e", "metadata": {}, "outputs": [], "source": [ "X_train = train_data.drop(\"class\", axis=1)\n", "y_train = train_data[\"class\"]" ] }, { "cell_type": "code", "execution_count": null, "id": "b7ee6d06", "metadata": {}, "outputs": [], "source": [ "from sklearn.preprocessing import StandardScaler\n", "from sklearn.pipeline import Pipeline\n", "\n", "scale_pipe = Pipeline([(\"scaler\", StandardScaler())])\n", "X_train_scaled = scale_pipe.fit_transform(X_train)" ] }, { "cell_type": "markdown", "id": "dd496de6", "metadata": {}, "source": [ "### Task 3: Training random forests regressor" ] }, { "cell_type": "code", "execution_count": null, "id": "8311b87a", "metadata": {}, "outputs": [], "source": [ "from sklearn.ensemble import RandomForestRegressor\n", "\n", "forest_reg = RandomForestRegressor(\n", " min_samples_split=2, bootstrap=False, random_state=42, n_jobs=-1\n", ")\n", "\n", "forest_reg.fit(X_train_scaled, y_train)" ] }, { "cell_type": "markdown", "id": "9afb9bde", "metadata": {}, "source": [ "\n", "\n", "### Task 4: Evaluating random forests regressor\n", "\n", "Let's first check the root mean squarred errr on the training. It is not advised to evaluate the model on the test data since we haven't improved it yet. we will make a function to make it easier and to avoid repetitions. " ] }, { "cell_type": "code", "execution_count": null, "id": "6962359e", "metadata": {}, "outputs": [], "source": [ "from sklearn.metrics import mean_squared_error\n", "\n", "\n", "def predict(input_data, model, labels):\n", " \"\"\"\n", " Take the input data, model and labels and return predictions\n", "\n", " \"\"\"\n", "\n", " preds = model.predict(input_data)\n", " mse = mean_squared_error(labels, preds)\n", " rmse = np.sqrt(mse)\n", "\n", " return rmse" ] }, { "cell_type": "code", "execution_count": null, "id": "a17fed16", "metadata": {}, "outputs": [], "source": [ "predict(X_train_scaled, forest_reg, y_train)" ] }, { "cell_type": "markdown", "id": "66a574a5", "metadata": { "tags": [] }, "source": [ "### Task 5: Improving random forests" ] }, { "cell_type": "code", "execution_count": null, "id": "0f92566d", "metadata": {}, "outputs": [], "source": [ "forest_reg.get_params()" ] }, { "cell_type": "markdown", "id": "af110f75", "metadata": {}, "source": [ "We will use GridSearch to find the best hyperparameters that we can use to retrain the model with. By setting the `refit` to `True`, the random forest will be automatically retrained on the dataset with the best hyperparameters. By default, `refit` is True.\n", "\n", "This will take too long. " ] }, { "cell_type": "code", "execution_count": null, "id": "056818ed", "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import GridSearchCV\n", "\n", "params_grid = {\n", " \"n_estimators\": [100, 200, 300, 400, 500],\n", " \"max_leaf_nodes\": list(range(2, 52)),\n", "}\n", "\n", "# refit is true by default. The best estimator is trained on the whole dataset\n", "\n", "grid_search = GridSearchCV(\n", " RandomForestRegressor(min_samples_split=2, bootstrap=False, random_state=42),\n", " params_grid,\n", " verbose=1,\n", " cv=5,\n", ")\n", "\n", "grid_search.fit(X_train_scaled, y_train)" ] }, { "cell_type": "code", "execution_count": null, "id": "55a5d7f3", "metadata": {}, "outputs": [], "source": [ "grid_search.best_params_" ] }, { "cell_type": "code", "execution_count": null, "id": "c4a1488d", "metadata": {}, "outputs": [], "source": [ "grid_search.best_estimator_" ] }, { "cell_type": "code", "execution_count": null, "id": "4995ab2d", "metadata": {}, "outputs": [], "source": [ "forest_best = grid_search.best_estimator_" ] }, { "cell_type": "markdown", "id": "44fc64ef", "metadata": {}, "source": [ "Let's make prediction on the training data again" ] }, { "cell_type": "code", "execution_count": null, "id": "85b1b211", "metadata": {}, "outputs": [], "source": [ "predict(X_train_scaled, forest_best, y_train)" ] }, { "cell_type": "markdown", "id": "40fb17de", "metadata": {}, "source": [ "Surprisingly, by searching model hyperparameters, the model did not improve. Can you guess why? We can observe many things while running Grid Search and reading about the [random forests](https://scikit-learn.org/stable/modules/ensemble.html#forests-of-randomized-trees). If you can't get good results, set the `bootstrap` to False. It is true by default, and that means that you are training on samples of the training set instead of the whole training set. Try going back to the orginal model and change it to True and note how the prediction changes. Also learn more about the other hyperparameters. " ] }, { "cell_type": "markdown", "id": "77ed447f", "metadata": { "tags": [] }, "source": [ "### Task 6: Feature importance " ] }, { "cell_type": "markdown", "id": "2a1e9c1f", "metadata": {}, "source": [ "Different to other machine learning models, random forests can show how each feature contributed to the model generalization. Let's find it.\n", "The results are values between 0 and 1. The closer to 1, the good the feature was to the model." ] }, { "cell_type": "code", "execution_count": null, "id": "92298569", "metadata": {}, "outputs": [], "source": [ "feat_import = forest_best.feature_importances_\n", "\n", "feat_dict = {\"Features\": X_train.columns, \"Feature Importance\": feat_import}\n", "\n", "pd.DataFrame(feat_dict)" ] }, { "cell_type": "markdown", "id": "a91ab967", "metadata": {}, "source": [ "As you can see above, the most 2 features which contributed to the prediction of the relative performance of the CPU are `MMAX` which is the Maximum Main Memory in Kilobytes and `CACH` (cache memory). \n", "\n", "It makes sense that the model was able to find that out. Main memory (RAM, Read Only Memory) and cache memory (which stores frequently used information thus facilitating faster processing and quick retrieval of information) are the two most factors of the CPU performance and if you are going to buy a new computer, you want to have high RAM and cache memory in order to have a powerful machine that can process/compute and retrieve things faster. " ] }, { "cell_type": "markdown", "id": "d5adbd33", "metadata": { "tags": [] }, "source": [ "### Task 7: Evaluating the Model on the Test Set" ] }, { "cell_type": "markdown", "id": "d4f82117", "metadata": {}, "source": [ "Let us evaluate the model on the test set. But we need first run the pipeline on the test data. Note that we only transform (not fit_transform)." ] }, { "cell_type": "code", "execution_count": null, "id": "3c5f805c", "metadata": {}, "outputs": [], "source": [ "train_data, test_data = train_test_split(machine_data, test_size=0.2, random_state=20)" ] }, { "cell_type": "code", "execution_count": null, "id": "07f6ff2e", "metadata": {}, "outputs": [], "source": [ "X_test = test_data.drop(\"class\", axis=1)\n", "y_test = test_data[\"class\"]\n", "\n", "test_scaled = scale_pipe.transform(X_test)" ] }, { "cell_type": "code", "execution_count": null, "id": "f03cc7fa", "metadata": {}, "outputs": [], "source": [ "predict(test_scaled, forest_best, y_test)" ] }, { "cell_type": "markdown", "id": "b8136b43", "metadata": {}, "source": [ "The results on the test set is not appealing, and it is a sign that the model is still overfitting the data(it is doing well on the training set and poor on the new data). One way to improve the model can be to regularize the model by searching more best hyperparameters and increasing the data and data quality. The later is what can improve the model in many scenarios. " ] }, { "cell_type": "markdown", "id": "af266a08", "metadata": {}, "source": [ "This is the end of this assignment. We have learned the fundamental idea behind the random forests, and used it to predict the CPU performance. In the next assignment, we will use it for classification task and we will use a real world dataset so that we can practically improve the random forests." ] }, { "cell_type": "markdown", "id": "17af5728", "metadata": {}, "source": [ "## Acknowledgments\n", "\n", "Thanks to Nyandwi for creating the open-source course [Machine Learning complete](https://github.com/Nyandwi/machine_learning_complete). It inspires the majority of the content in this chapter." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.16" } }, "nbformat": 4, "nbformat_minor": 5 }