{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Selecting the best model with Hyperparameter tuning.\n", "> The first three chapters focused on model validation techniques. In chapter 4 we apply these techniques, specifically cross-validation, while learning about hyperparameter tuning. After all, model validation makes tuning possible and helps us select the overall best model. This is the Summary of lecture \"Model Validation in Python\", via datacamp.\n", "\n", "- toc: true \n", "- badges: true\n", "- comments: true\n", "- author: Chanseok Kang\n", "- categories: [Python, Datacamp, Machine_Learning]\n", "- image: " ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction to hyperparameter tuning\n", "- Model Parameters\n", " - Learned or estimated from the data\n", " - The result of fitting a model\n", " - Used when making future predictions\n", " - Not manually set\n", "- Model Hyperparameters\n", " - Manually set before the training occurs\n", " - Specify how the training is supposed to happen\n", "- Hyperparameter tuning\n", " - Select hyperparameters\n", " - Run a single model type at different value sets\n", " - Create ranges of possible values to select from\n", " - Specify a single accuracy metric" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Creating Hyperparameters\n", "For a school assignment, your professor has asked your class to create a random forest model to predict the average test score for the final exam.\n", "\n", "After developing an initial random forest model, you are unsatisfied with the overall accuracy. You realize that there are too many hyperparameters to choose from, and each one has a lot of possible values. You have decided to make a list of possible ranges for the hyperparameters you might use in your next model.\n", "\n", "Your professor has provided de-identified data for the last ten quizzes to act as the training data. There are 30 students in your class." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "from sklearn.ensemble import RandomForestRegressor\n", "\n", "rfr = RandomForestRegressor(n_estimators='warn', max_features='auto', random_state=1111)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'bootstrap': True,\n", " 'ccp_alpha': 0.0,\n", " 'criterion': 'mse',\n", " 'max_depth': None,\n", " 'max_features': 'auto',\n", " 'max_leaf_nodes': None,\n", " 'max_samples': None,\n", " 'min_impurity_decrease': 0.0,\n", " 'min_impurity_split': None,\n", " 'min_samples_leaf': 1,\n", " 'min_samples_split': 2,\n", " 'min_weight_fraction_leaf': 0.0,\n", " 'n_estimators': 'warn',\n", " 'n_jobs': None,\n", " 'oob_score': False,\n", " 'random_state': 1111,\n", " 'verbose': 0,\n", " 'warm_start': False}" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rfr.get_params()" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "# Maximum Depth\n", "max_depth = [4, 8, 12]\n", "\n", "# Minimum samples for a split\n", "min_samples_split = [2, 5, 10]\n", "\n", "# Max features \n", "max_features = [4, 6, 8, 10]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Hyperparameter tuning requires selecting parameters to tune, as well the possible values these parameters can be set to." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Running a model using ranges\n", "You have just finished creating a list of hyperparameters and ranges to use when tuning a predictive model for an assignment." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'bootstrap': True, 'ccp_alpha': 0.0, 'criterion': 'mse', 'max_depth': 8, 'max_features': 6, 'max_leaf_nodes': None, 'max_samples': None, 'min_impurity_decrease': 0.0, 'min_impurity_split': None, 'min_samples_leaf': 1, 'min_samples_split': 5, 'min_weight_fraction_leaf': 0.0, 'n_estimators': 100, 'n_jobs': None, 'oob_score': False, 'random_state': None, 'verbose': 0, 'warm_start': False}\n" ] } ], "source": [ "import random\n", "\n", "# Fill in rfr using your variables\n", "rfr = RandomForestRegressor(\n", " n_estimators=100,\n", " max_depth=random.sample(max_depth, 1)[0],\n", " min_samples_split=random.sample(min_samples_split, 1)[0],\n", " max_features = random.sample(max_features, 1)[0]\n", ")\n", "\n", "# Print out the parameters\n", "print(rfr.get_params())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## RandomizedSearchCV\n", "- Grid Search\n", " - Benefits\n", " - Tests every possible combination\n", " - Drawbacks\n", " - Additional hyperparameters increase training time exponentially\n", "- Alternatives\n", " - Random searching\n", " - Bayesian optimization" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Preparing for RandomizedSearch\n", "Last semester your professor challenged your class to build a predictive model to predict final exam test scores. You tried running a few different models by randomly selecting hyperparameters. However, running each model required you to code it individually.\n", "\n", "After learning about `RandomizedSearchCV()`, you're revisiting your professors challenge to build the best model. In this exercise, you will prepare the three necessary inputs for completing a random search." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "from sklearn.metrics import make_scorer, mean_squared_error\n", "\n", "# Finish the dictionary by adding the max_depth parameter\n", "param_dist = {\n", " \"max_depth\": [2, 4, 6, 8],\n", " \"max_features\": [2, 4, 6, 8, 10],\n", " \"min_samples_split\": [2, 4, 8, 16]\n", "}\n", "\n", "# Create a random forest regression model\n", "rfr = RandomForestRegressor(n_estimators=10, random_state=1111)\n", "\n", "# Create a scorer to use (use the mean squared error)\n", "scorer = make_scorer(mean_squared_error)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To use `RandomizedSearchCV()`, you need a distribution dictionary, an estimator, and a scorer—once you've got these, you can run a random search to find the best parameters for your model." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Implementing RandomizedSearchCV\n", "You are hoping that using a random search algorithm will help you improve predictions for a class assignment. You professor has challenged your class to predict the overall final exam average score.\n", "\n", "In preparation for completing a random search, you have created:\n", "\n", "- `param_dist`: the hyperparameter distributions\n", "- `rfr`: a random forest regression model\n", "- `scorer`: a scoring method to use" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import RandomizedSearchCV\n", "\n", "# Build a random search using param_dist, rfr, and scorer\n", "random_search = RandomizedSearchCV(estimator=rfr,\n", " param_distributions=param_dist,\n", " n_iter=10,\n", " cv=5,\n", " scoring=scorer\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Although it takes a lot of steps, hyperparameter tuning with random search is well worth it and can improve the accuracy of your models. Plus, you are already using cross-validation to validate your best model." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Selecting your final model\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Best classification accuracy\n", "You are in a competition at work to build the best model for predicting the winner of a Tic-Tac-Toe game. You already ran a random search and saved the results of the most accurate model to `rs`." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "tic_tac_toe = pd.read_csv('./dataset/tic-tac-toe.csv')\n", "# Create dummy variables using pandas\n", "X = pd.get_dummies(tic_tac_toe.iloc[:, 0:9])\n", "y = tic_tac_toe.iloc[:, 9]\n", "y = tic_tac_toe['Class'].apply(lambda x: 1 if x == 'positive' else 0)" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "from sklearn.ensemble import RandomForestClassifier\n", "\n", "rfc = RandomForestClassifier(max_features='auto')\n", "\n", "param_dist = {\n", " 'max_depth': [2, 4, 8, 12],\n", " 'min_samples_split': [2, 4, 6, 8],\n", " 'n_estimators':[10, 20, 30]\n", "}\n", "\n", "rs = RandomizedSearchCV(estimator=rfc, param_distributions=param_dist, n_iter=10, \n", " cv=5, scoring=None, random_state=1111)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "RandomizedSearchCV(cv=5, error_score=nan,\n", " estimator=RandomForestClassifier(bootstrap=True,\n", " ccp_alpha=0.0,\n", " class_weight=None,\n", " criterion='gini',\n", " max_depth=None,\n", " max_features='auto',\n", " max_leaf_nodes=None,\n", " max_samples=None,\n", " min_impurity_decrease=0.0,\n", " min_impurity_split=None,\n", " min_samples_leaf=1,\n", " min_samples_split=2,\n", " min_weight_fraction_leaf=0.0,\n", " n_estimators=100,\n", " n_jobs=None,\n", " oob_score=False,\n", " random_state=None,\n", " verbose=0,\n", " warm_start=False),\n", " iid='deprecated', n_iter=10, n_jobs=None,\n", " param_distributions={'max_depth': [2, 4, 8, 12],\n", " 'min_samples_split': [2, 4, 6, 8],\n", " 'n_estimators': [10, 20, 30]},\n", " pre_dispatch='2*n_jobs', random_state=1111, refit=True,\n", " return_train_score=False, scoring=None, verbose=0)" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rs.fit(X, y)" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'n_estimators': 20, 'min_samples_split': 4, 'max_depth': 12}" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rs.best_params_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "From the result of random search, we can find that the optimal parameter is:\n", "```python\n", "{'n_estimators': 20, 'min_samples_split': 4, 'max_depth': 12}\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Selecting the best precision model\n", "Your boss has offered to pay for you to see three sports games this year. Of the 41 home games your favorite team plays, you want to ensure you go to three home games that they will definitely win. You build a model to decide which games your team will win.\n", "\n", "To do this, you will build a random search algorithm and focus on model precision (to ensure your team wins). You also want to keep track of your best model and best parameters, so that you can use them again next year (if the model does well, of course)." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "sports = pd.read_csv('./dataset/sports.csv')\n", "X = sports.drop('win', axis=1)\n", "y = sports['win']" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "rfc = RandomForestClassifier()\n", "\n", "param_dist = {\n", " 'max_depth': range(2, 12, 2),\n", " 'min_samples_split': range(2, 12, 2),\n", " 'n_estimators': [10, 25, 50]\n", "}" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The accuracy for each run was: [0.91107742 0.75556059 0.68148435 0.88886879 0.88136354 0.88578829\n", " 0.68821403 0.81709152 0.87505491 0.90675415].\n", "The best accuracy for a single model was: 0.9110774179109917\n" ] } ], "source": [ "from sklearn.metrics import precision_score\n", "\n", "# Create a precision scorer\n", "precision = make_scorer(precision_score)\n", "\n", "# Finalize the random search\n", "rs = RandomizedSearchCV(estimator=rfc, param_distributions=param_dist,\n", " scoring=precision, cv=5, n_iter=10,\n", " random_state=1111)\n", "\n", "rs.fit(X, y)\n", "\n", "# Print the mean test scores:\n", "print('The accuracy for each run was: {}.'.format(rs.cv_results_['mean_test_score']))\n", "# Print the best model scores:\n", "print('The best accuracy for a single model was: {}'.format(rs.best_score_))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Your model's precision was 91%! The best model accurately predicts a winning game 91% of the time. If you look at the mean test scores, you can tell some of the other parameter sets did really poorly. Also, since you used cross-validation, you can be confident in your predictions. Well done!" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }