{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Analysis of hyperparameter search results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the previous notebook we showed how to implement a randomized search for\n", "tuning the hyperparameters of a `HistGradientBoostingClassifier` to fit the\n", "`adult_census` dataset. In practice, a randomized hyperparameter search is\n", "usually run with a large number of iterations." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In order to avoid the computational cost and still make a decent analysis, we\n", "load the results obtained from a similar search with 500 iterations." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "\n", "cv_results = pd.read_csv(\n", " \"../figures/randomized_search_results.csv\", index_col=0\n", ")\n", "cv_results" ] }, { "cell_type": "markdown", "metadata": { "lines_to_next_cell": 2 }, "source": [ "We define a function to remove the prefixes in the hyperparameters column\n", "names." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def shorten_param(param_name):\n", " if \"__\" in param_name:\n", " return param_name.rsplit(\"__\", 1)[1]\n", " return param_name\n", "\n", "\n", "cv_results = cv_results.rename(shorten_param, axis=1)\n", "cv_results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we have more than 2 parameters in our randomized-search, we cannot\n", "visualize the results using a heatmap. We could still do it pair-wise, but\n", "having a two-dimensional projection of a multi-dimensional problem can lead to\n", "a wrong interpretation of the scores." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import seaborn as sns\n", "import numpy as np\n", "\n", "df = pd.DataFrame(\n", " {\n", " \"max_leaf_nodes\": cv_results[\"max_leaf_nodes\"],\n", " \"learning_rate\": cv_results[\"learning_rate\"],\n", " \"score_bin\": pd.cut(\n", " cv_results[\"mean_test_score\"], bins=np.linspace(0.5, 1.0, 6)\n", " ),\n", " }\n", ")\n", "sns.set_palette(\"YlGnBu_r\")\n", "ax = sns.scatterplot(\n", " data=df,\n", " x=\"max_leaf_nodes\",\n", " y=\"learning_rate\",\n", " hue=\"score_bin\",\n", " s=50,\n", " color=\"k\",\n", " edgecolor=None,\n", ")\n", "ax.set_xscale(\"log\")\n", "ax.set_yscale(\"log\")\n", "\n", "_ = ax.legend(\n", " title=\"mean_test_score\", loc=\"center left\", bbox_to_anchor=(1, 0.5)\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the previous plot we see that the top performing values are located in a\n", "band of learning rate between 0.01 and 1.0, but we have no control in how the\n", "other hyperparameters interact with such values for the learning rate.\n", "Instead, we can visualize all the hyperparameters at the same time using a\n", "parallel coordinates plot." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import plotly.express as px\n", "\n", "fig = px.parallel_coordinates(\n", " cv_results.rename(shorten_param, axis=1).apply(\n", " {\n", " \"learning_rate\": np.log10,\n", " \"max_leaf_nodes\": np.log2,\n", " \"max_bins\": np.log2,\n", " \"min_samples_leaf\": np.log10,\n", " \"l2_regularization\": np.log10,\n", " \"mean_test_score\": lambda x: x,\n", " }\n", " ),\n", " color=\"mean_test_score\",\n", " color_continuous_scale=px.colors.sequential.Viridis,\n", ")\n", "fig.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Note
\n", "We transformed most axis values by taking a log10 or log2 to\n", "spread the active ranges and improve the readability of the plot.
\n", "