{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Introduction to scikit-learn: basic model hyper-parameters tuning\n", "\n", "The process of learning a predictive model is driven by a set of internal\n", "parameters and a set of training data. These internal parameters are called\n", "hyper-parameters and are specific for each family of models. In addition, a\n", "specific set of parameters are optimal for a specific dataset and thus they\n", "need to be optimized.\n", "\n", "This notebook shows:\n", "* the influence of changing model parameters;\n", "* how to tune these hyper-parameters;\n", "* how to evaluate the model performance together with hyper-parameter\n", " tuning." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "\n", "df = pd.read_csv(\n", " \"https://www.openml.org/data/get_csv/1595261/adult-census.csv\")\n", "# Or use the local copy:\n", "# df = pd.read_csv(os.path.join(\"..\", \"datasets\", \"adult-census.csv\"))" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([' <=50K', ' <=50K', ' >50K', ..., ' <=50K', ' <=50K', ' >50K'],\n", " dtype=object)" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "target_name = \"class\"\n", "target = df[target_name].to_numpy()\n", "target" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ageworkclasseducationeducation-nummarital-statusoccupationrelationshipracesexcapital-gaincapital-losshours-per-weeknative-country
025Private11th7Never-marriedMachine-op-inspctOwn-childBlackMale0040United-States
138PrivateHS-grad9Married-civ-spouseFarming-fishingHusbandWhiteMale0050United-States
228Local-govAssoc-acdm12Married-civ-spouseProtective-servHusbandWhiteMale0040United-States
344PrivateSome-college10Married-civ-spouseMachine-op-inspctHusbandBlackMale7688040United-States
418?Some-college10Never-married?Own-childWhiteFemale0030United-States
\n", "
" ], "text/plain": [ " age workclass education education-num marital-status \\\n", "0 25 Private 11th 7 Never-married \n", "1 38 Private HS-grad 9 Married-civ-spouse \n", "2 28 Local-gov Assoc-acdm 12 Married-civ-spouse \n", "3 44 Private Some-college 10 Married-civ-spouse \n", "4 18 ? Some-college 10 Never-married \n", "\n", " occupation relationship race sex capital-gain \\\n", "0 Machine-op-inspct Own-child Black Male 0 \n", "1 Farming-fishing Husband White Male 0 \n", "2 Protective-serv Husband White Male 0 \n", "3 Machine-op-inspct Husband Black Male 7688 \n", "4 ? Own-child White Female 0 \n", "\n", " capital-loss hours-per-week native-country \n", "0 0 40 United-States \n", "1 0 50 United-States \n", "2 0 40 United-States \n", "3 0 40 United-States \n", "4 0 30 United-States " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = df.drop(columns=[target_name, \"fnlwgt\"])\n", "data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once the dataset is loaded, we split it into a training and testing sets." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "df_train, df_test, target_train, target_test = train_test_split(\n", " data, target, random_state=42)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, we define the preprocessing pipeline to transform differently\n", "the numerical and categorical data." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "from sklearn.compose import ColumnTransformer\n", "from sklearn.preprocessing import OrdinalEncoder\n", "\n", "categorical_columns = [\n", " 'workclass', 'education', 'marital-status', 'occupation',\n", " 'relationship', 'race', 'native-country', 'sex']\n", "\n", "categories = [\n", " data[column].unique() for column in data[categorical_columns]]\n", "\n", "categorical_preprocessor = OrdinalEncoder(categories=categories)\n", "\n", "preprocessor = ColumnTransformer([\n", " ('cat-preprocessor', categorical_preprocessor,\n", " categorical_columns),], remainder='passthrough',\n", " sparse_threshold=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, we use a tree-based classifier (i.e. histogram gradient-boosting) to\n", "predict whether or not a person earns more than 50,000 dollars a year." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The test accuracy score of the gradient boosting pipeline is: 0.88\n", "CPU times: user 10.9 s, sys: 342 ms, total: 11.2 s\n", "Wall time: 5.49 s\n" ] } ], "source": [ "%%time\n", "# for the moment this line is required to import HistGradientBoostingClassifier\n", "from sklearn.experimental import enable_hist_gradient_boosting\n", "from sklearn.ensemble import HistGradientBoostingClassifier\n", "from sklearn.pipeline import Pipeline\n", "\n", "model = Pipeline([\n", " (\"preprocessor\", preprocessor),\n", " (\"classifier\",\n", " HistGradientBoostingClassifier(max_leaf_nodes=16,\n", " learning_rate=0.05,\n", " random_state=42)),])\n", "model.fit(df_train, target_train)\n", "\n", "print(\n", " f\"The test accuracy score of the gradient boosting pipeline is: \"\n", " f\"{model.score(df_test, target_test):.2f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Quizz\n", "\n", "1. What is the default value of the `learning_rate` parameter of the `HistGradientBoostingClassifier` class? ([link to the API documentation](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.HistGradientBoostingClassifier.html#sklearn-ensemble-histgradientboostingclassifier))\n", "\n", "2. Try to edit the code of the previous cell to set the learning rate parameter to 10. Does this increase the accuracy of the model?\n", "\n", "3. Decrease progressively value of `learning_rate`: can you find a value that yields an accuracy higher than 0.87?\n", "\n", "4. Fix `learning_rate` to 0.05 and try setting the value of `max_leaf_nodes` to the minimum value of 2. Does not improve the accuracy?\n", "\n", "5. Try to progressively increase the value of `max_leaf_nodes` to 256 by taking powers of 2. What do you observe?" ] }, { "cell_type": "markdown", "metadata": { "lines_to_next_cell": 0 }, "source": [ "## The issue of finding the best model parameters\n", "\n", "In the previous example, we created an histogram gradient-boosting classifier\n", "using the default parameters by omitting to explicitely set these parameters.\n", "\n", "However, there is no reasons that this set of parameters are optimal for our\n", "dataset. For instance, fine-tuning the histogram gradient-boosting can be\n", "achieved by finding the best combination of the following parameters: (i)\n", "`learning_rate`, (ii) `min_samples_leaf`, and (iii) `max_leaf_nodes`.\n", "Nevertheless, finding this combination manually will be tedious. Indeed,\n", "there are relationship between these parameters which are difficult to find\n", "manually: increasing the depth of trees (increasing `max_samples_leaf`)\n", "should be associated with a lower learning-rate.\n", "\n", "Scikit-learn provides tools to explore and evaluate the parameters\n", "space." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Finding the best model hyper-parameters via exhaustive parameters search\n", "\n", "Our goal is to find the best combination of the parameters stated above.\n", "\n", "In short, we will set these parameters with some defined values, train our\n", "model on some data, and evaluate the model performance on some left out data.\n", "Ideally, we will select the parameters leading to the optimal performance on\n", "the testing set." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The first step is to find the name of the parameters to be set. We use the\n", "method `get_params()` to get this information. For instance, for a single\n", "model like the `HistGradientBoostingClassifier`, we can get the list such as:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The hyper-parameters are for a histogram GBDT model are:\n", "l2_regularization\n", "learning_rate\n", "loss\n", "max_bins\n", "max_depth\n", "max_iter\n", "max_leaf_nodes\n", "min_samples_leaf\n", "n_iter_no_change\n", "random_state\n", "scoring\n", "tol\n", "validation_fraction\n", "verbose\n", "warm_start\n" ] } ], "source": [ "print(\"The hyper-parameters are for a histogram GBDT model are:\")\n", "for param_name in HistGradientBoostingClassifier().get_params().keys(\n", "):\n", " print(param_name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When the model of interest is a `Pipeline`, i.e. a serie of transformers and\n", "a predictor, the name of the estimator will be added at the front of the\n", "parameter name with a double underscore (\"dunder\") in-between (e.g.\n", "`estimator__parameters`)." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The hyper-parameters are for the full-pipeline are:\n", "memory\n", "steps\n", "verbose\n", "preprocessor\n", "classifier\n", "preprocessor__n_jobs\n", "preprocessor__remainder\n", "preprocessor__sparse_threshold\n", "preprocessor__transformer_weights\n", "preprocessor__transformers\n", "preprocessor__verbose\n", "preprocessor__cat-preprocessor\n", "preprocessor__cat-preprocessor__categories\n", "preprocessor__cat-preprocessor__dtype\n", "classifier__l2_regularization\n", "classifier__learning_rate\n", "classifier__loss\n", "classifier__max_bins\n", "classifier__max_depth\n", "classifier__max_iter\n", "classifier__max_leaf_nodes\n", "classifier__min_samples_leaf\n", "classifier__n_iter_no_change\n", "classifier__random_state\n", "classifier__scoring\n", "classifier__tol\n", "classifier__validation_fraction\n", "classifier__verbose\n", "classifier__warm_start\n" ] } ], "source": [ "print(\"The hyper-parameters are for the full-pipeline are:\")\n", "for param_name in model.get_params().keys():\n", " print(param_name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The parameters that we want to set are:\n", "- `'classifier__learning_rate'`: this parameter will\n", " control the ability of a new tree to correct the error of the previous\n", " sequence of trees;\n", "- `'classifier__max_leaf_nodes'`: this parameter will\n", " control the depth of each tree." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercises:\n", "\n", "Use the previously defined model (called `model`) and using two nested `for`\n", "loops, make a search of the best combinations of the `learning_rate` and\n", "`max_leaf_nodes` parameters. In this regard, you will need to train and test\n", "the model by setting the parameters. The evaluation of the model should be\n", "performed using `cross_val_score`. We can propose to define the following\n", "parameters search:\n", "- `learning_rate` for the values 0.01, 0.1, and 1;\n", "- `max_leaf_nodes` for the values 5, 25, 45." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Automated parameter tuning via grid-search\n", "\n", "Instead of manually writting the two `for` loops, scikit-learn provides a\n", "class called `GridSearchCV` which implement the exhaustive search implemented\n", "during the exercise.\n", "\n", "Let see how to use the `GridSearchCV` estimator for doing such search.\n", "Since the grid-search will be costly, we will only explore the combination\n", "learning-rate and the maximum number of nodes." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/lesteve/miniconda3/envs/scikit-learn-tutorial/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py:706: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.\n", " \"timeout or by a memory leak.\", UserWarning\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "The test accuracy score of the grid-searched pipeline is: 0.88\n", "CPU times: user 8.52 s, sys: 302 ms, total: 8.82 s\n", "Wall time: 22.1 s\n" ] } ], "source": [ "%%time\n", "import numpy as np\n", "from sklearn.model_selection import GridSearchCV\n", "\n", "param_grid = {\n", " 'classifier__learning_rate': (0.05, 0.1, 0.5, 1, 5),\n", " 'classifier__max_leaf_nodes': (3, 10, 30, 100),}\n", "model_grid_search = GridSearchCV(model, param_grid=param_grid,\n", " n_jobs=4, cv=2)\n", "model_grid_search.fit(df_train, target_train)\n", "\n", "print(f\"The test accuracy score of the grid-searched pipeline is: \"\n", " f\"{model_grid_search.score(df_test, target_test):.2f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `GridSearchCV` estimator takes a `param_grid` parameter which defines\n", "all hyper-parameters and their associated values. The grid-search will be in\n", "charge of creating all possible combinations and test them.\n", "\n", "The number of combinations will be equal to the cardesian product of the\n", "number of values to explore for each parameter (e.g. in our example 3 x 3\n", "combinations). Thus, adding new parameters with their associated values to be\n", "explored become rapidly computationally expensive.\n", "\n", "Once the grid-search is fitted, it can be used as any other predictor by\n", "calling `predict` and `predict_proba`. Internally, it will use the model with\n", "the best parameters found during `fit`.\n", "\n", "Get predictions for the 5 first samples using the estimator with the best\n", "parameters." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([' <=50K', ' <=50K', ' >50K', ' <=50K', ' >50K'], dtype=object)" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_grid_search.predict(df_test.iloc[0:5])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can know about these parameters by looking at the `best_params_`\n", "attribute." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The best set of parameters is: {'classifier__learning_rate': 0.1, 'classifier__max_leaf_nodes': 30}\n" ] } ], "source": [ "print(f\"The best set of parameters is: \"\n", " f\"{model_grid_search.best_params_}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In addition, we can inspect all results which are stored in the attribute\n", "`cv_results_` of the grid-search. We will filter some specific columns to\n", "from these results" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
mean_fit_timestd_fit_timemean_score_timestd_score_timeparam_classifier__learning_rateparam_classifier__max_leaf_nodesparamssplit0_test_scoresplit1_test_scoremean_test_scorestd_test_scorerank_test_score
61.3655160.0278080.3057870.0364080.130{'classifier__learning_rate': 0.1, 'classifier...0.8689670.8692330.8691000.0001331
21.3614450.1439950.3518520.0298140.0530{'classifier__learning_rate': 0.05, 'classifie...0.8677660.8679220.8678440.0000782
33.4681210.0729540.4978760.0357720.05100{'classifier__learning_rate': 0.05, 'classifie...0.8683660.8669400.8676530.0007133
50.8841600.0224270.3097840.0263640.110{'classifier__learning_rate': 0.1, 'classifier...0.8664010.8679770.8671890.0007884
80.5765200.0436080.2289520.0319580.53{'classifier__learning_rate': 0.5, 'classifier...0.8647630.8671580.8659610.0011985
\n", "
" ], "text/plain": [ " mean_fit_time std_fit_time mean_score_time std_score_time \\\n", "6 1.365516 0.027808 0.305787 0.036408 \n", "2 1.361445 0.143995 0.351852 0.029814 \n", "3 3.468121 0.072954 0.497876 0.035772 \n", "5 0.884160 0.022427 0.309784 0.026364 \n", "8 0.576520 0.043608 0.228952 0.031958 \n", "\n", " param_classifier__learning_rate param_classifier__max_leaf_nodes \\\n", "6 0.1 30 \n", "2 0.05 30 \n", "3 0.05 100 \n", "5 0.1 10 \n", "8 0.5 3 \n", "\n", " params split0_test_score \\\n", "6 {'classifier__learning_rate': 0.1, 'classifier... 0.868967 \n", "2 {'classifier__learning_rate': 0.05, 'classifie... 0.867766 \n", "3 {'classifier__learning_rate': 0.05, 'classifie... 0.868366 \n", "5 {'classifier__learning_rate': 0.1, 'classifier... 0.866401 \n", "8 {'classifier__learning_rate': 0.5, 'classifier... 0.864763 \n", "\n", " split1_test_score mean_test_score std_test_score rank_test_score \n", "6 0.869233 0.869100 0.000133 1 \n", "2 0.867922 0.867844 0.000078 2 \n", "3 0.866940 0.867653 0.000713 3 \n", "5 0.867977 0.867189 0.000788 4 \n", "8 0.867158 0.865961 0.001198 5 " ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cv_results = pd.DataFrame(model_grid_search.cv_results_).sort_values(\n", " \"mean_test_score\", ascending=False)\n", "cv_results.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us focus on the most interesting columns and shorten the parameter names to remove the `\"param_classifier__\"` prefix for readability:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "# get the parameter names\n", "column_results = [f\"param_{name}\" for name in param_grid.keys()]\n", "column_results += [\n", " \"mean_test_score\", \"std_test_score\", \"rank_test_score\"]\n", "cv_results = cv_results[column_results]" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
learning_ratemax_leaf_nodesmean_test_scorestd_test_scorerank_test_score
60.1300.8691000.0001331
20.05300.8678440.0000782
30.051000.8676530.0007133
50.1100.8671890.0007884
80.530.8659610.0011985
70.11000.8659330.0010146
90.5100.8657690.0003597
12130.8652510.0001068
10.05100.8606370.0006799
100.5300.8577710.00093210
131100.8537300.00568211
40.130.8534570.00016812
00.0530.8489260.00141513
110.51000.8457590.00142414
141300.8251210.00333515
1511000.8009880.00004916
16530.7589470.00000717
175100.7589470.00000717
1951000.7589470.00000717
185300.4999930.25894720
\n", "
" ], "text/plain": [ " learning_rate max_leaf_nodes mean_test_score std_test_score \\\n", "6 0.1 30 0.869100 0.000133 \n", "2 0.05 30 0.867844 0.000078 \n", "3 0.05 100 0.867653 0.000713 \n", "5 0.1 10 0.867189 0.000788 \n", "8 0.5 3 0.865961 0.001198 \n", "7 0.1 100 0.865933 0.001014 \n", "9 0.5 10 0.865769 0.000359 \n", "12 1 3 0.865251 0.000106 \n", "1 0.05 10 0.860637 0.000679 \n", "10 0.5 30 0.857771 0.000932 \n", "13 1 10 0.853730 0.005682 \n", "4 0.1 3 0.853457 0.000168 \n", "0 0.05 3 0.848926 0.001415 \n", "11 0.5 100 0.845759 0.001424 \n", "14 1 30 0.825121 0.003335 \n", "15 1 100 0.800988 0.000049 \n", "16 5 3 0.758947 0.000007 \n", "17 5 10 0.758947 0.000007 \n", "19 5 100 0.758947 0.000007 \n", "18 5 30 0.499993 0.258947 \n", "\n", " rank_test_score \n", "6 1 \n", "2 2 \n", "3 3 \n", "5 4 \n", "8 5 \n", "7 6 \n", "9 7 \n", "12 8 \n", "1 9 \n", "10 10 \n", "13 11 \n", "4 12 \n", "0 13 \n", "11 14 \n", "14 15 \n", "15 16 \n", "16 17 \n", "17 17 \n", "19 17 \n", "18 20 " ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def shorten_param(param_name):\n", " if \"__\" in param_name:\n", " return param_name.rsplit(\"__\", 1)[1]\n", " return param_name\n", "\n", "\n", "cv_results = cv_results.rename(shorten_param, axis=1)\n", "cv_results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With only 2 parameters, we might want to visualize the grid-search as a\n", "heatmap. We need to transform our `cv_results` into a dataframe where the\n", "rows will correspond to the learning-rate values and the columns will\n", "correspond to the maximum number of leaf and the content of the dataframe\n", "will be the mean test scores." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
max_leaf_nodes31030100
learning_rate
0.050.8489260.8606370.8678440.867653
0.100.8534570.8671890.8691000.865933
0.500.8659610.8657690.8577710.845759
1.000.8652510.8537300.8251210.800988
5.000.7589470.7589470.4999930.758947
\n", "
" ], "text/plain": [ "max_leaf_nodes 3 10 30 100\n", "learning_rate \n", "0.05 0.848926 0.860637 0.867844 0.867653\n", "0.10 0.853457 0.867189 0.869100 0.865933\n", "0.50 0.865961 0.865769 0.857771 0.845759\n", "1.00 0.865251 0.853730 0.825121 0.800988\n", "5.00 0.758947 0.758947 0.499993 0.758947" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pivoted_cv_results = cv_results.pivot_table(\n", " values=\"mean_test_score\", index=[\"learning_rate\"],\n", " columns=[\"max_leaf_nodes\"])\n", "\n", "pivoted_cv_results" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "from seaborn import heatmap\n", "\n", "ax = heatmap(pivoted_cv_results, annot=True, cmap=\"YlGnBu\", vmin=0.7,\n", " vmax=0.9)\n", "ax.invert_yaxis()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The above tables highlights the following things:\n", "\n", "- for too high values of the value of `learning_rate`, the performance of the model is degraded and adjusting the value of `max_leaf_nodes` cannot fix that problem;\n", "- outside of this pathological region, we observe that the optimal choice of `max_leaf_nodes` depends on the value of `learning_rate`;\n", "- in particular, we observe a \"diagonal\" of good models with an accuracy close to the maximal of 0.87: when the value of `max_leaf_nodes` is increased, one should increase the value of `learning_rate` accordingly to preserve a good accuracy.\n", "\n", "The precise meaning of those two parameters will be explained in a latter notebook.\n", "\n", "For now we will note that, in general, **there is no unique optimal parameter setting**: 6 models out of the 16 parameter configuration reach the maximal accuracy (up to smal random fluctuations caused by the sampling of the training set)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Hyper-parameter tuning with Random Search\n", "\n", "\n", "With the `GridSearchCV` estimator, the parameters need to be specified\n", "explicitely. We mentioned that exploring a large number of values for\n", "different parameters will be quickly untractable.\n", "\n", "Instead, we can randomly generate the parameter candidates. The\n", "`RandomSearchCV` allows for such stochastic search. It is used similarly to\n", "the `GridSearchCV` but the sampling distributions need to be specified\n", "instead of the parameter values. For instance, we will draw candidates using\n", "a log-uniform distribution also called reciprocal distribution. In addition,\n", "we will optimize 2 other parameters:\n", "- `max_iter`: it corresponds to the number of trees in the ensemble;\n", "- `min_samples_leaf`: it corresponds to the minimum number of samples\n", " required in a leaf." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The test accuracy score of the best model is 0.88\n" ] } ], "source": [ "from scipy.stats import reciprocal\n", "from sklearn.model_selection import RandomizedSearchCV\n", "from pprint import pprint\n", "\n", "\n", "class reciprocal_int:\n", " \"\"\"Integer valued version of the log-uniform distribution\"\"\"\n", " def __init__(self, a, b):\n", " self._distribution = reciprocal(a, b)\n", "\n", " def rvs(self, *args, **kwargs):\n", " \"\"\"Random variable sample\"\"\"\n", " return self._distribution.rvs(*args, **kwargs).astype(int)\n", "\n", "\n", "param_distributions = {\n", " 'classifier__l2_regularization': reciprocal(1e-6, 1e3),\n", " 'classifier__learning_rate': reciprocal(0.001, 10),\n", " 'classifier__max_leaf_nodes': reciprocal_int(2, 256),\n", " 'classifier__min_samples_leaf': reciprocal_int(1, 100),\n", " 'classifier__max_bins': reciprocal_int(2, 255),}\n", "model_random_search = RandomizedSearchCV(\n", " model, param_distributions=param_distributions, n_iter=10,\n", " n_jobs=4, cv=5)\n", "model_random_search.fit(df_train, target_train)\n", "\n", "print(f\"The test accuracy score of the best model is \"\n", " f\"{model_random_search.score(df_test, target_test):.2f}\")" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The best parameters are:\n", "{'classifier__l2_regularization': 2.855328641639947e-05,\n", " 'classifier__learning_rate': 0.713602698128458,\n", " 'classifier__max_bins': 191,\n", " 'classifier__max_leaf_nodes': 5,\n", " 'classifier__min_samples_leaf': 2}\n" ] } ], "source": [ "print(\"The best parameters are:\")\n", "pprint(model_random_search.best_params_)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can inspect the results using the attributes `cv_results` as we previously\n", "did." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
l2_regularizationlearning_ratemax_leaf_nodesmin_samples_leafmax_binsmean_test_scorestd_test_scorerank_test_score
42.85533e-050.713603521910.8663430.0027101
70.001792180.094623616051550.8544680.0028462
124.43740.37101518074610.8536760.0024743
60.0002020980.026101727360.8408450.0036664
91.7478e-060.0176826516300.8375420.0022705
81.93292e-050.0788139233170.8374330.0043296
21.972831.578957530220.8133000.0042037
00.0001978350.0050048562340.7589470.0000138
314.67070.0040739142240.7589470.0000138
52.14164e-060.0025080870591400.7589470.0000138
\n", "
" ], "text/plain": [ " l2_regularization learning_rate max_leaf_nodes min_samples_leaf max_bins \\\n", "4 2.85533e-05 0.713603 5 2 191 \n", "7 0.00179218 0.0946236 160 51 55 \n", "1 24.4374 0.371015 180 74 61 \n", "6 0.000202098 0.0261017 27 3 6 \n", "9 1.7478e-06 0.0176826 5 16 30 \n", "8 1.93292e-05 0.0788139 2 33 17 \n", "2 1.97283 1.57895 75 30 22 \n", "0 0.000197835 0.00500485 6 23 4 \n", "3 14.6707 0.00407391 4 22 4 \n", "5 2.14164e-06 0.00250808 70 59 140 \n", "\n", " mean_test_score std_test_score rank_test_score \n", "4 0.866343 0.002710 1 \n", "7 0.854468 0.002846 2 \n", "1 0.853676 0.002474 3 \n", "6 0.840845 0.003666 4 \n", "9 0.837542 0.002270 5 \n", "8 0.837433 0.004329 6 \n", "2 0.813300 0.004203 7 \n", "0 0.758947 0.000013 8 \n", "3 0.758947 0.000013 8 \n", "5 0.758947 0.000013 8 " ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# get the parameter names\n", "column_results = [\n", " f\"param_{name}\" for name in param_distributions.keys()]\n", "column_results += [\n", " \"mean_test_score\", \"std_test_score\", \"rank_test_score\"]\n", "\n", "cv_results = pd.DataFrame(model_random_search.cv_results_)\n", "cv_results = cv_results[column_results].sort_values(\n", " \"mean_test_score\", ascending=False)\n", "cv_results = cv_results.rename(shorten_param, axis=1)\n", "cv_results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In practice, a randomized hyper-parameter search is usually run with a large number of\n", "iterations. In order to avoid the computation cost and still make a decent\n", "analysis, we load the results obtained from a similar search with 200\n", "iterations." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "# model_random_search = RandomizedSearchCV(\n", "# model, param_distributions=param_distributions, n_iter=500,\n", "# n_jobs=4, cv=5)\n", "# model_random_search.fit(df_train, target_train)\n", "# cv_results = pd.DataFrame(model_random_search.cv_results_)\n", "# cv_results.to_csv(\"../figures/randomized_search_results.csv\")" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "cv_results = pd.read_csv(\"../figures/randomized_search_results.csv\",\n", " index_col=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we have more than 2 paramters in our grid-search, we cannot visualize the\n", "results using a heatmap. However, we can us a parallel coordinates plot." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
l2_regularizationlearning_ratemax_leaf_nodesmin_samples_leafmax_binsmean_test_scorestd_test_scorerank_test_score
3570.0000263.075318368310.2410530.000013500
2000.0004446.23632522300.3446290.207156499
4130.0000018.8285746411440.4482050.253714497
3440.0000037.09107951950.4482050.253714497
2320.0000979.97682328530.4482050.253714496
...........................
3274.7338080.0367866152410.8696730.0024175
3282.0362320.22470228492360.8698370.0008084
214.9949180.0770475371920.8707930.0019933
3430.0004040.24450315152290.8713390.0027412
2080.0117750.0766532421550.8713930.0015881
\n", "

500 rows × 8 columns

\n", "
" ], "text/plain": [ " l2_regularization learning_rate max_leaf_nodes min_samples_leaf \\\n", "357 0.000026 3.075318 3 68 \n", "200 0.000444 6.236325 2 2 \n", "413 0.000001 8.828574 64 1 \n", "344 0.000003 7.091079 5 1 \n", "232 0.000097 9.976823 28 5 \n", ".. ... ... ... ... \n", "327 4.733808 0.036786 61 5 \n", "328 2.036232 0.224702 28 49 \n", "21 4.994918 0.077047 53 7 \n", "343 0.000404 0.244503 15 15 \n", "208 0.011775 0.076653 24 2 \n", "\n", " max_bins mean_test_score std_test_score rank_test_score \n", "357 31 0.241053 0.000013 500 \n", "200 30 0.344629 0.207156 499 \n", "413 144 0.448205 0.253714 497 \n", "344 95 0.448205 0.253714 497 \n", "232 3 0.448205 0.253714 496 \n", ".. ... ... ... ... \n", "327 241 0.869673 0.002417 5 \n", "328 236 0.869837 0.000808 4 \n", "21 192 0.870793 0.001993 3 \n", "343 229 0.871339 0.002741 2 \n", "208 155 0.871393 0.001588 1 \n", "\n", "[500 rows x 8 columns]" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(cv_results[column_results].rename(\n", " shorten_param, axis=1).sort_values(\"mean_test_score\"))" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/html": [ " \n", " " ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "dimensions": [ { "label": "learning_rate", "values": [ -0.2595781951293343, -2.940675837596371, -2.369159728958732, -2.88453485761083, 0.7329870491877641, 0.2983046092805795, 0.5548569578245499, 0.33555983063809386, -0.686708835115216, -2.572870871572566, -1.6744133774307115, -0.42397615518455334, 0.8020929981107696, -1.251943943652591, 0.5316785497439898, -2.5027136892054846, -0.15829490353407974, -2.2292362288789365, -1.104156402277492, -2.2621942159399344, -1.3803933064246605, -1.113243543391706, -2.126969402719152, 0.5272500424575628, -2.0508649757917716, 0.7640162382545799, -1.5450673866012914, -0.8664751801081518, -0.11378287149568458, -0.9674805062568956, 0.6714092969754278, 0.5221675732189461, -0.9207772602951074, -2.9972108230084324, -0.6331752161225435, -0.26777873315563067, -2.4964235743377574, -2.7623146438184683, -2.290934346475262, -2.0056333921241847, -0.7509468749538888, -2.024531632848069, -0.9527541739407162, -0.6350720229821465, 0.5039668725099286, -1.1953425850496422, -0.8278258904106357, -1.6149092150039808, -1.7050543853203157, 0.8684292704594809, -2.1972496783517763, -0.059924436071205886, 0.9868379844080767, -2.331928543945373, 0.1362300808596091, 0.7111224902322464, -2.2192856976015483, 0.6815664897297933, 0.5234013969636965, -1.450928963121446, 0.5631888404618968, -2.569842234704439, -2.3672951213128646, -2.8279448214900156, -0.7376351766438147, -0.6142740988180662, -1.3729748169323202, -0.4641858298951105, -0.10198506597505563, -0.571717178472003, -0.5820140729090038, -0.21628665183337856, -2.157054628169675, -0.6253573617827728, -2.89739210927367, -0.22858863860033235, -0.5027245682114554, -2.977722530926816, -0.26342893200649486, -1.9699156440415724, -1.0127512843295423, -2.5288212937745147, -1.8928649660788754, -2.9967515454409686, 0.8254943525927452, 0.27593617858074093, -2.03968915360755, -1.6416629609108497, 0.5778657682616428, -2.982876774981418, -0.3452841579128365, -2.9195366651106274, -1.806587546827596, -2.8829264330859554, -1.7954981070522185, -2.8427283857705574, 0.3400136147919248, -2.4261457706983496, 0.9019119101461754, -1.8126918727600447, 0.7269860430843247, -1.9543926354114896, -2.091475173694103, -0.23532455617425052, -1.013773816551081, -0.49406850682630793, -2.6223408695386796, -2.9869358549812697, -2.407142778077177, 0.5778944046948142, -1.7260949832245047, 0.07617734413906702, -0.6830617703231066, -0.7785589643731572, -2.387746715792704, -0.49765875311108404, -1.993143694181787, -1.0549721193331658, -0.16867137811816416, -2.067619512555646, 0.1625476471236538, -0.6811051367314556, -1.033144790954561, 0.5385530667431349, 0.21290853258177658, -0.3617283507784492, 0.9728884713061225, -0.574388095702488, -1.6828382282110415, -2.92304569272864, 0.48790949398671785, 0.23497416747488217, -1.366166610321462, -2.2786903503307108, 0.943989970873992, 0.9517825548982595, -2.7735047520087925, -2.6389890869411663, -0.4074889671841264, -2.380590275076169, 0.18801705350590625, -2.331186607994521, -2.1124358686824687, -0.23294534333594186, -1.4104290336283283, -2.6823650381356843, -0.13552056556410538, -2.215153906457735, -0.6392499167325946, 0.7158616074000768, -2.53545583576139, 0.14958095357325132, 0.9386757653126838, -0.216914695326369, -0.3755893809451716, -0.6542251320297123, 0.2914909598302347, -1.7947769295771505, -2.43910997128511, -1.0247913589420081, -0.3856128181869884, -2.044647573191102, -0.9708910808880842, -0.6772710263226256, -1.3270596053728192, 0.19621261833971015, -0.3394937547563792, 0.6923543995460961, -2.2140393786262798, -1.677781858427228, -0.07975458447210347, -2.9465425349979455, -2.0613014897079465, 0.38390753989779686, -2.8212890364904335, -2.926208567768631, -0.022349164960157278, -0.30507051149543646, -1.5258045797085962, 0.6277275787793015, -2.5037069610732763, -0.08830307256367083, -0.9959860119904431, 0.6085355130539387, -0.07453237062554893, 0.13365009178239312, -2.950238040184685, -0.07496914891681476, -1.1932162010580067, -1.3157391653600738, -2.4746572933164974, 0.19209058449846725, -0.8352724459802809, 0.7280300027585414, -2.062118983326392, -0.6380873423249511, -1.1162806088930748, 0.2051826679870796, 0.2716310568307434, 0.4325192833847585, 0.7949287283610547, -2.750335640431575, -0.5049799519156961, -1.5443342425782172, -1.660318115406163, -0.12478197102045208, -2.615683604387317, 0.3221506536204322, -1.115468778843912, -2.293829216128953, 0.3506044542138058, 0.9221914954820294, -0.5891681688774794, -0.07778000891895859, -2.6707152997754187, -0.6767870369652282, -0.8147678711585109, -0.24743718241949603, -1.4177162600617508, 0.3923659310689551, -0.9896071331663348, -0.5785982126723903, -2.013556114145162, 0.1906952441896395, -2.2488023733104043, 0.9864567134266787, 0.46860611916245365, 0.41282435520654387, -1.3623866731245804, -0.023620240260366805, -0.8575685764574619, -1.0703932842585706, 0.9989922463661269, -2.021713294992165, -0.9412904464048237, 0.005806720101103885, 0.8537125811919135, 0.4201562543908151, 0.6648714156170695, -0.5098647610026372, 0.32565516571956016, -2.3258765936140513, -0.34295041572972446, 0.4758155016663239, 0.42581729402206303, 0.0447361381921443, 0.6871758540917687, -1.8470212115362963, -2.980012722205905, 0.1120622687254249, -0.7385091756960902, -2.3349159529952495, -0.9334491631900034, 0.29958847185310183, -2.3876192650587225, -0.14527536869397564, -0.6229002069662578, -2.128957976386775, 0.8095194950770059, -2.2979772454964325, -2.2297325967696824, 0.026386038455113874, 0.09582737418183254, 0.39529389203921594, 0.5574807127781858, -0.8163382271505708, -2.2349183505089947, 0.2634659290759234, 0.9191916448362186, 0.7264160046763256, -1.1541755543173653, -1.6530030102277427, 0.7005936209731911, -2.2318896426536603, -1.3763755429192206, 0.3947512302715146, 0.5705627478488919, -1.3540682300218516, 0.2975555982751458, -1.1421977301684074, -0.2520784883238506, -0.3122977581294011, -0.5993676058974201, -1.414646700831466, -0.8151255547393013, 0.23160221472264156, -1.6427921486521146, -1.034711110958109, -2.958102958178784, -2.843424306406283, -2.017042366042574, 0.6371680362924792, -2.9694620618011234, 0.34736245203357896, -2.023500661493724, -0.5976210779507518, -1.004379030215694, -2.633085691352211, -2.705049693659505, -0.34025248190510243, -0.666661710752591, 0.7736403002558481, -0.7391307047408127, -2.0553372732158, -1.6477234073774585, 0.1265617648055017, -0.9550866490985003, -1.9014239291079034, 0.15683280111714024, -1.01079180603138, -0.7512885072650999, 0.9755691585716635, -0.7003235037320001, -1.5032966735571494, -1.7344539536083126, -2.256003056160692, -1.1935420850192795, -1.3753491462554963, -2.569543195656894, -0.23072083735969515, -2.737957131484694, -1.7102557464750836, -2.2322457876651924, 0.9861941971270552, -1.8877018465524964, 0.09439008888290164, -0.9512417267882758, -1.4343121845326974, -0.6483933890533198, -1.586536498745096, -2.0283567942296554, -1.9598176181073628, -2.9257614168116404, -1.4908742012715939, 0.6231605173881318, -0.5673073334432863, -0.6029916017558756, -0.1401362947799987, -1.5067927050071739, 0.15433024879439822, -1.587582721443526, -0.5727980096427568, -2.7770176743534454, -0.6117161839891713, 0.8507123481311054, -0.18788379380402453, 0.41482940318561834, -0.45964301775509264, -0.4188215988538312, -1.4809350858225598, -2.2040627015467, -0.4894056452398461, -2.418911919454265, 0.8467301789174599, -0.7446012734342693, -2.0359182937252145, -2.2321174573248372, 0.48789007564864084, -2.1668917953320372, -0.7330932176440532, -0.34283677908238586, -0.018964053980859808, -2.2214484536996464, -1.634963753370044, -0.34635980322546617, 0.49539329076529104, 0.17975579573490785, -2.316921610120978, -0.7669441998505424, 0.30925763445725446, -2.5756861948188288, -1.6159667412497112, 0.062268451604119035, -1.5567186880044672, -1.8212560828365452, -2.7234996146053194, -2.5889481524573674, -0.046222922740853725, -0.11685272224746462, -1.1303870221817092, -1.3353492612379827, 0.08437288375768932, -0.20457863275151925, -2.6611354532887117, -1.4006560228117926, -2.446273869864684, -0.8852342323540228, -2.2695357094485558, -1.7522239294456297, -2.160084475206308, 0.9431818654967882, -2.029304663263264, -1.9211377704636403, 0.05145563535284082, -2.4985937203425697, -0.7077443395600773, 0.6126509504822114, -1.6851951875549793, -2.858914841121299, 0.9078646631444989, -2.130199455828023, -1.426956103260153, -2.8182931499016113, 0.22962595384018153, 0.9386187454249142, -0.9560211550502551, -1.93620410039542, 0.8288585700209392, -2.976272176570748, -1.8283942859070672, -0.04059881229611715, -0.02078669169428567, -2.6455339059737386, 0.945890550043547, -0.4568227609199682, -0.19788345818068534, -0.12985873363739886, 0.2306155063098521, 0.0772891748400016, -0.16279548969617202, -0.0318505408285366, -2.033701591724046, -1.9291896210313273, -1.525143775118011, -0.008357073823396437, -1.9633872471905276, -1.7389041337042137, -0.4050497442306153, -0.9478557410894725, -2.480804645985081, 0.6507854346834954, -2.8780059068754933, -0.9914433348112072, -0.46342657969549883, -0.7914550797719984, 0.9186432850757464, 0.5698339107019339, -0.9224888662747119, -2.0460011334279846, -0.962040365224345, -0.15244628021119896, -2.8716613388956604, -1.3010250277389828, -2.5767991613085166, -1.440820685076281, -0.7097882504632499, 0.41963876639976583, -2.05036282912279, -1.9538810349191453, 0.04030067674122727, -0.010715182377102575, -2.945921945043922, 0.42962361428506135, -2.134521306285257, 0.2633742989276975, -2.5163695787591376, 0.11113764726697588, -1.4832629146922698, -2.2135692312587056, 0.06971104354726741, -2.6316480611662634, -2.7441937482761225, -0.3748567476313957, -0.8336859474128331, -0.4984661011015268, -0.4032962256879555, -1.075282082881666, -2.903044713433388, -0.3905017951927556, -2.6761276573426764, -2.368571562897515, -0.5616551674111765, 0.6198652724557174, 0.8631374057371706, 0.6714556770887685, -1.4785532441056168, -1.0551434177109749, -2.9453498366811885, -1.9029827329802074, -2.697360872318744, 0.3427774437233197, -2.368976628253939, -1.8857146414948236, -1.3023251818500765, -0.8583415083167161, -2.650170925622028, -1.2229648090028586, 0.18502752292757887, -1.8741775860885843, -2.855878440969028, -2.279026223719677, -1.928814409113924, -2.8108809437074416, -0.5973118536187103, 0.5296988590246463, -0.4384539385731978, -2.9130137173865887, -1.3525395670730131, -0.540752282686964, -2.291446959605108 ] }, { "label": "max_leaf_nodes", "values": [ 4.459431618637297, 4.247927513443585, 4.087462841250339, 7.357552004618084, 3.0, 7.459431618637297, 2.807354922057604, 4.857980995127572, 5.857980995127572, 6.108524456778169, 5.554588851677638, 1.0, 4.857980995127572, 6.768184324776926, 3.321928094887362, 7.60733031374961, 7.592457037268081, 4.169925001442312, 3.584962500721156, 5.321928094887363, 1.0, 5.727920454563199, 2.321928094887362, 5.700439718141092, 4.169925001442312, 3.700439718141092, 5.930737337562887, 4.700439718141092, 5.584962500721156, 2.807354922057604, 3.700439718141092, 6.044394119358453, 5.285402218862249, 4.700439718141092, 1.584962500721156, 4.643856189774724, 2.0, 2.321928094887362, 6.149747119504682, 4.807354922057604, 5.321928094887363, 4.700439718141092, 1.0, 1.0, 2.321928094887362, 6.459431618637297, 3.700439718141092, 6.954196310386875, 6.357552004618084, 5.882643049361842, 6.539158811108031, 2.584962500721156, 3.169925001442312, 7.562242424221073, 7.149747119504682, 6.392317422778761, 2.584962500721156, 3.700439718141092, 5.857980995127572, 2.584962500721156, 5.491853096329675, 6.303780748177103, 4.700439718141092, 2.321928094887362, 3.0, 5.554588851677638, 2.0, 2.0, 5.247927513443585, 4.392317422778761, 6.507794640198696, 3.584962500721156, 7.857980995127572, 3.321928094887362, 6.392317422778761, 6.857980995127572, 3.4594316186372973, 6.832890014164741, 5.392317422778761, 6.599912842187128, 7.912889336229962, 4.857980995127572, 3.4594316186372973, 1.0, 5.700439718141092, 7.044394119358453, 3.807354922057604, 4.169925001442312, 4.169925001442312, 5.044394119358453, 6.882643049361842, 5.930737337562887, 2.0, 4.700439718141092, 5.044394119358453, 3.321928094887362, 7.321928094887363, 2.584962500721156, 1.584962500721156, 3.584962500721156, 7.577428828035749, 4.807354922057604, 6.754887502163468, 4.087462841250339, 6.169925001442312, 3.700439718141092, 7.357552004618084, 2.584962500721156, 2.321928094887362, 2.0, 4.169925001442312, 4.857980995127572, 6.459431618637297, 7.285402218862249, 2.584962500721156, 5.285402218862249, 3.0, 5.930737337562887, 4.584962500721156, 3.584962500721156, 7.366322214245816, 3.169925001442312, 7.614709844115208, 1.584962500721156, 3.0, 1.584962500721156, 4.392317422778761, 3.9068905956085187, 1.584962500721156, 7.531381460516312, 6.285402218862249, 5.491853096329675, 6.539158811108031, 2.807354922057604, 3.0, 6.930737337562887, 7.930737337562887, 1.584962500721156, 3.0, 1.0, 4.523561956057013, 7.2384047393250786, 1.584962500721156, 3.321928094887362, 4.954196310386875, 2.321928094887362, 4.169925001442312, 7.7073591320808825, 7.523561956057013, 5.523561956057013, 7.882643049361842, 4.169925001442312, 3.0, 4.0, 3.807354922057604, 5.087462841250339, 2.0, 3.169925001442312, 6.459431618637297, 7.826548487290915, 1.584962500721156, 7.257387842692652, 5.247927513443585, 4.857980995127572, 1.584962500721156, 2.807354922057604, 1.0, 6.442943495848728, 5.727920454563199, 1.0, 5.247927513443585, 7.066089190457772, 1.584962500721156, 5.906890595608519, 2.807354922057604, 4.087462841250339, 7.011227255423254, 5.584962500721156, 6.108524456778169, 2.0, 3.9068905956085187, 7.6653359171851765, 3.0, 5.247927513443585, 1.584962500721156, 6.7414669864011465, 5.807354922057604, 5.882643049361842, 5.459431618637297, 2.584962500721156, 5.426264754702098, 5.584962500721156, 5.584962500721156, 5.78135971352466, 3.0, 1.584962500721156, 2.321928094887362, 2.807354922057604, 3.807354922057604, 5.614709844115208, 1.0, 5.977279923499917, 1.584962500721156, 4.321928094887363, 5.882643049361842, 6.169925001442312, 6.169925001442312, 3.700439718141092, 4.584962500721156, 1.584962500721156, 1.584962500721156, 5.321928094887363, 6.321928094887363, 1.0, 3.321928094887362, 3.321928094887362, 2.807354922057604, 1.0, 6.845490050944375, 5.285402218862249, 1.584962500721156, 2.807354922057604, 3.0, 3.4594316186372973, 3.4594316186372973, 3.4594316186372973, 4.0, 1.584962500721156, 4.584962500721156, 2.0, 3.0, 7.693486957499325, 4.807354922057604, 2.0, 2.584962500721156, 5.930737337562887, 1.584962500721156, 5.321928094887363, 6.22881869049588, 4.459431618637297, 5.882643049361842, 6.714245517666122, 7.960001932068081, 3.0, 4.392317422778761, 5.426264754702098, 1.0, 3.700439718141092, 3.169925001442312, 2.584962500721156, 2.0, 1.584962500721156, 3.700439718141092, 5.044394119358453, 2.0, 1.584962500721156, 7.129283016944966, 4.584962500721156, 3.584962500721156, 6.442943495848728, 1.0, 3.321928094887362, 4.169925001442312, 3.321928094887362, 4.754887502163468, 3.169925001442312, 4.584962500721156, 3.700439718141092, 4.0, 1.0, 7.714245517666122, 7.219168520462161, 5.643856189774724, 6.78135971352466, 2.321928094887362, 5.491853096329675, 6.442943495848728, 7.417852514885898, 5.614709844115208, 3.584962500721156, 7.285402218862249, 3.807354922057604, 5.832890014164741, 2.321928094887362, 4.584962500721156, 7.7481928495894605, 1.584962500721156, 6.442943495848728, 3.4594316186372973, 7.108524456778169, 2.807354922057604, 7.1799090900149345, 6.569855608330948, 4.700439718141092, 6.066089190457772, 6.108524456778169, 3.9068905956085187, 2.584962500721156, 2.0, 6.870364719583405, 1.584962500721156, 2.584962500721156, 6.599912842187128, 3.584962500721156, 3.169925001442312, 7.098032082960526, 2.0, 4.584962500721156, 7.754887502163468, 6.339850002884624, 7.0, 2.0, 4.954196310386875, 4.700439718141092, 2.584962500721156, 1.0, 3.0, 2.0, 6.149747119504682, 3.169925001442312, 5.832890014164741, 3.700439718141092, 7.022367813028454, 5.129283016944966, 7.577428828035749, 5.044394119358453, 4.523561956057013, 5.930737337562887, 4.807354922057604, 2.321928094887362, 6.321928094887363, 5.954196310386875, 5.087462841250339, 1.0, 7.6865005271832185, 2.321928094887362, 1.584962500721156, 2.0, 5.491853096329675, 7.066089190457772, 2.0, 2.807354922057604, 3.700439718141092, 3.9068905956085187, 2.321928094887362, 1.0, 2.584962500721156, 7.169925001442312, 5.247927513443585, 4.321928094887363, 5.044394119358453, 4.459431618637297, 3.584962500721156, 1.584962500721156, 5.491853096329675, 1.0, 5.700439718141092, 1.584962500721156, 5.459431618637297, 2.807354922057604, 2.584962500721156, 2.584962500721156, 4.807354922057604, 2.0, 1.584962500721156, 5.754887502163468, 6.409390936137702, 4.0, 7.3750394313469245, 6.554588851677638, 4.906890595608519, 4.807354922057604, 6.930737337562887, 5.491853096329675, 3.169925001442312, 3.321928094887362, 4.247927513443585, 6.3750394313469245, 3.169925001442312, 7.3750394313469245, 1.584962500721156, 4.954196310386875, 6.266786540694901, 4.392317422778761, 5.491853096329675, 6.22881869049588, 5.129283016944966, 3.4594316186372973, 6.672425341971495, 1.0, 1.0, 7.98299357469431, 2.0, 5.0, 1.0, 3.700439718141092, 4.0, 3.4594316186372973, 5.906890595608519, 2.321928094887362, 7.727920454563199, 7.392317422778761, 3.169925001442312, 4.954196310386875, 4.459431618637297, 1.0, 4.700439718141092, 2.0, 2.0, 4.247927513443585, 6.882643049361842, 6.9886846867721655, 2.321928094887362, 6.0, 5.727920454563199, 1.0, 1.0, 1.584962500721156, 2.0, 1.0, 7.392317422778761, 7.523561956057013, 4.087462841250339, 3.0, 2.0, 6.870364719583405, 1.0, 6.523561956057013, 3.4594316186372973, 6.599912842187128, 4.247927513443585, 7.499845887083206, 4.857980995127572, 4.087462841250339, 2.807354922057604, 4.321928094887363, 1.584962500721156, 1.584962500721156, 3.321928094887362, 5.321928094887363, 7.727920454563199, 2.0, 4.087462841250339, 4.643856189774724, 1.0, 7.066089190457772, 4.0, 7.900866807980749, 4.906890595608519, 4.584962500721156, 1.0, 2.321928094887362, 7.22881869049588, 6.768184324776926, 7.199672344836364, 4.523561956057013, 7.03342300153745, 7.794415866350106, 5.0, 2.321928094887362, 5.129283016944966, 7.076815597050831, 1.584962500721156, 2.321928094887362, 7.2384047393250786, 2.0, 2.321928094887362, 1.584962500721156, 5.954196310386875, 4.459431618637297, 7.20945336562895, 2.321928094887362, 5.523561956057013, 2.0, 1.0, 5.672425341971495, 3.584962500721156, 7.199672344836364, 7.467605550082998, 4.0, 4.954196310386875, 6.9886846867721655, 1.584962500721156, 7.475733430966398, 2.321928094887362, 3.9068905956085187, 5.857980995127572, 5.754887502163468, 3.0, 3.584962500721156, 3.0, 2.321928094887362, 7.592457037268081, 2.807354922057604, 5.169925001442312, 4.087462841250339, 4.087462841250339, 2.807354922057604, 7.189824558880018, 3.0 ] }, { "label": "max_bins", "values": [ 6.426264754702098, 5.906890595608519, 7.2384047393250786, 4.169925001442312, 6.599912842187128, 1.0, 3.4594316186372973, 1.0, 4.0, 2.321928094887362, 3.169925001442312, 6.303780748177103, 3.584962500721156, 2.0, 6.807354922057604, 5.357552004618084, 6.044394119358453, 2.321928094887362, 1.0, 7.459431618637297, 3.0, 7.584962500721156, 3.0, 2.584962500721156, 3.9068905956085187, 1.584962500721156, 4.700439718141092, 4.754887502163468, 4.247927513443585, 5.044394119358453, 4.392317422778761, 5.930737337562887, 4.459431618637297, 2.0, 3.584962500721156, 3.700439718141092, 3.584962500721156, 1.584962500721156, 2.0, 6.700439718141092, 4.169925001442312, 1.584962500721156, 7.483815777264256, 7.912889336229962, 2.0, 5.491853096329675, 3.9068905956085187, 5.491853096329675, 4.643856189774724, 5.700439718141092, 2.584962500721156, 2.321928094887362, 7.066089190457772, 6.857980995127572, 6.189824558880018, 6.459431618637297, 4.392317422778761, 5.169925001442312, 1.0, 1.584962500721156, 2.807354922057604, 5.285402218862249, 4.857980995127572, 4.169925001442312, 2.321928094887362, 3.0, 5.832890014164741, 7.159871336778389, 3.9068905956085187, 6.409390936137702, 4.247927513443585, 5.247927513443585, 3.169925001442312, 3.584962500721156, 5.321928094887363, 2.321928094887362, 5.727920454563199, 5.643856189774724, 7.965784284662087, 7.139551352398794, 6.643856189774724, 2.0, 1.0, 6.857980995127572, 2.321928094887362, 7.592457037268081, 4.643856189774724, 2.584962500721156, 2.321928094887362, 4.321928094887363, 3.169925001442312, 4.459431618637297, 4.523561956057013, 5.129283016944966, 6.0, 4.392317422778761, 7.774787059601174, 3.807354922057604, 1.0, 6.882643049361842, 1.584962500721156, 5.20945336562895, 4.087462841250339, 5.672425341971495, 2.807354922057604, 1.584962500721156, 5.169925001442312, 2.321928094887362, 2.0, 5.044394119358453, 5.0, 5.247927513443585, 1.0, 6.0, 6.882643049361842, 3.700439718141092, 6.169925001442312, 5.129283016944966, 3.321928094887362, 4.247927513443585, 7.303780748177103, 7.442943495848728, 7.924812503605781, 7.022367813028454, 4.247927513443585, 6.491853096329675, 6.845490050944375, 6.108524456778169, 7.700439718141092, 2.0, 6.20945336562895, 7.03342300153745, 1.0, 6.20945336562895, 4.754887502163468, 6.977279923499917, 5.584962500721156, 7.882643049361842, 1.584962500721156, 2.0, 3.584962500721156, 2.0, 5.459431618637297, 4.807354922057604, 2.807354922057604, 6.189824558880018, 7.515699838284043, 6.539158811108031, 3.0, 3.700439718141092, 2.321928094887362, 5.643856189774724, 7.076815597050831, 5.20945336562895, 6.614709844115208, 6.392317422778761, 1.584962500721156, 1.0, 5.044394119358453, 3.321928094887362, 1.584962500721156, 6.807354922057604, 3.584962500721156, 5.727920454563199, 1.0, 6.807354922057604, 1.0, 2.0, 2.0, 2.0, 4.700439718141092, 5.832890014164741, 2.0, 4.087462841250339, 1.584962500721156, 4.0, 2.0, 1.0, 3.807354922057604, 5.857980995127572, 7.409390936137702, 6.149747119504682, 6.169925001442312, 2.807354922057604, 1.0, 7.98299357469431, 2.0, 5.584962500721156, 2.0, 5.584962500721156, 6.870364719583405, 1.0, 6.20945336562895, 7.339850002884624, 1.584962500721156, 7.451211111832329, 5.044394119358453, 6.599912842187128, 1.0, 5.584962500721156, 4.906890595608519, 5.357552004618084, 3.807354922057604, 5.832890014164741, 6.022367813028454, 3.584962500721156, 7.727920454563199, 6.882643049361842, 7.2761244052742375, 7.614709844115208, 5.491853096329675, 7.417852514885898, 1.0, 2.807354922057604, 5.247927513443585, 3.700439718141092, 3.0, 4.700439718141092, 5.832890014164741, 1.584962500721156, 1.0, 1.0, 1.584962500721156, 7.643856189774724, 1.584962500721156, 7.6293566200796095, 6.475733430966398, 7.658211482751795, 4.169925001442312, 5.554588851677638, 4.807354922057604, 3.584962500721156, 1.584962500721156, 7.734709620225838, 3.584962500721156, 4.087462841250339, 3.169925001442312, 6.426264754702098, 1.0, 3.169925001442312, 5.754887502163468, 2.0, 1.584962500721156, 5.459431618637297, 6.832890014164741, 2.584962500721156, 3.584962500721156, 6.727920454563199, 4.459431618637297, 6.491853096329675, 6.569855608330948, 5.247927513443585, 1.0, 3.9068905956085187, 4.643856189774724, 1.0, 1.0, 2.321928094887362, 5.614709844115208, 3.169925001442312, 3.807354922057604, 1.584962500721156, 3.584962500721156, 2.0, 7.022367813028454, 7.169925001442312, 7.409390936137702, 6.426264754702098, 6.9886846867721655, 4.0, 2.584962500721156, 5.0, 3.321928094887362, 3.169925001442312, 6.845490050944375, 3.807354922057604, 4.857980995127572, 3.700439718141092, 1.584962500721156, 4.954196310386875, 7.076815597050831, 3.321928094887362, 2.584962500721156, 1.584962500721156, 1.0, 4.754887502163468, 2.584962500721156, 7.546894459887636, 7.303780748177103, 2.807354922057604, 3.169925001442312, 6.599912842187128, 2.0, 3.584962500721156, 4.700439718141092, 5.672425341971495, 7.554588851677638, 1.584962500721156, 2.584962500721156, 7.087462841250339, 7.870364719583405, 3.321928094887362, 2.321928094887362, 3.807354922057604, 1.0, 4.087462841250339, 5.169925001442312, 3.9068905956085187, 2.0, 1.0, 6.507794640198696, 7.257387842692652, 2.0, 6.409390936137702, 3.169925001442312, 4.754887502163468, 7.451211111832329, 1.0, 6.539158811108031, 7.312882955284356, 6.266786540694901, 3.169925001442312, 7.820178962415188, 4.0, 6.285402218862249, 1.0, 4.392317422778761, 7.912889336229962, 7.882643049361842, 1.584962500721156, 3.169925001442312, 4.169925001442312, 1.584962500721156, 4.321928094887363, 1.0, 7.7414669864011465, 4.169925001442312, 7.826548487290915, 1.584962500721156, 2.321928094887362, 3.0, 6.554588851677638, 1.584962500721156, 7.839203788096944, 6.569855608330948, 2.584962500721156, 6.321928094887363, 6.22881869049588, 3.169925001442312, 1.0, 2.807354922057604, 4.807354922057604, 2.0, 1.0, 4.247927513443585, 6.918863237274595, 4.321928094887363, 4.954196310386875, 2.807354922057604, 7.219168520462161, 2.584962500721156, 2.0, 1.0, 3.9068905956085187, 6.303780748177103, 7.6865005271832185, 2.0, 6.977279923499917, 4.954196310386875, 7.577428828035749, 1.0, 1.584962500721156, 6.906890595608519, 6.442943495848728, 4.247927513443585, 6.754887502163468, 1.0, 3.169925001442312, 5.0, 4.392317422778761, 3.169925001442312, 7.159871336778389, 2.0, 2.321928094887362, 7.6653359171851765, 6.303780748177103, 2.584962500721156, 4.754887502163468, 2.0, 3.4594316186372973, 2.321928094887362, 7.60733031374961, 3.584962500721156, 6.539158811108031, 1.584962500721156, 3.0, 2.0, 5.614709844115208, 6.20945336562895, 6.672425341971495, 6.672425341971495, 6.918863237274595, 4.169925001442312, 3.584962500721156, 6.3750394313469245, 7.022367813028454, 5.832890014164741, 5.0, 3.0, 3.9068905956085187, 1.0, 4.643856189774724, 5.523561956057013, 7.169925001442312, 5.700439718141092, 5.832890014164741, 7.434628227636725, 7.888743248898259, 4.906890595608519, 1.584962500721156, 7.1799090900149345, 6.357552004618084, 3.169925001442312, 1.0, 7.960001932068081, 3.9068905956085187, 2.0, 4.459431618637297, 7.754887502163468, 2.807354922057604, 5.426264754702098, 6.643856189774724, 2.584962500721156, 2.584962500721156, 2.321928094887362, 6.357552004618084, 6.149747119504682, 1.584962500721156, 7.7481928495894605, 3.584962500721156, 7.523561956057013, 4.643856189774724, 1.584962500721156, 2.807354922057604, 3.0, 6.3750394313469245, 3.9068905956085187, 2.321928094887362, 7.813781191217037, 2.321928094887362, 3.9068905956085187, 1.584962500721156, 2.0, 7.066089190457772, 4.523561956057013, 4.247927513443585, 3.321928094887362, 6.727920454563199, 2.0, 4.643856189774724, 6.832890014164741, 2.321928094887362, 2.807354922057604, 1.0, 6.459431618637297, 7.451211111832329, 5.491853096329675, 6.523561956057013, 4.459431618637297, 1.0, 1.584962500721156, 2.321928094887362, 4.0, 4.643856189774724, 7.614709844115208, 6.714245517666122, 4.584962500721156, 3.4594316186372973, 3.807354922057604, 5.247927513443585, 4.754887502163468, 1.584962500721156, 5.882643049361842, 7.658211482751795, 4.754887502163468, 6.459431618637297, 3.807354922057604, 3.169925001442312, 3.807354922057604, 6.807354922057604, 3.700439718141092, 3.0, 5.087462841250339, 1.0, 3.4594316186372973, 6.523561956057013, 2.807354922057604, 4.0, 7.826548487290915, 4.247927513443585 ] }, { "label": "min_samples_leaf", "values": [ 0.7781512503836436, 0.0, 1.0, 1.568201724066995, 0.47712125471966244, 0.0, 0.0, 1.8512583487190752, 1.1139433523068367, 0.0, 0.0, 0.0, 1.9030899869919435, 1.8920946026904804, 0.0, 1.99563519459755, 1.505149978319906, 1.041392685158225, 1.2304489213782739, 0.0, 1.2304489213782739, 0.8450980400142568, 0.3010299956639812, 0.0, 1.6434526764861874, 1.7160033436347992, 1.806179973983887, 0.3010299956639812, 1.6532125137753437, 0.0, 1.414973347970818, 0.3010299956639812, 0.0, 1.2787536009528289, 1.414973347970818, 1.7708520116421442, 0.0, 0.0, 1.2041199826559248, 1.7558748556724915, 0.6020599913279624, 1.0, 0.9030899869919435, 1.8512583487190752, 0.3010299956639812, 0.0, 1.1139433523068367, 0.6989700043360189, 0.47712125471966244, 1.7558748556724915, 1.3010299956639813, 1.7993405494535817, 1.662757831681574, 0.6020599913279624, 1.5797835966168101, 1.255272505103306, 0.7781512503836436, 1.505149978319906, 1.8864907251724818, 0.47712125471966244, 0.0, 1.255272505103306, 1.414973347970818, 1.9138138523837167, 0.3010299956639812, 0.3010299956639812, 0.0, 1.1139433523068367, 0.0, 1.041392685158225, 1.5797835966168101, 1.041392685158225, 1.146128035678238, 1.255272505103306, 0.0, 0.0, 0.0, 1.4771212547196624, 0.6020599913279624, 0.0, 1.8976270912904414, 1.2304489213782739, 1.5797835966168101, 0.9030899869919435, 0.47712125471966244, 1.1760912590556813, 0.8450980400142568, 1.3010299956639813, 0.6020599913279624, 1.662757831681574, 0.0, 1.7781512503836436, 1.7853298350107671, 0.0, 1.1139433523068367, 0.0, 0.0, 1.792391689498254, 1.462397997898956, 1.2787536009528289, 1.0791812460476249, 1.7323937598229686, 1.7558748556724915, 1.7781512503836436, 1.7481880270062005, 0.0, 0.0, 0.0, 0.0, 0.9030899869919435, 1.041392685158225, 0.0, 0.0, 0.6020599913279624, 1.8195439355418688, 1.8195439355418688, 0.3010299956639812, 1.2787536009528289, 1.1760912590556813, 0.0, 0.9542425094393249, 1.5563025007672873, 1.8920946026904804, 0.0, 0.0, 0.0, 0.6020599913279624, 1.5797835966168101, 1.9822712330395684, 0.0, 1.9294189257142929, 0.0, 1.9822712330395684, 0.0, 0.6020599913279624, 1.9030899869919435, 1.5185139398778875, 1.6532125137753437, 0.6989700043360189, 1.2787536009528289, 0.3010299956639812, 1.4313637641589874, 1.2041199826559248, 1.662757831681574, 1.4771212547196624, 1.5797835966168101, 1.5185139398778875, 1.3979400086720377, 1.792391689498254, 0.0, 0.47712125471966244, 1.7160033436347992, 0.0, 0.0, 1.662757831681574, 0.0, 1.462397997898956, 0.47712125471966244, 0.6020599913279624, 0.0, 1.146128035678238, 1.1139433523068367, 1.146128035678238, 0.8450980400142568, 0.8450980400142568, 0.3010299956639812, 0.3010299956639812, 1.3010299956639813, 0.0, 0.3010299956639812, 1.505149978319906, 1.3424226808222062, 0.6989700043360189, 0.0, 1.8864907251724818, 1.8512583487190752, 1.6720978579357175, 1.3222192947339193, 1.863322860120456, 0.6020599913279624, 0.7781512503836436, 1.3010299956639813, 0.0, 0.6020599913279624, 0.6989700043360189, 1.0791812460476249, 0.0, 0.6989700043360189, 1.4771212547196624, 0.6989700043360189, 1.9637878273455553, 1.863322860120456, 0.6020599913279624, 0.0, 0.6020599913279624, 1.0791812460476249, 1.3010299956639813, 1.414973347970818, 1.6989700043360187, 0.0, 0.3010299956639812, 0.0, 1.0, 1.505149978319906, 0.3010299956639812, 0.7781512503836436, 0.0, 0.6020599913279624, 0.3010299956639812, 0.0, 0.0, 1.0791812460476249, 1.9493900066449128, 0.0, 1.146128035678238, 0.3010299956639812, 0.7781512503836436, 1.5185139398778875, 1.146128035678238, 1.6020599913279625, 0.6989700043360189, 0.0, 0.6989700043360189, 0.9542425094393249, 1.3424226808222062, 0.47712125471966244, 0.3010299956639812, 1.2041199826559248, 1.8920946026904804, 0.3010299956639812, 1.7634279935629373, 1.4771212547196624, 0.6989700043360189, 0.6020599913279624, 0.8450980400142568, 1.5185139398778875, 1.4771212547196624, 1.8692317197309762, 0.9030899869919435, 0.3010299956639812, 1.5440680443502757, 1.1139433523068367, 0.0, 0.6020599913279624, 1.4471580313422192, 1.255272505103306, 1.6020599913279625, 1.568201724066995, 0.8450980400142568, 1.7708520116421442, 0.47712125471966244, 1.2304489213782739, 0.8450980400142568, 0.47712125471966244, 1.0, 1.3617278360175928, 0.8450980400142568, 1.146128035678238, 0.3010299956639812, 1.1760912590556813, 1.0, 0.7781512503836436, 0.6020599913279624, 1.255272505103306, 0.3010299956639812, 0.0, 1.6812412373755872, 0.0, 1.9590413923210936, 0.3010299956639812, 1.2787536009528289, 1.5797835966168101, 0.0, 0.0, 0.0, 1.4771212547196624, 1.0791812460476249, 0.6989700043360189, 1.99563519459755, 1.6334684555795866, 1.9030899869919435, 1.5797835966168101, 0.0, 0.3010299956639812, 1.4471580313422192, 1.5563025007672873, 1.5185139398778875, 1.0, 0.7781512503836436, 1.7993405494535817, 0.0, 0.47712125471966244, 1.0, 0.0, 1.1760912590556813, 1.8864907251724818, 1.7075701760979363, 1.255272505103306, 1.6434526764861874, 1.3010299956639813, 1.568201724066995, 0.6989700043360189, 1.414973347970818, 0.6989700043360189, 0.0, 1.8750612633917, 0.0, 1.8920946026904804, 0.3010299956639812, 1.255272505103306, 0.47712125471966244, 1.1760912590556813, 0.0, 1.1760912590556813, 0.6989700043360189, 0.47712125471966244, 1.8129133566428555, 0.0, 1.6020599913279625, 0.6989700043360189, 0.0, 1.1760912590556813, 1.3979400086720377, 0.0, 0.8450980400142568, 1.662757831681574, 0.3010299956639812, 0.6989700043360189, 1.6901960800285136, 0.3010299956639812, 0.8450980400142568, 1.380211241711606, 0.3010299956639812, 1.1760912590556813, 0.6989700043360189, 1.6434526764861874, 0.6020599913279624, 1.8750612633917, 1.6720978579357175, 1.146128035678238, 0.0, 0.3010299956639812, 1.591064607026499, 1.1760912590556813, 0.0, 1.5314789170422551, 1.9084850188786497, 0.8450980400142568, 0.8450980400142568, 1.0, 1.8195439355418688, 1.8388490907372552, 0.0, 0.0, 1.8260748027008264, 1.9030899869919435, 1.0791812460476249, 1.8325089127062364, 1.0, 1.1760912590556813, 0.6989700043360189, 0.6020599913279624, 1.792391689498254, 1.5314789170422551, 0.0, 1.380211241711606, 0.3010299956639812, 1.0, 0.0, 1.5314789170422551, 0.47712125471966244, 1.414973347970818, 1.041392685158225, 0.0, 1.6127838567197355, 0.0, 1.6532125137753437, 0.3010299956639812, 0.0, 0.3010299956639812, 0.9542425094393249, 0.0, 0.0, 1.6901960800285136, 1.146128035678238, 0.0, 1.041392685158225, 1.0791812460476249, 1.6532125137753437, 1.0791812460476249, 0.0, 0.0, 0.0, 0.3010299956639812, 0.8450980400142568, 1.505149978319906, 0.3010299956639812, 0.3010299956639812, 1.845098040014257, 1.9542425094393248, 0.47712125471966244, 0.0, 1.0, 1.6334684555795866, 1.7323937598229686, 1.5440680443502757, 1.9138138523837167, 1.7323937598229686, 1.9242792860618816, 1.4471580313422192, 1.9777236052888478, 0.3010299956639812, 1.0791812460476249, 0.0, 1.2041199826559248, 1.255272505103306, 1.5563025007672873, 0.0, 1.2787536009528289, 0.6020599913279624, 0.47712125471966244, 1.7781512503836436, 1.0791812460476249, 0.0, 1.9493900066449128, 1.6232492903979006, 1.041392685158225, 1.3010299956639813, 0.6020599913279624, 0.0, 0.0, 0.7781512503836436, 0.47712125471966244, 0.0, 1.8129133566428555, 0.3010299956639812, 1.041392685158225, 1.8864907251724818, 1.7242758696007892, 0.47712125471966244, 0.0, 0.6020599913279624, 0.3010299956639812, 1.8195439355418688, 0.6989700043360189, 1.919078092376074, 1.0791812460476249, 1.2787536009528289, 0.3010299956639812, 1.7403626894942439, 0.3010299956639812, 0.3010299956639812, 1.4913616938342726, 0.47712125471966244, 0.0, 0.0, 0.0, 1.0, 1.7634279935629373, 0.47712125471966244, 0.0, 0.0, 1.7558748556724915, 1.8129133566428555, 0.47712125471966244, 1.414973347970818, 1.3617278360175928, 0.0, 0.0, 0.0, 1.6901960800285136, 0.9030899869919435, 0.47712125471966244, 0.0, 1.8750612633917, 1.414973347970818, 1.4771212547196624, 1.3617278360175928, 0.0, 0.0, 1.8573324964312685, 1.9030899869919435, 0.47712125471966244, 0.3010299956639812, 1.5563025007672873, 0.8450980400142568, 1.99563519459755, 0.7781512503836436, 0.0, 1.041392685158225, 0.6020599913279624, 1.3222192947339193, 1.5440680443502757, 0.3010299956639812, 0.6020599913279624, 0.6020599913279624, 0.0, 0.8450980400142568, 0.6989700043360189, 1.2787536009528289 ] }, { "label": "l2_regularization", "values": [ 0.39217750897745574, -1.8111032549372006, 0.039451183704711595, -2.441159857208718, -4.091189449701977, -5.944935433261408, -3.1913706789785827, -3.1028078367738465, -3.437105005701581, -0.9976802520608308, -5.638845721919441, 0.22162518472175766, 2.249301997455248, -3.287852637247264, -5.742096693683774, -4.4872517641557685, 1.4100784967076632, -3.0824690845587575, 2.855752868676887, -1.3926011768485635, -2.6345394609860353, 0.6985283253534601, -4.395646930980709, -5.997159259432995, 2.7901925205505025, 0.6539887021910622, -1.3482426623660166, -0.2071662174710686, -4.267886186803508, -2.886131587998138, -0.18417537173724785, -3.6921340583589846, -2.4444706776625496, -3.340761217822522, 0.4313782574413746, -1.3731436769328105, 2.2866430479659217, 2.0259517563181135, -3.099210072312905, -5.758702182717651, -2.035775135573452, 1.9245940739254421, -4.209631843175476, -5.545238296516993, -1.6289830348840857, -1.2269885924984798, -4.938800453898116, 0.4238457749337017, -2.723572285939581, -5.302850242236925, -2.0332338178408347, 0.5473564616360486, -4.040210504180727, -2.9230755261941646, -4.77945064929443, -5.722949854862035, 1.539355833631865, -4.6424164025168855, 0.10720165967972409, -3.820127501423609, -0.38410752905982726, -3.679012176282214, -5.561317254545729, -4.784385992566478, -2.6951141122398274, -4.625059463352652, 0.8205105752483941, -3.951402691210199, 0.9244118522442514, -5.428570691946971, -5.1419772292545405, -2.172501674006123, -5.1907306575847585, -2.5415889734947155, -4.324219668355871, -0.05296197170734998, 1.2754785285539822, 2.144063753462438, -1.1074918605801205, 0.7002464157673021, -0.8042162252141752, -5.354520080402092, -5.901594858523433, -3.178965970744434, 0.7576404996262753, -3.228602723762521, 0.6199060801592511, -3.81152069368004, 2.572096486326961, -3.761473254303262, 0.86669962586009, 1.1408249409840812, -4.499031464115445, -0.9037933756165684, 1.5290619876551133, -1.1899237399193212, -1.7237868696000629, -5.053329885892729, -1.1255884957175346, -4.520115009862745, 1.859178016312534, -5.209075977632116, 0.9137415441107178, -0.821161490687236, 1.3866402285883614, -5.055561133814948, -3.33819826948749, -3.522905889974974, 1.5674826737356107, 1.48296924381876, -1.8617313813907965, 1.6215964220399024, -2.625600369315443, -2.437262006594162, -4.968787881889586, -5.805921971495212, -0.5725302145915967, -2.1261698600938677, 1.3288119445730686, -4.995218082266305, 2.4577188434676804, 1.7302142947399954, -4.232382521881763, 0.35494823337983866, -0.6002525555115462, 2.942300607152658, -2.335087182596236, -1.95121238518984, -2.1694301568579375, -0.35119308482266925, 2.5321508243177298, -2.5117049404505587, 2.196587949632119, -1.8940728983759199, -2.541261971737956, -2.761122084182819, -0.6966480631220489, -2.0773723573845793, -3.71173365365077, -0.06226241297477755, -3.5221742188100325, -1.3822599162605318, -0.26677614312159786, -4.283509921144625, -2.1833995030461963, 2.4757101679636433, -0.28902659422710564, -2.097039490428089, -4.358605424916951, -2.2111708864883197, 2.3752353893206823, -5.243412932082516, -4.784702757741252, 0.018503200937925558, -1.599800943329297, -4.686685594907361, -0.5395315743197073, -1.6016185600420005, -4.732930955135696, -1.924126931065756, -0.4123290730568685, -1.2487137686322098, 2.7643354346891247, -4.717078371661138, 1.2412076088385477, -4.977970488482374, -3.72244632967046, -0.9161091665889347, 2.3519393808934836, -4.092108174055652, -0.44170876906476564, -4.1732879403193675, 2.4415784358286303, -4.1889738351843295, -3.0708907231186955, 1.8487726156065314, -2.7994347097002295, -3.9006077703906294, -1.544775512656199, -0.2417210159401152, 0.7159740589069304, -2.441341393071509, -5.677253739295252, -5.905306529706932, -0.7933383621247065, -3.8273892909364027, 2.3143475751174867, -4.378511937177755, -0.12040367841625928, -3.2169227889338816, 2.2163608518685187, -5.210490212878951, 2.770134299143809, -0.01211947043030274, 1.71982040782419, 0.5865857903421703, 1.5075436582423307, -5.849448753767749, -1.8462343131460734, -1.1954641616605355, -3.3526284029914892, 0.46660920688401913, -4.170321789457858, -3.0183101845010314, 1.9791646900293847, -5.5654138272780695, -1.4412591085773234, 2.680035102696677, -1.9290391594644727, -5.77894777966596, 1.1338979737476682, -4.968960858451975, 1.3654414359306035, 0.8566099502259962, -0.357491897850485, 1.5478004705202093, -5.467374098086973, 2.2561940463106467, 1.3477107016229026, 1.40028147597818, -1.3185372605728272, -3.904989481790129, -0.7972524198837658, -0.6957754582797393, -3.917486613157455, 1.198314272031414, -3.6758037363028637, 1.725268467601882, 0.9605822494798502, 1.7360278795538822, -5.004269169500854, -2.402127684966816, -4.015117741880515, -0.711091839134752, 1.45199134640619, -4.692632661974941, 2.8826476909700793, -4.874232130366026, 2.4826868145197696, -1.2379497118581264, -1.3295124329281622, -0.8565585254905668, -3.772244758699039, -3.466599963615018, 1.6216352875025635, 1.4090786051384379, -0.9027457800563907, -4.542855699567055, 0.30062222861639315, 0.3294640913786885, 2.78956427316138, -1.356291691899703, 2.39275059439978, -0.06904006609016536, -4.753661521116399, 0.720055092930634, -3.3155539654472923, 2.7845155968675517, -2.450684053077208, -4.032976741618281, -2.87416390287014, -5.003647533679086, -5.287734005335574, -3.804493617059199, -5.952951574092016, -3.9153363658782627, -5.687037866087168, -1.3389892730707285, -4.627592829693165, 2.9530433108934573, 1.5638826236563503, 2.7258827815906805, -2.3988218026868418, 0.8704116525135754, -2.5135099839165704, -4.452299077808705, -5.043750877657704, -1.5584600048718933, -3.674579909195432, 2.322971771956633, 1.8896625996233107, -1.269034962451977, -4.041028027136354, 0.09377725529393612, -1.3052340179930293, -3.3538624892545834, 1.0109719242913353, -5.016401529219368, 1.501998036491053, 2.2922789196508857, 0.8079115844064877, -2.531718229623495, -2.190318343266044, -2.298477649313243, -5.7694370996813, -1.4514784681742448, 1.1236425742568603, -3.773910532308415, 0.7325314828797065, -3.7522398299075355, 0.3635890546068817, -5.8348373685238295, 2.9012272879729473, -4.548930171702741, -3.7831620791177523, 1.737780440030896, -0.9032700388270002, 2.2162534391920494, 1.728044304749559, -1.8271523053550638, -4.479043495085786, -0.7924812820668339, -3.1150584404359094, 2.7776042267276955, -5.531823247885648, 1.0017155366545203, 1.5554401658650894, -5.496576774767555, 1.5138007422964253, -3.282683316280065, 0.4035148756165198, -0.2997609390357239, -0.4841784752249646, -5.649957545373754, 2.9234489093899967, -0.6062584334390863, -5.98904318556352, 0.6752106558740787, 0.30882720669369523, -0.2089403673008539, 2.6052490895159837, -1.6594748928197598, 1.6356841424390187, -4.230311554985187, 2.416414656914682, 1.384561412789248, -0.9615918980969406, 1.424434615015022, 2.4691930828743756, 2.719447712938665, -3.0617042888999175, 0.12887306759157638, 0.6006353157209491, -3.3941447203603525, -5.4686973908633885, 0.9421711823566754, 0.44067672637129784, -1.3882687418162554, 1.5988248746240525, -1.5481066426207357, -3.1397940728010716, 2.6466179411261166, -5.627250181199472, 0.644593244250717, 0.5633339517795586, -2.620701051931253, -3.8739547283259523, -4.58734379543001, 1.8327062297990582, -4.478355915661508, -3.485041957683678, 0.8698136216376813, -4.2852705100865025, -4.579205228461922, -5.144285116054608, -4.933318278685454, 0.3337283014172543, -5.696341576983337, -3.3338794141809207, 1.0748361769947914, 2.4751519678902114, 2.9148844516586614, 2.837048780407483, 0.1938746038672012, -4.445357489622957, 0.9038275703682148, -4.263843905060754, -4.860166493159663, -1.4167472876566554, -3.6081113041275925, 2.4706930694111446, 2.475645334109952, -2.8713900676894117, -5.077503058429246, -3.901284534072631, -3.5887002654720987, -0.8894109008602571, 0.03610568458348313, -4.069615848997274, 0.9792615282559699, -2.185057751388333, -5.524050808192397, -2.673932199398266, -3.9115386588369145, -5.181260782465351, -3.137694008002772, -1.0219068314763469, 0.5086782735099109, -2.6261004230456098, 2.0098587076881835, -5.123247323931873, -4.697173811688806, -5.724819523424092, -0.04930369830747384, -4.204316567162551, 1.9677378901839893, -3.458095273914747, 0.7841681059716259, -1.2391350837792003, 1.3646267927172804, 0.9036819963174529, -1.3255101906090605, -2.369719936630284, -5.896520052435687, -5.726579143388821, -5.744601836718494, -0.31771714425301456, -1.5214577951982455, -2.6798790795204215, -4.87231913935283, -2.2269483772556615, 1.074807745833628, -5.8777832725590855, -5.540753851794682, -2.9706006965487215, -1.13579147372646, 2.548261880494484, 1.3987657382930334, -0.7291421519664939, 1.2791865982296222, 0.9171603291798563, -5.816633370326653, 2.1706770987974084, 2.864135900727531, -0.9828381500624092, 0.7995303862013866, -0.7476153088268062, -4.153897940928988, -1.1008175949450116, -3.8651763337262586, 0.42424751465561894, 2.3761806664395615, -5.345472429268739, -0.12935005806486433, 0.43616594078101945, -1.0218535230085422, -2.151451964065952, 1.9337463756161273, 1.1596799223867653, -0.47796209147450164, 2.7844045202990917, 2.9597669311289447, -4.535239962437448, -1.853310495755378, 0.1800015843776288, -0.3798661745098703, -0.7104037431406105, 0.35184290597914974, 1.8155035768770194, 2.8982689249453446, -2.6051574579674774, -0.605997512215646, -1.708279169657238, -4.21068373088914, 0.07343154707991127, -3.249692170725387, -0.9487854511545109, -2.6612920383194134, -4.416277644501867, 1.3312613823897725, -2.2320561502418235, -3.292097788794713, 1.2149718662910862, 2.6737497294098995, -0.5807179792477413, -4.735181511601223, -2.6592665123255754, -1.0028384678498588, 0.533322818673267, -4.03394577727438, -5.170079539460583, -5.6166884514398205, 2.7261033900877414, -2.28978097507582, 2.7320729056804907, -3.9715685317349703, -0.9009826913797376, 2.519933941012472, -0.43519884796991143, 1.1607658092641588, -3.0161038279419916, -1.903510390441164, -2.5889636935634557, -2.7302208241534616, 1.5898861138149338, -4.126018472165817, 0.7046605174140823, 0.39093952083677574, -4.169056298737916, -0.35142738979149407 ] }, { "label": "mean_test_score", "values": [ 0.8575523735207098, 0.7589473389664464, 0.7713408881417616, 0.7589473389664464, 0.7589473389664464, 0.6873151263351892, 0.5517947318138392, 0.6892789212155939, 0.84865282916327, 0.7589473389664464, 0.8433022548212941, 0.8523383104930804, 0.7234035082267647, 0.8384702990271435, 0.6553578753387679, 0.7589473389664464, 0.8526111691800241, 0.7823427544322863, 0.8223909890758533, 0.8206436047571575, 0.8045098847746591, 0.8707925439739279, 0.7730341530532605, 0.7178607978799053, 0.7589473389664464, 0.6736927368700263, 0.8535121334220557, 0.8558325165121945, 0.8452404039956928, 0.8547406234727102, 0.7306645106836179, 0.6526683326874401, 0.8535940521337, 0.7589473389664464, 0.8447764339398024, 0.8466600464007316, 0.7589473389664464, 0.7589473389664464, 0.7886761595646542, 0.8506456454593483, 0.8532117958622667, 0.7778110393232545, 0.8491442520103636, 0.8534575370934044, 0.4522942846813461, 0.8552864787067053, 0.8504818452955479, 0.8540581041604653, 0.851109764553193, 0.6553578753387679, 0.7945180971385419, 0.8415004679232876, 0.7589473389664464, 0.8183777881880789, 0.8138188960080595, 0.6553710353901427, 0.7638611762127526, 0.5517947318138392, 0.7200743207567284, 0.8214900919009012, 0.7128367358884623, 0.7589473389664464, 0.7763367856857689, 0.7589473389664464, 0.8417188495119436, 0.8431111360004495, 0.8460049128126104, 0.8662336406160618, 0.8458955636654558, 0.860391513018782, 0.8507822089369789, 0.8543856895842701, 0.8147197782792158, 0.8473424539345107, 0.7589473389664464, 0.8363953814479268, 0.8575251703680802, 0.7589473389664464, 0.8646229054718205, 0.8529115179176596, 0.8605826281136777, 0.7589473389664464, 0.8082499288995809, 0.7589473389664464, 0.7561082180981185, 0.7962933999920487, 0.8297343275506233, 0.8386341140947392, 0.7915152468994424, 0.7589473389664464, 0.844066513999638, 0.7589473389664464, 0.8321093740364929, 0.7589473389664464, 0.8523109508505987, 0.7589473389664464, 0.7400013584809544, 0.7589473389664464, 0.7589473389664464, 0.8499086863083042, 0.7019462195796302, 0.8448583302957535, 0.8212988911091819, 0.8545768084051147, 0.8439846921626641, 0.8272775933619387, 0.7589473389664464, 0.7589473389664464, 0.7589473389664464, 0.7580738871307999, 0.8468511763994228, 0.8525839362198034, 0.8200705277407885, 0.8570610289185423, 0.7589473389664464, 0.8469875908390989, 0.8212441643723205, 0.8573886553277846, 0.8457864157195397, 0.8116073627879278, 0.8631214039703188, 0.8665066222593177, 0.8667249069733034, 0.6214169813323628, 0.7764728089008149, 0.8532937220258088, 0.5517815717624643, 0.8586170745854108, 0.8162481699536428, 0.7589473389664464, 0.8209988777069437, 0.7860268385312742, 0.8227459155123956, 0.7680373650074755, 0.7563265363456437, 0.7589473389664464, 0.7589473389664464, 0.7589473389664464, 0.8281238904822908, 0.7589473389664464, 0.8166581659143416, 0.7658269383941556, 0.7698400498591573, 0.8539761779969233, 0.8437663068480592, 0.7589473389664464, 0.8652780986751234, 0.827632206818777, 0.8417734570184414, 0.6093704893895968, 0.7589473389664464, 0.8394529882314787, 0.7589473389664464, 0.8553138756086758, 0.8624390895852612, 0.8589993047752019, 0.7368061784876305, 0.8124266654087862, 0.7589473389664464, 0.8417734048551573, 0.8276051042667666, 0.8474789876045505, 0.8444488373381509, 0.8585352303927436, 0.8175043885157164, 0.8071308587891091, 0.8162211680022518, 0.6418315073016861, 0.7589473389664464, 0.7703310926594671, 0.8474516205101711, 0.7589473389664464, 0.7589473389664464, 0.4766407374159524, 0.7589473389664464, 0.7589473389664464, 0.8304714618212639, 0.8223909778980067, 0.8489804294908702, 0.7589473389664464, 0.7589473389664464, 0.8393437247811479, 0.8573613813821265, 0.6553710353901429, 0.8230461450196677, 0.8374600638828836, 0.7589473389664464, 0.8386612054689029, 0.8392073587788069, 0.8499086676785599, 0.7589473389664464, 0.7411205515477386, 0.8569519145061658, 0.7054399486772901, 0.7759821871327263, 0.8609921508788713, 0.8516558023586821, 0.7339404506184908, 0.715350819882006, 0.5968670172710024, 0.3446289646098572, 0.7589473389664464, 0.8477247102059439, 0.8550955350054572, 0.8524201174262591, 0.8330648861698403, 0.7589473389664464, 0.8614017518889907, 0.87139310731504, 0.7589473389664464, 0.7098332485121075, 0.6553710353901427, 0.8221452962820507, 0.8418826794833347, 0.7589473389664464, 0.8483525922041004, 0.8455135607585451, 0.8493080857077036, 0.8569245027004001, 0.7736616550046331, 0.8194698712509549, 0.8234829417305193, 0.7905872136389401, 0.8255849991212351, 0.7707132818638209, 0.7615709509526896, 0.6411750211941286, 0.6950645783319456, 0.8547405824872728, 0.8541400862132402, 0.8552865681294779, 0.8442031259146041, 0.4482052681861609, 0.7969205703339728, 0.8461686533612287, 0.8331468086074336, 0.5266321733224968, 0.6531323213730748, 0.6805691692412041, 0.8450766224616368, 0.7092875646717601, 0.7589473389664464, 0.8185418379904528, 0.5517947318138392, 0.7698395729377032, 0.8382245056327218, 0.6834095457542981, 0.8505365198691253, 0.7589473389664464, 0.857033732617191, 0.8516558582479149, 0.7589473389664464, 0.8231007301704721, 0.7815233623886552, 0.7589473389664464, 0.8234555820880374, 0.8196337198520902, 0.7589473389664464, 0.6069407869598944, 0.7787120184690813, 0.7589473389664464, 0.8242473350430222, 0.8354125730132281, 0.7019721819912894, 0.5517815717624643, 0.8684720975426584, 0.8192786220219002, 0.8037735553089723, 0.6553710353901429, 0.724441993515284, 0.8420737386889974, 0.8268680929524385, 0.6981228781419857, 0.7846631673300161, 0.8479704067256953, 0.5626402796458754, 0.5486825287016359, 0.8458956419103819, 0.7191183801392619, 0.8532937890928881, 0.865305353991037, 0.8451312076124413, 0.8396987220107187, 0.8213536290238898, 0.8223636592409622, 0.7705767183861901, 0.8023258676868613, 0.868499412473754, 0.7589473389664464, 0.7589473389664464, 0.7863555603694815, 0.6972219772410846, 0.7589473389664464, 0.6941964434253229, 0.8382517795783798, 0.8547405564056308, 0.8683902235424004, 0.7589473389664464, 0.7589473389664464, 0.8638311785984776, 0.8603369688534148, 0.7589473389664464, 0.8422102611811907, 0.8036360268105867, 0.8166581212029553, 0.8425650758392672, 0.8530207739160925, 0.8245476167135782, 0.8354672773943962, 0.8216538808868549, 0.8597090868552586, 0.6553578753387679, 0.8377331833862474, 0.8488985145051748, 0.8274413376365063, 0.7589473389664464, 0.8583714078732504, 0.8172313845167674, 0.7589473389664464, 0.8685268093757245, 0.7589473389664464, 0.8355492855288134, 0.8224452947804932, 0.5517815717624643, 0.7589473389664464, 0.8193061307023365, 0.8560509316343902, 0.8696732689362688, 0.8698370206627338, 0.816303064358203, 0.7589473389664464, 0.8429473172069051, 0.7589473389664464, 0.7940268382331982, 0.7008532013035902, 0.8659332620708355, 0.8519287653721938, 0.8671617931069276, 0.8172587068997605, 0.8425105428517465, 0.8303897331330112, 0.8635855789533965, 0.7589473389664464, 0.8713385519718262, 0.4482052681861608, 0.8408453082535244, 0.7072956873147946, 0.8506729380347504, 0.8474789913304994, 0.823592175373259, 0.7899319235609669, 0.8559690278265413, 0.7589473389664464, 0.4502526770849413, 0.8540307631477277, 0.7808958865189247, 0.8055471255962591, 0.24105266103355366, 0.7589473389664464, 0.8676805197072701, 0.8411456458133133, 0.8387432657666041, 0.7841990445102224, 0.8323552307720453, 0.8553957794165246, 0.7437412237603312, 0.8118259232221291, 0.8065027383302258, 0.8495537524198641, 0.8335833370499675, 0.7589473389664464, 0.8083863247095129, 0.8673801076285039, 0.8595179792122607, 0.8401083341986849, 0.7589473389664464, 0.7589473389664464, 0.8279054231968113, 0.8536759224080093, 0.8530480553136484, 0.8275232600740994, 0.8653326763740303, 0.8291884648647307, 0.7589473389664464, 0.8691545982251592, 0.7589473389664464, 0.8439301144637572, 0.7816879487276648, 0.8378696611670543, 0.7589473389664464, 0.7589473389664464, 0.8555595982100691, 0.8057930270431977, 0.8508912786379689, 0.7589473389664464, 0.8463051982091152, 0.7346248962462967, 0.8432476324110008, 0.7589473389664464, 0.6967565764208319, 0.8439299505220074, 0.866397332727345, 0.7589473389664464, 0.8017795243535657, 0.6553710353901427, 0.8466327463734317, 0.8468511056063942, 0.5424946367760617, 0.7589473389664464, 0.8369142757159682, 0.8208348949716493, 0.826649484080902, 0.7589473389664464, 0.4482052681861608, 0.853757777778523, 0.8534302407920533, 0.8600367020866543, 0.8374597695329236, 0.8512734715682717, 0.8237012115407094, 0.8402719555169398, 0.8409271002829076, 0.8273321114456641, 0.8207803433543844, 0.8652235880432959, 0.8380880576595058, 0.7589473389664464, 0.8526111878097684, 0.8691000093484058, 0.7589473389664464, 0.7760635991153257, 0.7589473389664464, 0.8436025253140034, 0.8427835729323376, 0.8426197504128445, 0.7546619463365745, 0.7589473389664464, 0.8256395432866022, 0.8164941049341212, 0.8475608802345529, 0.8573338838795369, 0.7589473389664464, 0.8281511308944092, 0.7589473389664464, 0.8020802345082405, 0.858753518832678, 0.6276888077079151, 0.8128088285314978, 0.8534029184090601, 0.8363954671447505, 0.8442849887370155, 0.7589473389664464, 0.594028235464021, 0.8420191162787042, 0.8104880430388823, 0.7589473389664464, 0.8171493204931174, 0.8636674417758081, 0.7589473389664464, 0.853129914410111, 0.7589473389664464, 0.7589473389664464, 0.8446944407091808, 0.8232918489913168, 0.8564877097154973, 0.8675985637361372, 0.8516011650445934, 0.7589473389664464, 0.8457046311420543, 0.7589473389664464, 0.7589473389664464, 0.8421829462500952, 0.7578820268461322, 0.735387221657182, 0.7589473389664464, 0.8650324468667583, 0.8557779648949294, 0.7589473389664464, 0.8396714070796232, 0.7589473389664464, 0.6758788330372879, 0.7594387394578469, 0.7589473389664464, 0.8665065179327496, 0.8493080559001127, 0.7589473389664464, 0.8500177746390385, 0.8443668813670179, 0.8287517016874189, 0.7589473389664464, 0.7589473389664464, 0.8209167876016517, 0.7589473389664464, 0.8232917968280328, 0.7809776860002055, 0.8614289289599786, 0.7589473389664464, 0.8453770159106586, 0.8622205328770091, 0.7652807925361496 ] } ], "domain": { "x": [ 0.0, 1.0 ], "y": [ 0.0, 1.0 ] }, "line": { "color": [ 0.8575523735207098, 0.7589473389664464, 0.7713408881417616, 0.7589473389664464, 0.7589473389664464, 0.6873151263351892, 0.5517947318138392, 0.6892789212155939, 0.84865282916327, 0.7589473389664464, 0.8433022548212941, 0.8523383104930804, 0.7234035082267647, 0.8384702990271435, 0.6553578753387679, 0.7589473389664464, 0.8526111691800241, 0.7823427544322863, 0.8223909890758533, 0.8206436047571575, 0.8045098847746591, 0.8707925439739279, 0.7730341530532605, 0.7178607978799053, 0.7589473389664464, 0.6736927368700263, 0.8535121334220557, 0.8558325165121945, 0.8452404039956928, 0.8547406234727102, 0.7306645106836179, 0.6526683326874401, 0.8535940521337, 0.7589473389664464, 0.8447764339398024, 0.8466600464007316, 0.7589473389664464, 0.7589473389664464, 0.7886761595646542, 0.8506456454593483, 0.8532117958622667, 0.7778110393232545, 0.8491442520103636, 0.8534575370934044, 0.4522942846813461, 0.8552864787067053, 0.8504818452955479, 0.8540581041604653, 0.851109764553193, 0.6553578753387679, 0.7945180971385419, 0.8415004679232876, 0.7589473389664464, 0.8183777881880789, 0.8138188960080595, 0.6553710353901427, 0.7638611762127526, 0.5517947318138392, 0.7200743207567284, 0.8214900919009012, 0.7128367358884623, 0.7589473389664464, 0.7763367856857689, 0.7589473389664464, 0.8417188495119436, 0.8431111360004495, 0.8460049128126104, 0.8662336406160618, 0.8458955636654558, 0.860391513018782, 0.8507822089369789, 0.8543856895842701, 0.8147197782792158, 0.8473424539345107, 0.7589473389664464, 0.8363953814479268, 0.8575251703680802, 0.7589473389664464, 0.8646229054718205, 0.8529115179176596, 0.8605826281136777, 0.7589473389664464, 0.8082499288995809, 0.7589473389664464, 0.7561082180981185, 0.7962933999920487, 0.8297343275506233, 0.8386341140947392, 0.7915152468994424, 0.7589473389664464, 0.844066513999638, 0.7589473389664464, 0.8321093740364929, 0.7589473389664464, 0.8523109508505987, 0.7589473389664464, 0.7400013584809544, 0.7589473389664464, 0.7589473389664464, 0.8499086863083042, 0.7019462195796302, 0.8448583302957535, 0.8212988911091819, 0.8545768084051147, 0.8439846921626641, 0.8272775933619387, 0.7589473389664464, 0.7589473389664464, 0.7589473389664464, 0.7580738871307999, 0.8468511763994228, 0.8525839362198034, 0.8200705277407885, 0.8570610289185423, 0.7589473389664464, 0.8469875908390989, 0.8212441643723205, 0.8573886553277846, 0.8457864157195397, 0.8116073627879278, 0.8631214039703188, 0.8665066222593177, 0.8667249069733034, 0.6214169813323628, 0.7764728089008149, 0.8532937220258088, 0.5517815717624643, 0.8586170745854108, 0.8162481699536428, 0.7589473389664464, 0.8209988777069437, 0.7860268385312742, 0.8227459155123956, 0.7680373650074755, 0.7563265363456437, 0.7589473389664464, 0.7589473389664464, 0.7589473389664464, 0.8281238904822908, 0.7589473389664464, 0.8166581659143416, 0.7658269383941556, 0.7698400498591573, 0.8539761779969233, 0.8437663068480592, 0.7589473389664464, 0.8652780986751234, 0.827632206818777, 0.8417734570184414, 0.6093704893895968, 0.7589473389664464, 0.8394529882314787, 0.7589473389664464, 0.8553138756086758, 0.8624390895852612, 0.8589993047752019, 0.7368061784876305, 0.8124266654087862, 0.7589473389664464, 0.8417734048551573, 0.8276051042667666, 0.8474789876045505, 0.8444488373381509, 0.8585352303927436, 0.8175043885157164, 0.8071308587891091, 0.8162211680022518, 0.6418315073016861, 0.7589473389664464, 0.7703310926594671, 0.8474516205101711, 0.7589473389664464, 0.7589473389664464, 0.4766407374159524, 0.7589473389664464, 0.7589473389664464, 0.8304714618212639, 0.8223909778980067, 0.8489804294908702, 0.7589473389664464, 0.7589473389664464, 0.8393437247811479, 0.8573613813821265, 0.6553710353901429, 0.8230461450196677, 0.8374600638828836, 0.7589473389664464, 0.8386612054689029, 0.8392073587788069, 0.8499086676785599, 0.7589473389664464, 0.7411205515477386, 0.8569519145061658, 0.7054399486772901, 0.7759821871327263, 0.8609921508788713, 0.8516558023586821, 0.7339404506184908, 0.715350819882006, 0.5968670172710024, 0.3446289646098572, 0.7589473389664464, 0.8477247102059439, 0.8550955350054572, 0.8524201174262591, 0.8330648861698403, 0.7589473389664464, 0.8614017518889907, 0.87139310731504, 0.7589473389664464, 0.7098332485121075, 0.6553710353901427, 0.8221452962820507, 0.8418826794833347, 0.7589473389664464, 0.8483525922041004, 0.8455135607585451, 0.8493080857077036, 0.8569245027004001, 0.7736616550046331, 0.8194698712509549, 0.8234829417305193, 0.7905872136389401, 0.8255849991212351, 0.7707132818638209, 0.7615709509526896, 0.6411750211941286, 0.6950645783319456, 0.8547405824872728, 0.8541400862132402, 0.8552865681294779, 0.8442031259146041, 0.4482052681861609, 0.7969205703339728, 0.8461686533612287, 0.8331468086074336, 0.5266321733224968, 0.6531323213730748, 0.6805691692412041, 0.8450766224616368, 0.7092875646717601, 0.7589473389664464, 0.8185418379904528, 0.5517947318138392, 0.7698395729377032, 0.8382245056327218, 0.6834095457542981, 0.8505365198691253, 0.7589473389664464, 0.857033732617191, 0.8516558582479149, 0.7589473389664464, 0.8231007301704721, 0.7815233623886552, 0.7589473389664464, 0.8234555820880374, 0.8196337198520902, 0.7589473389664464, 0.6069407869598944, 0.7787120184690813, 0.7589473389664464, 0.8242473350430222, 0.8354125730132281, 0.7019721819912894, 0.5517815717624643, 0.8684720975426584, 0.8192786220219002, 0.8037735553089723, 0.6553710353901429, 0.724441993515284, 0.8420737386889974, 0.8268680929524385, 0.6981228781419857, 0.7846631673300161, 0.8479704067256953, 0.5626402796458754, 0.5486825287016359, 0.8458956419103819, 0.7191183801392619, 0.8532937890928881, 0.865305353991037, 0.8451312076124413, 0.8396987220107187, 0.8213536290238898, 0.8223636592409622, 0.7705767183861901, 0.8023258676868613, 0.868499412473754, 0.7589473389664464, 0.7589473389664464, 0.7863555603694815, 0.6972219772410846, 0.7589473389664464, 0.6941964434253229, 0.8382517795783798, 0.8547405564056308, 0.8683902235424004, 0.7589473389664464, 0.7589473389664464, 0.8638311785984776, 0.8603369688534148, 0.7589473389664464, 0.8422102611811907, 0.8036360268105867, 0.8166581212029553, 0.8425650758392672, 0.8530207739160925, 0.8245476167135782, 0.8354672773943962, 0.8216538808868549, 0.8597090868552586, 0.6553578753387679, 0.8377331833862474, 0.8488985145051748, 0.8274413376365063, 0.7589473389664464, 0.8583714078732504, 0.8172313845167674, 0.7589473389664464, 0.8685268093757245, 0.7589473389664464, 0.8355492855288134, 0.8224452947804932, 0.5517815717624643, 0.7589473389664464, 0.8193061307023365, 0.8560509316343902, 0.8696732689362688, 0.8698370206627338, 0.816303064358203, 0.7589473389664464, 0.8429473172069051, 0.7589473389664464, 0.7940268382331982, 0.7008532013035902, 0.8659332620708355, 0.8519287653721938, 0.8671617931069276, 0.8172587068997605, 0.8425105428517465, 0.8303897331330112, 0.8635855789533965, 0.7589473389664464, 0.8713385519718262, 0.4482052681861608, 0.8408453082535244, 0.7072956873147946, 0.8506729380347504, 0.8474789913304994, 0.823592175373259, 0.7899319235609669, 0.8559690278265413, 0.7589473389664464, 0.4502526770849413, 0.8540307631477277, 0.7808958865189247, 0.8055471255962591, 0.24105266103355366, 0.7589473389664464, 0.8676805197072701, 0.8411456458133133, 0.8387432657666041, 0.7841990445102224, 0.8323552307720453, 0.8553957794165246, 0.7437412237603312, 0.8118259232221291, 0.8065027383302258, 0.8495537524198641, 0.8335833370499675, 0.7589473389664464, 0.8083863247095129, 0.8673801076285039, 0.8595179792122607, 0.8401083341986849, 0.7589473389664464, 0.7589473389664464, 0.8279054231968113, 0.8536759224080093, 0.8530480553136484, 0.8275232600740994, 0.8653326763740303, 0.8291884648647307, 0.7589473389664464, 0.8691545982251592, 0.7589473389664464, 0.8439301144637572, 0.7816879487276648, 0.8378696611670543, 0.7589473389664464, 0.7589473389664464, 0.8555595982100691, 0.8057930270431977, 0.8508912786379689, 0.7589473389664464, 0.8463051982091152, 0.7346248962462967, 0.8432476324110008, 0.7589473389664464, 0.6967565764208319, 0.8439299505220074, 0.866397332727345, 0.7589473389664464, 0.8017795243535657, 0.6553710353901427, 0.8466327463734317, 0.8468511056063942, 0.5424946367760617, 0.7589473389664464, 0.8369142757159682, 0.8208348949716493, 0.826649484080902, 0.7589473389664464, 0.4482052681861608, 0.853757777778523, 0.8534302407920533, 0.8600367020866543, 0.8374597695329236, 0.8512734715682717, 0.8237012115407094, 0.8402719555169398, 0.8409271002829076, 0.8273321114456641, 0.8207803433543844, 0.8652235880432959, 0.8380880576595058, 0.7589473389664464, 0.8526111878097684, 0.8691000093484058, 0.7589473389664464, 0.7760635991153257, 0.7589473389664464, 0.8436025253140034, 0.8427835729323376, 0.8426197504128445, 0.7546619463365745, 0.7589473389664464, 0.8256395432866022, 0.8164941049341212, 0.8475608802345529, 0.8573338838795369, 0.7589473389664464, 0.8281511308944092, 0.7589473389664464, 0.8020802345082405, 0.858753518832678, 0.6276888077079151, 0.8128088285314978, 0.8534029184090601, 0.8363954671447505, 0.8442849887370155, 0.7589473389664464, 0.594028235464021, 0.8420191162787042, 0.8104880430388823, 0.7589473389664464, 0.8171493204931174, 0.8636674417758081, 0.7589473389664464, 0.853129914410111, 0.7589473389664464, 0.7589473389664464, 0.8446944407091808, 0.8232918489913168, 0.8564877097154973, 0.8675985637361372, 0.8516011650445934, 0.7589473389664464, 0.8457046311420543, 0.7589473389664464, 0.7589473389664464, 0.8421829462500952, 0.7578820268461322, 0.735387221657182, 0.7589473389664464, 0.8650324468667583, 0.8557779648949294, 0.7589473389664464, 0.8396714070796232, 0.7589473389664464, 0.6758788330372879, 0.7594387394578469, 0.7589473389664464, 0.8665065179327496, 0.8493080559001127, 0.7589473389664464, 0.8500177746390385, 0.8443668813670179, 0.8287517016874189, 0.7589473389664464, 0.7589473389664464, 0.8209167876016517, 0.7589473389664464, 0.8232917968280328, 0.7809776860002055, 0.8614289289599786, 0.7589473389664464, 0.8453770159106586, 0.8622205328770091, 0.7652807925361496 ], "coloraxis": "coloraxis" }, "name": "", "type": "parcoords" } ], "layout": { "coloraxis": { "colorbar": { "title": { "text": "mean_test_score" } }, "colorscale": [ [ 0.0, "#440154" ], [ 0.1111111111111111, "#482878" ], [ 0.2222222222222222, "#3e4989" ], [ 0.3333333333333333, "#31688e" ], [ 0.4444444444444444, "#26828e" ], [ 0.5555555555555556, "#1f9e89" ], [ 0.6666666666666666, "#35b779" ], [ 0.7777777777777778, "#6ece58" ], [ 0.8888888888888888, "#b5de2b" ], [ 1.0, "#fde725" ] ] }, "legend": { "tracegroupgap": 0 }, "margin": { "t": 60 }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } } } }, "text/html": [ "
\n", " \n", " \n", "
\n", " \n", "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import plotly.express as px\n", "\n", "fig = px.parallel_coordinates(\n", " cv_results.rename(shorten_param, axis=1).apply({\n", " \"learning_rate\": np.log10,\n", " \"max_leaf_nodes\": np.log2,\n", " \"max_bins\": np.log2,\n", " \"min_samples_leaf\": np.log10,\n", " \"l2_regularization\": np.log10,\n", " \"mean_test_score\": lambda x: x,}),\n", " color=\"mean_test_score\",\n", " color_continuous_scale=px.colors.sequential.Viridis,\n", ")\n", "fig.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The parallel coordinates plot will display the values of the hyper-parameters\n", "on different columns while the performance metric is color coded. Thus, we\n", "are able to quickly inspect if there is a range of hyper-parameters which is\n", "working or not.\n", "\n", "Note that we **transformed most axis values by taking a log10 or log2** to\n", "spread the active ranges and improve the readability of the plot.\n", "\n", "It is possible to **select a range of results by clicking and holding on\n", "any axis** of the parallel coordinate plot. You can then slide (move)\n", "the range selection and cross two selections to see the intersections." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Quizz**\n", "\n", "\n", "Select the worst performing models (for instance models with a \"mean_test_score\" lower than 0.7): what do have all these moels in common (choose one):\n", "\n", "\n", "| | |\n", "|-------------------------------|------|\n", "| too large `l2_regularization` | |\n", "| too small `l2_regularization` | |\n", "| too large `learning_rate` | |\n", "| too low `learning_rate` | |\n", "| too large `max_bins` | |\n", "| too large `max_bins` | |\n", "\n", "\n", "Using the above plot, identify ranges of values for hyperparameter that always prevent the model to reach a test score higher than 0.86, irrespective of the other values:\n", "\n", "\n", "| | True | False |\n", "|-------------------------------|------|-------|\n", "| too large `l2_regularization` | | |\n", "| too small `l2_regularization` | | |\n", "| too large `learning_rate` | | |\n", "| too low `learning_rate` | | |\n", "| too large `max_bins` | | |\n", "| too large `max_bins` | | |" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercises:\n", "\n", "- Build a machine learning pipeline:\n", " * preprocess the categorical columns using a `OneHotEncoder` and use\n", " a `StandardScaler` to normalize the numerical data.\n", " * use a `LogisticRegression` as a predictive model.\n", "- Make an hyper-parameters search using `RandomizedSearchCV` and tuning the\n", " parameters:\n", " * `C` with values ranging from 0.001 to 10. You can use a reciprocal\n", " distribution (i.e. `scipy.stats.reciprocal`);\n", " * `solver` with possible values being `\"liblinear\"` and `\"lbfgs\"`;\n", " * `penalty` with possible values being `\"l2\"` and `\"l1\"`;\n", " * `drop` with possible values being `None` or `\"first\"`.\n", "\n", "You might get some `FitFailedWarning` and try to explain why." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Combining evaluation and hyper-parameters search\n", "\n", "Cross-validation was used for searching for the best model parameters. We\n", "previously evaluated model performance through cross-validation as well. If\n", "we would like to combine both aspects, we need to perform a **\"nested\"\n", "cross-validation**. The \"outer\" cross-validation is applied to assess the model\n", "while the \"inner\" cross-validation sets the hyper-parameters of the model on\n", "the data set provided by the \"outer\" cross-validation.\n", "\n", "\n", "In practice, it can be implemented by calling `cross_val_score` or\n", "`cross_validate` on an instance of `GridSearchCV`, `RandomSearchCV`, or any\n", "other `EstimatorCV` class." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import cross_val_score\n", "\n", "# recall the definition of our grid-search\n", "param_distributions = {\n", " 'classifier__max_iter': reciprocal_int(10, 50),\n", " 'classifier__learning_rate': reciprocal(0.01, 10),\n", " 'classifier__max_leaf_nodes': reciprocal_int(2, 16),\n", " 'classifier__min_samples_leaf': reciprocal_int(1, 50),}\n", "model_random_search = RandomizedSearchCV(\n", " model, param_distributions=param_distributions, n_iter=10,\n", " n_jobs=4, cv=5)\n", "\n", "scores = cross_val_score(model_random_search, data, target, n_jobs=4,\n", " cv=5)" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The cross-validated accuracy score is: 0.866 +- 0.004\n" ] } ], "source": [ "print(f\"The cross-validated accuracy score is:\"\n", " f\" {scores.mean():.3f} +- {scores.std():.3f}\")" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The scores obtained for each CV split are:\n", "[0.86569761 0.86794964 0.86885749 0.86885749 0.85933661]\n" ] } ], "source": [ "print(\"The scores obtained for each CV split are:\")\n", "print(scores)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Be aware that the best model found for each split of the outer cross-validation loop might not share the same hyper-parameter values.\n", "\n", "When analyzing such model, you should not only look at the\n", "overall model performance but look at the hyper-parameters variations as\n", "well." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## In this notebook, we have:\n", "\n", "* manually tuned the hyper-parameters of a machine-learning pipeline;\n", "* automatically tuned the hyper-parameters of a machine-learning pipeline by\n", " by exhaustively searching the best combination of parameters from a defined\n", " grid;\n", "* automatically tuned the hyper-parameters of a machine-learning pipeline by\n", " drawing values candidates from some predefined distributions;\n", "* nested an hyper-parameters tuning procedure within an cross-validation\n", " evaluation procedure.\n", "\n", "## Main take-away points\n", "\n", "* a grid-search is a costly exhaustive search and does scale with the number of\n", " parameters to search;\n", "* a randomized-search will always run with a fixed given budget;\n", "* when assessing the performance of a model, hyper-parameters search should\n", " be tuned on the training data of a predifined train test split;\n", "* alternatively it is possible to nest parameter tuning within a\n", " cross-validation scheme." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "jupytext": { "formats": "python_scripts//py:percent,notebooks//ipynb" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.4" } }, "nbformat": 4, "nbformat_minor": 2 }