{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Comparing Random Forests and Histogram Gradient Boosting models\n\nIn this example we compare the performance of Random Forest (RF) and Histogram\nGradient Boosting (HGBT) models in terms of score and computation time for a\nregression dataset, though **all the concepts here presented apply to\nclassification as well**.\n\nThe comparison is made by varying the parameters that control the number of\ntrees according to each estimator:\n\n- `n_estimators` controls the number of trees in the forest. It's a fixed number.\n- `max_iter` is the maximum number of iterations in a gradient boosting\n  based model. The number of iterations corresponds to the number of trees for\n  regression and binary classification problems. Furthermore, the actual number\n  of trees required by the model depends on the stopping criteria.\n\nHGBT uses gradient boosting to iteratively improve the model's performance by\nfitting each tree to the negative gradient of the loss function with respect to\nthe predicted value. RFs, on the other hand, are based on bagging and use a\nmajority vote to predict the outcome.\n\nSee the `User Guide <ensemble>` for more information on ensemble models or\nsee `sphx_glr_auto_examples_ensemble_plot_hgbt_regression.py` for an\nexample showcasing some other features of HGBT models.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Authors: The scikit-learn developers\n# SPDX-License-Identifier: BSD-3-Clause"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Load dataset\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.datasets import fetch_california_housing\n\nX, y = fetch_california_housing(return_X_y=True, as_frame=True)\nn_samples, n_features = X.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "HGBT uses a histogram-based algorithm on binned feature values that can\nefficiently handle large datasets (tens of thousands of samples or more) with\na high number of features (see `Why_it's_faster`). The scikit-learn\nimplementation of RF does not use binning and relies on exact splitting, which\ncan be computationally expensive.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print(f\"The dataset consists of {n_samples} samples and {n_features} features\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Compute score and computation times\n\nNotice that many parts of the implementation of\n:class:`~sklearn.ensemble.HistGradientBoostingClassifier` and\n:class:`~sklearn.ensemble.HistGradientBoostingRegressor` are parallelized by\ndefault.\n\nThe implementation of :class:`~sklearn.ensemble.RandomForestRegressor` and\n:class:`~sklearn.ensemble.RandomForestClassifier` can also be run on multiple\ncores by using the `n_jobs` parameter, here set to match the number of\nphysical cores on the host machine. See `parallelism` for more\ninformation.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import joblib\n\nN_CORES = joblib.cpu_count(only_physical_cores=True)\nprint(f\"Number of physical cores: {N_CORES}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Unlike RF, HGBT models offer an early-stopping option (see\n`sphx_glr_auto_examples_ensemble_plot_gradient_boosting_early_stopping.py`)\nto avoid adding new unnecessary trees. Internally, the algorithm uses an\nout-of-sample set to compute the generalization performance of the model at\neach addition of a tree. Thus, if the generalization performance is not\nimproving for more than `n_iter_no_change` iterations, it stops adding trees.\n\nThe other parameters of both models were tuned but the procedure is not shown\nhere to keep the example simple.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n\nfrom sklearn.ensemble import HistGradientBoostingRegressor, RandomForestRegressor\nfrom sklearn.model_selection import GridSearchCV, KFold\n\nmodels = {\n    \"Random Forest\": RandomForestRegressor(\n        min_samples_leaf=5, random_state=0, n_jobs=N_CORES\n    ),\n    \"Hist Gradient Boosting\": HistGradientBoostingRegressor(\n        max_leaf_nodes=15, random_state=0, early_stopping=False\n    ),\n}\nparam_grids = {\n    \"Random Forest\": {\"n_estimators\": [10, 20, 50, 100]},\n    \"Hist Gradient Boosting\": {\"max_iter\": [10, 20, 50, 100, 300, 500]},\n}\ncv = KFold(n_splits=4, shuffle=True, random_state=0)\n\nresults = []\nfor name, model in models.items():\n    grid_search = GridSearchCV(\n        estimator=model,\n        param_grid=param_grids[name],\n        return_train_score=True,\n        cv=cv,\n    ).fit(X, y)\n    result = {\"model\": name, \"cv_results\": pd.DataFrame(grid_search.cv_results_)}\n    results.append(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        ".. Note::\n Tuning the `n_estimators` for RF generally results in a waste of computer\n power. In practice one just needs to ensure that it is large enough so that\n doubling its value does not lead to a significant improvement of the testing\n score.\n\n## Plot results\nWe can use a [plotly.express.scatter](https://plotly.com/python-api-reference/generated/plotly.express.scatter.html)\nto visualize the trade-off between elapsed computing time and mean test score.\nPassing the cursor over a given point displays the corresponding parameters.\nError bars correspond to one standard deviation as computed in the different\nfolds of the cross-validation.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import plotly.colors as colors\nimport plotly.express as px\nfrom plotly.subplots import make_subplots\n\nfig = make_subplots(\n    rows=1,\n    cols=2,\n    shared_yaxes=True,\n    subplot_titles=[\"Train time vs score\", \"Predict time vs score\"],\n)\nmodel_names = [result[\"model\"] for result in results]\ncolors_list = colors.qualitative.Plotly * (\n    len(model_names) // len(colors.qualitative.Plotly) + 1\n)\n\nfor idx, result in enumerate(results):\n    cv_results = result[\"cv_results\"].round(3)\n    model_name = result[\"model\"]\n    param_name = list(param_grids[model_name].keys())[0]\n    cv_results[param_name] = cv_results[\"param_\" + param_name]\n    cv_results[\"model\"] = model_name\n\n    scatter_fig = px.scatter(\n        cv_results,\n        x=\"mean_fit_time\",\n        y=\"mean_test_score\",\n        error_x=\"std_fit_time\",\n        error_y=\"std_test_score\",\n        hover_data=param_name,\n        color=\"model\",\n    )\n    line_fig = px.line(\n        cv_results,\n        x=\"mean_fit_time\",\n        y=\"mean_test_score\",\n    )\n\n    scatter_trace = scatter_fig[\"data\"][0]\n    line_trace = line_fig[\"data\"][0]\n    scatter_trace.update(marker=dict(color=colors_list[idx]))\n    line_trace.update(line=dict(color=colors_list[idx]))\n    fig.add_trace(scatter_trace, row=1, col=1)\n    fig.add_trace(line_trace, row=1, col=1)\n\n    scatter_fig = px.scatter(\n        cv_results,\n        x=\"mean_score_time\",\n        y=\"mean_test_score\",\n        error_x=\"std_score_time\",\n        error_y=\"std_test_score\",\n        hover_data=param_name,\n    )\n    line_fig = px.line(\n        cv_results,\n        x=\"mean_score_time\",\n        y=\"mean_test_score\",\n    )\n\n    scatter_trace = scatter_fig[\"data\"][0]\n    line_trace = line_fig[\"data\"][0]\n    scatter_trace.update(marker=dict(color=colors_list[idx]))\n    line_trace.update(line=dict(color=colors_list[idx]))\n    fig.add_trace(scatter_trace, row=1, col=2)\n    fig.add_trace(line_trace, row=1, col=2)\n\nfig.update_layout(\n    xaxis=dict(title=\"Train time (s) - lower is better\"),\n    yaxis=dict(title=\"Test R2 score - higher is better\"),\n    xaxis2=dict(title=\"Predict time (s) - lower is better\"),\n    legend=dict(x=0.72, y=0.05, traceorder=\"normal\", borderwidth=1),\n    title=dict(x=0.5, text=\"Speed-score trade-off of tree-based ensembles\"),\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Both HGBT and RF models improve when increasing the number of trees in the\nensemble. However, the scores reach a plateau where adding new trees just\nmakes fitting and scoring slower. The RF model reaches such plateau earlier\nand can never reach the test score of the largest HGBDT model.\n\nNote that the results shown on the above plot can change slightly across runs\nand even more significantly when running on other machines: try to run this\nexample on your own local machine.\n\nOverall, one should often observe that the Histogram-based gradient boosting\nmodels uniformly dominate the Random Forest models in the \"test score vs\ntraining speed trade-off\" (the HGBDT curve should be on the top left of the RF\ncurve, without ever crossing). The \"test score vs prediction speed\" trade-off\ncan also be more disputed, but it's most often favorable to HGBDT. It's always\na good idea to check both kinds of model (with hyper-parameter tuning) and\ncompare their performance on your specific problem to determine which model is\nthe best fit but **HGBT almost always offers a more favorable speed-accuracy\ntrade-off than RF**, either with the default hyper-parameters or including the\nhyper-parameter tuning cost.\n\nThere is one exception to this rule of thumb though: when training a\nmulticlass classification model with a large number of possible classes, HGBDT\nfits internally one-tree per class at each boosting iteration while the trees\nused by the RF models are naturally multiclass which should improve the speed\naccuracy trade-off of the RF models in this case.\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.21"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}