{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n# Combine predictors using stacking\n\n.. currentmodule:: sklearn\n\nStacking refers to a method to blend estimators. In this strategy, some\nestimators are individually fitted on some training data while a final\nestimator is trained using the stacked predictions of these base estimators.\n\nIn this example, we illustrate the use case in which different regressors are\nstacked together and a final linear penalized regressor is used to output the\nprediction. We compare the performance of each individual regressor with the\nstacking strategy. Stacking slightly improves the overall performance.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Authors: The scikit-learn developers\n# SPDX-License-Identifier: BSD-3-Clause" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Download the dataset\n\nWe will use the `Ames Housing`_ dataset which was first compiled by Dean De Cock\nand became better known after it was used in Kaggle challenge. It is a set\nof 1460 residential homes in Ames, Iowa, each described by 80 features. We\nwill use it to predict the final logarithmic price of the houses. In this\nexample we will use only 20 most interesting features chosen using\nGradientBoostingRegressor() and limit number of entries (here we won't go\ninto the details on how to select the most interesting features).\n\nThe Ames housing dataset is not shipped with scikit-learn and therefore we\nwill fetch it from `OpenML`_.\n\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import numpy as np\n\nfrom sklearn.datasets import fetch_openml\nfrom sklearn.utils import shuffle\n\n\ndef load_ames_housing():\n df = fetch_openml(name=\"house_prices\", as_frame=True)\n X = df.data\n y = df.target\n\n features = [\n \"YrSold\",\n \"HeatingQC\",\n \"Street\",\n \"YearRemodAdd\",\n \"Heating\",\n \"MasVnrType\",\n \"BsmtUnfSF\",\n \"Foundation\",\n \"MasVnrArea\",\n \"MSSubClass\",\n \"ExterQual\",\n \"Condition2\",\n \"GarageCars\",\n \"GarageType\",\n \"OverallQual\",\n \"TotalBsmtSF\",\n \"BsmtFinSF1\",\n \"HouseStyle\",\n \"MiscFeature\",\n \"MoSold\",\n ]\n\n X = X.loc[:, features]\n X, y = shuffle(X, y, random_state=0)\n\n X = X.iloc[:600]\n y = y.iloc[:600]\n return X, np.log(y)\n\n\nX, y = load_ames_housing()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Make pipeline to preprocess the data\n\nBefore we can use Ames dataset we still need to do some preprocessing.\nFirst, we will select the categorical and numerical columns of the dataset to\nconstruct the first step of the pipeline.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.compose import make_column_selector\n\ncat_selector = make_column_selector(dtype_include=object)\nnum_selector = make_column_selector(dtype_include=np.number)\ncat_selector(X)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "num_selector(X)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, we will need to design preprocessing pipelines which depends on the\nending regressor. If the ending regressor is a linear model, one needs to\none-hot encode the categories. If the ending regressor is a tree-based model\nan ordinal encoder will be sufficient. Besides, numerical values need to be\nstandardized for a linear model while the raw numerical data can be treated\nas is by a tree-based model. However, both models need an imputer to\nhandle missing values.\n\nWe will first design the pipeline required for the tree-based models.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.compose import make_column_transformer\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import OrdinalEncoder\n\ncat_tree_processor = OrdinalEncoder(\n handle_unknown=\"use_encoded_value\",\n unknown_value=-1,\n encoded_missing_value=-2,\n)\nnum_tree_processor = SimpleImputer(strategy=\"mean\", add_indicator=True)\n\ntree_preprocessor = make_column_transformer(\n (num_tree_processor, num_selector), (cat_tree_processor, cat_selector)\n)\ntree_preprocessor" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, we will now define the preprocessor used when the ending regressor\nis a linear model.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.preprocessing import OneHotEncoder, StandardScaler\n\ncat_linear_processor = OneHotEncoder(handle_unknown=\"ignore\")\nnum_linear_processor = make_pipeline(\n StandardScaler(), SimpleImputer(strategy=\"mean\", add_indicator=True)\n)\n\nlinear_preprocessor = make_column_transformer(\n (num_linear_processor, num_selector), (cat_linear_processor, cat_selector)\n)\nlinear_preprocessor" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Stack of predictors on a single data set\n\nIt is sometimes tedious to find the model which will best perform on a given\ndataset. Stacking provide an alternative by combining the outputs of several\nlearners, without the need to choose a model specifically. The performance of\nstacking is usually close to the best model and sometimes it can outperform\nthe prediction performance of each individual model.\n\nHere, we combine 3 learners (linear and non-linear) and use a ridge regressor\nto combine their outputs together.\n\n

Note

Although we will make new pipelines with the processors which we wrote in\n the previous section for the 3 learners, the final estimator\n :class:`~sklearn.linear_model.RidgeCV()` does not need preprocessing of\n the data as it will be fed with the already preprocessed output from the 3\n learners.

\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.linear_model import LassoCV\n\nlasso_pipeline = make_pipeline(linear_preprocessor, LassoCV())\nlasso_pipeline" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.ensemble import RandomForestRegressor\n\nrf_pipeline = make_pipeline(tree_preprocessor, RandomForestRegressor(random_state=42))\nrf_pipeline" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.ensemble import HistGradientBoostingRegressor\n\ngbdt_pipeline = make_pipeline(\n tree_preprocessor, HistGradientBoostingRegressor(random_state=0)\n)\ngbdt_pipeline" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.ensemble import StackingRegressor\nfrom sklearn.linear_model import RidgeCV\n\nestimators = [\n (\"Random Forest\", rf_pipeline),\n (\"Lasso\", lasso_pipeline),\n (\"Gradient Boosting\", gbdt_pipeline),\n]\n\nstacking_regressor = StackingRegressor(estimators=estimators, final_estimator=RidgeCV())\nstacking_regressor" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Measure and plot the results\n\nNow we can use Ames Housing dataset to make the predictions. We check the\nperformance of each individual predictor as well as of the stack of the\nregressors.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import time\n\nimport matplotlib.pyplot as plt\n\nfrom sklearn.metrics import PredictionErrorDisplay\nfrom sklearn.model_selection import cross_val_predict, cross_validate\n\nfig, axs = plt.subplots(2, 2, figsize=(9, 7))\naxs = np.ravel(axs)\n\nfor ax, (name, est) in zip(\n axs, estimators + [(\"Stacking Regressor\", stacking_regressor)]\n):\n scorers = {\"R2\": \"r2\", \"MAE\": \"neg_mean_absolute_error\"}\n\n start_time = time.time()\n scores = cross_validate(\n est, X, y, scoring=list(scorers.values()), n_jobs=-1, verbose=0\n )\n elapsed_time = time.time() - start_time\n\n y_pred = cross_val_predict(est, X, y, n_jobs=-1, verbose=0)\n scores = {\n key: (\n f\"{np.abs(np.mean(scores[f'test_{value}'])):.2f} +- \"\n f\"{np.std(scores[f'test_{value}']):.2f}\"\n )\n for key, value in scorers.items()\n }\n\n display = PredictionErrorDisplay.from_predictions(\n y_true=y,\n y_pred=y_pred,\n kind=\"actual_vs_predicted\",\n ax=ax,\n scatter_kwargs={\"alpha\": 0.2, \"color\": \"tab:blue\"},\n line_kwargs={\"color\": \"tab:red\"},\n )\n ax.set_title(f\"{name}\\nEvaluation in {elapsed_time:.2f} seconds\")\n\n for name, score in scores.items():\n ax.plot([], [], \" \", label=f\"{name}: {score}\")\n ax.legend(loc=\"upper left\")\n\nplt.suptitle(\"Single predictors versus stacked predictors\")\nplt.tight_layout()\nplt.subplots_adjust(top=0.9)\nplt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The stacked regressor will combine the strengths of the different regressors.\nHowever, we also see that training the stacked regressor is much more\ncomputationally expensive.\n\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.21" } }, "nbformat": 4, "nbformat_minor": 0 }