{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Comparison of Calibration of Classifiers\n\nWell calibrated classifiers are probabilistic classifiers for which the output\nof :term:`predict_proba` can be directly interpreted as a confidence level.\nFor instance, a well calibrated (binary) classifier should classify the samples\nsuch that for the samples to which it gave a :term:`predict_proba` value close\nto 0.8, approximately 80% actually belong to the positive class.\n\nIn this example we will compare the calibration of four different\nmodels: `Logistic_regression`, `gaussian_naive_bayes`,\n`Random Forest Classifier <forest>` and `Linear SVM\n<svm_classification>`.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Authors: The scikit-learn developers\nSPDX-License-Identifier: BSD-3-Clause\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "#\n# Dataset\n# -------\n#\n# We will use a synthetic binary classification dataset with 100,000 samples\n# and 20 features. Of the 20 features, only 2 are informative, 2 are\n# redundant (random combinations of the informative features) and the\n# remaining 16 are uninformative (random numbers).\n#\n# Of the 100,000 samples, 100 will be used for model fitting and the remaining\n# for testing. Note that this split is quite unusual: the goal is to obtain\n# stable calibration curve estimates for models that are potentially prone to\n# overfitting. In practice, one should rather use cross-validation with more\n# balanced splits but this would make the code of this example more complicated\n# to follow.\n\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import train_test_split\n\nX, y = make_classification(\n    n_samples=100_000, n_features=20, n_informative=2, n_redundant=2, random_state=42\n)\n\ntrain_samples = 100  # Samples used for training the models\nX_train, X_test, y_train, y_test = train_test_split(\n    X,\n    y,\n    shuffle=False,\n    test_size=100_000 - train_samples,\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Calibration curves\n\nBelow, we train each of the four models with the small training dataset, then\nplot calibration curves (also known as reliability diagrams) using\npredicted probabilities of the test dataset. Calibration curves are created\nby binning predicted probabilities, then plotting the mean predicted\nprobability in each bin against the observed frequency ('fraction of\npositives'). Below the calibration curve, we plot a histogram showing\nthe distribution of the predicted probabilities or more specifically,\nthe number of samples in each predicted probability bin.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import numpy as np\n\nfrom sklearn.svm import LinearSVC\n\n\nclass NaivelyCalibratedLinearSVC(LinearSVC):\n    \"\"\"LinearSVC with `predict_proba` method that naively scales\n    `decision_function` output.\"\"\"\n\n    def fit(self, X, y):\n        super().fit(X, y)\n        df = self.decision_function(X)\n        self.df_min_ = df.min()\n        self.df_max_ = df.max()\n\n    def predict_proba(self, X):\n        \"\"\"Min-max scale output of `decision_function` to [0,1].\"\"\"\n        df = self.decision_function(X)\n        calibrated_df = (df - self.df_min_) / (self.df_max_ - self.df_min_)\n        proba_pos_class = np.clip(calibrated_df, 0, 1)\n        proba_neg_class = 1 - proba_pos_class\n        proba = np.c_[proba_neg_class, proba_pos_class]\n        return proba"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.calibration import CalibrationDisplay\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.linear_model import LogisticRegressionCV\nfrom sklearn.naive_bayes import GaussianNB\n\n# Define the classifiers to be compared in the study.\n#\n# Note that we use a variant of the logistic regression model that can\n# automatically tune its regularization parameter.\n#\n# For a fair comparison, we should run a hyper-parameter search for all the\n# classifiers but we don't do it here for the sake of keeping the example code\n# concise and fast to execute.\nlr = LogisticRegressionCV(\n    Cs=np.logspace(-6, 6, 101), cv=10, scoring=\"neg_log_loss\", max_iter=1_000\n)\ngnb = GaussianNB()\nsvc = NaivelyCalibratedLinearSVC(C=1.0)\nrfc = RandomForestClassifier(random_state=42)\n\nclf_list = [\n    (lr, \"Logistic Regression\"),\n    (gnb, \"Naive Bayes\"),\n    (svc, \"SVC\"),\n    (rfc, \"Random forest\"),\n]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\nfrom matplotlib.gridspec import GridSpec\n\nfig = plt.figure(figsize=(10, 10))\ngs = GridSpec(4, 2)\ncolors = plt.get_cmap(\"Dark2\")\n\nax_calibration_curve = fig.add_subplot(gs[:2, :2])\ncalibration_displays = {}\nmarkers = [\"^\", \"v\", \"s\", \"o\"]\nfor i, (clf, name) in enumerate(clf_list):\n    clf.fit(X_train, y_train)\n    display = CalibrationDisplay.from_estimator(\n        clf,\n        X_test,\n        y_test,\n        n_bins=10,\n        name=name,\n        ax=ax_calibration_curve,\n        color=colors(i),\n        marker=markers[i],\n    )\n    calibration_displays[name] = display\n\nax_calibration_curve.grid()\nax_calibration_curve.set_title(\"Calibration plots\")\n\n# Add histogram\ngrid_positions = [(2, 0), (2, 1), (3, 0), (3, 1)]\nfor i, (_, name) in enumerate(clf_list):\n    row, col = grid_positions[i]\n    ax = fig.add_subplot(gs[row, col])\n\n    ax.hist(\n        calibration_displays[name].y_prob,\n        range=(0, 1),\n        bins=10,\n        label=name,\n        color=colors(i),\n    )\n    ax.set(title=name, xlabel=\"Mean predicted probability\", ylabel=\"Count\")\n\nplt.tight_layout()\nplt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Analysis of the results\n\n:class:`~sklearn.linear_model.LogisticRegressionCV` returns reasonably well\ncalibrated predictions despite the small training set size: its reliability\ncurve is the closest to the diagonal among the four models.\n\nLogistic regression is trained by minimizing the log-loss which is a strictly\nproper scoring rule: in the limit of infinite training data, strictly proper\nscoring rules are minimized by the model that predicts the true conditional\nprobabilities. That (hypothetical) model would therefore be perfectly\ncalibrated. However, using a proper scoring rule as training objective is not\nsufficient to guarantee a well-calibrated model by itself: even with a very\nlarge training set, logistic regression could still be poorly calibrated, if\nit was too strongly regularized or if the choice and preprocessing of input\nfeatures made this model mis-specified (e.g. if the true decision boundary of\nthe dataset is a highly non-linear function of the input features).\n\nIn this example the training set was intentionally kept very small. In this\nsetting, optimizing the log-loss can still lead to poorly calibrated models\nbecause of overfitting. To mitigate this, the\n:class:`~sklearn.linear_model.LogisticRegressionCV` class was configured to\ntune the `C` regularization parameter to also minimize the log-loss via inner\ncross-validation so as to find the best compromise for this model in the\nsmall training set setting.\n\nBecause of the finite training set size and the lack of guarantee for\nwell-specification, we observe that the calibration curve of the logistic\nregression model is close but not perfectly on the diagonal. The shape of the\ncalibration curve of this model can be interpreted as slightly\nunder-confident: the predicted probabilities are a bit too close to 0.5\ncompared to the true fraction of positive samples.\n\nThe other methods all output less well calibrated probabilities:\n\n* :class:`~sklearn.naive_bayes.GaussianNB` tends to push probabilities to 0\n  or 1 (see histogram) on this particular dataset (over-confidence). This is\n  mainly because the naive Bayes equation only provides correct estimate of\n  probabilities when the assumption that features are conditionally\n  independent holds [2]_. However, features can be correlated and this is the case\n  with this dataset, which contains 2 features generated as random linear\n  combinations of the informative features. These correlated features are\n  effectively being 'counted twice', resulting in pushing the predicted\n  probabilities towards 0 and 1 [3]_. Note, however, that changing the seed\n  used to generate the dataset can lead to widely varying results for the\n  naive Bayes estimator.\n\n* :class:`~sklearn.svm.LinearSVC` is not a natural probabilistic classifier.\n  In order to interpret its prediction as such, we naively scaled the output\n  of the :term:`decision_function` into [0, 1] by applying min-max scaling in\n  the `NaivelyCalibratedLinearSVC` wrapper class defined above. This\n  estimator shows a typical sigmoid-shaped calibration curve on this data:\n  predictions larger than 0.5 correspond to samples with an even larger\n  effective positive class fraction (above the diagonal), while predictions\n  below 0.5 corresponds to even lower positive class fractions (below the\n  diagonal). This under-confident predictions are typical for maximum-margin\n  methods [1]_.\n\n* :class:`~sklearn.ensemble.RandomForestClassifier`'s prediction histogram\n  shows peaks at approx. 0.2 and 0.9 probability, while probabilities close to\n  0 or 1 are very rare. An explanation for this is given by [1]_:\n  \"Methods such as bagging and random forests that average\n  predictions from a base set of models can have difficulty making\n  predictions near 0 and 1 because variance in the underlying base models\n  will bias predictions that should be near zero or one away from these\n  values. Because predictions are restricted to the interval [0, 1], errors\n  caused by variance tend to be one-sided near zero and one. For example, if\n  a model should predict p = 0 for a case, the only way bagging can achieve\n  this is if all bagged trees predict zero. If we add noise to the trees that\n  bagging is averaging over, this noise will cause some trees to predict\n  values larger than 0 for this case, thus moving the average prediction of\n  the bagged ensemble away from 0. We observe this effect most strongly with\n  random forests because the base-level trees trained with random forests\n  have relatively high variance due to feature subsetting.\" This effect can\n  make random forests under-confident. Despite this possible bias, note that\n  the trees themselves are fit by minimizing either the Gini or Entropy\n  criterion, both of which lead to splits that minimize proper scoring rules:\n  the Brier score or the log-loss respectively. See `the user guide\n  <tree_mathematical_formulation>` for more details. This can explain why\n  this model shows a good enough calibration curve on this particular example\n  dataset. Indeed the Random Forest model is not significantly more\n  under-confident than the Logistic Regression model.\n\nFeel free to re-run this example with different random seeds and other\ndataset generation parameters to see how different the calibration plots can\nlook. In general, Logistic Regression and Random Forest will tend to be the\nbest calibrated classifiers, while SVC will often display the typical\nunder-confident miscalibration. The naive Bayes model is also often poorly\ncalibrated but the general shape of its calibration curve can vary widely\ndepending on the dataset.\n\nFinally, note that for some dataset seeds, all models are poorly calibrated,\neven when tuning the regularization parameter as above. This is bound to\nhappen when the training size is too small or when the model is severely\nmisspecified.\n\n## References\n\n.. [1] [Predicting Good Probabilities with Supervised Learning](https://dl.acm.org/doi/pdf/10.1145/1102351.1102430), A.\n       Niculescu-Mizil & R. Caruana, ICML 2005\n\n.. [2] [Beyond independence: Conditions for the optimality of the simple\n       bayesian classifier](https://www.ics.uci.edu/~pazzani/Publications/mlc96-pedro.pdf)\n       Domingos, P., & Pazzani, M., Proc. 13th Intl. Conf. Machine Learning.\n       1996.\n\n.. [3] [Obtaining calibrated probability estimates from decision trees and\n       naive Bayesian classifiers](https://citeseerx.ist.psu.edu/doc_view/pid/4f67a122ec3723f08ad5cbefecad119b432b3304)\n       Zadrozny, Bianca, and Charles Elkan. Icml. Vol. 1. 2001.\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.21"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}