{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Decision Boundaries of Multinomial and One-vs-Rest Logistic Regression\n\nThis example compares decision boundaries of multinomial and one-vs-rest\nlogistic regression on a 2D dataset with three classes.\n\nWe make a comparison of the decision boundaries of both methods that is equivalent\nto call the method `predict`. In addition, we plot the hyperplanes that correspond to\nthe line when the probability estimate for a class is of 0.5.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Authors: The scikit-learn developers\n# SPDX-License-Identifier: BSD-3-Clause"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Dataset Generation\n\nWe generate a synthetic dataset using :func:`~sklearn.datasets.make_blobs` function.\nThe dataset consists of 1,000 samples from three different classes,\ncentered around [-5, 0], [0, 1.5], and [5, -1]. After generation, we apply a linear\ntransformation to introduce some correlation between features and make the problem\nmore challenging. This results in a 2D dataset with three overlapping classes,\nsuitable for demonstrating the differences between multinomial and one-vs-rest\nlogistic regression.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\nimport numpy as np\n\nfrom sklearn.datasets import make_blobs\n\ncenters = [[-5, 0], [0, 1.5], [5, -1]]\nX, y = make_blobs(n_samples=1_000, centers=centers, random_state=40)\ntransformation = [[0.4, 0.2], [-0.4, 1.2]]\nX = np.dot(X, transformation)\n\nfig, ax = plt.subplots(figsize=(6, 4))\n\nscatter = ax.scatter(X[:, 0], X[:, 1], c=y, edgecolor=\"black\")\nax.set(title=\"Synthetic Dataset\", xlabel=\"Feature 1\", ylabel=\"Feature 2\")\n_ = ax.legend(*scatter.legend_elements(), title=\"Classes\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Classifier Training\n\nWe train two different logistic regression classifiers: multinomial and one-vs-rest.\nThe multinomial classifier handles all classes simultaneously, while the one-vs-rest\napproach trains a binary classifier for each class against all others.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.linear_model import LogisticRegression\nfrom sklearn.multiclass import OneVsRestClassifier\n\nlogistic_regression_multinomial = LogisticRegression().fit(X, y)\nlogistic_regression_ovr = OneVsRestClassifier(LogisticRegression()).fit(X, y)\n\naccuracy_multinomial = logistic_regression_multinomial.score(X, y)\naccuracy_ovr = logistic_regression_ovr.score(X, y)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Decision Boundaries Visualization\n\nLet's visualize the decision boundaries of both models that is provided by the\nmethod `predict` of the classifiers.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.inspection import DecisionBoundaryDisplay\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5), sharex=True, sharey=True)\n\nfor model, title, ax in [\n    (\n        logistic_regression_multinomial,\n        f\"Multinomial Logistic Regression\\n(Accuracy: {accuracy_multinomial:.3f})\",\n        ax1,\n    ),\n    (\n        logistic_regression_ovr,\n        f\"One-vs-Rest Logistic Regression\\n(Accuracy: {accuracy_ovr:.3f})\",\n        ax2,\n    ),\n]:\n    DecisionBoundaryDisplay.from_estimator(\n        model,\n        X,\n        ax=ax,\n        response_method=\"predict\",\n        alpha=0.8,\n    )\n    scatter = ax.scatter(X[:, 0], X[:, 1], c=y, edgecolor=\"k\")\n    legend = ax.legend(*scatter.legend_elements(), title=\"Classes\")\n    ax.add_artist(legend)\n    ax.set_title(title)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We see that the decision boundaries are different. This difference stems from their\napproaches:\n\n- Multinomial logistic regression considers all classes simultaneously during\n  optimization.\n- One-vs-rest logistic regression fits each class independently against all others.\n\nThese distinct strategies can lead to varying decision boundaries, especially in\ncomplex multi-class problems.\n\n## Hyperplanes Visualization\n\nWe also visualize the hyperplanes that correspond to the line when the probability\nestimate for a class is of 0.5.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def plot_hyperplanes(classifier, X, ax):\n    xmin, xmax = X[:, 0].min(), X[:, 0].max()\n    ymin, ymax = X[:, 1].min(), X[:, 1].max()\n    ax.set(xlim=(xmin, xmax), ylim=(ymin, ymax))\n\n    if isinstance(classifier, OneVsRestClassifier):\n        coef = np.concatenate([est.coef_ for est in classifier.estimators_])\n        intercept = np.concatenate([est.intercept_ for est in classifier.estimators_])\n    else:\n        coef = classifier.coef_\n        intercept = classifier.intercept_\n\n    for i in range(coef.shape[0]):\n        w = coef[i]\n        a = -w[0] / w[1]\n        xx = np.linspace(xmin, xmax)\n        yy = a * xx - (intercept[i]) / w[1]\n        ax.plot(xx, yy, \"--\", linewidth=3, label=f\"Class {i}\")\n\n    return ax.get_legend_handles_labels()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5), sharex=True, sharey=True)\n\nfor model, title, ax in [\n    (\n        logistic_regression_multinomial,\n        \"Multinomial Logistic Regression Hyperplanes\",\n        ax1,\n    ),\n    (logistic_regression_ovr, \"One-vs-Rest Logistic Regression Hyperplanes\", ax2),\n]:\n    hyperplane_handles, hyperplane_labels = plot_hyperplanes(model, X, ax)\n    scatter = ax.scatter(X[:, 0], X[:, 1], c=y, edgecolor=\"k\")\n    scatter_handles, scatter_labels = scatter.legend_elements()\n\n    all_handles = hyperplane_handles + scatter_handles\n    all_labels = hyperplane_labels + scatter_labels\n\n    ax.legend(all_handles, all_labels, title=\"Classes\")\n    ax.set_title(title)\n\nplt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "While the hyperplanes for classes 0 and 2 are quite similar between the two methods,\nwe observe that the hyperplane for class 1 is notably different. This difference stems\nfrom the fundamental approaches of one-vs-rest and multinomial logistic regression:\n\nFor one-vs-rest logistic regression:\n\n- Each hyperplane is determined independently by considering one class against all\n  others.\n- For class 1, the hyperplane represents the decision boundary that best separates\n  class 1 from the combined classes 0 and 2.\n- This binary approach can lead to simpler decision boundaries but may not capture\n  complex relationships between all classes simultaneously.\n- There is no possible interpretation of the conditional class probabilities.\n\nFor multinomial logistic regression:\n\n- All hyperplanes are determined simultaneously, considering the relationships between\n  all classes at once.\n- The loss minimized by the model is a proper scoring rule, which means that the model\n  is optimized to estimate the conditional class probabilities that are, therefore,\n  meaningful.\n- Each hyperplane represents the decision boundary where the probability of one class\n  becomes higher than the others, based on the overall probability distribution.\n- This approach can capture more nuanced relationships between classes, potentially\n  leading to more accurate classification in multi-class problems.\n\nThe difference in hyperplanes, especially for class 1, highlights how these methods\ncan produce different decision boundaries despite similar overall accuracy.\n\nIn practice, using multinomial logistic regression is recommended since it minimizes a\nwell-formulated loss function, leading to better-calibrated class probabilities and\nthus more interpretable results. When it comes to decision boundaries, one should\nformulate a utility function to transform the class probabilities into a meaningful\nquantity for the problem at hand. One-vs-rest allows for different decision boundaries\nbut does not allow for fine-grained control over the trade-off between the classes as\na utility function would.\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.21"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}