{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Multilabel classification using a classifier chain\nThis example shows how to use :class:`~sklearn.multioutput.ClassifierChain` to solve\na multilabel classification problem.\n\nThe most naive strategy to solve such a task is to independently train a binary\nclassifier on each label (i.e. each column of the target variable). At prediction\ntime, the ensemble of binary classifiers is used to assemble multitask prediction.\n\nThis strategy does not allow to model relationship between different tasks. The\n:class:`~sklearn.multioutput.ClassifierChain` is the meta-estimator (i.e. an estimator\ntaking an inner estimator) that implements a more advanced strategy. The ensemble\nof binary classifiers are used as a chain where the prediction of a classifier in the\nchain is used as a feature for training the next classifier on a new label. Therefore,\nthese additional features allow each chain to exploit correlations among labels.\n\nThe `Jaccard similarity <jaccard_similarity_score>` score for chain tends to be\ngreater than that of the set independent base models.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Authors: The scikit-learn developers\n# SPDX-License-Identifier: BSD-3-Clause"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Loading a dataset\nFor this example, we use the [yeast](https://www.openml.org/d/40597) dataset which contains\n2,417 datapoints each with 103 features and 14 possible labels. Each\ndata point has at least one label. As a baseline we first train a logistic\nregression classifier for each of the 14 labels. To evaluate the performance of\nthese classifiers we predict on a held-out test set and calculate the\nJaccard similarity for each sample.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\nimport numpy as np\n\nfrom sklearn.datasets import fetch_openml\nfrom sklearn.model_selection import train_test_split\n\n# Load a multi-label dataset from https://www.openml.org/d/40597\nX, Y = fetch_openml(\"yeast\", version=4, return_X_y=True)\nY = Y == \"TRUE\"\nX_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Fit models\nWe fit :class:`~sklearn.linear_model.LogisticRegression` wrapped by\n:class:`~sklearn.multiclass.OneVsRestClassifier` and ensemble of multiple\n:class:`~sklearn.multioutput.ClassifierChain`.\n\n### LogisticRegression wrapped by OneVsRestClassifier\nSince by default :class:`~sklearn.linear_model.LogisticRegression` can't\nhandle data with multiple targets, we need to use\n:class:`~sklearn.multiclass.OneVsRestClassifier`.\nAfter fitting the model we calculate Jaccard similarity.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import jaccard_score\nfrom sklearn.multiclass import OneVsRestClassifier\n\nbase_lr = LogisticRegression()\novr = OneVsRestClassifier(base_lr)\novr.fit(X_train, Y_train)\nY_pred_ovr = ovr.predict(X_test)\novr_jaccard_score = jaccard_score(Y_test, Y_pred_ovr, average=\"samples\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Chain of binary classifiers\nBecause the models in each chain are arranged randomly there is significant\nvariation in performance among the chains. Presumably there is an optimal\nordering of the classes in a chain that will yield the best performance.\nHowever, we do not know that ordering a priori. Instead, we can build a\nvoting ensemble of classifier chains by averaging the binary predictions of\nthe chains and apply a threshold of 0.5. The Jaccard similarity score of the\nensemble is greater than that of the independent models and tends to exceed\nthe score of each chain in the ensemble (although this is not guaranteed\nwith randomly ordered chains).\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.multioutput import ClassifierChain\n\nchains = [ClassifierChain(base_lr, order=\"random\", random_state=i) for i in range(10)]\nfor chain in chains:\n    chain.fit(X_train, Y_train)\n\nY_pred_chains = np.array([chain.predict_proba(X_test) for chain in chains])\nchain_jaccard_scores = [\n    jaccard_score(Y_test, Y_pred_chain >= 0.5, average=\"samples\")\n    for Y_pred_chain in Y_pred_chains\n]\n\nY_pred_ensemble = Y_pred_chains.mean(axis=0)\nensemble_jaccard_score = jaccard_score(\n    Y_test, Y_pred_ensemble >= 0.5, average=\"samples\"\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Plot results\nPlot the Jaccard similarity scores for the independent model, each of the\nchains, and the ensemble (note that the vertical axis on this plot does\nnot begin at 0).\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "model_scores = [ovr_jaccard_score] + chain_jaccard_scores + [ensemble_jaccard_score]\n\nmodel_names = (\n    \"Independent\",\n    \"Chain 1\",\n    \"Chain 2\",\n    \"Chain 3\",\n    \"Chain 4\",\n    \"Chain 5\",\n    \"Chain 6\",\n    \"Chain 7\",\n    \"Chain 8\",\n    \"Chain 9\",\n    \"Chain 10\",\n    \"Ensemble\",\n)\n\nx_pos = np.arange(len(model_names))\n\nfig, ax = plt.subplots(figsize=(7, 4))\nax.grid(True)\nax.set_title(\"Classifier Chain Ensemble Performance Comparison\")\nax.set_xticks(x_pos)\nax.set_xticklabels(model_names, rotation=\"vertical\")\nax.set_ylabel(\"Jaccard Similarity Score\")\nax.set_ylim([min(model_scores) * 0.9, max(model_scores) * 1.1])\ncolors = [\"r\"] + [\"b\"] * len(chain_jaccard_scores) + [\"g\"]\nax.bar(x_pos, model_scores, alpha=0.5, color=colors)\nplt.tight_layout()\nplt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Results interpretation\nThere are three main takeaways from this plot:\n\n- Independent model wrapped by :class:`~sklearn.multiclass.OneVsRestClassifier`\n  performs worse than the ensemble of classifier chains and some of individual chains.\n  This is caused by the fact that the logistic regression doesn't model relationship\n  between the labels.\n- :class:`~sklearn.multioutput.ClassifierChain` takes advantage of correlation\n  among labels but due to random nature of labels ordering, it could yield worse\n  result than an independent model.\n- An ensemble of chains performs better because it not only captures relationship\n  between labels but also does not make strong assumptions about their correct order.\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.21"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}