{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Visualizing cross-validation behavior in scikit-learn\n\nChoosing the right cross-validation object is a crucial part of fitting a\nmodel properly. There are many ways to split data into training and test\nsets in order to avoid model overfitting, to standardize the number of\ngroups in test sets, etc.\n\nThis example visualizes the behavior of several common scikit-learn objects\nfor comparison.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Authors: The scikit-learn developers\n# SPDX-License-Identifier: BSD-3-Clause\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom matplotlib.patches import Patch\n\nfrom sklearn.model_selection import (\n    GroupKFold,\n    GroupShuffleSplit,\n    KFold,\n    ShuffleSplit,\n    StratifiedGroupKFold,\n    StratifiedKFold,\n    StratifiedShuffleSplit,\n    TimeSeriesSplit,\n)\n\nrng = np.random.RandomState(1338)\ncmap_data = plt.cm.Paired\ncmap_cv = plt.cm.coolwarm\nn_splits = 4"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Visualize our data\n\nFirst, we must understand the structure of our data. It has 100 randomly\ngenerated input datapoints, 3 classes split unevenly across datapoints,\nand 10 \"groups\" split evenly across datapoints.\n\nAs we'll see, some cross-validation objects do specific things with\nlabeled data, others behave differently with grouped data, and others\ndo not use this information.\n\nTo begin, we'll visualize our data.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Generate the class/group data\nn_points = 100\nX = rng.randn(100, 10)\n\npercentiles_classes = [0.1, 0.3, 0.6]\ny = np.hstack([[ii] * int(100 * perc) for ii, perc in enumerate(percentiles_classes)])\n\n# Generate uneven groups\ngroup_prior = rng.dirichlet([2] * 10)\ngroups = np.repeat(np.arange(10), rng.multinomial(100, group_prior))\n\n\ndef visualize_groups(classes, groups, name):\n    # Visualize dataset groups\n    fig, ax = plt.subplots()\n    ax.scatter(\n        range(len(groups)),\n        [0.5] * len(groups),\n        c=groups,\n        marker=\"_\",\n        lw=50,\n        cmap=cmap_data,\n    )\n    ax.scatter(\n        range(len(groups)),\n        [3.5] * len(groups),\n        c=classes,\n        marker=\"_\",\n        lw=50,\n        cmap=cmap_data,\n    )\n    ax.set(\n        ylim=[-1, 5],\n        yticks=[0.5, 3.5],\n        yticklabels=[\"Data\\ngroup\", \"Data\\nclass\"],\n        xlabel=\"Sample index\",\n    )\n\n\nvisualize_groups(y, groups, \"no groups\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Define a function to visualize cross-validation behavior\n\nWe'll define a function that lets us visualize the behavior of each\ncross-validation object. We'll perform 4 splits of the data. On each\nsplit, we'll visualize the indices chosen for the training set\n(in blue) and the test set (in red).\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def plot_cv_indices(cv, X, y, group, ax, n_splits, lw=10):\n    \"\"\"Create a sample plot for indices of a cross-validation object.\"\"\"\n    use_groups = \"Group\" in type(cv).__name__\n    groups = group if use_groups else None\n    # Generate the training/testing visualizations for each CV split\n    for ii, (tr, tt) in enumerate(cv.split(X=X, y=y, groups=groups)):\n        # Fill in indices with the training/test groups\n        indices = np.array([np.nan] * len(X))\n        indices[tt] = 1\n        indices[tr] = 0\n\n        # Visualize the results\n        ax.scatter(\n            range(len(indices)),\n            [ii + 0.5] * len(indices),\n            c=indices,\n            marker=\"_\",\n            lw=lw,\n            cmap=cmap_cv,\n            vmin=-0.2,\n            vmax=1.2,\n        )\n\n    # Plot the data classes and groups at the end\n    ax.scatter(\n        range(len(X)), [ii + 1.5] * len(X), c=y, marker=\"_\", lw=lw, cmap=cmap_data\n    )\n\n    ax.scatter(\n        range(len(X)), [ii + 2.5] * len(X), c=group, marker=\"_\", lw=lw, cmap=cmap_data\n    )\n\n    # Formatting\n    yticklabels = list(range(n_splits)) + [\"class\", \"group\"]\n    ax.set(\n        yticks=np.arange(n_splits + 2) + 0.5,\n        yticklabels=yticklabels,\n        xlabel=\"Sample index\",\n        ylabel=\"CV iteration\",\n        ylim=[n_splits + 2.2, -0.2],\n        xlim=[0, 100],\n    )\n    ax.set_title(\"{}\".format(type(cv).__name__), fontsize=15)\n    return ax"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's see how it looks for the :class:`~sklearn.model_selection.KFold`\ncross-validation object:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "fig, ax = plt.subplots()\ncv = KFold(n_splits)\nplot_cv_indices(cv, X, y, groups, ax, n_splits)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As you can see, by default the KFold cross-validation iterator does not\ntake either datapoint class or group into consideration. We can change this\nby using either:\n\n- ``StratifiedKFold`` to preserve the percentage of samples for each class.\n- ``GroupKFold`` to ensure that the same group will not appear in two\n  different folds.\n- ``StratifiedGroupKFold`` to keep the constraint of ``GroupKFold`` while\n  attempting to return stratified folds.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "cvs = [StratifiedKFold, GroupKFold, StratifiedGroupKFold]\n\nfor cv in cvs:\n    fig, ax = plt.subplots(figsize=(6, 3))\n    plot_cv_indices(cv(n_splits), X, y, groups, ax, n_splits)\n    ax.legend(\n        [Patch(color=cmap_cv(0.8)), Patch(color=cmap_cv(0.02))],\n        [\"Testing set\", \"Training set\"],\n        loc=(1.02, 0.8),\n    )\n    # Make the legend fit\n    plt.tight_layout()\n    fig.subplots_adjust(right=0.7)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Next we'll visualize this behavior for a number of CV iterators.\n\n## Visualize cross-validation indices for many CV objects\n\nLet's visually compare the cross validation behavior for many\nscikit-learn cross-validation objects. Below we will loop through several\ncommon cross-validation objects, visualizing the behavior of each.\n\nNote how some use the group/class information while others do not.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "cvs = [\n    KFold,\n    GroupKFold,\n    ShuffleSplit,\n    StratifiedKFold,\n    StratifiedGroupKFold,\n    GroupShuffleSplit,\n    StratifiedShuffleSplit,\n    TimeSeriesSplit,\n]\n\n\nfor cv in cvs:\n    this_cv = cv(n_splits=n_splits)\n    fig, ax = plt.subplots(figsize=(6, 3))\n    plot_cv_indices(this_cv, X, y, groups, ax, n_splits)\n\n    ax.legend(\n        [Patch(color=cmap_cv(0.8)), Patch(color=cmap_cv(0.02))],\n        [\"Testing set\", \"Training set\"],\n        loc=(1.02, 0.8),\n    )\n    # Make the legend fit\n    plt.tight_layout()\n    fig.subplots_adjust(right=0.7)\nplt.show()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.21"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}