{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n# Analysis of the convergence of penalized logistic regression models\n\n.. currentmodule:: sklearn.callback\n\n\nThe purpose of this example is three-fold:\n\n1. Demonstrate registering a :class:`~ScoringMonitor` on the logistic\n regression step of a pipeline nested inside\n :class:`~sklearn.model_selection.GridSearchCV`.\n\n2. Show how to plot the metric values collected at each iteration of each fit\n of the logistic regression model during the grid search and analyze the\n convergence of the model for each hyperparameter combination.\n\n3. Show how the monitoring of diverse scoring metrics can inform us about the\n quality of the model and the trade-off between refinement and calibration.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Authors: The scikit-learn developers\n# SPDX-License-Identifier: BSD-3-Clause" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup\n\nLet's first define the pipeline and the grid search. Here we register a\n:class:`~ScoringMonitor` callback on the logistic regression model to monitor\nthe scores at each iteration of the L-BFGS solver.\n\nWe reuse the same scoring metrics for the grid search itself and use the D\u00b2\nlog-loss as the primary metric to select the best hyperparameter combination.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\n\nfrom sklearn.callback import ProgressBar, ScoringMonitor\nfrom sklearn.datasets import make_classification\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nX, y = make_classification(\n n_samples=1000, n_features=100, n_classes=10, n_informative=30, random_state=42\n)\n\nscoring_metrics = [\"d2_log_loss_score\", \"accuracy\", \"average_precision\"]\nscoring_monitor = ScoringMonitor(scoring=scoring_metrics)\nmodel = make_pipeline(\n StandardScaler(),\n LogisticRegression(solver=\"lbfgs\", max_iter=1000).set_callbacks(scoring_monitor),\n)\n\nparam_grid = {\n \"standardscaler__with_std\": [True, False],\n \"logisticregression__C\": np.geomspace(0.01, 100, 3),\n}\n\ngrid_search = GridSearchCV(\n model,\n param_grid,\n cv=5,\n scoring=scoring_metrics,\n n_jobs=2,\n error_score=\"raise\",\n refit=scoring_metrics[0],\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's fit the grid search with the auto-propagating progress bar callback.\nFeel free to set max_propagation_depth=3 in the ProgressBar constructor to\nget a more detailed output by displaying the progress bars for the pipeline,\nthe standard scaler and the logistic regression.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "grid_search.set_callbacks(ProgressBar()).fit(X, y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We use a grid search with 3 values for the regularization parameter ``C`` and\n2 values for the standardization of the features resulting in 6 parameter\ncombinations.\n\nSince we use 5-fold cross-validation (``cv=5``), we will have 5 fits of the\nlogistic regression model for each parameter combination resulting in 30 fits\nas subtasks of the \"search\" :term:`fit task`.\n\nIn addition, the grid search performs a final refit on the full dataset with\nthe best hyperparameter combination found during the grid search. This is\nvisible as the \"refit-with-best-params\" task in the output above.\n\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Consolidation of the grid search results\n\nLet's look at the results of the grid search.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "cv_results = pd.DataFrame(grid_search.cv_results_)\ncv_results.sort_values(by=\"rank_test_d2_log_loss_score\", ascending=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We observe that the best models use regularization (small ``C``). Feature\nstandardization does not seem to matter much but helps reduce the fit times.\nWe notice that many models have similar accuracy scores but different D\u00b2\nlog-loss scores and average precision scores. D\u00b2 log-loss and average\nprecision are more sensitive to the quality of the model than accuracy\nbecause they evaluate the entire probability distribution of the predictions\nrather than just the match of the top predicted class with the true class.\n\nLet's now refine this analysis by looking at the same metrics computed on the\ntraining set at each iteration of the L-BFGS solver and for each parameter\ncombination. Note that these are training-set scores recorded during L-BFGS\niterations, not the held-out CV scores from ``cv_results_``.\n\nThese values are stored in the `scoring_monitor` callback object:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "all_tasks_log = scoring_monitor.get_logs().data_as_pandas\nall_tasks_log" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's enrich this log with the candidate parameters and the split index so we\ncan plot the scores for each parameter combination for a particular CV split\nof interest.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "candidate_params = pd.DataFrame(grid_search.cv_results_[\"params\"]).add_prefix(\"param_\")\n\nn_splits = grid_search.n_splits_\nlbfgs_log = all_tasks_log.query(\n \"estimator_name == 'LogisticRegression' and task_name == 'lbfgs-iter'\"\n).copy()\n# Index 2 in ``task_id_path`` is the ``candidate-split-evaluation`` task id.\n# Future versions of scikit-learn will provide a more convenient way to\n# retrieve this task id.\nlbfgs_log[\"eval_task_id\"] = lbfgs_log[\"task_id_path\"].map(lambda path: path[2])\nlbfgs_log[\"candidate_idx\"] = lbfgs_log[\"eval_task_id\"] // n_splits\nlbfgs_log[\"split_idx\"] = lbfgs_log[\"eval_task_id\"] % n_splits\nlbfgs_log = lbfgs_log.query(\"split_idx == 0\").join(candidate_params, on=\"candidate_idx\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Exclude the final refit on the full dataset (``parent_task_id_path``\nstarts with ``(0, 1)`` instead of ``(0, 0)`` for cross-validation fits). Note\nthat it is possible to call `scoring_monitor.get_logs(include_lineage=True)`\nto retrieve the task name of the ancestor tasks if needed.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "cv_lbfgs_log = lbfgs_log[\n lbfgs_log[\"parent_task_id_path\"].map(lambda path: path[1]) == 0\n]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We define labels for plotting purposes and plot each metric separately.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "cv_lbfgs_log[\"param_label\"] = cv_lbfgs_log.apply(\n lambda row: (\n f\"with_std={row['param_standardscaler__with_std']}, \"\n f\"C={row['param_logisticregression__C']:.2g}\"\n ),\n axis=1,\n)\n\nmetrics = {\n \"d2_log_loss_score\": \"D\u00b2 log-loss (train)\",\n \"accuracy\": \"Accuracy (train)\",\n \"average_precision\": \"Average precision (train)\",\n}\n_, axes = plt.subplots(\n len(metrics),\n 1,\n figsize=(8, 2.5 * len(metrics)),\n sharex=True,\n constrained_layout=True,\n)\nfor idx, (metric, ylabel) in enumerate(metrics.items()):\n ax = axes[idx]\n for param_label, group in cv_lbfgs_log.groupby(\"param_label\", sort=False):\n ax.plot(group[\"task_id\"], group[metric], label=param_label)\n ax.set_ylabel(ylabel)\n if idx == 0:\n ax.set_title(\"CV split 0\")\n ax.legend(title=\"Hyperparameters\", fontsize=\"small\")\n\n_ = axes[-1].set_xlabel(\"L-BFGS iteration\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Analysis of the convergence of the logistic regression models\n\n### D\u00b2 log-loss convergence\n\nThe D\u00b2 log-loss scores generally improve monotonically for all models. This\nis expected because the logistic regression model is fitted by minimizing the\n(regularized) log-loss computed on the training set.\n\n### Accuracy fluctuations\n\nThe accuracy score improves with the number of iterations, albeit with some\nlocal fluctuations. This is expected because accuracy is discontinuous and\nnot directly optimized by the model. Instead the model minimizes the log-loss\nwhich is a smooth surrogate for the zero-one loss (and thus related to, but\nnot directly optimized by, accuracy).\n\n### Regularization and scaling\n\nWe also observe that the least regularized models (larger ``C`` values) tend\nto reach higher D\u00b2 log-loss scores, and models trained on scaled features\nconverge in much fewer iterations.\n\nFurthermore, models trained with high regularization (lower ``C`` values)\nconverge to a final D\u00b2 log-loss value that depends on the regularization\nstrength while this is not the case for models trained with low\nregularization: there is a strong coupling between the optimal regularization\nstrength and the feature scaling.\n\n### Average precision vs log-loss, refinement vs calibration\n\nFinally, we observe that the average precision value measured on the training\nset can improve quickly in the first iterations and then worsen even though\nthe D\u00b2 log-loss value continues to improve on the same training data. This is\nespecially noticeable for models trained with low regularization and feature\nstandardization. This counter-intuitive behavior can be explained as follows.\nFirst recall that average precision is a pure ranking metric that measures\nthe ability of the model to output predicted probabilities that rank the\nsamples of a given class higher than the samples of the other classes, but\ndoes not take into account the calibration of the predicted probabilities. In\nother words, average precision only evaluates if the predicted probabilities\nare well ordered relatively to one another but is insensitive to a rank\npreserving transformation of their absolute values. The log-loss, on the\nother hand, is a strictly proper scoring rule that accounts for both the\nrefinement (ranking power) of the model and the calibration of the predicted\nprobabilities.\n\nTherefore, the average precision curves of the low-regularized models trained\non scaled features suggest that the first iterations mostly improve\nrefinement of the models temporarily leaving calibration behind. In later\niterations, the log-loss score continues to improve but average precision\nvalues worsen, which suggests that the logistic regression model\nprogressively trades off refinement for calibration over the course of the\nfinal iterations. This phenomenon has been studied in [1]_.\n\nIt would be interesting to see if this also happens when evaluating the model\non a validation set so we could implement early stopping on average precision\nto explicitly select a model with high refinement on a validation set. This\nis not yet possible at the time of writing. Giving callbacks access to the\nvalidation set is planned for a future version of scikit-learn. Note that the\ncallbacks API is still experimental and may change without the usual\ndeprecation cycle.\n\n## References\n.. [1] :doi:`Berta, E., Holzm\u00fcller, D., Jordan, M. I., and Bach, F.\n \"Rethinking Early Stopping: Refine, Then Calibrate\" (2025).\n <10.48550/arXiv.2501.19195>`\n\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.6" } }, "nbformat": 4, "nbformat_minor": 0 }