{ "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "---\n", "title: \"Chapter 32: Sklearn pipelines and hyperparameter search\"\n", "---" ] }, { "cell_type": "markdown", "id": "1", "metadata": {}, "source": [ "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/sambaiga/ds-mlops-path/blob/main/tutorials/04-ml-intro/32-sklearn-pipeline.ipynb) [![Download Notebook](https://img.shields.io/badge/Download-Notebook-blue.svg?logo=jupyter&logoColor=white)](https://raw.githubusercontent.com/sambaiga/ds-mlops-path/main/tutorials/04-ml-intro/32-sklearn-pipeline.ipynb)" ] }, { "cell_type": "markdown", "id": "2", "metadata": {}, "source": [ "Six numeric columns and an F1 of 0.55, that was Chapter 31's stopping point. Add six categorical columns, hotel, meal plan, market segment, distribution channel, customer type, deposit type, change nothing else, and that same LogisticRegression jumps to 0.70. Almost the same gap Chapter 31 opened with between a useless model and a working one, closed by six columns Chapter 31 never touched, exactly the room type and season signal it named as missing.\n", "\n", "That jump is also where the risk gets worse. Chapter 31 counted six separate fit calls, scaler and model, twice over, as six places a leak could slip in. A categorical column needs its own encoder, fit its own way, on training data only, same as a scaler. Fourteen columns across two different treatments is no longer six fit calls to keep straight, it is closer to a dozen, and remembering the right order by hand stops being realistic." ] }, { "cell_type": "markdown", "id": "3", "metadata": {}, "source": [ "A `Pipeline` is scikit-learn's answer: every step, however many, wrapped into one object with one `fit` and one `predict`. Fit it and it fits every stage in sequence using only training data. Call `predict` and it applies every transformation in the same order, leakage no longer possible because there's no manual step left to get wrong. This chapter builds one for the full fourteen-column feature set, tunes it three different ways, and closes the ML landscape part: [Chapter 30](30-ml-workflow.qmd)'s workflow, [Chapter 31](31-sklearn-core.ipynb)'s two interfaces, combined into a single deployable object." ] }, { "cell_type": "markdown", "id": "4", "metadata": {}, "source": [ "::: {.callout-note collapse=\"true\" icon=false}\n", "## Learning objectives\n", "\n", "By the end of this chapter you will be able to:\n", "\n", "| # | Skill | Covered in |\n", "|---|---|---|\n", "| 1 | Wrap preprocessing and a model into a single Pipeline object, and explain what that buys over separate fit calls | Sec. 2 |\n", "| 2 | Route numeric and categorical columns to different treatments with ColumnTransformer | Sec. 3 |\n", "| 3 | Keep a Pipeline leakage-proof inside cross_val_score, GridSearchCV, and RandomizedSearchCV | Sec. 4 |\n", "| 4 | Compare model families on default hyperparameters before spending a tuning budget on any one of them | Sec. 5 |\n", "| 5 | Search hyperparameters with GridSearchCV, RandomizedSearchCV, and Optuna's TPE sampler | Sec. 6, 7 |\n", "| 6 | Read feature importances from a fitted pipeline, and recognise when a surprising one needs investigating rather than explaining away | Sec. 8 |\n", ":::" ] }, { "cell_type": "markdown", "id": "5", "metadata": {}, "source": [ "## 1. Growing the feature set\n", "\n", "Same loader as [Chapter 31](31-sklearn-core.ipynb), same dataset. What changes is how much of it gets used." ] }, { "cell_type": "code", "execution_count": null, "id": "6", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "from typing import Any\n", "\n", "from great_tables import GT, md as gt_md\n", "import joblib\n", "from lets_plot import (\n", " LetsPlot,\n", " aes,\n", " as_discrete,\n", " geom_bar,\n", " geom_line,\n", " geom_point,\n", " ggplot,\n", " labs,\n", " scale_fill_manual,\n", ")\n", "from loguru import logger\n", "import numpy as np\n", "import optuna\n", "import pandas as pd\n", "from scipy.stats import loguniform\n", "from sklearn.compose import ColumnTransformer\n", "from sklearn.dummy import DummyClassifier\n", "from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier\n", "from sklearn.impute import SimpleImputer\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.metrics import classification_report, f1_score\n", "from sklearn.model_selection import (\n", " GridSearchCV,\n", " RandomizedSearchCV,\n", " StratifiedKFold,\n", " cross_val_score,\n", " train_test_split,\n", ")\n", "from sklearn.pipeline import Pipeline, make_pipeline\n", "from sklearn.preprocessing import OneHotEncoder, StandardScaler\n", "\n", "from ark.plot.gt_style import metrics_report, themed_gt\n", "from ark.plot.theme import modern_theme\n", "from ark.plot.tokens import DANGER, INFO, SUCCESS\n", "\n", "optuna.logging.set_verbosity(optuna.logging.WARNING)\n", "LetsPlot.setup_html()" ] }, { "cell_type": "code", "execution_count": null, "id": "7", "metadata": {}, "outputs": [], "source": [ "DATA_URL: str = \"https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-02-11/hotels.csv\"\n", "DATA_PATH: Path = Path(\"data/hotel_bookings.csv\")\n", "MODEL_DIR: Path = Path(\"models\")\n", "RANDOM_STATE: int = 42\n", "\n", "\n", "def load_hotel_data(path: Path = DATA_PATH, url: str = DATA_URL) -> pd.DataFrame:\n", " \"\"\"Load hotel bookings, downloading from url on first call.\"\"\"\n", " if not path.exists():\n", " path.parent.mkdir(parents=True, exist_ok=True)\n", " raw = pd.read_csv(url)\n", " raw.to_csv(path, index=False)\n", " logger.success(f\"Downloaded {len(raw):,} rows to {path}\")\n", " else:\n", " raw = pd.read_csv(path)\n", " logger.info(f\"Loaded {len(raw):,} rows from cache: {path}\")\n", " df = raw[(raw[\"adr\"] > 0) & (raw[\"adr\"] <= 1000)].reset_index(drop=True)\n", " logger.info(f\"After cleaning: {len(df):,} rows\")\n", " return df\n", "\n", "\n", "df: pd.DataFrame = load_hotel_data()\n", "logger.info(f\"Shape: {df.shape[0]:,} rows x {df.shape[1]} columns\")" ] }, { "cell_type": "markdown", "id": "8", "metadata": {}, "source": [ "Eight numeric columns this time, [Chapter 31](31-sklearn-core.ipynb)'s original six plus two more, and six categorical ones that a plain `StandardScaler` has no way to handle: a hotel name or a deposit type isn't a number, and forcing one into a numeric column would invent an ordering between categories that was never there." ] }, { "cell_type": "code", "execution_count": null, "id": "9", "metadata": {}, "outputs": [], "source": [ "NUMERICAL_FEATURES: list[str] = [\n", " \"lead_time\",\n", " \"stays_in_weekend_nights\",\n", " \"stays_in_week_nights\",\n", " \"adults\",\n", " \"previous_cancellations\",\n", " \"booking_changes\",\n", " \"total_of_special_requests\",\n", " \"days_in_waiting_list\",\n", "]\n", "CATEGORICAL_FEATURES: list[str] = [\n", " \"hotel\",\n", " \"meal\",\n", " \"market_segment\",\n", " \"distribution_channel\",\n", " \"customer_type\",\n", " \"deposit_type\",\n", "]\n", "ALL_FEATURES: list[str] = NUMERICAL_FEATURES + CATEGORICAL_FEATURES\n", "TARGET_CLF: str = \"is_canceled\"" ] }, { "cell_type": "markdown", "id": "10", "metadata": {}, "source": [ "Cardinality, how many distinct values each categorical column has, matters before any encoding happens: it's exactly how many new binary columns `OneHotEncoder` is about to create." ] }, { "cell_type": "code", "execution_count": null, "id": "11", "metadata": {}, "outputs": [], "source": [ "cat_summary: list[dict[str, Any]] = [\n", " {\"feature\": c, \"n_unique\": df[c].nunique(), \"values\": str(sorted(df[c].unique().tolist())[:6])}\n", " for c in CATEGORICAL_FEATURES\n", "]\n", "themed_gt(GT(pd.DataFrame(cat_summary)), n_rows=len(CATEGORICAL_FEATURES)).tab_header(\n", " title=gt_md(\"**Categorical features**\"),\n", " subtitle=\"Cardinality determines how many one-hot columns each one becomes\",\n", ")" ] }, { "cell_type": "markdown", "id": "12", "metadata": {}, "source": [ "Twenty-eight possible one-hot columns from six features, `hotel` contributing 2 and `market_segment` contributing 8. Split before any of that encoding happens, same rule as [Chapter 31](31-sklearn-core.ipynb), now protecting an encoder's learned categories instead of just a scaler's mean:" ] }, { "cell_type": "code", "execution_count": null, "id": "13", "metadata": {}, "outputs": [], "source": [ "X: pd.DataFrame = df[ALL_FEATURES].copy()\n", "y: np.ndarray = df[TARGET_CLF].to_numpy(dtype=int)\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=RANDOM_STATE, stratify=y)\n", "logger.info(f\"Train: {X_train.shape} Test: {X_test.shape}\")\n", "logger.info(f\"Train cancel rate: {y_train.mean():.2%} Test cancel rate: {y_test.mean():.2%}\")" ] }, { "cell_type": "markdown", "id": "14", "metadata": {}, "source": [ "## 2. What a Pipeline actually buys you\n", "\n", "Rebuild [Chapter 31](31-sklearn-core.ipynb)'s six-numeric-feature model first, this time as a `Pipeline` instead of two separate objects called in sequence. Nothing about the model changes, only how the scaler and the classifier are held together." ] }, { "cell_type": "code", "execution_count": null, "id": "15", "metadata": {}, "outputs": [], "source": [ "NUM6: list[str] = NUMERICAL_FEATURES[:6]\n", "pipe_simple: Pipeline = Pipeline(\n", " [\n", " (\"scaler\", StandardScaler()),\n", " (\"model\", LogisticRegression(class_weight=\"balanced\", max_iter=1000, random_state=RANDOM_STATE)),\n", " ]\n", ")\n", "X_train_num: pd.DataFrame = X_train[NUM6]\n", "X_test_num: pd.DataFrame = X_test[NUM6]\n", "pipe_simple.fit(X_train_num, y_train)\n", "logger.success(\"Simple pipeline fitted\")" ] }, { "cell_type": "markdown", "id": "16", "metadata": {}, "source": [ "`named_steps` gives dictionary access into a fitted Pipeline, useful for checking that a stage learned what you expect without breaking the single-object abstraction:" ] }, { "cell_type": "code", "execution_count": null, "id": "17", "metadata": {}, "outputs": [], "source": [ "fitted_scaler: StandardScaler = pipe_simple.named_steps[\"scaler\"]\n", "logger.info(f\"Scaler means (first 4): {fitted_scaler.mean_[:4].round(2)}\")\n", "logger.info(f\"Pipeline steps: {list(pipe_simple.named_steps.keys())}\")" ] }, { "cell_type": "markdown", "id": "18", "metadata": {}, "source": [ "Check whether it can actually predict:" ] }, { "cell_type": "code", "execution_count": null, "id": "19", "metadata": {}, "outputs": [], "source": [ "y_pred_simple: np.ndarray = pipe_simple.predict(X_test_num)\n", "f1_simple: float = f1_score(y_test, y_pred_simple)\n", "logger.info(f\"Simple pipeline F1 (six numeric features): {f1_simple:.3f}\")" ] }, { "cell_type": "markdown", "id": "20", "metadata": {}, "source": [ "0.549, exactly [Chapter 31](31-sklearn-core.ipynb)'s number. Same model, same features, same split, wrapped differently. That's the guarantee worth trusting before anything gets more complicated: a `Pipeline` doesn't change what a model learns, it changes how safely you can call `fit` on it." ] }, { "cell_type": "markdown", "id": "21", "metadata": {}, "source": [ "
\n", " Key Concept: a Pipeline is itself a Predictor

\n", "Once assembled, a Pipeline exposes exactly the shape Chapter 31 named: fit(X, y) and predict(X), same as any single model. You can hand it to cross_val_score, GridSearchCV, or RandomizedSearchCV exactly as you would a bare classifier, and every one of those tools will re-fit every internal stage correctly on each fold's training data, never on its validation data. The rest of this chapter leans on that guarantee constantly, without restating it.\n", "
" ] }, { "cell_type": "markdown", "id": "22", "metadata": {}, "source": [ "
\n", " Activity 1: RobustScaler inside the Pipeline

\n", "Replace StandardScaler with RobustScaler inside pipe_simple and refit. Does it change F1 on this particular feature set, and does that match what you'd expect given how many outliers Chapter 31's feature ranges showed?\n", "
from sklearn.preprocessing import RobustScaler\n",
    "# your code here
\n", "
" ] }, { "cell_type": "markdown", "id": "23", "metadata": {}, "source": [ "## 3. Routing columns with ColumnTransformer\n", "\n", "A `StandardScaler` alone has no path for `deposit_type` or `hotel`. `ColumnTransformer` is what routes different columns to different treatments and concatenates the results back into one matrix: numeric columns to a scaler, categorical columns to an encoder, side by side." ] }, { "cell_type": "markdown", "id": "24", "metadata": {}, "source": [ "Here is the shape of what the next few cells build, before any code: fourteen raw columns splitting into two routes, then rejoining as one matrix.\n", "\n", "![Fourteen raw columns split into a numeric branch (SimpleImputer, StandardScaler, 8 columns out) and a categorical branch (SimpleImputer, OneHotEncoder, 27 columns out), both merging into one 35-column output matrix.](figs/column-transformer-routing.svg){fig-alt=\"Diagram showing 14 input columns branching into a green numeric path and a blue categorical path, converging into a dark output box labeled one matrix, 35 columns.\"}" ] }, { "cell_type": "code", "execution_count": null, "id": "25", "metadata": {}, "outputs": [], "source": [ "numeric_transformer: Pipeline = make_pipeline(SimpleImputer(strategy=\"median\"), StandardScaler())\n", "categorical_transformer: Pipeline = make_pipeline(\n", " SimpleImputer(strategy=\"most_frequent\"),\n", " OneHotEncoder(handle_unknown=\"ignore\", sparse_output=False),\n", ")" ] }, { "cell_type": "markdown", "id": "26", "metadata": {}, "source": [ "This dataset has no missing values in either feature group, checked directly, but `SimpleImputer` costs nothing to include and means a future data refresh with a few blank rows doesn't crash the pipeline the day it happens instead of the day someone tests for it. `handle_unknown=\"ignore\"` on the encoder does similar defensive work at inference time: a category never seen during training becomes an all-zero row instead of raising an error the moment production data drifts even slightly from training data." ] }, { "cell_type": "code", "execution_count": null, "id": "27", "metadata": {}, "outputs": [], "source": [ "preprocessor: ColumnTransformer = ColumnTransformer(\n", " transformers=[\n", " (\"num\", numeric_transformer, NUMERICAL_FEATURES),\n", " (\"cat\", categorical_transformer, CATEGORICAL_FEATURES),\n", " ],\n", " remainder=\"drop\",\n", ")\n", "X_prep: np.ndarray = preprocessor.fit_transform(X_train)\n", "logger.info(f\"Input shape: {X_train.shape}\")\n", "logger.info(f\"Output shape: {X_prep.shape}\")" ] }, { "cell_type": "markdown", "id": "28", "metadata": {}, "source": [ "Fourteen input columns became 35: the 8 numeric ones pass through unchanged in count, the 6 categorical ones expand to 27 one-hot columns between them. `get_feature_names_out()` recovers what each of those 27 actually means, since the raw column index no longer lines up with anything nameable on its own:" ] }, { "cell_type": "code", "execution_count": null, "id": "29", "metadata": {}, "outputs": [], "source": [ "feature_names: list[str] = preprocessor.get_feature_names_out().tolist()\n", "logger.info(f\"Total features after preprocessing: {len(feature_names)}\")\n", "logger.info(f\"First 4 names: {feature_names[:4]}\")\n", "logger.info(f\"Last 4 names: {feature_names[-4:]}\")" ] }, { "cell_type": "markdown", "id": "30", "metadata": {}, "source": [ "Chain that preprocessor to a model exactly the way Section 2's simpler pipeline chained a scaler, and every one of the fourteen raw columns, not just six, is now one `fit` call away from a prediction:" ] }, { "cell_type": "code", "execution_count": null, "id": "31", "metadata": {}, "outputs": [], "source": [ "pipe_full: Pipeline = Pipeline(\n", " [\n", " (\"preprocessor\", preprocessor),\n", " (\"model\", LogisticRegression(class_weight=\"balanced\", max_iter=1000, random_state=RANDOM_STATE)),\n", " ]\n", ")\n", "pipe_full.fit(X_train, y_train)\n", "y_pred_full: np.ndarray = pipe_full.predict(X_test)\n", "f1_full: float = f1_score(y_test, y_pred_full)\n", "logger.info(f\"Simple pipeline (6 numeric features): F1 = {f1_simple:.3f}\")\n", "logger.info(f\"Full pipeline (8 numeric + 6 categorical): F1 = {f1_full:.3f}\")" ] }, { "cell_type": "markdown", "id": "32", "metadata": {}, "source": [ "0.70, the jump this chapter opened with, now reproduced with the real pipeline instead of asserted in the hook. `deposit_type`, `market_segment`, and the four columns beside them were carrying real signal about cancellation risk that no numeric column could have proxied." ] }, { "cell_type": "markdown", "id": "33", "metadata": {}, "source": [ "
\n", " Activity 2: add arrival_date_month

\n", "Add arrival_date_month to CATEGORICAL_FEATURES, rebuild the preprocessor and pipe_full, and refit. Does the booking's arrival month move F1 further, and does the size of that move match how much you'd expect season to matter for a hotel?\n", "
" ] }, { "cell_type": "markdown", "id": "34", "metadata": {}, "source": [ "## 4. Keeping that guarantee inside cross-validation\n", "\n", "A single 80/20 split gave one F1 number. [Chapter 31](31-sklearn-core.ipynb) cross-validated for exactly this reason, a steadier estimate across five folds instead of trusting one split. Do that here by calling `preprocessor.fit_transform` once up front and passing the transformed array to `cross_val_score`, and the guarantee Section 2's Key Concept named quietly breaks: every fold's \"held-out\" score is computed against an encoder that already saw that fold's own categories during its one shared fit." ] }, { "cell_type": "markdown", "id": "35", "metadata": {}, "source": [ "
\n", " Common Mistake: fit_transform before cross_val_score

\n", "\n", "
\n",
    "# Leaky: the preprocessor is fitted once, on every row, before any fold is held out\n",
    "X_scaled = preprocessor.fit_transform(X_train)\n",
    "cross_val_score(model, X_scaled, y_train, cv=StratifiedKFold(5))\n",
    "\n",
    "# Correct: the Pipeline is what gets cross-validated, not its output\n",
    "pipe = Pipeline([(\"preprocessor\", preprocessor), (\"model\", model)])\n",
    "cross_val_score(pipe, X_train, y_train, cv=StratifiedKFold(5))\n",
    "
\n", "\n", "The leaky version's encoder has already learned every category in the full training set, including the ones sitting in whatever fold is about to be held out. The score that comes back is optimistic in a way that won't survive contact with real unseen data, and nothing about the code will tell you that happened.\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "36", "metadata": {}, "outputs": [], "source": [ "skf: StratifiedKFold = StratifiedKFold(n_splits=5, shuffle=True, random_state=RANDOM_STATE)\n", "cv_scores: np.ndarray = cross_val_score(pipe_full, X_train, y_train, cv=skf, scoring=\"f1\", n_jobs=-1)\n", "logger.info(f\"CV F1 per fold: {cv_scores.round(3)}\")\n", "logger.info(f\"Mean +/- std: {cv_scores.mean():.3f} +/- {cv_scores.std():.3f}\")" ] }, { "cell_type": "markdown", "id": "37", "metadata": {}, "source": [ "0.699 ± 0.004, right where the single split put it. Passing `pipe_full` itself, not its output, is what makes that number trustworthy: the preprocessor refits inside every fold, on that fold's training rows only." ] }, { "cell_type": "markdown", "id": "38", "metadata": {}, "source": [ "## 5. Which model family fits this data\n", "\n", "Before tuning any single model's hyperparameters, compare a few families with their defaults. The best default is a signal about which family's assumptions actually suit this data, and there's little point spending a tuning budget on a family that can't compete even before tuning starts." ] }, { "cell_type": "code", "execution_count": null, "id": "39", "metadata": {}, "outputs": [], "source": [ "MODEL_CANDIDATES: dict[str, Any] = {\n", " \"LogisticRegression\": LogisticRegression(class_weight=\"balanced\", max_iter=1000, random_state=RANDOM_STATE),\n", " \"RandomForest\": RandomForestClassifier(\n", " n_estimators=50, class_weight=\"balanced_subsample\", random_state=RANDOM_STATE, n_jobs=1\n", " ),\n", " \"HistGradientBoosting\": HistGradientBoostingClassifier(max_iter=50, random_state=RANDOM_STATE),\n", "}\n", "\n", "comparison_results: list[dict[str, Any]] = []\n", "for name, model in MODEL_CANDIDATES.items():\n", " candidate_pipe = Pipeline([(\"preprocessor\", preprocessor), (\"model\", model)])\n", " scores: np.ndarray = cross_val_score(\n", " candidate_pipe, X_train, y_train, cv=StratifiedKFold(3, shuffle=True, random_state=RANDOM_STATE), scoring=\"f1\"\n", " )\n", " comparison_results.append({\"Model\": name, \"F1_mean\": float(scores.mean()), \"F1_std\": float(scores.std())})\n", " logger.info(f\"{name:<22} F1 = {scores.mean():.3f} +/- {scores.std():.3f}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "40", "metadata": {}, "outputs": [], "source": [ "comparison_df: pd.DataFrame = pd.DataFrame(comparison_results)\n", "metrics_report(\n", " comparison_df,\n", " metrics=[\"F1_mean\", \"F1_std\"],\n", " maximize_cols=[\"F1_mean\"],\n", " minimize_cols=[\"F1_std\"],\n", " title=\"Model family comparison, default hyperparameters\",\n", " subtitle=\"3-fold stratified CV on training set | Hotel Booking Demand\",\n", ")" ] }, { "cell_type": "markdown", "id": "41", "metadata": {}, "source": [ "RandomForest wins outright, 0.739 against LogisticRegression's 0.699, with HistGradientBoosting in between at 0.723, using nothing but each family's defaults. If the only goal from here were the highest F1 achievable, tuning would start with RandomForest and this chapter would look different.\n", "\n", "It doesn't, on purpose. LogisticRegression gets the rest of this chapter's tuning budget because a coefficient anyone on the revenue team can read (Section 8 puts a number on exactly that) is worth more to a decision about who to call before a likely cancellation than an extra four points of F1 hidden inside fifty trees, and because `C` and `penalty` are also the clearest two knobs for showing what tuning actually searches over. That's a real trade a team makes, not a limitation of this notebook: accuracy against explainability, decided here in favour of the model a human can still argue with." ] }, { "cell_type": "markdown", "id": "42", "metadata": {}, "source": [ "## 6. Searching for good hyperparameters\n", "\n", "Inside a `Pipeline`, a hyperparameter belongs to a named step, so it's addressed as `step__param`, double underscore. Nested pipelines, like the categorical branch inside this `ColumnTransformer`, just add another layer of `__`." ] }, { "cell_type": "code", "execution_count": null, "id": "43", "metadata": {}, "outputs": [], "source": [ "example_params: dict[str, Any] = pipe_full.get_params()\n", "model_keys: list[str] = [k for k in example_params if \"model__\" in k]\n", "logger.info(f\"LogisticRegression parameters reachable via the pipeline: {model_keys}\")" ] }, { "cell_type": "markdown", "id": "44", "metadata": {}, "source": [ "Two of those are worth searching: `C`, the inverse of regularisation strength, and `penalty`, which kind of shrinkage it applies. `GridSearchCV` checks every combination in a fixed grid, exhaustive and simple:" ] }, { "cell_type": "code", "execution_count": null, "id": "45", "metadata": {}, "outputs": [], "source": [ "param_grid: dict[str, list[Any]] = {\n", " \"model__C\": [0.01, 0.1, 1.0, 10.0],\n", " \"model__penalty\": [\"l1\", \"l2\"],\n", " \"model__solver\": [\"liblinear\"],\n", "}\n", "grid_search: GridSearchCV = GridSearchCV(\n", " pipe_full, param_grid, cv=StratifiedKFold(5, shuffle=True, random_state=RANDOM_STATE), scoring=\"f1\", n_jobs=-1\n", ")\n", "grid_search.fit(X_train, y_train)\n", "logger.info(f\"Best grid params: {grid_search.best_params_}\")\n", "logger.info(f\"Best grid CV F1: {grid_search.best_score_:.3f}\")" ] }, { "cell_type": "markdown", "id": "46", "metadata": {}, "source": [ "`C=0.1`, `l1`, F1 barely above the default's 0.699, at 0.699. Eight combinations, five folds each, forty fits for a number that moved in the third decimal place. `RandomizedSearchCV` samples from a distribution instead of a fixed grid, useful once a parameter like `C` could plausibly need a value the grid never included:" ] }, { "cell_type": "code", "execution_count": null, "id": "47", "metadata": {}, "outputs": [], "source": [ "param_dist: dict[str, Any] = {\n", " \"model__C\": loguniform(1e-4, 1e2),\n", " \"model__penalty\": [\"l1\", \"l2\"],\n", " \"model__solver\": [\"liblinear\"],\n", "}\n", "rand_search: RandomizedSearchCV = RandomizedSearchCV(\n", " pipe_full,\n", " param_dist,\n", " n_iter=20,\n", " cv=StratifiedKFold(5, shuffle=True, random_state=RANDOM_STATE),\n", " scoring=\"f1\",\n", " random_state=RANDOM_STATE,\n", " n_jobs=-1,\n", ")\n", "rand_search.fit(X_train, y_train)\n", "logger.info(f\"Best random params: {rand_search.best_params_}\")\n", "logger.info(f\"Best random CV F1: {rand_search.best_score_:.3f}\")" ] }, { "cell_type": "markdown", "id": "48", "metadata": {}, "source": [ "| Strategy | What it tried | Best C | CV F1 |\n", "|---|---|---|---|\n", "| Default | `C=1.0` | 1.0 | 0.699 |\n", "| GridSearchCV | 8 fixed combinations | 0.1 | 0.699 |\n", "| RandomizedSearchCV | 20 sampled combinations | 0.063 | 0.699 |\n", "\n", "All three land within a rounding error of each other. That's a real finding, not a disappointing one: this feature set's ceiling with LogisticRegression sits right around 0.70 regardless of how `C` is tuned, which matches Section 5's own comparison showing a different model family, not a different hyperparameter, is where the next real gain would come from." ] }, { "cell_type": "markdown", "id": "49", "metadata": {}, "source": [ "
\n", " Activity 3: search over RandomForest

\n", "Define a param_dist for RandomForestClassifier (model__n_estimators, model__max_depth, model__min_samples_split) and run RandomizedSearchCV with 10 trials. Given Section 5's comparison, do you expect tuning to close much of the gap to LogisticRegression, or does RandomForest's advantage come from the model family itself rather than its specific settings?\n", "
" ] }, { "cell_type": "markdown", "id": "50", "metadata": {}, "source": [ "## 7. Bayesian optimisation: making each trial count\n", "\n", "Grid and random search both treat every trial as independent, learning nothing from the trials before it. Optuna's TPE sampler builds a running model of which regions of the search space score well and steers new trials toward them, useful when a single trial is expensive enough that wasting one on a region already ruled out is a real cost. Three pieces: a **trial** (one set of hyperparameters and the score they got), a **study** (a collection of trials with a direction to optimise), and an **objective function** (takes a trial, builds a pipeline from its suggested values, returns the score)." ] }, { "cell_type": "code", "execution_count": null, "id": "51", "metadata": {}, "outputs": [], "source": [ "def optuna_objective(trial: optuna.Trial) -> float:\n", " \"\"\"Return 5-fold CV F1 for the hyperparameters suggested by trial.\"\"\"\n", " C: float = trial.suggest_float(\"C\", 1e-4, 1e2, log=True)\n", " penalty: str = trial.suggest_categorical(\"penalty\", [\"l1\", \"l2\"])\n", " candidate: Pipeline = Pipeline(\n", " [\n", " (\"preprocessor\", preprocessor),\n", " (\n", " \"model\",\n", " LogisticRegression(\n", " C=C,\n", " penalty=penalty,\n", " solver=\"liblinear\",\n", " class_weight=\"balanced\",\n", " max_iter=1000,\n", " random_state=RANDOM_STATE,\n", " ),\n", " ),\n", " ]\n", " )\n", " scores: np.ndarray = cross_val_score(\n", " candidate,\n", " X_train,\n", " y_train,\n", " cv=StratifiedKFold(5, shuffle=True, random_state=RANDOM_STATE),\n", " scoring=\"f1\",\n", " n_jobs=-1,\n", " )\n", " return float(scores.mean())" ] }, { "cell_type": "code", "execution_count": null, "id": "52", "metadata": {}, "outputs": [], "source": [ "study: optuna.Study = optuna.create_study(direction=\"maximize\", sampler=optuna.samplers.TPESampler(seed=RANDOM_STATE))\n", "study.optimize(optuna_objective, n_trials=25, show_progress_bar=False)\n", "logger.success(\n", " f\"Best Optuna trial: C={study.best_params['C']:.4f} penalty={study.best_params['penalty']} \"\n", " f\"F1={study.best_value:.3f}\"\n", ")" ] }, { "cell_type": "markdown", "id": "53", "metadata": {}, "source": [ "`C=0.122`, F1 0.699. Watch how it got there:" ] }, { "cell_type": "code", "execution_count": null, "id": "54", "metadata": {}, "outputs": [], "source": [ "trial_df: pd.DataFrame = pd.DataFrame(\n", " [\n", " {\"trial\": t.number, \"f1\": t.value, \"best\": max(t_.value for t_ in study.trials[: i + 1])}\n", " for i, t in enumerate(study.trials)\n", " ]\n", ")\n", "(\n", " ggplot(trial_df, aes(x=\"trial\"))\n", " + geom_point(aes(y=\"f1\"), color=INFO, alpha=0.5, size=2)\n", " + geom_line(aes(y=\"best\"), color=SUCCESS, size=1.5)\n", " + labs(\n", " title=\"Optuna optimisation history\",\n", " subtitle=\"Dots: each trial's score. Line: the running best.\",\n", " x=\"Trial number\",\n", " y=\"CV F1\",\n", " )\n", " + modern_theme(grid=True)\n", ")" ] }, { "cell_type": "markdown", "id": "55", "metadata": {}, "source": [ "The running best flattens out well before trial 25: TPE found the neighbourhood worth exploiting early and spent the remaining trials confirming it rather than searching blind, which is the entire point of steering trials instead of sampling them uniformly. Three different search strategies now agree on the same answer, C in the neighbourhood of 0.1, F1 at 0.70, which is itself the useful result: this is the ceiling, and no fourth search strategy would find something meaningfully different. Fit the final pipeline and evaluate it against the test set exactly once, now that every round of searching is finished:" ] }, { "cell_type": "code", "execution_count": null, "id": "56", "metadata": {}, "outputs": [], "source": [ "best_c: float = study.best_params[\"C\"]\n", "best_pen: str = study.best_params[\"penalty\"]\n", "pipe_optuna: Pipeline = Pipeline(\n", " [\n", " (\"preprocessor\", preprocessor),\n", " (\n", " \"model\",\n", " LogisticRegression(\n", " C=best_c,\n", " penalty=best_pen,\n", " solver=\"liblinear\",\n", " class_weight=\"balanced\",\n", " max_iter=1000,\n", " random_state=RANDOM_STATE,\n", " ),\n", " ),\n", " ]\n", ")\n", "pipe_optuna.fit(X_train, y_train)\n", "y_pred_optuna: np.ndarray = pipe_optuna.predict(X_test)\n", "report_str: str = classification_report(y_test, y_pred_optuna, target_names=[\"Kept\", \"Cancelled\"])\n", "logger.info(\"\\n\" + report_str)" ] }, { "cell_type": "markdown", "id": "57", "metadata": {}, "source": [ "0.70 F1 on cancellations that never touched any search, precision and recall both around 0.70-0.71. Testing only once here matters for the same reason [Chapter 30](30-ml-workflow.qmd#the-three-way-split) held a test set back in the first place: run the test set against every candidate along the way and the \"held-out\" score quietly becomes another number you tuned against, not a check on it." ] }, { "cell_type": "markdown", "id": "58", "metadata": {}, "source": [ "
\n", " Activity 4: extend the search to max_iter

\n", "Add max_iter to optuna_objective, searching trial.suggest_categorical(\"max_iter\", [200, 500, 1000, 2000]). Does letting Optuna pick the iteration budget change the optimal C it finds, or are the two parameters independent of each other here?\n", "
" ] }, { "cell_type": "markdown", "id": "59", "metadata": {}, "source": [ "## 8. Which features are driving cancellations\n", "\n", "`get_feature_names_out()` already recovered what each of the 35 post-encoding columns means. Attach that to the tuned model's coefficients and you get a ranked answer to what it actually learned:" ] }, { "cell_type": "code", "execution_count": null, "id": "60", "metadata": {}, "outputs": [], "source": [ "lr_model: LogisticRegression = pipe_optuna.named_steps[\"model\"]\n", "fnames: list[str] = pipe_optuna.named_steps[\"preprocessor\"].get_feature_names_out().tolist()\n", "importance_df: pd.DataFrame = (\n", " pd.DataFrame({\"feature\": fnames, \"coefficient\": lr_model.coef_[0]})\n", " .assign(abs_coef=lambda d: d[\"coefficient\"].abs())\n", " .sort_values(\"abs_coef\", ascending=False)\n", " .head(10)\n", " .reset_index(drop=True)\n", ")\n", "themed_gt(GT(importance_df[[\"feature\", \"coefficient\", \"abs_coef\"]]), n_rows=len(importance_df)).tab_header(\n", " title=gt_md(\"**Top 10 feature importances**\"),\n", " subtitle=\"Coefficient sign and magnitude for cancellation prediction\",\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "61", "metadata": {}, "outputs": [], "source": [ "plot_df: pd.DataFrame = importance_df.assign(\n", " color=lambda d: d[\"coefficient\"].apply(lambda v: \"positive\" if v >= 0 else \"negative\")\n", ")\n", "(\n", " ggplot(plot_df, aes(x=as_discrete(\"feature\", order_by=\"abs_coef\", order=1), y=\"coefficient\", fill=\"color\"))\n", " + geom_bar(stat=\"identity\", alpha=0.85)\n", " + scale_fill_manual(values={\"positive\": INFO, \"negative\": DANGER})\n", " + labs(\n", " title=\"Top 10 features: LogisticRegression coefficients\",\n", " subtitle=\"Positive raises predicted cancellation probability\",\n", " x=\"\",\n", " y=\"Coefficient\",\n", " )\n", " + modern_theme(x_axis_angle=40, legend_pos=\"none\")\n", ")" ] }, { "cell_type": "markdown", "id": "62", "metadata": {}, "source": [ "`deposit_type=Non Refund` towers over everything else at +4.91, five times the size of the next feature. Read that coefficient straight and it says a non-refundable deposit makes a booking dramatically *more* likely to cancel, the opposite of what a non-refundable deposit is supposed to do. That's not a bug in this pipeline, it's a widely-noted quirk of this exact dataset: a large share of the bookings marked both \"Non Refund\" and cancelled appear to be travel-agency block bookings that get released back as cancellations rather than individual guests changing plans, a pattern several published analyses of this dataset have flagged without fully resolving. The model found something real in the data. Whether that something means what the column name suggests is a separate question, and it's exactly the kind of question a top-ranked, counterintuitive feature should raise before it ships, not one a chapter should paper over with a tidy explanation that happens to fit." ] }, { "cell_type": "markdown", "id": "63", "metadata": {}, "source": [ "
\n", " Pro Tip: a surprising top feature is a reason to dig, not a reason to relax

\n", "When the highest-ranked feature's sign runs against domain intuition, as it does here, the right response is investigating the data's provenance, not inventing a story that makes the number feel comfortable. A model that ranks a booking-agency artefact above genuine guest behaviour is still accurate on this test set, and still not safe to explain to a stakeholder as \"non-refundable deposits predict cancellation\" without the caveat above attached.\n", "
" ] }, { "cell_type": "markdown", "id": "64", "metadata": {}, "source": [ "## Packaging the pattern for reuse\n", "\n", "Everything above fits into a model card: what was trained, on what data, scoring what, so the artefact isn't a black box to whoever inherits it next." ] }, { "cell_type": "code", "execution_count": null, "id": "65", "metadata": {}, "outputs": [], "source": [ "final_cv: np.ndarray = cross_val_score(\n", " pipe_optuna,\n", " X_train,\n", " y_train,\n", " cv=StratifiedKFold(5, shuffle=True, random_state=RANDOM_STATE),\n", " scoring=\"f1\",\n", " n_jobs=-1,\n", ")\n", "test_f1: float = f1_score(y_test, y_pred_optuna)\n", "model_card: dict[str, Any] = {\n", " \"model_class\": \"LogisticRegression\",\n", " \"target\": TARGET_CLF,\n", " \"features\": ALL_FEATURES,\n", " \"n_train\": len(X_train),\n", " \"n_test\": len(X_test),\n", " \"search_strategy\": \"Optuna TPE (25 trials)\",\n", " \"best_c\": best_c,\n", " \"best_penalty\": best_pen,\n", " \"cv_f1_mean\": float(final_cv.mean()),\n", " \"cv_f1_std\": float(final_cv.std()),\n", " \"test_f1\": test_f1,\n", " \"dataset\": \"Hotel Booking Demand (Antonio et al., 2019)\",\n", "}\n", "for k, v in model_card.items():\n", " logger.info(f\" {k:<22} {v}\")" ] }, { "cell_type": "markdown", "id": "66", "metadata": {}, "source": [ "Save it as one bundle, pipeline and metadata together:" ] }, { "cell_type": "code", "execution_count": null, "id": "67", "metadata": {}, "outputs": [], "source": [ "MODEL_DIR.mkdir(exist_ok=True)\n", "bundle_path: Path = MODEL_DIR / \"cancellation_pipeline.joblib\"\n", "joblib.dump({\"pipeline\": pipe_optuna, \"features\": ALL_FEATURES, \"model_card\": model_card}, bundle_path)\n", "logger.success(f\"Bundle saved to {bundle_path}\")" ] }, { "cell_type": "markdown", "id": "68", "metadata": {}, "source": [ "Reload it in a fresh call to confirm the bundle actually works end to end:" ] }, { "cell_type": "code", "execution_count": null, "id": "69", "metadata": {}, "outputs": [], "source": [ "loaded: dict[str, Any] = joblib.load(bundle_path)\n", "sample_row: pd.DataFrame = X_test.iloc[:1]\n", "prediction: int = int(loaded[\"pipeline\"].predict(sample_row)[0])\n", "actual: int = int(y_test[0])\n", "logger.info(f\"Predicted: {prediction} Actual: {actual} ({'match' if prediction == actual else 'mismatch'})\")" ] }, { "cell_type": "markdown", "id": "70", "metadata": {}, "source": [ "One bundle, pipeline and metadata together, the same discipline [Chapter 31](31-sklearn-core.ipynb) established for a scaler and a model. Wrapping the whole build in a typed function makes the pattern reusable rather than something to retype for the next dataset:" ] }, { "cell_type": "code", "execution_count": null, "id": "71", "metadata": {}, "outputs": [], "source": [ "def build_classification_pipeline(\n", " df: pd.DataFrame,\n", " numerical_features: list[str],\n", " categorical_features: list[str],\n", " target: str,\n", " n_trials: int = 25,\n", " out_dir: Path = Path(\"models\"),\n", ") -> dict[str, Any]:\n", " \"\"\"Build, tune, and save a classification pipeline. Returns metrics and artefact paths.\"\"\"\n", " all_feats = numerical_features + categorical_features\n", " X_all: pd.DataFrame = df[all_feats].copy()\n", " y_all: np.ndarray = df[target].to_numpy(dtype=int)\n", "\n", " Xtr, Xte, ytr, yte = train_test_split(X_all, y_all, test_size=0.2, random_state=RANDOM_STATE, stratify=y_all)\n", "\n", " num_pipe = make_pipeline(SimpleImputer(strategy=\"median\"), StandardScaler())\n", " cat_pipe = make_pipeline(\n", " SimpleImputer(strategy=\"most_frequent\"), OneHotEncoder(handle_unknown=\"ignore\", sparse_output=False)\n", " )\n", " prep = ColumnTransformer([(\"num\", num_pipe, numerical_features), (\"cat\", cat_pipe, categorical_features)])\n", "\n", " def _objective(trial: optuna.Trial) -> float:\n", " C = trial.suggest_float(\"C\", 1e-4, 1e2, log=True)\n", " pen = trial.suggest_categorical(\"penalty\", [\"l1\", \"l2\"])\n", " pipe = Pipeline(\n", " [\n", " (\"preprocessor\", prep),\n", " (\n", " \"model\",\n", " LogisticRegression(\n", " C=C,\n", " penalty=pen,\n", " solver=\"liblinear\",\n", " class_weight=\"balanced\",\n", " max_iter=1000,\n", " random_state=RANDOM_STATE,\n", " ),\n", " ),\n", " ]\n", " )\n", " scores = cross_val_score(\n", " pipe,\n", " Xtr,\n", " ytr,\n", " cv=StratifiedKFold(5, shuffle=True, random_state=RANDOM_STATE),\n", " scoring=\"f1\",\n", " n_jobs=-1,\n", " )\n", " return float(scores.mean())\n", "\n", " study_inner = optuna.create_study(direction=\"maximize\", sampler=optuna.samplers.TPESampler(seed=RANDOM_STATE))\n", " study_inner.optimize(_objective, n_trials=n_trials, show_progress_bar=False)\n", "\n", " best = study_inner.best_params\n", " final_pipe = Pipeline(\n", " [\n", " (\"preprocessor\", prep),\n", " (\n", " \"model\",\n", " LogisticRegression(\n", " C=best[\"C\"],\n", " penalty=best[\"penalty\"],\n", " solver=\"liblinear\",\n", " class_weight=\"balanced\",\n", " max_iter=1000,\n", " random_state=RANDOM_STATE,\n", " ),\n", " ),\n", " ]\n", " )\n", " final_pipe.fit(Xtr, ytr)\n", "\n", " test_f1: float = f1_score(yte, final_pipe.predict(Xte))\n", " out_dir.mkdir(exist_ok=True)\n", " path: Path = out_dir / f\"{target}_pipeline.joblib\"\n", " joblib.dump({\"pipeline\": final_pipe, \"features\": all_feats}, path)\n", " logger.success(f\"Saved pipeline bundle to {path}\")\n", " logger.info(f\"Test F1: {test_f1:.3f} Best params: {best}\")\n", " return {\"pipeline\": final_pipe, \"test_f1\": test_f1, \"best_params\": best, \"path\": path}" ] }, { "cell_type": "code", "execution_count": null, "id": "72", "metadata": {}, "outputs": [], "source": [ "result: dict[str, Any] = build_classification_pipeline(\n", " df, NUMERICAL_FEATURES, CATEGORICAL_FEATURES, TARGET_CLF, n_trials=25, out_dir=MODEL_DIR\n", ")" ] }, { "cell_type": "markdown", "id": "73", "metadata": {}, "source": [ "This pipeline is closer to shippable than anything in [Chapter 31](31-sklearn-core.ipynb): fourteen columns instead of six, three model families compared honestly rather than assumed, three search strategies agreeing on the same ceiling, and a top feature investigated rather than rationalised. It's still not a deployment decision by itself. 0.70 F1 says nothing yet about whether catching seven in ten cancellations, at whatever precision cost, is worth the cost of a revenue team acting on each flagged booking, that threshold question is exactly [Case Study 1](../05-classical-ml-projects/33-spend-regression.ipynb#key-findings-and-recommendation)'s closing move, and it's the one this chapter deliberately leaves for the case studies that follow: attach a dollar figure, not just a metric, before calling a model done.\n", "\n", "That closes the ML landscape part. [Chapter 30](30-ml-workflow.qmd) gave you the workflow and the vocabulary. [Chapter 31](31-sklearn-core.ipynb) gave you the two shapes every algorithm takes. This chapter gave you the one object that holds many of them together safely. The seven case studies ahead each start from here, on a new dataset, with a new business question, and the same discipline: baseline first, leakage-proof always, a threshold decided before a model is called finished." ] }, { "cell_type": "markdown", "id": "74", "metadata": {}, "source": [ "## Further reading\n", "\n", "- [Scikit-learn Pipeline docs](https://scikit-learn.org/stable/modules/compose.html)\n", "- [Optuna documentation](https://optuna.readthedocs.io/)\n", "- Akiba et al. (2019), *Optuna: A Next-generation Hyperparameter Optimization Framework*\n", "- Geron, A. (2022), *Hands-On Machine Learning*, Ch 2\n", "- Antonio et al. (2019), *Hotel booking demand datasets*, Data in Brief" ] }, { "cell_type": "markdown", "id": "75", "metadata": {}, "source": [ "## Summary\n", "\n", "| Concept | Key rule |\n", "|---|---|\n", "| Pipeline | Chains Transformers and a Predictor into one object with one `fit` and one `predict`; re-fits every stage inside each cross-validation fold automatically |\n", "| ColumnTransformer | Routes numeric and categorical columns to different treatments and concatenates the results; `get_feature_names_out()` recovers what each output column means |\n", "| Leaky cross-validation | Calling `fit_transform` before `cross_val_score` fits the preprocessor on validation rows too; always pass the whole Pipeline, never its output |\n", "| Compare families before tuning | A family's default score is a signal about whether its assumptions suit the data; don't spend a tuning budget on a family that can't compete untuned |\n", "| GridSearchCV / RandomizedSearchCV / Optuna | Exhaustive grid, sampled distribution, and TPE-guided search respectively; on this data all three converge on the same answer |\n", "| Test set discipline | Evaluate on the test set exactly once, after every round of searching is finished, or the \"held-out\" score becomes another number you tuned against |\n", "| Surprising feature importance | A top-ranked feature with a counterintuitive sign is a reason to investigate the data's provenance, not a reason to invent an explanation that fits |\n", "\n", "**Next:** The case studies ahead, starting with [Case Study 1](../05-classical-ml-projects/33-spend-regression.ipynb), apply this same discipline, baseline first, leakage-proof pipeline, a threshold decided before a model is called finished, to a new dataset and a new business question each." ] } ], "metadata": { "kernelspec": { "display_name": "ark (3.12.12.final.0)", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 5 }