{ "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "---\n", "title: \"Chapter 31: Classical ML with scikit-learn\"\n", "---" ] }, { "cell_type": "markdown", "id": "1", "metadata": {}, "source": [ "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/sambaiga/ds-mlops-path/blob/main/tutorials/04-ml-intro/31-sklearn-core.ipynb) [![Download Notebook](https://img.shields.io/badge/Download-Notebook-blue.svg?logo=jupyter&logoColor=white)](https://raw.githubusercontent.com/sambaiga/ds-mlops-path/main/tutorials/04-ml-intro/31-sklearn-core.ipynb)" ] }, { "cell_type": "markdown", "id": "2", "metadata": {}, "source": [ "Half of the bookings that end up cancelled were made 113 days before check-in. Half of the ones that stick were made 47 days out. Somewhere between a reservation landing in the system and a guest walking through the door, more than a third of all 117,429 bookings in this hotel group's three years of records fall through, and it happens far more at the City Hotel (42%) than at the Resort (28%). None of that is a guess. It's sitting in a CSV, waiting for someone to ask two questions of it: which of today's bookings will actually show up, and what should a room like this one have cost in the first place." ] }, { "cell_type": "markdown", "id": "3", "metadata": {}, "source": [ "Chapter 30 gave you the vocabulary to ask those questions properly: a spec card, a workflow, a way to reason about splits and about bias and variance before writing a line of code. Everything in that chapter was diagrams and definitions. This is where it becomes code you run yourself, on the two questions above, using scikit-learn, the tool nearly every classical model in this book runs through. Learn the shape every one of its algorithms takes and you can read unfamiliar sklearn code on sight, swap one model for another without touching your preprocessing, and follow how [Chapter 32](32-sklearn-pipeline.ipynb) chains today's separate steps into a single object." ] }, { "cell_type": "markdown", "id": "4", "metadata": {}, "source": [ "::: {.callout-note collapse=\"true\" icon=false}\n", "## Learning objectives\n", "\n", "By the end of this chapter you will be able to:\n", "\n", "| # | Skill | Covered in |\n", "|---|---|---|\n", "| 1 | Distinguish scikit-learn's Transformer and Predictor interfaces, and know which one any given algorithm implements | Sec. 2 |\n", "| 2 | Split data before fitting a scaler or a model, and explain why the opposite order causes leakage | Sec. 3 |\n", "| 3 | Establish a dummy baseline before judging whether a real model has learned anything | Sec. 4 |\n", "| 4 | Fit and evaluate LinearRegression and LogisticRegression, including class_weight for imbalanced targets | Sec. 5 |\n", "| 5 | Use cross-validation to get a steadier performance estimate than a single train/test split | Sec. 6 |\n", "| 6 | Read a learning curve to diagnose whether more data or better features would move a metric | Sec. 7 |\n", "| 7 | Bundle a fitted scaler and model together so they can't be loaded out of sync in production | Sec. 8 |\n", ":::" ] }, { "cell_type": "markdown", "id": "5", "metadata": {}, "source": [ "## 1. Two questions, one dataset\n", "\n", "Both questions point at the same table: 117,429 bookings from two Portuguese hotels, one row per reservation. `adr`, average daily rate, is what the room actually sold for, in euros. `is_canceled` is 1 if the booking fell through before check-in, 0 if the guest showed up. Predicting the first is regression; predicting the second is binary classification. Same data, two different kinds of answer." ] }, { "cell_type": "code", "execution_count": null, "id": "6", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "from typing import Any\n", "\n", "from great_tables import GT, md as gt_md\n", "import joblib\n", "from lets_plot import (\n", " LetsPlot,\n", " aes,\n", " as_discrete,\n", " geom_bar,\n", " geom_hline,\n", " geom_line,\n", " geom_point,\n", " geom_ribbon,\n", " ggplot,\n", " labs,\n", " scale_color_manual,\n", " scale_fill_manual,\n", ")\n", "from loguru import logger\n", "import numpy as np\n", "import pandas as pd\n", "from sklearn.dummy import DummyClassifier, DummyRegressor\n", "from sklearn.linear_model import LinearRegression, LogisticRegression\n", "from sklearn.metrics import (\n", " classification_report,\n", " f1_score,\n", " mean_absolute_error,\n", " mean_squared_error,\n", " r2_score,\n", ")\n", "from sklearn.model_selection import (\n", " KFold,\n", " StratifiedKFold,\n", " cross_val_score,\n", " learning_curve,\n", " train_test_split,\n", ")\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import MinMaxScaler, StandardScaler\n", "\n", "from ark.plot.gt_style import metrics_report, themed_gt\n", "from ark.plot.theme import modern_theme\n", "from ark.plot.tokens import DANGER, INFO, WARNING\n", "\n", "LetsPlot.setup_html()" ] }, { "cell_type": "markdown", "id": "7", "metadata": {}, "source": [ "A typed loader keeps the download-and-cache logic out of the way of everything that follows: it fetches the CSV once, caches it to disk, and every later run reads the local copy instead of hitting the network again." ] }, { "cell_type": "code", "execution_count": null, "id": "8", "metadata": {}, "outputs": [], "source": [ "DATA_URL: str = \"https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-02-11/hotels.csv\"\n", "DATA_PATH: Path = Path(\"data/hotel_bookings.csv\")\n", "MODEL_DIR: Path = Path(\"models\")\n", "\n", "\n", "def load_hotel_data(path: Path = DATA_PATH, url: str = DATA_URL) -> pd.DataFrame:\n", " \"\"\"Load hotel bookings, downloading from url on first call.\"\"\"\n", " if not path.exists():\n", " path.parent.mkdir(parents=True, exist_ok=True)\n", " raw = pd.read_csv(url)\n", " raw.to_csv(path, index=False)\n", " logger.success(f\"Downloaded {len(raw):,} rows to {path}\")\n", " else:\n", " raw = pd.read_csv(path)\n", " logger.info(f\"Loaded {len(raw):,} rows from cache: {path}\")\n", " df = raw[(raw[\"adr\"] > 0) & (raw[\"adr\"] <= 1000)].reset_index(drop=True)\n", " logger.info(f\"After cleaning: {len(df):,} rows\")\n", " return df" ] }, { "cell_type": "code", "execution_count": null, "id": "9", "metadata": {}, "outputs": [], "source": [ "df: pd.DataFrame = load_hotel_data()\n", "logger.info(f\"Shape: {df.shape[0]:,} rows x {df.shape[1]} columns\")\n", "logger.info(f\"Overall cancellation rate: {df['is_canceled'].mean():.1%}\")" ] }, { "cell_type": "markdown", "id": "10", "metadata": {}, "source": [ "| Target | Column | Task |\n", "|---|---|---|\n", "| Room price | `adr` | Regression |\n", "| Booking cancelled | `is_canceled` | Binary classification |\n", "\n", "Look at the shape of each target before building anything on top of it. A summary table gets you the numeric columns at a glance:" ] }, { "cell_type": "code", "execution_count": null, "id": "11", "metadata": {}, "outputs": [], "source": [ "stats: pd.DataFrame = df.describe().round(2).T.reset_index().rename(columns={\"index\": \"feature\"})\n", "(\n", " themed_gt(GT(stats), n_rows=len(stats))\n", " .tab_header(\n", " title=gt_md(\"**Hotel bookings: summary statistics**\"),\n", " subtitle=\"117k bookings from two Portuguese hotels (2015-2017)\",\n", " )\n", " .tab_source_note(\"Antonio et al., 2019 | Data in Brief\")\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "12", "metadata": {}, "outputs": [], "source": [ "TARGET_REG: str = \"adr\"\n", "TARGET_CLF: str = \"is_canceled\"" ] }, { "cell_type": "markdown", "id": "13", "metadata": {}, "source": [ "### What a room actually costs\n", "\n", "A table of means and quartiles hides the shape of the distribution. Chart it instead." ] }, { "cell_type": "code", "execution_count": null, "id": "14", "metadata": {}, "outputs": [], "source": [ "(\n", " ggplot(df[[\"adr\"]], aes(x=\"adr\"))\n", " + geom_bar(stat=\"bin\", fill=INFO, alpha=0.85, bins=60)\n", " + labs(\n", " title=\"Distribution of average daily rate\",\n", " subtitle=\"Regression target: room price per night, in euros\",\n", " x=\"ADR (euros)\",\n", " y=\"Bookings\",\n", " )\n", " + modern_theme(grid=True)\n", ")" ] }, { "cell_type": "markdown", "id": "15", "metadata": {}, "source": [ "Mean ADR is €103.49, median is €95.00: a real but modest right skew, a long tail of expensive suites and premium dates pulling the mean above where most bookings actually sit. Not skewed enough to demand a log transform, skewed enough that a model judged purely on raw error will care more about getting the tail right than about the ordinary €80-100 booking most guests actually make." ] }, { "cell_type": "markdown", "id": "16", "metadata": {}, "source": [ "### Which bookings cancel\n", "\n", "The hook opened on a lead-time gap. Chart cancellation by hotel and you get the other half of the picture: whether both hotels share the same risk, or whether one runs hotter than the other." ] }, { "cell_type": "code", "execution_count": null, "id": "17", "metadata": {}, "outputs": [], "source": [ "cancel_df: pd.DataFrame = (\n", " df.groupby(\"hotel\", as_index=False)[\"is_canceled\"]\n", " .agg(total=\"count\", canceled=\"sum\")\n", " .assign(rate=lambda d: d[\"canceled\"] / d[\"total\"])\n", ")\n", "(\n", " ggplot(cancel_df, aes(x=\"hotel\", y=\"rate\", fill=\"hotel\"))\n", " + geom_bar(stat=\"identity\", alpha=0.85)\n", " + scale_fill_manual(values=[INFO, WARNING])\n", " + labs(\n", " title=\"Cancellation rate by hotel type\",\n", " subtitle=\"37% overall: City Hotel cancels more than Resort\",\n", " x=\"Hotel type\",\n", " y=\"Cancellation rate\",\n", " )\n", " + modern_theme(legend_pos=\"none\")\n", ")" ] }, { "cell_type": "markdown", "id": "18", "metadata": {}, "source": [ "City Hotel bookings cancel 42% of the time, Resort bookings 28%. A City Hotel guest is booking a work trip or a short city break, easier to change plans on than a holiday already built around a resort stay. That's a hypothesis, not a fact the chart proves, but it's the kind of question worth carrying into feature selection: hotel type alone is doing real work before a single model gets involved." ] }, { "cell_type": "markdown", "id": "19", "metadata": {}, "source": [ "
\n", " Activity 1: confirm the hook's own number

\n", "The opening of this chapter claimed a 113-day median lead time for cancelled bookings against 47 days for the ones that stuck. Compute it yourself before trusting a number a book handed you:\n", "
df.groupby(\"is_canceled\")[\"lead_time\"].median()
\n", "
" ] }, { "cell_type": "markdown", "id": "20", "metadata": {}, "source": [ "## 2. Two shapes every scikit-learn model takes\n", "\n", "Regression and classification look like different problems, and in one sense they are: different targets, different metrics, different questions. But every algorithm you're about to use, whether it predicts a price or a cancellation, exposes one of exactly two interfaces. Learn to recognise both and the rest of this chapter, and most of the book's remaining case studies, stop looking like a new API every time the model changes.\n", "\n", "Start with a toy array, small enough to see everything happening, so the mechanics are visible before any hotel data gets involved." ] }, { "cell_type": "code", "execution_count": null, "id": "21", "metadata": {}, "outputs": [], "source": [ "demo_x: np.ndarray = np.array([[1.0, 100.0], [2.0, 200.0], [3.0, 300.0]])\n", "demo_scaler: StandardScaler = StandardScaler()\n", "demo_scaler.fit(demo_x)\n", "logger.info(f\"Learned mean: {demo_scaler.mean_}\")\n", "logger.info(f\"Learned scale: {demo_scaler.scale_}\")" ] }, { "cell_type": "markdown", "id": "22", "metadata": {}, "source": [ "`fit` did the learning: it looked at `demo_x` and worked out a mean and a spread for each column. Nothing was transformed yet. Apply those learned numbers to a new point and the same scaler reuses them, it doesn't relearn anything:" ] }, { "cell_type": "code", "execution_count": null, "id": "23", "metadata": {}, "outputs": [], "source": [ "new_point: np.ndarray = np.array([[4.0, 400.0]])\n", "scaled_point: np.ndarray = demo_scaler.transform(new_point)\n", "logger.info(f\"Raw input: {new_point}\")\n", "logger.info(f\"Scaled: {scaled_point}\")" ] }, { "cell_type": "markdown", "id": "24", "metadata": {}, "source": [ "That two-step shape, learn parameters with `fit`, apply them with `transform`, is what scikit-learn calls a **Transformer**. It never sees a target value, only the columns it's adjusting.\n", "\n", "A model that predicts something works differently: it still has a `fit` step, but this time it's learning a mapping from features to a known answer, and instead of `transform` it exposes `predict`." ] }, { "cell_type": "code", "execution_count": null, "id": "25", "metadata": {}, "outputs": [], "source": [ "demo_y: np.ndarray = np.array([1.0, 2.0, 3.0])\n", "demo_lr: LinearRegression = LinearRegression()\n", "demo_lr.fit(demo_x, demo_y)\n", "demo_pred: np.ndarray = demo_lr.predict(new_point)\n", "logger.info(f\"Predicted y: {demo_pred}\")" ] }, { "cell_type": "markdown", "id": "26", "metadata": {}, "source": [ "
\n", " Key Concept: Transformer and Predictor

\n", "A Transformer has fit and transform. It learns something about the data's shape, a mean, a scale, a set of categories, and never touches a target. A Predictor has fit(X, y) and predict(X). It learns a mapping to a known answer. StandardScaler, OneHotEncoder, and every other preprocessing step you meet in this book are Transformers. LinearRegression, LogisticRegression, and every model that follows are Predictors. Chapter 32's Pipeline exists because a real project chains several of the first into exactly one of the second, and having a name for each half is what makes that chain readable.\n", "
" ] }, { "cell_type": "markdown", "id": "27", "metadata": {}, "source": [ "![Two parallel flow diagrams: a Transformer calls fit then transform and never sees a target, a Predictor calls fit with X and y then predict, and a Pipeline chains several Transformers into one Predictor.](figs/transformer-predictor.svg){fig-alt=\"Diagram contrasting a Transformer's fit-transform shape against a Predictor's fit-predict shape, both converging on a Pipeline chaining several of the first into one of the second.\"}" ] }, { "cell_type": "markdown", "id": "28", "metadata": {}, "source": [ "## 3. Selecting and scaling features, before fitting anything real\n", "\n", "Six interpretable columns, kept deliberately small so the mechanics stay visible: how far ahead someone booked, how many nights of each kind, how many adults, prior cancellations, and post-booking changes. [Chapter 32](32-sklearn-pipeline.ipynb) grows this into a full mixed feature set once the pattern here is solid." ] }, { "cell_type": "code", "execution_count": null, "id": "29", "metadata": {}, "outputs": [], "source": [ "FEATURES: list[str] = [\n", " \"lead_time\",\n", " \"stays_in_weekend_nights\",\n", " \"stays_in_week_nights\",\n", " \"adults\",\n", " \"previous_cancellations\",\n", " \"booking_changes\",\n", "]\n", "X: np.ndarray = df[FEATURES].to_numpy(dtype=float)\n", "y_reg: np.ndarray = df[TARGET_REG].to_numpy(dtype=float)\n", "y_clf: np.ndarray = df[TARGET_CLF].to_numpy(dtype=int)\n", "logger.info(f\"X: {X.shape} y_reg: {y_reg.shape} y_clf: {y_clf.shape}\")" ] }, { "cell_type": "markdown", "id": "30", "metadata": {}, "source": [ "`lead_time` runs into the hundreds of days, `adults` rarely exceeds four. A model that updates its parameters by gradient, or measures distance between points, treats a feature with a bigger numeric range as more important by default, regardless of what it actually measures. Scaling puts every feature on the same footing before that happens.\n", "\n", "Here's the part that's easy to get backwards: the scaler itself has to learn its mean and spread from training data only. Fit it on the full dataset before splitting, and the test set has quietly leaked its own distribution into the very transformation meant to be evaluated against it. The result looks like a working model and tells you nothing about how it performs on data it hasn't seen." ] }, { "cell_type": "code", "execution_count": null, "id": "31", "metadata": {}, "outputs": [], "source": [ "for i, feat in enumerate(FEATURES):\n", " logger.info(f\" {feat:<30} min={X[:, i].min():>7.1f} max={X[:, i].max():>7.1f}\")" ] }, { "cell_type": "markdown", "id": "32", "metadata": {}, "source": [ "Split first, before the scaler learns anything:" ] }, { "cell_type": "code", "execution_count": null, "id": "33", "metadata": {}, "outputs": [], "source": [ "X_tr, X_te, y_tr, y_te = train_test_split(X, y_reg, test_size=0.2, random_state=42)\n", "logger.info(f\"Train: {X_tr.shape} Test: {X_te.shape}\")" ] }, { "cell_type": "markdown", "id": "34", "metadata": {}, "source": [ "With the split done, fit the scaler on the training portion only:" ] }, { "cell_type": "code", "execution_count": null, "id": "35", "metadata": {}, "outputs": [], "source": [ "scaler: StandardScaler = StandardScaler()\n", "scaler.fit(X_tr)\n", "logger.info(f\"Learned means: {scaler.mean_.round(2)}\")\n", "logger.info(f\"Learned stds: {scaler.scale_.round(2)}\")" ] }, { "cell_type": "markdown", "id": "36", "metadata": {}, "source": [ "The same fitted scaler now applies to both sets. Same parameters, different input:" ] }, { "cell_type": "code", "execution_count": null, "id": "37", "metadata": {}, "outputs": [], "source": [ "X_train_s: np.ndarray = scaler.transform(X_tr)\n", "X_test_s: np.ndarray = scaler.transform(X_te)\n", "for i, feat in enumerate(FEATURES):\n", " logger.info(f\" {feat:<30} min={X_train_s[:, i].min():>6.2f} max={X_train_s[:, i].max():>6.2f}\")" ] }, { "cell_type": "markdown", "id": "38", "metadata": {}, "source": [ "`StandardScaler` centres on the mean. `MinMaxScaler` squeezes everything into `[0, 1]` instead, useful for algorithms that expect bounded input, like a neural network's first layer:" ] }, { "cell_type": "code", "execution_count": null, "id": "39", "metadata": {}, "outputs": [], "source": [ "mms: MinMaxScaler = MinMaxScaler()\n", "mms.fit(X_tr)\n", "X_mm: np.ndarray = mms.transform(X_tr)\n", "for i, feat in enumerate(FEATURES):\n", " logger.info(f\" {feat:<30} min={X_mm[:, i].min():.2f} max={X_mm[:, i].max():.2f}\")" ] }, { "cell_type": "markdown", "id": "40", "metadata": {}, "source": [ "
\n", " Common Mistake: fitting the scaler before the split

\n", "\n", "
\n",
    "# Wrong: the scaler sees every row, including the ones about to become test data\n",
    "X_scaled = StandardScaler().fit_transform(X)\n",
    "X_train_s, X_test_s = X_scaled[:n_train], X_scaled[n_train:]\n",
    "\n",
    "# Correct: split first, fit on training rows only\n",
    "X_tr, X_te, y_tr, y_te = train_test_split(X, y)\n",
    "sc = StandardScaler().fit(X_tr)\n",
    "X_train_s = sc.transform(X_tr)\n",
    "X_test_s = sc.transform(X_te)\n",
    "
\n", "\n", "The wrong version still runs. It still produces a number. That number is just measuring something other than what it claims to.\n", "
" ] }, { "cell_type": "markdown", "id": "41", "metadata": {}, "source": [ "
\n", " Activity 2: RobustScaler

\n", "Fit a RobustScaler on X_tr (it centres on the median and scales by interquartile range instead of mean and standard deviation, so outliers pull it around less). Compare the scaled ranges against StandardScaler's above.\n", "
from sklearn.preprocessing import RobustScaler\n",
    "# your code here
\n", "Which feature moves the most compared to StandardScaler, and does that match which one had the widest raw range back in the first cell of this section?\n", "
" ] }, { "cell_type": "markdown", "id": "42", "metadata": {}, "source": [ "## 4. Establishing baselines before anything smarter\n", "\n", "Before any real model gets a turn, ask what the dumbest possible answer would score. `DummyRegressor` always predicts the training mean. `DummyClassifier` always predicts the training set's most common class. Neither one learns anything about the six features you just scaled, and that's the point: whatever a real model scores, it has to beat this first." ] }, { "cell_type": "code", "execution_count": null, "id": "43", "metadata": {}, "outputs": [], "source": [ "dummy_reg: DummyRegressor = DummyRegressor(strategy=\"mean\")\n", "dummy_reg.fit(X_train_s, y_tr)\n", "dummy_pred: np.ndarray = dummy_reg.predict(X_test_s)\n", "dummy_mae: float = mean_absolute_error(y_te, dummy_pred)\n", "dummy_const: float = float(dummy_reg.constant_.flat[0])\n", "logger.info(f\"Dummy always predicts: {dummy_const:.2f} euros\")\n", "logger.info(f\"Dummy MAE: {dummy_mae:.2f}\")" ] }, { "cell_type": "markdown", "id": "44", "metadata": {}, "source": [ "€103.55 for every single booking, off by €35.66 on average. That's the number Section 5's model has to beat, and by how much determines whether the six features are worth anything at all.\n", "\n", "Classification needs its own split, stratified so the 37% cancellation rate holds in both halves rather than drifting by chance:" ] }, { "cell_type": "code", "execution_count": null, "id": "45", "metadata": {}, "outputs": [], "source": [ "Xc_tr, Xc_te, yc_tr, yc_te = train_test_split(X, y_clf, test_size=0.2, random_state=42, stratify=y_clf)\n", "sc_clf: StandardScaler = StandardScaler()\n", "Xc_train_s: np.ndarray = sc_clf.fit_transform(Xc_tr)\n", "Xc_test_s: np.ndarray = sc_clf.transform(Xc_te)\n", "logger.info(f\"Train cancel rate: {yc_tr.mean():.2%} Test cancel rate: {yc_te.mean():.2%}\")" ] }, { "cell_type": "markdown", "id": "46", "metadata": {}, "source": [ "The classifier's dummy baseline works the same way:" ] }, { "cell_type": "code", "execution_count": null, "id": "47", "metadata": {}, "outputs": [], "source": [ "dummy_clf: DummyClassifier = DummyClassifier(strategy=\"most_frequent\")\n", "dummy_clf.fit(Xc_train_s, yc_tr)\n", "dummy_clf_pred: np.ndarray = dummy_clf.predict(Xc_test_s)\n", "dummy_f1: float = f1_score(yc_te, dummy_clf_pred, zero_division=0)\n", "majority_class: int = int(dummy_clf.classes_[np.argmax(dummy_clf.class_prior_)])\n", "logger.info(f\"Dummy always predicts class: {majority_class}\")\n", "logger.info(f\"Dummy F1: {dummy_f1:.3f}\")" ] }, { "cell_type": "markdown", "id": "48", "metadata": {}, "source": [ "Zero. Not a low score, zero: predicting \"not cancelled\" for every booking gets roughly 63% of predictions right, since that's the class majority, but it never once identifies an actual cancellation, and F1 punishes that completely. A model judged on plain accuracy could hide behind that same 63% and look almost as good as doing nothing. This is exactly why [Chapter 30](30-ml-workflow.qmd#evaluation-metrics-that-match-the-problem) picked F1 over accuracy for imbalanced problems: accuracy would have called this dummy baseline a decent model." ] }, { "cell_type": "markdown", "id": "49", "metadata": {}, "source": [ "## 5. Linear models: a first real attempt\n", "\n", "`LinearRegression` for ADR, `LogisticRegression` for cancellation, the two simplest Predictors that can actually learn from the six features instead of ignoring them. One evaluation helper covers every regression comparison the rest of this notebook makes:" ] }, { "cell_type": "code", "execution_count": null, "id": "50", "metadata": {}, "outputs": [], "source": [ "def evaluate_regressor(model: Any, X_eval: np.ndarray, y_eval: np.ndarray, name: str) -> dict[str, Any]:\n", " \"\"\"Return MAE, RMSE, and R2 for a fitted regression model.\"\"\"\n", " preds: np.ndarray = model.predict(X_eval)\n", " return {\n", " \"Model\": name,\n", " \"MAE\": float(mean_absolute_error(y_eval, preds)),\n", " \"RMSE\": float(mean_squared_error(y_eval, preds) ** 0.5),\n", " \"R2\": float(r2_score(y_eval, preds)),\n", " }" ] }, { "cell_type": "markdown", "id": "51", "metadata": {}, "source": [ "With that ready, fit the model it will judge:" ] }, { "cell_type": "code", "execution_count": null, "id": "52", "metadata": {}, "outputs": [], "source": [ "lin_reg: LinearRegression = LinearRegression()\n", "lin_reg.fit(X_train_s, y_tr)\n", "logger.success(\"LinearRegression fitted\")" ] }, { "cell_type": "markdown", "id": "53", "metadata": {}, "source": [ "Compare it against the baseline:" ] }, { "cell_type": "code", "execution_count": null, "id": "54", "metadata": {}, "outputs": [], "source": [ "reg_results: pd.DataFrame = pd.DataFrame(\n", " [\n", " evaluate_regressor(dummy_reg, X_test_s, y_te, \"DummyRegressor (mean)\"),\n", " evaluate_regressor(lin_reg, X_test_s, y_te, \"LinearRegression\"),\n", " ]\n", ")\n", "metrics_report(\n", " reg_results,\n", " metrics=[\"MAE\", \"RMSE\", \"R2\"],\n", " minimize_cols=[\"MAE\", \"RMSE\"],\n", " maximize_cols=[\"R2\"],\n", " title=\"Regression performance: ADR prediction\",\n", " subtitle=\"Test set | Hotel Booking Demand dataset\",\n", ")" ] }, { "cell_type": "markdown", "id": "55", "metadata": {}, "source": [ "€33.80 MAE against the dummy's €35.66: a real improvement, but a small one, and R2 of 0.10 says these six features explain barely a tenth of why one booking costs more than another. That's not a failed model, it's an honest one: lead time, party size, and a handful of stay-length counts were never going to price a hotel room on their own, room type and season do most of that work, and neither is in `FEATURES` yet. The coefficients show which of the six is carrying what little signal there is:" ] }, { "cell_type": "code", "execution_count": null, "id": "56", "metadata": {}, "outputs": [], "source": [ "coef_df: pd.DataFrame = (\n", " pd.DataFrame({\"feature\": FEATURES, \"coefficient\": lin_reg.coef_})\n", " .sort_values(\"coefficient\", key=abs, ascending=False)\n", " .reset_index(drop=True)\n", ")\n", "themed_gt(GT(coef_df), n_rows=len(coef_df)).tab_header(\n", " title=gt_md(\"**LinearRegression coefficients**\"),\n", " subtitle=\"Standardised features: magnitude reflects relative importance\",\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "57", "metadata": {}, "outputs": [], "source": [ "coef_plot: pd.DataFrame = coef_df.assign(\n", " color=lambda d: d[\"coefficient\"].apply(lambda v: \"positive\" if v >= 0 else \"negative\")\n", ")\n", "(\n", " ggplot(coef_plot, aes(x=as_discrete(\"feature\", order_by=\"coefficient\", order=1), y=\"coefficient\", fill=\"color\"))\n", " + geom_bar(stat=\"identity\", alpha=0.85)\n", " + scale_fill_manual(values={\"positive\": INFO, \"negative\": DANGER})\n", " + labs(\n", " title=\"LinearRegression coefficients\",\n", " subtitle=\"Standardised features: positive raises ADR, negative lowers it\",\n", " x=\"\",\n", " y=\"Coefficient\",\n", " )\n", " + modern_theme(x_axis_angle=30, legend_pos=\"none\")\n", ")" ] }, { "cell_type": "markdown", "id": "58", "metadata": {}, "source": [ "More adults pushes the price up (+14.01, the largest effect by far, a bigger room for a bigger party), and a longer lead time pulls it down (-6.00): book far enough ahead and you tend to catch an early, lower rate. A residual plot checks whether the model is missing something systematic, or just noisy in a way six weak features would predict:" ] }, { "cell_type": "code", "execution_count": null, "id": "59", "metadata": {}, "outputs": [], "source": [ "y_pred_lin: np.ndarray = lin_reg.predict(X_test_s)\n", "resid_df: pd.DataFrame = pd.DataFrame({\"predicted\": y_pred_lin, \"residual\": y_te - y_pred_lin})\n", "(\n", " ggplot(resid_df, aes(x=\"predicted\", y=\"residual\"))\n", " + geom_point(color=INFO, alpha=0.12, size=1)\n", " + geom_hline(yintercept=0, color=DANGER, linetype=\"dashed\", size=1)\n", " + labs(\n", " title=\"Residuals vs predicted ADR\",\n", " subtitle=\"Random scatter around zero: no systematic bias, just weak signal\",\n", " x=\"Predicted ADR (euros)\",\n", " y=\"Residual (actual minus predicted)\",\n", " )\n", " + modern_theme(grid=True)\n", ")" ] }, { "cell_type": "markdown", "id": "60", "metadata": {}, "source": [ "No curve, no funnel, just a wide, roughly even scatter. The model isn't systematically wrong about any particular price range, it's just working with features that don't carry enough information to narrow the scatter further. That's a feature problem, not a modelling-technique problem, the same lesson [Case Study 1](../05-classical-ml-projects/33-spend-regression.ipynb#model-selection) makes with real dollar figures on a different dataset.\n", "\n", "Cancellation gets `LogisticRegression`, with one adjustment `LinearRegression` didn't need: `class_weight=\"balanced\"` reweights the loss so the majority class (kept bookings) can't dominate the way it did for the dummy baseline. Without it, a model minimising raw error learns to predict \"not cancelled\" almost everywhere, for the same reason the dummy classifier scored zero F1." ] }, { "cell_type": "code", "execution_count": null, "id": "61", "metadata": {}, "outputs": [], "source": [ "log_reg: LogisticRegression = LogisticRegression(class_weight=\"balanced\", max_iter=1000, random_state=42)\n", "log_reg.fit(Xc_train_s, yc_tr)\n", "logger.success(\"LogisticRegression fitted\")" ] }, { "cell_type": "markdown", "id": "62", "metadata": {}, "source": [ "See how it actually performs:" ] }, { "cell_type": "code", "execution_count": null, "id": "63", "metadata": {}, "outputs": [], "source": [ "yc_pred: np.ndarray = log_reg.predict(Xc_test_s)\n", "report: str = classification_report(yc_te, yc_pred, target_names=[\"Kept\", \"Cancelled\"])\n", "logger.info(\"\\n\" + report)" ] }, { "cell_type": "markdown", "id": "64", "metadata": {}, "source": [ "0.55 F1 on the cancelled class, against the dummy's 0.0. Recall of 0.54 means it catches just over half of the bookings that will actually cancel, precision of 0.56 means just over half of what it flags really does cancel. Not good enough to automate a decision on yet, but a real, measurable signal where the baseline had none." ] }, { "cell_type": "markdown", "id": "65", "metadata": {}, "source": [ "
\n", " Activity 3: Ridge vs LinearRegression

\n", "Ridge adds an L2 penalty that shrinks coefficients toward zero, which can reduce overfitting on noisier data.\n", "
from sklearn.linear_model import Ridge\n",
    "ridge = Ridge(alpha=1.0)\n",
    "ridge.fit(X_train_s, y_tr)\n",
    "# compare evaluate_regressor(ridge, ...) against lin_reg
\n", "Given R2 was already only 0.10 with plain LinearRegression, do you expect Ridge to move the number much? Check whether it does.\n", "
" ] }, { "cell_type": "markdown", "id": "66", "metadata": {}, "source": [ "## 6. Cross-validation: trusting the number more\n", "\n", "One train/test split gives one number, and that number carries whatever noise happened to land in that particular 20%. Cross-validation rotates the held-out portion across five folds, in turn: each fold takes a turn as validation while the other four train, producing five scores instead of one, and the mean plus standard deviation of those five is a far steadier estimate than any single split.\n", "\n", "Wrapping the scaler and model in a `Pipeline` first isn't optional here. Pass the model alone to `cross_val_score` and every fold's \"training\" score is still computed against a scaler fitted on all five folds at once, leaking exactly the way Section 3's Common Mistake did. `Pipeline` re-fits the scaler inside each fold automatically, [Chapter 32](32-sklearn-pipeline.ipynb) is built entirely around making that guarantee explicit." ] }, { "cell_type": "code", "execution_count": null, "id": "67", "metadata": {}, "outputs": [], "source": [ "reg_pipe: Pipeline = Pipeline([(\"scaler\", StandardScaler()), (\"model\", LinearRegression())])\n", "kf: KFold = KFold(n_splits=5, shuffle=True, random_state=42)\n", "cv_reg: np.ndarray = cross_val_score(reg_pipe, X_tr, y_tr, cv=kf, scoring=\"neg_mean_absolute_error\")\n", "logger.info(f\"CV MAE per fold: {(-cv_reg).round(2)}\")\n", "logger.info(f\"Mean +/- std: {-cv_reg.mean():.2f} +/- {cv_reg.std():.2f}\")" ] }, { "cell_type": "markdown", "id": "68", "metadata": {}, "source": [ "€33.57 ± €0.14: five folds within fifteen cents of each other. That tight a spread says the single-split MAE from Section 5 (€33.80) wasn't a lucky or unlucky split, it's a stable, reproducible number. Classification needs `StratifiedKFold` instead of plain `KFold`, for the same reason the earlier split used `stratify=y_clf`: with only 37% of bookings cancelling, an unlucky fold boundary could hand one fold almost no cancellations at all, and a metric like F1 becomes meaningless on a fold with too few positive examples to compute it from." ] }, { "cell_type": "code", "execution_count": null, "id": "69", "metadata": {}, "outputs": [], "source": [ "clf_pipe: Pipeline = Pipeline(\n", " [\n", " (\"scaler\", StandardScaler()),\n", " (\"model\", LogisticRegression(class_weight=\"balanced\", max_iter=1000, random_state=42)),\n", " ]\n", ")\n", "skf: StratifiedKFold = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n", "cv_clf: np.ndarray = cross_val_score(clf_pipe, Xc_tr, yc_tr, cv=skf, scoring=\"f1\")\n", "logger.info(f\"CV F1 per fold: {cv_clf.round(3)}\")\n", "logger.info(f\"Mean +/- std: {cv_clf.mean():.3f} +/- {cv_clf.std():.3f}\")" ] }, { "cell_type": "markdown", "id": "70", "metadata": {}, "source": [ "0.554 ± 0.003, just as tight. Both cross-validated numbers land almost exactly where the single split already put them, which is itself the useful finding: on a dataset this size, a single stratified split was already a trustworthy estimate. That won't hold on every dataset this book uses, which is exactly why the check is worth running rather than assuming." ] }, { "cell_type": "markdown", "id": "71", "metadata": {}, "source": [ "## 7. Learning curves: more data, or a better model?\n", "\n", "Section 5 left an open question: is 0.10 R2 a ceiling this feature set can't get past, or would a bigger training set push it higher? A learning curve answers this by refitting the pipeline at increasing training sizes and watching what happens to the gap between training and validation error." ] }, { "cell_type": "code", "execution_count": null, "id": "72", "metadata": {}, "outputs": [], "source": [ "train_sizes: np.ndarray\n", "train_lc_scores: np.ndarray\n", "val_lc_scores: np.ndarray\n", "train_sizes, train_lc_scores, val_lc_scores = learning_curve(\n", " reg_pipe,\n", " X_tr,\n", " y_tr,\n", " train_sizes=np.linspace(0.1, 1.0, 8),\n", " cv=KFold(n_splits=5, shuffle=True, random_state=42),\n", " scoring=\"neg_mean_absolute_error\",\n", " n_jobs=-1,\n", ")\n", "train_sizes" ] }, { "cell_type": "code", "execution_count": null, "id": "73", "metadata": {}, "outputs": [], "source": [ "lc_df: pd.DataFrame = pd.DataFrame(\n", " {\n", " \"n\": np.tile(train_sizes, 2),\n", " \"score\": np.concatenate([-train_lc_scores.mean(1), -val_lc_scores.mean(1)]),\n", " \"err\": np.concatenate([train_lc_scores.std(1), val_lc_scores.std(1)]),\n", " \"split\": [\"Train\"] * len(train_sizes) + [\"Validation\"] * len(train_sizes),\n", " }\n", ").assign(ymin=lambda d: d[\"score\"] - d[\"err\"], ymax=lambda d: d[\"score\"] + d[\"err\"])\n", "(\n", " ggplot(lc_df, aes(x=\"n\", y=\"score\", color=\"split\", fill=\"split\"))\n", " + geom_line(size=1.2)\n", " + geom_ribbon(aes(ymin=\"ymin\", ymax=\"ymax\"), alpha=0.15, color=None)\n", " + scale_color_manual(values={\"Train\": INFO, \"Validation\": WARNING})\n", " + scale_fill_manual(values={\"Train\": INFO, \"Validation\": WARNING})\n", " + labs(\n", " title=\"Learning curve: LinearRegression on ADR\",\n", " subtitle=\"Flat from the smallest training size tested: a features problem, not a data problem\",\n", " x=\"Training set size\",\n", " y=\"MAE (lower is better)\",\n", " )\n", " + modern_theme(grid=True, legend_pos=\"bottom\")\n", ")" ] }, { "cell_type": "markdown", "id": "74", "metadata": {}, "source": [ "Training and validation MAE sit within a few cents of each other at every size tested, including the smallest, 7,515 rows. There's no gap to close: this isn't a model straining against too little data, it's a model that hit its ceiling almost immediately and stayed there while the training set grew tenfold. More bookings would not move this number. A feature that actually explains price, room type, season, how far in advance the rate was locked in, would. That's the same conclusion Section 5's weak R2 already pointed to, confirmed a second way." ] }, { "cell_type": "markdown", "id": "75", "metadata": {}, "source": [ "
\n", " Activity 4: learning curve for LogisticRegression

\n", "Compute and plot a learning curve for clf_pipe using scoring=\"f1\" and StratifiedKFold in place of KFold. Does the cancellation classifier show the same flat, converged shape, or is there still a gap between train and validation that more data might close?\n", "
" ] }, { "cell_type": "markdown", "id": "76", "metadata": {}, "source": [ "## 8. Saving what you built\n", "\n", "A fitted model is only half of what production needs. Predict on raw, unscaled input with a model trained on scaled input and you get a number, just not a meaningful one, the model has no way to tell you it was handed the wrong kind of input. The scaler and the model have to travel together, saved as one bundle, not two files someone has to remember to load in the right order." ] }, { "cell_type": "code", "execution_count": null, "id": "77", "metadata": {}, "outputs": [], "source": [ "MODEL_DIR.mkdir(exist_ok=True)\n", "model_path: Path = MODEL_DIR / \"lin_reg_adr.joblib\"\n", "bundle: dict[str, Any] = {\"scaler\": scaler, \"model\": lin_reg, \"features\": FEATURES, \"target\": TARGET_REG}\n", "joblib.dump(bundle, model_path)\n", "logger.success(f\"Bundle saved to {model_path}\")" ] }, { "cell_type": "markdown", "id": "78", "metadata": {}, "source": [ "Reload it in a fresh call to confirm the bundle actually works end to end:" ] }, { "cell_type": "code", "execution_count": null, "id": "79", "metadata": {}, "outputs": [], "source": [ "loaded: dict[str, Any] = joblib.load(model_path)\n", "sample: np.ndarray = X_te[:1]\n", "sample_scaled: np.ndarray = loaded[\"scaler\"].transform(sample)\n", "predicted: float = float(loaded[\"model\"].predict(sample_scaled)[0])\n", "actual: float = float(y_te[0])\n", "logger.info(f\"Predicted ADR: {predicted:.2f} Actual ADR: {actual:.2f}\")" ] }, { "cell_type": "markdown", "id": "80", "metadata": {}, "source": [ "
\n", " Pro Tip: save the scaler inside the bundle, never separately

\n", "A model evaluated on unscaled inputs in production is one of the most common silent bugs in deployed ML systems: it doesn't crash, it just predicts wrong, quietly, at whatever confidence it would have had on correctly scaled data. Bundling scaler and model as one object removes the chance to load one without the other.\n", "
" ] }, { "cell_type": "markdown", "id": "81", "metadata": {}, "source": [ "## Packaging the pattern for reuse\n", "\n", "Split, scale, fit, evaluate, save: the same five steps ran twice in this chapter, once for ADR and once for cancellation, with only the model and the metric changing. Writing it once as a typed function isn't just tidier, it's the exact shape [Chapter 32](32-sklearn-pipeline.ipynb) inherits and formalises into a `Pipeline` object." ] }, { "cell_type": "code", "execution_count": null, "id": "82", "metadata": {}, "outputs": [], "source": [ "def train_and_evaluate(\n", " df: pd.DataFrame,\n", " features: list[str],\n", " target: str,\n", " task: str,\n", " out_dir: Path = Path(\"models\"),\n", ") -> dict[str, Any]:\n", " \"\"\"Run the full train-evaluate-save workflow for one task.\"\"\"\n", " X_all: np.ndarray = df[features].to_numpy(dtype=float)\n", " y_all: np.ndarray = df[target].to_numpy()\n", "\n", " stratify: np.ndarray | None = y_all if task == \"classification\" else None\n", " Xtr, Xte, ytr, yte = train_test_split(X_all, y_all, test_size=0.2, random_state=42, stratify=stratify)\n", " sc: StandardScaler = StandardScaler()\n", " Xtr_s: np.ndarray = sc.fit_transform(Xtr)\n", " Xte_s: np.ndarray = sc.transform(Xte)\n", "\n", " if task == \"regression\":\n", " model: Any = LinearRegression()\n", " baseline: Any = DummyRegressor(strategy=\"mean\")\n", " else:\n", " model = LogisticRegression(class_weight=\"balanced\", max_iter=1000, random_state=42)\n", " baseline = DummyClassifier(strategy=\"most_frequent\")\n", "\n", " baseline.fit(Xtr_s, ytr)\n", " model.fit(Xtr_s, ytr)\n", "\n", " if task == \"regression\":\n", " m: dict[str, float] = {\n", " \"baseline_mae\": float(mean_absolute_error(yte, baseline.predict(Xte_s))),\n", " \"model_mae\": float(mean_absolute_error(yte, model.predict(Xte_s))),\n", " \"model_r2\": float(r2_score(yte, model.predict(Xte_s))),\n", " }\n", " else:\n", " m = {\n", " \"baseline_f1\": float(f1_score(yte, baseline.predict(Xte_s), zero_division=0)),\n", " \"model_f1\": float(f1_score(yte, model.predict(Xte_s))),\n", " }\n", "\n", " out_dir.mkdir(exist_ok=True)\n", " save_path: Path = out_dir / f\"{target}.joblib\"\n", " joblib.dump({\"scaler\": sc, \"model\": model, \"features\": features, \"target\": target}, save_path)\n", " logger.success(f\"Saved {task} bundle for '{target}' to {save_path}\")\n", " logger.info(f\"Metrics: {m}\")\n", " return {\"model\": model, \"scaler\": sc, \"metrics\": m}" ] }, { "cell_type": "code", "execution_count": null, "id": "83", "metadata": {}, "outputs": [], "source": [ "reg_result: dict[str, Any] = train_and_evaluate(df, FEATURES, TARGET_REG, task=\"regression\", out_dir=MODEL_DIR)" ] }, { "cell_type": "code", "execution_count": null, "id": "84", "metadata": {}, "outputs": [], "source": [ "clf_result: dict[str, Any] = train_and_evaluate(df, FEATURES, TARGET_CLF, task=\"classification\", out_dir=MODEL_DIR)" ] }, { "cell_type": "markdown", "id": "85", "metadata": {}, "source": [ "Neither model is ready to run a hotel on. The regression model can't yet explain the price a room should carry, only rank it weakly against six thin proxies, missing room type and season entirely. The classifier catches barely half of the bookings that will cancel. Both are honest first attempts, and both point at the same next move: better features, not a fancier algorithm on the same six columns, the exact lesson [Case Study 1](../05-classical-ml-projects/33-spend-regression.ipynb#model-selection) proves with a dollar figure attached later in this book.\n", "\n", "What's solid is the shape: a Transformer and a Predictor, fit only on training data, evaluated against a baseline, cross-validated instead of trusted on one split, and saved as a single bundle. Six fit calls ran across this chapter, scaler and model, twice over, plus a third pair inside the capstone function. Six places a leak or a mismatched scaler could have crept in. [Chapter 32](32-sklearn-pipeline.ipynb) collapses all of it into one object that makes those six calls impossible to get out of order." ] }, { "cell_type": "markdown", "id": "86", "metadata": {}, "source": [ "## Further reading\n", "\n", "- [Scikit-learn user guide](https://scikit-learn.org/stable/user_guide.html)\n", "- Geron, A. (2022), *Hands-On Machine Learning*, Ch 2-4\n", "- Antonio et al. (2019), *Hotel booking demand datasets*, Data in Brief" ] }, { "cell_type": "markdown", "id": "87", "metadata": {}, "source": [ "## Summary\n", "\n", "| Concept | Key rule |\n", "|---|---|\n", "| Transformer | Has `fit` and `transform`; never sees a target; e.g. `StandardScaler`, `OneHotEncoder` |\n", "| Predictor | Has `fit(X, y)` and `predict(X)`; learns a mapping to a known answer |\n", "| Split before scaling | Fit a scaler (or any transformer) on training data only; fitting before the split leaks the test distribution into the transformation |\n", "| Baseline first | `DummyRegressor`/`DummyClassifier` show what \"no model\" scores; a real model has to beat that number, not just produce one |\n", "| `class_weight=\"balanced\"` | Reweights the loss so a majority class can't win by default; without it, an imbalanced target gets predicted as one class almost everywhere |\n", "| Cross-validation | Rotates the held-out fold across five splits; mean +/- std is a steadier estimate than any single split's score |\n", "| Learning curve | A flat train/validation gap at every size says more data won't help; the ceiling is the features, not the sample size |\n", "| Model bundle | Save the scaler and the model together; a model scored on unscaled input predicts wrong without ever raising an error |\n", "\n", "**Next:** Chapter 32 wraps every one of these separate fit calls into a single Pipeline object, and grows the six-feature model here into fourteen columns across two treatments." ] } ], "metadata": { "kernelspec": { "display_name": "ark (3.12.12.final.0)", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 5 }