{ "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "---\n", "title: \"Chapter 33: Predicting household spend\"\n", "---" ] }, { "cell_type": "markdown", "id": "1", "metadata": {}, "source": [ "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/sambaiga/ds-mlops-path/blob/main/tutorials/05-classical-ml-projects/33-spend-regression.ipynb) [![Download Notebook](https://img.shields.io/badge/Download-Notebook-blue.svg?logo=jupyter&logoColor=white)](https://raw.githubusercontent.com/sambaiga/ds-mlops-path/main/tutorials/05-classical-ml-projects/33-spend-regression.ipynb)" ] }, { "cell_type": "markdown", "id": "2", "metadata": {}, "source": [ "In November 2019, a single-person household in Alabama, reporting an income between $25,000 and $49,999, bought a 65-inch Samsung 4K television, a PlayStation 4 Pro, a Microsoft Surface tablet, an Apple Watch, a GoPro camera bundle, a trampoline, and a kayak. Fifty-eight items, spread across four days. The total came to $11,709.16.\n", "\n", "For the three months before that, the same household had spent an average of $49.47 a month. Across the roughly four years this account appears in the data, its lifetime total is $16,237.77. One month accounts for 72 percent of everything it has ever bought on the platform.\n", "\n", "Nothing in this household's survey answers, age bracket, income, education, state, explains it. There's no flag for it and no obvious trigger. It's one row among 1.85 million, sitting in the data whether or not anyone thinks to look for it." ] }, { "cell_type": "markdown", "id": "3", "metadata": {}, "source": [ "Chapters 28 through 32 gave you the machinery for exactly this kind of question: how to turn a vague business problem into [a five-component specification](../04-ml-intro/30-ml-workflow.qmd#translating-a-business-problem-into-a-machine-learning-problem), how [the nine-stage workflow](../04-ml-intro/30-ml-workflow.qmd#the-end-to-end-machine-learning-workflow) moves data through a model and into production, how scikit-learn's [Transformer and Predictor interfaces](../04-ml-intro/31-sklearn-core.ipynb) work, how [a Pipeline strings them together](../04-ml-intro/32-sklearn-pipeline.ipynb) without leaking information across a split. You used all of it on hotel booking data that was already clean enough to teach with.\n", "\n", "Here, you use it on a dataset nobody cleaned for teaching: five years of real Amazon purchase histories from 5,027 U.S. households, linked to a demographic and lifestyle survey each household completed once. The household above is what that data looks like up close. It's also the first of seven case studies built the same way, one real dataset and one real business question each, so the habit you build in this chapter (reason through the ask, prototype in the notebook, promote what works into a tested module) is the one the other six will follow." ] }, { "cell_type": "markdown", "id": "4", "metadata": {}, "source": [ "::: {.callout-note collapse=\"true\" icon=false}\n", "## Learning objectives\n", "\n", "By the end of this chapter you will be able to:\n", "\n", "| # | Skill | Covered in |\n", "|---|---|---|\n", "| 1 | Turn a vague business ask into a five-component ML specification through your own follow-up questions | Sec. 1 |\n", "| 2 | Build a tested, cached data-download utility instead of a one-off script | Sec. 2 |\n", "| 3 | Let exploratory checks be driven by the spec card instead of habit | Sec. 3 |\n", "| 4 | Engineer temporally-safe features and catch a leakage bug with a purpose-built test | Sec. 4 |\n", "| 5 | Demonstrate the Evaluation-to-Feature-Engineering feedback loop with an honest ablation | Sec. 5 |\n", "| 6 | Diagnose a model with a learning curve and permutation importance | Sec. 6 |\n", "| 7 | Attach a back-of-envelope dollar value to a model's metric before calling it done | Sec. 7 |\n", ":::" ] }, { "cell_type": "markdown", "id": "5", "metadata": {}, "source": [ "## 1. Problem framing\n", "\n", "Here's the request, exactly as it would actually arrive. You're the data scientist on a retention team at a company sitting on data like this, purchase histories linked to a demographic survey, for a panel of customers. Your manager stops by your desk and says:\n", "\n", "\"We want to know which customers are worth investing more in.\"\n", "\n", "That's the whole brief. No target variable, no metric, no mention of a model. This is what a real ask sounds like, and your job right now is to figure out what it actually means, not to open a notebook." ] }, { "cell_type": "markdown", "id": "6", "metadata": {}, "source": [ "### What does \"worth investing in\" even mean?\n", "\n", "Before you touch a single file, you need to pull that sentence apart. It's collapsing at least four different problems into one, and they are not interchangeable:\n", "\n", "- Customers who already spend the most. A ranking of the past, nothing predictive about it.\n", "- Customers likely to spend more going forward. A forecast.\n", "- Customers likely to leave if nobody intervenes. A churn problem.\n", "- Customers who look like existing high spenders but haven't been identified yet. A segmentation problem.\n", "\n", "Each one is a legitimate data science problem. Each one produces a different target variable, a different model, and a different action the business takes on your output. Guess wrong here and you could spend six months building the right system for the wrong question." ] }, { "cell_type": "markdown", "id": "7", "metadata": {}, "source": [ "You go back and ask a follow-up question. It turns out \"worth investing in\" means deciding who gets priority for a loyalty program with a fixed, limited budget. The team wants customers ranked by how much they're likely to spend going forward, using the demographic and behavioural data already on file, before that spending happens. Not a scoreboard of who has already spent the most.\n", "\n", "That answer resolves the first fork: this is a forecast of future value, not a description of past value, which rules out simply sorting by historical total. It also starts pinning down [Chapter 30's five-component spec](../04-ml-intro/30-ml-workflow.qmd#translating-a-business-problem-into-a-machine-learning-problem), one piece at a time, following wherever your own follow-up questions lead next:" ] }, { "cell_type": "markdown", "id": "8", "metadata": {}, "source": [ "
\n", " Activity 1: write the spec card before reading further

\n", "Using [Chapter 30's five-component spec](../04-ml-intro/30-ml-workflow.qmd#translating-a-business-problem-into-a-machine-learning-problem) (target variable, task type and paradigm, success metric, acceptable threshold, data specification), write your own answer for this problem before reading the discussion below. Compare where you landed once you get there. If your target variable differs, that's fine, it's the kind of decision [Chapter 30's Pro Tip](../04-ml-intro/30-ml-workflow.qmd#translating-a-business-problem-into-a-machine-learning-problem) already warned about: agree it with the stakeholder who will actually use the model, because a model optimised toward the wrong objective will hit that objective and still miss the point.\n", "
" ] }, { "cell_type": "markdown", "id": "9", "metadata": {}, "source": [ "**Target variable.** Not last month's spend, what you're predicting is next year's. You still need a window, a month, a year, a lifetime span all answer the question differently, and the right one depends on how often the loyalty program actually reallocates its budget. If the business revisits the list annually, that settles it: a household's spend over the coming twelve months.\n", "\n", "**Task type and paradigm.** You're predicting a number from labelled historical examples where the outcome is already known. That's regression, supervised.\n", "\n", "**Success metric.** The team isn't going to read predictions one row at a time, they're going to act on a ranked list. So the metric that matters is whether your model orders households correctly relative to each other, and whether the top decile it flags genuinely outspends everyone else. Mean absolute error is still worth tracking, mostly as a sanity check that the numbers themselves aren't nonsense.\n", "\n", "**Acceptable threshold.** A model that can't beat \"assume everyone spends the average\" isn't worth shipping. Here's the real bar: households you rank in the top decile need to outspend a randomly chosen household by enough that the loyalty program's cost per member actually pays for itself, a number the business has to hand you, you can't derive it from the data alone.\n", "\n", "**Data specification.** One demographic and lifestyle survey per household, answered once, and multi-year purchase history at the individual-transaction level, for 5,027 households." ] }, { "cell_type": "markdown", "id": "10", "metadata": {}, "source": [ "::: {.callout-note collapse=\"true\" icon=false}\n", "## The resolved spec card\n", "\n", "| Component | Specification |\n", "|---|---|\n", "| Target variable | Household's total spend over the coming 12 months |\n", "| Task type and paradigm | Regression, supervised |\n", "| Success metric | Rank quality (does the top decile of predicted spend actually outspend the rest), MAE as a secondary check |\n", "| Acceptable threshold | Top-decile households must outspend a randomly chosen household by a margin the loyalty program's budget can justify |\n", "| Data specification | One-time demographic and lifestyle survey plus multi-year, transaction-level purchase history, 5,027 U.S. households |\n", ":::\n", "\n", "That's the spec, and it took five follow-up questions to get there, not a target variable declared in the first paragraph. Every one of those questions is something a real stakeholder conversation would actually raise, and you'll ask versions of them again in every case study that follows.\n", "\n", "Next: turning this into something durable enough that the rest of the project can't quietly drift from it, a Pydantic object, not a paragraph you could reinterpret three sections from now." ] }, { "cell_type": "markdown", "id": "11", "metadata": {}, "source": [ "### Making the decision durable\n", "\n", "Right now the spec card is a table in a notebook. Nothing stops the next person, or you, in three weeks, from reading \"predicted spend\" and quietly building a model for last month's spend instead, because nothing enforces which one was actually decided. A Pydantic object does. Once `settings.spec.target_variable` is a real attribute, every later section imports it instead of re-reading a paragraph and hoping it remembers correctly.\n", "\n", "This is also where the practical decisions live, the ones that don't belong in a business spec but still need a fixed answer somewhere: which Harvard Dataverse files this pulls from, where they get cached, and which survey questions never make it into a feature, no matter how predictive they turn out to be." ] }, { "cell_type": "code", "execution_count": null, "id": "12", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "from pydantic import BaseModel\n", "from pydantic_settings import BaseSettings, SettingsConfigDict\n", "\n", "\n", "class SpecCard(BaseModel):\n", " \"\"\"The five-component spec resolved above, kept as data, not prose.\"\"\"\n", "\n", " target_variable: str = \"Household's total spend over the coming 12 months\"\n", " task_type: str = \"Regression, supervised\"\n", " success_metric: str = \"Rank quality (top-decile lift over random), MAE as a secondary check\"\n", " acceptable_threshold: str = (\n", " \"Top-decile households must outspend a random household by a margin the loyalty program's budget can justify\"\n", " )\n", " data_specification: str = (\n", " \"One-time demographic and lifestyle survey plus multi-year, \"\n", " \"transaction-level purchase history, 5,027 U.S. households\"\n", " )\n", "\n", "\n", "class SpendRegressionSettings(BaseSettings):\n", " \"\"\"Paths, source data, and policy decisions for this case study.\"\"\"\n", "\n", " model_config = SettingsConfigDict(env_file=\".env\", env_prefix=\"SPEND_\", case_sensitive=False)\n", "\n", " spec: SpecCard = SpecCard()\n", "\n", " data_dir: Path = Path(\"data\")\n", " dataverse_base_url: str = \"https://dataverse.harvard.edu/api/access/datafile\"\n", " purchases_file_id: int = 7616235 # amazon-purchases.csv, ~298.5 MB\n", " survey_file_id: int = 7616231 # survey.csv, tab-separated despite the .csv extension\n", "\n", " random_seed: int = 42\n", "\n", " excluded_survey_columns: list[str] = [\n", " \"Q-substance-use-cigarettes\",\n", " \"Q-substance-use-marijuana\",\n", " \"Q-substance-use-alcohol\",\n", " \"Q-personal-diabetes\",\n", " \"Q-personal-wheelchair\",\n", " \"Q-sexual-orientation\",\n", " ]\n", "\n", "\n", "settings = SpendRegressionSettings()\n", "settings.spec.target_variable" ] }, { "cell_type": "code", "execution_count": null, "id": "13", "metadata": {}, "outputs": [], "source": [ "from great_tables import GT, md as gt_md\n", "import pandas as pd\n", "\n", "from ark.plot.gt_style import themed_gt\n", "\n", "LABELS = {\n", " \"target_variable\": \"Target variable\",\n", " \"task_type\": \"Task type and paradigm\",\n", " \"success_metric\": \"Success metric\",\n", " \"acceptable_threshold\": \"Acceptable threshold\",\n", " \"data_specification\": \"Data specification\",\n", "}\n", "spec_df = pd.DataFrame([{\"Component\": LABELS[k], \"Specification\": v} for k, v in settings.spec.model_dump().items()])\n", "themed_gt(GT(spec_df), n_rows=len(spec_df)).tab_header(\n", " title=gt_md(\"**The spec card, as code**\"),\n", " subtitle=\"Generated from settings.spec, not retyped from the table above\",\n", ")" ] }, { "cell_type": "markdown", "id": "14", "metadata": {}, "source": [ "Same five rows as the resolved spec card earlier in this chapter. The difference is where they came from: that one was a markdown table anyone could edit without noticing, this one is read straight off the object the rest of the pipeline will actually import.\n", "\n", "This is what \"promoted\" means, and it's worth being explicit since nothing here does it automatically. `ark` isn't a third-party library, it's [this book's own package](../01-python-basics/16-data-storytelling.ipynb#from-default-to-branded), the same one behind `ark.plot.theme` and `ark.plot.gt_style` every chart and table in this chapter already uses. `ark/case_studies/spend_regression/` is a new corner of that same package, created for this case study specifically, and every future one will get its own folder next to it. Promoting code means exactly what it sounds like: once something written in a notebook has been tried and works, a person opens the real file, `ark/case_studies/spend_regression/config.py` in this case, and puts the working code there by hand. There's no button for it, and nothing enforces that the notebook and the file stay in sync except doing it carefully each time, which is what the rest of this chapter does out loud, one function at a time.\n", "\n", "The two classes above now live in that file, word for word. Proving that means checking the values survive the move, then using the real, imported class for everything from here on, not the copy defined above:" ] }, { "cell_type": "code", "execution_count": null, "id": "15", "metadata": {}, "outputs": [], "source": [ "before_spec = settings.spec.model_dump()\n", "before_exclusions = settings.excluded_survey_columns\n", "\n", "from ark.case_studies.spend_regression.config import SpendRegressionSettings\n", "\n", "settings = SpendRegressionSettings()\n", "settings.spec.model_dump() == before_spec and settings.excluded_survey_columns == before_exclusions" ] }, { "cell_type": "markdown", "id": "16", "metadata": {}, "source": [ "No test file for this one. Pydantic already validates the structure on construction, and there's no logic here yet worth a regression test, just decisions. That changes at the next step: downloading the actual data is where something can genuinely go wrong (a bad URL, a schema drift, a file Harvard renames), and where this chapter writes its first real test." ] }, { "cell_type": "markdown", "id": "17", "metadata": {}, "source": [ "## 2. Data collection\n", "\n", "The data specification says a one-time survey plus multi-year purchase history, for 5,027 households, hosted on Harvard Dataverse. Downloading a file isn't a data science problem, it's infrastructure, so you're not building it by hand in this notebook. It's a utility: written once, tested, imported everywhere it's needed, the same way you'd reach for a library function instead of re-deriving it." ] }, { "cell_type": "code", "execution_count": null, "id": "18", "metadata": {}, "outputs": [], "source": [ "from ark.case_studies.spend_regression.data import load_survey" ] }, { "cell_type": "markdown", "id": "19", "metadata": {}, "source": [ "Behind that import is `download_dataverse_file`, which handles the parts that have nothing to do with this dataset specifically and everything to do with downloading files correctly:\n", "\n", "- **Idempotent.** You won't wait through a 300 MB download twice, once a file is on disk, this skips the network entirely.\n", "- **Streamed in chunks.** Not read into memory in one call, so a large file doesn't need to fit in RAM before it's even written to disk.\n", "- **HTTPS-only, on purpose.** It refuses any URL that isn't `https://` before opening a connection. Dataverse would never hand back anything else, but you shouldn't write a function that fetches arbitrary URLs and quietly trusts whatever string it's given.\n", "- **A plain header fixes a confusing error.** Dataverse's server rejects Python's default `urllib` user agent with a 403, even though the same URL works fine in `curl` or a browser. You send a normal-looking header instead, one line, and the \"broken API\" turns out to be a request that just looked wrong.\n", "\n", "None of that is specific to spend prediction. You're looking at code worth writing once, testing, and never thinking about again. That's why it gets its own test, next." ] }, { "cell_type": "markdown", "id": "20", "metadata": {}, "source": [ "This is also the repo's first real test file, `tests/case_studies/test_spend_regression.py`, three tests that never touch the network: one confirms the cache short-circuit works without a real download, one confirms the HTTPS guard rejects a bad scheme, one confirms the tab-separator handling on a small fixture file. Run it yourself here rather than take it on faith:" ] }, { "cell_type": "code", "execution_count": null, "id": "21", "metadata": {}, "outputs": [], "source": [ "!uv run pytest ../../tests/case_studies/test_spend_regression.py -v --override-ini=addopts=\"\"" ] }, { "cell_type": "markdown", "id": "22", "metadata": {}, "source": [ "### Loading the survey\n", "\n", "Before loading it: `survey.csv` is tab-separated. Nothing about the extension warns you, and `pd.read_csv` with default settings would silently parse it as one giant column instead of raising an error, a worse failure than a crash because it looks like it worked. `load_survey` already accounts for this." ] }, { "cell_type": "code", "execution_count": null, "id": "23", "metadata": {}, "outputs": [], "source": [ "survey_df = load_survey(settings)\n", "themed_gt(GT(survey_df.head(5)), n_rows=5).tab_header(\n", " title=gt_md(\"**First 5 survey responses**\"),\n", " subtitle=f\"{survey_df.shape[1]} columns per household\",\n", ")" ] }, { "cell_type": "markdown", "id": "24", "metadata": {}, "source": [ "Scroll through the columns above and you'll find `Q-demos-age`, `Q-demos-income`, `Q-demos-education` sitting a few columns away from `Q-substance-use-cigarettes`, `Q-personal-diabetes`, `Q-personal-wheelchair`, and `Q-sexual-orientation`. One survey, one consent form, questions about spending habits next to questions that have nothing to do with them. `excluded_survey_columns` in `config.py` is what keeps the second group out of the feature set, whether they're worth excluding on predictive grounds is a question for EDA, next." ] }, { "cell_type": "markdown", "id": "25", "metadata": {}, "source": [ "## 3. EDA\n", "\n", "[Chapter 16's Datasaurus Dozen](../01-python-basics/16-data-storytelling.ipynb#the-datasaurus-dozen-always-visualize) made this point once already: never trust a summary statistic you haven't plotted. You're about to load 1.85 million purchase records, the largest dataset this book has put in front of you, and the pull is to jump straight to a model. Look first." ] }, { "cell_type": "markdown", "id": "26", "metadata": {}, "source": [ "
\n", " Activity 2: decide what EDA has to answer before you run any of it

\n", "Go back to the resolved spec card: target variable, task type, success metric, threshold, data spec. Before reading further, write down two or three things about this data you'd want to check before building a single feature, and *why* each one matters given that specific spec, not \"what EDA usually covers.\" Compare your list against the four checks below.\n", "
" ] }, { "cell_type": "markdown", "id": "27", "metadata": {}, "source": [ "::: {.callout-note collapse=\"true\" icon=false}\n", "## What this section actually checks, and why\n", "\n", "This section checks four things, each one earns its place because of something in the spec card, not because it's customary EDA:\n", "\n", "- **The shape of household spend.** The target itself. If it's skewed enough, a plain mean baseline misleads you and the choice of success metric stops being academic.\n", "- **Whether spend changed by year.** The target is a forward-looking number. If a household's history looks different depending on when it was collected, that's a fact any feature you build has to respect.\n", "- **Missingness in `Category` and `Title`.** Section 4 needs a category for every row. A blank one has to be handled on purpose, not discovered as a crash.\n", "- **Whether the excluded survey columns actually cost you anything.** Section 2 left this open. It's answerable with the same data you're about to load.\n", "\n", "\n", "If your list looked different, that's fine, the point isn't to predict this exact list, it's to arrive at EDA already knowing what question you're trying to answer instead of poking at charts until something looks interesting.\n", ":::" ] }, { "cell_type": "code", "execution_count": null, "id": "28", "metadata": {}, "outputs": [], "source": [ "from lets_plot import (\n", " LetsPlot,\n", " aes,\n", " geom_bar,\n", " geom_line,\n", " geom_point,\n", " geom_vline,\n", " ggplot,\n", " labs,\n", " scale_x_continuous,\n", ")\n", "\n", "from ark.case_studies.spend_regression.data import load_purchases\n", "from ark.plot.theme import modern_theme\n", "from ark.plot.tokens import DANGER, INFO\n", "\n", "LetsPlot.setup_html()\n", "\n", "purchases_df = load_purchases(settings)\n", "purchases_df.shape" ] }, { "cell_type": "markdown", "id": "29", "metadata": {}, "source": [ "Purchases are logged per line item, one row per thing bought. To see how spend behaves, you need one row per household first." ] }, { "cell_type": "code", "execution_count": null, "id": "30", "metadata": {}, "outputs": [], "source": [ "hh_total = purchases_df.groupby(\"Survey ResponseID\")[\"total\"].sum().rename(\"total_spend\").reset_index()\n", "(\n", " ggplot(hh_total, aes(x=\"total_spend\"))\n", " + geom_bar(stat=\"bin\", fill=INFO, alpha=0.85, bins=60)\n", " + geom_vline(xintercept=16237.77, color=DANGER, linetype=\"dashed\", size=1)\n", " + labs(\n", " title=\"Household lifetime spend\",\n", " subtitle=\"Dashed line: the household from the opening of this chapter, 85th percentile\",\n", " x=\"Total spend ($)\",\n", " y=\"Households\",\n", " )\n", " + modern_theme(grid=True)\n", ")" ] }, { "cell_type": "markdown", "id": "31", "metadata": {}, "source": [ "Median household spend across the whole dataset is $5,706.99. The mean is $8,763.40, more than half again as much, which is what you'd expect from a right-skewed distribution: most households cluster well below the mean, and a smaller group of very high spenders pulls it upward. You can see where the household from the opening lands: the 85th percentile of lifetime spend, comfortably above average, but nowhere near the top. Whatever happened in November 2019 wasn't a top spender having a big month. It was an ordinary-to-good household having an extraordinary one.\n", "\n", "This is also why Section 1's success metric was rank quality, not raw error. A skew this wide means a handful of households can swing a mean-based metric like MAE far more than they should, while barely moving where a household sits relative to the rest. You picked the metric before seeing this chart. The chart is why that choice holds up." ] }, { "cell_type": "markdown", "id": "32", "metadata": {}, "source": [ "### The COVID-era shift\n", "\n", "The household-level total hides when spending happened. Aggregate by year instead and you get a shape most of the developed world lived through directly." ] }, { "cell_type": "code", "execution_count": null, "id": "33", "metadata": {}, "outputs": [], "source": [ "purchases_df[\"year\"] = purchases_df[\"Order Date\"].dt.year\n", "yearly = (\n", " purchases_df[purchases_df[\"year\"].between(2018, 2022)]\n", " .groupby([\"Survey ResponseID\", \"year\"])[\"total\"]\n", " .sum()\n", " .reset_index()\n", " .groupby(\"year\")[\"total\"]\n", " .mean()\n", " .reset_index()\n", ")\n", "(\n", " ggplot(yearly, aes(x=\"year\", y=\"total\"))\n", " + geom_line(color=INFO, size=1.2)\n", " + geom_point(color=INFO, size=3)\n", " + scale_x_continuous(breaks=[2018, 2019, 2020, 2021, 2022], format=\"d\")\n", " + labs(\n", " title=\"Mean annual household spend, 2018-2022\",\n", " subtitle=\"2023-2024 excluded: data collection tapers off, not household behaviour\",\n", " x=\"Year\",\n", " y=\"Mean spend ($)\",\n", " )\n", " + modern_theme(grid=True)\n", ")" ] }, { "cell_type": "markdown", "id": "34", "metadata": {}, "source": [ "Mean annual spend rises every year from 2018 ($1,286) through 2021 ($2,287), nearly doubling, then holds roughly flat in 2022 ($2,204). None of this needed a model, a `groupby` and a chart found it. What it means for you as you build a spend-prediction model matters more: a household's 2018 history looks nothing like the same household's 2021 history, so the features you build in Section 4 need to account for when the data was collected, not only what was bought." ] }, { "cell_type": "markdown", "id": "35", "metadata": {}, "source": [ "### Missingness\n", "\n", "`Category` and `Title` are each missing on about 4.8% of rows, and it's almost always the same rows: 88,832 of the 89,458 records missing `Category` are also missing `Title`, over 99%. That's not data-entry noise scattered evenly across the file, you're looking at one class of purchase record, probably a discontinued listing or a third-party item Amazon never fully catalogued, that's incomplete in a consistent way." ] }, { "cell_type": "code", "execution_count": null, "id": "36", "metadata": {}, "outputs": [], "source": [ "missing_df = pd.DataFrame(\n", " {\n", " \"Column\": [\"Category\", \"Title\"],\n", " \"Missing (count)\": [purchases_df[\"Category\"].isna().sum(), purchases_df[\"Title\"].isna().sum()],\n", " \"Missing (%)\": [\n", " round(purchases_df[\"Category\"].isna().mean() * 100, 2),\n", " round(purchases_df[\"Title\"].isna().mean() * 100, 2),\n", " ],\n", " }\n", ")\n", "themed_gt(GT(missing_df), n_rows=len(missing_df)).tab_header(\n", " title=gt_md(\"**Missing values**\"),\n", " subtitle=\"Category and Title are missing on almost exactly the same rows\",\n", ")" ] }, { "cell_type": "markdown", "id": "37", "metadata": {}, "source": [ "You need a category for every row regardless of whether the source data has one. Section 4 handles that." ] }, { "cell_type": "markdown", "id": "38", "metadata": {}, "source": [ "### Does any of this actually predict spend?\n", "\n", "Section 2 left one question open for you: whether the columns `config.py` excludes, substance use, disability, sexual orientation, would have predicted spend better than the demographic columns that stayed in." ] }, { "cell_type": "markdown", "id": "39", "metadata": {}, "source": [ "
\n", " Activity 3: guess before you check

\n", "Before running the next cell, write down a guess for each of the six excluded columns: do you expect smoking status, marijuana use, alcohol use, diabetes, wheelchair use, and sexual orientation to show a real gap in mean spend, or none worth mentioning? You have one piece of evidence already, the spec card's data specification says nothing about health or identity driving spend, which is exactly the kind of assumption worth checking rather than trusting.\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "40", "metadata": {}, "outputs": [], "source": [ "hh_spend = purchases_df.groupby(\"Survey ResponseID\")[\"total\"].sum().rename(\"total_spend\")\n", "survey_with_spend = survey_df.merge(hh_spend, on=\"Survey ResponseID\", how=\"inner\")\n", "\n", "comparisons = [\n", " (\"Q-substance-use-cigarettes\", \"No\", \"Yes\"),\n", " (\"Q-substance-use-marijuana\", \"No\", \"Yes\"),\n", " (\"Q-substance-use-alcohol\", \"No\", \"Yes\"),\n", " (\"Q-personal-diabetes\", \"No\", \"Yes\"),\n", " (\"Q-personal-wheelchair\", \"No\", \"Yes\"),\n", " (\"Q-sexual-orientation\", \"heterosexual (straight)\", \"LGBTQ+\"),\n", "]\n", "rows = []\n", "for col, baseline, comparison in comparisons:\n", " baseline_mean = survey_with_spend.loc[survey_with_spend[col] == baseline, \"total_spend\"].mean()\n", " comparison_mean = survey_with_spend.loc[survey_with_spend[col] == comparison, \"total_spend\"].mean()\n", " rows.append(\n", " {\n", " \"Column\": col,\n", " \"Baseline\": f\"{baseline} (${baseline_mean:,.2f})\",\n", " \"Comparison\": f\"{comparison} (${comparison_mean:,.2f})\",\n", " \"Gap\": f\"{(comparison_mean - baseline_mean) / baseline_mean:+.2%}\",\n", " }\n", " )\n", "excluded_check = pd.DataFrame(rows)\n", "themed_gt(GT(excluded_check), n_rows=len(excluded_check)).tab_header(\n", " title=gt_md(\"**Mean household spend by excluded column**\"),\n", " subtitle=\"Checking whether the exclusion decision costs any real predictive power\",\n", ")" ] }, { "cell_type": "markdown", "id": "41", "metadata": {}, "source": [ "If you guessed \"no meaningful gap anywhere,\" you were half right, and the half where you were wrong is the more important half. Three of the six show almost no gap: alcohol use, diabetes, and using a wheelchair. Two show a real one: households where someone smokes cigarettes or uses marijuana spend noticeably less, not more, and LGBTQ+ households spend about 17% less than heterosexual ones. You're not looking at rounding errors.\n", "\n", "Which makes your exclusion decision more interesting, not less defensible. A real correlation between sexual orientation and spend doesn't mean sexual orientation causes spend, it almost certainly reflects something else these groups differ on that already has its own column, income, age, region, and a demographic proxy can pick up someone else's signal without you gaining anything a more direct feature wouldn't explain better. Even if it were a genuinely independent predictor, using a protected characteristic to price or prioritise a customer carries consequences beyond model accuracy, and [Chapter 30's spec card](../04-ml-intro/30-ml-workflow.qmd#translating-a-business-problem-into-a-machine-learning-problem) never asked what's technically permitted to use. It asked what the acceptable threshold and success metric are, not how to hit them by any means available. `excluded_survey_columns` was the right call before you had this table. Now you have evidence for it, not just a policy.\n", "\n", "Section 4 turns all of this, the skew, the year effect, the missingness, into features a model can actually use." ] }, { "cell_type": "markdown", "id": "42", "metadata": {}, "source": [ "## 4. Feature engineering\n", "\n", "The spec card's target is a household's spend over the *coming* 12 months. That word, coming, is the trap. Build a \"purchase frequency\" feature from a household's entire multi-year history, then predict \"future\" spend over a window that history already includes, and you've let the target leak into the input. The model would look uncannily accurate in this notebook and fall apart the moment you pointed it at a real future period, the exact failure Chapter 30 built its whole workflow around." ] }, { "cell_type": "markdown", "id": "43", "metadata": {}, "source": [ "
\n", " Activity 4: find the leak before it's built

\n", "A household has 50 months of purchase history. Say you compute \"average monthly spend\" as a feature and \"total spend\" as the target, both from that household's entire history. Every number would be technically correct. Write down what's wrong with the resulting model anyway, then compare your answer against the design below.\n", "
" ] }, { "cell_type": "markdown", "id": "44", "metadata": {}, "source": [ "::: {.callout-note collapse=\"true\" icon=false}\n", "## The temporal design\n", "\n", "Every feature has to come from data available before the target period starts, the same rule Chapter 30 applied to train/test splits, just applied one level earlier, to how the target itself gets defined.\n", "\n", "For each household: the first 12 months of its history become the feature window (demographics plus early purchase behaviour), and the next 12 months become the target window (the number you're actually predicting). A household with 50 months of history uses only its first 24, months 25 onward are simply never touched by this model. That's not wasted data, it's the difference between a feature and a leak.\n", "\n", "Not every household has a full year of purchases on both sides of that boundary. Checking directly: 4,778 of 5,027 households (95%) have at least one purchase in both windows and are usable. The other 5% are dropped, not imputed, a household that stopped buying before month 13 doesn't have a \"coming 12 months\" to predict.\n", ":::" ] }, { "cell_type": "markdown", "id": "45", "metadata": {}, "source": [ "Here's the shape of what the rest of this section builds toward, before any code: many purchase rows plus one survey row, collapsing into a single feature row per household.\n", "\n", "![Many purchase rows and one survey response collapsing into a single feature-matrix row for one household: early_spend, n_purchases, n_categories, nine demographic columns, and target_spend drawn from a separate, later time window.](figs/household-aggregation.svg){fig-alt=\"Diagram showing purchase records in a green feature-window box and a survey response in a blue box, both arrows converging via groupby on Survey ResponseID into a single dark output box listing early_spend, n_purchases, n_categories, nine demographic columns, and target_spend.\"}" ] }, { "cell_type": "markdown", "id": "46", "metadata": {}, "source": [ "### The target: next-12-months spend\n", "\n", "Every household's clock starts at its own first purchase, not a calendar date, two households can both be \"in year one\" while being years apart on the actual timeline. Days since first purchase is the anchor everything else is measured against." ] }, { "cell_type": "code", "execution_count": null, "id": "47", "metadata": {}, "outputs": [], "source": [ "first_purchase = purchases_df.groupby(\"Survey ResponseID\")[\"Order Date\"].transform(\"min\")\n", "purchases_df[\"days_since_first\"] = (purchases_df[\"Order Date\"] - first_purchase).dt.days\n", "\n", "feature_window = purchases_df[purchases_df[\"days_since_first\"] < 365]\n", "target_window = purchases_df[purchases_df[\"days_since_first\"].between(365, 729)]\n", "\n", "household_target = target_window.groupby(\"Survey ResponseID\")[\"total\"].sum().rename(\"target_spend\")\n", "household_target.shape" ] }, { "cell_type": "markdown", "id": "48", "metadata": {}, "source": [ "
\n", " Activity 5: keep only the households you can actually score

\n", "feature_window right now includes every household that ever made a purchase, but household_target only has an entry for the 4,778 with activity in months 13-24. Filter feature_window down to only the households present in household_target's index, before building a single feature from it:\n", "
feature_window = feature_window[...]
\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "49", "metadata": {}, "outputs": [], "source": [ "feature_window = feature_window[feature_window[\"Survey ResponseID\"].isin(household_target.index)]\n", "\n", "target_df = pd.DataFrame(\n", " {\n", " \"min\": [household_target.min()],\n", " \"median\": [household_target.median()],\n", " \"mean\": [household_target.mean()],\n", " \"max\": [household_target.max()],\n", " \"n households\": [household_target.shape[0]],\n", " }\n", ").round(2)\n", "themed_gt(GT(target_df), n_rows=1).tab_header(\n", " title=gt_md(\"**Target: spend in months 13-24**\"),\n", " subtitle=\"4,778 households, the same right skew as Section 3's lifetime-spend chart\",\n", ")" ] }, { "cell_type": "markdown", "id": "50", "metadata": {}, "source": [ "### Behavioural features from the feature window\n", "\n", "Three numbers per household, all computed only from the first 365 days, the window you already committed to: how much they spent, how often they bought, and how many different kinds of things." ] }, { "cell_type": "code", "execution_count": null, "id": "51", "metadata": {}, "outputs": [], "source": [ "early_spend = feature_window.groupby(\"Survey ResponseID\")[\"total\"].sum().rename(\"early_spend\")\n", "n_purchases = feature_window.groupby(\"Survey ResponseID\").size().rename(\"n_purchases\")\n", "early_spend.shape, n_purchases.shape" ] }, { "cell_type": "markdown", "id": "52", "metadata": {}, "source": [ "Category diversity looks like one more `.groupby(...).nunique()` call. Try the obvious version first:" ] }, { "cell_type": "code", "execution_count": null, "id": "53", "metadata": {}, "outputs": [], "source": [ "naive_categories = feature_window.groupby(\"Survey ResponseID\")[\"Category\"].nunique()\n", "naive_categories.head(3)" ] }, { "cell_type": "markdown", "id": "54", "metadata": {}, "source": [ "That ran without error, which is exactly the problem.\n", "\n", "
\n", " Common Mistake: nunique() silently drops missing values

\n", "Series.nunique() excludes NaN by default. Section 3 found that Category is missing on about 4.8% of rows, not an edge case, a purchase with no category recorded is still a real purchase. A household whose only \"unusual\" item was one of those missing-category rows gets undercounted by exactly one category, and nothing in the output tells you it happened.\n", "
" ] }, { "cell_type": "markdown", "id": "55", "metadata": {}, "source": [ "
\n", " Activity 6: fix the undercount

\n", "Fill the missing values in Category with a placeholder before counting, so a missing category becomes its own countable group instead of vanishing:\n", "
fixed_categories = feature_window.assign(Category=...).groupby(\"Survey ResponseID\")[\"Category\"].nunique()
\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "56", "metadata": {}, "outputs": [], "source": [ "fixed_categories = (\n", " feature_window.assign(Category=feature_window[\"Category\"].fillna(\"UNKNOWN\"))\n", " .groupby(\"Survey ResponseID\")[\"Category\"]\n", " .nunique()\n", " .rename(\"n_categories\")\n", ")\n", "n_undercounted = (fixed_categories - naive_categories.reindex(fixed_categories.index).fillna(0)).gt(0).sum()\n", "n_undercounted" ] }, { "cell_type": "markdown", "id": "57", "metadata": {}, "source": [ "3,906 of 4,778 households, over 80%, were undercounted by the naive version. Not a rare edge case, the default behaviour of a pandas method almost everyone reaches for without checking." ] }, { "cell_type": "code", "execution_count": null, "id": "58", "metadata": {}, "outputs": [], "source": [ "purchase_features = pd.concat([early_spend, n_purchases, fixed_categories], axis=1)\n", "themed_gt(GT(purchase_features.describe().round(2).reset_index()), n_rows=8).tab_header(\n", " title=gt_md(\"**Behavioural features, summary statistics**\"),\n", " subtitle=\"4,778 households, computed only from days 0-364\",\n", ")" ] }, { "cell_type": "markdown", "id": "59", "metadata": {}, "source": [ "### Adding demographics\n", "\n", "`survey_df` has 23 columns. Six are already in `settings.excluded_survey_columns`, and several more, life changes, willingness to sell data, whether the respondent works in a small business, are survey-methodology questions, not demographics or lifestyle in any sense the spec card meant. The feature list names exactly what's relevant, rather than subtracting a blocklist and hoping nothing irrelevant slips through." ] }, { "cell_type": "code", "execution_count": null, "id": "60", "metadata": {}, "outputs": [], "source": [ "DEMOGRAPHIC_COLUMNS = [\n", " \"Q-demos-age\",\n", " \"Q-demos-hispanic\",\n", " \"Q-demos-race\",\n", " \"Q-demos-education\",\n", " \"Q-demos-income\",\n", " \"Q-demos-gender\",\n", " \"Q-demos-state\",\n", " \"Q-amazon-use-hh-size\",\n", " \"Q-amazon-use-how-oft\",\n", "]\n", "if set(DEMOGRAPHIC_COLUMNS) & set(settings.excluded_survey_columns):\n", " raise ValueError(\"a demographic column overlaps the exclusion list, remove it\")\n", "DEMOGRAPHIC_COLUMNS" ] }, { "cell_type": "markdown", "id": "61", "metadata": {}, "source": [ "That check isn't decorative. `DEMOGRAPHIC_COLUMNS` doesn't currently include anything from the exclusion list, but \"currently\" is doing the work in that sentence. It fails loudly the moment someone adds `Q-sexual-orientation` back in without checking, instead of silently training on it." ] }, { "cell_type": "code", "execution_count": null, "id": "62", "metadata": {}, "outputs": [], "source": [ "demographics = survey_df.set_index(\"Survey ResponseID\")[DEMOGRAPHIC_COLUMNS]\n", "\n", "feature_matrix = purchase_features.join(demographics, how=\"inner\").join(household_target, how=\"inner\")\n", "feature_matrix.shape" ] }, { "cell_type": "markdown", "id": "63", "metadata": {}, "source": [ "Every join used `how=\"inner\"`, on purpose. A household missing from `purchase_features`, `demographics`, or `household_target` doesn't belong in a training set, it belongs nowhere near one, and an inner join drops it rather than filling in a guess." ] }, { "cell_type": "code", "execution_count": null, "id": "64", "metadata": {}, "outputs": [], "source": [ "n_missing = feature_matrix.isna().sum().sum()\n", "themed_gt(GT(feature_matrix.head(5).reset_index()), n_rows=5).tab_header(\n", " title=gt_md(\"**Feature matrix, first 5 households**\"),\n", " subtitle=f\"{feature_matrix.shape[0]:,} households x {feature_matrix.shape[1]} columns, {n_missing} missing values\",\n", ")" ] }, { "cell_type": "markdown", "id": "65", "metadata": {}, "source": [ "### Promoting it\n", "\n", "Same move as Section 1's `config.py`: everything above, the temporal windows, the behavioural features, the demographics join, now lives at `ark/case_studies/spend_regression/features.py`, as `build_feature_matrix`. It's joined by a new file, `ark/case_studies/spend_regression/schema.py`, holding `FeatureMatrixSchema`, a Pandera check on the way out: no negative spend, no zero purchase counts, every demographic column present. Rebuild from the promoted versions and confirm they match what you built by hand:" ] }, { "cell_type": "code", "execution_count": null, "id": "66", "metadata": {}, "outputs": [], "source": [ "from ark.case_studies.spend_regression.features import build_feature_matrix\n", "from ark.case_studies.spend_regression.schema import FeatureMatrixSchema\n", "\n", "promoted_matrix = build_feature_matrix(purchases_df, survey_df, settings)\n", "promoted_matrix = FeatureMatrixSchema.validate(promoted_matrix)\n", "\n", "same_shape = feature_matrix.shape == promoted_matrix.shape\n", "same_values = feature_matrix.equals(promoted_matrix.reindex(columns=feature_matrix.columns))\n", "same_shape and same_values" ] }, { "cell_type": "code", "execution_count": null, "id": "67", "metadata": {}, "outputs": [], "source": [ "n_rows = len(promoted_matrix)\n", "schema_checks = pd.DataFrame(\n", " [\n", " {\"Field(s)\": \"early_spend, target_spend\", \"Rule\": \"float, >= 0\", \"Rows passing\": n_rows},\n", " {\"Field(s)\": \"n_purchases, n_categories\", \"Rule\": \"int, >= 1\", \"Rows passing\": n_rows},\n", " {\"Field(s)\": \"9 demographic columns\", \"Rule\": \"present, non-null string\", \"Rows passing\": n_rows},\n", " ]\n", ")\n", "themed_gt(GT(schema_checks), n_rows=len(schema_checks)).tab_header(\n", " title=gt_md(\"**Schema validation summary**\"),\n", " subtitle=f\"FeatureMatrixSchema.validate() passed: all {n_rows:,} rows satisfy every rule\",\n", ")" ] }, { "cell_type": "markdown", "id": "68", "metadata": {}, "source": [ "
\n", " Activity 7: write the test that would have caught the leak

\n", "This case study's entire feature-window argument is only worth something if a future edit can't quietly break it. Here's a fixture: household R_a buys $50 and $30 in the feature window (days 0-364) and $100 in the target window (day 397). If someone later changed the window boundary and accidentally let that $100 purchase into the features, what assertion would catch it? Fill in the blank:\n", "
result = build_feature_matrix(purchases_df, survey_df, settings)\n",
    "assert result.loc[\"R_a\", \"early_spend\"] == ...
\n", "
" ] }, { "cell_type": "markdown", "id": "69", "metadata": {}, "source": [ "If you wrote `80.0` (the $50 plus the $30, not the $100), that's exactly `test_build_feature_matrix_computes_features_only_from_the_feature_window` in `tests/case_studies/test_spend_regression.py`. It exists specifically so that if someone changes `FEATURE_WINDOW_DAYS` or fumbles the boundary condition later, this test fails immediately instead of quietly producing a leaky model six months from now.\n", "\n", "Four more tests sit next to it: the cache-hit path from Section 2, a household with no purchases in the target window getting dropped rather than imputed, the missing-`Category` fix from the last activity, and the schema rejecting a negative spend value outright. Run the whole file:" ] }, { "cell_type": "code", "execution_count": null, "id": "70", "metadata": {}, "outputs": [], "source": [ "!uv run pytest ../../tests/case_studies/test_spend_regression.py -v --override-ini=addopts=\"\"" ] }, { "cell_type": "markdown", "id": "71", "metadata": {}, "source": [ "Section 5 turns this feature matrix into a model: a baseline, then something that can actually beat it." ] }, { "cell_type": "markdown", "id": "72", "metadata": {}, "source": [ "## 5. Model selection\n", "\n", "Back to the spec card: regression, supervised, success measured by whether the top decile of predicted spend actually outspends everyone else, MAE as a secondary sanity check. Three candidates, baseline, Ridge, one boosting model, per the project spine, not a bake-off. If a fourth model would have changed the story, the first three weren't different enough to be worth comparing." ] }, { "cell_type": "code", "execution_count": null, "id": "73", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "from sklearn.compose import ColumnTransformer\n", "from sklearn.dummy import DummyRegressor\n", "from sklearn.ensemble import HistGradientBoostingRegressor\n", "from sklearn.impute import SimpleImputer\n", "from sklearn.linear_model import Ridge\n", "from sklearn.metrics import mean_absolute_error\n", "from sklearn.model_selection import KFold, cross_val_score, train_test_split\n", "from sklearn.pipeline import Pipeline, make_pipeline\n", "from sklearn.preprocessing import OneHotEncoder, StandardScaler\n", "\n", "NUMERIC_FEATURES = [\"early_spend\", \"n_purchases\", \"n_categories\"]\n", "\n", "X = promoted_matrix[NUMERIC_FEATURES + DEMOGRAPHIC_COLUMNS]\n", "y = promoted_matrix[\"target_spend\"]\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=settings.random_seed)\n", "\n", "num_pipe = make_pipeline(SimpleImputer(strategy=\"median\"), StandardScaler())\n", "cat_pipe = make_pipeline(\n", " SimpleImputer(strategy=\"most_frequent\"),\n", " OneHotEncoder(handle_unknown=\"ignore\", sparse_output=False),\n", ")\n", "preprocessor = ColumnTransformer([(\"num\", num_pipe, NUMERIC_FEATURES), (\"cat\", cat_pipe, DEMOGRAPHIC_COLUMNS)])" ] }, { "cell_type": "markdown", "id": "74", "metadata": {}, "source": [ "That's the same `ColumnTransformer` shape [Chapter 32](../04-ml-intro/32-sklearn-pipeline.ipynb) built: numeric columns imputed and scaled, categorical columns imputed and one-hot encoded, combined into one preprocessing step so cross-validation can't leak a fold's own statistics into itself.\n", "\n", "MAE is a mechanical metric, scikit-learn ships it. Rank quality isn't, because it's specific to this business decision, and nothing in sklearn knows what \"top decile\" means for a loyalty program." ] }, { "cell_type": "markdown", "id": "75", "metadata": {}, "source": [ "
\n", " Activity 8: implement the metric the spec card actually asked for

\n", "Write top_decile_lift(y_true, y_pred): find the predicted-spend threshold marking the top 10% of predictions, then return the ratio of (actual spend of households at or above that threshold) to (actual spend of every household). A model with no predictive power at all should score close to 1.0, it can't tell the top decile from anyone else. A useful model should score well above 1.0.\n", "
def top_decile_lift(y_true, y_pred):\n",
    "    threshold = ...\n",
    "    return ...
\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "76", "metadata": {}, "outputs": [], "source": [ "def top_decile_lift(y_true: np.ndarray, y_pred: np.ndarray) -> float:\n", " \"\"\"Ratio of actual spend among the top decile of predictions to the overall mean.\"\"\"\n", " y_true = np.asarray(y_true)\n", " y_pred = np.asarray(y_pred)\n", " threshold = np.quantile(y_pred, 0.9)\n", " return y_true[y_pred >= threshold].mean() / y_true.mean()\n", "\n", "\n", "# Sanity check: a model that predicts the same value for everyone can't\n", "# distinguish a top decile from anyone else, lift should land near 1.0.\n", "sanity_check = top_decile_lift(y_test, np.full_like(y_test, y_train.mean()))\n", "sanity_check" ] }, { "cell_type": "markdown", "id": "77", "metadata": {}, "source": [ "### Setting a real threshold, before fitting anything\n", "\n", "Section 1 left the acceptable threshold as \"a number the business has to supply.\" For this chapter, say the loyalty program's economics work out to needing at least **2.5x lift**: the top decile has to spend two and a half times the average household, or targeting them isn't worth the program's cost. That number is illustrative, a real team would get it from finance, not a notebook, but a spec card without a number to clear isn't a spec card, it's a wish." ] }, { "cell_type": "markdown", "id": "78", "metadata": {}, "source": [ "### First attempt: predict spend from demographics alone\n", "\n", "Every one of Chapter 30's nine stages exists because skipping it costs you later, but the workflow is a cycle, not a checklist, and the honest way to show that is to prove what happens if you stop at the wrong point. Before behavioural features existed at all, the reasonable first move was demographics alone: age, income, education, household size, someone's profile should say *something* about how much they spend." ] }, { "cell_type": "code", "execution_count": null, "id": "79", "metadata": {}, "outputs": [], "source": [ "demo_preprocessor = ColumnTransformer([(\"cat\", cat_pipe, DEMOGRAPHIC_COLUMNS)])\n", "baseline_pipe = Pipeline([(\"preprocessor\", demo_preprocessor), (\"model\", DummyRegressor(strategy=\"mean\"))])\n", "demo_pipe = Pipeline([(\"preprocessor\", demo_preprocessor), (\"model\", Ridge(alpha=1.0))])\n", "\n", "X_demo_train, X_demo_test = X_train[DEMOGRAPHIC_COLUMNS], X_test[DEMOGRAPHIC_COLUMNS]\n", "\n", "results = {}\n", "for name, pipe in [(\"Baseline (mean)\", baseline_pipe), (\"Demographics only (Ridge)\", demo_pipe)]:\n", " pipe.fit(X_demo_train, y_train)\n", " preds = pipe.predict(X_demo_test)\n", " results[name] = {\"MAE\": mean_absolute_error(y_test, preds), \"Lift\": top_decile_lift(y_test, preds)}\n", "\n", "results" ] }, { "cell_type": "markdown", "id": "80", "metadata": {}, "source": [ "Demographics beat the baseline, $1,080 MAE and 2.00x lift against $1,239 and 1.00x, so the demographic columns aren't useless. But 2.00x doesn't clear the 2.5x bar. This is the exact moment [Chapter 30's dashed arrow from Evaluation back to Feature Engineering](../04-ml-intro/30-ml-workflow.qmd#the-end-to-end-machine-learning-workflow) means something concrete: the fix here isn't a fancier model on the same weak inputs, it's better inputs. A boosting model fit on the same three demographic-only columns wouldn't close a gap this size, there's simply not enough signal in age and income alone to predict what someone spends." ] }, { "cell_type": "markdown", "id": "81", "metadata": {}, "source": [ "### Why Section 4 didn't stop at demographics\n", "\n", "This is also why Section 4 built `early_spend`, `n_purchases`, and `n_categories` before this evaluation ever ran, not as a guess that paid off by luck, but because early purchase behaviour is a direct measurement of the thing you're predicting, and a demographic column is at best a proxy for it. Refit the identical Ridge model, same regularisation, same random state, with the behavioural features added back in:" ] }, { "cell_type": "code", "execution_count": null, "id": "82", "metadata": {}, "outputs": [], "source": [ "full_pipe = Pipeline([(\"preprocessor\", preprocessor), (\"model\", Ridge(alpha=1.0))])\n", "full_pipe.fit(X_train, y_train)\n", "full_preds = full_pipe.predict(X_test)\n", "\n", "results[\"Demographics + behaviour (Ridge)\"] = {\n", " \"MAE\": mean_absolute_error(y_test, full_preds),\n", " \"Lift\": top_decile_lift(y_test, full_preds),\n", "}\n", "results[\"Demographics + behaviour (Ridge)\"]" ] }, { "cell_type": "markdown", "id": "83", "metadata": {}, "source": [ "3.11x, clear of the 2.5x bar, and $756 MAE, almost a third lower. Same model, same target, same random state, the only thing that changed between the two attempts is which features it saw.\n", "\n", "
\n", " Key Concept: a model can't recover information it was never given

\n", "No amount of regularisation, tuning, or model complexity closes the gap between the demographics-only attempt and this one, because the missing ingredient was never a modelling choice. It was data the first attempt didn't have. This is [Chapter 30's dashed arrow from Evaluation back to Feature Engineering](../04-ml-intro/30-ml-workflow.qmd#the-end-to-end-machine-learning-workflow) made concrete: when a model underperforms, the first question is what it was fed, not which algorithm to swap in next. An evaluation that stops at \"the number's too low\" without asking that question is only half a diagnosis.\n", "
" ] }, { "cell_type": "markdown", "id": "84", "metadata": {}, "source": [ "Ridge already cleared the 2.5x bar using the full feature set. The bake-off Section 5 opened with still owes one more model, not because Ridge looks fragile, but because \"the simple model won\" is only a finding once something built to catch non-linear interactions has had the identical shot and lost. `HistGradientBoostingRegressor` runs on the same `X_train`/`X_test` split, the same preprocessing, the same target, nothing else changes." ] }, { "cell_type": "code", "execution_count": null, "id": "85", "metadata": {}, "outputs": [], "source": [ "boosting_pipe = Pipeline(\n", " [(\"preprocessor\", preprocessor), (\"model\", HistGradientBoostingRegressor(random_state=settings.random_seed))]\n", ")\n", "boosting_pipe.fit(X_train, y_train)\n", "boosting_preds = boosting_pipe.predict(X_test)\n", "\n", "results[\"Demographics + behaviour (Boosting)\"] = {\n", " \"MAE\": mean_absolute_error(y_test, boosting_preds),\n", " \"Lift\": top_decile_lift(y_test, boosting_preds),\n", "}\n", "results[\"Demographics + behaviour (Boosting)\"]" ] }, { "cell_type": "markdown", "id": "86", "metadata": {}, "source": [ "$765.39 MAE and 3.01x lift, both a little behind Ridge's $756.21 and 3.11x. With 3,822 households in the training set, a boosting model has far less signal per split to exploit than it would need to beat a linear fit convincingly, and the relationship between early spend and later spend turns out to be close enough to linear that Ridge's extra assumption costs nothing here. That answers Section 5's opening bet: a fourth model would not have changed this story, three candidates were already enough to see it. The lesson to carry forward isn't \"Ridge beats boosting\", it's to reach for the simplest model that clears the threshold, not the most powerful one on the shelf." ] }, { "cell_type": "code", "execution_count": null, "id": "87", "metadata": {}, "outputs": [], "source": [ "from ark.plot.gt_style import metrics_report\n", "\n", "results_df = pd.DataFrame([{\"Model\": name, **metrics} for name, metrics in results.items()])\n", "metrics_report(\n", " results_df,\n", " metrics=[\"MAE\", \"Lift\"],\n", " minimize_cols=[\"MAE\"],\n", " maximize_cols=[\"Lift\"],\n", " title=\"Spend Prediction: Model Comparison\",\n", " subtitle=\"Test set | Open e-commerce households, 12-month feature window\",\n", ")" ] }, { "cell_type": "markdown", "id": "88", "metadata": {}, "source": [ "### Promoting the winner into `model.py`\n", "\n", "Same move Section 4 already made twice for `config.py` and `features.py`: working code moves out of the notebook and into `ark/case_studies/spend_regression/model.py` once it's proven itself here. Four pieces make the move: `build_preprocessor` (the `ColumnTransformer` above, as a function of which columns are numeric versus categorical), `top_decile_lift`, an `evaluate_model` helper that returns the same row shape `metrics_report()` just consumed, and a dollar-value calculator the closing section needs next. Refitting the promoted pipeline should reproduce the exact numbers already on the board." ] }, { "cell_type": "code", "execution_count": null, "id": "89", "metadata": {}, "outputs": [], "source": [ "from ark.case_studies.spend_regression.model import build_preprocessor, evaluate_model\n", "\n", "promoted_preprocessor = build_preprocessor(NUMERIC_FEATURES, DEMOGRAPHIC_COLUMNS)\n", "promoted_pipe = Pipeline([(\"preprocessor\", promoted_preprocessor), (\"model\", Ridge(alpha=1.0))])\n", "promoted_pipe.fit(X_train, y_train)\n", "\n", "promoted_metrics = evaluate_model(promoted_pipe, X_test, y_test, \"Demographics + behaviour (Ridge)\")\n", "reference = results[\"Demographics + behaviour (Ridge)\"]\n", "promoted_metrics[\"MAE\"] == reference[\"MAE\"] and promoted_metrics[\"Lift\"] == reference[\"Lift\"]" ] }, { "cell_type": "markdown", "id": "90", "metadata": {}, "source": [ "`model.py` picked up two tests of its own alongside the six already covering `features.py` and `data.py`: `top_decile_lift` at a constant prediction (must equal 1.0x, the same sanity check from earlier in this section), and the dollar-value formula against hand-worked numbers. The full suite runs the same way it has throughout this chapter." ] }, { "cell_type": "code", "execution_count": null, "id": "91", "metadata": {}, "outputs": [], "source": [ "!uv run pytest ../../tests/case_studies/test_spend_regression.py -v --override-ini=addopts=\"\"" ] }, { "cell_type": "markdown", "id": "92", "metadata": {}, "source": [ "## 6. Evaluation\n", "\n", "Section 5 picked the metric before comparing models, one of [Chapter 30 Section 4](../04-ml-intro/30-ml-workflow.qmd#evaluation-metrics-that-match-the-problem)'s own arguments made concrete: choosing what to optimise after seeing which model wins lets the metric follow the result instead of judging it. Two questions from [Chapter 30 Section 5](../04-ml-intro/30-ml-workflow.qmd#diagnosing-failure-bias-variance-and-learning-curves) are still open now that Ridge has a number attached to it: would more of the same data move that number further, and which features is a $756 MAE actually resting on?" ] }, { "cell_type": "markdown", "id": "93", "metadata": {}, "source": [ "### Would more data help?\n", "\n", "A learning curve answers this directly: refit the promoted pipeline at increasing training-set sizes and watch what happens to the gap between training and validation error." ] }, { "cell_type": "code", "execution_count": null, "id": "94", "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import learning_curve\n", "\n", "train_sizes, train_lc_scores, val_lc_scores = learning_curve(\n", " promoted_pipe,\n", " X_train,\n", " y_train,\n", " train_sizes=np.linspace(0.1, 1.0, 6),\n", " cv=KFold(n_splits=5, shuffle=True, random_state=settings.random_seed),\n", " scoring=\"neg_mean_absolute_error\",\n", ")\n", "train_sizes" ] }, { "cell_type": "code", "execution_count": null, "id": "95", "metadata": {}, "outputs": [], "source": [ "from lets_plot import geom_ribbon, scale_color_manual, scale_fill_manual\n", "\n", "from ark.plot.tokens import WARNING\n", "\n", "lc_df = pd.DataFrame(\n", " {\n", " \"n\": np.tile(train_sizes, 2),\n", " \"mae\": np.concatenate([-train_lc_scores.mean(axis=1), -val_lc_scores.mean(axis=1)]),\n", " \"err\": np.concatenate([train_lc_scores.std(axis=1), val_lc_scores.std(axis=1)]),\n", " \"split\": [\"Train\"] * len(train_sizes) + [\"Validation\"] * len(train_sizes),\n", " }\n", ").assign(ymin=lambda d: d[\"mae\"] - d[\"err\"], ymax=lambda d: d[\"mae\"] + d[\"err\"])\n", "\n", "(\n", " ggplot(lc_df, aes(x=\"n\", y=\"mae\", color=\"split\", fill=\"split\"))\n", " + geom_line(size=1.2)\n", " + geom_ribbon(aes(ymin=\"ymin\", ymax=\"ymax\"), alpha=0.15, color=None)\n", " + scale_color_manual(values={\"Train\": INFO, \"Validation\": WARNING})\n", " + scale_fill_manual(values={\"Train\": INFO, \"Validation\": WARNING})\n", " + labs(\n", " title=\"Learning curve: promoted Ridge pipeline\",\n", " subtitle=\"Gap between train and validation = variance still to close\",\n", " x=\"Training set size (households)\",\n", " y=\"MAE, $ (lower is better)\",\n", " )\n", " + modern_theme(grid=True, legend_pos=\"bottom\")\n", ")" ] }, { "cell_type": "markdown", "id": "96", "metadata": {}, "source": [ "The gap starts wide, $875 validation MAE against $707 training MAE at the smallest size, and narrows to under $26 by the largest training size tested. Most of the variance a small training set would cause is already resolved. But validation MAE is still edging down at the far right of the curve, from $782 to $775 over the last two points, not flat. That combination, a mostly closed gap with a curve that hasn't fully levelled, says more rows of the same three behavioural features would help a little, not a lot. The bigger lever is the one Section 5's ablation already demonstrated: a feature the model has never seen moves the number more than a feature it already has, just measured again." ] }, { "cell_type": "markdown", "id": "97", "metadata": {}, "source": [ "### Which features are the model actually using?\n", "\n", "Section 4's demographic columns are one-hot encoded at different frequencies, so reading `Ridge.coef_` directly is misleading: a rare category's coefficient can be large simply because it was fit on a handful of households, not because it matters. Permutation importance sidesteps that: it shuffles one original column at a time and measures how much worse the fitted pipeline gets, so a coefficient's scale and a category's rarity both wash out." ] }, { "cell_type": "code", "execution_count": null, "id": "98", "metadata": {}, "outputs": [], "source": [ "from sklearn.inspection import permutation_importance\n", "\n", "perm = permutation_importance(\n", " promoted_pipe, X_test, y_test, n_repeats=20, random_state=settings.random_seed, scoring=\"neg_mean_absolute_error\"\n", ")\n", "importance_df = (\n", " pd.Series(perm.importances_mean, index=X_test.columns)\n", " .rename(\"MAE increase ($)\")\n", " .sort_values(ascending=False)\n", " .reset_index()\n", " .rename(columns={\"index\": \"Feature\"})\n", " .head(6)\n", ")\n", "themed_gt(GT(importance_df), n_rows=len(importance_df)).fmt_number(columns=\"MAE increase ($)\", decimals=2).tab_header(\n", " title=gt_md(\"**Which features drive the prediction**\"),\n", " subtitle=\"Increase in test-set MAE when a column is shuffled, top 6 of 12\",\n", ")" ] }, { "cell_type": "markdown", "id": "99", "metadata": {}, "source": [ "`early_spend` shuffled costs the model $366 of MAE, more than every other feature combined, `n_categories` is a distant second at $95. One demographic column, how often the household reports using Amazon, adds a real $52. Every other demographic column, including the ones the Section 5 ablation showed mattered as a block, is worth a few dollars or less on its own once behaviour is in the model. That's not a contradiction: demographics-only scored 2.00x because that was the only signal available in that attempt. With behaviour present, no single demographic column is doing much of the remaining work, their earlier contribution was a joint effect, not any one of them carrying the model alone.\n", "\n", "That closes the two open questions from Chapter 30 Section 5: variance is mostly resolved, and the metric's biggest lever is a feature that already exists, used well, not a new one. [Chapter 30 Section 6's decision framework](../04-ml-intro/30-ml-workflow.qmd#when-performance-is-insufficient-the-decision-framework), what to do if a model falls short of its threshold, isn't needed here since Ridge cleared 2.5x; the honest note for a real team is that if it hadn't, that section is where \"collect more data\" versus \"accept a lower threshold\" versus \"reframe the problem\" gets decided, and this evaluation is exactly the evidence that decision would run on." ] }, { "cell_type": "markdown", "id": "100", "metadata": {}, "source": [ "## 7. Key findings and recommendation\n", "\n", "**The business decision.** The loyalty team asked which households are worth investing more in. This model answers a narrower, checkable version of that: rank households by predicted spend over the next 12 months, and prioritise the top decile for the coming year's retention budget, exclusive early access, a dedicated support line, whatever the program's next round is. It's a targeting recommendation, not a spend forecast used for anything else, matching the spec card Section 1 resolved before any code was written.\n", "\n", "**The dollar-value estimate.** `estimate_incremental_value` gives the back-of-envelope version: the top-decile households in the test set spent $3,147.01 more than the household average, targeting 10% of the 4,778 usable households is 478 households, and assuming the loyalty program's perks convert 15% of that spend gap into incremental purchases (not the full gap, since most targeted households won't respond), that's roughly **$225,640.86** in incremental annual revenue. Every number in that sentence is a stated assumption, not a measurement: change the conversion rate and the estimate moves with it. That's the point, a reader challenging the 15% figure is arguing about the right thing." ] }, { "cell_type": "code", "execution_count": null, "id": "101", "metadata": {}, "outputs": [], "source": [ "from ark.case_studies.spend_regression.model import estimate_incremental_value\n", "\n", "incremental_value = estimate_incremental_value(\n", " np.asarray(y_test), promoted_pipe.predict(X_test), total_households=len(promoted_matrix), conversion_rate=0.15\n", ")\n", "f\"${incremental_value:,.2f}\"" ] }, { "cell_type": "markdown", "id": "102", "metadata": {}, "source": [ "**What's next.** This chapter stops at Evaluation, Chapter 30's workflow has two stages still ahead, Deployment and Monitoring, and a feedback loop from Monitoring back to Problem Framing that this notebook has only demonstrated one half of. The half it did demonstrate, Evaluation feeding back into Feature Engineering, showed up directly in Section 5's ablation. The other half is a live question a real team faces after shipping this model: household spending patterns shift, Section 3's COVID-era chart already proved that, so the demographic and behavioural relationships this Ridge model learned from 2018-2022 data will drift. Watching for that drift, and deciding when it's severe enough to reopen Section 1's problem framing rather than just retrain, is Monitoring's job, covered in this book's MLOps part, not here.\n", "\n", "Everything above, the resolved spec card, the honest ablation, the model comparison, the dollar-value assumptions, is what a production team would compress into a one-page model card before shipping this. The full version lives in this notebook so you could see how each line of that card gets earned, not asserted, before it ever gets compressed down to one page." ] }, { "cell_type": "markdown", "id": "103", "metadata": {}, "source": [ "## Summary\n", "\n", "| Concept | Key rule |\n", "|---|---|\n", "| Five-component spec card | Target variable, task type and paradigm, success metric, acceptable threshold, data specification; resolved through follow-up questions, not assumed from a vague ask |\n", "| Promotion | Code proven in the notebook moves into `ark/case_studies/spend_regression/` by hand; nothing keeps the two in sync automatically |\n", "| Temporal feature window | Features come only from days before the target window starts; the target itself must respect the same boundary a train/test split would |\n", "| `nunique()` and missing values | `Series.nunique()` drops `NaN` by default; fill missing categories before counting or undercounted households never show up as an error |\n", "| Evaluation-to-Feature-Engineering loop | An evaluation that stops at \"the number's too low\" is half a diagnosis; the fix is often what the model was fed, not which algorithm ran |\n", "| `top_decile_lift` | The spec card's real success metric: whether the top decile of predicted spend actually outspends everyone else, not just whether MAE is low |\n", "| Dollar-value estimate | State the assumption behind every dollar figure explicitly; a stakeholder challenging the assumption is arguing about the right thing |\n", "\n", "**Next:** Case Study 2 applies the same spine, spec card, baseline, honest evaluation, dollar-value close, to a churn classification problem, picking up the exact spec card Chapter 30 already worked out by hand." ] } ], "metadata": { "kernelspec": { "display_name": "ark (3.12.12.final.0)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 5 }