{
"cells": [
{
"cell_type": "markdown",
"id": "0",
"metadata": {},
"source": [
"---\n",
"title: \"Chapter 16: Data storytelling and house style\"\n",
"---"
]
},
{
"cell_type": "markdown",
"id": "1",
"metadata": {},
"source": [
"[](https://colab.research.google.com/github/sambaiga/ds-mlops-path/blob/main/tutorials/01-python-basics/16-data-storytelling.ipynb) [](https://raw.githubusercontent.com/sambaiga/ds-mlops-path/main/tutorials/01-python-basics/16-data-storytelling.ipynb)"
]
},
{
"cell_type": "markdown",
"id": "2",
"metadata": {},
"source": [
"You have built the chart. The axes are labelled, the data is correct. You export it and paste it into the slide deck. Ten minutes later someone asks: *\"What am I supposed to take from this?\"* The chart was not wrong. It was just not a story.\n",
"\n",
"Chapters 14 and 15 taught you the mechanics: matplotlib's object model, lets-plot's grammar of graphics. This notebook is about judgment: which chart to pick, what ink to cut, how your eye is deceived by common tricks, and how to apply a house style so every chart you publish looks like it belongs to the same project. Chapter 17 puts every one of these judgment calls to work on a single real case, start to finish.\n",
"\n",
"> Callout markers: [book cover page](../../index.qmd#callout-guide)."
]
},
{
"cell_type": "markdown",
"id": "3",
"metadata": {},
"source": [
"::: {.callout-note collapse=\"true\" icon=false}\n",
"## Learning objectives\n",
"\n",
"By the end of Chapter 16 you will be able to:\n",
"\n",
"| # | Skill | Covered in |\n",
"|---|---|---|\n",
"| 1 | Choose a chart type based on the question, not habit | Sec. 1 |\n",
"| 2 | Apply Gestalt principles to explain why grouped charts work | Sec. 2 |\n",
"| 3 | Recognise and avoid the most common ways charts mislead | Sec. 3 |\n",
"| 4 | Decide when a table communicates better than a chart | Sec. 4 |\n",
"| 5 | Apply `ark.plot`'s house theme to matplotlib and lets-plot charts | Sec. 5 |\n",
"| 6 | Identify preattentive attributes and use them deliberately | Sec. 6 |\n",
"| 7 | Add annotations that carry the chart's narrative | Sec. 7 |\n",
"| 8 | Explain why summary statistics alone are never sufficient (Datasaurus) | Sec. 8 |\n",
"| 9 | Produce one polished, captioned chart from a messy dataset | Capstone |\n",
":::"
]
},
{
"cell_type": "markdown",
"id": "4",
"metadata": {},
"source": [
"## 1. Picking the Right Chart, and Cutting What Does Not Help\n",
"\n",
"The most common visualisation mistake isn't a styling problem, it's a **selection** problem: building a chart type out of habit instead of asking what question it needs to answer. A pie chart can't answer \"which course improved the most,\" because comparing angles is something people are measurably worse at than comparing positions along a shared axis. That isn't opinion, it's the finding of a well-known study by Cleveland and McGill (1984) ranking how accurately people read different visual encodings: position along a common scale first, then length, then angle, then area, with colour saturation last."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"# Full platform export, used directly wherever a chart doesn't need a fixed category count\n",
"df7 = pd.read_csv(\"data/university_analytics.csv\")\n",
"\n",
"# Same three-course, three-Fall-cohort slice as Chapters 14 and 15, so every\n",
"# chart in this notebook is comparing the same students\n",
"_courses = [\"Machine Learning\", \"Data Structures\", \"Python Programming\"]\n",
"_semesters = [\"Fall 2022\", \"Fall 2023\", \"Fall 2024\"]\n",
"results = df7[df7[\"course\"].isin(_courses) & df7[\"semester\"].isin(_semesters)].copy()\n",
"results.head()"
]
},
{
"cell_type": "markdown",
"id": "6",
"metadata": {},
"source": [
"**Data-ink ratio**, a term from Edward Tufte's *The Visual Display of Quantitative Information*, is the proportion of a chart's ink that represents actual data, as opposed to gridlines, borders, redundant legends, and other decoration. Maximising it doesn't mean a bare chart, it means every mark earns its place. Alberto Cairo calls this \"functional art\": a chart still has to be engaging, it just can't be engaging *instead* of being clear. Compare matplotlib's defaults to a version with the same data and nothing extra:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7",
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"course_means = results.groupby(\"course\")[\"final_score\"].mean()\n",
"\n",
"fig, axes = plt.subplots(1, 2, figsize=(10, 3.5))\n",
"\n",
"# Left: matplotlib defaults. Box on all four sides, both gridlines, a\n",
"# legend the bar colours already make redundant since each bar has its\n",
"# own label directly underneath it.\n",
"axes[0].bar(course_means.index, course_means.values, color=[\"#4477AA\", \"#EE6677\", \"#228833\"])\n",
"axes[0].set_title(\"Before: matplotlib defaults\")\n",
"axes[0].grid(True)\n",
"\n",
"# Right: same data, decluttered. No top/right spines, no gridlines, the\n",
"# value printed directly on each bar instead of relying on the y-axis.\n",
"axes[1].bar(course_means.index, course_means.values, color=\"#4477AA\")\n",
"axes[1].spines[\"top\"].set_visible(False)\n",
"axes[1].spines[\"right\"].set_visible(False)\n",
"axes[1].set_yticks([])\n",
"for i, value in enumerate(course_means.values):\n",
" axes[1].text(i, value + 1, f\"{value:.0f}\", ha=\"center\")\n",
"axes[1].set_title(\"After: higher data-ink ratio\")\n",
"\n",
"fig.tight_layout()"
]
},
{
"cell_type": "markdown",
"id": "8",
"metadata": {},
"source": [
"\n",
" Key Concept: Data-ink ratio
\n",
"Every gridline, border, and redundant legend entry competes with your data for the reader's attention. The right-hand chart above removes the y-axis entirely (the labelled bars make it redundant) and the top/right spines, and prints each value once, directly on its bar, instead of forcing the reader to trace a line back to an axis. Decluttering isn't about making a chart sparse for its own sake, it's about removing anything that doesn't carry information.\n",
"
"
]
},
{
"cell_type": "markdown",
"id": "9",
"metadata": {},
"source": [
"**One message per chart** is the last principle worth internalising before the chart-lie gallery in Section 2: a chart that tries to answer three questions at once usually answers none of them clearly. If you find yourself adding a third colour dimension, a secondary y-axis, and a trend line to the same chart, that is a sign you need two charts, not one crowded one."
]
},
{
"cell_type": "markdown",
"id": "10",
"metadata": {},
"source": [
"\n",
" Activity 1 - Cut the Ink
\n",
"Apply the data-ink ratio principle to the comparison chart from this section.
1. Copy the fig, axes = plt.subplots(1, 2) comparison code from above into a new cell.
2. On the 'before' panel, add two extra elements that increase ink without adding information: ax.grid(True) and ax.set_facecolor('#F0F0F0').
3. On the 'after' panel, remove every element you can: spines, tick labels, background: keep only what is needed to read the values.
4. Add a markdown cell: 'I removed X because...' for each removed element.
Expected: the 'after' panel communicates the same data with fewer marks.\n",
"
"
]
},
{
"cell_type": "markdown",
"id": "11",
"metadata": {},
"source": [
"## 2. Gestalt Principles: Why Grouped Charts Work\n",
"\n",
"Preattentive attributes tell you what stands out before conscious attention. Gestalt principles tell you how the brain groups what it sees. The two work together: colour creates salience (preattentive); proximity creates grouping (Gestalt). Four principles appear everywhere in DS charts."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "12",
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.patches as mpatches\n",
"import matplotlib.pyplot as plt\n",
"import pandas as pd\n",
"\n",
"fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n",
"\n",
"# Proximity: dots grouped by closeness\n",
"x_left = [1, 1.2, 1.4, 3, 3.2, 3.4]\n",
"y_left = [1] * 6\n",
"axes[0].scatter(x_left, y_left, s=200, color=\"#38BDF8\")\n",
"axes[0].set_xlim(0, 4.5)\n",
"axes[0].set_ylim(0, 2)\n",
"axes[0].set_title(\"Proximity: spacing creates groups\", pad=10)\n",
"axes[0].axis(\"off\")\n",
"\n",
"# Similarity: dots grouped by colour\n",
"x_right = [1, 2, 3, 4, 5, 6]\n",
"colors = [\"#38BDF8\", \"#7C3AED\", \"#38BDF8\", \"#7C3AED\", \"#38BDF8\", \"#7C3AED\"]\n",
"axes[1].scatter(x_right, [1] * 6, s=200, c=colors)\n",
"axes[1].set_xlim(0, 7)\n",
"axes[1].set_ylim(0, 2)\n",
"axes[1].set_title(\"Similarity: colour creates groups\", pad=10)\n",
"axes[1].axis(\"off\")\n",
"\n",
"plt.suptitle(\"Gestalt: Proximity vs Similarity\", fontweight=\"bold\")\n",
"plt.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "13",
"metadata": {},
"source": [
"\n",
" Key Concept: Four Gestalt Principles in Chart Design
\n",
"The four Gestalt principles used in chart design: (1) Proximity: elements near each other are perceived as a group: this is why grouped bars and facets work; (2) Similarity: elements sharing colour, shape, or size seem to belong together; (3) Enclosure: a box, border, or shaded region creates a visual group; (4) Continuity: the eye follows a smooth path: this is why line charts work for time series but not categorical comparisons.\n",
"
"
]
},
{
"cell_type": "markdown",
"id": "14",
"metadata": {},
"source": [
"\n",
" Activity 2 - Proximity Without a Legend
\n",
"Use layout alone to group related series without adding a legend.
1. Create a fig, axes = plt.subplots(1, 2, figsize=(9, 3.5), sharey=True).
2. On the left panel plot Maths and Statistics scores as two overlapping histograms with alpha=0.6 in brand colours.
3. On the right panel plot Programming and Data Science the same way.
4. Add a title to each panel that says what it shows: no legend.
Expected: proximity (left vs right) groups the courses; the title replaces the legend.\n",
"
"
]
},
{
"cell_type": "markdown",
"id": "15",
"metadata": {},
"source": [
"## 3. Common Chart Lies\n",
"\n",
"None of the charts below contain incorrect numbers. Each one is drawn in a way that makes the reader's brain compute the wrong conclusion from correct data. Recognising these on sight, in your own draft charts, is the actual skill."
]
},
{
"cell_type": "markdown",
"id": "16",
"metadata": {},
"source": [
"\n",
" Key Concept: A misleading chart shows correct numbers in a deceptive frame
\n",
"Truncating the y-axis at a non-zero value makes small differences look large. A dual y-axis on the same plot invites the reader to see causation in coincidence. A pie chart with six slices is almost impossible to read because humans compare angles poorly. None of these charts lie about the numbers: they lie about what the numbers mean.\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "17",
"metadata": {},
"outputs": [],
"source": [
"fig, axes = plt.subplots(1, 2, figsize=(9, 3.5))\n",
"\n",
"axes[0].bar(course_means.index, course_means.values, color=\"#4477AA\")\n",
"axes[0].set_ylim(0, 100)\n",
"axes[0].set_title(\"Honest: y-axis starts at 0\")\n",
"\n",
"axes[1].bar(course_means.index, course_means.values, color=\"#EE6677\")\n",
"axes[1].set_ylim(50, 66)\n",
"axes[1].set_title(\"Misleading: y-axis starts at 50\")\n",
"\n",
"fig.tight_layout()"
]
},
{
"cell_type": "markdown",
"id": "18",
"metadata": {},
"source": [
"\n",
" Common Mistake: Truncating the y-axis
\n",
"The two charts above show the exact same three numbers. The right-hand one starts its y-axis at 50 instead of 0, which makes a 7-point difference between courses look like one course scores roughly twice as high as another. Bar charts in particular must start at zero, because the reader judges bar height (and therefore a ratio) automatically. Line charts can sometimes justify a non-zero baseline if it's labelled clearly, bars almost never can.\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "19",
"metadata": {},
"outputs": [],
"source": [
"fig, ax = plt.subplots(figsize=(6, 3.5))\n",
"\n",
"cohort_size = np.array([320, 340, 410, 580])\n",
"avg_score = np.array([71, 72, 70, 74])\n",
"years = [\"2022\", \"2023\", \"2024\", \"2025\"]\n",
"\n",
"ax.plot(years, cohort_size, color=\"#4477AA\", marker=\"o\", label=\"Cohort size\")\n",
"ax.set_ylabel(\"Cohort size\", color=\"#4477AA\")\n",
"\n",
"ax2 = ax.twinx()\n",
"ax2.plot(years, avg_score, color=\"#EE6677\", marker=\"o\", label=\"Average score\")\n",
"ax2.set_ylabel(\"Average score\", color=\"#EE6677\")\n",
"\n",
"ax.set_title(\"Two independently scaled axes invite a false correlation\");"
]
},
{
"cell_type": "markdown",
"id": "20",
"metadata": {},
"source": [
"\n",
" Common Mistake: Dual axes with independent scales
\n",
"Cohort size and average score both happen to rise together in this chart, but each axis was scaled independently to make the two lines fit nicely, which is exactly what makes dual axes dangerous: you can almost always pick scales that make two unrelated series appear to move together. If two series genuinely need comparing, plot their percent change from a common baseline on one shared axis instead, or use two separate charts stacked vertically with the same x-axis.\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "21",
"metadata": {},
"outputs": [],
"source": [
"enrollment = pd.Series(\n",
" {\"ML\": 145, \"Data Structures\": 132, \"Statistics\": 98, \"Databases\": 41, \"Networks\": 28, \"Ethics\": 15}\n",
")\n",
"\n",
"fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n",
"\n",
"axes[0].pie(enrollment.values, labels=enrollment.index, autopct=\"%1.0f%%\")\n",
"axes[0].set_title(\"Pie: hard to rank six similar slices\")\n",
"\n",
"axes[1].barh(enrollment.index[::-1], enrollment.values[::-1], color=\"#4477AA\")\n",
"axes[1].set_title(\"Bar: ranking is immediate\")\n",
"\n",
"fig.tight_layout()"
]
},
{
"cell_type": "markdown",
"id": "22",
"metadata": {},
"source": [
"\n",
" Common Mistake: Too many pie slices
\n",
"Six categories is already past where a pie chart works: Ethics and Networks are visibly different bars, but as pie slices they are nearly impossible to rank by eye, exactly the angle-judgment weakness Cleveland and McGill measured. A horizontal bar chart, sorted by value, answers \"which course has the most enrollment\" instantly. Reserve pie charts for two or three categories where one slice is clearly dominant, and prefer a bar chart almost everywhere else.\n",
"
"
]
},
{
"cell_type": "markdown",
"id": "23",
"metadata": {},
"source": [
"\n",
" Activity 3 - Fix the Truncated Y-Axis
\n",
"Spot and correct a chart lie in this section.
1. Re-run the truncated y-axis bar chart from this section.
2. Add axes[0].set_ylim(0, None) to start the scale at zero.
3. In a markdown cell below, describe in one sentence how the story changes when the axis starts at zero.
Expected: the differences between courses look smaller; the story is more honest.\n",
"
"
]
},
{
"cell_type": "markdown",
"id": "24",
"metadata": {},
"source": [
"## 4. When a Table Beats a Chart\n",
"\n",
"A chart is for spotting a pattern at a glance. A table is for looking up an exact number, or comparing many numbers a reader needs to cite precisely, such as a results appendix someone will quote in a report. This project's `ark.plot.gt_style` module wraps `great_tables` with the same brand colours used everywhere else, so a results table looks like it belongs in the same document as the chart next to it:"
]
},
{
"cell_type": "markdown",
"id": "25",
"metadata": {},
"source": [
"\n",
" Key Concept: Use a chart for patterns, a table for lookups
\n",
"A chart answers: 'is there a trend, a cluster, an outlier?' A table answers: 'what was the exact value for X in period Y?' If your reader needs to look up a specific number, a table is faster. If your reader needs to see a shape across many values at once, a chart is faster. Showing both for the same data wastes space and splits attention.\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "26",
"metadata": {},
"outputs": [],
"source": [
"from great_tables import GT, md as gt_md\n",
"\n",
"from ark.plot.gt_style import themed_gt\n",
"\n",
"summary = results.groupby(\"course\")[\"final_score\"].agg(mean=\"mean\", std=\"std\", n=\"count\").round(1).reset_index()\n",
"\n",
"table = themed_gt(\n",
" GT(summary)\n",
" .tab_header(title=gt_md(\"**Final Score Summary**\"), subtitle=\"Mean, spread, and sample size per course\")\n",
" .cols_label(course=\"Course\", mean=\"Mean\", std=\"Std. Dev.\", n=\"N\"),\n",
" n_rows=len(summary),\n",
")\n",
"table"
]
},
{
"cell_type": "markdown",
"id": "27",
"metadata": {},
"source": [
"\n",
" Pro Tip: Pair a chart with a table, don't choose between them
\n",
"A results section in a report often wants both: the histogram from Chapter 14 to show the shape of the distribution, and a table like the one above to give the exact numbers someone will want to quote. Neither replaces the other.\n",
"
"
]
},
{
"cell_type": "markdown",
"id": "28",
"metadata": {},
"source": [
"\n",
" Activity 4 - Chart vs Table Decision
\n",
"Decide which form is right for the question.
1. Create the summary table from this section (mean, std, n per course).
2. Also create a line chart of mean final score by course using matplotlib.
3. In a markdown cell, answer: a department head wants to compare pass rates at a glance: which do you show and why?
4. Now answer: a student wants to know their course's exact mean: which do you show and why?
Expected: different questions, different answers.\n",
"
"
]
},
{
"cell_type": "markdown",
"id": "29",
"metadata": {},
"source": [
"## 5. From Default to Branded\n",
"\n",
"Chapters 14 and 15 used each library's own defaults on purpose, so the charts in those notebooks teach the mechanics without a styling layer in the way. This project's `ark.plot` module exists precisely so you don't have to restate the same colours, fonts, and spacing on every single chart by hand. Compare a lets-plot chart with and without it:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "30",
"metadata": {},
"outputs": [],
"source": [
"from lets_plot import LetsPlot, aes, geom_histogram, gggrid, ggplot, ggsize\n",
"\n",
"LetsPlot.setup_html()\n",
"\n",
"default_chart = ggplot(results, aes(x=\"final_score\", fill=\"course\")) + geom_histogram(bins=20, alpha=0.6)\n",
"default_chart + ggsize(450, 300)"
]
},
{
"cell_type": "markdown",
"id": "31",
"metadata": {},
"source": [
"Apply the same transformation to the full dataset:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "32",
"metadata": {},
"outputs": [],
"source": [
"from lets_plot import labs, scale_fill_manual\n",
"\n",
"from ark.plot.theme import modern_theme, pro_colors\n",
"\n",
"branded_chart = (\n",
" ggplot(results, aes(x=\"final_score\", fill=\"course\"))\n",
" + geom_histogram(bins=20, alpha=0.85)\n",
" + scale_fill_manual(values=pro_colors)\n",
" + labs(\n",
" title=\"Final score distribution by course\",\n",
" x=\"Final score (0-100)\",\n",
" y=\"Number of students\",\n",
" fill=\"Course\",\n",
" )\n",
" + modern_theme(grid=True)\n",
")\n",
"branded_chart + ggsize(450, 300)"
]
},
{
"cell_type": "markdown",
"id": "33",
"metadata": {},
"source": [
"`modern_theme()` is one more `+` layer, exactly like every `geom_*()` in Chapter 15. `scale_fill_manual(values=pro_colors)` pins each course to the brand palette, and `labs()` replaces the auto-generated column names with reader-facing text. `gggrid()` puts both charts side by side in one output, which is the cleanest way to show a before/after comparison in a single cell:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "34",
"metadata": {},
"outputs": [],
"source": [
"gggrid([default_chart + ggsize(580, 380), branded_chart + ggsize(580, 380)], ncol=1)"
]
},
{
"cell_type": "markdown",
"id": "35",
"metadata": {},
"source": [
"The matplotlib equivalent, `ark.plot.matplot_theme.configure_matplotlib_style()`, works differently: it updates matplotlib's global `rcParams`, so it applies to every chart drawn after you call it, not just one. That is the right trade-off for a notebook or script that draws many charts you want to look consistent, at the cost of not being able to mix styled and unstyled charts in the same figure:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "36",
"metadata": {},
"outputs": [],
"source": [
"from ark.plot.matplot_theme import configure_matplotlib_style\n",
"\n",
"configure_matplotlib_style(font_size=10, fig_width=5)\n",
"\n",
"fig, ax = plt.subplots()\n",
"ax.hist(results[\"final_score\"], bins=20)\n",
"ax.set_xlabel(\"Final score\")\n",
"ax.set_ylabel(\"Number of students\")\n",
"ax.set_title(\"Same distribution as Section 1, after configure_matplotlib_style()\");"
]
},
{
"cell_type": "markdown",
"id": "37",
"metadata": {},
"source": [
"\n",
" Key Concept: House style is a default, not a style applied chart by chart
\n",
"Calling configure_matplotlib_style() once at the top of a notebook applies the same font size, spine removal, and grid style to every subsequent matplotlib chart. Passing modern_theme() and a branded colour scale once defines the look for every lets-plot chart in the session. The goal is consistency across an entire report, not per-chart customisation.\n",
"
"
]
},
{
"cell_type": "markdown",
"id": "38",
"metadata": {},
"source": [
"\n",
"
Activity 5 - Brand a Boxplot\n",
"\n",
"Rebuild Chapter 14's seaborn boxplot of
final_score by
course, but after calling
configure_matplotlib_style() from this section. No other code changes: the same
sns.boxplot() call should now pick up the brand colour cycle automatically.\n",
"
import seaborn as sns\n",
"\n",
"fig, ax = plt.subplots()\n",
"sns.boxplot(data=results, x=\"course\", y=\"final_score\", ax=ax)\n",
"ax.set_title(...)\n",
"
\n",
" Pro Tip: A house theme is a contract, not a one-off setting
\n",
"The value of ark.plot isn't any single colour choice, it's that every chart in this project, in any notebook, in any report, reaches for the same module instead of redefining its own palette. When the brand colours change, they change in one file.\n",
"
"
]
},
{
"cell_type": "markdown",
"id": "41",
"metadata": {},
"source": [
"## 6. Preattentive Attributes: What the Brain Sees First\n",
"\n",
"Before you consciously read a chart, your visual system has already processed certain features. These are called **preattentive attributes**: colour hue, luminance, shape, orientation, size, position, and enclosure. They register in under 250 milliseconds, before attention is directed.\n",
"\n",
"The design implication is simple but easy to violate: **use at most one preattentive attribute for emphasis per chart**. Two or more compete for attention and cancel each other out."
]
},
{
"cell_type": "markdown",
"id": "42",
"metadata": {},
"source": [
"\n",
" Key Concept: Preattentive Attributes
\n",
"Visual properties processed before conscious attention: colour hue, luminance, shape, orientation, size, position, and motion. The practical rule: encode at most one categorical dimension preattentively, and make sure it carries information, not decoration.\n",
"
\n",
"Reference: Ware, C. (2012). Information Visualization: Perception for Design, 3rd ed.\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "43",
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"df7 = pd.read_csv(\"data/university_analytics.csv\")\n",
"course_means = df7.groupby(\"course\")[\"final_score\"].mean().sort_values()\n",
"\n",
"# Left: every bar the same colour: no preattentive signal\n",
"# Right: one bar highlighted with colour: attention goes there instantly\n",
"TARGET = \"Machine Learning\"\n",
"\n",
"fig, axes = plt.subplots(1, 2, figsize=(11, 3.8))\n",
"\n",
"for ax, use_highlight in zip(axes, [False, True], strict=False):\n",
" colors = (\n",
" [\"#009E73\" if c == TARGET else \"#ADB5BD\" for c in course_means.index]\n",
" if use_highlight\n",
" else [\"#ADB5BD\"] * len(course_means)\n",
" )\n",
" ax.barh(course_means.index, course_means.values, color=colors)\n",
" ax.set_xlabel(\"Mean final score\")\n",
" ax.spines[[\"top\", \"right\"]].set_visible(False)\n",
" ax.set_xlim(0, 100)\n",
" title = \"No preattentive signal\" if not use_highlight else f\"Colour highlights {TARGET!r}\"\n",
" ax.set_title(title, fontsize=11, fontweight=\"bold\", color=\"#1E293B\")\n",
"\n",
"fig.suptitle(\"Same data: preattentive colour changes where the eye goes first\", fontsize=10, color=\"#6B7280\")\n",
"fig.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "44",
"metadata": {},
"source": [
"\n",
"
Activity 6 - Highlight by Region\n",
"\n",
"Build a horizontal bar chart of mean
midterm_score by
region.\n",
"Highlight the region with the highest mean in
#0369A1 and colour every other bar
#ADB5BD.\n",
"Add a concise title stating which region leads.\n",
"
region_means = df7.groupby(\"region\")[\"midterm_score\"].mean().sort_values()\n",
"# ... build the chart with one highlighted bar ...\n",
"
\n",
" Key Concept: Annotation as the \"So What?\"
\n",
"Knaflic's rule (2015): \"the most important single thing you can do to improve your graph is to add a descriptive title.\"\n",
"Add arrows and labels that direct attention to the exact point you want the viewer to remember.\n",
"A good annotation answers: what should I notice here, and why does it matter?\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "48",
"metadata": {},
"outputs": [],
"source": [
"fig, axes = plt.subplots(1, 2, figsize=(12, 4.2))\n",
"\n",
"# ── Left: unannotated: what's the takeaway? ──────────────────────────────────\n",
"program_means = df7.groupby(\"program\")[\"final_score\"].mean().sort_values()\n",
"\n",
"for ax in axes:\n",
" bars = ax.bar(program_means.index, program_means.values, color=[\"#ADB5BD\"] * len(program_means))\n",
" ax.set_ylabel(\"Mean final score\")\n",
" ax.spines[[\"top\", \"right\"]].set_visible(False)\n",
" ax.set_ylim(0, 85)\n",
" ax.tick_params(axis=\"x\", rotation=15)\n",
"\n",
"axes[0].set_title(\"Without annotation\\n(viewer guesses the message)\", fontsize=10, fontweight=\"bold\")\n",
"\n",
"# ── Right: annotated: narrative is explicit ──────────────────────────────────\n",
"best_idx = list(program_means.index).index(program_means.idxmax())\n",
"axes[1].patches[best_idx].set_facecolor(\"#0369A1\")\n",
"\n",
"axes[1].annotate(\n",
" f\"Data Science leads\\nat {program_means.max():.1f}\",\n",
" xy=(best_idx, program_means.max()),\n",
" xytext=(best_idx + 0.6, program_means.max() - 4),\n",
" fontsize=9,\n",
" color=\"#0369A1\",\n",
" fontweight=\"bold\",\n",
" arrowprops={\"arrowstyle\": \"->\", \"color\": \"#0369A1\", \"lw\": 1.4},\n",
" ha=\"left\",\n",
")\n",
"axes[1].set_title(\"With annotation\\n(message is explicit)\", fontsize=10, fontweight=\"bold\")\n",
"\n",
"fig.suptitle(\"Annotation turns a picture into an argument\", fontsize=10, color=\"#6B7280\")\n",
"fig.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "49",
"metadata": {},
"source": [
"\n",
"
Activity 7 - Annotate a Trend\n",
"\n",
"Plot mean
final_score by
semester as a line chart.\n",
"Add
ax.annotate() to flag the semester with the highest mean and write a one-sentence insight as the chart title.\n",
"
sem_means = df7.groupby(\"semester\")[\"final_score\"].mean()\n",
"# ... line chart + annotation ...\n",
"
\n",
" Key Concept: Datasaurus Dozen
\n",
"Matejka, J. & Fitzmaurice, G. (2017). Same stats, different graphs. Proceedings of CHI 2017.\n",
"The original Anscombe's Quartet dates from 1973. Both show the same principle: visualize before summarizing.\n",
"A modern corollary for ML: never report only RMSE without looking at a residual plot.\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "53",
"metadata": {},
"outputs": [],
"source": [
"# Demonstrate \"same stats, different shapes\" with two student subsets\n",
"# that have matching means and standard deviations but different distributions.\n",
"\n",
"rng = np.random.default_rng(7)\n",
"\n",
"# Group A: normally distributed scores (typical class)\n",
"n = 200\n",
"group_a = pd.DataFrame(\n",
" {\n",
" \"group\": \"Group A (normal)\",\n",
" \"midterm\": rng.normal(68, 12, n).clip(20, 100),\n",
" \"final\": rng.normal(70, 12, n).clip(20, 100),\n",
" }\n",
")\n",
"\n",
"# Group B: bimodal: two subpopulations merged (e.g. two campuses with different resources)\n",
"group_b = pd.DataFrame(\n",
" {\n",
" \"group\": \"Group B (bimodal)\",\n",
" \"midterm\": np.concatenate([rng.normal(55, 6, n // 2), rng.normal(82, 6, n // 2)]),\n",
" \"final\": np.concatenate([rng.normal(57, 6, n // 2), rng.normal(84, 6, n // 2)]),\n",
" }\n",
")\n",
"\n",
"combined = pd.concat([group_a, group_b], ignore_index=True)\n",
"\n",
"# Summary stats: almost identical\n",
"print(\"Summary statistics:\")\n",
"print(combined.groupby(\"group\")[[\"midterm\", \"final\"]].describe().round(1).to_string())\n",
"print()\n",
"\n",
"fig, axes = plt.subplots(1, 2, figsize=(11, 4), sharey=False)\n",
"\n",
"for ax, (name, grp) in zip(axes, combined.groupby(\"group\"), strict=False):\n",
" ax.scatter(grp[\"midterm\"], grp[\"final\"], alpha=0.4, s=18, color=\"#0369A1\")\n",
" ax.set_xlabel(\"Midterm score\")\n",
" ax.set_ylabel(\"Final score\")\n",
" ax.set_title(name, fontweight=\"bold\")\n",
" ax.spines[[\"top\", \"right\"]].set_visible(False)\n",
" # Annotate means\n",
" ax.axvline(grp[\"midterm\"].mean(), color=\"#D97706\", lw=1.2, ls=\"--\", label=f\"mean={grp['midterm'].mean():.1f}\")\n",
" ax.axhline(grp[\"final\"].mean(), color=\"#D97706\", lw=1.2, ls=\"--\")\n",
" ax.legend(fontsize=8)\n",
"\n",
"fig.suptitle(\"Same means and standard deviations: completely different structures\", fontsize=10, color=\"#6B7280\")\n",
"fig.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "54",
"metadata": {},
"source": [
"\n",
" Common Mistake: Reporting only mean ± std for a bimodal distribution
\n",
"Group B above has the same mean and standard deviation as Group A. A table of statistics alone would show no difference.\n",
"The scatter plot reveals that Group B is actually two separate subpopulations: a finding with direct intervention implications.\n",
"
\n",
"Rule: for any continuous variable you care about, always plot a distribution (histogram, KDE, strip plot, or box plot) before reporting a single number.\n",
"
"
]
},
{
"cell_type": "markdown",
"id": "55",
"metadata": {},
"source": [
"\n",
"
Activity 8 - Unmask the Hidden Structure\n",
"\n",
"From
university_analytics.csv, compare the distribution of
final_score for two programs side-by-side using overlapping KDE plots (or box plots).\n",
"Before plotting, print the mean and standard deviation for each program. Then reflect: would the statistics alone have revealed the difference?\n",
"
import seaborn as sns\n",
"# ... print stats, then plot KDE for each program ...\n",
"
\n",
"
Capstone Exercise - Fix the Chart\n",
"\n",
"\n",
"
\n",
"- Fix the y-axis to start at 0
\n",
"- Add an x and y axis label
\n",
"- Apply the house style with
configure_matplotlib_style() \n",
"- Add one print statement with a one-sentence caption stating the chart's single message
\n",
"
\n",
"
Hint: configure_matplotlib_style() only affects charts drawn after it runs, so call it before
plt.subplots(), not after.\n",
"