{ "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "---\n", "title: \"Chapter 7: NumPy advanced\"\n", "---" ] }, { "cell_type": "markdown", "id": "1", "metadata": {}, "source": [ "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/sambaiga/ds-mlops-path/blob/main/tutorials/01-python-basics/07-numpy-advanced.ipynb) [![Download Notebook](https://img.shields.io/badge/Download-Notebook-blue.svg?logo=jupyter&logoColor=white)](https://raw.githubusercontent.com/sambaiga/ds-mlops-path/main/tutorials/01-python-basics/07-numpy-advanced.ipynb)" ] }, { "cell_type": "markdown", "id": "2", "metadata": {}, "source": [ "Subtracting a shape-`(3,)` mean from a shape-`(5, 3)` matrix should raise an error. The shapes do not match. NumPy does it anyway, without complaining, without copying data. If you do not know why, you will write code that works by accident until the shapes change and it suddenly fails in production with no obvious cause.\n", "\n", "The answer is broadcasting: NumPy's rule for combining arrays of different shapes by stretching size-1 dimensions without copying data. Once you understand that rule, the gap between code that works and code that works *quickly* closes fast.\n", "\n", "The key concept is broadcasting. Once you understand it, the rest follows: memory layout, vectorisation, linear algebra, saving and loading arrays, and the gotchas that bite you when shapes are almost right but not quite. Chapter 6 built the array fundamentals; this chapter shows what those arrays can do at speed. By the end you will have everything you need for Chapter 14: matplotlib visualisations of the arrays you have been building.\n", "\n", "> Callout markers: [book cover page](../../index.qmd#callout-guide)." ] }, { "cell_type": "markdown", "id": "3", "metadata": {}, "source": [ "::: {.callout-note collapse=\"true\" icon=false}\n", "## Learning Objectives\n", "\n", "| # | Skill | Covered in |\n", "|---|---|---|\n", "| 1 | Apply broadcasting rules to combine arrays of different shapes | Sec. 7 |\n", "| 2 | Interpret `arr.strides` and `arr.itemsize` to understand memory layout | Sec. 7 |\n", "| 3 | Replace Python loops with vectorised NumPy expressions | Sec. 8 |\n", "| 4 | Compute dot products and matrix operations with `@` | Sec. 9 |\n", "| 5 | Save and load arrays with `.npy` and `.npz` | Sec. 10 |\n", "| 6 | Identify and fix the three most common NumPy gotchas | Sec. 11 |\n", ":::" ] }, { "cell_type": "markdown", "id": "4", "metadata": {}, "source": [ "## Setup\n", "\n", "Re-run this cell to restore the imports and student data matrix from Chapter 6." ] }, { "cell_type": "code", "execution_count": null, "id": "5", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "# Student data matrix: 5 students x 3 measurements (study_hours, attendance_pct, prior_gpa)\n", "student_data = np.array(\n", " [\n", " [12.0, 85.0, 3.1],\n", " [5.0, 60.0, 2.4],\n", " [18.0, 95.0, 3.8],\n", " [9.0, 70.0, 2.9],\n", " [22.0, 98.0, 3.9],\n", " ]\n", ")\n", "column_names = [\"study_hours\", \"attendance_pct\", \"prior_gpa\"]\n", "print(f\"student_data.shape={student_data.shape} student_data.dtype={student_data.dtype}\")" ] }, { "cell_type": "markdown", "id": "6", "metadata": {}, "source": [ "## 7. Broadcasting\n", "\n", "**Broadcasting** is the rule NumPy uses to apply an operation between two arrays of *different* shapes, by virtually \"stretching\" the smaller one, without actually copying any data. It is what lets you write `student_data - student_data.mean(axis=0)` instead of a loop over rows.\n", "\n", "
\n", " Key Concept: the broadcasting rule

\n", "Compare shapes from the right-hand side. Two dimensions are compatible when they are equal, or when one of them is 1 (it gets stretched to match). Missing leading dimensions are treated as 1.

\n", "(5, 3) and (3,) → treat (3,) as (1, 3) → stretch to (5, 3). (valid)
\n", "(5, 3) and (5,) → treat (5,) as (1, 5) → 3 != 5 and neither is 1. (error)\n", "
" ] }, { "cell_type": "markdown", "id": "7", "metadata": {}, "source": [ "The code below normalises each column of the student data matrix. `(5, 3) - (3,)` broadcasts without a loop: NumPy treats the `(3,)` mean as `(1, 3)` and stretches it across all five rows." ] }, { "cell_type": "code", "execution_count": null, "id": "8", "metadata": {}, "outputs": [], "source": [ "student_data = np.array(\n", " [\n", " [12.0, 85.0, 3.1],\n", " [5.0, 60.0, 2.4],\n", " [18.0, 95.0, 3.8],\n", " [9.0, 70.0, 2.9],\n", " [22.0, 98.0, 3.9],\n", " ]\n", ")\n", "\n", "col_mean = student_data.mean(axis=0) # shape (3,): one mean per column\n", "col_std = student_data.std(axis=0) # shape (3,)\n", "\n", "print(f\"student_data.shape : {student_data.shape}\")\n", "print(f\"col_mean.shape : {col_mean.shape}\")\n", "\n", "# (5, 3) - (3,) broadcasts the mean across every row: no loop needed\n", "student_data_z = (student_data - col_mean) / col_std\n", "print(f\"normalised:\\n{student_data_z}\")\n", "print(f\"new per-column mean (~0): {student_data_z.mean(axis=0).round(6)}\")" ] }, { "cell_type": "markdown", "id": "9", "metadata": {}, "source": [ "The rule is: align shapes from the right. Dimensions of size 1 stretch to match. Everything else must match exactly or NumPy raises an error." ] }, { "cell_type": "markdown", "id": "10", "metadata": {}, "source": [ "![NumPy broadcasting: a (5,3) matrix plus a (3,) vector. The vector is stretched to (5,3) by repeating it across rows, producing a (5,3) result.](figs/broadcasting.svg){fig-alt=\"Left: blue 5x3 grid. Center: plus sign. Right: green 1x3 vector with ghost rows showing it stretches to 5x3. Arrow to purple 5x3 result grid.\"}" ] }, { "cell_type": "markdown", "id": "11", "metadata": {}, "source": [ "
\n", " Common Mistake: broadcasting a (5,) array against a (5, 3) matrix fails for a subtle reason

\n", "If you compute per_row_mean = student_data.mean(axis=1) (shape (5,)) and try student_data - per_row_mean, NumPy raises ValueError: operands could not be broadcast together. It is comparing the trailing dimensions 3 vs 5, not what you intended. Fix it by giving the per-row result an explicit column shape with keepdims=True: student_data.mean(axis=1, keepdims=True) has shape (5, 1), which broadcasts correctly against (5, 3).\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "12", "metadata": {}, "outputs": [], "source": [ "row_mean = student_data.mean(axis=1, keepdims=True) # shape (5, 1), NOT (5,)\n", "print(f\"row_mean.shape : {row_mean.shape}\")\n", "\n", "# (5, 3) - (5, 1) broadcasts the per-row mean across every column\n", "centered_per_row = student_data - row_mean\n", "print(f\"centered_per_row:\\n{centered_per_row}\")" ] }, { "cell_type": "markdown", "id": "13", "metadata": {}, "source": [ "### Memory layout and strides\n", "\n", "Every NumPy array is, at bottom, a 1D block of bytes. `arr.strides` exposes the step sizes that let NumPy navigate a multi-dimensional shape through that flat block. `arr.itemsize` is the number of bytes in one element." ] }, { "cell_type": "markdown", "id": "14", "metadata": {}, "source": [ "
\n", " Key Concept: strides map a multi-dimensional shape onto flat memory

\n", "For a float64 array of shape (5, 3): itemsize is 8 bytes, strides are (24, 8). Moving one column right costs 8 bytes (= itemsize). Moving one row down costs 24 bytes (= 3 columns × 8 bytes). Basic slicing adjusts strides without copying data: that is why reshape() and [::-1] return views instantly.\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "15", "metadata": {}, "outputs": [], "source": [ "student_data = np.array(\n", " [\n", " [12.0, 85.0, 3.1],\n", " [5.0, 60.0, 2.4],\n", " [18.0, 95.0, 3.8],\n", " [9.0, 70.0, 2.9],\n", " [22.0, 98.0, 3.9],\n", " ]\n", ")\n", "\n", "print(f\"shape : {student_data.shape}\")\n", "print(f\"itemsize : {student_data.itemsize} bytes per element\") # 8 for float64\n", "print(f\"strides : {student_data.strides}\") # (bytes/row, bytes/col) = (24, 8)\n", "\n", "# A reversed view uses a negative stride: no data is copied\n", "rev = student_data[::-1]\n", "print(f\"reversed strides : {rev.strides}\") # (-24, 8)\n", "print(f\"shares memory original : {np.shares_memory(rev, student_data)}\")" ] }, { "cell_type": "markdown", "id": "16", "metadata": {}, "source": [ "
\n", " Activity 1: min-max scale every column

\n", "\n", "Write a one-line expression that scales every column of student_data to the [0, 1] range using the formula (student_data - student_data.min(axis=0)) / (student_data.max(axis=0) - student_data.min(axis=0)). Confirm the result's per-column min is 0 and max is 1.\n", "
student_data_scaled.min(axis=0)  # -> array([0., 0., 0.])\n",
    "student_data_scaled.max(axis=0)  # -> array([1., 1., 1.])
\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "17", "metadata": {}, "outputs": [], "source": [ "student_data = np.array(\n", " [\n", " [12.0, 85.0, 3.1],\n", " [5.0, 60.0, 2.4],\n", " [18.0, 95.0, 3.8],\n", " [9.0, 70.0, 2.9],\n", " [22.0, 98.0, 3.9],\n", " ]\n", ")\n", "\n", "student_data_scaled = ... # TODO: min-max scale every column to [0, 1]\n", "\n", "# print(f\"min per column: {student_data_scaled.min(axis=0)}\")\n", "# print(f\"max per column: {student_data_scaled.max(axis=0)}\")" ] }, { "cell_type": "markdown", "id": "18", "metadata": {}, "source": [ "## 8. Vectorisation vs Python loops\n", "\n", "Now that you can express normalisation as `(student_data - mean) / std` with no loop, it's worth seeing *why* that matters. Every NumPy operation you have used so far runs as a single compiled loop over contiguous memory; a Python `for` loop pays the cost of the interpreter on every single element." ] }, { "cell_type": "markdown", "id": "19", "metadata": {}, "source": [ "
\n", " Key Concept: replace loops with vectorised expressions, 10-100x faster

\n", "A Python loop over one million floats takes ~500ms. The same operation as (arr - arr.mean()) / arr.std() takes ~5ms. NumPy calls compiled C code: the Python interpreter never enters the inner loop. The speedup grows with array size.\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "20", "metadata": {}, "outputs": [], "source": [ "import time\n", "\n", "big = np.random.default_rng(0).normal(size=1_000_000)\n", "\n", "\n", "def zscore_loop(values: np.ndarray) -> list[float]:\n", " mean = sum(values) / len(values)\n", " variance = sum((v - mean) ** 2 for v in values) / len(values)\n", " std = variance**0.5\n", " return [(v - mean) / std for v in values]\n", "\n", "\n", "start = time.perf_counter()\n", "_ = zscore_loop(big)\n", "loop_time = time.perf_counter() - start\n", "\n", "start = time.perf_counter()\n", "_ = (big - big.mean()) / big.std()\n", "vector_time = time.perf_counter() - start\n", "\n", "print(f\"Python loop time : {loop_time:.4f}s\")\n", "print(f\"Vectorised time : {vector_time:.4f}s\")\n", "print(f\"Speedup : {loop_time / vector_time:,.0f}x\")" ] }, { "cell_type": "markdown", "id": "21", "metadata": {}, "source": [ "
\n", " Pro Tip: if you're writing a for loop over an array, stop and look for a vectorised way

\n", "Almost every elementwise transformation, filter, or aggregation you would write as a Python loop already has a NumPy equivalent: arithmetic operators, np.where, boolean masks, axis aggregations. Reach for those first: a hand-written loop over a large array is one of the most common NumPy performance bugs, and it's usually a 10-100x slowdown for no benefit.\n", "
" ] }, { "cell_type": "markdown", "id": "22", "metadata": {}, "source": [ "
\n", " Activity 2: benchmark loop vs vectorised z-score

\n", "Measure the speedup of vectorisation.

\n", "1. Create arr = np.random.default_rng(0).normal(size=100_000).
\n", "2. Implement a Python loop z-score (iterate element by element).
\n", "3. Implement the vectorised version: (arr - arr.mean()) / arr.std().
\n", "4. Time both with import time and print the speedup ratio.

\n", "Expected: vectorised is 50-200x faster.\n", "
" ] }, { "cell_type": "markdown", "id": "23", "metadata": {}, "source": [ "## 9. Linear algebra essentials\n", "\n", "A linear model's prediction is a **dot product**: multiply each measurement by a learned weight, sum the results, add a bias. `@` (matrix multiplication) computes this for every student row at once, no loop required." ] }, { "cell_type": "markdown", "id": "24", "metadata": {}, "source": [ "
\n", " Key Concept: @ is matrix multiplication; use it, never loop

\n", "student_data @ w computes every row's dot product with w in one call. Manually looping for row in student_data: sum(row * w) is correct but 100x slower. Every linear model, transformer attention layer, and PCA uses @ internally.\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "25", "metadata": {}, "outputs": [], "source": [ "student_data = np.array(\n", " [\n", " [12.0, 85.0, 3.1],\n", " [5.0, 60.0, 2.4],\n", " [18.0, 95.0, 3.8],\n", " [9.0, 70.0, 2.9],\n", " [22.0, 98.0, 3.9],\n", " ]\n", ") # shape (5 students, 3 measurements)\n", "\n", "# Suppose a (already-fitted) linear model has these learned weights and bias\n", "weights = np.array([1.5, 0.3, 8.0]) # one weight per column, shape (3,)\n", "bias = 10.0\n", "\n", "# student_data @ weights: (5, 3) @ (3,) -> (5,): one prediction per student\n", "predicted_scores = student_data @ weights + bias\n", "print(f\"predicted_scores: {predicted_scores.round(1)}\")" ] }, { "cell_type": "markdown", "id": "26", "metadata": {}, "source": [ "`@` on a matrix and a vector is shorthand for: for each row, multiply element-wise by `weights` and sum: exactly `(student_data * weights).sum(axis=1)`. Verify the two are equivalent, then measure prediction error against the true scores with `np.linalg.norm` (the Euclidean / RMS-style distance):" ] }, { "cell_type": "code", "execution_count": null, "id": "27", "metadata": {}, "outputs": [], "source": [ "# @ is equivalent to elementwise multiply + sum along axis=1\n", "manual = (student_data * weights).sum(axis=1) + bias\n", "print(f\"@ matches manual sum: {np.allclose(predicted_scores, manual)}\")\n", "\n", "actual_scores = np.array([88.0, 65.0, 95.0, 78.0, 99.0])\n", "errors = predicted_scores - actual_scores\n", "rmse = np.linalg.norm(errors) / np.sqrt(len(errors))\n", "print(f\"errors : {errors.round(1)}\")\n", "print(f\"RMSE : {rmse:.2f}\")" ] }, { "cell_type": "markdown", "id": "28", "metadata": {}, "source": [ "
\n", " Common Mistake: comparing floats with ==

\n", "np.allclose(a, b) was used above instead of (a == b).all() on purpose: floating-point arithmetic accumulates tiny rounding errors, so two mathematically-equal results can differ in their last bit. Always compare floats with a tolerance: np.allclose, or abs(a - b) < 1e-9, never with exact ==.\n", "
" ] }, { "cell_type": "markdown", "id": "29", "metadata": {}, "source": [ "
\n", " Activity 3: linear prediction with @

\n", "Use matrix multiplication to compute predictions.

\n", "Using student_data from the setup section:
\n", "1. Define weights = np.array([0.3, 0.4, 1.5]) and bias = -2.0.
\n", "2. Compute predicted scores as preds = student_data @ weights + bias.
\n", "3. Confirm preds.shape == (5,).
\n", "4. Print the index of the highest-scoring student using np.argmax(preds).\n", "
" ] }, { "cell_type": "markdown", "id": "30", "metadata": {}, "source": [ "## 10. Saving and loading arrays\n", "\n", "A typical pipeline computes a student data matrix once and reuses it across many later steps (training, evaluation, serving). `.npy` stores a single array in NumPy's own binary format: far smaller and faster to read than CSV for numeric data, and it preserves `dtype` and `shape` exactly. `.npz` bundles several named arrays together." ] }, { "cell_type": "markdown", "id": "31", "metadata": {}, "source": [ "
\n", " Key Concept: use .npy for one array, .npz for multiple

\n", "np.save('student_data.npy', student_data) and np.load('student_data.npy') preserve dtype and shape exactly. np.savez('data.npz', features=student_data, target=y) bundles multiple arrays in one file. Both load in microseconds: much faster than re-parsing a CSV.\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "32", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "tmp_dir = Path(\"tmp_numpy_activity\")\n", "tmp_dir.mkdir(exist_ok=True)\n", "\n", "student_data = np.array([[12.0, 85.0, 3.1], [5.0, 60.0, 2.4], [18.0, 95.0, 3.8]])\n", "y = np.array([88.0, 65.0, 95.0])\n", "\n", "# Save a single array\n", "np.save(tmp_dir / \"student_data.npy\", student_data)\n", "student_data_loaded = np.load(tmp_dir / \"student_data.npy\")\n", "print(f\"round-trip equal: {np.array_equal(student_data, student_data_loaded)}\")\n", "\n", "# Save several named arrays together in one file\n", "np.savez(tmp_dir / \"dataset.npz\", features=student_data, target=y)\n", "bundle = np.load(tmp_dir / \"dataset.npz\")\n", "print(f\"keys : {list(bundle.keys())}\")\n", "print(f\"target : {bundle['target']}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "33", "metadata": {}, "outputs": [], "source": [ "import shutil\n", "\n", "shutil.rmtree(tmp_dir)\n", "print(f\"cleaned up: {tmp_dir.exists()}\")" ] }, { "cell_type": "markdown", "id": "34", "metadata": {}, "source": [ "## 11. Common gotchas\n", "\n", "Like the Python gotchas in Chapter 4, none of these raise an exception. They silently produce a wrong (or surprising) result. Recognise them now so you do not lose hours to them later." ] }, { "cell_type": "markdown", "id": "35", "metadata": {}, "source": [ "
\n", " Key Concept: three gotchas that never raise an exception

\n", "1. View vs copy: sub = student_data[:2] shares memory; mutating sub mutates student_data. Fix: student_data[:2].copy().
\n", "2. Integer overflow: np.array([120, 10], dtype=np.int8) wraps silently. Fix: check dtype before arithmetic.
\n", "3. Shape mismatch in broadcasting: (5, 3) - (3, 5) may broadcast unexpectedly. Always check shapes before subtracting.\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "36", "metadata": {}, "outputs": [], "source": [ "# GOTCHA 1: basic slicing returns a VIEW, not a copy\n", "scores = np.array([62, 78, 85, 91, 55])\n", "top_three = scores[:3]\n", "top_three[0] = 0 # mutating the \"slice\"...\n", "\n", "print(f\"top_three : {top_three}\")\n", "print(f\"scores : {scores}\") # ...changed the original too!\n", "\n", "# Fix: copy explicitly when you need an independent array\n", "safe_copy = scores[:3].copy()" ] }, { "cell_type": "code", "execution_count": null, "id": "37", "metadata": {}, "outputs": [], "source": [ "# GOTCHA 2: integer dtype truncates on division-like ops, and can overflow\n", "small = np.array([120, 10], dtype=np.int8)\n", "print(f\"int8 + 50 : {small + 50}\") # 120 + 50 = 170, overflows int8's max of 127!\n", "\n", "# Fix: use a wide-enough dtype, or let NumPy infer (default is int64/float64)\n", "safe = small.astype(np.int64) + 50\n", "print(f\"int64 + 50: {safe}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "38", "metadata": {}, "outputs": [], "source": [ "# GOTCHA 3: {} is a dict, not a set: same trap as in plain Python (Chapter 4)\n", "empty_dict = {}\n", "empty_set = set()\n", "print(f\"type({{}}) : {type(empty_dict)}\")\n", "print(f\"type(set()) : {type(empty_set)}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "39", "metadata": {}, "outputs": [], "source": [ "# GOTCHA 4: comparing floats with == (see Sec. 9 for the fix: np.allclose)\n", "a = np.array([0.1 + 0.2])\n", "print(f\"0.1 + 0.2 == 0.3 : {a == 0.3}\") # False: 0.30000000000000004 != 0.3\n", "print(f\"np.allclose : {np.allclose(a, 0.3)}\") # True: tolerant comparison" ] }, { "cell_type": "markdown", "id": "40", "metadata": {}, "source": [ "## 12. Capstone exercises\n", "\n", "Apply everything from this notebook together. Each exercise is self-contained." ] }, { "cell_type": "markdown", "id": "41", "metadata": {}, "source": [ "
\n", " Activity 4: vectorised anomaly detector

\n", "\n", "Write a vectorised anomaly detector without any explicit loop. Flag any reading more than 2 standard deviations from the overall mean using a single boolean mask.\n", "
readings = [36.5, 36.7, 36.8, 36.6, 36.9, 39.5, 36.7, 36.8]\n",
    "# Expected: reading 39.5 (index 5) flagged as anomaly
\n", "Hint: z = (readings - readings.mean()) / readings.std(), then mask np.abs(z) > 2.\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "42", "metadata": {}, "outputs": [], "source": [ "readings = np.array([36.5, 36.7, 36.8, 36.6, 36.9, 39.5, 36.7, 36.8])\n", "\n", "z_scores = ... # TODO\n", "anomaly_mask = ... # TODO\n", "\n", "if z_scores is not ...:\n", " print(f\"z_scores : {z_scores.round(2)}\")\n", " print(f\"anomaly_mask : {anomaly_mask}\")\n", " print(f\"anomalies : {readings[anomaly_mask]}\")\n", "else:\n", " print(\"(implement z_scores and anomaly_mask above to see output)\")" ] }, { "cell_type": "markdown", "id": "43", "metadata": {}, "source": [ "## What's new in NumPy 2.0\n", "\n", "NumPy 2.0 (released June 2024) is the first major version bump in almost two decades. Two changes matter most for day-to-day data science work:\n", "\n", "### Removed type aliases\n", "\n", "The old Python-builtin aliases (`np.int`, `np.float`, `np.bool`, `np.complex`, `np.object`, `np.str`) were deprecated for years and are now **fully removed**. They were just aliases to the Python built-ins anyway, so the fix is mechanical:\n", "\n", "| Old (removed) | Replacement |\n", "|---|---|\n", "| `np.bool` | `np.bool_` or Python `bool` |\n", "| `np.int` | `np.intp` or Python `int` |\n", "| `np.float` | `np.float64` or Python `float` |\n", "| `np.complex` | `np.complex128` or Python `complex` |\n", "| `np.object` | `np.object_` or Python `object` |\n", "| `np.str` | `np.str_` or Python `str` |\n", "\n", "### `StringDType` for proper string arrays\n", "\n", "NumPy 2.0 introduced `np.dtypes.StringDType()`, a real variable-length string dtype backed by UTF-8 memory. The old `np.str_` stored fixed-width UCS-4 strings (one array dtype for every character count), `StringDType` stores arbitrary-length strings efficiently.\n", "\n", "
\n", " Pro Tip: use np.float64 not np.float in new code

\n", "If you see a module 'numpy' has no attribute 'float' error, the codebase was written against NumPy 1.x. A global search-and-replace of np.float → np.float64 (and so on for the others in the table) is the complete fix.\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "44", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "# Old aliases are removed in NumPy 2.0: this would raise AttributeError:\n", "# arr = np.array([1, 2, 3], dtype=np.float) # removed in NumPy 2.0\n", "\n", "# Use the explicit dtype names instead:\n", "arr = np.array([1, 2, 3], dtype=np.float64) # correct\n", "print(f\"dtype: {arr.dtype}\")\n", "\n", "# StringDType: variable-length strings, efficient UTF-8 storage (NumPy 2.0+)\n", "names = np.array([\"Alice\", \"Bob\", \"Charlie\"], dtype=np.dtypes.StringDType())\n", "print(f\"names : {names}\")\n", "print(f\"dtype : {names.dtype}\")\n", "\n", "# The old fixed-width str_ is still available but StringDType is the modern choice\n", "old_style = np.array([\"Alice\", \"Bob\", \"Charlie\"]) # infers str_ (fixed width)\n", "print(f\"old dtype: {old_style.dtype}\") # \n", " Pro Tip: use np.dtypes.StringDType() for string arrays that need fast operations

\n", "The default np.array([\"a\", \"b\"]) stores strings as object dtype: each element is a Python object pointer, so any operation loops at the Python level. np.dtypes.StringDType() stores strings as contiguous UTF-8 bytes, avoids the object-dtype overhead, and unlocks the entire np.strings.* API for compiled, vectorised string work.\n", "" ] }, { "cell_type": "markdown", "id": "48", "metadata": {}, "source": [ "## Further Reading\n", "\n", "| Resource | Why it matters |\n", "|---|---|\n", "| Harris, C.R. et al. (2020). [Array programming with NumPy](https://doi.org/10.1038/s41586-020-2649-2). *Nature* 585, 357–362. | The primary citation for NumPy; the paper explains the design decisions behind broadcasting and ufuncs |\n", "| VanderPlas, J. (2016). *Python Data Science Handbook*, Ch. 2. O'Reilly. | Free online: the most readable treatment of fancy indexing, broadcasting, and structured arrays |\n", "| [NumPy documentation: Broadcasting](https://numpy.org/doc/stable/user/basics.broadcasting.html) | Official broadcasting rules with diagrams; bookmark for the next time the shapes don't align |\n", "| [NumPy documentation: Indexing](https://numpy.org/doc/stable/user/basics.indexing.html) | Covers basic, advanced, and boolean indexing in one place |\n" ] }, { "cell_type": "markdown", "id": "49", "metadata": {}, "source": [ "## Summary\n", "\n", "| Concept | Key rule |\n", "|---|---|\n", "| `ndarray` | One dtype, contiguous memory, fast because it skips the Python interpreter |\n", "| Creation | `np.array`, `arange`, `linspace`, `zeros`/`ones`; `np.random.default_rng(seed)` for reproducible random data |\n", "| `shape`/`dtype` | Always check before trusting a result; mixed-type input silently upcasts |\n", "| `reshape` | Returns a **view**, same data, same `size`, different shape |\n", "| `flatten` | Always returns a **copy** |\n", "| Slicing | Basic slices (`student_data[:, 0]`) are views; fancy/boolean indexing always copies |\n", "| Boolean masks | `&` / `\\|` (not `and`/`or`) on arrays, each side parenthesised |\n", "| `np.where` / `np.select` | Vectorised if/else and multi-branch labelling |\n", "| `axis=0` vs `axis=1` | 0 collapses rows -> one value per **column**; 1 collapses columns -> one value per **row** |\n", "| Broadcasting | Compare shapes right-to-left; dims match if equal or one is `1`; use `keepdims=True` to broadcast a per-row stat |\n", "| Vectorisation | A NumPy expression beats a Python loop by 10-100x, look for one before writing `for` |\n", "| `@` / `np.dot` | Matrix multiplication: `student_data @ weights` predicts every row in one call |\n", "| `np.allclose` | Always compare floats with a tolerance, never `==` |\n", "| `.npy` / `.npz` | Compact, dtype/shape-preserving array storage between pipeline stages |\n", "\n", "\n", "**Next:** Chapter 14 (`14-matplotlib.ipynb`), covering how to visualise arrays and DataFrames with matplotlib." ] } ], "metadata": { "kernelspec": { "display_name": "ark", "language": "python", "name": "ark" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 5 }