{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Lesson 42: Instance Segmentation\n",
"\n",
"Semantic segmentation (Lesson 41) answers \"which pixels are circle pixels?\" It has no notion of *how many* circles there are — two touching circles are just one connected blob of \"circle\" pixels. **Instance segmentation** answers the harder question: which pixels belong to *this specific* object, as opposed to that other object of the same class. This lesson shows the failure concretely, then fixes it with an idea that traces straight back to Lesson 12's Hough transform: instead of only classifying pixels, have the network also predict, for every foreground pixel, a vote for where its object's center is — then cluster the votes to recover individual instances, the same accumulator-and-peak-finding pattern used for lines and circles, now finding object centers instead."
]
},
{
"cell_type": "code",
"id": "004a784d",
"source": "import numpy as np\nimport cv2\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport matplotlib.pyplot as plt",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "557bc204",
"source": "## Overlapping instances\n\nTwo circles per scene, deliberately placed close enough to touch or overlap. Ground truth includes both a semantic mask (0=background, 1=circle) and an *instance* mask (0=background, 1=first circle, 2=second circle).",
"metadata": {}
},
{
"cell_type": "code",
"id": "68daaf46",
"source": "SIZE = 32\n\ndef make_scene(rng, size=SIZE, n_circles=2, r=5):\n scene = np.zeros((size, size), dtype=np.float32)\n sem_mask = np.zeros((size, size), dtype=np.int64)\n inst_mask = np.zeros((size, size), dtype=np.int64)\n centers = []\n for k in range(n_circles):\n if k == 0:\n cx, cy = rng.integers(r + 8, size - r - 8), rng.integers(r + 8, size - r - 8)\n else:\n px, py = centers[-1]\n angle = rng.uniform(0, 2 * np.pi)\n dist = rng.uniform(6, 9) # overlapping (< 2r) but centers still separable\n cx = int(np.clip(px + dist * np.cos(angle), r, size - r - 1))\n cy = int(np.clip(py + dist * np.sin(angle), r, size - r - 1))\n centers.append((cx, cy))\n yy, xx = np.mgrid[0:size, 0:size]\n m = ((xx - cx) ** 2 + (yy - cy) ** 2) <= r ** 2\n scene[m] = 1.0\n sem_mask[m] = 1\n inst_mask[m] = k + 1 # later circles paint over earlier ones at overlaps\n scene = np.clip(scene + rng.normal(0, 0.05, scene.shape), 0, 1).astype(np.float32)\n return scene, sem_mask, inst_mask, centers\n\nrng = np.random.default_rng(17)\nN = 300\nscenes, sem_masks, inst_masks, all_centers = [], [], [], []\nfor _ in range(N):\n s, sm, im, c = make_scene(rng)\n scenes.append(s); sem_masks.append(sm); inst_masks.append(im); all_centers.append(c)\nscenes = np.array(scenes, dtype=np.float32)\nsem_masks = np.array(sem_masks, dtype=np.int64)\n\nfig, axes = plt.subplots(2, 4, figsize=(9, 4.5))\nfor i in range(4):\n axes[0, i].imshow(scenes[i], cmap='gray'); axes[0, i].axis('off')\n axes[1, i].imshow(inst_masks[i], cmap='viridis'); axes[1, i].axis('off')\naxes[0, 0].set_title('input', fontsize=9, loc='left')\naxes[1, 0].set_title('true instance mask', fontsize=9, loc='left')\nplt.show()",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "7c31f82a",
"source": "## Why semantic segmentation isn't enough\n\nTrain Lesson 41's exact U-Net architecture on the semantic task (background vs. circle) only, then try to recover instances the \"obvious\" way: run connected-components on the predicted foreground mask.",
"metadata": {}
},
{
"cell_type": "code",
"id": "d4c3a3a6",
"source": "# build offset targets: for every foreground pixel, the (dx, dy) to ITS instance's center\nyy, xx = np.mgrid[0:SIZE, 0:SIZE]\noffset_targets = np.zeros((N, 2, SIZE, SIZE), dtype=np.float32)\nfor i in range(N):\n im = inst_masks[i]\n for k, (cx, cy) in enumerate(all_centers[i]):\n m = im == (k + 1)\n offset_targets[i, 0][m] = (cx - xx[m]) / SIZE\n offset_targets[i, 1][m] = (cy - yy[m]) / SIZE\n\nsplit = int(0.85 * N)\nXtr, Str, Otr = scenes[:split], sem_masks[:split], offset_targets[:split]\nXte, Ste, Ote = scenes[split:], sem_masks[split:], offset_targets[split:]\ninst_te, centers_te = inst_masks[split:], all_centers[split:]\n\nclass InstanceNet(nn.Module):\n \"\"\"Lesson 41's U-Net, with a second head: per-pixel offset-to-center regression\"\"\"\n def __init__(self):\n super().__init__()\n self.enc1 = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.ReLU())\n self.enc2 = nn.Sequential(nn.Conv2d(16, 32, 3, padding=1), nn.ReLU())\n self.pool = nn.MaxPool2d(2)\n self.up = nn.Upsample(scale_factor=2, mode='nearest')\n self.dec1 = nn.Sequential(nn.Conv2d(32 + 16, 16, 3, padding=1), nn.ReLU())\n self.sem_head = nn.Conv2d(16, 2, 1) # background vs. circle\n self.offset_head = nn.Conv2d(16, 2, 1) # (dx, dy) to this pixel's instance center\n\n def forward(self, x):\n f1 = self.enc1(x)\n f2 = self.enc2(self.pool(f1))\n d1 = self.dec1(torch.cat([self.up(f2), f1], dim=1))\n return self.sem_head(d1), self.offset_head(d1)\n\ntorch.manual_seed(0)\nmodel = InstanceNet()\nopt = torch.optim.Adam(model.parameters(), lr=0.01)\nXt = torch.tensor(Xtr).unsqueeze(1); St = torch.tensor(Str); Ot = torch.tensor(Otr)\nfg_mask_t = (St == 1).unsqueeze(1).float()\n\nfor _ in range(300):\n opt.zero_grad()\n sem_logits, offset_pred = model(Xt)\n sem_loss = F.cross_entropy(sem_logits, St)\n offset_loss = (F.mse_loss(offset_pred, Ot, reduction='none') * fg_mask_t).sum() / fg_mask_t.sum().clamp(min=1)\n (sem_loss + 2.0 * offset_loss).backward()\n opt.step()\n\nwith torch.no_grad():\n sem_logits_te, offset_pred_te = model(torch.tensor(Xte).unsqueeze(1))\n sem_preds = sem_logits_te.argmax(1).numpy()\n offset_preds = offset_pred_te.numpy()\n\nprint(f'semantic pixel accuracy: {(sem_preds == Ste).mean():.1%}')\n\nn_cc_correct = 0\nfor i in range(len(Xte)):\n binary_mask = (sem_preds[i] == 1).astype(np.uint8)\n n_components, _ = cv2.connectedComponents(binary_mask)\n n_components -= 1 # subtract the background label\n true_n = len(set(inst_te[i].ravel()) - {0})\n if n_components == true_n:\n n_cc_correct += 1\nprint(f'connected-components recovers the correct instance count: {n_cc_correct}/{len(Xte)} scenes '\n f'({n_cc_correct/len(Xte):.1%})')",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "d11fb241",
"source": "## Fix: vote for the center, then cluster (Hough, again)\n\nThe second head above was already trained to predict, for every foreground pixel, an offset pointing toward *its own instance's* center — supervised directly from the ground-truth instance masks. Each foreground pixel casts one vote (its predicted center location) into an accumulator, exactly like Lesson 12's Hough line/circle voting. Because every pixel belonging to the same circle votes for approximately the same point, the accumulator forms one tight cluster of votes per instance, however tangled the pixels themselves are. Finding instances becomes: find the vote clusters, then assign each pixel to its nearest cluster.",
"metadata": {}
},
{
"cell_type": "code",
"id": "bdf53382",
"source": "def cluster_instances(binary_mask, offset_pred, size=SIZE, peak_dist=4, vote_thresh=3):\n ys, xs = np.where(binary_mask)\n if len(xs) == 0:\n return np.zeros_like(binary_mask, dtype=np.int64), []\n pred_cx = xs + offset_pred[0][ys, xs] * size\n pred_cy = ys + offset_pred[1][ys, xs] * size\n\n votes = np.zeros((size, size), dtype=np.float32)\n for cx, cy in zip(pred_cx, pred_cy):\n cxi = int(np.clip(round(cx), 0, size - 1)); cyi = int(np.clip(round(cy), 0, size - 1))\n votes[cyi, cxi] += 1\n\n # greedily take the highest-voted peak, suppress its neighborhood, repeat\n # (the same non-max-suppression idea as Lesson 39's detection boxes)\n peaks = []\n votes_work = votes.copy()\n for _ in range(6):\n idx = np.unravel_index(votes_work.argmax(), votes_work.shape)\n if votes_work[idx] < vote_thresh:\n break\n peaks.append((idx[1], idx[0]))\n y0, y1 = max(0, idx[0] - peak_dist), min(size, idx[0] + peak_dist + 1)\n x0, x1 = max(0, idx[1] - peak_dist), min(size, idx[1] + peak_dist + 1)\n votes_work[y0:y1, x0:x1] = 0\n\n if not peaks:\n return np.zeros_like(binary_mask, dtype=np.int64), []\n inst_pred = np.zeros_like(binary_mask, dtype=np.int64)\n peaks_arr = np.array(peaks)\n for y, x in zip(ys, xs):\n dists = (peaks_arr[:, 0] - x) ** 2 + (peaks_arr[:, 1] - y) ** 2\n inst_pred[y, x] = np.argmin(dists) + 1\n return inst_pred, peaks\n\nn_correct = 0\ninst_preds, all_peaks = [], []\nfor i in range(len(Xte)):\n binary_mask = sem_preds[i] == 1\n inst_pred, peaks = cluster_instances(binary_mask, offset_preds[i])\n inst_preds.append(inst_pred); all_peaks.append(peaks)\n true_n = len(set(inst_te[i].ravel()) - {0})\n if len(peaks) == true_n:\n n_correct += 1\n\nprint(f'offset-voting recovers the correct instance count: {n_correct}/{len(Xte)} scenes '\n f'({n_correct/len(Xte):.1%})')\nprint(f'(vs. {n_cc_correct}/{len(Xte)} for connected components: {n_cc_correct/len(Xte):.1%})')",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"id": "afe7a96d",
"source": "fig, axes = plt.subplots(3, 4, figsize=(9, 6.5))\nfor i in range(4):\n axes[0, i].imshow(scenes[split + i], cmap='gray'); axes[0, i].axis('off')\n axes[1, i].imshow(inst_te[i], cmap='viridis'); axes[1, i].axis('off')\n axes[2, i].imshow(inst_preds[i], cmap='viridis'); axes[2, i].axis('off')\nfor r, name in enumerate(['input', 'true instances', 'recovered via offset voting']):\n axes[r, 0].set_title(name, fontsize=9, loc='left')\nplt.tight_layout()\nplt.show()",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "0a46bfd2",
"source": "## Mask R-CNN, and the three flavors of \"segmentation\"\n\nThe offset-voting approach in this lesson is one real family of instance segmentation methods (related to techniques sometimes called \"instance embedding\" or center-voting). The other dominant approach, **Mask R-CNN** (He et al., 2017★), takes a more direct route: extend a two-stage detector (Lesson 40's R-CNN family) with a third output per detected box — alongside the existing class label and refined box coordinates, predict a small binary mask *within that box*. Detection already solves the \"how many objects, and roughly where\" problem via region proposals and NMS; Mask R-CNN just adds \"and here's this one's exact silhouette.\"\n\nPutting Lessons 37, 40, 41, and this lesson together, there are three distinct tasks that are easy to conflate:\n- **Classification** (Lesson 37): one label for the whole image.\n- **Semantic segmentation** (Lesson 41): one label per pixel, with no notion of separate objects of the same class.\n- **Instance segmentation** (this lesson): one label per pixel, *and* a distinct identity for every individual object — \"background,\" \"circle #1,\" \"circle #2,\" not just \"background,\" \"circle.\"\n\n(A fourth term, **panoptic segmentation**, unifies the last two: every pixel gets both a semantic class and, for countable \"thing\" classes like circles or cars, an instance ID — while uncountable \"stuff\" classes like sky or road are labeled semantically only, since instance identity doesn't make sense for them.)\n\n### Exercise\n\n1. Reduce `dist` in `make_scene` from the range `(6, 9)` to `(2, 5)`, making the circles overlap much more heavily (centers closer together). Does the offset-voting method's instance-count accuracy hold up, or does it start failing too — and does the failure mode look different from connected-components' failure?\n2. The clustering here uses `peak_dist=4` for non-max suppression on the vote accumulator. Try `peak_dist=8`. Does accuracy improve, get worse, or become sensitive to exactly how close two true circle centers happen to be in a given scene?\n3. This lesson only ever has 2 circles per scene. Extend `make_scene` to place a random number of circles (1 to 4) and adjust `cluster_instances`'s loop bound accordingly. Does the offset-voting approach's instance count accuracy hold up as the true number of instances grows?",
"metadata": {}
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 5
}