Photo by Felix Hanspach on Unsplash
" ] }, { "cell_type": "markdown", "id": "ae9b4441", "metadata": {}, "source": [ "The cyclist is cleanly labeled `person`, with the bicycle's wheels correctly labeled `bicycle` just below — both real, correctly-placed detections. The tram is where it gets interesting: the model splits it between `train` (the upper, windowed section) and `bus` (the lower body) rather than picking one label for the whole vehicle. That's not really a mistake so much as an honest reflection of VOC's limited vocabulary — a tram is a genuinely ambiguous case between \"train\" and \"bus\" for a model that was never given a \"tram\" class to choose from, and the two guesses roughly track a real visual seam in the vehicle (windows vs. body) rather than being random. A second, smaller `person` blob on the right edge catches the blurred pedestrian in the background." ] }, { "cell_type": "markdown", "id": "4433b6cd", "metadata": {}, "source": [ "## Adding skip connections: U-Net\n", "\n", "`FCNTiny`'s decoder only sees the pooled, upsampled bottleneck features. After two rounds of pooling, that bottleneck has just a quarter of the input's spatial resolution, and small or closely-packed shapes can blur together or lose their boundaries entirely at that resolution. **U-Net** (Ronneberger, Fischer & Brox, 2015★) fixes this with **skip connections**: at each decoder stage, it concatenates the encoder's feature map from the *matching resolution*, preserving the fine-grained details that would otherwise be lost during pooling. This is similar in spirit to Lesson 37's residual connections, but the purpose and mechanism differ: U-Net concatenates features across the encoder-decoder boundary, rather than adding a residual update within a single network stack." ] }, { "cell_type": "code", "execution_count": null, "id": "eb504d2b", "metadata": {}, "outputs": [], "source": [ "class UNetTiny(nn.Module):\n", " def __init__(self, n_classes=3):\n", " super().__init__()\n", " self.enc1 = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.ReLU())\n", " self.enc2 = nn.Sequential(nn.Conv2d(16, 32, 3, padding=1), nn.ReLU())\n", " self.enc3 = nn.Sequential(nn.Conv2d(32, 64, 3, padding=1), nn.ReLU())\n", " self.pool = nn.MaxPool2d(2)\n", " self.up = nn.Upsample(scale_factor=2, mode='nearest')\n", " self.dec2 = nn.Sequential(nn.Conv2d(64 + 32, 32, 3, padding=1), nn.ReLU())\n", " self.dec1 = nn.Sequential(nn.Conv2d(32 + 16, 16, 3, padding=1), nn.ReLU())\n", " self.out = nn.Conv2d(16, n_classes, 1)\n", "\n", " def forward(self, x):\n", " f1 = self.enc1(x) # (B,16,H,W)\n", " f2 = self.enc2(self.pool(f1)) # (B,32,H/2,W/2)\n", " f3 = self.enc3(self.pool(f2)) # (B,64,H/4,W/4)\n", " d2 = self.dec2(torch.cat([self.up(f3), f2], dim=1)) # skip from f2\n", " d1 = self.dec1(torch.cat([self.up(d2), f1], dim=1)) # skip from f1\n", " return self.out(d1)\n", "\n", "torch.manual_seed(0)\n", "model_unet = UNetTiny()\n", "opt2 = torch.optim.Adam(model_unet.parameters(), lr=0.01)\n", "for _ in range(200):\n", " opt2.zero_grad()\n", " loss = F.cross_entropy(model_unet(Xt), Mt)\n", " loss.backward()\n", " opt2.step()\n", "\n", "with torch.no_grad():\n", " preds_unet = model_unet(torch.tensor(Xte).unsqueeze(1)).argmax(1).numpy()\n", "\n", "pixel_acc_unet = (preds_unet == Mte).mean()\n", "ious_unet = mean_iou(preds_unet, Mte)\n", "\n", "print(f'{\"\":>18} {\"pixel acc\":>10} {\"mean IoU\":>10}')\n", "print(f'{\"FCN (no skip)\":>18} {pixel_acc_fcn:>10.1%} {np.mean(ious_fcn):>10.3f}')\n", "print(f'{\"U-Net (skip)\":>18} {pixel_acc_unet:>10.1%} {np.mean(ious_unet):>10.3f}')" ] }, { "cell_type": "code", "execution_count": null, "id": "2fcb0fbc", "metadata": {}, "outputs": [], "source": [ "fig, axes = plt.subplots(4, 4, figsize=(9, 9))\n", "for i in range(4):\n", " axes[0, i].imshow(Xte[i], cmap='gray')\n", " axes[1, i].imshow(Mte[i], cmap='viridis', vmin=0, vmax=2)\n", " axes[2, i].imshow(preds_fcn[i], cmap='viridis', vmin=0, vmax=2)\n", " axes[3, i].imshow(preds_unet[i], cmap='viridis', vmin=0, vmax=2)\n", " for r in range(4):\n", " axes[r, i].axis('off')\n", "for r, name in enumerate(['input', 'ground truth mask', 'pred (FCN)', 'pred (U-Net)']):\n", " axes[r, 0].set_title(name, fontsize=9, loc='left')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "b4cd6aaa", "metadata": {}, "source": [ "For small, tightly packed shapes, skip connections provide a substantial mean-IoU improvement over a plain FCN. An FCN cannot recover detail that was lost in the low-resolution bottleneck, because interpolation can only work on existing information. Skip connections fix this problem by giving the decoder a second, non-bottlenecked path to the encoder's high-resolution features, so precise boundaries never have to survive the bottleneck in the first place.\n", "\n", "This helps explain why U-Net remains popular for tasks requiring precise boundaries, especially medical imaging. But skip connections are not universal: popular networks like plain **FCN** and **DeepLabV3** have none, and even **DeepLabV3+** adds only a single one, not a full U-Net-style connection at every resolution. So while skip connections are a useful tool, they are not a requirement — their value depends on how much fine spatial detail the task demands." ] }, { "cell_type": "markdown", "id": "d58f6e24", "metadata": {}, "source": [ "### Exercises\n", "\n", "1. Increase `n_shapes` from 5 to 10, making the scene more crowded. Does the `FCNTiny`-vs-`UNetTiny` gap in mean IoU get larger or smaller? What does that suggest about when skip connections matter most?\n", "2. This lesson's loss is plain per-pixel cross-entropy. Print the per-pixel class weights implied by `class pixel fractions` above and try `F.cross_entropy(logits, Mt, weight=inverse_class_freq)` (Lesson 36's imbalance fix, applied here) to see whether it changes the circle/square IoU balance.\n", "3. `nn.Upsample(mode='nearest')` was used for simplicity. Try `mode='bilinear', align_corners=False` instead (Lesson 9's bilinear interpolation, now inside a network) and compare mean IoU for both `FCNTiny` and `UNetTiny`. Does the smoother upsampling help more or less than the skip connection does?\n", "4. Print the confidence (softmax probability of the predicted class) at a few pixels along the seam where the real FCN result switches from `train` to `bus` on the tram. Is the model confidently split, or genuinely uncertain right at that boundary — and what would you expect from a network being asked to choose between two classes when the true object matches neither one exactly?" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.0" } }, "nbformat": 4, "nbformat_minor": 5 }