{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": "# Lesson 55: Stereo Depth and FoundationStereo\n\nLesson 21 built classical stereo matching: slide a small window along corresponding scanlines, compare intensity patches, take the disparity with lowest cost. It works well on textured surfaces and fails predictably on flat, low-texture regions — the **aperture problem**, where a small window genuinely cannot tell which shift is correct. This lesson builds a small *learned* stereo network (a differentiable cost volume plus a convolutional refinement step, the core idea behind modern deep stereo systems up through **FoundationStereo**, Wen et al., 2025) and shows concretely why learning beats pure local matching: a network's receptive field lets it borrow context from *outside* the ambiguous region, something a fixed local window structurally cannot do."
},
{
"cell_type": "code",
"id": "4df836d0",
"source": "import numpy as np\nimport cv2\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport matplotlib.pyplot as plt",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "4773fd1a",
"source": "## A stereo pair with a deliberately textureless band\n\nGenerate a textured \"world\" strip, wide enough that shifting it produces a valid left/right pair at a known integer disparity — Lesson 21's synthetic stereo setup, extended with one twist: a horizontal band is flattened to constant intensity, with no texture at all inside it.",
"metadata": {}
},
{
"cell_type": "code",
"id": "78772704",
"source": "SIZE = 32\nMAX_DISP = 8\n\ndef make_stereo_pair(rng, size=SIZE, max_disp=MAX_DISP, textureless=True):\n world = rng.uniform(0, 1, (size, size + max_disp)).astype(np.float32)\n world = cv2.GaussianBlur(world, (3, 3), 0)\n for _ in range(15): # a few sharp features, so texture is patchy rather than uniform\n cx, cy = rng.integers(0, size + max_disp), rng.integers(0, size)\n world[max(0, cy-1):cy+2, max(0, cx-1):cx+2] = rng.uniform(0, 1)\n\n disparity = rng.integers(1, max_disp)\n band = None\n if textureless:\n b0 = rng.integers(0, size - size // 3)\n band = slice(b0, b0 + size // 3)\n world[band, :] = 0.5 # a flat strip: no texture, so no local cue to match against\n\n left = world[:, max_disp:max_disp + size]\n right = world[:, max_disp - disparity: max_disp - disparity + size]\n disp_map = np.full((size, size), float(disparity), dtype=np.float32)\n return left.astype(np.float32), right.astype(np.float32), disp_map, band\n\nrng = np.random.default_rng(11)\nleft0, right0, disp0, band0 = make_stereo_pair(rng)\n\nfig, axes = plt.subplots(1, 3, figsize=(9, 3))\naxes[0].imshow(left0, cmap='gray'); axes[0].set_title('left image'); axes[0].axis('off')\naxes[1].imshow(right0, cmap='gray'); axes[1].set_title('right image'); axes[1].axis('off')\naxes[2].imshow(disp0, cmap='viridis'); axes[2].set_title(f'true disparity ({disp0[0,0]:.0f} px)'); axes[2].axis('off')\nfor ax in axes:\n ax.axhline(band0.start, color='red', linestyle='--', linewidth=0.7)\n ax.axhline(band0.stop, color='red', linestyle='--', linewidth=0.7)\nplt.show()",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "72a3e3de",
"source": "## Classical block matching, and where it fails\n\nLesson 21's block matching: for every pixel, compare a small window against candidate windows at every disparity, keep the lowest-cost match.",
"metadata": {}
},
{
"cell_type": "code",
"id": "703cc7a7",
"source": "def block_match(left, right, max_disp=MAX_DISP, win=3):\n H, W = left.shape\n half = win // 2\n disp_map = np.zeros((H, W), dtype=np.float32)\n left_pad = np.pad(left, half, mode='edge')\n right_pad = np.pad(right, half, mode='edge')\n for y in range(H):\n for x in range(W):\n patch_l = left_pad[y:y+win, x:x+win]\n best_d, best_cost = 0, np.inf\n for d in range(max_disp):\n x2 = x + d\n if x2 + win > right_pad.shape[1]:\n continue\n patch_r = right_pad[y:y+win, x2:x2+win]\n cost = np.sum((patch_l - patch_r) ** 2)\n if cost < best_cost:\n best_cost, best_d = cost, d\n disp_map[y, x] = best_d\n return disp_map\n\nbm_disp0 = block_match(left0, right0)\nerr_textured = np.abs(bm_disp0[:band0.start, :] - disp0[:band0.start, :]).mean()\nerr_textureless = np.abs(bm_disp0[band0, :] - disp0[band0, :]).mean()\nprint(f'block matching MAE, textured region: {err_textured:.2f} px')\nprint(f'block matching MAE, textureless region: {err_textureless:.2f} px (true disparity = {disp0[0,0]:.0f})')",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "53a68156",
"source": "Block matching is nearly exact where texture exists, and badly wrong inside the flat band — a small window there genuinely contains no information to disambiguate disparity, so it locks onto whatever noise happens to look marginally better. No amount of clever thresholding fixes this: the window is too small to see anything else.\n\n## A learned cost volume\n\nDeep stereo networks (PSMNet, RAFT-Stereo, and now FoundationStereo) replace hand-picked window comparison with three learned pieces:\n\n1. A small CNN extracts a **feature map** from each image (not raw pixels — learned features, more robust than intensity alone).\n2. A **cost volume** is built by correlating the left feature map against the right feature map, shifted by every candidate disparity — the same all-disparities-at-once idea as block matching's search, just done at the feature level.\n3. A **refinement network** (a few more conv layers, working across the whole cost volume) converts a soft, smooth probability distribution over disparities into a final estimate — this step has a receptive field spanning far more of the image than any local window, so it can borrow context from well outside an ambiguous region.",
"metadata": {}
},
{
"cell_type": "code",
"id": "bc1e51d5",
"source": "class FeatureNet(nn.Module):\n def __init__(self, out_ch=8):\n super().__init__()\n self.net = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.Conv2d(16, out_ch, 3, padding=1))\n\n def forward(self, x):\n return self.net(x)\n\nclass StereoNet(nn.Module):\n def __init__(self, max_disp=MAX_DISP):\n super().__init__()\n self.feat = FeatureNet()\n self.max_disp = max_disp\n self.refine = nn.Sequential(\n nn.Conv2d(max_disp, 16, 3, padding=1), nn.ReLU(),\n nn.Conv2d(16, max_disp, 3, padding=1),\n )\n\n def forward(self, left, right):\n fl, fr = self.feat(left), self.feat(right)\n cost_volume = []\n for d in range(self.max_disp):\n shifted = fr if d == 0 else F.pad(fr[:, :, :, :-d], (d, 0))\n cost_volume.append((fl * shifted).sum(dim=1)) # correlation at this disparity\n cost_volume = self.refine(torch.stack(cost_volume, dim=1)) # regularize across space\n probs = F.softmax(cost_volume, dim=1)\n disp_range = torch.arange(self.max_disp, dtype=torch.float32).view(1, -1, 1, 1)\n return (probs * disp_range).sum(dim=1) # soft-argmin: expected disparity, sub-pixel\n\nrng2 = np.random.default_rng(11)\nN = 300\nlefts, rights, disp_maps = [], [], []\nfor _ in range(N):\n l, r, d, _ = make_stereo_pair(rng2)\n lefts.append(l); rights.append(r); disp_maps.append(d)\nlefts, rights, disp_maps = np.array(lefts), np.array(rights), np.array(disp_maps)\n\nsplit = int(0.85 * N)\nLtr, Rtr, Dtr = lefts[:split], rights[:split], disp_maps[:split]\nLte, Rte, Dte = lefts[split:], rights[split:], disp_maps[split:]\n\ntorch.manual_seed(0)\nmodel = StereoNet()\nopt = torch.optim.Adam(model.parameters(), lr=0.01)\nLt = torch.tensor(Ltr).unsqueeze(1); Rt = torch.tensor(Rtr).unsqueeze(1); Dt = torch.tensor(Dtr)\nfor _ in range(300):\n opt.zero_grad()\n loss = F.l1_loss(model(Lt, Rt), Dt)\n loss.backward()\n opt.step()\n\nwith torch.no_grad():\n pred_te = model(torch.tensor(Lte).unsqueeze(1), torch.tensor(Rte).unsqueeze(1))\nmae = (pred_te - torch.tensor(Dte)).abs().mean().item()\nprint(f'learned stereo network MAE (overall, includes textureless bands): {mae:.3f} px')",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "c4e1bf4b",
"source": "## Head to head, on the textureless band specifically\n\nCompare both methods on a single fresh pair, restricted to exactly the pixels inside the flat, featureless band — precisely where block matching's local window has nothing to go on.",
"metadata": {}
},
{
"cell_type": "code",
"id": "b4a3eec9",
"source": "test_rng = np.random.default_rng(99)\nleft1, right1, disp1, band1 = make_stereo_pair(test_rng)\nbm_disp1 = block_match(left1, right1)\nwith torch.no_grad():\n learned_disp1 = model(torch.tensor(left1[None, None]), torch.tensor(right1[None, None]))[0].numpy()\n\nbm_err_band = np.abs(bm_disp1[band1, :] - disp1[band1, :]).mean()\nlearned_err_band = np.abs(learned_disp1[band1, :] - disp1[band1, :]).mean()\nprint(f'textureless-band MAE: block matching = {bm_err_band:.2f} px, learned = {learned_err_band:.2f} px')\n\nfig, axes = plt.subplots(1, 4, figsize=(11, 3))\naxes[0].imshow(left1, cmap='gray'); axes[0].set_title('left'); axes[0].axis('off')\naxes[1].imshow(disp1, cmap='viridis', vmin=0, vmax=MAX_DISP); axes[1].set_title('true disparity'); axes[1].axis('off')\naxes[2].imshow(bm_disp1, cmap='viridis', vmin=0, vmax=MAX_DISP); axes[2].set_title('block matching'); axes[2].axis('off')\naxes[3].imshow(learned_disp1, cmap='viridis', vmin=0, vmax=MAX_DISP); axes[3].set_title('learned'); axes[3].axis('off')\nfor ax in axes:\n ax.axhline(band1.start, color='red', linestyle='--', linewidth=0.7)\n ax.axhline(band1.stop, color='red', linestyle='--', linewidth=0.7)\nplt.show()",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "6d0c20b7",
"source": "The learned network's error inside the flat band is far lower than block matching's. Nothing in the band itself became less ambiguous — the refinement network's convolutions have a receptive field extending well beyond the band's own pixels, so the disparity estimate for a pixel in the middle of the flat region is influenced by evidence from the textured pixels around and above it. A fixed 3x3 window can never do this; it only ever sees what's directly inside it.\n\n## FoundationStereo, and what \"foundation\" adds here\n\n**FoundationStereo** (NVIDIA, 2025) is this same recipe — feature extraction, cost volume, learned refinement (via an iterative update scheme derived from RAFT and RAFT-Stereo, rather than the one-shot softmax used here) — trained at foundation-model scale on a mix of large synthetic stereo datasets and real captured data, specifically to generalize to *new* cameras, scenes, and domains without per-dataset fine-tuning. That's the same \"learn a prior at scale, use it zero-shot\" story as Lesson 49's CLIP and Lesson 51's SAM, now applied to metric depth from a calibrated stereo pair rather than a single monocular image — which sidesteps Lesson 53's fundamental scale-ambiguity problem entirely, since a calibrated stereo baseline gives disparity a real geometric meaning (Lesson 21-23) that a single image never has.\n\n### Exercise\n\n1. Increase the textureless band's width from `size // 3` to `size // 2` in `make_stereo_pair`. Does the learned network's advantage over block matching grow, shrink, or stay about the same as the ambiguous region gets larger relative to the image?\n2. `StereoNet`'s `refine` step is two convolutional layers with a 3x3 kernel each, giving a limited receptive field. Add a third conv layer and rerun the textureless-band comparison — does a larger receptive field (more surrounding context reachable) improve the band's disparity estimate further?\n3. Remove the `refine` network entirely (use `probs = F.softmax(cost_volume, dim=1)` directly on the raw correlation cost volume, with no learned regularization step) and retrain. Does the network's textureless-band performance collapse back toward block matching's, confirming that spatial context — not the learned features alone — is what closes the gap?",
"metadata": {}
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 5
}