{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson 53: Monocular Depth Estimation\n",
    "\n",
    "Lesson 21's stereo matching recovered depth from *two* images by measuring disparity \u2014 a direct geometric signal. **Monocular depth estimation** asks for the same per-pixel depth map from a *single* image, with no disparity available at all. This is fundamentally **ill-posed**: infinitely many 3D scenes produce the exact same 2D image (a toy car photographed up close is indistinguishable, pixel-for-pixel, from a real car photographed from far away). Modern monocular depth networks work anyway, by learning statistical priors about the world — typical object sizes, perspective, occlusion — from massive training sets. This lesson builds a small depth network from one such prior (size/perspective cues), and then deliberately breaks the prior to show exactly where the ill-posedness the intro paragraph mentioned actually bites."
   ]
  },
  {
   "cell_type": "code",
   "id": "8f492615",
   "source": "import numpy as np\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport matplotlib.pyplot as plt",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "d38872d9",
   "source": "## A scene with a size-based depth cue\n\nCircles at different depths, with **deliberately uniform brightness** \u2014 the only information available about depth is each circle's *size* (closer objects are drawn larger) and its vertical position (closer objects sit lower in the frame, a simple ground-plane perspective cue). This is a synthetic stand-in for the single most important prior a real monocular depth network learns: how large an object of a given kind \"should\" look at a given distance.",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "id": "da08a076",
   "source": "SIZE = 32\n\ndef make_scene(rng, size=SIZE, n_objects=4):\n    img = np.zeros((size, size), dtype=np.float32)\n    depth = np.full((size, size), 1.0, dtype=np.float32)  # background = far (depth = 1)\n    depths_used = rng.uniform(0.2, 0.9, n_objects)\n    depths_used.sort()\n    for d in depths_used[::-1]:  # draw closest (smallest depth) last, so it occludes farther ones\n        r = max(2, int(6 * (1 - d) + 1))       # closer (small d) -> larger radius\n        cy = int(size * (0.3 + 0.6 * d))\n        cx = rng.integers(r, size - r)\n        yy, xx = np.mgrid[0:size, 0:size]\n        m = ((xx - cx) ** 2 + (yy - cy) ** 2) <= r ** 2\n        img[m] = 0.8  # uniform intensity: size/position are the ONLY depth cues\n        depth[m] = d\n    img = np.clip(img + rng.normal(0, 0.03, img.shape), 0, 1).astype(np.float32)\n    return img, depth\n\nrng = np.random.default_rng(3)\nN = 300\nimgs, depths = [], []\nfor _ in range(N):\n    im, d = make_scene(rng)\n    imgs.append(im); depths.append(d)\nimgs = np.array(imgs, dtype=np.float32)\ndepths = np.array(depths, dtype=np.float32)\n\nsplit = int(0.85 * N)\nXtr, Dtr = imgs[:split], depths[:split]\nXte, Dte = imgs[split:], depths[split:]\n\nfig, axes = plt.subplots(2, 4, figsize=(9, 4.5))\nfor i in range(4):\n    axes[0, i].imshow(Xtr[i], cmap='gray'); axes[0, i].axis('off')\n    axes[1, i].imshow(Dtr[i], cmap='viridis_r', vmin=0.2, vmax=1.0); axes[1, i].axis('off')\naxes[0, 0].set_title('image', fontsize=9, loc='left')\naxes[1, 0].set_title('true depth (bright=near)', fontsize=9, loc='left')\nplt.show()",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "09b6895d",
   "source": "## A depth-regression U-Net\n\nDepth estimation is per-pixel regression rather than per-pixel classification (Lesson 41) \u2014 the exact same encoder-decoder-with-skip-connections architecture, with a single continuous output per pixel instead of a class distribution.",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "id": "4b182d0b",
   "source": "class UNetTiny(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.enc1 = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.ReLU())\n        self.enc2 = nn.Sequential(nn.Conv2d(16, 32, 3, padding=1), nn.ReLU())\n        self.pool = nn.MaxPool2d(2)\n        self.up = nn.Upsample(scale_factor=2, mode='nearest')\n        self.dec1 = nn.Sequential(nn.Conv2d(32 + 16, 16, 3, padding=1), nn.ReLU())\n        self.out = nn.Conv2d(16, 1, 1)\n\n    def forward(self, x):\n        f1 = self.enc1(x)\n        f2 = self.enc2(self.pool(f1))\n        d1 = self.dec1(torch.cat([self.up(f2), f1], dim=1))\n        return torch.sigmoid(self.out(d1)).squeeze(1)  # depth in (0, 1)\n\ntorch.manual_seed(0)\nmodel = UNetTiny()\nopt = torch.optim.Adam(model.parameters(), lr=0.01)\nXt = torch.tensor(Xtr).unsqueeze(1); Dt = torch.tensor(Dtr)\nfor _ in range(300):\n    opt.zero_grad()\n    loss = F.mse_loss(model(Xt), Dt)\n    loss.backward()\n    opt.step()\n\nwith torch.no_grad():\n    pred_te = model(torch.tensor(Xte).unsqueeze(1))\n\nmae = (pred_te - torch.tensor(Dte)).abs().mean().item()\nmean_depth_baseline = np.abs(Dte - Dtr.mean()).mean()\nprint(f'mean absolute depth error:                        {mae:.4f}  (depth range is [0.2, 1.0])')\nprint(f'baseline (predict the mean training depth everywhere): {mean_depth_baseline:.4f}')",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "id": "1605053e",
   "source": "fig, axes = plt.subplots(3, 4, figsize=(9, 6.5))\nfor i in range(4):\n    axes[0, i].imshow(Xte[i], cmap='gray'); axes[0, i].axis('off')\n    axes[1, i].imshow(Dte[i], cmap='viridis_r', vmin=0.2, vmax=1.0); axes[1, i].axis('off')\n    axes[2, i].imshow(pred_te[i], cmap='viridis_r', vmin=0.2, vmax=1.0); axes[2, i].axis('off')\nfor r, name in enumerate(['input', 'true depth', 'predicted depth']):\n    axes[r, 0].set_title(name, fontsize=9, loc='left')\nplt.tight_layout()\nplt.show()",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "66a8b711",
   "source": "## Breaking the prior: the \"toy car\" problem\n\nThe network has clearly learned \"bigger circle = closer\" \u2014 but that's a *learned statistical association*, not a measurement. If an object's size doesn't follow the training distribution's rule (a miniature scale model, a projected image, a genuinely huge object) the network has no way to tell, because it never had access to true 3D geometry in the first place. Test this directly: two circles with the *identical* true depth, but one sized correctly for that depth and one sized as if it were much closer.",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "id": "e1bacabe",
   "source": "def make_single_object(cx, cy, r, size=SIZE):\n    img = np.zeros((size, size), dtype=np.float32)\n    yy, xx = np.mgrid[0:size, 0:size]\n    mask = ((xx - cx) ** 2 + (yy - cy) ** 2) <= r ** 2\n    img[mask] = 0.8\n    return img, mask\n\ntrue_depth = 0.8  # actually far away, in both cases below\nnormal_r = max(2, int(6 * (1 - true_depth) + 1))   # the radius a far object should have\nfaked_r = max(2, int(6 * (1 - 0.2) + 1))            # the radius as if it were very close instead\n\nimg_normal, mask_normal = make_single_object(16, 16, normal_r)\nimg_faked, mask_faked = make_single_object(16, 16, faked_r)\n\nwith torch.no_grad():\n    pred_normal = model(torch.tensor(img_normal[None, None]))[0].numpy()\n    pred_faked = model(torch.tensor(img_faked[None, None]))[0].numpy()\n\nprint(f'true depth in both cases: {true_depth}  (identical \u2014 nothing about the actual geometry changed)')\nprint(f'predicted depth, correctly-sized-for-its-depth object (r={normal_r}): {pred_normal[mask_normal].mean():.3f}')\nprint(f'predicted depth, oversized object (r={faked_r}, same true depth):     {pred_faked[mask_faked].mean():.3f}')\n\nfig, axes = plt.subplots(1, 2, figsize=(6, 3))\naxes[0].imshow(img_normal, cmap='gray'); axes[0].set_title(f'normal size\\npred depth={pred_normal[mask_normal].mean():.2f}', fontsize=9)\naxes[1].imshow(img_faked, cmap='gray'); axes[1].set_title(f'oversized\\npred depth={pred_faked[mask_faked].mean():.2f}', fontsize=9)\nfor ax in axes: ax.axis('off')\nplt.show()",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "0db6603c",
   "source": "The network confidently reports two very different depths for two objects at the exact same true distance \u2014 it isn't measuring depth, it's pattern-matching against \"how big things of this kind usually look at this distance,\" and that pattern-match can be fooled by anything that violates the training distribution's assumptions (miniatures, forced-perspective photography, an object of an unusual size). This is the real, well-documented failure mode of monocular depth networks in practice, and it is exactly what \"ill-posed without additional information\" predicted at the start of this lesson: the 2D image alone never contained the true depth, only cues correlated with it in typical training data.\n\nThe practical fix real systems use is exactly the one this lesson opened by contrasting against: whenever a second geometric measurement is available \u2014 a second camera (Lesson 21's stereo, Lesson 55), LiDAR, or known camera motion (structure from motion, Lesson 28) \u2014 it should be trusted over a monocular size prior. Monocular depth is most valuable precisely where those aren't available (a single photograph, a single video frame with no motion baseline), and least trustworthy exactly where its training-distribution assumptions break down.\n\n### Exercise\n\n1. Retrain with brightness restored as a depth cue (`img[m] = 0.4 + 0.5 * (1 - d)`, matching the very first version of this experiment) instead of uniform intensity. Rerun the fooling test \u2014 does the size-based illusion get weaker, because the model now has a second, un-fooled cue (brightness) to fall back on?\n2. The `silog` (scale-invariant log RMSE) metric from real monocular-depth research measures error up to a global multiplicative scale factor, since monocular predictions are often only correct *up to scale*. Implement it (`sqrt(mean((log(pred) - log(true))^2) - mean(log(pred) - log(true))^2)`) and compare it to plain MAE \u2014 does the model's ranking of \"how good is this prediction\" change between the two metrics?\n3. Train a second model on scenes with `n_objects` fixed to exactly 1 instead of drawn from `make_scene`'s default. Does the single-object model's depth-from-size prior transfer to the original multi-object test scenes, or does it fail specifically when circles occlude each other?",
   "metadata": {}
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}