{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": "# Lesson 54: Depth Anything\n\nLesson 53 showed monocular depth's central weakness: a network trained on a small, narrow dataset only learns priors valid *within* that dataset's distribution, and gets confidently fooled outside it. **Depth Anything** (Yang et al., 2024) is a monocular depth foundation model built specifically to close that gap, using **self-training on unlabeled data**: a teacher model pseudo-labels a huge pool of unlabeled images, and a student trains on labeled data plus those pseudo-labels, at a scale (over 60 million unlabeled images) no manually-annotated depth dataset gets close to. This lesson builds that pipeline — and its result is a genuine, useful negative finding: pseudo-labeling from a biased teacher does not, by itself, fix the bias. Understanding *why* is what actually explains what Depth Anything gets right that a naive version of the same idea doesn't." }, { "cell_type": "code", "id": "f3d12ded", "source": "import numpy as np\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport matplotlib.pyplot as plt", "metadata": {}, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "b4c29825", "source": "## Setup: a small labeled set, a large unlabeled pool, a distribution gap\n\nReuse Lesson 53's size-cue depth scenes. The **labeled** set is small (20 scenes) and covers only a narrow depth range (0.4-0.6). The **unlabeled** pool is much larger (150 scenes, no depth labels — just images) and, like real-world unlabeled photo collections, spans the *full* range of conditions (depths 0.2-0.9). The **test** set matches that full range too — deliberately including depths the labeled set never showed.", "metadata": {} }, { "cell_type": "code", "id": "72eb14e2", "source": "SIZE = 32\n\ndef make_scene(rng, size=SIZE, n_objects=4, depth_range=(0.2, 0.9)):\n img = np.zeros((size, size), dtype=np.float32)\n depth = np.full((size, size), 1.0, dtype=np.float32)\n depths_used = rng.uniform(*depth_range, n_objects)\n depths_used.sort()\n for d in depths_used[::-1]:\n r = max(2, int(6 * (1 - d) + 1))\n cy = int(size * (0.3 + 0.6 * d))\n cx = rng.integers(r, size - r)\n yy, xx = np.mgrid[0:size, 0:size]\n m = ((xx - cx) ** 2 + (yy - cy) ** 2) <= r ** 2\n img[m] = 0.8\n depth[m] = d\n img = np.clip(img + rng.normal(0, 0.03, img.shape), 0, 1).astype(np.float32)\n return img, depth\n\nlabel_rng = np.random.default_rng(10)\nN_LABELED = 20\nX_labeled, D_labeled = [], []\nfor _ in range(N_LABELED):\n im, d = make_scene(label_rng, n_objects=label_rng.integers(2, 4), depth_range=(0.4, 0.6))\n X_labeled.append(im); D_labeled.append(d)\nX_labeled, D_labeled = np.array(X_labeled, dtype=np.float32), np.array(D_labeled, dtype=np.float32)\n\nunlabeled_rng = np.random.default_rng(20)\nN_UNLABELED = 150\nX_unlabeled = []\nfor _ in range(N_UNLABELED):\n im, _ = make_scene(unlabeled_rng, n_objects=unlabeled_rng.integers(2, 6), depth_range=(0.2, 0.9))\n X_unlabeled.append(im)\nX_unlabeled = np.array(X_unlabeled, dtype=np.float32)\n\ntest_rng = np.random.default_rng(30)\nN_TEST = 100\nX_test, D_test = [], []\nfor _ in range(N_TEST):\n im, d = make_scene(test_rng, n_objects=test_rng.integers(2, 6), depth_range=(0.2, 0.9))\n X_test.append(im); D_test.append(d)\nX_test, D_test = np.array(X_test, dtype=np.float32), np.array(D_test, dtype=np.float32)\n\nprint(f'labeled: {N_LABELED} scenes, depth range [0.4, 0.6]')\nprint(f'unlabeled: {N_UNLABELED} scenes, depth range [0.2, 0.9], no depth labels')\nprint(f'test: {N_TEST} scenes, depth range [0.2, 0.9]')", "metadata": {}, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "47b2dca4", "source": "## Teacher, pseudo-labels, student\n\nTrain a **teacher** (Lesson 41's U-Net) on the small labeled set only. Use it to pseudo-label every image in the unlabeled pool. Train a **student**, identical architecture, on the labeled set plus the pseudo-labeled pool combined — exactly Depth Anything's core recipe.", "metadata": {} }, { "cell_type": "code", "id": "41b907a9", "source": "class UNetTiny(nn.Module):\n def __init__(self):\n super().__init__()\n self.enc1 = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.ReLU())\n self.enc2 = nn.Sequential(nn.Conv2d(16, 32, 3, padding=1), nn.ReLU())\n self.pool = nn.MaxPool2d(2)\n self.up = nn.Upsample(scale_factor=2, mode='nearest')\n self.dec1 = nn.Sequential(nn.Conv2d(32 + 16, 16, 3, padding=1), nn.ReLU())\n self.out = nn.Conv2d(16, 1, 1)\n\n def forward(self, x):\n f1 = self.enc1(x)\n f2 = self.enc2(self.pool(f1))\n d1 = self.dec1(torch.cat([self.up(f2), f1], dim=1))\n return torch.sigmoid(self.out(d1)).squeeze(1)\n\ndef train_model(X, D, seed, epochs=200, lr=0.01):\n torch.manual_seed(seed)\n model = UNetTiny()\n opt = torch.optim.Adam(model.parameters(), lr=lr)\n Xt, Dt = torch.tensor(X).unsqueeze(1), torch.tensor(D)\n for _ in range(epochs):\n opt.zero_grad()\n loss = F.mse_loss(model(Xt), Dt)\n loss.backward()\n opt.step()\n return model\n\ndef eval_mae(model, X, D):\n with torch.no_grad():\n pred = model(torch.tensor(X).unsqueeze(1))\n return (pred - torch.tensor(D)).abs().mean().item()\n\nteacher = train_model(X_labeled, D_labeled, seed=0)\nteacher_mae = eval_mae(teacher, X_test, D_test)\nprint(f'teacher (small labeled set only) test MAE: {teacher_mae:.4f}')\n\nwith torch.no_grad():\n pseudo_labels = teacher(torch.tensor(X_unlabeled).unsqueeze(1)).numpy()\n\nX_combined = np.concatenate([X_labeled, X_unlabeled])\nD_combined = np.concatenate([D_labeled, pseudo_labels])\nstudent = train_model(X_combined, D_combined, seed=1)\nstudent_mae = eval_mae(student, X_test, D_test)\nprint(f'student (labeled + pseudo-labeled) test MAE: {student_mae:.4f}')", "metadata": {}, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "5e1f75ce", "source": "## Why didn't self-training help?\n\nThe student's error is essentially unchanged from the teacher's. This is not a bug — it's the expected outcome of naive self-training, and it's worth understanding precisely why. The pseudo-labels for the unlabeled pool came *entirely* from the teacher, and the teacher was never shown a depth outside [0.4, 0.6]. For any unlabeled scene with a true depth of, say, 0.8, the teacher's pseudo-label isn't a noisy version of the right answer — it's the teacher's same systematic extrapolation error from Lesson 53's fooling experiment, confidently applied and then handed to the student as if it were ground truth. Training on that just teaches the student to reproduce the teacher's mistake more efficiently. No genuinely new information about depths in [0.6, 0.9] ever entered the pipeline — self-training redistributes what a model already knows, and cannot manufacture what it doesn't.\n\n## What actually makes Depth Anything work\n\nThe gap between this toy failure and Depth Anything's real, well-documented success comes from three ingredients this lesson's setup deliberately lacks:\n\n- **Genuine distributional coverage.** Depth Anything's unlabeled pool is not a narrow synthetic distribution shifted slightly from the labeled set — it's 62 million *real* images spanning an enormous range of scenes, lighting, and object types. The teacher's errors on any single unlabeled image are still real errors, but averaged over that much genuine diversity, the student's training signal is far less systematically biased than this lesson's 150-image, single-cue toy pool.\n- **A strong pretrained backbone.** The teacher isn't a small CNN trained from scratch on a handful of labeled examples — it starts from a DINOv2 (Lesson 48) backbone already carrying rich, general-purpose visual priors learned from unlabeled data at a completely different scale, before ever seeing a single depth label.\n- **Aggressive perturbation during student training.** Depth Anything specifically injects strong image augmentations (and, in later variants, additional auxiliary losses) when training the student on pseudo-labeled data, so the student cannot simply memorize the teacher's exact outputs — it has to learn something more robust to reproduce them under distortion, which is what actually squeezes new generalization out of the process, rather than merely copying the teacher.\n\nNone of these fix this toy pipeline's fundamental problem (there is no genuinely new information about depths in [0.6, 0.9] anywhere in this dataset), but at real scale, \"genuinely new information\" is rarely completely absent from a sufficiently large and diverse unlabeled pool — which is exactly the condition self-training needs to be worth doing.\n\n### Exercise\n\n1. Change the unlabeled pool's `depth_range` to match the labeled set's `(0.4, 0.6)` exactly, instead of the full `(0.2, 0.9)` range. Does self-training help *now* — and does that support the claim that self-training's value depends entirely on whether the unlabeled pool's true (unobserved) labels actually differ from what the teacher already believes?\n2. Add Gaussian pixel noise to the unlabeled images *before* generating pseudo-labels but train the student on the *clean* images with those noisy-derived labels (or vice versa) — a crude stand-in for Depth Anything's perturbation trick. Does forcing this mismatch between what's pseudo-labeled and what's trained on change the student's test MAE at all?\n3. Retrain the teacher on a labeled set covering the full `(0.2, 0.9)` depth range (same 20 examples, just resample with `depth_range=(0.2, 0.9)`) instead of the narrow one. Does self-training help *this* teacher improve further on the same test set, now that there's no fundamental information gap to begin with?", "metadata": {} } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.0" } }, "nbformat": 4, "nbformat_minor": 5 }