{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": "# Lesson 56: Feed-Forward Camera Pose Estimation\n\nLesson 22's RANSAC and Lesson 28's structure from motion recover camera geometry through an iterative pipeline: detect features, match them, run RANSAC to reject outliers, solve for the essential matrix, decompose it into rotation and translation. Each stage is a separate, hand-designed algorithm. **VGGT** (Visual Geometry Grounded Transformer, Wang et al., 2025) replaces the entire pipeline with a single feed-forward network: feed it correspondences (or even raw images) from multiple views, and it directly regresses camera poses in one forward pass, with no explicit RANSAC step anywhere. This lesson builds a small version of that idea and compares it against the classical pipeline it replaces — including exactly where each one wins."
},
{
"cell_type": "code",
"id": "5cb7f003",
"source": "import numpy as np\nimport cv2\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport matplotlib.pyplot as plt",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "677dd18d",
"source": "## Synthetic two-view correspondences, with outliers\n\nA random 3D point cloud, projected into two camera views related by a known random rotation and translation *direction* (Lesson 25's pinhole projection). Translation *direction* is the most a calibrated two-view pair can ever recover — absolute scale is fundamentally unobservable from two views alone (the same monocular scale ambiguity from Lesson 53, in a different guise). A controllable fraction of the correspondences are replaced with random mismatches, simulating the false matches Lesson 22's RANSAC was built to survive.",
"metadata": {}
},
{
"cell_type": "code",
"id": "76fa094c",
"source": "K = np.array([[50., 0, 32], [0, 50., 32], [0, 0, 1]])\nN_POINTS = 40\n\ndef random_rotation(rng, max_angle_deg=30):\n axis = rng.normal(size=3); axis /= np.linalg.norm(axis)\n angle = np.radians(rng.uniform(-max_angle_deg, max_angle_deg))\n Kx = np.array([[0, -axis[2], axis[1]], [axis[2], 0, -axis[0]], [-axis[1], axis[0], 0]])\n return np.eye(3) + np.sin(angle) * Kx + (1 - np.cos(angle)) * (Kx @ Kx)\n\ndef make_pair(rng, n_points=N_POINTS, outlier_frac=0.0):\n pts3d = rng.uniform(-2, 2, (n_points, 3)) + np.array([0, 0, 8])\n R_true = random_rotation(rng)\n t_dir = rng.uniform(-1, 1, 3); t_dir /= np.linalg.norm(t_dir)\n\n def project(pts, R, t):\n cam_pts = (R @ pts.T).T + t\n proj = (K @ cam_pts.T).T\n return proj[:, :2] / proj[:, 2:3]\n\n pts1 = project(pts3d, np.eye(3), np.zeros(3))\n pts2 = project(pts3d, R_true, t_dir * 2.0)\n\n n_outliers = int(outlier_frac * n_points)\n if n_outliers > 0:\n idx = rng.choice(n_points, n_outliers, replace=False)\n pts2[idx] = rng.uniform(0, 64, (n_outliers, 2))\n\n K_inv = np.linalg.inv(K) # normalize to camera coordinates, so the network never needs to learn K\n npts1 = (K_inv @ np.hstack([pts1, np.ones((n_points, 1))]).T).T[:, :2]\n npts2 = (K_inv @ np.hstack([pts2, np.ones((n_points, 1))]).T).T[:, :2]\n return npts1.astype(np.float32), npts2.astype(np.float32), R_true.astype(np.float32), t_dir.astype(np.float32)\n\ndef rotation_angle_error(R_est, R_true):\n R_diff = R_est.T @ R_true\n cos_angle = np.clip((np.trace(R_diff) - 1) / 2, -1, 1)\n return np.degrees(np.arccos(cos_angle))\n\ndef translation_angle_error(t_est, t_true):\n t_est = t_est / (np.linalg.norm(t_est) + 1e-8)\n t_true = t_true / np.linalg.norm(t_true)\n return np.degrees(np.arccos(np.clip(np.dot(t_est, t_true), -1, 1)))",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "341abd85",
"source": "## The classical pipeline: essential matrix + RANSAC\n\n`cv2.findEssentialMat` with `RANSAC` (Lesson 22) followed by `cv2.recoverPose` (Lesson 26) — the standard textbook approach to two-view relative pose, and the exact pipeline VGGT-style networks are built to replace.",
"metadata": {}
},
{
"cell_type": "code",
"id": "b06f7567",
"source": "def classical_pose(npts1, npts2):\n pts1_px = (K @ np.hstack([npts1, np.ones((N_POINTS, 1))]).T).T[:, :2].astype(np.float32)\n pts2_px = (K @ np.hstack([npts2, np.ones((N_POINTS, 1))]).T).T[:, :2].astype(np.float32)\n E, mask = cv2.findEssentialMat(pts1_px, pts2_px, K, method=cv2.RANSAC, prob=0.999, threshold=1.0)\n if E is None:\n return np.eye(3), np.array([0, 0, 1.0])\n _, R, t, _ = cv2.recoverPose(E, pts1_px, pts2_px, K)\n return R, t.ravel()\n\nrng = np.random.default_rng(0)\nfor outlier_frac in [0.0, 0.2, 0.4]:\n rot_errs, trans_errs = [], []\n for _ in range(20):\n p1, p2, R_true, t_true = make_pair(rng, outlier_frac=outlier_frac)\n R_est, t_est = classical_pose(p1, p2)\n rot_errs.append(rotation_angle_error(R_est, R_true))\n trans_errs.append(translation_angle_error(t_est, t_true))\n print(f'outlier_frac={outlier_frac}: classical rot err={np.mean(rot_errs):.2f} deg, '\n f'trans dir err={np.mean(trans_errs):.2f} deg')",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "19d5e99b",
"source": "## A feed-forward pose network\n\nEvery correspondence `(x1, y1, x2, y2)` becomes a token; a small Transformer encoder (Lesson 45) lets every correspondence attend to every other one — this is the piece a naive PointNet-style max-pooling architecture is missing: recovering a two-view geometric relationship (especially translation direction) fundamentally requires reasoning about how correspondences relate to *each other*, not just processing each one independently and pooling. Rotation is regressed via a continuous 6D representation (Zhou et al., 2019 — two 3D vectors, Gram-Schmidt-orthonormalized into a valid rotation matrix, avoiding the discontinuities of quaternions or Euler angles), and translation direction as a unit 3-vector.",
"metadata": {}
},
{
"cell_type": "code",
"id": "8814b2e1",
"source": "def rot6d_to_matrix(x):\n a1, a2 = x[..., :3], x[..., 3:]\n b1 = F.normalize(a1, dim=-1)\n b2 = a2 - (b1 * a2).sum(-1, keepdim=True) * b1\n b2 = F.normalize(b2, dim=-1)\n b3 = torch.cross(b1, b2, dim=-1)\n return torch.stack([b1, b2, b3], dim=-1)\n\nclass PoseNet(nn.Module):\n def __init__(self, hidden=64, n_heads=4, n_layers=2):\n super().__init__()\n self.embed = nn.Linear(4, hidden)\n layer = nn.TransformerEncoderLayer(hidden, n_heads, dim_feedforward=hidden * 2,\n batch_first=True, dropout=0.0)\n self.encoder = nn.TransformerEncoder(layer, num_layers=n_layers)\n self.head = nn.Sequential(nn.Linear(hidden, hidden), nn.ReLU(), nn.Linear(hidden, 9))\n\n def forward(self, corr): # corr: (B, N, 4) = [x1, y1, x2, y2]\n tok = self.encoder(self.embed(corr))\n pooled = tok.mean(dim=1)\n out = self.head(pooled)\n R = rot6d_to_matrix(out[..., :6])\n t = F.normalize(out[..., 6:], dim=-1)\n return R, t\n\ndef make_batch(rng, batch_size, outlier_frac):\n corrs, Rs, ts = [], [], []\n for _ in range(batch_size):\n p1, p2, R, t = make_pair(rng, outlier_frac=outlier_frac)\n corrs.append(np.concatenate([p1, p2], axis=1))\n Rs.append(R); ts.append(t)\n return (torch.tensor(np.array(corrs)), torch.tensor(np.array(Rs)), torch.tensor(np.array(ts)))\n\ntorch.manual_seed(0)\nmodel = PoseNet()\nopt = torch.optim.Adam(model.parameters(), lr=0.001)\ntrain_rng = np.random.default_rng(500)\nfor epoch in range(1400):\n outlier_frac = train_rng.uniform(0, 0.5) # train across a range of outlier ratios\n corr, R_true, t_true = make_batch(train_rng, 32, outlier_frac)\n R_pred, t_pred = model(corr)\n rot_loss = ((R_pred - R_true) ** 2).sum(dim=(1, 2)).mean()\n trans_loss = (1 - (t_pred * t_true).sum(-1)).mean()\n loss = rot_loss + trans_loss\n opt.zero_grad()\n loss.backward()\n opt.step()\n\nprint(f'final training loss: {loss.item():.4f}')",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "e322b23c",
"source": "## Head to head: classical RANSAC vs. feed-forward network, across outlier ratios",
"metadata": {}
},
{
"cell_type": "code",
"id": "8ef27b04",
"source": "eval_rng = np.random.default_rng(999)\nresults = {}\nfor outlier_frac in [0.0, 0.2, 0.4]:\n learned_rot, learned_trans, classical_rot, classical_trans = [], [], [], []\n for _ in range(20):\n p1, p2, R_true, t_true = make_pair(eval_rng, outlier_frac=outlier_frac)\n with torch.no_grad():\n corr = torch.tensor(np.concatenate([p1, p2], axis=1))[None]\n R_pred, t_pred = model(corr)\n learned_rot.append(rotation_angle_error(R_pred[0].numpy(), R_true))\n learned_trans.append(translation_angle_error(t_pred[0].numpy(), t_true))\n\n R_c, t_c = classical_pose(p1, p2)\n classical_rot.append(rotation_angle_error(R_c, R_true))\n classical_trans.append(translation_angle_error(t_c, t_true))\n\n results[outlier_frac] = (np.mean(learned_rot), np.mean(learned_trans), np.mean(classical_rot), np.mean(classical_trans))\n print(f'outlier_frac={outlier_frac}:')\n print(f' learned: rot err={np.mean(learned_rot):.2f} deg, trans dir err={np.mean(learned_trans):.2f} deg')\n print(f' classical: rot err={np.mean(classical_rot):.2f} deg, trans dir err={np.mean(classical_trans):.2f} deg')",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "fbd8d758",
"source": "## Where each approach wins\n\nOn clean data (no outliers), classical RANSAC wins clearly — an exact, well-conditioned geometric solve is hard to beat when the input genuinely satisfies its assumptions. As the outlier fraction increases, the gap closes and then **reverses**: the feed-forward network, trained across a whole range of outlier ratios, becomes *more* robust than RANSAC with a fixed threshold and iteration budget. This isn't a fluke of this toy setup — it's the actual, documented motivation for VGGT-style architectures: RANSAC's robustness is bounded by its hyperparameters and by the geometric model it assumes (a single rigid essential matrix relating exactly two views), while a network trained on enough varied, messy, real-world correspondence data learns something closer to \"what does a plausible pose look like given evidence like this,\" which degrades more gracefully.\n\nThis mirrors Lesson 55's stereo story almost exactly: a fixed, hand-designed algorithm (block matching, RANSAC) is precise under its ideal assumptions and brittle outside them; a network trained across a distribution of conditions trades a little of that peak precision for much better robustness across the full range of real conditions it will actually see. Neither replaces the other outright — production 3D vision pipelines increasingly use exactly this kind of feed-forward network as a fast, robust *initialization*, with a classical geometric refinement (bundle adjustment, Lesson 28) as a final, exact-precision cleanup step where compute allows.\n\n### Exercise\n\n1. Increase `max_angle_deg` in `random_rotation` from 30 to 90. Does the gap between learned and classical performance change, and does classical RANSAC's known preference for small, well-conditioned baselines explain the direction of the shift?\n2. Remove the self-attention (`self.encoder(self.embed(corr))` → just `self.embed(corr)`, then mean-pool directly) and retrain. Does removing attention between correspondences hurt rotation accuracy, translation accuracy, or both — and does that match the claim that translation direction specifically needs *relational* reasoning across points?\n3. Train two separate models, one only ever seeing `outlier_frac=0.0` during training and one only ever seeing `outlier_frac=0.5`, then evaluate both across the full `[0.0, 0.5]` range. Does either specialist beat the general-purpose model (trained across the whole range) on its own home turf, and does the generalist lose much by comparison anywhere?",
"metadata": {}
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 5
}