{ "cells": [ { "cell_type": "markdown", "id": "e83ca348", "metadata": {}, "source": "# Lesson 27: Camera Calibration\n\nLesson 26 assumed the camera intrinsics $K$ were already known. This lesson shows how to actually get them: **camera calibration**, typically done by showing the camera a known flat pattern (a checkerboard) from several different angles. The classic approach (Zhang, 2000) works by extracting a **homography** between the checkerboard plane and each image — a direct callback to Lesson 23 — and combining constraints from several such homographies to pin down $K$." }, { "cell_type": "code", "execution_count": null, "id": "e73d1adb", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import cv2\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "id": "22d7aa2f", "metadata": {}, "source": "## What calibration recovers\n\nTwo separate things:\n\n1. **Intrinsics** $K = \\begin{bmatrix}f_x & s & c_x\\\\0 & f_y & c_y\\\\0&0&1\\end{bmatrix}$: focal lengths, principal point, and skew — the ideal pinhole part of the camera model used throughout Lessons 26 and 28.\n2. **Distortion coefficients** $(k_1, k_2, p_1, p_2, k_3, \\dots)$: real lenses aren't perfect pinholes. Radial distortion ($k_1, k_2, k_3$) bows straight lines into curves (barrel or pincushion, depending on sign); tangential distortion ($p_1, p_2$) accounts for the lens not being perfectly parallel to the sensor.\n\nEvery equation in Lessons 26 and 28 secretly assumed distortion was already removed — calibration is the step that makes that assumption true." }, { "cell_type": "markdown", "id": "4c04090a", "metadata": {}, "source": [ "## A synthetic calibration rig\n", "\n", "Since we don't have a physical camera and checkerboard handy, we build one entirely synthetically: define a ground-truth $K$ and distortion, define a flat checkerboard's 3D corner positions, and generate several \"photos\" of it from different poses by projecting the corners with `cv2.projectPoints` (which applies distortion exactly as a real lens would). This gives us the same kind of input `cv2.calibrateCamera` expects from real detected checkerboard corners — with the enormous benefit of also knowing the ground truth to check against." ] }, { "cell_type": "code", "execution_count": null, "id": "94a8b89d", "metadata": {}, "outputs": [], "source": [ "rng = np.random.default_rng(0)\n", "\n", "cols, rows = 9, 6 # internal corners of a 10x7-square checkerboard\n", "corners_3d = np.zeros((rows * cols, 3), dtype=np.float64)\n", "corners_3d[:, :2] = np.mgrid[0:cols, 0:rows].T.reshape(-1, 2)\n", "corners_3d[:, :2] -= corners_3d[:, :2].mean(axis=0) # center the board on its own origin\n", "\n", "K_true = np.array([[800, 0, 320], [0, 800, 240], [0, 0, 1]], dtype=np.float64)\n", "dist_true = np.array([-0.3, 0.1, 0.001, -0.0005, 0.02]) # k1, k2, p1, p2, k3\n", "image_size = (640, 480)\n", "\n", "object_points, image_points = [], []\n", "for _ in range(40):\n", " rvec = rng.uniform(-0.4, 0.4, 3) # a somewhat random tilt\n", " tvec = np.array([rng.uniform(-0.5, 0.5), rng.uniform(-0.4, 0.4), rng.uniform(9, 13)])\n", " projected, _ = cv2.projectPoints(corners_3d, rvec, tvec, K_true, dist_true)\n", " projected = projected.reshape(-1, 2)\n", " if np.all(projected >= 5) and np.all(projected[:, 0] < image_size[0] - 5) and np.all(projected[:, 1] < image_size[1] - 5):\n", " object_points.append(corners_3d.astype(np.float32))\n", " image_points.append(projected.astype(np.float32))\n", "\n", "print(f'{len(object_points)} usable synthetic views (out of 40 attempted; some fell outside the frame)')" ] }, { "cell_type": "markdown", "id": "7e6d9f39", "source": "### What the calibration target actually looks like\n\n`corners_3d` and `image_points` above are just numbers so far. To make the target concrete: a real checkerboard is a flat pattern of alternating squares, and since it's flat, the mapping from the board's own 2D surface coordinates to a photo of it is exactly a **homography** — the same one Zhang's method extracts. We can use that fact in reverse: `cv2.findHomography` between the board's flat-pattern pixel coordinates and two of the synthetic `image_points` views, then warp a picture of the actual checkerboard through it, to see what those two synthetic \"photos\" would really look like.", "metadata": {} }, { "cell_type": "code", "id": "e1c6af5a", "source": "sq = 60 # pixels per square in the flat target image\nboard = np.zeros(((rows + 1) * sq, (cols + 1) * sq), dtype=np.uint8)\nfor r in range(rows + 1):\n for c in range(cols + 1):\n if (r + c) % 2 == 0:\n board[r * sq:(r + 1) * sq, c * sq:(c + 1) * sq] = 255\nboard_bgr = cv2.cvtColor(board, cv2.COLOR_GRAY2BGR)\n\n# the internal corners' locations in the flat target's own pixel coordinates,\n# in the same order as corners_3d\nboard_pts = np.array([[(c + 1) * sq, (r + 1) * sq] for r in range(rows) for c in range(cols)],\n dtype=np.float32)\n\nfig, axes = plt.subplots(1, 3, figsize=(12, 4))\naxes[0].imshow(board, cmap='gray')\naxes[0].set_title('Flat checkerboard target')\naxes[0].axis('off')\n\nfor i, ax in zip([0, 1], axes[1:]):\n H, _ = cv2.findHomography(board_pts, image_points[i])\n photo = cv2.warpPerspective(board_bgr, H, image_size, borderValue=(128, 128, 128))\n cv2.drawChessboardCorners(photo, (cols, rows), image_points[i], True)\n ax.imshow(cv2.cvtColor(photo, cv2.COLOR_BGR2RGB))\n ax.set_title(f'Synthetic photo {i}')\n ax.axis('off')\nplt.tight_layout()\nplt.show()", "metadata": {}, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "f2d965d2", "metadata": {}, "source": [ "## Running calibration\n", "\n", "`cv2.calibrateCamera` takes the (known) 3D corner positions and their (detected/here, projected) 2D image positions across all views, and jointly solves for $K$, distortion, and each view's own pose." ] }, { "cell_type": "code", "execution_count": null, "id": "d7825897", "metadata": {}, "outputs": [], "source": [ "rms_error, K_estimated, dist_estimated, rvecs, tvecs = cv2.calibrateCamera(\n", " object_points, image_points, image_size, None, None)\n", "\n", "print(f'RMS reprojection error: {rms_error:.2e} pixels\\n')\n", "print('K (true):\\n', K_true)\n", "print('K (estimated):\\n', np.round(K_estimated, 3))\n", "print()\n", "print('distortion (true): ', dist_true)\n", "print('distortion (estimated):', np.round(dist_estimated.ravel(), 5))" ] }, { "cell_type": "markdown", "id": "0eb64125", "metadata": {}, "source": [ "With clean (noiseless) synthetic correspondences, calibration recovers the ground truth essentially exactly. Real calibration never sees noise-free data — corner detection on a real photo introduces sub-pixel jitter — which is exactly why in practice you'd use as many views, from as wide a variety of angles, as practical: more (and more diverse) constraints average out that noise." ] }, { "cell_type": "markdown", "id": "df786998", "metadata": {}, "source": [ "## Straight lines, bent by a lens\n", "\n", "To see distortion directly (rather than just as numbers), apply the same distortion model used above to points sampled along perfectly straight lines, in normalized (pre-$K$) camera coordinates:\n", "\n", "$$x_d = x(1 + k_1r^2 + k_2r^4 + k_3r^6) + 2p_1xy + p_2(r^2+2x^2)$$\n", "$$y_d = y(1 + k_1r^2 + k_2r^4 + k_3r^6) + p_1(r^2+2y^2) + 2p_2xy$$\n", "\n", "where $r^2 = x^2+y^2$. This is exactly the model `cv2.projectPoints` applied internally above." ] }, { "cell_type": "code", "execution_count": null, "id": "e62cbee8", "metadata": {}, "outputs": [], "source": "def distort_normalized(xy, dist):\n k1, k2, p1, p2, k3 = dist\n x, y = xy[:, 0], xy[:, 1]\n r2 = x**2 + y**2\n radial = 1 + k1 * r2 + k2 * r2**2 + k3 * r2**3\n xd = x * radial + 2 * p1 * x * y + p2 * (r2 + 2 * x**2)\n yd = y * radial + p1 * (r2 + 2 * y**2) + 2 * p2 * x * y\n return np.stack([xd, yd], axis=1)\n\ndef make_grid_lines():\n lines = []\n for xv in np.linspace(-0.38, 0.38, 9):\n ys = np.linspace(-0.28, 0.28, 50)\n lines.append(np.stack([np.full_like(ys, xv), ys], axis=1))\n for yv in np.linspace(-0.28, 0.28, 7):\n xs = np.linspace(-0.38, 0.38, 50)\n lines.append(np.stack([xs, np.full_like(xs, yv)], axis=1))\n return lines\n\ndef draw_lines(lines_px, height=480, width=640):\n img = np.full((height, width, 3), 255, dtype=np.uint8)\n for line in lines_px:\n for i in range(len(line) - 1):\n cv2.line(img, tuple(line[i].astype(int)), tuple(line[i + 1].astype(int)), (0, 0, 0), 1)\n return img\n\ngrid_lines_normalized = make_grid_lines()\n\nstraight_px = [line @ K_true[:2, :2].T + K_true[:2, 2] for line in grid_lines_normalized]\ndistorted_px = [distort_normalized(line, dist_true) @ K_true[:2, :2].T + K_true[:2, 2]\n for line in grid_lines_normalized]\n\nfig, axes = plt.subplots(1, 2, figsize=(9, 4))\naxes[0].imshow(draw_lines(straight_px))\naxes[0].set_title('Ideal pinhole (no distortion)')\naxes[1].imshow(draw_lines(distorted_px))\naxes[1].set_title(f'Through the lens\\n(k1={dist_true[0]}, k2={dist_true[1]})')\nfor ax in axes:\n ax.axis('off')\nplt.tight_layout()\nplt.show()" }, { "cell_type": "markdown", "id": "eea0726b", "metadata": {}, "source": [ "## Undoing distortion with the estimated parameters\n", "\n", "`cv2.undistortPoints` inverts the distortion model, converting distorted pixel coordinates back to normalized undistorted coordinates. Applying it to the bent grid lines above, using our *estimated* (not the true) calibration, should straighten them back out." ] }, { "cell_type": "code", "execution_count": null, "id": "25d974f1", "metadata": {}, "outputs": [], "source": [ "undistorted_normalized = [\n", " cv2.undistortPoints(px.reshape(-1, 1, 2).astype(np.float64), K_estimated, dist_estimated).reshape(-1, 2)\n", " for px in distorted_px\n", "]\n", "undistorted_px = [line @ K_estimated[:2, :2].T + K_estimated[:2, 2] for line in undistorted_normalized]\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(9, 4))\n", "axes[0].imshow(draw_lines(distorted_px))\n", "axes[0].set_title('Distorted (as captured)')\n", "axes[1].imshow(draw_lines(undistorted_px))\n", "axes[1].set_title('Undistorted using estimated K, dist')\n", "for ax in axes:\n", " ax.axis('off')\n", "plt.tight_layout()\n", "plt.show()\n", "\n", "original_normalized = np.concatenate(grid_lines_normalized)\n", "recovered_normalized = np.concatenate(undistorted_normalized)\n", "print(f'max recovery error (normalized coords): {np.abs(original_normalized - recovered_normalized).max():.2e}')" ] }, { "cell_type": "markdown", "id": "ccdd40ee", "metadata": {}, "source": "The lines are straight again, and the recovered points match the original ideal grid to within floating-point precision — the whole point of calibrating a real camera is to be able to do exactly this correction on real photos before handing them to any of the geometric machinery from Lessons 26 and 28, all of which implicitly assumes an ideal, distortion-free pinhole." }, { "cell_type": "markdown", "id": "5546eaca", "metadata": {}, "source": [ "### Exercise\n", "\n", "1. Add pixel noise (e.g. `rng.normal(0, 0.5, projected.shape)`) to the synthetic corner detections before calibrating. How much does the RMS reprojection error grow, and how does `K_estimated` drift from `K_true`?\n", "2. Reduce the number of usable calibration views from ~10-15 down to just 3. Does the estimate noticeably degrade, especially for the distortion coefficients (which need corners spread widely across the frame, including near the edges, to be well constrained)?\n", "3. Set `dist_true = np.zeros(5)` (a perfect pinhole, no distortion) and rerun the whole notebook. Confirm the \"distorted\" and \"ideal\" grids become identical, and that calibration still recovers `K_true` correctly even with zero distortion to estimate." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.x" } }, "nbformat": 4, "nbformat_minor": 5 }