{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": "# Lesson 58: Neural Rendering\n\nLesson 28's structure from motion recovers a sparse 3D point cloud from multiple images. **Neural rendering** — most famously **NeRF** (Mildenhall et al., 2020) — asks for something richer: a *continuous* representation of an entire 3D scene, dense enough to render a photorealistic image from any camera viewpoint, including ones never observed during training. The representation isn't a mesh or a point cloud at all; it's the weights of a small neural network. This lesson builds one from scratch: a coordinate network trained purely by comparing rendered pixels to real ones, with no 3D supervision anywhere in the loss."
},
{
"cell_type": "code",
"id": "a70686c6",
"source": "import numpy as np\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport matplotlib.pyplot as plt",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "a57d683c",
"source": "## Ground truth: a classical ray-tracer for a synthetic scene\n\nA single shaded sphere, photographed from 8 camera positions arranged in a ring around it (Lesson 25's pinhole camera model and image formation). Each pixel's color comes from a classical, non-learned ray-sphere intersection with simple directional shading — this is *only* used to generate training data and a ground-truth held-out view to grade against; the network itself never sees the sphere's true position, radius, or color.",
"metadata": {}
},
{
"cell_type": "code",
"id": "76bd7986",
"source": "IMG = 16\nSPHERE_CENTER = np.array([0.0, 0.0, 0.0])\nSPHERE_RADIUS = 1.0\nSPHERE_COLOR = np.array([1.0, 0.3, 0.2])\n\ndef look_at(cam_pos, target=np.array([0, 0, 0]), up=np.array([0, 1, 0])):\n z = (cam_pos - target); z = z / np.linalg.norm(z)\n x = np.cross(up, z); x = x / np.linalg.norm(x)\n y = np.cross(z, x)\n return np.stack([x, y, z], axis=1)\n\ndef get_rays(cam_pos, R, img_size=IMG, fov=60.0):\n f = img_size / (2 * np.tan(np.radians(fov) / 2))\n ys, xs = np.meshgrid(np.arange(img_size), np.arange(img_size), indexing='ij')\n dirs_cam = np.stack([(xs - img_size/2 + 0.5) / f, -(ys - img_size/2 + 0.5) / f,\n -np.ones_like(xs, dtype=np.float64)], axis=-1)\n dirs_world = dirs_cam @ R.T\n dirs_world = dirs_world / np.linalg.norm(dirs_world, axis=-1, keepdims=True)\n origins = np.broadcast_to(cam_pos, dirs_world.shape)\n return origins.astype(np.float32), dirs_world.astype(np.float32)\n\ndef analytic_render(cam_pos):\n R = look_at(cam_pos)\n origins, dirs = get_rays(cam_pos, R)\n oc = origins - SPHERE_CENTER\n a = np.sum(dirs * dirs, axis=-1)\n b = 2 * np.sum(oc * dirs, axis=-1)\n c = np.sum(oc * oc, axis=-1) - SPHERE_RADIUS ** 2\n disc = b ** 2 - 4 * a * c\n hit = disc > 0\n t = np.where(hit, (-b - np.sqrt(np.clip(disc, 0, None))) / (2 * a), np.inf)\n hit = hit & (t > 0)\n img = np.zeros((IMG, IMG, 3), dtype=np.float32)\n hit_pos = origins + np.where(hit, t, 0)[..., None] * dirs\n normal = (hit_pos - SPHERE_CENTER) / SPHERE_RADIUS\n light_dir = np.array([1.0, 1.0, 1.0]); light_dir = light_dir / np.linalg.norm(light_dir)\n shade = np.clip(np.sum(normal * light_dir, axis=-1), 0.2, 1.0)\n img[hit] = (SPHERE_COLOR[None, :] * shade[..., None])[hit]\n return img, origins, dirs, hit\n\nn_train_views = 8\nangles = np.linspace(0, 2 * np.pi, n_train_views, endpoint=False)\ncam_positions = [np.array([3 * np.cos(a), 1.0, 3 * np.sin(a)]) for a in angles]\n\ntrain_images, train_origins, train_dirs = [], [], []\nfor cp in cam_positions:\n img, origins, dirs, hit = analytic_render(cp)\n train_images.append(img); train_origins.append(origins); train_dirs.append(dirs)\ntrain_images = torch.tensor(np.array(train_images))\ntrain_origins = torch.tensor(np.array(train_origins))\ntrain_dirs = torch.tensor(np.array(train_dirs))\n\n# a held-out test view, exactly BETWEEN two training camera positions\ntest_angle = angles[0] + (angles[1] - angles[0]) / 2\ntest_cam = np.array([3 * np.cos(test_angle), 1.0, 3 * np.sin(test_angle)])\ntest_img, test_origins, test_dirs, test_hit = analytic_render(test_cam)\ntest_img_t, test_origins_t, test_dirs_t = torch.tensor(test_img), torch.tensor(test_origins), torch.tensor(test_dirs)\n\nfig, axes = plt.subplots(1, 5, figsize=(11, 2.5))\nfor ax, im in zip(axes[:4], train_images[:4]):\n ax.imshow(im.numpy()); ax.axis('off')\naxes[4].imshow(test_img); axes[4].axis('off'); axes[4].set_title('held-out\\n(never trained on)', fontsize=8)\naxes[0].set_title('training views', fontsize=8, loc='left')\nplt.show()",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "2fc803b5",
"source": "## The scene as a network: coordinates in, color and density out\n\nA NeRF is a small MLP: feed it a 3D point, it returns a color and a **density** (how opaque the scene is at that point — near 0 in empty space, large inside solid material). Raw `(x, y, z)` coordinates are first expanded through a sinusoidal positional encoding at several frequencies (exactly Lesson 45's positional encoding, applied to 3D position instead of sequence position) — an MLP fed raw low-dimensional coordinates struggles to represent sharp, high-frequency detail, and this encoding fixes that.",
"metadata": {}
},
{
"cell_type": "code",
"id": "2e7d9a66",
"source": "class TinyNeRF(nn.Module):\n def __init__(self, hidden=64, n_freqs=6):\n super().__init__()\n self.n_freqs = n_freqs\n in_dim = 3 * (2 * n_freqs + 1)\n self.net = nn.Sequential(\n nn.Linear(in_dim, hidden), nn.ReLU(),\n nn.Linear(hidden, hidden), nn.ReLU(),\n nn.Linear(hidden, hidden), nn.ReLU(),\n nn.Linear(hidden, 4), # rgb (3) + density (1)\n )\n # bootstrap: bias the initial density prediction upward so early gradients\n # actually reach the color head, instead of vanishing behind near-zero density\n with torch.no_grad():\n self.net[-1].bias[3] = 1.0\n\n def encode(self, x):\n out = [x]\n for f in range(self.n_freqs):\n out.append(torch.sin(2 ** f * np.pi * x))\n out.append(torch.cos(2 ** f * np.pi * x))\n return torch.cat(out, dim=-1)\n\n def forward(self, x):\n out = self.net(self.encode(x))\n rgb = torch.sigmoid(out[..., :3])\n sigma = F.softplus(out[..., 3])\n return rgb, sigma",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "01f65aa9",
"source": "## Volumetric rendering: turning (color, density) samples into a pixel\n\nFor each pixel's ray, sample points along it, evaluate the network at each, and composite the colors weighted by how much of the ray's \"light\" survives to reach that point — the same alpha-blending idea as Lesson 2's `cv2.addWeighted`, but chained across many samples along a ray instead of two whole images: each point's density determines how much it blocks the points behind it. This entire computation is differentiable end to end, so gradients flow from a rendered pixel's error all the way back into the network's weights.",
"metadata": {}
},
{
"cell_type": "code",
"id": "3be74956",
"source": "def render_rays(model, origins, dirs, n_samples=32, near=1.0, far=5.0):\n t_vals = torch.linspace(near, far, n_samples)\n pts = origins[..., None, :] + dirs[..., None, :] * t_vals[:, None] # (..., n_samples, 3)\n rgb, sigma = model(pts)\n delta = t_vals[1:] - t_vals[:-1]\n delta = torch.cat([delta, torch.tensor([1e10])])\n alpha = 1.0 - torch.exp(-sigma * delta)\n trans = torch.cumprod(torch.cat([torch.ones_like(alpha[..., :1]), 1.0 - alpha + 1e-10], dim=-1), dim=-1)[..., :-1]\n weights = alpha * trans # how much each sample actually contributes to the final pixel\n return (weights[..., None] * rgb).sum(dim=-2)\n\ntorch.manual_seed(0)\nmodel = TinyNeRF()\nopt = torch.optim.Adam(model.parameters(), lr=0.005)\nV = train_images.shape[0]\norigins_flat = train_origins.reshape(V, -1, 3)\ndirs_flat = train_dirs.reshape(V, -1, 3)\nimages_flat = train_images.reshape(V, -1, 3)\nfor epoch in range(800):\n v = np.random.default_rng(epoch).integers(V)\n idx = np.random.default_rng(epoch + 1000).permutation(origins_flat.shape[1]) # all 256 pixels, shuffled\n o, d, target = origins_flat[v, idx], dirs_flat[v, idx], images_flat[v, idx]\n pred = render_rays(model, o, d)\n loss = F.mse_loss(pred, target)\n opt.zero_grad()\n loss.backward()\n opt.step()\n\nprint(f'final training loss: {loss.item():.4f}')",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "91cec418",
"source": "## Novel view synthesis: rendering a camera position never seen in training\n\nThe real test isn't how well the network reproduces the 8 training photos — it's whether it can render a *new* camera position exactly halfway between two training views, which it never received a single pixel of supervision for.",
"metadata": {}
},
{
"cell_type": "code",
"id": "e6dafdab",
"source": "with torch.no_grad():\n pred_test = render_rays(model, test_origins_t.reshape(-1, 3), test_dirs_t.reshape(-1, 3)).reshape(IMG, IMG, 3)\n\nmse = F.mse_loss(pred_test, test_img_t).item()\npsnr = -10 * np.log10(mse) if mse > 0 else float('inf')\nmean_color = train_images.reshape(-1, 3).mean(dim=0)\nbaseline_mse = F.mse_loss(mean_color[None, None, :].expand(IMG, IMG, 3), test_img_t).item()\n\nprint(f'held-out test view MSE: {mse:.4f} (PSNR = {psnr:.1f} dB)')\nprint(f'baseline (mean training color everywhere) MSE: {baseline_mse:.4f}')\n\nfig, axes = plt.subplots(1, 2, figsize=(6, 3))\naxes[0].imshow(test_img); axes[0].set_title('ground truth\\n(held-out view)', fontsize=9); axes[0].axis('off')\naxes[1].imshow(pred_test.clamp(0, 1).numpy()); axes[1].set_title('NeRF rendering', fontsize=9); axes[1].axis('off')\nplt.show()",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "551409f5",
"source": "The rendered novel view beats the trivial mean-color baseline by a clear margin, and visibly reconstructs the sphere's shading and silhouette from a camera angle it never trained on — the network has implicitly learned the sphere's 3D geometry (where it is, how big it is) purely from 2D photographic evidence and the geometric constraint that every ray's color must be explained consistently across all 8 views simultaneously.\n\n## 3D Gaussian Splatting: the same problem, a different representation\n\n**3D Gaussian Splatting** (Kerbl et al., 2023) solves the identical problem — dense, photorealistic novel-view synthesis from posed images — with an *explicit* scene representation instead of NeRF's implicit MLP: the scene is a large collection of 3D Gaussian \"blobs,\" each with a position, size/orientation, color, and opacity, directly optimized (not queried through a network) to reproduce the training photos when splatted (projected and alpha-blended, the same compositing math this lesson just implemented) onto each camera view. The tradeoff is speed: rendering an MLP requires a full forward pass per sample point along every ray, while rendering a fixed set of Gaussians is a much cheaper rasterization operation — closer to classical graphics — which is why Gaussian Splatting can render in real time where NeRF historically couldn't. Both approaches optimize purely against 2D photometric loss, with no 3D ground truth ever in the loop; they differ in *what* gets optimized, not in *how* the training signal reaches it.\n\n### Exercise\n\n1. Reduce `n_train_views` from 8 to 4. Does the held-out view's PSNR drop substantially — and does the rendered image show visible artifacts on the side of the sphere that's now farther from any training camera?\n2. Increase `n_samples` in `render_rays` from 32 to 8. Does rendering quality degrade, and can you see why too few samples along a ray would fail to capture where the sphere's surface actually is?\n3. This lesson's scene is a single diffuse (Lambertian) sphere, so the network only ever needs to learn `position -> color, density`, never `(position, view direction) -> color`. Real NeRF also conditions color on viewing direction, to capture view-dependent effects like specular highlights. Sketch how you'd change `TinyNeRF.forward` to take a view direction as a second input, and describe a material (e.g. a shiny sphere) where this would visibly matter and this lesson's diffuse sphere wouldn't.",
"metadata": {}
},
{
"cell_type": "markdown",
"id": "dc5b1b5c",
"source": "## Closing the course\n\nThis lesson closes Part 4, and with it the course: from raw pixels and convolution (Part 1) through classical 3D geometry (Part 2), learned convolutional features (Part 3), and attention and self-supervision at scale (Part 4), neural rendering is a fitting endpoint because it ties the whole arc together in one place — Part 2's projective geometry and image formation, Part 3's convolutional feature learning, and Part 4's habit of replacing a hand-designed algorithm with a differentiable one trained purely against pixels, all combined to reconstruct a 3D scene from nothing but 2D photographs and the constraint that they must be mutually consistent.",
"metadata": {}
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 5
}