{ "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "# Create a Semantic Segmentation Table from Pascal VOC 2012\n", "\n", "Ingest the Pascal VOC 2012 segmentation set into 3LC tables — a 21-class semantic segmentation problem (background plus 20 objects) that steps up from the 3-class Oxford Pets example.\n", "\n", "![](../../images/pascal-voc-semseg.png)\n", "\n", "\n", "\n", "[Pascal VOC 2012](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/) stores each segmentation as a paletted\n", "PNG whose pixel indices *are* the class ids (0 = background, 1..20 = objects, 255 = the void boundary).\n", "That means no label remapping is needed — `np.asarray` of the PNG is the dense label map. We author the\n", "tables with the documented front door `Table.from_semantic_segmentation(...)`, which takes the images and\n", "masks directly. `background` is recorded in the column schema's metadata (not the value map) and omitted\n", "from per-row storage as the implicit fill; `void` is tagged and excluded from metrics.\n", "\n", "This is the *ingest* half of a pair: `huggingface-pascal-voc-mask2former-finetuning` fine-tunes a\n", "Mask2Former model on the tables created here." ] }, { "cell_type": "markdown", "id": "1", "metadata": {}, "source": [ "## Project setup" ] }, { "cell_type": "code", "execution_count": null, "id": "2", "metadata": { "tags": [ "parameters" ] }, "outputs": [], "source": [ "# Parameters\n", "PROJECT_NAME = \"3LC Tutorials - Pascal VOC 2012\"\n", "DATASET_NAME = \"pascal-voc-2012\"\n", "TABLE_NAME = \"train\" # the train table; the val table is written as \"val\"\n", "VAL_TABLE_NAME = \"val\"\n", "DOWNLOAD_PATH = \"../../transient_data\" # dataset root; point at an existing copy to skip the download\n", "N_TRAIN = None # cap the train rows for a quick run; None keeps the whole train split\n", "N_VAL = None # cap the val rows for a quick run; None keeps the whole val split\n", "SEED = 42\n", "INSTALL_DEPENDENCIES = True" ] }, { "cell_type": "markdown", "id": "3", "metadata": {}, "source": [ "## Install dependencies" ] }, { "cell_type": "code", "execution_count": null, "id": "4", "metadata": {}, "outputs": [], "source": [ "if INSTALL_DEPENDENCIES:\n", " %pip install -q 3lc torchvision matplotlib" ] }, { "cell_type": "markdown", "id": "5", "metadata": {}, "source": [ "## Imports" ] }, { "cell_type": "code", "execution_count": null, "id": "6", "metadata": {}, "outputs": [], "source": [ "import random\n", "from collections.abc import Sequence\n", "from pathlib import Path\n", "\n", "import numpy as np\n", "import tlc\n", "from PIL import Image\n", "from torchvision.datasets import VOCSegmentation" ] }, { "cell_type": "markdown", "id": "7", "metadata": {}, "source": [ "## Download the dataset\n", "\n", "`torchvision` fetches the VOC2012 devkit. One download provides both the `train` and `val` segmentation\n", "splits. The download only runs when the data is missing — if `DOWNLOAD_PATH/VOCdevkit/VOC2012/` already\n", "exists, torchvision uses it as-is and never writes (so a pre-seeded read-only cache is safe). To reuse an\n", "existing copy, point `DOWNLOAD_PATH` at its parent — e.g. `DOWNLOAD_PATH = \"~/data\"` when the devkit lives\n", "at `~/data/VOCdevkit/VOC2012`.\n", "\n", "> **Note:** the official VOC host is occasionally slow or unavailable. If the download fails, fetch\n", "> `VOCtrainval_11-May-2012.tar` manually and extract it under `DOWNLOAD_PATH` so that\n", "> `DOWNLOAD_PATH/VOCdevkit/VOC2012/` exists." ] }, { "cell_type": "code", "execution_count": null, "id": "8", "metadata": {}, "outputs": [], "source": [ "download_root = Path(DOWNLOAD_PATH).expanduser()\n", "data_root = download_root / \"VOCdevkit\" / \"VOC2012\"\n", "# Gate on the extracted dirs ourselves rather than passing download=True: torchvision's VOCSegmentation\n", "# re-runs download+extract whenever download=True (it keys the skip on the .tar's md5, not the extracted\n", "# tree), so it would re-extract — or fail on a read-only cache — even when the devkit is already here.\n", "have_data = (data_root / \"JPEGImages\").is_dir() and (data_root / \"SegmentationClass\").is_dir()\n", "print(\n", " f\"Dataset cache {'HIT' if have_data else 'MISS'} at {data_root.resolve()} \"\n", " f\"({'using existing data' if have_data else 'downloading'})\"\n", ")\n", "VOCSegmentation(root=download_root, year=\"2012\", image_set=\"train\", download=not have_data)\n", "print(\"Dataset at\", data_root.resolve())" ] }, { "cell_type": "markdown", "id": "9", "metadata": {}, "source": [ "## Define the class universe\n", "\n", "The 21 VOC classes in canonical order, plus `void`. Passing `background` and `void` to the front door\n", "gives them their roles: `background` (id 0) is recorded in the schema metadata and omitted from the value\n", "map and per-row storage (the implicit fill, rendered transparent), while `void` (id 255) is tagged and\n", "excluded from metrics. Downstream code reads these back rather than hard-coding ids." ] }, { "cell_type": "code", "execution_count": null, "id": "10", "metadata": {}, "outputs": [], "source": [ "VOC_CLASS_NAMES = [\n", " \"background\",\n", " \"aeroplane\",\n", " \"bicycle\",\n", " \"bird\",\n", " \"boat\",\n", " \"bottle\",\n", " \"bus\",\n", " \"car\",\n", " \"cat\",\n", " \"chair\",\n", " \"cow\",\n", " \"diningtable\",\n", " \"dog\",\n", " \"horse\",\n", " \"motorbike\",\n", " \"person\",\n", " \"pottedplant\",\n", " \"sheep\",\n", " \"sofa\",\n", " \"train\",\n", " \"tvmonitor\",\n", "]\n", "BACKGROUND_ID = 0\n", "VOID_ID = 255\n", "\n", "# Class universe handed to the front door: background + 20 objects + void.\n", "SEGMENTATION_CLASSES = {**{i: name for i, name in enumerate(VOC_CLASS_NAMES)}, VOID_ID: \"void\"}" ] }, { "cell_type": "markdown", "id": "11", "metadata": {}, "source": [ "## Locate the image/mask pairs\n", "\n", "Only images with a segmentation annotation are ingested — the ids listed in\n", "`ImageSets/Segmentation/{train,val}.txt` (the official VOC segmentation split, not the larger\n", "SBD/SegmentationClassAug set). `LazyMasks` decodes each PNG on indexed access so we never hold all masks\n", "in memory at once." ] }, { "cell_type": "code", "execution_count": null, "id": "12", "metadata": {}, "outputs": [], "source": [ "def collect_samples(root: Path, split: str) -> list[tuple[Path, Path]]:\n", " # Pair every id in ImageSets/Segmentation/.txt with its jpg + mask png.\n", " split_file = root / \"ImageSets\" / \"Segmentation\" / f\"{split}.txt\"\n", " if not split_file.is_file():\n", " raise FileNotFoundError(f\"Missing segmentation split file: {split_file}\")\n", " pairs = []\n", " for name in split_file.read_text().split():\n", " image_path = root / \"JPEGImages\" / f\"{name}.jpg\"\n", " label_path = root / \"SegmentationClass\" / f\"{name}.png\"\n", " if image_path.exists() and label_path.exists():\n", " pairs.append((image_path, label_path))\n", " return pairs\n", "\n", "\n", "def load_mask(label_path: Path) -> np.ndarray:\n", " # Decode a paletted SegmentationClass PNG; its indices are the class ids directly.\n", " return np.asarray(Image.open(label_path), dtype=np.int32)\n", "\n", "\n", "class LazyMasks(Sequence):\n", " def __init__(self, label_paths: list[Path]) -> None:\n", " self._label_paths = label_paths\n", "\n", " def __len__(self) -> int:\n", " return len(self._label_paths)\n", "\n", " def __getitem__(self, index: int) -> np.ndarray:\n", " return load_mask(self._label_paths[index])" ] }, { "cell_type": "markdown", "id": "13", "metadata": {}, "source": [ "## Write the tables\n", "\n", "`Table.from_semantic_segmentation` takes the image paths and the (lazy) masks plus the class universe and\n", "the `background` / `void` roles, and writes the table in one call." ] }, { "cell_type": "code", "execution_count": null, "id": "14", "metadata": {}, "outputs": [], "source": [ "def build_table(split: str, table_name: str, n_rows: int | None) -> tlc.Table:\n", " pairs = collect_samples(data_root, split)\n", " if n_rows is not None:\n", " random.Random(SEED).shuffle(pairs)\n", " pairs = pairs[:n_rows]\n", "\n", " images = [image_path for image_path, _ in pairs]\n", " masks = LazyMasks([label_path for _, label_path in pairs])\n", "\n", " table = tlc.Table.from_semantic_segmentation(\n", " images=images,\n", " masks=masks,\n", " classes=SEGMENTATION_CLASSES,\n", " background=BACKGROUND_ID,\n", " void=VOID_ID,\n", " project_name=PROJECT_NAME,\n", " dataset_name=DATASET_NAME,\n", " table_name=table_name,\n", " if_exists=\"overwrite\",\n", " )\n", " print(f\"Wrote {table_name}: {len(table)} rows -> {table.url}\")\n", " return table\n", "\n", "\n", "train_table = build_table(\"train\", TABLE_NAME, N_TRAIN)\n", "val_table = build_table(\"val\", VAL_TABLE_NAME, N_VAL)" ] }, { "cell_type": "markdown", "id": "15", "metadata": {}, "source": [ "## Inspect a sample\n", "\n", "Reading a row back hands you a `SemanticSegmentation` dataclass. We overlay the mask on the image to\n", "sanity-check the alignment and the class palette." ] }, { "cell_type": "code", "execution_count": null, "id": "16", "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "\n", "sample = train_table[0]\n", "seg = sample[\"mask\"]\n", "present = seg.present_class_ids.tolist()\n", "print(f\"Round-trip type: {type(seg).__name__}\")\n", "print(\"Classes present:\", [SEGMENTATION_CLASSES[c] for c in present])\n", "\n", "image = sample[\"image\"] # the image column reads back as a PIL image\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n", "axes[0].imshow(image)\n", "axes[0].set_title(\"image\")\n", "axes[1].imshow(image)\n", "axes[1].imshow(seg.mask, alpha=0.5, cmap=\"tab20\", interpolation=\"nearest\")\n", "axes[1].set_title(\"segmentation\")\n", "for ax in axes:\n", " ax.axis(\"off\")\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "17", "metadata": {}, "source": [ "## Next steps\n", "\n", "The `train` and `val` tables are now in your 3LC project — open the project in the\n", "[3LC Dashboard](https://dashboard.3lc.ai) to browse the images and their 21-class segmentations.\n", "\n", "Continue with `huggingface-pascal-voc-mask2former-finetuning` to fine-tune a Mask2Former model on\n", "these tables and collect per-sample predictions and IoU back into a 3LC `Run`." ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 5 }