{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Convert Semantic to Instance Segmentation\n", "\n", "Convert a semantic segmentation dataset into the instance segmentation format used by 3LC, using the simplest possible mapping: every semantic class becomes a single \"instance\".\n", "\n", "![img](../../images/ade-20-semseg.jpg)\n", "\n", "\n", "\n", "We load ADE20K's semantic masks and store them as `SegmentationMasks` rows by collapsing all pixels of a given class into one mask per class. This fits a semantic-segmentation dataset into 3LC's instance-segmentation column type for visualization and analysis; the trade-off is that individual objects of the same class are merged together, so instance identity is lost. Recovering it would require an extra step such as connected-components labeling or a model that predicts instance ids." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Project Setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "parameters" ] }, "outputs": [], "source": [ "PROJECT_NAME = \"3LC Tutorials - Create Tables\"\n", "DATASET_NAME = \"ADE20k_toy_dataset\"\n", "DOWNLOAD_PATH = \"../../transient_data\"\n", "INSTALL_DEPENDENCIES = True" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Install dependencies" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "if INSTALL_DEPENDENCIES:\n", " %pip install -q 3lc\n", " %pip install -q huggingface-hub\n", " %pip install -q git+https://github.com/3lc-ai/3lc-examples.git\n", " %pip install -q matplotlib" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Imports" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import json\n", "from pathlib import Path\n", "\n", "import cv2\n", "import numpy as np\n", "import tlc\n", "from huggingface_hub import hf_hub_download\n", "\n", "from tlc_tools.common import download_and_extract_zipfile" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Download the dataset" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "DATASET_ROOT = (Path(DOWNLOAD_PATH) / \"ADE20k_toy_dataset\").resolve()\n", "\n", "if not DATASET_ROOT.exists():\n", " print(\"Downloading data...\")\n", " download_and_extract_zipfile(\n", " url=\"https://www.dropbox.com/s/l1e45oht447053f/ADE20k_toy_dataset.zip?dl=1\",\n", " location=DOWNLOAD_PATH,\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fetch the label map from the Hugging Face Hub" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Load id2label mapping from a JSON on the hub. The JSON is 0-indexed\n", "# ({\"0\": \"wall\", \"1\": \"building\", ...}), but the ADE20K mask PNGs are 1-indexed\n", "# with 0 reserved for \"unknown\" — so a mask pixel value of N maps to id2label[N-1].\n", "with open(\n", " hf_hub_download(\n", " repo_id=\"huggingface/label-files\",\n", " filename=\"ade20k-id2label.json\",\n", " repo_type=\"dataset\",\n", " )\n", ") as f:\n", " id2label = {int(k): v for k, v in json.load(f).items()}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load the images and segmentation maps\n", "\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "image_paths = list(sorted(DATASET_ROOT.glob(\"**/images/training/*.jpg\")))\n", "segmentation_map_paths = list(sorted(DATASET_ROOT.glob(\"**/annotations/training/*.png\")))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Call .to_relative() to ensure aliases are applied\n", "image_paths = [tlc.Url(p).to_relative().to_str() for p in image_paths]\n", "print(image_paths[0])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Transform the segmentation maps to instance segmentation masks" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def single_channel_map_to_per_class_masks(map: np.ndarray) -> tuple[np.ndarray, list[int]]:\n", " \"\"\"Convert a single channel segmentation map to a stack of per-class masks.\n", "\n", " The input map uses ADE20K's 1-indexed convention (0 = unknown, 1..150 = real\n", " classes). Returned ``labels`` are 0-indexed to match the HuggingFace id2label\n", " JSON, so we subtract 1.\n", "\n", " Args:\n", " map: A numpy array of shape (H, W) representing a single channel segmentation map.\n", "\n", " Returns:\n", " - A stack of per-class masks of shape (H, W, N), where N is the number of classes present.\n", " - A list of length N with 0-indexed class IDs.\n", " \"\"\"\n", " masks = []\n", " labels = []\n", " for class_id in np.unique(map):\n", " if class_id == 0:\n", " continue\n", " mask = (map == class_id).astype(np.uint8)\n", " masks.append(mask)\n", " labels.append(int(class_id) - 1)\n", " if not masks:\n", " h, w = map.shape\n", " return np.zeros((h, w, 0), dtype=np.uint8), []\n", " return np.stack(masks, axis=-1), labels" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Build the column of instance segmentations in the format required by 3LC\n", "instances = []\n", "\n", "for mask_path in segmentation_map_paths:\n", " map_np = cv2.imread(mask_path, cv2.IMREAD_GRAYSCALE)\n", " h, w = map_np.shape\n", " masks, labels = single_channel_map_to_per_class_masks(map_np)\n", "\n", " instances.append(\n", " tlc.data_types.SegmentationMasks(\n", " image_height=h,\n", " image_width=w,\n", " masks=masks,\n", " labels=labels,\n", " )\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Write the instance segmentation masks to a table" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "table_writer = tlc.TableWriter(\n", " table_name=\"ADE20K-instance-segmentation\",\n", " dataset_name=DATASET_NAME,\n", " project_name=PROJECT_NAME,\n", " schema={\n", " \"image\": tlc.schemas.ImageSchema(),\n", " \"instances\": tlc.data_types.SegmentationMasks.schema(classes=id2label),\n", " },\n", " if_exists=\"rename\",\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Add all rows (images and instance segmentations) to the table in one go\n", "table_writer.add_batch(\n", " {\n", " \"image\": image_paths,\n", " \"instances\": instances,\n", " }\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "table = table_writer.finalize()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Visualize a sample instance segmentation mask" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "\n", "example_mask = table[0][\"instances\"].masks[:, :, 0]\n", "\n", "plt.imshow(example_mask, cmap=\"gray\")\n", "plt.axis(\"off\")\n", "plt.show()" ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.10" } }, "nbformat": 4, "nbformat_minor": 2 }