{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Import instance segmentation dataset from multiple image masks\n", "\n", "In this tutorial, we will import the LIACi (Lifecycle Inspection, Analysis and\n", "Condition information) Segmentation Dataset for Underwater Ship Inspections,\n", "introduced in\n", "[this](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9998080) paper.\n", "\n", "The dataset contains 1893 images of underwater ship hulls, together with\n", "corresponding annotations. The dataset contains both COCO-style annotations\n", "(bounding boxes and segmentation polygons) and pixel-wise annotations stored as\n", "single-channel bitmap images with one image per class.\n", "\n", "In this notebook, we will create two different `tlc.Table`s from the dataset,\n", "in order to showcase different ways of working with annotated image data in 3LC:\n", "\n", "1. Build a 3LC Instance Segmentation Table using the individual per-class bitmaps.\n", "2. Build a 3LC Instance Segmentation Table using the COCO-style annotations.\n", "\n", "Since the downloaded data includes pre-computed embeddings, we will also add the\n", "embeddings to a Run, reduce the dimensionality of the embeddings and visualize\n", "the results." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Project setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "PROJECT_NAME = \"3LC Tutorials\"\n", "DATASET_NAME = \"LIACI\"\n", "INSTANCE_SEGMENTATION_TABLE_NAME = \"instance-segmentation\"\n", "COCO_TABLE_NAME = \"coco-segmentation\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Imports" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import json\n", "from pathlib import Path\n", "\n", "import cv2\n", "import numpy as np\n", "import tlc\n", "import tqdm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Prepare dataset\n", "\n", "The dataset is available for download from the [official website](https://data.sintef.no/product/details/dp-9e112cec-3a59-4b58-86b3-ecb1f2878c60), and must be downloaded and extracted to a local directory manually.\n", "\n", "The dataset is stored in the following layout: \n", "\n", "```\n", "LIACi_dataset_pretty\n", "│\n", "├── images\n", "│ ├── image_0001.jpg\n", "│ ├── image_0002.jpg\n", "│ ├── image_0003.jpg\n", "│ └── ...\n", "│\n", "├── masks\n", "│ ├── anode\n", "│ │ ├── image_0001.bmp\n", "│ │ ├── image_0002.bmp\n", "│ │ ├── image_0003.bmp\n", "│ │ └── ...\n", "│ ├── bilge_keel\n", "│ ├── corrosion\n", "│ ├── defect\n", "│ ├── marine_growth\n", "│ ├── over_board_valves\n", "│ ├── paint_peel\n", "│ ├── propeller\n", "│ ├── saliency\n", "│ ├── sea_chest_grating\n", "│ ├── segmentation # merged masks\n", "│ └── ship_hull\n", "│\n", "├── coco-annotations.json\n", "├── train_test_split.csv\n", "├── embeddings_resnet101.json\n", "...\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Replace with your own path, after downloading and extracting the dataset\n", "DATASET_ROOT = Path(\"C:/Data/LIACi_dataset_pretty\")\n", "\n", "# Register the dataset root as an alias, enabling easy sharing/moving of the table\n", "tlc.register_url_alias(\"LIACI_DATASET_ROOT\", DATASET_ROOT.as_posix())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Approach 1: instance segmentations from per-class masks" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "image_dir = DATASET_ROOT / \"images\"\n", "masks_dir = DATASET_ROOT / \"masks\"\n", "\n", "# Exclude merged masks, we are only interested in per-class masks\n", "mask_dirs = [d.name for d in masks_dir.iterdir() if d.is_dir() and d.name != \"segmentation\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will now collect the data which will go into our Table.\n", "For the segmentation column, we will stack all the per-class masks into a single array." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "image_urls = []\n", "instance_segmentations = []\n", "\n", "images = list(image_dir.iterdir())\n", "\n", "for image in tqdm.tqdm(images, total=len(images), desc=\"Processing images\"):\n", " image_url = tlc.Url(image).to_relative() # .to_relative() applies the alias to the path\n", " mask_filename = image_url.name.replace(\"jpg\", \"bmp\")\n", "\n", " image_urls.append(image_url.to_str())\n", "\n", " masks = []\n", " cat_ids = []\n", "\n", " for cat_id, category in enumerate(mask_dirs):\n", " mask_path = masks_dir / category / mask_filename\n", " if not mask_path.exists():\n", " print(f\"Skipping category {category} for image {image_url}\")\n", " continue\n", "\n", " mask = cv2.imread(mask_path, cv2.IMREAD_GRAYSCALE)\n", "\n", " if not np.any(mask):\n", " # If the mask is empty, we skip the category\n", " continue\n", "\n", " mask[mask == 255] = 1\n", " masks.append(mask)\n", " cat_ids.append(cat_id)\n", "\n", " # Masks in 3LC instance segmentation masks format\n", " instance_segmentation = {\n", " \"image_height\": masks[0].shape[0],\n", " \"image_width\": masks[0].shape[1],\n", " \"instance_properties\": {\n", " \"label\": cat_ids,\n", " },\n", " \"masks\": np.stack(masks, axis=-1),\n", " }\n", " instance_segmentations.append(instance_segmentation)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create a `TableWriter` to write the data to the Table with the correct column schemas." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "table_writer = tlc.TableWriter(\n", " table_name=INSTANCE_SEGMENTATION_TABLE_NAME,\n", " project_name=PROJECT_NAME,\n", " dataset_name=DATASET_NAME,\n", " column_schemas={\n", " \"image\": tlc.ImagePath(\"image\"),\n", " \"segmentations\": tlc.InstanceSegmentationMasks(\n", " \"segmentations\",\n", " instance_properties_structure={\n", " \"label\": tlc.CategoricalLabel(\"label\", mask_dirs),\n", " },\n", " ),\n", " },\n", ")\n", "\n", "table_writer.add_batch(\n", " {\n", " \"image\": image_urls,\n", " \"segmentations\": instance_segmentations,\n", " }\n", ")\n", "\n", "instance_segmentation_table = table_writer.finalize()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sample_masks = instance_segmentation_table[0][\"segmentations\"][\"masks\"]\n", "mask_labels = instance_segmentation_table[0][\"segmentations\"][\"instance_properties\"][\"label\"]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "\n", "fig, axes = plt.subplots(2, 3, figsize=(12, 8))\n", "for i in range(2):\n", " for j in range(3):\n", " idx = i * 3 + j\n", " axes[i, j].imshow(sample_masks[:, :, idx], cmap=\"gray\")\n", " axes[i, j].set_title(f\"{mask_dirs[mask_labels[idx]]}\")\n", " axes[i, j].axis(\"off\")\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Approach 2: COCO-style annotations\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "annotations_path = DATASET_ROOT / \"coco-labels.json\"\n", "image_folder = DATASET_ROOT / \"images\"\n", "\n", "coco_table = tlc.Table.from_coco(\n", " annotations_path,\n", " image_folder,\n", " project_name=PROJECT_NAME,\n", " dataset_name=DATASET_NAME,\n", " table_name=COCO_TABLE_NAME,\n", " task=\"segment\",\n", " segmentation_format=\"masks\",\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "coco_table.rows_schema[\"segmentations\"].sample_type" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Prepare data for YOLO (return polygons and split into train/test)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from tlc_tools.derived_tables import masks_to_polygons\n", "from tlc_tools.split import split_table\n", "\n", "polygons_table = masks_to_polygons(coco_table)\n", "splits = split_table(polygons_table)\n", "\n", "print(f\"Train split: {splits['train']}\")\n", "print(f\"Validation split: {splits['val']}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Extra: Reduce and visualize embeddings" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Load the pre-computed embeddings (2048-dimensional)\n", "embeddings_json_path = Path(DATASET_ROOT) / \"embeddings_resnet101.json\"\n", "\n", "with open(embeddings_json_path) as f:\n", " embeddings_json = json.load(f)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create a run to store the embeddings." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "run = tlc.init(\n", " PROJECT_NAME,\n", " \"LIACi-Embeddings\",\n", " description=\"Inspect 2D and 3D embeddings of the LIACi dataset\",\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Link the rows of the embeddings metrics table to the instance segmentation table, using the `foreign_table_url` parameter." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "foreign_table_url = tlc.Url.create_table_url(\n", " INSTANCE_SEGMENTATION_TABLE_NAME,\n", " DATASET_NAME,\n", " PROJECT_NAME,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "embedding_table_writer = tlc.MetricsTableWriter(\n", " run_url=run.url,\n", " foreign_table_url=foreign_table_url,\n", " column_schemas={\n", " \"embedding\": tlc.Schema(\n", " value=tlc.Float32Value(number_role=tlc.NUMBER_ROLE_NN_EMBEDDING),\n", " size0=tlc.DimensionNumericValue(value_min=2048, value_max=2048),\n", " writable=False,\n", " )\n", " },\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Add the original embeddings to the Run." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "embeddings = embeddings_json[\"embeddings\"]\n", "for i, embedding in tqdm.tqdm(enumerate(embeddings.values()), total=len(embeddings), delay=1.0):\n", " embedding_table_writer.add_row({tlc.EXAMPLE_ID: i, \"embedding\": embedding})" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "embedding_table = embedding_table_writer.finalize()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Reduce the dimensionality of the embeddings to 3D. Keep the original embeddings in the table." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "reduced_3d = tlc.reduce_embeddings(\n", " embedding_table,\n", " method=\"pacmap\",\n", " n_components=3,\n", " n_neighbors=20,\n", " MN_ratio=0.7,\n", " FP_ratio=3.0,\n", " target_embedding_column=\"embedding_3d\",\n", " retain_source_embedding_column=True, # We need the source embedding column for the 2D reduction\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "reduced_3d.columns" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Reduce the dimensionality of the embeddings to 2D. Delete the original embeddings from the table." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "reduced_2d = tlc.reduce_embeddings(\n", " reduced_3d,\n", " method=\"pacmap\",\n", " n_components=2,\n", " n_neighbors=20,\n", " MN_ratio=0.7,\n", " FP_ratio=3.0,\n", " target_embedding_column=\"embedding_2d\",\n", " retain_source_embedding_column=False, # We don't need the source embedding column anymore\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Add the metrics Table to the Run." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "run.add_metrics_table(reduced_2d)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "run.set_status_completed()" ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.9" }, "test_marks": [ "dependent" ] }, "nbformat": 4, "nbformat_minor": 2 }