{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Promptable segmentation with SAM 3\n", "\n", "Segment every object matching a concept (a short text phrase or a bounding box) using Meta's [Segment Anything Model 3](https://huggingface.co/facebook/sam3).\n", "\n", "**What's in this recipe:**\n", "\n", "- Segment a single object from a text prompt\n", "- Segment many instances of a concept at once\n", "- Segment a specific region with a bounding box\n", "- Segment objects across the frames of a video\n", "- Apply SAM 3 to three customer use cases: satellite imagery, medical scans, and solar inspection" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem\n", "\n", "Classical segmentation models output a fixed set of class labels. SAM 3 instead performs *promptable concept segmentation*: you describe what you want with a short noun phrase (\"zebra\", \"car\") or a bounding box, and the model returns one instance mask for every matching object.\n", "\n", "| Approach | Output | Prompting |\n", "|----------|--------|-----------|\n", "| Panoptic segmentation (DETR) | Fixed COCO classes | None |\n", "| Promptable segmentation (SAM 3) | Per-instance binary masks | Text and/or boxes |" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Solution\n", "\n", "Use the [`sam3_for_segmentation()`](https://docs.pixeltable.com/sdk/latest/huggingface#udf-sam3_for_segmentation) UDF in [`pixeltable.functions.huggingface`](https://docs.pixeltable.com/sdk/latest/huggingface). It returns a typed result with one entry per detected instance: a confidence `score`, a bounding `box`, and a binary `mask`. The [`overlay_segmentation()`](https://docs.pixeltable.com/sdk/latest/vision#udf-overlay_segmentation) vision UDF accepts those masks directly, so you can visualize results without any glue code.\n", "\n", "> **Note:** `facebook/sam3` is a gated model. Request access on its [model page](https://huggingface.co/facebook/sam3), then authenticate with `huggingface-cli login` (or set the `HF_TOKEN` environment variable) before running this notebook." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install -qU pixeltable torch torchvision transformers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Set up a table of images\n", "\n", "We load five images: a grazing zebra, a bento box full of food, an aerial view of the Davis-Monthan AMARG aircraft boneyard, a chest X-ray, and a solar array. A Pixeltable table stores the originals; every example below computes its segmentation on the fly with `select`, so nothing extra is persisted." ] }, { "cell_type": "code", "execution_count": 1, "id": "b1ca013a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/cpestano/.pixeltable/pgdata\n", "Created directory 'sam3_demo'.\n", "Created table 'images'.\n", "Inserted 5 rows with 0 errors in 0.84 s (5.98 rows/s)\n" ] }, { "data": { "text/plain": [ "5 rows inserted." ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pixeltable as pxt\n", "from pixeltable.functions.huggingface import (\n", " sam3_for_segmentation,\n", " sam_automatic_mask_generation,\n", ")\n", "from pixeltable.functions.vision import bboxes_draw, overlay_segmentation\n", "\n", "pxt.drop_dir('sam3_demo', force=True)\n", "pxt.create_dir('sam3_demo')\n", "\n", "base_url = 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images'\n", "\n", "images = pxt.create_table(\n", " 'sam3_demo/images', {'name': pxt.String, 'image': pxt.Image}\n", ")\n", "images.insert(\n", " [\n", " {'name': 'zebra', 'image': f'{base_url}/000000000034.jpg'},\n", " {'name': 'bento', 'image': f'{base_url}/000000000009.jpg'},\n", " {'name': 'boneyard', 'image': f'{base_url}/amarg-boneyard.jpg'},\n", " {'name': 'chest_xray', 'image': f'{base_url}/chest-xray.jpg'},\n", " {'name': 'solar_farm', 'image': f'{base_url}/solar-farm.jpg'},\n", " ]\n", ")" ] }, { "cell_type": "markdown", "id": "671220be", "metadata": {}, "source": [ "### Segment from a text prompt\n", "\n", "Pass a short noun phrase as `text`. SAM 3 returns one binary mask per matching instance. Here we segment the zebra and overlay the mask on the original image with [`overlay_segmentation()`](https://docs.pixeltable.com/sdk/latest/vision#udf-overlay_segmentation)." ] }, { "cell_type": "code", "execution_count": 5, "id": "b58bceb2", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
imagesegmented
\n", " \n", "
\n", " \n", "
" ], "text/plain": [ " image \\\n", "0 \n", " \n", " \n", " segmented\n", " \n", " \n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "" ], "text/plain": [ " segmented\n", "0 \n", " \n", " \n", " image\n", " segmented\n", " \n", " \n", " \n", " \n", "
\n", " \n", "
\n", "
\n", " \n", "
\n", " \n", " \n", "" ], "text/plain": [ " image \\\n", "0 \n", " \n", " \n", " segmented_video\n", " \n", " \n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "" ], "text/plain": [ " segmented_video\n", "0 /Users/cpestano/.pixeltable/tmp/fe42ee8c-eaad-..." ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# bangkok.mp4 plays at 5 fps; rebuild at the same rate to keep the tempo.\n", "# Note: You can improve the final frame's quality by padding the end then trimming it off\n", "# this is because SAMv3 looks ahead a few frames for object tracking\n", "frames.group_by(videos).select(\n", " segmented_video=make_video(\n", " frames.pos,\n", " bboxes_draw(\n", " img=overlay_segmentation(\n", " frames.frame, frames.masks, draw_contours=True\n", " ),\n", " boxes=frames.boxes.to_list(),\n", " labels=frames.labels.to_list(),\n", " ),\n", " fps=5,\n", " )\n", ").collect()" ] }, { "cell_type": "markdown", "id": "516d5346", "metadata": {}, "source": [ "## Defense and intelligence: count aircraft in a satellite image\n", "\n", "Geospatial imagery often shows hundreds of vehicles, vessels, or aircraft in a single frame, and the assets of interest change from mission to mission. SAM 3's open-vocabulary text prompt picks every instance out of the scene in a single pass; the `scores` column lets you threshold by confidence before counting or feeding the masks into a downstream tracker." ] }, { "cell_type": "code", "execution_count": 14, "id": "3144964a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Added 15 column values with 0 errors in 34.06 s (0.44 rows/s)\n" ] }, { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
stitched
\n", " \n", "
" ], "text/plain": [ " stitched\n", "0 \n", " \n", " \n", " image\n", " segmented\n", " \n", " \n", " \n", " \n", "
\n", " \n", "
\n", "
\n", " \n", "
\n", " \n", " \n", "" ], "text/plain": [ " image \\\n", "0 \n", " \n", " \n", " image\n", " segmented\n", " \n", " \n", " \n", " \n", "
\n", " \n", "
\n", "
\n", " \n", "
\n", " \n", " \n", "" ], "text/plain": [ " image \\\n", "0