{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", " Try in Google Colab\n", " \n", " \n", " \n", " \n", " Share via nbviewer\n", " \n", " \n", " \n", " \n", " View on GitHub\n", " \n", " \n", " \n", " \n", " Download notebook\n", " \n", "
\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Load WildMe Conservation Data into FiftyOne\n", "\n", "This notebook walks you through how to load data from the WildMe Collection in the Labeled Information Library of Alexandria: Biology and Conservation [LILA BC](https://lila.science/datasets) dataset!\n", "\n", "First, we'll download the data. Then, we'll load the data into FiftyOne. Finally, we'll add some visualization and similarity indexes to the data, as a bonus.\n", "\n", "**Note**: You can also browse this dataset for free at [try.fiftyone.ai](https://try.fiftyone.ai/datasets/wildme/samples)!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![WildMe Thumbnail](https://user-images.githubusercontent.com/12500356/260644073-17d5612a-aca3-45bb-b221-c429f3b7c545.gif)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To run this code, you will need to install the [FiftyOne open source library](https://github.com/voxel51/fiftyone) for dataset curation." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install fiftyone" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will import all of the necessary modules:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from datetime import datetime\n", "import json\n", "import numpy as np \n", "import os\n", "\n", "import fiftyone as fo\n", "import fiftyone.brain as fob\n", "from fiftyone import ViewField as F" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Downloading Data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "All of the raw data is hosted in Google Cloud buckets. We will be creating one combined dataset out of three collections:\n", "\n", "- [Leopard ID 2022](https://lila.science/datasets/leopard-id-2022/)\n", "- [Hyena ID 2022](https://lila.science/datasets/hyena-id-2022/)\n", "- [Beluga ID 2022](https://lila.science/datasets/beluga-id-2022/)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Run the following cell to batch download the zip files containing the X-ray images:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "## 3 collections of images and annotations from WildMe\n", "subsets = [\"beluga\", \"hyena\", \"leopard\"]\n", "\n", "for s in subsets:\n", " ## Download the data\n", " !wget https://storage.googleapis.com/public-datasets-lila/wild-me/{s}.coco.tar.gz\n", " ## Unzip the data\n", " !gunzip {s}.coco.tar.gz\n", " ## Untar the data\n", " !tar -xvf {s}.coco.tar\n", "\n", " ## Move the data to the correct location for COCO Import\n", " !mkdir {s}/data\n", " !mv {s}/images/train2022* {s}/data/\n", " !mv {s}/annotations/instances_train2022.json labels.json" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loading the data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have the data downloaded, we can create a FiftyOne dataset for all of it. First, let's create an empty dataset:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "dataset = fo.Dataset(\"WildMe\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, we can loop over the subdatasets, import each of them in COCO format, and add them to the main dataset. We also delete the `segmentations` sample field from the annotations, as it is not used." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "DATASET_TYPE = fo.types.COCODetectionDataset\n", "\n", "for subset in subsets:\n", " dataset_dir = f\"{subset}.coco/\"\n", "\n", " subset = fo.Dataset.from_dir(\n", " dataset_dir=dataset_dir,\n", " dataset_type=DATASET_TYPE,\n", " )\n", "\n", " ## delete unused segmentations field\n", " subset.delete_sample_field(\"segmentations\")\n", " dataset.add_samples(subset)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we can make the dataset persistent so it can be used in the future without having to re-download the data." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dataset.persistent=True" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Additionally, we can use the `add_dynamic_sample_Fields()` method to make all of the non-standard attributes on the dataset visible and filterable in the FiftyOne App:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dataset.add_dynamic_sample_fields()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In order to easily differentiate between the sub-collections in the dataset, we will save them each as their own view, and also tag samples with the sub-collection name. This will allow us to easily filter the dataset by sub-collection." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "beluga_view = dataset.match_labels(filter = F(\"label\") == \"beluga_whale\")\n", "dataset.save_view(\"beluga_view\", beluga_view)\n", "beluga_view.tag_samples(\"beluga\")\n", "\n", "hyena_view = dataset.match_labels(filter = F(\"label\") == \"hyena\")\n", "dataset.save_view(\"hyena_view\", hyena_view)\n", "hyena_view.tag_samples(\"hyena\")\n", "\n", "leopard_view = dataset.match_labels(filter = F(\"label\") == \"leopard\")\n", "dataset.save_view(\"leopard_view\", leopard_view)\n", "leopard_view.tag_samples(\"leopard\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Add Embeddings, Similarity, and Visualization" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In order to capture visual and conceptual similarity, we will use [DreamSim](https://dreamsim-nights.github.io/). We will compute embeddings once so that we can use them for the rest of the notebook. If you would like, you can swap out DreamSim for another embedding model, such as ResNet50." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install dreamsim" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from dreamsim import dreamsim\n", "from PIL import Image\n", "model, preprocess = dreamsim(pretrained=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Iterate through samples in the dataset, adding dreamsim embedding to each:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dataset.add_sample_field(\"dreamsim_embedding\", fo.ArrayField)\n", "for sample in dataset.iter_samples(autosave=True, progress=True):\n", " img1 = preprocess(Image.open(sample.filepath)).to(\"cuda\")\n", " sample[\"dreamsim_embedding\"] = np.array(model.embed(img1).cpu())[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can use these embeddings to compute an [image similarity index](https://docs.voxel51.com/user_guide/app.html#image-similarity) on the dataset:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fob.compute_similarity(\n", " dataset,\n", " embeddings = \"dreamsim_embedding\",\n", " brain_key = \"dreamsim_sim\",\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![WildMe Image Sim](https://user-images.githubusercontent.com/12500356/260641532-e7ff0833-9d93-4d79-93f6-d9f08e4985d8.gif)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As well as an embedding visualization, which we can generate by running UMAP on the embeddings to reduce them to 2 dimensions:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "fob.compute_visualization(\n", " dataset,\n", " embeddings = \"dreamsim_embedding\",\n", " brain_key = \"dreamsim_vis\",\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![WildMe Vis](https://user-images.githubusercontent.com/12500356/260641522-100cebf7-c6d2-4dbb-ac19-50d9f8aec0e0.gif)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also add a similarity index to the detection patches, making them searchable as well. Let's use a CLIP model so that we can search through the object detection patches with natural language queries:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "fob.compute_similarity(\n", " dataset,\n", " patches_field = \"detections\",\n", " model = \"clip-vit-base32-torch\",\n", " brain_key = \"clip_sim\"\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![WildMe Text Sim](https://user-images.githubusercontent.com/12500356/260641528-5fe9f705-8896-4051-814b-06c5e30ac9de.gif)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 4 }