{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Adding Classifier Predictions to a Dataset\n", "\n", "This recipe provides a glimpse into the possibilities for integrating FiftyOne into your ML workflows. Specifically, it covers:\n", "\n", "- [Loading](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html) an image classification dataset in FiftyOne\n", "- [Adding classifier predictions](https://voxel51.com/docs/fiftyone/user_guide/using_datasets.html#classification) to a dataset\n", "- Launching the [FiftyOne App](https://voxel51.com/docs/fiftyone/user_guide/app.html) and visualizing/exploring your data\n", "- Integrating the App into your data analysis workflow" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup\n", "\n", "If you haven't already, install FiftyOne:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install fiftyone" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You'll also need to install `torch` and `torchvision`, if necessary:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "scrolled": true }, "outputs": [], "source": [ "!pip install torch torchvision" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this example, we'll work with the test split of the CIFAR-10 dataset, which is conveniently available for download from the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_zoo/datasets.html#dataset-zoo-cifar10):" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Downloading split 'test' to '/Users/Brian/fiftyone/cifar10/test'\n", "Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to /Users/Brian/fiftyone/cifar10/tmp-download/cifar-10-python.tar.gz\n", "170500096it [00:05, 30536650.39it/s] \n", "Extracting /Users/Brian/fiftyone/cifar10/tmp-download/cifar-10-python.tar.gz to /Users/Brian/fiftyone/cifar10/tmp-download\n", " 100% |███| 10000/10000 [5.7s elapsed, 0s remaining, 1.9K samples/s] \n", "Dataset info written to '/Users/Brian/fiftyone/cifar10/info.json'\n" ] } ], "source": [ "# Downloads the test split of CIFAR-10\n", "!fiftyone zoo datasets download cifar10 --splits test" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll also download a pre-trained CIFAR-10 PyTorch model that we'll use to generate some predictions:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Cloning into 'PyTorch_CIFAR10'...\n", "remote: Enumerating objects: 82, done.\u001b[K\n", "remote: Counting objects: 100% (82/82), done.\u001b[K\n", "remote: Compressing objects: 100% (57/57), done.\u001b[K\n", "remote: Total 82 (delta 13), reused 59 (delta 12), pack-reused 0\u001b[K\n", "Unpacking objects: 100% (82/82), done.\n", "Note: checking out '2a2e76a56f943b70403796387d968704e74971ae'.\n", "\n", "You are in 'detached HEAD' state. You can look around, make experimental\n", "changes and commit them, and you can discard any commits you make in this\n", "state without impacting any branches by performing another checkout.\n", "\n", "If you want to create a new branch to retain commits you create, you may\n", "do so (now or later) by using -b with the checkout command again. Example:\n", "\n", " git checkout -b \n", "\n", "Downloading '1dGfpeFK_QG0kV-U6QDHMX2EOGXPqaNzu' to 'PyTorch_CIFAR10/cifar10_models/state_dicts/resnet50.pt'\n", " 100% |████| 719.8Mb/719.8Mb [2.7s elapsed, 0s remaining, 276.2Mb/s] \n" ] } ], "source": [ "# Download the software\n", "!git clone --depth 1 --branch v2.1 https://github.com/huyvnphan/PyTorch_CIFAR10.git\n", "\n", "# Download the pretrained model (90MB)\n", "!eta gdrive download --public \\\n", " 1dGfpeFK_QG0kV-U6QDHMX2EOGXPqaNzu \\\n", " PyTorch_CIFAR10/cifar10_models/state_dicts/resnet50.pt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loading an image classification dataset\n", "\n", "Suppose you have an image classification dataset on disk in the following\n", "format:\n", "\n", "```\n", "/\n", " data/\n", " .\n", " .\n", " ...\n", " labels.json\n", "```\n", "\n", "where `labels.json` is a JSON file in the following format:\n", "\n", "```\n", "{\n", " \"classes\": [\n", " ,\n", " ,\n", " ...\n", " ],\n", " \"labels\": {\n", " : ,\n", " : ,\n", " ...\n", " }\n", "}\n", "```\n", "\n", "In your current workflow, you may parse this data into a list of `(image_path, label)` tuples as follows:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('/Users/Brian/fiftyone/cifar10/test/data/000001.jpg', 'cat')\n", "('/Users/Brian/fiftyone/cifar10/test/data/000002.jpg', 'ship')\n", "('/Users/Brian/fiftyone/cifar10/test/data/000003.jpg', 'ship')\n", "('/Users/Brian/fiftyone/cifar10/test/data/000004.jpg', 'airplane')\n", "('/Users/Brian/fiftyone/cifar10/test/data/000005.jpg', 'frog')\n" ] } ], "source": [ "import json\n", "import os\n", "\n", "# The location of the dataset on disk that you downloaded above\n", "dataset_dir = os.path.expanduser(\"~/fiftyone/cifar10/test\")\n", "\n", "# Maps image UUIDs to image paths\n", "images_dir = os.path.join(dataset_dir, \"data\")\n", "image_uuids_to_paths = {\n", " os.path.splitext(n)[0]: os.path.join(images_dir, n)\n", " for n in os.listdir(images_dir)\n", "}\n", "\n", "labels_path = os.path.join(dataset_dir, \"labels.json\")\n", "with open(labels_path, \"rt\") as f:\n", " _labels = json.load(f)\n", "\n", "# Get classes\n", "classes = _labels[\"classes\"]\n", "\n", "# Maps image UUIDs to int targets\n", "labels = _labels[\"labels\"]\n", "\n", "# Make a list of (image_path, label) samples\n", "data = [(image_uuids_to_paths[u], classes[t]) for u, t in labels.items()]\n", "\n", "# Print a few data\n", "for sample in data[:5]:\n", " print(sample)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Building a [FiftyOne dataset](https://voxel51.com/docs/fiftyone/user_guide/using_datasets.html) from your samples in this format is simple:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 100% |███████████████████| 10000/10000 [44.9s elapsed, 0s remaining, 221.4 samples/s] \n", "Name: classifier-recipe\n", "Media type: image\n", "Num samples: 10000\n", "Persistent: False\n", "Info: {}\n", "Tags: []\n", "Sample fields:\n", " filepath: fiftyone.core.fields.StringField\n", " tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n", " metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n", " ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)\n" ] } ], "source": [ "import fiftyone as fo\n", "\n", "# Load the data into FiftyOne samples\n", "samples = []\n", "for image_path, label in data:\n", " samples.append(\n", " fo.Sample(\n", " filepath=image_path,\n", " ground_truth=fo.Classification(label=label),\n", " )\n", " )\n", "\n", "# Add the samples to a dataset\n", "dataset = fo.Dataset(\"classifier-recipe\")\n", "dataset.add_samples(samples)\n", "\n", "# Print some information about the dataset\n", "print(dataset)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ ",\n", "}>\n" ] } ], "source": [ "# Print a sample from the dataset\n", "print(dataset.first())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Working with views\n", "\n", "FiftyOne provides a powerful notion of [dataset views](https://voxel51.com/docs/fiftyone/user_guide/using_views.html) that you can use to explore subsets of the samples in your dataset.\n", "\n", "Here's an example operation:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset: classifier-recipe\n", "Media type: image\n", "Num samples: 5\n", "Tags: []\n", "Sample fields:\n", " filepath: fiftyone.core.fields.StringField\n", " tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n", " metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n", " ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)\n", "View stages:\n", " 1. Match(filter={'$expr': {'$eq': [...]}})\n", " 2. Limit(limit=5)\n" ] } ], "source": [ "# Used to write view expressions that involve sample fields\n", "from fiftyone import ViewField as F\n", "\n", "# Gets five airplanes from the dataset\n", "view = (\n", " dataset.match(F(\"ground_truth.label\") == \"airplane\")\n", " .limit(5)\n", ")\n", "\n", "# Print some information about the view you created\n", "print(view)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ ",\n", "}>\n" ] } ], "source": [ "# Print a sample from the view\n", "print(view.first())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Iterating over the samples in a view is easy:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/Users/Brian/fiftyone/cifar10/test/data/000004.jpg\n", "/Users/Brian/fiftyone/cifar10/test/data/000011.jpg\n", "/Users/Brian/fiftyone/cifar10/test/data/000022.jpg\n", "/Users/Brian/fiftyone/cifar10/test/data/000028.jpg\n", "/Users/Brian/fiftyone/cifar10/test/data/000045.jpg\n" ] } ], "source": [ "for sample in view:\n", " print(sample.filepath)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Adding model predictions\n", "\n", "Now let's [add our classifier's predictions](https://voxel51.com/docs/fiftyone/user_guide/using_datasets.html#classification) to our FiftyOne dataset in a new `predictions` field:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 100% |███████████████████████████| 5/5 [1.4s elapsed, 0s remaining, 3.6 batches/s] \n" ] } ], "source": [ "import sys\n", "\n", "import numpy as np\n", "import torch\n", "import torchvision\n", "from torch.utils.data import DataLoader\n", "\n", "import fiftyone.utils.torch as fout\n", "\n", "sys.path.insert(1, \"PyTorch_CIFAR10\")\n", "from cifar10_models import resnet50\n", "\n", "\n", "def make_cifar10_data_loader(image_paths, sample_ids, batch_size):\n", " mean = [0.4914, 0.4822, 0.4465]\n", " std = [0.2023, 0.1994, 0.2010]\n", " transforms = torchvision.transforms.Compose(\n", " [\n", " torchvision.transforms.ToTensor(),\n", " torchvision.transforms.Normalize(mean, std),\n", " ]\n", " )\n", " dataset = fout.TorchImageDataset(\n", " image_paths, sample_ids=sample_ids, transform=transforms\n", " )\n", " return DataLoader(dataset, batch_size=batch_size, num_workers=4)\n", "\n", "\n", "def predict(model, imgs):\n", " logits = model(imgs).detach().cpu().numpy()\n", " predictions = np.argmax(logits, axis=1)\n", " odds = np.exp(logits)\n", " confidences = np.max(odds, axis=1) / np.sum(odds, axis=1)\n", " return predictions, confidences\n", "\n", "\n", "#\n", "# Load model\n", "#\n", "# Model performance numbers are available at:\n", "# https://github.com/huyvnphan/PyTorch_CIFAR10\n", "#\n", "\n", "model = resnet50(pretrained=True)\n", "\n", "#\n", "# Extract a few images to process\n", "#\n", "\n", "num_samples = 25\n", "batch_size = 5\n", "\n", "view = dataset.take(num_samples, seed=51)\n", "\n", "image_paths, sample_ids = zip(*[(s.filepath, s.id) for s in view])\n", "data_loader = make_cifar10_data_loader(image_paths, sample_ids, batch_size)\n", "\n", "#\n", "# Perform prediction and store results in dataset\n", "#\n", "\n", "with fo.ProgressBar() as pb:\n", " for imgs, sample_ids in pb(data_loader):\n", " predictions, confidences = predict(model, imgs)\n", "\n", " # Add predictions to your FiftyOne dataset\n", " for sample_id, prediction, confidence in zip(\n", " sample_ids, predictions, confidences\n", " ):\n", " sample = dataset[sample_id]\n", " sample[\"predictions\"] = fo.Classification(\n", " label=classes[prediction],\n", " confidence=confidence,\n", " )\n", " sample.save()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can print our dataset to verify that a `predictions` field has been added to its schema:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Name: classifier-recipe\n", "Media type: image\n", "Num samples: 10000\n", "Persistent: False\n", "Info: {}\n", "Tags: []\n", "Sample fields:\n", " filepath: fiftyone.core.fields.StringField\n", " tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n", " metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n", " ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)\n", " predictions: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)\n" ] } ], "source": [ "print(dataset)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's explore the predictions we added by creating a view that sorts the samples in order of prediction confidence:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of samples with predictions: 25\n", "\n", "Highest confidence prediction:\n", "\n", ",\n", " 'predictions': ,\n", "}>\n" ] } ], "source": [ "pred_view = (\n", " dataset\n", " .exists(\"predictions\")\n", " .sort_by(\"predictions.confidence\", reverse=True)\n", ")\n", "\n", "print(\"Number of samples with predictions: %s\\n\" % len(pred_view))\n", "print(\"Highest confidence prediction:\\n\")\n", "print(pred_view.first())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using the FiftyOne App\n", "\n", "The [FiftyOne App](https://voxel51.com/docs/fiftyone/user_guide/app.html) allows you easily visualize, explore, search, filter, your datasets.\n", "\n", "You can explore the App interactively through the GUI, and you can even interact with it in real-time from your Python interpreter!" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Open the dataset in the App\n", "session = fo.launch_app(dataset)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Show five random samples in the App\n", "session.view = dataset.take(5)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Show the samples for which we added predictions above\n", "session.view = pred_view" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Show the full dataset again\n", "session.view = None" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can select samples in the App by clicking on the images. Try it!\n", "\n", "After you've selected some images in the App, you can hop back over to Python and make a view that contains those samples!" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset: classifier-recipe\n", "Media type: image\n", "Num samples: 8\n", "Tags: []\n", "Sample fields:\n", " filepath: fiftyone.core.fields.StringField\n", " tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n", " metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n", " ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)\n", " predictions: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)\n", "View stages:\n", " 1. Select(sample_ids=['602fe4c1b6fdaf68aad0a07a', '602fe4c1b6fdaf68aad0a082', '602fe4c1b6fdaf68aad0a086', ...])\n" ] } ], "source": [ "# Make a view containing the currently selected samples in the App\n", "selected_view = dataset.select(session.selected)\n", "\n", "# Print details about the selected samples\n", "print(selected_view)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "session.freeze() # screenshot the active App for sharing" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 4 }