{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "d1KVLcqHWdgI" }, "source": [ "# Downloading and Evaluating Open Images\n", "\n", "Downloading Google's [Open Images dataset](https://storage.googleapis.com/openimages/web/download.html) is now easier than ever with the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_zoo/index.html#dataset-zoo-open-images-v7)! You can load all three splits of Open Images V7, including image-level labels, detections, segmentations, visual relationships, and point labels.\n", "\n", "FiftyOne also natively supports [Open Images-style evaluation](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html#open-images-style-evaluation), so you can easily evaluate your object detection models and explore the results directly in the library.\n", "\n", "This walkthrough covers:\n", "\n", "- Downloading [Open Images](https://storage.googleapis.com/openimages/web/index.html) from the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_zoo/index.html)\n", "- Computing predictions using a model from the [FiftyOne Model Zoo](https://voxel51.com/docs/fiftyone/user_guide/model_zoo/index.html)\n", "- Performing [Open Images-style evaluation](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html#open-images-style-evaluation) in FiftyOne to evaluate a model and compute its mAP\n", "- Exploring the dataset and [evaluation results](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html)\n", "- [Visualizing embeddings](https://voxel51.com/docs/fiftyone/user_guide/brain.html#visualizing-embeddings) through [interactive plots](https://voxel51.com/docs/fiftyone/user_guide/plots.html)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**So, what's the takeaway?**\n", "\n", "Starting a new ML project takes data and time, and the datasets in the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_zoo/index.html) can help jump start the development process.\n", "\n", "Open Images in particular is one of the largest publicly available datasets for object detections, classification, segmentation, and more. Additionally, with [Open Images evaluation](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html#open-images-style-evaluation) available natively in FiftyOne, you can quickly evaluate your models and compute mAP and PR curves.\n", "\n", "While metrics like mAP are often used to compare models, the best way to improve your model's performance isn't to look at aggregate metrics but instead to get hands-on with your evaluation and visualize how your model performs on individual samples. All of this is made easy with FiftyOne!" ] }, { "cell_type": "markdown", "metadata": { "id": "R5SJCYnQXnOL" }, "source": [ "## Setup" ] }, { "cell_type": "markdown", "metadata": { "id": "Swt8yYgIbOkv" }, "source": [ "If you haven't already, install FiftyOne:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "627MxtHOUjU6", "outputId": "0c87eadb-60e4-452d-fa13-a5b8d06b218f" }, "outputs": [], "source": [ "!pip install fiftyone" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this tutorial, we’ll use some [TensorFlow models](https://github.com/tensorflow/models) and [PyTorch](https://pytorch.org/vision/stable/index.html) to generate predictions and embeddings, and we’ll use the [UMAP method](https://github.com/lmcinnes/umap) to reduce the dimensionality of embeddings, so we need to install the corresponding packages:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "!pip install tensorflow torch torchvision umap-learn" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This tutorial also includes some of FiftyOne's [interactive plotting capabilities](https://voxel51.com/docs/fiftyone/user_guide/plots.html).\n", "\n", "The recommended way to work with FiftyOne’s interactive plots is in [Jupyter notebooks](https://jupyter.org/) or [JupyterLab](https://jupyterlab.readthedocs.io/en/stable/). In these environments, you can leverage the full power of plots by [attaching them to the FiftyOne App](https://voxel51.com/docs/fiftyone/user_guide/plots.html#attaching-plots) and bidirectionally interacting with the plots and the App to identify interesting subsets of your data.\n", "\n", "To use interactive plots in Jupyter notebooks, ensure that you have the `ipywidgets` package installed:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "!pip install 'ipywidgets>=8,<9'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you’re working in JupyterLab, refer to [these instructions](https://voxel51.com/docs/fiftyone/user_guide/plots.html#working-in-notebooks) to get setup.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "

Support for interactive plots in non-notebook contexts and Google Colab is coming soon! In the meantime, you can still use FiftyOne's plotting features in those environments, but you must manually call plot.show() to update the state of a plot to match the state of a connected session, and any callbacks that would normally be triggered in response to interacting with a plot will not be triggered.

\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "wx-yhHuKrsOK" }, "source": [ "## Loading Open Images\n", "\n", "In this section, we'll load various subsets of Open Images from the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_zoo/index.html) and visualize them using FiftyOne.\n", "\n", "Let's start by downloading a small sample of 100 randomly chosen images + annotations:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "id": "q23aIrZSr6FR" }, "outputs": [], "source": [ "import fiftyone as fo\n", "import fiftyone.zoo as foz" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "id": "Nr1Fq8PPh95d" }, "outputs": [], "source": [ "dataset = foz.load_zoo_dataset(\n", " \"open-images-v7\",\n", " split=\"validation\",\n", " max_samples=100,\n", " seed=51,\n", " shuffle=True,\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "9nVJIHp7Ay6e" }, "source": [ "Now let's launch the [FiftyOne App](https://voxel51.com/docs/fiftyone/user_guide/app.html) so we can explore the [dataset](https://voxel51.com/docs/fiftyone/user_guide/using_datasets.html#datasets) we just downloaded." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 821, "resources": { "https://localhost:5151/polling?sessionId=d41be399-4e98-4781-865a-a7821a9e37b4": { "data": "eyJtZXNzYWdlcyI6IFtdfQ==", "headers": [ [ "access-control-allow-headers", "x-requested-with" ], [ "content-type", "text/html; charset=UTF-8" ] ], "ok": true, "status": 200, "status_text": "" } } }, "id": "D5Xw1YZZtPsW", "outputId": "967d1fe3-0be0-4633-c5f3-7f6892511e34" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Connected to FiftyOne on port 5151 at localhost.\n", "If you are not connecting to a remote session, you may need to start a new session and specify a port\n" ] }, { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "session = fo.launch_app(dataset.view())" ] }, { "cell_type": "markdown", "metadata": { "id": "TS7hzPZx2iQf" }, "source": [ "Loading Open Images with FiftyOne also automatically stores relevant [labels](https://voxel51.com/docs/fiftyone/user_guide/using_datasets.html#labels) and [metadata](https://voxel51.com/docs/fiftyone/user_guide/using_datasets.html#metadata) like classes, attributes, and a class hierarchy that is used for evaluation in the dataset's `info` dictionary:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "L_yrMhwu2xGQ", "outputId": "b8a45faf-d9c1-498a-ea70-7a24b8ed4f66" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "dict_keys(['hierarchy', 'attributes_map', 'attributes', 'segmentation_classes', 'point_classes', 'classes_map'])\n" ] } ], "source": [ "print(dataset.info.keys())" ] }, { "cell_type": "markdown", "metadata": { "id": "ZE9ObTIXA4Ab" }, "source": [ "When loading Open Images from the dataset zoo, there are a [variety of available parameters](https://voxel51.com/docs/fiftyone/api/fiftyone.zoo.datasets.base.html#fiftyone.zoo.datasets.base.OpenImagesV7Dataset) that you can pass to `load_zoo_dataset()` to specify a subset of the images and/or label types to download:\n", "\n", "- `label_types` - a list of label types to load. The supported values are (`\"detections\", \"classifications\", \"points\", \"segmentations\", \"relationships\"`) for Open Images V7. Open Images v6 is the same except that it does not contain point labels. By default, all available labels types will be loaded. Specifying `[]` will load only the images\n", "- `classes` - a list of classes of interest. If specified, only samples with at least one object, segmentation, or image-level label in the specified classes will be downloaded\n", "- `attrs` - a list of attributes of interest. If specified, only download samples if they contain at least one attribute in `attrs` or one class in `classes` (only applicable when `label_types` contains `\"relationships\"`)\n", "- `load_hierarchy` - whether to load the class hierarchy into `dataset.info[\"hierarchy\"]`\n", "- `image_ids` - an array of specific image IDs to download\n", "- `image_ids_file` - a path to a `.txt`, `.csv`, or `.json` file containing image IDs to download\n", "\n", "In addition, [like all other zoo datasets](https://voxel51.com/docs/fiftyone/user_guide/dataset_zoo/datasets.html), you can specify:\n", "\n", "- `max_samples` - the maximum number of samples to load\n", "- `shuffle` - whether to randomly chose which samples to load if `max_samples` is given\n", "- `seed` - a random seed to use when shuffling" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's use some of these parameters to download a 100 sample subset of Open Images containing segmentations and image-level labels for the classes \"Burrito\", \"Cheese\", and \"Popcorn\"." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "1CkIkqtzt9z4", "outputId": "66b29d52-4e19-43bc-99a8-edfdb18fa839" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Downloading split 'validation' to 'datasets/open-images-v7/validation' if necessary\n", "Only found 83 (<100) samples matching your requirements\n", "Necessary images already downloaded\n", "Existing download of split 'validation' is sufficient\n", "Loading existing dataset 'open-images-food'. To reload from disk, either delete the existing dataset or provide a custom `dataset_name` to use\n" ] } ], "source": [ "dataset = foz.load_zoo_dataset(\n", " \"open-images-v7\", \n", " split=\"validation\", \n", " label_types=[\"segmentations\", \"classifications\"], \n", " classes = [\"Burrito\", \"Cheese\", \"Popcorn\"],\n", " max_samples=100,\n", " seed=51,\n", " shuffle=True,\n", " dataset_name=\"open-images-food\",\n", ")" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 821, "resources": { "https://localhost:5151/polling?sessionId=a3fc3576-1f7f-4b13-9a4b-147674a3012b": { "data": "eyJtZXNzYWdlcyI6IFtdfQ==", "headers": [ [ "access-control-allow-headers", "x-requested-with" ], [ "content-type", "text/html; charset=UTF-8" ] ], "ok": true, "status": 200, "status_text": "" } } }, "id": "aeQd9A6-uDMy", "outputId": "e89bded4-1aa5-4b9c-dee2-325d19105021" }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "session.view = dataset.view()" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "id": "T1FA-QpAFa0r" }, "outputs": [], "source": [ "session.freeze() # screenshots App for sharing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can do the same for visual relationships. For example, we can download only samples that contain a relationship with the \"Wooden\" attribute." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5PqZEKcCffUe", "outputId": "a6dd7529-dd6a-468b-9d80-649d49c6bf7c" }, "outputs": [], "source": [ "dataset = foz.load_zoo_dataset(\n", " \"open-images-v7\",\n", " split=\"validation\", \n", " label_types=[\"relationships\"], \n", " attrs=[\"Wooden\"],\n", " max_samples=100,\n", " seed=51,\n", " shuffle=True,\n", " dataset_name=\"open-images-relationships\", \n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can visualize relationships in the App by clicking on a sample to open the [App's expanded view](https://voxel51.com/docs/fiftyone/user_guide/app.html#viewing-a-sample). From there, you can hover over objects to see their attributes in a tooltip.\n", "\n", "Alternatively, you can use the settings menu in the lower-right corner of the media player to set `show_attributes` to True to make attributes appear as persistent boxes (as shown below). This can also be achieved programmatically by [configuring the App](https://voxel51.com/docs/fiftyone/user_guide/config.html#configuring-the-app):" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 821, "resources": { "https://localhost:5151/polling?sessionId=416bb834-a5e4-4292-bb73-5ba727a5617f": { "data": "eyJtZXNzYWdlcyI6IFtdfQ==", "headers": [ [ "access-control-allow-headers", "x-requested-with" ], [ "content-type", "text/html; charset=UTF-8" ] ], "ok": true, "status": 200, "status_text": "" } } }, "id": "EcqIvwTYftNO", "outputId": "59520b38-69da-4390-8365-ee35f5c878d7" }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", "