{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Step 4: Evaluating Detections\n",
        "This step demonstrates how to use FiftyOne to perform hands-on evaluation of your detection model.\n",
        "\n",
        "It covers the following concepts:\n",
        "\n",
        "- Evaluating your model using FiftyOne’s evaluation API\n",
        "- Viewing the best and worst performing samples in your dataset\n",
        "\n",
        "## Load a Detection Dataset\n",
        "\n",
        "In this example, we’ll load the [quickstart](https://docs.voxel51.com/user_guide/dataset_zoo/datasets.html#dataset-zoo-quickstart) dataset again from the FiftyOne Dataset Zoo, which has ground truth annotations and predictions from a PyTorch Faster-RCNN model for a few samples from the COCO dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import fiftyone as fo\n",
        "import fiftyone.zoo as foz\n",
        "\n",
        "dataset = foz.load_zoo_dataset(\"quickstart\")\n",
        "\n",
        "session = fo.launch_app(dataset)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Evaluate Detections\n",
        "\n",
        "Now that we have samples with ground truth and predicted objects, let’s use FiftyOne to evaluate the quality of the detections.\n",
        "\n",
        "FiftyOne provides a powerful [evaluation API](https://docs.voxel51.com/user_guide/evaluation.html) that contains a collection of methods for performing evaluation of model predictions. Since we’re working with object detections here, we’ll use [detection evaluation](https://docs.voxel51.com/user_guide/evaluation.html#detections).\n",
        "\n",
        "We can run evaluation on our samples via evaluate_detections(). Note that this method is available on both the `Dataset` and `DatasetView` classes, which means that we can run evaluation on our high_conf_view to assess the quality of only the high confidence predictions in our dataset.\n",
        "\n",
        "By default, this method will use the [COCO evaluation](https://cocodataset.org/#detection-eval) protocol, plus some extra goodies that we will use later."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Evaluate the predictions in the `predictions` field of our dataset\n",
        "# with respect to the objects in the `ground_truth` field\n",
        "results = dataset.evaluate_detections(\n",
        "    \"predictions\",\n",
        "    gt_field=\"ground_truth\",\n",
        "    eval_key=\"eval\",\n",
        "    compute_mAP=True,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The `results` object returned by the evaluation routine provides a number of convenient methods for analyzing our predictions.\n",
        "\n",
        "For example, let’s print a classification report for the top-10 most common classes in the dataset, as well as the mAP score:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Get the 10 most common classes in the dataset\n",
        "counts = dataset.count_values(\"ground_truth.detections.label\")\n",
        "classes_top10 = sorted(counts, key=counts.get, reverse=True)[:10]\n",
        "\n",
        "# Print a classification report for the top-10 classes\n",
        "results.print_report(classes=classes_top10)\n",
        "\n",
        "# Print out the mAP score\n",
        "print(f\"mAP score: {results.mAP()}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Model Evaluation Panel\n",
        "You can observe all of your model evaluation results in the FiftyOne app with the [Model Evaluation Panel](https://docs.voxel51.com/user_guide/evaluation.html#model-evaluation-panel-sub-new)! \n",
        "\n",
        "When you load a dataset in the App that contains one or more evaluations, you can open the [Model Evaluation panel](https://docs.voxel51.com/user_guide/app.html#app-model-evaluation-panel) to visualize and interactively explore the evaluation results in the App:\n",
        "\n",
        "![model_eval_1](https://cdn.voxel51.com/model_eval_1.webp)\n",
        "\n",
        "You can even click into individual classes to see model performance on filtered samples as well!\n",
        "\n",
        "![model_eval_2](https://cdn.voxel51.com/model_eval_2.webp)\n",
        "\n",
        "Follow to [Model Evaluation Doc Page](https://docs.voxel51.com/user_guide/app.html#app-model-evaluation-panel) to learn more about how you can perform model eval, compare multiple models, and more!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "OSS310",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.16"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}