{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "de-cell-0",
   "metadata": {},
   "source": "# Building Annotation Workflows and Ontologies with FiftyOne\n\nThis tutorial builds a complete annotation loop in FiftyOne:\n\n- **Curate** \u2014 select what to label with [embeddings](https://docs.voxel51.com/brain.html#brain-embeddings-visualization), [similarity](https://docs.voxel51.com/brain.html#brain-image-similarity), [uniqueness](https://docs.voxel51.com/brain.html#brain-image-uniqueness), and zero-shot tagging\n- **Annotate** \u2014 define an [ontology](https://docs.voxel51.com/user_guide/annotation.html), route samples through a multi-stage [workflow](https://voxel51.com/annotation), then draft a first pass with an agent or [zoo models](https://docs.voxel51.com/model_zoo/models.html)\n- **Review** \u2014 rank labels by [estimated error](https://docs.voxel51.com/brain.html#brain-label-mistakes) and fix the worst\n- **Evaluate** \u2014 [score the model](https://docs.voxel51.com/user_guide/evaluation.html#evaluating-detections-coco) by scenario and feed failures back into the next curation pass"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-1",
   "metadata": {},
   "source": "<div class=\"admonition note\">\n<p class=\"admonition-title\">This tutorial uses FiftyOne Enterprise</p>\n<p>Workflows, ontologies, and Agentic Labeling are\n<a href=\"https://voxel51.com/annotation\">FiftyOne Enterprise</a> features, shown here\nthrough the App UI. The analysis steps \u2014 embeddings, similarity, uniqueness,\nmistakenness, evaluation, and zero-shot pre-labeling \u2014 also run in\n<a href=\"https://github.com/voxel51/fiftyone\">open-source FiftyOne</a>.</p>\n</div>"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-2",
   "metadata": {},
   "source": "![The FiftyOne annotation loop: curate, annotate, review, evaluate](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_flywheel.webp)"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-3",
   "metadata": {},
   "source": "## Setup\n\nInstall FiftyOne and the extras used below:"
  },
  {
   "cell_type": "code",
   "id": "de-cell-4",
   "metadata": {},
   "source": "!pip install fiftyone fiftyone-brain umap-learn\n# Optional, for the training step at the end:\n# !pip install ultralytics",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-5",
   "metadata": {},
   "source": "### Connect to FiftyOne Enterprise\n\nSet these two variables before importing `fiftyone` (generate a key under\n**Settings \u2192 API keys**). Skip this cell to run open source against a local dataset."
  },
  {
   "cell_type": "code",
   "id": "de-cell-6",
   "metadata": {},
   "source": "import os\n\n# Uncomment and fill in to target your FiftyOne Enterprise deployment:\n# os.environ[\"FIFTYONE_API_URI\"] = \"https://<your-deployment>-api.fiftyone.ai\"\n# os.environ[\"FIFTYONE_API_KEY\"] = \"<your-api-key>\"\n\nimport fiftyone as fo\nimport fiftyone.brain as fob\nimport fiftyone.zoo as foz\nfrom fiftyone import ViewField as F",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-7",
   "metadata": {},
   "source": "### Load the example data\n\nUse the [quickstart](https://docs.voxel51.com/dataset_zoo/datasets/quickstart.html)\ndataset (200 COCO images with `ground_truth` and `predictions`) from the\n[FiftyOne Dataset Zoo](https://docs.voxel51.com/dataset_zoo/index.html) as the\nrunning example. The agentic-labeling section later pulls in a second dataset from\nthe [Hugging Face Hub](https://docs.voxel51.com/integrations/huggingface.html).\n\n> Enterprise datasets reference cloud media (`gs://`, `s3://`) \u2014 point sample\n> `filepath`s at storage your deployment can read."
  },
  {
   "cell_type": "code",
   "id": "de-cell-8",
   "metadata": {},
   "source": "dataset = foz.load_zoo_dataset(\"quickstart\")\ndataset.persistent = True\nprint(dataset)",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "id": "de-cell-9",
   "metadata": {},
   "source": "session = fo.launch_app(dataset)",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-10",
   "metadata": {},
   "source": "## Stage 1 \u2014 Curate: select what to label\n\nDecide what's worth labeling before annotation starts."
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-11",
   "metadata": {},
   "source": "### Visualize in embedding space\n\nCompute a 2D\n[visualization](https://docs.voxel51.com/brain.html#brain-embeddings-visualization),\nthen open the **Embeddings panel** in the App."
  },
  {
   "cell_type": "code",
   "id": "de-cell-12",
   "metadata": {},
   "source": "fob.compute_visualization(\n    dataset,\n    model=\"clip-vit-base32-torch\",\n    method=\"umap\",\n    brain_key=\"img_viz\",\n)\nsession.view = dataset.view()",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-13",
   "metadata": {},
   "source": "![The Embeddings panel: the quickstart dataset visualized in 2D with the img_viz brain key](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_embeddings.webp)"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-14",
   "metadata": {},
   "source": "### Score redundancy and coverage\n\nRank by [uniqueness](https://docs.voxel51.com/brain.html#brain-image-uniqueness) and\n[representativeness](https://docs.voxel51.com/brain.html#brain-image-representativeness),\nand flag [near-duplicates](https://docs.voxel51.com/brain.html#brain-near-duplicates)."
  },
  {
   "cell_type": "code",
   "id": "de-cell-15",
   "metadata": {},
   "source": "fob.compute_uniqueness(dataset)\nfob.compute_representativeness(dataset, method=\"cluster-center\")\n\n# Most unique (least redundant) samples first\nunique_view = dataset.sort_by(\"uniqueness\", reverse=True)\nprint(unique_view.first().uniqueness)",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "id": "de-cell-16",
   "metadata": {},
   "source": "# Flag near-duplicates so you don't pay to label the same thing twice\nindex = fob.compute_near_duplicates(dataset)\nprint(index.duplicates_view())",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-17",
   "metadata": {},
   "source": "### Search by text and similarity\n\nBuild a [similarity index](https://docs.voxel51.com/brain.html#brain-image-similarity),\nthen query by [natural language](https://docs.voxel51.com/brain.html#brain-similarity-text)\nor by an example image."
  },
  {
   "cell_type": "code",
   "id": "de-cell-18",
   "metadata": {},
   "source": "fob.compute_similarity(\n    dataset,\n    model=\"clip-vit-base32-torch\",\n    brain_key=\"img_sim\",\n)\n\n# Natural-language query\nnight_view = dataset.sort_by_similarity(\n    \"a photo taken at night\", k=25, brain_key=\"img_sim\"\n)",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "id": "de-cell-19",
   "metadata": {},
   "source": "# Few-shot: retrieve samples similar to an example you care about\nquery_id = dataset.first().id\nsimilar_view = dataset.sort_by_similarity(query_id, k=25, brain_key=\"img_sim\")",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-20",
   "metadata": {},
   "source": "### Auto-tag by metadata\n\nTag each image as an indoor or outdoor scene with a zero-shot\n[CLIP model](https://docs.voxel51.com/model_zoo/models.html) \u2014 a field you can color\nthe Embeddings panel by and slice evaluation on later."
  },
  {
   "cell_type": "code",
   "id": "de-cell-21",
   "metadata": {},
   "source": "# Zero-shot scene tagging with CLIP\nscene_model = foz.load_zoo_model(\n    \"clip-vit-base32-torch\",\n    text_prompt=\"A photo of \",\n    classes=[\"an indoor scene\", \"an outdoor scene\"],\n)\ndataset.apply_model(scene_model, label_field=\"scene\")\nprint(dataset.count_values(\"scene.label\"))",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-22",
   "metadata": {},
   "source": "### Build the subset to annotate\n\nSave the most-unique samples as a\n[saved view](https://docs.voxel51.com/user_guide/using_views.html#saving-views) to\nsend to annotation."
  },
  {
   "cell_type": "code",
   "id": "de-cell-23",
   "metadata": {},
   "source": "to_annotate = dataset.sort_by(\"uniqueness\", reverse=True).limit(10)\ndataset.save_view(\"to_annotate\", to_annotate, overwrite=True)\nprint(dataset.load_saved_view(\"to_annotate\"))",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-24",
   "metadata": {},
   "source": "## Stage 2 \u2014 Annotate: ontologies, workflows, and model-assisted labeling\n\nDefine a shared ontology, route the curated view through a multi-stage workflow, and\nspeed up the first pass with models \u2014 so annotators verify rather than draw from\nscratch."
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-25",
   "metadata": {},
   "source": "### Define an ontology in code, apply it in the App\n\nAn [ontology](https://docs.voxel51.com/user_guide/annotation.html) is your shared\nlabel schema \u2014 classes, attributes, and a taxonomy. Define it **once in code**, then\n**apply it to a field in the App** so every annotator gets the same classes and\nattributes. Conditional attributes (`when=...`) appear only when relevant."
  },
  {
   "cell_type": "code",
   "id": "de-cell-26",
   "metadata": {},
   "source": "ontology = fo.AnnotationOntology(\n    name=\"coco-objects\",\n    description=\"Common object classes for the quickstart (COCO) dataset\",\n    attributes=[\n        fo.AttributeSpec(name=\"occluded\", type=\"bool\", component=\"checkbox\"),\n        fo.AttributeSpec(name=\"truncated\", type=\"bool\", component=\"checkbox\"),\n        fo.AttributeSpec(\n            name=\"pose\",\n            type=\"str\",\n            component=\"dropdown\",\n            values=[\"standing\", \"sitting\", \"lying\"],\n            # Only show this attribute for person labels\n            when=fo.WhenEquals(field=\"label\", value=\"person\"),\n        ),\n    ],\n)\nontology.save()\nprint(fo.list_ontologies())",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-27",
   "metadata": {},
   "source": "> The ontology SDK writes to the database directly, so run it with a direct\n> database connection (not API mode)."
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-28",
   "metadata": {},
   "source": "Now **apply it in the App**, click by click:\n\n1. Open the dataset, then open the **field schema editor** (the schema/settings control).\n2. Click the field you want to govern \u2014 here **ground_truth**.\n3. In **Edit field schema**, turn the **Ontology** toggle **on**.\n4. Open the **Chosen ontology** dropdown and select **coco-objects**.\n5. Click **Save**.\n\nEvery annotation task on that field now offers exactly the classes and attributes\nyou defined in code."
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-29",
   "metadata": {},
   "source": "### Manage annotation with multi-stage workflows\n\n*FiftyOne Enterprise.* The **Annotate** tab turns your curated saved view into a\nmanaged, multi-stage labeling effort \u2014 assign stages, track progress per task, and\nautomatically loop rejected work back to annotators. Follow it click by click."
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-30",
   "metadata": {},
   "source": "**Step 1 \u2014 Create a workflow.** Open the **Annotate** tab. Click\n**+ New workflow** (top right). In the dialog, type a **Name** (for example,\n`Annotate Most Unique`), optionally add a description, and click **Create workflow**.\n\n![The New workflow dialog: name the workflow, then click Create workflow](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_wf_new.webp)"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-31",
   "metadata": {},
   "source": "**Step 2 \u2014 Pick a template.** On the next screen, under **Start from a\ntemplate**, click a card to add its stages to the canvas:\n\n- **Human-in-the-loop** \u2014 Annotate \u2192 Review (one QA pass).\n- **Two-tier review** \u2014 Annotate \u2192 Review \u2192 Final review (a second sign-off). Click\n  this one for the rest of the walkthrough.\n- Or click **+ Blank canvas** to add stages yourself.\n\n![The template picker: choose Two-tier review, or start from a blank canvas](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_wf_templates.webp)"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-32",
   "metadata": {},
   "source": "**Step 3 \u2014 Point it at your data and start.** On the canvas, click the\n**Input samples** node. In the **Configuration** panel on the right, select\n**Saved view**, then choose **to_annotate** from the dropdown (or pick **Whole\ndataset**). The dashed red **Rejected \u2192 Annotate** edges are already wired by the\ntemplate, so rejected samples will loop back. Click **Start workflow** (top right).\n\n![Configuring Input samples to the to_annotate saved view, then Start workflow](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_wf_canvas.webp)"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-33",
   "metadata": {},
   "source": "**Step 4 \u2014 Open your task queue.** Starting the workflow generates tasks and\nreturns you to the **Annotate** tab. Under **My tasks**, each stage assigned to you\nshows its progress (for example, `4 / 10 samples \u00b7 6 left`) and an action button \u2014\n**Label** for annotate stages, **Review** for review stages. The workflow now shows\nas **Running**.\n\n![The Annotate dashboard: My tasks with Label and Review actions; the workflow is Running](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_wf_tasks.webp)"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-34",
   "metadata": {},
   "source": "**Step 5 \u2014 Label.** Click **Label** next to the Annotate task. FiftyOne opens a\n**locked** labeling view of only your assigned samples. Draw labels against the\nontology, then mark each finished sample **Labeled** \u2014 it gets a green **LABELED**\nbadge and the progress bar advances. Use **Resume labeling** to pick up where you\nleft off; close the task when the count reads `0 left`.\n\n![The locked labeling view: samples marked with green LABELED badges as you finish them](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_wf_label.webp)"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-35",
   "metadata": {},
   "source": "**Step 6 \u2014 Review: approve or reject.** Back on the **Annotate** tab, click\n**Review** next to the **First review** task. For each submitted sample, click\n**Approve** (\u2713) to accept it or **Reject** (\u2717) to send it back. Approved samples\nadvance to **Final review**; rejected samples follow the **Rejected** edge back to\nthe Annotate stage. Click **Task complete** when the queue is empty.\n\n![The review view: each sample marked APPROVED or REJECTED, then Task complete](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_wf_review.webp)"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-36",
   "metadata": {},
   "source": "**Step 7 \u2014 Close out the loop.** Rejected samples reappear as new **Annotate**\ntasks (the remaining count grows back). Re-label them, send them through review\nagain, and clear the **Final review** task. When every stage reads `0 left`, the\nworkflow flips to **Complete** \u2014 10 / 10 samples at 100%.\n\n![The dashboard with every task at 0 left and the workflow marked Complete (10/10, 100%)](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_wf_complete.webp)"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-37",
   "metadata": {},
   "source": "### Agentic labeling\n\n*FiftyOne Enterprise.* **Agentic Labeling** is a new annotation feature: train a\nlabeling *agent* from a plain-language prompt plus a few example crops, test it, then\nrun it across the dataset \u2014 so you can capture a concept that isn't in your schema\nyet. Here we detect construction workers who are **not** wearing a hard hat.\n\nLoad the\n[Voxel51 hard-hat detection dataset](https://huggingface.co/datasets/Voxel51/hard-hat-detection)\nfrom the [Hugging Face Hub](https://docs.voxel51.com/integrations/huggingface.html).\nIts `ground_truth` field holds Helmet / Person / Head boxes, but there is no\n\"worker without a hard hat\" class \u2014 that is exactly what the agent will add."
  },
  {
   "cell_type": "code",
   "id": "de-cell-38",
   "metadata": {},
   "source": "import fiftyone.utils.huggingface as fouh\n\nhardhat = fouh.load_from_hub(\n    \"Voxel51/hard-hat-detection\",\n    name=\"hard-hat-detection\",\n    max_samples=200,\n    persistent=True,\n)\nsession.dataset = hardhat  # point the running App at the hard-hat dataset",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-39",
   "metadata": {},
   "source": "**Step 1 \u2014 Open the Agentic Labeler.** In the App, click the **+** beside the\nsample-grid tab and choose **Agentic Labeler**. The panel opens on the **Runs** /\n**Agents** tabs. Click **+ Train Agent**.\n\n![The Agentic Labeler panel, showing the Runs and Agents tabs and the Train Agent button](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_agentic_panel.webp)"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-40",
   "metadata": {},
   "source": "**Step 2 \u2014 Describe the labels.** In the **Train new agent** form, type the\ninstruction in the **TEXT PROMPT** box \u2014 here, *Find site workers not wearing hard\nhats*. Under **SETTINGS \u2192 Task**, click the **Detection** card (bounding boxes). In\nthe **Allowed classes** box, type `no_hardhat` and press Enter so the agent assigns\nexactly that one label.\n\n![The Train new agent form: text prompt, the Detection task card, and the no_hardhat allowed class](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_agentic_train.webp)"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-41",
   "metadata": {},
   "source": "**Step 3 \u2014 Add visual prompts.** Scroll to **VISUAL PROMPTS** and give the\nagent examples. Under **Positive**, click **Pick from grid** and select five crops\nthat *do* show the concept (workers with no hard hat). Under **Negative**, select\nfive crops that *do not* (workers wearing one).\n\n![Selecting five positive and five negative example crops as visual prompts](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_agentic_visual.webp)"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-42",
   "metadata": {},
   "source": "**Step 4 \u2014 Test, then save.** Click **Run test** to preview the prompt on five\nsample images. Set **Offload to field** to `predictions` (where the labels will land)\nand inspect the thumbnails \u2014 `no_hardhat` boxes where the agent fired, and *I did not\nfind any bounding boxes* where it correctly abstained. When the previews look right,\nclick **Save agent**.\n\n![Test results on five samples, set to offload to the predictions field, with the Save agent button](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_agentic_test.webp)"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-43",
   "metadata": {},
   "source": "**Step 5 \u2014 Run the agent.** Back on the **Runs** tab, click **New Run**, choose\nthe saved agent and the samples to label, and start it. The run card reports progress\nand flips to **Completed** \u2014 here `no_hardhat_detector` labeled 60 samples with 60\nlabels.\n\n![The Runs tab with the no_hardhat_detector run marked Completed: 60 samples, 60 labels](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_agentic_run.webp)"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-44",
   "metadata": {},
   "source": "**Step 6 \u2014 Review the agent's labels.** Open a sample to see the agent's\n`predictions` (the blue `no_hardhat` boxes) alongside the original `ground_truth`.\nToggle either field in the right sidebar, filter by confidence, and bulk-approve or\ncorrect \u2014 the same review loop you built above.\n\n![A sample with no_hardhat predictions next to ground_truth, both fields listed in the sidebar](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_agentic_results.webp)"
  },
  {
   "cell_type": "code",
   "id": "de-cell-45",
   "metadata": {},
   "source": "# Done with the agentic detour \u2014 point the App back to the quickstart dataset\nsession.dataset = dataset",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-46",
   "metadata": {},
   "source": "### A scriptable first pass (open source)\n\nDraft labels from the SDK with [zoo models](https://docs.voxel51.com/model_zoo/models.html):\n**YOLO-World** for open-vocabulary detection, **SAM2** for masks."
  },
  {
   "cell_type": "code",
   "id": "de-cell-47",
   "metadata": {},
   "source": "# Open-vocabulary detection \u2014 pass the classes you care about at load time\nyolo_world = foz.load_zoo_model(\n    \"yolov8l-world-torch\",\n    classes=[\"car\", \"truck\", \"person\", \"traffic light\", \"stop sign\"],\n)\ndataset.apply_model(yolo_world, label_field=\"auto_labels\")",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "id": "de-cell-48",
   "metadata": {},
   "source": "# Pixel-accurate masks from the boxes above with SAM2\nsam2 = foz.load_zoo_model(\"segment-anything-2.1-hiera-base-plus-image-torch\")\ndataset.apply_model(sam2, label_field=\"auto_masks\", prompt_field=\"auto_labels\")",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-49",
   "metadata": {},
   "source": "## Stage 3 \u2014 Review: find label mistakes\n\nRank labels by estimated error so you review the exceptions, not everything."
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-50",
   "metadata": {},
   "source": "### Rank by mistakenness\n\nThe Brain's [mistakenness](https://docs.voxel51.com/brain.html#brain-label-mistakes)\nscore (`compute_mistakenness()`) uses model predictions to flag how likely each label\nis to be wrong."
  },
  {
   "cell_type": "code",
   "id": "de-cell-51",
   "metadata": {},
   "source": "fob.compute_mistakenness(dataset, \"predictions\", label_field=\"ground_truth\")\n\n# Triage queue: most-likely mistakes first\nneeds_review = dataset.sort_by(\"mistakenness\", reverse=True)\ndataset.save_view(\"needs_review\", needs_review, overwrite=True)\n\n# Objects most likely to be mislabeled\nworst = dataset.filter_labels(\"ground_truth\", F(\"mistakenness\") > 0.7)\nprint(worst.count(\"ground_truth.detections\"), \"suspect labels\")",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-52",
   "metadata": {},
   "source": "> Mistakenness covers detections and classifications. For semantic segmentation,\n> rank by per-sample mIoU from\n> [evaluate_segmentations()](https://docs.voxel51.com/user_guide/evaluation.html#evaluating-segmentations)."
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-53",
   "metadata": {},
   "source": "![Filtering to high-mistakenness samples to surface likely label errors](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_mistakenness.webp)"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-54",
   "metadata": {},
   "source": "### Embedding-based review\n\nIn the Embeddings panel, lasso outliers that sit far from their class cluster and\nsend just those samples to the review queue."
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-55",
   "metadata": {},
   "source": "### Compare against a reference model\n\nWhere predictions and labels disagree, one is likely wrong.\n[Evaluate](https://docs.voxel51.com/user_guide/evaluation.html#evaluating-detections-coco)\nand sort by unmatched ground truth; the Enterprise **AI Insights** panel automates\nthis."
  },
  {
   "cell_type": "code",
   "id": "de-cell-56",
   "metadata": {},
   "source": "# Audit ground truth against a trusted model: many false negatives on a sample\n# often means missing or incorrect labels there\nresults = dataset.evaluate_detections(\n    \"predictions\", gt_field=\"ground_truth\", eval_key=\"gt_audit\"\n)\naudit_view = dataset.sort_by(\"gt_audit_fn\", reverse=True)\nprint(audit_view.first().gt_audit_fn, \"unmatched ground-truth objects\")",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-57",
   "metadata": {},
   "source": "### Fix mistakes\n\nCorrect labels in the App, or tag suspects from the SDK and route them back through\nthe workflow."
  },
  {
   "cell_type": "code",
   "id": "de-cell-58",
   "metadata": {},
   "source": "# Tag suspect labels for rework\nhigh_mistakenness = dataset.filter_labels(\"ground_truth\", F(\"mistakenness\") > 0.7)\nfor s in high_mistakenness.iter_samples(autosave=True):\n    for det in s.ground_truth.detections:\n        if det.mistakenness and det.mistakenness > 0.7:\n            det.tags.append(\"needs_rework\")\n\nrework = dataset.select_labels(tags=\"needs_rework\")\nprint(rework.count(\"ground_truth.detections\"), \"labels queued for rework\")\n# rework.annotate(\"rework_round\", backend=\"cvat\", label_field=\"ground_truth\", ...)",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-59",
   "metadata": {},
   "source": "## Stage 4 \u2014 Train and evaluate: close the loop\n\nTrain on the verified set, evaluate by scenario, and feed the weakest slice back\ninto curation."
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-60",
   "metadata": {},
   "source": "### Export and train\n\nExport training-ready labels in\n[YOLOv5 format](https://docs.voxel51.com/user_guide/export_datasets.html#yolov5). In\nEnterprise, run training as a **delegated operation** on GPU."
  },
  {
   "cell_type": "code",
   "id": "de-cell-61",
   "metadata": {},
   "source": "train_view = dataset.load_saved_view(\"to_annotate\")\ntrain_view.export(\n    export_dir=\"/tmp/data_engine_yolo\",\n    dataset_type=fo.types.YOLOv5Dataset,\n    label_field=\"ground_truth\",\n)\n\n# Train with Ultralytics (uncomment to run):\n# from ultralytics import YOLO\n# model = YOLO(\"yolov8n.pt\")\n# model.train(data=\"/tmp/data_engine_yolo/dataset.yaml\", epochs=50)\n#\n# Then load predictions back in to evaluate:\n# dataset.apply_model(model, label_field=\"predictions\")",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-62",
   "metadata": {},
   "source": "### Evaluate by scenario\n\nCompute [COCO mAP](https://docs.voxel51.com/user_guide/evaluation.html#evaluating-detections-coco)\noverall and per scene type to find the weakest slice. The Enterprise\n**Model Evaluation panel** does this interactively."
  },
  {
   "cell_type": "code",
   "id": "de-cell-63",
   "metadata": {},
   "source": "results = dataset.evaluate_detections(\n    \"predictions\",\n    gt_field=\"ground_truth\",\n    eval_key=\"eval\",\n    method=\"coco\",\n    compute_mAP=True,\n)\nresults.print_report()\nprint(\"Overall mAP:\", round(results.mAP(), 3))\n\n# Scenario analysis: mAP per scene type to find the weakest slice\nfor scene in dataset.distinct(\"scene.label\"):\n    subset = dataset.match(F(\"scene.label\") == scene)\n    r = subset.evaluate_detections(\n        \"predictions\", gt_field=\"ground_truth\", method=\"coco\", compute_mAP=True\n    )\n    print(f\"{scene:>16}: mAP = {r.mAP():.3f}\")",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-64",
   "metadata": {},
   "source": "![The Model Evaluation panel: compare runs and slice performance by scenario](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_model_eval.webp)"
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-65",
   "metadata": {},
   "source": "### Close the loop\n\nMine more of the weakest scenario and save it as the next batch to annotate."
  },
  {
   "cell_type": "code",
   "id": "de-cell-66",
   "metadata": {},
   "source": "# Suppose evaluation showed \"an indoor scene\" is the weakest slice. Mine more\n# images like it from the embedding space for the next annotation round.\nnext_batch = dataset.sort_by_similarity(\n    \"an indoor scene\", k=50, brain_key=\"img_sim\"\n)\ndataset.save_view(\"next_to_annotate\", next_batch, overwrite=True)\nprint(\"Queued\", len(next_batch), \"samples for the next annotation round\")",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "de-cell-67",
   "metadata": {},
   "source": "## Summary\n\nThe loop, end to end:\n\n1. **Curate** \u2014 embeddings, uniqueness, similarity, and zero-shot tagging picked the subset to label.\n2. **Annotate** \u2014 an ontology fixed the schema; a multi-stage workflow moved samples through label \u2192 review \u2192 send-back; an agent and zoo models drafted the first pass.\n3. **Review** \u2014 mistakenness, embedding outliers, and model comparison surfaced bad labels to fix.\n4. **Evaluate** \u2014 scenario analysis named the weakest slice and fed it back into curation.\n\n### Next steps\n\n- [FiftyOne Annotation](https://voxel51.com/annotation)\n- [FiftyOne Brain](https://docs.voxel51.com/brain.html)\n- [Dataset Zoo](https://docs.voxel51.com/dataset_zoo/index.html)\n- [Evaluating models](https://docs.voxel51.com/user_guide/evaluation.html)"
  },
  {
   "cell_type": "code",
   "id": "de-cell-68",
   "metadata": {},
   "source": "session.freeze()  # screenshot the active App for sharing",
   "outputs": [],
   "execution_count": null
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}