{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Integrations, Plugins, and Model Evaluation\n",
    "\n",
    "In this notebook, we will explore **FiftyOne's integration capabilities**, **plugin system**, and **model evaluation tools**. \n",
    "This is particularly useful when working with external frameworks, custom plugins, and evaluating AI models.\n",
    "\n",
    "![integrations](https://cdn.voxel51.com/getting_started_manufacturing/notebook5/integrations.webp)\n",
    "\n",
    "## Learning Objectives:\n",
    "- Understand how FiftyOne integrates with external tools.\n",
    "- Learn about FiftyOne’s plugin system and how to configure it.\n",
    "- Use the **voxel51/evaluation** plugin to evaluate models.\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FiftyOne Integrations and Plugins\n",
    "\n",
    "FiftyOne provides a flexible architecture that enables **seamless integration** with external libraries, models, and cloud services. \n",
    "This is achieved through **integrations** and **plugins** that extend FiftyOne’s core functionalities.\n",
    "\n",
    "| Feature      | Description |\n",
    "|-------------|-------------|\n",
    "| **Integrations** | Connects FiftyOne with platforms like PyTorch, Ultralytics, CVAT, Lightning Flash, and  Albumentations. |\n",
    "|  | Enables easy dataset ingestion from sources like Hugging Face and Benchmark datasets |\n",
    "| **Plugins** | Allow users to **extend FiftyOne’s functionality**. |\n",
    "|  | Can be used for custom dataset visualizations, model evaluations, and interactive tools. |\n",
    "\n",
    "\n",
    "🔗 **Relevant Documentation:**  \n",
    "- [FiftyOne Integrations](https://voxel51.com/docs/fiftyone/integrations/index.html)  \n",
    "- [FiftyOne Plugins](https://voxel51.com/docs/fiftyone/plugins/index.html)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuring Plugins in FiftyOne\n",
    "\n",
    "To use plugins in FiftyOne, we need to enable and configure them properly.\n",
    "\n",
    "### Minimum Configuration Steps:\n",
    "1. Ensure FiftyOne is installed and up to date.\n",
    "2. Download plugind using `fiftyone` CLI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!fiftyone plugins download https://github.com/voxel51/fiftyone-plugins --plugin-names @voxel51/evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install ipywidgets anywidget"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up the dataset into Anomalib (custom folder) and train the model\n",
    "\n",
    "I want to clarify that Anomalib natively supports MVTec and you can use Datamodule callback for using MVTec AD dataset. However for the education purposes of this workshop we will manage MVTec Bottle subset as a custom folder in Anomalib. Selecting `OBJECT=bottle` in the next cell and adding normal and abnormal folders to the tree directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import os\n",
    "# from anomalib import TaskType\n",
    "# from pathlib import Path\n",
    "# from torchvision.transforms.v2 import Resize\n",
    "# import fiftyone as fo # base library and app\n",
    "# import fiftyone.utils.huggingface as fouh # Hugging Face integration\n",
    "# from fiftyone import ViewField as F # helper for defining views\n",
    "# from anomalib.data import Folder\n",
    "\n",
    "\n",
    "\n",
    "# dataset = fouh.load_from_hub(\"Voxel51/mvtec-ad\", persistent=True, overwrite=True)\n",
    "# #dataset = fo.load_dataset(\"Voxel51/mvtec-ad\") # Use this CLI if you already have the dataset \n",
    "#                                                # in your disk or if this is not the first time you run this notebook \n",
    "\n",
    "# # Try this for already loaded dataset\n",
    "# # dataset = fo.load_dataset('mvtec-ad-staging')\n",
    "# OBJECTS_LIST = [\n",
    "#     'pill',\n",
    "#     'zipper',\n",
    "#     'tile',\n",
    "#     'bottle',\n",
    "#     'transistor',\n",
    "#     'wood',\n",
    "#     'cable',\n",
    "#     'capsule',\n",
    "#     'carpet',\n",
    "#     'grid',\n",
    "#     'hazelnut',\n",
    "#     'leather',\n",
    "#     'metal_nut',\n",
    "#     'screw',\n",
    "#     'toothbrush'\n",
    "# ]\n",
    "# OBJECT = \"bottle\" ## object to train on\n",
    "\n",
    "# ROOT_DIR = Path(\"/tmp/mvtec_ad\") ## root directory to store data for anomalib\n",
    "# TASK = TaskType.SEGMENTATION ## task type for the model\n",
    "# IMAGE_SIZE = (256, 256) ## preprocess image size for uniformity\n",
    "\n",
    "# # training and inference:\n",
    "# # step-1 create data loader\n",
    "\n",
    "# def create_datamodule(object_type, transform=None):\n",
    "#     ## Build transform\n",
    "#     if transform is None:\n",
    "#         transform = Resize(IMAGE_SIZE, antialias=True)\n",
    "#     normal_data = dataset.match(F(\"category.label\") == object_type).match(\n",
    "#         F(\"split\") == \"train\"\n",
    "#     )\n",
    "#     abnormal_data = (\n",
    "#         dataset.match(F(\"category.label\") == object_type)\n",
    "#         .match(F(\"split\") == \"test\")\n",
    "#         .match(F(\"defect.label\") != \"good\")\n",
    "#     )\n",
    "#     normal_dir = ROOT_DIR / object_type / \"normal\"\n",
    "#     abnormal_dir = ROOT_DIR / object_type / \"abnormal\"\n",
    "#     mask_dir = ROOT_DIR / object_type / \"mask\"\n",
    "#     # create directories if they do not exist\n",
    "#     os.makedirs(normal_dir, exist_ok=True)\n",
    "#     os.makedirs(abnormal_dir, exist_ok=True)\n",
    "#     os.makedirs(mask_dir, exist_ok=True)\n",
    "#     if os.path.exists(str(normal_dir)):\n",
    "#         normal_data.export(\n",
    "#             export_dir=str(normal_dir),\n",
    "#             dataset_type=fo.types.ImageDirectory,\n",
    "#             export_media=\"symlink\",\n",
    "#         )\n",
    "#     for sample in abnormal_data.iter_samples():\n",
    "#         base_filename = sample.filename\n",
    "#         dir_name = os.path.dirname(sample.filepath).split(\"/\")[-1]\n",
    "#         new_filename = f\"{dir_name}_{base_filename}\"\n",
    "#         if not os.path.exists(str(abnormal_dir / new_filename)):\n",
    "#             os.symlink(sample.filepath, str(abnormal_dir / new_filename))\n",
    "#         if not os.path.exists(str(mask_dir / new_filename)):\n",
    "#             os.symlink(sample.defect_mask.mask_path, str(mask_dir / new_filename))\n",
    "#     datamodule = Folder(\n",
    "#         name=object_type,\n",
    "#         root=ROOT_DIR,\n",
    "#         normal_dir=normal_dir,\n",
    "#         abnormal_dir=abnormal_dir,\n",
    "#         mask_dir=mask_dir,\n",
    "#     )\n",
    "#     datamodule.setup()\n",
    "#     return datamodule\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from anomalib.engine import Engine\n",
    "\n",
    "# # train and save model to disk\n",
    "# def train_and_export_model(object_type, model, transform=None):\n",
    "#     engine = Engine()\n",
    "#     datamodule = create_datamodule(object_type, transform=transform)\n",
    "#     engine.train(model=model, datamodule=datamodule)\n",
    " \n",
    "#     return engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from anomalib.models import Padim, Patchcore\n",
    "\n",
    "# # Load and train padim\n",
    "# model = Padim()\n",
    "# engine = train_and_export_model(OBJECT, model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Evaluation in FiftyOne for anomaly detection in a bottle factory\n",
    "\n",
    "FiftyOne provides a built-in model evaluation system that supports various evaluation types, such as:\n",
    "- **Classification Evaluation** (Accuracy, Precision, Recall, F1-score)\n",
    "- **Object Detection Evaluation** (IoU, mAP)\n",
    "- **Segmentation Evaluation**\n",
    "\n",
    "We will use the [**voxel51/evaluation plugin**](https://github.com/voxel51/fiftyone-plugins/blob/main/plugins/evaluation/README.md) to evaluate our model’s performance.\n",
    "\n",
    "### Steps:\n",
    "1. Load a dataset with ground truth labels and predictions.\n",
    "2. Select an evaluation method.\n",
    "3. Run the evaluation and analyze the results.\n",
    "\n",
    "**Relevant Documentation:** [Evaluating Models in FiftyOne](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the data into FiftyOne"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fiftyone as fo # base library and app\n",
    "from fiftyone import ViewField as F # helper for defining views\n",
    "import fiftyone.utils.huggingface as fouh # Hugging Face integration\n",
    "\n",
    "# Load the dataset\n",
    "dataset_ = fouh.load_from_hub(\"Voxel51/mvtec-ad\", persistent=True, overwrite=True)\n",
    "#dataset = fo.load_dataset(\"Voxel51/mvtec-ad\") # Use this CLI if you already have the dataset \n",
    "                                               # in your disk or if this is not the first time you run this notebook \n",
    "\n",
    "# Define the new dataset name\n",
    "dataset_name = \"mvtec-ad_5\"\n",
    "\n",
    "# Check if the dataset exists\n",
    "if dataset_name in fo.list_datasets():\n",
    "    print(f\"Dataset '{dataset_name}' exists. Loading...\")\n",
    "    dataset = fo.load_dataset(dataset_name)\n",
    "else:\n",
    "    print(f\"Dataset '{dataset_name}' does not exist. Creating a new one...\")\n",
    "    # Clone the dataset with a new name and make it persistent\n",
    "    dataset = dataset_.clone(dataset_name, persistent=True)\n",
    "\n",
    "# Define the new dataset name for split set\n",
    "dataset_name_split = \"mvtec-bottle_2\"\n",
    "\n",
    "if dataset_name_split in fo.list_datasets():\n",
    "    print(f\"Dataset '{dataset_name_split}' exists. Loading...\")\n",
    "    dataset = fo.load_dataset(dataset_name_split)\n",
    "else:\n",
    "    print(f\"Dataset '{dataset_name_split}' does not exist. Creating a new one...\")\n",
    "    ## get the test split of the dataset\n",
    "    test_split = dataset.match(F(\"category.label\") == 'bottle')\n",
    "\n",
    "    # Clone the dataset into a new one called \"mvtec_bottle\"\n",
    "    dataset = test_split.clone(\"mvtec-bottle_2\", persistent=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(dataset.last())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check if there are previous evaluations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(dataset.list_evaluations())\n",
    "#dataset.delete_evaluation('padim_eval')\n",
    "#if \"eval_key\" in dataset.get_field_schema():\n",
    "#    dataset.delete_sample_field(\"eval_key\")\n",
    "\n",
    "#if \"padim_eval\"in dataset.get_field_schema():\n",
    "#    dataset.delete_sample_field(\"padim_eval\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running Inference with Anomalib's Engine\n",
    "\n",
    "This code demonstrates how to set up and run inference using Anomalib's `Engine` with a **PaDiM** model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from anomalib.engine import Engine\n",
    "from anomalib.models import Padim\n",
    "\n",
    "#Set up my model with Anomalib\n",
    "\n",
    "engine = Engine(accelerator=\"cpu\") \n",
    "model = Padim()\n",
    "\n",
    "# Specify the path to your trained model checkpoint\n",
    "ckpt_path_model = \"/path/to/your/model.ckpt\"\n",
    "\n",
    "# Specify the path to the image you want to run inference on\n",
    "image_path = \"/path/to/your/image.png\"\n",
    "\n",
    "# Run inference on the specified image\n",
    "predictions = engine.predict(model=model, data_path=image_path, ckpt_path=ckpt_path_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing Anomaly Predictions in Anomalib\n",
    "\n",
    "This code processes and visualizes the anomaly detection results from Anomalib. It extracts prediction details, displays anomaly maps, and overlays them on the input image.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from PIL import Image\n",
    "from matplotlib import pyplot as plt\n",
    "from anomalib.utils.post_processing import superimpose_anomaly_map\n",
    "\n",
    "# Extract the first prediction result\n",
    "prediction = predictions[0]\n",
    "\n",
    "# Print key prediction details\n",
    "print(\n",
    "    f\"Image Shape: {prediction.image.shape},\\n\"\n",
    "    f\"Anomaly Map Shape: {prediction.anomaly_map.shape}, \\n\"\n",
    "    f\"Prediction Score: {prediction.pred_score}, \\n\"\n",
    "    f\"Predicted Mask Shape: {prediction.pred_mask.shape}\",\n",
    ")\n",
    "\n",
    "# Extract anomaly score and label\n",
    "pred_score = prediction.pred_score[0]\n",
    "pred_labels = prediction.pred_label[0]\n",
    "print(pred_score, pred_labels)\n",
    "\n",
    "# Load and resize the input image\n",
    "image_path = prediction.image_path[0]\n",
    "image_size = prediction.image.shape[-2:]\n",
    "image = np.array(Image.open(image_path).resize(image_size))\n",
    "\n",
    "# Extract the predicted anomaly mask\n",
    "pred_masks = prediction.pred_mask[0].squeeze().cpu().numpy()\n",
    "plt.imshow(pred_masks)\n",
    "plt.title(\"Predicted Anomaly Mask\")\n",
    "\n",
    "# Extract and display the anomaly map\n",
    "pred_anomaly_map = prediction.anomaly_map[0].squeeze().cpu().numpy()\n",
    "plt.imshow(pred_anomaly_map)\n",
    "plt.title(\"Anomaly Map\")\n",
    "\n",
    "# Superimpose anomaly map on the input image for visualization\n",
    "heat_map = superimpose_anomaly_map(anomaly_map=pred_anomaly_map, image=image, normalize=True)\n",
    "plt.imshow(heat_map)\n",
    "plt.title(\"Superimposed Anomaly Map\")\n",
    "\n",
    "score_value = prediction.pred_score[0, 0].item()\n",
    "print(score_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running Anomaly Detection Inference and Grouping Results in FiftyOne\n",
    "\n",
    "This function, `run_inference`, performs **anomaly detection** on a collection of images using a pre-trained model in **Anomalib**. The results, including anomaly scores, masks, and heatmaps, are stored in [**FiftyOne's grouped dataset**](https://docs.voxel51.com/user_guide/groups.html?highlight=group) format for better visualization and analysis.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import cv2\n",
    "\n",
    "def run_inference(sample_collection, key, engine, threshold=0.5, ckpt_path=None):\n",
    "    # (Optional) if your engine requires a model argument, ensure it is defined or passed in.\n",
    "    # For this example, we assume your engine has the model already loaded.\n",
    "    \n",
    "    # Directory to save visualization images\n",
    "    output_dir = \"temp_predictions\"\n",
    "    os.makedirs(output_dir, exist_ok=True)\n",
    "    \n",
    "    # Define the new dataset name\n",
    "    gdataset_name = \"anomaly_predictions_grouped\"\n",
    "\n",
    "    \n",
    "    # Check if the dataset exists\n",
    "    if gdataset_name in fo.list_datasets():\n",
    "        print(f\"Dataset '{gdataset_name}' exists. Loading...\")\n",
    "        grouped_dataset = fo.load_dataset(gdataset_name)\n",
    "        return grouped_dataset\n",
    "        \n",
    "    else:\n",
    "        print(f\"Dataset '{gdataset_name}' does not exist. Creating a new one...\")\n",
    "        # Create a new grouped dataset\n",
    "        grouped_dataset = fo.Dataset(gdataset_name, overwrite=True, persistent=True)\n",
    "    \n",
    "    grouped_dataset.add_group_field(\"group\", default=\"original\")\n",
    "    grouped_samples = []  # Will hold all slices (samples) of all groups\n",
    "    \n",
    "    # Iterate over samples in the input collection\n",
    "    for sample in sample_collection.iter_samples(autosave=True, progress=True):\n",
    "        # Run inference on the sample's filepath\n",
    "        predictions = engine.predict(model=engine.model, data_path=sample.filepath, ckpt_path=ckpt_path)\n",
    "        prediction = predictions[0]\n",
    "        \n",
    "        # --- Process Prediction Outputs ---\n",
    "        # Extract the prediction score and label\n",
    "        conf = prediction.pred_score[0, 0].item()\n",
    "        anomaly = \"normal\" if conf < threshold else \"abnormal\"\n",
    "        \n",
    "        # Load original image using the prediction's image path and resize to expected dimensions\n",
    "        image_path = prediction.image_path[0]\n",
    "        image_size = prediction.image.shape[-2:]\n",
    "        image = np.array(Image.open(image_path).resize(image_size))\n",
    "        \n",
    "        # Convert predicted mask and anomaly map to NumPy arrays\n",
    "        pred_masks = prediction.pred_mask[0].squeeze().cpu().numpy()\n",
    "        pred_anomaly_map = prediction.anomaly_map[0].squeeze().cpu().numpy()\n",
    "        # Generate heat map overlay using the provided utility\n",
    "        heat_map = superimpose_anomaly_map(anomaly_map=pred_anomaly_map, image=image, normalize=True)\n",
    "        \n",
    "        # --- Save Metadata on the Original Sample ---\n",
    "        sample[f\"pred_anomaly_score_{key}\"] = conf\n",
    "        sample[f\"pred_anomaly_classification_{key}\"] = fo.Classification(label=anomaly)\n",
    "        sample[f\"pred_anomaly_map{key}\"] = fo.Heatmap(map=pred_anomaly_map) \n",
    "        sample[f\"pred_heat_map{key}\"] = fo.Heatmap(map=heat_map) \n",
    "        sample[f\"pred_anomaly_mask{key}\"] = fo.Segmentation(mask=pred_masks) \n",
    "\n",
    "        sample.save()\n",
    "                \n",
    "        # --- Create a Group for This Prediction ---\n",
    "        # Use the sample's unique id to build a group id\n",
    "        group = fo.Group()\n",
    "        group_id = f\"sample_{sample.id}\"\n",
    "        \n",
    "        # ----- Original Image Slice -----\n",
    "        # Ensure image is in uint8 (if not already)\n",
    "        orig = image.copy()\n",
    "        if orig.dtype != np.uint8:\n",
    "            orig = (orig * 255).astype(np.uint8)\n",
    "        orig_path = os.path.join(output_dir, f\"{group_id}_original.png\")\n",
    "        # OpenCV expects BGR, so convert RGB to BGR\n",
    "        cv2.imwrite(orig_path, cv2.cvtColor(orig, cv2.COLOR_RGB2BGR))\n",
    "        sample_orig = fo.Sample(filepath=orig_path, group=group.element(\"original\"))\n",
    "        sample_orig[\"pred_score\"] = conf\n",
    "        sample_orig[\"pred_label\"] = anomaly\n",
    "        # --- Save Metadata on the Original Sample ---\n",
    "        sample_orig[f\"pred_anomaly_score_{key}\"] = conf\n",
    "        sample_orig[f\"pred_anomaly_classification_{key}\"] = fo.Classification(label=anomaly)\n",
    "        sample_orig[f\"pred_anomaly_map{key}\"] = fo.Heatmap(map=pred_anomaly_map) \n",
    "        sample_orig[f\"pred_heat_map{key}\"] = fo.Heatmap(map=heat_map) \n",
    "        sample_orig[f\"pred_anomaly_mask{key}\"] = fo.Segmentation(mask=pred_masks) \n",
    "\n",
    "        sample.save()\n",
    "        \n",
    "        # ----- Heat Map Slice -----\n",
    "        heat = heat_map.copy()\n",
    "        if heat.dtype != np.uint8:\n",
    "            heat = (heat * 255).astype(np.uint8)\n",
    "        heat_path = os.path.join(output_dir, f\"{group_id}_heat.png\")\n",
    "        cv2.imwrite(heat_path, heat)\n",
    "        sample_heat = fo.Sample(filepath=heat_path, group=group.element(\"heat_map\"))\n",
    "        \n",
    "        # ----- Predicted Mask Slice -----\n",
    "        mask = pred_masks.copy()\n",
    "        if mask.dtype != np.uint8:\n",
    "            mask = (mask * 255).astype(np.uint8)\n",
    "        mask_path = os.path.join(output_dir, f\"{group_id}_mask.png\")\n",
    "        cv2.imwrite(mask_path, mask)\n",
    "        sample_mask = fo.Sample(filepath=mask_path, group=group.element(\"pred_mask\"))\n",
    "        \n",
    "        # ----- Anomaly Map Slice -----\n",
    "        anom = pred_anomaly_map.copy()\n",
    "        if anom.dtype != np.uint8:\n",
    "            anom = (anom * 255).astype(np.uint8)\n",
    "        anom_path = os.path.join(output_dir, f\"{group_id}_anomaly.png\")\n",
    "        cv2.imwrite(anom_path, anom)\n",
    "        sample_anom = fo.Sample(filepath=anom_path, group=group.element(\"anomaly_map\"))\n",
    "        \n",
    "        # Add all four slices for this prediction to the list\n",
    "        grouped_samples.extend([sample_orig, sample_heat, sample_mask, sample_anom])\n",
    "    \n",
    "    # Add all grouped samples to the grouped dataset\n",
    "    grouped_dataset.add_samples(grouped_samples)\n",
    "    \n",
    "    # Optionally launch the FiftyOne App to inspect the grouped dataset\n",
    "    \n",
    "    return grouped_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Function Overview**\n",
    "```python\n",
    "def run_inference(sample_collection, key, engine, threshold=0.5, ckpt_path=None):\n",
    "```\n",
    "- **sample_collection**: The input FiftyOne dataset containing images for inference.\n",
    "- **key**: Identifier for Anomalib model.\n",
    "- **engine**: The inference engine (e.g., Anomalib's Engine) with the model loaded.\n",
    "- **threshold**: Score threshold for classifying images as \"normal\" or \"abnormal\".\n",
    "- **ckpt_path:** Path to the trained model's checkpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped_dataset = run_inference(dataset, \"padim\", engine, threshold=0.5, ckpt_path=ckpt_path_model)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reloading and Visualizing the Grouped Dataset in FiftyOne\n",
    "\n",
    "After running inference, we reload the grouped dataset to ensure all predictions are updated and then visualize the results in the **FiftyOne App**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped_dataset.reload()\n",
    "print(grouped_dataset.last())\n",
    "\n",
    "session = fo.launch_app(grouped_dataset, port=5155, auto=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![grouped_prediction](https://cdn.voxel51.com/getting_started_manufacturing/notebook5/grouped_prediction.webp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating Anomaly Classification with the Model Evaluation Plugin in FiftyOne\n",
    "\n",
    "When using the **Model Evaluation Plugin** in FiftyOne, we can evaluate model predictions using either:\n",
    "1. **The Model Evaluation Panel in the FiftyOne App** – Interactive UI-based evaluation.\n",
    "2. **The FiftyOne SDK** – Programmatic evaluation via Python scripts.\n",
    "\n",
    "**Which one to use?**  \n",
    "- If you prefer an **interactive visual analysis**, the **FiftyOne App** provides an intuitive panel for evaluating models.  \n",
    "- If you need **automated and reproducible results**, the **FiftyOne SDK** allows for batch evaluation and detailed reporting.\n",
    "\n",
    "   \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "for sample in dataset.iter_samples(progress=True):\n",
    "    # If the sample already has \"defect2\", skip it\n",
    "    # if \"defect\" in sample and sample[\"defect2\"] is not None:\n",
    "    #     continue\n",
    "\n",
    "    # Otherwise, create \"defect2\"\n",
    "    if sample[\"defect\"].label == \"good\":\n",
    "        sample[\"defect_mask\"] = fo.Segmentation(\n",
    "            mask=np.zeros_like(sample[\"pred_anomaly_maskpadim\"].mask)\n",
    "        )\n",
    "        sample[\"defect2\"] = fo.Classification(label=\"normal\")\n",
    "    else:\n",
    "        sample[\"defect2\"] = fo.Classification(label=\"abnormal\")\n",
    "    \n",
    "    sample.save()\n",
    "\n",
    "eval_key = \"padim_eval\"\n",
    "\n",
    "# Now evaluate on the \"defect2\" field\n",
    "eval_classif_padim = dataset.evaluate_classifications(\n",
    "    \"pred_anomaly_classification_padim\",\n",
    "    gt_field=\"defect2\",\n",
    "    method=\"binary\", #method is important to see data in the FO app\n",
    "    classes=[\"normal\", \"abnormal\"],\n",
    "    eval_key=eval_key,  # <--- store this run under \"padim_eval\"\n",
    ")\n",
    "eval_classif_padim.print_report()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(dataset)\n",
    "print(dataset.last())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot a confusion matrix\n",
    "plot = eval_classif_padim.plot_confusion_matrix(backend=\"matplotlib\")\n",
    "plot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot a PR curve\n",
    "plot = eval_classif_padim.plot_pr_curve(backend=\"matplotlib\")\n",
    "plot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Visualizing Evaluation Results\n",
    "\n",
    "After running the evaluation, we can inspect the results using the FiftyOne App:\n",
    "\n",
    "In the App, you can:\n",
    "\n",
    "Filter samples by correct or incorrect predictions.\n",
    "View IoU scores, confusion matrices, and detection overlaps.\n",
    "Generate customized evaluation reports.\n",
    "\n",
    "Relevant Documentation: [Interactive Model Evaluation in FiftyOne](https://github.com/voxel51/fiftyone-plugins/blob/main/plugins/evaluation/README.md)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = fo.launch_app(dataset, port=5155, auto=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![visualizate_evaluation](https://cdn.voxel51.com/getting_started_manufacturing/notebook5/visualizate_evaluation.webp)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Next Steps:\n",
    "Try using other plugins or **create your own** to extend FiftyOne’s capabilities! 🚀\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py311_anomalib200b3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}