{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", " Try in Google Colab\n", " \n", " \n", " \n", " \n", " Share via nbviewer\n", " \n", " \n", " \n", " \n", " View on GitHub\n", " \n", " \n", " \n", " \n", " Download notebook\n", " \n", "
\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Exploring Image Hardness\n", "\n", "Say you have a repository of data with millions of unlabeled images in it. You managed to label a subset of data and trained an image classification model on it, but it's not performing as well as you hope. **How do you decide \n", "which new samples to annotate and add to your training set?**\n", "\n", "You could just randomly select new samples to annotate, but there is a better way. Hard sample mining is a tried and true method to distill a large amount of raw unlabeled data into smaller high quality labeled datasets.\n", "\n", "*A hard sample is one that is difficult for your machine learning (ML) model to correctly predict the label of.* \n", "\n", "In an image classification dataset, a hard sample could be anything from a cat that looks like a dog to a blurry resolution image. If you expect your model to perform well on these hard samples, then you may need to \"mine\" more examples of these hard samples to add to your training dataset. Exposing your model to more hard samples during training will allow it to perform better on those types of samples later on.\n", "\n", "\n", "Hard samples are useful for more than just training data, they are also necessary to include in your test set. If your test data is composed primarily of easy samples, then your [performance will soon reach an upper bound causing progress to stagnate](https://www.sciencedirect.com/science/article/abs/pii/S0925231219316984). Adding hard samples to a test set will give you a better idea of how models perform in harder edge cases and can provide more insight into which models are more reliable.\n", "\n", "## Overview:\n", "\n", "In this walkthrough, we explore how [FiftyOne’s image hardness tool](https://voxel51.com/docs/fiftyone/user_guide/brain.html) can be used to analyze and improve datasets.\n", "\n", "We’ll cover the following concepts:\n", "\n", "* Loading a dataset from the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_zoo/index.html)\n", "\n", "* Applying [FiftyOne’s sample hardness algorithm](https://voxel51.com/docs/fiftyone/user_guide/brain.html) to your dataset\n", "\n", "* Launching the [FiftyOne App and visualizing/exploring your data](https://voxel51.com/docs/fiftyone/user_guide/app.html)\n", "\n", "* Identifying the hardest samples in your dataset" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install fiftyone" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install torch torchvision" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load your data\n", "\n", "For this example, we will be using the test split of the image classification dataset, [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html). This dataset contains 10,000 test images labeled across 10 different classes. This is one of the dozens of datasets in the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_zoo/index.html), so we can easily load it up." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import fiftyone as fo" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import fiftyone.zoo as foz" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Split 'test' already downloaded\n", "Loading 'cifar10' split 'test'\n", " 100% |████████████████████████████████████████████████████████████| 10000/10000 [9.2s elapsed, 0s remaining, 1.1K samples/s] \n", "Dataset 'cifar10-test' created\n" ] } ], "source": [ "dataset = foz.load_zoo_dataset(\"cifar10\", split=\"test\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Note**: You can also [load your own dataset into FiftyOne](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/index.html). It supports labels for many computer vision tasks including [classification, detection, segmentation, keypoints, and more](https://voxel51.com/docs/fiftyone/user_guide/using_datasets.html#labels)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Add logits\n", "In order to calculate hardness on images in FiftyOne, you first need to use a model to compute logits for those images. You can use any model you want, but ideally, it would be one trained similar data and on the same task you will be using these new images for.\n", "\n", "In this example, we will be using code from the [PyTorch CIFAR-10 repository](https://github.com/huyvnphan/PyTorch_CIFAR10/tree/v2.1), namely the pretrained ResNet50 classifier.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Download the software\n", "!git clone --depth 1 --branch v2.1 https://github.com/huyvnphan/PyTorch_CIFAR10.git\n", "\n", "# Download the pretrained model (90MB)\n", "!eta gdrive download --public \\\n", " 1dGfpeFK_QG0kV-U6QDHMX2EOGXPqaNzu \\\n", " PyTorch_CIFAR10/cifar10_models/state_dicts/resnet50.pt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "You can easily add a classification field with [logits to your samples in a FiftyOne dataset.](https://voxel51.com/docs/fiftyone/recipes/model_inference.html?highlight=logits)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "import sys\n", "\n", "import numpy as np\n", "import torch\n", "import torchvision\n", "from torch.utils.data import DataLoader\n", "\n", "import fiftyone.utils.torch as fout\n", "\n", "sys.path.insert(1, \"PyTorch_CIFAR10\")\n", "from cifar10_models import resnet50\n", "\n", "\n", "# Set up a data loader in accordance to PyTorch CIFAR10\n", "def make_cifar10_data_loader(image_paths, sample_ids, batch_size):\n", " mean = [0.4914, 0.4822, 0.4465]\n", " std = [0.2023, 0.1994, 0.2010]\n", " transforms = torchvision.transforms.Compose(\n", " [\n", " torchvision.transforms.ToTensor(),\n", " torchvision.transforms.Normalize(mean, std),\n", " ]\n", " )\n", " dataset = fout.TorchImageDataset(\n", " image_paths, sample_ids=sample_ids, transform=transforms\n", " )\n", " return DataLoader(dataset, batch_size=batch_size, num_workers=4)\n", "\n", "\n", "# Run inference on the model to generate predictions and logits\n", "def predict(model, imgs):\n", " logits = model(imgs).detach().cpu().numpy()\n", " predictions = np.argmax(logits, axis=1)\n", " odds = np.exp(logits)\n", " confidences = np.max(odds, axis=1) / np.sum(odds, axis=1)\n", " return predictions, confidences, logits" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "#\n", "# Load a pretrained model\n", "#\n", "# Model performance numbers are available at:\n", "# https://github.com/huyvnphan/PyTorch_CIFAR10\n", "#\n", "\n", "model = resnet50(pretrained=True)\n", "model_name = \"resnet50\"\n", "\n", "view = dataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Note**: If you want this notebook to run faster, select a subset of samples from the dataset" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# Uncomment the lines below to select a random subset of 1000 samples\n", "\n", "# num_samples = 10000\n", "# view = dataset.take(num_samples, seed=51)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "batch_size = 20\n", "\n", "# Get the list of classes from the dataset information\n", "classes = dataset.info[\"classes\"]\n", "\n", "image_paths, sample_ids = zip(\n", " *[(s.filepath, s.id) for s in view.select_fields([\"filepath\", \"id\"])]\n", ")\n", "\n", "# Create a PyTorch data loader\n", "data_loader = make_cifar10_data_loader(image_paths, sample_ids, batch_size)\n", "\n", "#\n", "# Perform prediction and store results in dataset\n", "#\n", "\n", "for imgs, sample_ids in data_loader:\n", " predictions, confidences, logits_ = predict(model, imgs)\n", "\n", " # Add predictions to your FiftyOne dataset\n", " for sample_id, prediction, confidence, logits in zip(sample_ids, predictions, confidences, logits_):\n", " sample = dataset[sample_id]\n", " sample.tags.append(\"processed\")\n", " sample[model_name] = fo.Classification(\n", " label=classes[prediction], logits=logits, confidence=confidence\n", " )\n", " sample.save()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "processed_view = dataset.match_tags([\"processed\"])" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Dataset: cifar10-test\n", "Media type: image\n", "Num samples: 10000\n", "Tags: ['processed', 'test']\n", "Sample fields:\n", " filepath: fiftyone.core.fields.StringField\n", " tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n", " metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n", " ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)\n", " resnet50: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)\n", "View stages:\n", " 1. MatchTags(tag=['processed'])" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "processed_view" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can use the [FiftyOne App](https://voxel51.com/docs/fiftyone/user_guide/app.html) to take a look at this dataset." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "session = fo.launch_app(view=processed_view)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Compute hardness\n", "\n", "[The FiftyOne Brain](https://voxel51.com/docs/fiftyone/user_guide/brain.html) contains various useful methods that can provide insights into your data. You can compute the uniqueness of your data, the hardest samples, as well as annotation mistakes. These are all different ways to generate scalar metrics on your dataset that will let you better understand the quality of existing data as well as select help high-quality new samples of data.\n", "\n", "Once you have loaded your dataset and added logits to your samples, you calculate hardness in one line of code. The hardness algorithm is closed-source, but the basic idea is to leverage the relative uncertainty of the model's predictions to assign a scalar hardness value to each sample." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "import fiftyone.brain as fob" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Computing hardness...\n", " 100% |████████████████████████████████████████████████████████████| 10000/10000 [24.1s elapsed, 0s remaining, 409.4 samples/s] \n", "Hardness computation complete\n" ] } ], "source": [ "fob.compute_hardness(processed_view, label_field=model_name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Explore and identify the hardest samples\n", "\n", "You can visualize your dataset and explore the samples with the highest and lowest hardness scores with the FiftyOne App." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "scrolled": false }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "session.view = processed_view.sort_by(\"hardness\", reverse=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "While this example is using small images from CIFAR-10, FiftyOne also works with high-resolution images and videos.\n", "\n", "We can write some queries to dig a bit deeper into these hardness calculations and how they relate to other aspects of the data. For example, we can see the distribution of hardness on correct and incorrect predictions of the model separately." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "from fiftyone import ViewField as F" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset: cifar10-test\n", "Media type: image\n", "Num samples: 8183\n", "Tags: ['processed', 'test']\n", "Sample fields:\n", " filepath: fiftyone.core.fields.StringField\n", " tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n", " metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n", " ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)\n", " resnet50: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)\n", " hardness: fiftyone.core.fields.FloatField\n", "View stages:\n", " 1. MatchTags(tag=['processed'])\n", " 2. Match(filter={'$expr': {'$eq': [...]}})\n", "Dataset: cifar10-test\n", "Media type: image\n", "Num samples: 1817\n", "Tags: ['processed', 'test']\n", "Sample fields:\n", " filepath: fiftyone.core.fields.StringField\n", " tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n", " metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n", " ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)\n", " resnet50: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)\n", " hardness: fiftyone.core.fields.FloatField\n", "View stages:\n", " 1. MatchTags(tag=['processed'])\n", " 2. Match(filter={'$expr': {'$ne': [...]}})\n" ] } ], "source": [ "# Correct Preds\n", "correct_view = processed_view.match(F(\"ground_truth.label\") == F(model_name+\".label\"))\n", "\n", "# Incorrect Preds\n", "incorrect_view = processed_view.match(F(\"ground_truth.label\") != F(model_name+\".label\"))\n", "\n", "print(correct_view)\n", "print(incorrect_view)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Incorrect predictions avg hardness: 0.898047\n", "Correct predictions avg hardness: 0.518320\n", "Total avg hardness: 0.587316\n" ] } ], "source": [ "print(\"Incorrect predictions avg hardness: %f\" % (incorrect_view.sum(\"hardness\")/incorrect_view.count()))\n", "print(\"Correct predictions avg hardness: %f\" % (correct_view.sum(\"hardness\")/correct_view.count()))\n", "print(\"Total avg hardness: %f\" % (processed_view.sum(\"hardness\")/processed_view.count()))" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "incorrect_view.clone_sample_field(\"hardness\", \"hardness_incorrect\")\n", "correct_view.clone_sample_field(\"hardness\", \"hardness_correct\")" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "for sample in processed_view:\n", " if sample.id in correct_view:\n", " sample[\"prediction\"] = fo.Classification(label=\"correct\")\n", " else:\n", " sample[\"prediction\"] = fo.Classification(label=\"incorrect\")\n", " sample.save()" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "session.view = processed_view" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you might expect, the figure above shows that the distribution of hardness for correct predictions skews towards lower hardness values while incorrect predictions are spread more evenly at high hardness values. This indicates that samples that the model predicts incorrectly tend to be harder samples. Thus, adding harder samples to the training set should improve model performance.\n", "\n", "We can also see how the hardness of samples is distributed across different classes." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Average classwise hardness\n", "\n", "cat: 0.694020\n", "dog: 0.624073\n", "deer: 0.591152\n", "bird: 0.584566\n", "truck: 0.581653\n", "frog: 0.581470\n", "airplane: 0.576446\n", "horse: 0.565170\n", "ship: 0.542408\n", "automobile: 0.532203\n" ] } ], "source": [ "cls_hardness = []\n", "\n", "for label in processed_view.distinct(\"ground_truth.label\"):\n", " label_view = processed_view.match(F(\"ground_truth.label\")==label)\n", " avg_hardness = label_view.sum(\"hardness\")/label_view.count()\n", " \n", " num_correct = correct_view.match(F(\"ground_truth.label\")==label).count()\n", " accuracy = num_correct/label_view.count()\n", " \n", " cls_hardness.append([avg_hardness, label, accuracy])\n", "\n", "print(\"Average classwise hardness\\n\")\n", "for avg_hardness, label, _ in sorted(cls_hardness, reverse=True):\n", " print(\"%s: %f\" % (label, avg_hardness))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It seems that cat and dog tend to be the hardest classes so it would be worthwhile adding more examples of these before other classes." ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEGCAYAAACtqQjWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAu3ElEQVR4nO3de3hU1b3/8feXCAKiBEu0cg893CHcAqKAUKiCVhFUBEWEWrFIUUt78og/bQ966lN6sFahKF5BqYiFakSxxSLQogKSSOQqcjFHCJ6KQBAhKJfv74/ZiZOQywRmMoF8Xs8zT/Zee6813z0M+Wavtfde5u6IiIhEQ7V4ByAiImcOJRUREYkaJRUREYkaJRUREYkaJRUREYmas+IdQEWoX7++N2vWLN5hiIicVjIzM79096Ty1KkSSaVZs2ZkZGTEOwwRkdOKmf1veeuo+0tERKJGSUUqvdzcXJ544omotLVs2TKuvvrqqLQlIidSUpFKr6SkcvTo0ThEIyKlUVKRSm/ixIls27aNTp060a1bN3r37s2gQYNo27Yt2dnZtG/fvmDfRx55hEmTJgGwdetWfvSjH9GxY0e6dOnCtm3bCrW7evVqOnfufEK5iJy8KjFQL6e3yZMns379erKysli2bBk//vGPWb9+PcnJyWRnZ5dYb8SIEUycOJEhQ4Zw+PBhjh8/zo4dOwB4//33ueuuu3j99ddp0qRJBR2JyJlPZyoxlp6ezsaNG2P6HkX/Wg93++23F7x/s2bN+PLLL2May6lIX5NDz8lLSJ64kJ6Tl5C+JqfY/bp3705ycnKpbR04cICcnByGDBkCQM2aNalduzYAmzZt4o477uCNN95QQhGJMiWVGKuIpFKaZ599lrZt28bt/SOVviaH+15dR05uHg7k5OZx36vrik0s55xzTsHyWWedxfHjxwvWDx8+XOZ7XXTRRdSsWZM1a9ZEJXYR+Y6SykkYPHgwXbt2pV27djz99NMA1KlTp2D7/PnzGT16NO+//z4LFiwgLS2NTp06sW3bNrKysujRowcpKSkMGTKEffv2AdC3b18mTJhAamoqbdq0YfXq1Vx33XW0aNGCBx54oKDtRx99lPbt29O+fXsee+yxgvKjR48yYsQI2rRpww033MChQ4cK2i3uHp0///nPdO/enU6dOvGzn/2MY8eOxeKjitiURZvJO1I4hrwjx5iyaDPnnnsuBw4cKLbehRdeyBdffMGePXv45ptvePPNNwE499xzadSoEenp6QB88803BZ9JYmIiCxcu5L777mPZsmUxOyaRqkhJ5SQ8//zzZGZmkpGRwdSpU9mzZ0+x+1166aUMGjSIKVOmkJWVxQ9+8ANuvfVWfv/737N27Vo6dOjAgw8+WLB/jRo1yMjIYOzYsVx77bVMnz6d9evXM2vWLPbs2UNmZiYzZ85k1apVrFy5kmeeeabgr+3Nmzczbtw4Nm3axHnnnVfqJbibNm3ilVde4b333iMrK4uEhAReeuml6H5I5bQrN6/E8u9973v07NmT9u3bk5aWVmh79erV+c1vfkP37t25/PLLad26dcG22bNnM3XqVFJSUrj00kv5v//7v4JtF154IW+++SY///nPWbVqVWwOSqQK0kB9CdLX5DBl0WZ25ebRILEWaQNaMbhzQwCmTp3Ka6+9BsCOHTvYsmVLRG3u37+f3Nxc+vTpA8CoUaMYOnRowfZBgwYB0KFDB9q1a8dFF10EQPPmzdmxYwfvvvsuQ4YMKej+ue6661i+fDmDBg2icePG9OzZE4BbbrmFqVOn8p//+Z/FxvHOO++QmZlJt27dAMjLy+OCCy4o1+cTbQ0Sa5FTTGJpkFgLgDlz5pRY9+677+buu+8+obxFixYsWbKkUFnz5s3p27cvAE2aNGHDhg2nELWIFKWkUoz8/v387pj8/n2AxP1bWLx4MStWrKB27dr07duXw4cPY2YF9SPp1y/O2WefDUC1atUKlvPXy7onI/z9i1sP5+6MGjWK3/3udycVZyykDWhV6DMHqFU9gbQBreIYlYiUl7q/ilFa//7+/fupV68etWvX5uOPP2blypVAqDtl06ZNHD9+vOAsBig0HlC3bl3q1avH8uXLgVD3TP5ZSyR69+5Neno6hw4d4uDBg7z22mv07t0bgM8++4wVK1YAob/qe/XqVWI7/fv3Z/78+XzxxRcA7N27l//933I/4ieqBnduyO+u60DDxFoY0DCxFr+7rkPB2aGInB50plKM0vr3Bw4cyIwZM2jTpg2tWrWiR48eQOheiquvvpqkpCRSU1P5+uuvARg+fDhjxoxh6tSpzJ8/nxdeeIGxY8dy6NAhmjdvzsyZMyOOq0uXLowePZru3bsDocuFO3fuTHZ2Nq1atWL69OncdttttG3bljvvvLPEdtq2bctvf/tbrrjiCo4fP0716tWZPn06TZs2jTiWWBjcuaGSiMhpztw93jHEXGpqqpfnKcU9Jy8ptn+/YWIt3pvYL5qhiYhUWmaW6e6p5amj7q9ipA1oRa3qCYXKKrJ/v6SbFBcsWMDkyZMrJAYRkZOh7q9i5HfBlHT1V7wMGjSo4AoxEZHKSEmlBBXVv3/w4EFuvPFGdu7cybFjx/j1r38NwLRp03jjjTc4cuQI8+bNo3Xr1syaNYuMjAz+9Kc/MXr0aGrWrElGRgZfffUVjz76qB7pLiJxp+6vOPv73/9OgwYN+Oijj1i/fj0DBw4EoH79+nz44YfceeedPPLII8XWzc7O5oMPPmDhwoWMHTv2pC9lFhGJlpgmFTMbaGabzWyrmU0sZnsTM1tqZmvMbK2ZXRWUjzCzrLDXcTPrFGxbFrSZvy2+d+1FoLQHJXbo0IF//OMf3HvvvSxfvpy6desCoRsbAbp27Vrik3hvvPFGqlWrRosWLWjevDkff/xxzI9FRKQ0Mev+MrMEYDpwObATWG1mC9w9/OmKDwB/cfcnzawt8BbQzN1fAl4K2ukApLt7Vli9Ee5+Wkw6X9qNlIM7N6Rly5Z8+OGHvPXWWzzwwAP0798f+O5GyISEhBJvfCzPDY8iIhUhlmcq3YGt7r7d3b8F5gLXFtnHgfOC5brArmLauSmoe1oq7UZKgF27dlG7dm1uueUW0tLS+PDDDyNue968eRw/fpxt27axfft2WrXS3eciEl+xHKhvCOwIW98JXFxkn0nA22Z2F3AO8KNi2hnGicloppkdA/4K/NaLudnGzO4A7gDiOmdGaTdSAqxbt460tDSqVatG9erVefLJJ7nhhhsiartJkyZ0796dr776ihkzZlCzZs2oxS0icjJidvOjmd0ADHT324P1kcDF7j4+bJ9fBjH8wcwuAZ4D2rv78WD7xcCz7t4hrE5Dd88xs3MJJZU/u/uLpcVS3psfoylWN1KOHj2aq6++OuIEFA3Z2dlcffXVrF+/vsLeU0Tip7Ld/JgDNA5bbxSUhfsp8BcAd18B1ATqh20fDrwcXsHdc4KfB4A5hLrZKq1430hZWZT1QEwROTPEMqmsBlqYWbKZ1SCUIBYU2eczoD+AmbUhlFR2B+vVgBsJG08xs7PMrH6wXB24GqjUfzbH6kGJs2bNqtCzlHzHjh1jzJgxtGvXjiuuuIK8vLxSJx77xS9+QWpqKo8//jjz5s2jffv2dOzYkcsuu6ygvbS0NLp160ZKSgpPPfVUhR+TiESRu8fsBVwFfAJsA+4Pyh4CBgXLbYH3gI+ALOCKsLp9gZVF2jsHyATWAhuAx4GEsuLo2rWry6n79NNPPSEhwdesWePu7kOHDvXZs2d7hw4dfNmyZe7u/utf/9rvueced3fv06eP33nnnQX127dv7zt37nR393379rm7+1NPPeX//d//7e7uhw8f9q5du/r27dsr5oBEpFRAhpfz935M76h397cIXSYcXvabsOWNQM8S6i4DehQpOwh0jXqgUqC0yckAkpOT6dSpExC6h2bbtm2lTjw2bNiwguWePXsyevRobrzxxoL7cN5++23Wrl3L/PnzgdBEZlu2bCE5OTnWhyoiMaDHtEiBsu6pAQpNHpaQkEBubm6pbebPUgkwY8YMVq1axcKFC+natSuZmZm4O9OmTWPAgAFRPhoRiQc9pkUKlHVPTXHKM/HYtm3buPjii3nooYdISkpix44dDBgwgCeffJIjR44A8Mknn3Dw4MEoHZGIVDSdqUiBsu6pKUmkE4+lpaWxZcsW3J3+/fvTsWNHUlJSyM7OpkuXLrg7SUlJpKenn+qhiEicaJIuKaDJyUQkXGW7T0VOM7qnRkROlbq/pEBlnZxMRE4fSipSSEVNTiYiZyZ1f4mISNQoqYiISNQoqYiISNQoqYiISNQoqYiISNQoqYiISNQoqYiISNQoqYiISNQoqYiISNTENKmY2UAz22xmW81sYjHbm5jZUjNbY2ZrzeyqoLyZmeWZWVbwmhFWp6uZrQvanGpmFstjEBGRyMUsqZhZAjAduJLQtME3mVnbIrs9APzF3TsTmsP+ibBt29y9U/AaG1b+JDAGaBG8BsbqGEREpHxieabSHdjq7tvd/VtgLnBtkX0cOC9YrgvsKq1BM7sIOM/dVwbzJ78IDI5q1CIictJimVQaAjvC1ncGZeEmAbeY2U5Cc9nfFbYtOegW+6eZ9Q5rc2cZbQJgZneYWYaZZezevfsUDkNERCIV74H6m4BZ7t4IuAqYbWbVgM+BJkG32C+BOWZ2XintnMDdn3b3VHdPTUpKinrgIiJyolg++j4HaBy23igoC/dTgjERd19hZjWB+u7+BfBNUJ5pZtuAlkH9RmW0KSIicRLLM5XVQAszSzazGoQG4hcU2eczoD+AmbUBagK7zSwpGOjHzJoTGpDf7u6fA1+ZWY/gqq9bgddjeAwiIlIOMTtTcfejZjYeWAQkAM+7+wYzewjIcPcFwK+AZ8xsAqFB+9Hu7mZ2GfCQmR0BjgNj3X1v0PQ4YBZQC/hb8BIRkUrAQhdRndlSU1M9IyMj3mGIiJxWzCzT3VPLUyfeA/UiInIGUVIREZGoUVIREZGoUVIREZGoUVIREZGoUVKR08LUqVNp06YNI0aMiHcoIlKKWN5RLxI1TzzxBIsXL6ZRo+8eqHD06FHOOktfYZHKRGcqUumNHTuW7du3c+WVV1K3bl1GjhxJz549GTlyJNnZ2fTr14+UlBT69+/PZ599BsC2bdvo0aMHHTp04IEHHqBOnTpxPgqRqkFJRSq9GTNm0KBBA5YuXcqECRPYuHEjixcv5uWXX+auu+5i1KhRrF27lhEjRnD33XcDcM8993DPPfewbt26Qmc3IhJbSipy2hk0aBC1atUCYMWKFdx8880AjBw5knfffbegfOjQoQAF20Uk9tQhLZVG+pocpizazK7cPBok1iJtQCsGdz5xupxzzjknDtGJSCR0piKVQvqaHO57dR05uXk4kJObx32vriN9TekzG1x66aXMnTsXgJdeeonevUPzufXo0YO//vWvAAXbRST2lFSkUpiyaDN5R44VKss7cowpizaXWm/atGnMnDmTlJQUZs+ezeOPPw7AY489xqOPPkpKSgpbt26lbt26MYtdRL6j7i+pFHbl5hVa//e8/yLpmjR25YbWs7OzAZg0aVKh/Zo2bcqSJUsAGD16NB988AFNmjShYcOGrFy5EjNj7ty5bN5cenISkehQUpFKoUFiLXLCEsuFQx8sKM/n7rg71aqVfYKdmZnJ+PHjcXcSExN5/vnnox+0iJwgpt1fZjbQzDab2VYzm1jM9iZmttTM1pjZWjO7Kii/3MwyzWxd8LNfWJ1lQZtZweuCWB6DxN7gwYP5vxd+wefPjeNA1t8B2PnkbVQ/8jWjOtSmVatW3HrrrbRv354dO3ZQp04dJkyYQLt27ejfvz+7d+8+oc2lS5dSo0YNjh8/TuvWrfnBD34AQN++fbn33nvp3r07LVu2ZPny5QAcO3aMtLQ0unXrRkpKCk899VTFfQAiZ5CYJZVgOuDpwJVAW+AmM2tbZLcHgL+4e2dC0w0/EZR/CVzj7h2AUcDsIvVGuHun4PVFrI5BKsbzzz/Ptk1reXHBO+RlvcnxvK84q5rx6x+35Yp232fLli2MGzeODRs20LRpUw4ePEhqaiobNmygT58+PPjggye0OX78eFavXs369evJy8vjzTffLNh29OhRPvjgAx577LGCus899xx169Zl9erVrF69mmeeeYZPP/20wj4DkTNFLLu/ugNb3X07gJnNBa4FNobt48B5wXJdYBeAu68J22cDUMvMznb3b2IYr8RIWZcKT506lddeew2A6nl7+ctNyQx/vSZXpVzE119/TdOmTenRo0fB/tWqVWPYsGEA3HLLLVx33XUnvOfSpUv5n//5Hw4dOsTevXtp164d11xzDUDB/l27di0Yq3n77bdZu3Yt8+fPB2D//v1s2bKF5OTk6H8gImewWCaVhsCOsPWdwMVF9pkEvG1mdwHnAD8qpp3rgQ+LJJSZZnYM+CvwW68KcyKfpvIvFc6/siv/UmGAwZ0bsmzZMhYvXsyKFSuoXbs2ffv25fDhw4XaKOu+FDMrtH748GHGjRtHRkYGjRs3ZtKkSYXaPPvsswFISEjg6NGjQGi8Ztq0aQwYMODUDlikiov3JcU3AbPcvRFwFTDbzApiMrN2wO+Bn4XVGRF0i/UOXiOLa9jM7jCzDDPLKK7PXSpGWZcK79+/n3r16lG7dm0+/vhjVq5cWWabx48fLzijmDNnDr169Sq0PT+B1K9fn6+//rpg39IMGDCAJ598kiNHjgDwySefcPDgwbIPUEQKieWZSg7QOGy9UVAW7qfAQAB3X2FmNYH6wBdm1gh4DbjV3bflV3D3nODnATObQ6ib7cWib+7uTwNPA6SmpupMJk6KXipctHzgwIHMmDGDNm3a0KpVq0LdXCU555xz+OCDD/jtb3/LBRdcwCuvvFJoe2JiImPGjKF9+/Z8//vfp1u3bmW2efvtt5OdnU2XLl1wd5KSkkhPTy/7AEWkEItVz5GZnQV8AvQnlExWAze7+4awff4GvOLus8ysDfAOoW6zusA/gQfd/dUibSa6+5dmVh14GVjs7jNKiyU1NdUzMjKie4ASkZ6TlxS6VDhfw8RavDexXzE1ylanTh2+/vrrUw1NRMpgZpnunlqeOjHr/nL3o8B4YBGwidBVXhvM7CEzGxTs9itgjJl9RChBjA7GR8YD/wH8psilw2cDi8xsLZBFKFk9E6tjkFOXNqAVtaonFCqrVT2BtAGt4hSRiMRSzM5UKhOdqcRXpA+KFJHK5WTOVHRHvcTc4M4NlUREqoh4X/0lIiJnECUVERGJGiUVERGJGiUVERGJGiUVERGJGiUVERGJGiUVqbImTZrEI488Eu8wRM4oSioipyD/KcciEqKkIlXKww8/TMuWLenVq1fBvPXbtm1j4MCBdO3ald69e/Pxxx8DsHv3bq6//nq6detGt27deO+994DQGc7IkSPp2bMnI0cW+5BskSpLd9RLlZGZmcncuXPJysri6NGjdOnSha5du3LHHXcwY8YMWrRowapVqxg3bhxLlizhnnvuYcKECfTq1YvPPvuMAQMGsGnTJgA2btzIu+++S61ateJ8VCKVS5lJxcyuARa6+/EKiEfklJX0rLHly5czZMgQateuDcCgQYM4fPgw77//PkOHDi2o/803ofngFi9ezMaN301U+tVXXxU8HXnQoEFKKCLFiORMZRjwmJn9FXje3T+OcUwiJ620mSaLc/z4cRITE8nKyip228qVK6lZs+YJ28qajVKkqipzTMXdbwE6A9uAWWa2IphV8dyYRydSTqXNNHnZZZeRnp5OXl4eBw4c4I033qB27dokJyczb948IDSt8EcffQTAFVdcwbRp0wraKS7xiEhhEQ3Uu/tXwHxgLnARMAT4MJhbXqTSKG2myS5dujBs2DA6duzIlVdeWTAj5EsvvcRzzz1Hx44dadeuHa+//joAU6dOJSMjg5SUFNq2bcuMGaXOBSciRDCfSjCh1k8ITZr1IvCCu39hZrWBje7eLOZRniLNp1J1xGKmSZGqKlYzP14P/NHdO7j7FHf/AsDdDxGaY760gAaa2WYz22pmE4vZ3sTMlprZGjNba2ZXhW27L6i32cwGRNqmVG2aaVIkviIZqJ8EfJ6/Yma1gAvdPdvd3ympkpklANOBy4GdwGozW+DuG8N2e4DQNMNPmllb4C2gWbA8HGgHNAAWm1nLoE5ZbUoVlj8ZmGaaFImPSJLKPODSsPVjQVm3Mup1B7a6+3YAM5sLXAuEJwAHzguW6wK7guVrgbnu/g3wqZltDdojgjalitNMkyLxE0n311nu/m3+SrBcI4J6DYEdYes7g7Jwk4BbzGwnobOU/IH/kupG0iYAwRVqGWaWsXv37gjCFRGRUxVJUtkdDNYDYGbXAl9G6f1vAma5eyPgKmC2mUXl0THu/rS7p7p7alJSUjSaFBGRMkTyC3ws8P/M7DMz2wHcC/wsgno5QOOw9UZBWbifAn8BcPcVQE2gfil1I2lTqojs7Gzat29/Qvntt99e6E74ksyaNYvx48fHIjSRKqvMMRV33wb0MLM6wfrXEba9GmhhZsmEfvEPB24uss9nQH9CN1W2IZRUdgMLgDlm9iihgfoWwAeARdCmVHHPPvtsseXHjh0jISGh2G0iEh0RdTWZ2Y+BccAvzew3Zvabsuq4+1FgPLAI2EToKq8NZvZQWHfar4AxZvYR8DIw2kM2EDqD2Qj8Hfi5ux8rqc3yHLCcWY4ePcqIESNo06YNN9xwA4cOHaJv377k35dUp04dfvWrX9GxY0dWrFjBzJkzadmyJd27dy946rCIRE8kD5ScAdQGfgg8C9xA6KyhTO7+FqEB+PCy34QtbwR6llD3YeDhSNqUqmvz5s0899xz9OzZk9tuu40nnnii0PaDBw9y8cUX84c//IHPP/+cm2++mczMTOrWrcsPf/hDOnfuHKfIRc5MkZypXOrutwL73P1B4BKgZRl1RKImfU0OPScvIXniQnpOXkL6mu+G0Ro3bkzPnqG/S2655RbefffdQnUTEhK4/vrrAVi1ahV9+/YlKSmJGjVqMGzYsIo7CJEqIpL7VA4HPw+ZWQNgD6Hnf4nEXGlPHe5UD8ys0P5F12vWrKlxFJEKFMmZyhtmlghMAT4EsoE5MYxJpEBpTx0G+Oyzz1ixYgUAc+bMoVevXiW2dfHFF/PPf/6TPXv2cOTIkYInE4tI9JSaVIJ7Rt5x91x3/yvQFGgdPi4iEkulPXUYoFWrVkyfPp02bdqwb98+7rzzzhLbuuiii5g0aRKXXHIJPXv2pE2bNjGJWaQqi+QpxWvc/bQezdRTik9feuqwSPzE6inF75jZ9Va0s1qkAuipwyKnl0gG6n8G/BI4amaHCd2A6O5+XunVRE6dnjoscnqJ5I56TRsscaWnDoucPiK5+fGy4srd/V/RD0dERE5nkXR/pYUt1yQ0r0kmoFFSEREpJJLur2vC182sMfBYrAISEZHT18nMXbIT0AX+IiJygkjGVKYRmvYXQkmoE6E760VERAqJZEwl/K7Bo8DL7q5nhouIyAkiSSrzgcPufgzAzBLMrLa7H4ptaCIicrqJ6I56oFbYei1gcWzCERGR01kkSaVm+BTCwXLtSBo3s4FmttnMtprZxGK2/9HMsoLXJ2aWG5T/MKw8y8wOm9ngYNssM/s0bFunSGIREZHYi6T766CZdXH3DwHMrCtQ/KNjw5hZAjAduJzQFWOrzWxBMNsjAO4+IWz/u4DOQflSQhcEYGbnA1uBt8OaT3P3+RHELiIiFSiSpPILYJ6Z7SL03K/vA5FMmdcd2Oru2wHMbC5wLaF554tzE/BfxZTfAPxNYzgiIpVfmd1f7r4aaA3cCYwF2rh7ZgRtNwR2hK3vDMpOYGZNgWRgSTGbhwMvFyl72MzWBt1nZ5fQ5h1mlmFmGbt3744gXBEROVVlJhUz+zlwjruvd/f1QB0zGxflOIYD8/OvMAt774uADsCisOL7CCW5bsD5wL3FNejuT7t7qrunJiUlRTlcEREpTiQD9WPcPTd/xd33AWMiqJcDNA5bbxSUFae4sxGAG4HX3P1I2Pt/7iHfADMJdbOJiEglEElSSQifoCsYgK8RQb3VQAszSzazGoQSx4KiO5lZa6AesKKYNm6iSLIJzl4IYhoMrI8gFhERqQCRDNT/HXjFzJ4K1n8G/K2sSu5+1MzGE+q6SgCed/cNZvYQkOHu+QlmODDXi8xrbGbNCJ3p/LNI0y+ZWRKhiwayCI3ziIhIJRDJHPXVgDuA/kHRWuD77v7zGMcWNZqjXkSk/GIyR727HwdWAdmExi/6AZtOJkARETmzldj9ZWYtCY1p3AR8CbwC4O4/rJjQRETkdFPamMrHwHLganffCmBmE0rZX0REqrjSur+uAz4HlprZM2bWn9DguIiISLFKTCrunu7uwwndaLiU0ONaLjCzJ83sigqKT0RETiORDNQfdPc5wVz1jYA1lHAXu4iIVG3lmqPe3fcFjz/pX/beIiJS1ZQrqYiIiJRGSUVERKJGSUVERKJGSUVERKJGSUVERKJGSUVERKJGSUVERKJGSUVERKImpknFzAaa2WYz22pmE4vZ/kczywpen5hZbti2Y2HbFoSVJ5vZqqDNV4JZJUVEpBKIWVIJph2eDlwJtAVuMrO24fu4+wR37+TunYBpwKthm/Pyt7n7oLDy3wN/dPf/APYBP43VMYiISPnE8kylO7DV3be7+7fAXODaUvY/YT76ooJ56fsB84OiFwjNUy8iIpVALJNKQ2BH2PrOoOwEZtYUSAaWhBXXNLMMM1tpZoODsu8Bue5+tKw2RUSk4pU2SVdFGg7Md/djYWVN3T3HzJoDS8xsHbA/0gbN7A7gDoAmTZpENVgRESleLM9UcoDGYeuNgrLiDKdI15e75wQ/twPLgM7AHiDRzPKTYYltBk9TTnX31KSkpJM9BhERKYdYJpXVQIvgaq0ahBLHgqI7mVlroB6wIqysnpmdHSzXB3oCG93dCU0YdkOw6yjg9Rgeg4iIlEPMkkow7jEeWARsAv7i7hvM7CEzC7+aazgwN0gY+doAGWb2EaEkMtndNwbb7gV+aWZbCY2xPBerYxARkfKxwr/Lz0ypqamekZER7zBERE4rZpbp7qnlqaM76kVEJGqUVEREJGqUVEREJGqUVEREJGqUVEREJGqUVEREJGqUVESqmEmTJvHII4/EOww5QympiIhI1CipiFQBDz/8MC1btqRXr15s3rwZgKysLHr06EFKSgpDhgxh3759AKxevZqUlBQ6depEWloa7du3j2focppRUhE5w2VmZjJ37lyysrJ46623WL16NQC33norv//971m7di0dOnTgwQcfBOAnP/kJTz31FFlZWSQkJMQzdDkNKamInCHS1+TQc/ISkicupOfkJaSvCT3Ae/ny5QwZMoTatWtz3nnnMWjQIA4ePEhubi59+vQBYNSoUfzrX/8iNzeXAwcOcMkllwBw8803x+145PRUWeZTEZFTkL4mh/teXUfekdCURDm5edz36ro4RyVVkc5URM4AUxZtLkgo+fKOHGPKos1cdtllpKenk5eXx4EDB3jjjTc455xzqFevHsuXLwdg9uzZ9OnTh8TERM4991xWrVoFwNy5cyv8WOT0pjMVkTPArty8Esu7dOnHsGHD6NixIxdccAHdunUD4IUXXmDs2LEcOnSI5s2bM3PmTACee+45xowZQ7Vq1ejTpw9169atsOOQ05+SisgZoEFiLXKKSSwNEmsBcP/993P//fefsH3lypUnlLVr1461a9cCMHnyZFJTy/Xkc6ni1P0lcgZIG9CKWtULX6lVq3oCaQNalbuthQsX0qlTJ9q3b8/y5ct54IEHohWmVAExnaTLzAYCjwMJwLPuPrnI9j8CPwxWawMXuHuimXUCngTOA44BD7v7K0GdWUAfYH9Qb7S7Z5UWhybpkqogfU0OUxZtZlduHg0Sa5E2oBWDOzeMd1hyGjuZSbpillTMLAH4BLgc2ElozvqbwqYFLrr/XUBnd7/NzFoC7u5bzKwBkAm0cffcIKm86e7zI41FSUVEpPwq28yP3YGt7r7d3b8F5gLXlrL/TcDLAO7+ibtvCZZ3AV8ASTGMVUREoiCWSaUhsCNsfWdQdgIzawokA0uK2dYdqAFsCyt+2MzWmtkfzezsEtq8w8wyzCxj9+7dJ3sMIiJSDpVloH44MN/dC11ob2YXAbOBn7j78aD4PqA10A04H7i3uAbd/Wl3T3X31KQkneSIiFSEWCaVHKBx2HqjoKw4wwm6vvKZ2XnAQuB+dy+47tHdP/eQb4CZhLrZRESkEohlUlkNtDCzZDOrQShxLCi6k5m1BuoBK8LKagCvAS8WHZAPzl4wMwMGA+tjdQAiIlI+Mbv50d2Pmtl4YBGhS4qfd/cNZvYQkOHu+QlmODDXC1+GdiNwGfA9MxsdlOVfOvySmSUBBmQBY2N1DCIiUj4xvU+lstAlxSIi5VfZLikWEZEqRklFRESiRklFRESiRklFRESiRklFRESiRklFRESiRklFRESiRklFRESiRklFRESiRklFRESiRklFRESiRklFRESiRklFRESiRklFRESiRklFRESiRklFRESiJqZJxcwGmtlmM9tqZhOL2f5HM8sKXp+YWW7YtlFmtiV4jQor72pm64I2pwbTCouISCUQs+mEzSwBmA5cDuwEVpvZAnffmL+Pu08I2/8uoHOwfD7wX0Aq4EBmUHcf8CQwBlgFvAUMBP4Wq+MQEZHIxfJMpTuw1d23u/u3wFzg2lL2vwl4OVgeAPzD3fcGieQfwEAzuwg4z91XBnPavwgMjtkRiIhIucQyqTQEdoSt7wzKTmBmTYFkYEkZdRsGy5G0eYeZZZhZxu7du0/qAEREpHwqy0D9cGC+ux+LVoPu/rS7p7p7alJSUrSaFRGRUsQyqeQAjcPWGwVlxRnOd11fpdXNCZYjaVNERCpYLJPKaqCFmSWbWQ1CiWNB0Z3MrDVQD1gRVrwIuMLM6plZPeAKYJG7fw58ZWY9gqu+bgVej+ExiIhIOcTs6i93P2pm4wkliATgeXffYGYPARnunp9ghgNzg4H3/Lp7zey/CSUmgIfcfW+wPA6YBdQidNWXrvwSEakkLOx3+RkrNTXVMzIy4h2GiMhpxcw+AX7i7u9HWqeyDNSLiEjlcy5waXkqKKmIiFQxL774IikpKXTs2JGRI0fyxhtvcPHFF9O5c2d+9KMf8e9//5vs7GyAJGBC8NST3pG0re4vEZEqZMOGDQwZMoT333+f+vXrs3fvXsyMxMREzIxnn32WTZs28Yc//AEz+xx41N0fibT9mA3Ui4hIfKSvyWHKos3sys2jQWIt0ga0YnDn0H3iS5YsYejQodSvXx+A888/n3Xr1jFs2DA+//xzvv32W5KTk0/6vdX9JSJyBklfk8N9r64jJzcPB3Jy87jv1XWkryn5lr677rqL8ePHs27dOp566ikOHz580u+vpCIicgaZsmgzeUcKP5wk78gxpizaDEC/fv2YN28ee/bsAWDv3r3s37+fhg1DZzIvvPBCeNVjhAbrI6buLxGRM8iu3LxSy9u1a8f9999Pnz59SEhIoHPnzkyaNImhQ4dSr149+vXrx6effppfLRcYYmbXAne5+/Ky3l8D9SIiZ5Cek5eQU0xiaZhYi/cm9itXW2aW6e6p5amj7i8RkTNI2oBW1KqeUKisVvUE0ga0qpD3V/eXiMgZJP8qr5Ku/oo1JRURkTPM4M4NKyyJFKXuLxERiRolFRERiRolFRERiRolFRERiRolFRERiZoqcfOjme0G/jcOb10f+DIO71uayhgTKK7yqIwxQeWMqzLGBKdPXE3dPak8DVSJpBIvZpZR3rtRY60yxgSKqzwqY0xQOeOqjDHBmR2Xur9ERCRqlFRERCRqlFRi6+l4B1CMyhgTKK7yqIwxQeWMqzLGBGdwXBpTERGRqNGZioiIRI2SioiIRI2Sykkws4FmttnMtprZxGK2jzaz3WaWFbxuL7L9PDPbaWZ/qixxmVkTM3vbzDaZ2UYza1ZJ4vofM9sQxDXVzKwiYgr2uTH4LDaY2Zyw8lFmtiV4jYpGPKcal5l1MrMVQdlaMxsW75jCtsXl+15aXPH8vpcRV1y+72b2x7D/f5+YWW7YtvJ9391dr3K8gARgG9AcqAF8BLQtss9o4E+ltPE4MKe0fSo6LmAZcHmwXAeoHe+4gEuB94I2EoAVQN8KiqkFsAaoF6xfEPw8H9ge/KwXLNerwM+qpLhaAi2C5QbA50BiPGOqBN/3EuOK8/e9pH/DuH3fi+x/F/D8yX7fdaZSft2Bre6+3d2/BeYC10Za2cy6AhcCb1eWuMysLXCWu/8DwN2/dvdD8Y4LcKAmof8IZwPVgX9XUExjgOnuvg/A3b8IygcA/3D3vcG2fwADoxDTKcXl7p+4+5ZgeRfwBVCuO6GjHRPE/ftebFyV4Pte0ucVz+97uJuAl4Plcn/flVTKryGwI2x9Z1BW1PVBN8R8M2sMYGbVgD8A/1mZ4iL0V26umb1qZmvMbIqZJRRTt0LjcvcVwFJCf3V/Dixy900VFFNLoKWZvWdmK81sYDnqxiOuAmbWndAvpm3xjKkSfN9L+qzi/X0vNq44f98BMLOmQDKwpLx18ympxMYbQDN3TyGU2V8IyscBb7n7zkoW11lAb0L/+bsROk0eHe+4zOw/gDZAI0Jf5H5m1ruCYjqLUDdFX0J/uT1jZokV9N6lKTUuM7sImA38xN2PxzmmeH/fS4or3t/3YuOK8/c933BgvrsfO9kGlFTKLwdoHLbeKCgr4O573P2bYPVZoGuwfAkw3syygUeAW81sciWIayeQFZweHwXSgS6VIK4hwMqge+Jr4G+EPsOYx0ToM1ng7kfc/VPgE0K/CCKpG4+4MLPzgIXA/e6+shLEFNfveylxxfX7Xkpc8fy+5xvOd11f5a0bcqqDQFXtReivjO2EThHzB73aFdnnorDl/C9K0XZGE92By5OOi9BA3kdAUrA+E/h5JYhrGLA4aKM68A5wTQXFNBB4IViuT6gL4HuEBiw/JTRoWS9YPr8CP6uS4qoRfD6/iMP3vdiYKsH3vaTPKt7f95Liitv3PdivNZBNcFN8UFbu73vUvnxV6QVcReivi22E/ioEeAgYFCz/DtgQ/OMtBVoX00ZU/5OdalzA5cBaYB0wC6gR77iC//xPAZuAjcCjFRiTAY8G77sOGB5W9zZga/D6SQX/GxYbF3ALcATICnt1ivdnFefve2n/hvH8vpf0bxi373uwPgmYXEzdcn3f9ZgWERGJGo2piIhI1CipiIhI1CipiIhI1CipiIhI1CipiIhI1CipiETAzL4usj76VJ+6a2bZZlb/1CITqVyUVEQqgJmdFe8YRCqCkorIKTKza8xsVfBwwsVmdmFQPsnMZpvZe8BsM/teMIfHBjN7ltBNcJhZs2D+jGeCbW+bWa1g2w/M7O9mlmlmy82sdVA+1MzWm9lHZvavoKydmX0QzImx1sxaxOcTkapMNz+KRMDMjhG6+znf+YSe3zTezOoBue7uFppgrI27/8rMJgHXAL3cPc/MpgJfuvtDZvZj4E1Cj6evQ+hu5VR3zzKzvwRt/9nM3gHGuvsWM7sY+J279zOzdcBAd88xs0R3zzWzaYQecfOSmdUAEtw9r2I+IZEQnZKLRCbP3Tvlr5jZaCA1WG0EvBI8IbgGoecj5VsQ9ov9MuA6AHdfaGb7wvb71N2zguVMoJmZ1SE0cdO8sAkAzw5+vgfMChLQq0HZCuB+M2sEvOrB/CoiFUndXyKnbhqh51p1AH5GaKKlfAcjbOObsOVjhP7gq0boDKhT2KsNgLuPBR4g9ATZTDP7nrvPAQYBecBbZtbvlI5K5CQoqYicurp89zjwUaXs9y/gZgAzu5LQU19L5O5fAZ+a2dCgjplZx2D5B+6+yt1/A+wGGptZc2C7u08FXgdSTuGYRE6KkorIqZtEqIsqE/iylP0eBC4zsw2EusE+i6DtEcBPzewjQk9yzp8GdoqZrTOz9cD7hJ7wfCOw3syygPbAiydxLCKnRAP1IiISNTpTERGRqFFSERGRqFFSERGRqFFSERGRqFFSERGRqFFSERGRqFFSERGRqPn/0tKK+/fD4eoAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "import matplotlib.pyplot as plt\n", "\n", "avg_hardness = [i[0] for i in cls_hardness]\n", "labels = [i[1] for i in cls_hardness]\n", "acc = [i[2] for i in cls_hardness]\n", "\n", "plt.scatter(avg_hardness, acc)\n", "plt.xlabel(\"Hardness\")\n", "plt.ylabel(\"Accuracy\")\n", "for i, label in enumerate(labels):\n", " plt.annotate(label, (avg_hardness[i]+.002, acc[i]))\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that there is a fairly strong anti-correlation between the average hardness of the samples in a class and the accuracy of the model on that class." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take a look at the incorrectly predicted samples of the hardest class, \"cat\"." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "session.view = incorrect_view.match(F(\"ground_truth.label\")==\"cat\").sort_by(\"hardness\", reverse=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's take a look at the correctly predicted images of cats with the lowest hardness." ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", " \n", "
\n", "\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "session.view = correct_view.match(F(\"ground_truth.label\")==\"cat\").sort_by(\"hardness\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Comparing the hardest incorrectly predicted cat images with the easiest correctly predicted cat images, we can see that the model has a much easier time classifying images of cats faces looking directly at the camera. The images of cats that the model struggles the most with are ones of cats in poor lighting, complex backgrounds, and poses where they are not sitting and facing the camera. Now we have an idea of the types of cat images to look for to add to this dataset." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What's next?\n", "\n", "This example was done on a previously annotated set of data in order to show how hardness relates to other aspects of a dataset. In a real-world application, you would now apply this method to new unlabeled data.\n", "\n", "Once you've identified the hardest samples you have available, it's time to update your dataset. You can select the X samples with the highest hardness value to send off to get annotated and added to your train or test set. Alternatively, you could select samples proportionally to the per-class hardness calculated above.\n", "\n", "Retraining your model on this new data should now allow it to perform better on harder edge cases. Additionally, adding these samples to your test set will let you be more confident in the ability of your model to perform well on new unseen data if it performs well on your test set.\n", "\n", "Now it's time to keep improving your dataset by [fixing annotation mistakes](https://voxel51.com/docs/fiftyone/recipes/detection_mistakenness.html) and [adding more unqiue samples](https://voxel51.com/docs/fiftyone/tutorials/uniqueness.html)." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 2 }