{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
"
\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mY1EbrPz4IaX"
},
"source": [
"# PyTorch object detection model training\n",
"\n",
"[PyTorch](https://pytorch.org/) datasets provide a great starting point for loading complex datasets, letting you define a class to load individual samples from disk and then creating data loaders to efficiently supply the data to your model. Problems arise when you want to start iterating over your dataset itself. PyTorch datasets are fairly rigid and require you to either rewrite them or the underlying data on disk if you want to make any changes to the data you are training or testing your model on. That is where [FiftyOne](http://fiftyone.ai) comes in.\n",
"\n",
"PyTorch datasets can synergize well with [FiftyOne datasets](https://voxel51.com/docs/fiftyone/user_guide/using_datasets.html#using-fiftyone-datasets) for hard computer vision problems like [classification, object detection, segmentation, and more](https://voxel51.com/docs/fiftyone/user_guide/using_datasets.html#labels). \n",
"The flexibility of FiftyOne datasets lets you easily experiment with and finetune the datasets you use for training and testing to create better-performing models, faster.\n",
"In this example, I am focusing on object detection since that is one of the most common vision tasks while also being fairly complex. However, these methods work for most machine learning tasks. Specifically, this notebook covers:\n",
"* Loading your labeled dataset into FiftyOne\n",
"* Writing a PyTorch object detection dataset that utilizes your loaded FiftyOne dataset\n",
"* Exploring views into your FiftyOne dataset for training and evaluation\n",
"* Training a Torchvision object detection model on your FiftyOne dataset views\n",
"* Evaluating your models in FiftyOne to refine your dataset"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1HNRS3cs41WZ"
},
"source": [
"## Setup\n",
"\n",
"To start, we need to install FiftyOne:\n",
"\n",
"*If you're working in Google Colab, be sure to [enable a GPU runtime](https://colab.research.google.com/drive/1P7okDVh6viCIOkii6UAF2O9sTAcKGNWq) before running any cell*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Gtvfsp3wWRH_",
"outputId": "2eed0796-a5f8-488d-dabb-e7469092f88e"
},
"outputs": [],
"source": [
"!pip install fiftyone"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install torch torchvision"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We'll also need pytorch, and torchvision, as well as clone the torchvision GitHub repository to use the training and evaluation utilities provided for the [Torchvision Object Deteciton Tutorial](https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#defining-the-dataset) that we are using to train a basic object detection model."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "7dVQMF0SWg5I",
"outputId": "8205dedc-77dc-4963-dbc4-ded52613b7c4"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Cloning into 'vision'...\n",
"remote: Enumerating objects: 126, done.\u001b[K\n",
"remote: Counting objects: 100% (126/126), done.\u001b[K\n",
"remote: Compressing objects: 100% (115/115), done.\u001b[K\n",
"remote: Total 18391 (delta 78), reused 42 (delta 7), pack-reused 18265\u001b[K\n",
"Receiving objects: 100% (18391/18391), 17.21 MiB | 24.65 MiB/s, done.\n",
"Resolving deltas: 100% (13365/13365), done.\n",
"Note: checking out 'v0.3.0'.\n",
"\n",
"You are in 'detached HEAD' state. You can look around, make experimental\n",
"changes and commit them, and you can discard any commits you make in this\n",
"state without impacting any branches by performing another checkout.\n",
"\n",
"If you want to create a new branch to retain commits you create, you may\n",
"do so (now or later) by using -b with the checkout command again. Example:\n",
"\n",
" git checkout -b \n",
"\n",
"HEAD is now at be376084 version check against PyTorch's CUDA version\n"
]
},
{
"data": {
"text/plain": []
},
"execution_count": 7,
"metadata": {
"tags": []
},
"output_type": "execute_result"
}
],
"source": [
"%%shell\n",
"\n",
"# Download TorchVision repo to use some files from\n",
"# references/detection\n",
"git clone https://github.com/pytorch/vision.git\n",
"cd vision\n",
"git checkout v0.3.0\n",
"\n",
"cp references/detection/utils.py ../\n",
"cp references/detection/transforms.py ../\n",
"cp references/detection/coco_eval.py ../\n",
"cp references/detection/engine.py ../\n",
"cp references/detection/coco_utils.py ../"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vZDc2VuwNDm0"
},
"source": [
"## Loading your data\n",
"\n",
"Getting your data into FiftyOne is oftentimes actually easier than getting it into a PyTorch dataset. Additionally, once the data is in FiftyOne it is much more flexible allowing you to easily find and access even the most specific subsets of data that you can then use to train or evaluate your model.\n",
"\n",
"If you have data that follows a certain format on disk ([for example a directory tree for classification, the COCO detection format, or many more](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/index.html#id1)), then you can load it into FiftyOne in [one line of code](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/index.html).\n",
"\n",
"In this notebook, I am going to work with [COCO-2017](https://cocodataset.org/#home) and load it from the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_zoo/index.html)."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "pZ2cvwpPWXBt",
"outputId": "6444f42e-7465-4625-e4b1-3c207385fce9"
},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 4,
"metadata": {
"tags": []
},
"output_type": "execute_result"
}
],
"source": [
"import torch\n",
"\n",
"torch.manual_seed(1)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "TDvJBjDk2wpy"
},
"outputs": [],
"source": [
"import fiftyone as fo\n",
"import fiftyone.zoo as foz"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "5crNDNsRWdPT",
"outputId": "4f3ff734-ca0a-4312-a811-7f84db514fac"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Downloading split 'validation' to '/root/fiftyone/coco-2017/validation'\n",
"Downloading images zip to '/root/fiftyone/coco-2017/tmp-download/val2017.zip'\n",
" 100% |██████| 6.1Gb/6.1Gb [10.6s elapsed, 0s remaining, 502.3Mb/s] \n",
"Extracting images to '/root/fiftyone/coco-2017/tmp-download/val2017'\n",
"Downloading annotations zip to '/root/fiftyone/coco-2017/tmp-download/annotations_trainval2017.zip'\n",
" 100% |██████| 1.9Gb/1.9Gb [4.1s elapsed, 0s remaining, 502.3Mb/s] \n",
"Extracting annotations to '/root/fiftyone/coco-2017/tmp-download/annotations/instances_val2017.json'\n",
"loading annotations into memory...\n",
"Done (t=0.54s)\n",
"creating index...\n",
"index created!\n",
" 100% |███████████████| 5000/5000 [1.7m elapsed, 0s remaining, 52.2 samples/s] \n",
"Dataset info written to '/root/fiftyone/coco-2017/info.json'\n",
"Loading 'coco-2017' split 'validation'\n",
" 100% |███████████████| 5000/5000 [37.3s elapsed, 0s remaining, 130.1 samples/s] \n",
"Dataset 'coco-2017-validation' created\n"
]
}
],
"source": [
"fo_dataset = foz.load_zoo_dataset(\"coco-2017\", \"validation\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sc43DB9jNqGD"
},
"source": [
"We will be needing the height and width of images later in this notebook so we need to compute metadata on our dataset."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "YOdY2KQeGz1r",
"outputId": "06a1a270-d816-4f01-f3fe-2cc6d628a3bc"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" 100% |███████████████| 5000/5000 [55.2s elapsed, 0s remaining, 142.6 samples/s] \n"
]
}
],
"source": [
"fo_dataset.compute_metadata()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "99mcj42PN4OE"
},
"source": [
"We can create a session and visualize this dataset in the [FiftyOne App](https://voxel51.com/docs/fiftyone/user_guide/app.html)."
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 821,
"resources": {
"https://localhost:5151/polling?sessionId=93ede2db-8164-481d-9a83-1b5f3e83c66f": {
"data": "eyJtZXNzYWdlcyI6IFtdfQ==",
"headers": [
[
"access-control-allow-headers",
"x-requested-with"
],
[
"content-type",
"text/html; charset=UTF-8"
]
],
"ok": true,
"status": 200,
"status_text": ""
}
}
},
"id": "m63jPSOmCtK1",
"outputId": "b0fac4a2-0ec3-4846-b645-00f96880dfd4"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"
\n",
" Activate \n",
"
\n",
"
"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"d659815a-86a3-11eb-bf60-0242ac1c0002\"] = google.colab.output.getActiveOutputArea();\n",
"//# sourceURL=js_9491c43f12"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"d65a0b3e-86a3-11eb-bf60-0242ac1c0002\"] = document.querySelector(\"#focontainer-622230b7-07b5-49dc-aea0-612e84993d7e\");\n",
"//# sourceURL=js_d685c0094f"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"d65b5af2-86a3-11eb-bf60-0242ac1c0002\"] = google.colab.output.setActiveOutputArea(window[\"d65a0b3e-86a3-11eb-bf60-0242ac1c0002\"]);\n",
"//# sourceURL=js_e681dc4146"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" "
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"d65d5e24-86a3-11eb-bf60-0242ac1c0002\"] = google.colab.output.setActiveOutputArea(window[\"d659815a-86a3-11eb-bf60-0242ac1c0002\"]);\n",
"//# sourceURL=js_a4ed17f006"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
}
],
"source": [
"session = fo.launch_app(fo_dataset)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b8uR5eEUOCUQ"
},
"source": [
"## PyTorch dataset and training setup\n",
"\n",
"A [PyTorch dataset](https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class) is a class that defines how to load a static dataset and its labels from disk via a simple iterator interface. They differ from FiftyOne datasets which are flexible representations of your data geared towards visualization, querying, and understanding.\n",
"\n",
"Every PyTorch model expects data and labels to pass into it in a certain format. Before being able to write up a PyTorch dataset class, you first need to understand the format that the model requires. Namely, we need to know exactly what format the data loader is expected to output when iterating through the dataset so that we can properly define the `__getitem__` method in the PyTorch dataset.\n",
"In this example, I am following the [Torchvision object detection tutorial](https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#defining-the-dataset) and construct a PyTorch dataset to work with their RCNN-based models."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"id": "IRR2fVMlWeKI"
},
"outputs": [],
"source": [
"import torch\n",
"import fiftyone.utils.coco as fouc\n",
"from PIL import Image\n",
"\n",
"\n",
"class FiftyOneTorchDataset(torch.utils.data.Dataset):\n",
" \"\"\"A class to construct a PyTorch dataset from a FiftyOne dataset.\n",
" \n",
" Args:\n",
" fiftyone_dataset: a FiftyOne dataset or view that will be used for training or testing\n",
" transforms (None): a list of PyTorch transforms to apply to images and targets when loading\n",
" gt_field (\"ground_truth\"): the name of the field in fiftyone_dataset that contains the \n",
" desired labels to load\n",
" classes (None): a list of class strings that are used to define the mapping between\n",
" class names and indices. If None, it will use all classes present in the given fiftyone_dataset.\n",
" \"\"\"\n",
"\n",
" def __init__(\n",
" self,\n",
" fiftyone_dataset,\n",
" transforms=None,\n",
" gt_field=\"ground_truth\",\n",
" classes=None,\n",
" ):\n",
" self.samples = fiftyone_dataset\n",
" self.transforms = transforms\n",
" self.gt_field = gt_field\n",
"\n",
" self.img_paths = self.samples.values(\"filepath\")\n",
"\n",
" self.classes = classes\n",
" if not self.classes:\n",
" # Get list of distinct labels that exist in the view\n",
" self.classes = self.samples.distinct(\n",
" \"%s.detections.label\" % gt_field\n",
" )\n",
"\n",
" if self.classes[0] != \"background\":\n",
" self.classes = [\"background\"] + self.classes\n",
"\n",
" self.labels_map_rev = {c: i for i, c in enumerate(self.classes)}\n",
"\n",
" def __getitem__(self, idx):\n",
" img_path = self.img_paths[idx]\n",
" sample = self.samples[img_path]\n",
" metadata = sample.metadata\n",
" img = Image.open(img_path).convert(\"RGB\")\n",
"\n",
" boxes = []\n",
" labels = []\n",
" area = []\n",
" iscrowd = []\n",
" detections = sample[self.gt_field].detections\n",
" for det in detections:\n",
" category_id = self.labels_map_rev[det.label]\n",
" coco_obj = fouc.COCOObject.from_label(\n",
" det, metadata, category_id=category_id,\n",
" )\n",
" x, y, w, h = coco_obj.bbox\n",
" boxes.append([x, y, x + w, y + h])\n",
" labels.append(coco_obj.category_id)\n",
" area.append(coco_obj.area)\n",
" iscrowd.append(coco_obj.iscrowd)\n",
"\n",
" target = {}\n",
" target[\"boxes\"] = torch.as_tensor(boxes, dtype=torch.float32)\n",
" target[\"labels\"] = torch.as_tensor(labels, dtype=torch.int64)\n",
" target[\"image_id\"] = torch.as_tensor([idx])\n",
" target[\"area\"] = torch.as_tensor(area, dtype=torch.float32)\n",
" target[\"iscrowd\"] = torch.as_tensor(iscrowd, dtype=torch.int64)\n",
"\n",
" if self.transforms is not None:\n",
" img, target = self.transforms(img, target)\n",
"\n",
" return img, target\n",
"\n",
" def __len__(self):\n",
" return len(self.img_paths)\n",
"\n",
" def get_classes(self):\n",
" return self.classes"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9aLHV0VXVW3l"
},
"source": [
"The following code loads Faster-RCNN with a ResNet50 backbone from Torchvision and modifies the classifier for the number of classes we are training on:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"id": "m4PuSY1rWlTy"
},
"outputs": [],
"source": [
"import torchvision\n",
"from torchvision.models.detection.faster_rcnn import FastRCNNPredictor\n",
"\n",
"def get_model(num_classes):\n",
" # load a model pre-trained pre-trained on COCO\n",
" model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)\n",
" \n",
" # get number of input features for the classifier\n",
" in_features = model.roi_heads.box_predictor.cls_score.in_features\n",
" # replace the pre-trained head with a new one\n",
" model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)\n",
" return model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "coKEeFj3WfhF"
},
"source": [
"For this example, we are going to write a simple training loop. This function is going to take a model and our PyTorch datasets as input and use the [`train_one_epoch()`](https://github.com/pytorch/vision/blob/master/references/detection/engine.py) and [`evaluate()`](https://github.com/pytorch/vision/blob/master/references/detection/engine.py) functions from the Torchvision object detection code"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "JX5MuzD-VX5i"
},
"outputs": [],
"source": [
"# Import functions from the torchvision references we cloned\n",
"from engine import train_one_epoch, evaluate\n",
"import utils\n",
"\n",
"def do_training(model, torch_dataset, torch_dataset_test, num_epochs=4):\n",
" # define training and validation data loaders\n",
" data_loader = torch.utils.data.DataLoader(\n",
" torch_dataset, batch_size=2, shuffle=True, num_workers=2,\n",
" collate_fn=utils.collate_fn)\n",
" \n",
" data_loader_test = torch.utils.data.DataLoader(\n",
" torch_dataset_test, batch_size=1, shuffle=False, num_workers=2,\n",
" collate_fn=utils.collate_fn)\n",
"\n",
" # train on the GPU or on the CPU, if a GPU is not available\n",
" device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n",
" print(\"Using device %s\" % device)\n",
"\n",
" # move model to the right device\n",
" model.to(device)\n",
"\n",
" # construct an optimizer\n",
" params = [p for p in model.parameters() if p.requires_grad]\n",
" optimizer = torch.optim.SGD(params, lr=0.005,\n",
" momentum=0.9, weight_decay=0.0005)\n",
" # and a learning rate scheduler\n",
" lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,\n",
" step_size=3,\n",
" gamma=0.1)\n",
"\n",
" for epoch in range(num_epochs):\n",
" # train for one epoch, printing every 10 iterations\n",
" train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)\n",
"\n",
" # update the learning rate\n",
" lr_scheduler.step()\n",
" # evaluate on the test dataset\n",
" evaluate(model, data_loader_test, device=device)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qRKqLZrWV0VR"
},
"source": [
"## FiftyOne views and datasets\n",
"\n",
"One of the primary ways of interacting with your FiftyOne dataset is through different [views](https://voxel51.com/docs/fiftyone/user_guide/using_views.html) into your dataset. These are constructed by applying operations like filtering, sorting, slicing, etc, that result in a specific view into certain labels/samples of your dataset. These operations make it easier to experiment with different subsets of data and continue to finetune your dataset to train better models.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PqU6Ckq4WKHK"
},
"source": [
"For example, cluttered images make it difficult for models to localize objects. We can use FiftyOne to create a view containing only samples with more than, say, 10 objects. You can perform the same operations on views as datasets, so we can create an instance of our PyTorch dataset from this view:"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {
"id": "kLACOukJFUxd"
},
"outputs": [],
"source": [
"from fiftyone import ViewField as F\n",
"\n",
"busy_view = fo_dataset.match(F(\"ground_truth.detections\").length() > 10)\n",
"\n",
"busy_torch_dataset = FiftyOneTorchDataset(busy_view)"
]
},
{
"cell_type": "code",
"execution_count": 65,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 821,
"resources": {
"https://localhost:5151/polling?sessionId=49f79393-944b-457e-80fe-c7ae20d8f582": {
"data": "eyJtZXNzYWdlcyI6IFtdfQ==",
"headers": [
[
"access-control-allow-headers",
"x-requested-with"
],
[
"content-type",
"text/html; charset=UTF-8"
]
],
"ok": true,
"status": 200,
"status_text": ""
}
}
},
"id": "xVs_jYcLFXII",
"outputId": "3a011751-868b-46ea-c7cd-01bdbf72293c"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"
\n",
" Activate \n",
"
\n",
"
"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"d36d8100-86a6-11eb-bf60-0242ac1c0002\"] = google.colab.output.getActiveOutputArea();\n",
"//# sourceURL=js_f0539534d6"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"d36decf8-86a6-11eb-bf60-0242ac1c0002\"] = document.querySelector(\"#focontainer-952d86f2-8449-4a2c-9f1b-65913a91f258\");\n",
"//# sourceURL=js_dd7515f086"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"d36e33b6-86a6-11eb-bf60-0242ac1c0002\"] = google.colab.output.setActiveOutputArea(window[\"d36decf8-86a6-11eb-bf60-0242ac1c0002\"]);\n",
"//# sourceURL=js_4559905460"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" "
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"d37089fe-86a6-11eb-bf60-0242ac1c0002\"] = google.colab.output.setActiveOutputArea(window[\"d36d8100-86a6-11eb-bf60-0242ac1c0002\"]);\n",
"//# sourceURL=js_5704c3f7cc"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
}
],
"source": [
"session.view = busy_view"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xKsE_7TOWXBE"
},
"source": [
"Another example is if we want to train a model that is used primarily for road vehicle detection. We can easily create training and testing views (and corresponding PyTorch datasets) that only contain the classes car, truck, and bus:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "TELK0NWmWrMT",
"outputId": "8bf582cf-e483-4643-8f6b-7c664a2d6c5f"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"707\n"
]
}
],
"source": [
"from fiftyone import ViewField as F\n",
"\n",
"vehicles_list = [\"car\", \"truck\", \"bus\"]\n",
"vehicles_view = fo_dataset.filter_labels(\"ground_truth\",\n",
" F(\"label\").is_in(vehicles_list))\n",
"\n",
"print(len(vehicles_view))"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 821,
"resources": {
"https://localhost:5151/polling?sessionId=fdd7eeab-e102-4dae-99e5-e8d548dd1870": {
"data": "eyJtZXNzYWdlcyI6IFt7InR5cGUiOiAiZGVhY3RpdmF0ZSJ9XX0=",
"headers": [
[
"access-control-allow-headers",
"x-requested-with"
],
[
"content-type",
"text/html; charset=UTF-8"
]
],
"ok": true,
"status": 200,
"status_text": ""
}
}
},
"id": "COjr-9RvErG-",
"outputId": "fc7067ba-d8b3-467c-8886-e9f0360127aa"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"
\n",
" Activate \n",
"
\n",
"
"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"f45ed032-86a4-11eb-bf60-0242ac1c0002\"] = google.colab.output.getActiveOutputArea();\n",
"//# sourceURL=js_a533c9fdd9"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"f45f403a-86a4-11eb-bf60-0242ac1c0002\"] = document.querySelector(\"#focontainer-bf9ada08-2b21-4326-910d-71abadc00e58\");\n",
"//# sourceURL=js_f498cf2b00"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"f45fbb14-86a4-11eb-bf60-0242ac1c0002\"] = google.colab.output.setActiveOutputArea(window[\"f45f403a-86a4-11eb-bf60-0242ac1c0002\"]);\n",
"//# sourceURL=js_d3b742dcec"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" "
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"f461ebb4-86a4-11eb-bf60-0242ac1c0002\"] = google.colab.output.setActiveOutputArea(window[\"f45ed032-86a4-11eb-bf60-0242ac1c0002\"]);\n",
"//# sourceURL=js_9e5d709f9a"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
}
],
"source": [
"session.view = vehicles_view"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"id": "9BjUPS6FWndn"
},
"outputs": [],
"source": [
"# From the torchvision references we cloned\n",
"import transforms as T\n",
"\n",
"train_transforms = T.Compose([T.ToTensor(), T.RandomHorizontalFlip(0.5)])\n",
"test_transforms = T.Compose([T.ToTensor()])"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "8g6hT1XkWsrn"
},
"outputs": [],
"source": [
"# split the dataset in train and test set\n",
"train_view = vehicles_view.take(500, seed=51)\n",
"test_view = vehicles_view.exclude([s.id for s in train_view])"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "506vhQ9gZmWX"
},
"outputs": [],
"source": [
"# use our dataset and defined transformations\n",
"torch_dataset = FiftyOneTorchDataset(train_view, train_transforms,\n",
" classes=vehicles_list)\n",
"torch_dataset_test = FiftyOneTorchDataset(test_view, test_transforms, \n",
" classes=vehicles_list)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "C5je6lVBWz5r"
},
"source": [
"## Training and Evaluation\n",
"\n",
"In this section, we use the functions and datasets we defined above to initialize, train, and evaluate a model continuing with the vehicle example. "
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 103,
"referenced_widgets": [
"acbb3df601244291b8b2fb9ea1137573",
"32b6ec3046e64d04b4134553dc434fe0",
"d8c6a316609d4ca5bfee139b93177ef5",
"a1645bdfb02b42fba268f7000f183639",
"4a4788a4fd6841788b20cfbf54a3d10b",
"5d836b94d13e459d82429606496e4d4f",
"a410071b34034a91aeda7ef1114969c2",
"c063e7d90f6a4027b53d1b70c8c07742"
]
},
"id": "bHa6KRbEWuxz",
"outputId": "3b4ebd0b-aa69-4a4d-b73c-b8cf24d8b461"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Downloading: \"https://download.pytorch.org/models/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth\" to /root/.cache/torch/hub/checkpoints/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "acbb3df601244291b8b2fb9ea1137573",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"HBox(children=(FloatProgress(value=0.0, max=167502836.0), HTML(value='')))"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"source": [
"model = get_model(len(vehicles_list)+1)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "o3xZ6YMjWwY5",
"outputId": "530a2a74-6a5e-4240-eae2-dfed097c450b"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Using device cuda\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n",
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch: [0] [ 0/250] eta: 0:05:48 lr: 0.000025 loss: 2.0424 (2.0424) loss_classifier: 1.3071 (1.3071) loss_box_reg: 0.6820 (0.6820) loss_objectness: 0.0361 (0.0361) loss_rpn_box_reg: 0.0172 (0.0172) time: 1.3925 data: 0.2820 max mem: 3094\n",
"Epoch: [0] [ 10/250] eta: 0:02:39 lr: 0.000226 loss: 1.8390 (1.7483) loss_classifier: 1.2389 (1.1791) loss_box_reg: 0.5072 (0.4188) loss_objectness: 0.0438 (0.0689) loss_rpn_box_reg: 0.0219 (0.0814) time: 0.6632 data: 0.0323 max mem: 3506\n",
"Epoch: [0] [ 20/250] eta: 0:02:21 lr: 0.000426 loss: 1.2446 (1.3204) loss_classifier: 0.7553 (0.8323) loss_box_reg: 0.2559 (0.3675) loss_objectness: 0.0349 (0.0595) loss_rpn_box_reg: 0.0190 (0.0612) time: 0.5744 data: 0.0070 max mem: 3506\n",
"Epoch: [0] [ 30/250] eta: 0:02:12 lr: 0.000627 loss: 0.9709 (1.1985) loss_classifier: 0.5002 (0.7143) loss_box_reg: 0.3701 (0.3838) loss_objectness: 0.0290 (0.0493) loss_rpn_box_reg: 0.0190 (0.0511) time: 0.5688 data: 0.0068 max mem: 3506\n",
"Epoch: [0] [ 40/250] eta: 0:02:03 lr: 0.000827 loss: 0.6146 (1.0203) loss_classifier: 0.2428 (0.5860) loss_box_reg: 0.2571 (0.3442) loss_objectness: 0.0233 (0.0463) loss_rpn_box_reg: 0.0165 (0.0437) time: 0.5660 data: 0.0068 max mem: 3506\n",
"Epoch: [0] [ 50/250] eta: 0:01:57 lr: 0.001028 loss: 0.6146 (0.9627) loss_classifier: 0.2233 (0.5214) loss_box_reg: 0.2774 (0.3545) loss_objectness: 0.0233 (0.0430) loss_rpn_box_reg: 0.0140 (0.0438) time: 0.5606 data: 0.0067 max mem: 3506\n",
"Epoch: [0] [ 60/250] eta: 0:01:53 lr: 0.001229 loss: 0.6338 (0.9093) loss_classifier: 0.2331 (0.4718) loss_box_reg: 0.3477 (0.3581) loss_objectness: 0.0210 (0.0399) loss_rpn_box_reg: 0.0155 (0.0395) time: 0.6124 data: 0.0067 max mem: 3862\n",
"Epoch: [0] [ 70/250] eta: 0:01:48 lr: 0.001429 loss: 0.5526 (0.8670) loss_classifier: 0.1829 (0.4322) loss_box_reg: 0.3477 (0.3564) loss_objectness: 0.0244 (0.0405) loss_rpn_box_reg: 0.0119 (0.0380) time: 0.6477 data: 0.0071 max mem: 4211\n",
"Epoch: [0] [ 80/250] eta: 0:01:43 lr: 0.001630 loss: 0.4572 (0.8247) loss_classifier: 0.1649 (0.3991) loss_box_reg: 0.2320 (0.3494) loss_objectness: 0.0285 (0.0382) loss_rpn_box_reg: 0.0138 (0.0380) time: 0.6457 data: 0.0072 max mem: 4211\n",
"Epoch: [0] [ 90/250] eta: 0:01:38 lr: 0.001830 loss: 0.4399 (0.7853) loss_classifier: 0.1504 (0.3728) loss_box_reg: 0.2342 (0.3415) loss_objectness: 0.0101 (0.0360) loss_rpn_box_reg: 0.0122 (0.0350) time: 0.6507 data: 0.0069 max mem: 4211\n",
"Epoch: [0] [100/250] eta: 0:01:32 lr: 0.002031 loss: 0.3349 (0.7592) loss_classifier: 0.1167 (0.3501) loss_box_reg: 0.1766 (0.3313) loss_objectness: 0.0155 (0.0385) loss_rpn_box_reg: 0.0069 (0.0392) time: 0.6604 data: 0.0068 max mem: 4211\n",
"Epoch: [0] [110/250] eta: 0:01:27 lr: 0.002232 loss: 0.3243 (0.7329) loss_classifier: 0.1398 (0.3329) loss_box_reg: 0.1679 (0.3222) loss_objectness: 0.0209 (0.0385) loss_rpn_box_reg: 0.0129 (0.0392) time: 0.6575 data: 0.0070 max mem: 4211\n",
"Epoch: [0] [120/250] eta: 0:01:21 lr: 0.002432 loss: 0.5199 (0.7187) loss_classifier: 0.1594 (0.3210) loss_box_reg: 0.2173 (0.3169) loss_objectness: 0.0252 (0.0414) loss_rpn_box_reg: 0.0180 (0.0392) time: 0.6643 data: 0.0071 max mem: 4211\n",
"Epoch: [0] [130/250] eta: 0:01:15 lr: 0.002633 loss: 0.6149 (0.7129) loss_classifier: 0.2029 (0.3155) loss_box_reg: 0.3126 (0.3191) loss_objectness: 0.0282 (0.0402) loss_rpn_box_reg: 0.0186 (0.0381) time: 0.6511 data: 0.0069 max mem: 4211\n",
"Epoch: [0] [140/250] eta: 0:01:08 lr: 0.002833 loss: 0.5098 (0.6932) loss_classifier: 0.1946 (0.3066) loss_box_reg: 0.2652 (0.3109) loss_objectness: 0.0195 (0.0388) loss_rpn_box_reg: 0.0166 (0.0368) time: 0.6045 data: 0.0067 max mem: 4211\n",
"Epoch: [0] [150/250] eta: 0:01:02 lr: 0.003034 loss: 0.4439 (0.6817) loss_classifier: 0.1885 (0.3004) loss_box_reg: 0.2257 (0.3060) loss_objectness: 0.0195 (0.0391) loss_rpn_box_reg: 0.0125 (0.0363) time: 0.5870 data: 0.0066 max mem: 4211\n",
"Epoch: [0] [160/250] eta: 0:00:56 lr: 0.003235 loss: 0.5237 (0.6770) loss_classifier: 0.1961 (0.2968) loss_box_reg: 0.2549 (0.3057) loss_objectness: 0.0277 (0.0386) loss_rpn_box_reg: 0.0162 (0.0358) time: 0.6235 data: 0.0072 max mem: 4211\n",
"Epoch: [0] [170/250] eta: 0:00:49 lr: 0.003435 loss: 0.5588 (0.6670) loss_classifier: 0.2055 (0.2910) loss_box_reg: 0.2656 (0.3019) loss_objectness: 0.0268 (0.0385) loss_rpn_box_reg: 0.0195 (0.0356) time: 0.6290 data: 0.0072 max mem: 4211\n",
"Epoch: [0] [180/250] eta: 0:00:43 lr: 0.003636 loss: 0.4949 (0.6569) loss_classifier: 0.1841 (0.2846) loss_box_reg: 0.2374 (0.2987) loss_objectness: 0.0319 (0.0385) loss_rpn_box_reg: 0.0167 (0.0350) time: 0.6007 data: 0.0066 max mem: 4211\n",
"Epoch: [0] [190/250] eta: 0:00:37 lr: 0.003837 loss: 0.4744 (0.6498) loss_classifier: 0.1813 (0.2792) loss_box_reg: 0.2171 (0.2947) loss_objectness: 0.0337 (0.0398) loss_rpn_box_reg: 0.0209 (0.0361) time: 0.6212 data: 0.0065 max mem: 4211\n",
"Epoch: [0] [200/250] eta: 0:00:30 lr: 0.004037 loss: 0.4094 (0.6356) loss_classifier: 0.1452 (0.2727) loss_box_reg: 0.1871 (0.2890) loss_objectness: 0.0243 (0.0392) loss_rpn_box_reg: 0.0139 (0.0348) time: 0.6045 data: 0.0065 max mem: 4211\n",
"Epoch: [0] [210/250] eta: 0:00:24 lr: 0.004238 loss: 0.3599 (0.6242) loss_classifier: 0.1452 (0.2670) loss_box_reg: 0.1616 (0.2835) loss_objectness: 0.0208 (0.0390) loss_rpn_box_reg: 0.0073 (0.0346) time: 0.6035 data: 0.0067 max mem: 4211\n",
"Epoch: [0] [220/250] eta: 0:00:18 lr: 0.004438 loss: 0.3919 (0.6147) loss_classifier: 0.1589 (0.2621) loss_box_reg: 0.1741 (0.2798) loss_objectness: 0.0278 (0.0383) loss_rpn_box_reg: 0.0073 (0.0344) time: 0.6041 data: 0.0068 max mem: 4211\n",
"Epoch: [0] [230/250] eta: 0:00:12 lr: 0.004639 loss: 0.4691 (0.6127) loss_classifier: 0.1637 (0.2601) loss_box_reg: 0.2205 (0.2802) loss_objectness: 0.0179 (0.0378) loss_rpn_box_reg: 0.0173 (0.0347) time: 0.5973 data: 0.0066 max mem: 4211\n",
"Epoch: [0] [240/250] eta: 0:00:06 lr: 0.004840 loss: 0.5367 (0.6078) loss_classifier: 0.1789 (0.2572) loss_box_reg: 0.2651 (0.2790) loss_objectness: 0.0164 (0.0374) loss_rpn_box_reg: 0.0207 (0.0342) time: 0.6496 data: 0.0069 max mem: 4211\n",
"Epoch: [0] [249/250] eta: 0:00:00 lr: 0.005000 loss: 0.3983 (0.6044) loss_classifier: 0.1707 (0.2538) loss_box_reg: 0.2203 (0.2755) loss_objectness: 0.0173 (0.0402) loss_rpn_box_reg: 0.0246 (0.0350) time: 0.6465 data: 0.0069 max mem: 4211\n",
"Epoch: [0] Total time: 0:02:35 (0.6206 s / it)\n",
"creating index...\n",
"index created!\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n",
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Test: [ 0/207] eta: 0:01:08 model_time: 0.1717 (0.1717) evaluator_time: 0.0182 (0.0182) time: 0.3311 data: 0.1394 max mem: 4211\n",
"Test: [100/207] eta: 0:00:13 model_time: 0.1220 (0.1176) evaluator_time: 0.0048 (0.0063) time: 0.1298 data: 0.0034 max mem: 4211\n",
"Test: [200/207] eta: 0:00:00 model_time: 0.1123 (0.1173) evaluator_time: 0.0035 (0.0061) time: 0.1228 data: 0.0036 max mem: 4211\n",
"Test: [206/207] eta: 0:00:00 model_time: 0.1130 (0.1173) evaluator_time: 0.0061 (0.0062) time: 0.1256 data: 0.0036 max mem: 4211\n",
"Test: Total time: 0:00:26 (0.1292 s / it)\n",
"Averaged stats: model_time: 0.1130 (0.1173) evaluator_time: 0.0061 (0.0062)\n",
"Accumulating evaluation results...\n",
"DONE (t=0.20s).\n",
"IoU metric: bbox\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.062\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.147\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.046\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.059\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.068\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.106\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.079\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.221\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.286\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.271\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.313\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.333\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n",
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch: [1] [ 0/250] eta: 0:03:36 lr: 0.005000 loss: 0.7783 (0.7783) loss_classifier: 0.2188 (0.2188) loss_box_reg: 0.3122 (0.3122) loss_objectness: 0.2095 (0.2095) loss_rpn_box_reg: 0.0379 (0.0379) time: 0.8654 data: 0.3291 max mem: 4211\n",
"Epoch: [1] [ 10/250] eta: 0:02:30 lr: 0.005000 loss: 0.4352 (0.4338) loss_classifier: 0.1951 (0.1744) loss_box_reg: 0.1519 (0.1611) loss_objectness: 0.0443 (0.0770) loss_rpn_box_reg: 0.0099 (0.0213) time: 0.6275 data: 0.0362 max mem: 4211\n",
"Epoch: [1] [ 20/250] eta: 0:02:21 lr: 0.005000 loss: 0.4352 (0.4672) loss_classifier: 0.1951 (0.1943) loss_box_reg: 0.1854 (0.1962) loss_objectness: 0.0252 (0.0554) loss_rpn_box_reg: 0.0140 (0.0214) time: 0.6041 data: 0.0066 max mem: 4211\n",
"Epoch: [1] [ 30/250] eta: 0:02:17 lr: 0.005000 loss: 0.4343 (0.4460) loss_classifier: 0.1785 (0.1889) loss_box_reg: 0.1912 (0.1886) loss_objectness: 0.0244 (0.0472) loss_rpn_box_reg: 0.0160 (0.0213) time: 0.6212 data: 0.0065 max mem: 4211\n",
"Epoch: [1] [ 40/250] eta: 0:02:11 lr: 0.005000 loss: 0.3589 (0.4416) loss_classifier: 0.1737 (0.1831) loss_box_reg: 0.1693 (0.1881) loss_objectness: 0.0266 (0.0445) loss_rpn_box_reg: 0.0145 (0.0260) time: 0.6329 data: 0.0066 max mem: 4400\n",
"Epoch: [1] [ 50/250] eta: 0:02:04 lr: 0.005000 loss: 0.4596 (0.4623) loss_classifier: 0.1906 (0.1905) loss_box_reg: 0.1693 (0.1995) loss_objectness: 0.0280 (0.0416) loss_rpn_box_reg: 0.0186 (0.0306) time: 0.6211 data: 0.0067 max mem: 4400\n",
"Epoch: [1] [ 60/250] eta: 0:01:57 lr: 0.005000 loss: 0.5062 (0.4784) loss_classifier: 0.1928 (0.1908) loss_box_reg: 0.2222 (0.2038) loss_objectness: 0.0280 (0.0451) loss_rpn_box_reg: 0.0186 (0.0387) time: 0.6120 data: 0.0069 max mem: 4400\n",
"Epoch: [1] [ 70/250] eta: 0:01:51 lr: 0.005000 loss: 0.3885 (0.4576) loss_classifier: 0.1602 (0.1834) loss_box_reg: 0.1471 (0.1954) loss_objectness: 0.0327 (0.0436) loss_rpn_box_reg: 0.0175 (0.0352) time: 0.6113 data: 0.0068 max mem: 4400\n",
"Epoch: [1] [ 80/250] eta: 0:01:45 lr: 0.005000 loss: 0.3554 (0.4643) loss_classifier: 0.1489 (0.1844) loss_box_reg: 0.1471 (0.2005) loss_objectness: 0.0369 (0.0437) loss_rpn_box_reg: 0.0170 (0.0358) time: 0.6318 data: 0.0070 max mem: 4400\n",
"Epoch: [1] [ 90/250] eta: 0:01:39 lr: 0.005000 loss: 0.4147 (0.4566) loss_classifier: 0.1612 (0.1827) loss_box_reg: 0.1991 (0.1985) loss_objectness: 0.0163 (0.0406) loss_rpn_box_reg: 0.0125 (0.0348) time: 0.6395 data: 0.0070 max mem: 4400\n",
"Epoch: [1] [100/250] eta: 0:01:34 lr: 0.005000 loss: 0.3852 (0.4562) loss_classifier: 0.1597 (0.1812) loss_box_reg: 0.1708 (0.1985) loss_objectness: 0.0162 (0.0407) loss_rpn_box_reg: 0.0125 (0.0358) time: 0.6478 data: 0.0065 max mem: 4400\n",
"Epoch: [1] [110/250] eta: 0:01:27 lr: 0.005000 loss: 0.4013 (0.4543) loss_classifier: 0.1404 (0.1778) loss_box_reg: 0.1708 (0.1998) loss_objectness: 0.0290 (0.0407) loss_rpn_box_reg: 0.0287 (0.0360) time: 0.6278 data: 0.0066 max mem: 4400\n",
"Epoch: [1] [120/250] eta: 0:01:21 lr: 0.005000 loss: 0.4627 (0.4577) loss_classifier: 0.1577 (0.1810) loss_box_reg: 0.2247 (0.2018) loss_objectness: 0.0277 (0.0394) loss_rpn_box_reg: 0.0148 (0.0354) time: 0.6221 data: 0.0067 max mem: 4400\n",
"Epoch: [1] [130/250] eta: 0:01:15 lr: 0.005000 loss: 0.4507 (0.4546) loss_classifier: 0.1595 (0.1793) loss_box_reg: 0.1688 (0.2000) loss_objectness: 0.0304 (0.0399) loss_rpn_box_reg: 0.0142 (0.0355) time: 0.6517 data: 0.0065 max mem: 4400\n",
"Epoch: [1] [140/250] eta: 0:01:08 lr: 0.005000 loss: 0.3684 (0.4500) loss_classifier: 0.1507 (0.1780) loss_box_reg: 0.1688 (0.1984) loss_objectness: 0.0249 (0.0386) loss_rpn_box_reg: 0.0216 (0.0349) time: 0.6242 data: 0.0067 max mem: 4400\n",
"Epoch: [1] [150/250] eta: 0:01:02 lr: 0.005000 loss: 0.3214 (0.4417) loss_classifier: 0.1337 (0.1745) loss_box_reg: 0.1418 (0.1952) loss_objectness: 0.0187 (0.0375) loss_rpn_box_reg: 0.0112 (0.0346) time: 0.5884 data: 0.0068 max mem: 4400\n",
"Epoch: [1] [160/250] eta: 0:00:55 lr: 0.005000 loss: 0.3145 (0.4387) loss_classifier: 0.1253 (0.1726) loss_box_reg: 0.1309 (0.1941) loss_objectness: 0.0194 (0.0372) loss_rpn_box_reg: 0.0238 (0.0348) time: 0.5747 data: 0.0066 max mem: 4400\n",
"Epoch: [1] [170/250] eta: 0:00:49 lr: 0.005000 loss: 0.3692 (0.4370) loss_classifier: 0.1361 (0.1720) loss_box_reg: 0.1405 (0.1940) loss_objectness: 0.0231 (0.0370) loss_rpn_box_reg: 0.0238 (0.0339) time: 0.5676 data: 0.0068 max mem: 4400\n",
"Epoch: [1] [180/250] eta: 0:00:43 lr: 0.005000 loss: 0.4429 (0.4432) loss_classifier: 0.1848 (0.1734) loss_box_reg: 0.2007 (0.1971) loss_objectness: 0.0277 (0.0368) loss_rpn_box_reg: 0.0147 (0.0359) time: 0.6171 data: 0.0068 max mem: 4400\n",
"Epoch: [1] [190/250] eta: 0:00:37 lr: 0.005000 loss: 0.4324 (0.4437) loss_classifier: 0.1839 (0.1742) loss_box_reg: 0.2128 (0.1977) loss_objectness: 0.0277 (0.0362) loss_rpn_box_reg: 0.0144 (0.0357) time: 0.6573 data: 0.0067 max mem: 4400\n",
"Epoch: [1] [200/250] eta: 0:00:30 lr: 0.005000 loss: 0.4066 (0.4450) loss_classifier: 0.1546 (0.1747) loss_box_reg: 0.2128 (0.1983) loss_objectness: 0.0236 (0.0357) loss_rpn_box_reg: 0.0137 (0.0362) time: 0.6202 data: 0.0066 max mem: 4400\n",
"Epoch: [1] [210/250] eta: 0:00:24 lr: 0.005000 loss: 0.3613 (0.4456) loss_classifier: 0.1479 (0.1745) loss_box_reg: 0.1647 (0.1983) loss_objectness: 0.0265 (0.0360) loss_rpn_box_reg: 0.0116 (0.0368) time: 0.6014 data: 0.0066 max mem: 4400\n",
"Epoch: [1] [220/250] eta: 0:00:18 lr: 0.005000 loss: 0.3613 (0.4454) loss_classifier: 0.1539 (0.1747) loss_box_reg: 0.1647 (0.1981) loss_objectness: 0.0215 (0.0354) loss_rpn_box_reg: 0.0175 (0.0372) time: 0.5941 data: 0.0067 max mem: 4400\n",
"Epoch: [1] [230/250] eta: 0:00:12 lr: 0.005000 loss: 0.2813 (0.4400) loss_classifier: 0.1284 (0.1726) loss_box_reg: 0.1242 (0.1957) loss_objectness: 0.0129 (0.0348) loss_rpn_box_reg: 0.0175 (0.0369) time: 0.6086 data: 0.0066 max mem: 4400\n",
"Epoch: [1] [240/250] eta: 0:00:06 lr: 0.005000 loss: 0.2813 (0.4358) loss_classifier: 0.1191 (0.1710) loss_box_reg: 0.1294 (0.1945) loss_objectness: 0.0161 (0.0342) loss_rpn_box_reg: 0.0126 (0.0361) time: 0.6469 data: 0.0065 max mem: 4400\n",
"Epoch: [1] [249/250] eta: 0:00:00 lr: 0.005000 loss: 0.3646 (0.4367) loss_classifier: 0.1397 (0.1709) loss_box_reg: 0.1754 (0.1956) loss_objectness: 0.0184 (0.0348) loss_rpn_box_reg: 0.0110 (0.0355) time: 0.6435 data: 0.0065 max mem: 4400\n",
"Epoch: [1] Total time: 0:02:35 (0.6207 s / it)\n",
"creating index...\n",
"index created!\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n",
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Test: [ 0/207] eta: 0:01:08 model_time: 0.1723 (0.1723) evaluator_time: 0.0091 (0.0091) time: 0.3287 data: 0.1455 max mem: 4400\n",
"Test: [100/207] eta: 0:00:13 model_time: 0.1208 (0.1177) evaluator_time: 0.0029 (0.0056) time: 0.1285 data: 0.0034 max mem: 4400\n",
"Test: [200/207] eta: 0:00:00 model_time: 0.1131 (0.1174) evaluator_time: 0.0025 (0.0050) time: 0.1211 data: 0.0033 max mem: 4400\n",
"Test: [206/207] eta: 0:00:00 model_time: 0.1138 (0.1174) evaluator_time: 0.0038 (0.0052) time: 0.1240 data: 0.0033 max mem: 4400\n",
"Test: Total time: 0:00:26 (0.1283 s / it)\n",
"Averaged stats: model_time: 0.1138 (0.1174) evaluator_time: 0.0038 (0.0052)\n",
"Accumulating evaluation results...\n",
"DONE (t=0.13s).\n",
"IoU metric: bbox\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.281\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.497\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.302\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.128\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.259\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.345\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.263\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.474\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.508\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.363\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.509\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.595\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n",
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch: [2] [ 0/250] eta: 0:03:35 lr: 0.005000 loss: 0.2030 (0.2030) loss_classifier: 0.0742 (0.0742) loss_box_reg: 0.0981 (0.0981) loss_objectness: 0.0236 (0.0236) loss_rpn_box_reg: 0.0070 (0.0070) time: 0.8602 data: 0.2654 max mem: 4400\n",
"Epoch: [2] [ 10/250] eta: 0:02:32 lr: 0.005000 loss: 0.2678 (0.3346) loss_classifier: 0.1189 (0.1501) loss_box_reg: 0.1305 (0.1487) loss_objectness: 0.0176 (0.0219) loss_rpn_box_reg: 0.0094 (0.0138) time: 0.6360 data: 0.0300 max mem: 4400\n",
"Epoch: [2] [ 20/250] eta: 0:02:23 lr: 0.005000 loss: 0.2678 (0.3265) loss_classifier: 0.1222 (0.1480) loss_box_reg: 0.1305 (0.1383) loss_objectness: 0.0154 (0.0217) loss_rpn_box_reg: 0.0147 (0.0186) time: 0.6119 data: 0.0065 max mem: 4400\n",
"Epoch: [2] [ 30/250] eta: 0:02:18 lr: 0.005000 loss: 0.3089 (0.3398) loss_classifier: 0.1295 (0.1498) loss_box_reg: 0.1501 (0.1533) loss_objectness: 0.0154 (0.0197) loss_rpn_box_reg: 0.0124 (0.0170) time: 0.6232 data: 0.0066 max mem: 4400\n",
"Epoch: [2] [ 40/250] eta: 0:02:10 lr: 0.005000 loss: 0.4012 (0.3535) loss_classifier: 0.1544 (0.1528) loss_box_reg: 0.2280 (0.1654) loss_objectness: 0.0083 (0.0182) loss_rpn_box_reg: 0.0113 (0.0171) time: 0.6179 data: 0.0065 max mem: 4400\n",
"Epoch: [2] [ 50/250] eta: 0:02:02 lr: 0.005000 loss: 0.4555 (0.3739) loss_classifier: 0.1544 (0.1556) loss_box_reg: 0.2198 (0.1762) loss_objectness: 0.0121 (0.0189) loss_rpn_box_reg: 0.0126 (0.0231) time: 0.5944 data: 0.0064 max mem: 4400\n",
"Epoch: [2] [ 60/250] eta: 0:01:58 lr: 0.005000 loss: 0.4607 (0.3821) loss_classifier: 0.1651 (0.1577) loss_box_reg: 0.2198 (0.1833) loss_objectness: 0.0161 (0.0185) loss_rpn_box_reg: 0.0105 (0.0227) time: 0.6251 data: 0.0064 max mem: 4401\n",
"Epoch: [2] [ 70/250] eta: 0:01:52 lr: 0.005000 loss: 0.4061 (0.3902) loss_classifier: 0.1718 (0.1623) loss_box_reg: 0.2065 (0.1846) loss_objectness: 0.0172 (0.0186) loss_rpn_box_reg: 0.0122 (0.0247) time: 0.6442 data: 0.0065 max mem: 4401\n",
"Epoch: [2] [ 80/250] eta: 0:01:46 lr: 0.005000 loss: 0.3456 (0.3744) loss_classifier: 0.1480 (0.1568) loss_box_reg: 0.1291 (0.1749) loss_objectness: 0.0167 (0.0186) loss_rpn_box_reg: 0.0134 (0.0241) time: 0.6319 data: 0.0065 max mem: 4401\n",
"Epoch: [2] [ 90/250] eta: 0:01:38 lr: 0.005000 loss: 0.3226 (0.3736) loss_classifier: 0.1227 (0.1560) loss_box_reg: 0.1452 (0.1744) loss_objectness: 0.0157 (0.0186) loss_rpn_box_reg: 0.0171 (0.0246) time: 0.5941 data: 0.0066 max mem: 4401\n",
"Epoch: [2] [100/250] eta: 0:01:32 lr: 0.005000 loss: 0.3428 (0.3759) loss_classifier: 0.1466 (0.1543) loss_box_reg: 0.1719 (0.1779) loss_objectness: 0.0183 (0.0186) loss_rpn_box_reg: 0.0185 (0.0251) time: 0.5790 data: 0.0066 max mem: 4401\n",
"Epoch: [2] [110/250] eta: 0:01:27 lr: 0.005000 loss: 0.3428 (0.3753) loss_classifier: 0.1355 (0.1535) loss_box_reg: 0.1755 (0.1758) loss_objectness: 0.0140 (0.0186) loss_rpn_box_reg: 0.0141 (0.0274) time: 0.6470 data: 0.0065 max mem: 4401\n",
"Epoch: [2] [120/250] eta: 0:01:21 lr: 0.005000 loss: 0.4148 (0.3833) loss_classifier: 0.1514 (0.1569) loss_box_reg: 0.1740 (0.1789) loss_objectness: 0.0164 (0.0189) loss_rpn_box_reg: 0.0152 (0.0285) time: 0.6660 data: 0.0067 max mem: 4401\n",
"Epoch: [2] [130/250] eta: 0:01:14 lr: 0.005000 loss: 0.4149 (0.3835) loss_classifier: 0.1687 (0.1578) loss_box_reg: 0.1765 (0.1772) loss_objectness: 0.0166 (0.0187) loss_rpn_box_reg: 0.0229 (0.0298) time: 0.6217 data: 0.0066 max mem: 4401\n",
"Epoch: [2] [140/250] eta: 0:01:08 lr: 0.005000 loss: 0.4996 (0.3905) loss_classifier: 0.1934 (0.1604) loss_box_reg: 0.1858 (0.1812) loss_objectness: 0.0138 (0.0188) loss_rpn_box_reg: 0.0157 (0.0301) time: 0.5995 data: 0.0066 max mem: 4401\n",
"Epoch: [2] [150/250] eta: 0:01:02 lr: 0.005000 loss: 0.3816 (0.3883) loss_classifier: 0.1685 (0.1580) loss_box_reg: 0.1860 (0.1809) loss_objectness: 0.0111 (0.0189) loss_rpn_box_reg: 0.0160 (0.0305) time: 0.6182 data: 0.0065 max mem: 4401\n",
"Epoch: [2] [160/250] eta: 0:00:55 lr: 0.005000 loss: 0.3247 (0.3888) loss_classifier: 0.1164 (0.1580) loss_box_reg: 0.1533 (0.1820) loss_objectness: 0.0085 (0.0188) loss_rpn_box_reg: 0.0232 (0.0301) time: 0.6283 data: 0.0065 max mem: 4401\n",
"Epoch: [2] [170/250] eta: 0:00:49 lr: 0.005000 loss: 0.3346 (0.3865) loss_classifier: 0.1255 (0.1569) loss_box_reg: 0.1665 (0.1806) loss_objectness: 0.0071 (0.0184) loss_rpn_box_reg: 0.0196 (0.0307) time: 0.6233 data: 0.0066 max mem: 4401\n",
"Epoch: [2] [180/250] eta: 0:00:43 lr: 0.005000 loss: 0.3093 (0.3840) loss_classifier: 0.1184 (0.1546) loss_box_reg: 0.1458 (0.1799) loss_objectness: 0.0098 (0.0183) loss_rpn_box_reg: 0.0119 (0.0312) time: 0.6190 data: 0.0066 max mem: 4401\n",
"Epoch: [2] [190/250] eta: 0:00:37 lr: 0.005000 loss: 0.3315 (0.3878) loss_classifier: 0.1270 (0.1566) loss_box_reg: 0.1589 (0.1811) loss_objectness: 0.0141 (0.0191) loss_rpn_box_reg: 0.0146 (0.0310) time: 0.6022 data: 0.0070 max mem: 4401\n",
"Epoch: [2] [200/250] eta: 0:00:30 lr: 0.005000 loss: 0.3494 (0.3882) loss_classifier: 0.1283 (0.1560) loss_box_reg: 0.1492 (0.1812) loss_objectness: 0.0205 (0.0194) loss_rpn_box_reg: 0.0145 (0.0316) time: 0.5932 data: 0.0074 max mem: 4401\n",
"Epoch: [2] [210/250] eta: 0:00:24 lr: 0.005000 loss: 0.4358 (0.3902) loss_classifier: 0.1283 (0.1561) loss_box_reg: 0.1408 (0.1811) loss_objectness: 0.0244 (0.0198) loss_rpn_box_reg: 0.0097 (0.0332) time: 0.6046 data: 0.0070 max mem: 4401\n",
"Epoch: [2] [220/250] eta: 0:00:18 lr: 0.005000 loss: 0.4121 (0.3934) loss_classifier: 0.1361 (0.1569) loss_box_reg: 0.2099 (0.1836) loss_objectness: 0.0162 (0.0200) loss_rpn_box_reg: 0.0153 (0.0330) time: 0.6274 data: 0.0066 max mem: 4401\n",
"Epoch: [2] [230/250] eta: 0:00:12 lr: 0.005000 loss: 0.3618 (0.3916) loss_classifier: 0.1251 (0.1554) loss_box_reg: 0.2102 (0.1836) loss_objectness: 0.0104 (0.0197) loss_rpn_box_reg: 0.0167 (0.0329) time: 0.6337 data: 0.0065 max mem: 4401\n",
"Epoch: [2] [240/250] eta: 0:00:06 lr: 0.005000 loss: 0.2496 (0.3902) loss_classifier: 0.0994 (0.1550) loss_box_reg: 0.1357 (0.1835) loss_objectness: 0.0076 (0.0193) loss_rpn_box_reg: 0.0126 (0.0324) time: 0.6587 data: 0.0068 max mem: 4401\n",
"Epoch: [2] [249/250] eta: 0:00:00 lr: 0.005000 loss: 0.2581 (0.3889) loss_classifier: 0.1132 (0.1550) loss_box_reg: 0.1357 (0.1830) loss_objectness: 0.0084 (0.0192) loss_rpn_box_reg: 0.0121 (0.0317) time: 0.6822 data: 0.0068 max mem: 4401\n",
"Epoch: [2] Total time: 0:02:36 (0.6244 s / it)\n",
"creating index...\n",
"index created!\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n",
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Test: [ 0/207] eta: 0:01:05 model_time: 0.1593 (0.1593) evaluator_time: 0.0059 (0.0059) time: 0.3145 data: 0.1474 max mem: 4401\n",
"Test: [100/207] eta: 0:00:13 model_time: 0.1216 (0.1176) evaluator_time: 0.0018 (0.0031) time: 0.1269 data: 0.0034 max mem: 4401\n",
"Test: [200/207] eta: 0:00:00 model_time: 0.1130 (0.1175) evaluator_time: 0.0017 (0.0030) time: 0.1205 data: 0.0033 max mem: 4401\n",
"Test: [206/207] eta: 0:00:00 model_time: 0.1130 (0.1174) evaluator_time: 0.0024 (0.0032) time: 0.1228 data: 0.0034 max mem: 4401\n",
"Test: Total time: 0:00:26 (0.1262 s / it)\n",
"Averaged stats: model_time: 0.1130 (0.1174) evaluator_time: 0.0024 (0.0032)\n",
"Accumulating evaluation results...\n",
"DONE (t=0.09s).\n",
"IoU metric: bbox\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.341\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.547\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.363\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.186\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.319\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.433\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.304\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.533\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.553\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.438\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.537\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.680\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n",
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch: [3] [ 0/250] eta: 0:03:52 lr: 0.000500 loss: 0.3354 (0.3354) loss_classifier: 0.1450 (0.1450) loss_box_reg: 0.1744 (0.1744) loss_objectness: 0.0091 (0.0091) loss_rpn_box_reg: 0.0069 (0.0069) time: 0.9287 data: 0.2996 max mem: 4401\n",
"Epoch: [3] [ 10/250] eta: 0:02:30 lr: 0.000500 loss: 0.3590 (0.3596) loss_classifier: 0.1284 (0.1324) loss_box_reg: 0.1657 (0.1712) loss_objectness: 0.0196 (0.0204) loss_rpn_box_reg: 0.0134 (0.0354) time: 0.6271 data: 0.0325 max mem: 4401\n",
"Epoch: [3] [ 20/250] eta: 0:02:22 lr: 0.000500 loss: 0.3198 (0.3061) loss_classifier: 0.1136 (0.1162) loss_box_reg: 0.1374 (0.1451) loss_objectness: 0.0121 (0.0153) loss_rpn_box_reg: 0.0134 (0.0295) time: 0.6062 data: 0.0064 max mem: 4401\n",
"Epoch: [3] [ 30/250] eta: 0:02:20 lr: 0.000500 loss: 0.3198 (0.3471) loss_classifier: 0.1263 (0.1277) loss_box_reg: 0.1078 (0.1740) loss_objectness: 0.0081 (0.0147) loss_rpn_box_reg: 0.0154 (0.0307) time: 0.6434 data: 0.0067 max mem: 4401\n",
"Epoch: [3] [ 40/250] eta: 0:02:11 lr: 0.000500 loss: 0.4186 (0.3519) loss_classifier: 0.1500 (0.1302) loss_box_reg: 0.1878 (0.1738) loss_objectness: 0.0082 (0.0146) loss_rpn_box_reg: 0.0171 (0.0333) time: 0.6285 data: 0.0066 max mem: 4401\n",
"Epoch: [3] [ 50/250] eta: 0:02:05 lr: 0.000500 loss: 0.2651 (0.3384) loss_classifier: 0.0985 (0.1239) loss_box_reg: 0.1453 (0.1643) loss_objectness: 0.0080 (0.0175) loss_rpn_box_reg: 0.0109 (0.0327) time: 0.6171 data: 0.0068 max mem: 4401\n",
"Epoch: [3] [ 60/250] eta: 0:01:58 lr: 0.000500 loss: 0.2385 (0.3261) loss_classifier: 0.0935 (0.1192) loss_box_reg: 0.1164 (0.1576) loss_objectness: 0.0068 (0.0166) loss_rpn_box_reg: 0.0112 (0.0327) time: 0.6215 data: 0.0069 max mem: 4401\n",
"Epoch: [3] [ 70/250] eta: 0:01:51 lr: 0.000500 loss: 0.2588 (0.3302) loss_classifier: 0.0986 (0.1237) loss_box_reg: 0.1287 (0.1604) loss_objectness: 0.0084 (0.0164) loss_rpn_box_reg: 0.0108 (0.0298) time: 0.5899 data: 0.0067 max mem: 4401\n",
"Epoch: [3] [ 80/250] eta: 0:01:45 lr: 0.000500 loss: 0.2999 (0.3289) loss_classifier: 0.0924 (0.1234) loss_box_reg: 0.1696 (0.1622) loss_objectness: 0.0083 (0.0155) loss_rpn_box_reg: 0.0077 (0.0278) time: 0.6020 data: 0.0067 max mem: 4401\n",
"Epoch: [3] [ 90/250] eta: 0:01:39 lr: 0.000500 loss: 0.2691 (0.3220) loss_classifier: 0.0815 (0.1220) loss_box_reg: 0.1510 (0.1586) loss_objectness: 0.0054 (0.0146) loss_rpn_box_reg: 0.0066 (0.0268) time: 0.6222 data: 0.0069 max mem: 4401\n",
"Epoch: [3] [100/250] eta: 0:01:32 lr: 0.000500 loss: 0.2969 (0.3313) loss_classifier: 0.0992 (0.1239) loss_box_reg: 0.1518 (0.1621) loss_objectness: 0.0081 (0.0148) loss_rpn_box_reg: 0.0122 (0.0304) time: 0.6207 data: 0.0068 max mem: 4401\n",
"Epoch: [3] [110/250] eta: 0:01:26 lr: 0.000500 loss: 0.3117 (0.3274) loss_classifier: 0.0989 (0.1221) loss_box_reg: 0.1554 (0.1603) loss_objectness: 0.0095 (0.0144) loss_rpn_box_reg: 0.0120 (0.0306) time: 0.6052 data: 0.0068 max mem: 4401\n",
"Epoch: [3] [120/250] eta: 0:01:19 lr: 0.000500 loss: 0.2960 (0.3287) loss_classifier: 0.1061 (0.1217) loss_box_reg: 0.1614 (0.1621) loss_objectness: 0.0076 (0.0142) loss_rpn_box_reg: 0.0106 (0.0307) time: 0.5862 data: 0.0067 max mem: 4401\n",
"Epoch: [3] [130/250] eta: 0:01:13 lr: 0.000500 loss: 0.2624 (0.3192) loss_classifier: 0.0816 (0.1182) loss_box_reg: 0.1548 (0.1582) loss_objectness: 0.0054 (0.0136) loss_rpn_box_reg: 0.0097 (0.0292) time: 0.6022 data: 0.0066 max mem: 4401\n",
"Epoch: [3] [140/250] eta: 0:01:07 lr: 0.000500 loss: 0.2624 (0.3223) loss_classifier: 0.0943 (0.1193) loss_box_reg: 0.1460 (0.1601) loss_objectness: 0.0069 (0.0136) loss_rpn_box_reg: 0.0102 (0.0293) time: 0.6332 data: 0.0065 max mem: 4401\n",
"Epoch: [3] [150/250] eta: 0:01:01 lr: 0.000500 loss: 0.3056 (0.3211) loss_classifier: 0.1139 (0.1187) loss_box_reg: 0.1678 (0.1602) loss_objectness: 0.0126 (0.0136) loss_rpn_box_reg: 0.0092 (0.0286) time: 0.6537 data: 0.0065 max mem: 4401\n",
"Epoch: [3] [160/250] eta: 0:00:55 lr: 0.000500 loss: 0.2875 (0.3213) loss_classifier: 0.0929 (0.1180) loss_box_reg: 0.1600 (0.1617) loss_objectness: 0.0104 (0.0134) loss_rpn_box_reg: 0.0080 (0.0282) time: 0.6329 data: 0.0071 max mem: 4401\n",
"Epoch: [3] [170/250] eta: 0:00:49 lr: 0.000500 loss: 0.2942 (0.3189) loss_classifier: 0.0995 (0.1175) loss_box_reg: 0.1735 (0.1613) loss_objectness: 0.0044 (0.0129) loss_rpn_box_reg: 0.0113 (0.0272) time: 0.6375 data: 0.0072 max mem: 4401\n",
"Epoch: [3] [180/250] eta: 0:00:43 lr: 0.000500 loss: 0.2773 (0.3176) loss_classifier: 0.1070 (0.1169) loss_box_reg: 0.1463 (0.1608) loss_objectness: 0.0038 (0.0127) loss_rpn_box_reg: 0.0140 (0.0272) time: 0.6413 data: 0.0067 max mem: 4401\n",
"Epoch: [3] [190/250] eta: 0:00:37 lr: 0.000500 loss: 0.2640 (0.3150) loss_classifier: 0.1010 (0.1160) loss_box_reg: 0.1205 (0.1588) loss_objectness: 0.0065 (0.0125) loss_rpn_box_reg: 0.0216 (0.0276) time: 0.6065 data: 0.0065 max mem: 4401\n",
"Epoch: [3] [200/250] eta: 0:00:31 lr: 0.000500 loss: 0.2203 (0.3109) loss_classifier: 0.0895 (0.1143) loss_box_reg: 0.1031 (0.1571) loss_objectness: 0.0051 (0.0122) loss_rpn_box_reg: 0.0082 (0.0273) time: 0.6428 data: 0.0064 max mem: 4401\n",
"Epoch: [3] [210/250] eta: 0:00:24 lr: 0.000500 loss: 0.2134 (0.3106) loss_classifier: 0.0739 (0.1139) loss_box_reg: 0.1254 (0.1575) loss_objectness: 0.0045 (0.0122) loss_rpn_box_reg: 0.0062 (0.0269) time: 0.6389 data: 0.0068 max mem: 4401\n",
"Epoch: [3] [220/250] eta: 0:00:18 lr: 0.000500 loss: 0.2134 (0.3077) loss_classifier: 0.0802 (0.1135) loss_box_reg: 0.1171 (0.1559) loss_objectness: 0.0097 (0.0122) loss_rpn_box_reg: 0.0087 (0.0262) time: 0.5995 data: 0.0070 max mem: 4401\n",
"Epoch: [3] [230/250] eta: 0:00:12 lr: 0.000500 loss: 0.2028 (0.3019) loss_classifier: 0.0658 (0.1111) loss_box_reg: 0.0987 (0.1533) loss_objectness: 0.0056 (0.0118) loss_rpn_box_reg: 0.0066 (0.0256) time: 0.6096 data: 0.0066 max mem: 4401\n",
"Epoch: [3] [240/250] eta: 0:00:06 lr: 0.000500 loss: 0.1990 (0.3019) loss_classifier: 0.0799 (0.1111) loss_box_reg: 0.1058 (0.1530) loss_objectness: 0.0060 (0.0120) loss_rpn_box_reg: 0.0057 (0.0259) time: 0.6291 data: 0.0065 max mem: 4401\n",
"Epoch: [3] [249/250] eta: 0:00:00 lr: 0.000500 loss: 0.2175 (0.3016) loss_classifier: 0.0886 (0.1113) loss_box_reg: 0.1263 (0.1528) loss_objectness: 0.0089 (0.0118) loss_rpn_box_reg: 0.0076 (0.0257) time: 0.6131 data: 0.0065 max mem: 4401\n",
"Epoch: [3] Total time: 0:02:35 (0.6209 s / it)\n",
"creating index...\n",
"index created!\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n",
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Test: [ 0/207] eta: 0:01:08 model_time: 0.1713 (0.1713) evaluator_time: 0.0073 (0.0073) time: 0.3321 data: 0.1515 max mem: 4401\n",
"Test: [100/207] eta: 0:00:13 model_time: 0.1211 (0.1176) evaluator_time: 0.0018 (0.0031) time: 0.1284 data: 0.0035 max mem: 4401\n",
"Test: [200/207] eta: 0:00:00 model_time: 0.1128 (0.1175) evaluator_time: 0.0016 (0.0030) time: 0.1211 data: 0.0037 max mem: 4401\n",
"Test: [206/207] eta: 0:00:00 model_time: 0.1129 (0.1174) evaluator_time: 0.0026 (0.0032) time: 0.1239 data: 0.0037 max mem: 4401\n",
"Test: Total time: 0:00:26 (0.1267 s / it)\n",
"Averaged stats: model_time: 0.1129 (0.1174) evaluator_time: 0.0026 (0.0032)\n",
"Accumulating evaluation results...\n",
"DONE (t=0.09s).\n",
"IoU metric: bbox\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.359\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.578\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.378\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.215\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.346\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.460\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.313\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.541\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.570\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.436\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.550\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.690\n"
]
}
],
"source": [
"do_training(model, torch_dataset, torch_dataset_test, num_epochs=4)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Lj3vLT1eXFnk"
},
"source": [
"One of the main draws of FiftyOne is the ability to find failure modes of your model. The [built-in evaluation protocols](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html#evaluating-models) will help you find where your model got things right and where it got things wrong. Before we can evaluate the model, we need to run it on our test set and store the results in FiftyOne. Doing this is fairly simple and just requires us to run inference for the test images, get their corresponding [FiftyOne samples](https://voxel51.com/docs/fiftyone/user_guide/using_datasets.html#samples), and add a new field called `predictions` to each sample to store the detections."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"id": "zpg0WNh_WyzQ"
},
"outputs": [],
"source": [
"import fiftyone as fo\n",
"\n",
"def convert_torch_predictions(preds, det_id, s_id, w, h, classes):\n",
" # Convert the outputs of the torch model into a FiftyOne Detections object\n",
" dets = []\n",
" for bbox, label, score in zip(\n",
" preds[\"boxes\"].cpu().detach().numpy(), \n",
" preds[\"labels\"].cpu().detach().numpy(), \n",
" preds[\"scores\"].cpu().detach().numpy()\n",
" ):\n",
" # Parse prediction into FiftyOne Detection object\n",
" x0,y0,x1,y1 = bbox\n",
" coco_obj = fouc.COCOObject(det_id, s_id, int(label), [x0, y0, x1-x0, y1-y0])\n",
" det = coco_obj.to_detection((w,h), classes)\n",
" det[\"confidence\"] = float(score)\n",
" dets.append(det)\n",
" det_id += 1\n",
" \n",
" detections = fo.Detections(detections=dets)\n",
" \n",
" return detections, det_id\n",
"\n",
"def add_detections(model, torch_dataset, view, field_name=\"predictions\"):\n",
" # Run inference on a dataset and add results to FiftyOne\n",
" torch.set_num_threads(1)\n",
" device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n",
" print(\"Using device %s\" % device)\n",
"\n",
" model.eval()\n",
" model.to(device)\n",
" image_paths = torch_dataset.img_paths\n",
" classes = torch_dataset.classes\n",
" det_id = 0\n",
" \n",
" with fo.ProgressBar() as pb:\n",
" for img, targets in pb(torch_dataset):\n",
" # Get FiftyOne sample indexed by unique image filepath\n",
" img_id = int(targets[\"image_id\"][0])\n",
" img_path = image_paths[img_id]\n",
" sample = view[img_path]\n",
" s_id = sample.id\n",
" w = sample.metadata[\"width\"]\n",
" h = sample.metadata[\"height\"]\n",
" \n",
" # Inference\n",
" preds = model(img.unsqueeze(0).to(device))[0]\n",
" \n",
" detections, det_id = convert_torch_predictions(\n",
" preds, \n",
" det_id, \n",
" s_id, \n",
" w, \n",
" h, \n",
" classes,\n",
" )\n",
" \n",
" sample[field_name] = detections\n",
" sample.save()"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "CkzG1i3AW1O7",
"outputId": "ec7971c3-66ef-4a57-e710-248cb53dee8e"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Using device cuda\n",
" 100% |█████████████████| 207/207 [31.5s elapsed, 0s remaining, 6.5 samples/s] \n"
]
}
],
"source": [
"add_detections(model, torch_dataset_test, fo_dataset, field_name=\"predictions\")"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "gEw_hYOzW3yz",
"outputId": "337e47d6-d784-4538-eb3a-1cb5bfa901dd"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Evaluating detections...\n",
" 100% |█████████████████| 207/207 [5.5s elapsed, 0s remaining, 36.5 samples/s] \n",
"Performing IoU sweep...\n",
" 100% |█████████████████| 207/207 [8.1s elapsed, 0s remaining, 22.2 samples/s] \n"
]
}
],
"source": [
"results = fo.evaluate_detections(\n",
" test_view, \n",
" \"predictions\", \n",
" classes=[\"car\", \"bus\", \"truck\"], \n",
" eval_key=\"eval\", \n",
" compute_mAP=True\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6oet0wg9XVlA"
},
"source": [
"The [DetectionResults](https://voxel51.com/docs/fiftyone/api/fiftyone.utils.eval.detection.html#fiftyone.utils.eval.detection.DetectionResults) object that is returned stores information like the mAP and contains functions that let you [plot confusion matrices, precision-recall curves, and more](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html). Also, these evaluation runs are tracked in FiftyOne and can be managed through functions like [list_evaluations()](https://voxel51.com/docs/fiftyone/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.list_evaluations)."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "7uYdXrhgYdJ_",
"outputId": "2eb792e9-342f-4dc8-f6c0-e1503d8bf193"
},
"outputs": [
{
"data": {
"text/plain": [
"0.3598547617444416"
]
},
"execution_count": 23,
"metadata": {
"tags": []
},
"output_type": "execute_result"
}
],
"source": [
"results.mAP()"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "VcJBOM76aJPR",
"outputId": "ac452527-7608-4e8d-f57e-18a0470acd30"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" precision recall f1-score support\n",
"\n",
" car 0.20 0.87 0.32 610\n",
" bus 0.13 0.88 0.23 68\n",
" truck 0.10 0.88 0.18 112\n",
"\n",
" micro avg 0.17 0.87 0.28 790\n",
" macro avg 0.14 0.88 0.24 790\n",
"weighted avg 0.18 0.87 0.29 790\n",
"\n"
]
}
],
"source": [
"results.print_report()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nddFfGSnXo7i"
},
"source": [
"By default, objects are only matched with other objects of the same class. In order to get an interesting confusion matrix, we need to match interclass objects by setting `classwise=False`."
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "4_53aCMna2Vt",
"outputId": "4db71f31-73e3-4036-f623-efd8e2ac85bf"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Evaluating detections...\n",
" 100% |█████████████████| 207/207 [6.3s elapsed, 0s remaining, 30.2 samples/s] \n",
"Performing IoU sweep...\n",
" 100% |█████████████████| 207/207 [9.7s elapsed, 0s remaining, 16.9 samples/s] \n"
]
}
],
"source": [
"results_interclass = fo.evaluate_detections(\n",
" test_view, \n",
" \"predictions\", \n",
" classes=[\"car\", \"bus\", \"truck\"], \n",
" compute_mAP=True, \n",
" classwise=False\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 297
},
"id": "Nbqf-NuAZ7Ps",
"outputId": "571cd947-c94a-4b9a-ed93-2330fbddea7e"
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAVAAAAEYCAYAAAAK467YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5xcddn//9d7S3rPJiEVQpMOgdBRAyggosgtCsIt5Q6/gCCCClL09kZv9YeIoHRzi0iRjghSEpoIRFqCdAiEkk7Cpvdkd6/vH+csDCHZnZ2dyZnNvp+Pxzx25pwz51wzhGs+7Xw+igjMzKzlKrIOwMysrXICNTMrkBOomVmBnEDNzArkBGpmViAnUDOzAjmBmtlGS9L7kl6R9KKkiem2PpIelvR2+rd3ul2SLpM0RdLLknZt9vwb6zjQmj6VsdnQ6qzDKGtvvdwl6xDKnqoqsw6hrK2oX8LqhpUq5jkP3r9rzJtfn9exk15eNT4iDlnffknvAyMjojZn20XA/Ii4UNK5QO+IOEfSocDpwKHAnsDvI2LPpq5flVeUbdBmQ6t5bvzQrMMoawcP2iXrEMpeZe++WYdQ1p5ecFfRzzlvfj3PjR+W17GVA9+uKeAShwOj0ufXA48D56Tbb4ikVPmMpF6SBkbE7PWdaKNNoGbWNgXBmqjL9/Caxqp5amxEjP3E6eAhSQH8Id03ICcpfgAMSJ8PBqbnvHdGus0J1MzahgAayLtpsTYiRjaxf7+ImCmpP/CwpDc/ca2ISJNrQZxAzazsNNBQlPNExMz071xJdwN7AHMaq+aSBgJz08NnArntfkPSbevlXngzKytBUB/5PZoiqauk7o3PgYOAV4F7gePTw44H7kmf3wscl/bG7wUsaqr9E1wCNbMy1IIqfFMGAHdLgiTX3RwR4yQ9D9wuaTQwFfhmevwDJD3wU4DlwInNXcAJ1MzKSgD1RUigEfEusPM6ts8DDlzH9gBOa8k1nEDNrKwEsCaK0wZaak6gZlZ22kb6dAI1szITRFGq8BuCE6iZlZeA+raRP51Azay8JAPp2wYnUDMrM6Keos5PUjJOoGZWVpJeeCdQM7MWS8aBOoGamRWkwSVQM7OWcwnUzKxAgahvI/McOYGaWdlxFd7MrACBWB1tYy0qJ1AzKyvJQHpX4c3MCuJOJDOzAkSI+nAJ1MysIA0ugZqZtVzSidQ2UlPbiNLM2g13IpmZtUK9x4GambWc70QyM2uFBvfCm5m1XDKZiBOomVmLBWKNb+Vs+47bYzs6d6unogIqq4Irxr31if0RcPV/D+a5x3rQqXMDP7x0GlvttKJV11y8oJJfnbIZc2Z0YMCQ1fz4D+/TvVc9j/21N7df2Z8I6Ny1gdMvnM4W269s1bXKxQ8umcaeX1jCwtoqTj7gM1mHUxZqBqzkh798nd59VxMhxt01iHv+MpThWy/hu/89mc5d6pkzqxMXnbs9K5ZtXP8bR9BmBtK3jSgzdNEdU7j6kcmfSp4Azz/WnZnvdeS6CW9wxkXTufy8IXmf96V/dePiM4d9avvtV/RnxH5LuG7CG4zYbwm3XdEfgAFDV/Gbu6bwh8cmc+z3P+D3Pxpa+IcqMw/d1ocfHzs86zDKSn29+ONvt+KUI/biB/+5G4cdNYOhmy/jjAve5LrfbcGpX9+Tfz3ajyNPmJZ1qCUgGvJ8ZK3NJlBJmf/sPj2+J184cj4SbLvbcpYtqmTenCSsO67qx+lf2ppTDvwMN/xmk5ad85vzAfjCN+fz9LieAGy/+3K696oHYJtdl1M7u7rInyY7rz7bjSULMv/PWVYW1HbknTe6A7BieRXT3utKTf9VDN50Oa9O6gXAv5/uw75fmJtlmCURJCXQfB5Zyz4CQNJxkl6W9JKkGyV9RdKzkv4t6RFJA9LjLkj3TwBuLH1gwfnf2oLTDt6aB27q+6ndtR9U02/Qmo9e1wxaw7wPqpn0eFIyveyBt7jq4cm8/UpnXnmma16XXFBbTd8BdQD06V/HgtpPJ8pxt/Rh9/2XFPihrK3pP2gFW2yzhDdf6cHUd7qy9/61AHz2oLnUbLIq4+hKo56KvB5Zy/xnX9L2wE+AfSKiVlIfkh+hvSIiJJ0E/Aj4YfqW7YD9IqJ1jY15uORvU6gZuIaFtVWce/QWDN1yJTvutazZ9036Z3de+GcPTv1i0p63YnkFM9/tyI57LeN7X96KNasqWLG8giULK/nOF5JjRv9kFiNHfTIpSiDFJ7a9OKEb42/pyyV/e7tIn9LKWafOdfz4klcZe9FWrFhWxe9+ui2nnPsWR5/8Ps8+XkPdmuyrscUWyBMqt8ABwB0RUQsQEfMl7QjcJmkg0AF4L+f4e9eXPCWNAcYADBvc+o9WMzApXfaqqWPfQxbx5r+7fCKB1myyhg9nfVxCrJ1VTd9N1hDAUafP4cvfnvepc152f5L4XvpXNx6+vQ9n/e6TbVi9a9Ywb04VfQfUMW9OFb361n20793XO/G7s4byi5vepUef+lZ/PitvlVUN/PiSV3n8/gH869GkLXzG+135ySkjABi86XJ2/2xtliGWRLKscTmkpuZlXwZet8uBKyJiR+BkoFPOvvUWASNibESMjIiR/fq2bhjEyuUVLF9a8dHzSf/szmbbfLLXe6+DFvPInX2IgDcmdaFLj3r6Dqhj5OeXMP7WPqxYlry/dnY1C2vz+wex10GLeeT2PgA8cnsf9j54EQBzZ1Tz85OGc/ZlUxmyxcZZbbNcwZk/e5Pp73Xh7hs/7mzs2Wc1kNRMjh7zPg/cMTirAEtI1Of5yFo5pPnHgLslXRIR89IqfE9gZrr/+CyCWvBhFT8bnfQM19fB/kcsZPf9l3DfDUlb6GHHzWOPAxfz/KPdOXGfbemYDmMC2G3UEqZN6ciZX9kKSIYd/ejyqfSqaf66R313Dr88ZTPG3dqX/oOTYUwAf7l0E5YsqOSK85Le93UNq2qrzr1qKjvtvZSefeq4aeLr3PjbAYy/5dNtzu3JdiMWceBXPuC9t7py+e3PAXD9ZZszeNMVHHbUDAAmPNqPh/82MMswSyJoO3ciKSKaP6rUQUjHA2cD9cC/gbuBS4EFJAl294gYJekCYGlEXNzcOUfu3CmeG7/xDPUphYMH7ZJ1CGWvsqZ9J/LmPL3gLhat+bCoRcEhO/SM027fN69jz9/+wUkRMbKY12+JciiBEhHXA9evtfmedRx3wQYJyMwyE6E2UwJtG1GaWbtSzHGgkirTIZH3pa+Hp8Mkp0i6TVKHdHvH9PWUdP9mzZ3bCdTMykrjvfD5PPJ0BvBGzutfA5dGxJYkzYSj0+2jgQXp9kvT45rkBGpmZSXpRFJej+ZIGgJ8Gfhj+lokQyfvTA+5Hvha+vxwPm5KvBM4MD1+vcqiDdTMLFcR7zL6HcmNON3T132BhRHROMB6BtA4FmwwMB0gIuokLUqPX+9gW5dAzaysNN6JlGcJtEbSxJzHmMbzSDoMmBsRk0oVq0ugZlZ2WrCoXG0Tw5j2Bb4q6VCSm3F6AL8HekmqSkuhQ/h4zPlMYCgwI52sqCfw6dsJc7gEamZlJZkPVHk9mj5PnBcRQyJiM+Bo4LGIOBb4B3BketjxfDxk8l4+vnHnyPT4JgfKuwRqZmUlEHUNJZ2R/hzgVkm/ILlx59p0+7XAjZKmAPNJkm6TnEDNrOwU+z73iHgceDx9/i6wxzqOWQl8oyXndQI1s7LSOIypLXACNbMy03Zu5XQCNbOyUw7rHeXDCdTMykoErCltJ1LROIGaWVnxkh5mZq3gKryZWQHcC29m1gruhTczK0SeU9WVAydQMysrAdS5BGpm1nJuAzUzawUnUDOzAngcqJlZK3gcqJlZIcJVeDOzggRQ1+BeeDOzFnMbqJlZK4QTqJlZYdyJZGZWgHAnkplZ4VyFNzMriKh3L3y23nq5CwcP2iXrMMpa5dZbZB1C2Wt4b3rWIZS1qG8o/jlxFd7MrDCRtIO2BU6gZlZ23AtvZlaAwJ1IZmYF8p1IZmYFa2hwAjUza7EIV+HNzArmKryZWYE8jMnMrECuwpuZFcDzgZqZFcqdSGZmrdBG2kDbxpQnZtauRCivR1MkdZL0nKSXJL0m6Wfp9uGSnpU0RdJtkjqk2zumr6ek+zdrLs71lkAlXU4TvwMR8b3mTm5mVogi9cKvAg6IiKWSqoGnJD0I/AC4NCJulXQNMBq4Ov27ICK2lHQ08GvgqKYu0FQVfmJRPoKZWQsU6174iAhgafqyOn0EcABwTLr9euACkgR6ePoc4E7gCklKz7NO602gEXF97mtJXSJieYs/hZlZSwRE/rdy1kjKLeyNjYixjS8kVQKTgC2BK4F3gIURUZceMgMYnD4fDEwHiIg6SYuAvkDt+i7ebCeSpL2Ba4FuwDBJOwMnR8Sp+X0+M7MWyr8KXxsRI9d7moh6YBdJvYC7gW1aH9zH8ulE+h1wMDAvDegl4HPFDMLM7GP5dSC1pJofEQuBfwB7A70kNRYehwAz0+czgaEA6f6epHlvffLqhY+Itdc1qM8vbDOzAkSejyZI6peWPJHUGfgi8AZJIj0yPex44J70+b3pa9L9jzXV/gn5jQOdLmkfINKerDPSIMzMiq94A+kHAten7aAVwO0RcZ+k14FbJf0C+DdJEyXp3xslTQHmA0c3d4F8EugpwO9JGlhnAeOB01r6SczM8laEYUwR8TIwYh3b3wX2WMf2lcA3WnKNZhNoRNQCx7bkpGZmrdJGbuVstg1U0uaS/i7pQ0lzJd0jafMNEZyZtVNFaAPdEPLpRLoZuJ2kPWEQcAdwSymDMrN2LEhKoPk8MpZPAu0SETdGRF36uAnoVOrAzKz9isjvkbWm7oXvkz59UNK5wK0kvw1HAQ9sgNjMrL0qg+SYj6Y6kSaRfIzGcvLJOfsCOK9UQZlZO1cG1fN8NHUv/PANGYiZGQABasg6iPzkNaGypB2A7chp+4yIG0oVlJm1Z+XRQZSPfCYT+R9gFEkCfQD4EvAU4ARqZqXRRtpA8+mFPxI4EPggIk4Edia5yd7MrDTayDjQfKrwKyKiQVKdpB7AXNIZSyw/I0ct5pT/nUVlRfDgLX24/YoBWYeUuetuHceK5VXUN4iGenHGyQdw7k+fZfCwZP7bbt3WsHRpNaefdGDGkZaHI0Z/wCFHf0iEeP/Nzvz27OGsWbURr8hTBskxH/kk0InpjCb/R9IzvxR4urUXTtcbuS8idmjtucpZRUVw2q9mct7Rm1M7u5rLH3ibZ8b3ZNrbHkp77vc/y+JFHT96feHP9/zo+UnfeZlly6qzCKvs9B2wmsNPnMOYA3dk9aoKzr9yCqO+Mp+H76zJOrTSaBxI3wY0+xMWEadGxMKIuIZkOqjj06q85eEzI5Yz6/0OfDCtI3VrKnj8nl7sffCirMMqc8Fn95/JPx91RadRZWXQoVMDFZVBx84NzJuzcf+4qCG/R9aaGki/a1P7IuKFYlxf0l+AXYHXgOOA14GREVEraSRwcUSMkvR5klmhIPmN+lxELClCDCXVd5M1fDirw0eva2dXs82uXhklAn7xm6eIEA/+fTjj7vt41NwOO81j4YKOzJrZLcMIy8e8OR24c+wm3Pj0S6xaWcELT/bghSfdDVEOmqrC/7aJfY0LM7XWZ4DRETFB0p+AppYJOQs4LT22G7By7QMkjQHGAHSiSxHCs1I5+/TPM6+2Mz17reSXF09gxrTuvPpyUiX9/IHTedylz49061HH3gct5IT9dmLp4kp+fNU7HHBELY/dvZFW4QG1kTbQ9VbhI2L/Jh7FSJ4A0yNiQvr8JmC/Jo6dAFwi6XtAr5xFoXJjHhsRIyNiZDUdP32GDMz7oJp+g1Z/9Lpm4BpqZ2/c1a98zKvtDMCihZ14+qmBbL3tfAAqKhvY57OzeOIfg5t6e7syYr/FzJnekUXzq6mvq2DCuN5su9vS5t/Ylm1Ek4mU0tq/MwHU8XFcuQP3LwROAjoDEyQVdXGoUpn8YhcGD1/NgKGrqKpuYNThC3nmofZd/erYqY7Ondd89HzEyLlMfa8HACN2m8uMad2Z96FrEI3mzurANiOW0rFTPRDssu9ipk/pnHVYpZPvEKYyKKXmdSdSCQ2TtHdEPE2yTvNTQHdgN+BB4OuNB0raIiJeAV6RtDvJ6npvZhBzizTUiyt/PJhf3fwuFZXw0K19mPpW++6B7917FT/532cAqKxs4PFHhzLpuU0A+NwBM/jnY0OyDK/sTH6xG08+0Icr7n+d+nrxzmtdePDmflmHVVLl0EGUDzWzZlLpLpwMYxoHTCRJmK8D306fXwssBh4n6VAaJelyYH+ggaTD6YSIWLW+8/dQn9hTHkPYlMqtt8g6hLLX8N7a6ylarmfWjGNxw7yi1qU7Dh0aQ878fl7HvnvWDyc1taxxqeVzK6dIlvTYPCJ+LmkYsElEPNeaC0fE+6x7jeYnga3XcfzprbmembUhZVA9z0c+baBXkayl/K309RLgypJFZGbtmiL/R9byaQPdMyJ2lfRvgIhYIKlDc28yMytYGfSw5yOfBLomXVc5IFmsnqQd0sysNMqgdJmPfBLoZcDdQH9JvySZneknJY3KzNq1ttILn8+68H+RNIlkSjsBX4uIN0oemZm1T2XSvpmPfHrhhwHLgb/nbouIaaUMzMzasY0lgQL38/Hicp2A4cBkYPsSxmVm7dnGkkAjYsfc1+ksTU1N+mFm1iptpQrf4nvh02ns9mz2QDOzjVw+baA/yHlZQTJ356ySRWRm7dtGtqxx95zndSRtoneVJhwzMzaONtB0AH33iDhrA8VjZtb2E6ikqoiok7TvhgzIzNo30XY6kZoqgT5H0t75oqR7gTuAZY07I+KvJY7NzNqrNpJA8+mF7wTMI1kD6TDgK+lfM7PiK9JsTJKGSvqHpNclvSbpjHR7H0kPS3o7/ds73S5Jl0maIunlphbWbNRUCbR/2gP/Kh8PpM/5iGZmJVKcXvg64IcR8YKk7sAkSQ8DJwCPRsSFks4FzgXOAb4EbJU+9gSuppkhm00l0EqgG59MnI2cQM2sZIrRBhoRs4HZ6fMlkt4ABgOHA6PSw64nWfninHT7DZEs0/GMpF6SBqbnWaemEujsiPh5qz+FmVlL5Z9AayRNzHk9NiLGrn1QuoTQCOBZYEBOUvwAGJA+HwzkruEyI91WUAJtGzOamtnGpWUrbtY2tyaSpG4kY9fPjIjFySpF6aUiQiq8vNtUJ5JXZDOzTBRrSQ9J1STJ8y85I4fmSBqY7h8IzE23zwSG5rx9SLptvdabQCNifvPhmZmVQBHWhU8XxLwWeCMiLsnZdS9wfPr8eOCenO3Hpb3xewGLmmr/hOzXhTcz+5Qi3Qu/L8lS6a9IejHddj5wIXC7pNHAVOCb6b4HgEOBKSRzIJ/Y3AWcQM2svLSsDXT9p4l4ivX35XyqiTLtfT+tJddwAjWzsiLaTg+2E6iZlZ82MtLcCdTMys7GMJmImVk2NqIJlc3MNpyNaVljM7MNzgnUzKwwLoGamRXKCTRbqqqisk+/rMMoaw3vTW/+oHauYuvhWYdQ1jSlQ2nO6wRqZlaAwL3wZmaF2FgWlTMzy4YTqJlZYRRtI4M6gZpZeSnSbEwbghOomZUdt4GamRWoSBMql5wTqJmVH5dAzcwK4MlEzMxawQnUzKzlPJDezKw1PA7UzKwA4V54M7OCOYGamRWqbdTgnUDNrPy4E8nMrBCBO5HMzArlEqiZWQGEO5HMzAoT4Sq8mVmhXIU3MyuUE6iZWWFcAjUzK0QA9W0jgzqBmlnZaSsl0IqsAzAz+5TGnvjmHs2Q9CdJcyW9mrOtj6SHJb2d/u2dbpekyyRNkfSypF2bO78TqJmVHUV+jzz8GThkrW3nAo9GxFbAo+lrgC8BW6WPMcDVzZ3cCdTMyku04NHcqSKeAOavtflw4Pr0+fXA13K23xCJZ4BekgY2dX63gZpZWUlmpM+7EbRG0sSc12MjYmwz7xkQEbPT5x8AA9Lng4HpOcfNSLfNZj2cQM2s7Cj/XvjaiBhZ6HUiIqTCu6xchTez8lLEKvx6zGmsmqd/56bbZwJDc44bkm5bL5dAS6C6Qz0XXTeJ6uoGKquCpx7uz1+u3oLDjp7O146dxqBhKzj6859j8cIOWYdaNo4Y/QGHHP0hEeL9Nzvz27OHs2ZV+/5979p1NWf8YCKbbraIAH538e68+UYNAEccOZn/7+SXOPrrh7N4ccdsAy26kt8Lfy9wPHBh+veenO3flXQrsCewKKeqv04lSaCSegHHRMRVRTjXKOCsiDis1YFtIGtWV3DeSbuyckUVlVUNXPzniUx8qobXX+zJc0/syq//OCnrEMtK3wGrOfzEOYw5cEdWr6rg/CunMOor83n4zpqsQ8vUyaf+m0kTN+FX/7sPVVX1dOxYD0BNv+XsutsHzJ3TJeMIS6dY40Al3QKMImkrnQH8D0nivF3SaGAq8M308AeAQ4EpwHLgxObOX6oSaC/gVOATCVRSVUTUleiaZUSsXJF8tVVVQWVV8q/h3Td7ZBlUWausDDp0aqCuTnTs3MC8OdVZh5SpLl1Ws8OOtVzymz0AqKurpK6uEoAxp7zIn/5vZ376s6eyDLG0ilQCjYhvrWfXges4NoDTWnL+UiXQC4EtJL0IrAFWAguAbSQdBNwXETsASDoL6BYRF0jaErgG6AfUA9/IPamk3YGxwJER8U6JYi+Kiorg97c8y6BhK7jvtiFMfqVn1iGVrXlzOnDn2E248emXWLWyghee7MELT7bv72uTgctYtKgj3z/7eTbffCFT3u7NNVeNYMSIOcyb15n33u2VdYil04ZW5SxVI9O5wDsRsQtwNrArcEZEbN3M+/4CXBkROwP7kDN8QNI+JMn18PUlT0ljJE2UNHF1w4pifI6CNTSI04/ai+MO2o+td1jMplsuzTSectatRx17H7SQE/bbiWP32JlOnRs44IjarMPKVGVlsOVWC3jg71tw+ncOYuXKKo799msc9a03uPHP22cdXuk1RH6PjG2oVvrnIuK9pg6Q1B0YHBF3A0TEyohYnu7elqTk+ZWImLa+c0TE2IgYGREjO1R0LlbsrbJsSTUvP9+b3faZl3UoZWvEfouZM70ji+ZXU19XwYRxvdl2t/b9g1P7YWdqP+zM5Df7AvDUE0PYcqsFDNhkGVf+4SGuu/E+avqt4LKrH6Z372wLC6WgiLweWdtQvfDLcp7X8cnE3SmP989OjxsBzCpiXCXRo/dq6uvEsiXVdOhYz4i95nPndZtmHVbZmjurA9uMWErHTvWsWlnBLvsu5u1XumYdVqYWLOjMhx92YfCQxcyc0YNdRsxhytu9Of9Hoz465rob7+OM0764EfbC0+5npF8CdF/PvjlAf0l9gaXAYcC4iFgiaYakr0XE3yR1BCrT9ywERgMPS1oWEY+XKO6i6FOzih/+4jUqKkAVwZMPDeC5J/rx1WOmceQJU+nddzVX3vEME5+q4fc/2y7rcDM3+cVuPPlAH664/3Xq68U7r3XhwZv7ZR1W5q65cgQ/Ou9Zqqoa+GB2Vy69eI+sQ9owAmgjbaCKEmV6STcDOwErgDm5w5AkfQ84g2SQ6rvA+2kn0lbAH4Aaks6nbwDDSIcxSRoGPAj8V0Q829T1e1b3j737HFmCT7bxaFi4KOsQyl7F1sOzDqGsPT3lWhatmK1inrNn10Gx13Yn53XsQxMvmNSaO5Faq2RV+Ig4pol9lwGXrWP728ABa21+F3g83T8NaAct6GbtXDuvwpuZFcYz0puZFa4cetjz4QRqZuXHCdTMrBAln0ykaJxAzay8BE6gZmYFayPjQJ1AzazsqKFtZFAnUDMrL0FZTBSSDydQMysz7kQyMyucE6iZWYGcQM3MChAB9fVZR5EXJ1AzKz8ugZqZFcC98GZmreASqJlZgZxAzcwK4XGgZmaFCdwLb2ZWMJdAzcwKEe6FNzMrSECEZ2MyMyuMS6BmZgVyG6iZWQF8L7yZWeHCM9KbmRXCA+nNzArThiYTqcg6ADOzT4mG/B7NkHSIpMmSpkg6t9hhugRqZmUlgChCCVRSJXAl8EVgBvC8pHsj4vVWnzzlBGpm5SWCKE4v/B7AlIh4F0DSrcDhQNESqKvwZlZ+ilOFHwxMz3k9I91WNBttCXRx3Ye14+dePTXrOHLUALVZB1Hmyu87ejXrAD6l3L6jTYt9wiUsGP9I3FmT5+GdJE3MeT02IsYWO6b12WgTaET0yzqGXJImRsTIrOMoZ/6OmtcevqOIOKRIp5oJDM15PSTdVjSuwpvZxup5YCtJwyV1AI4G7i3mBTbaEqiZtW8RUSfpu8B4oBL4U0S8VsxrOIFuOBusXaYN83fUPH9HLRARDwAPlOr8ijZyy5SZWblxG6iZWYGcQM3MCuQEmhFJyjoGM2sdJ9ANTNJeAOHG53XyD0vz1v6OJPn/44z4i9+AJJ0A/ExSn6xjKUeS1PjDImlg1vGUsW6NT9J/U+dkF0r75gS6gUjaF/gmcFZEzE9nirEcOcnzTOBmST0yDqnsSNoUuEXS7ummauCdDENq15xAS0xShaQqYCTQH/i6pA4RUe/q6qdJ+i+SO0a+HRGLXVr/lBXAk8D5knYimf2te7YhtV8eB1pikgZExJz0+beBPYGngDsak6jbQz8m6QfAFGAlsAMwBriWZAD54vb6XUmqiHSxdEk1wH8CnwdWA9OA2wCRTDbyQuO/OSst34lUQpJOAw6X9BLwakRcL6ka2JtkFpkbI6JtLD9YYpIOA14GJgOnp5uvA84DjgFui4hFGYWXqfRHtjF5fotkPsvbgQbgeyQ/youAEUBf4NsZhdruOIGWSNq4fwxwLHARcFBaGr0oTazbAV2BxdlFWR4kdQQ+C3wX+BbwD5La0TJJXySZRWdNhiFmKqdt+BSS7+irETFL0i3AKuBQ4OaI+EWGYbZLbgMtAUkjgSXAYemjB0lJ4T8knRURVwK/jIh2nzwBImIV8BvgGeAmoH+aPE8Ffg2MiYjZWcaYJSUGk/wg/0dEvJuWSj8E/g48C/xSUmcPadqw3AZaZJK+AxwEnE1Srfo/4KSIqJX0V6ADcFxEzM8wzMys1ZZ3FLBLRJyXvgajpRwAAAYFSURBVO4DfJ+kdH4qMAhY1LgkQ3uVJsXOwB3ANyNiadoRuVrSIJJSaLTXf1NZ8q9VEUn6KvAd4MyImELSRNID2Dqt0jcAJ7Tnf+g5ybMf8Arwn5JOT/fNBx4BhgO/BV5y8tR+wCkRsQxYCPwZIE2eJwBXAMva87+pLLkNtLgGAbdGxFRJ1RExW9L9JJ0iw4DTIqKclmPYYCTtAwyLiFvThDmGZJqx54D/ltSQNm0MAh4CLmtMtu1J46iMtNQpktL4zmlpfTRwg6QnSKrtBwAnRsTK7CJu35xAi2sq8DVJd0XE5HTbZGAeSS/yiuxCy1xv4P+XtD2wBfC19O/7QD/gFEl7APsDh0TErKwCzVLOMK0hETFN0k0kVfR9gcqI+IakI0jGf/4hrelYRpxAi2sCsA9wgqQJQC/gDOBb7Tx5EhH3S1oNXEpSNX9H0gySVRM3Jyl1vgWsaY/Jc63bWAcBT0o6JSIelHQn0BE4Pr0p4+aIqMsyXku4DbSI0l71q0hKoqcCXwZGu5SQiIiHgR8Dh0o6KiJWRcQbwNZAj4iY6uSpH5GMEz4f+JWkgyJiWbrSZAdgJ3LuhbdsuQRaZOlwm2sk/Sl9vTrjkMpKRNyT3pF1maRtgRdJSqAvZhtZdnKS56HAKJJ29GmSArhE0jkka/qsAH4bEQszC9Y+wQm0RJw41y8i7kuroncB9wGHt8fedkn9gX4R8Vrao34uMCUipgFExM1ps8dPSJLnme15PGw58jhQy4ykzwNTI+L9rGPJgqStgCuB2SSjNP4EnAlcHxGX5RzXE6hLhzJZGXEJ1DITEf/MOoYsRcTbkl4mGdJ1TkTcKKkWODltFr08Pa5dzgHQFjiBmmXrGuAl4AeS5kfEbZLmAldJqo2IWzKOz5rgBGqWoXSExhRJC0nuZ18IdCKZpu6ZTIOzZjmBmpWBiPi7pDXAxcAykuFv72UcljXDnUhmZSTtmY90piUrc06gZmYF8p1IZmYFcgI1MyuQE6iZWYGcQM3MCuQEamZWICfQdkxSvaQXJb0q6Q5JXVpxrj9LOjJ9/kdJ2zVx7Kh0hvqWXuP9dE30vLavdczSFl7rAklntTRGa1+cQNu3FRGxS0TsQHLnyym5O9MZk1osIk6KiNebOGQUycTTZm2aE6g1ehLYMi0dPinpXuB1SZWSfiPpeUkvSzoZPlpq9wpJkyU9AvRvPJGkx9OlnZF0iKQXJL0k6VFJm5Ek6u+npd/PSuon6a70Gs9L2jd9b19JD0l6TdIfSdYIapKkv0malL5nzFr7Lk23P5ouaoekLSSNS9/zpKRtivFlWvvgWzmtsaT5JWBcumlXYIeIeC9NQosiYndJHYEJkh4CRgCfIVn0bADwOsl0bLnn7UeyrPPn0nP1iYj5kq4BlkbExelxNwOXRsRTkoYB44Ftgf8BnoqIn0v6Msmias35r/QanYHn0/Wp5gFdgYkR8X1JP03P/V1gLMmql29L2pNkRYEDCvgarR1yAm3fOktqnAn+SeBakqr1czn3YR8E7NTYvgn0BLYCPgfcEhH1wCxJj63j/HsBTzSeq4mld78AbCd9VMDsIalbeo3/SN97v6QFeXym76WLrgEMTWOdR7Kk9G3p9puAv6bX2Ae4I+faHfO4hhngBNrerYiIXXI3pIkkd+JeAadHxPi1jju0iHFUAHutvTxvTlLLi6RRJMl474hYLulxkpmN1iXS6y5c+zswy5fbQK0544HvSKoGkLS1pK7AE8BRaRvpQJLliNf2DPA5ScPT9/ZJty8Buucc9xBweuMLSY0J7QngmHTbl0iWRm5KT2BBmjy3ISkBN6oAGkvRx5A0DSwG3pP0jfQakrRzM9cw+4gTqDXnjyTtmy9IehX4A0nN5W7g7XTfDcDTa78xnVFoDEl1+SU+rkL/HTiisRMJ+B4wMu2kep2PRwP8jCQBv0ZSlZ/WTKzjgCpJbwAX8sn5NJcBe6Sf4QDg5+n2Y4HRaXyvAYfn8Z2YAZ6NycysYC6BmpkVyAnUzKxATqBmZgVyAjUzK5ATqJlZgZxAzcwK5ARqZlag/wdOafPRBSEk9gAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light",
"tags": []
},
"output_type": "display_data"
}
],
"source": [
"results_interclass.plot_confusion_matrix()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jG_n8iWMX2vn"
},
"source": [
"Note that there appears to be confusion between car and truck classes."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ElSV7tTbYKLr"
},
"source": [
"The [detection evaluation](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html#detections) also added the attributes `eval_fp`, `eval_tp`, and `eval_fn` to every predicted detection indicating if it is a false positive, true positive, or false negative. \n",
"Let's create a view to find the worst samples by sorting by `eval_fp` using the [FiftyOne App](https://voxel51.com/docs/fiftyone/user_guide/app.html) to visualize the results. "
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 786,
"resources": {
"https://localhost:5151/polling?sessionId=de0b710e-15f8-4c57-ba46-ae7955f716b1": {
"data": "eyJtZXNzYWdlcyI6IFtdfQ==",
"headers": [
[
"access-control-allow-headers",
"x-requested-with"
],
[
"content-type",
"text/html; charset=UTF-8"
]
],
"ok": true,
"status": 200,
"status_text": ""
}
}
},
"id": "Pm4Z52rd8AC1",
"outputId": "62d39076-7ef3-4fe3-95ae-500d0f8f8a3f"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"
\n",
" Activate \n",
"
\n",
"
"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"2a2446e8-86a0-11eb-bf60-0242ac1c0002\"] = google.colab.output.getActiveOutputArea();\n",
"//# sourceURL=js_90f1267f32"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"2a248a54-86a0-11eb-bf60-0242ac1c0002\"] = document.querySelector(\"#focontainer-d9a61328-7b16-4c7b-862f-42fdbd762d7a\");\n",
"//# sourceURL=js_f9b7b3c002"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"2a24dbf8-86a0-11eb-bf60-0242ac1c0002\"] = google.colab.output.setActiveOutputArea(window[\"2a248a54-86a0-11eb-bf60-0242ac1c0002\"]);\n",
"//# sourceURL=js_a8b2ac749a"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" "
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"2a2798de-86a0-11eb-bf60-0242ac1c0002\"] = google.colab.output.setActiveOutputArea(window[\"2a2446e8-86a0-11eb-bf60-0242ac1c0002\"]);\n",
"//# sourceURL=js_4e68cecd22"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
}
],
"source": [
"session.view = test_view.sort_by(\"eval_fp\", reverse=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_0QENj62YuTQ"
},
"source": [
"Looking through some of these samples, we can see the confusion between the classes \"car\" and \"truck\" that we found earlier. However, this seems to be, at least in part, due to annotation errors on vans and SUVs where they are interchangably labeled as \"car\" and \"truck\". The example below shows an SUV annotated as \"truck\" but predicted as \"car\"."
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 786,
"resources": {
"https://localhost:5151/polling?sessionId=ebbc318d-3578-4fb1-9ae7-68596117572b": {
"data": "eyJtZXNzYWdlcyI6IFtdfQ==",
"headers": [
[
"access-control-allow-headers",
"x-requested-with"
],
[
"content-type",
"text/html; charset=UTF-8"
]
],
"ok": true,
"status": 200,
"status_text": ""
}
}
},
"id": "njLG0l5K-ucV",
"outputId": "bda6f02d-d8fe-49be-d212-31e0e70779e3"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"
\n",
" Activate \n",
"
\n",
"
"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"4812aa82-86a0-11eb-bf60-0242ac1c0002\"] = google.colab.output.getActiveOutputArea();\n",
"//# sourceURL=js_b5313c3f4d"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"481345be-86a0-11eb-bf60-0242ac1c0002\"] = document.querySelector(\"#focontainer-5d7d3846-57d9-4fcc-8b51-b504f5b640d3\");\n",
"//# sourceURL=js_6ee34118f3"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"481396cc-86a0-11eb-bf60-0242ac1c0002\"] = google.colab.output.setActiveOutputArea(window[\"481345be-86a0-11eb-bf60-0242ac1c0002\"]);\n",
"//# sourceURL=js_95362935da"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
" "
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"data": {
"application/javascript": [
"window[\"48159bf2-86a0-11eb-bf60-0242ac1c0002\"] = google.colab.output.setActiveOutputArea(window[\"4812aa82-86a0-11eb-bf60-0242ac1c0002\"]);\n",
"//# sourceURL=js_36adea015c"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
}
],
"source": [
"session.view = test_view.sort_by(\"eval_fp\", reverse=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ReXDVFgLZLtf"
},
"source": [
"It would be best to get this [data reannotated to fix these mistakes](https://towardsdatascience.com/managing-annotation-mistakes-with-fiftyone-and-labelbox-fc6e87b51102), but in the meantime, we can easily remedy this by simply creating a new view that remaps the labels `car`, `truck`, and `bus` all to `vehicle` and then retraining the model with that. This is only possible because we are backing our data in FiftyOne and loading views into PyTorch as needed. Without FiftyOne, the PyTorch dataset class or the underlying data would need to be changed to remap these classes."
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {
"id": "9php90KrW9ST"
},
"outputs": [],
"source": [
"# map labels to single vehicle class \n",
"vehicles_map = {c: \"vehicle\" for c in vehicles_list}\n",
"\n",
"train_map_view = train_view.map_labels(\"ground_truth\", vehicles_map)\n",
"test_map_view = test_view.map_labels(\"ground_truth\", vehicles_map)\n",
"\n",
"# use our dataset and defined transformations\n",
"torch_map_dataset = FiftyOneTorchDataset(train_map_view, train_transforms)\n",
"torch_map_dataset_test = FiftyOneTorchDataset(test_map_view, test_transforms)"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {
"id": "ynRCHQv8XB_v"
},
"outputs": [],
"source": [
"# Only 2 classes (background and vehicle)\n",
"vehicle_model = get_model(2)"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "lCM_g65EXDzS",
"outputId": "14756cff-7a55-4086-f957-b70f516f06c6"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Using device cuda\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n",
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch: [0] [ 0/250] eta: 0:04:47 lr: 0.000025 loss: 1.5123 (1.5123) loss_classifier: 0.9323 (0.9323) loss_box_reg: 0.5551 (0.5551) loss_objectness: 0.0071 (0.0071) loss_rpn_box_reg: 0.0178 (0.0178) time: 1.1514 data: 0.2843 max mem: 4401\n",
"Epoch: [0] [ 10/250] eta: 0:02:35 lr: 0.000226 loss: 1.1389 (1.1536) loss_classifier: 0.8300 (0.7482) loss_box_reg: 0.3211 (0.3252) loss_objectness: 0.0288 (0.0414) loss_rpn_box_reg: 0.0125 (0.0387) time: 0.6473 data: 0.0319 max mem: 4401\n",
"Epoch: [0] [ 20/250] eta: 0:02:28 lr: 0.000426 loss: 1.0465 (1.0374) loss_classifier: 0.5113 (0.5782) loss_box_reg: 0.3211 (0.3764) loss_objectness: 0.0261 (0.0420) loss_rpn_box_reg: 0.0125 (0.0407) time: 0.6196 data: 0.0066 max mem: 4401\n",
"Epoch: [0] [ 30/250] eta: 0:02:24 lr: 0.000627 loss: 0.7177 (0.9652) loss_classifier: 0.3202 (0.5004) loss_box_reg: 0.4171 (0.3951) loss_objectness: 0.0187 (0.0346) loss_rpn_box_reg: 0.0168 (0.0350) time: 0.6625 data: 0.0066 max mem: 4401\n",
"Epoch: [0] [ 40/250] eta: 0:02:15 lr: 0.000827 loss: 0.5643 (0.8700) loss_classifier: 0.2055 (0.4228) loss_box_reg: 0.3185 (0.3841) loss_objectness: 0.0171 (0.0338) loss_rpn_box_reg: 0.0102 (0.0292) time: 0.6452 data: 0.0066 max mem: 4401\n",
"Epoch: [0] [ 50/250] eta: 0:02:07 lr: 0.001028 loss: 0.5635 (0.8148) loss_classifier: 0.1749 (0.3717) loss_box_reg: 0.3185 (0.3814) loss_objectness: 0.0303 (0.0343) loss_rpn_box_reg: 0.0115 (0.0274) time: 0.6009 data: 0.0067 max mem: 4401\n",
"Epoch: [0] [ 60/250] eta: 0:01:59 lr: 0.001229 loss: 0.4994 (0.7727) loss_classifier: 0.1450 (0.3337) loss_box_reg: 0.3095 (0.3741) loss_objectness: 0.0259 (0.0373) loss_rpn_box_reg: 0.0115 (0.0276) time: 0.6004 data: 0.0072 max mem: 4401\n",
"Epoch: [0] [ 70/250] eta: 0:01:52 lr: 0.001429 loss: 0.4577 (0.7237) loss_classifier: 0.1191 (0.3026) loss_box_reg: 0.2745 (0.3532) loss_objectness: 0.0240 (0.0376) loss_rpn_box_reg: 0.0115 (0.0302) time: 0.5992 data: 0.0073 max mem: 4401\n",
"Epoch: [0] [ 80/250] eta: 0:01:45 lr: 0.001630 loss: 0.4577 (0.6851) loss_classifier: 0.1058 (0.2799) loss_box_reg: 0.1997 (0.3380) loss_objectness: 0.0226 (0.0361) loss_rpn_box_reg: 0.0169 (0.0311) time: 0.5929 data: 0.0071 max mem: 4401\n",
"Epoch: [0] [ 90/250] eta: 0:01:38 lr: 0.001830 loss: 0.3704 (0.6538) loss_classifier: 0.1109 (0.2630) loss_box_reg: 0.2411 (0.3249) loss_objectness: 0.0264 (0.0361) loss_rpn_box_reg: 0.0129 (0.0298) time: 0.5899 data: 0.0070 max mem: 4401\n",
"Epoch: [0] [100/250] eta: 0:01:32 lr: 0.002031 loss: 0.3704 (0.6327) loss_classifier: 0.1113 (0.2498) loss_box_reg: 0.2208 (0.3154) loss_objectness: 0.0264 (0.0357) loss_rpn_box_reg: 0.0142 (0.0318) time: 0.5875 data: 0.0070 max mem: 4401\n",
"Epoch: [0] [110/250] eta: 0:01:26 lr: 0.002232 loss: 0.3363 (0.6076) loss_classifier: 0.1264 (0.2381) loss_box_reg: 0.1662 (0.3018) loss_objectness: 0.0287 (0.0369) loss_rpn_box_reg: 0.0120 (0.0308) time: 0.6119 data: 0.0070 max mem: 4401\n",
"Epoch: [0] [120/250] eta: 0:01:21 lr: 0.002432 loss: 0.3157 (0.5894) loss_classifier: 0.1040 (0.2283) loss_box_reg: 0.1581 (0.2951) loss_objectness: 0.0128 (0.0353) loss_rpn_box_reg: 0.0135 (0.0307) time: 0.6737 data: 0.0066 max mem: 4533\n",
"Epoch: [0] [130/250] eta: 0:01:15 lr: 0.002633 loss: 0.4046 (0.5778) loss_classifier: 0.1315 (0.2218) loss_box_reg: 0.1998 (0.2909) loss_objectness: 0.0128 (0.0353) loss_rpn_box_reg: 0.0149 (0.0298) time: 0.6756 data: 0.0070 max mem: 4533\n",
"Epoch: [0] [140/250] eta: 0:01:08 lr: 0.002833 loss: 0.4122 (0.5675) loss_classifier: 0.1386 (0.2157) loss_box_reg: 0.1871 (0.2833) loss_objectness: 0.0280 (0.0372) loss_rpn_box_reg: 0.0151 (0.0313) time: 0.6227 data: 0.0073 max mem: 4533\n",
"Epoch: [0] [150/250] eta: 0:01:02 lr: 0.003034 loss: 0.3331 (0.5530) loss_classifier: 0.1093 (0.2089) loss_box_reg: 0.1555 (0.2757) loss_objectness: 0.0371 (0.0377) loss_rpn_box_reg: 0.0119 (0.0308) time: 0.6210 data: 0.0071 max mem: 4533\n",
"Epoch: [0] [160/250] eta: 0:00:56 lr: 0.003235 loss: 0.2743 (0.5364) loss_classifier: 0.0888 (0.2014) loss_box_reg: 0.1502 (0.2681) loss_objectness: 0.0178 (0.0367) loss_rpn_box_reg: 0.0075 (0.0302) time: 0.6430 data: 0.0073 max mem: 4533\n",
"Epoch: [0] [170/250] eta: 0:00:50 lr: 0.003435 loss: 0.3487 (0.5292) loss_classifier: 0.1035 (0.1986) loss_box_reg: 0.1502 (0.2623) loss_objectness: 0.0256 (0.0374) loss_rpn_box_reg: 0.0131 (0.0310) time: 0.6522 data: 0.0074 max mem: 4533\n",
"Epoch: [0] [180/250] eta: 0:00:43 lr: 0.003636 loss: 0.3487 (0.5192) loss_classifier: 0.1153 (0.1948) loss_box_reg: 0.1466 (0.2577) loss_objectness: 0.0216 (0.0363) loss_rpn_box_reg: 0.0206 (0.0305) time: 0.6394 data: 0.0073 max mem: 4533\n",
"Epoch: [0] [190/250] eta: 0:00:37 lr: 0.003837 loss: 0.2860 (0.5115) loss_classifier: 0.1059 (0.1900) loss_box_reg: 0.1257 (0.2518) loss_objectness: 0.0165 (0.0382) loss_rpn_box_reg: 0.0163 (0.0314) time: 0.6321 data: 0.0074 max mem: 4533\n",
"Epoch: [0] [200/250] eta: 0:00:31 lr: 0.004037 loss: 0.3070 (0.5043) loss_classifier: 0.0995 (0.1862) loss_box_reg: 0.1395 (0.2493) loss_objectness: 0.0185 (0.0380) loss_rpn_box_reg: 0.0135 (0.0309) time: 0.6431 data: 0.0078 max mem: 4533\n",
"Epoch: [0] [210/250] eta: 0:00:25 lr: 0.004238 loss: 0.3828 (0.5003) loss_classifier: 0.1123 (0.1837) loss_box_reg: 0.2287 (0.2490) loss_objectness: 0.0276 (0.0372) loss_rpn_box_reg: 0.0117 (0.0304) time: 0.6685 data: 0.0071 max mem: 4722\n",
"Epoch: [0] [220/250] eta: 0:00:18 lr: 0.004438 loss: 0.4111 (0.4963) loss_classifier: 0.1251 (0.1814) loss_box_reg: 0.2181 (0.2471) loss_objectness: 0.0243 (0.0376) loss_rpn_box_reg: 0.0092 (0.0302) time: 0.6522 data: 0.0065 max mem: 4722\n",
"Epoch: [0] [230/250] eta: 0:00:12 lr: 0.004639 loss: 0.4111 (0.4931) loss_classifier: 0.1093 (0.1783) loss_box_reg: 0.2089 (0.2443) loss_objectness: 0.0484 (0.0397) loss_rpn_box_reg: 0.0213 (0.0308) time: 0.6243 data: 0.0066 max mem: 4722\n",
"Epoch: [0] [240/250] eta: 0:00:06 lr: 0.004840 loss: 0.3850 (0.4962) loss_classifier: 0.1090 (0.1783) loss_box_reg: 0.1370 (0.2424) loss_objectness: 0.0574 (0.0430) loss_rpn_box_reg: 0.0196 (0.0326) time: 0.6253 data: 0.0066 max mem: 4722\n",
"Epoch: [0] [249/250] eta: 0:00:00 lr: 0.005000 loss: 0.3593 (0.4939) loss_classifier: 0.1267 (0.1766) loss_box_reg: 0.1234 (0.2388) loss_objectness: 0.0706 (0.0445) loss_rpn_box_reg: 0.0209 (0.0340) time: 0.6304 data: 0.0066 max mem: 4722\n",
"Epoch: [0] Total time: 0:02:37 (0.6313 s / it)\n",
"creating index...\n",
"index created!\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n",
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Test: [ 0/207] eta: 0:01:10 model_time: 0.1596 (0.1596) evaluator_time: 0.0073 (0.0073) time: 0.3383 data: 0.1694 max mem: 4722\n",
"Test: [100/207] eta: 0:00:13 model_time: 0.1203 (0.1171) evaluator_time: 0.0024 (0.0046) time: 0.1296 data: 0.0037 max mem: 4722\n",
"Test: [200/207] eta: 0:00:00 model_time: 0.1130 (0.1172) evaluator_time: 0.0025 (0.0047) time: 0.1215 data: 0.0033 max mem: 4722\n",
"Test: [206/207] eta: 0:00:00 model_time: 0.1128 (0.1171) evaluator_time: 0.0034 (0.0049) time: 0.1248 data: 0.0033 max mem: 4722\n",
"Test: Total time: 0:00:26 (0.1281 s / it)\n",
"Averaged stats: model_time: 0.1128 (0.1171) evaluator_time: 0.0034 (0.0049)\n",
"Accumulating evaluation results...\n",
"DONE (t=0.07s).\n",
"IoU metric: bbox\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.262\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.507\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.245\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.197\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.308\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.416\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.134\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.360\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.419\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.338\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.451\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.595\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n",
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch: [1] [ 0/250] eta: 0:03:48 lr: 0.005000 loss: 0.2260 (0.2260) loss_classifier: 0.0742 (0.0742) loss_box_reg: 0.1216 (0.1216) loss_objectness: 0.0229 (0.0229) loss_rpn_box_reg: 0.0073 (0.0073) time: 0.9124 data: 0.3390 max mem: 4722\n",
"Epoch: [1] [ 10/250] eta: 0:02:36 lr: 0.005000 loss: 0.4380 (0.4907) loss_classifier: 0.1351 (0.1584) loss_box_reg: 0.1715 (0.2241) loss_objectness: 0.0317 (0.0482) loss_rpn_box_reg: 0.0167 (0.0600) time: 0.6532 data: 0.0360 max mem: 4722\n",
"Epoch: [1] [ 20/250] eta: 0:02:29 lr: 0.005000 loss: 0.3960 (0.4844) loss_classifier: 0.1281 (0.1492) loss_box_reg: 0.1715 (0.2056) loss_objectness: 0.0358 (0.0673) loss_rpn_box_reg: 0.0177 (0.0623) time: 0.6360 data: 0.0061 max mem: 4722\n",
"Epoch: [1] [ 30/250] eta: 0:02:17 lr: 0.005000 loss: 0.3671 (0.4388) loss_classifier: 0.1112 (0.1335) loss_box_reg: 0.1653 (0.1906) loss_objectness: 0.0371 (0.0578) loss_rpn_box_reg: 0.0173 (0.0570) time: 0.6098 data: 0.0066 max mem: 4722\n",
"Epoch: [1] [ 40/250] eta: 0:02:10 lr: 0.005000 loss: 0.3153 (0.4099) loss_classifier: 0.0969 (0.1271) loss_box_reg: 0.1344 (0.1804) loss_objectness: 0.0310 (0.0536) loss_rpn_box_reg: 0.0164 (0.0487) time: 0.5919 data: 0.0068 max mem: 4722\n",
"Epoch: [1] [ 50/250] eta: 0:02:04 lr: 0.005000 loss: 0.3379 (0.4230) loss_classifier: 0.1128 (0.1333) loss_box_reg: 0.1771 (0.1905) loss_objectness: 0.0257 (0.0509) loss_rpn_box_reg: 0.0170 (0.0482) time: 0.6143 data: 0.0067 max mem: 4722\n",
"Epoch: [1] [ 60/250] eta: 0:01:58 lr: 0.005000 loss: 0.3574 (0.4030) loss_classifier: 0.1075 (0.1279) loss_box_reg: 0.1640 (0.1838) loss_objectness: 0.0239 (0.0457) loss_rpn_box_reg: 0.0098 (0.0456) time: 0.6247 data: 0.0066 max mem: 4722\n",
"Epoch: [1] [ 70/250] eta: 0:01:51 lr: 0.005000 loss: 0.3245 (0.3916) loss_classifier: 0.0975 (0.1251) loss_box_reg: 0.1371 (0.1810) loss_objectness: 0.0138 (0.0424) loss_rpn_box_reg: 0.0082 (0.0431) time: 0.6243 data: 0.0067 max mem: 4722\n",
"Epoch: [1] [ 80/250] eta: 0:01:45 lr: 0.005000 loss: 0.3548 (0.4036) loss_classifier: 0.1278 (0.1334) loss_box_reg: 0.1713 (0.1831) loss_objectness: 0.0274 (0.0433) loss_rpn_box_reg: 0.0204 (0.0439) time: 0.6185 data: 0.0067 max mem: 4722\n",
"Epoch: [1] [ 90/250] eta: 0:01:38 lr: 0.005000 loss: 0.3772 (0.3997) loss_classifier: 0.1479 (0.1351) loss_box_reg: 0.1760 (0.1809) loss_objectness: 0.0274 (0.0412) loss_rpn_box_reg: 0.0271 (0.0424) time: 0.6057 data: 0.0066 max mem: 4722\n",
"Epoch: [1] [100/250] eta: 0:01:32 lr: 0.005000 loss: 0.3772 (0.4120) loss_classifier: 0.1295 (0.1391) loss_box_reg: 0.2125 (0.1897) loss_objectness: 0.0202 (0.0406) loss_rpn_box_reg: 0.0215 (0.0426) time: 0.6011 data: 0.0069 max mem: 4722\n",
"Epoch: [1] [110/250] eta: 0:01:26 lr: 0.005000 loss: 0.4882 (0.4238) loss_classifier: 0.1633 (0.1429) loss_box_reg: 0.2770 (0.1992) loss_objectness: 0.0210 (0.0398) loss_rpn_box_reg: 0.0212 (0.0419) time: 0.6070 data: 0.0071 max mem: 4722\n",
"Epoch: [1] [120/250] eta: 0:01:20 lr: 0.005000 loss: 0.4424 (0.4215) loss_classifier: 0.1509 (0.1426) loss_box_reg: 0.2372 (0.1982) loss_objectness: 0.0243 (0.0390) loss_rpn_box_reg: 0.0212 (0.0417) time: 0.6373 data: 0.0067 max mem: 4722\n",
"Epoch: [1] [130/250] eta: 0:01:14 lr: 0.005000 loss: 0.3083 (0.4120) loss_classifier: 0.1214 (0.1402) loss_box_reg: 0.1500 (0.1943) loss_objectness: 0.0229 (0.0372) loss_rpn_box_reg: 0.0158 (0.0403) time: 0.6323 data: 0.0068 max mem: 4722\n",
"Epoch: [1] [140/250] eta: 0:01:07 lr: 0.005000 loss: 0.3083 (0.4095) loss_classifier: 0.1214 (0.1390) loss_box_reg: 0.1667 (0.1947) loss_objectness: 0.0101 (0.0358) loss_rpn_box_reg: 0.0214 (0.0400) time: 0.5895 data: 0.0070 max mem: 4722\n",
"Epoch: [1] [150/250] eta: 0:01:01 lr: 0.005000 loss: 0.3217 (0.4036) loss_classifier: 0.0945 (0.1362) loss_box_reg: 0.1705 (0.1923) loss_objectness: 0.0110 (0.0352) loss_rpn_box_reg: 0.0163 (0.0399) time: 0.6008 data: 0.0068 max mem: 4722\n",
"Epoch: [1] [160/250] eta: 0:00:55 lr: 0.005000 loss: 0.3606 (0.3998) loss_classifier: 0.0966 (0.1352) loss_box_reg: 0.1771 (0.1923) loss_objectness: 0.0123 (0.0339) loss_rpn_box_reg: 0.0119 (0.0383) time: 0.6264 data: 0.0074 max mem: 4722\n",
"Epoch: [1] [170/250] eta: 0:00:49 lr: 0.005000 loss: 0.3350 (0.3959) loss_classifier: 0.1060 (0.1343) loss_box_reg: 0.1939 (0.1920) loss_objectness: 0.0128 (0.0330) loss_rpn_box_reg: 0.0104 (0.0366) time: 0.6413 data: 0.0078 max mem: 4722\n",
"Epoch: [1] [180/250] eta: 0:00:43 lr: 0.005000 loss: 0.3195 (0.3966) loss_classifier: 0.1070 (0.1351) loss_box_reg: 0.1639 (0.1908) loss_objectness: 0.0158 (0.0338) loss_rpn_box_reg: 0.0133 (0.0369) time: 0.6152 data: 0.0073 max mem: 4722\n",
"Epoch: [1] [190/250] eta: 0:00:37 lr: 0.005000 loss: 0.2176 (0.3884) loss_classifier: 0.1070 (0.1330) loss_box_reg: 0.1231 (0.1859) loss_objectness: 0.0226 (0.0331) loss_rpn_box_reg: 0.0175 (0.0364) time: 0.5990 data: 0.0070 max mem: 4722\n",
"Epoch: [1] [200/250] eta: 0:00:30 lr: 0.005000 loss: 0.2095 (0.3844) loss_classifier: 0.0809 (0.1321) loss_box_reg: 0.0945 (0.1839) loss_objectness: 0.0196 (0.0325) loss_rpn_box_reg: 0.0175 (0.0358) time: 0.6188 data: 0.0070 max mem: 4722\n",
"Epoch: [1] [210/250] eta: 0:00:24 lr: 0.005000 loss: 0.2791 (0.3822) loss_classifier: 0.1129 (0.1319) loss_box_reg: 0.1148 (0.1827) loss_objectness: 0.0196 (0.0321) loss_rpn_box_reg: 0.0172 (0.0355) time: 0.6508 data: 0.0070 max mem: 4722\n",
"Epoch: [1] [220/250] eta: 0:00:18 lr: 0.005000 loss: 0.4018 (0.3878) loss_classifier: 0.1442 (0.1338) loss_box_reg: 0.1818 (0.1860) loss_objectness: 0.0208 (0.0321) loss_rpn_box_reg: 0.0172 (0.0358) time: 0.6436 data: 0.0070 max mem: 4722\n",
"Epoch: [1] [230/250] eta: 0:00:12 lr: 0.005000 loss: 0.4029 (0.3874) loss_classifier: 0.1442 (0.1342) loss_box_reg: 0.1804 (0.1855) loss_objectness: 0.0271 (0.0320) loss_rpn_box_reg: 0.0165 (0.0357) time: 0.6020 data: 0.0072 max mem: 4722\n",
"Epoch: [1] [240/250] eta: 0:00:06 lr: 0.005000 loss: 0.4094 (0.3896) loss_classifier: 0.1413 (0.1351) loss_box_reg: 0.1645 (0.1871) loss_objectness: 0.0163 (0.0314) loss_rpn_box_reg: 0.0117 (0.0360) time: 0.6465 data: 0.0075 max mem: 4722\n",
"Epoch: [1] [249/250] eta: 0:00:00 lr: 0.005000 loss: 0.4166 (0.3906) loss_classifier: 0.1732 (0.1352) loss_box_reg: 0.1937 (0.1879) loss_objectness: 0.0163 (0.0313) loss_rpn_box_reg: 0.0117 (0.0361) time: 0.6474 data: 0.0070 max mem: 4722\n",
"Epoch: [1] Total time: 0:02:35 (0.6214 s / it)\n",
"creating index...\n",
"index created!\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n",
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Test: [ 0/207] eta: 0:01:11 model_time: 0.1580 (0.1580) evaluator_time: 0.0092 (0.0092) time: 0.3455 data: 0.1763 max mem: 4722\n",
"Test: [100/207] eta: 0:00:13 model_time: 0.1207 (0.1171) evaluator_time: 0.0023 (0.0044) time: 0.1285 data: 0.0037 max mem: 4722\n",
"Test: [200/207] eta: 0:00:00 model_time: 0.1136 (0.1171) evaluator_time: 0.0022 (0.0044) time: 0.1219 data: 0.0040 max mem: 4722\n",
"Test: [206/207] eta: 0:00:00 model_time: 0.1140 (0.1171) evaluator_time: 0.0034 (0.0047) time: 0.1263 data: 0.0040 max mem: 4722\n",
"Test: Total time: 0:00:26 (0.1280 s / it)\n",
"Averaged stats: model_time: 0.1140 (0.1171) evaluator_time: 0.0034 (0.0047)\n",
"Accumulating evaluation results...\n",
"DONE (t=0.07s).\n",
"IoU metric: bbox\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.375\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.663\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.403\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.291\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.444\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.508\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.165\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.457\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.517\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.443\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.579\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.620\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n",
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch: [2] [ 0/250] eta: 0:03:53 lr: 0.005000 loss: 0.0637 (0.0637) loss_classifier: 0.0306 (0.0306) loss_box_reg: 0.0283 (0.0283) loss_objectness: 0.0016 (0.0016) loss_rpn_box_reg: 0.0031 (0.0031) time: 0.9345 data: 0.3557 max mem: 4722\n",
"Epoch: [2] [ 10/250] eta: 0:02:39 lr: 0.005000 loss: 0.2129 (0.2930) loss_classifier: 0.1072 (0.1128) loss_box_reg: 0.1368 (0.1522) loss_objectness: 0.0115 (0.0165) loss_rpn_box_reg: 0.0088 (0.0114) time: 0.6641 data: 0.0378 max mem: 4722\n",
"Epoch: [2] [ 20/250] eta: 0:02:31 lr: 0.005000 loss: 0.2121 (0.2590) loss_classifier: 0.0808 (0.0958) loss_box_reg: 0.0980 (0.1370) loss_objectness: 0.0114 (0.0135) loss_rpn_box_reg: 0.0087 (0.0126) time: 0.6431 data: 0.0063 max mem: 4722\n",
"Epoch: [2] [ 30/250] eta: 0:02:24 lr: 0.005000 loss: 0.1546 (0.2640) loss_classifier: 0.0635 (0.0953) loss_box_reg: 0.0903 (0.1413) loss_objectness: 0.0073 (0.0118) loss_rpn_box_reg: 0.0068 (0.0156) time: 0.6540 data: 0.0066 max mem: 4722\n",
"Epoch: [2] [ 40/250] eta: 0:02:15 lr: 0.005000 loss: 0.2453 (0.2806) loss_classifier: 0.0893 (0.0993) loss_box_reg: 0.1305 (0.1536) loss_objectness: 0.0066 (0.0124) loss_rpn_box_reg: 0.0067 (0.0154) time: 0.6350 data: 0.0074 max mem: 4722\n",
"Epoch: [2] [ 50/250] eta: 0:02:08 lr: 0.005000 loss: 0.3236 (0.2907) loss_classifier: 0.1158 (0.1029) loss_box_reg: 0.1868 (0.1593) loss_objectness: 0.0070 (0.0125) loss_rpn_box_reg: 0.0084 (0.0160) time: 0.6131 data: 0.0078 max mem: 4722\n",
"Epoch: [2] [ 60/250] eta: 0:02:01 lr: 0.005000 loss: 0.3636 (0.3096) loss_classifier: 0.1372 (0.1094) loss_box_reg: 0.1964 (0.1661) loss_objectness: 0.0158 (0.0160) loss_rpn_box_reg: 0.0105 (0.0181) time: 0.6274 data: 0.0071 max mem: 4722\n",
"Epoch: [2] [ 70/250] eta: 0:01:54 lr: 0.005000 loss: 0.4012 (0.3217) loss_classifier: 0.1363 (0.1135) loss_box_reg: 0.1976 (0.1713) loss_objectness: 0.0220 (0.0169) loss_rpn_box_reg: 0.0148 (0.0199) time: 0.6331 data: 0.0070 max mem: 4722\n",
"Epoch: [2] [ 80/250] eta: 0:01:47 lr: 0.005000 loss: 0.4053 (0.3383) loss_classifier: 0.1319 (0.1175) loss_box_reg: 0.2088 (0.1788) loss_objectness: 0.0234 (0.0188) loss_rpn_box_reg: 0.0204 (0.0232) time: 0.6052 data: 0.0070 max mem: 4722\n",
"Epoch: [2] [ 90/250] eta: 0:01:41 lr: 0.005000 loss: 0.3911 (0.3339) loss_classifier: 0.1216 (0.1147) loss_box_reg: 0.1711 (0.1777) loss_objectness: 0.0156 (0.0182) loss_rpn_box_reg: 0.0158 (0.0234) time: 0.6115 data: 0.0068 max mem: 4722\n",
"Epoch: [2] [100/250] eta: 0:01:34 lr: 0.005000 loss: 0.2519 (0.3337) loss_classifier: 0.0933 (0.1142) loss_box_reg: 0.1363 (0.1777) loss_objectness: 0.0126 (0.0182) loss_rpn_box_reg: 0.0153 (0.0236) time: 0.6373 data: 0.0067 max mem: 4722\n",
"Epoch: [2] [110/250] eta: 0:01:28 lr: 0.005000 loss: 0.2847 (0.3340) loss_classifier: 0.0933 (0.1129) loss_box_reg: 0.1560 (0.1786) loss_objectness: 0.0145 (0.0183) loss_rpn_box_reg: 0.0132 (0.0241) time: 0.6365 data: 0.0069 max mem: 4722\n",
"Epoch: [2] [120/250] eta: 0:01:21 lr: 0.005000 loss: 0.2334 (0.3217) loss_classifier: 0.0636 (0.1092) loss_box_reg: 0.1219 (0.1716) loss_objectness: 0.0098 (0.0177) loss_rpn_box_reg: 0.0117 (0.0232) time: 0.6018 data: 0.0067 max mem: 4722\n",
"Epoch: [2] [130/250] eta: 0:01:15 lr: 0.005000 loss: 0.2218 (0.3242) loss_classifier: 0.0836 (0.1094) loss_box_reg: 0.0942 (0.1714) loss_objectness: 0.0125 (0.0182) loss_rpn_box_reg: 0.0137 (0.0251) time: 0.5867 data: 0.0066 max mem: 4722\n",
"Epoch: [2] [140/250] eta: 0:01:08 lr: 0.005000 loss: 0.2659 (0.3248) loss_classifier: 0.0960 (0.1098) loss_box_reg: 0.1178 (0.1704) loss_objectness: 0.0221 (0.0193) loss_rpn_box_reg: 0.0200 (0.0253) time: 0.5923 data: 0.0067 max mem: 4722\n",
"Epoch: [2] [150/250] eta: 0:01:02 lr: 0.005000 loss: 0.2990 (0.3265) loss_classifier: 0.1055 (0.1110) loss_box_reg: 0.1368 (0.1713) loss_objectness: 0.0179 (0.0191) loss_rpn_box_reg: 0.0099 (0.0251) time: 0.6004 data: 0.0070 max mem: 4722\n",
"Epoch: [2] [160/250] eta: 0:00:56 lr: 0.005000 loss: 0.3695 (0.3301) loss_classifier: 0.1121 (0.1121) loss_box_reg: 0.1810 (0.1731) loss_objectness: 0.0163 (0.0191) loss_rpn_box_reg: 0.0113 (0.0258) time: 0.6330 data: 0.0071 max mem: 4722\n",
"Epoch: [2] [170/250] eta: 0:00:49 lr: 0.005000 loss: 0.3311 (0.3337) loss_classifier: 0.0969 (0.1125) loss_box_reg: 0.1938 (0.1741) loss_objectness: 0.0157 (0.0192) loss_rpn_box_reg: 0.0162 (0.0279) time: 0.6402 data: 0.0073 max mem: 4722\n",
"Epoch: [2] [180/250] eta: 0:00:43 lr: 0.005000 loss: 0.3311 (0.3343) loss_classifier: 0.0988 (0.1123) loss_box_reg: 0.1752 (0.1751) loss_objectness: 0.0107 (0.0190) loss_rpn_box_reg: 0.0238 (0.0278) time: 0.6250 data: 0.0071 max mem: 4722\n",
"Epoch: [2] [190/250] eta: 0:00:37 lr: 0.005000 loss: 0.3159 (0.3355) loss_classifier: 0.1008 (0.1132) loss_box_reg: 0.1315 (0.1762) loss_objectness: 0.0107 (0.0187) loss_rpn_box_reg: 0.0114 (0.0275) time: 0.6239 data: 0.0067 max mem: 4722\n",
"Epoch: [2] [200/250] eta: 0:00:31 lr: 0.005000 loss: 0.3492 (0.3401) loss_classifier: 0.1314 (0.1145) loss_box_reg: 0.1712 (0.1779) loss_objectness: 0.0145 (0.0193) loss_rpn_box_reg: 0.0155 (0.0284) time: 0.6070 data: 0.0065 max mem: 4722\n",
"Epoch: [2] [210/250] eta: 0:00:24 lr: 0.005000 loss: 0.3398 (0.3361) loss_classifier: 0.1102 (0.1135) loss_box_reg: 0.1528 (0.1758) loss_objectness: 0.0113 (0.0187) loss_rpn_box_reg: 0.0114 (0.0280) time: 0.5989 data: 0.0065 max mem: 4722\n",
"Epoch: [2] [220/250] eta: 0:00:18 lr: 0.005000 loss: 0.2471 (0.3361) loss_classifier: 0.0930 (0.1129) loss_box_reg: 0.1283 (0.1754) loss_objectness: 0.0072 (0.0186) loss_rpn_box_reg: 0.0097 (0.0292) time: 0.6036 data: 0.0068 max mem: 4722\n",
"Epoch: [2] [230/250] eta: 0:00:12 lr: 0.005000 loss: 0.2549 (0.3378) loss_classifier: 0.1038 (0.1127) loss_box_reg: 0.1310 (0.1763) loss_objectness: 0.0109 (0.0188) loss_rpn_box_reg: 0.0194 (0.0300) time: 0.6081 data: 0.0067 max mem: 4722\n",
"Epoch: [2] [240/250] eta: 0:00:06 lr: 0.005000 loss: 0.3443 (0.3367) loss_classifier: 0.0949 (0.1124) loss_box_reg: 0.1797 (0.1764) loss_objectness: 0.0107 (0.0185) loss_rpn_box_reg: 0.0137 (0.0295) time: 0.6298 data: 0.0066 max mem: 4722\n",
"Epoch: [2] [249/250] eta: 0:00:00 lr: 0.005000 loss: 0.3654 (0.3391) loss_classifier: 0.1281 (0.1133) loss_box_reg: 0.2087 (0.1779) loss_objectness: 0.0129 (0.0186) loss_rpn_box_reg: 0.0137 (0.0292) time: 0.5967 data: 0.0065 max mem: 4722\n",
"Epoch: [2] Total time: 0:02:34 (0.6194 s / it)\n",
"creating index...\n",
"index created!\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n",
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Test: [ 0/207] eta: 0:01:13 model_time: 0.1681 (0.1681) evaluator_time: 0.0099 (0.0099) time: 0.3561 data: 0.1761 max mem: 4722\n",
"Test: [100/207] eta: 0:00:13 model_time: 0.1202 (0.1168) evaluator_time: 0.0027 (0.0050) time: 0.1313 data: 0.0049 max mem: 4722\n",
"Test: [200/207] eta: 0:00:00 model_time: 0.1133 (0.1170) evaluator_time: 0.0028 (0.0050) time: 0.1222 data: 0.0033 max mem: 4722\n",
"Test: [206/207] eta: 0:00:00 model_time: 0.1137 (0.1170) evaluator_time: 0.0036 (0.0052) time: 0.1265 data: 0.0035 max mem: 4722\n",
"Test: Total time: 0:00:26 (0.1288 s / it)\n",
"Averaged stats: model_time: 0.1137 (0.1170) evaluator_time: 0.0036 (0.0052)\n",
"Accumulating evaluation results...\n",
"DONE (t=0.09s).\n",
"IoU metric: bbox\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.395\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.651\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.424\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.296\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.454\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.588\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.173\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.476\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.544\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.454\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.600\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.703\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n",
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch: [3] [ 0/250] eta: 0:03:42 lr: 0.000500 loss: 0.2147 (0.2147) loss_classifier: 0.0755 (0.0755) loss_box_reg: 0.1179 (0.1179) loss_objectness: 0.0143 (0.0143) loss_rpn_box_reg: 0.0070 (0.0070) time: 0.8897 data: 0.2764 max mem: 4722\n",
"Epoch: [3] [ 10/250] eta: 0:02:37 lr: 0.000500 loss: 0.2830 (0.3095) loss_classifier: 0.1161 (0.1141) loss_box_reg: 0.1569 (0.1598) loss_objectness: 0.0143 (0.0162) loss_rpn_box_reg: 0.0080 (0.0194) time: 0.6581 data: 0.0305 max mem: 4722\n",
"Epoch: [3] [ 20/250] eta: 0:02:23 lr: 0.000500 loss: 0.2830 (0.3167) loss_classifier: 0.1070 (0.1103) loss_box_reg: 0.1588 (0.1722) loss_objectness: 0.0136 (0.0172) loss_rpn_box_reg: 0.0091 (0.0170) time: 0.6128 data: 0.0061 max mem: 4722\n",
"Epoch: [3] [ 30/250] eta: 0:02:15 lr: 0.000500 loss: 0.2080 (0.2815) loss_classifier: 0.0735 (0.0976) loss_box_reg: 0.1270 (0.1549) loss_objectness: 0.0091 (0.0153) loss_rpn_box_reg: 0.0081 (0.0137) time: 0.5902 data: 0.0065 max mem: 4722\n",
"Epoch: [3] [ 40/250] eta: 0:02:10 lr: 0.000500 loss: 0.1987 (0.2955) loss_classifier: 0.0735 (0.0988) loss_box_reg: 0.1267 (0.1667) loss_objectness: 0.0070 (0.0137) loss_rpn_box_reg: 0.0106 (0.0163) time: 0.6134 data: 0.0066 max mem: 4722\n",
"Epoch: [3] [ 50/250] eta: 0:02:04 lr: 0.000500 loss: 0.2251 (0.2821) loss_classifier: 0.0764 (0.0932) loss_box_reg: 0.1267 (0.1589) loss_objectness: 0.0055 (0.0126) loss_rpn_box_reg: 0.0151 (0.0174) time: 0.6356 data: 0.0066 max mem: 4722\n",
"Epoch: [3] [ 60/250] eta: 0:01:57 lr: 0.000500 loss: 0.1786 (0.2780) loss_classifier: 0.0556 (0.0907) loss_box_reg: 0.0982 (0.1566) loss_objectness: 0.0079 (0.0127) loss_rpn_box_reg: 0.0123 (0.0180) time: 0.6201 data: 0.0067 max mem: 4722\n",
"Epoch: [3] [ 70/250] eta: 0:01:51 lr: 0.000500 loss: 0.2088 (0.2767) loss_classifier: 0.0584 (0.0914) loss_box_reg: 0.0982 (0.1552) loss_objectness: 0.0088 (0.0123) loss_rpn_box_reg: 0.0123 (0.0178) time: 0.6088 data: 0.0072 max mem: 4722\n",
"Epoch: [3] [ 80/250] eta: 0:01:45 lr: 0.000500 loss: 0.2329 (0.2771) loss_classifier: 0.0765 (0.0907) loss_box_reg: 0.1354 (0.1557) loss_objectness: 0.0077 (0.0118) loss_rpn_box_reg: 0.0201 (0.0190) time: 0.6281 data: 0.0070 max mem: 4722\n",
"Epoch: [3] [ 90/250] eta: 0:01:39 lr: 0.000500 loss: 0.2785 (0.2762) loss_classifier: 0.0871 (0.0901) loss_box_reg: 0.1580 (0.1558) loss_objectness: 0.0066 (0.0113) loss_rpn_box_reg: 0.0120 (0.0189) time: 0.6214 data: 0.0066 max mem: 4722\n",
"Epoch: [3] [100/250] eta: 0:01:32 lr: 0.000500 loss: 0.2862 (0.2802) loss_classifier: 0.0974 (0.0914) loss_box_reg: 0.1737 (0.1580) loss_objectness: 0.0055 (0.0112) loss_rpn_box_reg: 0.0066 (0.0196) time: 0.6024 data: 0.0066 max mem: 4722\n",
"Epoch: [3] [110/250] eta: 0:01:26 lr: 0.000500 loss: 0.2487 (0.2801) loss_classifier: 0.1053 (0.0922) loss_box_reg: 0.1395 (0.1579) loss_objectness: 0.0071 (0.0111) loss_rpn_box_reg: 0.0089 (0.0189) time: 0.6126 data: 0.0066 max mem: 4722\n",
"Epoch: [3] [120/250] eta: 0:01:19 lr: 0.000500 loss: 0.2201 (0.2756) loss_classifier: 0.0611 (0.0902) loss_box_reg: 0.1097 (0.1553) loss_objectness: 0.0080 (0.0108) loss_rpn_box_reg: 0.0080 (0.0192) time: 0.5932 data: 0.0066 max mem: 4722\n",
"Epoch: [3] [130/250] eta: 0:01:13 lr: 0.000500 loss: 0.2385 (0.2720) loss_classifier: 0.0609 (0.0889) loss_box_reg: 0.1028 (0.1537) loss_objectness: 0.0081 (0.0108) loss_rpn_box_reg: 0.0080 (0.0187) time: 0.5834 data: 0.0065 max mem: 4722\n",
"Epoch: [3] [140/250] eta: 0:01:07 lr: 0.000500 loss: 0.2385 (0.2705) loss_classifier: 0.0700 (0.0880) loss_box_reg: 0.1259 (0.1524) loss_objectness: 0.0061 (0.0105) loss_rpn_box_reg: 0.0129 (0.0195) time: 0.5881 data: 0.0067 max mem: 4722\n",
"Epoch: [3] [150/250] eta: 0:01:00 lr: 0.000500 loss: 0.2545 (0.2780) loss_classifier: 0.0898 (0.0902) loss_box_reg: 0.1450 (0.1560) loss_objectness: 0.0064 (0.0107) loss_rpn_box_reg: 0.0090 (0.0211) time: 0.5857 data: 0.0067 max mem: 4722\n",
"Epoch: [3] [160/250] eta: 0:00:55 lr: 0.000500 loss: 0.3263 (0.2809) loss_classifier: 0.1172 (0.0913) loss_box_reg: 0.1450 (0.1559) loss_objectness: 0.0093 (0.0111) loss_rpn_box_reg: 0.0086 (0.0225) time: 0.6324 data: 0.0065 max mem: 4722\n",
"Epoch: [3] [170/250] eta: 0:00:49 lr: 0.000500 loss: 0.1752 (0.2739) loss_classifier: 0.0567 (0.0891) loss_box_reg: 0.0837 (0.1515) loss_objectness: 0.0068 (0.0108) loss_rpn_box_reg: 0.0100 (0.0225) time: 0.6591 data: 0.0066 max mem: 4722\n",
"Epoch: [3] [180/250] eta: 0:00:43 lr: 0.000500 loss: 0.1413 (0.2706) loss_classifier: 0.0515 (0.0879) loss_box_reg: 0.0699 (0.1493) loss_objectness: 0.0047 (0.0106) loss_rpn_box_reg: 0.0088 (0.0227) time: 0.6477 data: 0.0065 max mem: 4722\n",
"Epoch: [3] [190/250] eta: 0:00:37 lr: 0.000500 loss: 0.1904 (0.2686) loss_classifier: 0.0618 (0.0874) loss_box_reg: 0.1134 (0.1488) loss_objectness: 0.0046 (0.0105) loss_rpn_box_reg: 0.0048 (0.0219) time: 0.6486 data: 0.0065 max mem: 4722\n",
"Epoch: [3] [200/250] eta: 0:00:31 lr: 0.000500 loss: 0.2096 (0.2679) loss_classifier: 0.0618 (0.0869) loss_box_reg: 0.1405 (0.1486) loss_objectness: 0.0046 (0.0105) loss_rpn_box_reg: 0.0062 (0.0220) time: 0.6733 data: 0.0067 max mem: 4722\n",
"Epoch: [3] [210/250] eta: 0:00:24 lr: 0.000500 loss: 0.2528 (0.2669) loss_classifier: 0.0830 (0.0864) loss_box_reg: 0.1412 (0.1479) loss_objectness: 0.0059 (0.0104) loss_rpn_box_reg: 0.0099 (0.0222) time: 0.6832 data: 0.0068 max mem: 4722\n",
"Epoch: [3] [220/250] eta: 0:00:18 lr: 0.000500 loss: 0.2576 (0.2698) loss_classifier: 0.0855 (0.0871) loss_box_reg: 0.1504 (0.1504) loss_objectness: 0.0076 (0.0104) loss_rpn_box_reg: 0.0099 (0.0220) time: 0.6381 data: 0.0068 max mem: 4722\n",
"Epoch: [3] [230/250] eta: 0:00:12 lr: 0.000500 loss: 0.2824 (0.2752) loss_classifier: 0.0971 (0.0889) loss_box_reg: 0.1677 (0.1526) loss_objectness: 0.0075 (0.0104) loss_rpn_box_reg: 0.0176 (0.0233) time: 0.5935 data: 0.0068 max mem: 4722\n",
"Epoch: [3] [240/250] eta: 0:00:06 lr: 0.000500 loss: 0.2824 (0.2744) loss_classifier: 0.0890 (0.0886) loss_box_reg: 0.1466 (0.1524) loss_objectness: 0.0096 (0.0104) loss_rpn_box_reg: 0.0176 (0.0230) time: 0.5977 data: 0.0072 max mem: 4722\n",
"Epoch: [3] [249/250] eta: 0:00:00 lr: 0.000500 loss: 0.2845 (0.2753) loss_classifier: 0.0816 (0.0888) loss_box_reg: 0.1415 (0.1522) loss_objectness: 0.0104 (0.0105) loss_rpn_box_reg: 0.0116 (0.0238) time: 0.6015 data: 0.0070 max mem: 4722\n",
"Epoch: [3] Total time: 0:02:35 (0.6210 s / it)\n",
"creating index...\n",
"index created!\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n",
"/usr/local/lib/python3.7/dist-packages/pymongo/topology.py:162: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe\n",
" \"MongoClient opened before fork. Create MongoClient only \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Test: [ 0/207] eta: 0:01:11 model_time: 0.1685 (0.1685) evaluator_time: 0.0073 (0.0073) time: 0.3448 data: 0.1672 max mem: 4722\n",
"Test: [100/207] eta: 0:00:13 model_time: 0.1207 (0.1173) evaluator_time: 0.0019 (0.0033) time: 0.1277 data: 0.0034 max mem: 4722\n",
"Test: [200/207] eta: 0:00:00 model_time: 0.1126 (0.1174) evaluator_time: 0.0018 (0.0033) time: 0.1206 data: 0.0035 max mem: 4722\n",
"Test: [206/207] eta: 0:00:00 model_time: 0.1136 (0.1174) evaluator_time: 0.0026 (0.0035) time: 0.1245 data: 0.0035 max mem: 4722\n",
"Test: Total time: 0:00:26 (0.1267 s / it)\n",
"Averaged stats: model_time: 0.1136 (0.1174) evaluator_time: 0.0026 (0.0035)\n",
"Accumulating evaluation results...\n",
"DONE (t=0.05s).\n",
"IoU metric: bbox\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.428\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.688\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.459\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.315\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.507\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.615\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.182\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.502\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.544\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.441\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.614\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.713\n"
]
}
],
"source": [
"do_training(vehicle_model, torch_map_dataset, torch_map_dataset_test)"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Y-mrVOl4XFbp",
"outputId": "6d8bec76-ebe8-4a36-959a-52bb1aab8498"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Using device cuda\n",
" 100% |█████████████████| 207/207 [34.8s elapsed, 0s remaining, 5.5 samples/s] \n"
]
}
],
"source": [
"add_detections(vehicle_model, torch_map_dataset_test, test_map_view, field_name=\"vehicle_predictions\")"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "hfd3xvhaXhl_",
"outputId": "d9c4a2fe-538a-4979-c3f8-f5ede0c98aa1"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Evaluating detections...\n",
" 100% |█████████████████| 207/207 [5.4s elapsed, 0s remaining, 36.1 samples/s] \n",
"Performing IoU sweep...\n",
" 100% |█████████████████| 207/207 [10.0s elapsed, 0s remaining, 15.6 samples/s] \n"
]
}
],
"source": [
"vehicle_results = fo.evaluate_detections(\n",
" test_map_view, \n",
" \"vehicle_predictions\", \n",
" classes=[\"vehicle\"], \n",
" eval_key=\"vehicle_eval\", \n",
" compute_mAP=True\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "kFvddH3rk0NR",
"outputId": "59572ba2-f9ad-4dd2-e9ac-90877190ff99"
},
"outputs": [
{
"data": {
"text/plain": [
"0.43066431144550577"
]
},
"execution_count": 48,
"metadata": {
"tags": []
},
"output_type": "execute_result"
}
],
"source": [
"vehicle_results.mAP()"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "rwbhq18sk1PL",
"outputId": "d6985867-5049-4678-cc88-d5041a0079ed"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" precision recall f1-score support\n",
"\n",
" vehicle 0.18 0.89 0.30 791\n",
"\n",
" micro avg 0.18 0.89 0.30 791\n",
" macro avg 0.18 0.89 0.30 791\n",
"weighted avg 0.18 0.89 0.30 791\n",
"\n"
]
}
],
"source": [
"vehicle_results.print_report()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gJMAkJbWZ_u1"
},
"source": [
"Due to our ability to easily visualize and manage our dataset with FiftyOne, we were able to spot and take action on a dataset issue that would otherwise have gone unnoticed if we only concerned ourselves with dataset-wide evaluation metrics and fixed dataset representations. Through these efforts, we managed to increase the mAP of the model to 43%.\n",
"\n",
"Even though this example workflow may not work in all situations, this kind of class-merging strategy can be effective in cases where more fine-grained discrimination is not called for."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a0ZyjvCaaMTV"
},
"source": [
"## Summary\n",
"\n",
"PyTorch and related frameworks provide quick and easy methods to bootstrap your model development and training pipelines. However, they largely overlook the need to massage and finetune datasets to efficiently improve performance. FiftyOne makes it easy to load your datasets into a flexible format that works well with existing tools allowing you to provide better data for training and testing. As they say, \"garbage in, garbage out\"."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"name": "fiftyone_pytorch_training.ipynb",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.13"
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"32b6ec3046e64d04b4134553dc434fe0": {
"model_module": "@jupyter-widgets/base",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"4a4788a4fd6841788b20cfbf54a3d10b": {
"model_module": "@jupyter-widgets/controls",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": "initial"
}
},
"5d836b94d13e459d82429606496e4d4f": {
"model_module": "@jupyter-widgets/base",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"a1645bdfb02b42fba268f7000f183639": {
"model_module": "@jupyter-widgets/controls",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_c063e7d90f6a4027b53d1b70c8c07742",
"placeholder": "",
"style": "IPY_MODEL_a410071b34034a91aeda7ef1114969c2",
"value": " 160M/160M [01:05<00:00, 2.55MB/s]"
}
},
"a410071b34034a91aeda7ef1114969c2": {
"model_module": "@jupyter-widgets/controls",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"acbb3df601244291b8b2fb9ea1137573": {
"model_module": "@jupyter-widgets/controls",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_d8c6a316609d4ca5bfee139b93177ef5",
"IPY_MODEL_a1645bdfb02b42fba268f7000f183639"
],
"layout": "IPY_MODEL_32b6ec3046e64d04b4134553dc434fe0"
}
},
"c063e7d90f6a4027b53d1b70c8c07742": {
"model_module": "@jupyter-widgets/base",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"d8c6a316609d4ca5bfee139b93177ef5": {
"model_module": "@jupyter-widgets/controls",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "100%",
"description_tooltip": null,
"layout": "IPY_MODEL_5d836b94d13e459d82429606496e4d4f",
"max": 167502836,
"min": 0,
"orientation": "horizontal",
"style": "IPY_MODEL_4a4788a4fd6841788b20cfbf54a3d10b",
"value": 167502836
}
}
}
}
},
"nbformat": 4,
"nbformat_minor": 1
}
![](\"https://user-images.githubusercontent.com/25985824/104791629-6e618700-5769-11eb-857f-d176b37d2496.png\")
\n",
" Try in Google Colab\n",
" \n",
" ![](\"https://user-images.githubusercontent.com/25985824/104791634-6efa1d80-5769-11eb-8a4c-71d6cb53ccf0.png\")
\n",
" Share via nbviewer\n",
" \n",
" ![](\"https://user-images.githubusercontent.com/25985824/104791633-6efa1d80-5769-11eb-8ee3-4b2123fe4b66.png\")
\n",
" View on GitHub\n",
" \n",
" ![](\"https://user-images.githubusercontent.com/25985824/104792428-60f9cc00-576c-11eb-95a4-5709d803023a.png\")
\n",
" Download notebook\n",
" \n",
" 
\n",
" \n",
" "
],
"text/plain": [
"
\n",
" \n",
" "
],
"text/plain": [
"
\n",
" \n",
" "
],
"text/plain": [
"
\n",
" \n",
" "
],
"text/plain": [
"
\n",
" \n",
" "
],
"text/plain": [
"