{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Convert Dataset Formats\n",
"\n",
"This recipe demonstrates how to use FiftyOne to convert datasets on disk between common formats."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you haven't already, install FiftyOne:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pip install fiftyone"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook contains bash commands. To run it as a notebook, you must install the [Jupyter bash kernel](https://github.com/takluyver/bash_kernel) via the command below.\n",
"\n",
"Alternatively, you can just copy + paste the code blocks into your shell."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"pip install bash_kernel\n",
"python -m bash_kernel.install"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this recipe we'll use the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/zoo_datasets.html) to download some open source datasets to work with.\n",
"\n",
"Specifically, we'll need [TensorFlow](https://www.tensorflow.org/) and [TensorFlow Datasets](https://www.tensorflow.org/datasets) installed to [access the datasets](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/zoo_datasets.html#customizing-your-ml-backend):"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"pip install tensorflow tensorflow-datasets"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Download datasets\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Download the test split of the [CIFAR-10 dataset](https://www.cs.toronto.edu/~kriz/cifar.html) from the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/zoo_datasets.html) using the command below:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Downloading split 'test' to '~/fiftyone/cifar10/test'\n",
"Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ~/fiftyone/cifar10/tmp-download/cifar-10-python.tar.gz\n",
"170500096it [00:04, 35887670.65it/s] \n",
"Extracting ~/fiftyone/cifar10/tmp-download/cifar-10-python.tar.gz to ~/fiftyone/cifar10/tmp-download\n",
" 100% |███| 10000/10000 [5.2s elapsed, 0s remaining, 1.8K samples/s] \n",
"Dataset info written to '~/fiftyone/cifar10/info.json'\n"
]
}
],
"source": [
"# Download the test split of CIFAR-10\n",
"fiftyone zoo datasets download cifar10 --split test"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Download the validation split of the [KITTI dataset]( http://www.cvlibs.net/datasets/kitti) from the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/zoo_datasets.html) using the command below:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Split 'validation' already downloaded\n"
]
}
],
"source": [
"# Download the validation split of KITTI\n",
"fiftyone zoo datasets download kitti --split validation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The fiftyone convert command"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The [FiftyOne CLI](https://voxel51.com/docs/fiftyone/cli/index.html) provides a number of utilities for importing and exporting datasets in a variety of common (or custom) formats.\n",
"\n",
"Specifically, the `fiftyone convert` command provides a convenient way to convert datasets on disk between formats by specifying the [fiftyone.types.Dataset](https://voxel51.com/docs/fiftyone/api/fiftyone.types.html#fiftyone.types.dataset_types.Dataset) type of the input and desired output.\n",
"\n",
"FiftyOne provides a collection of [builtin types](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#supported-formats) that you can use to read/write datasets in common formats out-of-the-box:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"convert-recipes-table\">\n",
"\n",
"| Dataset format | Import Supported? | Export Supported? | Conversion Supported? |\n",
"| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | ----------------- | --------------------- |\n",
"| [ImageDirectory](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#imagedirectory) | ✓ | ✓ | ✓ |\n",
"| [VideoDirectory](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#videodirectory) | ✓ | ✓ | ✓ |\n",
"| [FiftyOneImageClassificationDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#fiftyoneimageclassificationdataset) | ✓ | ✓ | ✓ |\n",
"| [ImageClassificationDirectoryTree](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#imageclassificationdirectorytree) | ✓ | ✓ | ✓ |\n",
"| [TFImageClassificationDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#tfimageclassificationdataset) | ✓ | ✓ | ✓ |\n",
"| [FiftyOneImageDetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#fiftyoneimagedetectiondataset) | ✓ | ✓ | ✓ |\n",
"| [COCODetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#cocodetectiondataset) | ✓ | ✓ | ✓ |\n",
"| [VOCDetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#vocdetectiondataset) | ✓ | ✓ | ✓ |\n",
"| [KITTIDetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#kittidetectiondataset) | ✓ | ✓ | ✓ |\n",
"| [YOLODataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#yolodataset) | ✓ | ✓ | ✓ |\n",
"| [TFObjectDetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#tfobjectdetectiondataset) | ✓ | ✓ | ✓ |\n",
"| [CVATImageDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#cvatimagedataset) | ✓ | ✓ | ✓ |\n",
"| [CVATVideoDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#cvatvideodataset) | ✓ | ✓ | ✓ |\n",
"| [FiftyOneImageLabelsDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#fiftyoneimagelabelsdataset) | ✓ | ✓ | ✓ |\n",
"| [FiftyOneVideoLabelsDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#fiftyonevideolabelsdataset) | ✓ | ✓ | ✓ |\n",
"| [BDDDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#bdddataset) | ✓ | ✓ | ✓ |\n",
"\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In addition, you can define your own [custom dataset types](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#custom-formats) to read/write datasets in your own formats.\n",
"\n",
"The usage of the `fiftyone convert` command is as follows:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"usage: fiftyone convert [-h] --input-type INPUT_TYPE --output-type OUTPUT_TYPE\n",
" [--input-dir INPUT_DIR]\n",
" [--input-kwargs KEY=VAL [KEY=VAL ...]]\n",
" [--output-dir OUTPUT_DIR]\n",
" [--output-kwargs KEY=VAL [KEY=VAL ...]] [-o]\n",
"\n",
"Convert datasets on disk between supported formats.\n",
"\n",
" Examples::\n",
"\n",
" # Convert an image classification directory tree to TFRecords format\n",
" fiftyone convert \\\n",
" --input-dir /path/to/image-classification-directory-tree \\\n",
" --input-type fiftyone.types.ImageClassificationDirectoryTree \\\n",
" --output-dir /path/for/tf-image-classification-dataset \\\n",
" --output-type fiftyone.types.TFImageClassificationDataset\n",
"\n",
" # Convert a COCO detection dataset to CVAT image format\n",
" fiftyone convert \\\n",
" --input-dir /path/to/coco-detection-dataset \\\n",
" --input-type fiftyone.types.COCODetectionDataset \\\n",
" --output-dir /path/for/cvat-image-dataset \\\n",
" --output-type fiftyone.types.CVATImageDataset\n",
"\n",
" # Perform a customized conversion via optional kwargs\n",
" fiftyone convert \\\n",
" --input-dir /path/to/coco-detection-dataset \\\n",
" --input-type fiftyone.types.COCODetectionDataset \\\n",
" --input-kwargs max_samples=100 shuffle=True \\\n",
" --output-dir /path/for/cvat-image-dataset \\\n",
" --output-type fiftyone.types.TFObjectDetectionDataset \\\n",
" --output-kwargs force_rgb=True \\\n",
" --overwrite\n",
"\n",
"optional arguments:\n",
" -h, --help show this help message and exit\n",
" --input-dir INPUT_DIR\n",
" the directory containing the dataset\n",
" --input-kwargs KEY=VAL [KEY=VAL ...]\n",
" additional keyword arguments for `fiftyone.utils.data.convert_dataset(..., input_kwargs=)`\n",
" --output-dir OUTPUT_DIR\n",
" the directory to which to write the output dataset\n",
" --output-kwargs KEY=VAL [KEY=VAL ...]\n",
" additional keyword arguments for `fiftyone.utils.data.convert_dataset(..., output_kwargs=)`\n",
" -o, --overwrite whether to overwrite an existing output directory\n",
"\n",
"required arguments:\n",
" --input-type INPUT_TYPE\n",
" the fiftyone.types.Dataset type of the input dataset\n",
" --output-type OUTPUT_TYPE\n",
" the fiftyone.types.Dataset type to output\n"
]
}
],
"source": [
"fiftyone convert -h"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Convert CIFAR-10 dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When you downloaded the test split of the CIFAR-10 dataset above, it was written to disk as a dataset in [fiftyone.types.FiftyOneImageClassificationDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#fiftyoneimageclassificationdataset) format.\n",
"\n",
"You can verify this by printing information about the downloaded dataset:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"***** Dataset description *****\n",
"The CIFAR-10 dataset consists of 60000 32 x 32 color images in 10\n",
" classes, with 6000 images per class. There are 50000 training images and\n",
" 10000 test images.\n",
"\n",
" Dataset size:\n",
" 132.40 MiB\n",
"\n",
" Source:\n",
" https://www.cs.toronto.edu/~kriz/cifar.html\n",
" \n",
"***** Supported splits *****\n",
"test, train\n",
"\n",
"***** Dataset location *****\n",
"~/fiftyone/cifar10\n",
"\n",
"***** Dataset info *****\n",
"{\n",
" \"name\": \"cifar10\",\n",
" \"zoo_dataset\": \"fiftyone.zoo.datasets.torch.CIFAR10Dataset\",\n",
" \"dataset_type\": \"fiftyone.types.dataset_types.FiftyOneImageClassificationDataset\",\n",
" \"num_samples\": 10000,\n",
" \"downloaded_splits\": {\n",
" \"test\": {\n",
" \"split\": \"test\",\n",
" \"num_samples\": 10000\n",
" }\n",
" },\n",
" \"classes\": [\n",
" \"airplane\",\n",
" \"automobile\",\n",
" \"bird\",\n",
" \"cat\",\n",
" \"deer\",\n",
" \"dog\",\n",
" \"frog\",\n",
" \"horse\",\n",
" \"ship\",\n",
" \"truck\"\n",
" ]\n",
"}\n"
]
}
],
"source": [
"fiftyone zoo datasets info cifar10"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The snippet below uses `fiftyone convert` to convert the test split of the CIFAR-10 dataset to [fiftyone.types.ImageClassificationDirectoryTree](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#imageclassificationdirectorytree) format, which stores classification datasets on disk in a directory tree structure with images organized per-class:\n",
"\n",
"```\n",
"<dataset_dir>\n",
"├── <classA>/\n",
"│ ├── <image1>.<ext>\n",
"│ ├── <image2>.<ext>\n",
"│ └── ...\n",
"├── <classB>/\n",
"│ ├── <image1>.<ext>\n",
"│ ├── <image2>.<ext>\n",
"│ └── ...\n",
"└── ...\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loading dataset from '~/fiftyone/cifar10/test'\n",
"Input format 'fiftyone.types.dataset_types.FiftyOneImageClassificationDataset'\n",
" 100% |███| 10000/10000 [4.2s elapsed, 0s remaining, 2.4K samples/s] \n",
"Import complete\n",
"Exporting dataset to '/tmp/fiftyone/cifar10-dir-tree'\n",
"Export format 'fiftyone.types.dataset_types.ImageClassificationDirectoryTree'\n",
" 100% |███| 10000/10000 [6.2s elapsed, 0s remaining, 1.7K samples/s] \n",
"Export complete\n"
]
}
],
"source": [
"INPUT_DIR=$(fiftyone zoo datasets find cifar10 --split test)\n",
"OUTPUT_DIR=/tmp/fiftyone/cifar10-dir-tree\n",
"\n",
"fiftyone convert \\\n",
" --input-dir ${INPUT_DIR} --input-type fiftyone.types.FiftyOneImageClassificationDataset \\\n",
" --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.ImageClassificationDirectoryTree"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's verify that the conversion happened as expected:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 0\n",
"drwxr-xr-x 12 voxel51 wheel 384B Jul 14 11:08 .\n",
"drwxr-xr-x 3 voxel51 wheel 96B Jul 14 11:08 ..\n",
"drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 airplane\n",
"drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 automobile\n",
"drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 bird\n",
"drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 cat\n",
"drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 deer\n",
"drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 dog\n",
"drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 frog\n",
"drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 horse\n",
"drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 ship\n",
"drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 truck\n"
]
}
],
"source": [
"ls -lah /tmp/fiftyone/cifar10-dir-tree/"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 8000\n",
"drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 .\n",
"drwxr-xr-x 12 voxel51 wheel 384B Jul 14 11:08 ..\n",
"-rw-r--r-- 1 voxel51 wheel 1.2K Jul 14 11:23 000004.jpg\n",
"-rw-r--r-- 1 voxel51 wheel 1.1K Jul 14 11:23 000011.jpg\n",
"-rw-r--r-- 1 voxel51 wheel 1.1K Jul 14 11:23 000022.jpg\n",
"-rw-r--r-- 1 voxel51 wheel 1.3K Jul 14 11:23 000028.jpg\n",
"-rw-r--r-- 1 voxel51 wheel 1.2K Jul 14 11:23 000045.jpg\n",
"-rw-r--r-- 1 voxel51 wheel 1.2K Jul 14 11:23 000053.jpg\n",
"-rw-r--r-- 1 voxel51 wheel 1.3K Jul 14 11:23 000075.jpg\n"
]
}
],
"source": [
"ls -lah /tmp/fiftyone/cifar10-dir-tree/airplane/ | head"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's convert the classification directory tree to [TFRecords](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#tfimageclassificationdataset) format!"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loading dataset from '/tmp/fiftyone/cifar10-dir-tree'\n",
"Input format 'fiftyone.types.dataset_types.ImageClassificationDirectoryTree'\n",
" 100% |███| 10000/10000 [4.0s elapsed, 0s remaining, 2.5K samples/s] \n",
"Import complete\n",
"Exporting dataset to '/tmp/fiftyone/cifar10-tfrecords'\n",
"Export format 'fiftyone.types.dataset_types.TFImageClassificationDataset'\n",
" 0% ||--| 1/10000 [23.2ms elapsed, 3.9m remaining, 43.2 samples/s] 2020-07-14 11:24:15.187387: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n",
"2020-07-14 11:24:15.201384: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f83df428f60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
"2020-07-14 11:24:15.201405: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version\n",
" 100% |███| 10000/10000 [8.2s elapsed, 0s remaining, 1.3K samples/s] \n",
"Export complete\n"
]
}
],
"source": [
"INPUT_DIR=/tmp/fiftyone/cifar10-dir-tree\n",
"OUTPUT_DIR=/tmp/fiftyone/cifar10-tfrecords\n",
"\n",
"fiftyone convert \\\n",
" --input-dir ${INPUT_DIR} --input-type fiftyone.types.ImageClassificationDirectoryTree \\\n",
" --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.TFImageClassificationDataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's verify that the conversion happened as expected:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 29696\n",
"drwxr-xr-x 3 voxel51 wheel 96B Jul 14 11:24 .\n",
"drwxr-xr-x 4 voxel51 wheel 128B Jul 14 11:24 ..\n",
"-rw-r--r-- 1 voxel51 wheel 14M Jul 14 11:24 tf.records\n"
]
}
],
"source": [
"ls -lah /tmp/fiftyone/cifar10-tfrecords"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Convert KITTI dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When you downloaded the validation split of the KITTI dataset above, it was written to disk as a dataset in [fiftyone.types.FiftyOneImageDetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/datasets.html#fiftyoneimagedetectiondataset) format.\n",
"\n",
"You can verify this by printing information about the downloaded dataset:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"***** Dataset description *****\n",
"KITTI contains a suite of vision tasks built using an autonomous\n",
" driving platform.\n",
"\n",
" The full benchmark contains many tasks such as stereo, optical flow, visual\n",
" odometry, etc. This dataset contains the object detection dataset,\n",
" including the monocular images and bounding boxes. The dataset contains\n",
" 7481 training images annotated with 3D bounding boxes. A full description\n",
" of the annotations can be found in the README of the object development kit\n",
" on the KITTI homepage.\n",
"\n",
" Dataset size:\n",
" 5.27 GiB\n",
"\n",
" Source:\n",
" http://www.cvlibs.net/datasets/kitti\n",
" \n",
"***** Supported splits *****\n",
"test, train, validation\n",
"\n",
"***** Dataset location *****\n",
"~/fiftyone/kitti\n",
"\n",
"***** Dataset info *****\n",
"{\n",
" \"name\": \"kitti\",\n",
" \"zoo_dataset\": \"fiftyone.zoo.datasets.tf.KITTIDataset\",\n",
" \"dataset_type\": \"fiftyone.types.dataset_types.FiftyOneImageDetectionDataset\",\n",
" \"num_samples\": 423,\n",
" \"downloaded_splits\": {\n",
" \"validation\": {\n",
" \"split\": \"validation\",\n",
" \"num_samples\": 423\n",
" }\n",
" },\n",
" \"classes\": [\n",
" \"Car\",\n",
" \"Van\",\n",
" \"Truck\",\n",
" \"Pedestrian\",\n",
" \"Person_sitting\",\n",
" \"Cyclist\",\n",
" \"Tram\",\n",
" \"Misc\"\n",
" ]\n",
"}\n"
]
}
],
"source": [
"fiftyone zoo datasets info kitti"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The snippet below uses `fiftyone convert` to convert the test split of the CIFAR-10 dataset to [fiftyone.types.COCODetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#cocodetectiondataset) format, which writes the dataset to disk with annotations in [COCO format](https://cocodataset.org/#format-data)."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loading dataset from '~/fiftyone/kitti/validation'\n",
"Input format 'fiftyone.types.dataset_types.FiftyOneImageDetectionDataset'\n",
" 100% |███████| 423/423 [1.2s elapsed, 0s remaining, 351.0 samples/s] \n",
"Import complete\n",
"Exporting dataset to '/tmp/fiftyone/kitti-coco'\n",
"Export format 'fiftyone.types.dataset_types.COCODetectionDataset'\n",
" 100% |███████| 423/423 [4.4s elapsed, 0s remaining, 96.1 samples/s] \n",
"Export complete\n"
]
}
],
"source": [
"INPUT_DIR=$(fiftyone zoo datasets find kitti --split validation)\n",
"OUTPUT_DIR=/tmp/fiftyone/kitti-coco\n",
"\n",
"fiftyone convert \\\n",
" --input-dir ${INPUT_DIR} --input-type fiftyone.types.FiftyOneImageDetectionDataset \\\n",
" --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.COCODetectionDataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's verify that the conversion happened as expected:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 880\n",
"drwxr-xr-x 4 voxel51 wheel 128B Jul 14 11:24 .\n",
"drwxr-xr-x 5 voxel51 wheel 160B Jul 14 11:24 ..\n",
"drwxr-xr-x 425 voxel51 wheel 13K Jul 14 11:24 data\n",
"-rw-r--r-- 1 voxel51 wheel 437K Jul 14 11:24 labels.json\n"
]
}
],
"source": [
"ls -lah /tmp/fiftyone/kitti-coco/"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 171008\n",
"drwxr-xr-x 425 voxel51 wheel 13K Jul 14 11:24 .\n",
"drwxr-xr-x 4 voxel51 wheel 128B Jul 14 11:24 ..\n",
"-rw-r--r-- 1 voxel51 wheel 195K Jul 14 11:24 000001.jpg\n",
"-rw-r--r-- 1 voxel51 wheel 191K Jul 14 11:24 000002.jpg\n",
"-rw-r--r-- 1 voxel51 wheel 167K Jul 14 11:24 000003.jpg\n",
"-rw-r--r-- 1 voxel51 wheel 196K Jul 14 11:24 000004.jpg\n",
"-rw-r--r-- 1 voxel51 wheel 224K Jul 14 11:24 000005.jpg\n",
"-rw-r--r-- 1 voxel51 wheel 195K Jul 14 11:24 000006.jpg\n",
"-rw-r--r-- 1 voxel51 wheel 177K Jul 14 11:24 000007.jpg\n"
]
}
],
"source": [
"ls -lah /tmp/fiftyone/kitti-coco/data | head"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"info\": {\n",
" \"year\": \"\",\n",
" \"version\": \"\",\n",
" \"description\": \"Exported from FiftyOne\",\n",
" \"contributor\": \"\",\n",
" \"url\": \"https://voxel51.com/fiftyone\",\n",
" \"date_created\": \"2020-07-14T11:24:40\"\n",
" },\n",
" \"licenses\": [],\n",
" \"categories\": [\n",
" {\n",
" \"id\": 0,\n",
" \"name\": \"Car\",\n",
" \"supercategory\": \"none\"\n",
" },\n",
" {\n",
" \"id\": 1,\n",
" \"name\": \"Cyclist\",\n",
" \"supercategory\": \"none\"\n",
"...\n",
" \"area\": 4545.8,\n",
" \"segmentation\": null,\n",
" \"iscrowd\": 0\n",
" },\n",
" {\n",
" \"id\": 3196,\n",
" \"image_id\": 422,\n",
" \"category_id\": 3,\n",
" \"bbox\": [\n",
" 367.2,\n",
" 107.3,\n",
" 36.2,\n",
" 105.2\n",
" ],\n",
" \"area\": 3808.2,\n",
" \"segmentation\": null,\n",
" \"iscrowd\": 0\n",
" }\n",
" ]\n",
"}\n"
]
}
],
"source": [
"cat /tmp/fiftyone/kitti-coco/labels.json | python -m json.tool 2> /dev/null | head -20\n",
"echo \"...\"\n",
"cat /tmp/fiftyone/kitti-coco/labels.json | python -m json.tool 2> /dev/null | tail -20"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's convert from COCO format to [CVAT Image format](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#cvatimageformat) format!"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loading dataset from '/tmp/fiftyone/kitti-coco'\n",
"Input format 'fiftyone.types.dataset_types.COCODetectionDataset'\n",
" 100% |███████| 423/423 [2.0s elapsed, 0s remaining, 206.4 samples/s] \n",
"Import complete\n",
"Exporting dataset to '/tmp/fiftyone/kitti-cvat'\n",
"Export format 'fiftyone.types.dataset_types.CVATImageDataset'\n",
" 100% |███████| 423/423 [1.3s elapsed, 0s remaining, 323.7 samples/s] \n",
"Export complete\n"
]
}
],
"source": [
"INPUT_DIR=/tmp/fiftyone/kitti-coco\n",
"OUTPUT_DIR=/tmp/fiftyone/kitti-cvat\n",
"\n",
"fiftyone convert \\\n",
" --input-dir ${INPUT_DIR} --input-type fiftyone.types.COCODetectionDataset \\\n",
" --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.CVATImageDataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's verify that the conversion happened as expected:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 584\n",
"drwxr-xr-x 4 voxel51 wheel 128B Jul 14 11:25 .\n",
"drwxr-xr-x 6 voxel51 wheel 192B Jul 14 11:25 ..\n",
"drwxr-xr-x 425 voxel51 wheel 13K Jul 14 11:25 data\n",
"-rw-r--r-- 1 voxel51 wheel 289K Jul 14 11:25 labels.xml\n"
]
}
],
"source": [
"ls -lah /tmp/fiftyone/kitti-cvat"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
"<annotations>\n",
" <version>1.1</version>\n",
" <meta>\n",
" <task>\n",
" <size>423</size>\n",
" <mode>annotation</mode>\n",
" <labels>\n",
" <label>\n",
" <name>Car</name>\n",
" <attributes>\n",
" </attributes>\n",
" </label>\n",
" <label>\n",
" <name>Cyclist</name>\n",
" <attributes>\n",
" </attributes>\n",
" </label>\n",
" <label>\n",
" <name>Misc</name>\n",
"...\n",
" <box label=\"Pedestrian\" xtl=\"360\" ytl=\"116\" xbr=\"402\" ybr=\"212\">\n",
" </box>\n",
" <box label=\"Pedestrian\" xtl=\"396\" ytl=\"120\" xbr=\"430\" ybr=\"212\">\n",
" </box>\n",
" <box label=\"Pedestrian\" xtl=\"413\" ytl=\"112\" xbr=\"483\" ybr=\"212\">\n",
" </box>\n",
" <box label=\"Pedestrian\" xtl=\"585\" ytl=\"80\" xbr=\"646\" ybr=\"215\">\n",
" </box>\n",
" <box label=\"Pedestrian\" xtl=\"635\" ytl=\"94\" xbr=\"688\" ybr=\"212\">\n",
" </box>\n",
" <box label=\"Pedestrian\" xtl=\"422\" ytl=\"85\" xbr=\"469\" ybr=\"210\">\n",
" </box>\n",
" <box label=\"Pedestrian\" xtl=\"457\" ytl=\"93\" xbr=\"520\" ybr=\"213\">\n",
" </box>\n",
" <box label=\"Pedestrian\" xtl=\"505\" ytl=\"101\" xbr=\"548\" ybr=\"206\">\n",
" </box>\n",
" <box label=\"Pedestrian\" xtl=\"367\" ytl=\"107\" xbr=\"403\" ybr=\"212\">\n",
" </box>\n",
" </image>\n",
"</annotations>"
]
}
],
"source": [
"cat /tmp/fiftyone/kitti-cvat/labels.xml | head -20\n",
"echo \"...\"\n",
"cat /tmp/fiftyone/kitti-cvat/labels.xml | tail -20"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Cleanup\n",
"\n",
"You can cleanup the files generated by this recipe by running the command below:"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"rm -rf /tmp/fiftyone"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Bash",
"language": "bash",
"name": "bash"
},
"language_info": {
"codemirror_mode": "shell",
"file_extension": ".sh",
"mimetype": "text/x-sh",
"name": "bash"
}
},
"nbformat": 4,
"nbformat_minor": 4
}