{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Convert Dataset Formats\n", "\n", "This recipe demonstrates how to use FiftyOne to convert datasets on disk between common formats." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you haven't already, install FiftyOne:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pip install fiftyone" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This notebook contains bash commands. To run it as a notebook, you must install the [Jupyter bash kernel](https://github.com/takluyver/bash_kernel) via the command below.\n", "\n", "Alternatively, you can just copy + paste the code blocks into your shell." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "pip install bash_kernel\n", "python -m bash_kernel.install" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this recipe we'll use the [FiftyOne Dataset Zoo](https://docs.voxel51.com/dataset_zoo/index.html) to download some open source datasets to work with.\n", "\n", "Specifically, we'll need [TensorFlow](https://www.tensorflow.org/) and [TensorFlow Datasets](https://www.tensorflow.org/datasets) installed to [access the datasets](https://docs.voxel51.com/dataset_zoo/api.html#customizing-your-ml-backend):" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "pip install tensorflow tensorflow-datasets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Download datasets\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Download the test split of the [CIFAR-10 dataset](https://www.cs.toronto.edu/~kriz/cifar.html) from the [FiftyOne Dataset Zoo](https://docs.voxel51.com/dataset_zoo/index.html) using the command below:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Downloading split 'test' to '~/fiftyone/cifar10/test'\n", "Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ~/fiftyone/cifar10/tmp-download/cifar-10-python.tar.gz\n", "170500096it [00:04, 35887670.65it/s] \n", "Extracting ~/fiftyone/cifar10/tmp-download/cifar-10-python.tar.gz to ~/fiftyone/cifar10/tmp-download\n", " 100% |███| 10000/10000 [5.2s elapsed, 0s remaining, 1.8K samples/s] \n", "Dataset info written to '~/fiftyone/cifar10/info.json'\n" ] } ], "source": [ "# Download the test split of CIFAR-10\n", "fiftyone zoo datasets download cifar10 --split test" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Download the validation split of the [KITTI dataset]( http://www.cvlibs.net/datasets/kitti) from the [FiftyOne Dataset Zoo](https://docs.voxel51.com/dataset_zoo/index.html) using the command below:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Split 'validation' already downloaded\n" ] } ], "source": [ "# Download the validation split of KITTI\n", "fiftyone zoo datasets download kitti --split validation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The fiftyone convert command" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The [FiftyOne CLI](https://voxel51.com/docs/fiftyone/cli/index.html) provides a number of utilities for importing and exporting datasets in a variety of common (or custom) formats.\n", "\n", "Specifically, the `fiftyone convert` command provides a convenient way to convert datasets on disk between formats by specifying the [fiftyone.types.Dataset](https://voxel51.com/docs/fiftyone/api/fiftyone.types.html#fiftyone.types.dataset_types.Dataset) type of the input and desired output.\n", "\n", "FiftyOne provides a collection of [builtin types](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#supported-import-formats) that you can use to read/write datasets in common formats out-of-the-box:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "| Dataset format | Import Supported? | Export Supported? | Conversion Supported? |\n", "| ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------- | ----------------- | --------------------- |\n", "| [Image Directory](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#ImageDirectory-import) | ✓ | ✓ | ✓ |\n", "| [Video Directory](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#VideoDirectory-import) | ✓ | ✓ | ✓ |\n", "| [Media Directory](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#MediaDirectory-import) | ✓ | ✓ | ✓ |\n", "| [Image Classification Directory Tree](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#ImageClassificationDirectoryTree-import) | ✓ | ✓ | ✓ |\n", "| [FiftyOne Image Classification](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#FiftyOneImageClassificationDataset-import) | ✓ | ✓ | ✓ |\n", "| [TF Image Classification](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#TFImageClassificationDataset-import) | ✓ | ✓ | ✓ |\n", "| [COCO](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#COCODetectionDataset-import) | ✓ | ✓ | ✓ |\n", "| [VOC](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#VOCDetectionDataset-import) | ✓ | ✓ | ✓ |\n", "| [KITTI](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#KITTIDetectionDataset-import) | ✓ | ✓ | ✓ |\n", "| [YOLOv4](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#YOLOv4Dataset-import) | ✓ | ✓ | ✓ |\n", "| [YOLOv5](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#YOLOv5Dataset-import) | ✓ | ✓ | ✓ |\n", "| [FiftyOne Object Detection](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#FiftyOneImageDetectionDataset-import) | ✓ | ✓ | ✓ |\n", "| [FiftyOne Temporal Detection](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#FiftyOneTemporalDetectionDataset-import) | ✓ | ✓ | ✓ |\n", "| [TF Object Detection](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#TFObjectDetectionDataset-import) | ✓ | ✓ | ✓ |\n", "| [Image Segmentation Directory](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#ImageSegmentationDirectory-import) | ✓ | ✓ | ✓ |\n", "| [CVAT Image](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#CVATImageDataset-import) | ✓ | ✓ | ✓ |\n", "| [CVAT Video](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#CVATVideoDataset-import) | ✓ | ✓ | ✓ |\n", "| [OpenLABEL Image](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#OpenLABELImageDataset-import) | ✓ | | |\n", "| [OpenLABEL Video](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#OpenLABELVideoDataset-import) | ✓ | | |\n", "| [BDD](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#BDDDataset-import) | ✓ | ✓ | ✓ |\n", "| [CSV](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#CSVDataset-import) | ✓ | ✓ | ✓ |\n", "| [DICOM](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#DICOMDataset-import) | ✓ | ✓ | ✓ |\n", "| [GeoJSON](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#GeoJSONDataset-import) | ✓ | ✓ | ✓ |\n", "| [GeoTIFF](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#GeoTIFFDataset-import) | ✓ | | |\n", "| [FiftyOne Dataset](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#FiftyOneDataset-import) | ✓ | ✓ | ✓ |\n", "| [FiftyOne Image Labels](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#FiftyOneImageLabelsDataset-import) | ✓ | ✓ | ✓ |\n", "| [FiftyOne Video Labels](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#FiftyOneVideoLabelsDataset-import) | ✓ | ✓ | ✓ |\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In addition, you can define your own [custom dataset types](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#custom-dataset-importer) to read/write datasets in your own formats.\n", "\n", "The usage of the `fiftyone convert` command is as follows:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "usage: fiftyone convert [-h] --input-type INPUT_TYPE --output-type OUTPUT_TYPE\n", " [--input-dir INPUT_DIR]\n", " [--input-kwargs KEY=VAL [KEY=VAL ...]]\n", " [--output-dir OUTPUT_DIR]\n", " [--output-kwargs KEY=VAL [KEY=VAL ...]] [-o]\n", "\n", "Convert datasets on disk between supported formats.\n", "\n", " Examples::\n", "\n", " # Convert an image classification directory tree to TFRecords format\n", " fiftyone convert \\\n", " --input-dir /path/to/image-classification-directory-tree \\\n", " --input-type fiftyone.types.ImageClassificationDirectoryTree \\\n", " --output-dir /path/for/tf-image-classification-dataset \\\n", " --output-type fiftyone.types.TFImageClassificationDataset\n", "\n", " # Convert a COCO detection dataset to CVAT image format\n", " fiftyone convert \\\n", " --input-dir /path/to/coco-detection-dataset \\\n", " --input-type fiftyone.types.COCODetectionDataset \\\n", " --output-dir /path/for/cvat-image-dataset \\\n", " --output-type fiftyone.types.CVATImageDataset\n", "\n", " # Perform a customized conversion via optional kwargs\n", " fiftyone convert \\\n", " --input-dir /path/to/coco-detection-dataset \\\n", " --input-type fiftyone.types.COCODetectionDataset \\\n", " --input-kwargs max_samples=100 shuffle=True \\\n", " --output-dir /path/for/cvat-image-dataset \\\n", " --output-type fiftyone.types.TFObjectDetectionDataset \\\n", " --output-kwargs force_rgb=True \\\n", " --overwrite\n", "\n", "optional arguments:\n", " -h, --help show this help message and exit\n", " --input-dir INPUT_DIR\n", " the directory containing the dataset\n", " --input-kwargs KEY=VAL [KEY=VAL ...]\n", " additional keyword arguments for `fiftyone.utils.data.convert_dataset(..., input_kwargs=)`\n", " --output-dir OUTPUT_DIR\n", " the directory to which to write the output dataset\n", " --output-kwargs KEY=VAL [KEY=VAL ...]\n", " additional keyword arguments for `fiftyone.utils.data.convert_dataset(..., output_kwargs=)`\n", " -o, --overwrite whether to overwrite an existing output directory\n", "\n", "required arguments:\n", " --input-type INPUT_TYPE\n", " the fiftyone.types.Dataset type of the input dataset\n", " --output-type OUTPUT_TYPE\n", " the fiftyone.types.Dataset type to output\n" ] } ], "source": [ "fiftyone convert -h" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Convert CIFAR-10 dataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When you downloaded the test split of the CIFAR-10 dataset above, it was written to disk as a dataset in [fiftyone.types.FiftyOneImageClassificationDataset](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#FiftyOneImageClassificationDataset-import) format.\n", "\n", "You can verify this by printing information about the downloaded dataset:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "***** Dataset description *****\n", "The CIFAR-10 dataset consists of 60000 32 x 32 color images in 10\n", " classes, with 6000 images per class. There are 50000 training images and\n", " 10000 test images.\n", "\n", " Dataset size:\n", " 132.40 MiB\n", "\n", " Source:\n", " https://www.cs.toronto.edu/~kriz/cifar.html\n", " \n", "***** Supported splits *****\n", "test, train\n", "\n", "***** Dataset location *****\n", "~/fiftyone/cifar10\n", "\n", "***** Dataset info *****\n", "{\n", " \"name\": \"cifar10\",\n", " \"zoo_dataset\": \"fiftyone.zoo.datasets.torch.CIFAR10Dataset\",\n", " \"dataset_type\": \"fiftyone.types.dataset_types.FiftyOneImageClassificationDataset\",\n", " \"num_samples\": 10000,\n", " \"downloaded_splits\": {\n", " \"test\": {\n", " \"split\": \"test\",\n", " \"num_samples\": 10000\n", " }\n", " },\n", " \"classes\": [\n", " \"airplane\",\n", " \"automobile\",\n", " \"bird\",\n", " \"cat\",\n", " \"deer\",\n", " \"dog\",\n", " \"frog\",\n", " \"horse\",\n", " \"ship\",\n", " \"truck\"\n", " ]\n", "}\n" ] } ], "source": [ "fiftyone zoo datasets info cifar10" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The snippet below uses `fiftyone convert` to convert the test split of the CIFAR-10 dataset to [fiftyone.types.ImageClassificationDirectoryTree](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#imageclassificationdirectorytree) format, which stores classification datasets on disk in a directory tree structure with images organized per-class:\n", "\n", "```\n", "\n", "├── /\n", "│ ├── .\n", "│ ├── .\n", "│ └── ...\n", "├── /\n", "│ ├── .\n", "│ ├── .\n", "│ └── ...\n", "└── ...\n", "```" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loading dataset from '~/fiftyone/cifar10/test'\n", "Input format 'fiftyone.types.dataset_types.FiftyOneImageClassificationDataset'\n", " 100% |███| 10000/10000 [4.2s elapsed, 0s remaining, 2.4K samples/s] \n", "Import complete\n", "Exporting dataset to '/tmp/fiftyone/cifar10-dir-tree'\n", "Export format 'fiftyone.types.dataset_types.ImageClassificationDirectoryTree'\n", " 100% |███| 10000/10000 [6.2s elapsed, 0s remaining, 1.7K samples/s] \n", "Export complete\n" ] } ], "source": [ "INPUT_DIR=$(fiftyone zoo datasets find cifar10 --split test)\n", "OUTPUT_DIR=/tmp/fiftyone/cifar10-dir-tree\n", "\n", "fiftyone convert \\\n", " --input-dir ${INPUT_DIR} --input-type fiftyone.types.FiftyOneImageClassificationDataset \\\n", " --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.ImageClassificationDirectoryTree" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's verify that the conversion happened as expected:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "total 0\n", "drwxr-xr-x 12 voxel51 wheel 384B Jul 14 11:08 .\n", "drwxr-xr-x 3 voxel51 wheel 96B Jul 14 11:08 ..\n", "drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 airplane\n", "drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 automobile\n", "drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 bird\n", "drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 cat\n", "drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 deer\n", "drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 dog\n", "drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 frog\n", "drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 horse\n", "drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 ship\n", "drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 truck\n" ] } ], "source": [ "ls -lah /tmp/fiftyone/cifar10-dir-tree/" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "total 8000\n", "drwxr-xr-x 1002 voxel51 wheel 31K Jul 14 11:08 .\n", "drwxr-xr-x 12 voxel51 wheel 384B Jul 14 11:08 ..\n", "-rw-r--r-- 1 voxel51 wheel 1.2K Jul 14 11:23 000004.jpg\n", "-rw-r--r-- 1 voxel51 wheel 1.1K Jul 14 11:23 000011.jpg\n", "-rw-r--r-- 1 voxel51 wheel 1.1K Jul 14 11:23 000022.jpg\n", "-rw-r--r-- 1 voxel51 wheel 1.3K Jul 14 11:23 000028.jpg\n", "-rw-r--r-- 1 voxel51 wheel 1.2K Jul 14 11:23 000045.jpg\n", "-rw-r--r-- 1 voxel51 wheel 1.2K Jul 14 11:23 000053.jpg\n", "-rw-r--r-- 1 voxel51 wheel 1.3K Jul 14 11:23 000075.jpg\n" ] } ], "source": [ "ls -lah /tmp/fiftyone/cifar10-dir-tree/airplane/ | head" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's convert the classification directory tree to [TFRecords](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#tfimageclassificationdataset) format!" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loading dataset from '/tmp/fiftyone/cifar10-dir-tree'\n", "Input format 'fiftyone.types.dataset_types.ImageClassificationDirectoryTree'\n", " 100% |███| 10000/10000 [4.0s elapsed, 0s remaining, 2.5K samples/s] \n", "Import complete\n", "Exporting dataset to '/tmp/fiftyone/cifar10-tfrecords'\n", "Export format 'fiftyone.types.dataset_types.TFImageClassificationDataset'\n", " 0% ||--| 1/10000 [23.2ms elapsed, 3.9m remaining, 43.2 samples/s] 2020-07-14 11:24:15.187387: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n", "2020-07-14 11:24:15.201384: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f83df428f60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n", "2020-07-14 11:24:15.201405: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version\n", " 100% |███| 10000/10000 [8.2s elapsed, 0s remaining, 1.3K samples/s] \n", "Export complete\n" ] } ], "source": [ "INPUT_DIR=/tmp/fiftyone/cifar10-dir-tree\n", "OUTPUT_DIR=/tmp/fiftyone/cifar10-tfrecords\n", "\n", "fiftyone convert \\\n", " --input-dir ${INPUT_DIR} --input-type fiftyone.types.ImageClassificationDirectoryTree \\\n", " --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.TFImageClassificationDataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's verify that the conversion happened as expected:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "total 29696\n", "drwxr-xr-x 3 voxel51 wheel 96B Jul 14 11:24 .\n", "drwxr-xr-x 4 voxel51 wheel 128B Jul 14 11:24 ..\n", "-rw-r--r-- 1 voxel51 wheel 14M Jul 14 11:24 tf.records\n" ] } ], "source": [ "ls -lah /tmp/fiftyone/cifar10-tfrecords" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Convert KITTI dataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When you downloaded the validation split of the KITTI dataset above, it was written to disk as a dataset in [fiftyone.types.FiftyOneImageDetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#FiftyOneImageDetectionDataset-import) format.\n", "\n", "You can verify this by printing information about the downloaded dataset:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "***** Dataset description *****\n", "KITTI contains a suite of vision tasks built using an autonomous\n", " driving platform.\n", "\n", " The full benchmark contains many tasks such as stereo, optical flow, visual\n", " odometry, etc. This dataset contains the object detection dataset,\n", " including the monocular images and bounding boxes. The dataset contains\n", " 7481 training images annotated with 3D bounding boxes. A full description\n", " of the annotations can be found in the README of the object development kit\n", " on the KITTI homepage.\n", "\n", " Dataset size:\n", " 5.27 GiB\n", "\n", " Source:\n", " http://www.cvlibs.net/datasets/kitti\n", " \n", "***** Supported splits *****\n", "test, train, validation\n", "\n", "***** Dataset location *****\n", "~/fiftyone/kitti\n", "\n", "***** Dataset info *****\n", "{\n", " \"name\": \"kitti\",\n", " \"zoo_dataset\": \"fiftyone.zoo.datasets.tf.KITTIDataset\",\n", " \"dataset_type\": \"fiftyone.types.dataset_types.FiftyOneImageDetectionDataset\",\n", " \"num_samples\": 423,\n", " \"downloaded_splits\": {\n", " \"validation\": {\n", " \"split\": \"validation\",\n", " \"num_samples\": 423\n", " }\n", " },\n", " \"classes\": [\n", " \"Car\",\n", " \"Van\",\n", " \"Truck\",\n", " \"Pedestrian\",\n", " \"Person_sitting\",\n", " \"Cyclist\",\n", " \"Tram\",\n", " \"Misc\"\n", " ]\n", "}\n" ] } ], "source": [ "fiftyone zoo datasets info kitti" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The snippet below uses `fiftyone convert` to convert the test split of the CIFAR-10 dataset to [fiftyone.types.COCODetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#cocodetectiondataset) format, which writes the dataset to disk with annotations in [COCO format](https://cocodataset.org/#format-data)." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loading dataset from '~/fiftyone/kitti/validation'\n", "Input format 'fiftyone.types.dataset_types.FiftyOneImageDetectionDataset'\n", " 100% |███████| 423/423 [1.2s elapsed, 0s remaining, 351.0 samples/s] \n", "Import complete\n", "Exporting dataset to '/tmp/fiftyone/kitti-coco'\n", "Export format 'fiftyone.types.dataset_types.COCODetectionDataset'\n", " 100% |███████| 423/423 [4.4s elapsed, 0s remaining, 96.1 samples/s] \n", "Export complete\n" ] } ], "source": [ "INPUT_DIR=$(fiftyone zoo datasets find kitti --split validation)\n", "OUTPUT_DIR=/tmp/fiftyone/kitti-coco\n", "\n", "fiftyone convert \\\n", " --input-dir ${INPUT_DIR} --input-type fiftyone.types.FiftyOneImageDetectionDataset \\\n", " --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.COCODetectionDataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's verify that the conversion happened as expected:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "total 880\n", "drwxr-xr-x 4 voxel51 wheel 128B Jul 14 11:24 .\n", "drwxr-xr-x 5 voxel51 wheel 160B Jul 14 11:24 ..\n", "drwxr-xr-x 425 voxel51 wheel 13K Jul 14 11:24 data\n", "-rw-r--r-- 1 voxel51 wheel 437K Jul 14 11:24 labels.json\n" ] } ], "source": [ "ls -lah /tmp/fiftyone/kitti-coco/" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "total 171008\n", "drwxr-xr-x 425 voxel51 wheel 13K Jul 14 11:24 .\n", "drwxr-xr-x 4 voxel51 wheel 128B Jul 14 11:24 ..\n", "-rw-r--r-- 1 voxel51 wheel 195K Jul 14 11:24 000001.jpg\n", "-rw-r--r-- 1 voxel51 wheel 191K Jul 14 11:24 000002.jpg\n", "-rw-r--r-- 1 voxel51 wheel 167K Jul 14 11:24 000003.jpg\n", "-rw-r--r-- 1 voxel51 wheel 196K Jul 14 11:24 000004.jpg\n", "-rw-r--r-- 1 voxel51 wheel 224K Jul 14 11:24 000005.jpg\n", "-rw-r--r-- 1 voxel51 wheel 195K Jul 14 11:24 000006.jpg\n", "-rw-r--r-- 1 voxel51 wheel 177K Jul 14 11:24 000007.jpg\n" ] } ], "source": [ "ls -lah /tmp/fiftyone/kitti-coco/data | head" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"info\": {\n", " \"year\": \"\",\n", " \"version\": \"\",\n", " \"description\": \"Exported from FiftyOne\",\n", " \"contributor\": \"\",\n", " \"url\": \"https://voxel51.com/fiftyone\",\n", " \"date_created\": \"2020-07-14T11:24:40\"\n", " },\n", " \"licenses\": [],\n", " \"categories\": [\n", " {\n", " \"id\": 0,\n", " \"name\": \"Car\",\n", " \"supercategory\": \"none\"\n", " },\n", " {\n", " \"id\": 1,\n", " \"name\": \"Cyclist\",\n", " \"supercategory\": \"none\"\n", "...\n", " \"area\": 4545.8,\n", " \"segmentation\": null,\n", " \"iscrowd\": 0\n", " },\n", " {\n", " \"id\": 3196,\n", " \"image_id\": 422,\n", " \"category_id\": 3,\n", " \"bbox\": [\n", " 367.2,\n", " 107.3,\n", " 36.2,\n", " 105.2\n", " ],\n", " \"area\": 3808.2,\n", " \"segmentation\": null,\n", " \"iscrowd\": 0\n", " }\n", " ]\n", "}\n" ] } ], "source": [ "cat /tmp/fiftyone/kitti-coco/labels.json | python -m json.tool 2> /dev/null | head -20\n", "echo \"...\"\n", "cat /tmp/fiftyone/kitti-coco/labels.json | python -m json.tool 2> /dev/null | tail -20" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's convert from COCO format to [CVAT Image format](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#cvatimageformat) format!" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loading dataset from '/tmp/fiftyone/kitti-coco'\n", "Input format 'fiftyone.types.dataset_types.COCODetectionDataset'\n", " 100% |███████| 423/423 [2.0s elapsed, 0s remaining, 206.4 samples/s] \n", "Import complete\n", "Exporting dataset to '/tmp/fiftyone/kitti-cvat'\n", "Export format 'fiftyone.types.dataset_types.CVATImageDataset'\n", " 100% |███████| 423/423 [1.3s elapsed, 0s remaining, 323.7 samples/s] \n", "Export complete\n" ] } ], "source": [ "INPUT_DIR=/tmp/fiftyone/kitti-coco\n", "OUTPUT_DIR=/tmp/fiftyone/kitti-cvat\n", "\n", "fiftyone convert \\\n", " --input-dir ${INPUT_DIR} --input-type fiftyone.types.COCODetectionDataset \\\n", " --output-dir ${OUTPUT_DIR} --output-type fiftyone.types.CVATImageDataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's verify that the conversion happened as expected:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "total 584\n", "drwxr-xr-x 4 voxel51 wheel 128B Jul 14 11:25 .\n", "drwxr-xr-x 6 voxel51 wheel 192B Jul 14 11:25 ..\n", "drwxr-xr-x 425 voxel51 wheel 13K Jul 14 11:25 data\n", "-rw-r--r-- 1 voxel51 wheel 289K Jul 14 11:25 labels.xml\n" ] } ], "source": [ "ls -lah /tmp/fiftyone/kitti-cvat" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", " 1.1\n", " \n", " \n", " 423\n", " annotation\n", " \n", " \n", " \n", " " ] } ], "source": [ "cat /tmp/fiftyone/kitti-cvat/labels.xml | head -20\n", "echo \"...\"\n", "cat /tmp/fiftyone/kitti-cvat/labels.xml | tail -20" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Cleanup\n", "\n", "You can cleanup the files generated by this recipe by running the command below:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "rm -rf /tmp/fiftyone" ] } ], "metadata": { "kernelspec": { "display_name": "Bash", "language": "bash", "name": "bash" }, "language_info": { "codemirror_mode": "shell", "file_extension": ".sh", "mimetype": "text/x-sh", "name": "bash" } }, "nbformat": 4, "nbformat_minor": 4 }