{
"cells": [
{
"cell_type": "markdown",
"id": "european-testimony",
"metadata": {},
"source": [
"# Annotating Datasets with Labelbox\n",
"\n",
"All successful computer vision projects start with the same thing: LOTS OF LABELED DATA!\n",
"\n",
"In this walkthrough, we'll cover how to use [the integration](https://voxel51.com/docs/fiftyone/integrations/labelbox.html) between [FiftyOne](https://fiftyone.ai) and the popular annotation tool [Labelbox](https://labelbox.com/) to build a high-quality labeled dataset.\n",
"\n",
"Specifically, this walkthrough covers:\n",
"\n",
"* Using the FiftyOne Brain to select unique samples for annotation\n",
"* Annotating unlabeled samples with Labelbox\n",
"* Improving existing annotations using FiftyOne tagging and Labelbox\n",
"* Additional utilities for managing Labelbox annotation projects\n",
"\n",
"**So, what's the takeaway?**\n",
"\n",
"FiftyOne Datasets are the best way to explore, understand, and improve your datasets and models. This walkthrough covers how to use our Labelbox integration to streamline the annotation creation and improvement process."
]
},
{
"cell_type": "markdown",
"id": "defined-spyware",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"To get started, you need to [install FiftyOne](https://voxel51.com/docs/fiftyone/getting_started/install.html) and the [Labelbox Python client](https://github.com/Labelbox/labelbox-python):"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "quantitative-weight",
"metadata": {},
"outputs": [],
"source": [
"!pip install fiftyone labelbox"
]
},
{
"cell_type": "markdown",
"id": "french-station",
"metadata": {},
"source": [
"You'll also need to set up a [Labelbox account](https://app.labelbox.com/). FiftyOne supports both standard Labelbox cloud accounts and Labelbox enterprise solutions.\n",
"\n",
"The easiest way to get started is to use the default Labelbox server, which simply requires creating an account and then providing your API key as shown below."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "authorized-delay",
"metadata": {},
"outputs": [],
"source": [
"!export FIFTYONE_LABELBOX_API_KEY=..."
]
},
{
"cell_type": "markdown",
"id": "national-footage",
"metadata": {},
"source": [
"Alternatively, for a more permanent solution, you can store your credentials in your FiftyOne annotation config located at `~/.fiftyone/annotation_config.json`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dutch-documentation",
"metadata": {},
"outputs": [],
"source": [
"{\n",
" \"backends\": {\n",
" \"labelbox\": {\n",
" \"api_key\": ...,\n",
" }\n",
" }\n",
"}"
]
},
{
"cell_type": "markdown",
"id": "german-checkout",
"metadata": {},
"source": [
"See [this page](https://voxel51.com/docs/fiftyone/integrations/labelbox.html#setup) for more advanced Labelbox setup options."
]
},
{
"cell_type": "markdown",
"id": "ranging-rating",
"metadata": {},
"source": [
"## Raw Data\n",
"\n",
"To start, you need to gather raw image or video data relevant to your task. The internet has a lot of places to look for free data. Assuming you have your raw data downloaded locally, you can [easily load it into FiftyOne](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html)."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "environmental-bridges",
"metadata": {},
"outputs": [],
"source": [
"import fiftyone as fo"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "other-framing",
"metadata": {},
"outputs": [],
"source": [
"dataset = fo.Dataset.from_dir(dataset_dir=\"/path/to/dir\", dataset_type=fo.types.ImageDirectory)"
]
},
{
"cell_type": "markdown",
"id": "processed-nature",
"metadata": {},
"source": [
"Another method is to use publically available datasets that may be relevant. For example, the [Open Images dataset](https://storage.googleapis.com/openimages/web/download.html) contains millions of images available for public use and can be accessed directly through the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_zoo/index.html)."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "mounted-memphis",
"metadata": {},
"outputs": [],
"source": [
"import fiftyone.zoo as foz"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "generous-sociology",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Downloading split 'validation' to '/home/eric/fiftyone/open-images-v6/validation' if necessary\n",
"Necessary images already downloaded\n",
"Existing download of split 'validation' is sufficient\n",
"Loading 'open-images-v6' split 'validation'\n",
" 100% |█████████████████| 100/100 [72.9ms elapsed, 0s remaining, 1.4K samples/s] \n",
"Dataset 'labelbox_dataset' created\n"
]
}
],
"source": [
"dataset = foz.load_zoo_dataset(\n",
" \"open-images-v6\",\n",
" split=\"validation\",\n",
" max_samples=100,\n",
" label_types=[],\n",
" dataset_name=\"labelbox_dataset\",\n",
")"
]
},
{
"cell_type": "markdown",
"id": "legitimate-conference",
"metadata": {},
"source": [
"Either way, once your data is in FiftyOne, we can [visualize it in the FiftyOne App](https://voxel51.com/docs/fiftyone/user_guide/app.html)."
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "ancient-partnership",
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"
\n",
"
\n",
" \n",
"
\n",
" \n",
"
\n",
"\n",
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"session = fo.launch_app(dataset)"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "brazilian-extension",
"metadata": {},
"outputs": [],
"source": [
"session.freeze() # Screenshot the App in this notebook"
]
},
{
"cell_type": "markdown",
"id": "certified-turkish",
"metadata": {},
"source": [
"FiftyOne provides a [variety of methods](https://voxel51.com/docs/fiftyone/user_guide/brain.html) that can help you understand the quality of the dataset and pick the best samples to annotate. For example, the [`compute_similarity()`](https://voxel51.com/docs/fiftyone/user_guide/brain.html#visual-similarity) the method can be used to find both the most similar, and the most unique samples, ensuring that your dataset will contain an even distribution of data."
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "rental-liberia",
"metadata": {},
"outputs": [],
"source": [
"import fiftyone.brain as fob"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "twelve-violin",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Computing embeddings...\n",
" 100% |█████████████████| 100/100 [1.0m elapsed, 0s remaining, 1.8 samples/s] \n"
]
}
],
"source": [
"results = fob.compute_similarity(dataset, brain_key=\"img_sim\")"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "sophisticated-coverage",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Generating index...\n",
"Index complete\n",
"Computing unique samples...\n",
"threshold: 1.000000, kept: 100, target: 10\n",
"threshold: 2.000000, kept: 100, target: 10\n",
"threshold: 4.000000, kept: 100, target: 10\n",
"threshold: 8.000000, kept: 70, target: 10\n",
"threshold: 16.000000, kept: 5, target: 10\n",
"threshold: 12.000000, kept: 14, target: 10\n",
"threshold: 14.000000, kept: 7, target: 10\n",
"threshold: 13.000000, kept: 10, target: 10\n",
"Uniqueness computation complete\n"
]
}
],
"source": [
"results.find_unique(10)"
]
},
{
"cell_type": "markdown",
"id": "linear-minneapolis",
"metadata": {},
"source": [
"Now to select only the slice of our dataset that contains the 10 most unique samples."
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "possible-motion",
"metadata": {},
"outputs": [],
"source": [
"unique_view = dataset.select(results.unique_ids)"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "forty-northwest",
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"
\n",
"
\n",
" \n",
"
\n",
" \n",
"
\n",
"\n",
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"session.view = unique_view"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "devoted-oracle",
"metadata": {},
"outputs": [],
"source": [
"session.freeze()"
]
},
{
"cell_type": "markdown",
"id": "recorded-wholesale",
"metadata": {},
"source": [
"## Annotation\n",
"\n",
"The [integration between FiftyOne and Labelbox](https://voxel51.com/docs/fiftyone/integrations/labelbox.html) allows you to begin annotating your image or video data by calling a single method!"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "palestinian-alaska",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Initializing Labelbox client at 'https://api.labelbox.com/graphql'\n",
"Uploading samples to Labelbox...\n",
"Upload complete\n",
"Initializing Labelbox client at 'https://api.labelbox.com/graphql'\n",
"Launching editor at 'https://editor.labelbox.com/?project=ckx83uywu09j610a9ertke8ny'...\n"
]
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"anno_key = \"annotation_run_1\"\n",
"\n",
"results = unique_view.annotate(\n",
" anno_key,\n",
" backend=\"labelbox\",\n",
" label_field=\"detections\",\n",
" classes=[\"vehicle\", \"animal\", \"plant\"],\n",
" label_type=\"detections\",\n",
" launch_editor=True,\n",
")"
]
},
{
"cell_type": "markdown",
"id": "direct-bailey",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"id": "european-basin",
"metadata": {},
"source": [
"The annotations can then be [loaded back into FiftyOne](https://voxel51.com/docs/fiftyone/integrations/labelbox.html#loading-annotations) in just one more line."
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "divided-apparatus",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Initializing Labelbox client at 'https://api.labelbox.com/graphql'\n",
"Downloading labels from Labelbox...\n",
" 133.4Kb [7.2ms elapsed, ? remaining, 91.0Mb/s] \n",
"Download complete\n",
"Loading labels for field 'detections'...\n",
" 100% |█████████████████████| 8/8 [26.2ms elapsed, 0s remaining, 305.1 samples/s] \n"
]
}
],
"source": [
"unique_view.load_annotations(anno_key)"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "taken-deputy",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"
\n",
"
\n",
" \n",
"
\n",
" \n",
"
\n",
"\n",
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"session.view = unique_view"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "universal-eleven",
"metadata": {},
"outputs": [],
"source": [
"session.freeze()"
]
},
{
"cell_type": "markdown",
"id": "outstanding-namibia",
"metadata": {},
"source": [
"This API provides [advanced customization options](https://voxel51.com/docs/fiftyone/integrations/labelbox.html#requesting-annotations) for your annotation tasks. For example, we can construct a sophisticated schema to define the annotations we want and even directly assign the annotators:"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "patient-reach",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Experimental features have been enabled\n",
"Initializing Labelbox client at 'https://api.labelbox.com/graphql'\n",
"Uploading samples to Labelbox...\n",
"Uploading existing labels in field 'detections' to Labelbox is not yet supported\n",
"Your organization has reached its limit of 5 members. Cannot invite new member fiftyone_labelbox_user3@gmail.com to project 'FiftyOne_labelbox_dataset'\n",
"Upload complete\n",
"Experimental features have been enabled\n",
"Initializing Labelbox client at 'https://api.labelbox.com/graphql'\n",
"Launching editor at 'https://editor.labelbox.com/?project=ckx845cbd0atq10a948ze988n'...\n"
]
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"anno_key = \"labelbox_assign_users\"\n",
"\n",
"members = [\n",
" (\"fiftyone_labelbox_user1@gmail.com\", \"LABELER\"),\n",
" (\"fiftyone_labelbox_user2@gmail.com\", \"REVIEWER\"),\n",
" (\"fiftyone_labelbox_user3@gmail.com\", \"TEAM_MANAGER\"),\n",
"]\n",
"\n",
"# Set up the Labelbox editor to reannotate existing \"detections\" labels and a new \"keypoints\" field\n",
"label_schema = {\n",
" \"detections_new\": {\n",
" \"type\": \"detections\",\n",
" \"classes\": dataset.distinct(\"detections.detections.label\"),\n",
" },\n",
" \"keypoints\": {\n",
" \"type\": \"keypoints\",\n",
" \"classes\": [\"Person\"],\n",
" }\n",
"}\n",
"\n",
"results = unique_view.annotate(\n",
" anno_key,\n",
" backend=\"labelbox\",\n",
" label_schema=label_schema,\n",
" members=members,\n",
" launch_editor=True,\n",
")"
]
},
{
"cell_type": "markdown",
"id": "substantial-window",
"metadata": {},
"source": [
"After you're finished annotating in Labelbox, you can easily download the results:"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "joined-parallel",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Experimental features have been enabled\n",
"Initializing Labelbox client at 'https://api.labelbox.com/graphql'\n",
"Downloading labels from Labelbox...\n",
" 24.0b [8.4ms elapsed, ? remaining, 58.0Kb/s] \n",
"Download complete\n",
"Loading labels for field 'detections'...\n",
" 100% |█████████████████████| 0/0 [12.1ms elapsed, ? remaining, ? samples/s] \n",
"Loading labels for field 'keypoints'...\n",
" 100% |█████████████████████| 0/0 [1.8ms elapsed, ? remaining, ? samples/s] \n",
"Experimental features have been enabled\n",
"Initializing Labelbox client at 'https://api.labelbox.com/graphql'\n",
"Deleting project 'ckx845cbd0atq10a948ze988n'...\n"
]
}
],
"source": [
"# Download results and clean the run from FiftyOne and Labelbox\n",
"unique_view.load_annotations(anno_key, cleanup=True)"
]
},
{
"cell_type": "markdown",
"id": "burning-learning",
"metadata": {},
"source": [
"## Next Steps\n",
"\n",
"Now that you have a labeled dataset, you can go ahead and start training a model. FiftyOne lets you [export your data to disk in a variety of formats](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html) (ex: COCO, YOLO, etc) expected by most training pipelines. It also provides workflows for using popular model training libraries like [PyTorch](https://towardsdatascience.com/stop-wasting-time-with-pytorch-datasets-17cac2c22fa8), [PyTorch Lightning Flash](https://voxel51.com/docs/fiftyone/integrations/lightning_flash.html), and [Tensorflow](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#tfobjectdetectiondataset).\n",
"\n",
"Once the model is trained, the [model predictions can be loaded](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#adding-model-predictions) back into FiftyOne. These [predictions can then be evaluated against the ground truth](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html) annotations to find where the model is performing well, and where it is performing poorly. This provides insight into the type of samples that need to be added to the training set, as well as any annotation errors that may exist."
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "gentle-president",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Dataset already downloaded\n",
"Loading existing dataset 'quickstart'. To reload from disk, either delete the existing dataset or provide a custom `dataset_name` to use\n"
]
}
],
"source": [
"# Load an existing dataset with predictions\n",
"dataset = foz.load_zoo_dataset(\"quickstart\")"
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "patent-rebecca",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Evaluating detections...\n",
" 100% |█████████████████| 200/200 [6.8s elapsed, 0s remaining, 29.6 samples/s] \n"
]
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Evaluate model predictions\n",
"dataset.evaluate_detections(\n",
" \"predictions\",\n",
" gt_field=\"ground_truth\",\n",
" eval_key=\"eval\",\n",
")"
]
},
{
"cell_type": "markdown",
"id": "revolutionary-albuquerque",
"metadata": {},
"source": [
"We can use the [powerful querying capabilities of the FiftyOne API](https://voxel51.com/docs/fiftyone/user_guide/using_views.html) to create a [view filtering](https://voxel51.com/docs/fiftyone/user_guide/using_views.html#filtering) these model results by false positives with high confidence which generally indicates an error in the ground truth annotation."
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "moral-theta",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"
\n",
"
\n",
" \n",
"
\n",
" \n",
"
\n",
"\n",
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from fiftyone import ViewField as F\n",
"\n",
"fp_view = dataset.filter_labels(\n",
" \"predictions\",\n",
" (F(\"confidence\") > 0.8) & (F(\"eval\") == \"fp\"),\n",
")\n",
"\n",
"session = fo.launch_app(view=fp_view)"
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "documentary-greece",
"metadata": {},
"outputs": [],
"source": [
"session.freeze()"
]
},
{
"cell_type": "markdown",
"id": "impossible-basis",
"metadata": {},
"source": [
"This sample appears to be missing a ground truth annotation of skis. Let's [tag it in FiftyOne](https://voxel51.com/docs/fiftyone/user_guide/app.html#tags-and-tagging), and send it to Labelbox for reannotation."
]
},
{
"cell_type": "code",
"execution_count": 41,
"id": "vertical-hawaiian",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"