{ "cells": [ { "cell_type": "markdown", "id": "07f4c93e-e2c4-4e0e-a4c6-afc5391dad40", "metadata": {}, "source": [ "\n", "\n", "# Using Label Studio for Annotations with Pixeltable\n", "\n", "This tutorial demonstrates how to integrate Pixeltable with Label Studio, in order to provide seamless management of annotations data across the annotation workflow. We'll assume that you're at least somewhat familiar with Pixeltable and have read the [Pixeltable Basics](https://pixeltable.readme.io/docs/pixeltable-basics) tutorial.\n", "\n", "__This tutorial can only be run in a local Pixeltable installation, not in Colab or Kaggle__, since it relies on spinning up a locally running Label Studio instance. See the [Installation Guide](https://pixeltable.readme.io/docs/installation) for instructions on how to set up a local Pixeltable instance.\n", "\n", "To begin, let's ensure the requisite dependencies are installed." ] }, { "cell_type": "code", "execution_count": null, "id": "ba177f97-6bbc-4c2a-8c81-a6eecb8a8e43", "metadata": {}, "outputs": [], "source": [ "%pip install -qU pixeltable label-studio label-studio-sdk torch transformers" ] }, { "cell_type": "markdown", "id": "80ba2580-8f6b-4292-8280-acbdf9b6e4f2", "metadata": {}, "source": [ "## Set up Label Studio\n", "\n", "Now let's spin up a Label Studio server process. (If you're already running Label Studio, you can choose to skip this step, and instead enter your existing Label Studio URL and access token in the subsequent step.) Be patient, as it may take a minute or two to start.\n", "\n", "This will open a new browser window containing the Label Studio interface. If you've never run Label Studio before, you'll need to create an account; a link to create one will appear in the Label Studio browser window. __Everything is running locally in this tutorial, so the account will exist only on your local system.__" ] }, { "cell_type": "code", "execution_count": 1, "id": "91c032cb-472d-4e66-9594-8edebf527b66", "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Performing system checks...\n", "\n", "System check identified no issues (1 silenced).\n", "August 14, 2024 - 04:24:46\n", "Django version 3.2.25, using settings 'label_studio.core.settings.label_studio'\n", "Starting development server at http://0.0.0.0:8080/\n", "Quit the server with CONTROL-C.\n" ] } ], "source": [ "import subprocess\n", "ls_process = subprocess.Popen(['label-studio'], stderr=subprocess.PIPE)" ] }, { "cell_type": "markdown", "id": "dcfa1986-c742-4097-b200-6c195f22cdd2", "metadata": {}, "source": [ "If for some reason the Label Studio browser window failed to open, you can always access it at: http://localhost:8080/\n", "\n", "Once you've created an account in Label Studio, you'll need to locate your API key. In the Label Studio browser window, log in, and click on \"Account & Settings\" in the top right. Copy the Access Token from the interface." ] }, { "cell_type": "markdown", "id": "0f780b92-b698-4c5a-b5a1-3ff4b4d6a45b", "metadata": {}, "source": [ "## Configure Pixeltable\n", "\n", "Next, we configure Pixeltable to communicate with Label Studio. Run the following command, pasting in the API key that you copied from the Label Studio interface." ] }, { "cell_type": "code", "execution_count": 2, "id": "d8d5660c-7986-4558-bbb7-8eca1ae7590f", "metadata": {}, "outputs": [ { "name": "stdin", "output_type": "stream", "text": [ "Label Studio API key: ········\n" ] } ], "source": [ "import getpass\n", "import os\n", "\n", "if 'LABEL_STUDIO_URL' not in os.environ:\n", " os.environ['LABEL_STUDIO_URL'] = 'http://localhost:8080/'\n", "\n", "if 'LABEL_STUDIO_API_KEY' not in os.environ:\n", " os.environ['LABEL_STUDIO_API_KEY'] = getpass.getpass('Label Studio API key: ')" ] }, { "cell_type": "markdown", "id": "5694554a-8a81-4dd8-b351-a89260ef3bf9", "metadata": {}, "source": [ "## Create a Table to Store Videos\n", "\n", "Now we create the master table that will hold our videos to be annotated. This only needs to be done once, when we initially set up the workflow." ] }, { "cell_type": "code", "execution_count": 3, "id": "876c5342-d166-4cf1-92a8-8130308697e3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Connected to Pixeltable database at: postgresql://postgres:@/pixeltable?host=/Users/asiegel/.pixeltable/pgdata\n", "Created directory `ls_demo`.\n", "Created table `videos`.\n" ] } ], "source": [ "import pixeltable as pxt\n", "\n", "schema = {\n", " 'video': pxt.Video,\n", " 'date': pxt.Timestamp\n", "}\n", "\n", "# Before creating the table, we drop the `ls_demo` dir and all its contents,\n", "# in order to ensure a clean environment for the demo.\n", "pxt.drop_dir('ls_demo', force=True)\n", "pxt.create_dir('ls_demo')\n", "videos_table = pxt.create_table('ls_demo.videos', schema)" ] }, { "cell_type": "markdown", "id": "4d25fe62-b59c-4602-a0ce-46170f378150", "metadata": {}, "source": [ "## Populate It with Data" ] }, { "cell_type": "markdown", "id": "4e8357ad-345d-4273-a3ec-8711f0b48245", "metadata": {}, "source": [ "Now let's add some videos to the table to populate it. For this tutorial, we'll use some randomly selected videos from the Multimedia Commons archive. The table also contains a `date` field, for which we'll use a fixed date (but in a production setting, it would typically be the date on which the video was imported)." ] }, { "cell_type": "code", "execution_count": 4, "id": "29bea73f-c455-4e8f-aed7-ee34faa421c3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Inserting rows into `videos`: 3 rows [00:00, 993.05 rows/s]\n", "Inserted 3 rows with 0 errors.\n" ] }, { "data": { "text/plain": [ "UpdateStatus(num_rows=3, num_computed_values=0, num_excs=0, updated_cols=[], cols_with_excs=[])" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from datetime import datetime\n", "\n", "url_prefix = 'http://multimedia-commons.s3-website-us-west-2.amazonaws.com/data/videos/mp4/'\n", "files = [\n", " '122/8ff/1228ff94bf742242ee7c88e4769ad5d5.mp4',\n", " '2cf/a20/2cfa205eae979b31b1144abd9fa4e521.mp4',\n", " 'ffe/ff3/ffeff3c6bf57504e7a6cecaff6aefbc9.mp4',\n", "]\n", "today = datetime(2024, 4, 22)\n", "videos_table.insert({'video': url_prefix + file, 'date': today} for file in files)" ] }, { "cell_type": "markdown", "id": "4727dd4b-5b21-4ff9-a6a8-4c0e53641841", "metadata": {}, "source": [ "Let's have a look at the table now." ] }, { "cell_type": "code", "execution_count": 5, "id": "c984cf21-0e24-4844-bafd-9772ceaeffc7", "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
video | \n", "date | \n", "
---|---|
\n",
" \n",
" | \n",
" 2024-04-22 | \n", "
\n",
" \n",
" | \n",
" 2024-04-22 | \n", "
\n",
" \n",
" | \n",
" 2024-04-22 | \n", "
Column Name | \n", "Type | \n", "Computed With | \n", "
---|---|---|
annotations | \n", "json | \n", "\n", " |
video | \n", "video | \n", "\n", " |
date | \n", "timestamp | \n", "\n", " |
video | \n", "annotations | \n", "
---|---|
\n",
" \n",
" | \n",
" [{"id": 35, "task": 141, "result": [{"id": "E_L95THtiU", "type": "choices", "value": {"choices": ["sports"]}, "origin": "manual", "to_name": "video", "from_name": "video-category"}], "project": 106, "import_id": null, "lead_time": 2.586, "created_at": "2024-08-14T04:25:17.201212Z", "updated_at": "2024-08-14T04:25:17.201239Z", "updated_by": 2, "created_ago": "0\\u00a0minutes", "last_action": null, "completed_by": 2, "ground_truth": false, "was_cancelled": false, "last_created_by": null, "created_username": " asiegel@pixeltable.com, 2", "draft_created_at": null, "parent_annotation": null, "parent_prediction": null}] | \n", "
\n",
" \n",
" | \n",
" None | \n", "
\n",
" \n",
" | \n",
" None | \n", "
video | \n", "annotations | \n", "video_category | \n", "
---|---|---|
\n",
" \n",
" | \n",
" [{"id": 35, "task": 141, "result": [{"id": "E_L95THtiU", "type": "choices", "value": {"choices": ["sports"]}, "origin": "manual", "to_name": "video", "from_name": "video-category"}], "project": 106, "import_id": null, "lead_time": 2.586, "created_at": "2024-08-14T04:25:17.201212Z", "updated_at": "2024-08-14T04:25:17.201239Z", "updated_by": 2, "created_ago": "0\\u00a0minutes", "last_action": null, "completed_by": 2, "ground_truth": false, "was_cancelled": false, "last_created_by": null, "created_username": " asiegel@pixeltable.com, 2", "draft_created_at": null, "parent_annotation": null, "parent_prediction": null}] | \n", "sports | \n", "
\n",
" \n",
" | \n",
" None | \n", "None | \n", "
\n",
" \n",
" | \n",
" None | \n", "None | \n", "
video | \n", "annotations | \n", "video_category | \n", "video_metadata | \n", "
---|---|---|---|
\n",
" \n",
" | \n",
" [{"id": 35, "task": 141, "result": [{"id": "E_L95THtiU", "type": "choices", "value": {"choices": ["sports"]}, "origin": "manual", "to_name": "video", "from_name": "video-category"}], "project": 106, "import_id": null, "lead_time": 2.586, "created_at": "2024-08-14T04:25:17.201212Z", "updated_at": "2024-08-14T04:25:17.201239Z", "updated_by": 2, "created_ago": "0\\u00a0minutes", "last_action": null, "completed_by": 2, "ground_truth": false, "was_cancelled": false, "last_created_by": null, "created_username": " asiegel@pixeltable.com, 2", "draft_created_at": null, "parent_annotation": null, "parent_prediction": null}] | \n", "sports | \n", "{"size": 815026, "streams": [{"type": "video", "width": 640, "frames": 235, "height": 480, "duration": 235235, "metadata": {"encoder": "AVC Coding", "language": "eng", "vendor_id": "[0][0][0][0]", "handler_name": "MP4 Video Media Handler", "creation_time": "2010-04-27T16:40:32.000000Z"}, "base_rate": 29.97, "time_base": 3.333e-05, "average_rate": 29.97, "guessed_rate": 29.97, "codec_context": {"name": "h264", "pix_fmt": "yuv420p", "profile": "High", "codec_tag": "avc1"}, "duration_seconds": 7.841}, {"type": "audio", "frames": 339, "duration": 347135, "metadata": {"language": "eng", "vendor_id": "[0][0][0][0]", "handler_name": "MP4 Sound Media Handler", "creation_time": "2010-04-27T16:40:32.000000Z"}, "time_base": 2.268e-05, "codec_context": {"name": "aac", "profile": "LC", "channels": 2, "codec_tag": "mp4a"}, "duration_seconds": 7.872}], "bit_rate": 828326, "metadata": {"major_brand": "mp42", "creation_time": "2010-04-27T16:40:32.000000Z", "minor_version": "0", "compatible_brands": "isom"}, "bit_exact": false} | \n", "
\n",
" \n",
" | \n",
" None | \n", "None | \n", "{"size": 1558736, "streams": [{"type": "video", "width": 640, "frames": 450, "height": 480, "duration": 450450, "metadata": {"encoder": "AVC Coding", "language": "eng", "vendor_id": "[0][0][0][0]", "handler_name": "MP4 Video Media Handler", "creation_time": "2009-05-20T00:53:00.000000Z"}, "base_rate": 29.97, "time_base": 3.333e-05, "average_rate": 29.97, "guessed_rate": 29.97, "codec_context": {"name": "h264", "pix_fmt": "yuv420p", "profile": "High", "codec_tag": "avc1"}, "duration_seconds": 15.015}, {"type": "audio", "frames": 648, "duration": 663551, "metadata": {"language": "eng", "vendor_id": "[0][0][0][0]", "handler_name": "MP4 Sound Media Handler", "creation_time": "2009-05-20T00:53:00.000000Z"}, "time_base": 2.268e-05, "codec_context": {"name": "aac", "profile": "LC", "channels": 2, "codec_tag": "mp4a"}, "duration_seconds": 15.047}], "bit_rate": 828756, "metadata": {"major_brand": "mp42", "creation_time": "2009-05-20T00:53:00.000000Z", "minor_version": "0", "compatible_brands": "isom"}, "bit_exact": false} | \n", "
\n",
" \n",
" | \n",
" None | \n", "None | \n", "{"size": 2099014, "streams": [{"type": "video", "width": 640, "frames": 600, "height": 360, "duration": 600600, "metadata": {"language": "eng", "vendor_id": "[0][0][0][0]", "handler_name": "VideoHandler"}, "base_rate": 29.97, "time_base": 3.333e-05, "average_rate": 29.97, "guessed_rate": 29.97, "codec_context": {"name": "h264", "pix_fmt": "yuv420p", "profile": "High", "codec_tag": "avc1"}, "duration_seconds": 20.02}, {"type": "audio", "frames": 863, "duration": 883712, "metadata": {"language": "eng", "vendor_id": "[0][0][0][0]", "handler_name": "SoundHandler"}, "time_base": 2.268e-05, "codec_context": {"name": "aac", "profile": "LC", "channels": 2, "codec_tag": "mp4a"}, "duration_seconds": 20.039}], "bit_rate": 836844, "metadata": {"encoder": "Lavf54.63.104", "major_brand": "isom", "minor_version": "512", "compatible_brands": "isomiso2avc1mp41"}, "bit_exact": false} | \n", "
frame | \n", "
---|
\n",
" \n",
" | \n",
"
\n",
" \n",
" | \n",
"
\n",
" \n",
" | \n",
"