{ "cells": [ { "cell_type": "markdown", "id": "07f4c93e-e2c4-4e0e-a4c6-afc5391dad40", "metadata": {}, "source": [ "\"Download\n", "\n", "# Using Label Studio for Annotations with Pixeltable\n", "\n", "This tutorial demonstrates how to integrate Pixeltable with Label Studio, in order to provide seamless management of annotations data across the annotation workflow. We'll assume that you're at least somewhat familiar with Pixeltable and have read the [Pixeltable Basics](https://pixeltable.readme.io/docs/pixeltable-basics) tutorial.\n", "\n", "__This tutorial can only be run in a local Pixeltable installation, not in Colab or Kaggle__, since it relies on spinning up a locally running Label Studio instance. See the [Installation Guide](https://pixeltable.readme.io/docs/installation) for instructions on how to set up a local Pixeltable instance.\n", "\n", "To begin, let's ensure the requisite dependencies are installed." ] }, { "cell_type": "code", "execution_count": null, "id": "ba177f97-6bbc-4c2a-8c81-a6eecb8a8e43", "metadata": {}, "outputs": [], "source": [ "%pip install -qU pixeltable label-studio label-studio-sdk torch transformers" ] }, { "cell_type": "markdown", "id": "80ba2580-8f6b-4292-8280-acbdf9b6e4f2", "metadata": {}, "source": [ "## Set up Label Studio\n", "\n", "Now let's spin up a Label Studio server process. (If you're already running Label Studio, you can choose to skip this step, and instead enter your existing Label Studio URL and access token in the subsequent step.) Be patient, as it may take a minute or two to start.\n", "\n", "This will open a new browser window containing the Label Studio interface. If you've never run Label Studio before, you'll need to create an account; a link to create one will appear in the Label Studio browser window. __Everything is running locally in this tutorial, so the account will exist only on your local system.__" ] }, { "cell_type": "code", "execution_count": 1, "id": "91c032cb-472d-4e66-9594-8edebf527b66", "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Performing system checks...\n", "\n", "System check identified no issues (1 silenced).\n", "August 14, 2024 - 04:24:46\n", "Django version 3.2.25, using settings 'label_studio.core.settings.label_studio'\n", "Starting development server at http://0.0.0.0:8080/\n", "Quit the server with CONTROL-C.\n" ] } ], "source": [ "import subprocess\n", "ls_process = subprocess.Popen(['label-studio'], stderr=subprocess.PIPE)" ] }, { "cell_type": "markdown", "id": "dcfa1986-c742-4097-b200-6c195f22cdd2", "metadata": {}, "source": [ "If for some reason the Label Studio browser window failed to open, you can always access it at: http://localhost:8080/\n", "\n", "Once you've created an account in Label Studio, you'll need to locate your API key. In the Label Studio browser window, log in, and click on \"Account & Settings\" in the top right. Copy the Access Token from the interface." ] }, { "cell_type": "markdown", "id": "0f780b92-b698-4c5a-b5a1-3ff4b4d6a45b", "metadata": {}, "source": [ "## Configure Pixeltable\n", "\n", "Next, we configure Pixeltable to communicate with Label Studio. Run the following command, pasting in the API key that you copied from the Label Studio interface." ] }, { "cell_type": "code", "execution_count": 2, "id": "d8d5660c-7986-4558-bbb7-8eca1ae7590f", "metadata": {}, "outputs": [ { "name": "stdin", "output_type": "stream", "text": [ "Label Studio API key: ········\n" ] } ], "source": [ "import getpass\n", "import os\n", "\n", "if 'LABEL_STUDIO_URL' not in os.environ:\n", " os.environ['LABEL_STUDIO_URL'] = 'http://localhost:8080/'\n", "\n", "if 'LABEL_STUDIO_API_KEY' not in os.environ:\n", " os.environ['LABEL_STUDIO_API_KEY'] = getpass.getpass('Label Studio API key: ')" ] }, { "cell_type": "markdown", "id": "5694554a-8a81-4dd8-b351-a89260ef3bf9", "metadata": {}, "source": [ "## Create a Table to Store Videos\n", "\n", "Now we create the master table that will hold our videos to be annotated. This only needs to be done once, when we initially set up the workflow." ] }, { "cell_type": "code", "execution_count": 3, "id": "876c5342-d166-4cf1-92a8-8130308697e3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Connected to Pixeltable database at: postgresql://postgres:@/pixeltable?host=/Users/asiegel/.pixeltable/pgdata\n", "Created directory `ls_demo`.\n", "Created table `videos`.\n" ] } ], "source": [ "import pixeltable as pxt\n", "\n", "schema = {\n", " 'video': pxt.Video,\n", " 'date': pxt.Timestamp\n", "}\n", "\n", "# Before creating the table, we drop the `ls_demo` dir and all its contents,\n", "# in order to ensure a clean environment for the demo.\n", "pxt.drop_dir('ls_demo', force=True)\n", "pxt.create_dir('ls_demo')\n", "videos_table = pxt.create_table('ls_demo.videos', schema)" ] }, { "cell_type": "markdown", "id": "4d25fe62-b59c-4602-a0ce-46170f378150", "metadata": {}, "source": [ "## Populate It with Data" ] }, { "cell_type": "markdown", "id": "4e8357ad-345d-4273-a3ec-8711f0b48245", "metadata": {}, "source": [ "Now let's add some videos to the table to populate it. For this tutorial, we'll use some randomly selected videos from the Multimedia Commons archive. The table also contains a `date` field, for which we'll use a fixed date (but in a production setting, it would typically be the date on which the video was imported)." ] }, { "cell_type": "code", "execution_count": 4, "id": "29bea73f-c455-4e8f-aed7-ee34faa421c3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Inserting rows into `videos`: 3 rows [00:00, 993.05 rows/s]\n", "Inserted 3 rows with 0 errors.\n" ] }, { "data": { "text/plain": [ "UpdateStatus(num_rows=3, num_computed_values=0, num_excs=0, updated_cols=[], cols_with_excs=[])" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from datetime import datetime\n", "\n", "url_prefix = 'http://multimedia-commons.s3-website-us-west-2.amazonaws.com/data/videos/mp4/'\n", "files = [\n", " '122/8ff/1228ff94bf742242ee7c88e4769ad5d5.mp4',\n", " '2cf/a20/2cfa205eae979b31b1144abd9fa4e521.mp4',\n", " 'ffe/ff3/ffeff3c6bf57504e7a6cecaff6aefbc9.mp4',\n", "]\n", "today = datetime(2024, 4, 22)\n", "videos_table.insert({'video': url_prefix + file, 'date': today} for file in files)" ] }, { "cell_type": "markdown", "id": "4727dd4b-5b21-4ff9-a6a8-4c0e53641841", "metadata": {}, "source": [ "Let's have a look at the table now." ] }, { "cell_type": "code", "execution_count": 5, "id": "c984cf21-0e24-4844-bafd-9772ceaeffc7", "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
videodate
\n", " \n", "
2024-04-22
\n", " \n", "
2024-04-22
\n", " \n", "
2024-04-22
" ], "text/plain": [ " video date\n", "0 /Users/asiegel/.pixeltable/file_cache/faebc341... 2024-04-22\n", "1 /Users/asiegel/.pixeltable/file_cache/faebc341... 2024-04-22\n", "2 /Users/asiegel/.pixeltable/file_cache/faebc341... 2024-04-22" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "videos_table.head()" ] }, { "cell_type": "markdown", "id": "f114e857-e749-4dc9-977d-8bb676b3c660", "metadata": {}, "source": [ "## Create a Label Studio project\n", "\n", "Next we'll create a new Label Studio project and link it to a new view on the Pixeltable table. You can link a Label Studio project to either a table or a view. For tables that are expecting a lot of input data, it's often easier to link to views. In this example, we'll create a view that filters the table down by date." ] }, { "cell_type": "code", "execution_count": 6, "id": "65857a69-582a-43ca-b370-caa2799f3df1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Inserting rows into `videos_2024_04_22`: 3 rows [00:00, 1864.69 rows/s]\n", "Created view `videos_2024_04_22` with 3 rows, 0 exceptions.\n", "Added 3 column values with 0 errors.\n", "Computing cells: 100%|███████████████████████████████████████████| 3/3 [00:00<00:00, 857.44 cells/s]\n", "Linked external store `ls_project_0` to table `videos_2024_04_22`.\n", "Created 3 new task(s) in LabelStudioProject `videos_2024_04_22`.\n" ] }, { "data": { "text/plain": [ "SyncStatus(external_rows_created=3, external_rows_deleted=0, external_rows_updated=0, pxt_rows_updated=0, num_excs=0)" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create a view to filter on the specified date\n", "\n", "v = pxt.create_view(\n", " 'ls_demo.videos_2024_04_22',\n", " videos_table.where(videos_table.date == today)\n", ")\n", "\n", "# Create a new Label Studio project and link it to the view. The\n", "# configuration uses Label Studio's standard XML format. This only\n", "# needs to be done once: after the view and project are linked,\n", "# the relationship is stored indefinitely in Pixeltable's metadata.\n", "\n", "label_config = '''\n", " \n", " \n", " '''\n", "\n", "pxt.io.create_label_studio_project(v, label_config)" ] }, { "cell_type": "markdown", "id": "a64d0394-ab86-427c-8b7e-5a71f9ece6bc", "metadata": {}, "source": [ "If you look in the Label Studio UI now, you'll see that there's a new project with the name `videos_2022_04_22`, with three tasks, one for each of the videos in the view. If you want to create the project without populating it with tasks (yet), you can set `sync_immediately=False` in the call to `create_label_studio_project()`. You can always sync the table and project by calling `v.sync()`.\n", "\n", "Note also that we didn't have to specify an explicit mapping between Pixeltable columns and Label Studio data fields. This is because, by default, Pixeltable assumes the Pixeltable and Label Studio field names coincide. The data field in the Label Studio project has the name `$video`, which Pixeltable maps, by default, to the column in `ls_demo.videos_2024_02_22` that is also called `video`. If you want to override this behavior to specify an explicit mapping of columns to fields, you can do that with the `col_mapping` parameter of `create_label_studio_project()`.\n", "\n", "Inspecting the view, we also see that Pixeltable created an additional column on the view, `annotations`, which will hold the output of our annotations workflow. The name of the output column can also be overridden by specifying a dict entry in `col_mapping` of the form `{'my_col_name': 'annotations'}`." ] }, { "cell_type": "code", "execution_count": 7, "id": "ccd06368-f770-4203-903d-1ac6dd4c61de", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Column NameTypeComputed With
annotationsjson
videovideo
datetimestamp
\n" ], "text/plain": [ "view 'videos_2024_04_22'\n", "\n", "Column Name Type Computed With\n", "annotations json \n", " video video \n", " date timestamp " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "v" ] }, { "cell_type": "markdown", "id": "ff94777f-dd99-42ee-afe2-1f5684d99929", "metadata": {}, "source": [ "## Add Some Annotations" ] }, { "cell_type": "markdown", "id": "c68ab5c2-eb10-4eec-abb6-72fc9b8371d2", "metadata": {}, "source": [ "Now, let's add some annotations to our Label Studio project to simulate a human-in-the-loop workflow. In the Label Studio UI, click on the new `videos_2024_02_22` project, and click on any of the three tasks. Select the appropriate category (\"city\", \"food\", or \"sports\"), and click \"Submit\"." ] }, { "cell_type": "markdown", "id": "0956c5e8-d48a-4e4a-b594-6c07e4664083", "metadata": {}, "source": [ "## Import the Annotations Back To Pixeltable" ] }, { "cell_type": "markdown", "id": "a0f2a373-f1b2-4bdc-941a-41ca85dec038", "metadata": {}, "source": [ "Now let's try importing annotations from Label Studio back to our view." ] }, { "cell_type": "code", "execution_count": 8, "id": "8aa21d6a-c3f8-4a80-921d-0e7df22e3a40", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created 0 new task(s) in LabelStudioProject `videos_2024_04_22`.\n", "Updated annotation(s) from 1 task(s) in LabelStudioProject `videos_2024_04_22`.\n" ] }, { "data": { "text/plain": [ "SyncStatus(external_rows_created=0, external_rows_deleted=0, external_rows_updated=0, pxt_rows_updated=1, num_excs=0)" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "v = pxt.get_table('ls_demo.videos_2024_04_22')\n", "v.sync()" ] }, { "cell_type": "markdown", "id": "013633e5-1802-47b9-b84f-81c3b3e4ac81", "metadata": {}, "source": [ "Let's see what effect that had. You'll see that any videos that you annotated now have their `annotations` field populated in the view." ] }, { "cell_type": "code", "execution_count": 9, "id": "006268eb-6f96-40f8-b1e2-f96eb7aeee9c", "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
videoannotations
\n", " \n", "
[{"id": 35, "task": 141, "result": [{"id": "E_L95THtiU", "type": "choices", "value": {"choices": ["sports"]}, "origin": "manual", "to_name": "video", "from_name": "video-category"}], "project": 106, "import_id": null, "lead_time": 2.586, "created_at": "2024-08-14T04:25:17.201212Z", "updated_at": "2024-08-14T04:25:17.201239Z", "updated_by": 2, "created_ago": "0\\u00a0minutes", "last_action": null, "completed_by": 2, "ground_truth": false, "was_cancelled": false, "last_created_by": null, "created_username": " asiegel@pixeltable.com, 2", "draft_created_at": null, "parent_annotation": null, "parent_prediction": null}]
\n", " \n", "
None
\n", " \n", "
None
" ], "text/plain": [ " video \\\n", "0 /Users/asiegel/.pixeltable/file_cache/faebc341... \n", "1 /Users/asiegel/.pixeltable/file_cache/faebc341... \n", "2 /Users/asiegel/.pixeltable/file_cache/faebc341... \n", "\n", " annotations \n", "0 [{'id': 35, 'task': 141, 'result': [{'id': 'E_... \n", "1 None \n", "2 None " ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "v.select(v.video, v.annotations).head()" ] }, { "cell_type": "markdown", "id": "955e0485-f026-4a2c-97fc-07f61d979da7", "metadata": {}, "source": [ "## Parse Annotations with a Computed Column\n", "\n", "Pixeltable pulls in all sorts of metadata from Label Studio during a sync: everything that Label Studio reports back about the annotations, including things like the user account that created the annotations. Let's say that all we care about is the annotation value. We can add a computed column to our table to pull it out." ] }, { "cell_type": "code", "execution_count": 10, "id": "432a983e-f3eb-44bd-a4c9-fab556c3a754", "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Computing cells: 100%|███████████████████████████████████████████| 3/3 [00:00<00:00, 394.63 cells/s]\n", "Added 3 column values with 0 errors.\n" ] }, { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
videoannotationsvideo_category
\n", " \n", "
[{"id": 35, "task": 141, "result": [{"id": "E_L95THtiU", "type": "choices", "value": {"choices": ["sports"]}, "origin": "manual", "to_name": "video", "from_name": "video-category"}], "project": 106, "import_id": null, "lead_time": 2.586, "created_at": "2024-08-14T04:25:17.201212Z", "updated_at": "2024-08-14T04:25:17.201239Z", "updated_by": 2, "created_ago": "0\\u00a0minutes", "last_action": null, "completed_by": 2, "ground_truth": false, "was_cancelled": false, "last_created_by": null, "created_username": " asiegel@pixeltable.com, 2", "draft_created_at": null, "parent_annotation": null, "parent_prediction": null}]sports
\n", " \n", "
NoneNone
\n", " \n", "
NoneNone
" ], "text/plain": [ " video \\\n", "0 /Users/asiegel/.pixeltable/file_cache/faebc341... \n", "1 /Users/asiegel/.pixeltable/file_cache/faebc341... \n", "2 /Users/asiegel/.pixeltable/file_cache/faebc341... \n", "\n", " annotations video_category \n", "0 [{'id': 35, 'task': 141, 'result': [{'id': 'E_... sports \n", "1 None None \n", "2 None None " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "v['video_category'] = v.annotations[0].result[0].value.choices[0]\n", "v.select(v.video, v.annotations, v.video_category).head()" ] }, { "cell_type": "markdown", "id": "97e3a868-55d0-4ee3-b480-6f4d0094add6", "metadata": {}, "source": [ "Another useful operation is the `get_metadata` function, which returns information about the video itself, such as the resolution and codec (independent of Label Studio). Let's add another computed column to hold such metadata." ] }, { "cell_type": "code", "execution_count": 11, "id": "9ae666d2-6822-421c-a703-f6280b135d4d", "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Computing cells: 100%|███████████████████████████████████████████| 3/3 [00:00<00:00, 138.97 cells/s]\n", "Added 3 column values with 0 errors.\n" ] }, { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
videoannotationsvideo_categoryvideo_metadata
\n", " \n", "
[{"id": 35, "task": 141, "result": [{"id": "E_L95THtiU", "type": "choices", "value": {"choices": ["sports"]}, "origin": "manual", "to_name": "video", "from_name": "video-category"}], "project": 106, "import_id": null, "lead_time": 2.586, "created_at": "2024-08-14T04:25:17.201212Z", "updated_at": "2024-08-14T04:25:17.201239Z", "updated_by": 2, "created_ago": "0\\u00a0minutes", "last_action": null, "completed_by": 2, "ground_truth": false, "was_cancelled": false, "last_created_by": null, "created_username": " asiegel@pixeltable.com, 2", "draft_created_at": null, "parent_annotation": null, "parent_prediction": null}]sports{"size": 815026, "streams": [{"type": "video", "width": 640, "frames": 235, "height": 480, "duration": 235235, "metadata": {"encoder": "AVC Coding", "language": "eng", "vendor_id": "[0][0][0][0]", "handler_name": "MP4 Video Media Handler", "creation_time": "2010-04-27T16:40:32.000000Z"}, "base_rate": 29.97, "time_base": 3.333e-05, "average_rate": 29.97, "guessed_rate": 29.97, "codec_context": {"name": "h264", "pix_fmt": "yuv420p", "profile": "High", "codec_tag": "avc1"}, "duration_seconds": 7.841}, {"type": "audio", "frames": 339, "duration": 347135, "metadata": {"language": "eng", "vendor_id": "[0][0][0][0]", "handler_name": "MP4 Sound Media Handler", "creation_time": "2010-04-27T16:40:32.000000Z"}, "time_base": 2.268e-05, "codec_context": {"name": "aac", "profile": "LC", "channels": 2, "codec_tag": "mp4a"}, "duration_seconds": 7.872}], "bit_rate": 828326, "metadata": {"major_brand": "mp42", "creation_time": "2010-04-27T16:40:32.000000Z", "minor_version": "0", "compatible_brands": "isom"}, "bit_exact": false}
\n", " \n", "
NoneNone{"size": 1558736, "streams": [{"type": "video", "width": 640, "frames": 450, "height": 480, "duration": 450450, "metadata": {"encoder": "AVC Coding", "language": "eng", "vendor_id": "[0][0][0][0]", "handler_name": "MP4 Video Media Handler", "creation_time": "2009-05-20T00:53:00.000000Z"}, "base_rate": 29.97, "time_base": 3.333e-05, "average_rate": 29.97, "guessed_rate": 29.97, "codec_context": {"name": "h264", "pix_fmt": "yuv420p", "profile": "High", "codec_tag": "avc1"}, "duration_seconds": 15.015}, {"type": "audio", "frames": 648, "duration": 663551, "metadata": {"language": "eng", "vendor_id": "[0][0][0][0]", "handler_name": "MP4 Sound Media Handler", "creation_time": "2009-05-20T00:53:00.000000Z"}, "time_base": 2.268e-05, "codec_context": {"name": "aac", "profile": "LC", "channels": 2, "codec_tag": "mp4a"}, "duration_seconds": 15.047}], "bit_rate": 828756, "metadata": {"major_brand": "mp42", "creation_time": "2009-05-20T00:53:00.000000Z", "minor_version": "0", "compatible_brands": "isom"}, "bit_exact": false}
\n", " \n", "
NoneNone{"size": 2099014, "streams": [{"type": "video", "width": 640, "frames": 600, "height": 360, "duration": 600600, "metadata": {"language": "eng", "vendor_id": "[0][0][0][0]", "handler_name": "VideoHandler"}, "base_rate": 29.97, "time_base": 3.333e-05, "average_rate": 29.97, "guessed_rate": 29.97, "codec_context": {"name": "h264", "pix_fmt": "yuv420p", "profile": "High", "codec_tag": "avc1"}, "duration_seconds": 20.02}, {"type": "audio", "frames": 863, "duration": 883712, "metadata": {"language": "eng", "vendor_id": "[0][0][0][0]", "handler_name": "SoundHandler"}, "time_base": 2.268e-05, "codec_context": {"name": "aac", "profile": "LC", "channels": 2, "codec_tag": "mp4a"}, "duration_seconds": 20.039}], "bit_rate": 836844, "metadata": {"encoder": "Lavf54.63.104", "major_brand": "isom", "minor_version": "512", "compatible_brands": "isomiso2avc1mp41"}, "bit_exact": false}
" ], "text/plain": [ " video \\\n", "0 /Users/asiegel/.pixeltable/file_cache/faebc341... \n", "1 /Users/asiegel/.pixeltable/file_cache/faebc341... \n", "2 /Users/asiegel/.pixeltable/file_cache/faebc341... \n", "\n", " annotations video_category \\\n", "0 [{'id': 35, 'task': 141, 'result': [{'id': 'E_... sports \n", "1 None None \n", "2 None None \n", "\n", " video_metadata \n", "0 {'size': 815026, 'streams': [{'type': 'video',... \n", "1 {'size': 1558736, 'streams': [{'type': 'video'... \n", "2 {'size': 2099014, 'streams': [{'type': 'video'... " ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from pixeltable.functions.video import get_metadata\n", "\n", "v['video_metadata'] = get_metadata(v.video)\n", "v.select(v.video, v.annotations, v.video_category, v.video_metadata).head()" ] }, { "cell_type": "markdown", "id": "3c8c5394-2539-4160-84df-68cb8df72de9", "metadata": {}, "source": [ "## Preannotations with Pixeltable and Label Studio" ] }, { "cell_type": "markdown", "id": "ca674ef6-46f0-4d65-bcd8-58399bdacef2", "metadata": {}, "source": [ "Frame extraction is another common operation in labeling workflows. In this example, we'll extract frames from our videos into a view, then use an object detection model to generate preannotations for each frame. The following code uses a Pixeltable `FrameIterator` to automatically extract frames into a new view, which we'll call `frames_2024_04_22`." ] }, { "cell_type": "code", "execution_count": 12, "id": "64ad60b8-fca5-419d-81bb-8c9e640b0037", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Inserting rows into `frames_2024_04_22`: 13 rows [00:00, 5434.66 rows/s]\n", "Created view `frames_2024_04_22` with 13 rows, 0 exceptions.\n" ] } ], "source": [ "from datetime import datetime\n", "from pixeltable.iterators import FrameIterator\n", "\n", "today = datetime(2024, 4, 22)\n", "videos_table = pxt.get_table('ls_demo.videos')\n", "\n", "# Create the view, using a `FrameIterator` to extract frames with a sample rate\n", "# of `fps=0.25`, or 1 frame per 4 seconds of video. Setting `fps=0` would use the\n", "# native framerate of the video, extracting every frame.\n", "\n", "frames = pxt.create_view(\n", " 'ls_demo.frames_2024_04_22',\n", " videos_table.where(videos_table.date == today),\n", " iterator=FrameIterator.create(video=videos_table.video, fps=0.25)\n", ")" ] }, { "cell_type": "code", "execution_count": 13, "id": "f923f511-7624-4537-9eaf-21a8337696fc", "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
frame
\n", " \n", "
\n", " \n", "
\n", " \n", "
" ], "text/plain": [ " frame\n", "0 \n", " \n", " \n", " frame\n", " detections\n", " \n", " \n", " \n", " \n", "
\n", " \n", "
\n", " {"boxes": [[584.916, 0.75, 639.989, 321.319], [46.766, 93.549, 294.693, 465.584]], "labels": [1, 1], "scores": [0.995, 0.999], "label_text": ["person", "person"]}\n", " \n", " \n", "
\n", " \n", "
\n", " {"boxes": [[562.879, 197., 640.206, 229.308], [415.185, 149.31, 427.469, 168.121], [182.444, 160.578, 219.743, 248.855], [413.599, 139.345, 426.33, 160.366]], "labels": [15, 4, 1, 1], "scores": [0.981, 0.995, 1., 0.988], "label_text": ["bench", "motorcycle", "person", "person"]}\n", " \n", " \n", "
\n", " \n", "
\n", " {"boxes": [[387.795, 196.154, 505.848, 372.086]], "labels": [1], "scores": [1.], "label_text": ["person"]}\n", " \n", " \n", "" ], "text/plain": [ " frame \\\n", "0 \n", " \n", " \n", " frame\n", " detections\n", " preannotations\n", " \n", " \n", " \n", " \n", "
\n", " \n", "
\n", " {"boxes": [[584.916, 0.75, 639.989, 321.319], [46.766, 93.549, 294.693, 465.584]], "labels": [1, 1], "scores": [0.995, 0.999], "label_text": ["person", "person"]}\n", " {"image": {"width": 640, "height": 480}, "annotations": [{"bbox": [584.916, 0.75, 55.073, 320.57], "category": 1}, {"bbox": [46.766, 93.549, 247.927, 372.035], "category": 1}]}\n", " \n", " \n", "
\n", " \n", "
\n", " {"boxes": [[562.879, 197., 640.206, 229.308], [415.185, 149.31, 427.469, 168.121], [182.444, 160.578, 219.743, 248.855], [413.599, 139.345, 426.33, 160.366]], "labels": [15, 4, 1, 1], "scores": [0.981, 0.995, 1., 0.988], "label_text": ["bench", "motorcycle", "person", "person"]}\n", " {"image": {"width": 640, "height": 480}, "annotations": [{"bbox": [562.879, 197., 77.326, 32.308], "category": 15}, {"bbox": [415.185, 149.31, 12.283, 18.811], "category": 4}, {"bbox": [182.444, 160.578, 37.299, 88.276], "category": 1}, {"bbox": [413.599, 139.345, 12.73, 21.021], "category": 1}]}\n", " \n", " \n", "
\n", " \n", "
\n", " {"boxes": [[387.795, 196.154, 505.848, 372.086]], "labels": [1], "scores": [1.], "label_text": ["person"]}\n", " {"image": {"width": 640, "height": 480}, "annotations": [{"bbox": [387.795, 196.154, 118.053, 175.932], "category": 1}]}\n", " \n", " \n", "" ], "text/plain": [ " frame \\\n", "0 \n", " \n", " \n", " \n", " \n", " '''\n", "\n", "pxt.io.create_label_studio_project(frames, frames_config)" ] }, { "cell_type": "markdown", "id": "e48446f9-39af-4017-9ab9-72a8951131b5", "metadata": {}, "source": [ "If you go into Label Studio and open up the new project, you can see the effect of adding the preannotations from Resnet-50 to our workflow." ] }, { "cell_type": "markdown", "id": "4f16ee59-4200-4a98-9965-eaa72727d103", "metadata": {}, "source": [ "## Incremental Updates\n", "\n", "As we saw in the [Pixeltable Basics](https://pixeltable.readme.io/docs/pixeltable-basics) tutorial, adding new data to Pixeltable results in incremental updates of everything downstream. We can see this by inserting a new video into our base videos table: all of the downstream views and computed columns are updated automatically, including the video metadata, frames, and preannotations.\n", "\n", "The update may take some time, so please be patient (it involves a sequence of operations, including frame extraction and object detection)." ] }, { "cell_type": "code", "execution_count": 17, "id": "dd371a24-795a-4c3e-9d92-e1310013d915", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Inserting rows into `videos`: 1 rows [00:00, 808.31 rows/s]\n", "Inserting rows into `videos_2024_04_22`: 1 rows [00:00, 849.57 rows/s]\n", "Inserting rows into `frames_2024_04_22`: 5 rows [00:00, 3225.89 rows/s]\n", "Inserted 7 rows with 0 errors.\n" ] }, { "data": { "text/plain": [ "UpdateStatus(num_rows=7, num_computed_values=0, num_excs=0, updated_cols=[], cols_with_excs=[])" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "videos_table.insert(\n", " video=url_prefix + '22a/948/22a9487a92956ac453a9c15e0fc4dd4.mp4',\n", " date=today\n", ")" ] }, { "cell_type": "markdown", "id": "1da98d43-fe42-4d91-ba3d-a5c4b32170cb", "metadata": {}, "source": [ "Note that the incremental updates do _not_ automatically sync the `Table` with the remote Label Studio projects. To issue a sync, we have to call the `sync()` methods separately. Note that tasks will be created only for the _newly added_ rows in the videos and frames views, not the existing ones." ] }, { "cell_type": "code", "execution_count": 18, "id": "9799e354-3fcf-4755-8192-135e94a6a8ef", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created 1 new task(s) in LabelStudioProject `videos_2024_04_22`.\n", "Created 5 new task(s) in LabelStudioProject `frames_2024_04_22`.\n" ] }, { "data": { "text/plain": [ "SyncStatus(external_rows_created=5, external_rows_deleted=0, external_rows_updated=0, pxt_rows_updated=0, num_excs=0)" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "v.sync()\n", "frames.sync()" ] }, { "cell_type": "markdown", "id": "eb7f04bb-d2ba-4cbc-86c0-a3995f2118a5", "metadata": {}, "source": [ "## Deleting a Project\n", "\n", "To remove a Label Studio project from a table or view, use `unlink_external_stores()`, as demonstrated by the following example. If you specify `delete_external_data=True`, then the Label Studio project will also be deleted, along with all existing data and annotations (be careful!) If `delete_external_data=False`, then the Label Studio project will be unlinked from Pixeltable, but the project and data will remain in Label Studio (so you'll need to delete the project manually if you later want to get rid of it)." ] }, { "cell_type": "code", "execution_count": 19, "id": "d5a28b50-e7ac-4302-b675-1394c57a9843", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['ls_project_0']" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "v.external_stores # Get a list of all external stores for `v`" ] }, { "cell_type": "code", "execution_count": 20, "id": "b0055b97-5912-4073-b6be-af9d894f5f1c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Deleted Label Studio project: videos_2024_04_22\n", "Unlinked external store from table `videos_2024_04_22`: ls_project_0\n" ] } ], "source": [ "v.unlink_external_stores('ls_project_0', delete_external_data=True)" ] }, { "cell_type": "markdown", "id": "20569f26-85ad-45e5-8746-ebf376d77519", "metadata": {}, "source": [ "## Configuring `media_import_method`\n", "\n", "All of the examples so far in this tutorial use HTTP file uploads to send media data to Label Studio. This is the simplest method and the easiest to configure, but it's undesirable for complex projects or projects with a lot of data. In fact, the Label Studio documentation includes this specific warning: \"Uploading data works fine for proof of concept projects, but it is not recommended for larger projects.\"\n", "\n", "In Pixeltable, you can configure linked Label Studio projects to use URLs for media data (instead of file uploads) by specifying the `media_import_method='url'` argument in `create_label_studio_project`. This is recommended for all production applications, and is mandatory for projects whose input configuration is more complex than a single media file (in the Label Studio parlance, projects with more than one \"data key\").\n", "\n", "If `media_import_method='url'`, then Pixeltable will simply pass the media data URLs directly to Label Studio. If the URLs are `http://` or `https://` URLs, then nothing more needs to be done.\n", "\n", "Label Studio also supports `s3://` URLs with credentialed access. To use them, you'll need to configure access to your bucket in the project configuration. The simplest way to do this is by specifying an `s3_configuration` in `create_label_studio_project`. Here's an example, though it won't work directly in this demo notebook, since it relies on having an access key. (If your AWS credentials are stored in `~/.aws/credentials`, then you can omit the access key and secret, and Pixeltable will fill them in automatically.)" ] }, { "cell_type": "code", "execution_count": null, "id": "d418d367-c188-443e-b911-9325fc37d841", "metadata": {}, "outputs": [], "source": [ "pxt.io.create_label_studio_project(\n", " v,\n", " label_config,\n", " media_import_method='url',\n", " s3_configuration={'bucket': 'pxt-test', 'aws_access_key_id': my_key, 'aws_secret_access_key': my_secret}\n", ")" ] }, { "cell_type": "markdown", "id": "f7df6cb2-d8a4-4141-91a7-ad97e75dabee", "metadata": {}, "source": [ "Before you can set up credentialed S3 access, you'll need to configure your S3 bucket to work with Label Studio; the details on how to do this are described here:\n", "- [Label Studio Docs: Amazon S3](https://labelstud.io/guide/storage.html#Amazon-S3)\n", "\n", "For the full documentation on `create_label_studio_project` usage, see:\n", "- [Pixeltable API Docs: create_label_studio_project](https://pixeltable.github.io/pixeltable/api/io/#pixeltable.io.create_label_studio_project)" ] }, { "cell_type": "markdown", "id": "de46c9d9-edb6-43eb-be6e-d4e857477622", "metadata": {}, "source": [ "## Notebook Cleanup\n", "\n", "That's the end of the tutorial! To conclude, let's terminate the running Label Studio process. (Of course, feel free to leave it running if you want to play around with it some more.)" ] }, { "cell_type": "code", "execution_count": 21, "id": "d44f708a-207b-41b6-bbdb-1026809e794d", "metadata": {}, "outputs": [], "source": [ "ls_process.kill()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.19" } }, "nbformat": 4, "nbformat_minor": 5 }