{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Working with Twelve Labs in Pixeltable\n", "\n", "Twelve Labs provides multimodal embeddings that project text, images, audio, and video into the **same semantic space**. This enables true **cross-modal search** - the most powerful feature of this integration.\n", "\n", "**What makes this special?** You can search a video index using *any* modality:\n", "\n", "| Query Type | Use Case |\n", "|------------|----------|\n", "| **Text to Video** | \"Find clips of a man giving a speech\" |\n", "| **Image to Video** | Find videos visually similar to a photo |\n", "| **Audio to Video** | Find videos with similar speech/sounds |\n", "| **Video to Video** | Find videos similar to a clip |\n", "\n", "This notebook demonstrates this cross-modal capability with video, then shows how to apply the same embeddings to other modalities.\n", "\n", "### Prerequisites\n", "\n", "- A Twelve Labs account with an API key ([playground.twelvelabs.io](https://playground.twelvelabs.io/))\n", "- Audio and video must be at least 4 seconds long" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install -qU pixeltable twelvelabs" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import getpass\n", "import os\n", "\n", "if 'TWELVELABS_API_KEY' not in os.environ:\n", " os.environ['TWELVELABS_API_KEY'] = getpass.getpass(\n", " 'Enter your Twelve Labs API key: '\n", " )" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/pjlb/.pixeltable/pgdata\n", "Created directory 'twelvelabs_demo'.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/opt/miniconda3/envs/pixeltable/lib/python3.11/site-packages/pixeltable/env.py:494: UserWarning: Progress reporting is disabled because ipywidgets is not installed. To fix this, run: `pip install ipywidgets`\n", " warnings.warn(\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pixeltable as pxt\n", "import pixeltable.functions as pxtf\n", "\n", "# Create a fresh directory for our demo\n", "pxt.drop_dir('twelvelabs_demo', force=True)\n", "pxt.create_dir('twelvelabs_demo')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Cross-Modal Video Search\n", "\n", "Let's index a video and search it using text, images, audio, and other videos - all against the same index." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Create Video Table and Index" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created table 'videos'.\n", "Inserted 1 row with 0 errors in 1.60 s (0.63 rows/s)\n" ] }, { "data": { "text/plain": [ "1 row inserted." ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create a table for videos\n", "video_t = pxt.create_table('twelvelabs_demo/videos', {'video': pxt.Video})\n", "\n", "# Insert a sample video\n", "video_url = 'https://github.com/pixeltable/pixeltable/raw/main/docs/resources/The-Pursuit-of-Happiness.mp4'\n", "video_t.insert([{'video': video_url}])" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "# Create a view that segments the video into searchable chunks\n", "# Twelve Labs requires minimum 4 second segments\n", "video_chunks = pxt.create_view(\n", " 'twelvelabs_demo/video_chunks',\n", " video_t,\n", " iterator=pxtf.video.video_splitter(\n", " video=video_t.video, duration=5.0, min_segment_duration=4.0\n", " ),\n", ")\n", "\n", "# Add embedding index for cross-modal search\n", "video_chunks.add_embedding_index(\n", " 'video_segment',\n", " embedding=pxtf.twelvelabs.embed.using(model_name='marengo3.0'),\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's look at the index we just added in the table metadata:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
view 'twelvelabs_demo/video_chunks' (of 'twelvelabs_demo/videos')
\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Column NameTypeComputed With
posRequired[Int]
segment_startFloat
segment_start_ptsInt
segment_endFloat
segment_end_ptsInt
video_segmentRequired[Video]
videoVideo
\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Index NameColumnMetricEmbedding
idx0video_segmentcosineembed(video_segment, model_name='marengo3.0', start_sec=None, end_sec=None, embedding_option=None)
\n" ], "text/plain": [ "view 'twelvelabs_demo/video_chunks' (of 'twelvelabs_demo/videos')\n", "\n", " Column Name Type Computed With\n", " pos Required[Int] \n", " segment_start Float \n", " segment_start_pts Int \n", " segment_end Float \n", " segment_end_pts Int \n", " video_segment Required[Video] \n", " video Video \n", "\n", " Index Name Column Metric Embedding\n", " idx0 video_segment cosine embed(video_segment, model_name='marengo3.0', ..." ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "video_chunks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The iterator created a larger table from our single video:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "51" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "video_chunks.count()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Text to Video Search\n", "\n", "Find video segments matching a text description." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
video_segmentscore
\n", "            \n", "        
0.424
\n", "            \n", "        
0.408
\n", "            \n", "        
0.223
" ], "text/plain": [ " video_segment score\n", "0 /Users/pjlb/.pixeltable/media/e4230d37e1ce484b... 0.424177\n", "1 /Users/pjlb/.pixeltable/media/e4230d37e1ce484b... 0.407893\n", "2 /Users/pjlb/.pixeltable/media/e4230d37e1ce484b... 0.223031" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sim = video_chunks.video_segment.similarity(string='pink')\n", "\n", "video_chunks.order_by(sim, asc=False).limit(3).select(\n", " video_chunks.video_segment, score=sim\n", ").collect()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Image to Video Search\n", "\n", "Find video segments similar to an image." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
video_segmentscore
\n", "            \n", "        
0.747
\n", "            \n", "        
0.702
" ], "text/plain": [ " video_segment score\n", "0 /Users/pjlb/.pixeltable/media/e4230d37e1ce484b... 0.747186\n", "1 /Users/pjlb/.pixeltable/media/e4230d37e1ce484b... 0.701641" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "image_query = 'https://github.com/pixeltable/pixeltable/raw/main/docs/resources/The-Pursuit-of-Happiness-Screenshot.png'\n", "\n", "sim = video_chunks.video_segment.similarity(image=image_query)\n", "\n", "video_chunks.order_by(sim, asc=False).limit(2).select(\n", " video_chunks.video_segment, score=sim\n", ").collect()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Video to Video Search\n", "\n", "Find video segments similar to another video clip." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
video_segmentscore
\n", "            \n", "        
0.875
\n", "            \n", "        
0.836
" ], "text/plain": [ " video_segment score\n", "0 /Users/pjlb/.pixeltable/media/e4230d37e1ce484b... 0.875341\n", "1 /Users/pjlb/.pixeltable/media/e4230d37e1ce484b... 0.836088" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "video_query = 'https://github.com/pixeltable/pixeltable/raw/main/docs/resources/The-Pursuit-of-Happiness-Video-Extract.mp4'\n", "\n", "sim = video_chunks.video_segment.similarity(video=video_query)\n", "\n", "video_chunks.order_by(sim, asc=False).limit(2).select(\n", " video_chunks.video_segment, score=sim\n", ").collect()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Audio to Video Search\n", "\n", "Find video segments with similar audio/speech content." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
video_segmentscore
\n", "            \n", "        
0.607
\n", "            \n", "        
0.587
" ], "text/plain": [ " video_segment score\n", "0 /Users/pjlb/.pixeltable/media/e4230d37e1ce484b... 0.606734\n", "1 /Users/pjlb/.pixeltable/media/e4230d37e1ce484b... 0.586691" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "audio_query = 'https://github.com/pixeltable/pixeltable/raw/main/docs/resources/The-Pursuit-of-Happiness-Audio-Extract.m4a'\n", "\n", "sim = video_chunks.video_segment.similarity(audio=audio_query)\n", "\n", "video_chunks.order_by(sim, asc=False).limit(2).select(\n", " video_chunks.video_segment, score=sim\n", ").collect()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Embedding Options\n", "\n", "For video embeddings, you can focus on specific aspects:\n", "\n", "- `'visual'` - Focus on what you see\n", "- `'audio'` - Focus on what you hear\n", "- `'transcription'` - Focus on what is said" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Added 51 column values with 0 errors in 19.81 s (2.57 rows/s)\n" ] }, { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
video_segmentvisual_embedding
\n", "            \n", "        
[ 0.034  0.071 -0.038  0.062 -0.01   0.061 ...  0.047 -0.069 -0.009 -0.021  0.036  0.002]
\n", "            \n", "        
[ 0.034  0.024 -0.005  0.068  0.016  0.008 ...  0.066 -0.005  0.011  0.023  0.051  0.007]
" ], "text/plain": [ " video_segment \\\n", "0 /Users/pjlb/.pixeltable/media/e4230d37e1ce484b... \n", "1 /Users/pjlb/.pixeltable/media/e4230d37e1ce484b... \n", "\n", " visual_embedding \n", "0 [0.034423828, 0.07080078, -0.037841797, 0.0620... \n", "1 [0.034179688, 0.024291992, -0.005279541, 0.067... " ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Add a visual-only embedding column\n", "video_chunks.add_computed_column(\n", " visual_embedding=pxtf.twelvelabs.embed(\n", " video_chunks.video_segment,\n", " model_name='marengo3.0',\n", " embedding_option=['visual'],\n", " )\n", ")\n", "\n", "video_chunks.select(\n", " video_chunks.video_segment, video_chunks.visual_embedding\n", ").limit(2).collect()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Other Modalities: Text, Images, and Documents\n", "\n", "Twelve Labs embeddings also work for text, images, and documents. Here's a compact example showing **multiple embedding indexes on a single table**." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created table 'content'.\n", "Added 0 column values with 0 errors in 0.01 s\n" ] } ], "source": [ "# Create a multimodal content table\n", "content_t = pxt.create_table(\n", " 'twelvelabs_demo/content',\n", " {\n", " 'title': pxt.String,\n", " 'description': pxt.String,\n", " 'thumbnail': pxt.Image,\n", " },\n", ")\n", "\n", "# Add computed column combining title and description\n", "content_t.add_computed_column(\n", " text_content=content_t.title + '. ' + content_t.description\n", ")\n", "\n", "# Add embedding index on combined text column\n", "content_t.add_embedding_index(\n", " 'text_content',\n", " embedding=pxtf.twelvelabs.embed.using(model_name='marengo3.0'),\n", ")\n", "\n", "# Add embedding index on image column\n", "content_t.add_embedding_index(\n", " 'thumbnail',\n", " embedding=pxtf.twelvelabs.embed.using(model_name='marengo3.0'),\n", ")" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Inserted 4 rows with 0 errors in 1.97 s (2.03 rows/s)\n" ] }, { "data": { "text/plain": [ "4 rows inserted." ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Insert sample content\n", "content_t.insert(\n", " [\n", " {\n", " 'title': 'Beach Sunset',\n", " 'description': 'A beautiful sunset over the ocean with palm trees.',\n", " 'thumbnail': 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000025.jpg',\n", " },\n", " {\n", " 'title': 'Mountain Hiking',\n", " 'description': 'Hikers climbing a steep mountain trail with scenic views.',\n", " 'thumbnail': 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000139.jpg',\n", " },\n", " {\n", " 'title': 'City Street',\n", " 'description': 'Busy urban street with cars and pedestrians.',\n", " 'thumbnail': 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000042.jpg',\n", " },\n", " {\n", " 'title': 'Wildlife Safari',\n", " 'description': 'Elephants and zebras on the African savanna.',\n", " 'thumbnail': 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000061.jpg',\n", " },\n", " ]\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see the two indexes we added in the schema:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
table 'twelvelabs_demo/content'
\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Column NameTypeComputed With
titleString
descriptionString
thumbnailImage
text_contentString(title + '. ') + description
\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Index NameColumnMetricEmbedding
idx4text_contentcosineembed(text_content, None, model_name='marengo3.0')
idx5thumbnailcosineembed(thumbnail, model_name='marengo3.0')
\n" ], "text/plain": [ "table 'twelvelabs_demo/content'\n", "\n", " Column Name Type Computed With\n", " title String \n", " description String \n", " thumbnail Image \n", " text_content String (title + '. ') + description\n", "\n", " Index Name Column Metric Embedding\n", " idx4 text_content cosine embed(text_content, None, model_name='marengo3...\n", " idx5 thumbnail cosine embed(thumbnail, model_name='marengo3.0')" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "content_t" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titletext_contentscore
Mountain HikingMountain Hiking. Hikers climbing a steep mountain trail with scenic views.0.539
Beach SunsetBeach Sunset. A beautiful sunset over the ocean with palm trees.0.435
" ], "text/plain": [ " title text_content \\\n", "0 Mountain Hiking Mountain Hiking. Hikers climbing a steep mount... \n", "1 Beach Sunset Beach Sunset. A beautiful sunset over the ocea... \n", "\n", " score \n", "0 0.538902 \n", "1 0.435235 " ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Search by text description\n", "sim = content_t.text_content.similarity(string='outdoor nature adventure')\n", "\n", "content_t.order_by(sim, asc=False).limit(2).select(\n", " content_t.title, content_t.text_content, score=sim\n", ").collect()" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titlethumbnailscore
Beach Sunset
\n", "                \n", "            
0.759
Mountain Hiking
\n", "                \n", "            
0.742
" ], "text/plain": [ " title thumbnail \\\n", "0 Beach Sunset \n", " \n", " \n", " title\n", " thumbnail\n", " score\n", " \n", " \n", " \n", " \n", " City Street\n", "
\n", "                \n", "            
\n", " 0.135\n", " \n", " \n", " Mountain Hiking\n", "
\n", "                \n", "            
\n", " -0.003\n", " \n", " \n", "" ], "text/plain": [ " title thumbnail \\\n", "0 City Street