{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Working with Voyage AI in Pixeltable\n", "\n", "Pixeltable's Voyage AI integration enables you to access state-of-the-art embedding and reranker models via the Voyage AI API.\n", "\n", "### Prerequisites\n", "\n", "- A Voyage AI account with an API key (https://www.voyageai.com/)\n", "\n", "### Important notes\n", "\n", "- Voyage AI usage may incur costs based on your Voyage AI plan.\n", "- Be mindful of sensitive data and consider security measures when integrating with external services." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First you'll need to install required libraries and enter your Voyage AI API key." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": { "iopub.execute_input": "2025-12-04T04:20:47.468703Z", "iopub.status.busy": "2025-12-04T04:20:47.468601Z", "iopub.status.idle": "2025-12-04T04:20:48.256737Z", "shell.execute_reply": "2025-12-04T04:20:48.256360Z" } }, "outputs": [], "source": [ "%pip install -qU voyageai" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "execution": { "iopub.execute_input": "2025-12-04T04:20:48.258691Z", "iopub.status.busy": "2025-12-04T04:20:48.258342Z", "iopub.status.idle": "2025-12-04T04:20:48.260718Z", "shell.execute_reply": "2025-12-04T04:20:48.260457Z" } }, "outputs": [], "source": [ "import getpass\n", "import os\n", "\n", "if 'VOYAGE_API_KEY' not in os.environ:\n", " os.environ['VOYAGE_API_KEY'] = getpass.getpass(\n", " 'Enter your Voyage AI API key:'\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's create a Pixeltable directory to hold the tables for our demo." ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "execution": { "iopub.execute_input": "2025-12-04T04:20:48.262319Z", "iopub.status.busy": "2025-12-04T04:20:48.262179Z", "iopub.status.idle": "2025-12-04T04:20:59.142625Z", "shell.execute_reply": "2025-12-04T04:20:59.142181Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created directory 'voyageai_demo'.\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pixeltable as pxt\n", "\n", "# Remove the 'voyageai_demo' directory and its contents, if it exists\n", "pxt.drop_dir('voyageai_demo', force=True)\n", "pxt.create_dir('voyageai_demo')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Text embeddings\n", "\n", "Voyage AI provides state-of-the-art embedding models for semantic search and RAG applications." ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "execution": { "iopub.execute_input": "2025-12-04T04:20:59.145492Z", "iopub.status.busy": "2025-12-04T04:20:59.144896Z", "iopub.status.idle": "2025-12-04T04:20:59.200290Z", "shell.execute_reply": "2025-12-04T04:20:59.199973Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created table 'documents'.\n", "Added 0 column values with 0 errors.\n" ] }, { "data": { "text/plain": [ "No rows affected." ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from pixeltable.functions import voyageai\n", "\n", "# Create a table for document embeddings\n", "docs_t = pxt.create_table('voyageai_demo/documents', {'text': pxt.String})\n", "\n", "# Add computed column with Voyage embeddings\n", "docs_t.add_computed_column(\n", " embedding=voyageai.embeddings(\n", " docs_t.text, model='voyage-3.5', input_type='document'\n", " )\n", ")" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "execution": { "iopub.execute_input": "2025-12-04T04:20:59.202386Z", "iopub.status.busy": "2025-12-04T04:20:59.202220Z", "iopub.status.idle": "2025-12-04T04:21:04.082526Z", "shell.execute_reply": "2025-12-04T04:21:04.082176Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Inserting rows into `documents`: 6 rows [00:00, 2561.67 rows/s]\n", "Inserted 6 rows with 0 errors.\n" ] }, { "data": { "text/plain": [ "6 rows inserted, 12 values computed." ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Insert some sample documents\n", "documents = [\n", " 'The Mediterranean diet emphasizes fish, olive oil, and vegetables, believed to reduce chronic diseases.',\n", " 'Photosynthesis in plants converts light energy into glucose and produces essential oxygen.',\n", " '20th-century innovations, from radios to smartphones, centered on electronic advancements.',\n", " 'Rivers provide water, irrigation, and habitat for aquatic species, vital for ecosystems.',\n", " \"Apple's conference call to discuss fourth fiscal quarter results is scheduled for Thursday, November 2, 2023.\",\n", " \"Shakespeare's works, like 'Hamlet' and 'A Midsummer Night's Dream,' endure in literature.\",\n", "]\n", "\n", "docs_t.insert({'text': doc} for doc in documents)" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "execution": { "iopub.execute_input": "2025-12-04T04:21:04.084472Z", "iopub.status.busy": "2025-12-04T04:21:04.083934Z", "iopub.status.idle": "2025-12-04T04:21:04.098347Z", "shell.execute_reply": "2025-12-04T04:21:04.098005Z" } }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textembedding
The Mediterranean diet emphasizes fish, olive oil, and vegetables, believed to reduce chronic diseases.[ 0.048 0.016 0.002 0.026 0.038 0.013 ... -0.015 -0.034 -0.016 0.007 0.046 -0.011]
Photosynthesis in plants converts light energy into glucose and produces essential oxygen.[ 0.013 0.023 -0.004 0.052 0.037 0.022 ... -0.013 -0.042 0.001 0.008 -0.02 -0.016]
20th-century innovations, from radios to smartphones, centered on electronic advancements.[ 4.373e-03 4.474e-02 -6.796e-05 2.745e-02 4.904e-02 6.148e-03 ... -1.833e-02 -4.274e-02 -4.713e-03 -1.739e-02 -1.540e-03 -2.306e-02]
" ], "text/plain": [ " text \\\n", "0 The Mediterranean diet emphasizes fish, olive ... \n", "1 Photosynthesis in plants converts light energy... \n", "2 20th-century innovations, from radios to smart... \n", "\n", " embedding \n", "0 [0.04790462180972099, 0.015555687248706818, 0.... \n", "1 [0.013055538758635521, 0.023066800087690353, -... \n", "2 [0.004373149946331978, 0.044735632836818695, -... " ] }, "execution_count": 43, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# View the embeddings\n", "docs_t.select(docs_t.text, docs_t.embedding).head(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Embedding index for similarity search\n", "\n", "You can use Voyage AI embeddings with Pixeltable's embedding index for efficient similarity search." ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "execution": { "iopub.execute_input": "2025-12-04T04:21:04.099684Z", "iopub.status.busy": "2025-12-04T04:21:04.099594Z", "iopub.status.idle": "2025-12-04T04:21:04.145485Z", "shell.execute_reply": "2025-12-04T04:21:04.145137Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created table 'search'.\n" ] } ], "source": [ "# Create a table with an embedding index\n", "search_t = pxt.create_table('voyageai_demo/search', {'text': pxt.String})\n", "\n", "# Add embedding index for similarity search\n", "embed_fn = voyageai.embeddings.using(\n", " model='voyage-3.5', input_type='document'\n", ")\n", "search_t.add_embedding_index('text', string_embed=embed_fn)" ] }, { "cell_type": "code", "execution_count": 45, "metadata": { "execution": { "iopub.execute_input": "2025-12-04T04:21:04.147332Z", "iopub.status.busy": "2025-12-04T04:21:04.147192Z", "iopub.status.idle": "2025-12-04T04:21:05.754182Z", "shell.execute_reply": "2025-12-04T04:21:05.753823Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Inserting rows into `search`: 6 rows [00:00, 973.68 rows/s]\n", "Inserted 6 rows with 0 errors.\n" ] }, { "data": { "text/plain": [ "6 rows inserted, 12 values computed." ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Insert documents\n", "search_t.insert({'text': doc} for doc in documents)" ] }, { "cell_type": "code", "execution_count": 46, "metadata": { "execution": { "iopub.execute_input": "2025-12-04T04:21:05.755476Z", "iopub.status.busy": "2025-12-04T04:21:05.755379Z", "iopub.status.idle": "2025-12-04T04:21:05.770442Z", "shell.execute_reply": "2025-12-04T04:21:05.770149Z" } }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textscore
The Mediterranean diet emphasizes fish, olive oil, and vegetables, believed to reduce chronic diseases.0.863
Photosynthesis in plants converts light energy into glucose and produces essential oxygen.0.649
Shakespeare's works, like 'Hamlet' and 'A Midsummer Night's Dream,' endure in literature.0.621
" ], "text/plain": [ " text score\n", "0 The Mediterranean diet emphasizes fish, olive ... 0.863046\n", "1 Photosynthesis in plants converts light energy... 0.649284\n", "2 Shakespeare's works, like 'Hamlet' and 'A Mids... 0.620859" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Perform similarity search\n", "sim = search_t.text.similarity(\n", " string='What are the health benefits of Mediterranean food?'\n", ")\n", "search_t.order_by(sim, asc=False).limit(3).select(\n", " search_t.text, score=sim\n", ").collect()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reranking\n", "\n", "Voyage AI's rerankers can refine search results by providing more accurate relevance scores." ] }, { "cell_type": "code", "execution_count": 47, "metadata": { "execution": { "iopub.execute_input": "2025-12-04T04:21:05.771729Z", "iopub.status.busy": "2025-12-04T04:21:05.771655Z", "iopub.status.idle": "2025-12-04T04:21:05.805079Z", "shell.execute_reply": "2025-12-04T04:21:05.804812Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created table 'rerank'.\n", "Added 0 column values with 0 errors.\n" ] }, { "data": { "text/plain": [ "No rows affected." ] }, "execution_count": 47, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create a table for reranking\n", "rerank_t = pxt.create_table(\n", " 'voyageai_demo/rerank', {'query': pxt.String, 'documents': pxt.Json}\n", ")\n", "\n", "# Add computed column with reranking results\n", "rerank_t.add_computed_column(\n", " reranked=voyageai.rerank(\n", " rerank_t.query, rerank_t.documents, model='rerank-2.5', top_k=3\n", " )\n", ")" ] }, { "cell_type": "code", "execution_count": 48, "metadata": { "execution": { "iopub.execute_input": "2025-12-04T04:21:05.806212Z", "iopub.status.busy": "2025-12-04T04:21:05.806145Z", "iopub.status.idle": "2025-12-04T04:21:05.952896Z", "shell.execute_reply": "2025-12-04T04:21:05.952561Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Inserting rows into `rerank`: 1 rows [00:00, 343.65 rows/s]\n", "Inserted 1 row with 0 errors.\n" ] }, { "data": { "text/plain": [ "1 row inserted, 2 values computed." ] }, "execution_count": 48, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Insert query and documents to rerank\n", "rerank_t.insert(\n", " [\n", " {\n", " 'query': \"When is Apple's conference call scheduled?\",\n", " 'documents': documents,\n", " }\n", " ]\n", ")" ] }, { "cell_type": "code", "execution_count": 49, "metadata": { "execution": { "iopub.execute_input": "2025-12-04T04:21:05.954281Z", "iopub.status.busy": "2025-12-04T04:21:05.954182Z", "iopub.status.idle": "2025-12-04T04:21:05.961709Z", "shell.execute_reply": "2025-12-04T04:21:05.961471Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Added 1 column value with 0 errors.\n" ] }, { "data": { "text/plain": [ "1 row updated, 1 value computed." ] }, "execution_count": 49, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Add computed column to extract top results using JSON path\n", "rerank_t.add_computed_column(top_results=rerank_t.reranked['results'])" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
querytop_documenttop_score
When is Apple's conference call scheduled?Apple's conference call to discuss fourth fiscal quarter results is scheduled for Thursday, November 2, 2023.0.93
" ], "text/plain": [ " query \\\n", "0 When is Apple's conference call scheduled? \n", "\n", " top_document top_score \n", "0 Apple's conference call to discuss fourth fisc... 0.929688 " ] }, "execution_count": 50, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Extract the top result's document and score\n", "rerank_t.select(\n", " rerank_t.query,\n", " top_document=rerank_t.top_results[0]['document'],\n", " top_score=rerank_t.top_results[0]['relevance_score'],\n", ").collect()" ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
querytop_results
When is Apple's conference call scheduled?[{"index": 4, "document": "Apple's conference call to discuss fourth fiscal quarter results is scheduled for Thursday, November 2, 2023.", "relevance_score": 0.93}, {"index": 2, "document": "20th-century innovations, from radios to smartphones, centered on electronic advancements.", "relevance_score": 0.283}, {"index": 0, "document": "The Mediterranean diet emphasizes fish, olive oil, and vegetables, believed to reduce chronic diseases.", "relevance_score": 0.264}]
" ], "text/plain": [ " query \\\n", "0 When is Apple's conference call scheduled? \n", "\n", " top_results \n", "0 [{'index': 4, 'document': 'Apple's conference ... " ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# View reranking results\n", "rerank_t.select(rerank_t.query, rerank_t.top_results).collect()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Multimodal Embeddings\n", "\n", "Voyage AI's multimodal model (`voyage-multimodal-3`) can embed both images and text into the same vector space, enabling cross-modal similarity search." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": { "iopub.execute_input": "2025-12-04T04:21:05.963058Z", "iopub.status.busy": "2025-12-04T04:21:05.962961Z", "iopub.status.idle": "2025-12-04T04:21:06.012180Z", "shell.execute_reply": "2025-12-04T04:21:06.011871Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created table 'multimodal'.\n", "Added 0 column values with 0 errors.\n", "Added 0 column values with 0 errors.\n" ] }, { "data": { "text/plain": [ "No rows affected." ] }, "execution_count": 52, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create a table for multimodal embeddings\n", "mm_t = pxt.create_table(\n", " 'voyageai_demo/multimodal',\n", " {'image': pxt.Image, 'caption': pxt.String},\n", " if_exists='replace',\n", ")\n", "\n", "# Add computed columns for image and text embeddings\n", "# multimodal_embed can embed either images or text independently\n", "mm_t.add_computed_column(\n", " image_embedding=voyageai.multimodal_embed(\n", " mm_t.image, model='voyage-multimodal-3.5', input_type='document'\n", " )\n", ")\n", "mm_t.add_computed_column(\n", " text_embedding=voyageai.multimodal_embed(\n", " mm_t.caption, model='voyage-multimodal-3.5', input_type='document'\n", " )\n", ")" ] }, { "cell_type": "code", "execution_count": 53, "metadata": { "execution": { "iopub.execute_input": "2025-12-04T04:21:06.013453Z", "iopub.status.busy": "2025-12-04T04:21:06.013318Z", "iopub.status.idle": "2025-12-04T04:21:06.510803Z", "shell.execute_reply": "2025-12-04T04:21:06.510430Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Inserting rows into `multimodal`: 1 rows [00:00, 520.00 rows/s]\n", "Inserted 1 row with 0 errors.\n" ] }, { "data": { "text/plain": [ "1 row inserted, 5 values computed." ] }, "execution_count": 53, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Insert a sample image with caption\n", "mm_t.insert(\n", " [\n", " {\n", " 'image': 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000139.jpg',\n", " 'caption': 'A person standing next to an elephant',\n", " }\n", " ]\n", ")" ] }, { "cell_type": "code", "execution_count": 54, "metadata": { "execution": { "iopub.execute_input": "2025-12-04T04:21:06.512276Z", "iopub.status.busy": "2025-12-04T04:21:06.512174Z", "iopub.status.idle": "2025-12-04T04:21:06.547733Z", "shell.execute_reply": "2025-12-04T04:21:06.547379Z" } }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
imagecaptionimage_embeddingtext_embedding
\n", " \n", "
A person standing next to an elephant[-0.017 -0.001 0.015 0.008 -0.025 -0.012 ... 0.01 0.058 -0.031 0.017 0.011 -0.055][ 0.012 -0.002 -0.024 0.012 -0.036 -0.003 ... -0.041 0.007 0.019 -0.013 0.027 0.001]
" ], "text/plain": [ " image \\\n", "0