{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "983yjns496tx" }, "source": [ "# Working with Gemini in Pixeltable\n", "\n", "Pixeltable's Gemini integration supports two authentication methods:\n", "\n", "- Google AI Studio: requires an API key from https://aistudio.google.com/app/apikey\n", "- Vertex AI: uses Application Default Credentials via the Google Cloud SDK\n", "\n", "### Prerequisites\n", "\n", "For Google AI Studio:\n", "\n", "- A Google AI Studio account with an API key (https://aistudio.google.com/app/apikey)\n", "\n", "For Vertex AI:\n", "\n", "- A Google Cloud project with the Vertex AI API enabled\n", "- The Google Cloud SDK installed and configured (`gcloud auth application-default login`)" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "9pckrD01ik-e", "outputId": "060b8b32-48a6-48a0-e720-4eacf94d83ef" }, "source": [ "%pip install -qU pixeltable google-genai" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "AQ6_Py7_7d0r", "outputId": "f82cfe36-be9e-4d43-f13e-9f6f5b680e8e" }, "source": [ "import os\n", "\n", "vertex_enabled = (\n", " os.environ.get('GOOGLE_GENAI_USE_VERTEXAI', '').lower() == 'true'\n", ")\n", "\n", "# Option 1: Google AI Studio (API key)\n", "# Set GEMINI_API_KEY or GOOGLE_API_KEY in your environment,\n", "# or add api_key to the [gemini] section of $PIXELTABLE_HOME/config.toml\n", "\n", "if (\n", " not vertex_enabled\n", " and 'GEMINI_API_KEY' not in os.environ\n", " and 'GOOGLE_API_KEY' not in os.environ\n", "):\n", " import getpass\n", "\n", " os.environ['GEMINI_API_KEY'] = getpass.getpass(\n", " 'Google AI Studio API Key:'\n", " )\n", "\n", "# Option 2: Vertex AI (Application Default Credentials)\n", "# Uncomment and set the following environment variables to use Vertex AI instead:\n", "# os.environ['GOOGLE_GENAI_USE_VERTEXAI'] = 'true'\n", "# os.environ['GOOGLE_CLOUD_PROJECT'] = 'your-project-id'\n", "# os.environ['GOOGLE_CLOUD_LOCATION'] = 'us-central1' # optional, defaults to us-central1\n", "# Then authenticate via: gcloud auth application-default login" ], "execution_count": 4, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create a Pixeltable directory for this notebook's demo tables." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "brtjK-88tTSS", "outputId": "55d08c91-438a-4c3e-c217-3cea72faca11" }, "source": [ "import pixeltable as pxt\n", "import pixeltable.functions as pxtf\n", "\n", "pxt.create_dir('gemini_demo', if_exists='replace_force')" ], "execution_count": 62, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Created directory 'gemini_demo'.\n" ] }, { "output_type": "execute_result", "metadata": {}, "execution_count": null, "data": { "text/plain": [ "" ] } } ] }, { "cell_type": "markdown", "metadata": { "id": "0kmjJoDq9Oqe" }, "source": [ "## Generate content\n", "\n", "Create a table with an input column, then add a computed column that calls Gemini on each row." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5ti10tXu5m3X", "outputId": "30848066-1e9b-4efd-aad7-b2271a031ec3" }, "source": [ "from google.genai.types import GenerateContentConfigDict\n", "\n", "# Create a table in Pixeltable and pick a model hosted on Google AI Studio with some parameters\n", "\n", "t = pxt.create_table('gemini_demo/text', {'input': pxt.String})\n", "\n", "config = GenerateContentConfigDict(\n", " max_output_tokens=500, temperature=1.0, top_p=0.95, top_k=40\n", ")\n", "t.add_computed_column(\n", " output=pxtf.gemini.generate_content(\n", " t.input, model='gemini-2.5-flash', config=config\n", " )\n", ")" ], "execution_count": 63, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Created table 'text'.\n", "Added 0 column values with 0 errors in 0.00 s\n" ] }, { "output_type": "execute_result", "metadata": {}, "execution_count": null, "data": { "text/plain": [ "No rows affected." ] } } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 599 }, "id": "IkMM7OYb5rQ_", "outputId": "8e94af3e-485c-49f2-d7ba-b5490ec83af9" }, "source": [ "# Ask Gemini for short responses so results are easy to inspect\n", "t.insert(\n", " [\n", " {'input': 'Write a two-sentence story about a magic backpack.'},\n", " {'input': 'Tell a one-line science joke.'},\n", " ]\n", ")" ], "execution_count": 64, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Inserted 2 rows with 0 errors in 5.90 s (0.34 rows/s)\n" ] }, { "output_type": "execute_result", "metadata": {}, "execution_count": null, "data": { "text/plain": [ "2 rows inserted." ] } } ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Parse the response text into its own column\n", "t.add_computed_column(\n", " response=t.output['candidates'][0]['content']['parts'][0]['text'],\n", " if_exists='replace',\n", ")" ], "execution_count": 66, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Added 2 column values with 0 errors in 0.01 s (184.71 rows/s)\n" ] }, { "output_type": "execute_result", "metadata": {}, "execution_count": null, "data": { "text/plain": [ "2 rows updated." ] } } ] }, { "cell_type": "code", "metadata": {}, "source": [ "t.select(t.input, t.response).collect()" ], "execution_count": 67, "outputs": [ { "output_type": "execute_result", "metadata": {}, "execution_count": null, "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
inputresponse
Write a two-sentence story about a magic backpack.Elara's worn canvas backpack always seemed to hold exactly what she needed, producing everything from a forgotten umbrella to an entire picnic lunch. One afternoon, caught unprepared for a sudden pop quiz, she unzipped it to find a perfectly highlighted textbook waiting inside.
Tell a one-line science joke.Why don't scientists trust atoms?\n", "\n", "Because they make up everything!
" ], "text/plain": [ " input \\\n", "0 Write a two-sentence story about a magic backp... \n", "1 Tell a one-line science joke. \n", "\n", " response \n", "0 Elara's worn canvas backpack always seemed to ... \n", "1 Why don't scientists trust atoms?\\n\\nBecause t... " ] } } ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Generate images with Nano Banana\n", "\n", "Use `pxtf.gemini.generate_content` with model `gemini-2.5-flash-image`.\n", "\n", "This returns a structured response, so extract the image from `response['candidates'][0]['content']['parts'][0]['inline_data']['data']`, as shown below." ] }, { "cell_type": "code", "metadata": {}, "source": [ "nano_t = pxt.create_table(\n", " 'gemini_demo/nano_banana_images', {'prompt': pxt.String}\n", ")\n", "\n", "config = GenerateContentConfigDict(\n", " response_modalities=['IMAGE'], image_config={'aspect_ratio': '16:9'}\n", ")\n", "nano_t.add_computed_column(\n", " response=pxtf.gemini.generate_content(\n", " nano_t.prompt, model='gemini-2.5-flash-image', config=config\n", " )\n", ")\n", "nano_t.add_computed_column(\n", " generated_image=(\n", " nano_t.response.candidates[0]\n", " .content.parts[0]\n", " .inline_data.data.astype(pxt.Image)\n", " )\n", ")" ], "execution_count": 11, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Created table 'nano_banana_images'.\n", "Added 0 column values with 0 errors in 0.00 s\n", "Added 0 column values with 0 errors in 0.00 s\n" ] }, { "output_type": "execute_result", "metadata": {}, "execution_count": null, "data": { "text/plain": [ "No rows affected." ] } } ] }, { "cell_type": "code", "metadata": {}, "source": [ "nano_t.insert(\n", " [{'prompt': 'A friendly dinosaur playing tennis in a cornfield'}]\n", ")" ], "execution_count": 12, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Inserted 1 row with 0 errors in 6.61 s (0.15 rows/s)\n" ] }, { "output_type": "execute_result", "metadata": {}, "execution_count": null, "data": { "text/plain": [ "1 row inserted." ] } } ] }, { "cell_type": "code", "metadata": {}, "source": [ "nano_t.select(nano_t.prompt, nano_t.generated_image).collect()\n" ], "execution_count": 14, "outputs": [ { "output_type": "execute_result", "metadata": {}, "execution_count": null, "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
promptgenerated_image
A friendly dinosaur playing tennis in a cornfield
\n", " \n", "
" ], "text/plain": [ " prompt \\\n", "0 A friendly dinosaur playing tennis in a cornfield \n", "\n", " generated_image \n", "0 \n", " \n", " \n", " prompt\n", " generated_image\n", " \n", " \n", " \n", " \n", " A friendly dinosaur playing tennis in a cornfield\n", "
\n", " \n", "
\n", " \n", " \n", "" ], "text/plain": [ " prompt \\\n", "0 A friendly dinosaur playing tennis in a cornfield \n", "\n", " generated_image \n", "0 \n", " \n", " \n", " prompt\n", " generated_video\n", " \n", " \n", " \n", " \n", " A giant pixel floating over the open ocean in a sea of data\n", "
\n", " \n", "
\n", " \n", " \n", "" ], "text/plain": [ " prompt \\\n", "0 A giant pixel floating over the open ocean in ... \n", "\n", " generated_video \n", "0 /Users/alison-pxt/.pixeltable/media/72e5e7e33e... " ] } } ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Generate video from an existing image" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll add a computed column to our `nano_t` table to animate the generated image. You can pass `images_t.generated_image` from the Imagen section above the same way." ] }, { "cell_type": "code", "metadata": {}, "source": [ "nano_t.add_computed_column(\n", " generated_video=pxtf.gemini.generate_videos(\n", " image=nano_t.generated_image, model='veo-2.0-generate-001'\n", " )\n", ")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "nano_t.select(nano_t.prompt, nano_t.generated_video).collect()\n" ], "execution_count": 25, "outputs": [ { "output_type": "execute_result", "metadata": {}, "execution_count": null, "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
promptgenerated_video
A friendly dinosaur playing tennis in a cornfield
\n", " \n", "
" ], "text/plain": [ " prompt \\\n", "0 A friendly dinosaur playing tennis in a cornfield \n", "\n", " generated_video \n", "0 /Users/alison-pxt/.pixeltable/media/16de97a6ac... " ] } } ] }, { "cell_type": "markdown", "metadata": { "id": "lTtQcjKQAlis" }, "source": [ "### Learn more\n", "\n", "For a deeper end-to-end example, see [RAG Operations in Pixeltable](https://docs.pixeltable.com/howto/use-cases/rag-operations)." ] } ], "metadata": { "colab": { "provenance": [] }, "kernelspec": { "display_name": "pxt", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.19" } }, "nbformat": 4, "nbformat_minor": 4 }