{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "983yjns496tx" }, "source": [ "# Working with Gemini in Pixeltable\n", "\n", "Pixeltable's Gemini integration enables you to access the Gemini LLM via the Google Gemini API.\n", "\n", "### Prerequisites\n", "\n", "- A Google AI Studio account with an API key (https://aistudio.google.com/app/apikey)\n", "\n", "### Important notes\n", "\n", "- Google AI Studio usage may incur costs based on your plan.\n", "- Be mindful of sensitive data and consider security measures when integrating with external services.\n", "\n", "First you'll need to install required libraries and enter a Gemini API key obtained via Google AI Studio." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "9pckrD01ik-e", "outputId": "060b8b32-48a6-48a0-e720-4eacf94d83ef" }, "outputs": [], "source": [ "%pip install -qU pixeltable google-genai" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "AQ6_Py7_7d0r", "outputId": "f82cfe36-be9e-4d43-f13e-9f6f5b680e8e" }, "outputs": [], "source": [ "import getpass\n", "import os\n", "\n", "if 'GEMINI_API_KEY' not in os.environ:\n", " os.environ['GEMINI_API_KEY'] = getpass.getpass(\n", " 'Google AI Studio API Key:'\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's create a Pixeltable directory to hold the tables for our demo." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "brtjK-88tTSS", "outputId": "55d08c91-438a-4c3e-c217-3cea72faca11" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/asiegel/.pixeltable/pgdata\n", "Created directory 'gemini_demo'.\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pixeltable as pxt\n", "\n", "# Remove the 'gemini_demo' directory and its contents, if it exists\n", "pxt.drop_dir('gemini_demo', force=True)\n", "pxt.create_dir('gemini_demo')" ] }, { "cell_type": "markdown", "metadata": { "id": "0kmjJoDq9Oqe" }, "source": [ "## Generate content\n", "\n", "Create a Table: In Pixeltable, create a table with columns to represent your input data and the columns where you want to store the results from Gemini." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5ti10tXu5m3X", "outputId": "30848066-1e9b-4efd-aad7-b2271a031ec3" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created table 'text'.\n", "Added 0 column values with 0 errors in 0.01 s\n" ] }, { "data": { "text/plain": [ "No rows affected." ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from google.genai.types import GenerateContentConfigDict\n", "from pixeltable.functions import gemini\n", "\n", "# Create a table in Pixeltable and pick a model hosted on Google AI Studio with some parameters\n", "\n", "t = pxt.create_table('gemini_demo/text', {'input': pxt.String})\n", "\n", "config = GenerateContentConfigDict(\n", " stop_sequences=['\\n'],\n", " max_output_tokens=300,\n", " temperature=1.0,\n", " top_p=0.95,\n", " top_k=40,\n", ")\n", "t.add_computed_column(\n", " output=gemini.generate_content(\n", " t.input, model='gemini-2.0-flash', config=config\n", " )\n", ")" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 599 }, "id": "IkMM7OYb5rQ_", "outputId": "8e94af3e-485c-49f2-d7ba-b5490ec83af9" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Inserted 2 rows with 0 errors in 1.43 s (1.39 rows/s)\n" ] }, { "data": { "text/plain": [ "2 rows inserted." ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Ask Gemini to generate some content based on the input\n", "t.insert(\n", " [\n", " {'input': 'Write a story about a magic backpack.'},\n", " {'input': 'Tell me a science joke.'},\n", " ]\n", ")" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Added 2 column values with 0 errors in 0.03 s (62.79 rows/s)\n" ] }, { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
inputresponse
Tell me a science joke.Why did the chemist make a terrible fisherman?
Write a story about a magic backpack.Maya hated Mondays. Especially this Monday. Her alarm hadn't gone off, she’d spilled orange juice down her favourite t-shirt, and to top it all off, her usual bus was late. As she stomped into Mr. Abernathy’s dusty old antique shop, looking for her forgotten library book, she was about as thrilled as a cat in a bathtub.
" ], "text/plain": [ " input \\\n", "0 Tell me a science joke. \n", "1 Write a story about a magic backpack. \n", "\n", " response \n", "0 Why did the chemist make a terrible fisherman? \n", "1 Maya hated Mondays. Especially this Monday. He... " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Parse the response into a new column\n", "t.add_computed_column(\n", " response=t.output['candidates'][0]['content']['parts'][0]['text']\n", ")\n", "t.select(t.input, t.response).head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Generate images with Imagen" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created table 'images'.\n", "Added 0 column values with 0 errors in 0.01 s\n" ] }, { "data": { "text/plain": [ "No rows affected." ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from google.genai.types import GenerateImagesConfigDict\n", "\n", "images_t = pxt.create_table('gemini_demo/images', {'prompt': pxt.String})\n", "\n", "config = GenerateImagesConfigDict(aspect_ratio='16:9')\n", "images_t.add_computed_column(\n", " generated_image=gemini.generate_images(\n", " images_t.prompt, model='imagen-4.0-generate-001', config=config\n", " )\n", ")" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Inserted 1 row with 0 errors in 9.41 s (0.11 rows/s)\n" ] }, { "data": { "text/plain": [ "1 row inserted." ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "images_t.insert(\n", " [{'prompt': 'A friendly dinosaur playing tennis in a cornfield'}]\n", ")" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
promptgenerated_image
A friendly dinosaur playing tennis in a cornfield
\n", " \n", "
" ], "text/plain": [ " prompt \\\n", "0 A friendly dinosaur playing tennis in a cornfield \n", "\n", " generated_image \n", "0 \n", " \n", " \n", " prompt\n", " generated_video\n", " \n", " \n", " \n", " \n", " A giant pixel floating over the open ocean in a sea of data\n", "
\n", " \n", "
\n", " \n", " \n", "" ], "text/plain": [ " prompt \\\n", "0 A giant pixel floating over the open ocean in ... \n", "\n", " generated_video \n", "0 /Users/asiegel/.pixeltable/media/6743560b75734... " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "videos_t.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Generate Video from an existing Image" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll add an additional computed column to our existing `images_t` to animate the generated images." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Added 1 column value with 0 errors in 40.00 s (0.03 rows/s)\n" ] }, { "data": { "text/plain": [ "1 row updated." ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "images_t.add_computed_column(\n", " generated_video=gemini.generate_videos(\n", " image=images_t.generated_image, model='veo-2.0-generate-001'\n", " )\n", ")" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
promptgenerated_imagegenerated_video
A friendly dinosaur playing tennis in a cornfield
\n", " \n", "
\n", " \n", "
" ], "text/plain": [ " prompt \\\n", "0 A friendly dinosaur playing tennis in a cornfield \n", "\n", " generated_image \\\n", "0