{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with RunwayML in Pixeltable\n",
    "\n",
    "Pixeltable's RunwayML integration lets you generate and transform images and videos using RunwayML's Gen-4.5, Veo, and other models, directly from your Pixeltable tables.\n",
    "\n",
    "## What is RunwayML?\n",
    "\n",
    "[RunwayML](https://runwayml.com/) is an AI creative platform offering state-of-the-art generative models for images and video. Pixeltable wraps four endpoints:\n",
    "\n",
    "- **`text_to_image`**: Generate images from text prompts with optional reference images\n",
    "- **`text_to_video`**: Generate videos from text prompts\n",
    "- **`image_to_video`**: Animate a still image into a video\n",
    "- **`video_to_video`**: Transform an existing video with text guidance\n",
    "\n",
    "### Documentation\n",
    "\n",
    "- [Pixeltable RunwayML SDK Reference](https://docs.pixeltable.com/sdk/latest/runwayml)\n",
    "- [RunwayML API Reference](https://docs.dev.runwayml.com/api/)\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "1. A RunwayML account with an API secret from [RunwayML](https://app.runwayml.com/)\n",
    "1. `pip install runwayml`\n",
    "\n",
    "**Important:** RunwayML API calls consume credits based on your plan. Image and video generation can be expensive — monitor your usage to avoid unexpected charges."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU pixeltable runwayml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "import os\n",
    "\n",
    "if 'RUNWAYML_API_SECRET' not in os.environ:\n",
    "    os.environ['RUNWAYML_API_SECRET'] = getpass.getpass(\n",
    "        'RunwayML API Secret: '\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To read more about working with API keys in Pixeltable, see [Configuration](https://docs.pixeltable.com/platform/configuration)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created directory 'runwayml_demo'.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<pixeltable.catalog.dir.Dir at 0x32bd20a10>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pixeltable as pxt\n",
    "from pixeltable.functions import runwayml\n",
    "\n",
    "pxt.drop_dir('runwayml_demo', force=True)\n",
    "pxt.create_dir('runwayml_demo')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text-to-Video Generation\n",
    "\n",
    "Generate videos from text prompts. The `text_to_video` function returns a JSON response containing the output video URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created table 'text_to_video'.\n",
      "Added 0 column values with 0 errors in 0.00 s\n",
      "Added 0 column values with 0 errors in 0.00 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "No rows affected."
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t2v = pxt.create_table(\n",
    "    'runwayml_demo.text_to_video', {'prompt': pxt.String}\n",
    ")\n",
    "\n",
    "t2v.add_computed_column(\n",
    "    response=runwayml.text_to_video(\n",
    "        t2v.prompt, model='gen4.5', ratio='1280:720', duration=5\n",
    "    )\n",
    ")\n",
    "\n",
    "# Extract the video URL from the response\n",
    "t2v.add_computed_column(video=t2v.response['output'][0].astype(pxt.Video))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inserted 1 row with 0 errors in 105.52 s (0.01 rows/s)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1 row inserted."
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t2v.insert(\n",
    "    [\n",
    "        {\n",
    "            'prompt': 'A cinematic aerial shot of a coastal city at golden hour, waves crashing against the shore'\n",
    "        }\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t2v.select(t2v.prompt, t2v.video).collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Image-to-Video Generation\n",
    "\n",
    "Animate a still image into a video. You can optionally add a text prompt to guide the motion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created table 'image_to_video'.\n",
      "Added 0 column values with 0 errors in 0.01 s\n",
      "Added 0 column values with 0 errors in 0.01 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "No rows affected."
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i2v = pxt.create_table(\n",
    "    'runwayml_demo.image_to_video',\n",
    "    {'image': pxt.Image, 'prompt': pxt.String},\n",
    ")\n",
    "\n",
    "i2v.add_computed_column(\n",
    "    response=runwayml.image_to_video(\n",
    "        i2v.image,\n",
    "        model='gen4.5',\n",
    "        ratio='1280:720',\n",
    "        prompt_text=i2v.prompt,\n",
    "        duration=5,\n",
    "    )\n",
    ")\n",
    "\n",
    "i2v.add_computed_column(video=i2v.response['output'][0].astype(pxt.Video))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inserted 1 row with 0 errors in 145.73 s (0.01 rows/s)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1 row inserted."
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i2v.insert(\n",
    "    [\n",
    "        {\n",
    "            'image': 'https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg',\n",
    "            'prompt': 'The ant slowly walks across the leaf',\n",
    "        }\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i2v.select(i2v.image, i2v.video).collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text-to-Image Generation\n",
    "\n",
    "Generate images from text prompts with reference images. The `text_to_image` function requires at least one reference image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created table 'text_to_image'.\n",
      "Added 0 column values with 0 errors in 0.01 s\n",
      "Added 0 column values with 0 errors in 0.01 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "No rows affected."
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t2i = pxt.create_table(\n",
    "    'runwayml_demo.text_to_image',\n",
    "    {'prompt': pxt.String, 'ref_image': pxt.Image},\n",
    ")\n",
    "\n",
    "t2i.add_computed_column(\n",
    "    response=runwayml.text_to_image(\n",
    "        t2i.prompt, [t2i.ref_image], model='gen4_image', ratio='1280:720'\n",
    "    )\n",
    ")\n",
    "\n",
    "# The response contains a list of output image URLs\n",
    "t2i.add_computed_column(image=t2i.response['output'][0].astype(pxt.Image))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inserted 1 row with 0 errors in 21.70 s (0.05 rows/s)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1 row inserted."
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t2i.insert(\n",
    "    [\n",
    "        {\n",
    "            'prompt': 'A photorealistic painting in the same style as the reference',\n",
    "            'ref_image': 'https://upload.wikimedia.org/wikipedia/commons/6/6a/Mona_Lisa.jpg',\n",
    "        }\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t2i.select(t2i.ref_image, t2i.image).collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Video-to-Video Transformation\n",
    "\n",
    "Transform an existing video with text guidance. Note that `video_to_video` requires a publicly accessible HTTPS URL for the input video (not a local file)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v2v = pxt.create_table(\n",
    "    'runwayml_demo.video_to_video',\n",
    "    {'video_url': pxt.String, 'style_prompt': pxt.String},\n",
    ")\n",
    "\n",
    "v2v.add_computed_column(\n",
    "    response=runwayml.video_to_video(\n",
    "        v2v.video_url, v2v.style_prompt, model='aleph2', ratio='1280:720'\n",
    "    )\n",
    ")\n",
    "\n",
    "v2v.add_computed_column(video=v2v.response['output'][0].astype(pxt.Video))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using `model_kwargs` for Advanced Parameters\n",
    "\n",
    "All RunwayML functions accept an optional `model_kwargs` parameter for passing additional API parameters not exposed as explicit arguments. Refer to the [RunwayML API docs](https://docs.dev.runwayml.com/api/) for the full list of supported parameters per endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example: pass additional parameters via model_kwargs\n",
    "advanced = pxt.create_table(\n",
    "    'runwayml_demo.advanced', {'prompt': pxt.String}\n",
    ")\n",
    "\n",
    "advanced.add_computed_column(\n",
    "    response=runwayml.text_to_video(\n",
    "        advanced.prompt,\n",
    "        model='gen4.5',\n",
    "        ratio='1280:720',\n",
    "        duration=5,\n",
    "        model_kwargs={'audio': True, 'seed': 42},\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learn More\n",
    "\n",
    "- [RunwayML SDK Reference](https://docs.pixeltable.com/sdk/latest/runwayml) — Full API details for all RunwayML functions\n",
    "- [RunwayML API Documentation](https://docs.dev.runwayml.com/api/) — Official RunwayML API reference\n",
    "- [Working with fal.ai](https://docs.pixeltable.com/howto/providers/working-with-fal) — Another image/video generation integration"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}