{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Extract structured data from images\n", "\n", "Use AI vision to extract JSON data from receipts, forms, documents, and other images." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem\n", "\n", "You have images containing structured information (receipts, forms, ID cards) and need to extract specific fields as JSON for downstream processing.\n", "\n", "| Image | Fields to extract |\n", "|-------|------------------|\n", "| receipt.jpg | vendor, total, date, items |\n", "| business_card.jpg | name, email, phone, company |\n", "| invoice.pdf | invoice_number, amount, due_date |" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Solution\n", "\n", "**What's in this recipe:**\n", "\n", "- Extract structured JSON from images using GPT-4o\n", "- Use `openai.chat_completions()` with multimodal messages\n", "- Access individual fields from the extracted data\n", "\n", "You use Pixeltable's `openai.chat_completions()` function with multimodal messages that include images directly. Request JSON output via `response_format` in `model_kwargs`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install -qU pixeltable openai\n", "\n", "import getpass\n", "import os\n", "\n", "if 'OPENAI_API_KEY' not in os.environ:\n", " os.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key: ')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load images" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/asiegel/.pixeltable/pgdata\n", "Created directory 'extraction_demo'.\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pixeltable as pxt\n", "from pixeltable.functions import openai\n", "\n", "# Create a fresh directory\n", "pxt.drop_dir('extraction_demo', force=True)\n", "pxt.create_dir('extraction_demo')" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created table 'images'.\n" ] } ], "source": [ "t = pxt.create_table('extraction_demo/images', {'image': pxt.Image})" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Inserted 2 rows with 0 errors in 0.03 s (60.43 rows/s)\n" ] }, { "data": { "text/plain": [ "2 rows inserted." ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Insert sample images\n", "t.insert(\n", " [\n", " {\n", " 'image': 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000036.jpg'\n", " },\n", " {\n", " 'image': 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000090.jpg'\n", " },\n", " ]\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Extract structured data\n", "\n", "Use `openai.chat_completions()` to analyze images and get JSON output:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Added 2 column values with 0 errors in 7.55 s (0.26 rows/s)\n" ] }, { "data": { "text/plain": [ "2 rows updated." ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Add extraction column using openai.chat_completions with multimodal messages\n", "PROMPT = \"\"\"Analyze this image and extract the following as JSON:\n", "- description: A brief description of the image\n", "- objects: List of objects visible in the image\n", "- dominant_colors: List of dominant colors\n", "- scene_type: Type of scene (indoor, outdoor, etc.)\"\"\"\n", "\n", "messages = [\n", " {\n", " 'role': 'user',\n", " 'content': [\n", " {'type': 'text', 'text': PROMPT},\n", " {'type': 'image_url', 'image_url': t.image},\n", " ],\n", " }\n", "]\n", "\n", "t.add_computed_column(\n", " data=openai.chat_completions(\n", " messages,\n", " model='gpt-4o-mini',\n", " model_kwargs={'response_format': {'type': 'json_object'}},\n", " )\n", ")" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
imagedatadata_choices0_message_content
\n", " \n", "
{"id": "chatcmpl-DCZGQMTD4imtZ6xI1Z6rwiebHO5Jd", "model": "gpt-4o-mini-2024-07-18", "usage": {"total_tokens": 14327, "prompt_tokens": 14229, "completion_tokens": 98, "prompt_tokens_details": {"audio_tokens": 0, "cached_tokens": 0}, "completion_tokens_details": {"audio_tokens": 0, "reasoning_tokens": 0, "accepted_prediction_tokens": 0, "rejected_prediction_tokens": 0}}, "object": "chat.completion", "choices": [{"index": 0, "message": {"role": "assistant", "content": "{\\n \\"description\\": \\"A woman holding a pink umbrella and wearing a colorful swimsuit, smiling against a backdrop of a lake and blue sky.\\",\\n \\"objec ...... \\"sky\\",\\n \\"trees\\"\\n ],\\n \\"dominant_colors\\": [\\n \\"pink\\",\\n \\"blue\\",\\n \\"green\\",\\n \\"white\\",\\n \\"purple\\"\\n ],\\n \\"scene_type\\": \\"outdoor\\"\\n}", "refusal": null, "annotations": []}, "logprobs": null, "finish_reason": "stop"}], "created": 1771887862, "service_tier": "default", "system_fingerprint": "fp_373a14eb6f"}{\n", " "description": "A woman holding a pink umbrella and wearing a colorful swimsuit, smiling against a backdrop of a lake and blue sky.",\n", " "objects": [\n", " "umbrella",\n", " "swimsuit",\n", " "water",\n", " "sky",\n", " "trees"\n", " ],\n", " "dominant_colors": [\n", " "pink",\n", " "blue",\n", " "green",\n", " "white",\n", " "purple"\n", " ],\n", " "scene_type": "outdoor"\n", "}
\n", " \n", "
{"id": "chatcmpl-DCZGU0lzNE9d1K8UgFaNs0P1OPleO", "model": "gpt-4o-mini-2024-07-18", "usage": {"total_tokens": 14321, "prompt_tokens": 14229, "completion_tokens": 92, "prompt_tokens_details": {"audio_tokens": 0, "cached_tokens": 0}, "completion_tokens_details": {"audio_tokens": 0, "reasoning_tokens": 0, "accepted_prediction_tokens": 0, "rejected_prediction_tokens": 0}}, "object": "chat.completion", "choices": [{"index": 0, "message": {"role": "assistant", "content": "{\\n \\"description\\": \\"A serene landscape featuring a cow grazing in a green field with a tree and distant buildings under a partly cloudy sky.\\",\\n \\" ...... \\"grass\\",\\n \\"buildings\\",\\n \\"sky\\"\\n ],\\n \\"dominant_colors\\": [\\n \\"green\\",\\n \\"blue\\",\\n \\"white\\",\\n \\"brown\\"\\n ],\\n \\"scene_type\\": \\"outdoor\\"\\n}", "refusal": null, "annotations": []}, "logprobs": null, "finish_reason": "stop"}], "created": 1771887866, "service_tier": "default", "system_fingerprint": "fp_373a14eb6f"}{\n", " "description": "A serene landscape featuring a cow grazing in a green field with a tree and distant buildings under a partly cloudy sky.",\n", " "objects": [\n", " "cow",\n", " "tree",\n", " "grass",\n", " "buildings",\n", " "sky"\n", " ],\n", " "dominant_colors": [\n", " "green",\n", " "blue",\n", " "white",\n", " "brown"\n", " ],\n", " "scene_type": "outdoor"\n", "}
" ], "text/plain": [ " image \\\n", "0 \n", " \n", " \n", " image\n", " description\n", " \n", " \n", " \n", " \n", "
\n", " \n", "
\n", " A woman holding a pink umbrella and wearing a colorful swimsuit, smiling against a backdrop of a lake and blue sky.\n", " \n", " \n", "
\n", " \n", "
\n", " A serene landscape featuring a cow grazing in a green field with a tree and distant buildings under a partly cloudy sky.\n", " \n", " \n", "" ], "text/plain": [ " image \\\n", "0 str:\n", " return json.loads(data).get('description', '')\n", "\n", "\n", "t.select(\n", " t.image,\n", " description=parse_description(\n", " t.data['choices'][0]['message']['content']\n", " ),\n", ").collect()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Explanation\n", "\n", "**Getting JSON output:**\n", "\n", "Pass `model_kwargs={'response_format': {'type': 'json_object'}}` to get structured JSON.\n", "\n", "**Other extraction use cases:**\n", "\n", "| Use Case | Fields to extract |\n", "|----------|------------------|\n", "| Receipts | vendor, total, date, items, tax |\n", "| Business cards | name, title, company, email, phone |\n", "| Product photos | brand, model, condition, defects |\n", "| Documents | title, date, author, summary |" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## See also\n", "\n", "- [Analyze images in batch](https://docs.pixeltable.com/howto/cookbooks/images/vision-batch-analysis)\n", "- [Configure API keys](https://docs.pixeltable.com/howto/cookbooks/core/workflow-api-keys)" ] } ], "metadata": { "kernelspec": { "display_name": "pixeltable", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.11" } }, "nbformat": 4, "nbformat_minor": 4 }