{"cells": [{"cell_type": "markdown", "id": "9ce3dae4-9dd8-48fe-9f3a-6e1ddd79e5a0", "metadata": {}, "source": ["# [Beta] 多模态ReAct Agent\n", "\n", "\"在Colab中打开\"/\n", "\n", "在本教程中，我们将向您展示如何构建一个多模态ReAct代理。\n", "\n", "这是一个可以接受文本和图像作为输入任务定义的代理，通过思维链和工具使用来尝试解决任务。\n", "\n", "这是通过我们的低级代理API实现的，允许我们明确地步进ReAct循环，以向您展示每个步骤中发生的事情。\n", "\n", "我们展示了两个用例：\n", "1. **RAG代理**：给定文本/图像，可以查询RAG管道以查找答案。（给定OpenAI Dev Day 2023的屏幕截图）\n", "2. **Web代理**：给定文本/图像，可以查询网络工具以从网络中查找相关信息（给定鞋子的图片）。\n", "\n", "**注意**：这明确是一个beta功能，抽象化可能会随时间改变！\n", "\n", "**注意**：目前仅适用于GPT-4V。\n"]}, {"cell_type": "markdown", "id": "d50c7396-e56a-4018-bb45-596c218a1947", "metadata": {}, "source": ["## 使用RAG管道增强图像分析\n", "\n", "在本节中，我们将创建一个配备了RAG工具的多模态代理。\n"]}, {"cell_type": "markdown", "id": "68bcdd1c-c69f-4da6-9712-29a8ad25b858", "metadata": {}, "source": ["### 设置数据\n"]}, {"cell_type": "code", "execution_count": null, "id": "2a820720", "metadata": {}, "outputs": [], "source": ["%pip install llama-index-llms-openai\n", "%pip install llama-index-readers-web\n", "%pip install llama-index-multi-modal-llms-openai\n", "%pip install llama-index-tools-metaphor"]}, {"cell_type": "code", "execution_count": null, "id": "54ce02e1-1edc-44fd-b160-0e15cc1961f1", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["--2024-01-02 20:25:25-- https://images.openai.com/blob/a2e49de2-ba5b-4869-9c2d-db3b4b5dcc19/new-models-and-developer-products-announced-at-devday.jpg?width=2000\n", "Resolving images.openai.com (images.openai.com)... 2606:4700:4400::6812:28cd, 2606:4700:4400::ac40:9333, 172.64.147.51, ...\n", "Connecting to images.openai.com (images.openai.com)|2606:4700:4400::6812:28cd|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 300894 (294K) [image/jpeg]\n", "Saving to: ‘other_images/openai/dev_day.png’\n", "\n", "other_images/openai 100%[===================>] 293.84K --.-KB/s in 0.02s \n", "\n", "2024-01-02 20:25:25 (13.8 MB/s) - ‘other_images/openai/dev_day.png’ saved [300894/300894]\n", "\n"]}], "source": ["# 下载我们之后用于运行查询的图片\n", "!wget \"https://images.openai.com/blob/a2e49de2-ba5b-4869-9c2d-db3b4b5dcc19/new-models-and-developer-products-announced-at-devday.jpg?width=2000\" -O other_images/openai/dev_day.png\n", "!wget \"https://drive.google.com/uc\\?id\\=1B4f5ZSIKN0zTTPPRlZ915Ceb3_uF9Zlq\\&export\\=download\" -O other_images/adidas.png"]}, {"cell_type": "code", "execution_count": null, "id": "d0522215-4d36-4bbc-8c77-945c4e1bc9a4", "metadata": {}, "outputs": [], "source": ["from llama_index.readers.web import SimpleWebPageReader\n", "\n", "url = \"https://openai.com/blog/new-models-and-developer-products-announced-at-devday\"\n", "reader = SimpleWebPageReader(html_to_text=True)\n", "documents = reader.load_data(urls=[url])"]}, {"cell_type": "markdown", "id": "e7dfd978-7f56-4a15-9e56-019c7a35f35e", "metadata": {}, "source": ["### 设置工具\n"]}, {"cell_type": "code", "execution_count": null, "id": "24b510d5-419e-4963-bba9-cb76efcf7ba4", "metadata": {}, "outputs": [], "source": ["from llama_index.llms.openai import OpenAI\n", "from llama_index.core import VectorStoreIndex\n", "from llama_index.core.tools import QueryEngineTool, ToolMetadata"]}, {"cell_type": "code", "execution_count": null, "id": "367e4884-bf62-45f1-9e0c-211379287bfb", "metadata": {}, "outputs": [], "source": ["from llama_index.core import Settings\n", "\n", "Settings.llm = OpenAI(temperature=0, model=\"gpt-3.5-turbo\")"]}, {"cell_type": "code", "execution_count": null, "id": "60b78d55-5a36-4943-8a25-468e26566c56", "metadata": {}, "outputs": [], "source": ["vector_index = VectorStoreIndex.from_documents(\n", " documents,\n", ")"]}, {"cell_type": "code", "execution_count": null, "id": "fc8bb089-f84f-465f-a9df-d883df82d839", "metadata": {}, "outputs": [], "source": ["query_tool = QueryEngineTool(\n", " query_engine=vector_index.as_query_engine(),\n", " metadata=ToolMetadata(\n", " name=f\"vector_tool\",\n", " description=(\n", " \"用于查找OpenAI宣布的新功能\"\n", " # \"用于查找有关图像的任何信息\"\n", " ),\n", " ),\n", ")"]}, {"cell_type": "markdown", "id": "6aaa1543-5c43-43bd-8d22-fe2729a51b25", "metadata": {}, "source": ["### 设置代理\n"]}, {"cell_type": "code", "execution_count": null, "id": "f698d34d-6cb6-419b-b4f8-983f85ee2914", "metadata": {}, "outputs": [], "source": ["from llama_index.core.agent.react_multimodal.step import (\n", " MultimodalReActAgentWorker,\n", ")\n", "from llama_index.core.multi_modal_llms import MultiModalLLM\n", "from llama_index.multi_modal_llms.openai import OpenAIMultiModal\n", "from llama_index.core.agent import Task\n", "\n", "mm_llm = OpenAIMultiModal(model=\"gpt-4-vision-preview\", max_new_tokens=1000)\n", "\n", "# 选项2：使用OpenAIAgentWorker进行初始化\n", "react_step_engine = MultimodalReActAgentWorker.from_tools(\n", " [query_tool],\n", " # [],\n", " multi_modal_llm=mm_llm,\n", " verbose=True,\n", ")\n", "agent = react_step_engine.as_agent()"]}, {"cell_type": "code", "execution_count": null, "id": "afbe933d-fa7c-4d56-8c0f-faf1746bc09d", "metadata": {}, "outputs": [], "source": ["query_str = (\n", " \"照片显示了OpenAI发布的一些新功能。\"\n", " \"你能在照片中找出这些功能并使用相关工具提供更多细节吗？\"\n", ")\n", "\n", "from llama_index.core.schema import ImageDocument\n", "\n", "# 图像文档\n", "image_document = ImageDocument(image_path=\"other_images/openai/dev_day.png\")\n", "\n", "task = agent.create_task(\n", " query_str,\n", " extra_state={\"image_docs\": [image_document]},\n", ")"]}, {"cell_type": "code", "execution_count": null, "id": "18c30caf-2f3a-4202-8fde-b1c3099db726", "metadata": {}, "outputs": [], "source": ["from llama_index.core.agent import AgentRunner\n", "\n", "\n", "def execute_step(agent: AgentRunner, task: Task):\n", " step_output = agent.run_step(task.task_id)\n", " if step_output.is_last:\n", " response = agent.finalize_response(task.task_id)\n", " print(f\"> Agent finished: {str(response)}\")\n", " return response\n", " else:\n", " return None\n", "\n", "\n", "def execute_steps(agent: AgentRunner, task: Task):\n", " response = execute_step(agent, task)\n", " while response is None:\n", " response = execute_step(agent, task)\n", " return response"]}, {"cell_type": "code", "execution_count": null, "id": "a0900fb2-9e16-4d75-9aa4-23180decffa7", "metadata": {}, "outputs": [], "source": ["# 如果您只想一次性运行所有内容，请运行此代码，而不是下面的代码。\n", "# response = execute_steps(agent, task)"]}, {"cell_type": "code", "execution_count": null, "id": "934df138-6fb7-4adb-903d-c7e7a008b1e1", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["\u001b[1;3;38;5;200mThought: I need to use a tool to help me identify the new features released by OpenAI as shown in the photo.\n", "Action: vector_tool\n", "Action Input: {'input': 'new features released by OpenAI'}\n", "\u001b[0m\u001b[1;3;34mObservation: OpenAI has released several new features, including the GPT-4 Turbo model, the Assistants API, and multimodal capabilities. The GPT-4 Turbo model is more capable, cheaper, and supports a 128K context window. The Assistants API makes it easier for developers to build their own assistive AI apps with goals and the ability to call models and tools. The multimodal capabilities include vision, image creation (DALL·E 3), and text-to-speech (TTS). These new features are being rolled out to OpenAI customers starting at 1pm PT today.\n", "\u001b[0m"]}], "source": ["response = execute_step(agent, task)"]}, {"cell_type": "code", "execution_count": null, "id": "85783870-d51b-46cb-8588-3de1bdf1feb9", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["\u001b[1;3;38;5;200mThought: The observation provided information about the new features released by OpenAI, which I can now relate to the image provided.\n", "Response: The photo shows a user interface with a section titled \"Playground\" and several options such as \"GPT-4.0-turbo,\" \"Code Interpreter,\" \"Translate,\" and \"Chat.\" Based on the observation from the tool, these features are part of the new releases by OpenAI. Specifically, \"GPT-4.0-turbo\" likely refers to the GPT-4 Turbo model, which is a more capable and cost-effective version of the language model with a larger context window. The \"Code Interpreter\" could be related to the Assistants API, which allows developers to build AI apps that can interpret and execute code. The \"Translate\" and \"Chat\" options might be part of the multimodal capabilities, with \"Translate\" possibly involving text-to-text language translation and \"Chat\" involving conversational AI capabilities. The multimodal capabilities also include vision and image creation, which could be represented in the Playground interface but are not visible in the provided section of the photo.\n", "\u001b[0m> Agent finished: The photo shows a user interface with a section titled \"Playground\" and several options such as \"GPT-4.0-turbo,\" \"Code Interpreter,\" \"Translate,\" and \"Chat.\" Based on the observation from the tool, these features are part of the new releases by OpenAI. Specifically, \"GPT-4.0-turbo\" likely refers to the GPT-4 Turbo model, which is a more capable and cost-effective version of the language model with a larger context window. The \"Code Interpreter\" could be related to the Assistants API, which allows developers to build AI apps that can interpret and execute code. The \"Translate\" and \"Chat\" options might be part of the multimodal capabilities, with \"Translate\" possibly involving text-to-text language translation and \"Chat\" involving conversational AI capabilities. The multimodal capabilities also include vision and image creation, which could be represented in the Playground interface but are not visible in the provided section of the photo.\n"]}], "source": ["response = execute_step(agent, task)"]}, {"cell_type": "code", "execution_count": null, "id": "c45113bd-73aa-44f5-b803-652231fcb92a", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["The photo shows a user interface with a section titled \"Playground\" and several options such as \"GPT-4.0-turbo,\" \"Code Interpreter,\" \"Translate,\" and \"Chat.\" Based on the observation from the tool, these features are part of the new releases by OpenAI. Specifically, \"GPT-4.0-turbo\" likely refers to the GPT-4 Turbo model, which is a more capable and cost-effective version of the language model with a larger context window. The \"Code Interpreter\" could be related to the Assistants API, which allows developers to build AI apps that can interpret and execute code. The \"Translate\" and \"Chat\" options might be part of the multimodal capabilities, with \"Translate\" possibly involving text-to-text language translation and \"Chat\" involving conversational AI capabilities. The multimodal capabilities also include vision and image creation, which could be represented in the Playground interface but are not visible in the provided section of the photo.\n"]}], "source": ["print(str(response))"]}, {"cell_type": "markdown", "id": "f60e6265-0b30-477f-9b09-d5268c76f040", "metadata": {}, "source": ["## 通过网络搜索增强图像分析\n", "\n", "在这个示例中，我们将向您展示如何设置一个由GPT-4V驱动的代理，以在网络上查找信息，帮助更好地解释给定的图像。\n"]}, {"cell_type": "code", "execution_count": null, "id": "2c3631cc-5a6c-4bb7-a972-34467e96c399", "metadata": {}, "outputs": [], "source": ["from llama_index.tools.metaphor import MetaphorToolSpec\n", "from llama_index.core.agent.react_multimodal.step import (\n", " MultimodalReActAgentWorker,\n", ")\n", "from llama_index.core.agent import AgentRunner\n", "from llama_index.core.multi_modal_llms import MultiModalLLM\n", "from llama_index.multi_modal_llms.openai import OpenAIMultiModal\n", "from llama_index.core.agent import Task\n", "\n", "metaphor_tool_spec = MetaphorToolSpec(\n", " api_key=\"\",\n", ")\n", "metaphor_tools = metaphor_tool_spec.to_tool_list()"]}, {"cell_type": "code", "execution_count": null, "id": "99c37097-cd5c-4819-b42e-f19cc4f268be", "metadata": {}, "outputs": [], "source": ["mm_llm = OpenAIMultiModal(model=\"gpt-4-vision-preview\", max_new_tokens=1000)\n", "\n", "# 选项2：使用OpenAIAgentWorker进行初始化\n", "react_step_engine = MultimodalReActAgentWorker.from_tools(\n", " metaphor_tools,\n", " # [],\n", " multi_modal_llm=mm_llm,\n", " verbose=True,\n", ")\n", "agent = react_step_engine.as_agent()"]}, {"cell_type": "code", "execution_count": null, "id": "9a692881-f70e-4815-a89a-3b584ec1b844", "metadata": {}, "outputs": [], "source": ["from llama_index.core.schema import ImageDocument\n", "\n", "query_str = \"Look up some reviews regarding these shoes.\"\n", "image_document = ImageDocument(image_path=\"other_images/adidas.png\")\n", "\n", "task = agent.create_task(\n", " query_str, extra_state={\"image_docs\": [image_document]}\n", ")"]}, {"cell_type": "code", "execution_count": null, "id": "6621c3aa-22c2-46e9-b405-9c99d3b4d657", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["\u001b[1;3;38;5;200mThought: The image shows a pair of shoes from a website that appears to be selling them. The user is asking for reviews of these shoes, but the image does not provide specific details such as the brand or model name. I will need to use a search tool to find reviews based on the visual information provided.\n", "Action: search\n", "Action Input: {'query': 'reviews for yellow and white running shoes with black stripes'}\n", "\u001b[0m[Metaphor Tool] Autoprompt: Here is a review for a great pair of yellow and white running shoes with black stripes:\n", "\u001b[1;3;34mObservation: [{'title': '| On', 'url': 'https://www.on-running.com/en-us/', 'id': 'bO8WCIY4qIAlfi5MbHMw7A'}, {'title': 'ASICS Gel Nimbus 21 Review 2023, Facts, Deals ($81)', 'url': 'https://runrepeat.com/asics-gel-nimbus-21', 'id': 'l-1YebY9dIRt1d8MeHyaBg'}, {'title': 'ASICS UK | Official Running Shoes & Clothing | ASICS', 'url': 'https://www.asics.com/gb/en-gb/', 'id': 'aAY5Tpax5jevadvNMj34_w'}, {'title': 'Asics Gel Nimbus', 'url': 'https://www.thepeacefulrunner.com/asics-gel-nimbus.html', 'id': 'l8joVUIb-c6H5BTKqu7fJw'}, {'title': 'hoka clifton', 'url': 'https://www.zappos.com/hoka-clifton?PID=3428536&AID=11554337&splash=none&cjevent=65a454d2018811ed82ca7c440a82b82a&zap_placement=', 'id': 'iPqidPvLy-wt02B9GdgG8g'}, {'title': '| On United States', 'url': 'https://www.on-running.com/en-us/shop?utm_source=facebook&utm_medium=post&utm_campaign=cloud_x_w45&utm_content=launch', 'id': 'pUaSVBqAbU-VNvj0KEkNlw'}, {'title': 'Shoe Review: Skechers GORun Speed Elite Hyper', 'url': 'https://athleticsillustrated.com/shoe-review-skechers-gorun-speed-elite-hyper/', 'id': 'B3ijN8qSaV1eBkoOCeV9yw'}, {'title': 'Saucony Running Shoes & Running Apparel | Saucony.com', 'url': 'https://www.saucony.com/en/country', 'id': 'ng-HmN8CxK2TAKmA9gPLTg'}, {'title': 'The entire Kalenji range and training plans on Kalenji.co.uk - Shoes, clothing and accessories for running and trail running.', 'url': 'https://www.kalenji.co.uk/', 'id': 'SHrJYCGb5g9yoehuu8zOzg'}, {'title': 'ONEMIX Men Running Shoes Trainers Comfortable Damping Outdoor Athletic Vulcanized Tennis Shoes Trail Sneakers', 'url': 'https://usa.banggood.com/ONEMIX-Men-Running-Shoes-Trainers-Comfortable-Damping-Outdoor-Athletic-Vulcanized-Tennis-Shoes-Trail-Sneakers-p-1745312.html?imageAb=2&p=H320105094792201606N&akmClientCountry=America&a=1675672269.4621&akmClientCountry=America', 'id': 'Dh81GLQU2gi4Efc2z8ZzYA'}]\n", "\u001b[0m"]}], "source": ["response = execute_step(agent, task)"]}, {"cell_type": "code", "execution_count": null, "id": "2e77161b-50d6-4fb9-934e-336a0eaf1e30", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["\u001b[1;3;38;5;200mThought: The search results returned a variety of shoe brands and models, but none of them seem to match the specific shoes in the image. To find reviews for the exact shoes, I need more information about the brand or model. Since the image shows a website interface with the brand likely being \"UltraBOOST 1.0 DNA\" and a discount code \"CYBER,\" I can use this information to refine my search for reviews.\n", "Action: search\n", "Action Input: {'query': 'UltraBOOST 1.0 DNA shoes reviews'}\n", "\u001b[0m[Metaphor Tool] Autoprompt: Here is a review of the UltraBOOST 1.0 DNA shoes:\n", "\u001b[1;3;34mObservation: [{'title': 'Shoe', 'url': 'https://therunningclinic.com/shoe/?id=81645', 'id': 'SR5Ar004nuT97AkWDTdekg'}, {'title': 'Shoe', 'url': 'https://therunningclinic.com/shoe/?id=81644', 'id': 'kiEvTAolb2Kh4OrvSxnqUA'}, {'title': 'On Cloudboom Review', 'url': 'https://www.doctorsofrunning.com/2020/12/on-cloudboom-review.html', 'id': 'J3ARnLPH1KWzrIdGogwxnw'}, {'title': 'Adidas UltraBOOST 5.0 DNA Glow now available', 'url': 'https://dlmag.com/adidas-ultraboost-5-0-dna-glow-now-available', 'id': 'wljZXtnx28OF5YtEXcbwXg'}, {'title': 'Ultraboost 19 Review in 2020 (comparison)', 'url': 'https://www.youtube.com/watch?v=n9_23gq7ER0&t=6s', 'id': 'B3QWxLBJgA9FR6HRmOnW2A'}, {'title': 'Adidas ULTRABOOST LIGHT : Detailed First Impressions', 'url': 'https://www.youtube.com/watch?v=780bVc3yjsk', 'id': 'LOUPO-1rU2tZJd31YFYYXA'}, {'title': 'Adidas Ultraboost Review : 7 pros, 2 cons (2023)', 'url': 'https://runrepeat.com/adidas-ultra-boost', 'id': 'Xqa5dR7IR24En7uL5BCTEg'}, {'title': 'VLOG # 999 | The MOST *Challenging* Running SHOE of 2018', 'url': 'https://www.youtube.com/watch?v=3KWIWLtyrSw', 'id': 'avKLkDhnPT3eTwAvI6Wi7g'}, {'title': 'NEW ARRIVALS', 'url': 'http://www.adidasultraboost.us.org/', 'id': 'NhKaTjMI_rIIS0tWLrJXAA'}, {'title': 'Brooks Hyperion Tempo Review (2020: Better Late Than Never)', 'url': 'https://www.doctorsofrunning.com/2023/01/brooks-hyperion-tempo-review-2020.html', 'id': '3U7runLSyYNIBlRHETTWCA'}]\n", "\u001b[0m"]}], "source": ["response = execute_step(agent, task)"]}, {"cell_type": "code", "execution_count": null, "id": "211cf323-790c-4ee1-a3b3-7abdf573b9b5", "metadata": {}, "outputs": [], "source": ["# 执行步骤，获取agent的响应\n", "response = execute_step(agent, task)"]}, {"cell_type": "code", "execution_count": null, "id": "13bdea5c-30a6-4cb8-be79-39cd4e69e299", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["\u001b[1;3;38;5;200mThought: I can answer without using any more tools.\n", "Response: The Adidas Ultraboost is reviewed as an expensive but versatile shoe suitable for various activities, from running a half marathon to going out on a date. They are considered a good value due to their durability, with the reviewer getting hundreds of miles out of them. The shoes are described as lightweight, breathable, and comfortable enough to wear without socks. However, they are not recommended for wet climates as they do not perform well in the rain. The reviewer also mentions owning seven different models of Adidas Boost, indicating a strong preference for the brand.\n", "\u001b[0m> Agent finished: The Adidas Ultraboost is reviewed as an expensive but versatile shoe suitable for various activities, from running a half marathon to going out on a date. They are considered a good value due to their durability, with the reviewer getting hundreds of miles out of them. The shoes are described as lightweight, breathable, and comfortable enough to wear without socks. However, they are not recommended for wet climates as they do not perform well in the rain. The reviewer also mentions owning seven different models of Adidas Boost, indicating a strong preference for the brand.\n"]}], "source": ["response = execute_step(agent, task)"]}, {"cell_type": "code", "execution_count": null, "id": "54e1b6d7-f940-4806-8418-ab86074dda17", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["The Adidas Ultraboost is reviewed as an expensive but versatile shoe suitable for various activities, from running a half marathon to going out on a date. They are considered a good value due to their durability, with the reviewer getting hundreds of miles out of them. The shoes are described as lightweight, breathable, and comfortable enough to wear without socks. However, they are not recommended for wet climates as they do not perform well in the rain. The reviewer also mentions owning seven different models of Adidas Boost, indicating a strong preference for the brand.\n"]}], "source": ["print(str(response))"]}], "metadata": {"kernelspec": {"display_name": "venv", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3"}}, "nbformat": 4, "nbformat_minor": 5}