{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Chat with PDF - test, evaluation and experimentation" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "\n", "We will walk you through how to use prompt flow Python SDK to test, evaluate and experiment with the \"Chat with PDF\" flow.\n", "\n", "## 0. Install dependencies" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install -r requirements.txt" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Create connections\n", "Connection in prompt flow is for managing settings of your application behaviors incl. how to talk to different services (Azure OpenAI for example)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import promptflow\n", "\n", "pf = promptflow.PFClient()\n", "\n", "# List all the available connections\n", "for c in pf.connections.list():\n", " print(c.name + \" (\" + c.type + \")\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You will need to have a connection named \"open_ai_connection\" to run the chat_with_pdf flow." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# create needed connection\n", "from promptflow.entities import AzureOpenAIConnection, OpenAIConnection\n", "\n", "try:\n", " conn_name = \"open_ai_connection\"\n", " conn = pf.connections.get(name=conn_name)\n", " print(\"using existing connection\")\n", "except:\n", " # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n", " connection = AzureOpenAIConnection(\n", " name=conn_name,\n", " api_key=\"\",\n", " api_base=\"\",\n", " api_type=\"azure\",\n", " api_version=\"\",\n", " )\n", "\n", " # use this if you have an existing OpenAI account\n", " # connection = OpenAIConnection(\n", " # name=conn_name,\n", " # api_key=\"\",\n", " # )\n", " conn = pf.connections.create_or_update(connection)\n", " print(\"successfully created connection\")\n", "\n", "print(conn)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Test the flow\n", "\n", "**Note**: this sample uses `predownloaded PDFs` and `prebuilt FAISS Index` to speed up execution time.\n", "You can remove the folders to start a fresh run." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ./chat_with_pdf/.pdfs/ stores predownloaded PDFs\n", "# ./chat_with_pdf/.index/ stores prebuilt index files\n", "\n", "output = pf.flows.test(\n", " \".\",\n", " inputs={\n", " \"chat_history\": [],\n", " \"pdf_url\": \"https://arxiv.org/pdf/1810.04805.pdf\",\n", " \"question\": \"what is BERT?\",\n", " },\n", ")\n", "print(output)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Run the flow with a data file" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "flow_path = \".\"\n", "data_path = \"./data/bert-paper-qna-3-line.jsonl\"\n", "\n", "config_2k_context = {\n", " \"EMBEDDING_MODEL_DEPLOYMENT_NAME\": \"text-embedding-ada-002\",\n", " \"CHAT_MODEL_DEPLOYMENT_NAME\": \"gpt-4\", # change this to the name of your deployment if you're using Azure OpenAI\n", " \"PROMPT_TOKEN_LIMIT\": 2000,\n", " \"MAX_COMPLETION_TOKENS\": 256,\n", " \"VERBOSE\": True,\n", " \"CHUNK_SIZE\": 1024,\n", " \"CHUNK_OVERLAP\": 64,\n", "}\n", "\n", "column_mapping = {\n", " \"question\": \"${data.question}\",\n", " \"pdf_url\": \"${data.pdf_url}\",\n", " \"chat_history\": \"${data.chat_history}\",\n", " \"config\": config_2k_context,\n", "}\n", "run_2k_context = pf.run(flow=flow_path, data=data_path, column_mapping=column_mapping)\n", "pf.stream(run_2k_context)\n", "\n", "print(run_2k_context)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pf.get_details(run_2k_context)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Evaluate the \"groundedness\"\n", "The `eval-groundedness flow` is using ChatGPT/GPT4 model to grade the answers generated by chat-with-pdf flow." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "eval_groundedness_flow_path = \"../../evaluation/eval-groundedness/\"\n", "eval_groundedness_2k_context = pf.run(\n", " flow=eval_groundedness_flow_path,\n", " run=run_2k_context,\n", " column_mapping={\n", " \"question\": \"${run.inputs.question}\",\n", " \"answer\": \"${run.outputs.answer}\",\n", " \"context\": \"${run.outputs.context}\",\n", " },\n", " display_name=\"eval_groundedness_2k_context\",\n", ")\n", "pf.stream(eval_groundedness_2k_context)\n", "\n", "print(eval_groundedness_2k_context)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pf.get_details(eval_groundedness_2k_context)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pf.get_metrics(eval_groundedness_2k_context)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pf.visualize(eval_groundedness_2k_context)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You will see a web page like this. It gives you detail about how each row is graded and even the details how the evaluation run executes:\n", "![pf-visualize-screenshot](./media/chat-with-pdf/pf-visualize-screenshot.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Try a different configuration and evaluate again - experimentation\n", "\n", "NOTE: since we only use 3 lines of test data in this example, and because of the non-deterministic nature of LLMs, don't be surprised if you see exact same metrics when you run this process." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "config_3k_context = {\n", " \"EMBEDDING_MODEL_DEPLOYMENT_NAME\": \"text-embedding-ada-002\",\n", " \"CHAT_MODEL_DEPLOYMENT_NAME\": \"gpt-4\", # change this to the name of your deployment if you're using Azure OpenAI\n", " \"PROMPT_TOKEN_LIMIT\": 3000,\n", " \"MAX_COMPLETION_TOKENS\": 256,\n", " \"VERBOSE\": True,\n", " \"CHUNK_SIZE\": 1024,\n", " \"CHUNK_OVERLAP\": 64,\n", "}\n", "\n", "run_3k_context = pf.run(flow=flow_path, data=data_path, column_mapping=column_mapping)\n", "pf.stream(run_3k_context)\n", "\n", "print(run_3k_context)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "eval_groundedness_3k_context = pf.run(\n", " flow=eval_groundedness_flow_path,\n", " run=run_3k_context,\n", " column_mapping={\n", " \"question\": \"${run.inputs.question}\",\n", " \"answer\": \"${run.outputs.answer}\",\n", " \"context\": \"${run.outputs.context}\",\n", " },\n", " display_name=\"eval_groundedness_3k_context\",\n", ")\n", "pf.stream(eval_groundedness_3k_context)\n", "\n", "print(eval_groundedness_3k_context)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pf.get_details(eval_groundedness_3k_context)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pf.visualize([eval_groundedness_2k_context, eval_groundedness_3k_context])" ] } ], "metadata": { "build_doc": { "author": [ "wangchao1230@github.com", "ttthree@github.com" ], "category": "local", "section": "Rag", "weight": 10 }, "description": "A tutorial of chat-with-pdf flow that allows user ask questions about the content of a PDF file and get answers", "kernelspec": { "display_name": "prompt-flow", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.19" } }, "nbformat": 4, "nbformat_minor": 2 }