{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d5dd7679-5ed6-4243-8e0f-549f8118bff7",
   "metadata": {},
   "source": [
    "# How to test and evaluate LLMs for SQL generation\n",
    "\n",
    "LLMs are fundamentally non-deterministic in their responses, this attribute makes them wonderfully creative and dynamic in their responses. However, this trait poses significant challenges in achieving consistency, a crucial aspect for integrating LLMs into production environments.\n",
    "\n",
    "The key to harnessing the potential of LLMs in practical applications lies in consistent and systematic evaluation. This enables the identification and rectification of inconsistencies and helps with monitoring progress over time as the application evolves.\n",
    "\n",
    "## Scope of this notebook\n",
    "\n",
    "This notebook aims to demonstrate a framework for evaluating LLMs, particularly focusing on:\n",
    "\n",
    "* **Unit Testing:** Essential for assessing individual components of the application.\n",
    "* **Evaluation Metrics:** Methods to quantitatively measure the model's effectiveness.\n",
    "* **Runbook Documentation:** A record of historical evaluations to track progress and regression.\n",
    "\n",
    "This example focuses on a natural language to SQL use case - code generation use cases fit well with this approach when you combine **code validation** with **code execution**, so your application can test code for real as it is generated to ensure consistency.\n",
    "\n",
    "Although this notebook uses SQL generation usecase to demonstrate the concept, the approach is generic and can be applied to a wide variety of LLM driven applications.\n",
    "\n",
    "We will use two versions of a prompt to perform SQL generation.  We will then use the unit tests and evaluation functions to test the perforamance of the prompts.  Specifically, in this demonstration, we will evaluate:\n",
    "\n",
    "1. The consistency of JSON response.\n",
    "2. Syntactic correctness of SQL in response.\n",
    "\n",
    "\n",
    "## Table of contents\n",
    "\n",
    "1. **[Setup](#Setup):** Install required libraries, download data consisting of SQL queries and corresponding natural language translations.\n",
    "2. **[Test Development](#Test-development):** Create unit tests and define evaluation metrics for the SQL generation process.\n",
    "3. **[Evaluation](#Evaluation):** Conduct tests using different prompts to assess the impact on performance.\n",
    "4. **[Reporting](#Report):** Compile a report that succinctly presents the performance differences observed across various tests."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2913d615",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Import our libraries and the dataset we'll use, which is the natural language to SQL [b-mc2/sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context) dataset from HuggingFace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c7f325fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment this to install all necessary dependencies\n",
    "# !pip install openai datasets pandas pydantic matplotlib python-dotenv numpy tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "245fcedb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "78577 rows\n"
     ]
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "from openai import OpenAI\n",
    "import pandas as pd\n",
    "import pydantic\n",
    "import os\n",
    "import sqlite3\n",
    "from sqlite3 import Error\n",
    "from pprint import pprint\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from dotenv import load_dotenv\n",
    "from tqdm.notebook import tqdm\n",
    "from IPython.display import HTML, display\n",
    "\n",
    "# Loads key from local .env file to setup API KEY in env variables\n",
    "%reload_ext dotenv\n",
    "%dotenv\n",
    "    \n",
    "GPT_MODEL = 'gpt-4o'\n",
    "dataset = load_dataset(\"b-mc2/sql-create-context\")\n",
    "\n",
    "print(dataset['train'].num_rows, \"rows\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04c7fde6-d7dc-4a0d-b9a0-32858f3bac25",
   "metadata": {},
   "source": [
    "### Looking at the dataset\n",
    "\n",
    "We use Huggingface datasets library to download SQL create context dataset.  This dataset consists of:\n",
    "\n",
    "1. Question, expressed in natural language\n",
    "2. Answer, expressed in SQL designed to answer the question in natural language.\n",
    "3. Context, expressed as a CREATE SQL statement, that describes the table that may be used to answer the question.\n",
    "\n",
    "In our demonstration today, we will use LLM to attempt to answer the question (in natural language).  The LLM will be expected to generate a CREATE SQL statement to create a context suitable to answer the user question and a coresponding SELECT SQL query designed to answer the user question completely.\n",
    "\n",
    "The dataset looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f8027115",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>question</th>\n",
       "      <th>context</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>SELECT COUNT(*) FROM head WHERE age &gt; 56</td>\n",
       "      <td>How many heads of the departments are older th...</td>\n",
       "      <td>CREATE TABLE head (age INTEGER)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>SELECT name, born_state, age FROM head ORDER B...</td>\n",
       "      <td>List the name, born state and age of the heads...</td>\n",
       "      <td>CREATE TABLE head (name VARCHAR, born_state VA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>SELECT creation, name, budget_in_billions FROM...</td>\n",
       "      <td>List the creation year, name and budget of eac...</td>\n",
       "      <td>CREATE TABLE department (creation VARCHAR, nam...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>SELECT MAX(budget_in_billions), MIN(budget_in_...</td>\n",
       "      <td>What are the maximum and minimum budget of the...</td>\n",
       "      <td>CREATE TABLE department (budget_in_billions IN...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SELECT AVG(num_employees) FROM department WHER...</td>\n",
       "      <td>What is the average number of employees of the...</td>\n",
       "      <td>CREATE TABLE department (num_employees INTEGER...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              answer  \\\n",
       "0           SELECT COUNT(*) FROM head WHERE age > 56   \n",
       "1  SELECT name, born_state, age FROM head ORDER B...   \n",
       "2  SELECT creation, name, budget_in_billions FROM...   \n",
       "3  SELECT MAX(budget_in_billions), MIN(budget_in_...   \n",
       "4  SELECT AVG(num_employees) FROM department WHER...   \n",
       "\n",
       "                                            question  \\\n",
       "0  How many heads of the departments are older th...   \n",
       "1  List the name, born state and age of the heads...   \n",
       "2  List the creation year, name and budget of eac...   \n",
       "3  What are the maximum and minimum budget of the...   \n",
       "4  What is the average number of employees of the...   \n",
       "\n",
       "                                             context  \n",
       "0                    CREATE TABLE head (age INTEGER)  \n",
       "1  CREATE TABLE head (name VARCHAR, born_state VA...  \n",
       "2  CREATE TABLE department (creation VARCHAR, nam...  \n",
       "3  CREATE TABLE department (budget_in_billions IN...  \n",
       "4  CREATE TABLE department (num_employees INTEGER...  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sql_df = dataset['train'].to_pandas()\n",
    "sql_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b04cb5eb",
   "metadata": {},
   "source": [
    "## Test development\n",
    "\n",
    "To test the output of the LLM generations, we'll develop two unit tests and an evaluation, which will combine to give us a basic evaluation framework to grade the quality of our LLM iterations.\n",
    "\n",
    "To re-iterate, our purpose is to measure the correctness and consistency of LLM output given our questions.\n",
    "\n",
    "### Unit tests\n",
    "\n",
    "Unit tests should test the most granular components of your LLM application.\n",
    "\n",
    "For this section we'll develop unit tests to test the following:\n",
    "- `test_valid_schema` will check that a parseable `create` and `select` statement are returned by the LLM.\n",
    "- `test_llm_sql` will execute both the `create` and `select` statements on a `sqlite` database to ensure they are syntactically correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "eb811101",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydantic import BaseModel\n",
    "\n",
    "\n",
    "class LLMResponse(BaseModel):\n",
    "    \"\"\"This is the structure that we expect the LLM to respond with.\n",
    "\n",
    "    The LLM should respond with a JSON string with `create` and `select` fields.\n",
    "    \"\"\"\n",
    "    create: str\n",
    "    select: str"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19fadf67-8b2f-4e17-95df-030a36aad90b",
   "metadata": {},
   "source": [
    "#### Prompting the LLM\n",
    "\n",
    "For this demonstration purposes, we use a fairly simple prompt requesting GPT to generate a `(context, answer)` pair. `context` is the `CREATE` SQL statement, and `answer` is the `SELECT` SQL statement. We supply the natural language question as part of the prompt.  We request the response to be in JSON format, so that it can be parsed easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c2be3ba4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: How many heads of the departments are older than 56 ?\n",
      "Answer: {\"create\":\"CREATE TABLE DepartmentHeads (\\n    id INT PRIMARY KEY,\\n    name VARCHAR(100),\\n    age INT,\\n    department VARCHAR(100)\\n);\",\"select\":\"SELECT COUNT(*) AS NumberOfHeadsOlderThan56 \\nFROM DepartmentHeads \\nWHERE age > 56;\"}\n"
     ]
    }
   ],
   "source": [
    "system_prompt = \"\"\"Translate this natural language request into a JSON\n",
    "object containing two SQL queries. The first query should be a CREATE \n",
    "tatement for a table answering the user's request, while the second\n",
    "should be a SELECT query answering their question.\"\"\"\n",
    "\n",
    "# Sending the message array to GPT, requesting a response (ensure that you\n",
    "# have API key loaded to Env for this step)\n",
    "client = OpenAI()\n",
    "\n",
    "def get_response(system_prompt, user_message, model=GPT_MODEL):\n",
    "    messages = []\n",
    "    messages.append({\"role\": \"system\", \"content\": system_prompt})\n",
    "    messages.append({\"role\": \"user\", \"content\": user_message})\n",
    "\n",
    "    response = client.beta.chat.completions.parse(\n",
    "        model=GPT_MODEL,\n",
    "        messages=messages,\n",
    "        response_format=LLMResponse,\n",
    "    )\n",
    "    return response.choices[0].message.content\n",
    "\n",
    "question = sql_df.iloc[0]['question']\n",
    "content = get_response(system_prompt, question)\n",
    "print(\"Question:\", question)\n",
    "print(\"Answer:\", content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "901e3bb7",
   "metadata": {},
   "source": [
    "#### Check JSON formatting\n",
    "\n",
    "Our first simple unit test checks that the LLM response is parseable into the `LLMResponse` Pydantic class that we've defined.\n",
    "\n",
    "We'll test that our first response passes, then create a failing example to check that the check fails. This logic will be wrapped in a simple function `test_valid_schema`.\n",
    "\n",
    "We expect GPT to respond with a valid SQL, we can validate this using LLMResponse base model.  `test_valid_schema` is designed to help us validate this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4c7133f1-74d6-43f1-9443-09a3f8308c35",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_valid_schema(content):\n",
    "    \"\"\"Tests whether the content provided can be parsed into our Pydantic model.\"\"\"\n",
    "    try:\n",
    "        LLMResponse.model_validate_json(content)\n",
    "        return True\n",
    "    # Catch pydantic's validation errors:\n",
    "    except pydantic.ValidationError as exc:\n",
    "        print(f\"ERROR: Invalid schema: {exc}\")\n",
    "        return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6a9a9128",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_valid_schema(content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78f1af23-4dd0-4860-8a1a-88e5146be6ed",
   "metadata": {},
   "source": [
    "#### Testing negative scenario\n",
    "\n",
    "To simulate a scenario in which we get an invalid JSON response from GPT, we hardcode an invalid JSON as response.  We expect `test_valid_schema` function to throw an exception."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a0a26690",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ERROR: Invalid schema: 1 validation error for LLMResponse\n",
      "  Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value='CREATE departments, select * from departments', input_type=str]\n",
      "    For further information visit https://errors.pydantic.dev/2.10/v/json_invalid\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "failing_query = 'CREATE departments, select * from departments'\n",
    "test_valid_schema(failing_query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5fdc420-94cc-4e47-80e1-82bf51e44f2a",
   "metadata": {},
   "source": [
    "As expected, we get an exception thrown from the `test_valid_schema` fucntion."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4e972cd-5734-43c0-a103-b9ceb41552fd",
   "metadata": {},
   "source": [
    "### Test SQL queries\n",
    "\n",
    "Next we'll validate the correctness of the SQL.  This test will be desined to validate:\n",
    "\n",
    "1. The CREATE SQL returned in GPT response is syntactically correct.\n",
    "2. The SELECT SQL returned in the GPT response is syntactically correct.\n",
    "\n",
    "To achieve this, we will use a sqlite instance. We will direct the retured SQL functions to a sqlite instance.  If the SQL statements are valid, sqlite instance will accept and execute the statements; otherwise we will expect an exception to be thrown.\n",
    "\n",
    "`create_connection` function below will setup a sqlite instance (in-memory by default) and create a connection to be used later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9cc95481",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up SQLite to act as our test database\n",
    "def create_connection(db_file=\":memory:\"):\n",
    "    \"\"\"create a database connection to a SQLite database\"\"\"\n",
    "    try:\n",
    "        conn = sqlite3.connect(db_file)\n",
    "        # print(sqlite3.version)\n",
    "    except Error as e:\n",
    "        print(e)\n",
    "        return None\n",
    "\n",
    "    return conn\n",
    "\n",
    "def close_connection(conn):\n",
    "    \"\"\"close a database connection\"\"\"\n",
    "    try:\n",
    "        conn.close()\n",
    "    except Error as e:\n",
    "        print(e)\n",
    "\n",
    "\n",
    "conn = create_connection()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa5c5cb8-1c81-403b-a3f2-f2784d8235fc",
   "metadata": {},
   "source": [
    "Next, we will create the following functions to carry out the syntactical correctness checks.\n",
    "\n",
    "\n",
    "- `test_create`: Function testing if the CREATE SQL statement succeeds.\n",
    "- `test_select`: Function testing if the SELECT SQL statement succeeds.\n",
    "- `test_llm_sql`: Wrapper function executing the two tests above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c6d2573d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_select(conn, cursor, select, should_log=True):\n",
    "    \"\"\"Tests that a SQLite select query can be executed successfully.\"\"\"\n",
    "    try:\n",
    "        if should_log:\n",
    "            print(f\"Testing select query: {select}\")\n",
    "        cursor.execute(select)\n",
    "        record = cursor.fetchall()\n",
    "        if should_log:\n",
    "            print(f\"Result of query: {record}\")\n",
    "\n",
    "        return True\n",
    "\n",
    "    except sqlite3.Error as error:\n",
    "        if should_log:\n",
    "            print(\"Error while executing select query:\", error)\n",
    "        return False\n",
    "\n",
    "\n",
    "def test_create(conn, cursor, create, should_log=True):\n",
    "    \"\"\"Tests that a SQLite create query can be executed successfully\"\"\"\n",
    "    try:\n",
    "        if should_log:\n",
    "            print(f\"Testing create query: {create}\")\n",
    "        cursor.execute(create)\n",
    "        conn.commit()\n",
    "\n",
    "        return True\n",
    "\n",
    "    except sqlite3.Error as error:\n",
    "        if should_log:\n",
    "            print(\"Error while creating the SQLite table:\", error)\n",
    "        return False\n",
    "\n",
    "\n",
    "def test_llm_sql(llm_response, should_log=True):\n",
    "    \"\"\"Runs a suite of SQLite tests\"\"\"\n",
    "    try:\n",
    "        conn = create_connection()\n",
    "        cursor = conn.cursor()\n",
    "\n",
    "        create_response = test_create(conn, cursor, llm_response.create, should_log=should_log)\n",
    "\n",
    "        select_response = test_select(conn, cursor, llm_response.select, should_log=should_log)\n",
    "\n",
    "        if conn:\n",
    "            close_connection(conn)\n",
    "\n",
    "        if create_response is not True:\n",
    "            return False\n",
    "\n",
    "        elif select_response is not True:\n",
    "            return False\n",
    "\n",
    "        else:\n",
    "            return True\n",
    "\n",
    "    except sqlite3.Error as error:\n",
    "        if should_log:\n",
    "            print(\"Error while creating a sqlite table\", error)\n",
    "        return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a9266753-4646-4901-bc14-632d3bf47aaa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CREATE SQL is: CREATE TABLE DepartmentHeads (\n",
      "    id INT PRIMARY KEY,\n",
      "    name VARCHAR(100),\n",
      "    age INT,\n",
      "    department VARCHAR(100)\n",
      ");\n",
      "SELECT SQL is: SELECT COUNT(*) AS NumberOfHeadsOlderThan56 \n",
      "FROM DepartmentHeads \n",
      "WHERE age > 56;\n"
     ]
    }
   ],
   "source": [
    "# Viewing CREATE and SELECT sqls returned by GPT\n",
    "\n",
    "test_query = LLMResponse.model_validate_json(content)\n",
    "print(f\"CREATE SQL is: {test_query.create}\")\n",
    "print(f\"SELECT SQL is: {test_query.select}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "83bc1f1b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing create query: CREATE TABLE DepartmentHeads (\n",
      "    id INT PRIMARY KEY,\n",
      "    name VARCHAR(100),\n",
      "    age INT,\n",
      "    department VARCHAR(100)\n",
      ");\n",
      "Testing select query: SELECT COUNT(*) AS NumberOfHeadsOlderThan56 \n",
      "FROM DepartmentHeads \n",
      "WHERE age > 56;\n",
      "Result of query: [(0,)]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Testing the CREATE and SELECT sqls are valid (we expect this to be succesful)\n",
    "\n",
    "test_llm_sql(test_query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "589c7cc7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing create query: CREATE TABLE departments (id INT, name VARCHAR(255), head_of_department VARCHAR(255))\n",
      "Testing select query: SELECT COUNT(*) FROM departments WHERE age > 56\n",
      "Error while executing select query: no such column: age\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Again we'll perform a negative test to confirm that a failing SELECT will return an error.\n",
    "\n",
    "test_failure_query = '{\"create\": \"CREATE TABLE departments (id INT, name VARCHAR(255), head_of_department VARCHAR(255))\", \"select\": \"SELECT COUNT(*) FROM departments WHERE age > 56\"}'\n",
    "test_failure_query = LLMResponse.model_validate_json(test_failure_query)\n",
    "test_llm_sql(test_failure_query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8148f820",
   "metadata": {},
   "source": [
    "### Using an LLM to evaluate relevancy\n",
    "\n",
    "Next, we **evaluate** whether the generated SQL actually answers the user's question. This test will be performed by `gpt-4o-mini`, and will assess how **relevant** the produced SQL query is when compared to the initial user request.\n",
    "\n",
    "This is a simple example which adapts an approach outlined in the [G-Eval paper](https://arxiv.org/abs/2303.16634), and tested in one of our other [cookbooks](https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "029c8426",
   "metadata": {},
   "outputs": [],
   "source": [
    "EVALUATION_MODEL = \"gpt-4o-mini\"\n",
    "\n",
    "EVALUATION_PROMPT_TEMPLATE = \"\"\"\n",
    "You will be given one summary written for an article. Your task is to rate the summary on one metric.\n",
    "Please make sure you read and understand these instructions very carefully. \n",
    "Please keep this document open while reviewing, and refer to it as needed.\n",
    "\n",
    "Evaluation Criteria:\n",
    "\n",
    "{criteria}\n",
    "\n",
    "Evaluation Steps:\n",
    "\n",
    "{steps}\n",
    "\n",
    "Example:\n",
    "\n",
    "Request:\n",
    "\n",
    "{request}\n",
    "\n",
    "Queries:\n",
    "\n",
    "{queries}\n",
    "\n",
    "Evaluation Form (scores ONLY):\n",
    "\n",
    "- {metric_name}\n",
    "\"\"\"\n",
    "\n",
    "# Relevance\n",
    "\n",
    "RELEVANCY_SCORE_CRITERIA = \"\"\"\n",
    "Relevance(1-5) - review of how relevant the produced SQL queries are to the original question. \\\n",
    "The queries should contain all points highlighted in the user's request. \\\n",
    "Annotators were instructed to penalize queries which contained redundancies and excess information.\n",
    "\"\"\"\n",
    "\n",
    "RELEVANCY_SCORE_STEPS = \"\"\"\n",
    "1. Read the request and the queries carefully.\n",
    "2. Compare the queries to the request document and identify the main points of the request.\n",
    "3. Assess how well the queries cover the main points of the request, and how much irrelevant or redundant information it contains.\n",
    "4. Assign a relevance score from 1 to 5.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "85cfb78d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_geval_score(\n",
    "    criteria: str, steps: str, request: str, queries: str, metric_name: str\n",
    "):\n",
    "    \"\"\"Given evaluation criteria and an observation, this function uses EVALUATION GPT to evaluate the observation against those criteria.\n",
    "\"\"\"\n",
    "    prompt = EVALUATION_PROMPT_TEMPLATE.format(\n",
    "        criteria=criteria,\n",
    "        steps=steps,\n",
    "        request=request,\n",
    "        queries=queries,\n",
    "        metric_name=metric_name,\n",
    "    )\n",
    "    response = client.chat.completions.create(\n",
    "        model=EVALUATION_MODEL,\n",
    "        messages=[{\"role\": \"user\", \"content\": prompt}],\n",
    "        temperature=0,\n",
    "        max_tokens=5,\n",
    "        top_p=1,\n",
    "        frequency_penalty=0,\n",
    "        presence_penalty=0,\n",
    "    )\n",
    "    return response.choices[0].message.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "607ee304",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test out evaluation on a few records\n",
    "\n",
    "evaluation_results = []\n",
    "\n",
    "for x,y in sql_df.head(3).iterrows():\n",
    "    score = get_geval_score(\n",
    "        RELEVANCY_SCORE_CRITERIA,\n",
    "        RELEVANCY_SCORE_STEPS,\n",
    "        y['question'],\n",
    "        y['context'] + '\\n' + y['answer'],'relevancy'\n",
    "    )\n",
    "    evaluation_results.append((y['question'],y['context'] + '\\n' + y['answer'],score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "bd1002c2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "User Question \t: How many heads of the departments are older than 56 ?\n",
      "CREATE SQL Returned \t: CREATE TABLE head (age INTEGER)\n",
      "SELECT SQL Returned \t: SELECT COUNT(*) FROM head WHERE age > 56\n",
      "5\n",
      "********************\n",
      "User Question \t: List the name, born state and age of the heads of departments ordered by age.\n",
      "CREATE SQL Returned \t: CREATE TABLE head (name VARCHAR, born_state VARCHAR, age VARCHAR)\n",
      "SELECT SQL Returned \t: SELECT name, born_state, age FROM head ORDER BY age\n",
      "4\n",
      "********************\n",
      "User Question \t: List the creation year, name and budget of each department.\n",
      "CREATE SQL Returned \t: CREATE TABLE department (creation VARCHAR, name VARCHAR, budget_in_billions VARCHAR)\n",
      "SELECT SQL Returned \t: SELECT creation, name, budget_in_billions FROM department\n",
      "4\n",
      "********************\n"
     ]
    }
   ],
   "source": [
    "for result in evaluation_results:\n",
    "    print(f\"User Question \\t: {result[0]}\")\n",
    "    print(f\"CREATE SQL Returned \\t: {result[1].splitlines()[0]}\")\n",
    "    print(f\"SELECT SQL Returned \\t: {result[1].splitlines()[1]}\")\n",
    "    print(f\"{result[2]}\")\n",
    "    print(\"*\" * 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61b68e2a",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "We will test these functions in combination including our unit test and evaluations to test out two system prompts.\n",
    "\n",
    "Each iteration of input/output and scores should be stored as a **run**. Optionally you can add GPT-4 annotation within your evaluations or as a separate step to review an entire run and highlight the reasons for errors.\n",
    "\n",
    "For this example, the second system prompt will include an extra line of clarification, so we can assess the impact of this for both SQL validity and quality of solution."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b578b00-1b27-49de-8fd1-15c00ec99729",
   "metadata": {},
   "source": [
    "### Building the test framework\n",
    "\n",
    "We want to build a function, `test_system_prompt`, which will run our unit tests and evaluation against a given system prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "40be5fae-4eb3-40ce-8645-613c24d5e0b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def execute_unit_tests(input_df, output_list, system_prompt):\n",
    "    \"\"\"Unit testing function that takes in a dataframe and appends test results to an output_list.\"\"\"\n",
    "\n",
    "    for x, y in tqdm(input_df.iterrows(), total=len(input_df)):\n",
    "        model_response = get_response(system_prompt, y['question'])\n",
    "\n",
    "        format_valid = test_valid_schema(model_response)\n",
    "\n",
    "        try:\n",
    "            test_query = LLMResponse.model_validate_json(model_response)\n",
    "            # Avoid logging since we're executing many rows at once\n",
    "            sql_valid = test_llm_sql(test_query, should_log=False)\n",
    "        except:\n",
    "            sql_valid = False\n",
    "\n",
    "        output_list.append((y['question'], model_response, format_valid, sql_valid))\n",
    "        \n",
    "def evaluate_row(row):\n",
    "    \"\"\"Simple evaluation function to categorize unit testing results.\n",
    "    \n",
    "    If the format or SQL are flagged it returns a label, otherwise it is correct\"\"\"\n",
    "    if row['format'] is False:\n",
    "        return 'Format incorrect'\n",
    "    elif row['sql'] is False:\n",
    "        return 'SQL incorrect'\n",
    "    else:\n",
    "        return 'SQL correct'\n",
    "\n",
    "def test_system_prompt(test_df, system_prompt):\n",
    "    # Execute unit tests and capture results\n",
    "    results = []\n",
    "    execute_unit_tests(\n",
    "        input_df=test_df,\n",
    "        output_list=results,\n",
    "        system_prompt=system_prompt\n",
    "    )\n",
    "    \n",
    "    results_df = pd.DataFrame(results)\n",
    "    results_df.columns = ['question','response','format','sql']\n",
    "    \n",
    "    # Use `apply` to calculate the geval score and unit test evaluation\n",
    "    # for each generated response\n",
    "    results_df['evaluation_score'] = results_df.apply(\n",
    "        lambda x: get_geval_score(\n",
    "            RELEVANCY_SCORE_CRITERIA,\n",
    "            RELEVANCY_SCORE_STEPS,\n",
    "            x['question'],\n",
    "            x['response'],\n",
    "            'relevancy'\n",
    "        ),\n",
    "        axis=1\n",
    "    )\n",
    "    results_df['unit_test_evaluation'] = results_df.apply(\n",
    "        lambda x: evaluate_row(x),\n",
    "        axis=1\n",
    "    )\n",
    "    return results_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6abc2c22-d7c6-4f15-b519-60cc58ff7774",
   "metadata": {},
   "source": [
    "### System Prompt 1\n",
    "\n",
    "The system under test is the first system prompt as shown below.  This `run` will generate responses for this system prompt and evaluate the responses using the functions we've created so far."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "85c44a17",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4d39ec72385f4b74bed652bfa54427f8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/50 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "system_prompt = \"\"\"Translate this natural language request into a JSON object\n",
    "containing two SQL queries.\n",
    "\n",
    "The first query should be a CREATE statement for a table answering the user's\n",
    "request, while the second should be a SELECT query answering their question. \n",
    "\"\"\"\n",
    "\n",
    "# Select 50 unseen queries to test this one\n",
    "test_df = sql_df.tail(50)\n",
    "\n",
    "results_df = test_system_prompt(test_df, system_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3dd9b04-44e2-476c-86fd-c0a261b1cbdd",
   "metadata": {},
   "source": [
    "We can now group the outcomes of:\n",
    "* the **unit tests**, which test the structure of response; and\n",
    "* the **evaluation**, which checks if the SQL is syntatically correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "15bf55ca-63e0-42a0-9846-703549710d4d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "unit_test_evaluation\n",
       "SQL correct      46\n",
       "SQL incorrect     4\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df['unit_test_evaluation'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b3f98f81",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "evaluation_score\n",
       "5    33\n",
       "4    16\n",
       "3     1\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df['evaluation_score'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "019f3a1d",
   "metadata": {},
   "source": [
    "### System Prompt 2\n",
    "\n",
    "We now use a new system prompt to run same unit test and evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "513a2da1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "032f5bc915c44d88a5a0aeb52e2c1bb4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/50 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "system_prompt_2 = \"\"\"Translate this natural language request into a JSON\n",
    "object containing two SQL queries.\n",
    "\n",
    "The first query should be a CREATE statement for a table answering the user's\n",
    "request, while the second should be a SELECT query answering their question.\n",
    "\n",
    "Ensure the SQL is always generated on one line, never use \\\\n to separate rows.\"\"\"\n",
    "\n",
    "\n",
    "results_2_df = test_system_prompt(test_df, system_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd95c3f9-f90d-451d-a32b-aeb066906779",
   "metadata": {},
   "source": [
    "As above, we can group the unit test and evaluation results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "cbaa4bdf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "unit_test_evaluation\n",
       "SQL correct      44\n",
       "SQL incorrect     6\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_2_df['unit_test_evaluation'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "1ada474e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "evaluation_score\n",
       "5    34\n",
       "4    15\n",
       "3     1\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_2_df['evaluation_score'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1908c933",
   "metadata": {},
   "source": [
    "## Reporting\n",
    "\n",
    "We'll make a simple dataframe to store and display the run performance - this is where you can use tools like Weights & Biases Prompts or Gantry to store the results for analytics on your different iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "d277222d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>response</th>\n",
       "      <th>format</th>\n",
       "      <th>sql</th>\n",
       "      <th>evaluation_score</th>\n",
       "      <th>unit_test_evaluation</th>\n",
       "      <th>run</th>\n",
       "      <th>Evaluating Model</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What venue did the parntership of shoaib malik...</td>\n",
       "      <td>{\"create\":\"CREATE TABLE cricket_partnerships (...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>5</td>\n",
       "      <td>SQL correct</td>\n",
       "      <td>1</td>\n",
       "      <td>gpt-4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What venue did the partnership of herschelle g...</td>\n",
       "      <td>{\"create\":\"CREATE TABLE CricketPartnerships (\\...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>5</td>\n",
       "      <td>SQL correct</td>\n",
       "      <td>1</td>\n",
       "      <td>gpt-4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is the number Played that has 310 Points ...</td>\n",
       "      <td>{\"create\":\"CREATE TABLE game_stats (\\n    numb...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>5</td>\n",
       "      <td>SQL correct</td>\n",
       "      <td>1</td>\n",
       "      <td>gpt-4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What Losing bonus has a Points against of 588?</td>\n",
       "      <td>{\"create\":\"CREATE TABLE BonusInfo (\\n    id IN...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>5</td>\n",
       "      <td>SQL correct</td>\n",
       "      <td>1</td>\n",
       "      <td>gpt-4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What Tries against has a Losing bonus of 7?</td>\n",
       "      <td>{\"create\":\"CREATE TABLE matches (\\n    id SERI...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>5</td>\n",
       "      <td>SQL correct</td>\n",
       "      <td>1</td>\n",
       "      <td>gpt-4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  What venue did the parntership of shoaib malik...   \n",
       "1  What venue did the partnership of herschelle g...   \n",
       "2  What is the number Played that has 310 Points ...   \n",
       "3     What Losing bonus has a Points against of 588?   \n",
       "4        What Tries against has a Losing bonus of 7?   \n",
       "\n",
       "                                            response  format   sql  \\\n",
       "0  {\"create\":\"CREATE TABLE cricket_partnerships (...    True  True   \n",
       "1  {\"create\":\"CREATE TABLE CricketPartnerships (\\...    True  True   \n",
       "2  {\"create\":\"CREATE TABLE game_stats (\\n    numb...    True  True   \n",
       "3  {\"create\":\"CREATE TABLE BonusInfo (\\n    id IN...    True  True   \n",
       "4  {\"create\":\"CREATE TABLE matches (\\n    id SERI...    True  True   \n",
       "\n",
       "  evaluation_score unit_test_evaluation  run Evaluating Model  \n",
       "0                5          SQL correct    1            gpt-4  \n",
       "1                5          SQL correct    1            gpt-4  \n",
       "2                5          SQL correct    1            gpt-4  \n",
       "3                5          SQL correct    1            gpt-4  \n",
       "4                5          SQL correct    1            gpt-4  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df['run'] = 1\n",
    "results_df['Evaluating Model'] = 'gpt-4'\n",
    "\n",
    "results_2_df['run'] = 2\n",
    "results_2_df['Evaluating Model'] = 'gpt-4'\n",
    "\n",
    "run_df = pd.concat([results_df,results_2_df])\n",
    "run_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0162a009-fc43-484c-90f6-d59a8e52f365",
   "metadata": {},
   "source": [
    "#### Plotting unit test results\n",
    "\n",
    "We can create a simple bar chart to visualise the results of unit tests for both runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "ed800f0c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Number of records</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>run</th>\n",
       "      <th>unit_test_evaluation</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">1</th>\n",
       "      <th>SQL correct</th>\n",
       "      <td>46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SQL incorrect</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">2</th>\n",
       "      <th>SQL correct</th>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SQL incorrect</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          Number of records\n",
       "run unit_test_evaluation                   \n",
       "1   SQL correct                          46\n",
       "    SQL incorrect                         4\n",
       "2   SQL correct                          44\n",
       "    SQL incorrect                         6"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unittest_df_pivot = pd.pivot_table(\n",
    "    run_df,\n",
    "    values='format',\n",
    "    index=['run','unit_test_evaluation'],\n",
    "    aggfunc='count'\n",
    ")\n",
    "unittest_df_pivot.columns = ['Number of records']\n",
    "unittest_df_pivot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "e2b4aa03-42f5-4c30-a610-e553937bf160",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0kAAAIjCAYAAADWYVDIAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAATSVJREFUeJzt3QeYFeXdP+5nqSoKWFA0omCJigq22DUWFDX2bjQqGo29N4xdY+9vbG/sxpKo2GMl9l5QrNiIDRVjARGRdv7X93l/Z/+zywK7yy5buO/rOrBnTpnnzMw5M595ylSUSqVSAgAAIGvzf/8BAAAQhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkaMb23HPP1LNnz6YuRquy3nrr5VtTqaioSKecckqTzZ+axTax3HLLpZbipptuSksvvXRq37596tq1a5qVNOTv4pgxY9If//jH1L179/zdPOyww1Jrc/311+fP9sorrzR1UaBFEZJgBsUBb+yA/vvf/9b4eBx4NdRB+dixY/P8nnjiiek+Nw4iolzTu8UOtCGceeaZ6e67767Vc//zn/9Ms0xnn312asn+9a9/CULT2CYPPvjgKR6LbToeu+OOO5qkbC3Je++9l4PC4osvnv72t7+l//3f/53u71P5FqEq1sMhhxySfvjhhzSri9+t+A3cf//9c/D8wx/+0Kjzm9bv8iabbJKaI9sQs6p2TV0AYOriAGjy5MlVQtKpp56a/55e8Lr44ovzWdLigfutt96aLrroojTffPNVTl9zzTUb7GBj++23T1tvvXWtX7PLLrukzTbbbIrpK664YmrJYllfdtllNQaln3/+ObVrN2v/9MZ2PXDgwLTQQgs1dVFapAiU8btwySWXpCWWWKJWr7niiivSnHPOmX766ac0ePDg9D//8z/ptddeS88880yalf373/9Oq6++ejr55JNn2jxXWGGFdOSRR04xvbl/H2xDzGpm7T01NHNxxq6+qoeVr776KoekmN5cmvCttNJKabfddkuzktlmmy3NypZddtk0bNiwXFt46aWXpllJBJvx48fP8DYwcuTI/H9dmtnFCYzyyZE//elPaeedd07/+Mc/0ksvvZRWXXXV1JyMGzcudejQIbVp0/iNXWJZ9u7du8Heb+LEiXk9R/mn5le/+lWL/N1rSdsQNATN7WAmKzcr+uc//5n+8pe/pIUXXjgfNG244Ybpww8/nGrb+2ii1q1bt/x31CaVmz7MaLOuv//972nllVdOs88+e5pnnnnyju+zzz6r8pwPPvggbbfddrndfpQ1yhzPGzVqVH48yhFnF2+44YbKckXZZ9Tmm2+eFltssRofW2ONNdIqq6xSef+6665LG2ywQZp//vlTx44d84FPnPmsbXv9WL41radi08ann3467bDDDmmRRRbJ8+jRo0c6/PDDc+1QWXzuqEUKxSYqZTWtsyFDhqRNN900de7cOZ+pjW3hhRdeqLGczz77bDriiCPyttCpU6e0zTbbpG+++abKc6PvQf/+/fMBTazXXr16pb322qvBlvWjjz6a1l577XyQHuVdaqml0vHHH59qI7bn3XffPdcmjRgxol59T8rNf4ri/kEHHZRuv/32vO7jc0e533zzzfz4VVddlWtdYvuNWtjq67vs1VdfzbWr5eV25ZVXTvGcX375Jdc8xPuVt4NjjjkmT6+pTDfffHMOh/Hchx56aJqf+fLLL698btQsHHjggVWaNMXyKNd6xDZQ39+AddZZJ///0UcfVZn+4osv5mZfXbp0SXPMMUf67W9/m7e56r744ou099575zJGWWNZRZO1CIFlH3/8cf6+xO9KvFfU2DzwwAM1fs9uu+22dMIJJ+QAEc8dPXp0fjya8EaT5Vhv8f9dd91V4+eJ18fv2FxzzZW/R8svv3yuaZua8nyHDx+ey1T+npa3iwhP8fkWWGCBPO++ffvm37eamg2ff/75ueY+mj/GsnjnnXfSjBo6dGje/uM7GfOP3974Dn/77bf1Whchts/p/XbM6DYU22dNv/3V+4LWZT8ITUVNEjSROJMeZ0qPOuqoHDbOPffctOuuu+aDlJrEji0O+mPnFzu3bbfdNk/v06dPvcsQO6cTTzwx7bjjjrnzcuwwownFuuuumw/c4yA4drRxwB072OhLEjvr2Cnff//9+eAtDqaiLX+8Ps4m7rvvvvm944BheqL5YE19uWK+0SRtp512ygfUL7/8cvrNb35T+fgnn3ySQ8R5551XOS2WTRxcbrnllvm19913XzrggAPyWd040GwIcQAeZY51MO+88+YzqLG8Pv/88/xY+QxrHPxHkIjlMj1vv/12PtiIA7s40I7awzigjwOKJ598Mq222mpVnh/rYO65584HynGQFgdncSAeZ3TLB3cbb7xx3l6OO+64vCzjeYMGDZpmOWq7rKO8EahiuzvttNPyQVkc1NR0ID01f/7zn9ONN97Y4LVJEWLvvffeyvV91lln5bLGco3wEdvD999/n79rccAZTa2K4rFo/hnfh2gKGgdwsa6jVqAcMmN7im0smhjFtr7MMsvkIBbNWN9///0p+uXFPOJ9Yh1FaJ1WLW6EnTgB0q9fvzzfqHGL7TrWSSzf2DZifceyi7BQbv5Un9+AchiIbalY1gjrETZi+4rfp/LJh1i25dqC2L7j7/j+xzKIASTiNyH6k8X3I5bX119/ncNm3I++K/F9iZARyy6eF79hRaeffnp+Xfwexm9N/P3II4/kkzMRemNdRkAYMGBAPqAuiu9arK84wD7nnHPytHfffTcvs0MPPbTGzx/rLb6fcZIj3q/c/C2+N3HSI75/sV3HeovQEd/vOPiPz1z9PWMZRe1XLIv4PkQonJYJEybU+LsXwSXCefkzRciMzxu/ufG9i75n8X98H8snCWqzLmr729EQ21Bj7wdhpioBM+Tkk08uxVfpm2++qfHxZZddtvTb3/628v7jjz+en7/MMsuUfvnll8rpl1xySZ7+5ptvVk7bY489Sosuumjl/ZhHPCfmWVfnnXdefu3w4cPz/f/85z+ltm3blv7yl79UeV7Mv127dpXThwwZkl93++23T/P9O3XqlMtbG1GGeM+p3Z5//vn8vFGjRpU6duxYOvLII6u8/txzzy1VVFSUPvnkk8ppY8eOnWI+/fv3Ly222GJVpsW6KK6P6667rspyqb6e4v9pzeOss86aoiwHHnhgfm1Nqq+/rbfeutShQ4fSRx99VDltxIgRpbnmmqu07rrrTlHOfv36lSZPnlw5/fDDD8/r8Ycffsj377rrrvy8l19+uVQXtV3WF1100TS392mJbfl3v/td/nvAgAGl2WabLX/W4vIubmfVt//q37miuB/lL67Hq666Kk/v3r17afTo0ZXTBw4cOMU6j20ipl1wwQWV0+L7ucIKK5Tmn3/+0vjx4/O0m266qdSmTZvS008/XWX+V155ZX79s88+W6VM8dy33357ustm5MiReTvYeOONS5MmTaqc/te//jW/z7XXXjvF56/NOig/d9iwYfn58b2P95p99tlL3bp1K/3000/5ebFNLbnkkvk7U9y+Ypvv1atXaaONNqqctvvuu+fPVdM2Vn7tYYcdludbXE4//vhjfq+ePXtWfsbyeo/vafXvVyz7BRdcsHLbDo888kh+fnG7OPTQQ0udO3cuTZw4sTQj22TZxRdfnOfx97//vXJarP811lijNOecc1ZuS+XfsZh3rL/azm9qv3vxWzKt35pbb701P++pp56q07qo7W/HjG5D5c9X036g+u9uXfaD0FQ0t4MmEmcIi2f5yk0X4uzhzBA1C3FWPM6ax1nN8i3OWi655JLp8ccfz8+LmqLw8MMP5zOTDSnOfMYZ0+q3ch+BqF2JM9txJv7/jjn/T5z5jKY70eytrHwGNsQZyfgs0VQolme5WeCMKs4jmhfGPOJseZQtat7qatKkSflsefQTKzZ1W3DBBdPvf//7XFtRbnZUXGbFpmax3cT7RI1PsZ9K1PTFGevaqu2yLr//PffcU2VQkbqKplXRf6MhRzKMmoRiTU25Fi5qI6IZVvXp1b9rUQMZNYFl8f2M+1E7F83wQtQoRC1EnLEvfm+itiWUvzdlsQ3Wps/LY489lmttYwjqYl+cffbZJ6+b6s3U6iqaREYtSSyfqBWLpoIPPvhgbtoWXn/99dysNra7qLEpf67YzmO5PvXUU3l9xy1qy7bYYosqTTDLyttmDF4SNRzRLLMsar1i+40aiOpN0vbYY48q368vv/wylymml3+DwkYbbTTF8oxtMsoZvx0NIcoev4NRO1UWtXhRIxaD4UQNb1FsX+Wm0LUR219Nv3vF+RWXRdRSxbqI72GIwRJCbddFbX87ZnQbaon7QZgWIQlmguo7q1A8wC82WYgmPzNDHBDFwXAEotjxFW/RVKXcOTyamkQ79quvvjo3F4qmd9HnpiGCR8w7mhZVv8VBYbEZWPSRev755yvbv8cBa0wviqY18dposhIHTfE5yv1kGiokffrpp7nJTTSniQO+mEccBNd3HtG8MYJnHHxUFwficRBUvX/Y9LabKE8ctEWzrVhfW221VW4OVL2/TE1qs6zj77XWWis3r4z+GtE3LYJVXQNThMIYbjmaEMUBcUOovmzKB9fRZ6im6dW/a9GnI7afol//+tdVmhbF9yaaPFX/zpSfV/7elMX3pzbKB6rVt4U4gIxlVdsD2am5884784H4Lbfckg+2o5zFA/H4XCFCSfXPFt/92H5iG49tNoL79K4pFeWd2nZd/LxTW07lx+M3orrq7xvNKGP5R8iPpnNxAD+9vl/TK3vMt/rAEbUt+/TE97Km371FF1208jnfffddbtYX37FYT7EeyvMp/9bUdl001D5nettQfTT1fhCmRZ8kmEHlkaqKnfeL4iC4ptGs2rZtW+Pzi2fxG1Mc1EZ4izOBNZUlQkDZBRdckMNB1B5EzUecUY0+AtE2vnr/gIYWZ0njTGUciEetTfwfBy/RIbwsDubjbHec3b/wwgvzQXEcXMYZ4egrMq0D+JoCbIgzrNXvx1nsOHg59thj87zigDra/8eymZFalbqY3nZTvtZQrJvolxU1gHHQGOswphXXa32WdRwURa1C1JhE7UYcjEZtU9SkxLYxtfJNrW9S9AuJfiQ1DR1f23VTNrV5N+R3LdZzDAoQ21lNqgeyGT2IbCjRz7A8Mlms5/gM0fcjQnCs4/L2G33PYojqmsS2E9t/Y5iR5RSDtUStU2zr8XsWtzgxEH3sqg+20BgaYx1HDf9zzz2Xjj766Lw+YtnHOopBNer7WzOj34PpbUPT+87WNP+m3g/CtAhJMIPKZ/+ik3X1A6QISHFmPjrSN4Sp7YDqIwZWiB1RnJ0snwWfltghxi2aScXOO2oTYuSvM844o8HLVhRBJDrfRzOnODCNA/JoklG8pkiEgTjTHZ32i2cmqzd9qkn5zGX1iyJWP1scnfOjY34cdMXBV1lNTXxquyzi7HCEkth2arpgaBx4VN+maivO9MYtBueIM79xMBMjgEUN0Iws6xDlilAat3heXCMrAk8s7zgjXpdtMIZCjoEqqg9QUV43NV2sckZrVaYmOsFHs61ibVKs81BuxhdlfuONN/Jnb8htvvg7Umx6GU3wYgS2uizX6YkD7ui8H02dIghHbWB5oJWoxZ3WvGKbjee89dZb0/08U9uuy49P7/XFGq6imt43TorEgXvcIkRE7VJsVzEwTW2vJVWcd4wuF+9TrE2qbdlnVNSixHWIojb4pJNOqpxefVnUdl00hpq2oel9Z6c2eiY0V5rbwQyKg6XYQcdIU9XP8EVTouh3Ec1AGkK57XdDXOU8RseLs3ixI65+1i7ul4eajeYc8RmKIizFwUOxCVccWDbW1dejiVccwEaznzhArd7Urnw2svg5oklKnE2envLBYdSOFM96xrqb3jzi75qGGS4fZE9vecR7RoCOGrrikNQxMlgEm+jPUWx6WNsDrOrrs1wzUNsmd9Na1jXVJNTl/auL0B19p2JUq5rWTazHOGAti6Z5UxsGekbFdh4H1sWAEvfjYDRGfCuf4Y/awxjCvLqoTY6QVR8RTOJ3JEb7K66/a665Ji+D3/3ud6khRWiOWuDyaHDx+WJ5x3DWxYtQl5WHio7vfdT6xYmJGGq+unLZY5TAGP2x3HQzxLKJ71UEzun104p+ebFdxUmJYlPWOClRvT9T9WGxo4zlEf/qs01G2eO6csVR32LbiJEsIxyUm9g2lpp+a0KMRldU23XRWKpvQyG2oaixLg4/Hv0jqzcbhpZATRLMoGjqEWf74mAvmiPEELcRZqK2JS7eGgfBcXazoZp1xMFF7Lyj9if6xkR79Nq2SS+KnVnUAg0cODAfoMfONjq3x1nrOAiNTr4xLGsMCxzDxEaTq5hnHCxEE6nYkUffl7I4yIrO51GzEDUPUUNVU+1AUXRAjus01VS2uMZN8aAlyhblqT7fEMu4fCY5OtrHQV4cxMa6mV5/lxg2PGpcYjlEAIhlGjUu1YNhNK+LckUZ4iA5wku00a+p7Xz5gDqaJUYfrihz+UxrdbEOytcdirPfMXhAHJjHwV1NwWF64qAyhruOIZajvD/++GNeFlHeWI7TM71lHcN+R6CMg/Y4ox79EmJ+cbBU7KRf19qkmppFxTKLpo3xWWJZRs1snIyI7bDceb0hxXYbB3zxfYh5xPcsmnHFgX35ws7RjyrOnO+333655ixqVCNURy1DTI8mXzV1op+eCGKxDcZJi2hSFb8jUWMSyzaGZG/oi4/G54k+L9GcK5pMxjwjGMcJnfhORA1BXLMotvX4nLH9xMF4iJrDaFoZYaE8DHp8z6IGMgYbiT6BMfx8/P7F+8W6i+9VrOP4fYnvTW0uFBtNemM7i+0qmozG9zOCSpSvGOSidjQeiyafsR1GrUU8L0JWuR9RXcRniu9gNKONpmQR6qIJa/R7jKBSHASkPmKZ1vS7FwEsfodjWce+JL7/cQIh1kMs71h21dVmXTSWmrahWBexrOLvOKEQTaHjs9bmkhDQ7DTZuHrQysRwsauvvnoeCjuGIl566aVLp556amncuHFVnlfTUMfF4WRjuNZpDYH83HPPlVZeeeU8XHBdhgOvPgR42Z133llae+21c7njFuWOIaxjuNfw8ccfl/baa6/S4osvnodsnmeeeUrrr79+6bHHHqvyPu+9914esjqGhY35TGs48OkNAV7Ta3fdddfKYWxrcu+995b69OmTyxhDDJ9zzjl5mNqahnouDkUbYvjteN9YbwsssEDp+OOPLz366KNTDAH+zjvv5OfFMMDzzTdfaZ999im98cYbU6y3GIr44IMPzsPjxvDZxZ/amtbZa6+9lodejvedY4458vKN9VxUHsa3+lC/1Ycqj/faZZddSossskj+PDF89eabb1565ZVXpro+6rKsBw8eXNpqq61KCy20UN4G4/+Y3/vvv1+v4ZbDBx98kIcirul7EUM+L7fccnleSy21VP6eTW0I8Nhua9rOYtuf3ncwtokYrj+WUwz1HNtRlDeG4K4uhoOO7SueH8t47rnnzt/J+L7HUOrTKtP0xPziO9i+ffu8Le6///6l77//vspz6jMEeE3PjbJ26dKlyvchhvzfdtttS/POO2/+bLEMdtxxx7zei2JI+Bh+OrbxeF4M4R2ftTicc3yvtt9++1LXrl3z8lx11VVL999/f61+D4u/TzFMdMyjd+/epUGDBk3xu3jHHXfkodNjW4/tJLb9P/3pT6Uvv/yy3tvk119/nYepj+95vOfyyy9f5Ts+re1revOb2u9e8TN9/vnnpW222SYvu1hHO+ywQx4uv6bfj+mti9r+djTUNhTD6P/qV7/KZVlrrbXyd2pqQ4DXZj8ITaUi/mnqoAYAANBc6JMEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAAMxKF5OdPHlyvnp8XPytoqKiqYsDAAA0kbj6UVxoPS4gPq0LW7f6kBQBqUePHk1dDAAAoJn47LPP0sILLzzrhqSoQSoviM6dOzd1cQAAgCYyevToXIFSzgizbEgqN7GLgCQkAQAAFdPphmPgBgAAgAIhCQAAoEBIAgAAmJX6JAEAQHMSw1BPnDgxTZo0qamL0uq0bds2tWvXboYv/SMkAQDATDJ+/Pj05ZdfprFjxzZ1UVqtOeaYIy244IKpQ4cO9X4PIQkAAGaCyZMnp+HDh+fajriYaRzEz2iNB1Vr6CKEfvPNN3k5L7nkktO8YOy0CEkAADATxAF8BKW4Tk/UdtDwZp999tS+ffv0ySef5OU922yz1et9DNwAAAAzUX1rN5h5y9caAgAAKBCSAAAACvRJAgCAJlYx9L6ZOr9Sny1m6vxaGjVJAADANO255555JL64xcAIvXr1Ssccc0waN25co8970KBBaeONN07zzjtvnv/rr7/e6PMUkgAAgOnaZJNN8jWePv7443TRRRelq666Kp188smNPt+ffvoprb322umcc85JM4uQBAAATFfHjh1T9+7d8xDmW2+9derXr1969NFHKx/v2bNnuvjii6u8ZoUVVkinnHJK5f2oCbr66qvTNttsk4dBj2sZ3XvvvdOc7x/+8Id00kkn5fnNLEISAABQJ2+99VZ67rnn8gVx6+rUU09NO+64Yxo6dGjabLPN0q677pq+++671JwISQAAwHTdf//9ac4558wXaF1++eXTyJEj09FHH12v/k277LJLWmKJJdKZZ56ZxowZk1566aXUnBjdDgAAmK71118/XXHFFbmPUPRJateuXdpuu+3q/D59+vSp/LtTp06pc+fOOXA1J2qSAACA6erUqVOu/enbt2+69tpr04svvpiuueaaysfbtGmTSqVSlddMmDBhiveJ0fGKop/S5MmTU3MiJAEAAHXSpk2bdPzxx6cTTjgh/fzzz3lat27d8uh3ZaNHj07Dhw9PLZHmdq38QmG0Pi7+BgA0BzvssEPuk3TZZZelo446Km2wwQbp+uuvT1tssUXq2rVrHpGubdu2MzyfGNTh008/TSNGjMj3hw0blv+Pkfbi1hiEJAAAaGIt8SRou3bt0kEHHZTOPffctP/++6eBAwfmmqPNN988denSJZ1++ukNUpMUQ4QPGDCg8v7OO++c/49rNBWHF29IFaXqDQdbmajmi5U0atSo3CmsqalJYlb8EQUAUho3blwODb169cojxDHzl3Nts4E+SQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAF7Yp3AACAma+iomKmzq9UKs3U+bU0apIAAIBp2nPPPXOQi1v79u1Tr1690jHHHJPGjRvXqPOdMGFCOvbYY9Pyyy+fOnXqlBZaaKG0++67pxEjRjTqfIUkAABgujbZZJP05Zdfpo8//jhddNFF6aqrrkonn3xyo85z7Nix6bXXXksnnnhi/n/QoEFp2LBhacstt2zU+QpJAADAdHXs2DF179499ejRI2299dapX79+6dFHH618vGfPnuniiy+u8poVVlghnXLKKZX3oybq6quvTttss02aY4450pJLLpnuvffeqc6zS5cueR477rhjWmqppdLqq6+e/vrXv6ZXX301ffrpp430SYUkAACgjt5666303HPPpQ4dOtT5taeeemoOPUOHDk2bbbZZ2nXXXdN3331X69ePGjUqh62uXbumxiIkAQAA03X//fenOeecM80222y5j9DIkSPT0UcfXa/+TbvssktaYokl0plnnpnGjBmTXnrppVq9NvpARR+leH3nzp1TYzG6HQAAMF3rr79+uuKKK9JPP/2U+yS1a9cubbfddnV+nz59+lT+HYMxRNiJwFWbQRyiBipG5otyNCYhCVqYmT1EKK2PYV8BqI9OnTrl2p9w7bXXpr59+6Zrrrkm7b333nlamzZtptjHRLCpLkbHq35sM3ny5FoFpE8++ST9+9//btRapKC5HQAAUCdt2rRJxx9/fDrhhBPSzz//nKd169Ytj35XNnr06DR8+PAZnlc5IH3wwQfpscceS/POO29qbEISAABQZzvssENq27Ztuuyyy/L9DTbYIN10003p6aefTm+++WbaY4898uMzGpC233779Morr6Sbb745TZo0KX311Vf5Nn78+NRYNLcDAIAm1hKbQrdr1y4ddNBB6dxzz037779/GjhwYK452nzzzfPQ3aeffvoM1yR98cUXlUOEx3DiRY8//nhab731UmOoKLXENVIHUc0XKymGCmzstou1UTH0vqYuAi1d38a9eBqtXyv/2QdotmJktggNvXr1yiPEMfOXc22zgeZ2AAAABUISAABAgZAEAABQICQBAAAUCEkAADATGUCn+S9fIQkAAGaC9u3b5//Hjh3b1EVp1cb+v+VbXt714TpJAAAwE8SFVbt27ZpGjhyZ788xxxypoqKiqYvVqmqQxo4dm5dvLOcZuZCtkAQAADNJ9+7d8//loETDi4BUXs71JSQBAMBMEjVHCy64YJp//vnThAkTmro4rU779u1nqAapTEgCAICZLA7kG+JgnsZh4AYAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAaI4h6eyzz04VFRXpsMMOq5w2bty4dOCBB6Z55503zTnnnGm77bZLX3/9dZOWEwAAaN2aRUh6+eWX01VXXZX69OlTZfrhhx+e7rvvvnT77benJ598Mo0YMSJtu+22TVZOAACg9WvykDRmzJi06667pr/97W9p7rnnrpw+atSodM0116QLL7wwbbDBBmnllVdO1113XXruuefSCy+80KRlBgAAWq8mD0nRnO53v/td6tevX5Xpr776apowYUKV6UsvvXRaZJFF0vPPPz/V9/vll1/S6NGjq9wAAABqq11qQrfddlt67bXXcnO76r766qvUoUOH1LVr1yrTF1hggfzY1Jx11lnp1FNPbZTyAgAArV+T1SR99tln6dBDD00333xzmm222RrsfQcOHJib6pVvMR8AAIBmH5KiOd3IkSPTSiutlNq1a5dvMTjDpZdemv+OGqPx48enH374ocrrYnS77t27T/V9O3bsmDp37lzlBgAA0Oyb22244YbpzTffrDJtwIABud/Rsccem3r06JHat2+fBg8enIf+DsOGDUuffvppWmONNZqo1AAAQGvXZCFprrnmSsstt1yVaZ06dcrXRCpP33vvvdMRRxyR5plnnlwjdPDBB+eAtPrqqzdRqQEAgNauSQdumJ6LLrootWnTJtckxah1/fv3T5dffnlTFwsAAGjFKkqlUim1YjEEeJcuXfIgDs2hf1LF0Puaugi0dH23bOoS0MK18p99AJjhbNDk10kCAABoToQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAAJiRkPTzzz+nsWPHVt7/5JNP0sUXX5weeeSRur4VAABAyw9JW221Vbrxxhvz3z/88ENabbXV0gUXXJCnX3HFFY1RRgAAgOYbkl577bW0zjrr5L/vuOOOtMACC+TapAhOl156aWOUEQAAoPmGpGhqN9dcc+W/o4ndtttum9q0aZNWX331HJYAAABmqZC0xBJLpLvvvjt99tln6eGHH04bb7xxnj5y5MjUuXPnxigjAABA8w1JJ510UjrqqKNSz549c3+kNdZYo7JWacUVV6zTe0Ufpj59+uRwFbd4rwcffLDy8XHjxqUDDzwwzTvvvGnOOedM2223Xfr666/rWmQAAIBaqyiVSqVUR1999VX68ssvU9++fXNTu/DSSy/loLP00kvX+n3uu+++1LZt27TkkkumKMYNN9yQzjvvvDRkyJC07LLLpv333z898MAD6frrr09dunRJBx10UJ7fs88+W+t5jB49Or921KhRzaKmq2LofU1dBFq6vls2dQlo4erxsw8ArUJts0G9QlJjmmeeeXJQ2n777VO3bt3SLbfckv8O7733XlpmmWXS888/n/tA1YaQRKsjJDGDmtnPPgDMNLXNBu1q82YxOENtDRo0KNXHpEmT0u23355++umn3Ozu1VdfTRMmTEj9+vWrfE7UUi2yyCLTDEm//PJLvhUXBAAAQIP2SYq0Vb5F4ho8eHB65ZVXKh+PQBPT4vG6evPNN3N/o44dO6b99tsv3XXXXal37965SV+HDh1S165dqzw/hhyPx6bmrLPOqlLeHj161LlMAADArKtWNUnXXXdd5d/HHnts2nHHHdOVV16Z+xOVa4EOOOCAejVnW2qppdLrr7+eq7ziukt77LFHevLJJ1N9DRw4MB1xxBFVapIEJQAAoEFDUtG1116bnnnmmcqAFOLvCCZrrrlm7k9UF1FbFMOKh5VXXjm9/PLL6ZJLLkk77bRTGj9+fPrhhx+q1CbF6Hbdu3ef6vtFjVTcAAAAZsoQ4BMnTswDKFQX0yZPnpxmVLxH9CmKwNS+ffvcjK9s2LBh6dNPP60cdhwAAKDJa5IGDBiQ9t577/TRRx+lVVddNU978cUX09lnn50fq2vTuE033TQPxvDjjz/mkeyeeOKJfJHa6E8U84kaqhjxLpryHXzwwTkg1XZkOwAAgEYPSeeff35u7nbBBRfkayWFBRdcMB199NHpyCOPrNN7jRw5Mu2+++75fSIUxYVlIyBttNFG+fGLLrooXxcpLiIbtUv9+/dPl19+eV2LDAAAUGt1uk5SNLWL2p4IKzHKXHl47eZw/aGpcZ0kWh3XSWIGuU4SALOq0bXMBnXqk9SuXbs8TPe4cePy/Xjj5hA8AAAAmmzghuiHNGTIkAYrAAAAQIvukxTXQ4q+R59//nkega5Tp05VHo9+RQAAALNMSNp5553z/4ccckjltIqKitzGPf6PC8sCAADMMiFp+PDhjVMSAACAlhiSFl100cYpCQAAQEsMSSEuJHvxxRend999N9/v3bt3OvTQQ9Piiy/e0OUDAABo3qPbxcVeIxS99NJLeZCGuL344otp2WWXTY8++mjjlBIAAKC51iQdd9xx6fDDD09nn332FNOPPfbYtNFGGzVk+QAAAJp3TVI0sdt7772nmL7XXnuld955p6HKBQAA0DJCUrdu3dLrr78+xfSYNv/88zdUuQAAAFpGc7t99tkn7bvvvunjjz9Oa665Zp727LPPpnPOOScdccQRjVFGAACA5huSTjzxxDTXXHOlCy64IA0cODBPW2ihhdIpp5xS5QKzAAAALVFFqVQq1ffFP/74Y/4/QlNzNXr06NSlS5c0atSo1Llz56YuTqoYel9TF4GWru+WTV0CWrgZ+NkHgBatttmgzjVJw4cPTxMnTkxLLrlklXD0wQcfpPbt26eePXvWv9QAAAAtbeCGPffcMz333HNTTI9rJcVjAAAAs1RIGjJkSFprrbWmmL766qvXOOodAABAqw5JFRUVlX2RiqJd36RJkxqqXAAAAC0jJK277rrprLPOqhKI4u+Ytvbaazd0+QAAAGaqOg/cENdDiqC01FJLpXXWWSdPe/rpp/NIEf/+978bo4wAAADNtyapd+/eaejQoWnHHXdMI0eOzE3vdt999/Tee++l5ZZbrnFKCQAA0FxrksoXjz3zzDMbvjQAAAAtrSap3Lxut912S2uuuWb64osv8rSbbropPfPMMw1dPgAAgOYdku68887Uv3//NPvss6fXXnst/fLLL5Wj26ldAgAAZrmQdMYZZ6Qrr7wy/e1vf0vt27evnB7XTorQBAAAMEuFpGHDhuXR7arr0qVL+uGHHxqqXAAAAC0jJHXv3j19+OGHU0yP/kiLLbZYQ5ULAACgZYSkffbZJx166KHpxRdfTBUVFWnEiBHp5ptvTkcddVTaf//9G6eUAAAAzXUI8OOOOy5Nnjw5bbjhhmns2LG56V3Hjh1zSDr44IMbp5QAAAAzSUWpVCrV54Xjx4/Pze7GjBmTLzA755xzpp9//jmPetecjB49OveXitH3Onfu3NTFSRVD72vqItDS9d2yqUtAC1fPn30AaPFqmw3qdZ2k0KFDhxyOVl111TzK3YUXXph69epV37cDAABoFmodkuJ6SAMHDkyrrLJKvojs3Xffnadfd911ORxddNFF6fDDD2/MsgIAADSfPkknnXRSuuqqq1K/fv3Sc889l3bYYYc0YMCA9MILL+RapLjftm3bxi0tAABAcwlJt99+e7rxxhvTlltumd56663Up0+fNHHixPTGG2/kUe4AAABmqeZ2n3/+eVp55ZXz38stt1we0S6a1wlIAADALBmSJk2alAdrKGvXrl0e0Q4AAGCWbG4XQ8buueeeuQYpjBs3Lu23336pU6dOVZ43aNCghi8lAABAcwtJe+yxR5X7u+22W2OUBwAAoGWEpBjqGwAAoLWr98VkAQAAWiMhCQAAoEBIAgAAKBCSAAAA6hqSVlpppfT999/nv0877bQ0duzY2rwMAACgdYakd999N/3000/571NPPTWNGTOmscsFAADQfIcAX2GFFdKAAQPS2muvnS8qe/7556c555yzxueedNJJDV1GAACA5hWSrr/++nTyySen+++/P1VUVKQHH3wwtWs35UvjMSEJAABo9SFpqaWWSrfddlv+u02bNmnw4MFp/vnnb+yyAQAANM+QVDR58uTGKQkAAEBLDEnho48+ShdffHEe0CH07t07HXrooWnxxRdv6PIBAAA07+skPfzwwzkUvfTSS6lPnz759uKLL6Zll102Pfroo41TSgAAgOZak3Tcccelww8/PJ199tlTTD/22GPTRhtt1JDlAwAAaN41SdHEbu+9955i+l577ZXeeeedhioXAABAywhJ3bp1S6+//voU02OaEe8AAIBZrrndPvvsk/bdd9/08ccfpzXXXDNPe/bZZ9M555yTjjjiiMYoIwAAQPMNSSeeeGKaa6650gUXXJAGDhyYpy200ELplFNOSYccckhjlBEAAGCmqSiVSqX6vvjHH3/M/0doaq5Gjx6dunTpkkaNGpU6d+7c1MVJFUPva+oi0NL13bKpS0ALNwM/+wDQotU2G9TrOkllzTkcAQAAzJSBGwAAAFozIQkAAKBASAIAAKhvSJowYULacMMN0wcffFCXlwEAALTOkNS+ffs0dOjQxisNAABAS2tut9tuu6VrrrmmcUoDAADQxOo8BPjEiRPTtddemx577LG08sorp06dOlV5/MILL2zI8gEAADTvkPTWW2+llVZaKf/9/vvvV3msoqKi4UoGAADQEkLS448/3jglAQAAaMlDgH/44Yfp4YcfTj///HO+XyqVGrJcAAAALSMkffvtt3kY8F//+tdps802S19++WWevvfee6cjjzyyMcoIAADQfEPS4YcfnocC//TTT9Mcc8xROX2nnXZKDz30UEOXDwAAoHn3SXrkkUdyM7uFF164yvQll1wyffLJJw1ZNgAAgOZfk/TTTz9VqUEq++6771LHjh0bqlwAAAAtIySts8466cYbb6wy7PfkyZPTueeem9Zff/2GLh8AAEDzbm4XYSgGbnjllVfS+PHj0zHHHJPefvvtXJP07LPPNk4pAQAAmmtN0nLLLZcvIrv22munrbbaKje/23bbbdOQIUPS4osv3jilBAAAaK41SaFLly7pz3/+c8OXBgAAoCWGpO+//z5dc8016d133833e/funQYMGJDmmWeehi4fAABA825u99RTT6WePXumSy+9NIeluMXfvXr1yo8BAADMUjVJBx54YL5w7BVXXJHatm2bp02aNCkdcMAB+bE333yzMcoJAADQPGuSPvzww3TkkUdWBqQQfx9xxBH5MQAAgFkqJK200kqVfZGKYlrfvn0bqlwAAADNt7nd0KFDK/8+5JBD0qGHHpprjVZfffU87YUXXkiXXXZZOvvssxuvpAAAADNBRalUKk3vSW3atEkVFRVpek+N50T/pOZk9OjRecjyUaNGpc6dOzd1cVLF0Puaugi0dH23bOoS0MLV4mcfAFql2maDWtUkDR8+vCHLBgAA0GzVKiQtuuiijV8SAACAlnox2REjRqRnnnkmjRw5Mk2ePLnKY9FnqbbOOuusNGjQoPTee++l2WefPa255prpnHPOSUsttVTlc8aNG5dH07vtttvSL7/8kvr3758uv/zytMACC9Sn6AAAAA0bkq6//vr0pz/9KXXo0CHNO++8uR9SWfxdl5D05JNP5msr/eY3v0kTJ05Mxx9/fNp4443TO++8kzp16pSfc/jhh6cHHngg3X777bn94EEHHZS23Xbb9Oyzz9a16AAAAA0zcENRjx490n777ZcGDhyYB3RoSN98802af/75c3had911c4eqbt26pVtuuSVtv/32+TlR67TMMsuk559/vnJ0vaKobYpbsXNWlNnADbQaBm5gBhm4AYBZ1ehaDtxQ55QzduzYtPPOOzd4QApR2DDPPPPk/1999dU0YcKE1K9fv8rnLL300mmRRRbJIWlqTfjig5dvEZAAAABqq85JZ++9985N3xpa9G067LDD0lprrZWWW265PO2rr77Kzfq6du1a5bnRHykeq0nUcEXYKt8+++yzBi8rAADQetW5T1LU1Gy++ebpoYceSssvv3xq3759lccvvPDCehUk+ia99dZbeUCIGdGxY8d8AwAAmGkh6eGHH64cga76wA31EYMx3H///empp55KCy+8cOX07t27p/Hjx6cffvihSm3S119/nR8DAABo8pB0wQUXpGuvvTbtueeeDdJ5+OCDD0533XVXeuKJJ1KvXr2qPL7yyivnmqrBgwen7bbbLk8bNmxY+vTTT9Maa6wxw/MHAACY4ZAUTdmi31BDiCZ2MXLdPffck+aaa67KfkYx4EJcNyn+jz5QRxxxRB7MIUagiFAVAammke0AAABm+sANhx56aPqf//mf1BCuuOKKPLjCeuutlxZccMHK2z/+8Y/K51x00UW5D1TUJMWw4NHMLi5ACwAA0Cyuk7TNNtukf//73/lCsssuu+wUAzc0twBT27HQZxbXSWKGuU4SM8h1kgCYVY2uZTaoc3O7GEBh2223ndHyAQAANEt1DknXXXdd45QEAACgJfZJAgAAaM3qXJMUw3RP63pIH3/88YyWCQAAoOWEpMMOO6zK/QkTJqQhQ4akhx56KB199NENWTYAAIDmH5JiCPCaXHbZZemVV15piDIBAAC0/D5Jm266abrzzjsb6u0AAABadki644470jzzzNNQbwcAANAymtutuOKKVQZuiIsSfvXVV+mbb75Jl19+eUOXDwAAoHmHpK233rrK/TZt2qRu3bql9dZbLy299NINWTYAAIDmH5JOPvnkxikJAABAM+BisgAAAPWpSYpmddO6iGyIxydOnFjbtwQAAGi5Iemuu+6a6mPPP/98uvTSS9PkyZMbqlwAAADNOyRttdVWU0wbNmxYOu6449J9992Xdt1113Taaac1dPkAAACaf5+kESNGpH322Sctv/zyuXnd66+/nm644Ya06KKLNnwJAQAAmmtIGjVqVDr22GPTEksskd5+++00ePDgXIu03HLLNV4JAQAAmmNzu3PPPTedc845qXv37unWW2+tsfkdAABAS1dRKpVKtR3dbvbZZ0/9+vVLbdu2nerzBg0alJqT0aNHpy5duuRasM6dOzd1cVLF0Puaugi0dH23bOoS0MLV8mcfAFqd2maDWtck7b777tMdAhwAAKClq3VIuv766xu3JAAAAC11dDsAAIDWSkgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAACaS0h66qmn0hZbbJEWWmihVFFRke6+++4qj5dKpXTSSSelBRdcMM0+++ypX79+6YMPPmiy8gIAAK1fk4akn376KfXt2zdddtllNT5+7rnnpksvvTRdeeWV6cUXX0ydOnVK/fv3T+PGjZvpZQUAAGYN7Zpy5ptuumm+1SRqkS6++OJ0wgknpK222ipPu/HGG9MCCyyQa5x23nnnmVxaAABgVtBs+yQNHz48ffXVV7mJXVmXLl3Saqutlp5//vmpvu6XX35Jo0ePrnIDAABo8SEpAlKImqOiuF9+rCZnnXVWDlPlW48ePRq9rAAAQOvRbENSfQ0cODCNGjWq8vbZZ581dZEAAIAWpNmGpO7du+f/v/766yrT4375sZp07Ngxde7cucoNAACgxYekXr165TA0ePDgymnRvyhGuVtjjTWatGwAAEDr1aSj240ZMyZ9+OGHVQZreP3119M888yTFllkkXTYYYelM844Iy255JI5NJ144on5mkpbb711UxYbAABoxZo0JL3yyitp/fXXr7x/xBFH5P/32GOPdP3116djjjkmX0tp3333TT/88ENae+2100MPPZRmm222Jiw1AADQmlWU4oJErVg00YtR7mIQh+bQP6li6H1NXQRaur5bNnUJaOFa+c8+AMxwNmi2fZIAAACagpAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABS0K94BAIDqKioqmroItHClUim1JGqSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAICCdsU7AEDrUzH0vqYuAkCLoiYJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAaGkh6bLLLks9e/ZMs802W1pttdXSSy+91NRFAgAAWqlmH5L+8Y9/pCOOOCKdfPLJ6bXXXkt9+/ZN/fv3TyNHjmzqogEAAK1Qsw9JF154Ydpnn33SgAEDUu/evdOVV16Z5phjjnTttdc2ddEAAIBWqF1qxsaPH59effXVNHDgwMppbdq0Sf369UvPP/98ja/55Zdf8q1s1KhR+f/Ro0enZmHM2KYuATCLaza/h8w89j1AExvdTPY95XKUSqWWG5L++9//pkmTJqUFFligyvS4/95779X4mrPOOiudeuqpU0zv0aNHo5UToCXp0qVLUxcBgFlMl2a27/nxxx+nWaZmHZLqI2qdog9T2eTJk9N3332X5p133lRRUdGkZYOGOPsRgf+zzz5LnTt3buriADALsO+hNYkapAhICy200DSf16xD0nzzzZfatm2bvv766yrT43737t1rfE3Hjh3zrahr166NWk6Y2WInZUcFwMxk38OsVKvVrAdu6NChQ1p55ZXT4MGDq9QMxf011lijScsGAAC0Ts26JilE07k99tgjrbLKKmnVVVdNF198cfrpp5/yaHcAAACzXEjaaaed0jfffJNOOumk9NVXX6UVVlghPfTQQ1MM5gCzgmhKGtcMq96kFAAai30Ps6KK0vTGvwMAAJiFNOs+SQAAADObkAQAAFAgJAEAABQISQAAAAVCEgDALKKioiLdfffdTV0MaPaEJKhBDDu///77p0UWWSQPedq9e/fUv3//9Oyzz1Z53nPPPZc222yzNPfcc6fZZpstLb/88unCCy9MkyZNmmV2Suutt1467LDDmroYAC1CU+9fvvzyy7Tpppum1ug///lPXh6vv/56UxeFVqDZXycJmsJ2222Xxo8fn2644Ya02GKLpa+//joNHjw4ffvtt5XPueuuu9KOO+6YL2z8+OOPp65du6bHHnssHXPMMen5559P//znP/OPdXMQn6VDhw5VpsWONsrXpo1zJQCzyv4lQllTiivPxP6nXbt2091PQZOK6yQB/7/vv/8+rh1WeuKJJ6b6nDFjxpTmnXfe0rbbbjvFY/fee29+/W233VY5Le7fdddddSrDvvvuW5p//vlLHTt2LC277LKl++67r/LxO+64o9S7d+9Shw4dSosuumjp/PPPr/L6mHbaaaeV/vCHP5Tmmmuu0h577FG67rrrSl26dCndc889pWWWWabUtm3b0vDhw0vjxo0rHXnkkaWFFlqoNMccc5RWXXXV0uOPP17l/Z555pnSb3/729Lss89e6tq1a2njjTcufffdd/l947MVb/GeADTP/Uvx+fF7HffvvPPO0nrrrZd/4/v06VN67rnnarUPCLEPOfjgg0vdunXL+6u11lqr9NJLL1W+NvYnMY9//etfpZVWWqnUvn37PC3e78ADDywdeuih+fPG/MObb75Z2mSTTUqdOnXK+8Dddtut9M0331S+36RJk0rnnHNOafHFF8/7wB49epTOOOOMys9WvMU8oL6cQoZq5pxzznyL5gu//PJLjc955JFH8lm/o446aorHtthii/TrX/863XrrrfWa/+TJk3NTiGh68fe//z2988476eyzz05t27bNj7/66qv5DOPOO++c3nzzzXTKKaekE088MV1//fVV3uf8889Pffv2TUOGDMmPh7Fjx6ZzzjknXX311entt99O888/fzrooIPymcnbbrstDR06NO2www5pk002SR988EF+TTRb2HDDDVPv3r3z85555pn8GeNM4CWXXJLWWGONtM8+++QmHHHr0aNHvT43QGvX1PuXqfnzn/+c5xe/9/H+u+yyS5o4ceJ09wEharfuvPPOXDP22muvpSWWWCI3H/zuu++qzOO4447L+7J333039enTJ0+L10TtUezvrrzyyvTDDz+kDTbYIK244orplVdeSQ899FCuaYt9XtnAgQPz+8R+LfaPt9xyS1pggQXyYy+99FL+P2rdYn80aNCgBl1OzGLqHa+gFYuamrnnnrs022yzldZcc83SwIEDS2+88Ubl42effXY+SxVnBWuy5ZZb5tqa+pzpe/jhh0tt2rQpDRs2rMbHf//735c22mijKtOOPvroXLNUrEnaeuutqzwnapKiHK+//nrltE8++STXKH3xxRdVnrvhhhvmzxx22WWXfGZwauJMXZwJBKB571+mVpN09dVXVz7+9ttv52nvvvvudPcBUesVNUM333xz5bTx48fnlgnnnntulZqku+++e4p9x4orrlhl2umnn55rqYo+++yz/PrYJ44ePTrXVv3tb3+rsTzlzzNkyJBaLw+YGjVJMJU24yNGjEj33ntvrlV54okn0korrTRFbc3/7W9qVt+21XHWbuGFF85n82oSZ+HWWmutKtPiftT8FDv0rrLKKjWWqXwGL0RNVLwm5lU+wxm3J598Mn300UdVziIC0LL3L1NT3C8suOCC+f+RI0dOdx8Q+4kJEyZU2Se1b98+rbrqqnlfVVTTPmnllVeucv+NN97IfbCK+6Oll166cl7xnlEDZ5/EzGDgBpiKGE1oo402yreo1v/jH/+YTj755LTnnnumJZdcMj8nfrDXXHPNKV4b01dYYYV6zXf22WdPDaFTp041vnexs++YMWNyM75owlduzlcWO6eGLA8ATbt/mZoINmXlfUQ0/W7sfVL1abFPiqZ80Sy8ughvH3/8cYOUBWpDTRLUUrTH/umnn/Lf0d56nnnmSRdccMEUz4uzg1GrEzu7+p7R+/zzz9P7779f4+PLLLPMFEPFxv2oDaoedKYn2n1HTVKcMYx25MVbeQSkKE+MvDStM5rVh6QFoPntX+pjWvuAxRdfvLJPUVnULL388sv5M9VV1KhFf9mePXtOsU+KQBUBMkLb1MpTrmGzT6IhCElQTXSYjY6jMWhCDGQwfPjwdPvtt6dzzz03bbXVVvk58WN91VVXpXvuuSftu++++XlxfYZrrrkm77xiIIO4vkVRvE80WyjeyjvFot/+9rdp3XXXzU0yHn300fy6Bx98MHdgDUceeWTeQZx++uk5SEXH17/+9a81dvKdnghWu+66a9p9991zB9eYV3R8Peuss9IDDzxQ2Uk2dngHHHBA/pzvvfdeuuKKK9J///vf/HjszF588cX8+WNa+ewjAM1r/1If09oHRFnjmk9HH3103kfFQApRvhgkaO+9967zvA488MA84EMMHBHzjCZ2Dz/8cB4KPYJP1MAde+yxebCIG2+8MT/+wgsv5GUTYjCiCFHlAR9GjRrVIMuAWdRUeyvBLCqGMz3uuOPyUKUxZHYMi73UUkuVTjjhhNLYsWOrPPepp54q9e/fv9S5c+fKIUdjaNLqqg9LWr49/fTTNZbh22+/LQ0YMCAPixqde5dbbrnS/fffP8UQ4NFhdpFFFimdd955VV4fAzdcdNFFVaaVhwCvLjrZnnTSSaWePXvm91twwQVL22yzTWno0KGVz4nhaqODcXSYjeFf4zOXOxVHZ9rVV189Dw1rCHCA5r1/qWnghuJAB+VhyouXgpjWPuDnn3/OQ4DPN9980xwCvPpAFFMb9Of999/P+6CYT+xXll566dJhhx1Wmjx5cuUQ4DHkd+znyvvAM888s/L1MahDDAseAyAZApwZURH/NHVQg9Zg3Lhx+UzgZ599lgc+6NatW1MXCYBWwP4FZj4hCRp4R3bxxRfndtPRXA4AGoL9C8xcQhIAAECBgRsAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJABqpWfPnnkIYppu2fznP/9JFRUV6fXXX2/0eQHMyoQkgFZsvfXWS4cddtgU06+//vrUtWvXOr3Xyy+/nPbdd9/K+3Gwfvfdd0/1+TGPeM60bnHQXx+1Lf/UyjDbbLOl5m7PPfdMW2+9dZVpPXr0SF9++WVabrnlmqxcALOCdk1dAABahm7dutXp+TvttFPaZJNNKu9vu+22+eD+tNNOq/d71kfnzp3TsGHDqkyLoNQStW3bNnXv3r2piwHQ6qlJAqCy1uL8889PCy64YJp33nnTgQcemCZMmFBjk7L4O2yzzTY5cJTvF80+++z5gL5869ChQ5pjjjkq70dtzp/+9KcclCLIbLDBBumNN96ofH38vf7666e55porP77yyiunV155JT3xxBNpwIABadSoUZU1Q6eccspUP1s8XixH3BZYYIH82P/+7/+mhRZaKE2ePLnKa7baaqu011575b8/+uijfD9eM+ecc6bf/OY36bHHHqtTk7gffvghT4uyh0mTJqW999479erVKy+npZZaKl1yySWVz4/Pc8MNN6R77rmn8jPGa2t67yeffDKtuuqqqWPHjnndHXfccWnixIlVahMPOeSQdMwxx6R55pknf/5pLS8AhCQA/p/HH388B4L4Pw7Qo6la3KbW9C5cd911uflX+X5d7LDDDmnkyJHpwQcfTK+++mpaaaWV0oYbbpi+++67/Piuu+6aFl544fze8Xgc/Ldv3z6tueaaOaxFcIp5x+2oo46q12eOMnz77bf5M5fF/B966KE8/zBmzJi02WabpcGDB6chQ4bk2rEtttgiffrpp6m+IpTFZ7v99tvTO++8k0466aR0/PHHp3/+85/58fg8O+64Y55X+TPG567uiy++yGWL4Bah8oorrkjXXHNNOuOMM6o8L9Znp06d0osvvpjOPffcXJv36KOP1rv8AK2d5nYAZHPPPXf661//mpt0Lb300ul3v/tdDgb77LPPFM8tN5OLfkH1af71zDPPpJdeeimHpKgBCVGLFX2c7rjjjtz3KULI0UcfncsSllxyycrXd+nSpbKGaHqixilqgIrWWWedHM7iM2+66abplltuyQEtxPznm2++XIsV+vbtm29lp59+errrrrvSvffemw466KBUHxH2Tj311Mr7UaP0/PPP55AU4SjKGzVMv/zyyzQ/4+WXX577KcV6i+URy2rEiBHp2GOPzcGrTZv/Oxfap0+fdPLJJ1cux3h+rNuNNtqoXuUHaO2EJACyZZddNgeksmi69eabbzbKvKLWI2poollf0c8//5xrs8IRRxyR/vjHP6abbrop9evXL9f6LL744nWeVzTXe+2116pMiwBSFjVGEQQjcERgu/nmm9POO+9cGTCinNE87YEHHsg1OtGULco5IzVJ4bLLLkvXXnttfp94v/Hjx6cVVlihTu/x7rvvpjXWWKNKH6u11lorl/nzzz9PiyyySGVIKop1GwEVgJoJSQCtWDRJi5qU6qKPTNTGVK/dKIoD7+p9dRpKHMTHgXq5j05RedS6CCa///3vcziJWp+oCbnttttyP6i6iLCzxBJLTPXxaDpXKpXyfKLZ2tNPP50uuuiiysej6Vs0TYuarnifCFjbb799DjVTm1+I9ywr9u0K8TnifS+44IIcciLInXfeebk5XGOYmesWoDUQkgBasRgQ4JFHHplietSs/PrXv57hA+8YgKA+ov/RV199ldq1a1fjoA9lUca4HX744WmXXXbJfaAiJMUgEPWdd3UxgESMvBc1SB9++GFeZlG+smeffTYPbFEOZxHwpjV0ebkpYtQ6rbjiivnv6tc1iveMPkYHHHBA5bRyDVpZbT7jMsssk+68884cyMq1SfHeEbqizxMA9WPgBoBWbP/990/vv/9+Ht1s6NCheSjsCy+8MN16663pyCOPnKH3jnAT/Voi7Hz//fd1em00n4salBhRL0JchI7nnnsu/fnPf84j2EXzs+jvEzVNn3zyST7wjwEcIhSU5x1hJeb/3//+N40dO3aq84oAEWWsfivWpESTu6hJiuZv5QEbyqIPz6BBg3LQiWaCUbs1rVqYqGlaffXV09lnn52bw8XocyeccMIU7xmf8+GHH87r58QTT5xi8Iv4jOV1Fp+xem1UiJD12WefpYMPPji99957eTS8qHGLporlGi0A6s4vKEArtthii6WnnnoqH0BHMFlttdXy4AAxqlrxGkb1EU3FohlaDBxQrjGpraj1+Ne//pXWXXfdPJx31BZFP6AIRDHUdvSNilHndt999/xYDGYQAyyUBzuIWpj99tsvX4spam5ixLapGT16dG7aV/1W7JMTw4/H8NgRSCIEFUWojAEeYp7RNK9///5VappqEmEr+i7FsOVxMd/qo83F0OdRexXlj3USn7VYqxSin1TUaq2yyir5M0ZQrO5Xv/pVXo4xCEYMLhHLJIYWrx7KAKibilKx0TQAAMAsTk0SAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAApP/f/weIN1NK40UeUAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1000x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "unittest_df_pivot.reset_index(inplace=True)\n",
    "\n",
    "# Plotting\n",
    "plt.figure(figsize=(10, 6))\n",
    "\n",
    "# Set the width of each bar\n",
    "bar_width = 0.35\n",
    "\n",
    "# OpenAI brand colors\n",
    "openai_colors = ['#00D1B2', '#000000']  # Green and Black\n",
    "\n",
    "# Get unique runs and unit test evaluations\n",
    "unique_runs = unittest_df_pivot['run'].unique()\n",
    "unique_unit_test_evaluations = unittest_df_pivot['unit_test_evaluation'].unique()\n",
    "\n",
    "# Ensure we have enough colors (repeating the pattern if necessary)\n",
    "colors = openai_colors * (len(unique_runs) // len(openai_colors) + 1)\n",
    "\n",
    "# Iterate over each run to plot\n",
    "for i, run in enumerate(unique_runs):\n",
    "    run_data = unittest_df_pivot[unittest_df_pivot['run'] == run]\n",
    "\n",
    "    # Position of bars for this run\n",
    "    positions = np.arange(len(unique_unit_test_evaluations)) + i * bar_width\n",
    "\n",
    "    plt.bar(positions, run_data['Number of records'], width=bar_width, label=f'Run {run}', color=colors[i])\n",
    "\n",
    "# Setting the x-axis labels to be the unit test evaluations, centered under the groups\n",
    "plt.xticks(np.arange(len(unique_unit_test_evaluations)) + bar_width / 2, unique_unit_test_evaluations)\n",
    "\n",
    "plt.xlabel('Unit Test Evaluation')\n",
    "plt.ylabel('Number of Records')\n",
    "plt.title('Unit Test Evaluations vs Number of Records for Each Run')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "786515fa-6841-4820-98f9-aa29ae76cf76",
   "metadata": {},
   "source": [
    "#### Plotting evaluation results\n",
    "\n",
    "We can similarly plot the results of the evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "7228eac7-e0a9-473d-9432-e558bbc91841",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Number of records</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>run</th>\n",
       "      <th>evaluation_score</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">1</th>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">2</th>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      Number of records\n",
       "run evaluation_score                   \n",
       "1   3                                 1\n",
       "    4                                16\n",
       "    5                                33\n",
       "2   3                                 1\n",
       "    4                                15\n",
       "    5                                34"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluation_df_pivot = pd.pivot_table(\n",
    "    run_df,\n",
    "    values='format',\n",
    "    index=['run','evaluation_score'],\n",
    "    aggfunc='count'\n",
    ")\n",
    "evaluation_df_pivot.columns = ['Number of records']\n",
    "evaluation_df_pivot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "b2a18a78-55ec-43f6-9d62-929707a94364",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0kAAAIjCAYAAADWYVDIAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAASy9JREFUeJzt3QecVOW9P/5nKaKCoGJBroi9CyYmtmiigjX2boyi8jMx0Viw4rVhQ42FmCgaY0+MiUaNeK8SNLZYYyFoLFEklghiNIKAIML8X9/nvmb/Z5cFFthlZnff79dr2JkzZ2aeOefM4XzOU05NqVQqJQAAALJ2//cHAACAICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkQRtTU1OTzjvvvIp89mOPPZY/P/5Cc29nd999d2oJPvroo7T//vun7t2753IPGzYstRX//Oc/83e+5ZZbmuT9HnroobTpppumJZdcMr/vZ599llqb1VdfPe2+++6VLga0ekISVEAcEMR/4HO7Pfvss6klu/baa5vsoKepzJ49O912221piy22SMsvv3xaZpll0rrrrpsOP/zwFr+8K7kNx8Hov/71rzme32677dLGG29ckbK1NCeddFIaOXJkGjx4cLr99tvTLrvsMtd56+8runbtmr7zne+k//mf/0lt3SeffJIOPPDAtNRSS6VrrrkmL8vOnTs32+e11P24bQgap0Mj5wOawfnnn5/WWGONOaavvfbaqaWHpBVWWCEdccQRdaZ/+9vfTl988UVaYoklFnuZjj/++HzgtNdee6VDDz00dejQIb355pvpwQcfTGuuuWbacsstF3uZWoMZM2akSy65JP385z+vdFFarD//+c95uzzllFMaNf+OO+6Yw32pVErvvvtuGj58eNpjjz3ytrzzzjuntuqvf/1r+vzzz9MFF1yQ+vfvv9g+tyXux21DMH9CElTQrrvumr7xjW+ktqJdu3a55qESzZkiuB199NHpl7/8ZZ3nomnTxx9/vNjK8tVXX+VarUoExeYQTZtuuOGGXAvSs2fP1JZMnTq1SWoqJk6cmJZddtlGzx81oN///vdrH++3335pww03TD/72c+q8gB32rRpaemll272z4nlGBZkWTbFOm6J+/GWtg1BJWhuB1Vq5syZuVnYkUceOcdzkydPzmGjfOb5yy+/TOecc07abLPNUrdu3fJ/6ttuu2169NFH5/s5UdsTbdzri35L0RSj6Oabb0477LBDWmmllVKnTp3yf6pxBrIo3uvvf/97evzxx2ubc0TTq3n1Sbrrrrty2aOZTNRAxX/e9ZtwRTm7dOmSp++99975/oorrpiXwaxZs+b5HceNG5fPmH7rW9+a47koT3yfoujHEE2g4rvE91x11VXzWdd///vfdQ7IBg4cmFZeeeW8Lvr27ZtuvfXWBvtbXH755TmMrbXWWvn9Xnvttfz8G2+8kfuixHqO94gDrfvvv3+O7WDIkCFpnXXWyfNEv5VtttkmjRo1aq7f94UXXsifW788IZp1xXMPPPBAfhxn3k888cTa7xrLIs4yv/TSS6kxzjzzzLz8ozZpYfue1O8nV972/vGPf+RtIbbpWNdnn312Xo/vv/9+rnmJZkI9evRIV1xxRYOfGeWK8sU88ZvYc88982vre+6553ITt/icOJiPpkdPPfVUnXnKZYp1973vfS8tt9xyeT3MyzvvvJMOOOCAvH7jfaO2stikqdxcK75T1HKWfy8LaoMNNsi/m7Fjx85Ry3fuuefmGo1Yt7169UqnnXZanl7fr3/967T55pvncsZ3i1rfP/3pT3XmiRMNG220UX6vCMTHHnvsHH1+ys0sX3zxxfwe8X6xDkLMG7/jWM4RZAYMGNBgn6EJEybk/V787uKzVllllby+Yxuam/jceL/wzW9+My/HYk32guxjYjnutttuuUlu1Do3hdgHbL311vn3G2WIssytz1xj1kX4y1/+kueL/ULUhkdz4oXV0DZU3j7rL/eG9uPl9R6/j+233z6X/b/+67/SZZddttBlgkoTkqCCJk2alA+8i7doVx86duyY9tlnn3TfffflEFQU0+JA5+CDD64NTb/61a/yf1SXXnppPqCL2pE4Izh69OgmK28Eot69e+eDnjgwjYOuH//4x/kAryzCQBzcrL/++rlPQNz++7//e67vGf8RRz+C9u3bp6FDh+bannvuuScfgNY/gIqD3vhOcaARBx1xMBvlqF87VF+UuXygFGe152XKlCk5YEbzsZ122imfWT3mmGNyoPnggw/yPNFkMJZ1fLc4iPrpT3+aD/ziICvmry/CZbzfD37wg1zeOGiOIBkHza+//no644wz8vQ4kI8AeO+999a+NtZlhKQ48PjFL36Rl+Vqq602zxATYSsOmn7/+9/P8dzvfve7fOBVPlsc3y3Wa5xJjoPgCJ1xEBflaoxoZhQBMmqTPvzww9SUDjrooFzrFgEs+pJdeOGFefuKEBcHYLGtRwCIMj/xxBNzvP6iiy7KoeT000/PzS0jWEYzrFh/xaZucRAav6EIFBdffHHe7uJkwPPPPz/He0boiW0o5ottdV61l3FQHKE0fiNRlunTp+egVl6/8bmxDYX4TuXfy8LsR/7zn//k9VoWyy0+K34n0Ywqtr/Ytq666qq8XIti+zrssMPyPieajsXj+G3HsiluhxGKIhzFthrby/XXX59/IxHki2IfFrUrUcsY6yu23QiCEXTi+0VAiXUZv6dysCmK945lFEEptslYdxHm33vvvbkug/hdxO8rxHeIz/nhD3+4wPuYqOmN30acLIhlF2VZlP14WewXvva1r+WyxbYTzX1jW6rfD6gx6yK8/fbb+QRLbDexPmLdx/4n9isLo6FtaEHF6+NkQ5wwijLF/wHx24smfNAilYDF7uabby7Fz6+hW6dOnWrnGzlyZJ42YsSIOq/fbbfdSmuuuWbt46+++qo0Y8aMOvP85z//Ka288sqlo446qs70eL9zzz239vGAAQNKvXv3nqOMMU/9XcS0adPmmG/nnXeuU5aw0UYblb7zne/MMe+jjz6a3zP+hi+//LK00korlTbeeOPSF198UTvfAw88kOc755xz6pQzpp1//vl13vNrX/taabPNNivNz+GHH55fv9xyy5X22Wef0uWXX156/fXX55gvPjPmu+eee+Z4bvbs2fnvsGHD8jy//vWva5+L77LVVluVunTpUpo8eXKeNm7cuDxf165dSxMnTqzzXv369SttsskmpenTp9d5/6233rq0zjrr1E7r27dv6bvf/W5pQQ0ePLjUsWPH0qefflo7LbaRZZddts420a1bt9Kxxx670NvwX//619LYsWNLHTp0KB1//PG1z8f6j+2grLws4nX11d8my9veD37wgzrb+KqrrlqqqakpXXLJJXW286WWWipvH/W3s//6r/+qXRfh97//fZ7+s5/9rHZ5x7KObbi8bsvb+RprrFHacccd5yjTIYcc0qjlc+KJJ+b5n3zyydppn3/+eX7f1VdfvTRr1qw637+x6yDmHThwYOnjjz/O29QLL7xQ2mWXXfL0n/70p7Xz3X777aV27drV+fxw3XXX5Xmfeuqp/Pitt97K88Vvolim8vIJ8TlLLLFEaaeddqozzy9+8Yv8XjfddFOd9R7T4nOK7rvvvjz9sssuq7NOt9122zrbRazP+t9lYbbJsoXZx5xxxhlNuh9vaN8Z5Yoy7bDDDrXTGrMuQuyv4zOeeOKJ2mmxjuIzTz755CbbhsrfL36789qPF9f7bbfdVmd/06NHj9J+++033zJBNVKTBBUUNTBxdrt4K551i7PZ0QQizv4Xz9bFfMWzwXGGtNzHJc4gf/rpp/mMaNQoNLbZVGNEDUP9s6dRmxPNiuLxgopmYdFsLc60F/sqffe7381nIRsabSlqPoqi1ic+f36iNidqYqLmI85SR+1DNDHp169fnWY3f/jDH/KZ0KjFq6/cFOp///d/cxOuQw45pPa5OPMbZ7yjJiqaGhbF2ehoLlYW6yfODMfZ7ThDXjz7HGex33rrrdoyRbOkODsc0xZEbB9xhj/OmJdFk504c17cduL9o7nZotQCRa1VnP2OGr3x48enpvL//t//q7ONx/Ycx3jRzLFY/vXWW6/BbSBquKLJVFmceY+mW7H+QtSyxnKN5nOx7MvrIfqhxHYRtVPxe5rX9jc38RnRFKrYJC+ackVtRzRfKje5XBg33nhj3p6itiOWySOPPJKb0Q0aNKh2nqg1je07fkfFGo7Yp4RyU9yolY7vGM11o89gQ9v7ww8/nGuzo1lmcZ6okYkmj/V/p9FErn4z4VgeUXvyox/9qM46/clPfjLHPib2ZdGUK/Z1i2ph9jHFMjbFfrz+vjO+V+wvY99V3D83Zl2URVPneH1ZbA9z+x0s7Da0oGL7LvZzivUYv4HGlgmqjYEboILiP5B5dfiNg4o4wL7jjjty87o4+IiD3jj4rd9kJvqfRBOHaBZWbP7S0KhLCyv6aUSTpGeeeWaOZmvxn340OVsQMapSiP/c64sDmGhzXxQHOcWwEaJ5SGMOpuKgI5oLxS0OiOO7XHfddflgJpotPvnkk3m+aJM/vyY2Ue7oI1T/QCYOSovfa27rIJrKxMF+9LGJW0PiwC6alEWTm2imFB2to81/NGeJQNKnT595ljGCXizDCNjlUBH3I3SXD5RD9BmIJk/RpCf6SURfjAgXEXwWxFlnnZWbOEXTuIaaHC6MaFZYFNtXbAPxHepPr9+8KcQ6qn+gGc3zyn0sysGzoSZfxe262ASpsb+n2AaiiWB9xW1kYYdIj+3huOOOy8ElRnSL5lvxeyxuj/Hdoslk/d9L/UEOYnuP18VB97y+S0O/0zgIju2k/vYe2239gUlingiocSBdVP89Yx8XzShPPvnk3N8vmqTGNYFim4wTEwtqQfcxsc+N5sJNuR8P0QcwmhhGMC/2CSuGn8asi7n9NhZkX9jYbWhBxXKrH+aiTGPGjFno94RKEpKgysUBfLT9j4P56FMQ/UziP/c4CC529I326PH8qaeems8Oltvf1+/MXd/cOorXHwwh3ifOrsdnX3nllfmgOg6E4gxx9HOof8a9OcR3agrRpyn6a8Qt+hZFzU8cTJX7LjW14lnkUF5WUZs1t5GkysMHR7+VWPZ//OMfc01Q9D2L5R0Br1jT0pAI0tEXJmoQokYlBoWI2q84ECyL2qw4Ix21a/H+0b8qDlIjjEe/ksaKg+U4ixy1SdHHamG3s/mt77ltA//XimjBlNdDfOfoP9OQ+gf19ddlJcTBaHmI6wi1ERrjgDf6/uy77761322TTTbJv9WGxO+3uSzqMooaq+hHFTUr0acrTiTEvixqX6NfT3OKkLYoQaEhcQIm9jXxW44+VhEWo+Y5arfjBNjCWNTfQWO2oQX9zTblbxOqgZAEVS7+Y43/VKMWIJruxIFC/YEQYpSkOEiNA9vif2xR6zM/caavoRGm6p8dHjFiRD4DGgfaxbOYDY2g19gRusqhJK5XVKzdKE9rrtBSFGeAIyRFM7H4vBiB7tVXX53na2K+ODsaB6LFA6qoxSs/Py/lWpo4UGrM9VzKoxzGLZrzxTYRHekbE5Ki43c0IYyz8jE4QXmwj6LYvqI5UtyihuHrX/96DlcLEpLKtUkR2CNk1Veujam/rdXfzppS/SaKcbAWtXjlWrhY1yGajDX1dXViG4htuL7GbiMLIgYoiOAcyz+aicbvL77b3/72t3xiY16/x5gvtuNo/je3oFj8nRZrGKMWIkaObMyyi/eIJl2x/RaDZ0PLqFyuqE2KW6zHKFvUlMf2tSCqYR8Tv7+oAY3AFyGsLELSgq6L5tLQNlSJ3yxUE32SoMrFQXj0pYiQEs2Zoq9R/aZ25TN4xTN20c8kmsXNT/zHHE2Kik0iIjAUR1ib22fE6+r/Rx9ilLaGgldDASVqvaJWpNgEJWrNoqlQ9BtoCjGkcEN9QOIgLw7cYhmXa26iqV0cXNb//sXvHmde4z2LfcVivcQIYnEAGP205iW+c9RgRQ1hQ314itdtqt+MLN4/ytrQMM4NNe2K2oQoZ9wiDEXAKp4Rrt+XLMoWI5g15v0b2paiNim+Vyyfoggicba6/ih0cWa9ucSQyNHnq3gyIZZ3OfxF88Ioc4xiFgfv9S3K9bNiG4nR8Yq/wejrFDVtMdx6Y5pUNVbUDEaYiN9M1DiWawijX1uMOlhfjO4XZQlR+xzbfzTrrF8bXN7eIwRFrfHVV19d5/cf/Vpi+2nM7zSWR/xGipcMiO2v/kWIo8lXjAJYFOsoakIXZptcXPuYeYl9Z4SOYg1MNPmMmrKixqyL5tLQNlQ+iVD8zcZ3mN9ootBaqEmCCor/qMtnloti6ODiGdsIRXEwETVDcdBb7tdQFm32oxYpzgDGf/pxdjcOCuJArKGDv6KoWYhhWuO1MfBAHKTEgUz0gSl2Ko6hfuNAKZrBxFnHeN84AIsDkPoH+nHwGe8RbfDjgD7mqX8Wt1yTErUOUUMSwSKagsXQydGnJQ4k41pFTSGGGo5+A1GGOLMefRuixuS3v/1tDkTRvKfczyWaK8bBdAzPe9RRR+XvEgMtRA1aLNNo5hid7yMIRBPHuB5MlDVeE/2cYsjj4mAB8+rsHTWDsT6jA3ys7/jucVAd5Y1yhViHEaiiHFGjFB3R47OiaUxjxLYTHcHjTHb0TSrWfEWAiGY3EcLje0UAi0760Udhbtcemp+o5YwwH2fp45o6RVHzFX2W4m8cvMbBV1wLqbnE8oplHNtXLNtYN7E9lofujmURzRcjNEVZY77oTxPhImpII9jFyYmFEU0OY/uK947fVZQl+g3GbzNqFpq6SVdsi7Ge4/cUB9vRby2a5sZAE/Fd4hphcYAb+5uYHrUasQ5iecQ6u+CCC3Kzy2hqFbUdsQ1EWI5mbtGvKS4WHLWS0Scumo7F+o2AG9ckKnbWn5vYb0QZYrlEQIjtOvZZ9UN6bA/xG42QF/PEwXucsIj111At6Pwsjn3M/PbjsU+OZo+x7GKQkNj3xO8/ln3x5FRj1kVzqr8NxW8i+oTFuo99YGzDd955Zw670CZUeng9aIvmNXRsQ8Mkx/CvvXr1ys9deOGFc7xfPH/xxRfnoWFjGNgYFjuGuG1oeO/6wy2HP/3pT3k42hjmd7311stDWzc0BPj9999f6tOnT2nJJZfMwxhfeumlefjf+sPETpgwIQ9bvcwyy+TnysOBNzR0bPjd736XyxxlX3755UuHHnpo6YMPPqgzT3yXzp07z/HdGypnfTEMdAz7HEM9xzDSMTR2lC2G7L7hhhvqDK8bPvnkk9Jxxx2Xh5COZRKvic//97//XTvPRx99VDryyCNLK6ywQp4nhvOuv97Kw17PbTjjGDo7hiaPYXKjTPF5u+++e+nuu++unSfW9+abb56H7o6hrtdff/3SRRddlIcQbowYVri8Xf3lL3+p81wM0XvqqafmYcZjecTyjfvXXnvtQg23XH8o5eIQ4OVhkGPo4Rh2PD7vwAMPzEMQz20I8BiiuDHbQP3hxsvb2W9/+9s8FHoMAR3LLrbJd999d47Xv/zyy6V999231L1797wNxm8myvbII4/Mt0zzEut3//33z+sufjOxHuN3Wd+CDgE+t3nPO++8OYbYj99oLJv4XjH8fQyXP2TIkNKkSZPqvDZ+x+XfYMwXy3TUqFF15okhv2P7i201Li/wox/9KA/ZPa91Uf93ddhhh+Uh8WMbiPux7Iv7vPiNxfeLz4l1HfNtscUWefj2RdkmF2Uf0xT78RtvvDEPNx+fH98tnpvbvmt+6yK2z4YuCxDzNXTphUXZhmIb7t+/fy5LrPMzzzwzl6WhIcAbWu9zu8QEtAQ18U+lgxoAAEC10CcJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAAGhLF5ONq1Z/+OGH+eKOccVrAACgbSqVSvli6nGR5nld2LvVh6QISL169ap0MQAAgCrx/vvvp1VXXbXthqSoQSoviK5du1a6OAAAQIVMnjw5V6CUM0KbDUnlJnYRkIQkAACgZj7dcAzcAAAAUCAkAQAAVEtIGj58eOrTp09tU7itttoqPfjgg7XPb7fddrkqrHg75phjKllkAACglaton6QYUeKSSy5J66yzTh6O79Zbb0177bVXevnll9NGG22U5zn66KPT+eefX/uapZdeusnLEZ/91VdfpVmzZjX5e7d17du3Tx06dDD8OgAALUZFQ9Iee+xR5/FFF12Ua5eeffbZ2pAUoahHjx7NVoYvv/wyjR8/Pk2bNq3ZPqOti3W4yiqrpCWWWKLSRQEAgPmqmtHtohbnrrvuSlOnTs3N7sp+85vfpF//+tc5KEWoOvvss+dZmzRjxox8Kw7zN68LzY4bNy7XdsQFpeIgXo1H09bQRQj9+OOP83KOGsN5XbQLAACqQcVD0iuvvJJD0fTp01OXLl3SvffemzbccMP83Pe+973Uu3fvHGDGjBmTTj/99PTmm2+me+65Z67vN3To0DRkyJBGfXYcwEdQirHSm6MZHykttdRSqWPHjundd9/Ny3vJJZesdJEAAGCeakpxur+C4sD5vffeS5MmTUp33313+tWvfpUef/zx2qBU9Oc//zn169cvvf3222mttdZqdE1ShKB4//rXSYpgFjUca6yxhoP3ZmQ5AwBQDSIbdOvWrcFsUFU1SdHEbe211873N9tss/TXv/41/exnP0vXX3/9HPNuscUW+e+8QlKnTp3yDQAAYGFUXQeRaP5WrAkqGj16dP4bgwAAAAA0h4rWJA0ePDjtuuuuabXVVkuff/55uuOOO9Jjjz2WRo4cmcaOHZsf77bbbql79+65T9JJJ52Uvv3tb+drKzWnmjEj0uJU6lN3lD8AAKCN1iRNnDgxHX744Wm99dbLfY2iqV0EpB133DE3w3v44YfTTjvtlNZff/108sknp/322y+NGLF4A0w1OuKII2ovrhuDIkRfn9NOOy33/WluMWhGrJMIrvH55do9AABoLSpak3TjjTfO9bkYbCEGcKBhu+yyS7r55pvTzJkz04svvpgGDBiQQ8ull17arJ8bQ7Rvs8026cADD8wX+gUAgNam6vok0TgxOEVcOyrC5N5775369++fRo0aVfv86quvnoYNG1bnNZtuumk677zzah9HqIrRBPfZZ588BHpcx+j++++f5+cedthh6ZxzzsmfBwAArZGQ1Aq8+uqr6emnn85NFBdUXFMqaoWiz1f0/zr00EPTp59+2izlBACAlkBIaqEeeOCBfPHduO7QJptskvt3nXrqqQvVv+mQQw7Jw7BffPHFacqUKen5559vljIDAEBLUPHrJLFwtt9++zR8+PDcR+iqq65KHTp0yANbLKjiSIGdO3fOF9WKwAUAAG2VmqQWKgJN1P707ds33XTTTem5556rMxBGu3btUqlUqvOaGOShvhgdryj6KcW1qgAAoK0SklqBCERnnnlmOuuss9IXX3yRp6244opp/PjxtfNMnjw5jRs3roKlBACAlkFIaiUOOOCA1L59+3TNNdfkxzvssEO6/fbb05NPPpleeeWVPER4PL+oYlCHuDbSa6+9lh+/+eab+fGECRMW+b0BABaX8jUn3RbPraXRJ6kBpT57pJYm+iQdd9xx6bLLLks/+tGP0uDBg3PN0e677566deuWLrjggiapSYohwo888sjaxwcffHD+e+6559YZXhwAAFqqmlL9jiutTDQzi5AwadKkPChB0fTp03NwWGONNfIocTQPyxkAqDYtsXajJStVSeSYVzYo0twOAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoKBD8QGVuQJztVyBGAAAEJJapCOOOCLdeuut+X6HDh3Sqquumg444IB0/vnnpyWXXLLZPnfmzJnprLPOSv/7v/+b3nnnndStW7fUv3//dMkll6SePXs22+cCQFtRM2ZEpYsAaG7Xcu2yyy5p/PjxOaxcddVV6frrr0/nnntus37mtGnT0ksvvZTOPvvs/Peee+5Jb775Ztpzzz2b9XMBAGBxEpJaqE6dOqUePXqkXr16pb333jvX6IwaNar2+dVXXz0NGzaszms23XTTdN5559VpVvirX/0q7bPPPmnppZdO66yzTrr//vvn+plRcxSfceCBB6b11lsvbbnllukXv/hFevHFF9N7773XTN8UAAAWLyGpFXj11VfT008/nZZYYokFfu2QIUNy6BkzZkzabbfd0qGHHpo+/fTTRr9+0qRJOWwtu+yyC/zZAABQjYSkFuqBBx5IXbp0yX2QNtlkkzRx4sR06qmnLlT/pkMOOSStvfba6eKLL05TpkxJzz//fKNeO3369HT66afn13ft2nUhvgUAAFQfAze0UNtvv30aPnx4mjp1au6TFAM47Lfffgv8Pn369Km937lz5xx2InA1ZhCHqIGKkfmiHAAA0FqoSWqhItBE7U/fvn3TTTfdlJ577rl044031j7frl27OYYWj2BTX8eOHes8jqZzs2fPblRAevfdd3MfJbVIAAC0JkJSKxCB6Mwzz8zDc3/xxRd52oorrphHvyubPHlyGjdu3CJ/VjkgvfXWW+nhhx9O3bt3X+T3BACAaiIktRJxnaT27duna665Jj/eYYcd0u23356efPLJ9Morr6QBAwbk5xc1IO2///7phRdeSL/5zW/SrFmz0oQJE/Ltyy+/bKJvAgAAlaVPUgPqN1NrCaJP0nHHHZcuu+yy9KMf/SgNHjw41xztvvvueejuCy64YJFrkv71r3/VDhEew4kXPfroo2m77bZbpPcHAIBqUFNqiYlgAUQzswgJMVR1/b4zMTpbBIc11lgjjxJH87CcAaBxasaMqHQR2o6+e1a6BG1KqUoix7yyQZHmdgAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJVdSRrLWyfAEAaEnadEjq2LFj/jtt2rRKF6VVKy/f8vIGAIBq1qavkxQXV1122WXTxIkT8+Oll1461dTUVLpYraoGKQJSLN9Yzot6MVsAAFgc2nRICj169Mh/y0GJphcBqbycAQCg2rX5kBQ1R6usskpaaaWV0syZMytdnFYnmtipQQIAoCVp8yGpLA7kHcwDAABteuAGAACA+oQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAaglJw4cPT3369Eldu3bNt6222io9+OCDtc9Pnz49HXvssal79+6pS5cuab/99ksfffRRJYsMAAC0chUNSauuumq65JJL0osvvpheeOGFtMMOO6S99tor/f3vf8/Pn3TSSWnEiBHprrvuSo8//nj68MMP07777lvJIgMAAK1cTalUKqUqsvzyy6ef/vSnaf/9908rrrhiuuOOO/L98MYbb6QNNtggPfPMM2nLLbds8PUzZszIt7LJkyenXr16pUmTJuXaKgCAalUzZkSli9B29N2z0iVoU0pVEjkiG3Tr1m2+2aBq+iTNmjUr3XnnnWnq1Km52V3ULs2cOTP179+/dp71118/rbbaajkkzc3QoUPzFy/fIiABAAA0VsVD0iuvvJL7G3Xq1Ckdc8wx6d57700bbrhhmjBhQlpiiSXSsssuW2f+lVdeOT83N4MHD87JsHx7//33F8O3AAAAWosOlS7Aeuutl0aPHp0Dzd13350GDBiQ+x8trAhbcQMAAGiRISlqi9Zee+18f7PNNkt//etf089+9rN00EEHpS+//DJ99tlndWqTYnS7Hj16VLDEAABAa1bx5nb1zZ49Ow+8EIGpY8eO6ZFHHql97s0330zvvfde7rMEAADQ6mqSov/Qrrvumgdj+Pzzz/NIdo899lgaOXJkHnRh4MCBadCgQXnEuxh94ic/+UkOSHMb2Q4AAKBFh6SJEyemww8/PI0fPz6HoriwbASkHXfcMT9/1VVXpXbt2uWLyEbt0s4775yuvfbaShYZAABo5aruOkmVGgsdAKDSXCdpMXKdpMWqVCWRo8VdJwkAAKAaCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAAFRLSBo6dGj65je/mZZZZpm00korpb333ju9+eabdebZbrvtUk1NTZ3bMcccU7EyAwAArVtFQ9Ljjz+ejj322PTss8+mUaNGpZkzZ6addtopTZ06tc58Rx99dBo/fnzt7bLLLqtYmQEAgNatQyU//KGHHqrz+JZbbsk1Si+++GL69re/XTt96aWXTj169KhACQEAgLamqvokTZo0Kf9dfvnl60z/zW9+k1ZYYYW08cYbp8GDB6dp06bN9T1mzJiRJk+eXOcGAADQImqSimbPnp1OPPHE9K1vfSuHobLvfe97qXfv3qlnz55pzJgx6fTTT8/9lu6555659nMaMmTIYiw5AADQmtSUSqVSqgI/+tGP0oMPPpj+8pe/pFVXXXWu8/35z39O/fr1S2+//XZaa621GqxJiltZ1CT16tUr11J17dq12coPALCoasaMqHQR2o6+e1a6BG1KqToiR84G3bp1m282qIqapOOOOy498MAD6YknnphnQApbbLFF/ju3kNSpU6d8AwAAWBgdKp0of/KTn6R77703PfbYY2mNNdaY72tGjx6d/66yyiqLoYQAAEBbU9GQFMN/33HHHemPf/xjvlbShAkT8vSoAltqqaXS2LFj8/O77bZb6t69e+6TdNJJJ+WR7/r06VPJogMAAK1URUPS8OHDay8YW3TzzTenI444Ii2xxBLp4YcfTsOGDcvXToq+Rfvtt18666yzKlRiAACgtat4c7t5iVAUF5wFAABok9dJAgAAqDQhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIACIQkAAKBASAIAACgQkgAAAAqEJAAAgAIhCQAAoEBIAgAAKBCSAAAACoQkAACAAiEJAACgQEgCAAAoEJIAAAAKhCQAAIBFCUlffPFFmjZtWu3jd999Nw0bNiz96U9/WtC3AgAAaPkhaa+99kq33XZbvv/ZZ5+lLbbYIl1xxRV5+vDhw5ujjAAAANUbkl566aW07bbb5vt33313WnnllXNtUgSnq6++ujnKCAAAUL0hKZraLbPMMvl+NLHbd999U7t27dKWW26ZwxIAAECbCklrr712uu+++9L777+fRo4cmXbaaac8feLEialr167NUUYAAIDqDUnnnHNOOuWUU9Lqq6+e+yNttdVWtbVKX/va15qjjAAAAItNhwV9wf7775+22WabNH78+NS3b9/a6f369Uv77LNPU5cPAACgukNS6NGjR74Vbb755k1VJgAAgOoOSTE4Q2Pdc889i1IeAACA6u+T1K1bt9pbDM7wyCOPpBdeeKH2+RdffDFPi+cBAABafU3SzTffXHv/9NNPTwceeGC67rrrUvv27fO0WbNmpR//+MdGtwMAANre6HY33XRTHt2uHJBC3B80aFB+DgAAoE2FpK+++iq98cYbc0yPabNnz26qcgEAALSM0e2OPPLINHDgwDR27NjaEe2ee+65dMkll+TnAAAA2lRIuvzyy/Pw31dccUW+VlJYZZVV0qmnnppOPvnk5igjAABAdYakaGp3xx13pAEDBqTTTjstTZ48OU83YAMAANAm+yR16NAhHXPMMWn69Om14UhAAgAA2vTADdEP6eWXX26e0gAAALS0PklxPaToe/TBBx+kzTbbLHXu3LnO83369GnK8gEAAFR3SDr44IPz3+OPP752Wk1NTSqVSvlvXFgWAACgzYSkcePGNU9JAAAAWmJI6t27d/OUBAAAoCWGpBAXkh02bFh6/fXX8+MNN9wwnXDCCWmttdZq6vIBAABU9+h2I0eOzKHo+eefz4M0xO25555LG220URo1alTzlBIAAKBaa5LOOOOMdNJJJ6VLLrlkjumnn3562nHHHZuyfAAAANVdkxRN7AYOHDjH9KOOOiq99tprTVUuAACAlhGSVlxxxTR69Og5pse0lVZaqanKBQAA0DKa2x199NHpBz/4QXrnnXfS1ltvnac99dRT6dJLL02DBg1qjjICAABUb0g6++yz0zLLLJOuuOKKNHjw4DytZ8+e6bzzzqtzgVkAAICWqKZUKpUW9sWff/55/huhqVpNnjw5devWLU2aNCl17dq10sUBAJirmjEjKl2EtqPvnpUuQZtSWvjIUZFssMA1SePGjUtfffVVWmeddeqEo7feeit17Ngxrb766gtfagAAgJY2cMMRRxyRnn766Tmmx7WS4jkAAIA2FZJefvnl9K1vfWuO6VtuuWWDo94BAAC06pBUU1NT2xepKNr1zZo1a4Hea+jQoemb3/xmbrYXw4fvvffe6c0336wzz/Tp09Oxxx6bunfvnrp06ZL222+/9NFHHy1osQEAAJonJH3729/O4aYYiOJ+TNtmm20W6L0ef/zxHICeffbZNGrUqDRz5sy00047palTp9bOc9JJJ6URI0aku+66K8//4Ycfpn333XdBiw0AANA8o9u99tprOSgtu+yyadttt83TnnzyyTxSxJ///Oe08cYbp4X18ccf5xqlCEPxGVE7FRevveOOO9L++++f53njjTfSBhtskJ555pncxG9+jG4HALQURrdbjIxut1iVWtjodgtck7ThhhumMWPGpAMPPDBNnDgxN707/PDDc3hZlIAUorBh+eWXz39ffPHFXLvUv3//2nnWX3/9tNpqq+WQ1JAZM2bkL1+8AQAANNYCDwFevnjsxRdfnJrS7Nmz04knnpgHhSiHrQkTJqQlllgi11oVrbzyyvm5hkSzvyFDhjRp2QAAgLZjgWuSys3rvv/976ett946/etf/8rTbr/99vSXv/xloQsSfZNeffXVdOedd6ZFMXjw4FwjVb69//77i/R+AABA27LAIekPf/hD2nnnndNSSy2VXnrppdy8LUQgWdjapeOOOy498MAD6dFHH02rrrpq7fQePXqkL7/8Mn322Wd15o/R7eK5hnTq1Cm3LyzeAAAAmi0kXXjhhem6665LN9xwQ+rYsWPt9GgmF6FpQTtwRUC6995786APa6yxRp3nN9tss/wZjzzySO20GCL8vffeS1tttdWCFh0AAKDp+yRFSImR5+qLUSLq1/g0poldjFz3xz/+MV8rqdzPKN4raqri78CBA9OgQYPyYA5RK/STn/wkB6TGjGwHAADQ7CEpmrm9/fbbafXVV68zPfojrbnmmgv0XsOHD89/t9tuuzrTb7755nTEEUfk+1dddVVq165dvohsNO2Lpn7XXnvtghYbAACgeULS0UcfnU444YR00003pZqamnxx1xiO+5RTTklnn312k4+XvuSSS6Zrrrkm3wAAAKouJJ1xxhl5uO5+/fqladOm5aZ3MVhChKRoCgcAANCS1ZQW8vK3MepcNLubMmVKvsBsly5d0hdffJH7ElWTxl5VFwCg0mrGjKh0EdqOvntWugRtSmnhIkfFssFCXScpxEVeIxxtvvnmeQS6K6+8co7R6QAAAFqaRoekGDQhLtT6jW98I19E9r777qsdZCHCUQywcNJJJzVnWQEAAKqnT9I555yTrr/++tS/f//09NNPpwMOOCAdeeSR6dlnn821SPG4ffv2zVtaAACAaglJd911V7rtttvSnnvumV599dXUp0+f9NVXX6W//e1veZQ7AACANtXc7oMPPkibbbZZvr/xxhvnEe2ieZ2ABAAAtMmQNGvWrDxYQ1mHDh3yiHYAAABtsrldDNt3xBFH5BqkMH369HTMMcekzp0715nvnnvuafpSAgAAVFtIGjBgQJ3H3//+95ujPAAAAC0jJMVQ3wAAAK3dQl9MFgAAoDUSkgAAAAqEJAAAgAIhCQAAYEFD0te//vX0n//8J98///zz07Rp0xrzMgAAgNYZkl5//fU0derUfH/IkCFpypQpzV0uAACA6h0CfNNNN01HHnlk2mabbfJFZS+//PLUpUuXBuc955xzmrqMAAAA1RWSbrnllnTuueemBx54INXU1KQHH3wwdegw50vjOSEJAABo9SFpvfXWS3feeWe+365du/TII4+klVZaqbnLBgAAUJ0hqWj27NnNUxIAAICWGJLC2LFj07Bhw/KADmHDDTdMJ5xwQlprrbWaunwAAADVfZ2kkSNH5lD0/PPPpz59+uTbc889lzbaaKM0atSo5iklAABAtdYknXHGGemkk05Kl1xyyRzTTz/99LTjjjs2ZfkAAACquyYpmtgNHDhwjulHHXVUeu2115qqXAAAAC0jJK244opp9OjRc0yPaUa8AwAA2lxzu6OPPjr94Ac/SO+8807aeuut87SnnnoqXXrppWnQoEHNUUYAAIDqDUlnn312WmaZZdIVV1yRBg8enKf17NkznXfeeen4449vjjICAAAsNjWlUqm0sC/+/PPP898ITdVq8uTJqVu3bmnSpEmpa9eulS4OAMBc1YwZUekitB1996x0CdqU0sJHjopkg4W6TlJZNYcjAACAxTJwAwAAQGsmJAEAABQISQAAAAsbkmbOnJn69euX3nrrrQV5GQAAQOsMSR07dkxjxoxpvtIAAAC0tOZ23//+99ONN97YPKUBAACosAUeAvyrr75KN910U3r44YfTZpttljp37lzn+SuvvLIpywcAAFDdIenVV19NX//61/P9f/zjH3Weq6mpabqSAQAAtISQ9OijjzZPSQAAAFryEOBvv/12GjlyZPriiy/y41Kp1JTlAgAAaBkh6ZNPPsnDgK+77rppt912S+PHj8/TBw4cmE4++eTmKCMAAED1hqSTTjopDwX+3nvvpaWXXrp2+kEHHZQeeuihpi4fAABAdfdJ+tOf/pSb2a266qp1pq+zzjrp3XffbcqyAQAAVH9N0tSpU+vUIJV9+umnqVOnTk1VLgAAgJYRkrbddtt022231Rn2e/bs2emyyy5L22+/fVOXDwAAoLqb20UYioEbXnjhhfTll1+m0047Lf3973/PNUlPPfVU85QSoJWqGTOi0kVoM0p99qh0EQBorTVJG2+8cb6I7DbbbJP22muv3Pxu3333TS+//HJaa621mqeUAAAA1VqTFLp165b++7//u+lLAwAA0BJD0n/+85904403ptdffz0/3nDDDdORRx6Zll9++aYuHwAAQHU3t3viiSfS6quvnq6++uocluIW99dYY438HAAAQJuqSTr22GPzhWOHDx+e2rdvn6fNmjUr/fjHP87PvfLKK81RTgAAgOqsSXr77bfTySefXBuQQtwfNGhQfg4AAKBNhaSvf/3rtX2RimJa3759m6pcAAAA1dvcbsyYMbX3jz/++HTCCSfkWqMtt9wyT3v22WfTNddcky655JLmKykAAMBiUFMqlUrzm6ldu3appqYmzW/WmCf6JzVWDPTw05/+NL344otp/Pjx6d57701777137fNHHHFEuvXWW+u8Zuedd04PPfRQoz9j8uTJecjySZMmpa5duzb6dQCLg4vJLj4uJktLYJ+wGPXds9IlaFNK848ci0Vjs0GjapLGjRuXmkNciDaa6B111FH5grQN2WWXXdLNN99c+7hTp07NUhYAAIBGh6TevXs3y9Ladddd821eIhT16NGjWT4fAACgSS4m++GHH6a//OUvaeLEiWn27Nl1nos+S03pscceSyuttFJabrnl0g477JAuvPDC1L1797nOP2PGjHwrVqkBAAA0W0i65ZZb0g9/+MO0xBJL5LAS/ZDK4n5ThqRoahfN8OJCtWPHjk1nnnlmrnl65pln6gxBXjR06NA0ZMiQJisDAK1D8f8r2k7/A4BmG7ihqFevXumYY45JgwcPzgM6NOV/XvUHbqjvnXfeSWuttVZ6+OGHU79+/RpdkxRlNnADUI100l6MdNJerISkhWOfsBjZJ7TJfcLkRg7csMApZ9q0aenggw9u0oDUWGuuuWZaYYUV5nnR2ujDFF+4eAMAAGisBU46AwcOTHfddVeqhA8++CB98sknaZVVVqnI5wMAAK3fAvdJij4/u+++e75W0SabbJI6duxY5/krr7yy0e81ZcqUOrVCMdT46NGj0/LLL59v0bdov/32y6PbRZ+k0047La299tr5WkkAAABVE5JGjhyZ1ltvvfy4/sANC+KFF15I22+/fe3jQYMG5b8DBgxIw4cPT2PGjMkXk/3ss89Sz54900477ZQuuOAC10oCAACqJyRdccUV6aabbkpHHHHEIn/4dtttN89OXBHGAAAAqrpPUtTifOtb32qe0gAAALS0kHTCCSekn//8581TGgAAgJbW3O75559Pf/7zn9MDDzyQNtpoozkGbrjnnnuasnwAAADVHZKWXXbZtO+++zZPaQAAAFpaSLr55pubpyQAAAAtsU8SAABAa7bANUlrrLHGPK+H9M477yxqmQAAAFpOSDrxxBPrPJ45c2Z6+eWX00MPPZROPfXUpiwbAABA9YekGAK8Iddcc0164YUXmqJMAAAALb9P0q677pr+8Ic/NNXbAQAAtOyQdPfdd6fll1++qd4OAACgZTS3+9rXvlZn4IZSqZQmTJiQPv7443Tttdc2dfkAAACqOyTtvffedR63a9curbjiimm77bZL66+/flOWDQAAoPpD0rnnnts8JQEAAKgCLiYLAACwMDVJ0axuXheRDfH8V1991di3BAAAaLkh6d57753rc88880y6+uqr0+zZs5uqXAAAANUdkvbaa685pr355pvpjDPOSCNGjEiHHnpoOv/885u6fAAAANXfJ+nDDz9MRx99dNpkk01y87rRo0enW2+9NfXu3bvpSwgAAFCtIWnSpEnp9NNPT2uvvXb6+9//nh555JFci7Txxhs3XwkBAACqsbndZZddli699NLUo0eP9Nvf/rbB5ncAAABtJiRF36Ollloq1yJF07q4NeSee+5pyvIBAABUZ0g6/PDD5zsEOAAAQJsJSbfcckvzlgQAAKCljm4HAADQWglJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAABUS0h64okn0h577JF69uyZampq0n333Vfn+VKplM4555y0yiqrpKWWWir1798/vfXWWxUrLwAA0PpVNCRNnTo19e3bN11zzTUNPn/ZZZelq6++Ol133XXpueeeS507d04777xzmj59+mIvKwAA0DZ0qOSH77rrrvnWkKhFGjZsWDrrrLPSXnvtlafddtttaeWVV841TgcffPBiLi0AANAWVG2fpHHjxqUJEybkJnZl3bp1S1tssUV65pln5vq6GTNmpMmTJ9e5AQAAtPiQFAEpRM1RUTwuP9eQoUOH5jBVvvXq1avZywoAALQeVRuSFtbgwYPTpEmTam/vv/9+pYsEAAC0IFUbknr06JH/fvTRR3Wmx+Pycw3p1KlT6tq1a50bAABAiw9Ja6yxRg5DjzzySO206F8Uo9xttdVWFS0bAADQelV0dLspU6akt99+u85gDaNHj07LL798Wm211dKJJ56YLrzwwrTOOuvk0HT22WfnayrtvffelSw2AADQilU0JL3wwgtp++23r308aNCg/HfAgAHplltuSaeddlq+ltIPfvCD9Nlnn6VtttkmPfTQQ2nJJZesYKkBAIDWrKYUFyRqxaKJXoxyF4M46J8EVJuaMSMqXYS2o++elS5Bm9LKDy+ajX3CYmSf0Cb3CZMbmQ2qtk8SAABAJQhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAtJSSdd955qaamps5t/fXXr3SxAACAVqxDqnIbbbRRevjhh2sfd+hQ9UUGAABasKpPHBGKevToUeliAAAAbURVN7cLb731VurZs2dac80106GHHpree++9ec4/Y8aMNHny5Do3AACAVhGStthii3TLLbekhx56KA0fPjyNGzcubbvttunzzz+f62uGDh2aunXrVnvr1avXYi0zAADQstWUSqVSaiE+++yz1Lt373TllVemgQMHzrUmKW5lUZMUQWnSpEmpa9eui7G0APNXM2ZEpYvQdvTds9IlaFNa0OFFVbFPWIzsE9rkPmHy5Mm5ImV+2aDq+yQVLbvssmnddddNb7/99lzn6dSpU74BAAC0uuZ29U2ZMiWNHTs2rbLKKpUuCgAA0EpVdUg65ZRT0uOPP57++c9/pqeffjrts88+qX379umQQw6pdNEAAIBWqqqb233wwQc5EH3yySdpxRVXTNtss0169tln830AAIA2F5LuvPPOShcBAABoY6q6uR0AAMDiJiQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUNCh+IDmVzNmRKWL0Hb03bPSJWhTSqVSpYsAANAk1CQBAAAUCEkAAAAFQhIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAA0NJC0jXXXJNWX331tOSSS6YtttgiPf/885UuEgAA0EpVfUj63e9+lwYNGpTOPffc9NJLL6W+ffumnXfeOU2cOLHSRQMAAFqhqg9JV155ZTr66KPTkUcemTbccMN03XXXpaWXXjrddNNNlS4aAADQCnVIVezLL79ML774Yho8eHDttHbt2qX+/funZ555psHXzJgxI9/KJk2alP9Onjw5VYUp0ypdAmgWVfMba2nsE2il7BMWkn0CrdTkKtknlMtRKpVabkj697//nWbNmpVWXnnlOtPj8RtvvNHga4YOHZqGDBkyx/RevXo1WzmBlLp161bpIgBVxD4BqOZ9wueffz7PMlV1SFoYUesUfZjKZs+enT799NPUvXv3VFNTU9GysXjPEkQwfv/991PXrl0rXRygwuwTgCL7hLarVCrlgNSzZ895zlfVIWmFFVZI7du3Tx999FGd6fG4R48eDb6mU6dO+Va07LLLNms5qV6x47PzA8rsE4Ai+4S2qVsjarWqeuCGJZZYIm222WbpkUceqVMzFI+32mqripYNAABonaq6JilE07kBAwakb3zjG2nzzTdPw4YNS1OnTs2j3QEAALS5kHTQQQeljz/+OJ1zzjlpwoQJadNNN00PPfTQHIM5QFE0uYxra9Vvegm0TfYJQJF9AvNTU5rf+HcAAABtSFX3SQIAAFjchCQAAIACIQkAAKBASAIAACgQkmhVhg8fnvr06VN7cbi4ntaDDz5Y6WIBVeKSSy5JNTU16cQTT6x0UYAKOO+88/I+oHhbf/31K10sqlDVDwEOC2LVVVfNB0HrrLNOioEbb7311rTXXnull19+OW200UaVLh5QQX/961/T9ddfn0+kAG1XHA88/PDDtY87dHA4zJzUJNGq7LHHHmm33XbLIWnddddNF110UerSpUt69tlnK100oIKmTJmSDj300HTDDTek5ZZbrtLFASooQlGPHj1qbyussEKli0QVEpJotWbNmpXuvPPONHXq1NzsDmi7jj322PTd73439e/fv9JFASrsrbfeSj179kxrrrlmPnny3nvvVbpIVCH1i7Q6r7zySg5F06dPz7VI9957b9pwww0rXSygQuJkyUsvvZSb2wFt2xZbbJFuueWWtN5666Xx48enIUOGpG233Ta9+uqraZlllql08agiQhKtTuz4Ro8enSZNmpTuvvvuNGDAgPT4448LStAGvf/+++mEE05Io0aNSksuuWSliwNU2K677lp7P/onRmjq3bt3+v3vf58GDhxY0bJRXWpK0bsdWrFoXrPWWmvlDttA23LfffelffbZJ7Vv375OU9wY0apdu3ZpxowZdZ4D2p5vfvOb+Vhh6NChlS4KVURNEq3e7Nmz84EQ0Pb069cvN8EtOvLII/OQv6effrqABG1cDOoyduzYdNhhh1W6KFQZIYlWZfDgwbkqfbXVVkuff/55uuOOO9Jjjz2WRo4cWemiARUQfQw23njjOtM6d+6cunfvPsd0oPU75ZRT8ki40cTuww8/TOeee24+WXLIIYdUumhUGSGJVmXixInp8MMPz50xu3XrltsbR0DacccdK100AKDCPvjggxyIPvnkk7TiiiumbbbZJl8mJO5DkT5JAAAABa6TBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAAAUCEkAAAAFQhIAAECBkATAYvHPf/4z1dTUpNGjRzf7Z91yyy1p2WWXbfbPAaB1EpIASEcccUQOMPVvu+yyS6p2q6++eho2bFidaQcddFD6xz/+0eyfPW7cuPS9730v9ezZMy255JJp1VVXTXvttVd64403mv2zAWg+HZrxvQFoQSIQ3XzzzXWmderUKbVESy21VL41p5kzZ6Ydd9wxrbfeeumee+5Jq6yySvrggw/Sgw8+mD777LNm/dyOHTs22/sDoCYJgEIg6tGjR53bcsstl5+L2pKonal/sL7CCiuk2267LT9+6KGH0jbbbJObuXXv3j3tvvvuaezYsQvUJO6+++7LNVhl8fqomVl55ZVTly5d0je/+c308MMP1z6/3XbbpXfffTeddNJJtbVfc3vv4cOHp7XWWistscQSOdjcfvvtdZ6P1/7qV79K++yzT1p66aXTOuusk+6///65lv/vf/97Lt+1116bttxyy9S7d+/0rW99K1144YX5cVkEp0MOOSQtv/zyqXPnzukb3/hGeu655xaoXDHPnnvumV9/0UUX5el//OMf09e//vVcg7XmmmumIUOGpK+++mqu5QWg8YQkAObr0EMPTSNGjEhTpkypnTZy5Mg0bdq0HCrC1KlT06BBg9ILL7yQHnnkkdSuXbv83OzZsxf6c+Pzdtttt/x+L7/8cq7t2mOPPdJ7772Xn48anGjidv7556fx48fnW0PuvffedMIJJ6STTz45vfrqq+mHP/xhOvLII9Ojjz5aZ74IGgceeGAaM2ZM/tz43p9++mmD77niiivm73j33XenWbNmzbX83/nOd9K//vWvHLj+9re/pdNOO612mTS2XOedd15elq+88ko66qij0pNPPpkOP/zw/NrXXnstXX/99TkYlgMUAIuoBECbN2DAgFL79u1LnTt3rnO76KKL8vMzZ84srbDCCqXbbrut9jWHHHJI6aCDDprre3788cel+G/mlVdeyY/HjRuXH7/88sv58c0331zq1q1bndfce++9eZ552WijjUo///nPax/37t27dNVVV9WZp/57b7311qWjjz66zjwHHHBAabfddqt9HJ971lln1T6eMmVKnvbggw/OtSy/+MUvSksvvXRpmWWWKW2//fal888/vzR27Nja56+//vr83CeffNLg6xtbrhNPPLHOPP369StdfPHFdabdfvvtpVVWWWWuZQWg8dQkAZBtv/32eeS54u2YY47Jz3Xo0CHXsPzmN7+prTWK5l5R01L21ltv5WZl0fSra9eueUCFUK71WRhRE3PKKaekDTbYIDefiyZ3r7/++gK/Z7wmmsIVxeOYXtSnT5/a+9G0Lb7HxIkT5/q+xx57bJowYUJeLltttVW666670kYbbZRGjRqVn49l+LWvfS03tVuUckUTvaKokYras1ge5dvRRx+da9Kidg+ARWPgBgBqQ8Haa6891+cjEEXTsQgNEQJiYITi6HfRDC765dxwww15tLdoUrbxxhunL7/8ssH3i6Zq/1dRUrefU1EEpPisyy+/PJctPnP//fef63suqvoDIkR/oPk1F1xmmWXyd49b9Efaeeed898Y1KGpBo+IdVM/PEbTwH333XeOeaOPEgCLRk0SAI2y9dZbp169eqXf/e53uebkgAMOqA0Vn3zySXrzzTfTWWedlfr165drfv7zn//M8/2iT8/nn3+ea6XK6l9D6amnnsrDk0d/nE022SQPJhHXWyqKAQ/m1ieoLMoT71X/vTfccMPUlCJUrb/++rXfKWqm4jvNrV/TwpYrBmyI5R3Bsf4twicAi0ZNEgDZjBkzctOxomhmFyPYlcUod9ddd12+BlFxcIEYBS9GtPvlL3+Zh8KO5nBnnHHGPD9viy22yKPInXnmmen444/PI77F4ANFMcJcDM4QtTQRQM4+++w5anaiWd8TTzyRDj744DxCX7G8ZaeeempuLhhN3/r3758HoYj3LY6Ut6Ai/Jx77rnpsMMOy6Emwtrjjz+ebrrppnT66afneaL54cUXX5z23nvvNHTo0LxsYgCKqGmL5nkLW65zzjknjx642mqr5Zq1CEbRBC8Gf4haLAAW0QL0XwKgFQ/cEP8l1L+tt956deZ77bXX8vQYLGH27Nl1nhs1alRpgw02KHXq1KnUp0+f0mOPPZbnjcEYGhq4IcRza6+9dmmppZYq7b777qVf/vKXdQZuiNfEgAjxfK9evfJACd/5zndKJ5xwQu08zzzzTP68+NzyaxsaFOLaa68trbnmmqWOHTuW1l133TqDUIRiWcviPeK95jYwxfHHH1/aeOONS126dMkDNGyyySalyy+/vDRr1qza+f75z3+W9ttvv1LXrl3zIA/f+MY3Ss8999wilSs89NBDeeCHWDbx3ptvvnlefgAsupr4Z1GDFgAAQGuh4TIAAECBkAQAAFAgJAEAABQISQAAAAVCEgAAQIGQBAAAUCAkAQAAFAhJAAAABUISAABAgZAEAABQICQBAACk/9//B1tVC3ghFo6+AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1000x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Reset index without dropping the 'run' and 'evaluation_score' columns\n",
    "evaluation_df_pivot.reset_index(inplace=True)\n",
    "\n",
    "# Plotting\n",
    "plt.figure(figsize=(10, 6))\n",
    "\n",
    "bar_width = 0.35\n",
    "\n",
    "# OpenAI brand colors\n",
    "openai_colors = ['#00D1B2', '#000000']  # Green, Black\n",
    "\n",
    "# Identify unique runs and evaluation scores\n",
    "unique_runs = evaluation_df_pivot['run'].unique()\n",
    "unique_evaluation_scores = evaluation_df_pivot['evaluation_score'].unique()\n",
    "\n",
    "# Repeat colors if there are more runs than colors\n",
    "colors = openai_colors * (len(unique_runs) // len(openai_colors) + 1)\n",
    "\n",
    "for i, run in enumerate(unique_runs):\n",
    "    # Select rows for this run only\n",
    "    run_data = evaluation_df_pivot[evaluation_df_pivot['run'] == run].copy()\n",
    "    \n",
    "    # Ensure every 'evaluation_score' is present\n",
    "    run_data.set_index('evaluation_score', inplace=True)\n",
    "    run_data = run_data.reindex(unique_evaluation_scores, fill_value=0)\n",
    "    run_data.reset_index(inplace=True)\n",
    "    \n",
    "    # Plot each bar\n",
    "    positions = np.arange(len(unique_evaluation_scores)) + i * bar_width\n",
    "    plt.bar(\n",
    "        positions,\n",
    "        run_data['Number of records'],\n",
    "        width=bar_width,\n",
    "        label=f'Run {run}',\n",
    "        color=colors[i]\n",
    "    )\n",
    "\n",
    "# Configure the x-axis to show evaluation scores under the grouped bars\n",
    "plt.xticks(np.arange(len(unique_evaluation_scores)) + bar_width / 2, unique_evaluation_scores)\n",
    "\n",
    "plt.xlabel('Evaluation Score')\n",
    "plt.ylabel('Number of Records')\n",
    "plt.title('Evaluation Scores vs Number of Records for Each Run')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "992f9aa0",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Now you have a framework to test SQL generation using LLMs, and with some tweaks this approach can be extended to many other code generation use cases. With GPT-4 and engaged human labellers you can aim to automate the evaluation of these test cases, making an iterative loop where new examples are added to the test set and this structure detects any performance regressions. \n",
    "\n",
    "We hope you find this useful, and please supply any feedback."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8368c786-38eb-4ca3-b5f4-cad63fec87bd",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}