{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "ecb7ce06-97cb-432c-8fa7-f4cf709d50ff",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "# MLflow 3 and LangGraph — Develop and evaluate a *clickbait agent*.\n",
    "\n",
    "This notebook demonstrates how to implement and evaluate a clickbait detector and rewriter agent using LangGraph and MLflow 3's GenAI tools. It was developed as part of the talk \"What Comes After Coding: Evaluating Agentic Behaviour\" presented at the Madrid Databricks User Group Meetup.\n",
    "\n",
    "The focus is on practical agent development and rigorous evaluation. Core topics include agent development with LangGraph, MLflow instrumentation and the use of custom scorers and built-in judges to assess agentic behavior directly from execution traces."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "a6aa73c6-4371-4c17-92bf-edf642820c56",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "## Environment Setup and Configuration\n",
    "\n",
    "Ensure that the configuration settings in this section are reviewed and updated before proceeding.\n",
    "\n",
    "These values determine where artifacts such as models and volumes will be stored, tokens to allow interaction with external tools, and addresses to volumes with datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "db253b14-0030-46bf-b7ca-626678ec5ccd",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[43mNote: you may need to restart the kernel using %restart_python or dbutils.library.restartPython() to use updated packages.\u001B[0m\nRequirement already satisfied: mlflow>=3.1.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from mlflow[databricks]>=3.1.0) (3.1.0)\nRequirement already satisfied: databricks-sdk in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (0.55.0)\nCollecting databricks-sdk\n  Downloading databricks_sdk-0.57.0-py3-none-any.whl.metadata (39 kB)\nRequirement already satisfied: mlflow-skinny==3.1.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (3.1.0)\nRequirement already satisfied: Flask<4 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (3.1.1)\nRequirement already satisfied: alembic!=1.10.0,<2 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (1.16.2)\nRequirement already satisfied: docker<8,>=4.0.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (7.1.0)\nRequirement already satisfied: graphene<4 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (3.4.3)\nRequirement already satisfied: gunicorn<24 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (23.0.0)\nRequirement already satisfied: matplotlib<4 in /databricks/python3/lib/python3.12/site-packages (from mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (3.8.4)\nRequirement already satisfied: numpy<3 in /databricks/python3/lib/python3.12/site-packages (from mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (1.26.4)\nRequirement already satisfied: pandas<3 in /databricks/python3/lib/python3.12/site-packages (from mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (1.5.3)\nRequirement already satisfied: pyarrow<21,>=4.0.0 in /databricks/python3/lib/python3.12/site-packages (from mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (15.0.2)\nRequirement already satisfied: scikit-learn<2 in /databricks/python3/lib/python3.12/site-packages (from mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (1.4.2)\nRequirement already satisfied: scipy<2 in /databricks/python3/lib/python3.12/site-packages (from mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (1.13.1)\nRequirement already satisfied: sqlalchemy<3,>=1.4.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (2.0.41)\nRequirement already satisfied: cachetools<7,>=5.0.0 in /databricks/python3/lib/python3.12/site-packages (from mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (5.3.3)\nRequirement already satisfied: click<9,>=7.0 in /databricks/python3/lib/python3.12/site-packages (from mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (8.1.7)\nRequirement already satisfied: cloudpickle<4 in /databricks/python3/lib/python3.12/site-packages (from mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (2.2.1)\nRequirement already satisfied: fastapi<1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (0.115.13)\nRequirement already satisfied: gitpython<4,>=3.1.9 in /databricks/python3/lib/python3.12/site-packages (from mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (3.1.37)\nRequirement already satisfied: importlib_metadata!=4.7.0,<9,>=3.7.0 in /databricks/python3/lib/python3.12/site-packages (from mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (6.0.0)\nRequirement already satisfied: opentelemetry-api<3,>=1.9.0 in /databricks/python3/lib/python3.12/site-packages (from mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (1.27.0)\nRequirement already satisfied: opentelemetry-sdk<3,>=1.9.0 in /databricks/python3/lib/python3.12/site-packages (from mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (1.27.0)\nRequirement already satisfied: packaging<26 in /databricks/python3/lib/python3.12/site-packages (from mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (24.1)\nRequirement already satisfied: protobuf<7,>=3.12.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (5.29.5)\nRequirement already satisfied: pydantic<3,>=1.10.8 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (2.11.7)\nRequirement already satisfied: pyyaml<7,>=5.1 in /databricks/python3/lib/python3.12/site-packages (from mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (6.0.1)\nRequirement already satisfied: requests<3,>=2.17.3 in /databricks/python3/lib/python3.12/site-packages (from mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (2.32.2)\nRequirement already satisfied: sqlparse<1,>=0.4.0 in /databricks/python3/lib/python3.12/site-packages (from mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (0.5.1)\nRequirement already satisfied: typing-extensions<5,>=4.0.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (4.14.0)\nRequirement already satisfied: uvicorn<1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (0.34.3)\nRequirement already satisfied: google-auth~=2.0 in /databricks/python3/lib/python3.12/site-packages (from databricks-sdk) (2.35.0)\nRequirement already satisfied: azure-storage-file-datalake>12 in /databricks/python3/lib/python3.12/site-packages (from mlflow[databricks]>=3.1.0) (12.17.0)\nRequirement already satisfied: google-cloud-storage>=1.30.0 in /databricks/python3/lib/python3.12/site-packages (from mlflow[databricks]>=3.1.0) (2.18.2)\nRequirement already satisfied: boto3>1 in /databricks/python3/lib/python3.12/site-packages (from mlflow[databricks]>=3.1.0) (1.34.69)\nRequirement already satisfied: botocore in /databricks/python3/lib/python3.12/site-packages (from mlflow[databricks]>=3.1.0) (1.34.69)\nRequirement already satisfied: databricks-agents<2.0,>=1.0.0rc3 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from mlflow[databricks]>=3.1.0) (1.1.0)\nRequirement already satisfied: Mako in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from alembic!=1.10.0,<2->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (1.3.10)\nRequirement already satisfied: azure-core>=1.30.0 in /databricks/python3/lib/python3.12/site-packages (from azure-storage-file-datalake>12->mlflow[databricks]>=3.1.0) (1.31.0)\nRequirement already satisfied: azure-storage-blob>=12.23.0 in /databricks/python3/lib/python3.12/site-packages (from azure-storage-file-datalake>12->mlflow[databricks]>=3.1.0) (12.23.0)\nRequirement already satisfied: isodate>=0.6.1 in /databricks/python3/lib/python3.12/site-packages (from azure-storage-file-datalake>12->mlflow[databricks]>=3.1.0) (0.6.1)\nRequirement already satisfied: jmespath<2.0.0,>=0.7.1 in /databricks/python3/lib/python3.12/site-packages (from boto3>1->mlflow[databricks]>=3.1.0) (1.0.1)\nRequirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /databricks/python3/lib/python3.12/site-packages (from boto3>1->mlflow[databricks]>=3.1.0) (0.10.2)\nRequirement already satisfied: python-dateutil<3.0.0,>=2.1 in /databricks/python3/lib/python3.12/site-packages (from botocore->mlflow[databricks]>=3.1.0) (2.9.0.post0)\nRequirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from botocore->mlflow[databricks]>=3.1.0) (2.5.0)\nRequirement already satisfied: databricks-connect in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from databricks-agents<2.0,>=1.0.0rc3->mlflow[databricks]>=3.1.0) (16.3.4)\nRequirement already satisfied: dataclasses-json in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from databricks-agents<2.0,>=1.0.0rc3->mlflow[databricks]>=3.1.0) (0.6.7)\nRequirement already satisfied: jinja2>=3.0.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from databricks-agents<2.0,>=1.0.0rc3->mlflow[databricks]>=3.1.0) (3.1.6)\nRequirement already satisfied: tenacity>=8.5 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from databricks-agents<2.0,>=1.0.0rc3->mlflow[databricks]>=3.1.0) (9.1.2)\nRequirement already satisfied: tiktoken>=0.8.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from databricks-agents<2.0,>=1.0.0rc3->mlflow[databricks]>=3.1.0) (0.9.0)\nRequirement already satisfied: tqdm in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from databricks-agents<2.0,>=1.0.0rc3->mlflow[databricks]>=3.1.0) (4.67.1)\nRequirement already satisfied: blinker>=1.9.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from Flask<4->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (1.9.0)\nRequirement already satisfied: itsdangerous>=2.2.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from Flask<4->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (2.2.0)\nRequirement already satisfied: markupsafe>=2.1.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from Flask<4->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (3.0.2)\nRequirement already satisfied: werkzeug>=3.1.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from Flask<4->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (3.1.3)\nRequirement already satisfied: pyasn1-modules>=0.2.1 in /databricks/python3/lib/python3.12/site-packages (from google-auth~=2.0->databricks-sdk) (0.2.8)\nRequirement already satisfied: rsa<5,>=3.1.4 in /databricks/python3/lib/python3.12/site-packages (from google-auth~=2.0->databricks-sdk) (4.9)\nRequirement already satisfied: google-api-core<3.0.0dev,>=2.15.0 in /databricks/python3/lib/python3.12/site-packages (from google-cloud-storage>=1.30.0->mlflow[databricks]>=3.1.0) (2.20.0)\nRequirement already satisfied: google-cloud-core<3.0dev,>=2.3.0 in /databricks/python3/lib/python3.12/site-packages (from google-cloud-storage>=1.30.0->mlflow[databricks]>=3.1.0) (2.4.1)\nRequirement already satisfied: google-resumable-media>=2.7.2 in /databricks/python3/lib/python3.12/site-packages (from google-cloud-storage>=1.30.0->mlflow[databricks]>=3.1.0) (2.7.2)\nRequirement already satisfied: google-crc32c<2.0dev,>=1.0 in /databricks/python3/lib/python3.12/site-packages (from google-cloud-storage>=1.30.0->mlflow[databricks]>=3.1.0) (1.6.0)\nRequirement already satisfied: graphql-core<3.3,>=3.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from graphene<4->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (3.2.6)\nRequirement already satisfied: graphql-relay<3.3,>=3.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from graphene<4->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (3.2.0)\nRequirement already satisfied: contourpy>=1.0.1 in /databricks/python3/lib/python3.12/site-packages (from matplotlib<4->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (1.2.0)\nRequirement already satisfied: cycler>=0.10 in /databricks/python3/lib/python3.12/site-packages (from matplotlib<4->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (0.11.0)\nRequirement already satisfied: fonttools>=4.22.0 in /databricks/python3/lib/python3.12/site-packages (from matplotlib<4->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (4.51.0)\nRequirement already satisfied: kiwisolver>=1.3.1 in /databricks/python3/lib/python3.12/site-packages (from matplotlib<4->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (1.4.4)\nRequirement already satisfied: pillow>=8 in /databricks/python3/lib/python3.12/site-packages (from matplotlib<4->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (10.3.0)\nRequirement already satisfied: pyparsing>=2.3.1 in /databricks/python3/lib/python3.12/site-packages (from matplotlib<4->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (3.0.9)\nRequirement already satisfied: pytz>=2020.1 in /databricks/python3/lib/python3.12/site-packages (from pandas<3->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (2024.1)\nRequirement already satisfied: charset-normalizer<4,>=2 in /databricks/python3/lib/python3.12/site-packages (from requests<3,>=2.17.3->mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (2.0.4)\nRequirement already satisfied: idna<4,>=2.5 in /databricks/python3/lib/python3.12/site-packages (from requests<3,>=2.17.3->mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (3.7)\nRequirement already satisfied: certifi>=2017.4.17 in /databricks/python3/lib/python3.12/site-packages (from requests<3,>=2.17.3->mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (2024.6.2)\nRequirement already satisfied: joblib>=1.2.0 in /databricks/python3/lib/python3.12/site-packages (from scikit-learn<2->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (1.4.2)\nRequirement already satisfied: threadpoolctl>=2.0.0 in /databricks/python3/lib/python3.12/site-packages (from scikit-learn<2->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (2.2.0)\nRequirement already satisfied: greenlet>=1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from sqlalchemy<3,>=1.4.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (3.2.3)\nRequirement already satisfied: six>=1.11.0 in /usr/lib/python3/dist-packages (from azure-core>=1.30.0->azure-storage-file-datalake>12->mlflow[databricks]>=3.1.0) (1.16.0)\nRequirement already satisfied: cryptography>=2.1.4 in /databricks/python3/lib/python3.12/site-packages (from azure-storage-blob>=12.23.0->azure-storage-file-datalake>12->mlflow[databricks]>=3.1.0) (42.0.5)\nRequirement already satisfied: starlette<0.47.0,>=0.40.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from fastapi<1->mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (0.46.2)\nRequirement already satisfied: gitdb<5,>=4.0.1 in /databricks/python3/lib/python3.12/site-packages (from gitpython<4,>=3.1.9->mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (4.0.11)\nRequirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /databricks/python3/lib/python3.12/site-packages (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage>=1.30.0->mlflow[databricks]>=3.1.0) (1.65.0)\nRequirement already satisfied: proto-plus<2.0.0dev,>=1.22.3 in /databricks/python3/lib/python3.12/site-packages (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage>=1.30.0->mlflow[databricks]>=3.1.0) (1.24.0)\nRequirement already satisfied: zipp>=0.5 in /databricks/python3/lib/python3.12/site-packages (from importlib_metadata!=4.7.0,<9,>=3.7.0->mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (3.17.0)\nRequirement already satisfied: deprecated>=1.2.6 in /databricks/python3/lib/python3.12/site-packages (from opentelemetry-api<3,>=1.9.0->mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (1.2.14)\nRequirement already satisfied: opentelemetry-semantic-conventions==0.48b0 in /databricks/python3/lib/python3.12/site-packages (from opentelemetry-sdk<3,>=1.9.0->mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (0.48b0)\nRequirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /databricks/python3/lib/python3.12/site-packages (from pyasn1-modules>=0.2.1->google-auth~=2.0->databricks-sdk) (0.4.8)\nRequirement already satisfied: annotated-types>=0.6.0 in /databricks/python3/lib/python3.12/site-packages (from pydantic<3,>=1.10.8->mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (0.7.0)\nRequirement already satisfied: pydantic-core==2.33.2 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from pydantic<3,>=1.10.8->mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (2.33.2)\nRequirement already satisfied: typing-inspection>=0.4.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from pydantic<3,>=1.10.8->mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (0.4.1)\nRequirement already satisfied: regex>=2022.1.18 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from tiktoken>=0.8.0->databricks-agents<2.0,>=1.0.0rc3->mlflow[databricks]>=3.1.0) (2024.11.6)\nRequirement already satisfied: h11>=0.8 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from uvicorn<1->mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (0.16.0)\nRequirement already satisfied: grpcio-status>=1.59.3 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from databricks-connect->databricks-agents<2.0,>=1.0.0rc3->mlflow[databricks]>=3.1.0) (1.71.0)\nRequirement already satisfied: grpcio>=1.59.3 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from databricks-connect->databricks-agents<2.0,>=1.0.0rc3->mlflow[databricks]>=3.1.0) (1.73.0)\nRequirement already satisfied: py4j==0.10.9.7 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from databricks-connect->databricks-agents<2.0,>=1.0.0rc3->mlflow[databricks]>=3.1.0) (0.10.9.7)\nRequirement already satisfied: setuptools>=68.0.0 in /usr/local/lib/python3.12/dist-packages (from databricks-connect->databricks-agents<2.0,>=1.0.0rc3->mlflow[databricks]>=3.1.0) (74.0.0)\nRequirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from dataclasses-json->databricks-agents<2.0,>=1.0.0rc3->mlflow[databricks]>=3.1.0) (3.26.1)\nRequirement already satisfied: typing-inspect<1,>=0.4.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from dataclasses-json->databricks-agents<2.0,>=1.0.0rc3->mlflow[databricks]>=3.1.0) (0.9.0)\nRequirement already satisfied: cffi>=1.12 in /databricks/python3/lib/python3.12/site-packages (from cryptography>=2.1.4->azure-storage-blob>=12.23.0->azure-storage-file-datalake>12->mlflow[databricks]>=3.1.0) (1.16.0)\nRequirement already satisfied: wrapt<2,>=1.10 in /databricks/python3/lib/python3.12/site-packages (from deprecated>=1.2.6->opentelemetry-api<3,>=1.9.0->mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (1.14.1)\nRequirement already satisfied: smmap<6,>=3.0.1 in /databricks/python3/lib/python3.12/site-packages (from gitdb<5,>=4.0.1->gitpython<4,>=3.1.9->mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (5.0.0)\nRequirement already satisfied: anyio<5,>=3.6.2 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from starlette<0.47.0,>=0.40.0->fastapi<1->mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (4.9.0)\nRequirement already satisfied: mypy-extensions>=0.3.0 in /databricks/python3/lib/python3.12/site-packages (from typing-inspect<1,>=0.4.0->dataclasses-json->databricks-agents<2.0,>=1.0.0rc3->mlflow[databricks]>=3.1.0) (1.0.0)\nRequirement already satisfied: sniffio>=1.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-fee525af-078a-43ee-a470-33f1df903818/lib/python3.12/site-packages (from anyio<5,>=3.6.2->starlette<0.47.0,>=0.40.0->fastapi<1->mlflow-skinny==3.1.0->mlflow>=3.1.0->mlflow[databricks]>=3.1.0) (1.3.1)\nRequirement already satisfied: pycparser in /databricks/python3/lib/python3.12/site-packages (from cffi>=1.12->cryptography>=2.1.4->azure-storage-blob>=12.23.0->azure-storage-file-datalake>12->mlflow[databricks]>=3.1.0) (2.21)\n\u001B[43mNote: you may need to restart the kernel using %restart_python or dbutils.library.restartPython() to use updated packages.\u001B[0m\n"
     ]
    }
   ],
   "source": [
    "%pip install -U -qqqq databricks-langchain langgraph==0.3.4 databricks-agents\n",
    "%pip install --upgrade mlflow[databricks]>=3.1.0 databricks-sdk\n",
    "%restart_python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "500e555e-103e-414f-8387-a088f7185315",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "### Global variables\n",
    "\n",
    "This notebook uses Unity Catalog to register models and other resources, and relies on a model serving endpoint for inference. You need to specify:\n",
    "\n",
    "- `UC_CATALOG`: the Unity Catalog catalog where resources will be stored.\n",
    "- `UC_SCHEMA`: the schema within the catalog to use.\n",
    "- `model_serving_endpoint`: the name of the Databricks model serving endpoint that will handle inference requests. You can use a default one or [create your own](https://docs.databricks.com/aws/en/machine-learning/model-serving/create-manage-serving-endpoints)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "9086a8a6-37c8-4280-b231-e3f44c788dee",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "UC_CATALOG = \"\"\n",
    "UC_SCHEMA = \"\"\n",
    "\n",
    "AGENT_PARAMETERS = {\n",
    "    \"model_serving_endpoint\": \"\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "66ac904e-fd4e-4420-abcf-8daf4e19c186",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "#### Jina AI token\n",
    "\n",
    "The agent will have access to an external tool to fetch website contents from urls. This tool is implemented using *Jina AI Reader API*, a service for extracting content from web pages and converting it into clean LLM-ready markdown text.\n",
    "\n",
    "A free Jina AI API key can be obtained by following [this tutorial](https://www.youtube.com/watch?v=SLv6tSEKYOg).\n",
    "\n",
    "The token is managed using [databricks secrets](https://docs.databricks.com/aws/en/security/secrets/). For testing purposes you could replace this definition by inlining the token string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "f16acaed-53ff-4f7b-aec1-6fa08c97505d",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "import base64\n",
    "from databricks.sdk import WorkspaceClient\n",
    "\n",
    "JINA_AI_TOKEN = base64.b64decode(\n",
    "    WorkspaceClient().secrets.get_secret(\"credentials\", \"jinaai\").value\n",
    ").decode()\n",
    "\n",
    "# JINA_AI_TOKEN = \"YOUR_JINA_AI_TOKEN\" # Replace with your Jina AI token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "f220c99a-4f53-495c-bb9b-11141e037499",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "### Clickbait dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "d5ed767f-4cb4-4231-8920-c352f165c67c",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "A corpus of clickbait and non-clickbait title examples will prove useful to evaluate our agent implementation. You can download a dataset with headlines and binary (clickbait or not) classifications from this [Kaggle dataset](https://www.kaggle.com/datasets/amananandrai/clickbait-dataset/data).\n",
    "\n",
    "The `CORPUS_FILE` variable stores the path to the dataset in csv format. In this example, the dataset is stored in a unity catalog volume."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "f7e4a12f-ff0a-43ad-b39e-e0aef7fa93c8",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "CORPUS_FILE = \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "99df8387-5425-43ea-ab85-954de7a87b16",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "assert UC_CATALOG != \"\", \"Please set UC_CATALOG to your catalog name\"\n",
    "assert UC_SCHEMA != \"\", \"Please set UC_SCHEMA to your schema name\"\n",
    "assert \"model_serving_endpoint\" in AGENT_PARAMETERS and AGENT_PARAMETERS[\"model_serving_endpoint\"] != \"\", \"Please set AGENT_PARAMETERS['model_serving_endpoint'] to your model serving endpoint name\"\n",
    "assert JINA_AI_TOKEN != \"\", \"Please set JINA_AI_TOKEN to your Jina AI token\"\n",
    "assert CORPUS_FILE != \"\", \"Please set CORPUS_FILE to the path of your corpus file\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "762d05d9-226b-43b5-8995-bbdfd6db9f60",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "## Implement the agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "f53cd785-b7d7-4831-835a-0609fcf74b99",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "### Register prompts for our agent\n",
    "\n",
    "With MLflow we can manage prompts using the [*prompt registry*](https://mlflow.org/docs/latest/genai/prompt-version-mgmt/prompt-registry/). The prompt registry provides many other features, like prompt version control, prompt aliasing, prompt engineering collaboration and even an automatic [prompt optimizer](https://mlflow.org/docs/latest/genai/prompt-version-mgmt/prompt-registry/optimize-prompts).\n",
    "\n",
    "Here, it's only used to register prompts and log them as model parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "fb34df2d-3963-4cd9-bfb8-52b4771c4874",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "import mlflow\n",
    "\n",
    "# List all registered prompts\n",
    "registered_prompts = mlflow.genai.search_prompts(f\"catalog='{UC_CATALOG}' AND schema='{UC_SCHEMA}'\")\n",
    "\n",
    "# Utility function to register prompts if they are not already registered\n",
    "def register_initial_prompt(name: str, prompt: str):\n",
    "    uc_name = f\"{UC_CATALOG}.{UC_SCHEMA}.{name}\"\n",
    "    # Check if prompt is already registered\n",
    "    if any([uc_name == prompt.name and int(prompt.tags['PromptVersionCount']) > 0 for prompt in registered_prompts]):\n",
    "        return None\n",
    "    # Register initial version of prompt\n",
    "    return mlflow.genai.register_prompt(\n",
    "        name=f\"{UC_CATALOG}.{UC_SCHEMA}.{name}\",\n",
    "        template=prompt.strip(),\n",
    "        commit_message=f\"Initial version of {name} prompt\",\n",
    "        tags={\n",
    "            \"agent\": \"clickbait_langgraph\",\n",
    "            \"language\": \"en\"\n",
    "        }\n",
    "    )\n",
    "\n",
    "register_initial_prompt(\n",
    "    name=\"title_extractor\",\n",
    "    prompt=\"\"\"\n",
    "You are a title and content extractor agent.\n",
    "You will receive a message containing a title and content of an article or similar.\n",
    "You can also just receive a webpage url.\n",
    "If you receive an url, you will call the necessary tool to fetch the title and content.\n",
    "If you receive a prompt containing a title and content, you will extract the title and content from the message.\n",
    "Be careful not to use the fetch tool if there is no valid url in the message (in that case you should always call the extract_title_and_content tool).\n",
    "    \"\"\"\n",
    ")\n",
    "\n",
    "register_initial_prompt(\n",
    "    name=\"clickbait_classifier\",\n",
    "    prompt=\"\"\"\n",
    "You are a clickbait detector intelligent agent.\n",
    "Given a headline you will analyze if it is clickbait or not.\n",
    "You will return a boolean clickbait value, and a list of reasons for your score.\n",
    "The reasons will be provided as bullet points (in markdown format, using '-' for each point).\n",
    "    \"\"\"\n",
    ")\n",
    "\n",
    "register_initial_prompt(\n",
    "    name=\"clickbait_rewriter\",\n",
    "    prompt=\"\"\"\n",
    "You are a clickbait headline rewriter. Given a headline that has been classified as clickbait, you will rewrite in a way that is not considered to be clickbait, and that is more informative and related to the actual content. You will just return the new headline. The new headline will be written in the original language.\n",
    "\n",
    "The headline: {{title}}\n",
    "\n",
    "The content: {{content}}\n",
    "    \"\"\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "2c847800-fbf8-44fd-9c4c-abfdfab7b8fe",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "### Tool definitions\n",
    "\n",
    "The agent will have the following two tools available:\n",
    "\n",
    "- `extract_title_and_content`: an identity function used to extract structured data from the user prompt.\n",
    "- `fetch_url_title_and_content`: fetches title and content from the given url, using *Jina AI reader API*.\n",
    "\n",
    " Note: Defining explicit extraction functions is not always necessary as LangGraph also provides the method `with_structured_outputs`. However, this method cannot be combined with `bind_tools`, that is needed for the web content fetching. More details about this limitation and different strategies for overcoming it can be found it in [this guide](https://langchain-ai.github.io/langgraph/how-tos/react-agent-structured-output)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "08e655ca-5e0e-4cc6-8674-cf8b4b2ff186",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "from urllib.parse import urlparse\n",
    "from langchain_core.tools import tool\n",
    "\n",
    "@tool\n",
    "def extract_title_and_content(title: str, content: str) -> dict[str, str]:\n",
    "    \"\"\"Structure title and content from provided text into a dict.\n",
    "    \n",
    "    Args:\n",
    "        title: The title of the article\n",
    "        content: The content/body of the article\n",
    "    \n",
    "    Returns:\n",
    "        dict: Dictionary with title and content\n",
    "    \"\"\"\n",
    "    return { \"title\": title, \"content\": content }\n",
    "\n",
    "@tool\n",
    "def fetch_url_title_and_content(url: str) -> dict[str, str]:\n",
    "    \"\"\"\n",
    "    Gets title and content from an url\n",
    "\n",
    "    Arguments:\n",
    "        url: The url to fetch data from\n",
    "    \n",
    "    Returns:\n",
    "        dict: Dictionary with title and content\n",
    "    \"\"\"\n",
    "    \n",
    "    parsed = urlparse(url)\n",
    "    if parsed.scheme not in (\"http\", \"https\"):\n",
    "        url = f\"https://{url}\"\n",
    "        parsed = urlparse(url)\n",
    "    if len(parsed.netloc) == 0:\n",
    "        raise ValueError(f\"Invalid URL: {url}\")\n",
    "    \n",
    "    response = requests.get(f\"https://r.jina.ai/{url}\", headers={\n",
    "        \"Accept\": \"application/json\",\n",
    "        \"Authorization\": f\"Bearer {JINA_AI_TOKEN}\",\n",
    "        \"X-Md-Heading-Style\": \"setext\",\n",
    "        \"X-Base\": \"final\",\n",
    "        \"X-Retain-Images\": \"none\",\n",
    "        \"X-Md-Link-Style\": \"discarded\",\n",
    "        \"X-Timeout\": \"10\",\n",
    "    }).json()\n",
    "\n",
    "    return {\n",
    "        \"title\": response[\"data\"].get(\"title\", \"\"),\n",
    "        \"content\": response[\"data\"].get(\"content\", \"\"),\n",
    "    }\n",
    "\n",
    "tools = [extract_title_and_content, fetch_url_title_and_content]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "66853e01-c64d-4c1d-9566-4f075b1afaa6",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "### Structured outputs and agent state\n",
    "\n",
    "- Sometimes it's useful to extract structured data from llm calls. This can be done using pydantic models and the `with_structured_output` method.\n",
    "- LangGraph agents also have a state, a data object that will be passed between graph nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "e61232d1-b9e6-4e72-9919-51e0e6345fbd",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "from typing import TypedDict\n",
    "from pydantic import BaseModel\n",
    "from langchain_core.messages import BaseMessage\n",
    "\n",
    "class Article(BaseModel):\n",
    "    title: str\n",
    "    content: str\n",
    "\n",
    "class Classification(BaseModel):\n",
    "    is_clickbait: bool\n",
    "    classification_reason: str\n",
    "\n",
    "class ClickbaitAgentState(BaseModel):\n",
    "    messages: list[BaseMessage] = []\n",
    "    title: str | None = None\n",
    "    content: str | None = None\n",
    "    is_clickbait: bool | None = None\n",
    "    classification_reason: str | None = None\n",
    "    rewritten_title: str | None = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "1ce34236-f353-47a0-bbc2-2354c8b2aa62",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "### LangGraph agent\n",
    "\n",
    "In the next code block the LangGraph agent is implemented by defining nodes and edges between them. \n",
    "- Some functions are declared to define behavior of nodes.\n",
    "  - Nodes receive a state parameter containing the current graph state. Each node will process messages or state and return state changes. Nodes only return the state keys that contain changes.\n",
    "  - Most nodes are wrappers around a model call, possibly using `bind_tools` or `with_structured_outputs`.\n",
    "  - The node `tool_call` is a wrapper around `ToolNode`. This node executes called tools and saves the outputs in the state.\n",
    "- A `StateGraph` is declared, based on the previously declared state dictionary `ClickbaitAgentState`.\n",
    "- These functions are registered as nodes in the graph with the `RunnableLambda` utility.\n",
    "- The edges between nodes are defined with:\n",
    "  - `add_edge` to declare static directional edges between nodes.\n",
    "  - `add_conditional_edges` to route based on conditions.\n",
    "- The graph entry point is configured with `set_entry_point`. The execution will stop on reaching the special `END` node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "7bf2da9b-6f9c-487a-a743-5749d5392a16",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "from typing import Sequence, Union, TypedDict\n",
    "\n",
    "from databricks_langchain import ChatDatabricks\n",
    "\n",
    "import mlflow\n",
    "\n",
    "from langchain_core.language_models import LanguageModelLike\n",
    "from langchain_core.runnables import RunnableConfig, RunnableLambda\n",
    "from langchain_core.tools import BaseTool\n",
    "from langchain_core.messages import AIMessage, SystemMessage, HumanMessage\n",
    "\n",
    "from langgraph.graph import END, StateGraph\n",
    "from langgraph.graph.graph import CompiledGraph\n",
    "from langgraph.graph.state import CompiledStateGraph\n",
    "from langgraph.prebuilt import ToolNode\n",
    "\n",
    "def create_clickbait_agent(\n",
    "    tools: Sequence[BaseTool],\n",
    "    model_serving_endpoint: str,\n",
    ") -> CompiledGraph:\n",
    "    \n",
    "    model = ChatDatabricks(endpoint=model_serving_endpoint)\n",
    "\n",
    "    # Load prompts from the registry\n",
    "    title_extractor_prompt = mlflow.genai.load_prompt(f\"prompts:/{UC_CATALOG}.{UC_SCHEMA}.title_extractor/1\")\n",
    "    clickbait_classifier_prompt = mlflow.genai.load_prompt(f\"prompts:/{UC_CATALOG}.{UC_SCHEMA}.clickbait_classifier/1\")\n",
    "    clickbait_rewriter_prompt = mlflow.genai.load_prompt(f\"prompts:/{UC_CATALOG}.{UC_SCHEMA}.clickbait_rewriter/1\")\n",
    "\n",
    "    def get_title_and_content(\n",
    "        state: ClickbaitAgentState,\n",
    "        config: RunnableConfig,\n",
    "    ):\n",
    "        chain = (\n",
    "            RunnableLambda(lambda state: [\n",
    "                SystemMessage(content=title_extractor_prompt.template)\n",
    "            ] + state.messages)\n",
    "            | model.bind_tools(tools)\n",
    "        )\n",
    "        response = chain.invoke(state, config)\n",
    "        return {\"messages\": [response]}\n",
    "    \n",
    "    def tool_call(state: ClickbaitAgentState, config: RunnableConfig):\n",
    "        result = ToolNode(tools).invoke(state, config)\n",
    "        article = Article.model_validate_json(result[\"messages\"][-1].content)\n",
    "        return {\n",
    "            \"messages\": state.messages + result[\"messages\"],\n",
    "            \"title\": article.title,\n",
    "            \"content\": article.content\n",
    "        }\n",
    "    \n",
    "    def classify(\n",
    "        state: ClickbaitAgentState,\n",
    "        config: RunnableConfig,\n",
    "    ):\n",
    "        chain = (\n",
    "            RunnableLambda(lambda state: [\n",
    "                SystemMessage(content=clickbait_classifier_prompt.template),\n",
    "                HumanMessage(content=state.title)\n",
    "            ])\n",
    "            | model.with_structured_output(Classification)\n",
    "        )\n",
    "        classification = chain.invoke(state, config)\n",
    "        return {\n",
    "            \"messages\": state.messages\n",
    "                + [AIMessage(content=f\"Classification result: {classification.model_dump()}\")],\n",
    "            \"is_clickbait\": classification.is_clickbait,\n",
    "            \"classification_reason\": classification.classification_reason,\n",
    "        }\n",
    "    \n",
    "    def is_clickbait(\n",
    "        state: ClickbaitAgentState,\n",
    "        config: RunnableConfig,\n",
    "    ):\n",
    "        assert state.is_clickbait is not None\n",
    "        return \"clickbait\" if state.is_clickbait else \"no_clickbait\"\n",
    "    \n",
    "    def clickbait_response(\n",
    "        state: ClickbaitAgentState,\n",
    "        config: RunnableConfig\n",
    "    ):\n",
    "        assert state.title is not None\n",
    "        assert state.content is not None\n",
    "        chain = (\n",
    "            RunnableLambda(lambda state: [\n",
    "                HumanMessage(content = clickbait_rewriter_prompt.format(\n",
    "                    title=state.title,\n",
    "                    content=state.content,\n",
    "                ))\n",
    "            ])\n",
    "            | model\n",
    "        )\n",
    "        response = chain.invoke(state, config)\n",
    "        return { \"messages\": state.messages + [response]}\n",
    "    \n",
    "    def no_clickbait_response(\n",
    "        state: ClickbaitAgentState,\n",
    "        config: RunnableConfig\n",
    "    ):\n",
    "        return { \"messages\": state.messages + [AIMessage(content=state.title)] }\n",
    "    \n",
    "    graph = StateGraph(ClickbaitAgentState)\n",
    "\n",
    "    graph.add_node(\"get_title_and_content\", RunnableLambda(get_title_and_content))\n",
    "    graph.add_node(\"tool_call\", RunnableLambda(tool_call))\n",
    "    graph.add_node(\"classify\", RunnableLambda(classify))\n",
    "    graph.add_node(\"clickbait_response\", RunnableLambda(clickbait_response))\n",
    "    graph.add_node(\"no_clickbait_response\", RunnableLambda(no_clickbait_response))\n",
    "\n",
    "    graph.set_entry_point(\"get_title_and_content\")\n",
    "    graph.add_edge(\"get_title_and_content\", \"tool_call\")\n",
    "    graph.add_edge(\"tool_call\", \"classify\")\n",
    "    graph.add_conditional_edges(\n",
    "        \"classify\",\n",
    "        is_clickbait,\n",
    "        {\n",
    "            \"clickbait\": \"clickbait_response\",\n",
    "            \"no_clickbait\": \"no_clickbait_response\",\n",
    "        },\n",
    "    )\n",
    "    graph.add_edge(\"clickbait_response\", END)\n",
    "    graph.add_edge(\"no_clickbait_response\", END)\n",
    "    \n",
    "    return graph.compile()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "abf08bcd-5c45-4a2f-8a45-bb89a27b6e90",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "### Logging a LangGraph agent into MLflow\n",
    "\n",
    "- Activating logging executions of our agent, called *traces*, is as simple as executing `mlflow.langchain.autolog`.\n",
    "- `set_active_model` is used to set the active model with the specified name. This model will be linked to traces generated from now on. The model name can include the version. In this notebook the version is not being associated to a versioning system. [This page](https://mlflow.org/docs/latest/genai/prompt-version-mgmt/version-tracking/track-application-versions-with-mlflow#step-3-link-traces-to-the-application-version) of MLflow  documentation gives some insights about agent versioning.\n",
    "- Model hyperparameters can be associated to the active model using `log_model_params`.\n",
    "- Finally the agent graph is instantiated. It's important to leave this step as the last one, after enabling logging and registering the active model. Once the agent graph has been initialized, it can be visualized using `display` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "ac527b9f-947d-40a3-93ea-b680f0a3629f",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2025/06/24 11:03:03 INFO mlflow.tracking.fluent: LoggedModel with name 'clickbait-langgraph-1750762982' does not exist, creating one...\n2025/06/24 11:03:04 INFO mlflow.tracking.fluent: Active model is set to the logged model with ID: m-3505dfc179ef47daba099a06eae6cec6\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAIrCAIAAAAmy9qIAAAAAXNSR0IArs4c6QAAIABJREFUeJzs3Xdc0/j/B/BPB7TsvRUR2YqC4t6KG889zoHzXKfnwi1uz4UTJy7EvUW9c5zn3oKCguICN6DMLkpb2t8f8dcvx0HltBBIX8/HPe4RkjR5p6avfvJJmrBUKhUBAGAiNt0FAACUFQQcADAWAg4AGAsBBwCMhYADAMZCwAEAY3HpLgBoI5UoM1PzJYICsUChUKgKZJXggiGeIVuPxzYy4RiZ69k46dNdDlR0CDidI8wpePlIkJIgzpco+cYcI1OOkSnX2JyrKqgEAadSkc/vpGKBQt+A8y5J4lrLyNXX2MXHkO66oIJi4UJf3aGQqW6dzRBkyi3t9V1rGju48umu6IfkiQqSE8Spb6RpKXlNulq71jKiuyKocBBwuiLhluDm6S9NulrXbmZGdy1alv1ZfvtsBpvFaj/IjqPHorscqEAQcDrh0sF0cxv9gEALugspQ18+yI5teN9rfBVbZx7dtUBFgYBjvjPbP7n7mXjVN6G7kPJwZO37DoPtzaz16C4EKgQEHMMdWffev6WFu78x3YWUn6Pr3jfsZOXsiTMPgOvgGO3y4c+1GpvpVLoRQvpMqvr3wXRxbgHdhQD9EHCMlXhXYGql59PQlO5CaDBwlsulg+l0VwH0Q8Ax1pWjn+u1ZfJZBQ30eSy7arwHf2XRXQjQDAHHTHf+zGzUyYqlw5dMNOpkde98FnqYdRwCjoFkUuWX9/nMviikNFr1sn10JYfuKoBOCDgGSk4QGxhzynmlM2fOjI6O/q+vev36dVBQUNlURKq4GyTezS2jhUOlgIBjoJQEcfVy/93S06dPy+1VpWRuo8dmk+x0WdmtAio4XAfHOCpyZO373hOrssumDXfr1q2oqKjExERra+s6depMmDDB2to6ICCAmmpsbHz16lWRSLRv3747d+68fv3a2tq6ZcuWY8eO5fP5hJC2bduOHDny8uXLjx49Gjx48N69e6kXTp48eeDAgVqvNvZyDpdL6rQw1/qSoVJAC45phDkKiaigjNItKSlp4sSJ9evXP3bs2PTp01+8eLFgwQIq9QghoaGhV69eJYQcOnQoMjJy8ODB69atmzhx4l9//RUREUEtQU9P7+TJk56enps2bfr111+Dg4Pt7e1jYmLKIt0IIYbG7IxPaMHpLtwuiWnEAoWRWVn9s8bFxfH5/OHDh7PZbHt7ex8fn1evXv17tkGDBrVt27Z69erUn/Hx8bdv3/7tt98IISwWy8zMLCQkpIwqLMLIlCvOVZTPuqACQsAxjURQYGRaVmcY/Pz8pFLppEmTGjZs2KJFi6pVq6oPTgvT09O7c+fO/PnzX7x4oVAoCCGWlpbqqT4+PmVU3r8ZmnLFAgSc7sIhKtOolESfX1YB5+XltWHDBhsbm/Dw8B49eowbNy4+Pv7fs4WHh0dERPTo0ePUqVMxMTHDhg0rPFVfv/zuxMvlsrh62Ml1F/7tmcbQhJObUYa9Tk2aNAkNDT1z5syCBQtyc3MnTZpEtdHUVCrV8ePH+/Xr16NHD3t7e0KIUCgsu3o0E+Uq9Hg6fLmzzkPAMY2hKUcsKKvfmcfGxt6+fZsQYmNjExQUNHXqVKFQmJqaWngeuVyel5dna2tL/SmTya5fv15G9XyTWKAwMkU/jO5CwDGNsYWeqWVZ3Q0tPj5++vTpJ06cyM7OTkhIOHTokI2NjYODA4/Hs7W1vXv3bkxMDJvNdnFxOX369IcPH3JychYtWuTn5ycQCMRi8b8X6OzsnJGRcfXq1bdv35ZFwTKp0soR97/UXQg4puFwCJtN3iVJymLhgwYN6tGjR1hYWLt27UaNGmVkZBQREcHlcgkhw4cPf/DgwdSpU/Py8n7//Xc+n9+7d+/u3bs3aNBg/PjxfD4/MDDw06dPRRbYrFkzPz+/kJCQCxculEXBSQ8ETpX80RPwI3ChLwM9uZmblS5r2cuG7kJoJhEWHFz1bsSi6nQXArRBC46Bqtc0EuXg2gjy4WVezUZMe8IO/Cfof2UgYwuugTHn6T1BSXe7lMvl7dq1K3aSTCbT09NjFXejJVdX1127dmm72K8iIyMjIyOLnWRsbCwSiYqdFBAQEBYWVtIyb0Z/6TfVWXs1QuWDQ1RmkoqV+5a9GbnEtaQZ/t0dRhGJRMbGxd/inMvlqs+Nap1QKCzpahKpVEr9jvXf9PX1ra2ti530+EZuzhdZi566fpyu4xBwjBV7KZtvxKnZWBdvWU4Iid76qctwB64+LoLTaeiDY6x6gRYvHgk/vMqjuxAaHN/4oX57S6QbIOCYrMc4p/ORqRKhku5CytWFveke/iaOuDoEcIjKeEol2bv0TaehDrZVdeJ614v70r0CTJ29DOguBCoEtOAYjs0mQ0Jdrhz9/Cq++BORjKGQqY6sfV/VwwDpBmpowemKW6czPibnNQ2ydnJj4Of/7p+Z755LWvW21ZGGKpQSAk6HpL/Lv30mw8JO374a36WmEd+w0rff095IP77Ku3s+s2EHq4BAC4KTCvBPCDid8/553vOHgpREsYOLgZEZx8iUS/2nUFSCcxEsNhFmKsQCBYtFnt4TmNvo16hjXKeFObvSZzWUCQSc7kp7I834lC8RFIgFCsJiScXavMmSQCD48OGD1m/ea2jK4XBYRqZcE0u9Km4GfCMEG2iCgIMyERsbGxERsW3bNroLAZ2GL0AAYCwEHAAwFgIOABgLAQcAjIWAAwDGQsABAGMh4ACAsRBwAMBYCDgAYCwEHAAwFgIOABgLAQcAjIWAAwDGQsABAGMh4ACAsRBwAMBYCDgAYCwEHAAwFgIOABgLAQcAjIWAAwDGQsABAGMh4ACAsRBwUCbYbLa5uTndVYCuQ8BBmVAqlTk5OXRXAboOAQcAjIWAAwDGQsABAGMh4ACAsRBwAMBYCDgAYCwEHAAwFgIOABgLAQcAjIWAAwDGQsABAGMh4ACAsRBwAMBYCDgAYCwEHAAwFkulUtFdAzBH37598/PzVSqVVCoVCoW2trbU8MWLF+kuDXQRWnCgTYGBgR8/fvz06VNWVpZcLqeGjYyM6K4LdBQCDrTp559/dnZ2LjyGxWJ17tyZvopApyHgQJtMTEw6duzIYrHUYxwdHQcMGEBrUaC7EHCgZT///LOTkxM1zGKxfvrpJxyiAl0QcKBlJiYmXbp0oRpx1apV6927N90Vge5CwIH29evXz8nJicvlBgUFmZmZ0V0O6C4u3QUwkERY8OVjviyvgO5CaMTq2GxoTExMXY/OLx8J6S6GNmwO29SKa2Wvz+awSjE7aB+ug9Mmeb7qr4Ppqcl5VTyMCmRKussBmvGNOelvpXr6LO+GpjUbmdJdji5CwGmNVKI8uelDg062tlX5dNcCFcuNE+nVvAxqNkbGlTf0wWnNobB3rfo6It3g35r3tHvzTJL0QHeP1umCgNOOhFsCN39TY3P0aULxGgfZPrmdq0K/RflCwGlH+nupoTHSDUqkx2OLshVigYLuQnQLAk475PkqU0t9uquACs2mCl+YjYArVwg47ZBKCgqUOPwATfLECpzTK2cIOABgLAQcADAWAg4AGAsBBwCMhYADAMZCwAEAYyHgAICxEHAAwFgIOABgLAQcADAWAg4AGAsBx3zzF0yfGjL23+MXLJwRMm0cDQUVkpOT3bptwJWrf9FbBjAVAq6S6dGr3afUj9+c7eSpI8tWzKeGW7Ro265d53+P11k//iakpLzuPyBIexVBWcEtzCqTtLTUnJzs0sz5/PlT9XDbNh2KHa+zfvxNeP4Cb2PlgICjzdOnT9atX/7h4ztfX//gQSO3Rqx3re42edIsQkhWVubmLWsSEuOlUmn9+o2DB42sWrXao7iYKVPHEEIGDurWtGnLJYtWl7TkSVNGxcc/JIRcvPjHtq379u/fJRIJV4dtKTK+8EsUCsXOXZvv3rv5+XNarVp+Pbr1bdSo2Tc34c6dG5evXHj85JFAkOvtVWvw4JH+fgFUE2nvvh3r1kTMXzj9zZtkV1e3Pr0HduzQlXrV35cv7N69RSAUNGnSol+fwaV8u+7cubE+fMWXL5/danh07963U8efqPG3bl3bExXx9l2KmZm5m5vnxAkz7OzsCSELF81ksViBbTstX7kgL0/i4+M7ZtREb+9aRd4ED3evxMTHe6IikpISzcwtGjdqPiR4FPWk6pK2Ynfk1qi9OwghrdsGLFywskXzNqXcBCh/OESlh1QqnT13soWF5a4dR0YMH7dpy5ovX9KphyUXFBRMnjo6Lj528qTZu3YctjC3HPfrkI+fPvj7BSxbuo4Qsn9ftIZ0I4SsWxPh7V2rffsuV/6O8XD3+uZ4QsiG8JXHjh/o0b3fgf1nWrZoO3/h9GvX//7mJixdNjc/P3/mjIW/L13n7OwyZ+7krKxMQoienp5IJNwQvnLa1NDLlx60bBG4ctWi9PQ0Qkhy8qulv89t3z5o395THdoHhW9cVZq3686dG6HzQ0YM/3X5sg3NmrVeuWrRpb/PE0JiYu/NWzCtffsuRw79OT90eXp66roNy6mXcLncxKeP/7r059Yte8/9cZOnz6MOS4u8CR8+vg+ZPk6aL90YvnvxwrDk5JeTp4xSKBQatmLY0DH9+wXb2dlf+TsG6VbBIeDocffezdzcnNGjJtrbO3i4e/0ycjz1+SeEPHkS9+7dm9mzFjds0MTS0mrsmEmmZubHjx8ou2Ly8/MvXDw74OehP3XtZWZq1rlTt7ZtOkbt3a75VXw+f0fEoalT5vj7Bfj7BYwZPSkvL+9JQhw1VS6XDwke5ePjy2KxOrQPUqlUr149J4REnz5qZ2sfPHikqYmpv19Aly49SlPh7sitLZq3aRfYqX5Ao8GDRvTrO1giERNCdu3e0qJ5m969BpiZmdesWXvc2Cl3795M+v8j0DyJZFrIPEcHJy6X27ZNx/fv30okkiJLvnTpnB5Xb/HCMGdnFxcX15CpoS9fPb9566rmrYDKAoeo9EhJeWVsbOzq6kb96e8XYGLy9ZlyTxLi9PT06vrXp/5ksVh+derFP35YdsW8ePFMJpPVD2isHuNXp96586dzBblmppqeSy+RiHfs3BgXH5uZmUGNKdxF6OVVkxqgNk0kEhJCPn5871K9xr/n0UCpVL5OfhkY2Ek9ZszoidRAcvLLli3aqsd7evgQQpKSEr08fQghVZ1dDA0NqUnGxiaEEKFQoB5DSUyM9/KqaWZmTv1pb+/g6Fjl8ZNHrVoGatgKqCwQcPQQioSGhkaFx5ibW1ADIpFQLpe3bhtQ7NSyQH1oJ0wcUWR8dlamhoBLT0+bOHlkXf8GoXN+p9o47To0KjwDdcRdhECQW6WKs/pPA77BN8uTSqVKpZLHK/o8RpFIlJ+fX3g8FV5U444QwmZ/+wBFJBImPX9a5N3OzsrUvBVQWSDg6MHn8WUyWeExmZlfqAErK2sDA4OlS9YWnsphc8quGCtrG0LI1ClznJyqFh5va2uv4VVXr/0lk8lmzlhoYGBQpO2mgampmTRfqv5THUYa8Hg8NpstFouKjOfz+YQQqTRPPUYsERNCrCytS1MJxdLK2tfXb9jQMYVHmpmal34JUJEh4Ojh5FQ1Jyc7KyvT0tKKEPIoLkbdPVSjhkdeXp6trb2TYxVqzKfUj+ZmZdiCq+LkzOPxqCNlakx2dpZKpSpyNFeEQJBrYmJKpRsh5JsnJSh2dg6371xXKpVU8+rO3RvffAmHw/H09FH37hFCtu/YKJPJfh03xdPDOzHxsXo8Nexaw700lVBquLpf/OuPOrXrqpt7b94kF25jQqWGkwz0aNSwGYfDCd+4SiwWf/j4fu/eHTY2ttSkenUbNGjQJCxscXp6Wm5uzqnoo2PGDj5//jTVqUQIuXr1r6fPEjQv38mp6rNnCQ8fPcjOzvrmeENDw6FDRkft3f7kSZxMJrt2/e+Q6ePWrV+ueRWuru6ZmRmnzxxXKBT37t9++PC+mZn5589pml/VqlW7nJzs8I2rVCrVo7iYU6eOfOutIoSQbl17P3hw5/CRvY/iYqJPHzt4aE/16jUIIT2697t56+rx4wcFQsGjuJjNW9bU9a/v7uapeWmF34TevQcqlcqNm1dLpdL3799ui9gwfGS/5JRXmpdQpYpzZmbGzZtXP39OL039QBe04OhhZWU9edKsnbs29+rT3t3da0jwqPCNq7hcPWrqsqXrTp85vmjJrKdPn1StWi0wsFPPnv0JIU6OVajrsGrVrLN2zTYNy+/apeeLF8+mTf91xfLw0ozv3y+4Rg2PA4ciHz68b2RkXNOn9tSpczVvQts2Hd6+TY7au33tumX1AxrNmL7g0OGoAwcjhUKBh4d3Sa+qH9BozOiJp08faxNY387Ofs6sJb9NGvnNh+l16BAkEObuiYoQi8VWVtajfpnQuVM3Qkj79l2+ZHw+fHTvxs2r7ezsA+o1+mXkeM2LKvImBNRruHPH4UOH9oweO+jduzdeXjWnhYQWuYbm3xo1bOZbyy90fsjsmYvUvxKBCoiFBzVqxaktn7wbmTu6ajqmK+Ljpw8mJqamJqaEEJVKFfRTy+FDx/bq9XNZlgl0Oh/5oWlXK0fXb59XAW1BC44eubk5434d4lbDY8SIXy0sLHfu3MRmsVu1akd3XQCMgoCjh5mZ+fLf12/fsXHe/BBZfr63d61NGyOtrEp7+u/Jk7jZcyaVNHXf3lPqC7t+xIGDkQcPRhY7qZqL68YNu358FWqz5kxKeBJX7KTOnbuPHVPixgJogENU7fiOQ9QflJr2qaRJDvaOWlmFUCQs6bpWLoerPiuiFZmZGTK5rNhJhgaGWslr2uEQtfyhBVdZaSvFNDAxNjExNinrtVBK33oFKD1cJgIAjIWAAwDGQsABAGMh4ACAsRBwAMBYCDgAYCwEHAAwFgIOABgLAQcAjIWA0w5TSy7Bb95AIyMzLlcPn7hyhbdbOwxNOBkf8+muAiq0lASRtROP7ip0CwJOO1x8jHO+IOCgRGkpeR51TUvxGBzQJrzf2mHvwrOvxrt9+jPdhUBFJBEW3IxOb9tfm/dfgdLA7ZK06dHVnA+vpA7VDawd+RwuHjen69gcVs4XmThXkXAra9Csavp8tCfKGwJOyz6+lr54KMwTFuRkFH93Mx0hlyvy8vJMTcvpbksVk4mlHptNHKsb+Ldmwv3sKiMEHJSJ2NjYiIiIbds0PRkHoKyhzQwAjIWAAwDGQsABAGMh4ACAsRBwAMBYCDgAYCwEHAAwFgIOABgLAQcAjIWAAwDGQsABAGMh4ACAsRBwAMBYCDgAYCwEHAAwFgIOABgLAQcAjIWAAwDGQsABAGMh4ACAsRBwAMBYCDgAYCwEHAAwFgIOygSHw3FwcKC7CtB1CDgoEwUFBampqXRXAboOAQcAjIWAAwDGQsABAGMh4ACAsRBwAMBYCDgAYCwEHAAwFgIOABgLAQcAjIWAAwDGQsABAGMh4ACAsRBwAMBYCDgAYCwEHAAwFkulUtFdAzDHgAEDcnJyVCqVTCaTSCQWFhYqlUoul1+6dInu0kAXoQUH2tSwYcPPnz9/+fIlNzdXLpdTw2ZmZnTXBToKAQfa1Ldv32rVqhUew2KxWrVqRV9FoNMQcKBNDg4OLVu2ZLFY6jHVqlXr1asXrUWB7kLAgZb16dPH2dmZGmaxWC1atHB0dKS7KNBRCDjQMgcHhxYtWlCNOBcXFzTfgEYIONC+3r17V61alc1mN23a1MnJie5yQHdx6S4AypxYUJAvKSjPNRpwbBrXa3e/4H6H1r2y0mTluWpCWJb2euW7Rqi4cB0ckz24mP34Zg7PgFOo05/hzG31UxJEbnVMGnexNLNG0uk6BBxj/X3osx6P69XAzMCYQ3ct5S3ns/zSwU89Rjua2yHjdBoCjpkuHfxsZKZfq6k53YXQ6cjqlP5TqxqZoR9Gd+EkAwN9fC1VKYmOpxshpHVfhzt/ZtFdBdAJAcdAXz5I2Vz8yxIza/3kBBHdVQCd8DFgoDxRgZUjj+4q6KdvwLapwhfnlusZZKhQ0D3BQPl5Si5PSXcVFUJ2ar7unEGGf0MLDgAYCwEHAIyFgAMAxkLAAQBjIeAAgLEQcADAWAg4AGAsBBwAMBYCDgAYCwEHAIyFgAMAxkLAgXYsWDgjZNo4rS/2+IlDge0bUsPdewZG7d2h9VUAgyHggBBCTp46smzFfLqrANAyBBwQQsjz50/pLgFA+3C7JCCTpoyKj39ICLl48Y9tW/d5uHvdunVtT1TE23cpZmbmbm6eEyfMsLOzp2bWMKk03r17s3rt0sePHzk6ODVv3mb4sLH6+vqEkBMnD9+9e+PZswR9Hq9O7bojRvzq5FilzLYYdAVacEDWrYnw9q7Vvn2XK3/HeLh7xcTem7dgWvv2XY4c+nN+6PL09NR1G5ZTc2qYVBppaanjJwzzreW3OmxLv37Bf18+vyF8JSHkyZO48I2ratass2hR2MwZC7Ozs5b+PrfMNhd0CFpwUNSu3VtaNG/Tu9cAQoiZmfm4sVNCpo1Lev7Uy9NHw6TSLPnY8QM8Pn/Y0DEcDqeuf319fX3q0NjHx3f3ziNVqjhzuVxCiEIunz13cq4g18zUrOw3F5gMAQdFJSe/bNmirfpPTw8fQkhSUqKXp4+GSaVcsru7F4fz9TGGHTt07dihKyGEw+F8+vRh0+bVz5ISxGIxNTUnOwsBBz8Ih6jwDyKRKD8/n8fjq8cYGhoSQiQSsYZJpVy4WCziF3q52q1b1+aETvH09Fm3ZvvlSw9WrtiojU0BQAsO/onP5xNCpNI89RixREwIsbK01jCplAs3MjIWF5eGZ/886evrN3LEr9SfIpHwh7cDgKAFB0VxuVxPD+/ExMfqMdSwaw13DZNKuXBPT5/ExHiFQkH9+fflCyHTxhUUFAgEuTbWturZbty4rL0NAp2GgANCCHFyqvrsWcLDRw+ys7N6dO9389bV48cPCoSCR3Exm7esqetf393NkxCiYVJpdOncXSaTrVn7e0zsvRs3r2zfEW5lbcPhcNxqeDyIufsoLkahUBw9tp+aOS09tSy3GHQCDlGBEEK6dun54sWzadN/XbE8vH37Ll8yPh8+unfj5tV2dvYB9Rr9MnI8NZuGSaVRpYrz8mUbwsIWnzt/msfjdWgfNHLkeELI8OHjJBLx3NApeXl5PXv0nzljYWrqx5mzfpsze0mZbTHoBJZKpaK7BtCyq8e+GJnre9XHKUhydHVK/xBnQ1MO3YUAPXCICgCMhUNU0KYDByMPHowsdlI1F9eNG3aVe0Wg0xBwoE1du/Zq3bp9sZO4HOxsUN6wz4E2mRibmBib0F0FwFfogwMAxkLAAQBjIeAAgLEQcADAWAg4AGAsBBwAMBYCDgAYCwEHAIyFgAMAxkLAMRDfiKOnj39ZQgixduIRFovuKoA2+BgwkLEZ58sHKd1V0C9PWJDxKd/QBDu57sK/PQPZVzNQyJV0V0G/zLR8Nz/8MFanIeAYyNpJ38pe72b0Z7oLoZNCprp88FOLHqV9IA4wEu7oy1gJtwUpiWLPAHMrB56+gQ59k+V8lgmz5NeOp/2y1JWrhw44nYaAY7KURHH8tRxhjiI3Q053LeXEobqBVFLg4mPUtKsV3bUA/RBwUCZiY2MjIiK2bdtW3itWEYJGG/w/HTpyAZ2AdINCEHAAwFgIOABgLAQcADAWAg4AGAsBBwCMhYADAMZCwAEAYyHgAICxEHAAwFgIOABgLAQcADAWAg4AGAsBBwCMhYADAMZCwAEAYyHgAICxEHAAwFgIOABgLAQcADAWAg4AGAsBBwCMhYADAMZCwEGZ4HK5VapUobsK0HUIOCgTCoXiw4cPdFcBug4BBwCMhYADAMZCwAEAYyHgAICxEHAAwFgIOABgLAQcADAWAg4AGAsBBwCMhYADAMZCwAEAYyHgAICxEHAAwFgIOABgLAQcADAWS6VS0V0DMMfw4cMlEolKpRKJRNnZ2c7OziqVSiqVRkdH010a6CIu3QUAo1StWvXMmTNs9tcjg1evXhFCbG1t6a4LdBQOUUGbBgwYYG9vX3iMUqkMCAigryLQaQg40CZPT8/69esXHuPo6Dho0CD6KgKdhoADLRs0aJCdnZ36T39/fw8PD1orAt2FgAMtc3d39/f3p4bt7e0HDhxId0WguxBwoH3BwcFUT5yfn5+Xlxfd5YDuwllU3aJSEVL21wW5u3n41fF/IHswaOBglbLMV0cIYeHLGoqD6+B0Qtob6cMrOZ+S8xRyZYGcgf/i9tUNZHkFrr7GDTta0l0LVCAIOOZLSZTcv5Dl38bKwlafb8Shu5yykpUmy07Lf3Qlc+h8FxaL7mqgYkDAMdyz+4LnsaK2AxzpLqScfH4vvXEybWioC92FQIWAfgsmk4qVLx7qULoRQmyr8ms3t3xwMZvuQqBCQMAxWfp7qbJc+vgrFHNr/TdPxXRXARUCAo7JcjPk9i4GdFdR3qwc+Fw9dMIBQcAxnDxfKcvTuSaciqjS3kjprgIqBAQcADAWAg4AGAsBBwCMhYADAMZCwAEAYyHgAICxEHAAwFgIOABgLAQcADAWAg4AGAsBBwCMhYCDUuneMzBq7w7tLvP4iUNt2zWghpOTX82YOaFdh0b7D+zW7lpAl+GZDEAbH+9agweNpIb/vnz+8ZNHC+evdHV1p7suYA4EHNDG27uWt3ctalgsFtnbOzZp0oLuooBREHDwDwUFBUeP7d8TFUEI8fH2HTpktK+vX5F5Tpw8fPfujWfPEvR5vDq1644Y8auTYxVCiEqlOn7i4IULZ99/eFvNuXpAQKPhw8ZyOJySxh8/cWjzljV//3V/wsQRCQnxhJDWbQM6d+r257no8PU7a9WqQ63u1asXv4wecPnSAxYetQD/Efrg4B8itodHRx9dtDBs7uylNjZ2M2ZNePfuTeEZnjyJC9+4qmbNOosWhc2csTA7O2vp73OpSSdOHNpIcalrAAAgAElEQVS3f1fvXgMOHTjbtWuvP/48dehwlIbxauHrd3b7qbeLi+uVv2OmhYTa2dlf+vuceuq165fMzMyRbvAd0IKD/xEIBUeO7ps0cWb9gEaEkIYNm0ok4sysDGfn/z3DxcfHd/fOI1WqOHO5XEKIQi6fPXdyriDXzNQs/vFDT0+fDh2CCCFBXXr4+9fPk0gIISWNL0nXoF6HD0dNGD+Nw+EQQq5c/atD+6ByeQOAaRBw8D9v3yQTQry8alJ/crncRQtXFZmHw+F8+vRh0+bVz5ISxOKvjz7Iyc4yMzWrVatOxPbwlasW1a7t37hxC+q4lRBS0viSdOncfeeuzffu3WrSpEVy8quPH9937tStDDYXmA8BB/8jEgkJIXweX8M8t25dmztv6sABw0aPmlijhntM7L3pM8ZTk3r3GmBoaHTr9rUVKxdyudxWrdqN/uU3a2ubksaXtApzc4umTVr+ffl8kyYtrl2/5OHuVa1a9TLYXGA+BBz8j5GRMSFEItH0SKqzf5709fUbOeJX6k8qEylsNjuoS4+gLj3evEl++PB+ZFSEWCz6fcnaksZrWEuXzt0XLp4pEApu3rrauVN37W0i6BacZID/cXV153K58Y8fUn+qVKqZsydeuHC28DwCQa6Nta36zxs3LquHL1w4m5LymhDi4uLas2f/Xj1/fvXquYbxGjRs2NTU1Ozw4ai3b1MC23bU9oaCrkDAwf8YGxu3C+wcHX303PnTj+Jiwjeuio29p75UjeJWw+NBzN1HcTEKheLosf3UyLT0VOpi3XkLpt2+fT1XkHv37s0bNy/XqllHw3gNWCxWp44/HT9xsEnjFmZm5mW50cBkOESFf5j424x165evXrO0oKDArYbHogWrCp9CJYQMHz5OIhHPDZ2Sl5fXs0f/mTMWpqZ+nDnrtzmzl0ydMnfjprA5oVMIIZaWVkFdevTpPYgQUtJ4zZo0abknanv7dl3KcnOB4VgqlYruGqCsxP6dLcpR1g20oruQ73HocNTp08f27T3FZv+344wCherg8uSxq2qUWWlQaaAFBxVOXFzsp9QPe6IiFsxf+V/TDaAwBBxUONNnjudwOCOGj2vYoAndtUDlhoCDCufi+Tt0lwAMgfY/ADAWAg4AGAsBBwCMhYADAMZCwAEAYyHgAICxEHAAwFgIOABgLAQcADAWAg4AGAs/1WIyrj5b34DuIsodi82yqarpruugO9CCYzITc+6XD1K6qyhvOen5CpmS7iqgQkDAMZm1I4+le//Cgiy5s5ch3VVAhaB7u7/OUKlU2yPX21TRu3P2C921lB9lgerGybSU3PO3b9+muxagHwKOsSZNmuTo6Nioo7Wlnd7Nk58lAgXdFZWtAoUq/a30wPLkEYtcO3fufPjw4ZSUFNywWsfhluVMc+bMmS9fvgwfPrzwyGf3BY9v5OZmyk0s9JQFDPwXN7fVf/tM7FnPpGVPG64+ixopFov19fUXLVo0ffp0ExMTumsEGiDgmEOlUr148eLQoUMzZ87k8Xj/mkzy85SiXGa241gcYmmrX+ykc+fO3bx5c+nSpeVeFNAPAccEHz9+/P3338PDw2UyGZ+PKySKt3XrViMjo8GDB9NdCJQf9MFVbnl5eYSQyMjI4OBgNpuNdNNgzJgxWVlZiYmJ+FLXHWjBVWIbNmywsLBAk+Q/USgUbDY7ODh43rx5Hh4edJcDZQstuMoqNjbW3Nwc6fZfcblcNpsdGhp67tw5Qkh2djbdFUEZQguukomJiVF/OOHHHT169NmzZ3PmzOFwOHTXAtqHFlylkZqaSgh59OhRVFQU3bUwR58+ffz8/BISEuguBMoEWnCVQF5e3syZMzt27NipUye6a2GyRo0a/f77723atKG7ENAatOAqNJFIRAh59epVv379kG5l7ebNmxkZGYSQlJQUumsB7UDAVVwnT57s3r07IcTX17dJkyZ0l8N8XC63b9++hJA3b94MGTJEKBTSXRH8KARcRZSUlEQI4fF4ly5dorsWXdS6devp06d/+PCBEEL9HyopBFzFkpmZ2bVrV4FAQAjp3Lkz3eXorpo1a3p7e1P3LMBZncoLAVdRPHnyhAq4iIiIBg0a0F0OfHXs2DEXFxdCSFxcHN21wH+GgKsQwsPDN23aRAjx8PBwcHCguxz4hxYtWhBCWCxW8+bN09PT6S4H/gNcJkInqVQaHx/fsGHDhw8f1q1bl+5y4Bvy8vJSU1NdXV1v3rzZrFkzusuBb0MLjjbv3r0LDAw0MzMjhCDdKgUDAwNXV1fqpnvLli2juxz4NrTgaHD27NmgoKC3b99Wq1aN7lrgO718+dLd3f369evu7u7oVaiw0IIrbzNmzHj27BkhBOlWqbm7u1P/iKNGjXr9+jXd5UDx0IIrJ0+fPv348WO7du3S0tLs7e3pLge06dOnT46OjlFRUcHBwXTXAv+AFlx5SExMXL58ee3atQkhSDfmcXR0JITIZLKxY8fSXQv8A1pwZaigoGDLli3jx4//8uWLjY0N3eVAmROLxUZGRn/88YeZmRlOs1YEaMGVoWHDhlHf7Ug3HWFkZEQIad68+bFjx+Lj4+kuB9CCKwNnzpyRy+U9e/akuxCgU05Ojrm5+dKlS8ePH09dDATlDy04Lbt3797Dhw+7dOlCdyFAM3Nzc+oec7Nnz6Ye6kh3Rbqo0rTgFIoK/UDPDx8+REZGzp07l+qFKXYeNpvNZuMbpfLRyr535syZ/Pz83r17a6Mi+nG5XLpLKJXKEXBKpTIrK4vuKoqnUqlYLJZAIDAwMNDT09MwJ4fDsbCwKMfSQAukUil129EfJxaL9fT09PWLf0B15WJiYlLMw8UrHjQofohYLM7PzyeEmJqaak43ACMjI2onycnJqeBHJIyBgPt++fn5LBYLz1qG0mOxWIQQY2Nj6ondSqWS7ooYDgH3n8lkMuphmjwez9DQkO5yoPLhcrkmJiaEELlcLhQKK0U3USWFgPsPqO9bmUyGs/6gFTweT09PTyaT0V0IYzEt4JYuXTpr1ixquG/fvgcOHNAw86lTp4q9Lfi/x6tUqtzcXKrfxNjYuNiToUuWLJk5c+YPbwHoFj6fT/XWZ2VlSaVS9fhTp06pLzb65q6VkpLSsWPHfz/dtaTxmpX0uaiMKse53u/Tq1cv6rb6P4I6SSqXyw0MDDSf/2rWrJn6q3jp0qUBAQEdOnT4wbWD7rC0tJRIJNQv/DgcTuFJhXetcuDl5TVgwABq+PTp0y9evAgJCSm3tWsXkwOuX79+P7iEvLw8qVRqYWFRmlP7rVq1Ug+/fPkyICDgB9cOuobq0lUqlQKBoHDHXOFdqxx4eXl5eXlRwy9fvizPVWtdJQ64e/fubdq0KSMjw9XVtWvXrv9uLvXt27d79+7Ud9H79+/Xr1+fkJDg4ODQtGnT4ODgIplVUFAwd+7c9PT0devWUWe7UlNTd+3a9fDhQ2tr6z59+gQGBlLXhRw/fjw2Nvbt27eWlpaNGjUKDg6mTqQuWbJEJBItX768Y8eOhJC1a9dGREQcP368fN8VKHP9+vUbPHiwQCDYt28fn8+vV6/emDFjrKysCCESiSQ8PDw+Pl4kEjk7O3fo0KFr167fXOC/d04TExOqw5dqzal3LUKIUCjcsWPHhQsXzMzM/P39hw8fbmtrW2SBBw4cOHz48MqVK6mdPD8/PyIi4saNGyqVqlWrVsOGDaNaiNHR0ffv309KStLX1/f19R06dCj10+lTp05FRET8+eef06ZNo56FdOnSpY0bN7q5uZXZm1pWKmsf3L179xYtWjR06NDFixc3bdp07dq1V65cKWnm9PT0yZMn16xZc/ny5b17975y5crmzZuLzLN27dqXL18uXbrUyMiIOlIICwtr3779vHnzfHx8wsLCqOdjRkdHHzlypFevXgsXLhwxYsT169f3799fZFHR0dGEkMmTJyPdGInL5R47dozNZh85cmT79u2JiYn79u2jJoWGhqamps6fP3/v3r3NmjXbtGnT8+fPNS+t2J2Ty+VSGSQUCsVisXpmhUIRGhqamZm5YsWKsWPHfvnyJTQ0tMgldVeuXImKipo5c6anpyc1ZvPmze7u7iEhIf379z927NiFCxcIIQkJCVu2bPHx8Zk3b15ISEhOTs7KlSuL1LZq1SovL6/AwMDz589XxnSrxC24qKiopk2btmnThhBSr149sVhMpVKxTp48yePxgoODORyOn5+fnp5ekYb3gQMHrl27tnTpUgcHB6q7raCgoFu3bvXr1yeE1KhR46+//rp69eqgQYN69uzZrFkzZ2dn6oVPnz6NiYkZMWJE2W8xVCCOjo79+/enzjjVq1eP2p3u37+fmJi4detW6jGD/fv3f/Dgwb59+xYvXqxhUZp3TnNzc+pKcuqIlWpwbd++vWrVqoSQKlWqHD9+nLpoifL48ePVq1ePGDGicePG6pH+/v6tW7cmhNSpU+fSpUvXrl3r3Lmzt7f3tm3bnJycqB9dKRSK+fPnCwQCU1PTMnvbaFApA06pVKakpFDpRhk5cqSG+VNSUtzc3NQdt+3bt2/fvj11HMpisahvvGnTplH3ENfT06OuxlR3ohkbG1erVi0tLY2aGhsbGxYWlpycTH1z4tdXOoi6XznFxMSE+nJ98+YNn8+n0k0929WrVzUvqqSdU039i6icnJzk5GQDAwMq3Qghbm5uM2bMIIRQPyZ7//799u3bW7du3adPn8JLqFevnnrYy8vr9u3b1A8HU1NTt23blpSUpG4c5OTkMCzgKuUhqlQqVSqVpf8pnFgsLnZmlUpVUFAQFhZGna03NjYuPLXwRbx8Pp962vyuXbv27dvXqVOnXbt2nT9//sfPYwBjZGVlFflZi4GBAfWLBQ1K2jmLYLFYBgYGYrFYwy9nNm3aJJFI/v2NW/juDwYGBtSefOfOnQULFri7u69atercuXNLly79Zg2VUaVswfF4PDabXbhvQjN1t1qxfvvtt4SEhA0bNvj4+BTeOaRSqXpnysvLc3BwUKlUf/zxR48ePTp16kSNL30NwHiGhoaFL2SjzjlQJx800LxzFsbj8YyMjPLy8pRKZbFXYgYGBnp6eq5fv75u3bp+fn7q8YWrkkgkVBvt3LlzNWvWHDZsGDWeqXtypWzBcTgcDw+PxMRE9Zjdu3dv27atpPk9PDyePn2q7ou9evXqrFmzCgoKqFsYdejQYdy4cXw+f8WKFYVf9erVK2pAIpG8e/fO0dFRLpdLpVJra2tqvEwmu3v3btlsIlQ+Hh4eUqlUvdsQQp4/f/7NZ6dp2DmLkEgkNWrUkEql6k669+/fT5s2LTk5mfozMDCwU6dOzZo1W7FiBdVMoxQu6eXLl9SpUqFQqN6TCSE3b9783u2u0CplwBFCunTpEhsbS90Y+uzZs0eOHCnc91FEx44d5XL5hg0bHj58eOvWrV27dllZWRW+ltLAwGDy5MlPnjxRn/fkcrl79+59//69QqHYs2ePQqFo2bKlvr5+1apVL168+OnTp9zc3LVr19asWVMoFBb5BubxeNbW1rGxsfHx8bhphO4ICAhwcHDYsGHDixcvsrKyIiMjk5KSevXqpflV39w51fLz8/39/R0dHXfu3Hnr1q3Y2NiNGzdmZGSoT3lRpkyZwuVyqY4XytWrVx88eEAIuXz5clJSUsuWLQkhrq6uDx8+pHbREydOUHOmp6cXWamjo2NSUlJcXFzhUxmVSKU8RCWEtGvXTigU7tu3TyKRWFpaDh8+XMPPBpycnBYvXrxu3bqLFy/yeLzAwEB1y1zNx8fn559/3r17d926dQsKCgwNDXv16jV9+vTs7Ozq1avPnDnTycmJEDJz5sxt27aNGjWKx+ONGjWqTp06MTEx/fr12759e+Gl9e/ff+/evTExMVFRUUW69oCpuFzu/Pnzd+zYMXHiRH19/erVq8+bN69WrVqaX1WanZNiaGiop6e3bNmyVatWUWdmGzZsuGjRoiL3njQ0NJw9e/aUKVOio6N9fHwIIUOHDt21a1doaCh1RSd1EmPIkCESiWTBggVSqbRbt24hISFpaWmhoaHUWQu1zp07v3z5cvbs2UuWLKmM59Nww8vygxteVkZavOElk+CGl5VMXl4eDiehIpNIJMX2zYEGlfUQVetkMlmxHR8AP+jw4cNHjhwpdlK1atXWrFlTyuXk5+cz43bn5QmHqF9RAVemGYdD1Mroxw9RRSJRSUvgcrmFT2Vqlp+fr6enV0GeW1RZDlHRgvsK341QRoyNjbVyoqlSBEpFUyG+DSoC9MFBBYc+uO9QOVpwLBarrL++bt265eXlRV0DWUYqyMEF/CdcLreCNJ0uX77coEGDb/40onxUlueiVo4qWSwW9ZCOsmNlZWVmZlbWa4FKR/2AGNpZW1tjF/2vKsdJBgCA74CDpq8OHz6s/k0fQAUUGRlJ3bMLSg8B99WtW7ew90BFdvnyZQb8nqecIeC+6tu3r6urK91VAJRo6NChDg4OdFdRyaAPDgAYCy24r9AHBxUc+uC+AwLuK/TBQQWHPrjvgID7Cn1wUMGhD+47oA8OABgLLbiv0AcHFRz64L4DAu4r9MFBBYc+uO+AgPsKfXBQwaEP7jugDw4AGAstuK/QBwcVHPrgvgMC7iv0wUEFhz6474CA+wp9cFDBoQ/uO1SOG16WnY4dO37+/LnwGKVSWa9evZ07d9JXFMD/1K1bl8ViEUJUKpV6oGnTpuHh4XSXVgnoegvO39+fupm4mqWl5fDhw+muC+Crhg0bUje1ZrPZLBaLxWLZ2Nj88ssvdNdVOeh6wPXv39/W1rbwGG9v76ZNm9JXEcA/DBgwwMzMrPCY2rVr165dm76KKhNdD7g6deoU3ldMTU0HDBhAa0UA/9C8eXNvb2/1n1ZWVoMHD6a1ospE1wOOEDJ48GD1k4o8PT3RfIOKZsCAAaamptSwr68vmm+lh4D73x5jZGSE70aogJo2beru7k49WGvIkCF0l1OZIOAIISQ4ONjCwsLd3b1JkyZ01wJQjEGDBvH5fF9fX19fX7prqUy+8VMtcW7Bo6vZXz7kS4QMf+p7VlaWoaERn18hHvFbRixseSw2caphULu5WSlmp1nSA+HbJImyQJWVlk93LRVCZmaWiYmJvr4e3YXQj2/E5eqxHFz4DTpaap5TU8B9Spae35Pq29zSwlafb8gpgzqhXLHYrOz0fHGuIjlB2H9KVVYFbr6f3Zlqbs0zsdKzcuAT/FwaiuCwxNlyUbbizh+fh4S6mFiUeD1viQH3Lkny4K+c9sGOZVkm0OPTK8mjK5n9Q6rSXUjxzkelWdjyfRqb010IVHgqcnrbu64jHUytim/YFv8lrixQ3fkzq91gpBszOboZegSY3/kjk+5CivH0rsDIVB/pBqXCIq37OV47nlHS9OID7v2LPJ4Bm8Uqy8qAVvbV+C8fieiuohiv4kU2Vfh0VwGVhokFNys9X5ApL3Zq8QGX/VlmX92wjAsDOplY6hmZc2V5SroLKUqpJNZOCDj4D6p6Gmd8lBU7qfjOuXyJUlF8IAJzZKfnFygrXP/95w9SXLwE/0m+pEAuK/6rGrsSADAWAg4AGAsBBwCMhYADAMZCwAEAYyHgAICxEHAAwFgIOABgLAQcADAWAg4AGAsBBwCMhYADAMYq84Dr3jMwau8OQsjxE4fatmugeeZuPdpSM5dy/PctDeA7LFg4I2TaOEJIcvKr1m0DHj9+pGHm+QumTw0ZW/rxmn3fq6BcW3A+3rUGDxpZbqsjhPTrO7i2rz813KNXu0+pH8tz7cBU5uYWwYNH2tral9saW7Ro265dZ2p44aKZf56LLrdVV3Yl3stc67y9a3l71yq31RFCBvw8lBpIS0vNyckuz1UDg1laWg0bOqY819i2TQf18PPnT+vXb1yea6/UtBZwBQUFR4/t3xMVQQjx8fYdOmS0r69f4RmOnzi0ecuav/+6X5qZCSFxcbHTZvz667ip3bv1ocacPHXk/PnTHz+9r+vfYMrk2ebmFoSQlJTXp88ce/joQVraJ5dqrp07d+/2U29q/m492vbq+bOvr9+UqWMIIQMHdWvatOWSRatL2oTk5Fcjfum/bOm6sDVLzM0tdkQcVCgUO3dtvnvv5ufPabVq+fXo1rdRo2bUzO/evdkduTUuPlalUtWsWbt/32BqE4J+ajng52HPnz+9fuOykZGRr6//7FmLTYxNqFdF7d1x4eLZjIzPtrb2fnXqTZ40i81mp6S8Hj6y3+ZNew4c2H3z1lUbG9vWrdqP+mUCh8MhhNy9d+vw4aik54mWlta1atUZNXKClZU1ISQrK3PzljUJifFSqbR+/cbBg0ZWrVpNW/+alcjJU0f27tuxbk3E/IXT37xJdnV169N7YMcOXamp7969Wbd++YuXzzgcrouL69Aho/39Ar65zDt3bqwPX/Hly2e3Gh7du/ft1PGnwlOp/WT92u21a/t/c2ZCSGZmxphxg328fRfMX0EIYbFYMbH3Dh+OSkiMr1HD47cJ0z3cvQghIpHo6LF99x/cefPmtZWldZMmLYcPG8vn86lDVJFIuDpsS+u2AYSQVWGLt2xdeyb6qoZN6NajbfCgkddvXn78+FH0qcumJqaJiY/3REUkJSWamVs0btR8SPAoIyMjQohQJNwdufXe3ZvZOVmeHj6BgZ26dO5OCJkTOkWPq1etWvVDh6OUSqVrdbdpIfPc3Dyo5d+6dW1PVMTbdylmZuZubp4TJ8yws7OnuqSGDR2Tm5uzJyrCwMCgfkDj8b+GUHtsSR8ZDZ+yH6e1Q9SI7eHR0UcXLQybO3upjY3djFkT3r17890zv32bMnfelJ9+6q1Ot3PnorOzM8eMmTRn1pK4uJiNm8Ko8Zs2r37w4M7E32YsX7ahc+fu6zesuHvvVuFF+fsFLFu6jhCyf1+0hnQjhOjp6RFCovbt6Nd38NQpcwkhG8JXHjt+oEf3fgf2n2nZou38hdOvXf+bECKTySZNGcXhcFYsD1+9aguXw50zd7JUKiWEcDjco8f2BwX1vHzpwcrlG9+9exO+cRW1/N2RW09FHxk7etKxoxdGDB939dpfR4/tV6939Zolbdt2vHj+zpxZS44c3Xfl6l+EkBcvk2bNnujvXz9y17HfJkx//frFipULqG+IyVNHx8XHTp40e9eOwxbmluN+HfLx04cf+AesrPT09EQi4YbwldOmhl6+9KBli8CVqxalp6cRQrKzs8ZPGGZrax+x7cCm8N0W5paLl8yWSCSaF3jnzo3Q+SEjhv+6fNmGZs1ar1y16NLf57975ry8vOkzx1tZWs+ZvYTFYhFC3r5LORV9ZMCAYb8vXadUKueGTqEe/HTi5KEDByP79R38+9J1o0dPvHrtL6oFUNj5P28RQqaFhGpON+ptOfvnSTc3z1UrNxkaGH74+D5k+jhpvnRj+O7FC8OSk19OnjJKoVAQQlauXPg08fGkSbMidx3z9q61dt2yxMTHhBAuh/soLoZa6Z7I45ZW1nPnTSkoKCCExMTem7dgWvv2XY4c+nN+6PL09NR1G5ar13v4cBSbzT518u89u48/SYiL3LNN80empE+ZVminBZcryD1ydN+kiTPrBzQihDRs2FQiEWdmZTg7u3zHzJmZGSHTx/n6+v86dor6VQaGhsOGjqF2kaCgnseOH5DJZPr6+qGhyyQSsYO9I5Vl58+fvv/gdqOGTb9jK6iF1w9o1Kf3QEJIfn7+hYtnB/w89KeuvQghnTt1S0iIj9q7vWWLtu/fv83OzurV82fqu3f+vOXxjx9SuwshxK2GB7VpPj6+3X7qvWPnpmlTQ6X50oOH9owdM7lZs1aEkFYtA5OTX+7bv7Nnj/7Uq1q2CGzVMpAQUqdOXUcHpxcvngW27ZjwJI7P5w8aOJzNZtvZ2Xt5+iSnvCKEPHkS9+7dm9VhW+r61yeEjB0z6dbta8ePH/htwvTv+ges3ORy+ZDgUT4+voSQDu2DdkduffXquZ2d/dFj+/V5vJCpc7lcLiFkWsi83n07RJ8++nN/TQ+H3x25tUXzNu0CO1E7g1gskkjE3zdzQUFB6LypErF4y+YofX19amR2dtak32ZaW9sQQoIH/zJr9sT4+Id+fvX69hnUskXbatWqU7MlJMTff3B79Kjfvu89YbFYpqZmE34Nof68dOmcHldv8cIwMzNzQkjI1NCfB3a9eetqq5aB8Y8f9u8XTO2xo36Z0LJloJnp1yf+yGT5gweNZLFYjg5Ow4aOGT1m0JMncX5+9Xbt3tKieZvevQYQQszMzMeNnRIybVzS86denj6EECenqoMGDieEEGOT+gGNX7x4Rggp6SOj4VP2fRtehHYC7k3Ka0KIl1fNrwvlchctXPUdM7NYrPx86fSZ401NzeaHLmez/9fADKjXiPX/T8Hx8fGVH5JnZH5xdHAiKtWJE4fu3b/1/v1baqqDg9OPbIuHuzc18OLFM5lMVj/gf/0dfnXqnTt/OleQW6WKs7m5xfKVC9oFdvarU69WrTqFD3zc3DzVw06OVeVy+adPH8QSsVwuL9wL6eHhLRKJPn58T338PDy81ZOMjU1EIiEhpJavn1QqnTVnUkC9ho0bt6jiVJVa0ZOEOD09PSrdqPfNr069+McPf2TDKzX17mRiYkoIod695JRX7u5e1NtLCDEyMqpapRr1eSuJUql8nfwyMLCTesyY0RO/Y2YWi8VisVaGLUp6nrhlUxTVnUKp4epOpRshpFbNOoSQT6kf/Pzq6enpPYi5s3zF/FevX1BflhYW33iqsWaeHj7q4cTEeC+vmlS6EULs7R0cHas8fvKoVctAX1+/I0f35ebm1Kldt379xp6F9sPq1d3U714VJ2eq+ennVy85+WXhAKJWlJSUSAVc4T3ZxMRULBYRQkr6yDx5Elfsp0wqlVKH5z9IOwFH7U98XqkK0jCzSqU6cnSfQqKBQwoAABcoSURBVKHw8fFVf+NRDA2N1MMGBoaEkNzcHHs7h5mzJ8rlsl9GjvfzCzAxNpkwccQPbos+j1e4zn8vMDsr08XFdf3a7X/8eerY8QM7d212dKwyNHiU+jwXr9Cm8Q0MCCFisSgrO7PIVlNbkZcnoT6ThdNczcPda/myDdev/x2xPXzzlrX16jYYOmR0rVp1RCKhXC6nemTUCn+KdA2ruEfAZWVmODn949mvfAMDSZ6mQ1SpVKpUKnml25M1zKxSqagWiomxSZEZjIyM1cOGhoaEEIEgl+q3+fPPU6NHT6wf0NjOzn7Hzk0/eLa08CdIJBImPX9aZIfJzsokhMyYvuD06WOXr1w4cnSfsZFxjx79ggf/QuVa4d2VihuxWCQSifLz8wtvFLUV6qZrsf8WPB6v2I9MSZ+yvDxJBQo46t9MQ0u+9DO7u3uNGjlh5uzfovZuHzpktHq8VJqnHqa+E8zMzF+8TEpKSgxbtble3a9X2IlEQhtr2x/eIEIIsbK2IYRMnTKnyIeEuj7A2dll7JhJw4aOefjw/rnzp39fPq+aiyvV/KbK+1p2Xh4hhM83oLY6r9BWUO+ApaW1XF78A4EoDRs0adigybChY2Jj7x0/cXD2nEknjv9lZWVtYGCwdMnawnNy2BytbDhjGBoZSfOlhcfkSSRUS6QkPB6PzWYX/hf87pmNjIwXzFuxeu3S5Svmrw7bov7YF94HRGIRIcTU1EylUp05e7x3rwFBXXp8nSQSlqaGUrK0svb19Sty8pc6FDU1MR00cPjAAcMSEuJv3Lyyd99OY2OTvn0GFd2TpVLqy5vKnX98HiViQoiVpbXmGor9yJT0KTP+//NyP0g7Jxnc3Dy5XK76EEmlUs2cPfHChbPfMXOjhs38/OqNGT0pau+Op0+fqF/16tVz9fDz50/19fVtrG1zc3MIIepEe/Mm+c2bZK1sEdUm5/F4VNce9Z9LNddqztUNDQ3fvXtz7vxp6mutSZMWC+av4HK56mOf+PhY9UJevnrO5XKdnKrWqOHB4XASE+PVk549SzAxNrGx0RTHcXGx9+7fJoRYW9t06BD067ipQpEwLT21Rg2PvLw8W1t7dW12dg6FD42BOnR69ixBLv/6gDiBUPD2XUr16jU0vITD4Xh6+jxJiFOP2b5j46bNa75j5hqu7n5+9RbOX/kkIW7/gd3qed69S6HCgtqTqT1NLpfn5eVZ//+eLJPJbt+5/gObXlQNV/fPn9Pq1K6r3mEszC2dnV1yBbknTh6WSqUsFsvX12/c2Mn+fgEvXiZRr3qd/JL6iFE9NoQQV1c3Lpfr6eFNnYigUMOuNdw1FFDSR6akTxl15u3HaSfgjI2N2wV2jo4+eu786UdxMeEbV8XG3ivpqrfSzNy9W5+GDZsuXDxTLP7a0Et58/rI0X0FBQUvXiZduHi2RfM2enp6LtVcuVzu4SN7BUIBdb6yfkCjtPTUImus6uxCCLl69a+nzxJKv1GGhoZDh4yO2rud6ia4dv3vkOnj1q1fTh1TrFy1aMvWdR8+vn///u3+A7sVCgXVn0II+ZLx+eix/QUFBe/evTn7x4nWrdvzeDxTE9N2gZ337d91+/Z1gVBw8eIfJ08d7t17YLFHpmoJifELFk4/c/ZETk7202cJJ04esra2sbdzqFe3QYMGTcLCFqenp+Xm5pyKPjpm7ODz50+Xfut0QdeuvcRi0eo1S9PT0968SV62fB6fx+/cqbvmV3Xr2vvBgzuHj+x9FBcTffrYwUN7NGTiN2d2dXX7ZeT4yD3b1KnB5xuErV4sEApycrL3H9hla2vn6+unr6/v7Oxy7vzpj58+5ObmrAxb5FvLTygUqPd/Co/Hs7GxjYm5+yguRn1SqzR69x6oVCo3bl4tlUrfv3+7LWLD8JH9klNecTncPVERCxbNSEiIz8rKvHjxj5evknxrfb1my9TUbEP4SoFQIBAKovZut7Ozp66c79G9381bV48fPygQCh7FxWzesqauf313jd+vJX1kNHzKtEJr18FN/G3GuvXLV69ZWlBQ4FbDY9GCVcWeQi39zDNnLBw+ou/KVQsXLlipUMh/7j8kMfHxlq3rjIyMqItrCCF2dvZzZi/ZExXRrXsbJ6eqc2YtzszKCJ0XMmRY7z27j6kX5eRYpWOHrrsjt9aqWWftmm2l36j+/YJr1PA4cCjy4cP7RkbGNX1qT506lxBSq1adKZNnR+7ZduToPkJIQL2Ga1ZvdXFxpV4V1KVHYuLjzVvWEkLq+tefMH4aNf7XcVPZbPbipbMVCoWjY5UBPw/TfDqPENK3z6CcnOyNm8LWrP1dX1+/TesOa9dEUP0jy5auO33m+KIls54+fVK1arXAwE49e/Yv/abpgipOVefPW753747+A4LMzMy9vWutX7eDuvhLgw4dggTC3D1REWKx2MrKetQvEzp36vYjM/ftM+j+/dsLFkzfueOwXCGvVbOOs3P1Pn07KpVKL6+aSxavoY5eQ+f8vmnz6qHDevP5/HFjp/j5Bdy/f7tHr8A9kccLL23ggOG7I7fef3D74IGzJqU+jjM1Md254/ChQ3tGjx307t0bL6+a00JCqR6VRQtWhW9aRfWCVa9eY8zoSepL+Vyru7m41Ojbr1N+fr6DveOSRWuoazPbt+/yJePz4aN7N25ebWdnH1Cv0S8jx2suQMNHpqRPmVawqGtwirh3LksuJ3Va/tBJHN1EXV0cPLhcf5T2fQ6HJQ+cWc3AqGL13EXMSe75mwuPj9tA0Ex9dTHdhXzbzZPprrUMPQOKiXvsRgDAWOX3W9SK4MmTuNlzJpU0dd/eU+oLhYDBuv7UqqRJM2YsaNa0xKkVCjO2oqzpVsD5+vpFFuqbK0Ir6RZ9Umu/MoEyomEfoK5JrBTKeisWLlj54wuhnW4FHCGE+t0v6DJm7APM2Iqyhj44AGAsBBwAMBYCDgAYCwEHAIyFgAMAxkLAAQBjIeAAgLEQcADAWMVf6MvhEhUL2cdwxub6RFXMzVfpZWKuV9wdYQFKpG/IYbGL32mKTzFDU27uF023mYXKTiFT5XzJNzCucF9jLBYRZcnprgIqk8yPUhPL4ttqxe/f1g48eb6yjKsCOuVmyKt5fePOaLRwcjMQIODgv1CpiJW9frGTig84W2cez4D1IlZQxoUBbW6fTa/XtiI+pKZRJ8vbp9PprgIqjfsXMtxqG+mXcAPB4m94STkXmWbpYODTyKwsy4PyppCRi/s+NO9m7eiqhacWlQVBpjx6W2r7wY6Gpjp3Mwj4T+6dyzAxYzfsVOKteTUFHCHk+okvr+JEptb6Fe2+r1qnVCqpZ1nSXUgZMjTjvn8uNrXUq9vG3NnTkO5yNMlKk90+m5nxMd/Z2yhPWEB3ORVCgVLJZpfQl65j9PicrDSpSqnyrGdSt42mA5FvBBwhRJ6vyviULxb8hydcVEa7du1q2rSppyeTH0zF4bAs7PTNbbTzvKJyIMxWZKXK5HJ0BxNCSHh4eI8ePapUqUJ3IRWCqaWelb0+R+8bgf/tQwA9HsuhegU9ltEikeqVRZUAtzrGpZgXyomJBdfEAkepX2XLntlW7+bmg130P6hwVwkAAGgLAg4AGAsBBwCMhYADAMZCwAEAYyHgAICxEHAAwFgIOABgLAQcADAWAg4AGAsBBwCMhYADAMZCwAEAYyHgAICxEHAAwFgIOABgLAQcADAWAg4AGAsBBwCMhYADAMZCwAEAYyHgAICxEHBfGRkZ3b9/XyaT0V0IQDGePHmiVCp5PB7dhVQy337ws46QSCTr168/c+ZM69atg4KCGjduTHdFACQjI+Ps2bNnz541MTH55ZdfmjRpQndFlQwCrqgLFy6cPXv25cuXQUFBQUFBLi4udFcEuuj8+fN//PHHq1evgoKCunTpgv3w+yDgikd9c/7xxx/GxsZdunQJCgri8/l0FwXM9/jx47Nnz545c6Zt27ZBQUGNGjWiu6LKDQH3DQkJCdQxQvPmzYOCgpo2bUp3RcBAX758OfN/7d17cFTVHQfws7v37ivZzSa7yWbzgjwQ0hhJkPDQUCaUBMju4hR1Rh2lr7FOp1ZtrR0FZyp1qp1C/UfodBgRa+2UVkGdvYtSYkgqUYwhxhBMCIEAm9dmN8lm34+7e/vHYVK04SHk5uze+/v8x83unS+T5Lv3nHPvic3GMIxOp7NYLFarlaZp0qGEAAruRh09epRhmN7eXqvVajabS0pKSCcCQvDhhx8yDHPu3Dmr1WqxWIqKikgnEhQouG9ncnLSZrPZ7XaVSoU/aWHoCm5CV1eX3W632Wz19fVmsxmGojyBgrtJp0+fxkPX2tpas9lcW1tLOhFIAePj43goqtfr8SoWRVGkQwkZFNytampqstvtPT09+Oe1tLSUdCKQjA4fPswwzIULF/BQtLCwkHQiUYCCmxsejwd/Msvlcjx0VavVpEMB8rq6uvCq6IYNGywWy4oVK0gnEhcouDnW29uLf6BXr15tsVjWrFlDOhEgYGxsDN9mZDAYzGaz1WqVyWSkQ4kRFBxfPvroI4Zhuru78ZCkrKyMdCIwH+x2O8MwDocDr7YXFBSQTiRqUHD8mp6exquuFEXhG4bT09NJhwJzr7OzEy86bdq0yWKx1NTUkE4EEBTc/Onr68O/ADU1NRaLZe3ataQTgTkwNjaG516NRiNeZZJKYQOLJAIFN99aWloYhjl58iT+fVi8eDHpROBm4Avz4eFhPBTNz88nnQjMAgqODJ/Phy/oOI7DTafVakmHAtfX0dGBZ9nwM/DLly8nnQhcCxQcYf39/bjpqqurLRZLXV0d6URgFkNDQ7jXCgoK8Fwq6UTghkDBJYvW1laGYdrb2/Gq65IlS0gnAigej+OPH5fLhe/2yM3NJR0KfAtQcMklEAjgyZ1YLIaHrjqdjnQoMWpvb2cY5siRI/i7UF1dTToRuBlQcElqYGAAP4xdWVlpNpvXr19POpEoOBwOfJ/2woULLRZLY2Mj6UTglkDBJbvjx48zDNPW1oYvJSoqKmZ92QMPPHDgwIF5TycQLMviC+eJiQn8pF1OTg7pUGAOQMGlhlAohCeDQqEQngzKysq68gXV1dVVVVX79+8nlzGpNTU17dq1i6Zpm8125fETJ07Y7fampia8KlpVVUUsIuABFFyKOX/+PB66lpeXm83mhoYGhFBjY+P4+DjHcatWrdqzZw/pjEmnvb19x44dTqdTKpW2t7cjhC5cuIBXRcvKysxm88aNG0lnBLyAgktVbW1tdru9tbXVarUeOnQokUjg43V1dTt37iSdLomcOnXqhRdeuHjxIkIokUhs376dYRifz4cvhA0GA+mAgEdQcKktEonYbLaXXnpp5gkhmUzW0NDw4osvko6WFAYHB5988smRkZGZIzRN7927t7KykmguME+g4FJefX391NTUlUdomt6yZcszzzxzI293j0THHWGPm/VPsxIkCfpY3pLeErlSptZI0zOoLCNdtEQtoyTXfcvk5OTWrVvHxsauPMhx3MmTJ/lMCpIIFFzKW7ZsmVQqnRmi4os4jUbT3Nx8jXdNjEa/aJ2++FVAIpWqdCoZJaEUFK2kuCvOk1S4BIqF2VgkLpVy7kvenEJV+Qrt7as113jL5s2b3W53OBy+fAaOw5uymUymbyw1AKGC/eBTntFoVCgUCCGDwWA0GvV6vdFozMzMvNrrfVPssXfcHlc8Ta8uqjLRqtT7GTAu0vsnQr1fRNps5++2GG6/a/bHeLdt2+ZwODwej9PpdDqdLpcrEolIJJJgMDjvkQEZcAUnBF6v9waf1f/EPtXziSenTK/LTeM/F+/YaMI5MKGQc+Yf5arSrz9oTSQSsJ2RqEDBiQizbywYoXOKhfbsVzQYHzjhuOdnefklKtJZQHKBghML5nVnjFNm5gl2P+HBz4c3P2rSm+APwoP/gYIThYO7h2XqdJ1JsO2GDXYMb3w421QM13HgMpiPEL7WQ25EqQTfbgih4uX57/55hI3BZza4DApO4AZPB9xOTr8gg3SQeVK2Mv/wfifpFCBZQMEJXMs7Lo1RRJuhy9PoQAD1fe4lHQQkBSg4ITv1ybQqQyVPwTvdbkV2cdbx9ydIpwBJAQpOyL76zG8oybqBF5Kx89UHD9r+OOenpRQyXZ729GdwEQeg4IRr/FIkFEhQtBi/xYp0eX+nn3QKQJ4Yf/pFYqDbn5alJp2CDE22evhsEMFqquiJa3ZGVCbGYtocvh5aiMfZD5r+0tvf5vGMFS9YetfK+7+z+G6E0Kjz3J92P/TEY683/+evPb2tGdqcqsr6xvqf46fcx8bPHzj4O6drsKzkzvVrf8xTNsy0SHuxL7igXKQVDzC4ghOskXNBSiHj6eTvMrs+/vQftSvv3/b0e5UV69488Gx3TzNCiJLRCKG333+5+o4Nf/jt8Yfu29Ha9vcvTzchhFg29tqbT+kycn7zxD/NDY+3HH/L53PzFA8hxMaQdyLG3/lBSoCCE6Z4jIuznIzi5fsbi0U6uuzr1vxg9YotaeqMlXdurr5jw9GWfTMvWFqxbunt36MourR4mT4zf2i4DyF06qtjnmnn5k2/zNTl5uaUfN/y61DYx0c8TErJ/NNJurcdmDdQcMIU8MbTMvh6KtMx0suy0dvKVs4cKV24bNQ5EAhO438W5JXPfEmp1OAic0845LQyK9OEj2s1Bl2GkaeECCFaSYcCSbq3HZg3MAcnTFIZiobjPJ08HPIjhPa89tNvHPf5J2RSCiEkkczywRkMeeWKr82I0ZSSp4QIoUQ8kYjDKoPYQcEJU5qWioT4Kjit1oAQuu+e5wxZhVcez8zI9V59Wk2t0kYiX9tpMhwJ8JQQIcRGWE0hX1OQIFVAwQmTRIrkCikbTVDyuZ+FyNYX0bQCIVRWcic+4vNPchynUKjR1WfVMnWmWCw86hwwGcsQQsOj/V6fa86zzYiz8fQM2FZE7GAOTrDyStSxMC/LiAqFuqHu0aPH9p2/2BVjo909zXvf+MUh5jrPJFSUf5ei5G+/93I0Gp72ut761/NqNY9bAMgknC5Hzt/5QUqAKzjByitVnO0JqLQKPk5et+aRPNNtxz5+8+y5z5XK9IWFlfffs+3ab1Ep03/y8Cv2f+9+/vfr5LTS3PB4Z/eR6+8yflMSLDc5EsgvzeXn9CBlwIaXguVxxQ7uHildVUA6CAGeUb9CGt70Qx5XaUFKgCGqYOmyab1JEQmI8WbXWCi6pOZaf1EQiAQMUYVs6RpNm32ioPKqI7Wdrz447R3//+OJRFwikUoks48gn33qYHranD0Etu9vvxq89OWsX1KrtMHQ7JuCbH/6fZVy9j2KQ95I1B8ursieq4QgdcEQVeAOvDKkNWWqdbPfceaZdiYS3/pukqzMvLmIdpnX62bj0Vm/FImEFIrZV0J1GblX+wOAji9H6+7VFyyCJVQABSd040ORj9+b0peK5XImOBWmUbD+IbH8f8G1wRycwOUUKJYsVzv7ebzjLHnEwuzoGRe0G5gBBSd8Fau0uYXU6Bnh7+I98Nnw1m1FpFOAJAJDVLH4omX6bE8kp1RPOggvIoHYwKdDj/2hlKJ5urUOpCQoOBHpPObp7Qgab8vm4/ktgryu4NSlyUeeWyCFZ0/B10HBiYvjbOiDN0Z1uZrs0qyr3ASSSnzukHtwsqQibe29wrwyBbcICk6MOps9ncc8Kp1SY0jTZqtRqjVdxB/zugKJWIymEmu3GPQmeOYUzA4KTqw41NfhO9Ppd5wJZOSoJFKJTC6jVfJEjK9Nlm4Rx3FsNB6PsnKlLBKIlt6Rvqgq3VjEy5O2QDCg4AAavxTxe9mgl2VjXDScpLvgyuVSlVaWpqU0mVSGga/NioHAQMEBAARLUKtpAABwJSg4AIBgQcEBAAQLCg4AIFhQcAAAwYKCAwAI1n8BTbH5adUxuA4AAAAASUVORK5CYII=",
      "text/plain": [
       "<langgraph.graph.state.CompiledStateGraph object at 0x718a1b2a30b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from time import time\n",
    "\n",
    "mlflow.langchain.autolog()\n",
    "active_model_info = mlflow.set_active_model(name=f\"clickbait-langgraph-{int(time())}\")\n",
    "mlflow.log_model_params(model_id=active_model_info.model_id, params=AGENT_PARAMETERS)\n",
    "agent = create_clickbait_agent(tools, **AGENT_PARAMETERS)\n",
    "display(agent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "5ab848b6-08be-452f-b2df-9561f5d321cd",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "## Invoke the agent\n",
    "\n",
    "With the agent declared, it is now possible to call the invoke method by passing an user message.\n",
    "\n",
    "We can confirm the autologging is working by checking the produced trace in the block results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "1cc2fdb6-5945-44d9-9d63-a5a1ca2b0ff1",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": [
       "{'messages': [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'fetch_url_title_and_content', 'type': 'function', 'function': {'name': 'fetch_url_title_and_content', 'arguments': '{\"url\":\"https://www.lavanguardia.com/cribeo/fast-news/20250527/10723991/popular-nombre-masculino-espana-jonathan-coeficiente-intelectual-bajo-mmn.amp.html\"}'}}]}, response_metadata={'model': 'gemini-2.0-flash-lite', 'usage': {'prompt_tokens': 290, 'completion_tokens': 71, 'total_tokens': 361}, 'object': 'chat.completion', 'id': None, 'created': 1750762987, 'model_name': 'gemini-2.0-flash-lite'}, id='run--e5151a31-08eb-47d2-8c57-fec7e228e3d4-0', tool_calls=[{'name': 'fetch_url_title_and_content', 'args': {'url': 'https://www.lavanguardia.com/cribeo/fast-news/20250527/10723991/popular-nombre-masculino-espana-jonathan-coeficiente-intelectual-bajo-mmn.amp.html'}, 'id': 'fetch_url_title_and_content', 'type': 'tool_call'}]),\n",
       "  ToolMessage(content='{\"title\": \"El popular nombre masculino en España que se asocia al coeficiente intelectual más bajo que hay\", \"content\": \"¿Puede el nombre que llevas influir en cómo te perciben los demás… o incluso en tu inteligencia? Según una reciente investigación realizada por un grupo de profesores de la Universidad de Stanford, existe una correlación estadística entre ciertos nombres y los resultados obtenidos en test de coeficiente intelectual.\\\\n\\\\nEl nombre masculino que, según los datos analizados, aparece con el CI medio más bajo es Jonathan, uno de los más comunes en países como España, Estados Unidos o México.\\\\n\\\\n### El nombre con peor puntuación en las pruebas de CI\\\\n\\\\nEl estudio, basado en una muestra de 70.000 personas, encontró que quienes se llamaban Jonathan registraban una puntuación media en los test de inteligencia de alrededor de 80 puntos, muy por debajo del promedio general establecido en 100. Esta cifra situaría a quienes llevan este nombre dentro del rango considerado de capacidad intelectual baja.\\\\n\\\\nEl calvario de David Castillo al encarnar a Jonathan en \\'Aída\\'\\\\n\\\\nCaptura Youtube/El sentido de la birra\\\\n\\\\nEste hallazgo resulta especialmente llamativo en España, donde más de 24.000 personas se llaman Jonathan, y existen además variantes como Jonatan (más de 19.000) y Yonatan (alrededor de 1.000), según los datos del INE.\\\\n\\\\n**Un vínculo cuestionado por expertos**. A pesar del impacto que puede tener un titular así, muchos especialistas en psicología y educación ponen en entredicho la validez de este tipo de investigaciones. ¿El motivo? Asociar directamente el nombre de una persona con su capacidad intelectual es, según señalan, metodológicamente problemático.\\\\n\\\\nEl nombre no es una variable intrínseca de la persona, sino que está profundamente ligado a factores culturales, sociales y económicos, que sí pueden tener una influencia real en el desarrollo cognitivo. Es decir, lo que puede existir no es una relación causal entre nombre e inteligencia, sino una correlación indirecta a través del entorno en el que nacen y crecen muchas personas con determinados nombres.\\\\n\\\\nCada vez tenemos puntuaciones de coeficiente intelectual más bajas\\\\n\\\\nOtras Fuentes\\\\n\\\\n### La inteligencia es más que un número\\\\n\\\\nOtro punto clave que critican los expertos es la reducción de la inteligencia al coeficiente intelectual. Aunque el CI puede ofrecer información sobre ciertas habilidades cognitivas como la memoria, la lógica o la capacidad matemática, no mide aspectos como la creatividad, la inteligencia emocional, la adaptabilidad, ni el pensamiento crítico, todos ellos esenciales para el desarrollo humano.\\\\n\\\\nPor tanto, este tipo de estudios, aunque llamativos, deben leerse con cautela. El riesgo de perpetuar estereotipos injustos y simplistas es elevado, y más aún si no se tiene en cuenta el contexto completo en el que vive cada persona.\"}', name='fetch_url_title_and_content', tool_call_id='fetch_url_title_and_content'),\n",
       "  AIMessage(content=\"Classification result: {'is_clickbait': True, 'classification_reason': 'The headline uses a sensationalist claim about a popular male name in Spain and its association with the lowest IQ, which is likely to be an exaggeration or a generalization.'}\", additional_kwargs={}, response_metadata={}),\n",
       "  AIMessage(content='Estudio encuentra correlación entre el nombre \"Jonathan\" y puntuaciones más bajas en pruebas de coeficiente intelectual, pero expertos cuestionan la validez.\\n', additional_kwargs={}, response_metadata={'model': 'gemini-2.0-flash-lite', 'usage': {'prompt_tokens': 670, 'completion_tokens': 31, 'total_tokens': 701}, 'object': 'chat.completion', 'id': None, 'created': 1750762993, 'model_name': 'gemini-2.0-flash-lite'}, id='run--d3dae01e-0492-4242-990c-1c2c8ad953a5-0')],\n",
       " 'title': 'El popular nombre masculino en España que se asocia al coeficiente intelectual más bajo que hay',\n",
       " 'content': \"¿Puede el nombre que llevas influir en cómo te perciben los demás… o incluso en tu inteligencia? Según una reciente investigación realizada por un grupo de profesores de la Universidad de Stanford, existe una correlación estadística entre ciertos nombres y los resultados obtenidos en test de coeficiente intelectual.\\n\\nEl nombre masculino que, según los datos analizados, aparece con el CI medio más bajo es Jonathan, uno de los más comunes en países como España, Estados Unidos o México.\\n\\n### El nombre con peor puntuación en las pruebas de CI\\n\\nEl estudio, basado en una muestra de 70.000 personas, encontró que quienes se llamaban Jonathan registraban una puntuación media en los test de inteligencia de alrededor de 80 puntos, muy por debajo del promedio general establecido en 100. Esta cifra situaría a quienes llevan este nombre dentro del rango considerado de capacidad intelectual baja.\\n\\nEl calvario de David Castillo al encarnar a Jonathan en 'Aída'\\n\\nCaptura Youtube/El sentido de la birra\\n\\nEste hallazgo resulta especialmente llamativo en España, donde más de 24.000 personas se llaman Jonathan, y existen además variantes como Jonatan (más de 19.000) y Yonatan (alrededor de 1.000), según los datos del INE.\\n\\n**Un vínculo cuestionado por expertos**. A pesar del impacto que puede tener un titular así, muchos especialistas en psicología y educación ponen en entredicho la validez de este tipo de investigaciones. ¿El motivo? Asociar directamente el nombre de una persona con su capacidad intelectual es, según señalan, metodológicamente problemático.\\n\\nEl nombre no es una variable intrínseca de la persona, sino que está profundamente ligado a factores culturales, sociales y económicos, que sí pueden tener una influencia real en el desarrollo cognitivo. Es decir, lo que puede existir no es una relación causal entre nombre e inteligencia, sino una correlación indirecta a través del entorno en el que nacen y crecen muchas personas con determinados nombres.\\n\\nCada vez tenemos puntuaciones de coeficiente intelectual más bajas\\n\\nOtras Fuentes\\n\\n### La inteligencia es más que un número\\n\\nOtro punto clave que critican los expertos es la reducción de la inteligencia al coeficiente intelectual. Aunque el CI puede ofrecer información sobre ciertas habilidades cognitivas como la memoria, la lógica o la capacidad matemática, no mide aspectos como la creatividad, la inteligencia emocional, la adaptabilidad, ni el pensamiento crítico, todos ellos esenciales para el desarrollo humano.\\n\\nPor tanto, este tipo de estudios, aunque llamativos, deben leerse con cautela. El riesgo de perpetuar estereotipos injustos y simplistas es elevado, y más aún si no se tiene en cuenta el contexto completo en el que vive cada persona.\",\n",
       " 'is_clickbait': True,\n",
       " 'classification_reason': 'The headline uses a sensationalist claim about a popular male name in Spain and its association with the lowest IQ, which is likely to be an exaggeration or a generalization.'}"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "output_type": "display_data",
     "data": {
      "application/databricks.mlflow.trace": "\"tr-70aaab048062ee7e5853c6a218d202b0\"",
      "text/plain": [
       "Trace(trace_id=tr-70aaab048062ee7e5853c6a218d202b0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "input_example = {\"messages\": [HumanMessage(content=\n",
    "    \"is this article clickbait https://www.lavanguardia.com/cribeo/fast-news/20250527/10723991/popular-nombre-masculino-espana-jonathan-coeficiente-intelectual-bajo-mmn.amp.html ?\"\n",
    ")]}\n",
    "\n",
    "output = agent.invoke(input_example)\n",
    "\n",
    "display(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "879508e6-953d-4b8c-922c-83917c55b383",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "## Agent evaluation\n",
    "\n",
    "Once an agent is implemented it is essential to define a set of evaluation scores that enable both quantitative and qualitative assessments of its behavior. This enables the gathering of meaningful insights about how the model—or specific components of it—are performing.\n",
    "\n",
    "MLflow 3's new GenAI API introduces the `evaluate`, which supports executing scorer functions on traces or custom function outputs, via the `predict_fn` parameter. This `predict_fn` function allows wrapping the entire agent agent or individual components-like single nodes-making it possible to reproduce classic testing paradigms such as *integration tests* and *unit tests* within the agent evaluation workflow.\n",
    "\n",
    "Evaluation metrics are defined using the `@scorer` decorator. Custom scorer functions can access inputs, outputs, traces, and expectations, giving them full context to evaluate agent behavior.\n",
    "\n",
    "Custom scorers can tipically be written in an implementation-agnostic way, allowing the agent logic to be replaced—potentially even with a different framework, such as DSPy—without requiring changes to the scorers themselves. The only likely adjustment needed when switching implementations is updating the `predict_fn`, particularly if the new implementation introduces a different function signature.\n",
    "\n",
    "Inside a scorer, it's possible to make custom calls to language models—for example, by passing them the output to assess specific properties. This approach resembles *property-based testing*, where evaluations are derived from general behavioral rules rather than hardcoded expected outputs. Scorers that rely on language models for evaluation are known as judges. Databricks provides several [built-in judges](https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/predefined-judge-scorers) for common evaluation scenarios, streamlining the process for standard use cases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "4e378244-a5ff-40c0-b23c-ad8ba1094336",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "### Useful datasets\n",
    "\n",
    "In the following block some datasets are imported from the *kaggle clickbait dataset* and the `urls_data.json`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "1369c1bc-e919-4424-9388-349934512d58",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "from mlflow.entities import Feedback, Trace\n",
    "from mlflow.genai.scorers import scorer\n",
    "import pandas as pd\n",
    "import random\n",
    "\n",
    "# Kaggle dataset\n",
    "corpus = pd.read_csv(CORPUS_FILE)\n",
    "SAMPLE_SIZE = 100\n",
    "RANDOM_SEED = 42   # fix the random seed to get the same sample every time\n",
    "clickbait_sample = corpus[corpus[\"clickbait\"] == 1].sample(n=SAMPLE_SIZE//2, random_state=RANDOM_SEED)\n",
    "non_clickbait_sample = corpus[corpus[\"clickbait\"] == 0].sample(n=SAMPLE_SIZE//2, random_state=RANDOM_SEED)\n",
    "corpus_sample = pd.concat([clickbait_sample, non_clickbait_sample], ignore_index=True)\n",
    "\n",
    "# dataset with url, title and content (20 rows)\n",
    "url_articles = pd.read_json(\"./urls_data.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "adfd0a27-3f41-4e29-b6d9-c9e11aa9d4db",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "### Classification correctness\n",
    "\n",
    "The `check_clickbait` scorer evaluates whether the agent correctly classifies input titles as clickbait or not.\n",
    "- The dataset consists of inputs with a title field and expectations containing the expected clickbait classification label.\n",
    "- To isolate and evaluate just the classification logic, the predict_fn wraps the invoke method of the `classify` node of the agent.\n",
    "- The scorer compares the agent’s output to the expected label, marking incorrect predictions as either false positives or false negatives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "6e342b43-032d-4765-b9c7-90c127fae6cb",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2025/06/24 11:03:15 INFO mlflow.genai.utils.data_validation: Testing model prediction with the first sample in the dataset.\n2025/06/24 11:03:21 INFO mlflow.tracking.fluent: Active model is set to the logged model with ID: m-3505dfc179ef47daba099a06eae6cec6\n2025/06/24 11:03:21 INFO mlflow.tracking.fluent: Use `mlflow.set_active_model` to set the active model to a different one if needed.\n2025/06/24 11:03:22 INFO mlflow.models.evaluation.utils.trace: Auto tracing is temporarily enabled during the model evaluation for computing some metrics and debugging. To disable tracing, call `mlflow.autolog(disable=True)`.\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a7d09c0a65294ed9996a9778610ee1e4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Evaluating:   0%|          | 0/100 [Elapsed: 00:00, Remaining: ?] "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "<!DOCTYPE html>\n",
       "<html lang=\"en\">\n",
       "<head>\n",
       "    <title>Evaluation output</title>\n",
       "    <meta charset=\"UTF-8\">\n",
       "    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n",
       "    <style>\n",
       "        body {\n",
       "            font-family: Arial, sans-serif;\n",
       "        }\n",
       "\n",
       "        .header {\n",
       "            a.button {\n",
       "                padding: 4px 8px;\n",
       "                line-height: 20px;\n",
       "                box-shadow: none;\n",
       "                height: 20px;\n",
       "                display: inline-flex;\n",
       "                align-items: center;\n",
       "                justify-content: center;\n",
       "                vertical-align: middle;\n",
       "                background-color: rgb(34, 114, 180);\n",
       "                color: rgb(255, 255, 255);\n",
       "                text-decoration: none;\n",
       "                animation-duration: 0s;\n",
       "                transition: none 0s ease 0s;\n",
       "                position: relative;\n",
       "                white-space: nowrap;\n",
       "                text-align: center;\n",
       "                border: 1px solid rgb(192, 205, 216);\n",
       "                cursor: pointer;\n",
       "                user-select: none;\n",
       "                touch-action: manipulation;\n",
       "                border-radius: 4px;\n",
       "                gap: 6px;\n",
       "            }\n",
       "\n",
       "            a.button:hover {\n",
       "                background-color: rgb(14, 83, 139) !important;\n",
       "                border-color: transparent !important;\n",
       "                color: rgb(255, 255, 255) !important;\n",
       "            }\n",
       "        }\n",
       "\n",
       "        .warnings-section {\n",
       "            margin-top: 8px;\n",
       "\n",
       "            ul {\n",
       "                list-style-type: none;\n",
       "            }\n",
       "        }\n",
       "\n",
       "        .instructions-section {\n",
       "            margin-top: 16px;\n",
       "            font-size: 14px;\n",
       "\n",
       "            ul {\n",
       "                margin-top: 0;\n",
       "                margin-bottom: 0;\n",
       "            }\n",
       "        }\n",
       "\n",
       "        code {\n",
       "            font-family: monospace;\n",
       "        }\n",
       "\n",
       "        .note {\n",
       "            color: #666;\n",
       "        }\n",
       "\n",
       "        a {\n",
       "            color: #2272B4;\n",
       "            text-decoration: none;\n",
       "        }\n",
       "\n",
       "        a:hover {\n",
       "            color: #005580;\n",
       "        }\n",
       "    </style>\n",
       "</head>\n",
       "<body>\n",
       "<div>\n",
       "    <div class=\"header\">\n",
       "        <a href=\"https://dbc-5310e2d8-d9d5.cloud.databricks.com/ml/experiments/ddd3d8c02f3640519cbd336f99d256d7/runs/d9a408b2b69e416282d2a651c5cc18d6/evaluations/\" class=\"button\">\n",
       "            View evaluation results.\n",
       "            <svg xmlns=\"http://www.w3.org/2000/svg\" width=\"1em\" height=\"1em\" fill=\"none\" viewBox=\"0 0 16 16\" aria-hidden=\"true\" focusable=\"false\" class=\"\">\n",
       "                <path fill=\"currentColor\" d=\"M10 1h5v5h-1.5V3.56L8.53 8.53 7.47 7.47l4.97-4.97H10z\"></path>\n",
       "                <path fill=\"currentColor\" d=\"M1 2.75A.75.75 0 0 1 1.75 2H8v1.5H2.5v10h10V8H14v6.25a.75.75 0 0 1-.75.75H1.75a.75.75 0 0 1-.75-.75z\"></path>\n",
       "            </svg>\n",
       "        </a>\n",
       "    </div>\n",
       "</div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "output_type": "display_data",
     "data": {
      "application/databricks.mlflow.trace": "[\"tr-4fa9dffeb4cf7bbbfa846667f9aaadec\", \"tr-ccf8b80af7b7a0e7ce842e76a008741a\", \"tr-6097b19e20f7a542e2ba90c5d0b92316\", \"tr-15e402862170e0507f62e343148f6f96\", \"tr-3dcf5af1483ba1ff0a4a5173baa4f5af\", \"tr-1faadd7bff3a118ba43da5beee9ad1ea\", \"tr-c1a38835f3056a584d3fa206e33a92d3\", \"tr-9b524e1d84757b8321527cb86141e231\", \"tr-7acf1d30d0fce450411873cf6445fb37\", \"tr-9a18ef6f23a39461f287719ec5ad7877\"]",
      "text/plain": [
       "[Trace(trace_id=tr-4fa9dffeb4cf7bbbfa846667f9aaadec), Trace(trace_id=tr-ccf8b80af7b7a0e7ce842e76a008741a), Trace(trace_id=tr-6097b19e20f7a542e2ba90c5d0b92316), Trace(trace_id=tr-15e402862170e0507f62e343148f6f96), Trace(trace_id=tr-3dcf5af1483ba1ff0a4a5173baa4f5af), Trace(trace_id=tr-1faadd7bff3a118ba43da5beee9ad1ea), Trace(trace_id=tr-c1a38835f3056a584d3fa206e33a92d3), Trace(trace_id=tr-9b524e1d84757b8321527cb86141e231), Trace(trace_id=tr-7acf1d30d0fce450411873cf6445fb37), Trace(trace_id=tr-9a18ef6f23a39461f287719ec5ad7877)]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "titles_dataset = (\n",
    "    corpus_sample\n",
    "    .apply(\n",
    "        lambda row: {\n",
    "            \"inputs\": {\"title\": row[\"headline\"]},\n",
    "            \"expectations\": {\"is_clickbait\": (row[\"clickbait\"] == 1)},\n",
    "        },\n",
    "        axis=1,\n",
    "        result_type=\"expand\",\n",
    "    )\n",
    "    .sample(frac=1, random_state=42)\n",
    ")\n",
    "\n",
    "def run_classify_node(title: str):\n",
    "    return agent.nodes[\"classify\"].invoke(\n",
    "        ClickbaitAgentState(title=title)\n",
    "    )\n",
    "\n",
    "@scorer\n",
    "def check_clickbait(outputs,  expectations):\n",
    "\n",
    "    value = outputs[\"is_clickbait\"] == expectations[\"is_clickbait\"] \n",
    "    rationale = (\n",
    "        \"The model predicted the wrong clickbait status\" if value else \"The model predicted the correct clickbait status\"\n",
    "        )\n",
    "    \n",
    "    feedback = [Feedback(\n",
    "            name = \"success\",\n",
    "            value = value,\n",
    "            rationale=rationale\n",
    "        )]       \n",
    "\n",
    "    if not value:\n",
    "        if expectations[\"is_clickbait\"]:\n",
    "            feedback.append(Feedback(\n",
    "                name = \"false negative\",\n",
    "                value = True))\n",
    "            feedback.append(Feedback(\n",
    "                name = \"false positive\",\n",
    "                value = False))\n",
    "        else :\n",
    "            feedback.append(Feedback(\n",
    "                name = \"false negative\",\n",
    "                value = False))\n",
    "            feedback.append(Feedback(\n",
    "                name = \"false positive\",\n",
    "                value = True))\n",
    "\n",
    "    return feedback\n",
    "\n",
    "evaluation = mlflow.genai.evaluate(\n",
    "    data=titles_dataset,\n",
    "    scorers=[check_clickbait],\n",
    "    predict_fn=run_classify_node\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "d505cad3-dba7-4834-a9be-5e520a8de04d",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "### Clickbait response correctness\n",
    "\n",
    "The `check_clickbait_response` scorer evaluates whether the agent successfully rewrites titles originally classified as clickbait, eliminating the clickbait aspect in the revised version.\n",
    "\n",
    "- The dataset includes title and content fields as inputs.\n",
    "- To isolate and evaluate the rewriting logic, the `predict_fn` wraps the invoke method of the `clickbait_response` node.\n",
    "- The scorer then uses the `classify` node as a judge, applying it to the rewritten output to determine whether it would still be classified as clickbait."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "2fcf15a4-1cae-4911-a498-870a36239817",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2025/06/24 11:03:48 INFO mlflow.genai.utils.data_validation: Testing model prediction with the first sample in the dataset.\n2025/06/24 11:03:49 INFO mlflow.tracking.fluent: Active model is set to the logged model with ID: m-3505dfc179ef47daba099a06eae6cec6\n2025/06/24 11:03:49 INFO mlflow.tracking.fluent: Use `mlflow.set_active_model` to set the active model to a different one if needed.\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d18a7c9f87f94d31b49b6c56b079f51d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Evaluating:   0%|          | 0/20 [Elapsed: 00:00, Remaining: ?] "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "<!DOCTYPE html>\n",
       "<html lang=\"en\">\n",
       "<head>\n",
       "    <title>Evaluation output</title>\n",
       "    <meta charset=\"UTF-8\">\n",
       "    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n",
       "    <style>\n",
       "        body {\n",
       "            font-family: Arial, sans-serif;\n",
       "        }\n",
       "\n",
       "        .header {\n",
       "            a.button {\n",
       "                padding: 4px 8px;\n",
       "                line-height: 20px;\n",
       "                box-shadow: none;\n",
       "                height: 20px;\n",
       "                display: inline-flex;\n",
       "                align-items: center;\n",
       "                justify-content: center;\n",
       "                vertical-align: middle;\n",
       "                background-color: rgb(34, 114, 180);\n",
       "                color: rgb(255, 255, 255);\n",
       "                text-decoration: none;\n",
       "                animation-duration: 0s;\n",
       "                transition: none 0s ease 0s;\n",
       "                position: relative;\n",
       "                white-space: nowrap;\n",
       "                text-align: center;\n",
       "                border: 1px solid rgb(192, 205, 216);\n",
       "                cursor: pointer;\n",
       "                user-select: none;\n",
       "                touch-action: manipulation;\n",
       "                border-radius: 4px;\n",
       "                gap: 6px;\n",
       "            }\n",
       "\n",
       "            a.button:hover {\n",
       "                background-color: rgb(14, 83, 139) !important;\n",
       "                border-color: transparent !important;\n",
       "                color: rgb(255, 255, 255) !important;\n",
       "            }\n",
       "        }\n",
       "\n",
       "        .warnings-section {\n",
       "            margin-top: 8px;\n",
       "\n",
       "            ul {\n",
       "                list-style-type: none;\n",
       "            }\n",
       "        }\n",
       "\n",
       "        .instructions-section {\n",
       "            margin-top: 16px;\n",
       "            font-size: 14px;\n",
       "\n",
       "            ul {\n",
       "                margin-top: 0;\n",
       "                margin-bottom: 0;\n",
       "            }\n",
       "        }\n",
       "\n",
       "        code {\n",
       "            font-family: monospace;\n",
       "        }\n",
       "\n",
       "        .note {\n",
       "            color: #666;\n",
       "        }\n",
       "\n",
       "        a {\n",
       "            color: #2272B4;\n",
       "            text-decoration: none;\n",
       "        }\n",
       "\n",
       "        a:hover {\n",
       "            color: #005580;\n",
       "        }\n",
       "    </style>\n",
       "</head>\n",
       "<body>\n",
       "<div>\n",
       "    <div class=\"header\">\n",
       "        <a href=\"https://dbc-5310e2d8-d9d5.cloud.databricks.com/ml/experiments/ddd3d8c02f3640519cbd336f99d256d7/runs/fa2ca816fe9843b4a6b6db206ec262b4/evaluations/\" class=\"button\">\n",
       "            View evaluation results.\n",
       "            <svg xmlns=\"http://www.w3.org/2000/svg\" width=\"1em\" height=\"1em\" fill=\"none\" viewBox=\"0 0 16 16\" aria-hidden=\"true\" focusable=\"false\" class=\"\">\n",
       "                <path fill=\"currentColor\" d=\"M10 1h5v5h-1.5V3.56L8.53 8.53 7.47 7.47l4.97-4.97H10z\"></path>\n",
       "                <path fill=\"currentColor\" d=\"M1 2.75A.75.75 0 0 1 1.75 2H8v1.5H2.5v10h10V8H14v6.25a.75.75 0 0 1-.75.75H1.75a.75.75 0 0 1-.75-.75z\"></path>\n",
       "            </svg>\n",
       "        </a>\n",
       "    </div>\n",
       "</div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "output_type": "display_data",
     "data": {
      "application/databricks.mlflow.trace": "[\"tr-8f1ec97bce68ac772ca165640f81f01c\", \"tr-e0f860c862b29f920ce46b2e846f46fc\", \"tr-46cc4059d4974b4f9bc787fa926ce7b3\", \"tr-7881e9d0764cd2b3bbd8b2e41f0e5971\", \"tr-ad79b7e6bfb02c6bbc3b9de27238fdf3\", \"tr-b41e8b69bb5bf9f10c5599ae64e922aa\", \"tr-d261cd3b76be6d57d8d9ab2d2cd8ac76\", \"tr-d0bf3896118a42fd1acddb6ddbd119c0\", \"tr-f1305d5fcc3172f03948a6591e9714ec\", \"tr-65cb34b61daadeb5894f49942d30df15\"]",
      "text/plain": [
       "[Trace(trace_id=tr-8f1ec97bce68ac772ca165640f81f01c), Trace(trace_id=tr-e0f860c862b29f920ce46b2e846f46fc), Trace(trace_id=tr-46cc4059d4974b4f9bc787fa926ce7b3), Trace(trace_id=tr-7881e9d0764cd2b3bbd8b2e41f0e5971), Trace(trace_id=tr-ad79b7e6bfb02c6bbc3b9de27238fdf3), Trace(trace_id=tr-b41e8b69bb5bf9f10c5599ae64e922aa), Trace(trace_id=tr-d261cd3b76be6d57d8d9ab2d2cd8ac76), Trace(trace_id=tr-d0bf3896118a42fd1acddb6ddbd119c0), Trace(trace_id=tr-f1305d5fcc3172f03948a6591e9714ec), Trace(trace_id=tr-65cb34b61daadeb5894f49942d30df15)]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "clickbait_titles_dataset = (\n",
    "    url_articles\n",
    "    .apply(\n",
    "        lambda row: {\n",
    "            \"inputs\": {\n",
    "                \"title\": row[\"title\"],\n",
    "                \"content\": row[\"content\"]\n",
    "            }\n",
    "        },\n",
    "        axis=1,\n",
    "        result_type=\"expand\",\n",
    "    )\n",
    ")\n",
    "\n",
    "def run_clickbait_response_node(title: str, content: str):\n",
    "    return agent.nodes[\"clickbait_response\"].invoke(\n",
    "        ClickbaitAgentState(title=title, content=content)\n",
    "    )\n",
    "\n",
    "@scorer\n",
    "def check_clickbait_response(inputs, outputs):\n",
    "    output_title = outputs[\"messages\"][-1].content\n",
    "    judge_output = agent.nodes[\"classify\"].invoke(\n",
    "        ClickbaitAgentState(title=output_title)\n",
    "    )\n",
    "    return Feedback(\n",
    "        name=\"response is no clickbait\",\n",
    "        value=not judge_output[\"is_clickbait\"],\n",
    "        rationale=judge_output[\"classification_reason\"],\n",
    "    )\n",
    "\n",
    "evaluation = mlflow.genai.evaluate(\n",
    "    data=clickbait_titles_dataset,\n",
    "    scorers=[check_clickbait_response],\n",
    "    predict_fn=run_clickbait_response_node\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "22e6391a-ca71-48ed-a68e-63325522264c",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "### Tool call correctness\n",
    "\n",
    "The `check_tool_use` scorer verifies whether the agent invoked the expected tools during execution.\n",
    "\n",
    "- The dataset consists of user messages (in this case, URLs) as inputs, with expectations specifying that the agent should call the `fetch_url_title_and_content` tool.\n",
    "- The `predict_fn` wraps the full agent invocation, simulating a user message being processed end-to-end.\n",
    "- The scorer inspects the execution trace and collects all tool invocations by searching for spans of type `\"TOOL\"`. It stores the tool names in a set and compares them to the `expected_tools`, which are also converted into a set.\n",
    "\n",
    "Note: Tool names are stored in sets to ignore the order of invocation. This simplifies comparison but does mean that repeated invocations of the same tool (e.g., retries or multiple uses) are ignored. MLflow automatically renames multiple executions of the same tool-appending suffixes like _1, _2, etc- so information about how many times each tool was called is not lost when storing the tool call names in a set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "3128e56a-2376-49bd-bdc3-51d678640a70",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2025/06/24 11:03:58 INFO mlflow.genai.utils.data_validation: Testing model prediction with the first sample in the dataset.\n2025/06/24 11:04:13 INFO mlflow.tracking.fluent: Active model is set to the logged model with ID: m-3505dfc179ef47daba099a06eae6cec6\n2025/06/24 11:04:13 INFO mlflow.tracking.fluent: Use `mlflow.set_active_model` to set the active model to a different one if needed.\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2acd28b455dc4520915a16ea88f7952a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Evaluating:   0%|          | 0/20 [Elapsed: 00:00, Remaining: ?] "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "<!DOCTYPE html>\n",
       "<html lang=\"en\">\n",
       "<head>\n",
       "    <title>Evaluation output</title>\n",
       "    <meta charset=\"UTF-8\">\n",
       "    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n",
       "    <style>\n",
       "        body {\n",
       "            font-family: Arial, sans-serif;\n",
       "        }\n",
       "\n",
       "        .header {\n",
       "            a.button {\n",
       "                padding: 4px 8px;\n",
       "                line-height: 20px;\n",
       "                box-shadow: none;\n",
       "                height: 20px;\n",
       "                display: inline-flex;\n",
       "                align-items: center;\n",
       "                justify-content: center;\n",
       "                vertical-align: middle;\n",
       "                background-color: rgb(34, 114, 180);\n",
       "                color: rgb(255, 255, 255);\n",
       "                text-decoration: none;\n",
       "                animation-duration: 0s;\n",
       "                transition: none 0s ease 0s;\n",
       "                position: relative;\n",
       "                white-space: nowrap;\n",
       "                text-align: center;\n",
       "                border: 1px solid rgb(192, 205, 216);\n",
       "                cursor: pointer;\n",
       "                user-select: none;\n",
       "                touch-action: manipulation;\n",
       "                border-radius: 4px;\n",
       "                gap: 6px;\n",
       "            }\n",
       "\n",
       "            a.button:hover {\n",
       "                background-color: rgb(14, 83, 139) !important;\n",
       "                border-color: transparent !important;\n",
       "                color: rgb(255, 255, 255) !important;\n",
       "            }\n",
       "        }\n",
       "\n",
       "        .warnings-section {\n",
       "            margin-top: 8px;\n",
       "\n",
       "            ul {\n",
       "                list-style-type: none;\n",
       "            }\n",
       "        }\n",
       "\n",
       "        .instructions-section {\n",
       "            margin-top: 16px;\n",
       "            font-size: 14px;\n",
       "\n",
       "            ul {\n",
       "                margin-top: 0;\n",
       "                margin-bottom: 0;\n",
       "            }\n",
       "        }\n",
       "\n",
       "        code {\n",
       "            font-family: monospace;\n",
       "        }\n",
       "\n",
       "        .note {\n",
       "            color: #666;\n",
       "        }\n",
       "\n",
       "        a {\n",
       "            color: #2272B4;\n",
       "            text-decoration: none;\n",
       "        }\n",
       "\n",
       "        a:hover {\n",
       "            color: #005580;\n",
       "        }\n",
       "    </style>\n",
       "</head>\n",
       "<body>\n",
       "<div>\n",
       "    <div class=\"header\">\n",
       "        <a href=\"https://dbc-5310e2d8-d9d5.cloud.databricks.com/ml/experiments/ddd3d8c02f3640519cbd336f99d256d7/runs/6091338efd38440ab79d91be152584e9/evaluations/\" class=\"button\">\n",
       "            View evaluation results.\n",
       "            <svg xmlns=\"http://www.w3.org/2000/svg\" width=\"1em\" height=\"1em\" fill=\"none\" viewBox=\"0 0 16 16\" aria-hidden=\"true\" focusable=\"false\" class=\"\">\n",
       "                <path fill=\"currentColor\" d=\"M10 1h5v5h-1.5V3.56L8.53 8.53 7.47 7.47l4.97-4.97H10z\"></path>\n",
       "                <path fill=\"currentColor\" d=\"M1 2.75A.75.75 0 0 1 1.75 2H8v1.5H2.5v10h10V8H14v6.25a.75.75 0 0 1-.75.75H1.75a.75.75 0 0 1-.75-.75z\"></path>\n",
       "            </svg>\n",
       "        </a>\n",
       "    </div>\n",
       "</div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "output_type": "display_data",
     "data": {
      "application/databricks.mlflow.trace": "[\"tr-876c166d0f8f9758bb65b67f4dd938b9\", \"tr-61e3d40289989544614fd161d26629af\", \"tr-60211a55456b541afa70094026c1c8ee\", \"tr-cefbb0219a51562cbf6f2da864f09760\", \"tr-81224a14aea007c3466283ea3d50b7f7\", \"tr-346361759f634cc47e789290b8eedede\", \"tr-d1459550022334ebc78f4c73f1e29577\", \"tr-d71bc888967ec6b5a8fb5ba80cf9ae49\", \"tr-43550366f00d88b637841e01f8a17dc3\", \"tr-dc13d75f9f48c5a074888024def6b72a\"]",
      "text/plain": [
       "[Trace(trace_id=tr-876c166d0f8f9758bb65b67f4dd938b9), Trace(trace_id=tr-61e3d40289989544614fd161d26629af), Trace(trace_id=tr-60211a55456b541afa70094026c1c8ee), Trace(trace_id=tr-cefbb0219a51562cbf6f2da864f09760), Trace(trace_id=tr-81224a14aea007c3466283ea3d50b7f7), Trace(trace_id=tr-346361759f634cc47e789290b8eedede), Trace(trace_id=tr-d1459550022334ebc78f4c73f1e29577), Trace(trace_id=tr-d71bc888967ec6b5a8fb5ba80cf9ae49), Trace(trace_id=tr-43550366f00d88b637841e01f8a17dc3), Trace(trace_id=tr-dc13d75f9f48c5a074888024def6b72a)]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tool_use_dataset = url_articles.apply(\n",
    "    lambda row: {\n",
    "        \"inputs\": {\"user_message\": row[\"url\"]},\n",
    "        \"expectations\": {\"expected_tools\": [\"fetch_url_title_and_content\"]},\n",
    "    },\n",
    "    axis=1,\n",
    "    result_type=\"expand\",\n",
    ")\n",
    "\n",
    "def invoke_agent(user_message: str):\n",
    "    return agent.invoke(\n",
    "        {\"messages\": [HumanMessage(content=user_message)]}\n",
    "    )\n",
    "\n",
    "@scorer\n",
    "def check_tool_use(trace, expectations):\n",
    "    tool_calls = {span.name for span in trace.search_spans(span_type=\"TOOL\")}\n",
    "\n",
    "    if not tool_calls:\n",
    "        return Feedback(value=False, rationale=\"No tool calls found\")\n",
    "    \n",
    "    expected_tools = set(expectations[\"expected_tools\"])\n",
    "\n",
    "    if expected_tools != tool_calls:\n",
    "        return Feedback(\n",
    "            value=False,\n",
    "            rationale=(\n",
    "                \"Tool calls did not match expectations.\\n\"\n",
    "                f\"Expected {expected_tools} but got {tool_calls}.\"\n",
    "            )\n",
    "        )\n",
    "    return Feedback(value=True)\n",
    "\n",
    "check_tool_use_eval_result = mlflow.genai.evaluate(\n",
    "    data=tool_use_dataset, scorers=[check_tool_use], predict_fn=invoke_agent\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "8d6e9121-9a55-45ea-9c86-b8629b7c981e",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "### Predefined judges\n",
    "\n",
    "MLflow GenAI provides several built-in judges, that can be used to evaluate common behavioral expectations without writing custom logic. Two such predefined judges are `Safety` and `Guidelines`.\n",
    "\n",
    "- The `Safety` scorer evaluates content—whether generated by the application or provided by a user—for harmful, unethical, or inappropriate material.\n",
    "- The `Guidelines` scorer enables fast and flexible evaluation based on natural language rules, framed as binary pass/fail conditions. These criteria can be tailored to specific application constraints or behavioral policies.\n",
    "\n",
    "Unlike earlier examples that require defining a `predict_fn` to execute and evaluate an agent function, this approach uses the data parameter to evaluate previously executed traces. This is particularly useful when agent outputs have already been logged—such as during experimentation or batch processing—and additional scoring needs to be applied without re-executing the agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "ee15813f-8539-4507-a781-473f608748b7",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2025/06/24 11:04:48 INFO mlflow.tracking.fluent: Active model is set to the logged model with ID: m-3505dfc179ef47daba099a06eae6cec6\n2025/06/24 11:04:48 INFO mlflow.tracking.fluent: Use `mlflow.set_active_model` to set the active model to a different one if needed.\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c4063d0b7f824b2aa540ac5126437cb7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Evaluating:   0%|          | 0/20 [Elapsed: 00:00, Remaining: ?] "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "<!DOCTYPE html>\n",
       "<html lang=\"en\">\n",
       "<head>\n",
       "    <title>Evaluation output</title>\n",
       "    <meta charset=\"UTF-8\">\n",
       "    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n",
       "    <style>\n",
       "        body {\n",
       "            font-family: Arial, sans-serif;\n",
       "        }\n",
       "\n",
       "        .header {\n",
       "            a.button {\n",
       "                padding: 4px 8px;\n",
       "                line-height: 20px;\n",
       "                box-shadow: none;\n",
       "                height: 20px;\n",
       "                display: inline-flex;\n",
       "                align-items: center;\n",
       "                justify-content: center;\n",
       "                vertical-align: middle;\n",
       "                background-color: rgb(34, 114, 180);\n",
       "                color: rgb(255, 255, 255);\n",
       "                text-decoration: none;\n",
       "                animation-duration: 0s;\n",
       "                transition: none 0s ease 0s;\n",
       "                position: relative;\n",
       "                white-space: nowrap;\n",
       "                text-align: center;\n",
       "                border: 1px solid rgb(192, 205, 216);\n",
       "                cursor: pointer;\n",
       "                user-select: none;\n",
       "                touch-action: manipulation;\n",
       "                border-radius: 4px;\n",
       "                gap: 6px;\n",
       "            }\n",
       "\n",
       "            a.button:hover {\n",
       "                background-color: rgb(14, 83, 139) !important;\n",
       "                border-color: transparent !important;\n",
       "                color: rgb(255, 255, 255) !important;\n",
       "            }\n",
       "        }\n",
       "\n",
       "        .warnings-section {\n",
       "            margin-top: 8px;\n",
       "\n",
       "            ul {\n",
       "                list-style-type: none;\n",
       "            }\n",
       "        }\n",
       "\n",
       "        .instructions-section {\n",
       "            margin-top: 16px;\n",
       "            font-size: 14px;\n",
       "\n",
       "            ul {\n",
       "                margin-top: 0;\n",
       "                margin-bottom: 0;\n",
       "            }\n",
       "        }\n",
       "\n",
       "        code {\n",
       "            font-family: monospace;\n",
       "        }\n",
       "\n",
       "        .note {\n",
       "            color: #666;\n",
       "        }\n",
       "\n",
       "        a {\n",
       "            color: #2272B4;\n",
       "            text-decoration: none;\n",
       "        }\n",
       "\n",
       "        a:hover {\n",
       "            color: #005580;\n",
       "        }\n",
       "    </style>\n",
       "</head>\n",
       "<body>\n",
       "<div>\n",
       "    <div class=\"header\">\n",
       "        <a href=\"https://dbc-5310e2d8-d9d5.cloud.databricks.com/ml/experiments/ddd3d8c02f3640519cbd336f99d256d7/runs/806bfa28c2974aaf88655ca4c2a0e151/evaluations/\" class=\"button\">\n",
       "            View evaluation results.\n",
       "            <svg xmlns=\"http://www.w3.org/2000/svg\" width=\"1em\" height=\"1em\" fill=\"none\" viewBox=\"0 0 16 16\" aria-hidden=\"true\" focusable=\"false\" class=\"\">\n",
       "                <path fill=\"currentColor\" d=\"M10 1h5v5h-1.5V3.56L8.53 8.53 7.47 7.47l4.97-4.97H10z\"></path>\n",
       "                <path fill=\"currentColor\" d=\"M1 2.75A.75.75 0 0 1 1.75 2H8v1.5H2.5v10h10V8H14v6.25a.75.75 0 0 1-.75.75H1.75a.75.75 0 0 1-.75-.75z\"></path>\n",
       "            </svg>\n",
       "        </a>\n",
       "    </div>\n",
       "</div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "<mlflow.models.evaluation.base.EvaluationResult at 0x718a1b2a0bc0>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "output_type": "display_data",
     "data": {
      "application/databricks.mlflow.trace": "[\"tr-5f1c1f65a456a1ce427e946bfebd8441\", \"tr-dc13d75f9f48c5a074888024def6b72a\", \"tr-2e2ba56a964708470c413552e29752fd\", \"tr-d31fe933873ee75a035583319bc5fbd2\", \"tr-d6e9124245995f7cb8b8bed9ab29f22f\", \"tr-cc042bd8881398b87123a483698016ac\", \"tr-af28a1961642df91f100d885e7cb4aca\", \"tr-61e3d40289989544614fd161d26629af\", \"tr-81224a14aea007c3466283ea3d50b7f7\", \"tr-346361759f634cc47e789290b8eedede\"]",
      "text/plain": [
       "[Trace(trace_id=tr-5f1c1f65a456a1ce427e946bfebd8441), Trace(trace_id=tr-dc13d75f9f48c5a074888024def6b72a), Trace(trace_id=tr-2e2ba56a964708470c413552e29752fd), Trace(trace_id=tr-d31fe933873ee75a035583319bc5fbd2), Trace(trace_id=tr-d6e9124245995f7cb8b8bed9ab29f22f), Trace(trace_id=tr-cc042bd8881398b87123a483698016ac), Trace(trace_id=tr-af28a1961642df91f100d885e7cb4aca), Trace(trace_id=tr-61e3d40289989544614fd161d26629af), Trace(trace_id=tr-81224a14aea007c3466283ea3d50b7f7), Trace(trace_id=tr-346361759f634cc47e789290b8eedede)]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from mlflow.genai.scorers import Safety, Guidelines\n",
    "\n",
    "traces = mlflow.search_traces(run_id=check_tool_use_eval_result.run_id)\n",
    "\n",
    "mlflow.genai.evaluate(\n",
    "    data=traces,\n",
    "    scorers=[\n",
    "        Safety(),\n",
    "        Guidelines(name=\"question\", guidelines=\"The response must not contain a question\")\n",
    "    ],\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "application/vnd.databricks.v1+notebook": {
   "computePreferences": null,
   "dashboards": [],
   "environmentMetadata": null,
   "inputWidgetPreferences": null,
   "language": "python",
   "notebookMetadata": {
    "pythonIndentUnit": 4
   },
   "notebookName": "Clickbait agent - LangGraph",
   "widgets": {}
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}