{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"source": [
"[](https://colab.research.google.com/drive/1HAsxm1U4iCCrWay-dTGut3hmCPy0xqNH?usp=sharing)"
],
"metadata": {
"id": "AnNmjSBygJv4"
}
},
{
"cell_type": "markdown",
"source": [
"# Monitoring Llama Agents with Portkey\n",
"\n",
"[llama-agents](https://github.com/run-llama/llama-agents) is an agent framework for building, iterating, and productionizing multi-agent systems, including multi-agent communication, distributed tool execution, human-in-the-loop, and more!\n",
"\n",
"One of the major challenges of building AI agents is the lack of visibility into key performance metrics such as:\n",
"\n",
"- **Number of API Requests**: How many requests are being made to LLMs.\n",
"- **Token Usage & Costs**: How many tokens are being consumed and the cost for each agent run.\n",
"- **Latency**: How long is the agent taking to complete a set of tasks.\n",
"\n",
"
\n",
"\n",
"**Portkey** is an open source [**AI Gateway**](https://github.com/Portkey-AI/gateway) that helps you manage access to 250+ LLMs through a unified API while providing visibility into\n",
"\n",
"✅ cost,
\n",
"✅ performance, and
\n",
"✅ accuracy metrics.\n",
"\n",
"This notebook demonstrates how you can bring visibility and flexbility to Llama Agents using Portkey's AI Gateway."
],
"metadata": {
"id": "hWK3BmwYgSw9"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "oPwH-Y0lUmTa"
},
"outputs": [],
"source": [
"!pip install -qU llama-agents llama-index portkey-ai"
]
},
{
"cell_type": "code",
"source": [
"import nest_asyncio\n",
"nest_asyncio.apply()"
],
"metadata": {
"id": "3QwmT8wva5Zi"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"import os\n",
"from google.colab import userdata\n",
"\n",
"os.environ[\"OPENAI_API_KEY\"] = userdata.get('OPENAI_API_KEY')"
],
"metadata": {
"id": "6xEQtUUTzufn"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"### Llama Agents Setup"
],
"metadata": {
"id": "YAPO4mXpy_eR"
}
},
{
"cell_type": "markdown",
"source": [
"
"
],
"metadata": {
"id": "dUm3DMAayHW3"
}
},
{
"cell_type": "markdown",
"source": [
"### Quickstart\n",
"\n",
"Since Portkey is fully compatible with the OpenAI signature, you can connect to the Portkey AI Gateway through the OpenAI client.\n",
"\n",
"- Set the `base_url` as `PORTKEY_GATEWAY_URL`\n",
"- Add `default_headers` to consume the headers needed by Portkey using the `createHeaders` helper method."
],
"metadata": {
"id": "L6Pgo8R-2Fso"
}
},
{
"cell_type": "code",
"source": [
"from llama_index.llms.openai import OpenAI\n",
"from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders\n",
"from google.colab import userdata\n",
"\n",
"\n",
"gpt_4o_config = {\n",
" \"provider\": \"openai\",\n",
" \"api_key\": userdata.get('OPENAI_API_KEY'),\n",
" \"override_params\": { \"model\":\"gpt-4o\" }\n",
"}\n",
"\n",
"gpt_4o = OpenAI(\n",
" api_base=PORTKEY_GATEWAY_URL,\n",
" default_headers=createHeaders(\n",
" api_key=userdata.get('PORTKEY_API_KEY'),\n",
" config=gpt_4o_config\n",
" )\n",
")\n",
"\n",
"llama3_groq_config = {\n",
" \"provider\": \"groq\",\n",
" \"api_key\": userdata.get('GROQ_API_KEY'),\n",
" \"override_params\": {\"model\":\"llama3-70b-8192\"}\n",
"}\n",
"\n",
"llama3 = OpenAI(\n",
" api_base=PORTKEY_GATEWAY_URL,\n",
" default_headers=createHeaders(\n",
" api_key=userdata.get('PORTKEY_API_KEY'),\n",
" config=llama3_groq_config\n",
" )\n",
")"
],
"metadata": {
"id": "OHp4PT6cbb2k"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"### Creating Agents"
],
"metadata": {
"id": "sTs1vDRt2MOq"
}
},
{
"cell_type": "code",
"source": [
"from llama_agents import (\n",
" AgentService,\n",
" AgentOrchestrator,\n",
" ControlPlaneServer,\n",
" SimpleMessageQueue,\n",
")\n",
"\n",
"from llama_index.core.agent import ReActAgent\n",
"from llama_index.core.tools import FunctionTool\n",
"from llama_index.llms.openai import OpenAI\n",
"\n",
"\n",
"# create an agent\n",
"def get_the_secret_fact() -> str:\n",
" \"\"\"Returns the secret fact.\"\"\"\n",
" return \"The secret fact is: A baby llama is called a 'Cria'.\"\n",
"\n",
"\n",
"tool = FunctionTool.from_defaults(fn=get_the_secret_fact)\n",
"\n",
"agent1 = ReActAgent.from_tools([tool], llm=llama3)\n",
"agent2 = ReActAgent.from_tools([], llm=llama3)\n",
"\n",
"# create our multi-agent framework components\n",
"message_queue = SimpleMessageQueue()\n",
"control_plane = ControlPlaneServer(\n",
" message_queue=message_queue,\n",
" orchestrator=AgentOrchestrator(llm=gpt_4o),\n",
")\n",
"\n",
"agent_server_1 = AgentService(\n",
" agent=agent1,\n",
" message_queue=message_queue,\n",
" description=\"Useful for getting the secret fact.\",\n",
" service_name=\"secret_fact_agent\"\n",
")\n",
"\n",
"agent_server_2 = AgentService(\n",
" agent=agent2,\n",
" message_queue=message_queue,\n",
" description=\"Useful for getting random dumb facts.\",\n",
" service_name=\"dumb_fact_agent\"\n",
")"
],
"metadata": {
"id": "rfuiemWcdASk"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"### Launching Agents"
],
"metadata": {
"id": "FdPF5lpa2Rah"
}
},
{
"cell_type": "code",
"source": [
"from llama_agents import LocalLauncher\n",
"\n",
"# launch it\n",
"launcher = LocalLauncher(\n",
" [agent_server_1, agent_server_2],\n",
" control_plane,\n",
" message_queue,\n",
")\n",
"result = launcher.launch_single(\"What is the secret fact?\")\n",
"\n",
"print(f\"Result: {result}\")"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Mm5Y1MyjdXVd",
"outputId": "50485721-e39c-4fd7-b23e-0d673078ac89"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stderr",
"text": [
"INFO:llama_agents.message_queues.simple - Consumer AgentService-86ae75c9-9d5f-4ddf-88ea-c67d79506506: secret_fact_agent has been registered.\n",
"INFO:llama_agents.message_queues.simple - Consumer AgentService-d2c4ba62-3a1f-4f08-8fdc-10c49d7f5140: dumb_fact_agent has been registered.\n",
"INFO:llama_agents.message_queues.simple - Consumer 230c00bf-9ff1-4770-a92b-84c0cfb7e290: human has been registered.\n",
"INFO:llama_agents.message_queues.simple - Consumer ControlPlaneServer-2578d1b4-9576-4f4e-8107-c8a6829ca7a3: control_plane has been registered.\n",
"INFO:llama_agents.services.agent - secret_fact_agent launch_local\n",
"INFO:llama_agents.services.agent - dumb_fact_agent launch_local\n",
"INFO:llama_agents.message_queues.base - Publishing message to 'control_plane' with action 'new_task'\n",
"INFO:llama_agents.message_queues.simple - Launching message queue locally\n",
"INFO:llama_agents.message_queues.base - Publishing message to 'secret_fact_agent' with action 'new_task'\n",
"INFO:llama_agents.message_queues.simple - Successfully published message 'control_plane' to consumer.\n",
"INFO:llama_agents.message_queues.simple - Successfully published message 'secret_fact_agent' to consumer.\n",
"INFO:llama_agents.message_queues.base - Publishing message to 'control_plane' with action 'completed_task'\n",
"INFO:llama_agents.message_queues.base - Publishing message to 'human' with action 'completed_task'\n",
"INFO:llama_agents.message_queues.simple - Successfully published message 'control_plane' to consumer.\n",
"INFO:llama_agents.message_queues.simple - Successfully published message 'human' to consumer.\n"
]
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"Result: A baby llama is called a 'Cria'.\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"## Telemetry with Portkey\n"
],
"metadata": {
"id": "SQ7vS5dA2czk"
}
},
{
"cell_type": "markdown",
"source": [
"### Part 1: Logs\n",
"\n",
"For the above agent run, we have initiated two agents utilizing the GPT-4o and Llama 3 models. The detailed logs for this request are available on the Portkey dashboard, providing comprehensive insights into the activities of each agent."
],
"metadata": {
"id": "8dGZSfhxquVh"
}
},
{
"cell_type": "markdown",
"source": [
"Sample Logs\n",
"\n",
""
],
"metadata": {
"id": "FaloY-u1qy3x"
}
},
{
"cell_type": "markdown",
"source": [
"### Part 2: Analytics\n",
"\n",
"The insights from this request can be viewed on the Portkey dashboard. We are able to infer tokens used and number of API requests. You can also review many other metrics from the dashboards like cost, latency, etc."
],
"metadata": {
"id": "FG5m1y9crMnh"
}
},
{
"cell_type": "markdown",
"source": [
"Sample Dashboard\n",
"\n",
"
"
],
"metadata": {
"id": "7-F6oTFrrWW1"
}
},
{
"cell_type": "code",
"source": [],
"metadata": {
"id": "D-8gc_R-daKl"
},
"execution_count": null,
"outputs": []
}
]
}