{ "cells": [ { "cell_type": "markdown", "id": "ed8376a3", "metadata": {}, "source": [ "# Session 5 – Multi-Agent Orchestrator\n", "\n", "Demonstrates a simple two-agent pipeline (Researcher -> Editor) using Foundry Local." ] }, { "cell_type": "markdown", "id": "e8bef206", "metadata": {}, "source": [ "### Explanation: Dependency Installation\n", "Installs `foundry-local-sdk` and `openai` needed for local model access and chat completions. Idempotent." ] }, { "cell_type": "markdown", "id": "ec32871a", "metadata": {}, "source": [ "# Scenario\n", "Implements a minimal two‑agent orchestrator pattern:\n", "- **Researcher agent** gathers concise factual bullets\n", "- **Editor agent** rewrites for executive clarity\n", "\n", "Demonstrates shared memory per agent, sequential passing of intermediate output, and a simple pipeline function. Extendable to more roles (e.g., Critic, Verifier) or parallel branches.\n", "\n", "**Environment Variables:**\n", "- `FOUNDRY_LOCAL_ALIAS` - Default model to use (default: phi-4-mini)\n", "- `AGENT_MODEL_PRIMARY` - Primary agent model (overrides ALIAS)\n", "- `AGENT_MODEL_EDITOR` - Editor agent model (defaults to primary)\n", "\n", "**SDK Reference:** https://github.com/microsoft/Foundry-Local/tree/main/sdk/python/foundry_local\n", "\n", "**How it works:**\n", "1. **FoundryLocalManager** automatically starts the Foundry Local service\n", "2. Downloads and loads the specified model (or uses cached version)\n", "3. Provides an OpenAI-compatible endpoint for interaction\n", "4. Each agent can use a different model for specialized tasks\n", "5. Built-in retry logic handles transient failures gracefully\n", "\n", "**Key Features:**\n", "- ✅ Automatic service discovery and initialization\n", "- ✅ Model lifecycle management (download, cache, load)\n", "- ✅ OpenAI SDK compatibility for familiar API\n", "- ✅ Multi-model support for agent specialization\n", "- ✅ Robust error handling with retry logic\n", "- ✅ Local inference (no cloud API required)\n" ] }, { "cell_type": "code", "execution_count": 16, "id": "d91c342c", "metadata": {}, "outputs": [], "source": [ "# Install dependencies\n", "!pip install -q foundry-local-sdk openai" ] }, { "cell_type": "markdown", "id": "45db22f6", "metadata": {}, "source": [ "### Explanation: Core Imports & Typing\n", "Introduces dataclasses for agent message storage and typing hints for clarity. Imports Foundry Local manager + OpenAI client for subsequent agent actions." ] }, { "cell_type": "code", "execution_count": 17, "id": "72d53845", "metadata": {}, "outputs": [], "source": [ "from dataclasses import dataclass, field\n", "from typing import List\n", "import os\n", "from foundry_local import FoundryLocalManager\n", "from openai import OpenAI" ] }, { "cell_type": "markdown", "id": "9803e00f", "metadata": {}, "source": [ "### Explanation: Model Initialization (SDK Pattern)\n", "Uses Foundry Local Python SDK for robust model management:\n", "- **FoundryLocalManager(alias)** - Auto-starts service and loads model by alias\n", "- **get_model_info(alias)** - Resolves alias to concrete model ID\n", "- **manager.endpoint** - Provides the service endpoint for OpenAI client\n", "- **manager.api_key** - Provides API key (optional for local usage)\n", "- Supports separate models for different agents (primary vs editor)\n", "- Built-in retry logic with exponential backoff for resilience\n", "- Connection verification to ensure service is ready\n", "\n", "**Key SDK Pattern:**\n", "```python\n", "manager = FoundryLocalManager(alias)\n", "model_info = manager.get_model_info(alias)\n", "client = OpenAI(base_url=manager.endpoint, api_key=manager.api_key)\n", "```\n", "\n", "**Lifecycle Management:**\n", "- Managers are stored globally for proper cleanup\n", "- Each agent can use a different model for specialization\n", "- Automatic service discovery and connection handling\n", "- Graceful retry with exponential backoff on failures\n", "\n", "This ensures proper initialization before agent orchestration begins.\n", "\n", "**Reference:** https://github.com/microsoft/Foundry-Local/tree/main/sdk/python/foundry_local\n" ] }, { "cell_type": "code", "execution_count": 18, "id": "6552004f", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "================================================================================\n", "Initializing Primary Model: phi-4-mini\n", "================================================================================\n", "[Init] Starting Foundry Local for 'phi-4-mini' (attempt 1/3)...\n", "[OK] Initialized 'phi-4-mini' -> Phi-4-mini-instruct-cuda-gpu:4 at http://127.0.0.1:59959/v1\n", "\n", "================================================================================\n", "Initializing Editor Model: gpt-oss-20b\n", "================================================================================\n", "[Init] Starting Foundry Local for 'gpt-oss-20b' (attempt 1/3)...\n", "[OK] Initialized 'gpt-oss-20b' -> gpt-oss-20b-cuda-gpu:1 at http://127.0.0.1:59959/v1\n", "\n", "================================================================================\n", "[Configuration Summary]\n", "================================================================================\n", " Primary Agent:\n", " - Alias: phi-4-mini\n", " - Model: Phi-4-mini-instruct-cuda-gpu:4\n", " - Endpoint: http://127.0.0.1:59959/v1\n", "\n", " Editor Agent:\n", " - Alias: gpt-oss-20b\n", " - Model: gpt-oss-20b-cuda-gpu:1\n", " - Endpoint: http://127.0.0.1:59959/v1\n", "================================================================================\n" ] } ], "source": [ "import time\n", "\n", "# Environment configuration\n", "PRIMARY_ALIAS = os.getenv('AGENT_MODEL_PRIMARY', os.getenv('FOUNDRY_LOCAL_ALIAS', 'phi-4-mini'))\n", "EDITOR_ALIAS = os.getenv('AGENT_MODEL_EDITOR', PRIMARY_ALIAS)\n", "\n", "# Store managers globally for proper lifecycle management\n", "primary_manager = None\n", "editor_manager = None\n", "\n", "def init_model(alias: str, max_retries: int = 3):\n", " \"\"\"Initialize Foundry Local manager with retry logic.\n", " \n", " Args:\n", " alias: Model alias to initialize\n", " max_retries: Number of retry attempts with exponential backoff\n", " \n", " Returns:\n", " Tuple of (manager, client, model_id, endpoint)\n", " \"\"\"\n", " delay = 2.0\n", " last_err = None\n", " \n", " for attempt in range(1, max_retries + 1):\n", " try:\n", " print(f\"[Init] Starting Foundry Local for '{alias}' (attempt {attempt}/{max_retries})...\")\n", " \n", " # Initialize manager - this starts the service and loads the model\n", " manager = FoundryLocalManager(alias)\n", " \n", " # Get model info to retrieve the actual model ID\n", " model_info = manager.get_model_info(alias)\n", " model_id = model_info.id\n", " \n", " # Create OpenAI client with manager's endpoint\n", " client = OpenAI(\n", " base_url=manager.endpoint,\n", " api_key=manager.api_key or 'not-needed'\n", " )\n", " \n", " # Verify the connection with a simple test\n", " models = client.models.list()\n", " print(f\"[OK] Initialized '{alias}' -> {model_id} at {manager.endpoint}\")\n", " \n", " return manager, client, model_id, manager.endpoint\n", " \n", " except Exception as e:\n", " last_err = e\n", " if attempt < max_retries:\n", " print(f\"[Retry {attempt}/{max_retries}] Failed to init '{alias}': {e}\")\n", " print(f\"[Retry] Waiting {delay:.1f}s before retry...\")\n", " time.sleep(delay)\n", " delay *= 2\n", " else:\n", " print(f\"[ERROR] Failed to initialize '{alias}' after {max_retries} attempts\")\n", " \n", " raise RuntimeError(f\"Failed to initialize '{alias}' after {max_retries} attempts: {last_err}\")\n", "\n", "# Initialize primary model (for researcher)\n", "print(f\"\\n{'='*80}\")\n", "print(f\"Initializing Primary Model: {PRIMARY_ALIAS}\")\n", "print('='*80)\n", "primary_manager, primary_client, PRIMARY_MODEL_ID, primary_endpoint = init_model(PRIMARY_ALIAS)\n", "\n", "# Initialize editor model (may be same as primary)\n", "if EDITOR_ALIAS != PRIMARY_ALIAS:\n", " print(f\"\\n{'='*80}\")\n", " print(f\"Initializing Editor Model: {EDITOR_ALIAS}\")\n", " print('='*80)\n", " editor_manager, editor_client, EDITOR_MODEL_ID, editor_endpoint = init_model(EDITOR_ALIAS)\n", "else:\n", " print(f\"\\n[Info] Editor using same model as primary\")\n", " editor_manager = primary_manager\n", " editor_client, EDITOR_MODEL_ID = primary_client, PRIMARY_MODEL_ID\n", " editor_endpoint = primary_endpoint\n", "\n", "print(f\"\\n{'='*80}\")\n", "print(f\"[Configuration Summary]\")\n", "print('='*80)\n", "print(f\" Primary Agent:\")\n", "print(f\" - Alias: {PRIMARY_ALIAS}\")\n", "print(f\" - Model: {PRIMARY_MODEL_ID}\")\n", "print(f\" - Endpoint: {primary_endpoint}\")\n", "print(f\"\\n Editor Agent:\")\n", "print(f\" - Alias: {EDITOR_ALIAS}\")\n", "print(f\" - Model: {EDITOR_MODEL_ID}\")\n", "print(f\" - Endpoint: {editor_endpoint}\")\n", "print('='*80)\n" ] }, { "cell_type": "markdown", "id": "817abb14", "metadata": {}, "source": [ "### Explanation: Agent & Memory Classes\n", "Defines lightweight `AgentMsg` for memory entries and `Agent` encapsulating:\n", "- **System role** - Agent's persona and instructions\n", "- **Message history** - Maintains conversation context\n", "- **act() method** - Executes actions with proper error handling\n", "\n", "The agent can use different models (primary vs editor) and maintains isolated context per agent. This pattern enables:\n", "- Memory persistence across actions\n", "- Flexible model assignment per agent\n", "- Error isolation and recovery\n", "- Easy chaining and orchestration" ] }, { "cell_type": "code", "execution_count": 19, "id": "3e535cdc", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[INFO] Agent classes initialized with Foundry SDK support\n", "[INFO] Using OpenAI SDK version: openai\n" ] } ], "source": [ "@dataclass\n", "class AgentMsg:\n", " role: str\n", " content: str\n", "\n", "@dataclass\n", "class Agent:\n", " name: str\n", " system: str\n", " client: OpenAI = None # Allow per-agent client assignment\n", " model_id: str = None # Allow per-agent model\n", " memory: List[AgentMsg] = field(default_factory=list)\n", "\n", " def _history(self):\n", " \"\"\"Return chat history in OpenAI messages format including system + memory.\"\"\"\n", " msgs = [{'role': 'system', 'content': self.system}]\n", " for m in self.memory[-6:]: # Keep last 6 messages to avoid context overflow\n", " msgs.append({'role': m.role, 'content': m.content})\n", " return msgs\n", "\n", " def act(self, prompt: str, temperature: float = 0.4, max_tokens: int = 300):\n", " \"\"\"Send a prompt, store user + assistant messages in memory, and return assistant text.\n", " \n", " Args:\n", " prompt: User input/task for the agent\n", " temperature: Sampling temperature (0.0-1.0)\n", " max_tokens: Maximum tokens to generate\n", " \n", " Returns:\n", " Assistant response text\n", " \"\"\"\n", " # Use agent-specific client/model or fall back to primary\n", " client_to_use = self.client or primary_client\n", " model_to_use = self.model_id or PRIMARY_MODEL_ID\n", " \n", " self.memory.append(AgentMsg('user', prompt))\n", " \n", " try:\n", " # Build messages including system prompt and history\n", " messages = self._history() + [{'role': 'user', 'content': prompt}]\n", " \n", " resp = client_to_use.chat.completions.create(\n", " model=model_to_use,\n", " messages=messages,\n", " max_tokens=max_tokens,\n", " temperature=temperature,\n", " )\n", " \n", " # Validate response\n", " if not resp.choices:\n", " raise RuntimeError(\"No completion choices returned\")\n", " \n", " out = resp.choices[0].message.content or \"\"\n", " \n", " if not out:\n", " raise RuntimeError(\"Empty response content\")\n", " \n", " except Exception as e:\n", " out = f\"[ERROR:{self.name}] {type(e).__name__}: {str(e)}\"\n", " print(f\"[Agent Error] {self.name}: {type(e).__name__}: {str(e)}\")\n", " \n", " self.memory.append(AgentMsg('assistant', out))\n", " return out\n", "\n", "print(\"[INFO] Agent classes initialized with Foundry SDK support\")\n", "print(f\"[INFO] Using OpenAI SDK version: {OpenAI.__module__}\")\n" ] }, { "cell_type": "markdown", "id": "6ca1f4bd", "metadata": {}, "source": [ "### Explanation: Orchestrated Pipeline\n", "Creates two specialized agents:\n", "- **Researcher**: Uses primary model, gathers factual information\n", "- **Editor**: Can use separate model (if configured), refines and rewrites\n", "\n", "The `pipeline` function:\n", "1. Researcher gathers raw information\n", "2. Editor refines into executive-ready output\n", "3. Returns both intermediate and final results\n", "\n", "This pattern enables:\n", "- Model specialization (different models for different roles)\n", "- Quality improvement through multi-stage processing\n", "- Traceability of information transformation\n", "- Easy extension to more agents or parallel processing" ] }, { "cell_type": "code", "execution_count": null, "id": "e60bb753", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "================================================================================\n", "[Pipeline] Question: Explain why edge AI matters for compliance and latency.\n", "\n", "[Stage 1: Research]\n", "Output: - **Data Sovereignty**: Edge AI allows data to be processed locally, which can help organizations comply with regional data protection regulations by keeping sensitive information within the borders o...\n", "\n", "[Stage 2: Editorial Refinement]\n" ] } ], "source": [ "# Create specialized agents with optional model assignment\n", "researcher = Agent(\n", " name='Researcher',\n", " system='You collect concise factual bullet points.',\n", " client=primary_client,\n", " model_id=PRIMARY_MODEL_ID\n", ")\n", "\n", "editor = Agent(\n", " name='Editor',\n", " system='You rewrite content for clarity and an executive, action-focused tone.',\n", " client=editor_client,\n", " model_id=EDITOR_MODEL_ID\n", ")\n", "\n", "def pipeline(q: str, verbose: bool = True):\n", " \"\"\"Execute multi-agent pipeline: Researcher -> Editor.\n", " \n", " Args:\n", " q: User question/task\n", " verbose: Print intermediate outputs\n", " \n", " Returns:\n", " Dictionary with research, final outputs, and metadata\n", " \"\"\"\n", " if verbose:\n", " print(f\"[Pipeline] Question: {q}\\n\")\n", " \n", " # Stage 1: Research\n", " if verbose:\n", " print(\"[Stage 1: Research]\")\n", " research = researcher.act(q)\n", " if verbose:\n", " print(f\"Output: {research[:200]}...\\n\")\n", " \n", " # Stage 2: Editorial refinement\n", " if verbose:\n", " print(\"[Stage 2: Editorial Refinement]\")\n", " rewrite = editor.act(\n", " f\"Rewrite professionally with a 1-sentence executive summary first. \"\n", " f\"Improve clarity, keep bullet structure if present. Source:\\n{research}\"\n", " )\n", " if verbose:\n", " print(f\"Output: {rewrite[:200]}...\\n\")\n", " \n", " return {\n", " 'question': q,\n", " 'research': research,\n", " 'final': rewrite,\n", " 'models': {\n", " 'researcher': PRIMARY_MODEL_ID,\n", " 'editor': EDITOR_MODEL_ID\n", " }\n", " }\n", "\n", "# Execute sample pipeline\n", "print(\"=\"*80)\n", "result = pipeline('Explain why edge AI matters for compliance and latency.')\n", "print(\"=\"*80)\n", "print(\"\\n[FINAL OUTPUT]\")\n", "print(result['final'])\n", "print(\"\\n[METADATA]\")\n", "print(f\"Models used: {result['models']}\")\n", "result" ] }, { "cell_type": "markdown", "id": "a7038f2d", "metadata": {}, "source": [ "### Explanation: Pipeline Execution & Results\n", "Executes the multi-agent pipeline on a compliance + latency themed question to demonstrate:\n", "- Multi-stage information transformation\n", "- Agent specialization and collaboration\n", "- Output quality improvement through refinement\n", "- Traceability (both intermediate and final outputs preserved)\n", "\n", "**Result Structure:**\n", "- `question` - Original user query\n", "- `research` - Raw research output (factual bullets)\n", "- `final` - Refined executive summary\n", "- `models` - Which models were used for each stage\n", "\n", "**Extension Ideas:**\n", "1. Add a Critic agent for quality review\n", "2. Implement parallel research agents for different aspects\n", "3. Add a Verifier agent for fact-checking\n", "4. Use different models for different complexity levels\n", "5. Implement feedback loops for iterative improvement" ] }, { "cell_type": "markdown", "id": "1d7391fc", "metadata": {}, "source": [ "### Advanced: Custom Agent Configuration\n", "\n", "Try customizing agent behavior by modifying environment variables before running the initialization cell:\n", "\n", "**Available Models:**\n", "- Use `foundry model ls` in terminal to see all available models\n", "- Examples: phi-4-mini, phi-3.5-mini, qwen2.5-7b, llama-3.2-3b, etc.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "c6af178a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Testing pipeline with multiple questions:\n", "\n", "\n", "================================================================================\n", "Question 1: What are 3 key benefits of using small language models?\n", "================================================================================\n", "\n", "[FINAL]: <|channel|>analysis<|message|>The user wants a rewrite of the entire block of text. The rewrite should be professional, include a one-sentence executive summary first, improve clarity, keep bullet structure if present. The user has provided a large amount of text. The user wants a rewrite of that te...\n", "[Models]: Researcher=Phi-4-mini-instruct-cuda-gpu:4, Editor=gpt-oss-20b-cuda-gpu:1\n", "\n", "================================================================================\n", "Question 2: How does RAG improve AI accuracy?\n", "================================================================================\n", "\n", "[FINAL]: <|channel|>final<|message|>**RAG (Retrieval‑Augmented Generation) empowers AI to produce highly accurate, contextually relevant responses by combining a retrieval system with a large language model (LLM).**<|return|>...\n", "[Models]: Researcher=Phi-4-mini-instruct-cuda-gpu:4, Editor=gpt-oss-20b-cuda-gpu:1\n", "\n", "================================================================================\n", "Question 3: Why is local inference important for privacy?\n", "================================================================================\n", "\n", "[FINAL]: <|channel|>final<|message|>**Local inference—processing data directly on the device—offers a privacy‑first, security‑enhancing approach that meets regulatory requirements and builds user trust.**<|return|>...\n", "[Models]: Researcher=Phi-4-mini-instruct-cuda-gpu:4, Editor=gpt-oss-20b-cuda-gpu:1\n" ] } ], "source": [ "# Example: Use different models for different agents\n", "# Uncomment and modify as needed:\n", "\n", "# import os\n", "# os.environ['AGENT_MODEL_PRIMARY'] = 'phi-4-mini' # Fast, good for research\n", "# os.environ['AGENT_MODEL_EDITOR'] = 'qwen2.5-7b' # Higher quality for editing\n", "\n", "# Then restart the kernel and re-run all cells\n", "\n", "# Test with different questions\n", "test_questions = [\n", " \"What are 3 key benefits of using small language models?\",\n", " \"How does RAG improve AI accuracy?\",\n", " \"Why is local inference important for privacy?\"\n", "]\n", "\n", "print(\"Testing pipeline with multiple questions:\\n\")\n", "for i, q in enumerate(test_questions, 1):\n", " print(f\"\\n{'='*80}\")\n", " print(f\"Question {i}: {q}\")\n", " print('='*80)\n", " r = pipeline(q, verbose=False)\n", " print(f\"\\n[FINAL]: {r['final'][:300]}...\")\n", " print(f\"[Models]: Researcher={r['models']['researcher']}, Editor={r['models']['editor']}\")\n" ] } ], "metadata": { "kernelspec": { "display_name": "demo", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.15" } }, "nbformat": 4, "nbformat_minor": 5 }