{ "cells": [ { "cell_type": "markdown", "id": "e49ba9a1", "metadata": {}, "source": [ "# Session 4 – SLM vs LLM Comparison\n", "\n", "Compare latency and sample response quality between a Small Language Model and a larger model running via Foundry Local." ] }, { "cell_type": "markdown", "id": "18d9330f", "metadata": {}, "source": [ "## ⚑ Quick Start\n", "\n", "**Memory-Optimized Setup (Updated):**\n", "1. Models auto-select CPU variants (works on any hardware)\n", "2. Uses `qwen2.5-3b` instead of 7B (saves ~4GB RAM)\n", "3. Port auto-detection (no manual configuration)\n", "4. Total RAM needed: ~8GB recommended (models + OS)\n", "\n", "**Terminal Setup (30 seconds):**\n", "```bash\n", "foundry service start\n", "foundry model run phi-4-mini\n", "foundry model run qwen2.5-3b\n", "```\n", "\n", "Then run this notebook! πŸš€" ] }, { "cell_type": "markdown", "id": "9941ea8c", "metadata": {}, "source": [ "### Explanation: Dependency Installation\n", "Installs minimal packages (`foundry-local-sdk`, `openai`, `numpy`) needed for timing and chat requests. Safe to re-run idempotently." ] }, { "cell_type": "markdown", "id": "4690c7c7", "metadata": {}, "source": [ "# Scenario\n", "Compare a representative Small Language Model (SLM) with a larger model on a single prompt to illustrate trade‑offs:\n", "- **Latency difference** (wall clock seconds)\n", "- **Token usage** (if available) as a proxy for throughput\n", "- **Sample qualitative output** for quick eyeballing\n", "- **Speedup calculation** to quantify performance gains\n", "\n", "**Environment Variables:**\n", "- `SLM_ALIAS` - Small language model (default: phi-4-mini, ~4GB RAM)\n", "- `LLM_ALIAS` - Larger language model (default: qwen2.5-7b, ~7GB RAM)\n", "- `COMPARE_PROMPT` - Test prompt for comparison\n", "- `COMPARE_RETRIES` - Retry attempts for resilience (default: 2)\n", "- `FOUNDRY_LOCAL_ENDPOINT` - Override service endpoint (auto-detected if not set)\n", "\n", "**How It Works (Official SDK Pattern):**\n", "1. **FoundryLocalManager** initializes and manages the Foundry Local service\n", "2. Service auto-starts if not running (no manual setup needed)\n", "3. Models are resolved from aliases to concrete IDs automatically\n", "4. Hardware-optimized variants selected (CUDA, NPU, or CPU)\n", "5. OpenAI-compatible client performs chat completions\n", "6. Metrics are captured: latency, tokens, output quality\n", "7. Results are compared to calculate speedup ratio\n", "\n", "This micro‑comparison helps decide when routing to a bigger model is justified for your use case.\n", "\n", "**SDK Reference:** \n", "- Python SDK: https://github.com/microsoft/Foundry-Local/tree/main/sdk/python/foundry_local\n", "- Workshop Utils: Uses the official pattern from ../samples/workshop_utils.py\n", "\n", "**Key Benefits:**\n", "- βœ… Automatic service discovery and initialization\n", "- βœ… Auto-start service if not running\n", "- βœ… Built-in model resolution and caching\n", "- βœ… Hardware optimization (CUDA/NPU/CPU)\n", "- βœ… OpenAI SDK compatibility\n", "- βœ… Robust error handling with retries\n", "- βœ… Local inference (no cloud API required)" ] }, { "cell_type": "markdown", "id": "ea4813a6", "metadata": {}, "source": [ "## 🚨 Prerequisites: Foundry Local Must Be Running!\n", "\n", "**Before running this notebook**, ensure Foundry Local service is set up:\n", "\n", "### Quick Start Commands (Run in Terminal):\n", "\n", "```bash\n", "# 1. Start the Foundry Local service\n", "foundry service start\n", "\n", "# 2. Load the default models used in this comparison (CPU-optimized)\n", "foundry model run phi-4-mini\n", "foundry model run qwen2.5-3b\n", "\n", "# 3. Verify models are loaded\n", "foundry model ls\n", "\n", "# 4. Check service health\n", "foundry service status\n", "```\n", "\n", "### Alternative Models (if defaults aren't available):\n", "\n", "```bash\n", "# Even smaller alternatives (if memory is very limited)\n", "foundry model run phi-3.5-mini\n", "foundry model run qwen2.5-0.5b\n", "\n", "# Or update the environment variables in this notebook:\n", "# SLM_ALIAS = 'phi-3.5-mini'\n", "# LLM_ALIAS = 'qwen2.5-1.5b' # Or qwen2.5-0.5b for minimum memory\n", "```\n", "\n", "⚠️ **If you skip these steps**, you'll see `APIConnectionError` when running the notebook cells below." ] }, { "cell_type": "code", "execution_count": 29, "id": "e8ea63f9", "metadata": {}, "outputs": [], "source": [ "# Install dependencies\n", "!pip install -q foundry-local-sdk openai numpy requests" ] }, { "cell_type": "markdown", "id": "eb3d2ae1", "metadata": {}, "source": [ "### Explanation: Core Imports\n", "Brings in timing utilities and Foundry Local / OpenAI clients used to fetch model info and perform chat completions." ] }, { "cell_type": "code", "execution_count": 30, "id": "b1233c05", "metadata": {}, "outputs": [], "source": [ "import os, time, json\n", "from foundry_local import FoundryLocalManager\n", "from openai import OpenAI\n", "import sys\n", "sys.path.append('../samples')\n", "from workshop_utils import get_client, chat_once" ] }, { "cell_type": "markdown", "id": "03b6aa79", "metadata": {}, "source": [ "### Explanation: Aliases & Prompt Setup\n", "Defines environment-configurable aliases for small vs. larger model plus a comparison prompt. Adjust env vars to experiment with different model families or tasks." ] }, { "cell_type": "code", "execution_count": 31, "id": "f46b14ab", "metadata": {}, "outputs": [], "source": [ "# Default to CPU models for better memory efficiency\n", "SLM = os.getenv('SLM_ALIAS', 'phi-4-mini') # Auto-selects CPU variant\n", "LLM = os.getenv('LLM_ALIAS', 'qwen2.5-3b') # Smaller LLM, more memory-friendly\n", "PROMPT = os.getenv('COMPARE_PROMPT', 'List 5 benefits of local AI inference.')\n", "# Endpoint is now managed by FoundryLocalManager - it auto-detects or can be overridden\n", "ENDPOINT = os.getenv('FOUNDRY_LOCAL_ENDPOINT', None)" ] }, { "cell_type": "markdown", "id": "c29375d5", "metadata": {}, "source": [ "### πŸ’‘ Memory-Optimized Configuration\n", "\n", "**This notebook uses memory-efficient models by default:**\n", "- `phi-4-mini` β†’ ~4GB RAM (Foundry Local auto-selects CPU variant)\n", "- `qwen2.5-3b` β†’ ~3GB RAM (instead of 7B which needs ~7GB+)\n", "\n", "**Port Auto-Detection:**\n", "- Foundry Local may use different ports (commonly 55769 or 59959)\n", "- The diagnostic cell below automatically detects the correct port\n", "- No manual configuration needed!\n", "\n", "**If you have limited RAM (<8GB), use even smaller models:**\n", "```python\n", "SLM = 'phi-3.5-mini' # ~2GB\n", "LLM = 'qwen2.5-0.5b' # ~500MB\n", "```" ] }, { "cell_type": "code", "execution_count": 32, "id": "1c17fc01", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "============================================================\n", "CURRENT CONFIGURATION\n", "============================================================\n", "SLM Model: phi-4-mini\n", "LLM Model: qwen2.5-7b\n", "SDK Pattern: FoundryLocalManager (official)\n", "Endpoint: Auto-detect\n", "Test Prompt: List 5 benefits of local AI inference....\n", "Retry Count: 2\n", "============================================================\n", "\n", "πŸ’‘ Using official Foundry SDK pattern from workshop_utils\n", " β†’ FoundryLocalManager handles service lifecycle\n", " β†’ Automatic model resolution and hardware optimization\n", " β†’ OpenAI-compatible API for inference\n" ] } ], "source": [ "# Display current configuration\n", "print(\"=\"*60)\n", "print(\"CURRENT CONFIGURATION\")\n", "print(\"=\"*60)\n", "print(f\"SLM Model: {SLM}\")\n", "print(f\"LLM Model: {LLM}\")\n", "print(f\"SDK Pattern: FoundryLocalManager (official)\")\n", "print(f\"Endpoint: {ENDPOINT or 'Auto-detect'}\")\n", "print(f\"Test Prompt: {PROMPT[:50]}...\")\n", "print(f\"Retry Count: 2\")\n", "print(\"=\"*60)\n", "print(\"\\nπŸ’‘ Using official Foundry SDK pattern from workshop_utils\")\n", "print(\" β†’ FoundryLocalManager handles service lifecycle\")\n", "print(\" β†’ Automatic model resolution and hardware optimization\")\n", "print(\" β†’ OpenAI-compatible API for inference\")" ] }, { "cell_type": "markdown", "id": "e638df58", "metadata": {}, "source": [ "### Explanation: Execution Helpers (Foundry SDK Pattern)\n", "Uses the official Foundry Local SDK pattern as documented in the Workshop samples:\n", "\n", "**Approach:**\n", "- **FoundryLocalManager** - Initializes and manages the Foundry Local service\n", "- **Auto-Detection** - Automatically discovers endpoint and handles service lifecycle\n", "- **Model Resolution** - Resolves aliases to full model IDs (e.g., phi-4-mini β†’ phi-4-mini-instruct-cpu)\n", "- **Hardware Optimization** - Selects best variant for available hardware (CUDA, NPU, or CPU)\n", "- **OpenAI Client** - Configured with manager's endpoint for OpenAI-compatible API access\n", "\n", "**Resilience Features:**\n", "- Exponential backoff retry logic (configurable via environment)\n", "- Automatic service startup if not running\n", "- Connection verification after initialization\n", "- Graceful error handling with detailed error reporting\n", "- Model caching to avoid repeated initialization\n", "\n", "**Result Structure:**\n", "- Latency measurement (wall clock time)\n", "- Token usage tracking (if available)\n", "- Sample output (truncated for readability)\n", "- Error details for failed requests\n", "\n", "This pattern leverages the workshop_utils module which follows the official SDK pattern.\n", "\n", "**SDK Reference:**\n", "- Main Repo: https://github.com/microsoft/Foundry-Local\n", "- Python SDK: https://github.com/microsoft/Foundry-Local/tree/main/sdk/python/foundry_local\n", "- Workshop Utils: ../samples/workshop_utils.py" ] }, { "cell_type": "code", "execution_count": 39, "id": "5e646fa6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "βœ… Execution helpers defined: setup(), run()\n", " β†’ Uses workshop_utils for proper SDK integration\n", " β†’ setup() initializes with FoundryLocalManager\n", " β†’ run() executes inference via OpenAI-compatible API\n", " β†’ Token counting: Uses API data or estimates if unavailable\n" ] } ], "source": [ "def setup(alias: str, endpoint: str = None, retries: int = 3):\n", " \"\"\"\n", " Initialize a Foundry Local model connection using official SDK pattern.\n", " \n", " This follows the workshop_utils pattern which uses FoundryLocalManager\n", " to properly initialize the Foundry Local service and resolve models.\n", " \n", " Args:\n", " alias: Model alias (e.g., 'phi-4-mini', 'qwen2.5-3b')\n", " endpoint: Optional endpoint override (usually auto-detected)\n", " retries: Number of connection attempts (default: 3)\n", " \n", " Returns:\n", " tuple: (manager, client, model_id, metadata) or (None, None, alias, error_metadata) if failed\n", " \"\"\"\n", " import time\n", " \n", " last_err = None\n", " current_delay = 2 # seconds\n", " \n", " for attempt in range(1, retries + 1):\n", " try:\n", " print(f\"[Init] Connecting to '{alias}' (attempt {attempt}/{retries})...\")\n", " \n", " # Use the workshop utility which follows the official SDK pattern\n", " manager, client, model_id = get_client(alias, endpoint=endpoint)\n", " \n", " print(f\"[OK] Connected to '{alias}' -> {model_id}\")\n", " print(f\" Endpoint: {manager.endpoint}\")\n", " \n", " return manager, client, model_id, {\n", " 'endpoint': manager.endpoint,\n", " 'resolved': model_id,\n", " 'attempts': attempt,\n", " 'status': 'success'\n", " }\n", " \n", " except Exception as e:\n", " last_err = e\n", " error_msg = str(e)\n", " \n", " # Provide helpful error messages\n", " if \"Connection error\" in error_msg or \"connection refused\" in error_msg.lower():\n", " print(f\"[ERROR] Cannot connect to Foundry Local service\")\n", " print(f\" β†’ Is the service running? Try: foundry service start\")\n", " print(f\" β†’ Is the model loaded? Try: foundry model run {alias}\")\n", " elif \"not found\" in error_msg.lower():\n", " print(f\"[ERROR] Model '{alias}' not found in catalog\")\n", " print(f\" β†’ Available models: Run 'foundry model ls' in terminal\")\n", " print(f\" β†’ Download model: Run 'foundry model download {alias}'\")\n", " else:\n", " print(f\"[ERROR] Setup failed: {type(e).__name__}: {error_msg}\")\n", " \n", " if attempt < retries:\n", " print(f\"[Retry] Waiting {current_delay:.1f}s before retry...\")\n", " time.sleep(current_delay)\n", " current_delay *= 2 # Exponential backoff\n", " \n", " # All retries failed - provide actionable guidance\n", " print(f\"\\n❌ Failed to initialize '{alias}' after {retries} attempts\")\n", " print(f\" Last error: {type(last_err).__name__}: {str(last_err)}\")\n", " print(f\"\\nπŸ’‘ Troubleshooting steps:\")\n", " print(f\" 1. Ensure Foundry Local service is running:\")\n", " print(f\" β†’ foundry service status\")\n", " print(f\" β†’ foundry service start (if not running)\")\n", " print(f\" 2. Ensure model is loaded:\")\n", " print(f\" β†’ foundry model run {alias}\")\n", " print(f\" 3. Check available models:\")\n", " print(f\" β†’ foundry model ls\")\n", " print(f\" 4. Try alternative models if '{alias}' isn't available\")\n", " \n", " return None, None, alias, {\n", " 'error': f\"{type(last_err).__name__}: {str(last_err)}\",\n", " 'endpoint': endpoint or 'auto-detect',\n", " 'attempts': retries,\n", " 'status': 'failed'\n", " }\n", "\n", "\n", "def run(client, model_id: str, prompt: str, max_tokens: int = 180, temperature: float = 0.5):\n", " \"\"\"\n", " Run inference with the configured model using OpenAI SDK.\n", " \n", " Args:\n", " client: OpenAI client instance (configured for Foundry Local)\n", " model_id: Model identifier (resolved from alias)\n", " prompt: Input prompt\n", " max_tokens: Maximum response tokens (default: 180)\n", " temperature: Sampling temperature (default: 0.5)\n", " \n", " Returns:\n", " dict: Response with timing, tokens, and content\n", " \"\"\"\n", " import time\n", " \n", " start = time.time()\n", " \n", " try:\n", " response = client.chat.completions.create(\n", " model=model_id,\n", " messages=[{\"role\": \"user\", \"content\": prompt}],\n", " max_tokens=max_tokens,\n", " temperature=temperature\n", " )\n", " \n", " elapsed = time.time() - start\n", " \n", " # Extract response details\n", " content = response.choices[0].message.content\n", " \n", " # Try to extract token usage from multiple possible locations\n", " usage_info = {}\n", " if hasattr(response, 'usage') and response.usage:\n", " usage_info['prompt_tokens'] = getattr(response.usage, 'prompt_tokens', None)\n", " usage_info['completion_tokens'] = getattr(response.usage, 'completion_tokens', None)\n", " usage_info['total_tokens'] = getattr(response.usage, 'total_tokens', None)\n", " \n", " # Calculate approximate token count if API doesn't provide it\n", " # Rough estimate: ~4 characters per token for English text\n", " if not usage_info.get('total_tokens'):\n", " estimated_prompt_tokens = len(prompt) // 4\n", " estimated_completion_tokens = len(content) // 4\n", " estimated_total = estimated_prompt_tokens + estimated_completion_tokens\n", " usage_info['estimated_tokens'] = estimated_total\n", " usage_info['estimated_prompt_tokens'] = estimated_prompt_tokens\n", " usage_info['estimated_completion_tokens'] = estimated_completion_tokens\n", " \n", " return {\n", " 'status': 'success',\n", " 'content': content,\n", " 'elapsed_sec': elapsed,\n", " 'tokens': usage_info.get('total_tokens') or usage_info.get('estimated_tokens'),\n", " 'usage': usage_info,\n", " 'model': model_id\n", " }\n", " \n", " except Exception as e:\n", " elapsed = time.time() - start\n", " return {\n", " 'status': 'error',\n", " 'error': f\"{type(e).__name__}: {str(e)}\",\n", " 'elapsed_sec': elapsed,\n", " 'model': model_id\n", " }\n", "\n", "\n", "print(\"βœ… Execution helpers defined: setup(), run()\")\n", "print(\" β†’ Uses workshop_utils for proper SDK integration\")\n", "print(\" β†’ setup() initializes with FoundryLocalManager\")\n", "print(\" β†’ run() executes inference via OpenAI-compatible API\")\n", "print(\" β†’ Token counting: Uses API data or estimates if unavailable\")" ] }, { "cell_type": "markdown", "id": "18ba23b5", "metadata": {}, "source": [ "### Explanation: Pre-Flight Self-Test\n", "Runs a lightweight connectivity check using FoundryLocalManager for both models. This verifies:\n", "- Service is accessible\n", "- Models can be initialized\n", "- Aliases resolve to actual model IDs\n", "- Connection is stable before running comparison\n", "\n", "The setup() function uses the official SDK pattern from workshop_utils." ] }, { "cell_type": "code", "execution_count": 34, "id": "e251bd89", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[Diagnostic] Checking Foundry Local service...\n", "\n", "❌ Foundry Local service not found!\n", "\n", "πŸ’‘ To fix this:\n", " 1. Open a terminal\n", " 2. Run: foundry service start\n", " 3. Run: foundry model run phi-4-mini\n", " 4. Run: foundry model run qwen2.5-3b\n", " 5. Re-run this notebook\n", "\n", "⚠️ No service detected - FoundryLocalManager will attempt to start it\n" ] } ], "source": [ "# Simplified diagnostic: Just verify service is accessible\n", "import requests\n", "\n", "def check_foundry_service():\n", " \"\"\"Quick diagnostic to verify Foundry Local is running.\"\"\"\n", " # Try common ports\n", " endpoints_to_try = [\n", " \"http://localhost:59959\",\n", " \"http://127.0.0.1:59959\", \n", " \"http://localhost:55769\",\n", " \"http://127.0.0.1:55769\",\n", " ]\n", " \n", " print(\"[Diagnostic] Checking Foundry Local service...\")\n", " \n", " for endpoint in endpoints_to_try:\n", " try:\n", " response = requests.get(f\"{endpoint}/health\", timeout=2)\n", " if response.status_code == 200:\n", " print(f\"βœ… Service is running at {endpoint}\")\n", " \n", " # Try to list models\n", " try:\n", " models_response = requests.get(f\"{endpoint}/v1/models\", timeout=2)\n", " if models_response.status_code == 200:\n", " models_data = models_response.json()\n", " model_count = len(models_data.get('data', []))\n", " print(f\"βœ… Found {model_count} models available\")\n", " if model_count > 0:\n", " print(\" Models:\", [m.get('id', 'unknown') for m in models_data.get('data', [])[:5]])\n", " except Exception as e:\n", " print(f\"⚠️ Could not list models: {e}\")\n", " \n", " return endpoint\n", " except requests.exceptions.ConnectionError:\n", " continue\n", " except Exception as e:\n", " print(f\"⚠️ Error checking {endpoint}: {e}\")\n", " \n", " print(\"\\n❌ Foundry Local service not found!\")\n", " print(\"\\nπŸ’‘ To fix this:\")\n", " print(\" 1. Open a terminal\")\n", " print(\" 2. Run: foundry service start\")\n", " print(\" 3. Run: foundry model run phi-4-mini\")\n", " print(\" 4. Run: foundry model run qwen2.5-3b\")\n", " print(\" 5. Re-run this notebook\")\n", " return None\n", "\n", "# Run diagnostic\n", "discovered_endpoint = check_foundry_service()\n", "\n", "if discovered_endpoint:\n", " print(f\"\\nβœ… Service detected (will be managed by FoundryLocalManager)\")\n", "else:\n", " print(f\"\\n⚠️ No service detected - FoundryLocalManager will attempt to start it\")" ] }, { "cell_type": "code", "execution_count": 35, "id": "35317769", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "⚠️ The commands above are commented out.\n", "Uncomment them if you want to start the service from the notebook.\n", "\n", "πŸ’‘ Recommended: Run these commands in a separate terminal instead:\n", " foundry service start\n", " foundry model run phi-4-mini\n", " foundry model run qwen2.5-3b\n" ] } ], "source": [ "# Quick Fix: Start service and load models from notebook\n", "# Uncomment the commands you need:\n", "\n", "# !foundry service start\n", "# !foundry model run phi-4-mini\n", "# !foundry model run qwen2.5-3b\n", "# !foundry model ls\n", "\n", "print(\"⚠️ The commands above are commented out.\")\n", "print(\"Uncomment them if you want to start the service from the notebook.\")\n", "print(\"\")\n", "print(\"πŸ’‘ Recommended: Run these commands in a separate terminal instead:\")\n", "print(\" foundry service start\")\n", "print(\" foundry model run phi-4-mini\")\n", "print(\" foundry model run qwen2.5-3b\")" ] }, { "cell_type": "markdown", "id": "0de2d0a5", "metadata": {}, "source": [ "### πŸ› οΈ Quick Fix: Start Foundry Local from Notebook (Optional)\n", "\n", "If the diagnostic above shows service isn't running, you can try starting it from here:\n", "\n", "**Note:** This works best on Windows. On other platforms, use terminal commands." ] }, { "cell_type": "markdown", "id": "781c0bd2", "metadata": {}, "source": [ "### ⚠️ Troubleshooting Connection Errors\n", "\n", "If you're seeing `APIConnectionError`, the Foundry Local service may not be running or models aren't loaded. Try these steps:\n", "\n", "**1. Check Service Status:**\n", "```bash\n", "# In a terminal (not in notebook):\n", "foundry service status\n", "```\n", "\n", "**2. Start Service (if not running):**\n", "```bash\n", "foundry service start\n", "```\n", "\n", "**3. Load Required Models:**\n", "```bash\n", "# Load the models needed for comparison\n", "foundry model run phi-4-mini\n", "foundry model run qwen2.5-7b\n", "\n", "# Or use alternative models:\n", "foundry model run phi-3.5-mini\n", "foundry model run qwen2.5-3b\n", "```\n", "\n", "**4. Verify Models Are Available:**\n", "```bash\n", "foundry model ls\n", "```\n", "\n", "**Common Issues:**\n", "- ❌ Service not running β†’ Run `foundry service start`\n", "- ❌ Models not loaded β†’ Run `foundry model run `\n", "- ❌ Port conflicts β†’ Check if another service is using the port\n", "- ❌ Firewall blocking β†’ Ensure local connections are allowed\n", "\n", "**Quick Fix:** Run the diagnostic cell below before the pre-flight check." ] }, { "cell_type": "code", "execution_count": 36, "id": "8a607078", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[Init] Connecting to 'phi-4-mini' (attempt 1/2)...\n", "[OK] Connected to 'phi-4-mini' -> Phi-4-mini-instruct-cuda-gpu:4\n", " Endpoint: http://127.0.0.1:59959/v1\n", "[Init] Connecting to 'qwen2.5-7b' (attempt 1/2)...\n", "[OK] Connected to 'qwen2.5-7b' -> qwen2.5-7b-instruct-cuda-gpu:3\n", " Endpoint: http://127.0.0.1:59959/v1\n", "\n", "[Pre-flight Check]\n", " βœ… phi-4-mini: success - Phi-4-mini-instruct-cuda-gpu:4\n", " βœ… qwen2.5-7b: success - qwen2.5-7b-instruct-cuda-gpu:3\n" ] }, { "data": { "text/plain": [ "{'phi-4-mini': {'endpoint': 'http://127.0.0.1:59959/v1',\n", " 'resolved': 'Phi-4-mini-instruct-cuda-gpu:4',\n", " 'attempts': 1,\n", " 'status': 'success'},\n", " 'qwen2.5-7b': {'endpoint': 'http://127.0.0.1:59959/v1',\n", " 'resolved': 'qwen2.5-7b-instruct-cuda-gpu:3',\n", " 'attempts': 1,\n", " 'status': 'success'}}" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "preflight = {}\n", "retries = 2 # Number of retry attempts\n", "\n", "for a in (SLM, LLM):\n", " mgr, c, mid, info = setup(a, endpoint=ENDPOINT, retries=retries)\n", " # Keep the original status from info (either 'success' or 'failed')\n", " preflight[a] = info\n", "\n", "print('\\n[Pre-flight Check]')\n", "for alias, details in preflight.items():\n", " status_icon = 'βœ…' if details['status'] == 'success' else '❌'\n", " print(f\" {status_icon} {alias}: {details['status']} - {details.get('resolved', details.get('error', 'unknown'))}\")\n", "\n", "preflight" ] }, { "cell_type": "markdown", "id": "ec76741a", "metadata": {}, "source": [ "### βœ… Pre-Flight Check: Model Availability\n", "\n", "This cell verifies both models can be reached at the configured endpoint before running the comparison." ] }, { "cell_type": "markdown", "id": "a22961df", "metadata": {}, "source": [ "### Explanation: Run Comparison & Collect Results\n", "Iterates over both aliases using the official Foundry SDK pattern:\n", "1. Initialize each model with setup() (uses FoundryLocalManager)\n", "2. Run inference with OpenAI-compatible API\n", "3. Capture latency, tokens, and sample output\n", "4. Produce JSON summary with comparative analysis\n", "\n", "This follows the same pattern as the Workshop samples in session04/model_compare.py." ] }, { "cell_type": "code", "execution_count": 40, "id": "a6f12b99", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[Init] Connecting to 'phi-4-mini' (attempt 1/2)...\n", "[OK] Connected to 'phi-4-mini' -> Phi-4-mini-instruct-cuda-gpu:4\n", " Endpoint: http://127.0.0.1:59959/v1\n", "[Init] Connecting to 'qwen2.5-7b' (attempt 1/2)...\n", "[OK] Connected to 'qwen2.5-7b' -> qwen2.5-7b-instruct-cuda-gpu:3\n", " Endpoint: http://127.0.0.1:59959/v1\n", "[Init] Connecting to 'qwen2.5-7b' (attempt 1/2)...\n", "[OK] Connected to 'qwen2.5-7b' -> qwen2.5-7b-instruct-cuda-gpu:3\n", " Endpoint: http://127.0.0.1:59959/v1\n", "[\n", " {\n", " \"alias\": \"phi-4-mini\",\n", " \"status\": \"success\",\n", " \"content\": \"1. Reduced Latency: Local AI inference can significantly reduce latency by processing data closer to the source, which is particularly beneficial for real-time applications such as autonomous vehicles or augmented reality.\\n\\n2. Enhanced Privacy: By keeping data processing local, sensitive information is less likely to be exposed to external networks, thereby enhancing privacy and security.\\n\\n3. Lower Bandwidth Usage: Local AI inference reduces the need for data transmission over the network, which can save bandwidth and reduce the risk of network congestion.\\n\\n4. Improved Reliability: Local processing can be more reliable, as it is less dependent on network connectivity. This is particularly important in scenarios where network connectivity is unreliable or intermittent.\\n\\n5. Scalability: Local AI inference can be easily scaled by adding more local processing units, making it easier to handle increasing data volumes or more complex AI models.\",\n", " \"elapsed_sec\": 51.97519612312317,\n", " \"tokens\": 247,\n", " \"usage\": {\n", " \"estimated_tokens\": 247,\n", " \"estimated_prompt_tokens\": 9,\n", " \"estimated_completion_tokens\": 238\n", " },\n", " \"model\": \"Phi-4-mini-instruct-cuda-gpu:4\"\n", " },\n", " {\n", " \"alias\": \"qwen2.5-7b\",\n", " \"status\": \"success\",\n", " \"content\": \"Local AI inference offers several advantages over cloud-based or remote inference solutions. Here are five key benefits:\\n\\n1. **Latency Reduction**: Local AI inference reduces latency because the data does not need to be sent to a remote server for processing. This is particularly important in applications where real-time response is critical, such as autonomous vehicles, medical imaging, and real-time analytics.\\n\\n2. **Data Privacy and Security**: Processing data locally can enhance privacy and security by keeping sensitive information within the user's control. This is especially important in industries like healthcare, finance, and government where data breaches can have severe consequences.\\n\\n3. **Cost Efficiency**: For certain applications, local inference can be more cost-effective than cloud inference. While initial setup costs for hardware might be higher, ongoing costs for cloud services can add up over time, especially for high-frequency or large-scale operations.\\n\\n4. **Offline Capabilities**: Devices\",\n", " \"elapsed_sec\": 329.4796121120453,\n", " \"tokens\": 264,\n", " \"usage\": {\n", " \"estimated_tokens\": 264,\n", " \"estimated_prompt_tokens\": 9,\n", " \"estimated_completion_tokens\": 255\n", " },\n", " \"model\": \"qwen2.5-7b-instruct-cuda-gpu:3\"\n", " }\n", "]\n", "\n", "================================================================================\n", "COMPARISON SUMMARY\n", "================================================================================\n", "Alias Status Latency(s) Tokens \n", "--------------------------------------------------------------------------------\n", "βœ… phi-4-mini success 51.975 ~247 (est.) \n", "βœ… qwen2.5-7b success 329.480 ~264 (est.) \n", "--------------------------------------------------------------------------------\n", "\n", "Detailed Token Usage:\n", "\n", " phi-4-mini:\n", " Estimated prompt: 9\n", " Estimated completion: 238\n", " Estimated total: 247\n", " (API did not provide token counts - using ~4 chars/token estimate)\n", "\n", " qwen2.5-7b:\n", " Estimated prompt: 9\n", " Estimated completion: 255\n", " Estimated total: 264\n", " (API did not provide token counts - using ~4 chars/token estimate)\n", "\n", "================================================================================\n", "\n", "πŸ’‘ SLM is 6.34x faster than LLM for this prompt\n", " SLM throughput: 4.8 tokens/sec\n", " LLM throughput: 0.8 tokens/sec\n" ] }, { "data": { "text/plain": [ "[{'alias': 'phi-4-mini',\n", " 'status': 'success',\n", " 'content': '1. Reduced Latency: Local AI inference can significantly reduce latency by processing data closer to the source, which is particularly beneficial for real-time applications such as autonomous vehicles or augmented reality.\\n\\n2. Enhanced Privacy: By keeping data processing local, sensitive information is less likely to be exposed to external networks, thereby enhancing privacy and security.\\n\\n3. Lower Bandwidth Usage: Local AI inference reduces the need for data transmission over the network, which can save bandwidth and reduce the risk of network congestion.\\n\\n4. Improved Reliability: Local processing can be more reliable, as it is less dependent on network connectivity. This is particularly important in scenarios where network connectivity is unreliable or intermittent.\\n\\n5. Scalability: Local AI inference can be easily scaled by adding more local processing units, making it easier to handle increasing data volumes or more complex AI models.',\n", " 'elapsed_sec': 51.97519612312317,\n", " 'tokens': 247,\n", " 'usage': {'estimated_tokens': 247,\n", " 'estimated_prompt_tokens': 9,\n", " 'estimated_completion_tokens': 238},\n", " 'model': 'Phi-4-mini-instruct-cuda-gpu:4'},\n", " {'alias': 'qwen2.5-7b',\n", " 'status': 'success',\n", " 'content': \"Local AI inference offers several advantages over cloud-based or remote inference solutions. Here are five key benefits:\\n\\n1. **Latency Reduction**: Local AI inference reduces latency because the data does not need to be sent to a remote server for processing. This is particularly important in applications where real-time response is critical, such as autonomous vehicles, medical imaging, and real-time analytics.\\n\\n2. **Data Privacy and Security**: Processing data locally can enhance privacy and security by keeping sensitive information within the user's control. This is especially important in industries like healthcare, finance, and government where data breaches can have severe consequences.\\n\\n3. **Cost Efficiency**: For certain applications, local inference can be more cost-effective than cloud inference. While initial setup costs for hardware might be higher, ongoing costs for cloud services can add up over time, especially for high-frequency or large-scale operations.\\n\\n4. **Offline Capabilities**: Devices\",\n", " 'elapsed_sec': 329.4796121120453,\n", " 'tokens': 264,\n", " 'usage': {'estimated_tokens': 264,\n", " 'estimated_prompt_tokens': 9,\n", " 'estimated_completion_tokens': 255},\n", " 'model': 'qwen2.5-7b-instruct-cuda-gpu:3'}]" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "results = []\n", "retries = 2 # Number of retry attempts\n", "\n", "for alias in (SLM, LLM):\n", " mgr, client, mid, info = setup(alias, endpoint=ENDPOINT, retries=retries)\n", " if client:\n", " r = run(client, mid, PROMPT)\n", " results.append({'alias': alias, **r})\n", " else:\n", " # If setup failed, record error\n", " results.append({\n", " 'alias': alias,\n", " 'status': 'error',\n", " 'error': info.get('error', 'Setup failed'),\n", " 'elapsed_sec': 0,\n", " 'tokens': None,\n", " 'model': alias\n", " })\n", "\n", "# Display results\n", "print(json.dumps(results, indent=2))\n", "\n", "# Quick comparative view\n", "print('\\n' + '='*80)\n", "print('COMPARISON SUMMARY')\n", "print('='*80)\n", "print(f\"{'Alias':<20} {'Status':<15} {'Latency(s)':<15} {'Tokens':<15}\")\n", "print('-'*80)\n", "\n", "for row in results:\n", " status = row.get('status', 'unknown')\n", " status_icon = 'βœ…' if status == 'success' else '❌'\n", " latency_str = f\"{row.get('elapsed_sec', 0):.3f}\" if row.get('elapsed_sec') else 'N/A'\n", " \n", " # Handle token display - show if available or indicate estimated\n", " tokens = row.get('tokens')\n", " usage = row.get('usage', {})\n", " if tokens:\n", " if 'estimated_tokens' in usage:\n", " tokens_str = f\"~{tokens} (est.)\"\n", " else:\n", " tokens_str = str(tokens)\n", " else:\n", " tokens_str = 'N/A'\n", " \n", " print(f\"{status_icon} {row['alias']:<18} {status:<15} {latency_str:<15} {tokens_str:<15}\")\n", "\n", "print('-'*80)\n", "\n", "# Show detailed token breakdown if available\n", "print(\"\\nDetailed Token Usage:\")\n", "for row in results:\n", " if row.get('status') == 'success' and row.get('usage'):\n", " usage = row['usage']\n", " print(f\"\\n {row['alias']}:\")\n", " if 'prompt_tokens' in usage and usage['prompt_tokens']:\n", " print(f\" Prompt tokens: {usage['prompt_tokens']}\")\n", " print(f\" Completion tokens: {usage['completion_tokens']}\")\n", " print(f\" Total tokens: {usage['total_tokens']}\")\n", " elif 'estimated_tokens' in usage:\n", " print(f\" Estimated prompt: {usage['estimated_prompt_tokens']}\")\n", " print(f\" Estimated completion: {usage['estimated_completion_tokens']}\")\n", " print(f\" Estimated total: {usage['estimated_tokens']}\")\n", " print(f\" (API did not provide token counts - using ~4 chars/token estimate)\")\n", "\n", "print('\\n' + '='*80)\n", "\n", "# Calculate speedup if both succeeded\n", "if len(results) == 2 and all(r.get('status') == 'success' and r.get('elapsed_sec') for r in results):\n", " speedup = results[1]['elapsed_sec'] / results[0]['elapsed_sec']\n", " print(f\"\\nπŸ’‘ SLM is {speedup:.2f}x faster than LLM for this prompt\")\n", " \n", " # Compare token throughput if available\n", " slm_tokens = results[0].get('tokens', 0)\n", " llm_tokens = results[1].get('tokens', 0)\n", " if slm_tokens and llm_tokens:\n", " slm_tps = slm_tokens / results[0]['elapsed_sec']\n", " llm_tps = llm_tokens / results[1]['elapsed_sec']\n", " print(f\" SLM throughput: {slm_tps:.1f} tokens/sec\")\n", " print(f\" LLM throughput: {llm_tps:.1f} tokens/sec\")\n", " \n", "elif any(r.get('status') == 'error' for r in results):\n", " print(f\"\\n⚠️ Some models failed - check errors above\")\n", " print(\" Ensure Foundry Local is running: foundry service start\")\n", " print(\" Ensure models are loaded: foundry model run \")\n", "\n", "results" ] }, { "cell_type": "markdown", "id": "592d052c", "metadata": {}, "source": [ "### Interpreting Results\n", "\n", "**Key Metrics:**\n", "- **Latency**: Lower is better - indicates faster response time\n", "- **Tokens**: Higher throughput = more tokens processed\n", "- **Route**: Confirms which API endpoint was used\n", "\n", "**When to Use SLM vs LLM:**\n", "- **SLM (Small Language Model)**: Fast responses, lower resource usage, good for simple tasks\n", "- **LLM (Large Language Model)**: Higher quality, better reasoning, use when quality matters most\n", "\n", "**Next Steps:**\n", "1. Try different prompts to see how complexity affects the comparison\n", "2. Experiment with other model pairs\n", "3. Use the Workshop router samples (Session 06) to intelligently route based on task complexity" ] }, { "cell_type": "code", "execution_count": 38, "id": "e8caed5a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "======================================================================\n", "VALIDATION SUMMARY\n", "======================================================================\n", "βœ… SLM Model: phi-4-mini\n", "βœ… LLM Model: qwen2.5-7b\n", "βœ… Using Foundry SDK Pattern: workshop_utils with FoundryLocalManager\n", "βœ… Pre-flight passed: True\n", "βœ… Comparison completed: True\n", "βœ… Both models responded: True\n", "======================================================================\n", "πŸŽ‰ ALL CHECKS PASSED! Notebook completed successfully.\n", " SLM (phi-4-mini) vs LLM (qwen2.5-7b) comparison completed.\n", " Performance: SLM is 5.14x faster\n", "======================================================================\n" ] } ], "source": [ "# Final Validation Check\n", "print(\"=\"*70)\n", "print(\"VALIDATION SUMMARY\")\n", "print(\"=\"*70)\n", "print(f\"βœ… SLM Model: {SLM}\")\n", "print(f\"βœ… LLM Model: {LLM}\")\n", "print(f\"βœ… Using Foundry SDK Pattern: workshop_utils with FoundryLocalManager\")\n", "print(f\"βœ… Pre-flight passed: {all(v['status'] == 'success' for v in preflight.values()) if 'preflight' in dir() else 'Not run yet'}\")\n", "print(f\"βœ… Comparison completed: {len(results) == 2 if 'results' in dir() else 'Not run yet'}\")\n", "print(f\"βœ… Both models responded: {all(r.get('status') == 'success' for r in results) if 'results' in dir() and results else 'Not run yet'}\")\n", "print(\"=\"*70)\n", "\n", "# Check for common configuration issues\n", "issues = []\n", "if 'LLM' in dir() and LLM not in ['qwen2.5-3b', 'qwen2.5-0.5b', 'qwen2.5-1.5b', 'qwen2.5-7b', 'phi-3.5-mini']:\n", " issues.append(f\"⚠️ LLM is '{LLM}' - expected qwen2.5-3b for memory efficiency\")\n", "if 'preflight' in dir() and not all(v['status'] == 'success' for v in preflight.values()):\n", " issues.append(\"⚠️ Pre-flight check failed - models not accessible\")\n", "if 'results' in dir() and results and not all(r.get('status') == 'success' for r in results):\n", " issues.append(\"⚠️ Comparison incomplete - check for errors above\")\n", "\n", "if not issues and 'results' in dir() and results and all(r.get('status') == 'success' for r in results):\n", " print(\"πŸŽ‰ ALL CHECKS PASSED! Notebook completed successfully.\")\n", " print(f\" SLM ({SLM}) vs LLM ({LLM}) comparison completed.\")\n", " if len(results) == 2:\n", " speedup = results[1]['elapsed_sec'] / results[0]['elapsed_sec'] if results[0]['elapsed_sec'] > 0 else 0\n", " print(f\" Performance: SLM is {speedup:.2f}x faster\")\n", "elif issues:\n", " print(\"\\n⚠️ Issues detected:\")\n", " for issue in issues:\n", " print(f\" {issue}\")\n", " print(\"\\nπŸ’‘ Troubleshooting:\")\n", " print(\" 1. Ensure service is running: foundry service start\")\n", " print(\" 2. Load models: foundry model run phi-4-mini && foundry model run qwen2.5-7b\")\n", " print(\" 3. Check model list: foundry model ls\")\n", "else:\n", " print(\"\\nπŸ’‘ Run all cells above first, then re-run this validation.\")\n", "print(\"=\"*70)" ] } ], "metadata": { "kernelspec": { "display_name": "demo", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.15" } }, "nbformat": 4, "nbformat_minor": 5 }