{ "cells": [ { "cell_type": "markdown", "id": "91644746", "metadata": {}, "source": [ "# Session 3 – Benchmark Open-Source Models\n", "\n", "Benchmark latency & approximate tokens/sec for multiple model aliases via Foundry Local." ] }, { "cell_type": "markdown", "id": "ef532433", "metadata": {}, "source": [ "## πŸ’Ύ Memory-Optimized Configuration\n", "\n", "**This notebook automatically prioritizes CPU models over CUDA variants for memory efficiency.**\n", "\n", "### Why CPU Models?\n", "- **30-50% less memory** usage compared to CUDA variants\n", "- **Works on any hardware** (no GPU required)\n", "- **Good performance** for benchmarking purposes\n", "- **Prevents memory issues** when testing multiple models\n", "\n", "### Automatic Model Selection\n", "The notebook automatically filters discovered models to prefer:\n", "1. βœ… **CPU-optimized models** (e.g., `phi-4-mini-cpu`, `qwen2.5-0.5b-cpu-int4`)\n", "2. βœ… **Quantized models** (e.g., `*-int4`, `*-q4`)\n", "3. ⚠️ **Other variants** (excluding CUDA if CPU available)\n", "4. ❌ **CUDA models** (only used if no CPU alternative exists)\n", "\n", "### Manual Override\n", "To benchmark specific models, set the `BENCH_MODELS` environment variable:\n", "```python\n", "import os\n", "os.environ['BENCH_MODELS'] = 'phi-4-mini,qwen2.5-0.5b' # Will auto-select CPU variants\n", "```\n", "\n", "### Recommended Models for Limited Memory\n", "- `phi-3.5-mini` (~2GB RAM)\n", "- `qwen2.5-0.5b` (~500MB RAM)\n", "- `phi-4-mini` (~4GB RAM)\n", "- `qwen2.5-3b` (~3GB RAM)" ] }, { "cell_type": "markdown", "id": "d2fe2c2d", "metadata": {}, "source": [ "### Explanation: Dependency Installation\n", "Installs minimal packages for benchmarking:\n", "- `foundry-local-sdk` for managing/attaching to local models.\n", "- `openai` for a simple chat completion client.\n", "- `numpy` (future extension or vector ops if needed).\n", "Idempotent; safe to re-run." ] }, { "cell_type": "markdown", "id": "84a5751c", "metadata": {}, "source": [ "# Scenario\n", "This benchmark notebook measures latency and approximate throughput (tokens/sec) for one or more locally hosted open‑source model aliases via Foundry Local. It:\n", "- Discovers available model IDs (or respects BENCH_MODELS env override).\n", "- Warms each model once to mitigate first-token cold start.\n", "- Executes multiple chat completion rounds per model and aggregates latency + token usage.\n", "- Outputs JSON plus a Markdown-friendly summary table.\n", "\n", "Use this to compare small language model trade-offs (speed vs. capability) before integrating routing or cost heuristics." ] }, { "cell_type": "code", "execution_count": 15, "id": "880bb753", "metadata": {}, "outputs": [], "source": [ "!pip install -q foundry-local-sdk openai numpy requests" ] }, { "cell_type": "markdown", "id": "38238d54", "metadata": {}, "source": [ "### Explanation: Service Diagnostic & Model Discovery\n", "Performs service health check and model discovery using multiple strategies:\n", "\n", "1. Direct health endpoint checks on common portsThis ensures the service is accessible before benchmarking begins.\n", "\n", "2. Model listing via REST API\n", "3. Provides actionable troubleshooting guidance" ] }, { "cell_type": "code", "execution_count": 16, "id": "4481aa72", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[Diagnostic] Checking Foundry Local service...\n", "βœ… Service auto-detected via SDK at http://127.0.0.1:59959/v1\n", "\n", "βœ… Service detected - ready for benchmarking\n" ] } ], "source": [ "import os, time, statistics, json\n", "import requests\n", "from foundry_local import FoundryLocalManager\n", "from openai import OpenAI\n", "\n", "def check_foundry_service():\n", " \"\"\"Quick diagnostic to verify Foundry Local is running and detect the endpoint automatically.\"\"\"\n", " print(\"[Diagnostic] Checking Foundry Local service...\")\n", " \n", " # Strategy 1: Use SDK to detect service automatically\n", " try:\n", " # Try to connect to any available model to detect the service\n", " # This will auto-discover the endpoint\n", " temp_manager = FoundryLocalManager()\n", " detected_endpoint = temp_manager.endpoint\n", " \n", " if detected_endpoint:\n", " print(f\"βœ… Service auto-detected via SDK at {detected_endpoint}\")\n", " \n", " # Verify by listing models\n", " try:\n", " models_response = requests.get(f\"{detected_endpoint}/v1/models\", timeout=2)\n", " if models_response.status_code == 200:\n", " models_data = models_response.json()\n", " model_count = len(models_data.get('data', []))\n", " print(f\"βœ… Found {model_count} models available\")\n", " if model_count > 0:\n", " model_ids = [m.get('id', 'unknown') for m in models_data.get('data', [])[:10]]\n", " print(f\" Models: {model_ids}\")\n", " return detected_endpoint\n", " except Exception as e:\n", " print(f\"⚠️ Could not list models: {e}\")\n", " return detected_endpoint\n", " except Exception as e:\n", " print(f\"⚠️ SDK auto-detection failed: {e}\")\n", " \n", " # Strategy 2: Fallback to manual port scanning\n", " print(\"[Diagnostic] Trying manual port detection...\")\n", " endpoints_to_try = [\n", " \"http://localhost:59959\",\n", " \"http://127.0.0.1:59959\", \n", " \"http://localhost:55769\",\n", " \"http://127.0.0.1:55769\",\n", " \"http://localhost:57127\",\n", " \"http://127.0.0.1:57127\",\n", " ]\n", " \n", " for endpoint in endpoints_to_try:\n", " try:\n", " response = requests.get(f\"{endpoint}/health\", timeout=2)\n", " if response.status_code == 200:\n", " print(f\"βœ… Service found at {endpoint}\")\n", " \n", " # Try to list models\n", " try:\n", " models_response = requests.get(f\"{endpoint}/v1/models\", timeout=2)\n", " if models_response.status_code == 200:\n", " models_data = models_response.json()\n", " model_count = len(models_data.get('data', []))\n", " print(f\"βœ… Found {model_count} models available\")\n", " if model_count > 0:\n", " model_ids = [m.get('id', 'unknown') for m in models_data.get('data', [])[:10]]\n", " print(f\" Models: {model_ids}\")\n", " return endpoint\n", " except Exception as e:\n", " print(f\"⚠️ Could not list models: {e}\")\n", " return endpoint\n", " except requests.exceptions.ConnectionError:\n", " continue\n", " except Exception as e:\n", " print(f\"⚠️ Error checking {endpoint}: {e}\")\n", " \n", " print(\"\\n❌ Foundry Local service not found!\")\n", " print(\"\\nπŸ’‘ To fix this:\")\n", " print(\" 1. Open a terminal\")\n", " print(\" 2. Run: foundry service start\")\n", " print(\" 3. Run: foundry model run phi-4-mini\")\n", " print(\" 4. Run: foundry model run qwen2.5-0.5b\")\n", " print(\" 5. Re-run this notebook\")\n", " return None\n", "\n", "# Run diagnostic\n", "discovered_endpoint = check_foundry_service()\n", "\n", "if discovered_endpoint:\n", " print(f\"\\nβœ… Service detected - ready for benchmarking\")\n", "else:\n", " print(f\"\\n⚠️ No service detected - benchmarking will likely fail\")\n" ] }, { "cell_type": "markdown", "id": "5b7df483", "metadata": {}, "source": [ "### Explanation: Benchmark Configuration & Model Filtering (Memory-Optimized)\n", "Sets environment-driven benchmarking parameters (rounds, prompt, generation settings). Uses auto-discovered endpoint or environment override. \n", "\n", "**Memory Optimization Strategy:**\n", "- Automatically filters discovered models to prefer CPU variants over CUDA\n", "- CPU models use 30-50% less memory while maintaining good performance\n", "- Prioritizes: CPU-optimized > Quantized models > Other variants > CUDA (only if no alternative)\n", "- Manual override available via BENCH_MODELS environment variable\n", "\n", "Discovered models are filtered to the most memory-efficient variants, with helpful logging to show which models were selected." ] }, { "cell_type": "code", "execution_count": 17, "id": "64c1a7c9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model discovery failed: Connection error.\n", "Warning: No models discovered at BASE_URL. Ensure Foundry Local is running and models are loaded.\n", "Notice: The following requested models were not discovered and may fail during benchmarking: ['phi-4-mini', 'gpt-oss-20b']\n", "Benchmarking models: ['phi-4-mini', 'gpt-oss-20b']\n", "Rounds: 3 Max Tokens: 120 Temp: 0.2\n" ] } ], "source": [ "# Benchmark configuration & model discovery (override via environment variables)\n", "BASE_URL = os.getenv('FOUNDRY_LOCAL_ENDPOINT', discovered_endpoint if 'discovered_endpoint' in dir() and discovered_endpoint else 'http://127.0.0.1:59959')\n", "if not BASE_URL.endswith('/v1'):\n", " BASE_URL = f\"{BASE_URL}/v1\"\n", "API_KEY = os.getenv('API_KEY','not-needed')\n", "\n", "_raw_models = os.getenv('BENCH_MODELS','').strip()\n", "requested_models = [m.strip() for m in _raw_models.split(',') if m.strip()] if _raw_models else []\n", "\n", "ROUNDS = int(os.getenv('BENCH_ROUNDS','3'))\n", "if ROUNDS < 1:\n", " raise ValueError('BENCH_ROUNDS must be >= 1')\n", "PROMPT = os.getenv('BENCH_PROMPT','Explain retrieval augmented generation briefly.')\n", "MAX_TOKENS = int(os.getenv('BENCH_MAX_TOKENS','120'))\n", "TEMPERATURE = float(os.getenv('BENCH_TEMPERATURE','0.2'))\n", "\n", "def _discover_models():\n", " try:\n", " c = OpenAI(base_url=BASE_URL, api_key=API_KEY)\n", " data = c.models.list().data\n", " return [m.id for m in data]\n", " except Exception as e:\n", " print(f\"Model discovery failed: {e}\")\n", " return []\n", "\n", "def _prefer_cpu_models(model_list):\n", " \"\"\"Filter models to prefer CPU variants over CUDA for memory efficiency.\n", " \n", " Priority order:\n", " 1. CPU-optimized models (e.g., *-cpu, *-cpu-int4)\n", " 2. Quantized models without CUDA (e.g., *-q4, *-int4)\n", " 3. Other models (excluding CUDA variants if CPU available)\n", " \"\"\"\n", " # Group models by base name (removing variant suffixes)\n", " from collections import defaultdict\n", " model_groups = defaultdict(list)\n", " \n", " for model in model_list:\n", " # Extract base name (before variant like -cpu, -cuda, -int4, etc.)\n", " base_name = model.split('-cpu')[0].split('-cuda')[0].split('-int4')[0].split('-q4')[0]\n", " model_groups[base_name].append(model)\n", " \n", " selected = []\n", " for base_name, variants in model_groups.items():\n", " # Prioritize CPU variants\n", " cpu_variants = [m for m in variants if '-cpu' in m.lower()]\n", " cuda_variants = [m for m in variants if '-cuda' in m.lower()]\n", " other_variants = [m for m in variants if m not in cpu_variants and m not in cuda_variants]\n", " \n", " if cpu_variants:\n", " # Prefer CPU variants\n", " selected.extend(cpu_variants)\n", " print(f\"βœ“ Selected CPU variant for {base_name}: {cpu_variants[0]}\")\n", " elif other_variants:\n", " # Use non-CUDA variants if available\n", " selected.extend(other_variants[:1]) # Take first one\n", " elif cuda_variants:\n", " # Only use CUDA if no other option\n", " selected.extend(cuda_variants[:1])\n", " print(f\"⚠️ Using CUDA variant for {base_name}: {cuda_variants[0]} (no CPU variant found)\")\n", " \n", " return selected\n", "\n", "_discovered = _discover_models()\n", "if not _discovered:\n", " print(\"Warning: No models discovered at BASE_URL. Ensure Foundry Local is running and models are loaded.\")\n", "\n", "if not requested_models or requested_models == ['auto'] or 'ALL' in requested_models:\n", " # Auto mode: discover and prefer CPU models\n", " MODELS = _prefer_cpu_models(_discovered)\n", " if len(MODELS) < len(_discovered):\n", " print(f\"πŸ’‘ Memory-optimized: Using {len(MODELS)} CPU models instead of all {len(_discovered)} variants\")\n", "else:\n", " # Filter requested models to those actually discovered\n", " MODELS = [m for m in requested_models if m in _discovered] or requested_models # fallback to requested even if not discovered\n", " missing = [m for m in requested_models if m not in _discovered]\n", " if missing:\n", " print(f\"Notice: The following requested models were not discovered and may fail during benchmarking: {missing}\")\n", "\n", "MODELS = [m for m in MODELS if m]\n", "if not MODELS:\n", " raise ValueError(\"No models available to benchmark. Start a model (e.g., 'foundry model run phi-4-mini') or set BENCH_MODELS.\")\n", "\n", "print(f\"Benchmarking models: {MODELS}\\nRounds: {ROUNDS} Max Tokens: {MAX_TOKENS} Temp: {TEMPERATURE}\")\n" ] }, { "cell_type": "markdown", "id": "92895bf4", "metadata": {}, "source": [ "### Explanation: Model Access Helper (Memory-Optimized)\n", "`ensure_loaded(alias)` follows the official Foundry Local SDK pattern with CPU preference:\n", "1. **FoundryLocalManager(alias)** - Automatically starts service and loads model if needed\n", "2. **CPU Preference** - Warns if CUDA variant is loaded, suggests CPU alternative for lower memory\n", "3. **Auto-detection** - Discovers endpoint and model variant\n", "4. **OpenAI client** - Returns configured client for chat completions\n", "5. **Model resolution** - Resolves alias to concrete model ID\n", "\n", "**Memory Optimization:** CPU variants typically use 30-50% less memory than CUDA variants while maintaining good performance for benchmarking purposes. The configuration cell automatically filters for CPU models when in auto-discovery mode." ] }, { "cell_type": "code", "execution_count": 18, "id": "65c4ff2e", "metadata": {}, "outputs": [], "source": [ "def ensure_loaded(alias):\n", " \"\"\"Return (manager, client, model_id) ensuring the alias is accessible.\n", " \n", " This follows the official Foundry Local SDK pattern with CPU preference:\n", " 1. FoundryLocalManager(alias) - Automatically starts service and loads model if needed\n", " 2. Prefers CPU variants over CUDA for memory efficiency\n", " 3. Create OpenAI client with manager's endpoint\n", " 4. Resolve model ID from alias\n", " \n", " Raises RuntimeError with guidance if the model cannot be accessed.\n", " \"\"\"\n", " try:\n", " # Initialize manager - this auto-starts service and loads model if needed\n", " # Note: By default, Foundry Local may select CUDA if available\n", " # For memory efficiency, we recommend using CPU-optimized aliases explicitly\n", " m = FoundryLocalManager(alias)\n", " \n", " # Get resolved model ID\n", " info = m.get_model_info(alias)\n", " model_id = getattr(info, 'id', alias)\n", " \n", " # Warn if CUDA variant was loaded\n", " if 'cuda' in model_id.lower():\n", " print(f\"⚠️ Loaded CUDA variant: '{alias}' -> '{model_id}'\")\n", " print(f\" πŸ’‘ For lower memory usage, use CPU variant with: foundry model run {alias.split('-cuda')[0]}-cpu\")\n", " else:\n", " print(f\"βœ“ Loaded model: '{alias}' -> '{model_id}' at {m.endpoint}\")\n", " if 'cpu' in model_id.lower():\n", " print(f\" βœ… Using memory-optimized CPU variant\")\n", " \n", " # Create OpenAI-compatible client for local Foundry service\n", " c = OpenAI(base_url=m.endpoint, api_key=m.api_key or 'not-needed')\n", " \n", " return m, c, model_id\n", " \n", " except Exception as e:\n", " raise RuntimeError(\n", " f\"Failed to load model '{alias}'.\\n\"\n", " f\"Original error: {e}\\n\\n\"\n", " f\"πŸ’‘ To fix:\\n\"\n", " f\" 1. Ensure Foundry Local service is running: foundry service start\\n\"\n", " f\" 2. Verify model is available: foundry model ls\\n\"\n", " f\" 3. For CPU-optimized models: foundry model run {alias}\\n\"\n", " f\" 4. Check available variants with: foundry model search {alias.split('-')[0]}\"\n", " )\n" ] }, { "cell_type": "markdown", "id": "b2944316", "metadata": {}, "source": [ "### Explanation: Single Round Execution\n", "`run_round` performs one chat completion and returns latency + token usage fields. If the API doesn't provide token counts, estimates them using ~4 chars/token heuristic. This ensures all benchmarks have comparable metrics." ] }, { "cell_type": "code", "execution_count": 19, "id": "6524ec41", "metadata": {}, "outputs": [], "source": [ "def run_round(client, model_id, prompt):\n", " \"\"\"Execute one chat completion round with comprehensive metric capture.\n", " \n", " Returns:\n", " Tuple of (latency_sec, total_tokens, prompt_tokens, completion_tokens, response_text)\n", " Token counts are estimated if API doesn't provide them.\n", " \"\"\"\n", " start = time.time()\n", " resp = client.chat.completions.create(\n", " model=model_id,\n", " messages=[{'role':'user','content':prompt}],\n", " max_tokens=MAX_TOKENS,\n", " temperature=TEMPERATURE,\n", " )\n", " end = time.time()\n", " latency = end - start\n", " \n", " # Extract response content\n", " content = resp.choices[0].message.content if resp.choices else \"\"\n", " \n", " # Try to get usage from API\n", " usage = getattr(resp, 'usage', None)\n", " prompt_tokens = getattr(usage, 'prompt_tokens', None) if usage else None\n", " completion_tokens = getattr(usage, 'completion_tokens', None) if usage else None\n", " total_tokens = getattr(usage, 'total_tokens', None) if usage else None\n", " \n", " # Estimate tokens if API doesn't provide them (~4 chars per token for English)\n", " if prompt_tokens is None:\n", " prompt_tokens = len(prompt) // 4\n", " if completion_tokens is None:\n", " completion_tokens = len(content) // 4\n", " if total_tokens is None:\n", " total_tokens = prompt_tokens + completion_tokens\n", " \n", " return latency, total_tokens, prompt_tokens, completion_tokens, content\n" ] }, { "cell_type": "markdown", "id": "d1d60c14", "metadata": {}, "source": [ "### Explanation: Benchmark Loop & Aggregation\n", "Iterates each model:\n", "- Warmup (excluded from stats) to mitigate cold start.\n", "- Multiple measured rounds capturing latency + tokens.\n", "- Aggregates mean, p95, and tokens/sec.\n", "Stores per-model summary dicts for later rendering." ] }, { "cell_type": "code", "execution_count": 20, "id": "a31f53b0", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "⚠️ Loaded CUDA variant: 'phi-4-mini' -> 'Phi-4-mini-instruct-cuda-gpu:4'\n", " πŸ’‘ For lower memory usage, use CPU variant with: foundry model run phi-4-mini-cpu\n", "⚠️ Loaded CUDA variant: 'gpt-oss-20b' -> 'gpt-oss-20b-cuda-gpu:1'\n", " πŸ’‘ For lower memory usage, use CPU variant with: foundry model run gpt-oss-20b-cpu\n" ] } ], "source": [ "summary = []\n", "for alias in MODELS:\n", " try:\n", " m, client, model_id = ensure_loaded(alias.strip())\n", " except Exception as e:\n", " print(e)\n", " continue\n", " \n", " # Warmup (not recorded)\n", " try:\n", " run_round(client, model_id, PROMPT)\n", " except Exception as e:\n", " print(f\"Warmup failed for {alias}: {e}\")\n", " continue\n", "\n", " latencies, tps = [], []\n", " prompt_tokens_total = 0\n", " completion_tokens_total = 0\n", " total_tokens_sum = 0\n", " sample_output = None\n", "\n", " for round_num in range(ROUNDS):\n", " try:\n", " latency, total_tokens, p_tokens, c_tokens, content = run_round(client, model_id, PROMPT)\n", " except Exception as e:\n", " print(f\"Round {round_num+1} failed for {alias}: {e}\")\n", " continue\n", " \n", " latencies.append(latency)\n", " prompt_tokens_total += p_tokens\n", " completion_tokens_total += c_tokens\n", " total_tokens_sum += total_tokens\n", " \n", " # Calculate tokens per second\n", " if total_tokens and latency > 0:\n", " tps.append(total_tokens / latency)\n", " \n", " # Capture first successful output as sample\n", " if sample_output is None:\n", " sample_output = content[:200] # First 200 chars\n", "\n", " if not latencies:\n", " print(f\"Skipping {alias}: no successful rounds.\")\n", " continue\n", "\n", " # Calculate statistics\n", " rounds_ok = len(latencies)\n", " latency_avg = statistics.mean(latencies)\n", " latency_min = min(latencies)\n", " latency_max = max(latencies)\n", " latency_p95 = statistics.quantiles(latencies, n=20)[-1] if len(latencies) > 1 else latencies[0]\n", " tokens_per_sec_avg = statistics.mean(tps) if tps else None\n", " \n", " # Average tokens per round\n", " avg_prompt_tokens = prompt_tokens_total / rounds_ok if rounds_ok else 0\n", " avg_completion_tokens = completion_tokens_total / rounds_ok if rounds_ok else 0\n", " avg_total_tokens = total_tokens_sum / rounds_ok if rounds_ok else 0\n", "\n", " summary.append({\n", " 'alias': alias,\n", " 'model_id': model_id,\n", " 'latency_avg_s': latency_avg,\n", " 'latency_min_s': latency_min,\n", " 'latency_max_s': latency_max,\n", " 'latency_p95_s': latency_p95,\n", " 'tokens_per_sec_avg': tokens_per_sec_avg,\n", " 'avg_prompt_tokens': avg_prompt_tokens,\n", " 'avg_completion_tokens': avg_completion_tokens,\n", " 'avg_total_tokens': avg_total_tokens,\n", " 'prompt_tokens_total': prompt_tokens_total,\n", " 'completion_tokens_total': completion_tokens_total,\n", " 'total_tokens_sum': total_tokens_sum,\n", " 'rounds_ok': rounds_ok,\n", " 'configured_rounds': ROUNDS,\n", " 'sample_output': sample_output,\n", " })" ] }, { "cell_type": "markdown", "id": "806e635d", "metadata": {}, "source": [ "### Explanation: Results Rendering\n", "Outputs a JSON summary (machine-friendly) and a Markdown table (human-friendly) with aligned columns. Table includes p95 latency for tail insights and tokens/sec if usage data was available." ] }, { "cell_type": "code", "execution_count": 21, "id": "93452b1a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "================================================================================\n", "BENCHMARK RESULTS\n", "================================================================================\n", "\n", "πŸ“Š PERFORMANCE SUMMARY TABLE\n", "================================================================================\n", "Model | Latency (avg) | Latency (P95) | Throughput | Tokens | Success | Rating\n", "-------------+---------------+---------------+------------+--------+---------+--------\n", "phi-4-mini | 🟒 38.815s | 39.191s | 🟒 4.6 | 179 | 3/3 | ⭐⭐⭐ \n", "gpt-oss-20b | πŸ”΄ 160.754s | 220.707s | πŸ”΄ 1.1 | 169 | 3/3 | ⭐ \n", "\n", "================================================================================\n", "Legend: 🟒 Best 🟑 Average πŸ”΄ Worst | Rating: ⭐⭐⭐ Excellent ⭐⭐ Good ⭐ Needs Improvement\n", "================================================================================\n", "\n", "================================================================================\n", "DETAILED METRICS PER MODEL\n", "================================================================================\n", "\n", "πŸ“Š phi-4-mini (Phi-4-mini-instruct-cuda-gpu:4)\n", " Latency:\n", " Average: 38.815s\n", " Min: 38.499s\n", " Max: 39.057s\n", " P95: 39.191s\n", " Tokens:\n", " Avg Prompt: 11\n", " Avg Completion: 168\n", " Avg Total: 179\n", " Throughput: 4.6 tok/s\n", " Rounds: 3/3 successful\n", " Sample Output: Retrieval Augmented Generation (RAG) is a method that combines the capabilities of retrieval and generation to create more accurate and contextually r...\n", "\n", "πŸ“Š gpt-oss-20b (gpt-oss-20b-cuda-gpu:1)\n", " Latency:\n", " Average: 160.754s\n", " Min: 134.951s\n", " Max: 191.753s\n", " P95: 220.707s\n", " Tokens:\n", " Avg Prompt: 11\n", " Avg Completion: 158\n", " Avg Total: 169\n", " Throughput: 1.1 tok/s\n", " Rounds: 3/3 successful\n", " Sample Output: <|channel|>analysis<|message|>We need to explain retrieval augmented generation briefly. Provide concise explanation.<|end|><|start|>assistant<|channe...\n", "\n", "================================================================================\n", "πŸ” PERFORMANCE COMPARISON\n", "================================================================================\n", "\n", "πŸ“ˆ Relative Performance (normalized to fastest model)\n", "--------------------------------------------------------------------------------\n", "Model | Speed vs Fastest | Latency Delta | Throughput | Efficiency\n", "--------------+------------------+---------------+------------+-----------\n", "πŸš€ phi-4-mini | β–ˆβ–ˆβ–ˆβ–ˆβ–ˆ 100% | baseline | 4.6 tok/s | 100% \n", "🐌 gpt-oss-20b | β–ˆβ–ˆβ–‘β–‘β–‘ 40% | +121.939s | 1.1 tok/s | 24% \n", "\n", "================================================================================\n", "πŸ“Š KEY FINDINGS\n", "================================================================================\n", "\n", "πŸƒ Fastest Model: phi-4-mini\n", " β”œβ”€ Average latency: 38.815s\n", " β”œβ”€ P95 latency: 39.191s\n", " └─ Throughput: 4.6 tok/s\n", "\n", "🐌 Slowest Model: gpt-oss-20b\n", " β”œβ”€ Average latency: 160.754s\n", " └─ Performance gap: 4.14x slower than fastest\n", "\n", "⚑ Highest Throughput: phi-4-mini\n", " β”œβ”€ Throughput: 4.6 tok/s\n", " └─ Latency: 38.815s\n", "\n", "πŸ’‘ Throughput Range: 4.32x difference between best and worst\n", "\n", "πŸ’Ύ Memory Efficiency:\n", "\n", "================================================================================\n", "JSON SUMMARY (for programmatic analysis)\n", "================================================================================\n", "[\n", " {\n", " \"alias\": \"phi-4-mini\",\n", " \"model_id\": \"Phi-4-mini-instruct-cuda-gpu:4\",\n", " \"latency_avg_s\": 38.81543548901876,\n", " \"latency_min_s\": 38.498725175857544,\n", " \"latency_max_s\": 39.05744504928589,\n", " \"latency_p95_s\": 39.19129209518432,\n", " \"tokens_per_sec_avg\": 4.6117357291016745,\n", " \"avg_prompt_tokens\": 11.0,\n", " \"avg_completion_tokens\": 168.0,\n", " \"avg_total_tokens\": 179.0,\n", " \"prompt_tokens_total\": 33,\n", " \"completion_tokens_total\": 504,\n", " \"total_tokens_sum\": 537,\n", " \"rounds_ok\": 3,\n", " \"configured_rounds\": 3,\n", " \"sample_output\": \"Retrieval Augmented Generation (RAG) is a method that combines the capabilities of retrieval and generation to create more accurate and contextually relevant responses. In this approach, a retrieval s\"\n", " },\n", " {\n", " \"alias\": \"gpt-oss-20b\",\n", " \"model_id\": \"gpt-oss-20b-cuda-gpu:1\",\n", " \"latency_avg_s\": 160.75422271092734,\n", " \"latency_min_s\": 134.9505536556244,\n", " \"latency_max_s\": 191.7526979446411,\n", " \"latency_p95_s\": 220.70732307434082,\n", " \"tokens_per_sec_avg\": 1.0664144910569093,\n", " \"avg_prompt_tokens\": 11.0,\n", " \"avg_completion_tokens\": 157.66666666666666,\n", " \"avg_total_tokens\": 168.66666666666666,\n", " \"prompt_tokens_total\": 33,\n", " \"completion_tokens_total\": 473,\n", " \"total_tokens_sum\": 506,\n", " \"rounds_ok\": 3,\n", " \"configured_rounds\": 3,\n", " \"sample_output\": \"<|channel|>analysis<|message|>We need to explain retrieval augmented generation briefly. Provide concise explanation.<|end|><|start|>assistant<|channel|>final<|message|>**Retrieval\\u2011Augmented Generatio\"\n", " }\n", "]\n", "\n", "================================================================================\n", "Benchmark completed: 2 models tested\n", "Configuration: 3 rounds, 120 max tokens, temp=0.2\n", "Prompt: Explain retrieval augmented generation briefly....\n", "================================================================================\n" ] } ], "source": [ "# Render results as JSON and markdown table\n", "import math\n", "\n", "print(\"=\"*80)\n", "print(\"BENCHMARK RESULTS\")\n", "print(\"=\"*80)\n", "\n", "if not summary:\n", " print(\"No results to display.\")\n", "else:\n", " # Calculate best/worst for highlighting\n", " if len(summary) > 0:\n", " best_latency = min(r['latency_avg_s'] for r in summary)\n", " worst_latency = max(r['latency_avg_s'] for r in summary)\n", " best_tps = max((r['tokens_per_sec_avg'] for r in summary if r['tokens_per_sec_avg']), default=None)\n", " worst_tps = min((r['tokens_per_sec_avg'] for r in summary if r['tokens_per_sec_avg']), default=None)\n", " \n", " # Enhanced comprehensive table with performance indicators\n", " print(\"\\nπŸ“Š PERFORMANCE SUMMARY TABLE\")\n", " print(\"=\"*80)\n", " headers = [\"Model\", \"Latency (avg)\", \"Latency (P95)\", \"Throughput\", \"Tokens\", \"Success\", \"Rating\"]\n", " rows = []\n", " \n", " for r in summary:\n", " # Performance indicators\n", " lat_indicator = \"🟒\" if r['latency_avg_s'] == best_latency else (\"πŸ”΄\" if r['latency_avg_s'] == worst_latency else \"🟑\")\n", " tps_indicator = \"\"\n", " if r['tokens_per_sec_avg']:\n", " if best_tps and r['tokens_per_sec_avg'] == best_tps:\n", " tps_indicator = \"🟒\"\n", " elif worst_tps and r['tokens_per_sec_avg'] == worst_tps:\n", " tps_indicator = \"πŸ”΄\"\n", " else:\n", " tps_indicator = \"🟑\"\n", " \n", " # Overall rating based on latency and throughput\n", " rating = \"\"\n", " if r['latency_avg_s'] == best_latency or (r['tokens_per_sec_avg'] and r['tokens_per_sec_avg'] == best_tps):\n", " rating = \"⭐⭐⭐\"\n", " elif r['latency_avg_s'] == worst_latency or (r['tokens_per_sec_avg'] and worst_tps and r['tokens_per_sec_avg'] == worst_tps):\n", " rating = \"⭐\"\n", " else:\n", " rating = \"⭐⭐\"\n", " \n", " rows.append([\n", " r['alias'][:20], # Truncate long names\n", " f\"{lat_indicator} {r['latency_avg_s']:.3f}s\",\n", " f\"{r['latency_p95_s']:.3f}s\",\n", " f\"{tps_indicator} {r['tokens_per_sec_avg']:.1f}\" if r['tokens_per_sec_avg'] else '-',\n", " f\"{r['avg_total_tokens']:.0f}\",\n", " f\"{r['rounds_ok']}/{r['configured_rounds']}\",\n", " rating\n", " ])\n", " \n", " col_widths = [max(len(str(cell)) for cell in col) for col in zip(headers, *rows)]\n", " def fmt_row(row):\n", " return \" | \".join(str(c).ljust(w) for c, w in zip(row, col_widths))\n", " \n", " print(fmt_row(headers))\n", " print(\"-\" + \"-+-\".join('-'*w for w in col_widths) + \"-\")\n", " for row in rows:\n", " print(fmt_row(row))\n", " \n", " print(\"\\n\" + \"=\"*80)\n", " print(\"Legend: 🟒 Best 🟑 Average πŸ”΄ Worst | Rating: ⭐⭐⭐ Excellent ⭐⭐ Good ⭐ Needs Improvement\")\n", " print(\"=\"*80)\n", " \n", " # Detailed metrics per model\n", " print(\"\\n\" + \"=\"*80)\n", " print(\"DETAILED METRICS PER MODEL\")\n", " print(\"=\"*80)\n", " for r in summary:\n", " print(f\"\\nπŸ“Š {r['alias']} ({r['model_id']})\")\n", " print(f\" Latency:\")\n", " print(f\" Average: {r['latency_avg_s']:.3f}s\")\n", " print(f\" Min: {r['latency_min_s']:.3f}s\")\n", " print(f\" Max: {r['latency_max_s']:.3f}s\")\n", " print(f\" P95: {r['latency_p95_s']:.3f}s\")\n", " print(f\" Tokens:\")\n", " print(f\" Avg Prompt: {r['avg_prompt_tokens']:.0f}\")\n", " print(f\" Avg Completion: {r['avg_completion_tokens']:.0f}\")\n", " print(f\" Avg Total: {r['avg_total_tokens']:.0f}\")\n", " if r['tokens_per_sec_avg']:\n", " print(f\" Throughput: {r['tokens_per_sec_avg']:.1f} tok/s\")\n", " print(f\" Rounds: {r['rounds_ok']}/{r['configured_rounds']} successful\")\n", " if r.get('sample_output'):\n", " print(f\" Sample Output: {r['sample_output'][:150]}...\")\n", " \n", " # Comparative analysis\n", " if len(summary) > 1:\n", " print(\"\\n\" + \"=\"*80)\n", " print(\"πŸ” PERFORMANCE COMPARISON\")\n", " print(\"=\"*80)\n", " \n", " # Sort by latency for speed comparison\n", " sorted_by_speed = sorted(summary, key=lambda x: x['latency_avg_s'])\n", " fastest = sorted_by_speed[0]\n", " slowest = sorted_by_speed[-1]\n", " \n", " # Create performance comparison table\n", " print(\"\\nπŸ“ˆ Relative Performance (normalized to fastest model)\")\n", " print(\"-\" * 80)\n", " comp_headers = [\"Model\", \"Speed vs Fastest\", \"Latency Delta\", \"Throughput\", \"Efficiency\"]\n", " comp_rows = []\n", " \n", " for r in sorted_by_speed:\n", " speedup = r['latency_avg_s'] / fastest['latency_avg_s']\n", " latency_delta = r['latency_avg_s'] - fastest['latency_avg_s']\n", " \n", " # Speed indicator\n", " if speedup <= 1.1:\n", " speed_bar = \"β–ˆβ–ˆβ–ˆβ–ˆβ–ˆ 100%\"\n", " speed_emoji = \"πŸš€\"\n", " elif speedup <= 1.5:\n", " speed_bar = \"β–ˆβ–ˆβ–ˆβ–ˆβ–‘ 80%\"\n", " speed_emoji = \"⚑\"\n", " elif speedup <= 2.0:\n", " speed_bar = \"β–ˆβ–ˆβ–ˆβ–‘β–‘ 60%\"\n", " speed_emoji = \"πŸƒ\"\n", " else:\n", " speed_bar = \"β–ˆβ–ˆβ–‘β–‘β–‘ 40%\"\n", " speed_emoji = \"🐌\"\n", " \n", " # Efficiency score (lower is better: combines latency and throughput)\n", " if r['tokens_per_sec_avg']:\n", " efficiency = f\"{r['tokens_per_sec_avg']:.1f} tok/s\"\n", " else:\n", " efficiency = \"N/A\"\n", " \n", " comp_rows.append([\n", " f\"{speed_emoji} {r['alias'][:18]}\",\n", " speed_bar,\n", " f\"+{latency_delta:.3f}s\" if latency_delta > 0 else \"baseline\",\n", " efficiency,\n", " f\"{(1/speedup)*100:.0f}%\"\n", " ])\n", " \n", " comp_widths = [max(len(str(cell)) for cell in col) for col in zip(comp_headers, *comp_rows)]\n", " def comp_fmt_row(row):\n", " return \" | \".join(str(c).ljust(w) for c, w in zip(row, comp_widths))\n", " \n", " print(comp_fmt_row(comp_headers))\n", " print(\"-+-\".join('-'*w for w in comp_widths))\n", " for row in comp_rows:\n", " print(comp_fmt_row(row))\n", " \n", " # Summary statistics\n", " print(\"\\n\" + \"=\"*80)\n", " print(\"πŸ“Š KEY FINDINGS\")\n", " print(\"=\"*80)\n", " \n", " print(f\"\\nπŸƒ Fastest Model: {fastest['alias']}\")\n", " print(f\" β”œβ”€ Average latency: {fastest['latency_avg_s']:.3f}s\")\n", " print(f\" β”œβ”€ P95 latency: {fastest['latency_p95_s']:.3f}s\")\n", " if fastest['tokens_per_sec_avg']:\n", " print(f\" └─ Throughput: {fastest['tokens_per_sec_avg']:.1f} tok/s\")\n", " \n", " if len(summary) > 1:\n", " print(f\"\\n🐌 Slowest Model: {slowest['alias']}\")\n", " print(f\" β”œβ”€ Average latency: {slowest['latency_avg_s']:.3f}s\")\n", " speedup = slowest['latency_avg_s'] / fastest['latency_avg_s']\n", " print(f\" └─ Performance gap: {speedup:.2f}x slower than fastest\")\n", " \n", " # Throughput comparison\n", " with_throughput = [r for r in summary if r['tokens_per_sec_avg']]\n", " if len(with_throughput) > 1:\n", " sorted_by_tps = sorted(with_throughput, key=lambda x: x['tokens_per_sec_avg'], reverse=True)\n", " highest_tps = sorted_by_tps[0]\n", " lowest_tps = sorted_by_tps[-1]\n", " \n", " print(f\"\\n⚑ Highest Throughput: {highest_tps['alias']}\")\n", " print(f\" β”œβ”€ Throughput: {highest_tps['tokens_per_sec_avg']:.1f} tok/s\")\n", " print(f\" └─ Latency: {highest_tps['latency_avg_s']:.3f}s\")\n", " \n", " if highest_tps['alias'] != lowest_tps['alias']:\n", " throughput_gap = highest_tps['tokens_per_sec_avg'] / lowest_tps['tokens_per_sec_avg']\n", " print(f\"\\nπŸ’‘ Throughput Range: {throughput_gap:.2f}x difference between best and worst\")\n", " \n", " # Memory efficiency note\n", " print(\"\\nπŸ’Ύ Memory Efficiency:\")\n", " cpu_models = [r for r in summary if 'cpu' in r['model_id'].lower()]\n", " if cpu_models:\n", " print(f\" β”œβ”€ {len(cpu_models)}/{len(summary)} models using CPU variants (30-50% memory savings)\")\n", " print(f\" └─ Recommended for systems with limited memory\")\n", " \n", " # Export JSON\n", " print(\"\\n\" + \"=\"*80)\n", " print(\"JSON SUMMARY (for programmatic analysis)\")\n", " print(\"=\"*80)\n", " print(json.dumps(summary, indent=2))\n", "\n", "print(\"\\n\" + \"=\"*80)\n", "print(f\"Benchmark completed: {len(summary)} models tested\")\n", "print(f\"Configuration: {ROUNDS} rounds, {MAX_TOKENS} max tokens, temp={TEMPERATURE}\")\n", "print(f\"Prompt: {PROMPT[:60]}...\")\n", "print(\"=\"*80)" ] }, { "cell_type": "markdown", "id": "c5468cfc", "metadata": {}, "source": [ "### Summary and Next Steps\n", "\n", "This benchmark notebook provides comprehensive performance metrics for comparing multiple models via Foundry Local:\n", "\n", "**Key Metrics Captured:**\n", "- βœ… **Latency**: Average, min, max, and P95 (tail latency)\n", "- βœ… **Throughput**: Tokens per second for each model\n", "- βœ… **Token Usage**: Prompt, completion, and total tokens (with estimation fallback)\n", "- βœ… **Reliability**: Success rate across multiple rounds\n", "- βœ… **Sample Output**: Preview of model responses\n", "\n", "**Environment Variables for Customization:**\n", "- `BENCH_MODELS`: Comma-separated list of model aliases to benchmark\n", "- `BENCH_ROUNDS`: Number of benchmark rounds per model (default: 3)\n", "- `BENCH_PROMPT`: Test prompt for benchmarking\n", "- `BENCH_MAX_TOKENS`: Maximum response tokens (default: 120)\n", "- `BENCH_TEMPERATURE`: Sampling temperature (default: 0.2)\n", "- `FOUNDRY_LOCAL_ENDPOINT`: Override service endpoint (auto-detected by default)\n", "\n", "**Next Steps:**\n", "1. Try benchmarking with different prompts to test various complexity levels\n", "2. Increase `BENCH_ROUNDS` for more statistical confidence\n", "3. Use results to inform routing decisions (see Session 06 notebooks)\n", "4. Compare memory usage and hardware optimization across model variants" ] }, { "cell_type": "code", "execution_count": 22, "id": "db617282", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "================================================================================\n", "VALIDATION SUMMARY\n", "================================================================================\n", "βœ… Service Auto-Detection Found at http://127.0.0.1:59959/v1\n", "βœ… Models Configuration 2 models configured: ['phi-4-mini', 'gpt-oss-20b']\n", "βœ… Benchmark Execution 2/2 models completed\n", "βœ… Metrics Completeness All models have comprehensive metrics\n", "================================================================================\n", "\n", "πŸŽ‰ ALL VALIDATIONS PASSED! Benchmark completed successfully.\n", " Successfully benchmarked 2 models\n", " Configuration: 3 rounds, 120 tokens, temp=0.2\n", "================================================================================\n" ] } ], "source": [ "# Final Validation Check\n", "print(\"=\"*80)\n", "print(\"VALIDATION SUMMARY\")\n", "print(\"=\"*80)\n", "\n", "validation_checks = []\n", "\n", "# Check service detection\n", "if 'discovered_endpoint' in dir() and discovered_endpoint:\n", " validation_checks.append((\"βœ…\", \"Service Auto-Detection\", f\"Found at {discovered_endpoint}\"))\n", "else:\n", " validation_checks.append((\"⚠️\", \"Service Auto-Detection\", \"Not detected - using default\"))\n", "\n", "# Check configuration\n", "if 'MODELS' in dir() and MODELS:\n", " validation_checks.append((\"βœ…\", \"Models Configuration\", f\"{len(MODELS)} models configured: {MODELS}\"))\n", "else:\n", " validation_checks.append((\"❌\", \"Models Configuration\", \"No models configured\"))\n", "\n", "# Check benchmark results\n", "if 'summary' in dir() and summary:\n", " successful = [r for r in summary if r['rounds_ok'] > 0]\n", " validation_checks.append((\"βœ…\", \"Benchmark Execution\", f\"{len(successful)}/{len(summary)} models completed\"))\n", " \n", " # Check all have complete metrics\n", " all_have_metrics = all(\n", " r.get('latency_avg_s') and \n", " r.get('tokens_per_sec_avg') and \n", " r.get('avg_total_tokens')\n", " for r in successful\n", " )\n", " if all_have_metrics:\n", " validation_checks.append((\"βœ…\", \"Metrics Completeness\", \"All models have comprehensive metrics\"))\n", " else:\n", " validation_checks.append((\"⚠️\", \"Metrics Completeness\", \"Some metrics missing\"))\n", "else:\n", " validation_checks.append((\"❌\", \"Benchmark Execution\", \"No results yet\"))\n", "\n", "# Display validation results\n", "for icon, check_name, status in validation_checks:\n", " print(f\"{icon} {check_name:<25} {status}\")\n", "\n", "print(\"=\"*80)\n", "\n", "# Overall status\n", "all_passed = all(icon == \"βœ…\" for icon, _, _ in validation_checks)\n", "if all_passed:\n", " print(\"\\nπŸŽ‰ ALL VALIDATIONS PASSED! Benchmark completed successfully.\")\n", " if 'summary' in dir() and len(summary) > 0:\n", " print(f\" Successfully benchmarked {len(summary)} models\")\n", " print(f\" Configuration: {ROUNDS} rounds, {MAX_TOKENS} tokens, temp={TEMPERATURE}\")\n", "else:\n", " print(\"\\n⚠️ Some validations did not pass. Review the issues above.\")\n", " print(\"\\nπŸ’‘ Common fixes:\")\n", " print(\" 1. Ensure Foundry Local service is running: foundry service start\")\n", " print(\" 2. Load models: foundry model run phi-4-mini && foundry model run qwen2.5-0.5b\")\n", " print(\" 3. Check model availability: foundry model ls\")\n", " print(\" 4. Re-run the benchmark cells\")\n", "\n", "print(\"=\"*80)" ] } ], "metadata": { "kernelspec": { "display_name": "demo", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.15" } }, "nbformat": 4, "nbformat_minor": 5 }