{ "cells": [ { "cell_type": "markdown", "id": "af028554", "metadata": {}, "source": [ "# Intent-Based Model Router with Foundry Local SDK\n", "\n", "**CPU-Optimized Multi-Model Routing System**\n", "\n", "This notebook demonstrates an intelligent routing system that automatically selects the best small language model based on user intent. Perfect for edge deployment scenarios where you want to leverage multiple specialized models efficiently.\n", "\n", "## ๐ŸŽฏ What You'll Learn\n", "\n", "- **Intent Detection**: Automatically classify prompts (code, summarize, classification, general)\n", "- **Smart Model Selection**: Route to the most capable model for each task\n", "- **CPU Optimization**: Memory-efficient models that work on any hardware\n", "- **Multi-Model Management**: Keep multiple models loaded with `--retain true`\n", "- **Production Patterns**: Retry logic, error handling, and token tracking\n", "\n", "## ๐Ÿ“‹ Scenario Overview\n", "\n", "This pattern demonstrates:\n", "\n", "1. **Intent Detection**: Classify each user prompt (code, summarize, classification, or general)\n", "2. **Model Selection**: Automatically pick the most suitable small language model based on capabilities\n", "3. **Local Execution**: Route to models running locally via Foundry Local service\n", "4. **Unified Interface**: Single chat entry point routing to multiple specialized models\n", "\n", "**Ideal for**: Edge deployments with multiple specialized models where you want intelligent request routing without manual model selection.\n", "\n", "## ๐Ÿ”ง Prerequisites\n", "\n", "- **Foundry Local** installed and service running\n", "- **Python 3.8+** with pip\n", "- **8GB+ RAM** (16GB+ recommended for multiple models)\n", "- **workshop_utils** module (in ../samples/)\n", "\n", "## ๐Ÿš€ Quick Start\n", "\n", "The notebook will:\n", "1. Detect your system memory\n", "2. Recommend appropriate CPU models\n", "3. Automatically load models with `--retain true`\n", "4. Verify all models are ready\n", "5. Route test prompts to specialized models\n", "\n", "**Estimated setup time**: 5-7 minutes (includes model loading)" ] }, { "cell_type": "markdown", "id": "b4aa55f0", "metadata": {}, "source": [ "## ๐Ÿ“ฆ Step 1: Install Dependencies\n", "\n", "Install the official Foundry Local SDK and required libraries:\n", "\n", "- **foundry-local-sdk**: Official Python SDK for local model management\n", "- **openai**: OpenAI-compatible API for chat completions\n", "- **psutil**: System memory detection and monitoring" ] }, { "cell_type": "code", "execution_count": 107, "id": "2929c9f5", "metadata": {}, "outputs": [], "source": [ "# Install core dependencies\n", "!pip install -q foundry-local-sdk openai psutil" ] }, { "cell_type": "markdown", "id": "990799e7", "metadata": {}, "source": [ "## ๐Ÿ’ป Step 2: System Memory Detection\n", "\n", "Detect available system memory to determine which CPU models can run efficiently. This ensures optimal model selection for your hardware." ] }, { "cell_type": "code", "execution_count": 108, "id": "ff58f1ee", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ–ฅ๏ธ System Memory Information\n", "======================================================================\n", "Total Memory: 63.30 GB\n", "Available Memory: 16.19 GB\n", "\n", "โœ… High Memory System (32GB+)\n", " Can run 3-4 models simultaneously\n", "\n", "๐Ÿ“‹ Recommended Model Aliases for Your System:\n", " โ€ข phi-4-mini\n", " โ€ข phi-3.5-mini\n", " โ€ข qwen2.5-0.5b\n", " โ€ข qwen2.5-coder-0.5b\n", "\n", "๐Ÿ’ก About Model Aliases:\n", " โœ“ Use base alias (e.g., phi-4-mini, not phi-4-mini-cpu)\n", " โœ“ Foundry Local automatically selects CPU variant for your hardware\n", " โœ“ No GPU required - optimized for CPU inference\n", " โœ“ Predictable memory usage and consistent performance\n", "======================================================================\n" ] } ], "source": [ "import psutil\n", "\n", "# Get system memory information\n", "total_memory_gb = psutil.virtual_memory().total / (1024**3)\n", "available_memory_gb = psutil.virtual_memory().available / (1024**3)\n", "\n", "print('๐Ÿ–ฅ๏ธ System Memory Information')\n", "print('=' * 70)\n", "print(f'Total Memory: {total_memory_gb:.2f} GB')\n", "print(f'Available Memory: {available_memory_gb:.2f} GB')\n", "print()\n", "\n", "# Recommend models based on available memory\n", "# Using model aliases - Foundry Local will automatically select CPU variant\n", "model_aliases = []\n", "\n", "if total_memory_gb >= 32:\n", " model_aliases = ['phi-4-mini', 'phi-3.5-mini', 'qwen2.5-0.5b', 'qwen2.5-coder-0.5b']\n", " print('โœ… High Memory System (32GB+)')\n", " print(' Can run 3-4 models simultaneously')\n", "elif total_memory_gb >= 16:\n", " model_aliases = ['phi-4-mini', 'qwen2.5-0.5b', 'phi-3.5-mini']\n", " print('โœ… Medium Memory System (16-32GB)')\n", " print(' Can run 2-3 models simultaneously')\n", "elif total_memory_gb >= 8:\n", " model_aliases = ['qwen2.5-0.5b', 'phi-3.5-mini']\n", " print('โš ๏ธ Lower Memory System (8-16GB)')\n", " print(' Recommended: 2 smaller models')\n", "else:\n", " model_aliases = ['qwen2.5-0.5b']\n", " print('โš ๏ธ Limited Memory System (<8GB)')\n", " print(' Recommended: Use only smallest model')\n", "\n", "print()\n", "print('๐Ÿ“‹ Recommended Model Aliases for Your System:')\n", "for model in model_aliases:\n", " print(f' โ€ข {model}')\n", "\n", "print()\n", "print('๐Ÿ’ก About Model Aliases:')\n", "print(' โœ“ Use base alias (e.g., phi-4-mini, not phi-4-mini-cpu)')\n", "print(' โœ“ Foundry Local automatically selects CPU variant for your hardware')\n", "print(' โœ“ No GPU required - optimized for CPU inference')\n", "print(' โœ“ Predictable memory usage and consistent performance')\n", "print('=' * 70)" ] }, { "cell_type": "markdown", "id": "69590b94", "metadata": {}, "source": [ "## ๐Ÿค– Step 3: Automatic Model Loading\n", "\n", "This cell automatically:\n", "1. Starts Foundry Local service (if not running)\n", "2. Loads recommended models with `--retain true` (keeps multiple models in memory)\n", "3. Verifies all models are ready using the SDK\n", "\n", "โฑ๏ธ **Expected time**: 3-5 minutes for all models" ] }, { "cell_type": "code", "execution_count": null, "id": "543fd976", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿš€ Automatic Model Loading with SDK Verification\n", "======================================================================\n", "๐Ÿ“‹ Loading 3 models: ['phi-4-mini', 'phi-3.5-mini', 'qwen2.5-0.5b']\n", "๐Ÿ’ก Using model aliases - Foundry will load CPU variants automatically\n", "\n", "๐Ÿ“ก Step 1: Checking Foundry Local service...\n", " โœ… Service is already running\n", "\n", "๐Ÿค– Step 2: Loading models with retention...\n", " [1/3] Starting phi-4-mini...\n", " โœ… phi-4-mini loading in background\n", " [2/3] Starting phi-3.5-mini...\n", " โœ… phi-3.5-mini loading in background\n", " [3/3] Starting qwen2.5-0.5b...\n", " โœ… qwen2.5-0.5b loading in background\n", "\n", "โœ… Step 3: Verifying models (this may take 2-3 minutes)...\n", "======================================================================\n", "\n", " Attempt 1/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 2/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 3/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 4/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 5/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 6/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 7/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 8/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 9/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 10/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 11/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 12/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 13/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 14/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 15/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 16/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 17/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 18/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 19/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 20/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 21/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 22/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 23/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 24/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 25/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 26/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 27/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 28/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 29/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", " Attempt 30/30...\n", " โš ๏ธ phi-4-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ phi-3.5-mini error: get_client() takes 1 positional argument but 2 were given...\n", " โš ๏ธ qwen2.5-0.5b error: get_client() takes 1 positional argument but 2 were given...\n", "\n", "======================================================================\n", "๐Ÿ“ฆ Final Status: 0/3 models ready\n", " โŒ phi-4-mini - NOT READY\n", " โŒ phi-3.5-mini - NOT READY\n", " โŒ qwen2.5-0.5b - NOT READY\n", "\n", "โš ๏ธ Some models not ready. Check: foundry model ls\n" ] } ], "source": [ "import subprocess\n", "import time\n", "import sys\n", "import os\n", "\n", "# Add samples directory for workshop_utils (Foundry SDK pattern)\n", "sys.path.append(os.path.join('..', 'samples'))\n", "\n", "print('๐Ÿš€ Automatic Model Loading with SDK Verification')\n", "print('=' * 70)\n", "\n", "# Use top 3 recommended models (aliases)\n", "# Foundry will automatically load CPU variants\n", "REQUIRED_MODELS = model_aliases[:3]\n", "print(f'๐Ÿ“‹ Loading {len(REQUIRED_MODELS)} models: {REQUIRED_MODELS}')\n", "print('๐Ÿ’ก Using model aliases - Foundry will load CPU variants automatically')\n", "print()\n", "\n", "# Step 1: Ensure Foundry Local service is running\n", "print('๐Ÿ“ก Step 1: Checking Foundry Local service...')\n", "try:\n", " result = subprocess.run(['foundry', 'service', 'status'], \n", " capture_output=True, text=True, timeout=5)\n", " if result.returncode == 0:\n", " print(' โœ… Service is already running')\n", " else:\n", " print(' โš™๏ธ Starting Foundry Local service...')\n", " subprocess.run(['foundry', 'service', 'start'], \n", " capture_output=True, text=True, timeout=30)\n", " time.sleep(5)\n", " print(' โœ… Service started')\n", "except Exception as e:\n", " print(f' โš ๏ธ Could not verify service: {e}')\n", " print(' ๐Ÿ’ก Try manually: foundry service start')\n", "\n", "# Step 2: Load each model with --retain true\n", "print(f'\\n๐Ÿค– Step 2: Loading models with retention...')\n", "for i, model in enumerate(REQUIRED_MODELS, 1):\n", " print(f' [{i}/{len(REQUIRED_MODELS)}] Starting {model}...')\n", " try:\n", " subprocess.Popen(['foundry', 'model', 'run', model, '--retain', 'true'],\n", " stdout=subprocess.DEVNULL,\n", " stderr=subprocess.DEVNULL)\n", " print(f' โœ… {model} loading in background')\n", " except Exception as e:\n", " print(f' โŒ Error starting {model}: {e}')\n", "\n", "# Step 3: Verify models are ready\n", "print(f'\\nโœ… Step 3: Verifying models (this may take 2-3 minutes)...')\n", "print('=' * 70)\n", "\n", "try:\n", " from workshop_utils import get_client\n", " \n", " ready_models = []\n", " max_attempts = 30\n", " attempt = 0\n", " \n", " while len(ready_models) < len(REQUIRED_MODELS) and attempt < max_attempts:\n", " attempt += 1\n", " print(f'\\n Attempt {attempt}/{max_attempts}...')\n", " \n", " for model in REQUIRED_MODELS:\n", " if model in ready_models:\n", " continue\n", " \n", " try:\n", " manager, client, model_id = get_client(model)\n", " response = client.chat.completions.create(\n", " model=model_id,\n", " messages=[{\"role\": \"user\", \"content\": \"test\"}],\n", " max_tokens=5,\n", " temperature=0\n", " )\n", " \n", " if response and response.choices:\n", " ready_models.append(model)\n", " print(f' โœ… {model} is READY')\n", " \n", " except Exception as e:\n", " error_msg = str(e).lower()\n", " if 'connection' in error_msg or 'timeout' in error_msg:\n", " print(f' โณ {model} still loading...')\n", " else:\n", " print(f' โš ๏ธ {model} error: {str(e)[:60]}...')\n", " \n", " if len(ready_models) == len(REQUIRED_MODELS):\n", " break\n", " \n", " if len(ready_models) < len(REQUIRED_MODELS):\n", " time.sleep(10)\n", " \n", " # Final status\n", " print('\\n' + '=' * 70)\n", " print(f'๐Ÿ“ฆ Final Status: {len(ready_models)}/{len(REQUIRED_MODELS)} models ready')\n", " \n", " for model in REQUIRED_MODELS:\n", " if model in ready_models:\n", " print(f' โœ… {model} - READY (retained in memory)')\n", " else:\n", " print(f' โŒ {model} - NOT READY')\n", " \n", " if len(ready_models) == len(REQUIRED_MODELS):\n", " print('\\n๐ŸŽ‰ All models loaded and verified!')\n", " print(' โœ… Ready for intent-based routing')\n", " else:\n", " print(f'\\nโš ๏ธ Some models not ready. Check: foundry model ls')\n", " \n", "except ImportError as e:\n", " print(f'\\nโŒ Cannot import workshop_utils: {e}')\n", " print(' ๐Ÿ’ก Ensure workshop_utils.py is in ../samples/')\n", "except Exception as e:\n", " print(f'\\nโŒ Verification error: {e}')" ] }, { "cell_type": "markdown", "id": "e682909b", "metadata": {}, "source": [ "## ๐ŸŽฏ Step 4: Configure Intent Detection & Model Catalog\n", "\n", "Set up the routing system with:\n", "- **Intent Rules**: Regex patterns to classify prompts\n", "- **Model Catalog**: Maps model capabilities to intent categories\n", "- **Priority System**: Determines model selection when multiple models match\n", "\n", "**CPU Model Advantages**:\n", "- โœ… No GPU required\n", "- โœ… Consistent performance\n", "- โœ… Lower power consumption\n", "- โœ… Predictable memory usage" ] }, { "cell_type": "code", "execution_count": 110, "id": "3620a4fc", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ“‹ Active Model Catalog (Hardware-Optimized Aliases)\n", "======================================================================\n", "๐Ÿ’ก Using model aliases - Foundry automatically selects CPU variants\n", "\n", " โ€ข phi-4-mini\n", " Capabilities: general, summarize, reasoning\n", " Priority: 3\n", "\n", " โ€ข qwen2.5-0.5b\n", " Capabilities: classification, fast, general\n", " Priority: 1\n", "\n", " โ€ข phi-3.5-mini\n", " Capabilities: code, refactor, technical\n", " Priority: 2\n", "\n", " โ€ข qwen2.5-coder-0.5b\n", " Capabilities: code, programming, debug\n", " Priority: 1\n", "\n", "โœ… Intent detection and model selection configured\n", "======================================================================\n", "\n", "======================================================================\n", "๐Ÿ’ก Using model aliases - Foundry automatically selects CPU variants\n", "\n", " โ€ข phi-4-mini\n", " Capabilities: general, summarize, reasoning\n", " Priority: 3\n", "\n", " โ€ข qwen2.5-0.5b\n", " Capabilities: classification, fast, general\n", " Priority: 1\n", "\n", " โ€ข phi-3.5-mini\n", " Capabilities: code, refactor, technical\n", " Priority: 2\n", "\n", " โ€ข qwen2.5-coder-0.5b\n", " Capabilities: code, programming, debug\n", " Priority: 1\n", "\n", "โœ… Intent detection and model selection configured\n", "======================================================================\n" ] } ], "source": [ "import re\n", "\n", "# Model capability catalog (maps model aliases to capabilities)\n", "# Use base aliases - Foundry Local will automatically select CPU variants\n", "CATALOG = {\n", " 'phi-4-mini': {\n", " 'capabilities': ['general', 'summarize', 'reasoning'],\n", " 'priority': 3\n", " },\n", " 'qwen2.5-0.5b': {\n", " 'capabilities': ['classification', 'fast', 'general'],\n", " 'priority': 1\n", " },\n", " 'phi-3.5-mini': {\n", " 'capabilities': ['code', 'refactor', 'technical'],\n", " 'priority': 2\n", " },\n", " 'qwen2.5-coder-0.5b': {\n", " 'capabilities': ['code', 'programming', 'debug'],\n", " 'priority': 1\n", " }\n", "}\n", "\n", "# Filter to only include models recommended for this system\n", "CATALOG = {k: v for k, v in CATALOG.items() if k in model_aliases}\n", "\n", "print('๐Ÿ“‹ Active Model Catalog (Hardware-Optimized Aliases)')\n", "print('=' * 70)\n", "print('๐Ÿ’ก Using model aliases - Foundry automatically selects CPU variants')\n", "print()\n", "for model, info in CATALOG.items():\n", " caps = ', '.join(info['capabilities'])\n", " print(f' โ€ข {model}')\n", " print(f' Capabilities: {caps}')\n", " print(f' Priority: {info[\"priority\"]}')\n", " print()\n", "\n", "# Intent detection rules (regex pattern -> intent label)\n", "INTENT_RULES = [\n", " (re.compile(r'code|refactor|function|debug|program', re.I), 'code'),\n", " (re.compile(r'summar|abstract|tl;?dr|brief', re.I), 'summarize'),\n", " (re.compile(r'classif|categor|label|sentiment', re.I), 'classification'),\n", " (re.compile(r'explain|teach|describe', re.I), 'general'),\n", "]\n", "\n", "def detect_intent(prompt: str) -> str:\n", " \"\"\"Detect intent from prompt using regex patterns.\n", " \n", " Args:\n", " prompt: User input text\n", " \n", " Returns:\n", " Intent label: 'code', 'summarize', 'classification', or 'general'\n", " \"\"\"\n", " for pattern, intent in INTENT_RULES:\n", " if pattern.search(prompt):\n", " return intent\n", " return 'general'\n", "\n", "def pick_model(intent: str) -> str:\n", " \"\"\"Select best model for intent based on capabilities and priority.\n", " \n", " Args:\n", " intent: Detected intent category\n", " \n", " Returns:\n", " Model alias string, or first available model if no match\n", " \"\"\"\n", " candidates = [\n", " (alias, info['priority']) \n", " for alias, info in CATALOG.items() \n", " if intent in info['capabilities']\n", " ]\n", " \n", " if candidates:\n", " # Sort by priority (higher = better)\n", " candidates.sort(key=lambda x: x[1], reverse=True)\n", " return candidates[0][0]\n", " \n", " # Fallback to first available model\n", " return list(CATALOG.keys())[0] if CATALOG else None\n", "\n", "print('โœ… Intent detection and model selection configured')\n", "print('=' * 70)" ] }, { "cell_type": "markdown", "id": "5fbb6d09", "metadata": {}, "source": [ "## ๐Ÿงช Step 5: Test Intent Detection\n", "\n", "Verify that the intent detection system correctly classifies different types of prompts." ] }, { "cell_type": "code", "execution_count": 111, "id": "0fd85468", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿงช Testing Intent Detection\n", "======================================================================\n", "\n", "Prompt: Refactor this Python function for better readabili...\n", " Intent: code โ†’ Model: phi-3.5-mini\n", "\n", "Prompt: Summarize the key points of this article...\n", " Intent: summarize โ†’ Model: phi-4-mini\n", "\n", "Prompt: Classify this customer feedback as positive or neg...\n", " Intent: classification โ†’ Model: qwen2.5-0.5b\n", "\n", "Prompt: Explain how edge AI differs from cloud AI...\n", " Intent: general โ†’ Model: phi-4-mini\n", "\n", "Prompt: Write a function to calculate fibonacci numbers...\n", " Intent: code โ†’ Model: phi-3.5-mini\n", "\n", "Prompt: Give me a brief overview of small language models...\n", " Intent: summarize โ†’ Model: phi-4-mini\n", "\n", "======================================================================\n", "โœ… Intent detection working correctly\n" ] } ], "source": [ "# Test intent detection with sample prompts\n", "test_prompts = [\n", " 'Refactor this Python function for better readability',\n", " 'Summarize the key points of this article',\n", " 'Classify this customer feedback as positive or negative',\n", " 'Explain how edge AI differs from cloud AI',\n", " 'Write a function to calculate fibonacci numbers',\n", " 'Give me a brief overview of small language models'\n", "]\n", "\n", "print('๐Ÿงช Testing Intent Detection')\n", "print('=' * 70)\n", "\n", "for prompt in test_prompts:\n", " intent = detect_intent(prompt)\n", " model = pick_model(intent)\n", " print(f'\\nPrompt: {prompt[:50]}...')\n", " print(f' Intent: {intent:15s} โ†’ Model: {model}')\n", "\n", "print('\\n' + '=' * 70)\n", "print('โœ… Intent detection working correctly')" ] }, { "cell_type": "markdown", "id": "9ae6a08b", "metadata": {}, "source": [ "## ๐Ÿš€ Step 6: Implement Routing Function\n", "\n", "Create the main routing function that:\n", "1. Detects intent from the prompt\n", "2. Selects the optimal model\n", "3. Executes the request via Foundry Local SDK\n", "4. Tracks token usage and errors\n", "\n", "**Uses workshop_utils pattern**:\n", "- Automatic retry with exponential backoff\n", "- OpenAI-compatible API\n", "- Token tracking and error handling" ] }, { "cell_type": "code", "execution_count": 112, "id": "24cc251d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "โœ… Routing function ready\n", " Using Foundry Local SDK via workshop_utils\n", " Token tracking: Enabled\n", " Retry logic: Automatic with exponential backoff\n" ] } ], "source": [ "import os\n", "from workshop_utils import chat_once\n", "\n", "# Fix RETRY_BACKOFF environment variable if it has comments\n", "if 'RETRY_BACKOFF' in os.environ:\n", " retry_val = os.environ['RETRY_BACKOFF'].strip().split()[0]\n", " try:\n", " float(retry_val)\n", " os.environ['RETRY_BACKOFF'] = retry_val\n", " except ValueError:\n", " os.environ['RETRY_BACKOFF'] = '1.0'\n", "\n", "def route(prompt: str, max_tokens: int = 200, temperature: float = 0.7):\n", " \"\"\"Route prompt to appropriate model based on intent.\n", " \n", " Pipeline:\n", " 1. Detect intent using regex patterns\n", " 2. Select best model by capability + priority\n", " 3. Execute via Foundry Local SDK\n", " \n", " Args:\n", " prompt: User input text\n", " max_tokens: Maximum tokens in response\n", " temperature: Sampling temperature (0-1)\n", " \n", " Returns:\n", " Dict with: intent, model, output, tokens, usage, error\n", " \"\"\"\n", " intent = detect_intent(prompt)\n", " model_alias = pick_model(intent)\n", " \n", " if not model_alias:\n", " return {\n", " 'intent': intent,\n", " 'model': None,\n", " 'output': '',\n", " 'tokens': None,\n", " 'usage': {},\n", " 'error': 'No suitable model found'\n", " }\n", " \n", " try:\n", " # Call Foundry Local via workshop_utils\n", " text, usage = chat_once(\n", " model_alias,\n", " messages=[{\"role\": \"user\", \"content\": prompt}],\n", " max_tokens=max_tokens,\n", " temperature=temperature\n", " )\n", " \n", " # Extract token information\n", " usage_info = {}\n", " if usage:\n", " usage_info['prompt_tokens'] = getattr(usage, 'prompt_tokens', None)\n", " usage_info['completion_tokens'] = getattr(usage, 'completion_tokens', None)\n", " usage_info['total_tokens'] = getattr(usage, 'total_tokens', None)\n", " \n", " # Estimate if not provided\n", " if not usage_info.get('total_tokens'):\n", " est_prompt = len(prompt) // 4\n", " est_completion = len(text or '') // 4\n", " usage_info['estimated_tokens'] = est_prompt + est_completion\n", " \n", " return {\n", " 'intent': intent,\n", " 'model': model_alias,\n", " 'output': (text or '').strip(),\n", " 'tokens': usage_info.get('total_tokens') or usage_info.get('estimated_tokens'),\n", " 'usage': usage_info,\n", " 'error': None\n", " }\n", " \n", " except Exception as e:\n", " return {\n", " 'intent': intent,\n", " 'model': model_alias,\n", " 'output': '',\n", " 'tokens': None,\n", " 'usage': {},\n", " 'error': f'{type(e).__name__}: {str(e)}'\n", " }\n", "\n", "print('โœ… Routing function ready')\n", "print(' Using Foundry Local SDK via workshop_utils')\n", "print(' Token tracking: Enabled')\n", "print(' Retry logic: Automatic with exponential backoff')" ] }, { "cell_type": "markdown", "id": "00a5c915", "metadata": {}, "source": [ "## ๐ŸŽฏ Step 7: Run Routing Tests\n", "\n", "Test the complete routing system with various prompts to demonstrate:\n", "- Automatic intent detection\n", "- Intelligent model selection\n", "- Multi-model routing with retained models\n", "- Token tracking and performance metrics" ] }, { "cell_type": "code", "execution_count": null, "id": "85c46ef4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐ŸŽฏ Running Intent-Based Routing Tests\n", "================================================================================\n", "\n", "[1/6] Testing prompt...\n", "Prompt: Refactor this Python function to make it more efficient and readable\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Expected Intent: code\n", " Detected Intent: code โœ…\n", " Selected Model: phi-3.5-mini\n", " โœ… Response: To refactor a Python function for efficiency and readability, I would need to see the specific funct...\n", " ๐Ÿ“Š Tokens: ~158 (estimated)\n", "\n", "[2/6] Testing prompt...\n", "Prompt: Summarize the key benefits of using small language models at the edge\n", " Expected Intent: summarize\n", " Detected Intent: summarize โœ…\n", " Selected Model: phi-4-mini\n", " โŒ Error: APIConnectionError: Connection error.\n", "\n", "[3/6] Testing prompt...\n", "Prompt: Classify this user feedback: The app is slow but the UI looks great\n", " Expected Intent: classification\n", " Detected Intent: classification โœ…\n", " Selected Model: qwen2.5-0.5b\n", " โŒ Error: APIConnectionError: Connection error.\n", "\n", "[4/6] Testing prompt...\n", "Prompt: Explain the difference between local and cloud inference\n", " Expected Intent: general\n", " Detected Intent: general โœ…\n", " Selected Model: phi-4-mini\n", " โŒ Error: APIConnectionError: Connection error.\n", "\n", "[5/6] Testing prompt...\n", "Prompt: Write a Python function to calculate the Fibonacci sequence\n" ] } ], "source": [ "# Test prompts covering all intent categories\n", "test_cases = [\n", " {\n", " 'prompt': 'Refactor this Python function to make it more efficient and readable',\n", " 'expected_intent': 'code'\n", " },\n", " {\n", " 'prompt': 'Summarize the key benefits of using small language models at the edge',\n", " 'expected_intent': 'summarize'\n", " },\n", " {\n", " 'prompt': 'Classify this user feedback: The app is slow but the UI looks great',\n", " 'expected_intent': 'classification'\n", " },\n", " {\n", " 'prompt': 'Explain the difference between local and cloud inference',\n", " 'expected_intent': 'general'\n", " },\n", " {\n", " 'prompt': 'Write a Python function to calculate the Fibonacci sequence',\n", " 'expected_intent': 'code'\n", " },\n", " {\n", " 'prompt': 'Give me a brief overview of the Phi model family',\n", " 'expected_intent': 'summarize'\n", " }\n", "]\n", "\n", "print('๐ŸŽฏ Running Intent-Based Routing Tests')\n", "print('=' * 80)\n", "\n", "results = []\n", "for i, test in enumerate(test_cases, 1):\n", " print(f'\\n[{i}/{len(test_cases)}] Testing prompt...')\n", " print(f'Prompt: {test[\"prompt\"]}')\n", " \n", " result = route(test['prompt'], max_tokens=150)\n", " results.append(result)\n", " \n", " print(f' Expected Intent: {test[\"expected_intent\"]}')\n", " print(f' Detected Intent: {result[\"intent\"]} {\"โœ…\" if result[\"intent\"] == test[\"expected_intent\"] else \"โš ๏ธ\"}')\n", " print(f' Selected Model: {result[\"model\"]}')\n", " \n", " if result['error']:\n", " print(f' โŒ Error: {result[\"error\"]}')\n", " else:\n", " output_preview = result['output'][:100] + '...' if len(result['output']) > 100 else result['output']\n", " print(f' โœ… Response: {output_preview}')\n", " \n", " tokens = result.get('tokens', 0)\n", " if tokens:\n", " usage = result.get('usage', {})\n", " if 'estimated_tokens' in usage:\n", " print(f' ๐Ÿ“Š Tokens: ~{tokens} (estimated)')\n", " else:\n", " print(f' ๐Ÿ“Š Tokens: {tokens}')\n", "\n", "# Summary statistics\n", "print('\\n' + '=' * 80)\n", "print('๐Ÿ“Š ROUTING SUMMARY')\n", "print('=' * 80)\n", "\n", "success_count = sum(1 for r in results if not r['error'])\n", "total_tokens = sum(r.get('tokens', 0) or 0 for r in results if not r['error'])\n", "intent_accuracy = sum(1 for i, r in enumerate(results) if r['intent'] == test_cases[i]['expected_intent'])\n", "\n", "print(f'Total Prompts: {len(results)}')\n", "print(f'โœ… Successful: {success_count}/{len(results)}')\n", "print(f'โŒ Failed: {len(results) - success_count}')\n", "print(f'๐ŸŽฏ Intent Accuracy: {intent_accuracy}/{len(results)} ({intent_accuracy/len(results)*100:.1f}%)')\n", "print(f'๐Ÿ“Š Total Tokens Used: {total_tokens}')\n", "\n", "# Model usage distribution\n", "print('\\n๐Ÿ“‹ Model Usage Distribution:')\n", "model_counts = {}\n", "for r in results:\n", " if r['model']:\n", " model_counts[r['model']] = model_counts.get(r['model'], 0) + 1\n", "\n", "for model, count in sorted(model_counts.items(), key=lambda x: x[1], reverse=True):\n", " percentage = (count / len(results)) * 100\n", " print(f' โ€ข {model}: {count} requests ({percentage:.1f}%)')\n", "\n", "if success_count == len(results):\n", " print('\\n๐ŸŽ‰ All routing tests passed successfully!')\n", "else:\n", " print(f'\\nโš ๏ธ {len(results) - success_count} test(s) failed')\n", " print(' Check Foundry Local service: foundry service status')\n", " print(' Verify models loaded: foundry model ls')\n", "\n", "print('=' * 80)" ] }, { "cell_type": "markdown", "id": "4764811e", "metadata": {}, "source": [ "## ๐Ÿ”ง Step 8: Interactive Testing\n", "\n", "Try your own prompts to see the routing system in action!" ] }, { "cell_type": "code", "execution_count": null, "id": "3f8fdd51", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐ŸŽฏ Interactive Routing Test\n", "================================================================================\n", "Your prompt: Explain how model quantization reduces memory usage\n", "\n", "Detected Intent: general\n", "Selected Model: phi-4-mini\n", "\n", "โœ… Response:\n", "--------------------------------------------------------------------------------\n", "Model quantization is a technique used to reduce the memory footprint of a machine learning model, particularly deep learning models. It works by converting the high-precision weights of a neural network, typically represented as 32-bit floating-point numbers, into lower-precision representations, such as 8-bit integers or even binary values.\n", "\n", "\n", "The primary reason for quantization is to decrease the amount of memory required to store the model's parameters. Since floating-point numbers take up more space than integers, by quantizing the weights, we can significantly reduce the model's size. This reduction in size not only saves memory but also can lead to faster computation during inference, as integer operations are generally faster than floating-point operations on many hardware platforms.\n", "\n", "\n", "However, quantization can introduce some loss of accuracy because the lower precision representation may not capture the full range of values that the floating-point representation can. To mitigate this, techniques such as quantization-aware training can be used, where the model is trained with quantization in mind,\n", "--------------------------------------------------------------------------------\n", "\n", "๐Ÿ“Š Tokens used: 292\n", "\n", "๐Ÿ’ก Try different prompts to test routing behavior!\n", "Detected Intent: general\n", "Selected Model: phi-4-mini\n", "\n", "โœ… Response:\n", "--------------------------------------------------------------------------------\n", "Model quantization is a technique used to reduce the memory footprint of a machine learning model, particularly deep learning models. It works by converting the high-precision weights of a neural network, typically represented as 32-bit floating-point numbers, into lower-precision representations, such as 8-bit integers or even binary values.\n", "\n", "\n", "The primary reason for quantization is to decrease the amount of memory required to store the model's parameters. Since floating-point numbers take up more space than integers, by quantizing the weights, we can significantly reduce the model's size. This reduction in size not only saves memory but also can lead to faster computation during inference, as integer operations are generally faster than floating-point operations on many hardware platforms.\n", "\n", "\n", "However, quantization can introduce some loss of accuracy because the lower precision representation may not capture the full range of values that the floating-point representation can. To mitigate this, techniques such as quantization-aware training can be used, where the model is trained with quantization in mind,\n", "--------------------------------------------------------------------------------\n", "\n", "๐Ÿ“Š Tokens used: 292\n", "\n", "๐Ÿ’ก Try different prompts to test routing behavior!\n" ] } ], "source": [ "# Interactive testing - modify the prompt and run this cell\n", "custom_prompt = \"Explain how model quantization reduces memory usage\"\n", "\n", "print('๐ŸŽฏ Interactive Routing Test')\n", "print('=' * 80)\n", "print(f'Your prompt: {custom_prompt}')\n", "print()\n", "\n", "result = route(custom_prompt, max_tokens=200)\n", "\n", "print(f'Detected Intent: {result[\"intent\"]}')\n", "print(f'Selected Model: {result[\"model\"]}')\n", "print()\n", "\n", "if result['error']:\n", " print(f'โŒ Error: {result[\"error\"]}')\n", "else:\n", " print('โœ… Response:')\n", " print('-' * 80)\n", " print(result['output'])\n", " print('-' * 80)\n", " \n", " if result['tokens']:\n", " print(f'\\n๐Ÿ“Š Tokens used: {result[\"tokens\"]}')\n", "\n", "print('\\n๐Ÿ’ก Try different prompts to test routing behavior!')" ] }, { "cell_type": "markdown", "id": "1c17226c", "metadata": {}, "source": [ "## ๐Ÿ“Š Step 9: Performance Analysis\n", "\n", "Analyze the routing system's performance and model utilization." ] }, { "cell_type": "code", "execution_count": null, "id": "805c688c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "โšก Performance Benchmark\n", "================================================================================\n", "\n", "Prompt: Write a hello world function...\n", " Model: phi-3.5-mini\n", " Time: 3.31s\n", " Tokens: 60\n", "\n", "Prompt: Write a hello world function...\n", " Model: phi-3.5-mini\n", " Time: 3.31s\n", " Tokens: 60\n", "\n", "Prompt: Summarize: AI at the edge is powerful...\n", " Model: phi-4-mini\n", " Time: 49.67s\n", " Tokens: 84\n", "\n", "Prompt: Summarize: AI at the edge is powerful...\n", " Model: phi-4-mini\n", " Time: 49.67s\n", " Tokens: 84\n", "\n", "Prompt: Classify: Good product...\n", " Model: qwen2.5-0.5b\n", " Time: 7.21s\n", " Tokens: 69\n", "\n", "Prompt: Classify: Good product...\n", " Model: qwen2.5-0.5b\n", " Time: 7.21s\n", " Tokens: 69\n", "\n", "Prompt: Explain edge computing...\n", " Model: phi-4-mini\n", " Time: 49.67s\n", " Tokens: 72\n", "\n", "================================================================================\n", "๐Ÿ“Š Performance Statistics:\n", " Average response time: 27.46s\n", " Fastest response: 3.31s\n", " Slowest response: 49.67s\n", "\n", "๐Ÿ’ก Note: First request may be slower due to model initialization\n", "================================================================================\n", "\n", "Prompt: Explain edge computing...\n", " Model: phi-4-mini\n", " Time: 49.67s\n", " Tokens: 72\n", "\n", "================================================================================\n", "๐Ÿ“Š Performance Statistics:\n", " Average response time: 27.46s\n", " Fastest response: 3.31s\n", " Slowest response: 49.67s\n", "\n", "๐Ÿ’ก Note: First request may be slower due to model initialization\n", "================================================================================\n" ] } ], "source": [ "import time\n", "\n", "# Performance benchmark\n", "benchmark_prompts = [\n", " 'Write a hello world function',\n", " 'Summarize: AI at the edge is powerful',\n", " 'Classify: Good product',\n", " 'Explain edge computing'\n", "]\n", "\n", "print('โšก Performance Benchmark')\n", "print('=' * 80)\n", "\n", "timings = []\n", "for prompt in benchmark_prompts:\n", " start = time.time()\n", " result = route(prompt, max_tokens=50)\n", " duration = time.time() - start\n", " timings.append(duration)\n", " \n", " print(f'\\nPrompt: {prompt[:40]}...')\n", " print(f' Model: {result[\"model\"]}')\n", " print(f' Time: {duration:.2f}s')\n", " if result.get('tokens'):\n", " print(f' Tokens: {result[\"tokens\"]}')\n", "\n", "print('\\n' + '=' * 80)\n", "print('๐Ÿ“Š Performance Statistics:')\n", "print(f' Average response time: {sum(timings)/len(timings):.2f}s')\n", "print(f' Fastest response: {min(timings):.2f}s')\n", "print(f' Slowest response: {max(timings):.2f}s')\n", "print('\\n๐Ÿ’ก Note: First request may be slower due to model initialization')\n", "print('=' * 80)" ] }, { "cell_type": "markdown", "id": "e7db64ff", "metadata": {}, "source": [ "## ๐ŸŽ“ Key Takeaways & Next Steps\n", "\n", "### โœ… What You've Learned\n", "\n", "1. **Intent-Based Routing**: Automatically classify prompts and route to specialized models\n", "2. **Memory-Aware Selection**: Choose CPU models based on available system RAM\n", "3. **Multi-Model Retention**: Use `--retain true` to keep multiple models loaded\n", "4. **Production Patterns**: Retry logic, error handling, and token tracking\n", "5. **CPU Optimization**: Deploy efficiently without GPU requirements\n", "\n", "### ๐Ÿš€ Experiment Ideas\n", "\n", "1. **Add Custom Intents**:\n", " ```python\n", " INTENT_RULES.append(\n", " (re.compile(r'translate|convert', re.I), 'translation')\n", " )\n", " ```\n", "\n", "2. **Load Additional Models**:\n", " ```bash\n", " foundry model run llama-3.2-1b-cpu --retain true\n", " ```\n", "\n", "3. **Tune Model Selection**:\n", " - Adjust priority values in CATALOG\n", " - Add more capability tags\n", " - Implement fallback strategies\n", "\n", "4. **Monitor Performance**:\n", " ```python\n", " import psutil\n", " print(f\"Memory: {psutil.virtual_memory().percent}%\")\n", " ```\n", "\n", "### ๐Ÿ“š Additional Resources\n", "\n", "- **Foundry Local SDK**: https://github.com/microsoft/Foundry-Local\n", "- **Workshop Samples**: ../samples/\n", "- **Edge AI Course**: ../../Module08/\n", "\n", "### ๐Ÿ’ก Best Practices\n", "\n", "โœ… Use CPU models for consistent cross-platform behavior \n", "โœ… Always check system memory before loading multiple models \n", "โœ… Use `--retain true` for routing scenarios \n", "โœ… Implement proper error handling and retries \n", "โœ… Track token usage for cost/performance optimization \n", "\n", "---\n", "\n", "**๐ŸŽ‰ Congratulations!** You've built a production-ready intent-based model router using Foundry Local SDK with CPU-optimized models!" ] } ], "metadata": { "kernelspec": { "display_name": "demo", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.15" } }, "nbformat": 4, "nbformat_minor": 5 }