{ "cells": [ { "cell_type": "markdown", "id": "e49ba9a1", "metadata": {}, "source": [ "# Сесия 4 – Сравнение между SLM и LLM\n", "\n", "Сравнете латентността и качеството на примерните отговори между Малък езиков модел и по-голям модел, работещ чрез Foundry Local.\n" ] }, { "cell_type": "markdown", "id": "18d9330f", "metadata": {}, "source": [ "## ⚡ Бърз старт\n", "\n", "**Оптимизирана за памет настройка (Обновена):**\n", "1. Моделите автоматично избират CPU варианти (работи на всякакъв хардуер)\n", "2. Използва `qwen2.5-3b` вместо 7B (спестява ~4GB RAM)\n", "3. Автоматично откриване на порт (без ръчна конфигурация)\n", "4. Препоръчителна обща RAM: ~8GB (модели + операционна система)\n", "\n", "**Настройка в терминала (30 секунди):**\n", "```bash\n", "foundry service start\n", "foundry model run phi-4-mini\n", "foundry model run qwen2.5-3b\n", "```\n", "\n", "След това стартирайте този notebook! 🚀\n" ] }, { "cell_type": "markdown", "id": "9941ea8c", "metadata": {}, "source": [ "### Обяснение: Инсталиране на зависимости\n", "Инсталира минимални пакети (`foundry-local-sdk`, `openai`, `numpy`), необходими за измерване на време и заявки за чат. Може безопасно да се изпълнява многократно без промени.\n" ] }, { "cell_type": "markdown", "id": "4690c7c7", "metadata": {}, "source": [ "# Сценарий\n", "Сравнете представителен Малък Езиков Модел (SLM) с по-голям модел върху един и същ подаден текст, за да илюстрирате компромисите:\n", "- **Разлика в латентността** (време по стенен часовник в секунди)\n", "- **Използване на токени** (ако е налично) като индикатор за пропускателна способност\n", "- **Пример за качествен изход** за бърза визуална оценка\n", "- **Изчисление на ускорението** за количествено измерване на подобренията в производителността\n", "\n", "**Променливи на средата:**\n", "- `SLM_ALIAS` - Малък езиков модел (по подразбиране: phi-4-mini, ~4GB RAM)\n", "- `LLM_ALIAS` - По-голям езиков модел (по подразбиране: qwen2.5-7b, ~7GB RAM)\n", "- `COMPARE_PROMPT` - Тестов подаден текст за сравнение\n", "- `COMPARE_RETRIES` - Опити за повторение за устойчивост (по подразбиране: 2)\n", "- `FOUNDRY_LOCAL_ENDPOINT` - Замяна на крайна точка на услугата (автоматично откриване, ако не е зададено)\n", "\n", "**Как работи (Официален SDK модел):**\n", "1. **FoundryLocalManager** инициализира и управлява локалната услуга Foundry\n", "2. Услугата се стартира автоматично, ако не работи (не е необходима ръчна настройка)\n", "3. Моделите се преобразуват от псевдоними в конкретни идентификатори автоматично\n", "4. Избират се оптимизирани за хардуер варианти (CUDA, NPU или CPU)\n", "5. Клиент, съвместим с OpenAI, изпълнява завършване на чатове\n", "6. Метриките се събират: латентност, токени, качество на изхода\n", "7. Резултатите се сравняват за изчисление на съотношението на ускорение\n", "\n", "Това микро-сравнение помага да се реши кога е оправдано насочването към по-голям модел за вашия случай на употреба.\n", "\n", "**Референция за SDK:** \n", "- Python SDK: https://github.com/microsoft/Foundry-Local/tree/main/sdk/python/foundry_local\n", "- Workshop Utils: Използва официалния модел от ../samples/workshop_utils.py\n", "\n", "**Основни предимства:**\n", "- ✅ Автоматично откриване и инициализация на услугата\n", "- ✅ Автоматично стартиране на услугата, ако не работи\n", "- ✅ Вградена резолюция на модели и кеширане\n", "- ✅ Оптимизация за хардуер (CUDA/NPU/CPU)\n", "- ✅ Съвместимост с OpenAI SDK\n", "- ✅ Надеждно обработване на грешки с повторения\n", "- ✅ Локално извеждане (не е необходим облачен API)\n" ] }, { "cell_type": "markdown", "id": "ea4813a6", "metadata": {}, "source": [ "## 🚨 Предварителни условия: Foundry Local трябва да работи!\n", "\n", "**Преди да стартирате този тетрадка**, уверете се, че услугата Foundry Local е настроена:\n", "\n", "### Команди за бърз старт (стартирайте в терминала):\n", "\n", "```bash\n", "# 1. Start the Foundry Local service\n", "foundry service start\n", "\n", "# 2. Load the default models used in this comparison (CPU-optimized)\n", "foundry model run phi-4-mini\n", "foundry model run qwen2.5-3b\n", "\n", "# 3. Verify models are loaded\n", "foundry model ls\n", "\n", "# 4. Check service health\n", "foundry service status\n", "```\n", "\n", "### Алтернативни модели (ако стандартните не са налични):\n", "\n", "```bash\n", "# Even smaller alternatives (if memory is very limited)\n", "foundry model run phi-3.5-mini\n", "foundry model run qwen2.5-0.5b\n", "\n", "# Or update the environment variables in this notebook:\n", "# SLM_ALIAS = 'phi-3.5-mini'\n", "# LLM_ALIAS = 'qwen2.5-1.5b' # Or qwen2.5-0.5b for minimum memory\n", "```\n", "\n", "⚠️ **Ако пропуснете тези стъпки**, ще получите `APIConnectionError`, когато изпълнявате клетките в тетрадката по-долу.\n" ] }, { "cell_type": "code", "execution_count": 29, "id": "e8ea63f9", "metadata": {}, "outputs": [], "source": [ "# Install dependencies\n", "!pip install -q foundry-local-sdk openai numpy requests" ] }, { "cell_type": "markdown", "id": "eb3d2ae1", "metadata": {}, "source": [ "### Обяснение: Основни импорти\n", "Включва инструменти за управление на времето и клиенти на Foundry Local / OpenAI, използвани за извличане на информация за модела и изпълнение на чат завършвания.\n" ] }, { "cell_type": "code", "execution_count": 30, "id": "b1233c05", "metadata": {}, "outputs": [], "source": [ "import os, time, json\n", "from foundry_local import FoundryLocalManager\n", "from openai import OpenAI\n", "import sys\n", "sys.path.append('../samples')\n", "from workshop_utils import get_client, chat_once" ] }, { "cell_type": "markdown", "id": "03b6aa79", "metadata": {}, "source": [ "### Обяснение: Псевдоними и Настройка на Подканата\n", "Определя псевдоними, конфигурируеми чрез средата, за малък и голям модел, както и подканата за сравнение. Настройте променливите на средата, за да експериментирате с различни семейства модели или задачи.\n" ] }, { "cell_type": "code", "execution_count": 31, "id": "f46b14ab", "metadata": {}, "outputs": [], "source": [ "# Default to CPU models for better memory efficiency\n", "SLM = os.getenv('SLM_ALIAS', 'phi-4-mini') # Auto-selects CPU variant\n", "LLM = os.getenv('LLM_ALIAS', 'qwen2.5-3b') # Smaller LLM, more memory-friendly\n", "PROMPT = os.getenv('COMPARE_PROMPT', 'List 5 benefits of local AI inference.')\n", "# Endpoint is now managed by FoundryLocalManager - it auto-detects or can be overridden\n", "ENDPOINT = os.getenv('FOUNDRY_LOCAL_ENDPOINT', None)" ] }, { "cell_type": "markdown", "id": "c29375d5", "metadata": {}, "source": [ "### 💡 Конфигурация за оптимизирана памет\n", "\n", "**Този notebook използва модели, които са ефективни по отношение на паметта:**\n", "- `phi-4-mini` → ~4GB RAM (Foundry Local автоматично избира варианта за CPU)\n", "- `qwen2.5-3b` → ~3GB RAM (вместо 7B, който изисква ~7GB+)\n", "\n", "**Автоматично откриване на порт:**\n", "- Foundry Local може да използва различни портове (обикновено 55769 или 59959)\n", "- Диагностичната клетка по-долу автоматично открива правилния порт\n", "- Не е необходима ръчна конфигурация!\n", "\n", "**Ако разполагате с ограничена RAM (<8GB), използвайте още по-малки модели:**\n", "```python\n", "SLM = 'phi-3.5-mini' # ~2GB\n", "LLM = 'qwen2.5-0.5b' # ~500MB\n", "```\n" ] }, { "cell_type": "code", "execution_count": 32, "id": "1c17fc01", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "============================================================\n", "CURRENT CONFIGURATION\n", "============================================================\n", "SLM Model: phi-4-mini\n", "LLM Model: qwen2.5-7b\n", "SDK Pattern: FoundryLocalManager (official)\n", "Endpoint: Auto-detect\n", "Test Prompt: List 5 benefits of local AI inference....\n", "Retry Count: 2\n", "============================================================\n", "\n", "💡 Using official Foundry SDK pattern from workshop_utils\n", " → FoundryLocalManager handles service lifecycle\n", " → Automatic model resolution and hardware optimization\n", " → OpenAI-compatible API for inference\n" ] } ], "source": [ "# Display current configuration\n", "print(\"=\"*60)\n", "print(\"CURRENT CONFIGURATION\")\n", "print(\"=\"*60)\n", "print(f\"SLM Model: {SLM}\")\n", "print(f\"LLM Model: {LLM}\")\n", "print(f\"SDK Pattern: FoundryLocalManager (official)\")\n", "print(f\"Endpoint: {ENDPOINT or 'Auto-detect'}\")\n", "print(f\"Test Prompt: {PROMPT[:50]}...\")\n", "print(f\"Retry Count: 2\")\n", "print(\"=\"*60)\n", "print(\"\\n💡 Using official Foundry SDK pattern from workshop_utils\")\n", "print(\" → FoundryLocalManager handles service lifecycle\")\n", "print(\" → Automatic model resolution and hardware optimization\")\n", "print(\" → OpenAI-compatible API for inference\")" ] }, { "cell_type": "markdown", "id": "e638df58", "metadata": {}, "source": [ "### Обяснение: Помощници за изпълнение (Foundry SDK Pattern)\n", "Използва официалния модел на Foundry Local SDK, както е документирано в примерите от Workshop:\n", "\n", "**Подход:**\n", "- **FoundryLocalManager** - Инициализира и управлява услугата Foundry Local\n", "- **Автоматично откриване** - Автоматично открива крайна точка и управлява жизнения цикъл на услугата\n", "- **Решаване на модели** - Превръща псевдоними в пълни идентификатори на модели (напр. phi-4-mini → phi-4-mini-instruct-cpu)\n", "- **Оптимизация на хардуера** - Избира най-добрия вариант за наличния хардуер (CUDA, NPU или CPU)\n", "- **OpenAI клиент** - Конфигуриран с крайна точка на мениджъра за достъп до API, съвместим с OpenAI\n", "\n", "**Функции за устойчивост:**\n", "- Логика за повторение с експоненциално забавяне (конфигурируема чрез среда)\n", "- Автоматично стартиране на услугата, ако не работи\n", "- Проверка на връзката след инициализация\n", "- Гъвкаво обработване на грешки с подробни отчети за грешки\n", "- Кеширане на модели за избягване на повторна инициализация\n", "\n", "**Структура на резултата:**\n", "- Измерване на латентност (реално време)\n", "- Проследяване на използването на токени (ако е налично)\n", "- Примерен изход (съкратен за четимост)\n", "- Подробности за грешки при неуспешни заявки\n", "\n", "Този модел използва модула workshop_utils, който следва официалния SDK модел.\n", "\n", "**SDK Референция:**\n", "- Основно хранилище: https://github.com/microsoft/Foundry-Local\n", "- Python SDK: https://github.com/microsoft/Foundry-Local/tree/main/sdk/python/foundry_local\n", "- Workshop Utils: ../samples/workshop_utils.py\n" ] }, { "cell_type": "code", "execution_count": 39, "id": "5e646fa6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "✅ Execution helpers defined: setup(), run()\n", " → Uses workshop_utils for proper SDK integration\n", " → setup() initializes with FoundryLocalManager\n", " → run() executes inference via OpenAI-compatible API\n", " → Token counting: Uses API data or estimates if unavailable\n" ] } ], "source": [ "def setup(alias: str, endpoint: str = None, retries: int = 3):\n", " \"\"\"\n", " Initialize a Foundry Local model connection using official SDK pattern.\n", " \n", " This follows the workshop_utils pattern which uses FoundryLocalManager\n", " to properly initialize the Foundry Local service and resolve models.\n", " \n", " Args:\n", " alias: Model alias (e.g., 'phi-4-mini', 'qwen2.5-3b')\n", " endpoint: Optional endpoint override (usually auto-detected)\n", " retries: Number of connection attempts (default: 3)\n", " \n", " Returns:\n", " tuple: (manager, client, model_id, metadata) or (None, None, alias, error_metadata) if failed\n", " \"\"\"\n", " import time\n", " \n", " last_err = None\n", " current_delay = 2 # seconds\n", " \n", " for attempt in range(1, retries + 1):\n", " try:\n", " print(f\"[Init] Connecting to '{alias}' (attempt {attempt}/{retries})...\")\n", " \n", " # Use the workshop utility which follows the official SDK pattern\n", " manager, client, model_id = get_client(alias, endpoint=endpoint)\n", " \n", " print(f\"[OK] Connected to '{alias}' -> {model_id}\")\n", " print(f\" Endpoint: {manager.endpoint}\")\n", " \n", " return manager, client, model_id, {\n", " 'endpoint': manager.endpoint,\n", " 'resolved': model_id,\n", " 'attempts': attempt,\n", " 'status': 'success'\n", " }\n", " \n", " except Exception as e:\n", " last_err = e\n", " error_msg = str(e)\n", " \n", " # Provide helpful error messages\n", " if \"Connection error\" in error_msg or \"connection refused\" in error_msg.lower():\n", " print(f\"[ERROR] Cannot connect to Foundry Local service\")\n", " print(f\" → Is the service running? Try: foundry service start\")\n", " print(f\" → Is the model loaded? Try: foundry model run {alias}\")\n", " elif \"not found\" in error_msg.lower():\n", " print(f\"[ERROR] Model '{alias}' not found in catalog\")\n", " print(f\" → Available models: Run 'foundry model ls' in terminal\")\n", " print(f\" → Download model: Run 'foundry model download {alias}'\")\n", " else:\n", " print(f\"[ERROR] Setup failed: {type(e).__name__}: {error_msg}\")\n", " \n", " if attempt < retries:\n", " print(f\"[Retry] Waiting {current_delay:.1f}s before retry...\")\n", " time.sleep(current_delay)\n", " current_delay *= 2 # Exponential backoff\n", " \n", " # All retries failed - provide actionable guidance\n", " print(f\"\\n❌ Failed to initialize '{alias}' after {retries} attempts\")\n", " print(f\" Last error: {type(last_err).__name__}: {str(last_err)}\")\n", " print(f\"\\n💡 Troubleshooting steps:\")\n", " print(f\" 1. Ensure Foundry Local service is running:\")\n", " print(f\" → foundry service status\")\n", " print(f\" → foundry service start (if not running)\")\n", " print(f\" 2. Ensure model is loaded:\")\n", " print(f\" → foundry model run {alias}\")\n", " print(f\" 3. Check available models:\")\n", " print(f\" → foundry model ls\")\n", " print(f\" 4. Try alternative models if '{alias}' isn't available\")\n", " \n", " return None, None, alias, {\n", " 'error': f\"{type(last_err).__name__}: {str(last_err)}\",\n", " 'endpoint': endpoint or 'auto-detect',\n", " 'attempts': retries,\n", " 'status': 'failed'\n", " }\n", "\n", "\n", "def run(client, model_id: str, prompt: str, max_tokens: int = 180, temperature: float = 0.5):\n", " \"\"\"\n", " Run inference with the configured model using OpenAI SDK.\n", " \n", " Args:\n", " client: OpenAI client instance (configured for Foundry Local)\n", " model_id: Model identifier (resolved from alias)\n", " prompt: Input prompt\n", " max_tokens: Maximum response tokens (default: 180)\n", " temperature: Sampling temperature (default: 0.5)\n", " \n", " Returns:\n", " dict: Response with timing, tokens, and content\n", " \"\"\"\n", " import time\n", " \n", " start = time.time()\n", " \n", " try:\n", " response = client.chat.completions.create(\n", " model=model_id,\n", " messages=[{\"role\": \"user\", \"content\": prompt}],\n", " max_tokens=max_tokens,\n", " temperature=temperature\n", " )\n", " \n", " elapsed = time.time() - start\n", " \n", " # Extract response details\n", " content = response.choices[0].message.content\n", " \n", " # Try to extract token usage from multiple possible locations\n", " usage_info = {}\n", " if hasattr(response, 'usage') and response.usage:\n", " usage_info['prompt_tokens'] = getattr(response.usage, 'prompt_tokens', None)\n", " usage_info['completion_tokens'] = getattr(response.usage, 'completion_tokens', None)\n", " usage_info['total_tokens'] = getattr(response.usage, 'total_tokens', None)\n", " \n", " # Calculate approximate token count if API doesn't provide it\n", " # Rough estimate: ~4 characters per token for English text\n", " if not usage_info.get('total_tokens'):\n", " estimated_prompt_tokens = len(prompt) // 4\n", " estimated_completion_tokens = len(content) // 4\n", " estimated_total = estimated_prompt_tokens + estimated_completion_tokens\n", " usage_info['estimated_tokens'] = estimated_total\n", " usage_info['estimated_prompt_tokens'] = estimated_prompt_tokens\n", " usage_info['estimated_completion_tokens'] = estimated_completion_tokens\n", " \n", " return {\n", " 'status': 'success',\n", " 'content': content,\n", " 'elapsed_sec': elapsed,\n", " 'tokens': usage_info.get('total_tokens') or usage_info.get('estimated_tokens'),\n", " 'usage': usage_info,\n", " 'model': model_id\n", " }\n", " \n", " except Exception as e:\n", " elapsed = time.time() - start\n", " return {\n", " 'status': 'error',\n", " 'error': f\"{type(e).__name__}: {str(e)}\",\n", " 'elapsed_sec': elapsed,\n", " 'model': model_id\n", " }\n", "\n", "\n", "print(\"✅ Execution helpers defined: setup(), run()\")\n", "print(\" → Uses workshop_utils for proper SDK integration\")\n", "print(\" → setup() initializes with FoundryLocalManager\")\n", "print(\" → run() executes inference via OpenAI-compatible API\")\n", "print(\" → Token counting: Uses API data or estimates if unavailable\")" ] }, { "cell_type": "markdown", "id": "18ba23b5", "metadata": {}, "source": [ "### Обяснение: Самотест преди стартиране\n", "Извършва лек тест за свързаност чрез FoundryLocalManager за двата модела. Това проверява:\n", "- Достъпността на услугата\n", "- Възможността за инициализация на моделите\n", "- Разрешаването на псевдонимите към действителни ID на модели\n", "- Стабилността на връзката преди изпълнение на сравнението\n", "\n", "Функцията setup() използва официалния SDK модел от workshop_utils.\n" ] }, { "cell_type": "code", "execution_count": 34, "id": "e251bd89", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[Diagnostic] Checking Foundry Local service...\n", "\n", "❌ Foundry Local service not found!\n", "\n", "💡 To fix this:\n", " 1. Open a terminal\n", " 2. Run: foundry service start\n", " 3. Run: foundry model run phi-4-mini\n", " 4. Run: foundry model run qwen2.5-3b\n", " 5. Re-run this notebook\n", "\n", "⚠️ No service detected - FoundryLocalManager will attempt to start it\n" ] } ], "source": [ "# Simplified diagnostic: Just verify service is accessible\n", "import requests\n", "\n", "def check_foundry_service():\n", " \"\"\"Quick diagnostic to verify Foundry Local is running.\"\"\"\n", " # Try common ports\n", " endpoints_to_try = [\n", " \"http://localhost:59959\",\n", " \"http://127.0.0.1:59959\", \n", " \"http://localhost:55769\",\n", " \"http://127.0.0.1:55769\",\n", " ]\n", " \n", " print(\"[Diagnostic] Checking Foundry Local service...\")\n", " \n", " for endpoint in endpoints_to_try:\n", " try:\n", " response = requests.get(f\"{endpoint}/health\", timeout=2)\n", " if response.status_code == 200:\n", " print(f\"✅ Service is running at {endpoint}\")\n", " \n", " # Try to list models\n", " try:\n", " models_response = requests.get(f\"{endpoint}/v1/models\", timeout=2)\n", " if models_response.status_code == 200:\n", " models_data = models_response.json()\n", " model_count = len(models_data.get('data', []))\n", " print(f\"✅ Found {model_count} models available\")\n", " if model_count > 0:\n", " print(\" Models:\", [m.get('id', 'unknown') for m in models_data.get('data', [])[:5]])\n", " except Exception as e:\n", " print(f\"⚠️ Could not list models: {e}\")\n", " \n", " return endpoint\n", " except requests.exceptions.ConnectionError:\n", " continue\n", " except Exception as e:\n", " print(f\"⚠️ Error checking {endpoint}: {e}\")\n", " \n", " print(\"\\n❌ Foundry Local service not found!\")\n", " print(\"\\n💡 To fix this:\")\n", " print(\" 1. Open a terminal\")\n", " print(\" 2. Run: foundry service start\")\n", " print(\" 3. Run: foundry model run phi-4-mini\")\n", " print(\" 4. Run: foundry model run qwen2.5-3b\")\n", " print(\" 5. Re-run this notebook\")\n", " return None\n", "\n", "# Run diagnostic\n", "discovered_endpoint = check_foundry_service()\n", "\n", "if discovered_endpoint:\n", " print(f\"\\n✅ Service detected (will be managed by FoundryLocalManager)\")\n", "else:\n", " print(f\"\\n⚠️ No service detected - FoundryLocalManager will attempt to start it\")" ] }, { "cell_type": "code", "execution_count": 35, "id": "35317769", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "⚠️ The commands above are commented out.\n", "Uncomment them if you want to start the service from the notebook.\n", "\n", "💡 Recommended: Run these commands in a separate terminal instead:\n", " foundry service start\n", " foundry model run phi-4-mini\n", " foundry model run qwen2.5-3b\n" ] } ], "source": [ "# Quick Fix: Start service and load models from notebook\n", "# Uncomment the commands you need:\n", "\n", "# !foundry service start\n", "# !foundry model run phi-4-mini\n", "# !foundry model run qwen2.5-3b\n", "# !foundry model ls\n", "\n", "print(\"⚠️ The commands above are commented out.\")\n", "print(\"Uncomment them if you want to start the service from the notebook.\")\n", "print(\"\")\n", "print(\"💡 Recommended: Run these commands in a separate terminal instead:\")\n", "print(\" foundry service start\")\n", "print(\" foundry model run phi-4-mini\")\n", "print(\" foundry model run qwen2.5-3b\")" ] }, { "cell_type": "markdown", "id": "0de2d0a5", "metadata": {}, "source": [ "### 🛠️ Бързо решение: Стартиране на Foundry Local от Notebook (по избор)\n", "\n", "Ако диагностиката по-горе показва, че услугата не работи, можете да опитате да я стартирате оттук:\n", "\n", "**Забележка:** Това работи най-добре на Windows. На други платформи използвайте команди в терминала.\n" ] }, { "cell_type": "markdown", "id": "781c0bd2", "metadata": {}, "source": [ "### ⚠️ Отстраняване на грешки при свързване\n", "\n", "Ако виждате `APIConnectionError`, услугата Foundry Local може да не работи или моделите да не са заредени. Опитайте следните стъпки:\n", "\n", "**1. Проверете състоянието на услугата:**\n", "```bash\n", "# In a terminal (not in notebook):\n", "foundry service status\n", "```\n", "\n", "**2. Стартирайте услугата (ако не работи):**\n", "```bash\n", "foundry service start\n", "```\n", "\n", "**3. Заредете необходимите модели:**\n", "```bash\n", "# Load the models needed for comparison\n", "foundry model run phi-4-mini\n", "foundry model run qwen2.5-7b\n", "\n", "# Or use alternative models:\n", "foundry model run phi-3.5-mini\n", "foundry model run qwen2.5-3b\n", "```\n", "\n", "**4. Уверете се, че моделите са налични:**\n", "```bash\n", "foundry model ls\n", "```\n", "\n", "**Чести проблеми:**\n", "- ❌ Услугата не работи → Изпълнете `foundry service start`\n", "- ❌ Моделите не са заредени → Изпълнете `foundry model run `\n", "- ❌ Конфликти с портове → Проверете дали друга услуга използва порта\n", "- ❌ Блокиране от защитната стена → Уверете се, че локалните връзки са разрешени\n", "\n", "**Бързо решение:** Изпълнете диагностичната клетка по-долу преди проверката за готовност.\n" ] }, { "cell_type": "code", "execution_count": 36, "id": "8a607078", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[Init] Connecting to 'phi-4-mini' (attempt 1/2)...\n", "[OK] Connected to 'phi-4-mini' -> Phi-4-mini-instruct-cuda-gpu:4\n", " Endpoint: http://127.0.0.1:59959/v1\n", "[Init] Connecting to 'qwen2.5-7b' (attempt 1/2)...\n", "[OK] Connected to 'qwen2.5-7b' -> qwen2.5-7b-instruct-cuda-gpu:3\n", " Endpoint: http://127.0.0.1:59959/v1\n", "\n", "[Pre-flight Check]\n", " ✅ phi-4-mini: success - Phi-4-mini-instruct-cuda-gpu:4\n", " ✅ qwen2.5-7b: success - qwen2.5-7b-instruct-cuda-gpu:3\n" ] }, { "data": { "text/plain": [ "{'phi-4-mini': {'endpoint': 'http://127.0.0.1:59959/v1',\n", " 'resolved': 'Phi-4-mini-instruct-cuda-gpu:4',\n", " 'attempts': 1,\n", " 'status': 'success'},\n", " 'qwen2.5-7b': {'endpoint': 'http://127.0.0.1:59959/v1',\n", " 'resolved': 'qwen2.5-7b-instruct-cuda-gpu:3',\n", " 'attempts': 1,\n", " 'status': 'success'}}" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "preflight = {}\n", "retries = 2 # Number of retry attempts\n", "\n", "for a in (SLM, LLM):\n", " mgr, c, mid, info = setup(a, endpoint=ENDPOINT, retries=retries)\n", " # Keep the original status from info (either 'success' or 'failed')\n", " preflight[a] = info\n", "\n", "print('\\n[Pre-flight Check]')\n", "for alias, details in preflight.items():\n", " status_icon = '✅' if details['status'] == 'success' else '❌'\n", " print(f\" {status_icon} {alias}: {details['status']} - {details.get('resolved', details.get('error', 'unknown'))}\")\n", "\n", "preflight" ] }, { "cell_type": "markdown", "id": "ec76741a", "metadata": {}, "source": [ "### ✅ Предварителна проверка: Наличност на модела\n", "\n", "Тази клетка проверява дали и двата модела могат да бъдат достъпни на конфигурирания крайна точка преди да се изпълни сравнението.\n" ] }, { "cell_type": "markdown", "id": "a22961df", "metadata": {}, "source": [ "### Обяснение: Сравнение на изпълнение и събиране на резултати\n", "Извършва итерация през двата алиаса, използвайки официалния модел на Foundry SDK:\n", "1. Инициализиране на всеки модел с setup() (използва FoundryLocalManager)\n", "2. Изпълнение на инференция с API, съвместим с OpenAI\n", "3. Заснемане на латентност, токени и примерен изход\n", "4. Създаване на JSON обобщение с сравнителен анализ\n", "\n", "Това следва същия модел като примерите от Workshop в session04/model_compare.py.\n" ] }, { "cell_type": "code", "execution_count": 40, "id": "a6f12b99", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[Init] Connecting to 'phi-4-mini' (attempt 1/2)...\n", "[OK] Connected to 'phi-4-mini' -> Phi-4-mini-instruct-cuda-gpu:4\n", " Endpoint: http://127.0.0.1:59959/v1\n", "[Init] Connecting to 'qwen2.5-7b' (attempt 1/2)...\n", "[OK] Connected to 'qwen2.5-7b' -> qwen2.5-7b-instruct-cuda-gpu:3\n", " Endpoint: http://127.0.0.1:59959/v1\n", "[Init] Connecting to 'qwen2.5-7b' (attempt 1/2)...\n", "[OK] Connected to 'qwen2.5-7b' -> qwen2.5-7b-instruct-cuda-gpu:3\n", " Endpoint: http://127.0.0.1:59959/v1\n", "[\n", " {\n", " \"alias\": \"phi-4-mini\",\n", " \"status\": \"success\",\n", " \"content\": \"1. Reduced Latency: Local AI inference can significantly reduce latency by processing data closer to the source, which is particularly beneficial for real-time applications such as autonomous vehicles or augmented reality.\\n\\n2. Enhanced Privacy: By keeping data processing local, sensitive information is less likely to be exposed to external networks, thereby enhancing privacy and security.\\n\\n3. Lower Bandwidth Usage: Local AI inference reduces the need for data transmission over the network, which can save bandwidth and reduce the risk of network congestion.\\n\\n4. Improved Reliability: Local processing can be more reliable, as it is less dependent on network connectivity. This is particularly important in scenarios where network connectivity is unreliable or intermittent.\\n\\n5. Scalability: Local AI inference can be easily scaled by adding more local processing units, making it easier to handle increasing data volumes or more complex AI models.\",\n", " \"elapsed_sec\": 51.97519612312317,\n", " \"tokens\": 247,\n", " \"usage\": {\n", " \"estimated_tokens\": 247,\n", " \"estimated_prompt_tokens\": 9,\n", " \"estimated_completion_tokens\": 238\n", " },\n", " \"model\": \"Phi-4-mini-instruct-cuda-gpu:4\"\n", " },\n", " {\n", " \"alias\": \"qwen2.5-7b\",\n", " \"status\": \"success\",\n", " \"content\": \"Local AI inference offers several advantages over cloud-based or remote inference solutions. Here are five key benefits:\\n\\n1. **Latency Reduction**: Local AI inference reduces latency because the data does not need to be sent to a remote server for processing. This is particularly important in applications where real-time response is critical, such as autonomous vehicles, medical imaging, and real-time analytics.\\n\\n2. **Data Privacy and Security**: Processing data locally can enhance privacy and security by keeping sensitive information within the user's control. This is especially important in industries like healthcare, finance, and government where data breaches can have severe consequences.\\n\\n3. **Cost Efficiency**: For certain applications, local inference can be more cost-effective than cloud inference. While initial setup costs for hardware might be higher, ongoing costs for cloud services can add up over time, especially for high-frequency or large-scale operations.\\n\\n4. **Offline Capabilities**: Devices\",\n", " \"elapsed_sec\": 329.4796121120453,\n", " \"tokens\": 264,\n", " \"usage\": {\n", " \"estimated_tokens\": 264,\n", " \"estimated_prompt_tokens\": 9,\n", " \"estimated_completion_tokens\": 255\n", " },\n", " \"model\": \"qwen2.5-7b-instruct-cuda-gpu:3\"\n", " }\n", "]\n", "\n", "================================================================================\n", "COMPARISON SUMMARY\n", "================================================================================\n", "Alias Status Latency(s) Tokens \n", "--------------------------------------------------------------------------------\n", "✅ phi-4-mini success 51.975 ~247 (est.) \n", "✅ qwen2.5-7b success 329.480 ~264 (est.) \n", "--------------------------------------------------------------------------------\n", "\n", "Detailed Token Usage:\n", "\n", " phi-4-mini:\n", " Estimated prompt: 9\n", " Estimated completion: 238\n", " Estimated total: 247\n", " (API did not provide token counts - using ~4 chars/token estimate)\n", "\n", " qwen2.5-7b:\n", " Estimated prompt: 9\n", " Estimated completion: 255\n", " Estimated total: 264\n", " (API did not provide token counts - using ~4 chars/token estimate)\n", "\n", "================================================================================\n", "\n", "💡 SLM is 6.34x faster than LLM for this prompt\n", " SLM throughput: 4.8 tokens/sec\n", " LLM throughput: 0.8 tokens/sec\n" ] }, { "data": { "text/plain": [ "[{'alias': 'phi-4-mini',\n", " 'status': 'success',\n", " 'content': '1. Reduced Latency: Local AI inference can significantly reduce latency by processing data closer to the source, which is particularly beneficial for real-time applications such as autonomous vehicles or augmented reality.\\n\\n2. Enhanced Privacy: By keeping data processing local, sensitive information is less likely to be exposed to external networks, thereby enhancing privacy and security.\\n\\n3. Lower Bandwidth Usage: Local AI inference reduces the need for data transmission over the network, which can save bandwidth and reduce the risk of network congestion.\\n\\n4. Improved Reliability: Local processing can be more reliable, as it is less dependent on network connectivity. This is particularly important in scenarios where network connectivity is unreliable or intermittent.\\n\\n5. Scalability: Local AI inference can be easily scaled by adding more local processing units, making it easier to handle increasing data volumes or more complex AI models.',\n", " 'elapsed_sec': 51.97519612312317,\n", " 'tokens': 247,\n", " 'usage': {'estimated_tokens': 247,\n", " 'estimated_prompt_tokens': 9,\n", " 'estimated_completion_tokens': 238},\n", " 'model': 'Phi-4-mini-instruct-cuda-gpu:4'},\n", " {'alias': 'qwen2.5-7b',\n", " 'status': 'success',\n", " 'content': \"Local AI inference offers several advantages over cloud-based or remote inference solutions. Here are five key benefits:\\n\\n1. **Latency Reduction**: Local AI inference reduces latency because the data does not need to be sent to a remote server for processing. This is particularly important in applications where real-time response is critical, such as autonomous vehicles, medical imaging, and real-time analytics.\\n\\n2. **Data Privacy and Security**: Processing data locally can enhance privacy and security by keeping sensitive information within the user's control. This is especially important in industries like healthcare, finance, and government where data breaches can have severe consequences.\\n\\n3. **Cost Efficiency**: For certain applications, local inference can be more cost-effective than cloud inference. While initial setup costs for hardware might be higher, ongoing costs for cloud services can add up over time, especially for high-frequency or large-scale operations.\\n\\n4. **Offline Capabilities**: Devices\",\n", " 'elapsed_sec': 329.4796121120453,\n", " 'tokens': 264,\n", " 'usage': {'estimated_tokens': 264,\n", " 'estimated_prompt_tokens': 9,\n", " 'estimated_completion_tokens': 255},\n", " 'model': 'qwen2.5-7b-instruct-cuda-gpu:3'}]" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "results = []\n", "retries = 2 # Number of retry attempts\n", "\n", "for alias in (SLM, LLM):\n", " mgr, client, mid, info = setup(alias, endpoint=ENDPOINT, retries=retries)\n", " if client:\n", " r = run(client, mid, PROMPT)\n", " results.append({'alias': alias, **r})\n", " else:\n", " # If setup failed, record error\n", " results.append({\n", " 'alias': alias,\n", " 'status': 'error',\n", " 'error': info.get('error', 'Setup failed'),\n", " 'elapsed_sec': 0,\n", " 'tokens': None,\n", " 'model': alias\n", " })\n", "\n", "# Display results\n", "print(json.dumps(results, indent=2))\n", "\n", "# Quick comparative view\n", "print('\\n' + '='*80)\n", "print('COMPARISON SUMMARY')\n", "print('='*80)\n", "print(f\"{'Alias':<20} {'Status':<15} {'Latency(s)':<15} {'Tokens':<15}\")\n", "print('-'*80)\n", "\n", "for row in results:\n", " status = row.get('status', 'unknown')\n", " status_icon = '✅' if status == 'success' else '❌'\n", " latency_str = f\"{row.get('elapsed_sec', 0):.3f}\" if row.get('elapsed_sec') else 'N/A'\n", " \n", " # Handle token display - show if available or indicate estimated\n", " tokens = row.get('tokens')\n", " usage = row.get('usage', {})\n", " if tokens:\n", " if 'estimated_tokens' in usage:\n", " tokens_str = f\"~{tokens} (est.)\"\n", " else:\n", " tokens_str = str(tokens)\n", " else:\n", " tokens_str = 'N/A'\n", " \n", " print(f\"{status_icon} {row['alias']:<18} {status:<15} {latency_str:<15} {tokens_str:<15}\")\n", "\n", "print('-'*80)\n", "\n", "# Show detailed token breakdown if available\n", "print(\"\\nDetailed Token Usage:\")\n", "for row in results:\n", " if row.get('status') == 'success' and row.get('usage'):\n", " usage = row['usage']\n", " print(f\"\\n {row['alias']}:\")\n", " if 'prompt_tokens' in usage and usage['prompt_tokens']:\n", " print(f\" Prompt tokens: {usage['prompt_tokens']}\")\n", " print(f\" Completion tokens: {usage['completion_tokens']}\")\n", " print(f\" Total tokens: {usage['total_tokens']}\")\n", " elif 'estimated_tokens' in usage:\n", " print(f\" Estimated prompt: {usage['estimated_prompt_tokens']}\")\n", " print(f\" Estimated completion: {usage['estimated_completion_tokens']}\")\n", " print(f\" Estimated total: {usage['estimated_tokens']}\")\n", " print(f\" (API did not provide token counts - using ~4 chars/token estimate)\")\n", "\n", "print('\\n' + '='*80)\n", "\n", "# Calculate speedup if both succeeded\n", "if len(results) == 2 and all(r.get('status') == 'success' and r.get('elapsed_sec') for r in results):\n", " speedup = results[1]['elapsed_sec'] / results[0]['elapsed_sec']\n", " print(f\"\\n💡 SLM is {speedup:.2f}x faster than LLM for this prompt\")\n", " \n", " # Compare token throughput if available\n", " slm_tokens = results[0].get('tokens', 0)\n", " llm_tokens = results[1].get('tokens', 0)\n", " if slm_tokens and llm_tokens:\n", " slm_tps = slm_tokens / results[0]['elapsed_sec']\n", " llm_tps = llm_tokens / results[1]['elapsed_sec']\n", " print(f\" SLM throughput: {slm_tps:.1f} tokens/sec\")\n", " print(f\" LLM throughput: {llm_tps:.1f} tokens/sec\")\n", " \n", "elif any(r.get('status') == 'error' for r in results):\n", " print(f\"\\n⚠️ Some models failed - check errors above\")\n", " print(\" Ensure Foundry Local is running: foundry service start\")\n", " print(\" Ensure models are loaded: foundry model run \")\n", "\n", "results" ] }, { "cell_type": "markdown", "id": "592d052c", "metadata": {}, "source": [ "### Интерпретиране на резултатите\n", "\n", "**Основни показатели:**\n", "- **Закъснение**: Колкото по-ниско, толкова по-добре - показва по-бързо време за отговор\n", "- **Токени**: По-висока пропускателна способност = повече обработени токени\n", "- **Маршрут**: Потвърждава кой API крайна точка е използвана\n", "\n", "**Кога да използвате SLM срещу LLM:**\n", "- **SLM (Малък езиков модел)**: Бързи отговори, ниска употреба на ресурси, подходящ за прости задачи\n", "- **LLM (Голям езиков модел)**: По-високо качество, по-добро разсъждение, използвайте, когато качеството е най-важно\n", "\n", "**Следващи стъпки:**\n", "1. Опитайте различни подканвания, за да видите как сложността влияе на сравнението\n", "2. Експериментирайте с други двойки модели\n", "3. Използвайте примерите за маршрутизатор от Workshop (Сесия 06), за да маршрутизирате интелигентно според сложността на задачата\n" ] }, { "cell_type": "code", "execution_count": 38, "id": "e8caed5a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "======================================================================\n", "VALIDATION SUMMARY\n", "======================================================================\n", "✅ SLM Model: phi-4-mini\n", "✅ LLM Model: qwen2.5-7b\n", "✅ Using Foundry SDK Pattern: workshop_utils with FoundryLocalManager\n", "✅ Pre-flight passed: True\n", "✅ Comparison completed: True\n", "✅ Both models responded: True\n", "======================================================================\n", "🎉 ALL CHECKS PASSED! Notebook completed successfully.\n", " SLM (phi-4-mini) vs LLM (qwen2.5-7b) comparison completed.\n", " Performance: SLM is 5.14x faster\n", "======================================================================\n" ] } ], "source": [ "# Final Validation Check\n", "print(\"=\"*70)\n", "print(\"VALIDATION SUMMARY\")\n", "print(\"=\"*70)\n", "print(f\"✅ SLM Model: {SLM}\")\n", "print(f\"✅ LLM Model: {LLM}\")\n", "print(f\"✅ Using Foundry SDK Pattern: workshop_utils with FoundryLocalManager\")\n", "print(f\"✅ Pre-flight passed: {all(v['status'] == 'success' for v in preflight.values()) if 'preflight' in dir() else 'Not run yet'}\")\n", "print(f\"✅ Comparison completed: {len(results) == 2 if 'results' in dir() else 'Not run yet'}\")\n", "print(f\"✅ Both models responded: {all(r.get('status') == 'success' for r in results) if 'results' in dir() and results else 'Not run yet'}\")\n", "print(\"=\"*70)\n", "\n", "# Check for common configuration issues\n", "issues = []\n", "if 'LLM' in dir() and LLM not in ['qwen2.5-3b', 'qwen2.5-0.5b', 'qwen2.5-1.5b', 'qwen2.5-7b', 'phi-3.5-mini']:\n", " issues.append(f\"⚠️ LLM is '{LLM}' - expected qwen2.5-3b for memory efficiency\")\n", "if 'preflight' in dir() and not all(v['status'] == 'success' for v in preflight.values()):\n", " issues.append(\"⚠️ Pre-flight check failed - models not accessible\")\n", "if 'results' in dir() and results and not all(r.get('status') == 'success' for r in results):\n", " issues.append(\"⚠️ Comparison incomplete - check for errors above\")\n", "\n", "if not issues and 'results' in dir() and results and all(r.get('status') == 'success' for r in results):\n", " print(\"🎉 ALL CHECKS PASSED! Notebook completed successfully.\")\n", " print(f\" SLM ({SLM}) vs LLM ({LLM}) comparison completed.\")\n", " if len(results) == 2:\n", " speedup = results[1]['elapsed_sec'] / results[0]['elapsed_sec'] if results[0]['elapsed_sec'] > 0 else 0\n", " print(f\" Performance: SLM is {speedup:.2f}x faster\")\n", "elif issues:\n", " print(\"\\n⚠️ Issues detected:\")\n", " for issue in issues:\n", " print(f\" {issue}\")\n", " print(\"\\n💡 Troubleshooting:\")\n", " print(\" 1. Ensure service is running: foundry service start\")\n", " print(\" 2. Load models: foundry model run phi-4-mini && foundry model run qwen2.5-7b\")\n", " print(\" 3. Check model list: foundry model ls\")\n", "else:\n", " print(\"\\n💡 Run all cells above first, then re-run this validation.\")\n", "print(\"=\"*70)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n---\n\n**Отказ от отговорност**: \nТози документ е преведен с помощта на AI услуга за превод [Co-op Translator](https://github.com/Azure/co-op-translator). Въпреки че се стремим към точност, моля, имайте предвид, че автоматизираните преводи може да съдържат грешки или неточности. Оригиналният документ на неговия роден език трябва да се счита за авторитетен източник. За критична информация се препоръчва професионален човешки превод. Не носим отговорност за недоразумения или погрешни интерпретации, произтичащи от използването на този превод.\n" ] } ], "metadata": { "kernelspec": { "display_name": "demo", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.15" }, "coopTranslator": { "original_hash": "8220e33feaffbd35ece0f93e8214c24f", "translation_date": "2025-10-08T14:42:20+00:00", "source_file": "Workshop/notebooks/session04_model_compare.ipynb", "language_code": "bg" } }, "nbformat": 4, "nbformat_minor": 5 }