{
"cells": [
{
"cell_type": "markdown",
"id": "65ca1c8b",
"metadata": {},
"source": [
"# Сесия 1 – Стартиране на чат (Foundry Local)\n",
"\n",
"Този бележник стартира Foundry Local, изтегля предпочитания псевдоним на модела и изпълнява както стандартно, така и стрийминг завършване на чат.\n"
]
},
{
"cell_type": "markdown",
"id": "126def3c",
"metadata": {},
"source": [
"# Сценарий\n",
"Тази сесия представя абсолютния минимум за настройка на локален малък езиков модел, който да отговаря чрез Foundry Local. Ще направите следното:\n",
"- Инсталирате SDK / клиентските зависимости.\n",
"- Инициализирате Foundry Local мениджъра за избран псевдоним (по подразбиране: `phi-4-mini`).\n",
"- Приложите защитен monkey‑patch, за да се справите с опционални полета в метаданните на модела.\n",
"- Изпратите стандартна заявка за завършване на чат.\n",
"- Стриймвате отговор токен по токен.\n",
"\n",
"Целта е да валидирате вашата локална среда и мрежовия път, преди да преминете към RAG, маршрутизиране или агенти.\n"
]
},
{
"cell_type": "markdown",
"id": "79dbb732",
"metadata": {},
"source": [
"### Обяснение: Инсталиране на зависимости\n",
"Инсталира Python пакетите, необходими за този минимален чат поток:\n",
"- `foundry-local-sdk`: Управление на локални модели и жизнен цикъл на услугите.\n",
"- `openai`: Позната клиентска абстракция за завършване на чатове.\n",
"- `rich`: Красиво форматиране за по-ясен изход в notebook.\n",
"\n",
"Повторното изпълнение е безопасно (идемпотентно). Пропуснете, ако вашата среда вече разполага с тези.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "09accd63",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.3\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
"Note: you may need to restart the kernel to use updated packages.\n"
]
}
],
"source": [
"# Install required libraries (idempotent)\n",
"%pip install -q foundry-local-sdk openai rich"
]
},
{
"cell_type": "markdown",
"id": "1c32e699",
"metadata": {},
"source": [
"### Обяснение: Основни импорти\n",
"Включва модули, използвани в целия тетрадката:\n",
"- `FoundryLocalManager` за взаимодействие с локалната среда на модела.\n",
"- `OpenAI` клиент, за да можем да използваме познатия API за чат завършване.\n",
"- `rich.print` за стилизирано извеждане.\n",
"\n",
"Тук не се извършват мрежови заявки—това просто подготвя пространството от имена.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "73cb8c38",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"from foundry_local import FoundryLocalManager\n",
"from foundry_local.models import FoundryModelInfo\n",
"from openai import OpenAI\n",
"from rich import print"
]
},
{
"cell_type": "markdown",
"id": "5fda4e9a",
"metadata": {},
"source": [
"### Обяснение: Инициализация на мениджъра и корекция на метаданни\n",
"Инициализира `FoundryLocalManager` за избрания псевдоним и прилага защитен monkey‑patch, за да обработва плавно отговорите на услугата, когато `promptTemplate` може да бъде `null`.\n",
"\n",
"Основни резултати:\n",
"- Потвърждава състоянието на услугата и крайната точка.\n",
"- Изброява кешираните модели (проверява локалното хранилище).\n",
"- Разрешава конкретния ID на модела за псевдонима (използван при последващи чат заявки).\n",
"\n",
"Ако срещнете проблеми с валидирането в суровите метаданни на услугата, този модел показва как да ги изчистите, без да се налага да разклонявате SDK.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "2c0e087c",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Service running: True\n",
"\n"
],
"text/plain": [
"\u001b[1;32mService running:\u001b[0m \u001b[3;92mTrue\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Endpoint: http://127.0.0.1:50262/v1\n",
"\n"
],
"text/plain": [
"Endpoint: \u001b[4;94mhttp://127.0.0.1:50262/v1\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Cached models:\n",
"[\n",
" FoundryModelInfo(\n",
" alias='phi-4-mini',\n",
" id='Phi-4-mini-instruct-generic-gpu:4',\n",
" version='4',\n",
" execution_provider='WebGpuExecutionProvider',\n",
" device_type=<DeviceType.GPU: 'GPU'>,\n",
" uri='azureml://registries/azureml/models/Phi-4-mini-instruct-generic-gpu/versions/4',\n",
" file_size_mb=3809,\n",
" prompt_template={\n",
" 'system': '<|system|>{Content}<|end|>',\n",
" 'user': '<|user|>{Content}<|end|>',\n",
" 'assistant': '<|assistant|>{Content}<|end|>',\n",
" 'prompt': '<|user|>{Content}<|end|><|assistant|>'\n",
" },\n",
" provider='AzureFoundry',\n",
" publisher='Microsoft',\n",
" license='MIT',\n",
" task='chat-completion',\n",
" ep_override=None\n",
" ),\n",
" FoundryModelInfo(\n",
" alias='qwen2.5-0.5b',\n",
" id='qwen2.5-0.5b-instruct-generic-gpu:3',\n",
" version='3',\n",
" execution_provider='WebGpuExecutionProvider',\n",
" device_type=<DeviceType.GPU: 'GPU'>,\n",
" uri='azureml://registries/azureml/models/qwen2.5-0.5b-instruct-generic-gpu/versions/3',\n",
" file_size_mb=700,\n",
" prompt_template={\n",
" 'system': '<|im_start|>system\\n{Content}<|im_end|>',\n",
" 'user': '<|im_start|>user\\n{Content}<|im_end|>',\n",
" 'assistant': '<|im_start|>assistant\\n{Content}<|im_end|>',\n",
" 'prompt': '<|im_start|>user\\n{Content}<|im_end|>\\n<|im_start|>assistant'\n",
" },\n",
" provider='AzureFoundry',\n",
" publisher='Microsoft',\n",
" license='apache-2.0',\n",
" task='chat-completion',\n",
" ep_override=None\n",
" ),\n",
" FoundryModelInfo(\n",
" alias='phi-3.5-mini',\n",
" id='Phi-3.5-mini-instruct-generic-gpu:1',\n",
" version='1',\n",
" execution_provider='WebGpuExecutionProvider',\n",
" device_type=<DeviceType.GPU: 'GPU'>,\n",
" uri='azureml://registries/azureml/models/Phi-3.5-mini-instruct-generic-gpu/versions/1',\n",
" file_size_mb=2211,\n",
" prompt_template={\n",
" 'prompt': '<|user|>\\n{Content}<|end|>\\n<|assistant|>',\n",
" 'assistant': '<|assistant|>\\n{Content}<|end|>'\n",
" },\n",
" provider='AzureFoundry',\n",
" publisher='Microsoft',\n",
" license='MIT',\n",
" task='chat-completion',\n",
" ep_override=None\n",
" )\n",
"]\n",
"\n"
],
"text/plain": [
"Cached models:\n",
"\u001b[1m[\u001b[0m\n",
" \u001b[1;35mFoundryModelInfo\u001b[0m\u001b[1m(\u001b[0m\n",
" \u001b[33malias\u001b[0m=\u001b[32m'phi-4-mini'\u001b[0m,\n",
" \u001b[33mid\u001b[0m=\u001b[32m'Phi-4-mini-instruct-generic-gpu:4'\u001b[0m,\n",
" \u001b[33mversion\u001b[0m=\u001b[32m'4'\u001b[0m,\n",
" \u001b[33mexecution_provider\u001b[0m=\u001b[32m'WebGpuExecutionProvider'\u001b[0m,\n",
" \u001b[33mdevice_type\u001b[0m=\u001b[1m<\u001b[0m\u001b[1;95mDeviceType.GPU:\u001b[0m\u001b[39m \u001b[0m\u001b[32m'GPU'\u001b[0m\u001b[39m>,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33muri\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'azureml://registries/azureml/models/Phi-4-mini-instruct-generic-gpu/versions/4'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mfile_size_mb\u001b[0m\u001b[39m=\u001b[0m\u001b[1;36m3809\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mprompt_template\u001b[0m\u001b[39m=\u001b[0m\u001b[1;39m{\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'system'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|system|>\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|end|>'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'user'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|user|>\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|end|>'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'assistant'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|assistant|>\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|end|>'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'prompt'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|user|>\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|end|><|assistant|>'\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[1;39m}\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mprovider\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'AzureFoundry'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mpublisher\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'Microsoft'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mlicense\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'MIT'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mtask\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'chat-completion'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mep_override\u001b[0m\u001b[39m=\u001b[0m\u001b[3;35mNone\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[1;39m)\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[1;35mFoundryModelInfo\u001b[0m\u001b[1;39m(\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33malias\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'qwen2.5-0.5b'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mid\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'qwen2.5-0.5b-instruct-generic-gpu:3'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mversion\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'3'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mexecution_provider\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'WebGpuExecutionProvider'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mdevice_type\u001b[0m\u001b[39m=,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33muri\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'azureml://registries/azureml/models/qwen2.5-0.5b-instruct-generic-gpu/versions/3'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mfile_size_mb\u001b[0m\u001b[39m=\u001b[0m\u001b[1;36m700\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mprompt_template\u001b[0m\u001b[39m=\u001b[0m\u001b[1;39m{\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'system'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|im_start|>system\\n\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|im_end|>'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'user'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|im_start|>user\\n\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|im_end|>'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'assistant'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|im_start|>assistant\\n\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|im_end|>'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'prompt'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|im_start|>user\\n\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|im_end|>\\n<|im_start|>assistant'\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[1;39m}\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mprovider\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'AzureFoundry'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mpublisher\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'Microsoft'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mlicense\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'apache-2.0'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mtask\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'chat-completion'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mep_override\u001b[0m\u001b[39m=\u001b[0m\u001b[3;35mNone\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[1;39m)\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[1;35mFoundryModelInfo\u001b[0m\u001b[1;39m(\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33malias\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'phi-3.5-mini'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mid\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'Phi-3.5-mini-instruct-generic-gpu:1'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mversion\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'1'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mexecution_provider\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'WebGpuExecutionProvider'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mdevice_type\u001b[0m\u001b[39m=,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33muri\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'azureml://registries/azureml/models/Phi-3.5-mini-instruct-generic-gpu/versions/1'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mfile_size_mb\u001b[0m\u001b[39m=\u001b[0m\u001b[1;36m2211\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mprompt_template\u001b[0m\u001b[39m=\u001b[0m\u001b[1;39m{\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'prompt'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|user|>\\n\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|end|>\\n<|assistant|>'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'assistant'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|assistant|>\\n\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|end|\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m\n",
" \u001b[1m}\u001b[0m,\n",
" \u001b[33mprovider\u001b[0m=\u001b[32m'AzureFoundry'\u001b[0m,\n",
" \u001b[33mpublisher\u001b[0m=\u001b[32m'Microsoft'\u001b[0m,\n",
" \u001b[33mlicense\u001b[0m=\u001b[32m'MIT'\u001b[0m,\n",
" \u001b[33mtask\u001b[0m=\u001b[32m'chat-completion'\u001b[0m,\n",
" \u001b[33mep_override\u001b[0m=\u001b[3;35mNone\u001b[0m\n",
" \u001b[1m)\u001b[0m\n",
"\u001b[1m]\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Using model id: Phi-4-mini-instruct-generic-gpu:4\n",
"\n"
],
"text/plain": [
"Using model id: Phi-\u001b[1;36m4\u001b[0m-mini-instruct-generic-gpu:\u001b[1;36m4\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Monkeypatch to tolerate service responses where promptTemplate is null\n",
"_original_from_list_response = FoundryModelInfo.from_list_response\n",
"\n",
"def _safe_from_list_response(response): # type: ignore\n",
" try:\n",
" if isinstance(response, dict) and response.get(\"promptTemplate\") is None:\n",
" # Normalize to empty dict so pydantic validation passes\n",
" response[\"promptTemplate\"] = {}\n",
" except Exception as e: # pragma: no cover\n",
" print(f\"[yellow]Warning: safe wrapper encountered issue normalizing promptTemplate: {e}[/yellow]\")\n",
" return _original_from_list_response(response)\n",
"\n",
"# Apply patch only once\n",
"if getattr(FoundryModelInfo.from_list_response, \"__name__\", \"\") != \"_safe_from_list_response\":\n",
" FoundryModelInfo.from_list_response = staticmethod(_safe_from_list_response) # type: ignore\n",
"\n",
"ALIAS = os.getenv('FOUNDRY_LOCAL_ALIAS', 'phi-4-mini')\n",
"manager = FoundryLocalManager(ALIAS)\n",
"print(f'[bold green]Service running:[/bold green] {manager.is_service_running()}')\n",
"print(f'Endpoint: {manager.endpoint}')\n",
"print('Cached models:', manager.list_cached_models())\n",
"model_id = manager.get_model_info(ALIAS).id\n",
"print(f'Using model id: {model_id}')"
]
},
{
"cell_type": "markdown",
"id": "3fbb2059",
"metadata": {},
"source": [
"### Обяснение: Основно завършване на чат\n",
"Създава клиент, съвместим с `OpenAI`, който сочи към локалната крайна точка на Foundry и изпълнява едно завършване на чат без поточно предаване. Основни акценти тук:\n",
"- Уверете се, че моделът отговаря без грешка.\n",
"- Проверете латентността / формата на изхода.\n",
"- Поддържайте `max_tokens` скромен, за да пестите ресурси.\n",
"\n",
"Ако това не успее, проверете отново дали локалната услуга Foundry работи и дали псевдонимът се разрешава правилно.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "28fb04dd",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Local inference for privacy refers to the practice of performing data analysis on a local device without sending \n",
"sensitive information to a central server. Two benefits of this approach are:\n",
"\n",
"\n",
"1. **Enhanced Privacy**: Local inference keeps personal data on the user's device, reducing the risk of data \n",
"breaches and unauthorized access. Since the data is not transmitted over the network, it is less susceptible to \n",
"interception by malicious actors.\n",
"\n",
"\n",
"2. **Data Sovereignty**: Users retain control over their data, as it does not leave their device. This means that \n",
"individuals or organizations can comply with local data protection regulations, such as the General\n",
"\n"
],
"text/plain": [
"Local inference for privacy refers to the practice of performing data analysis on a local device without sending \n",
"sensitive information to a central server. Two benefits of this approach are:\n",
"\n",
"\n",
"\u001b[1;36m1\u001b[0m. **Enhanced Privacy**: Local inference keeps personal data on the user's device, reducing the risk of data \n",
"breaches and unauthorized access. Since the data is not transmitted over the network, it is less susceptible to \n",
"interception by malicious actors.\n",
"\n",
"\n",
"\u001b[1;36m2\u001b[0m. **Data Sovereignty**: Users retain control over their data, as it does not leave their device. This means that \n",
"individuals or organizations can comply with local data protection regulations, such as the General\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"client = OpenAI(base_url=manager.endpoint, api_key=manager.api_key or 'not-needed')\n",
"prompt = 'List two benefits of local inference for privacy.'\n",
"resp = client.chat.completions.create(\n",
" model=model_id,\n",
" messages=[{'role':'user','content':prompt}],\n",
" max_tokens=120,\n",
" temperature=0.5\n",
")\n",
"print(resp.choices[0].message.content)"
]
},
{
"cell_type": "markdown",
"id": "f2091aa0",
"metadata": {},
"source": [
"### Обяснение: Поточно завършване на чат\n",
"Демонстрира поточно предаване на токени за подобрена възприемана латентност и интерактивен потребителски интерфейс. Цикълът отпечатва постепенни промени, докато пристигат:\n",
"- Полезно за чат интерфейси, където ранният частичен изход е важен.\n",
"- Позволява ви да измерите скоростта на токените спрямо латентността на пълното завършване.\n",
"\n",
"Можете да адаптирате този модел, за да натрупвате токени, да актуализирате прогресивен уиджет или да прекъснете генерирането по средата.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "5e85a2e8",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Edge
\n"
],
"text/plain": [
"Edge"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" AI
\n"
],
"text/plain": [
" AI"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" refers
\n"
],
"text/plain": [
" refers"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" to
\n"
],
"text/plain": [
" to"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" artificial
\n"
],
"text/plain": [
" artificial"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" intelligence
\n"
],
"text/plain": [
" intelligence"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" algorithms
\n"
],
"text/plain": [
" algorithms"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" and
\n"
],
"text/plain": [
" and"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" models
\n"
],
"text/plain": [
" models"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" that
\n"
],
"text/plain": [
" that"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" are
\n"
],
"text/plain": [
" are"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" deployed
\n"
],
"text/plain": [
" deployed"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" at
\n"
],
"text/plain": [
" at"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" the
\n"
],
"text/plain": [
" the"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" edge
\n"
],
"text/plain": [
" edge"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" of
\n"
],
"text/plain": [
" of"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" the
\n"
],
"text/plain": [
" the"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" network
\n"
],
"text/plain": [
" network"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
",
\n"
],
"text/plain": [
","
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" closer
\n"
],
"text/plain": [
" closer"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" to
\n"
],
"text/plain": [
" to"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" the
\n"
],
"text/plain": [
" the"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" source
\n"
],
"text/plain": [
" source"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" of
\n"
],
"text/plain": [
" of"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" data
\n"
],
"text/plain": [
" data"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
",
\n"
],
"text/plain": [
","
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" to
\n"
],
"text/plain": [
" to"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" enable
\n"
],
"text/plain": [
" enable"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" real
\n"
],
"text/plain": [
" real"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"-time
\n"
],
"text/plain": [
"-time"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" processing
\n"
],
"text/plain": [
" processing"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" and
\n"
],
"text/plain": [
" and"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" decision
\n"
],
"text/plain": [
" decision"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"-making
\n"
],
"text/plain": [
"-making"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" with
\n"
],
"text/plain": [
" with"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" reduced
\n"
],
"text/plain": [
" reduced"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" latency
\n"
],
"text/plain": [
" latency"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" and
\n"
],
"text/plain": [
" and"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" bandwidth
\n"
],
"text/plain": [
" bandwidth"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" usage
\n"
],
"text/plain": [
" usage"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
".
\n"
],
"text/plain": [
"."
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Streaming example\n",
"stream = client.chat.completions.create(\n",
" model=model_id,\n",
" messages=[{'role':'user','content':'Give a one-sentence definition of edge AI.'}],\n",
" stream=True,\n",
" max_tokens=60,\n",
" temperature=0.4\n",
")\n",
"for chunk in stream:\n",
" delta = chunk.choices[0].delta\n",
" if delta and delta.content:\n",
" print(delta.content, end='', flush=True)\n",
"print()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n**Отказ от отговорност**: \nТози документ е преведен с помощта на AI услуга за превод [Co-op Translator](https://github.com/Azure/co-op-translator). Въпреки че се стремим към точност, моля, имайте предвид, че автоматизираните преводи може да съдържат грешки или неточности. Оригиналният документ на неговия роден език трябва да се счита за авторитетен източник. За критична информация се препоръчва професионален човешки превод. Ние не носим отговорност за недоразумения или погрешни интерпретации, произтичащи от използването на този превод.\n\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.12"
},
"coopTranslator": {
"original_hash": "3c07eacf0aa63b8dadf6733414138d47",
"translation_date": "2025-11-11T21:10:52+00:00",
"source_file": "Workshop/notebooks/session01_chat_bootstrap.ipynb",
"language_code": "bg"
}
},
"nbformat": 4,
"nbformat_minor": 5
}