{
"cells": [
{
"cell_type": "markdown",
"id": "65ca1c8b",
"metadata": {},
"source": [
"# Session 1 – Chat Bootstrap (Foundry Local)\n",
"\n",
"This notebook bootstraps Foundry Local, downloads the preferred model alias, and performs both a standard and streaming chat completion."
]
},
{
"cell_type": "markdown",
"id": "126def3c",
"metadata": {},
"source": [
"# Scenario\n",
"This session introduces the absolute minimum to get a local small language model responding via Foundry Local. You will:\n",
"- Install the SDK / client dependencies.\n",
"- Initialize the Foundry Local manager for a chosen alias (default: `phi-4-mini`).\n",
"- Apply a defensive monkey‑patch to tolerate optional fields in model metadata.\n",
"- Send a standard chat completion request.\n",
"- Stream a response token‑by‑token.\n",
"\n",
"The goal is to validate your local runtime & network path before moving to RAG, routing, or agents."
]
},
{
"cell_type": "markdown",
"id": "79dbb732",
"metadata": {},
"source": [
"### Explanation: Dependency Installation\n",
"Installs the Python packages required for this minimal chat flow:\n",
"- `foundry-local-sdk`: Manage local models and service lifecycle.\n",
"- `openai`: Familiar client abstraction for chat completions.\n",
"- `rich`: Pretty printing for clearer notebook output.\n",
"\n",
"Re-running is safe (idempotent). Skip if your environment already has these."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "09accd63",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.3\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
"Note: you may need to restart the kernel to use updated packages.\n"
]
}
],
"source": [
"# Install required libraries (idempotent)\n",
"%pip install -q foundry-local-sdk openai rich"
]
},
{
"cell_type": "markdown",
"id": "1c32e699",
"metadata": {},
"source": [
"### Explanation: Core Imports\n",
"Brings in modules used throughout the notebook:\n",
"- `FoundryLocalManager` to interface with the local model runtime.\n",
"- `OpenAI` client so we can reuse the familiar chat completion API surface.\n",
"- `rich.print` for styled output.\n",
"\n",
"No network calls happen here—this just prepares the namespace."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "73cb8c38",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"from foundry_local import FoundryLocalManager\n",
"from foundry_local.models import FoundryModelInfo\n",
"from openai import OpenAI\n",
"from rich import print"
]
},
{
"cell_type": "markdown",
"id": "5fda4e9a",
"metadata": {},
"source": [
"### Explanation: Manager Initialization & Metadata Patch\n",
"Initializes `FoundryLocalManager` for the chosen alias and applies a defensive monkey‑patch to gracefully handle service responses where `promptTemplate` might be `null`.\n",
"\n",
"Key outcomes:\n",
"- Confirms service state and endpoint.\n",
"- Lists cached models (verifies local store).\n",
"- Resolves the concrete model ID for the alias (used in later chat calls).\n",
"\n",
"If you encounter validation issues in raw service metadata, this pattern shows how to sanitize without forking the SDK."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "2c0e087c",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Service running: True\n",
"\n"
],
"text/plain": [
"\u001b[1;32mService running:\u001b[0m \u001b[3;92mTrue\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Endpoint: http://127.0.0.1:50262/v1\n",
"\n"
],
"text/plain": [
"Endpoint: \u001b[4;94mhttp://127.0.0.1:50262/v1\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Cached models:\n",
"[\n",
" FoundryModelInfo(\n",
" alias='phi-4-mini',\n",
" id='Phi-4-mini-instruct-generic-gpu:4',\n",
" version='4',\n",
" execution_provider='WebGpuExecutionProvider',\n",
" device_type=<DeviceType.GPU: 'GPU'>,\n",
" uri='azureml://registries/azureml/models/Phi-4-mini-instruct-generic-gpu/versions/4',\n",
" file_size_mb=3809,\n",
" prompt_template={\n",
" 'system': '<|system|>{Content}<|end|>',\n",
" 'user': '<|user|>{Content}<|end|>',\n",
" 'assistant': '<|assistant|>{Content}<|end|>',\n",
" 'prompt': '<|user|>{Content}<|end|><|assistant|>'\n",
" },\n",
" provider='AzureFoundry',\n",
" publisher='Microsoft',\n",
" license='MIT',\n",
" task='chat-completion',\n",
" ep_override=None\n",
" ),\n",
" FoundryModelInfo(\n",
" alias='qwen2.5-0.5b',\n",
" id='qwen2.5-0.5b-instruct-generic-gpu:3',\n",
" version='3',\n",
" execution_provider='WebGpuExecutionProvider',\n",
" device_type=<DeviceType.GPU: 'GPU'>,\n",
" uri='azureml://registries/azureml/models/qwen2.5-0.5b-instruct-generic-gpu/versions/3',\n",
" file_size_mb=700,\n",
" prompt_template={\n",
" 'system': '<|im_start|>system\\n{Content}<|im_end|>',\n",
" 'user': '<|im_start|>user\\n{Content}<|im_end|>',\n",
" 'assistant': '<|im_start|>assistant\\n{Content}<|im_end|>',\n",
" 'prompt': '<|im_start|>user\\n{Content}<|im_end|>\\n<|im_start|>assistant'\n",
" },\n",
" provider='AzureFoundry',\n",
" publisher='Microsoft',\n",
" license='apache-2.0',\n",
" task='chat-completion',\n",
" ep_override=None\n",
" ),\n",
" FoundryModelInfo(\n",
" alias='phi-3.5-mini',\n",
" id='Phi-3.5-mini-instruct-generic-gpu:1',\n",
" version='1',\n",
" execution_provider='WebGpuExecutionProvider',\n",
" device_type=<DeviceType.GPU: 'GPU'>,\n",
" uri='azureml://registries/azureml/models/Phi-3.5-mini-instruct-generic-gpu/versions/1',\n",
" file_size_mb=2211,\n",
" prompt_template={\n",
" 'prompt': '<|user|>\\n{Content}<|end|>\\n<|assistant|>',\n",
" 'assistant': '<|assistant|>\\n{Content}<|end|>'\n",
" },\n",
" provider='AzureFoundry',\n",
" publisher='Microsoft',\n",
" license='MIT',\n",
" task='chat-completion',\n",
" ep_override=None\n",
" )\n",
"]\n",
"\n"
],
"text/plain": [
"Cached models:\n",
"\u001b[1m[\u001b[0m\n",
" \u001b[1;35mFoundryModelInfo\u001b[0m\u001b[1m(\u001b[0m\n",
" \u001b[33malias\u001b[0m=\u001b[32m'phi-4-mini'\u001b[0m,\n",
" \u001b[33mid\u001b[0m=\u001b[32m'Phi-4-mini-instruct-generic-gpu:4'\u001b[0m,\n",
" \u001b[33mversion\u001b[0m=\u001b[32m'4'\u001b[0m,\n",
" \u001b[33mexecution_provider\u001b[0m=\u001b[32m'WebGpuExecutionProvider'\u001b[0m,\n",
" \u001b[33mdevice_type\u001b[0m=\u001b[1m<\u001b[0m\u001b[1;95mDeviceType.GPU:\u001b[0m\u001b[39m \u001b[0m\u001b[32m'GPU'\u001b[0m\u001b[39m>,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33muri\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'azureml://registries/azureml/models/Phi-4-mini-instruct-generic-gpu/versions/4'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mfile_size_mb\u001b[0m\u001b[39m=\u001b[0m\u001b[1;36m3809\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mprompt_template\u001b[0m\u001b[39m=\u001b[0m\u001b[1;39m{\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'system'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|system|>\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|end|>'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'user'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|user|>\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|end|>'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'assistant'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|assistant|>\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|end|>'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'prompt'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|user|>\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|end|><|assistant|>'\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[1;39m}\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mprovider\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'AzureFoundry'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mpublisher\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'Microsoft'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mlicense\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'MIT'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mtask\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'chat-completion'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mep_override\u001b[0m\u001b[39m=\u001b[0m\u001b[3;35mNone\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[1;39m)\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[1;35mFoundryModelInfo\u001b[0m\u001b[1;39m(\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33malias\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'qwen2.5-0.5b'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mid\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'qwen2.5-0.5b-instruct-generic-gpu:3'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mversion\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'3'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mexecution_provider\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'WebGpuExecutionProvider'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mdevice_type\u001b[0m\u001b[39m=,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33muri\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'azureml://registries/azureml/models/qwen2.5-0.5b-instruct-generic-gpu/versions/3'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mfile_size_mb\u001b[0m\u001b[39m=\u001b[0m\u001b[1;36m700\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mprompt_template\u001b[0m\u001b[39m=\u001b[0m\u001b[1;39m{\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'system'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|im_start|>system\\n\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|im_end|>'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'user'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|im_start|>user\\n\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|im_end|>'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'assistant'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|im_start|>assistant\\n\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|im_end|>'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'prompt'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|im_start|>user\\n\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|im_end|>\\n<|im_start|>assistant'\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[1;39m}\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mprovider\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'AzureFoundry'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mpublisher\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'Microsoft'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mlicense\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'apache-2.0'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mtask\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'chat-completion'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mep_override\u001b[0m\u001b[39m=\u001b[0m\u001b[3;35mNone\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[1;39m)\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[1;35mFoundryModelInfo\u001b[0m\u001b[1;39m(\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33malias\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'phi-3.5-mini'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mid\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'Phi-3.5-mini-instruct-generic-gpu:1'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mversion\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'1'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mexecution_provider\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'WebGpuExecutionProvider'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mdevice_type\u001b[0m\u001b[39m=,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33muri\u001b[0m\u001b[39m=\u001b[0m\u001b[32m'azureml://registries/azureml/models/Phi-3.5-mini-instruct-generic-gpu/versions/1'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mfile_size_mb\u001b[0m\u001b[39m=\u001b[0m\u001b[1;36m2211\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[33mprompt_template\u001b[0m\u001b[39m=\u001b[0m\u001b[1;39m{\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'prompt'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|user|>\\n\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|end|>\\n<|assistant|>'\u001b[0m\u001b[39m,\u001b[0m\n",
"\u001b[39m \u001b[0m\u001b[32m'assistant'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'<|assistant|>\\n\u001b[0m\u001b[32m{\u001b[0m\u001b[32mContent\u001b[0m\u001b[32m}\u001b[0m\u001b[32m<|end|\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m\n",
" \u001b[1m}\u001b[0m,\n",
" \u001b[33mprovider\u001b[0m=\u001b[32m'AzureFoundry'\u001b[0m,\n",
" \u001b[33mpublisher\u001b[0m=\u001b[32m'Microsoft'\u001b[0m,\n",
" \u001b[33mlicense\u001b[0m=\u001b[32m'MIT'\u001b[0m,\n",
" \u001b[33mtask\u001b[0m=\u001b[32m'chat-completion'\u001b[0m,\n",
" \u001b[33mep_override\u001b[0m=\u001b[3;35mNone\u001b[0m\n",
" \u001b[1m)\u001b[0m\n",
"\u001b[1m]\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Using model id: Phi-4-mini-instruct-generic-gpu:4\n",
"\n"
],
"text/plain": [
"Using model id: Phi-\u001b[1;36m4\u001b[0m-mini-instruct-generic-gpu:\u001b[1;36m4\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Monkeypatch to tolerate service responses where promptTemplate is null\n",
"_original_from_list_response = FoundryModelInfo.from_list_response\n",
"\n",
"def _safe_from_list_response(response): # type: ignore\n",
" try:\n",
" if isinstance(response, dict) and response.get(\"promptTemplate\") is None:\n",
" # Normalize to empty dict so pydantic validation passes\n",
" response[\"promptTemplate\"] = {}\n",
" except Exception as e: # pragma: no cover\n",
" print(f\"[yellow]Warning: safe wrapper encountered issue normalizing promptTemplate: {e}[/yellow]\")\n",
" return _original_from_list_response(response)\n",
"\n",
"# Apply patch only once\n",
"if getattr(FoundryModelInfo.from_list_response, \"__name__\", \"\") != \"_safe_from_list_response\":\n",
" FoundryModelInfo.from_list_response = staticmethod(_safe_from_list_response) # type: ignore\n",
"\n",
"ALIAS = os.getenv('FOUNDRY_LOCAL_ALIAS', 'phi-4-mini')\n",
"manager = FoundryLocalManager(ALIAS)\n",
"print(f'[bold green]Service running:[/bold green] {manager.is_service_running()}')\n",
"print(f'Endpoint: {manager.endpoint}')\n",
"print('Cached models:', manager.list_cached_models())\n",
"model_id = manager.get_model_info(ALIAS).id\n",
"print(f'Using model id: {model_id}')"
]
},
{
"cell_type": "markdown",
"id": "3fbb2059",
"metadata": {},
"source": [
"### Explanation: Basic Chat Completion\n",
"Creates an `OpenAI`-compatible client pointing at the local Foundry endpoint and performs a single non‑streaming chat completion. Focus here:\n",
"- Ensure the model responds without error.\n",
"- Validate latency / output format.\n",
"- Keep `max_tokens` modest to conserve resources.\n",
"\n",
"If this fails, re-check that the Foundry Local service is running and the alias resolves correctly."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "28fb04dd",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Local inference for privacy refers to the practice of performing data analysis on a local device without sending \n",
"sensitive information to a central server. Two benefits of this approach are:\n",
"\n",
"\n",
"1. **Enhanced Privacy**: Local inference keeps personal data on the user's device, reducing the risk of data \n",
"breaches and unauthorized access. Since the data is not transmitted over the network, it is less susceptible to \n",
"interception by malicious actors.\n",
"\n",
"\n",
"2. **Data Sovereignty**: Users retain control over their data, as it does not leave their device. This means that \n",
"individuals or organizations can comply with local data protection regulations, such as the General\n",
"\n"
],
"text/plain": [
"Local inference for privacy refers to the practice of performing data analysis on a local device without sending \n",
"sensitive information to a central server. Two benefits of this approach are:\n",
"\n",
"\n",
"\u001b[1;36m1\u001b[0m. **Enhanced Privacy**: Local inference keeps personal data on the user's device, reducing the risk of data \n",
"breaches and unauthorized access. Since the data is not transmitted over the network, it is less susceptible to \n",
"interception by malicious actors.\n",
"\n",
"\n",
"\u001b[1;36m2\u001b[0m. **Data Sovereignty**: Users retain control over their data, as it does not leave their device. This means that \n",
"individuals or organizations can comply with local data protection regulations, such as the General\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"client = OpenAI(base_url=manager.endpoint, api_key=manager.api_key or 'not-needed')\n",
"prompt = 'List two benefits of local inference for privacy.'\n",
"resp = client.chat.completions.create(\n",
" model=model_id,\n",
" messages=[{'role':'user','content':prompt}],\n",
" max_tokens=120,\n",
" temperature=0.5\n",
")\n",
"print(resp.choices[0].message.content)"
]
},
{
"cell_type": "markdown",
"id": "f2091aa0",
"metadata": {},
"source": [
"### Explanation: Streaming Chat Completion\n",
"Demonstrates token streaming for improved perceived latency and interactive UX. The loop prints incremental deltas as they arrive:\n",
"- Useful for chat UIs where early partial output matters.\n",
"- Lets you measure token throughput vs. full completion latency.\n",
"\n",
"You can adapt this pattern to accumulate tokens, update a progress widget, or abort mid‑generation."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "5e85a2e8",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Edge
\n"
],
"text/plain": [
"Edge"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" AI
\n"
],
"text/plain": [
" AI"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" refers
\n"
],
"text/plain": [
" refers"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" to
\n"
],
"text/plain": [
" to"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" artificial
\n"
],
"text/plain": [
" artificial"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" intelligence
\n"
],
"text/plain": [
" intelligence"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" algorithms
\n"
],
"text/plain": [
" algorithms"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" and
\n"
],
"text/plain": [
" and"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" models
\n"
],
"text/plain": [
" models"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" that
\n"
],
"text/plain": [
" that"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" are
\n"
],
"text/plain": [
" are"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" deployed
\n"
],
"text/plain": [
" deployed"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" at
\n"
],
"text/plain": [
" at"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" the
\n"
],
"text/plain": [
" the"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" edge
\n"
],
"text/plain": [
" edge"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" of
\n"
],
"text/plain": [
" of"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" the
\n"
],
"text/plain": [
" the"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" network
\n"
],
"text/plain": [
" network"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
",
\n"
],
"text/plain": [
","
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" closer
\n"
],
"text/plain": [
" closer"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" to
\n"
],
"text/plain": [
" to"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" the
\n"
],
"text/plain": [
" the"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" source
\n"
],
"text/plain": [
" source"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" of
\n"
],
"text/plain": [
" of"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" data
\n"
],
"text/plain": [
" data"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
",
\n"
],
"text/plain": [
","
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" to
\n"
],
"text/plain": [
" to"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" enable
\n"
],
"text/plain": [
" enable"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" real
\n"
],
"text/plain": [
" real"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"-time
\n"
],
"text/plain": [
"-time"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" processing
\n"
],
"text/plain": [
" processing"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" and
\n"
],
"text/plain": [
" and"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" decision
\n"
],
"text/plain": [
" decision"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"-making
\n"
],
"text/plain": [
"-making"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" with
\n"
],
"text/plain": [
" with"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" reduced
\n"
],
"text/plain": [
" reduced"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" latency
\n"
],
"text/plain": [
" latency"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" and
\n"
],
"text/plain": [
" and"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" bandwidth
\n"
],
"text/plain": [
" bandwidth"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" usage
\n"
],
"text/plain": [
" usage"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
".
\n"
],
"text/plain": [
"."
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Streaming example\n",
"stream = client.chat.completions.create(\n",
" model=model_id,\n",
" messages=[{'role':'user','content':'Give a one-sentence definition of edge AI.'}],\n",
" stream=True,\n",
" max_tokens=60,\n",
" temperature=0.4\n",
")\n",
"for chunk in stream:\n",
" delta = chunk.choices[0].delta\n",
" if delta and delta.content:\n",
" print(delta.content, end='', flush=True)\n",
"print()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}