{ "cells": [ { "cell_type": "markdown", "id": "ffeb64bf", "metadata": {}, "source": [ "# Сесия 2 – Минимален RAG Пайплайн\n", "\n", "Създайте лек пайплайн за Генериране с Подобрено Извличане, използвайки Foundry Local + sentence-transformers embeddings.\n" ] }, { "cell_type": "markdown", "id": "77a46eea", "metadata": {}, "source": [ "### Обяснение: Инсталиране на зависимости\n", "Инсталира минимални пакети за този процес:\n", "- `foundry-local-sdk` за локално управление на модели (ако не се използва чист BASE_URL път).\n", "- `openai` за съвместими SDK структури (някои помощни функции).\n", "- `sentence-transformers` за вграждания.\n", "- `numpy` за векторна математика.\n", "Безопасно за повторно изпълнение; пропуснете, ако средата вече е удовлетворена.\n" ] }, { "cell_type": "markdown", "id": "a857bce7", "metadata": {}, "source": [ "# Сценарий\n", "Този ноутбук изгражда минимална локална тръбопроводна система за Възстановяване-Подпомогнато Генериране (RAG), която работи изцяло локално:\n", "- Свързва се с локален модел на Foundry (автоматично разпознаване чрез SDK или BASE_URL).\n", "- Създава малък корпус от документи в паметта и го вгражда с Sentence Transformers.\n", "- Реализира наивно извличане на векторна прилика (без външен индекс) за прозрачност.\n", "- Изпраща заявки за генериране, базирани на данни, чрез няколко резервни HTTP маршрута (`/v1/chat/completions`, `/v1/completions`, `/v1/responses`).\n", "- Осигурява помощна функция `answer()`, която опитва алтернативни форми на модела, когато първоначалните опити се провалят.\n", "\n", "Използвайте това като диагностичен шаблон преди мащабиране към по-големи корпуси, постоянни хранилища на вектори или метрики за оценка (вижте ноутбука за оценка на RAG).\n" ] }, { "cell_type": "code", "execution_count": 5, "id": "7b2c5757", "metadata": {}, "outputs": [], "source": [ "# Install dependencies\n", "!pip install -q foundry-local-sdk openai sentence-transformers numpy" ] }, { "cell_type": "markdown", "id": "9bed3131", "metadata": {}, "source": [ "### Обяснение: Основни импорти\n", "Зарежда основни библиотеки, необходими за вграждане + локално извеждане:\n", "- SentenceTransformer за плътни векторни вграждания.\n", "- FoundryLocalManager (по избор) за управление на локална услуга.\n", "- OpenAI клиент за познати обектни форми (въпреки че по-късно използваме HTTP директно).\n" ] }, { "cell_type": "code", "execution_count": 6, "id": "5cf135fe", "metadata": {}, "outputs": [], "source": [ "import os, numpy as np\n", "from sentence_transformers import SentenceTransformer\n", "from foundry_local import FoundryLocalManager\n", "from openai import OpenAI" ] }, { "cell_type": "markdown", "id": "725143b8", "metadata": {}, "source": [ "### Обяснение: Играчка Корпус от Документи\n", "Определя малък списък с изявления от домейна, съхраняван в паметта. Поддържа итерацията бърза и контролирана, така че фокусът да остане върху механиката на тръбопровода (извличане + обосновка), вместо върху обработката на данни.\n" ] }, { "cell_type": "code", "execution_count": 7, "id": "6f6d493c", "metadata": {}, "outputs": [], "source": [ "DOCS = [\n", " 'Foundry Local provides an OpenAI-compatible local inference endpoint.',\n", " 'Retrieval Augmented Generation improves answer grounding by injecting relevant context.',\n", " 'Edge AI reduces latency and preserves privacy via local execution.',\n", " 'Small Language Models can offer competitive quality with lower resource usage.',\n", " 'Vector similarity search retrieves semantically relevant documents.'\n", "]" ] }, { "cell_type": "markdown", "id": "ef0e66b3", "metadata": {}, "source": [ "### Обяснение: Свързване, Избор на Модел и Инициализация на Вграждане\n", "Логика за надеждно свързване:\n", "1. По избор използва явен `BASE_URL` (чист HTTP път), в противен случай се връща към FoundryLocalManager.\n", "2. Проверява `/v1/models` и избира най-добре съвпадащия конкретен идентификатор на модел (точен псевдоним > канонично семейство > първия наличен).\n", "3. Реализира цикъл за повторение с конфигурируеми `FOUNDRY_CONNECT_RETRIES` и забавяне.\n", "4. Инициализира SentenceTransformer вграждания (нормализирани вектори) за игрален корпус.\n", "5. Улавя версията на OpenAI SDK за възпроизводимост.\n", "Ако услугата липсва, отпечатва указания за стартиране вместо да се срине.\n" ] }, { "cell_type": "code", "execution_count": 12, "id": "dae7f5cd", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[OK] Foundry Local manager endpoint: http://127.0.0.1:59778/v1 | base=http://127.0.0.1:59778 | alias=phi-4-mini\n", "[OK] Model resolved: deepseek-r1-distill-qwen-7b-cuda-gpu:0 (total_models=11)\n", "[OK] Embedded 5 docs using sentence-transformers/all-MiniLM-L6-v2 shape=(5, 384)\n", "OpenAI SDK version: 1.109.1\n" ] } ], "source": [ "import os, time, json, requests, re\n", "# Native Foundry Local SDK preferred; fall back to explicit BASE_URL if provided\n", "os.environ.setdefault('FOUNDRY_LOCAL_ALIAS', 'phi-4-mini')\n", "alias = os.getenv('FOUNDRY_LOCAL_ALIAS', os.getenv('TARGET_MODEL', 'phi-4-mini'))\n", "base_url_env = os.getenv('BASE_URL', '').strip()\n", "manager = None\n", "client = None\n", "endpoint = None\n", "\n", "def _canonicalize(model_id: str) -> str:\n", " \"\"\"Remove CUDA suffix and version tags from model name.\"\"\"\n", " b = model_id.split(':')[0]\n", " return re.sub(r'-cuda.*', '', b)\n", "\n", "try:\n", " if base_url_env:\n", " # Allow user override; normalize by removing trailing / and optional /v1\n", " root = base_url_env.rstrip('/')\n", " if root.endswith('/v1'):\n", " root = root[:-3]\n", " endpoint = root\n", " print(f'[INFO] Using explicit BASE_URL override: {endpoint}')\n", " else:\n", " from foundry_local import FoundryLocalManager\n", " manager = FoundryLocalManager(alias)\n", " # Manager endpoint already includes /v1 - remove it for our base\n", " raw_endpoint = manager.endpoint.rstrip('/')\n", " if raw_endpoint.endswith('/v1'):\n", " endpoint = raw_endpoint[:-3]\n", " else:\n", " endpoint = raw_endpoint\n", " print(f'[OK] Foundry Local manager endpoint: {manager.endpoint} | base={endpoint} | alias={alias}')\n", " \n", " # Probe models list (endpoint does NOT include /v1 here)\n", " models_resp = requests.get(endpoint + '/v1/models', timeout=5)\n", " models_resp.raise_for_status()\n", " payload = models_resp.json() if models_resp.headers.get('content-type','').startswith('application/json') else {}\n", " data = payload.get('data', []) if isinstance(payload, dict) else []\n", " ids = [m.get('id') for m in data if isinstance(m, dict)]\n", " \n", " # Select best matching model\n", " chosen = None\n", " if alias in ids:\n", " chosen = alias\n", " else:\n", " for mid in ids:\n", " if _canonicalize(mid) == _canonicalize(alias):\n", " chosen = mid\n", " break\n", " if not chosen and ids:\n", " chosen = ids[0]\n", " model_name = chosen or alias\n", " \n", " # Initialize OpenAI client\n", " from openai import OpenAI as _OpenAI\n", " client = _OpenAI(\n", " base_url=endpoint + '/v1', # OpenAI client needs full base URL with /v1\n", " api_key=(getattr(manager, 'api_key', None) or os.getenv('API_KEY') or 'not-needed')\n", " )\n", " print(f'[OK] Model resolved: {model_name} (total_models={len(ids)})')\n", "except Exception as e:\n", " print('[ERROR] Failed to initialize Foundry Local client:', e)\n", " client = None\n", " model_name = alias\n", "\n", "# Expose BASE for downstream compatibility (without /v1)\n", "BASE = endpoint\n", "\n", "# Embeddings setup\n", "embed_model_name = os.getenv('EMBED_MODEL', 'sentence-transformers/all-MiniLM-L6-v2')\n", "try:\n", " from sentence_transformers import SentenceTransformer\n", " embedder = SentenceTransformer(embed_model_name)\n", " doc_emb = embedder.encode(DOCS, convert_to_numpy=True, normalize_embeddings=True)\n", " print(f'[OK] Embedded {len(DOCS)} docs using {embed_model_name} shape={doc_emb.shape}')\n", "except Exception as e:\n", " print('[ERROR] Embedding init failed:', e)\n", " embedder = None\n", " doc_emb = None\n", "\n", "try:\n", " import openai as _openai\n", " openai_version = getattr(_openai, '__version__', 'unknown')\n", " print('OpenAI SDK version:', openai_version)\n", "except Exception:\n", " openai_version = 'unknown'\n", "\n", "if client is None:\n", " print('\\nNEXT: Start/verify service then re-run this cell:')\n", " print(' foundry service start')\n", " print(' foundry model run phi-4-mini')\n", " print(' (optional) set BASE_URL=http://127.0.0.1:57127')" ] }, { "cell_type": "markdown", "id": "e6537337", "metadata": {}, "source": [ "### Обяснение: Функция за извличане (Векторна сходност)\n", "`retrieve(query, k=3)` кодира заявката, изчислява косинусната сходност (скаларно произведение на нормализирани вектори) и връща индексите на топ-k документи. Това остава минимално и в паметта за прозрачност.\n" ] }, { "cell_type": "code", "execution_count": 9, "id": "373d8845", "metadata": {}, "outputs": [], "source": [ "def retrieve(query, k=3):\n", " q = embedder.encode([query], convert_to_numpy=True, normalize_embeddings=True)[0]\n", " sims = doc_emb @ q\n", " return sims.argsort()[::-1][:k]" ] }, { "cell_type": "markdown", "id": "ea42bed1", "metadata": {}, "source": [ "### Обяснение: Генериране на основата на SDK и помощник за отговори\n", "Преработено за използване на Foundry Local SDK + методи на клиент, съвместими с OpenAI, вместо ръчни сурови HTTP заявки:\n", "- Основен път: `client.chat.completions.create` (структурирани съобщения).\n", "- Резервни варианти: `client.completions.create` (наследен prompt) и след това `client.responses.create` (оптимизиран API за отговори).\n", "- Нормализира алтернативни идентификатори на модели (RAW срещу опростен ALT), за да разшири съвместимостта.\n", "- `answer()` създава обоснован prompt от топ-k извлечени документи и записва подредени следи от опити.\n", "Това запазва логиката четима, като същевременно предлага плавна деградация при използване на развиващите се съвместими с OpenAI крайни точки.\n" ] }, { "cell_type": "code", "execution_count": 14, "id": "526983ad", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[INFO] SDK generation mode active.\n", " RAW_MODEL = deepseek-r1-distill-qwen-7b-cuda-gpu:0\n", " ALT_MODEL = deepseek-r1-distill-qwen-7b\n" ] } ], "source": [ "# SDK-based generation (Foundry Local manager + OpenAI client methods)\n", "import re, time, json\n", "\n", "def _strip_model_name(name: str) -> str:\n", " \"\"\"Strip CUDA suffix and version tags from model name.\"\"\"\n", " base = name.split(':')[0]\n", " base = re.sub(r'-cuda.*', '', base)\n", " return base\n", "\n", "# Use the actual resolved model name from connection cell\n", "RAW_MODEL = model_name\n", "ALT_MODEL = _strip_model_name(RAW_MODEL)\n", "\n", "def _try_via_client(messages, prompt, model_id: str, max_tokens=220, temperature=0.2):\n", " \"\"\"Try generating response using OpenAI client with multiple fallback routes.\"\"\"\n", " attempts = []\n", " \n", " # 1. Try chat.completions endpoint (preferred for chat models)\n", " try:\n", " resp = client.chat.completions.create(\n", " model=model_id, \n", " messages=messages, \n", " max_tokens=max_tokens, \n", " temperature=temperature\n", " )\n", " content = resp.choices[0].message.content\n", " attempts.append(('chat.completions', 200, (content or '')[:160]))\n", " if content and content.strip():\n", " return content, attempts\n", " except Exception as e:\n", " attempts.append(('chat.completions', None, str(e)[:160]))\n", " \n", " # 2. Try legacy completions endpoint\n", " try:\n", " comp = client.completions.create(\n", " model=model_id, \n", " prompt=prompt, \n", " max_tokens=max_tokens, \n", " temperature=temperature\n", " )\n", " txt = comp.choices[0].text if comp.choices else ''\n", " attempts.append(('completions', 200, (txt or '')[:160]))\n", " if txt and txt.strip():\n", " return txt, attempts\n", " except Exception as e:\n", " attempts.append(('completions', None, str(e)[:160]))\n", " \n", " return None, attempts\n", "\n", "def retrieve(query, k=3):\n", " \"\"\"Retrieve top-k most similar documents using cosine similarity.\"\"\"\n", " if embedder is None or doc_emb is None:\n", " raise RuntimeError(\"Embeddings not initialized.\")\n", " q_emb = embedder.encode([query], normalize_embeddings=True)[0]\n", " scores = doc_emb @ q_emb\n", " idxs = np.argsort(scores)[::-1][:k]\n", " return idxs\n", "\n", "def answer(query, k=3, max_tokens=220, temperature=0.2, try_alternate=True):\n", " \"\"\"\n", " Answer a query using RAG pipeline:\n", " 1. Retrieve relevant documents using vector similarity\n", " 2. Generate grounded response using Foundry Local model via OpenAI SDK\n", " \n", " Args:\n", " query: User question\n", " k: Number of documents to retrieve\n", " max_tokens: Maximum tokens for generation\n", " temperature: Sampling temperature\n", " try_alternate: Whether to try alternate model name on failure\n", " \n", " Returns:\n", " Dictionary with query, answer, docs, context, route, and tried attempts\n", " \"\"\"\n", " if client is None:\n", " raise RuntimeError('Model client not initialized. Re-run connection cell after starting Foundry Local.')\n", " if embedder is None or doc_emb is None:\n", " raise RuntimeError('Embeddings not initialized.')\n", " \n", " # Retrieve relevant documents\n", " idxs = retrieve(query, k=k)\n", " context = '\\n'.join(f'Doc {i}: {DOCS[i]}' for i in idxs)\n", " \n", " # Construct grounded generation prompt\n", " system_content = 'Use ONLY provided context. If insufficient, say \"I\\'m not sure.\"'\n", " user_content = f'Context:\\n{context}\\n\\nQuestion: {query}'\n", " messages = [\n", " {'role': 'system', 'content': system_content},\n", " {'role': 'user', 'content': user_content}\n", " ]\n", " prompt = f'System: {system_content}\\n{user_content}\\nAnswer:'\n", " \n", " # Try generation with primary model\n", " tried = []\n", " ans, attempts = _try_via_client(messages, prompt, RAW_MODEL, max_tokens=max_tokens, temperature=temperature)\n", " tried.append({'model': RAW_MODEL, 'attempts': attempts})\n", " \n", " if ans and ans.strip():\n", " return {\n", " 'query': query, \n", " 'answer': ans.strip(), \n", " 'docs': idxs.tolist(), \n", " 'context': context, \n", " 'route': 'chat-first', \n", " 'tried': tried\n", " }\n", " \n", " # Try alternate model name if available\n", " if try_alternate and ALT_MODEL != RAW_MODEL:\n", " ans2, attempts2 = _try_via_client(messages, prompt, ALT_MODEL, max_tokens=max_tokens, temperature=temperature)\n", " tried.append({'model': ALT_MODEL, 'attempts': attempts2})\n", " if ans2 and ans2.strip():\n", " return {\n", " 'query': query, \n", " 'answer': ans2.strip(), \n", " 'docs': idxs.tolist(), \n", " 'context': context, \n", " 'route': 'chat-alt', \n", " 'tried': tried\n", " }\n", " \n", " # All routes failed\n", " return {\n", " 'query': query, \n", " 'answer': 'I\\'m not sure. (All SDK routes failed)', \n", " 'docs': idxs.tolist(), \n", " 'context': context, \n", " 'route': 'failed', \n", " 'tried': tried\n", " }\n", "\n", "print('[INFO] SDK generation mode active.')\n", "print(f' RAW_MODEL = {RAW_MODEL}')\n", "print(f' ALT_MODEL = {ALT_MODEL}')" ] }, { "cell_type": "code", "execution_count": 15, "id": "b19a21cb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'alt_model': 'deepseek-r1-distill-qwen-7b',\n", " 'answer_preview': 'Okay, so I need to figure out why someone would use '\n", " 'Retrieval Augmented Generation (RAG) with local inference. '\n", " 'Let me start by understanding each part of the qu',\n", " 'base': 'http://127.0.0.1:59778',\n", " 'raw_model': 'deepseek-r1-distill-qwen-7b-cuda-gpu:0',\n", " 'retrieved_indices': [0, 3, 1],\n", " 'route': 'chat-first'}\n" ] } ], "source": [ "# Self-test cell: validates connectivity, embeddings, and answer() basic functionality (SDK mode)\n", "import math, pprint\n", "\n", "def rag_self_test(sample_query: str = 'Why use RAG with local inference?', expect_docs: int = 3):\n", " report = {'base': BASE, 'raw_model': RAW_MODEL, 'alt_model': ALT_MODEL}\n", " if not BASE:\n", " report['error'] = 'BASE not resolved'\n", " return report\n", " if embedder is None or doc_emb is None:\n", " report['error'] = 'Embeddings not initialized'\n", " return report\n", " if getattr(doc_emb, 'shape', (0,))[0] != len(DOCS):\n", " report['warning_embeddings'] = f\"doc_emb count {getattr(doc_emb,'shape',('?'))} mismatch DOCS {len(DOCS)}\"\n", " try:\n", " idxs = retrieve(sample_query, k=expect_docs)\n", " report['retrieved_indices'] = idxs.tolist() if hasattr(idxs, 'tolist') else list(idxs)\n", " except Exception as e:\n", " report['error_retrieve'] = str(e)\n", " return report\n", " try:\n", " ans = answer(sample_query, k=expect_docs, max_tokens=80, temperature=0.2)\n", " report['route'] = ans.get('route')\n", " report['answer_preview'] = ans.get('answer','')[:160]\n", " if ans.get('route') == 'failed':\n", " report['warning_generation'] = 'All SDK routes failed for sample query'\n", " except Exception as e:\n", " report['error_generation'] = str(e)\n", " return report\n", "\n", "pprint.pprint(rag_self_test())" ] }, { "cell_type": "markdown", "id": "1ccaa474", "metadata": {}, "source": [ "### Обяснение: Тест за проверка на групови заявки\n", "Изпълнява няколко представителни потребителски въпроси чрез `answer()`, за да валидира:\n", "- Индексите за извличане съответстват на правдоподобни подпомагащи документи.\n", "- Резервното маршрутизиране работи (стойността на маршрута не е 'failed').\n", "- Отговорите спазват инструкциите за основа (без халюцинации).\n", "Запазва последния обект с резултати за допълнителна проверка.\n" ] }, { "cell_type": "code", "execution_count": 16, "id": "c38a64fb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Q: Why use RAG with local inference?\n", "A: Okay, so I need to figure out why someone would use Retrieval Augmented Generation (RAG) with local inference. Let me start by understanding each part of the question.\n", "\n", "First, RAG. From the context given, Doc 1 says that RAG improves answer grounding by injecting relevant context. So RAG is a method that uses retrieval techniques to find the most relevant parts of a document or corpus to augment the generation process. This probably helps in making the generated answers more accurate because they're backed by real data.\n", "\n", "Then, local inference. Doc 0 mentions that Foundry Local provides an OpenAI-compatible local inference endpoint. So local inference means running the model on the user's device rather than sending the request to a remote server. This is good for privacy and reducing latency, but it might have limitations in terms of model size or capabilities compared to cloud-based options.\n", "\n", "Now, combining RAG with local inference. The context says that small language models can offer competitive quality with lower resource usage (Doc 3). So using a smaller model might be more efficient, but maybe RAG\n", "Docs: [0, 3, 1]\n", "---\n", "Q: What does vector similarity search do?\n", "A: Okay, so I need to figure out what vector similarity search does based on the provided context. Let me start by looking at the given documents.\n", "\n", "Doc 4 says, \"Vector similarity search retrieves semantically relevant documents.\" Hmm, that seems straightforward. So vector similarity search is a method that retrieves documents based on their semantic relevance. It probably uses some kind of vector representation for the documents, maybe through techniques like word embeddings or TF-IDF vectors. Then, when a query comes in, it's converted into a vector, and the system finds documents with similar vectors, meaning they share similar meanings or topics.\n", "\n", "I should make sure I'm not missing anything. The other documents mention edge AI and retrieval augmented generation, but they don't directly relate to vector similarity search. So, the key point here is that it retrieves semantically relevant documents. That makes sense because vector-based methods are commonly used in information retrieval to find documents that are not just syntactically similar but actually about the same topic.\n", "\n", "I think that's the main idea. Vector similarity search focuses on semantic relevance, using vector representations to\n", "Docs: [4, 2, 1]\n", "---\n", "Q: Explain privacy benefits.\n", "A: Okay, so I need to explain the privacy benefits mentioned in the provided context. Let me look at the context again. The context includes three documents:\n", "\n", "Doc 2 says Edge AI reduces latency and preserves privacy via local execution.\n", "Doc 3 mentions Small Language Models can offer competitive quality with lower resource usage.\n", "Doc 1 states Retrieval Augmented Generation improves answer grounding by injecting relevant context.\n", "\n", "The question is about explaining the privacy benefits. So, I should focus on the parts of the context that talk about privacy. \n", "\n", "Looking at Doc 2, it mentions Edge AI reduces latency and preserves privacy via local execution. That seems directly related to privacy. I think \"local execution\" means that the AI processes data on the device itself rather than sending it to a server. This could mean that data doesn't have to be transmitted, which might help protect user privacy because it avoids centralizing data that could be intercepted.\n", "\n", "Doc 3 talks about small language models with lower resource usage. I'm not sure how this relates to privacy. It might be more about efficiency rather than privacy benefits.\n", "\n", "Doc 1\n", "Docs: [2, 3, 1]\n", "---\n" ] }, { "data": { "text/plain": [ "{'query': 'Explain privacy benefits.',\n", " 'answer': 'Okay, so I need to explain the privacy benefits mentioned in the provided context. Let me look at the context again. The context includes three documents:\\n\\nDoc 2 says Edge AI reduces latency and preserves privacy via local execution.\\nDoc 3 mentions Small Language Models can offer competitive quality with lower resource usage.\\nDoc 1 states Retrieval Augmented Generation improves answer grounding by injecting relevant context.\\n\\nThe question is about explaining the privacy benefits. So, I should focus on the parts of the context that talk about privacy. \\n\\nLooking at Doc 2, it mentions Edge AI reduces latency and preserves privacy via local execution. That seems directly related to privacy. I think \"local execution\" means that the AI processes data on the device itself rather than sending it to a server. This could mean that data doesn\\'t have to be transmitted, which might help protect user privacy because it avoids centralizing data that could be intercepted.\\n\\nDoc 3 talks about small language models with lower resource usage. I\\'m not sure how this relates to privacy. It might be more about efficiency rather than privacy benefits.\\n\\nDoc 1',\n", " 'docs': [2, 3, 1],\n", " 'context': 'Doc 2: Edge AI reduces latency and preserves privacy via local execution.\\nDoc 3: Small Language Models can offer competitive quality with lower resource usage.\\nDoc 1: Retrieval Augmented Generation improves answer grounding by injecting relevant context.',\n", " 'route': 'chat-first',\n", " 'tried': [{'model': 'deepseek-r1-distill-qwen-7b-cuda-gpu:0',\n", " 'attempts': [('chat.completions',\n", " 200,\n", " 'Okay, so I need to explain the privacy benefits mentioned in the provided context. Let me look at the context again. The context includes three documents:\\n\\nDoc ')]}]}" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Quick test queries\n", "\n", "queries = [\n", "\n", " \"Why use RAG with local inference?\",\n", "\n", " \"What does vector similarity search do?\",\n", "\n", " \"Explain privacy benefits.\"\n", "\n", "]\n", "\n", "\n", "\n", "last_result = None\n", "\n", "for q in queries:\n", "\n", " try:\n", "\n", " r = answer(q)\n", "\n", " last_result = r\n", "\n", " print(f\"Q: {q}\\nA: {r['answer']}\\nDocs: {r['docs']}\\n---\")\n", "\n", " except Exception as e:\n", "\n", " print(f\"Failed answering '{q}': {e}\")\n", "\n", "\n", "\n", "last_result" ] }, { "cell_type": "markdown", "id": "0f8282fc", "metadata": {}, "source": [ "### Обяснение: Удобно извикване за единичен отговор\n", "Последно бързо извикване за единичен въпрос, предназначено за лесно копиране/поставяне или последващо препращане. Демонстрира идемпотентно използване на `answer()` след предварителни подготвителни заявки.\n" ] }, { "cell_type": "code", "execution_count": 17, "id": "d4d25165", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'query': 'Why use RAG with local inference?',\n", " 'answer': \"Okay, so I need to figure out why someone would use Retrieval Augmented Generation (RAG) with local inference. Let me start by understanding each part of the question.\\n\\nFirst, RAG. From the context given, Doc 1 says that RAG improves answer grounding by injecting relevant context. So RAG is a method that uses retrieval techniques to find the most relevant parts of a document or corpus to augment the generation process. This probably helps in making the generated answers more accurate because they're backed by real data.\\n\\nThen, local inference. Doc 0 mentions that Foundry Local provides an OpenAI-compatible local inference endpoint. So local inference means running the model on the user's device rather than sending the request to a remote server. This is good for privacy and reducing latency, but it might have limitations in terms of model size or capabilities compared to cloud-based options.\\n\\nNow, combining RAG with local inference. The context says that small language models can offer competitive quality with lower resource usage (Doc 3). So using a smaller model might be more efficient, but maybe RAG\",\n", " 'docs': [0, 3, 1],\n", " 'context': 'Doc 0: Foundry Local provides an OpenAI-compatible local inference endpoint.\\nDoc 3: Small Language Models can offer competitive quality with lower resource usage.\\nDoc 1: Retrieval Augmented Generation improves answer grounding by injecting relevant context.',\n", " 'route': 'chat-first',\n", " 'tried': [{'model': 'deepseek-r1-distill-qwen-7b-cuda-gpu:0',\n", " 'attempts': [('chat.completions',\n", " 200,\n", " 'Okay, so I need to figure out why someone would use Retrieval Augmented Generation (RAG) with local inference. Let me start by understanding each part of the qu')]}]}" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "result = answer('Why use RAG with local inference?')\n", "result" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n---\n\n**Отказ от отговорност**: \nТози документ е преведен с помощта на AI услуга за превод [Co-op Translator](https://github.com/Azure/co-op-translator). Въпреки че се стремим към точност, моля, имайте предвид, че автоматизираните преводи може да съдържат грешки или неточности. Оригиналният документ на неговия роден език трябва да се счита за авторитетен източник. За критична информация се препоръчва професионален човешки превод. Ние не носим отговорност за каквито и да било недоразумения или погрешни интерпретации, произтичащи от използването на този превод.\n" ] } ], "metadata": { "kernelspec": { "display_name": "demo", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.15" }, "coopTranslator": { "original_hash": "9f009936a177f8847d250fe1771d25f6", "translation_date": "2025-10-08T14:34:39+00:00", "source_file": "Workshop/notebooks/session02_rag_pipeline.ipynb", "language_code": "bg" } }, "nbformat": 4, "nbformat_minor": 5 }