{ "cells": [ { "cell_type": "markdown", "id": "ffeb64bf", "metadata": {}, "source": [ "# Session 2 – Minimal RAG Pipeline\n", "\n", "Build a lightweight Retrieval-Augmented Generation pipeline using Foundry Local + sentence-transformers embeddings." ] }, { "cell_type": "markdown", "id": "77a46eea", "metadata": {}, "source": [ "### Explanation: Dependency Installation\n", "Installs minimal packages for this pipeline:\n", "- `foundry-local-sdk` for local model management (if not using pure BASE_URL path).\n", "- `openai` for compatible SDK structures (some utilities).\n", "- `sentence-transformers` for embeddings.\n", "- `numpy` for vector math.\n", "Safe to re-run; skip if environment already satisfied." ] }, { "cell_type": "markdown", "id": "a857bce7", "metadata": {}, "source": [ "# Scenario\n", "This notebook builds a minimal Retrieval-Augmented Generation (RAG) pipeline running entirely locally:\n", "- Connects to a Foundry Local model (auto-detects via SDK or BASE_URL).\n", "- Creates a tiny in-memory document corpus and embeds it with Sentence Transformers.\n", "- Implements naive vector similarity retrieval (no external index) for transparency.\n", "- Issues grounded generation requests via multiple HTTP fallback routes (`/v1/chat/completions`, `/v1/completions`, `/v1/responses`).\n", "- Provides an `answer()` helper that retries alternate model forms when initial attempts fail.\n", "\n", "Use this as a diagnostic template before scaling to larger corpora, persistent vector stores, or evaluation metrics (see RAG evaluation notebook)." ] }, { "cell_type": "code", "execution_count": 5, "id": "7b2c5757", "metadata": {}, "outputs": [], "source": [ "# Install dependencies\n", "!pip install -q foundry-local-sdk openai sentence-transformers numpy" ] }, { "cell_type": "markdown", "id": "9bed3131", "metadata": {}, "source": [ "### Explanation: Core Imports\n", "Loads core libraries needed for embedding + local inference:\n", "- SentenceTransformer for dense vector embeddings.\n", "- FoundryLocalManager (optional) to manage local service.\n", "- OpenAI client for familiar object shapes (even though we later hit HTTP directly)." ] }, { "cell_type": "code", "execution_count": 6, "id": "5cf135fe", "metadata": {}, "outputs": [], "source": [ "import os, numpy as np\n", "from sentence_transformers import SentenceTransformer\n", "from foundry_local import FoundryLocalManager\n", "from openai import OpenAI" ] }, { "cell_type": "markdown", "id": "725143b8", "metadata": {}, "source": [ "### Explanation: Toy Document Corpus\n", "Defines a small in-memory list of domain statements. Keeps iteration fast and controlled so focus stays on pipeline mechanics (retrieval + grounding) rather than data wrangling." ] }, { "cell_type": "code", "execution_count": 7, "id": "6f6d493c", "metadata": {}, "outputs": [], "source": [ "DOCS = [\n", " 'Foundry Local provides an OpenAI-compatible local inference endpoint.',\n", " 'Retrieval Augmented Generation improves answer grounding by injecting relevant context.',\n", " 'Edge AI reduces latency and preserves privacy via local execution.',\n", " 'Small Language Models can offer competitive quality with lower resource usage.',\n", " 'Vector similarity search retrieves semantically relevant documents.'\n", "]" ] }, { "cell_type": "markdown", "id": "ef0e66b3", "metadata": {}, "source": [ "### Explanation: Connection, Model Selection & Embedding Init\n", "Robust connection logic:\n", "1. Optionally uses explicit `BASE_URL` (pure HTTP path) else falls back to FoundryLocalManager.\n", "2. Probes `/v1/models` and selects best matching concrete model id (exact alias > canonical family > first available).\n", "3. Implements retry loop with configurable `FOUNDRY_CONNECT_RETRIES` & delay.\n", "4. Initializes SentenceTransformer embeddings (normalized vectors) for the toy corpus.\n", "5. Captures OpenAI SDK version for reproducibility.\n", "If service is absent, prints guidance to start it instead of crashing." ] }, { "cell_type": "code", "execution_count": 12, "id": "dae7f5cd", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[OK] Foundry Local manager endpoint: http://127.0.0.1:59778/v1 | base=http://127.0.0.1:59778 | alias=phi-4-mini\n", "[OK] Model resolved: deepseek-r1-distill-qwen-7b-cuda-gpu:0 (total_models=11)\n", "[OK] Embedded 5 docs using sentence-transformers/all-MiniLM-L6-v2 shape=(5, 384)\n", "OpenAI SDK version: 1.109.1\n" ] } ], "source": [ "import os, time, json, requests, re\n", "# Native Foundry Local SDK preferred; fall back to explicit BASE_URL if provided\n", "os.environ.setdefault('FOUNDRY_LOCAL_ALIAS', 'phi-4-mini')\n", "alias = os.getenv('FOUNDRY_LOCAL_ALIAS', os.getenv('TARGET_MODEL', 'phi-4-mini'))\n", "base_url_env = os.getenv('BASE_URL', '').strip()\n", "manager = None\n", "client = None\n", "endpoint = None\n", "\n", "def _canonicalize(model_id: str) -> str:\n", " \"\"\"Remove CUDA suffix and version tags from model name.\"\"\"\n", " b = model_id.split(':')[0]\n", " return re.sub(r'-cuda.*', '', b)\n", "\n", "try:\n", " if base_url_env:\n", " # Allow user override; normalize by removing trailing / and optional /v1\n", " root = base_url_env.rstrip('/')\n", " if root.endswith('/v1'):\n", " root = root[:-3]\n", " endpoint = root\n", " print(f'[INFO] Using explicit BASE_URL override: {endpoint}')\n", " else:\n", " from foundry_local import FoundryLocalManager\n", " manager = FoundryLocalManager(alias)\n", " # Manager endpoint already includes /v1 - remove it for our base\n", " raw_endpoint = manager.endpoint.rstrip('/')\n", " if raw_endpoint.endswith('/v1'):\n", " endpoint = raw_endpoint[:-3]\n", " else:\n", " endpoint = raw_endpoint\n", " print(f'[OK] Foundry Local manager endpoint: {manager.endpoint} | base={endpoint} | alias={alias}')\n", " \n", " # Probe models list (endpoint does NOT include /v1 here)\n", " models_resp = requests.get(endpoint + '/v1/models', timeout=5)\n", " models_resp.raise_for_status()\n", " payload = models_resp.json() if models_resp.headers.get('content-type','').startswith('application/json') else {}\n", " data = payload.get('data', []) if isinstance(payload, dict) else []\n", " ids = [m.get('id') for m in data if isinstance(m, dict)]\n", " \n", " # Select best matching model\n", " chosen = None\n", " if alias in ids:\n", " chosen = alias\n", " else:\n", " for mid in ids:\n", " if _canonicalize(mid) == _canonicalize(alias):\n", " chosen = mid\n", " break\n", " if not chosen and ids:\n", " chosen = ids[0]\n", " model_name = chosen or alias\n", " \n", " # Initialize OpenAI client\n", " from openai import OpenAI as _OpenAI\n", " client = _OpenAI(\n", " base_url=endpoint + '/v1', # OpenAI client needs full base URL with /v1\n", " api_key=(getattr(manager, 'api_key', None) or os.getenv('API_KEY') or 'not-needed')\n", " )\n", " print(f'[OK] Model resolved: {model_name} (total_models={len(ids)})')\n", "except Exception as e:\n", " print('[ERROR] Failed to initialize Foundry Local client:', e)\n", " client = None\n", " model_name = alias\n", "\n", "# Expose BASE for downstream compatibility (without /v1)\n", "BASE = endpoint\n", "\n", "# Embeddings setup\n", "embed_model_name = os.getenv('EMBED_MODEL', 'sentence-transformers/all-MiniLM-L6-v2')\n", "try:\n", " from sentence_transformers import SentenceTransformer\n", " embedder = SentenceTransformer(embed_model_name)\n", " doc_emb = embedder.encode(DOCS, convert_to_numpy=True, normalize_embeddings=True)\n", " print(f'[OK] Embedded {len(DOCS)} docs using {embed_model_name} shape={doc_emb.shape}')\n", "except Exception as e:\n", " print('[ERROR] Embedding init failed:', e)\n", " embedder = None\n", " doc_emb = None\n", "\n", "try:\n", " import openai as _openai\n", " openai_version = getattr(_openai, '__version__', 'unknown')\n", " print('OpenAI SDK version:', openai_version)\n", "except Exception:\n", " openai_version = 'unknown'\n", "\n", "if client is None:\n", " print('\\nNEXT: Start/verify service then re-run this cell:')\n", " print(' foundry service start')\n", " print(' foundry model run phi-4-mini')\n", " print(' (optional) set BASE_URL=http://127.0.0.1:57127')" ] }, { "cell_type": "markdown", "id": "e6537337", "metadata": {}, "source": [ "### Explanation: Retrieve Function (Vector Similarity)\n", "`retrieve(query, k=3)` encodes the query, computes cosine similarity (dot product on normalized vectors), and returns top-k doc indices. This stays minimal & in-memory for transparency." ] }, { "cell_type": "code", "execution_count": 9, "id": "373d8845", "metadata": {}, "outputs": [], "source": [ "def retrieve(query, k=3):\n", " q = embedder.encode([query], convert_to_numpy=True, normalize_embeddings=True)[0]\n", " sims = doc_emb @ q\n", " return sims.argsort()[::-1][:k]" ] }, { "cell_type": "markdown", "id": "ea42bed1", "metadata": {}, "source": [ "### Explanation: SDK-Based Generation & Answer Helper\n", "Reworked to use the Foundry Local SDK + OpenAI-compatible client methods instead of manual raw HTTP posts:\n", "- Primary path: `client.chat.completions.create` (structured messages).\n", "- Fallbacks: `client.completions.create` (legacy prompt) then `client.responses.create` (streamlined responses API).\n", "- Normalizes alternate model ids (RAW vs stripped ALT) to widen compatibility.\n", "- `answer()` constructs a grounded prompt from top-k retrieved docs and records ordered attempt traces.\n", "This keeps logic readable while still offering graceful degradation across evolving OpenAI-compatible endpoints." ] }, { "cell_type": "code", "execution_count": 14, "id": "526983ad", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[INFO] SDK generation mode active.\n", " RAW_MODEL = deepseek-r1-distill-qwen-7b-cuda-gpu:0\n", " ALT_MODEL = deepseek-r1-distill-qwen-7b\n" ] } ], "source": [ "# SDK-based generation (Foundry Local manager + OpenAI client methods)\n", "import re, time, json\n", "\n", "def _strip_model_name(name: str) -> str:\n", " \"\"\"Strip CUDA suffix and version tags from model name.\"\"\"\n", " base = name.split(':')[0]\n", " base = re.sub(r'-cuda.*', '', base)\n", " return base\n", "\n", "# Use the actual resolved model name from connection cell\n", "RAW_MODEL = model_name\n", "ALT_MODEL = _strip_model_name(RAW_MODEL)\n", "\n", "def _try_via_client(messages, prompt, model_id: str, max_tokens=220, temperature=0.2):\n", " \"\"\"Try generating response using OpenAI client with multiple fallback routes.\"\"\"\n", " attempts = []\n", " \n", " # 1. Try chat.completions endpoint (preferred for chat models)\n", " try:\n", " resp = client.chat.completions.create(\n", " model=model_id, \n", " messages=messages, \n", " max_tokens=max_tokens, \n", " temperature=temperature\n", " )\n", " content = resp.choices[0].message.content\n", " attempts.append(('chat.completions', 200, (content or '')[:160]))\n", " if content and content.strip():\n", " return content, attempts\n", " except Exception as e:\n", " attempts.append(('chat.completions', None, str(e)[:160]))\n", " \n", " # 2. Try legacy completions endpoint\n", " try:\n", " comp = client.completions.create(\n", " model=model_id, \n", " prompt=prompt, \n", " max_tokens=max_tokens, \n", " temperature=temperature\n", " )\n", " txt = comp.choices[0].text if comp.choices else ''\n", " attempts.append(('completions', 200, (txt or '')[:160]))\n", " if txt and txt.strip():\n", " return txt, attempts\n", " except Exception as e:\n", " attempts.append(('completions', None, str(e)[:160]))\n", " \n", " return None, attempts\n", "\n", "def retrieve(query, k=3):\n", " \"\"\"Retrieve top-k most similar documents using cosine similarity.\"\"\"\n", " if embedder is None or doc_emb is None:\n", " raise RuntimeError(\"Embeddings not initialized.\")\n", " q_emb = embedder.encode([query], normalize_embeddings=True)[0]\n", " scores = doc_emb @ q_emb\n", " idxs = np.argsort(scores)[::-1][:k]\n", " return idxs\n", "\n", "def answer(query, k=3, max_tokens=220, temperature=0.2, try_alternate=True):\n", " \"\"\"\n", " Answer a query using RAG pipeline:\n", " 1. Retrieve relevant documents using vector similarity\n", " 2. Generate grounded response using Foundry Local model via OpenAI SDK\n", " \n", " Args:\n", " query: User question\n", " k: Number of documents to retrieve\n", " max_tokens: Maximum tokens for generation\n", " temperature: Sampling temperature\n", " try_alternate: Whether to try alternate model name on failure\n", " \n", " Returns:\n", " Dictionary with query, answer, docs, context, route, and tried attempts\n", " \"\"\"\n", " if client is None:\n", " raise RuntimeError('Model client not initialized. Re-run connection cell after starting Foundry Local.')\n", " if embedder is None or doc_emb is None:\n", " raise RuntimeError('Embeddings not initialized.')\n", " \n", " # Retrieve relevant documents\n", " idxs = retrieve(query, k=k)\n", " context = '\\n'.join(f'Doc {i}: {DOCS[i]}' for i in idxs)\n", " \n", " # Construct grounded generation prompt\n", " system_content = 'Use ONLY provided context. If insufficient, say \"I\\'m not sure.\"'\n", " user_content = f'Context:\\n{context}\\n\\nQuestion: {query}'\n", " messages = [\n", " {'role': 'system', 'content': system_content},\n", " {'role': 'user', 'content': user_content}\n", " ]\n", " prompt = f'System: {system_content}\\n{user_content}\\nAnswer:'\n", " \n", " # Try generation with primary model\n", " tried = []\n", " ans, attempts = _try_via_client(messages, prompt, RAW_MODEL, max_tokens=max_tokens, temperature=temperature)\n", " tried.append({'model': RAW_MODEL, 'attempts': attempts})\n", " \n", " if ans and ans.strip():\n", " return {\n", " 'query': query, \n", " 'answer': ans.strip(), \n", " 'docs': idxs.tolist(), \n", " 'context': context, \n", " 'route': 'chat-first', \n", " 'tried': tried\n", " }\n", " \n", " # Try alternate model name if available\n", " if try_alternate and ALT_MODEL != RAW_MODEL:\n", " ans2, attempts2 = _try_via_client(messages, prompt, ALT_MODEL, max_tokens=max_tokens, temperature=temperature)\n", " tried.append({'model': ALT_MODEL, 'attempts': attempts2})\n", " if ans2 and ans2.strip():\n", " return {\n", " 'query': query, \n", " 'answer': ans2.strip(), \n", " 'docs': idxs.tolist(), \n", " 'context': context, \n", " 'route': 'chat-alt', \n", " 'tried': tried\n", " }\n", " \n", " # All routes failed\n", " return {\n", " 'query': query, \n", " 'answer': 'I\\'m not sure. (All SDK routes failed)', \n", " 'docs': idxs.tolist(), \n", " 'context': context, \n", " 'route': 'failed', \n", " 'tried': tried\n", " }\n", "\n", "print('[INFO] SDK generation mode active.')\n", "print(f' RAW_MODEL = {RAW_MODEL}')\n", "print(f' ALT_MODEL = {ALT_MODEL}')" ] }, { "cell_type": "code", "execution_count": 15, "id": "b19a21cb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'alt_model': 'deepseek-r1-distill-qwen-7b',\n", " 'answer_preview': 'Okay, so I need to figure out why someone would use '\n", " 'Retrieval Augmented Generation (RAG) with local inference. '\n", " 'Let me start by understanding each part of the qu',\n", " 'base': 'http://127.0.0.1:59778',\n", " 'raw_model': 'deepseek-r1-distill-qwen-7b-cuda-gpu:0',\n", " 'retrieved_indices': [0, 3, 1],\n", " 'route': 'chat-first'}\n" ] } ], "source": [ "# Self-test cell: validates connectivity, embeddings, and answer() basic functionality (SDK mode)\n", "import math, pprint\n", "\n", "def rag_self_test(sample_query: str = 'Why use RAG with local inference?', expect_docs: int = 3):\n", " report = {'base': BASE, 'raw_model': RAW_MODEL, 'alt_model': ALT_MODEL}\n", " if not BASE:\n", " report['error'] = 'BASE not resolved'\n", " return report\n", " if embedder is None or doc_emb is None:\n", " report['error'] = 'Embeddings not initialized'\n", " return report\n", " if getattr(doc_emb, 'shape', (0,))[0] != len(DOCS):\n", " report['warning_embeddings'] = f\"doc_emb count {getattr(doc_emb,'shape',('?'))} mismatch DOCS {len(DOCS)}\"\n", " try:\n", " idxs = retrieve(sample_query, k=expect_docs)\n", " report['retrieved_indices'] = idxs.tolist() if hasattr(idxs, 'tolist') else list(idxs)\n", " except Exception as e:\n", " report['error_retrieve'] = str(e)\n", " return report\n", " try:\n", " ans = answer(sample_query, k=expect_docs, max_tokens=80, temperature=0.2)\n", " report['route'] = ans.get('route')\n", " report['answer_preview'] = ans.get('answer','')[:160]\n", " if ans.get('route') == 'failed':\n", " report['warning_generation'] = 'All SDK routes failed for sample query'\n", " except Exception as e:\n", " report['error_generation'] = str(e)\n", " return report\n", "\n", "pprint.pprint(rag_self_test())" ] }, { "cell_type": "markdown", "id": "1ccaa474", "metadata": {}, "source": [ "### Explanation: Batch Query Smoke Test\n", "Executes several representative user questions through `answer()` to validate:\n", "- Retrieval indices correspond to plausible supporting docs.\n", "- Fallback routing works (route value not 'failed').\n", "- Answers honor the grounding instruction (no hallucinations).\n", "Captures the last result object for ad‑hoc inspection." ] }, { "cell_type": "code", "execution_count": 16, "id": "c38a64fb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Q: Why use RAG with local inference?\n", "A: Okay, so I need to figure out why someone would use Retrieval Augmented Generation (RAG) with local inference. Let me start by understanding each part of the question.\n", "\n", "First, RAG. From the context given, Doc 1 says that RAG improves answer grounding by injecting relevant context. So RAG is a method that uses retrieval techniques to find the most relevant parts of a document or corpus to augment the generation process. This probably helps in making the generated answers more accurate because they're backed by real data.\n", "\n", "Then, local inference. Doc 0 mentions that Foundry Local provides an OpenAI-compatible local inference endpoint. So local inference means running the model on the user's device rather than sending the request to a remote server. This is good for privacy and reducing latency, but it might have limitations in terms of model size or capabilities compared to cloud-based options.\n", "\n", "Now, combining RAG with local inference. The context says that small language models can offer competitive quality with lower resource usage (Doc 3). So using a smaller model might be more efficient, but maybe RAG\n", "Docs: [0, 3, 1]\n", "---\n", "Q: What does vector similarity search do?\n", "A: Okay, so I need to figure out what vector similarity search does based on the provided context. Let me start by looking at the given documents.\n", "\n", "Doc 4 says, \"Vector similarity search retrieves semantically relevant documents.\" Hmm, that seems straightforward. So vector similarity search is a method that retrieves documents based on their semantic relevance. It probably uses some kind of vector representation for the documents, maybe through techniques like word embeddings or TF-IDF vectors. Then, when a query comes in, it's converted into a vector, and the system finds documents with similar vectors, meaning they share similar meanings or topics.\n", "\n", "I should make sure I'm not missing anything. The other documents mention edge AI and retrieval augmented generation, but they don't directly relate to vector similarity search. So, the key point here is that it retrieves semantically relevant documents. That makes sense because vector-based methods are commonly used in information retrieval to find documents that are not just syntactically similar but actually about the same topic.\n", "\n", "I think that's the main idea. Vector similarity search focuses on semantic relevance, using vector representations to\n", "Docs: [4, 2, 1]\n", "---\n", "Q: Explain privacy benefits.\n", "A: Okay, so I need to explain the privacy benefits mentioned in the provided context. Let me look at the context again. The context includes three documents:\n", "\n", "Doc 2 says Edge AI reduces latency and preserves privacy via local execution.\n", "Doc 3 mentions Small Language Models can offer competitive quality with lower resource usage.\n", "Doc 1 states Retrieval Augmented Generation improves answer grounding by injecting relevant context.\n", "\n", "The question is about explaining the privacy benefits. So, I should focus on the parts of the context that talk about privacy. \n", "\n", "Looking at Doc 2, it mentions Edge AI reduces latency and preserves privacy via local execution. That seems directly related to privacy. I think \"local execution\" means that the AI processes data on the device itself rather than sending it to a server. This could mean that data doesn't have to be transmitted, which might help protect user privacy because it avoids centralizing data that could be intercepted.\n", "\n", "Doc 3 talks about small language models with lower resource usage. I'm not sure how this relates to privacy. It might be more about efficiency rather than privacy benefits.\n", "\n", "Doc 1\n", "Docs: [2, 3, 1]\n", "---\n" ] }, { "data": { "text/plain": [ "{'query': 'Explain privacy benefits.',\n", " 'answer': 'Okay, so I need to explain the privacy benefits mentioned in the provided context. Let me look at the context again. The context includes three documents:\\n\\nDoc 2 says Edge AI reduces latency and preserves privacy via local execution.\\nDoc 3 mentions Small Language Models can offer competitive quality with lower resource usage.\\nDoc 1 states Retrieval Augmented Generation improves answer grounding by injecting relevant context.\\n\\nThe question is about explaining the privacy benefits. So, I should focus on the parts of the context that talk about privacy. \\n\\nLooking at Doc 2, it mentions Edge AI reduces latency and preserves privacy via local execution. That seems directly related to privacy. I think \"local execution\" means that the AI processes data on the device itself rather than sending it to a server. This could mean that data doesn\\'t have to be transmitted, which might help protect user privacy because it avoids centralizing data that could be intercepted.\\n\\nDoc 3 talks about small language models with lower resource usage. I\\'m not sure how this relates to privacy. It might be more about efficiency rather than privacy benefits.\\n\\nDoc 1',\n", " 'docs': [2, 3, 1],\n", " 'context': 'Doc 2: Edge AI reduces latency and preserves privacy via local execution.\\nDoc 3: Small Language Models can offer competitive quality with lower resource usage.\\nDoc 1: Retrieval Augmented Generation improves answer grounding by injecting relevant context.',\n", " 'route': 'chat-first',\n", " 'tried': [{'model': 'deepseek-r1-distill-qwen-7b-cuda-gpu:0',\n", " 'attempts': [('chat.completions',\n", " 200,\n", " 'Okay, so I need to explain the privacy benefits mentioned in the provided context. Let me look at the context again. The context includes three documents:\\n\\nDoc ')]}]}" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Quick test queries\n", "\n", "queries = [\n", "\n", " \"Why use RAG with local inference?\",\n", "\n", " \"What does vector similarity search do?\",\n", "\n", " \"Explain privacy benefits.\"\n", "\n", "]\n", "\n", "\n", "\n", "last_result = None\n", "\n", "for q in queries:\n", "\n", " try:\n", "\n", " r = answer(q)\n", "\n", " last_result = r\n", "\n", " print(f\"Q: {q}\\nA: {r['answer']}\\nDocs: {r['docs']}\\n---\")\n", "\n", " except Exception as e:\n", "\n", " print(f\"Failed answering '{q}': {e}\")\n", "\n", "\n", "\n", "last_result" ] }, { "cell_type": "markdown", "id": "0f8282fc", "metadata": {}, "source": [ "### Explanation: Single Answer Convenience Call\n", "Final quick single-question call for easy copy/paste reuse or downstream referencing. Demonstrates idempotent use of `answer()` after prior warm-up queries." ] }, { "cell_type": "code", "execution_count": 17, "id": "d4d25165", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'query': 'Why use RAG with local inference?',\n", " 'answer': \"Okay, so I need to figure out why someone would use Retrieval Augmented Generation (RAG) with local inference. Let me start by understanding each part of the question.\\n\\nFirst, RAG. From the context given, Doc 1 says that RAG improves answer grounding by injecting relevant context. So RAG is a method that uses retrieval techniques to find the most relevant parts of a document or corpus to augment the generation process. This probably helps in making the generated answers more accurate because they're backed by real data.\\n\\nThen, local inference. Doc 0 mentions that Foundry Local provides an OpenAI-compatible local inference endpoint. So local inference means running the model on the user's device rather than sending the request to a remote server. This is good for privacy and reducing latency, but it might have limitations in terms of model size or capabilities compared to cloud-based options.\\n\\nNow, combining RAG with local inference. The context says that small language models can offer competitive quality with lower resource usage (Doc 3). So using a smaller model might be more efficient, but maybe RAG\",\n", " 'docs': [0, 3, 1],\n", " 'context': 'Doc 0: Foundry Local provides an OpenAI-compatible local inference endpoint.\\nDoc 3: Small Language Models can offer competitive quality with lower resource usage.\\nDoc 1: Retrieval Augmented Generation improves answer grounding by injecting relevant context.',\n", " 'route': 'chat-first',\n", " 'tried': [{'model': 'deepseek-r1-distill-qwen-7b-cuda-gpu:0',\n", " 'attempts': [('chat.completions',\n", " 200,\n", " 'Okay, so I need to figure out why someone would use Retrieval Augmented Generation (RAG) with local inference. Let me start by understanding each part of the qu')]}]}" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "result = answer('Why use RAG with local inference?')\n", "result" ] } ], "metadata": { "kernelspec": { "display_name": "demo", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.15" } }, "nbformat": 4, "nbformat_minor": 5 }