{ "cells": [ { "cell_type": "markdown", "id": "5392a8a8", "metadata": {}, "source": [ "# Session 2 – RAG Evaluation with ragas\n", "\n", "Evaluate minimal RAG pipeline using ragas metrics: answer_relevancy, faithfulness, context_precision." ] }, { "cell_type": "markdown", "id": "2b34473b", "metadata": {}, "source": [ "# Scenario\n", "This scenario evaluates a minimal Retrieval Augmented Generation (RAG) pipeline locally. We:\n", "- Define a tiny synthetic document corpus.\n", "- Embed docs & implement a naive similarity retriever.\n", "- Generate grounded answers using a local model (Foundry Local / OpenAI-compatible).\n", "- Compute ragas metrics (`answer_relevancy`, `faithfulness`, `context_precision`).\n", "- Support a FAST mode (env `RAG_FAST=1`) to only compute answer relevancy for quick iteration.\n", "\n", "Use this notebook to validate that your local model + embeddings stack produces factually grounded answers before scaling to larger corpora." ] }, { "cell_type": "markdown", "id": "e9eb1aa2", "metadata": {}, "source": [ "### Explanation: Dependency Installation\n", "Installs required libraries:\n", "- `foundry-local-sdk` for local model management.\n", "- `openai` client interface.\n", "- `sentence-transformers` for dense embeddings.\n", "- `ragas` + `datasets` for evaluation & metric computation.\n", "- `langchain-openai` adapter for ragas LLM interface.\n", "\n", "Safe to re-run; skip if environment already prepared." ] }, { "cell_type": "code", "execution_count": 1, "id": "ff641221", "metadata": {}, "outputs": [], "source": [ "# Install libraries (ragas pulls datasets, evaluate, etc.)\n", "!pip install -q foundry-local-sdk openai sentence-transformers ragas datasets numpy langchain-openai" ] }, { "cell_type": "markdown", "id": "61e82678", "metadata": {}, "source": [ "### Explanation: Core Imports & Metrics\n", "Loads core libraries and ragas metrics. Key pieces:\n", "- SentenceTransformer for embeddings.\n", "- `evaluate` + selected ragas metrics.\n", "- `Dataset` for constructing evaluation corpus.\n", "These imports do not trigger remote calls (other than potential model cache load for embeddings)." ] }, { "cell_type": "code", "execution_count": 2, "id": "519dabae", "metadata": {}, "outputs": [], "source": [ "import os, numpy as np\n", "from sentence_transformers import SentenceTransformer\n", "from foundry_local import FoundryLocalManager\n", "from openai import OpenAI\n", "from ragas import evaluate\n", "from ragas.metrics import answer_relevancy, faithfulness, context_precision\n", "from datasets import Dataset" ] }, { "cell_type": "markdown", "id": "74f01938", "metadata": {}, "source": [ "### Explanation: Toy Corpus & QA Ground Truth\n", "Defines a miniature in-memory corpus (`DOCS`), a set of user questions, and expected ground truth answers. These allow fast, deterministic metric computation without external data fetches. In real scenarios you'd sample production queries + curated answers." ] }, { "cell_type": "code", "execution_count": 3, "id": "27307d2e", "metadata": {}, "outputs": [], "source": [ "DOCS = [\n", " 'Foundry Local exposes a local OpenAI-compatible endpoint.',\n", " 'RAG retrieves relevant context snippets before generation.',\n", " 'Local inference improves privacy and reduces latency.',\n", "]\n", "QUESTIONS = [\n", " 'What advantage does local inference offer?',\n", " 'How does RAG improve grounding?',\n", "]\n", "GROUND_TRUTH = [\n", " 'It reduces latency and preserves privacy.',\n", " 'It adds retrieved context snippets for factual grounding.',\n", "]" ] }, { "cell_type": "markdown", "id": "eaf3b2ec", "metadata": {}, "source": [ "### Explanation: Service Init, Embeddings & Safety Patch\n", "Initializes Foundry Local manager, applies a schema-drift safety patch for `promptTemplate`, resolves model id, creates OpenAI-compatible client, and pre-computes dense embeddings for the document corpus. This sets up reusable state for retrieval + generation." ] }, { "cell_type": "code", "execution_count": 4, "id": "156a7bc9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Service running: True | Endpoint: http://127.0.0.1:57127/v1\n", "Cached models: [FoundryModelInfo(alias=gpt-oss-20b, id=gpt-oss-20b-cuda-gpu:1, execution_provider=CUDAExecutionProvider, device_type=GPU, file_size=9882 MB, license=apache-2.0), FoundryModelInfo(alias=phi-3.5-mini, id=Phi-3.5-mini-instruct-cuda-gpu:1, execution_provider=CUDAExecutionProvider, device_type=GPU, file_size=2181 MB, license=MIT), FoundryModelInfo(alias=phi-4-mini, id=Phi-4-mini-instruct-cuda-gpu:4, execution_provider=CUDAExecutionProvider, device_type=GPU, file_size=3686 MB, license=MIT), FoundryModelInfo(alias=qwen2.5-0.5b, id=qwen2.5-0.5b-instruct-cuda-gpu:3, execution_provider=CUDAExecutionProvider, device_type=GPU, file_size=528 MB, license=apache-2.0), FoundryModelInfo(alias=qwen2.5-7b, id=qwen2.5-7b-instruct-cuda-gpu:3, execution_provider=CUDAExecutionProvider, device_type=GPU, file_size=4843 MB, license=apache-2.0), FoundryModelInfo(alias=qwen2.5-coder-7b, id=qwen2.5-coder-7b-instruct-cuda-gpu:3, execution_provider=CUDAExecutionProvider, device_type=GPU, file_size=4843 MB, license=apache-2.0)]\n", "Using model id: Phi-4-mini-instruct-cuda-gpu:4\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "c:\\Users\\leestott\\AppData\\Local\\miniforge\\envs\\demo\\lib\\site-packages\\transformers\\models\\bert\\modeling_bert.py:440: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at C:\\cb\\pytorch_1000000000000\\work\\aten\\src\\ATen\\native\\transformers\\cuda\\sdp_utils.cpp:555.)\n", " attn_output = torch.nn.functional.scaled_dot_product_attention(\n" ] } ], "source": [ "import os\n", "from foundry_local import FoundryLocalManager\n", "from foundry_local.models import FoundryModelInfo\n", "from openai import OpenAI\n", "\n", "# --- Safe monkeypatch for potential null promptTemplate field (schema drift guard) ---\n", "_original_from_list_response = FoundryModelInfo.from_list_response\n", "\n", "def _safe_from_list_response(response): # type: ignore\n", " try:\n", " if isinstance(response, dict) and response.get(\"promptTemplate\") is None:\n", " response[\"promptTemplate\"] = {}\n", " except Exception as e: # pragma: no cover\n", " print(f\"Warning normalizing promptTemplate: {e}\")\n", " return _original_from_list_response(response)\n", "\n", "if getattr(FoundryModelInfo.from_list_response, \"__name__\", \"\") != \"_safe_from_list_response\":\n", " FoundryModelInfo.from_list_response = staticmethod(_safe_from_list_response) # type: ignore\n", "# --- End monkeypatch ---\n", "\n", "alias = os.getenv('FOUNDRY_LOCAL_ALIAS','phi-3.5-mini')\n", "manager = FoundryLocalManager(alias)\n", "print(f\"Service running: {manager.is_service_running()} | Endpoint: {manager.endpoint}\")\n", "print('Cached models:', manager.list_cached_models())\n", "model_info = manager.get_model_info(alias)\n", "model_id = model_info.id\n", "print(f\"Using model id: {model_id}\")\n", "\n", "# OpenAI-compatible client\n", "client = OpenAI(base_url=manager.endpoint, api_key=manager.api_key or 'not-needed')\n", "\n", "from sentence_transformers import SentenceTransformer\n", "embedder = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')\n", "import numpy as np\n", "doc_emb = embedder.encode(DOCS, convert_to_numpy=True, normalize_embeddings=True)\n" ] }, { "cell_type": "markdown", "id": "566d24a8", "metadata": {}, "source": [ "### Explanation: Retriever Function\n", "Defines a naive vector similarity retriever using dot product over normalized embeddings. Returns top-k docs (k=2 default). In production swap with ANN index (FAISS, Chroma, Milvus) for scale & latency." ] }, { "cell_type": "code", "execution_count": 5, "id": "0af32d21", "metadata": {}, "outputs": [], "source": [ "def retrieve(query, k=2):\n", " q = embedder.encode([query], convert_to_numpy=True, normalize_embeddings=True)[0]\n", " sims = doc_emb @ q\n", " return [DOCS[i] for i in sims.argsort()[::-1][:k]]" ] }, { "cell_type": "markdown", "id": "b00f284e", "metadata": {}, "source": [ "### Explanation: Generation Function\n", "`generate` constructs a constrained prompt (system instructs to use ONLY context) and calls the local model. Low temperature (0.1) favors faithful extraction over creativity. Returns trimmed answer text." ] }, { "cell_type": "code", "execution_count": 6, "id": "7798ef22", "metadata": {}, "outputs": [], "source": [ "def generate(query, contexts):\n", " ctx = \"\\n\".join(contexts)\n", " messages = [\n", " {'role':'system','content':'Answer using ONLY the provided context.'},\n", " {'role':'user','content':f\"Context:\\n{ctx}\\n\\nQuestion: {query}\"}\n", " ]\n", " resp = client.chat.completions.create(model=model_id, messages=messages, max_tokens=120, temperature=0.1)\n", " return resp.choices[0].message.content.strip()\n" ] }, { "cell_type": "markdown", "id": "1fbde788", "metadata": {}, "source": [ "### Explanation: Fallback Client Initialization\n", "Ensures `client` exists even if earlier initialization cell was skipped or failed—prevents NameError during later evaluation steps." ] }, { "cell_type": "code", "execution_count": 7, "id": "6e71f8fc", "metadata": {}, "outputs": [], "source": [ "# Fallback client initialization (added after patch failure)\n", "try:\n", " client # type: ignore\n", "except NameError:\n", " from openai import OpenAI\n", " client = OpenAI(base_url=manager.endpoint, api_key=manager.api_key or 'not-needed')\n", " print('Initialized OpenAI-compatible client (late init).')\n" ] }, { "cell_type": "markdown", "id": "d17386ee", "metadata": {}, "source": [ "### Explanation: Evaluation Loop & Metrics\n", "Builds the evaluation dataset (required columns: question, answer, contexts, ground_truths, reference) then iterates selected ragas metrics.\n", "\n", "Optimization:\n", "- FAST_MODE restricts to answer relevancy for quick smoke tests.\n", "- Per-metric loop avoids full recomputation when one metric fails.\n", "\n", "Outputs a dict of metric -> score (NaN if failure)." ] }, { "cell_type": "code", "execution_count": 8, "id": "521a9163", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Evaluation dataset columns: ['question', 'answer', 'contexts', 'ground_truths', 'reference']\n", "Metrics to compute: ['answer_relevancy']\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.\n", "LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.\n", "LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "answer_relevancy finished in 78.1s -> 0.6975427764759168\n", "RAG evaluation results: {'answer_relevancy': 0.6975427764759168}\n" ] }, { "data": { "text/plain": [ "{'answer_relevancy': 0.6975427764759168}" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Build evaluation dataset with required columns (including 'reference' for context_precision)\n", "records = []\n", "for q, gt in zip(QUESTIONS, GROUND_TRUTH):\n", " ctxs = retrieve(q)\n", " ans = generate(q, ctxs)\n", " records.append({\n", " 'question': q,\n", " 'answer': ans,\n", " 'contexts': ctxs,\n", " 'ground_truths': [gt],\n", " 'reference': gt\n", " })\n", "\n", "from datasets import Dataset\n", "from ragas import evaluate\n", "from ragas.metrics import answer_relevancy, faithfulness, context_precision\n", "from langchain_openai import ChatOpenAI\n", "from ragas.run_config import RunConfig\n", "import math, time, os\n", "import numpy as np\n", "\n", "ragas_llm = ChatOpenAI(model=model_id, base_url=manager.endpoint, api_key=manager.api_key or 'not-needed', temperature=0.0, timeout=60)\n", "\n", "class LocalEmbeddings:\n", " def embed_documents(self, texts):\n", " return embedder.encode(texts, convert_to_numpy=True, normalize_embeddings=True).tolist()\n", " def embed_query(self, text):\n", " return embedder.encode([text], convert_to_numpy=True, normalize_embeddings=True)[0].tolist()\n", "\n", "# Fast mode: only answer_relevancy unless RAG_FAST=0\n", "FAST_MODE = os.getenv('RAG_FAST','1') == '1'\n", "metrics = [answer_relevancy] if FAST_MODE else [answer_relevancy, faithfulness, context_precision]\n", "\n", "base_timeout = 45 if FAST_MODE else 120\n", "\n", "ds = Dataset.from_list(records)\n", "print('Evaluation dataset columns:', ds.column_names)\n", "print('Metrics to compute:', [m.name for m in metrics])\n", "\n", "results_dict = {}\n", "for metric in metrics:\n", " t0 = time.time()\n", " try:\n", " cfg = RunConfig(timeout=base_timeout, max_workers=1)\n", " partial = evaluate(ds, metrics=[metric], llm=ragas_llm, embeddings=LocalEmbeddings(), run_config=cfg, show_progress=False)\n", " raw_val = partial[metric.name]\n", " if isinstance(raw_val, list):\n", " numeric = [v for v in raw_val if isinstance(v, (int, float))]\n", " score = float(np.nanmean(numeric)) if numeric else math.nan\n", " else:\n", " score = float(raw_val)\n", " results_dict[metric.name] = score\n", " except Exception as e:\n", " results_dict[metric.name] = math.nan\n", " print(f\"Metric {metric.name} failed: {e}\")\n", " finally:\n", " print(f\"{metric.name} finished in {time.time()-t0:.1f}s -> {results_dict[metric.name]}\")\n", "\n", "print('RAG evaluation results:', results_dict)\n", "results_dict" ] } ], "metadata": { "kernelspec": { "display_name": "demo", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.15" } }, "nbformat": 4, "nbformat_minor": 5 }