--- name: RAG Regression Testing description: Gate RAG pipelines in CI with versioned golden eval sets, per-metric thresholds, baseline drift detection, and a build that fails when retrieval or answer quality regresses. version: 1.0.0 author: thetestingacademy license: MIT tags: [rag, regression-testing, llm-evals, ci, golden-dataset, drift-detection, ragas, deepeval, baseline] testingTypes: [regression, llm-evals, integration] frameworks: [ragas, deepeval, pytest] languages: [python] domains: [ai, llm, api] agents: [claude-code, cursor, github-copilot, windsurf, codex, aider, continue, cline, zed, bolt, gemini-cli, amp] --- # RAG Regression Testing Skill You are an expert in shipping RAG systems without quality regressions. When the user asks you to add CI gates, detect drift, or stop a build from merging when answer quality drops, you build a versioned golden eval set, compare every run against a committed baseline, and fail the build on absolute-threshold breaches or relative drops. You treat prompts and retriever configs as versioned artifacts, because a prompt change is a behavior change. ## Core Principles 1. **A RAG pipeline regresses silently.** Code tests stay green while answer quality rots from a model update, a prompt tweak, a chunking change, or an index rebuild. Only an eval gate catches this. 2. **The golden set is the regression contract.** Every metric is scored against a fixed, committed dataset. Changing the dataset is a deliberate, reviewed event - never an accident. 3. **Gate on two conditions: absolute floor and relative drop.** Fail if any metric falls below its hard floor, *and* fail if it drops more than N points versus the committed baseline - even while still "passing." 4. **Baselines are committed artifacts.** Store `baseline_metrics.json` in the repo. A score is only meaningful as a delta against a known-good baseline. 5. **Version the prompt and retriever, not just the code.** Tag each eval run with prompt and retriever versions so a regression can be traced to the exact change that caused it. 6. **Pin everything that scores.** Judge model, judge temperature, embedding model, and `top_k`. An unpinned judge makes "regression" indistinguishable from judge noise. 7. **Fail fast and loud in CI; allow an explicit baseline-update path.** The only way to move the baseline is a reviewed PR that regenerates and commits it. 8. **Quarantine, do not delete, flaky golden samples.** Mark them, investigate, fix the data or the pipeline - never silently drop a hard question to make the gate pass. ## Repository Layout ``` rag-evals/ golden/ dataset.v3.json # versioned golden set; bump filename on change baseline/ baseline_metrics.json # committed known-good scores config/ eval_config.py # pinned models, thresholds, drift budget run_eval.py # produces scores, writes report.json gate.py # compares scores vs baseline + floors -> exit code update_baseline.py # regenerates baseline (run intentionally) .github/ workflows/ rag-regression.yml ``` ## Pinned Evaluation Config ```python # config/eval_config.py from dataclasses import dataclass @dataclass(frozen=True) class EvalConfig: # Everything that influences a score is pinned. judge_model: str = "gpt-4o-mini" judge_temperature: float = 0.0 embedding_model: str = "text-embedding-3-small" top_k: int = 5 # Identifies the system under test for traceability. prompt_version: str = "answer-v4" retriever_version: str = "hybrid-bm25+dense-v2" dataset_path: str = "rag-evals/golden/dataset.v3.json" baseline_path: str = "rag-evals/baseline/baseline_metrics.json" # Absolute floors: build fails if a metric drops below these, ever. HARD_FLOORS = { "faithfulness": 0.88, "context_precision": 0.78, "context_recall": 0.78, "answer_relevancy": 0.72, } # Drift budget: build fails if a metric drops more than this vs baseline, # even if still above the hard floor. Catches slow erosion. MAX_REGRESSION = { "faithfulness": 0.03, "context_precision": 0.05, "context_recall": 0.05, "answer_relevancy": 0.05, } CONFIG = EvalConfig() ``` ## Running the Eval and Producing a Report ```python # run_eval.py import json from datasets import Dataset from ragas import evaluate from ragas.metrics import ( context_precision, context_recall, faithfulness, answer_relevancy, ) from ragas.llms import LangchainLLMWrapper from langchain_openai import ChatOpenAI, OpenAIEmbeddings from config.eval_config import CONFIG from my_rag_app import rag_pipeline def load_golden(path: str) -> list[dict]: with open(path) as f: return json.load(f)["samples"] def main() -> None: golden = load_golden(CONFIG.dataset_path) rows = {"question": [], "answer": [], "contexts": [], "ground_truth": []} for s in golden: out = rag_pipeline(s["question"], top_k=CONFIG.top_k) rows["question"].append(s["question"]) rows["answer"].append(out["answer"]) rows["contexts"].append(out["contexts"]) rows["ground_truth"].append(s["ground_truth"]) judge = LangchainLLMWrapper( ChatOpenAI(model=CONFIG.judge_model, temperature=CONFIG.judge_temperature) ) result = evaluate( Dataset.from_dict(rows), metrics=[context_precision, context_recall, faithfulness, answer_relevancy], llm=judge, embeddings=OpenAIEmbeddings(model=CONFIG.embedding_model), ) df = result.to_pandas() means = { m: round(float(df[m].mean()), 4) for m in ["context_precision", "context_recall", "faithfulness", "answer_relevancy"] } report = { "metrics": means, "n_samples": len(golden), "below_floor_counts": { m: int((df[m] < 0.5).sum()) # count of egregious per-sample failures for m in means }, "prompt_version": CONFIG.prompt_version, "retriever_version": CONFIG.retriever_version, "judge_model": CONFIG.judge_model, } with open("report.json", "w") as f: json.dump(report, f, indent=2) print(json.dumps(report, indent=2)) if __name__ == "__main__": main() ``` ## The Gate: Floors + Drift Detection ```python # gate.py import json import sys from config.eval_config import CONFIG, HARD_FLOORS, MAX_REGRESSION def load(path: str) -> dict: with open(path) as f: return json.load(f) def main() -> int: report = load("report.json") current = report["metrics"] baseline = load(CONFIG.baseline_path)["metrics"] failures: list[str] = [] for metric, score in current.items(): floor = HARD_FLOORS.get(metric) if floor is not None and score < floor: failures.append(f"[FLOOR] {metric}={score:.3f} < hard floor {floor:.2f}") base = baseline.get(metric) budget = MAX_REGRESSION.get(metric) if base is not None and budget is not None: drop = base - score if drop > budget: failures.append( f"[DRIFT] {metric} dropped {drop:.3f} " f"(baseline {base:.3f} -> {score:.3f}, budget {budget:.2f})" ) if failures: print("RAG REGRESSION DETECTED:\n " + "\n ".join(failures)) print(f"\nprompt={report['prompt_version']} retriever={report['retriever_version']}") return 1 print("RAG eval passed. No regression vs baseline.") for m, s in current.items(): print(f" {m}: {s:.3f} (baseline {baseline.get(m, float('nan')):.3f})") return 0 if __name__ == "__main__": sys.exit(main()) ``` ## Updating the Baseline (Intentional Only) ```python # update_baseline.py """Run ONLY when a quality change is intended and reviewed. The resulting baseline_metrics.json must be committed in the same PR.""" import json import shutil from config.eval_config import CONFIG # run_eval.py must have been run first to produce report.json with open("report.json") as f: report = json.load(f) shutil.copy(CONFIG.baseline_path, CONFIG.baseline_path + ".bak") with open(CONFIG.baseline_path, "w") as f: json.dump({"metrics": report["metrics"], "prompt_version": report["prompt_version"], "retriever_version": report["retriever_version"]}, f, indent=2) print("Baseline updated. Commit this file with a justification in the PR.") ``` ## CI Workflow (GitHub Actions) ```yaml # .github/workflows/rag-regression.yml name: RAG Regression Gate on: pull_request: paths: - "rag-evals/**" - "src/prompts/**" - "src/retriever/**" - "src/rag/**" workflow_dispatch: concurrency: group: rag-eval-${{ github.ref }} cancel-in-progress: true jobs: rag-eval-gate: runs-on: ubuntu-latest timeout-minutes: 20 steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" - name: Install deps run: | python -m pip install --upgrade pip pip install -r requirements.txt # ragas, datasets, langchain-openai, etc. - name: Run RAG evaluation env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} run: python rag-evals/run_eval.py - name: Gate on thresholds + drift run: python rag-evals/gate.py # non-zero exit fails the job - name: Upload eval report if: always() uses: actions/upload-artifact@v4 with: name: rag-eval-report path: report.json - name: Comment metrics on PR if: always() && github.event_name == 'pull_request' uses: actions/github-script@v7 with: script: | const fs = require('fs'); const r = JSON.parse(fs.readFileSync('report.json', 'utf8')); const rows = Object.entries(r.metrics) .map(([k, v]) => `| ${k} | ${v.toFixed(3)} |`).join('\n'); const body = `### RAG Eval (\`${r.prompt_version}\` / \`${r.retriever_version}\`)\n` + `| metric | score |\n|---|---|\n${rows}`; await github.rest.issues.createComment({ owner: context.repo.owner, repo: context.repo.repo, issue_number: context.issue.number, body, }); ``` The gate runs on PRs that touch prompts, retriever, or the eval set. A merge is blocked until the gate passes - so the only way to ship a quality change is to also commit the new baseline. ## Detecting Drift Over Time For nightly scheduled runs against production traffic samples, append each run's metrics to a time series and alert on a moving-window drop: ```python # drift_alert.py import statistics def detect_trend_drift(history: list[dict], metric: str, window: int = 7) -> str | None: """history: list of {date, metrics:{...}} newest last.""" series = [h["metrics"][metric] for h in history if metric in h["metrics"]] if len(series) < window + 1: return None recent = statistics.mean(series[-3:]) baseline_window = statistics.mean(series[-(window + 1):-3]) drop = baseline_window - recent if drop > 0.04: return (f"{metric} trending down: {baseline_window:.3f} -> {recent:.3f} " f"over {window} days (drop {drop:.3f})") return None ``` ## Best Practices 1. **Commit the baseline; never compute it at runtime.** A regression is a delta from a known-good, reviewed file - not from yesterday's accidental score. 2. **Gate on both a hard floor and a drift budget.** Floors catch cliffs; drift budgets catch slow erosion that stays "green." 3. **Version the golden dataset in the filename (`dataset.v3.json`).** Bumping the version is a reviewable, deliberate act. 4. **Tag every report with prompt and retriever versions.** When the gate fails, you know exactly which artifact regressed. 5. **Make baseline updates a separate, justified PR step.** Require a written reason in the PR description for any baseline move. 6. **Scope the gate to RAG-relevant paths, and publish the report.** Use `paths:` to keep paid LLM-judge calls off unrelated PRs; upload `report.json` as an artifact and comment scores on the PR. 7. **Run a nightly scheduled eval against fresh data for trend drift.** PR gates catch deliberate changes; scheduled runs catch upstream model drift. ## Anti-Patterns to Avoid 1. **No committed baseline.** Comparing against "last run" lets a slow daily 0.5% decline accumulate into a disaster, each step individually passing. 2. **Floor-only gating.** A metric sliding from 0.95 to 0.89 (still above a 0.88 floor) is a real regression a drift budget would catch. 3. **Editing the golden set in the same PR as a pipeline change.** You can no longer tell whether the score moved because the system changed or the test changed. 4. **Unpinned judge or embedding model in CI.** Judge noise gets misread as regression, and the gate becomes flaky and ignored. 5. **Deleting hard golden questions to make the build green.** That is removing the smoke detector because it keeps going off. 6. **Silencing the gate (`continue-on-error: true`) to unblock a deadline.** A non-blocking quality gate is theater. ## When to Trigger This Skill Trigger when the user asks to: - Add a CI gate or build check for RAG / LLM answer quality - Detect quality drift or regression in a RAG pipeline over time - Set up a golden eval set with baselines and thresholds for CI - Fail a build when faithfulness, retrieval, or relevancy drops - Version prompts and retrievers for regression traceability - Wire Ragas/DeepEval into GitHub Actions or another CI system For the *definitions and scoring* of the underlying metrics (faithfulness, context precision/recall, answer relevancy), use the RAG Evaluation Metrics skill. This skill assumes those metrics exist and focuses on gating and drift over time.