Test Case ID,Eval Set,Quality Signal,Root Cause Type,Root Cause Detail,What Went Wrong,Diagnostic Path,Remediation Action,Status,Won't Fix Rationale,Verification Result,Verification Date,Date Triaged,Triaged By
KG-003,Knowledge Grounding,Factual accuracy,Eval Setup,Outdated expected answer,"Expected answer said '30-day return window'; source updated to '15 business days'",Step 1 Q1.2 — expected answer outdated,Updated expected value to '15 business days',Resolved,,Pass,2026-02-15,2026-02-14,
KG-005,Knowledge Grounding,Knowledge grounding / hallucination prevention,Agent Config,Hallucination,"Agent claimed '3-year extended warranty' when source says '2-year standard warranty'","Step 1 passed → Step 2 Q2.4 (no source used) + Q2.5 (contradicted source)","Added grounding instruction: 'Only answer based on information found in your knowledge sources'",Resolved,,Pass,2026-02-16,2026-02-14,
TI-002,Tool Invocation,Tool call correctness,Platform Limitation,Retrieval ranking,Retrieval consistently returns FAQ page instead of product manual for product spec queries,"Step 2 Q2.1 (wrong source) → Step 3 (persists across config changes)",Restructured doc headings; duplicated critical specs in FAQ; escalated to platform team,Resolved (workaround),,Pass (with workaround),2026-02-20,2026-02-18,
FA-019,Factual Accuracy,Factual accuracy,Platform Limitation,Retrieval ranking — ambiguous query,Ambiguous query cannot reliably retrieve correct source document,"Step 2 Q2.1 → Step 3 (persists across config changes, query inherently ambiguous)",Documented as known limitation; monitoring in production,Won't Fix,Query is genuinely ambiguous; no source restructuring can fully resolve. Monitoring in production for user impact.,N/A,,2026-02-18,
,,,,,,,,,,,,
,,,,,,,,,,,,
,,,,,,,,,,,,