--- name: hallucination-guard version: 1.0.0 author: reikys description: Execution-based verification guardrail with 14 check items for AI agent output tags: [verification, quality, safety, guard, hallucination] trigger: "hallucination check" --- # πŸ›‘οΈ hallucination-guard > **Solving the biggest trust problem with AI agents.** > Not with "double-check that" prompts, but with 14 execution-based verification items. --- ## 🎯 Problem Definition AI agents carry the following trust issues: | Problem Type | Example | |-------------|---------| | **Path hallucination** | References non-existent files/directories as if they exist | | **Command hallucination** | Describes uninstalled binaries as if they run normally | | **Library hallucination** | Writes code that `import`s packages that don't exist on npm/pip | | **Numerical hallucination** | States unsourced statistics/percentages as fact | | **Completeness hallucination** | Reports "done" while leaving TODO/PLACEHOLDER behind | | **Consistency hallucination** | Uses different names for the same concept within a document | ### Limitations of Existing Solutions - **truth-check, verification-before-completion**: Just tells the agent "check again" β€” still hallucination-based - **hallucination-guard**: 14 concrete items Γ— executable commands Γ— structured PASS/FAIL report --- ## ⚑ Quick Start ### 1. Trigger Method Call anytime during agent conversation with the following phrases: ``` hallucination check hallucination check on hallucination check --scope=fact,completeness ``` ### 2. Auto-Execution Method Add to the system prompt so the agent automatically runs this skill before completing a task: ``` Before completing any task, you must run all 14 checks from the hallucination-guard SKILL.md, output the PASS/FAIL report, and fix any FAIL items. ``` ### 3. Quick Check (3-Minute Version) When the full 14 items feel like too much, run only the essential 5: ```bash # Run only H-1, H-2, H-9, H-10, H-12 hallucination check --quick ``` --- ## πŸ“‹ 14 Check Items in Detail ### πŸ”΅ Fact Verification (H-1 ~ H-5) --- #### H-1: File Path Existence Verification **Purpose:** Verify that files/directories referenced by the agent actually exist **Verification Commands:** ```bash # macOS / Linux stat ls -la # Existence check only (exit code based) [ -e "" ] && echo "PASS: $path exists" || echo "FAIL: $path not found" # Batch check for multiple paths for p in path1 path2 path3; do [ -e "$p" ] && echo "βœ… $p" || echo "❌ $p" done ``` **Windows (PowerShell):** ```powershell Test-Path "C:\path\to\check" ``` **Pass Criteria:** All referenced paths confirmed to exist via `stat`/`Test-Path` β†’ PASS --- #### H-2: Command/Binary Existence Verification **Purpose:** Verify that CLI commands used in code or documentation are actually installed **Verification Commands:** ```bash # macOS / Linux which command -v # Example: batch check for multiple binaries for cmd in git node python3 docker jq; do command -v "$cmd" &>/dev/null \ && echo "βœ… $cmd: $(which $cmd)" \ || echo "❌ $cmd: not installed" done ``` **Windows (PowerShell):** ```powershell Get-Command -ErrorAction SilentlyContinue ``` **Pass Criteria:** All CLI commands appearing in documents/code confirmed via `command -v` β†’ PASS --- #### H-3: URL Validity Check (Optional) **Purpose:** Verify that links/API endpoints embedded in documentation are actually accessible **Verification Commands:** ```bash # Check HTTP status code (5-second timeout) curl -sI --max-time 5 | head -1 # Batch check script urls=( "https://example.com/api" "https://docs.example.com" ) for url in "${urls[@]}"; do status=$(curl -sI --max-time 5 "$url" | head -1 | awk '{print $2}') if [[ "$status" =~ ^[23] ]]; then echo "βœ… $url β†’ HTTP $status" else echo "❌ $url β†’ HTTP $status (or unreachable)" fi done ``` **Note:** False positive/negative possible depending on network environment. Manual verification recommended for internal network URLs. **Pass Criteria:** External reference URLs respond with 2xx/3xx β†’ PASS (optional execution) --- #### H-4: Code Syntax Validity Check **Purpose:** Verify that code generated by the agent is actually parseable **Verification Commands:** **Python:** ```bash python3 -c " import ast, sys with open('target.py') as f: src = f.read() try: ast.parse(src) print('βœ… Python syntax valid') except SyntaxError as e: print(f'❌ SyntaxError: {e}') sys.exit(1) " ``` **JavaScript/TypeScript:** ```bash # Node.js node --check target.js # TypeScript npx tsc --noEmit target.ts ``` **JSON:** ```bash jq . target.json > /dev/null && echo "βœ… JSON valid" || echo "❌ JSON parse failed" ``` **YAML:** ```bash python3 -c "import yaml; yaml.safe_load(open('target.yaml'))" \ && echo "βœ… YAML valid" || echo "❌ YAML parse failed" ``` **Shell:** ```bash bash -n target.sh && echo "βœ… Shell syntax valid" || echo "❌ Shell syntax error" ``` **Pass Criteria:** All generated code files pass their respective language parsers β†’ PASS --- #### H-5: Numerical Data Cross-Verification **Purpose:** Verify that statistics/numbers mentioned by the agent are substantiated **Verification Method:** ``` Checklist: β–‘ Is the source (URL, paper, official docs) specified for the number? β–‘ Can the source be cross-verified with 2+ references? β–‘ Is the data current? (check date) β–‘ Is uncertainty appropriately expressed? ("approximately X%", "roughly Nx") ``` **Auto-detection Pattern (grep):** ```bash # Detect unsourced number patterns grep -En "[0-9]+%" | grep -v "http\|source\|ref\|reference" grep -En "[0-9]+(x|times)" | grep -v "http\|source" ``` **Pass Criteria:** All numbers have cited sources or are labeled "needs verification:" β†’ PASS --- ### 🟑 Consistency (H-6 ~ H-8) --- #### H-6: No Self-Contradiction **Purpose:** Verify there are no conflicting claims within the same document **Verification Method:** ```bash # Manual check for negation/affirmation pairs grep -n "cannot\|impossible\|prohibited\|not available\|not supported" grep -n "possible\|supported\|available\|can be\|is able to" # Agent self-verification instruction """ Read the document below and list all pairs of contradicting claims. If none exist, respond with "No self-contradiction found." [document content] """ ``` **Pass Criteria:** 0 conflicting claim pairs β†’ PASS --- #### H-7: Plan-Result Alignment **Purpose:** 1:1 mapping to verify all initially promised deliverables were actually generated **Verification Method:** ```bash # Extract deliverable list (e.g., ## Deliverables section) grep -A 20 "deliverable\|output\|result" PLAN.md # Verify actual file existence promised_files=( "src/main.py" "README.md" "tests/test_main.py" ) for f in "${promised_files[@]}"; do [ -f "$f" ] && echo "βœ… $f" || echo "❌ $f missing (promise not fulfilled)" done ``` **Pass Criteria:** All deliverables specified in the plan actually exist β†’ PASS --- #### H-8: Terminology Consistency **Purpose:** Verify that the same concept is called by the same name throughout the document **Auto-detection Example:** ```bash # Detect synonym mixing (customize as needed) echo "=== 'user' related terms ===" grep -oin "user\|customer\|client\|end-user\|end user" | sort | uniq -c | sort -rn echo "=== 'error' related terms ===" grep -oin "error\|failure\|fault\|exception\|bug" | sort | uniq -c | sort -rn ``` **Pass Criteria:** Core terms are not mixed with 2+ different names β†’ PASS (Intentional synonym usage must be stated in comments/definitions) --- ### 🟒 Completeness (H-9 ~ H-11) --- #### H-9: No Remaining TODO/FIXME **Purpose:** Verify that no incomplete markers remain **Verification Commands:** ```bash # Basic search grep -rn "TODO\|FIXME\|HACK\|XXX\|TEMP\|BUG" # Count count=$(grep -rn "TODO\|FIXME\|HACK\|XXX" | wc -l) if [ "$count" -eq 0 ]; then echo "βœ… H-9 PASS: No remaining markers" else echo "❌ H-9 FAIL: $count incomplete markers found" grep -rn "TODO\|FIXME\|HACK\|XXX" fi ``` **Windows (PowerShell):** ```powershell Select-String -Path ".\*" -Pattern "TODO|FIXME|HACK|XXX" -Recurse ``` **Pass Criteria:** 0 TODO/FIXME/HACK/XXX occurrences β†’ PASS --- #### H-10: No Placeholders **Purpose:** Verify no placeholders remain that weren't filled with actual values **Verification Commands:** ```bash # Search for placeholder patterns grep -rn \ "PLACEHOLDER\|CHANGEME\|TBD\|INSERT_HERE\| # Count count=$(grep -rn "PLACEHOLDER\|CHANGEME\|TBD\|INSERT_HERE\|YOUR_API_KEY" | wc -l) [ "$count" -eq 0 ] \ && echo "βœ… H-10 PASS: No placeholders" \ || echo "❌ H-10 FAIL: $count found" ``` **Pass Criteria:** 0 placeholder pattern occurrences β†’ PASS --- #### H-11: All Deliverables Exist **Purpose:** Verify that files promised in specifications (README, PLAN, conversation) actually exist **Verification Method:** ```bash #!/bin/bash # Extract file list from spec file (path pattern matching) spec_file="PLAN.md" # or README.md echo "=== Deliverable Existence Check ===" missing=0 # Extract paths from markdown code blocks grep -oE '`[^`]+\.(py|js|ts|md|json|yaml|sh)`' "$spec_file" | tr -d '`' | while read f; do if [ -e "$f" ]; then echo "βœ… $f" else echo "❌ $f (not found)" missing=$((missing + 1)) fi done ``` **Pass Criteria:** All paths mentioned in spec files actually exist β†’ PASS --- ### πŸ”΄ Hallucination Patterns (H-12 ~ H-14) --- #### H-12: No Fictional Library/API References **Purpose:** Verify that no non-existent packages are `import`ed/`require`d **npm Package Verification:** ```bash # Verify all dependencies in package.json exist node -e " const pkg = require('./package.json'); const deps = {...(pkg.dependencies||{}), ...(pkg.devDependencies||{})}; const names = Object.keys(deps); console.log('Packages to check:', names.length); " # Actual npm registry lookup npm_check() { local pkg=$1 result=$(curl -sI "https://registry.npmjs.org/$pkg" | head -1) echo "$result" | grep -q "200" \ && echo "βœ… npm: $pkg exists" \ || echo "❌ npm: $pkg not found (suspected hallucination)" } # Usage example npm_check "some-package-name" ``` **pip Package Verification:** ```bash pip_check() { local pkg=$1 result=$(curl -sI "https://pypi.org/pypi/$pkg/json" | head -1) echo "$result" | grep -q "200" \ && echo "βœ… PyPI: $pkg exists" \ || echo "❌ PyPI: $pkg not found (suspected hallucination)" } # Registry lookup preferred since dry-run install is not available pip_check "some-library" ``` **Extracting and Batch-Verifying Imports from Code:** ```bash # Extract Python imports python3 -c " import ast, sys tree = ast.parse(open('target.py').read()) imports = set() for node in ast.walk(tree): if isinstance(node, ast.Import): for alias in node.names: imports.add(alias.name.split('.')[0]) elif isinstance(node, ast.ImportFrom): if node.module: imports.add(node.module.split('.')[0]) # Exclude standard library import sys stdlib = set(sys.stdlib_module_names) if hasattr(sys, 'stdlib_module_names') else set() third_party = imports - stdlib print('Third-party imports:', sorted(third_party)) " ``` **Pass Criteria:** All import/require packages actually exist in their registries β†’ PASS --- #### H-13: No Unsourced Statistics **Purpose:** Verify no numbers/statistics/ratios are presented definitively without sources **Auto-detection Patterns:** ```bash # Detect numbers without sources echo "=== Unsourced number candidates ===" # Percentage patterns grep -En "[0-9]+(\.[0-9]+)?%" | grep -iv "http\|source\|ref\|reference\|needs verification" # Multiplier/ratio patterns grep -En "[0-9]+(x|times)" | grep -iv "http\|source\|reference\|needs verification" # Large number patterns (millions, billions) grep -En "[0-9]+(million|billion|thousand)" | grep -iv "http\|source\|reference" ``` **Manual Check Checklist:** ``` β–‘ Do all statistics have "[source: URL or paper name]"? β–‘ Are uncertainty expressions used appropriately? ("approximately", "roughly", "estimated") β–‘ Are there statistics without dates? (unclear recency) β–‘ If internal data, is "based on internal data" stated? ``` **Pass Criteria:** All numbers have cited sources or "needs verification:" label β†’ PASS --- #### H-14: No Overconfident Expressions **Purpose:** Verify uncertain matters aren't stated definitively with "always", "never", "guaranteed", etc. **Auto-detection Commands:** ```bash echo "=== Overconfident expression detection ===" grep -En \ "never\|always\|definitely\|certainly\|guaranteed\|without doubt\|100%\|absolutely\|undoubtedly\|impossible\|must be true" \ ``` **Allowed vs. Not Allowed:** ``` ❌ Not allowed: "This method is guaranteed to work" βœ… Allowed: "This method works in most cases (needs verification: edge case X)" ❌ Not allowed: "Errors will never occur" βœ… Allowed: "Error likelihood is low in typical usage scenarios" ``` **Pass Criteria:** 0 definitive expressions, or if present, accompanied by supporting evidence β†’ PASS --- ## πŸ“Š Report Output Format ### Standard PASS/FAIL Table ``` ╔══════════════════════════════════════════════════════════╗ β•‘ πŸ›‘οΈ HALLUCINATION GUARD REPORT β•‘ β•‘ Executed: 2026-03-26 15:21 KST β•‘ ╠══════════════════════════════════════════════════════════╣ β•‘ Target: /path/to/target Scope: ALL β•‘ ╠══════╦══════════╦════════╦════════════════════════════╣ β•‘ ID β•‘ Category β•‘ Result β•‘ Details β•‘ ╠══════╬══════════╬════════╬════════════════════════════╣ β•‘ H-1 β•‘ Fact β•‘ PASS β•‘ All 4 referenced paths exist β•‘ β•‘ H-2 β•‘ Fact β•‘ PASS β•‘ 3 CLIs confirmed installed β•‘ β•‘ H-3 β•‘ Fact β•‘ SKIP β•‘ Network check not selected β•‘ β•‘ H-4 β•‘ Fact β•‘ PASS β•‘ Python/JS syntax valid β•‘ β•‘ H-5 β•‘ Fact β•‘ FAIL β•‘ 3 numbers missing sources β•‘ ╠══════╬══════════╬════════╬════════════════════════════╣ β•‘ H-6 β•‘ Consist. β•‘ PASS β•‘ No self-contradiction β•‘ β•‘ H-7 β•‘ Consist. β•‘ FAIL β•‘ README.md promised but absent β•‘ β•‘ H-8 β•‘ Consist. β•‘ PASS β•‘ Core term consistency OK β•‘ ╠══════╬══════════╬════════╬════════════════════════════╣ β•‘ H-9 β•‘ Complete β•‘ PASS β•‘ TODO/FIXME: 0 found β•‘ β•‘ H-10 β•‘ Complete β•‘ FAIL β•‘ YOUR_API_KEY: 2 found β•‘ β•‘ H-11 β•‘ Complete β•‘ PASS β•‘ Deliverables: 5/5 exist β•‘ ╠══════╬══════════╬════════╬════════════════════════════╣ β•‘ H-12 β•‘ Halluc. β•‘ PASS β•‘ All packages registry-verifiedβ•‘ β•‘ H-13 β•‘ Halluc. β•‘ FAIL β•‘ 2 unsourced numbers found β•‘ β•‘ H-14 β•‘ Halluc. β•‘ PASS β•‘ Overconfident expressions: 0 β•‘ ╠══════╩══════════╩════════╩════════════════════════════╣ β•‘ Total: PASS 9 / FAIL 4 / SKIP 1 β•‘ β•‘ Score: 9/13 = 69.2% (recommended threshold: 85%) β•‘ β•‘ Verdict: ❌ FAIL β€” Fix required, then re-run β•‘ β•šβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β• [FAIL Item Fix Guide] H-5: Add source URLs to numbers on lines 42, 67, 89 H-7: Create README.md or remove promise from PLAN.md H-10: Check .env.example β€” replace YOUR_API_KEY with actual key or comment out H-13: Add "[needs verification]" label to statistics on lines 15, 33 ``` ### Brief Mode (--brief) ``` πŸ›‘οΈ HG: PASS 9/13 (H-5,H-7,H-10,H-13 FAIL) β†’ Fix required ``` ### JSON Mode (--format=json) ```json { "timestamp": "2026-03-26T15:21:00+09:00", "target": "/path/to/target", "score": 0.692, "threshold": 0.85, "verdict": "FAIL", "results": [ {"id": "H-1", "category": "fact", "status": "PASS", "detail": "All 4 referenced paths exist"}, {"id": "H-5", "category": "fact", "status": "FAIL", "detail": "3 numbers missing sources", "lines": [42, 67, 89]}, ... ] } ``` --- ## 🌍 Platform-Specific Branches ### macOS ```bash OS="macos" STAT_CMD="stat" WHICH_CMD="which" GREP_CMD="grep" # If GNU grep not available, brew install grep recommended ``` ### Linux ```bash OS="linux" STAT_CMD="stat" WHICH_CMD="which" GREP_CMD="grep" # Most GNU tools installed by default ``` ### Windows (PowerShell) ```powershell # stat β†’ Test-Path / Get-Item # which β†’ Get-Command # grep β†’ Select-String # curl β†’ Invoke-WebRequest # Wrapper function definitions function hg-pathcheck { param($p) Test-Path $p } function hg-cmdcheck { param($c) Get-Command $c -EA SilentlyContinue } function hg-grep { param($p, $f) Select-String -Pattern $p -Path $f } ``` ### Docker/CI Environment ```yaml # GitHub Actions example - name: Hallucination Guard run: | bash scripts/hallucination-guard.sh \ --scope=all \ --format=json \ --output=hg-report.json - name: Check HG Result run: | score=$(jq '.score' hg-report.json) python3 -c "import sys; sys.exit(0 if float('$score') >= 0.85 else 1)" ``` --- ## πŸ”Œ Integration with Other Skills ### Integration with coding-agent ``` [Auto-trigger after coding-agent completion] 1. coding-agent completes code generation 2. hallucination-guard auto-runs (H-1, H-2, H-4, H-9, H-10, H-12 prioritized) 3. FAIL items β†’ Send fix instructions to coding-agent 4. Re-run β†’ Confirm PASS before final report ``` ### Integration with finishing-a-development-branch ``` [Add HG to branch completion checklist] β–‘ Tests passing β–‘ hallucination-guard 85%+ PASS β–‘ PR created ``` ### Integration with docs-writer ``` [Verify after documentation writing] docs-writer complete β†’ hallucination-guard --scope=fact,consistency,completeness ``` ### Integration with adversarial-code-review ``` [Dual verification pipeline] Code generation β†’ hallucination-guard (automated/execution-based verification) β†’ adversarial-code-review (logic/security verification) β†’ Final approval ``` --- ## βž• How to Add Custom Items Create `.hallucination-guard.yaml` at the project root: ```yaml # .hallucination-guard.yaml version: "1.0" custom_checks: - id: "H-C1" name: "Internal API endpoint verification" category: "fact" command: | curl -sI --max-time 5 https://internal-api.company.com/health | head -1 pass_pattern: "200" severity: "error" # error | warning | info - id: "H-C2" name: "Company terminology compliance" category: "consistency" command: | grep -n "end-user\|enduser\|end user" {target} | grep -v "# allowed" fail_pattern: "." # Match = FAIL severity: "warning" - id: "H-C3" name: "License header presence" category: "completeness" command: | head -3 {target} | grep -i "copyright\|license\|MIT\|Apache" pass_pattern: "." severity: "error" # Threshold customization thresholds: pass_rate: 0.90 # Require 90% PASS error_tolerance: 0 # Allow 0 error-level FAILs warning_tolerance: 2 # Allow up to 2 warning-level FAILs # Exclude patterns exclude: paths: - "node_modules/**" - "*.min.js" - ".git/**" checks: - "H-3" # Disable URL check (internal network environment) ``` --- ## πŸ“– 3 Real-World Usage Examples ### Example 1: Immediate Verification After Code Generation **Situation:** coding-agent generated a Python script, and you want to verify it actually works ``` [Conversation] User: Implement a file upload feature Agent: [Code generation complete] Created src/uploader.py... User: hallucination check on src/uploader.py Agent: πŸ›‘οΈ Running H-1... βœ… src/uploader.py exists πŸ›‘οΈ Running H-2... βœ… python3 confirmed installed ❌ boto3: not installed (import boto3 used in code) πŸ›‘οΈ Running H-4... βœ… Python syntax valid πŸ›‘οΈ Running H-9... ❌ TODO: S3 bucket name needs to be configured (line 23) πŸ›‘οΈ Running H-12... βœ… boto3: confirmed on PyPI (just not installed locally) [Result] PASS 4/5 β€” Need to install boto3 and handle TODO Recommend: pip install boto3, then retry ``` --- ### Example 2: Technical Documentation Quality Verification **Situation:** Agent wrote team API documentation, needs pre-deployment verification ``` [Execution] hallucination check on docs/api-reference.md --scope=fact,consistency [Result] H-3: ❌ https://api.example.com/v2/users β†’ HTTP 404 (Endpoint in documentation doesn't actually exist) H-5: ❌ "Average response time 23ms" β€” no source ❌ "99.9% uptime guaranteed" β€” no source/evidence H-8: ⚠️ Mixed usage of "user" / "client" / "end-user" detected β†’ Unification recommended [Fix Instructions] 1. Change /v2/users β†’ /v1/users or create the endpoint 2. Add "[internal measurement, 2026-03]" to response time figure 3. Replace uptime claim with SLA document link 4. Unify to "user" throughout ``` --- ### Example 3: Full Scan Before Release (CI Pipeline) **Situation:** Auto-run in GitHub Actions before PR merge ```yaml # .github/workflows/hallucination-guard.yml name: Hallucination Guard on: pull_request: types: [opened, synchronize] jobs: hg-check: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Run Hallucination Guard run: | # H-9, H-10 (completeness) checks echo "=== H-9: TODO/FIXME check ===" count=$(grep -rn "TODO\|FIXME\|HACK\|XXX" src/ | wc -l) [ "$count" -gt 0 ] && { echo "FAIL: $count found"; exit 1; } || echo "PASS" echo "=== H-10: Placeholder check ===" count=$(grep -rn "PLACEHOLDER\|YOUR_API_KEY\|CHANGEME\|TBD" src/ | wc -l) [ "$count" -gt 0 ] && { echo "FAIL: $count found"; exit 1; } || echo "PASS" echo "=== H-4: Syntax validity ===" find src -name "*.py" -exec python3 -m py_compile {} \; && echo "PASS" echo "βœ… Hallucination Guard complete" - name: Comment PR if: failure() uses: actions/github-script@v7 with: script: | github.rest.issues.createComment({ issue_number: context.issue.number, owner: context.repo.owner, repo: context.repo.repo, body: '❌ Hallucination Guard FAIL β€” Please fix TODO/placeholder/syntax errors.' }) ``` --- ## πŸ”§ Skill Execution Principles (For Agents) Agents that have read this skill follow these principles: 1. **On receiving "hallucination check" trigger** β€” Immediately start executing all 14 items 2. **When target is unclear** β€” Automatically set the current working file/directory as target 3. **On finding FAIL items** β€” Attempt to fix before reporting to user (state explicitly if fix is not possible) 4. **--quick flag** β€” Run only H-1, H-2, H-9, H-10, H-12 (essential 5) 5. **When --scope is specified** β€” `fact`(H-1~5), `consistency`(H-6~8), `completeness`(H-9~11), `hallucination`(H-12~14) 6. **Threshold** β€” Default 85% PASS (customizable via `.hallucination-guard.yaml`) 7. **Uncertain items** β€” Label as "needs verification:", no arbitrary judgment --- ## πŸ“Œ References and Sources - Python AST parsing: [Python Official Docs - ast module](https://docs.python.org/3/library/ast.html) - npm registry API: `https://registry.npmjs.org/` (public API) - PyPI JSON API: `https://pypi.org/pypi//json` (public API) - GNU grep patterns: POSIX standard compliant - The above APIs are publicly accessible as of March 2026 (needs verification: subject to future changes) --- *hallucination-guard v1.0.0 β€” by reikys* *"Proving trust through execution, not prompts"*