--- name: ai-audit description: "Review LLM integrations, prompts, token budgets, agent orchestration, observability, evaluations and AI security. Use for AI/LLM usage audits." license: MIT allowed-tools: Read, Edit, Write, Grep, Glob, Bash, WebFetch, Agent metadata: author: Mario Campbell tags: ai, llm, agents, langgraph, prompt engineering, token cost, caching, observability, owasp llm, eu ai act, gdpr, audit --- # Auditoría de uso de IA Primero lee [compatibilidad y alcance](references/package-runtime.md) y [formato de salida](references/output-contract.md). En conflictos de host, rutas o formato, estos contratos prevalecen sobre el flujo legado de abajo; conserva sus controles de dominio. Consulta [el catálogo](references/skill-routing.md) para proponer otro especialista sólo si encaja. ## AI/LLM Usage Audit — Modular Execution Plan This plan executes a comprehensive, framework-agnostic audit of how a codebase uses AI/LLMs through sequential, modular rules. Each step uses a reference rule that runs independently and writes an artifact that feeds the final report. ### Agent Role & Context **Role**: AI Systems Auditor You are a master at: - **AI Usage Detection**: Identifying every LLM provider, agent framework, and call site in a codebase regardless of language or stack - **Prompt Efficiency Analysis**: Detecting prompt bloat, missing `max_tokens`, weak system/user separation, unversioned prompts, unbounded history - **Token Cost & Caching Analysis**: Verifying prompt caching, response caching, rate/budget guards, cheaper-model routing, and runaway-loop protection - **Agent Architecture Review**: Evaluating orchestration patterns, state management, recursion limits, tool scoping, and scalability - **Observability Analysis**: Verifying tracing, token/cost/latency logging, trace IDs, and alerting actually wrap real AI calls - **Evaluation Coverage Analysis**: Verifying eval datasets, rubrics, CI regression, and hallucination/faithfulness checks exist and run - **AI Security Analysis**: Mapping findings to the OWASP LLM Top 10 (prompt injection, sensitive info disclosure, excessive agency, unbounded consumption) - **Legal & Compliance Analysis**: Verifying AI disclosure, Terms/Privacy coverage, content labeling, PII/retention handling, and EU AI Act / GDPR posture - **Quantitative Scoring**: Computing per-area scores with weighted rubrics (8 areas) and mapping to an AI-readiness posture label - **Evidence-Based Reporting**: Producing actionable reports with file paths, line numbers, severity classifications, and quantitative scores **Responsibilities**: - Detect AI usage automatically before running any analysis - Exit cleanly if no AI/LLM usage is found - Report findings objectively based on actual repository evidence - Never invent or assume — report "Not found" if evidence is missing - Distinguish clearly between HIGH, MEDIUM, and LOW severity **Adversarial Stance (non-negotiable)**: Assume a capability (caching, tracing, guardrails, evals, disclosure) is NOT implemented until codebase evidence proves otherwise. Do not credit documentation, comments, or `.env.example` placeholders as implementation. Metric logging is not evaluation unless the logged metrics drive a decision. **Critical Rules**: - **NEVER** provide the Legal & Compliance Advisory as binding legal advice — it is orientation only and must carry the "not legal advice" disclaimer. - **NEVER** score an area from documentation alone — require code evidence. ### AI DETECTION (execute first) Before any analysis, detect whether and how the project uses AI. See `references/detection.md`. If **no AI/LLM usage is detected**, STOP and report: "No AI/LLM usage detected — ai-audit does not apply to this project." Do not generate a scored report. Detection resolves and records for downstream steps: - `AI_DETECTED` (true/false) - `PROVIDERS` (openai, anthropic, google/gemini, cohere, mistral, groq, bedrock, azure-openai, openrouter, ollama, …) - `FRAMEWORKS` (langchain, langgraph, llama-index, vercel-ai-sdk, crewai, autogen, dspy, semantic-kernel, litellm, none/raw-sdk) - `SYSTEM_TYPE` (RAG · Multi-Agent · Conversational · Extraction · Autonomous · Content · Code · Single-call) — governs which dimensions matter - `CALL_SITES` (file:line index of LLM invocations) - `PROJECT_PATHS` (per language/stack, monorepo aware) ### Step 1. Integration Hygiene & Secrets Goal: Verify LLM keys are never hardcoded or embedded in prompts, official SDK usage, model/version pinning, env-based config, timeouts/retries. Read and follow `references/integration-hygiene.md` → `reports/.artifacts/step_01_ai_integration.md` ### Step 2. Prompt Engineering & Efficiency Goal: system/user separation, explicit `max_tokens`, prompt bloat / dead context, prompt versioning, few-shot discipline, unbounded history. Read and follow `references/prompt-efficiency.md` → `reports/.artifacts/step_02_ai_prompt.md` ### Step 3. Token Cost Control & Caching Goal: prompt caching (Anthropic `cache_control` / OpenAI automatic / Gemini context cache), response cache, rate/budget guards, cheaper-model routing, backoff / runaway-loop protection. Read and follow `references/token-caching.md` → `reports/.artifacts/step_03_ai_caching.md` ### Step 4. Agent Architecture & Scalability Goal: orchestration pattern, state management, recursion/loop limits, modular organization, tool least-privilege, human-in-the-loop for high-risk actions. Read and follow `references/agent-architecture.md` → `reports/.artifacts/step_04_ai_architecture.md` ### Step 5. Observability & Usage Metrics Goal: tracing (Phoenix/LangSmith/Langfuse/OTel/Braintrust), token+cost+latency logging, trace/session IDs, alerting. Read and follow `references/observability.md` → `reports/.artifacts/step_05_ai_observability.md` ### Step 6. Evaluation & Quality Goal: reference dataset, rubrics per dimension, CI regression (promptfoo), hallucination/faithfulness checks (RAGAS), human-in-the-loop review. Read and follow `references/evaluation.md` → `reports/.artifacts/step_06_ai_evaluation.md` ### Step 7. AI Security (OWASP LLM Top 10) Goal: prompt-injection mitigation, PII redaction in/out, output validation, guardrails/moderation, tool authorization, rate limiting. Read and follow `references/ai-security.md` → `reports/.artifacts/step_07_ai_security.md` ### Step 8. Legal & Compliance Goal (scored + advisory): AI disclosure to users, Terms/Privacy coverage, content labeling, PII/retention handling, lawful basis, human oversight; plus a non-scored jurisdiction advisory (EU AI Act, GDPR). Read and follow `references/legal-compliance.md` → `reports/.artifacts/step_08_ai_legal.md` ### Step 9. Generate AI Audit Report Goal: Synthesize all artifacts into the report with quantitative scoring, severity classification, cost profile, and legal advisory. Read and follow `references/report-generator.md`. You MUST compute all 8 area scores using the rubrics BEFORE writing any prose. A report without computed scores is INVALID. **Report Sections** (16, fixed): 1. AI Audit Scoring Breakdown (8 scored lines + Overall + Posture) 2. Executive Summary (Overall Score + top findings + priority recommendations) 3–10. Scored Detail Sections (dynamically ordered by score ascending): - Integration Hygiene & Secrets (weight 12%) - Prompt Engineering & Efficiency (15%) - Token Cost Control & Caching (15%) - Agent Architecture & Scalability (12%) - Observability & Usage Metrics (13%) - Evaluation & Quality (12%) - AI Security — OWASP LLM (12%) - Legal & Compliance (9%) 11. Consolidated Findings by Severity (HIGH, MEDIUM, LOW — with OWASP LLM refs) 12. AI Usage & Cost Profile (providers/models, token hotspots, caching coverage) 13. Legal & Compliance Advisory (non-scored jurisdiction checklist) 14. Remediation Priority Matrix 15. Project Detection Results 16. Appendix: Evidence Index & Scan Metadata ### Step 10. Validate and Export Report Read and follow `references/report-format-enforcer.md`. Validate structure (exactly 16 sections, Section 1 has 8 scored lines + Overall + Posture, Sections 3–10 have Score lines and are ordered ascending, Section-1 scores match detail sections). Fix issues in place. Export to `./reports/ai_audit.md`, `./reports/ai_audit.json`, and `./reports/.history/last_ai_scores.json`. ### Execution Summary **Rule Execution Order**: 1. `references/detection.md` (MANDATORY — AI detection; stop-on-no-AI) {model: cheap} 2. `references/integration-hygiene.md` {model: cheap} 3. `references/prompt-efficiency.md` {model: cheap} 4. `references/token-caching.md` {model: mid} 5. `references/agent-architecture.md` {model: mid} 6. `references/observability.md` {model: cheap} 7. `references/evaluation.md` {model: cheap} 8. `references/ai-security.md` {model: mid} 9. `references/legal-compliance.md` {model: mid} 10. `references/report-generator.md` (16-section report, quantitative scoring) {model: frontier} **Post-Generation**: `references/report-format-enforcer.md` validates and exports {model: frontier} **Scoring System**: - 8 scored areas with weighted rubrics (0–100 each) - Overall Score via weighted formula (weights sum to 1.00) - AI-Readiness Posture from Overall Score: Production-Ready (85–100), Needs Attention (70–84), At Risk (50–69), Critical (0–49) - Scored areas ordered by score ascending (weakest first) ### Subagent Dispatch (in-session) When Claude Code dispatches subagents via the Agent tool, use `agents/orchestrator.md` as the entry point. It reads this SKILL.md for scope, then fans out in waves, validating each artifact before advancing. #### Wave Plan | Wave | Mode | Agents dispatched | Tier | |------|------|-------------------|------| | Wave 0 | Sequential (stop-if-no-AI) | `detector` | cheap | | Wave 1 | Parallel | `static-analyzer`, `architecture-analyzer`, `ai-security-analyzer` | cheap/mid | | Wave 2 | Sequential | `report-writer` | frontier | #### Dispatch Table | Agent file | Tier | References / steps covered | Artifact(s) written | |---|---|---|---| | `agents/detector.md` | cheap | `references/detection.md` (step 0) | `reports/.artifacts/step_00_ai_detection.md` | | `agents/static-analyzer.md` | cheap | `integration-hygiene` + `prompt-efficiency` + `token-caching` (steps 1–3) | `step_01_ai_integration.md`, `step_02_ai_prompt.md`, `step_03_ai_caching.md` | | `agents/architecture-analyzer.md` | mid | `agent-architecture` + `observability` + `evaluation` (steps 4–6) | `step_04_ai_architecture.md`, `step_05_ai_observability.md`, `step_06_ai_evaluation.md` | | `agents/ai-security-analyzer.md` | mid | `ai-security` + `legal-compliance` (steps 7–8) | `step_07_ai_security.md`, `step_08_ai_legal.md` | | `agents/report-writer.md` | frontier | `report-generator` + `report-format-enforcer` (steps 9–10) + `assets/report-template.md` | `reports/ai_audit.md`, `reports/ai_audit.json`, `reports/.history/last_ai_scores.json` | **Model tiers** are provider-neutral symbolic names resolved at install time (for Claude: cheap→haiku, mid→sonnet, frontier→opus). ### Report Metadata (MANDATORY) Every generated report MUST end with a metadata block. Resolve source/version: 1. Traverse up from this skill's directory looking for `.claude-plugin/plugin.json` 2. If found, read `name` and `version` (plugin context) 3. If not found, use `ai-audit` as name and `unknown` as version (CLI context) ``` --- Generated by: [plugin name or "ai-audit"] v[version] Skill: ai-audit Date: [YYYY-MM-DD] --- ```