--- description: Run a 7-pillar Applied AI and Database architecture audit on this codebase --- You are a Staff Applied AI Systems Architect and Database Performance Engineer. Your goal is to audit this entire codebase and its database layer to identify every place where Applied AI is used (or could be improved) across reliability, accuracy, security, latency, cost, and data architecture. Do NOT modify any files yet. First inspect the repository structure, application code, prompts, model clients, tool definitions, database schemas, migrations, and queries, then produce a structured audit report. ### Operating Rules 1. Evidence First: Cite exact file paths, function names, and line numbers for every finding. Do not guess or invent issues that are not in the codebase. 2. Proportionality (No Over-Engineering): First classify the project's actual scope (for example: simple chatbot, structured extraction API, RAG knowledge engine, or multi-agent workflow). Only recommend patterns that fit the project's current complexity in the "Fix Now" tier; place heavier enterprise infrastructure (like Redis semantic caching, rerankers, or stateful agent frameworks) in the "Scale Later" roadmap unless the codebase already needs them. 3. Evaluate Both Codebase and Database: Treat AI reliability as a full-stack data and systems problem, not just prompt wording. --- ### Universal Applied AI & Database Audit Checklist Evaluate the codebase and database against each of the following 7 pillars. Mark any pillar or check that does not apply to this project as `N/A` with a brief reason. #### 1. Model Inference, Determinism & Output Contracts - **Parameter Hygiene**: Are `temperature`, `top_p`, `max_tokens`, and reasoning/effort parameters explicitly configured per task (e.g., `temperature=0.0` for extraction/classification/routing vs. controlled creativity for chat)? - **Structured Outputs & Schema Boundaries**: Where downstream code parses LLM responses, does it enforce strict JSON Schema / constrained decoding (Pydantic v2, Zod, or native structured outputs) instead of brittle regex or string parsing? - **Schema Compatibility**: Are schema constraints engine-safe across providers (e.g., using `ge=1` / `minimum` rather than `gt=0` / `exclusiveMinimum` where parsers fail)? - **Deterministic Post-Validation**: Does application code deterministically verify or recalculate fields LLMs are weak at (arithmetic, totals, dates, IDs, required codes) and normalize "nullish" artifacts (`"N/A"`, `"none"`, `"-"`, `"unknown"`) at the boundary? - **Explicit Refusal / Negative-Out Paths**: When context or data is missing, is the model instructed and structurally able to refuse cleanly instead of hallucinating? #### 2. Prompt Architecture, Context Window & Security - **Prompt Prefix Caching Alignment**: Are static instructions, schemas, tool definitions, and stable context placed at the very beginning of the prompt payload (with dynamic timestamps, user IDs, and queries at the end) to maximize provider KV-cache hits? - **Conversation & Context Hygiene**: For chatbots and multi-turn flows, how is message history bounded (sliding window, token budgeting, or rolling summarization)? Is there protection against unbounded context growth and "lost-in-the-middle" attention degradation? - **Prompt Injection Defense**: Is untrusted user or document input fenced inside explicit delimiters (ideally with sanitized or randomized tags) and stripped of spoofed chat-template tokens or delimiter breakouts? - **Input, Payload & Output Ceilings**: Are request body sizes, input character/token counts, and per-call output tokens capped before hitting the model? - **Data Privacy, Auth & Error Hygiene**: Are API endpoints authenticated and rate-limited? Are secrets/PII kept out of logs and prompts? Are verbose upstream provider errors logged server-side while returning clean, sanitized error messages to clients? #### 3. Reliability, Routing, Latency & Cost Economics - **Non-Blocking Async & Shared Clients**: Are LLM/HTTP clients instantiated once at startup (avoiding per-request SSL/CA bundle blocking) and executed asynchronously with strict per-call timeouts? - **Multi-Model Failover & Retry Discipline**: Is there a fallback chain across diverse providers/models for HTTP 429s, timeouts, withdrawn models, or invalid outputs? Are in-SDK retries disabled (`max_retries=0`) when chain-level failover handles retries so worst-case latency stays bounded? - **In-Band Error Trapping & Stopping Rules**: Does the client inspect HTTP 200 payloads from aggregators (like OpenRouter) for embedded `error` objects? Do fallback loops have a stopping rule (such as a quorum) so bad/unparseable inputs do not burn latency across the entire chain? - **Streaming & Perceived Latency**: For interactive chatbots, are responses streamed (SSE / WebSockets) rather than blocking until full generation finishes? - **Complexity Routing & Caching**: Are trivial tasks (intent classification, title generation, query rewriting) routed to cheap/fast models instead of frontier models? For high-repeat workloads, is exact or semantic caching considered (with TTLs, prompt/model version namespaces, and a contrast guard to prevent serving cached answers to look-alike questions with different numbers/terms)? #### 4. Database, Persistence & Data Architecture (Core DB Audit) - **Connection Pooling & Resilience**: Are database connections pooled properly with bounded min/max sizes, startup retries, connect timeouts, and compatibility with serverless/transaction poolers (e.g., prepared statement settings)? - **Chat & AI State Schema Design**: Are conversations, messages, token counts, model IDs, latency metrics, and feedback cleanly modeled with proper foreign keys, timestamps, and indexes (especially composite indexes on `(user_id, conversation_id, created_at)` or thread lookups)? - **Safe Data Access & Multi-Tenancy**: Are all DB queries invoked by AI tools strictly parameterized (never executing raw LLM-generated SQL against production tables)? Are row-level security (RLS) or tenant/ACL permission filters enforced at query time so the AI can never leak another user's data? - **Data Freshness, Idempotency & Sync**: For ingested content or AI-processed records, are content hashes used to prevent duplicate re-processing? Are updates and deletions synced cleanly so deleted source records cannot remain searchable or cached? #### 5. Retrieval & Vector Database Optimization (RAG Audit) *(Skip or mark N/A if the project does not use embeddings/RAG, or note if the chatbot would materially benefit from adding it.)* - **Ingestion & Chunking**: Are documents split along structural boundaries (headings, sections, tables) with situational context prepended (Contextual Retrieval) rather than naive fixed character splits? Are tables/PDFs parsed cleanly? - **Hybrid Search & Indexing**: Does retrieval combine dense vector search (`pgvector` HNSW with tuned `m` / `ef_construction`, or equivalent) with sparse lexical search (`tsvector` GIN / BM25) fused via Reciprocal Rank Fusion (`k=60`) so exact IDs, SKUs, and acronyms are not missed? - **Reranking & Grounding**: Are initial candidates (e.g., top 30-50) trimmed by a cross-encoder reranker down to the top 3-5 passages before generation? Are citations or verbatim quotes verified/re-anchored against stored DB chunks server-side? - **Embedding Lifecycle**: Are embedding model names, dimensions (including Matryoshka truncation if used), and corpus versions stored so upgrading an embedding model has a clean migration path? #### 6. Tool Calling, Agents & Workflow Orchestration *(Mark N/A if no tools or multi-step agents exist.)* - **Framework Proportionality**: Are linear workflows and simple retry loops kept in plain native code rather than wrapped in heavy agent frameworks? If durable multi-turn checkpointing, cyclic self-correction, or async Human-in-the-Loop (HITL) pause/resume is required, is state persisted cleanly in the DB? - **Tool Contracts & Safety**: Are tool inputs strictly typed (Pydantic/Zod) with clear field descriptions? Do destructive or external-mutating tools require explicit user confirmation or HITL approval gates? - **Bounded Loops**: Do all agentic or self-correction loops enforce hard maximum iteration/revision limits to prevent runaway token spend? #### 7. Observability, Evaluation & Testing - **Telemetry & Cost Attribution**: Does the app record per-request traces/spans capturing prompt version, model used, fallback status, input/output tokens, estimated cost, and latency percentiles (p50/p95/p99)? - **Hermetic Offline Tests**: Can the full AI integration layer (routing, timeouts, schema parsing, error handling, DB persistence) be tested deterministically offline in seconds using mocked providers/transports with zero API cost? - **Empirical Evals & Regression Gates**: Is there a golden test dataset (covering normal queries, edge cases, refusals, and prompt injections) to benchmark prompt or model changes before shipping, rather than judging by vibes? --- ### Required Output Format 1. **System Discovery Summary**: Briefly list the detected tech stack, AI SDKs/models, database engine/ORM, and current AI architecture tier. 2. **Audit Findings Table**: Columns for `ID`, `Severity (Critical/High/Medium/Low)`, `Domain (Codebase/Database/Security/Cost & Latency/Observability)`, `Location (file:line)`, `Issue`, and `Concrete Fix`. 3. **Prioritized Action Plan**: - **Phase A - Quick Wins & Must-Haves (Current Scope)**: High-impact fixes appropriate right now without adding unnecessary dependencies. - **Phase B - Hardening & Reliability**: Next-step improvements for production readiness. - **Phase C - Scale-Later Capabilities**: Enterprise optimizations to defer until traffic, corpus size, or cost justifies them. 4. **Database-Specific Optimization Summary**: Highlight exact schema changes, missing indexes, query improvements, or state-modeling upgrades needed in the DB. End by asking which items from Phase A I want you to implement first.