A multi-LLM deliberation system where multiple LLMs collaboratively answer questions through peer review and synthesis. Available as a Python library, MCP server, or HTTP API.
## What is This? Instead of asking a single LLM for answers, this MCP server: 1. **Stage 1**: Sends your question to multiple LLMs in parallel (GPT, Claude, Gemini, Grok, etc.) 2. **Stage 2**: Each LLM reviews and ranks the other responses (anonymized to prevent bias) 3. **Stage 3**: A Chairman LLM synthesizes all responses into a final, high-quality answer ## Quick Deploy Deploy your own LLM Council instance: | Platform | Deploy | Best For | |----------|--------|----------| | **Railway** | [](https://railway.com/deploy/llm-council?referralCode=K9dsYj) | Production, webhooks | | **Render** | [](https://render.com/deploy?repo=https://github.com/amiable-dev/llm-council) | Evaluation, free tier | **Required Environment Variables:** - `OPENROUTER_API_KEY` - Your [OpenRouter](https://openrouter.ai) API key - `LLM_COUNCIL_API_TOKEN` - A secure token for API authentication (generate with `openssl rand -hex 16`) > **Note**: Railway is recommended for [n8n integration](https://llm-council.dev/integrations/n8n/) (no cold-start). Render Free tier spins down after 15 minutes which may cause webhook timeouts. For detailed deployment instructions, see the [Deployment Guide](https://llm-council.dev/deployment/). ## Credits & Attribution This project is a derivative work based on the original [llm-council](https://github.com/karpathy/llm-council) by Andrej Karpathy. Karpathy's original README stated: > "I'm not going to support it in any way, it's provided here as is for other people's inspiration and I don't intend to improve it. Code is ephemeral now and libraries are over, ask your LLM to change it in whatever way you like." ...the irony of producing a derivative work that packages the core concept for broader use via the Model Context Protocol! ## Installation ```bash # Recommended (isolated tool install; puts `llm-council` on PATH): uv tool install "llm-council-core[mcp,secure]" # Or with pip: pip install "llm-council-core[mcp,secure]" ``` For core library only (no MCP server): ```bash pip install llm-council-core ``` > **Why `[mcp,secure]` and not just `[mcp]`?** The `secure` extra provides > keychain support. Without it, `llm-council setup-key` is unavailable AND a > key already stored in your keychain is **silently ignored** — the server > starts but `council_health_check` reports `api_key_configured: false`. > Include both extras unless you only ever use environment variables. ## Setup ### 1. Choose Your Gateway The council supports three gateway options for accessing LLMs: | Gateway | Best For | API Keys Needed | |---------|----------|-----------------| | **OpenRouter** (default) | Easiest setup, 100+ models via single key | `OPENROUTER_API_KEY` | | **Requesty** | BYOK mode, analytics, cost tracking | `REQUESTY_API_KEY` + provider keys | | **Direct** | Maximum control, direct provider APIs | Provider keys (Anthropic, OpenAI, Google) | **Quick Start (OpenRouter):** ```bash # Sign up at openrouter.ai and get your API key export OPENROUTER_API_KEY="sk-or-v1-..." ``` **Direct Provider Access:** ```bash # Use your existing provider API keys directly export ANTHROPIC_API_KEY="sk-ant-..." export OPENAI_API_KEY="sk-..." export GOOGLE_API_KEY="..." export LLM_COUNCIL_DEFAULT_GATEWAY=direct ``` **Requesty with BYOK:** ```bash export REQUESTY_API_KEY="..." export ANTHROPIC_API_KEY="sk-ant-..." # Your own key, routed through Requesty export LLM_COUNCIL_DEFAULT_GATEWAY=requesty ``` ### 2. Store Your API Keys Securely Choose one of these options (in order of recommendation): #### Option A: System Keychain (Most Secure) Store keys encrypted in your OS keychain: ```bash # Install with keychain support pip install "llm-council-core[mcp,secure]" # Store key securely (prompts for key, no echo) llm-council setup-key # For CI/CD automation, pipe from stdin: echo "$OPENROUTER_API_KEY" | llm-council setup-key --stdin ``` #### Option B: Environment Variables Set in your shell profile (`~/.zshrc`, `~/.bashrc`): ```bash # OpenRouter (default gateway) export OPENROUTER_API_KEY="sk-or-v1-..." # Or use direct provider APIs export ANTHROPIC_API_KEY="sk-ant-..." export OPENAI_API_KEY="sk-..." export GOOGLE_API_KEY="..." ``` #### Option C: Environment File Create a `.env` file (ensure it's in `.gitignore`): ```bash # For OpenRouter echo "OPENROUTER_API_KEY=sk-or-v1-..." > .env # Or for direct APIs cat > .env << 'EOF' ANTHROPIC_API_KEY=sk-ant-... OPENAI_API_KEY=sk-... GOOGLE_API_KEY=... LLM_COUNCIL_DEFAULT_GATEWAY=direct EOF ``` > **Security Note**: Never put API keys in command-line arguments or JSON config files that might be committed to version control. ### 3. Customize Models (Optional) You can customize which models participate in the council using three methods (in priority order): #### Option 1: Environment Variables (Recommended) ```bash # Comma-separated list of council models export LLM_COUNCIL_MODELS="openai/gpt-4,anthropic/claude-3-opus,google/gemini-pro" # Chairman model (synthesizes final response) export LLM_COUNCIL_CHAIRMAN="anthropic/claude-3-opus" # Skip chairman synthesis, return the top-ranked Stage 1 response directly # (cheaper/faster, but NEVER enable this for council-verify/council-gate -- # no verdict is computed; see docs/guides/verify.md) export LLM_COUNCIL_CHAIRMAN_DISABLED=false ``` #### Option 2: YAML Configuration (Recommended) Create `llm_council.yaml` in your project root or `~/.config/llm-council/llm_council.yaml`: ```yaml council: # Tier configuration (ADR-022) tiers: default: high pools: quick: models: - openai/gpt-4o-mini - anthropic/claude-3-5-haiku-20241022 timeout_seconds: 30 balanced: models: - openai/gpt-4o - anthropic/claude-3-5-sonnet-20241022 timeout_seconds: 90 high: models: - openai/gpt-4o - anthropic/claude-opus-4-7 - google/gemini-3-pro timeout_seconds: 180 # Triage configuration (ADR-020) triage: enabled: false wildcard: enabled: true prompt_optimization: enabled: true # Gateway configuration (ADR-023, ADR-025a) gateways: default: openrouter fallback: enabled: true chain: [openrouter, ollama] # Can use Ollama as fallback # Provider-specific configuration providers: ollama: enabled: true base_url: http://localhost:11434 timeout_seconds: 120.0 hardware_profile: recommended # minimum|recommended|professional|enterprise openrouter: enabled: true base_url: https://openrouter.ai/api/v1/chat/completions requesty: enabled: true base_url: https://router.requesty.ai/v1/chat/completions api_key: ${REQUESTY_API_KEY} # Per-gateway model-id translation (e.g. Requesty rejects OpenRouter's # ":free" suffix and uses provider-prefixed names for some models) model_name_map: requesty: "some/model:free": "some/model" # Webhook notifications (ADR-025a) webhooks: enabled: false # Opt-in timeout_seconds: 5.0 max_retries: 3 https_only: true default_events: - council.complete - council.error observability: log_escalations: true ``` **Priority**: YAML config > Environment variables > Defaults #### Option 3: JSON Configuration (Legacy) Create `~/.config/llm-council/config.json`: ```json { "council_models": [ "openai/gpt-4-turbo", "anthropic/claude-3-opus", "google/gemini-pro", "meta-llama/llama-3-70b-instruct" ], "chairman_model": "anthropic/claude-3-opus", "synthesis_mode": "consensus", "exclude_self_votes": true, "style_normalization": false, "max_reviewers": null } ``` #### Option 4: Use Defaults If you don't configure anything, these defaults are used: - Council: GPT-5.1, Gemini 3 Pro, Claude Sonnet 4.5, Grok 4 - Chairman: Gemini 3 Pro - Mode: consensus - Self-vote exclusion: enabled **Finding Models**: - OpenRouter: [openrouter.ai/models](https://openrouter.ai/models) - Anthropic: [docs.anthropic.com/models](https://docs.anthropic.com/en/docs/about-claude/models) - OpenAI: [platform.openai.com/docs/models](https://platform.openai.com/docs/models) - Google: [ai.google.dev/gemini-api/docs/models](https://ai.google.dev/gemini-api/docs/models/gemini) ## Usage ### With Claude Code ```bash # First, store your API key securely (one-time setup) llm-council setup-key # Then add the MCP server (key is read from keychain or environment) claude mcp add --transport stdio llm-council --scope user -- llm-council ``` Then in Claude Code: ``` Consult the LLM council about best practices for error handling ``` ### With Claude Desktop First ensure your API key is available (via keychain, environment variable, or `.env` file). Add to `~/Library/Application Support/Claude/claude_desktop_config.json`: ```json { "mcpServers": { "llm-council": { "command": "llm-council" } } } ``` > **Note**: No `env` block needed—the key is resolved from your system keychain or environment automatically. > **Client timeouts (`high`/`reasoning` tiers):** deliberation can exceed MCP > clients' default transport timeout (~60s). Set `MCP_TIMEOUT` (milliseconds) > in your client config to at least the tier budget (`high` ≈ 180000, > `reasoning` ≈ 600000) or the client cuts the connection mid-deliberation. ### With Other MCP Clients Any MCP client can use the server by running: ```bash llm-council ``` ### Quality Benchmark (`llm-council bench`) A golden dataset (`bench/dataset/v1/`, 20 items across coding/reasoning/factual/judgment) guards council quality against drift (ADR-048). **Every run costs real API spend** — it runs nightly/on-demand, never per-PR: ```bash llm-council bench run [--items id1,id2] [--max-usd 2.00] # exit 0 ok / 1 drift / 2 aborted-partial llm-council bench baseline --set # snapshot the last run as baseline llm-council bench report [--format md|json] # render the last run + baseline deltas ``` Caps: `LLM_COUNCIL_BENCH_MAX_USD` per run (default $2.00, enforced against actuals mid-run with graceful partial abort) and `LLM_COUNCIL_BENCH_MONTHLY_USD` month-to-date guard (default $30, from run artefacts under `.council/bench/runs/`). Dataset governance: [`bench/dataset/GOVERNANCE.md`](bench/dataset/GOVERNANCE.md). ### Streaming Deliberation (SSE) `GET /v1/council/stream?prompt=...` streams the deliberation live (ADR-046). Beyond the coarse stage events, the stream now emits **rich per-model events**, each wrapped in a versioned envelope (`v: 1`, `session_id`, `ts`, monotonic `seq`): | Event | Payload | When | |---|---|---| | `stage1.response` | `model`, `response`, `latency_ms`, `usage` | each model's answer, as it lands | | `stage2.review` | `reviewer`, `ranking`, `parse_ok` | each peer review, as it lands | | `consensus.early_termination` | `leader`, `votes_saved`, … | ADR-044 early consensus (flag-on) | | `stage3.start` | `chairman` | synthesis begins | | `synthesis.delta` | `text` | chairman tokens, live (opt-in: `&stream_tokens=true`) | | `council.complete` | full result | terminal | | `council.error` | `error` | terminal | Schema changes are additive-only within `v: 1`. Non-streaming endpoints are byte-identical — the rich events are wired only when a stream consumer is attached. Chairman token streaming (`stream_tokens=true`) assembles the identical final result object as the non-streamed path; if the stream transport fails, stage 3 silently falls back to the regular call. Streamed synthesis reports usage as *unknown* (the streaming wire protocol carries no usage data) — ADR-011 semantics, never a fabricated cost. ### Server Discovery (MCP Server Card) The HTTP server (`llm-council serve`) publishes an [MCP Server Card](https://github.com/modelcontextprotocol/experimental-ext-server-card) — public discovery metadata generated from the live tool registry — at: - `/server-card` (SEP-2127 experimental-extension path) - `/.well-known/mcp/server-card.json` (SEP-1649 well-known path) A static copy for registry submission lives at [`server-card.json`](server-card.json) and is regenerated with: ```bash llm-council server-card --output server-card.json ``` The card follows the SEP-2127 RC schema (v1); it will be re-validated against the final schema after the MCP 2026-07-28 spec release. ## Available Tools ### `consult_council` Ask the LLM council a question and get synthesized guidance. Every response includes a one-line **Cost & Tokens** summary (total tokens, plus USD cost when the provider reports it); pass `include_details=true` for the full per-model/per-stage breakdown. **Arguments:** - `query` (string, required): The question to ask the council - `confidence` (string, optional): Response quality level (default: "high") - `"quick"`: fast/mini-class models, ~30s — simple questions - `"balanced"`: mid-tier models, ~90s — most questions - `"high"`: the full frontier council, ~180s — important decisions - `"reasoning"`: deep-thinking models, ~600s — complex analysis The concrete model pools are configured in `llm_council.yaml` (or `LLM_COUNCIL_MODELS_*`) — run `council_health_check` to see exactly what YOUR council runs; any list printed here would drift. - `include_details` (boolean, optional): Include individual model responses, rankings, and the full per-model/per-stage **Cost & Tokens** breakdown (default: false) - `verdict_type` (string, optional): Type of verdict to render (default: "synthesis") - `"synthesis"`: Free-form natural language synthesis - `"binary"`: Go/no-go decision (approved/rejected) with confidence score - `"tie_breaker"`: Chairman resolves deadlocked decisions - `include_dissent` (boolean, optional): Extract minority opinions from Stage 2 (default: false) - `evidence` (list, optional): Caller-supplied grounding context — retrieved snippets, tool output, repo files — rendered into the question for every council member under a per-tier budget (ADR-042) - `on_partial` (string, optional): What to do when the council **doesn't complete** (default: `"synthesise"`) - `"synthesise"`: answer anyway, with the shortfall declared in the heading and above the content - `"error"`: return a `council_incomplete` blob **instead of** an answer - `"return_raw"`: return the surviving members' responses individually, with no chairman synthesis over the top **Reading a degraded answer.** A run can complete with fewer models than it asked for. The heading says which kind of answer you got — `### Chairman's Synthesis` is reserved for a full deliberation, and a chairman-failed fallback is labelled as one model's raw response. Every response also ends with a machine-readable `### Council Status` block so automation can branch on `status` / `peer_review` without parsing prose. This matters because the models that time out are the slowest, hence generally the strongest — a short council is skewed, not merely smaller. See the [MCP guide](https://llm-council.dev/guides/mcp/#reading-a-degraded-answer). **Example:** ``` Use consult_council to ask: "What are the trade-offs between microservices and monolithic architecture?" ``` **Example with confidence level:** ``` Use consult_council with confidence="quick" to ask: "What's the syntax for a Python list comprehension?" ``` **Example with Jury Mode (binary verdict):** ``` Use consult_council with verdict_type="binary" and include_dissent=true to ask: "Should we approve this PR that adds caching?" ``` ### `council_health_check` Verify the council is working before expensive operations. Returns API connectivity status, the models a real run would use, and each tier's server budget. **Arguments:** - `deep` (boolean, optional): Probe the configured **chairman** model as well as general API reachability (default: **true**). Costs one small extra call (~2-3s). Pass `deep=false` for a cheap connectivity-only ping. Skipped automatically if general connectivity has already failed. - `tier` (string, optional): Report readiness for the tier a real run would use (default: `"high"`) **Returns:** - `api_key_configured`: Whether an API key was found - `key_source`: Where the key came from ("environment", "keychain", or "config_file") - `default_tier` / `council_size` / `models`: The tier reported on, and the models a real `consult_council` run at that tier would actually use — resolved from the tier pool, not the flat config list - `chairman_model` / `chairman_connectivity`: The synthesis model, and its probe result when `deep` ran - `estimated_duration`: Each tier's configured **server budget** (an upper bound, not a typical latency) - `config_warnings`: Present only when two config surfaces disagree — e.g. `council.models` naming different models than the resolved tier pool - `ready`: Whether the council is ready to accept queries - `ready_scope`: What `ready` is a statement about — `chairman_probed` (default) or `connectivity_only`. Read this before trusting `ready`: a connectivity-only check answers *"is the API up"*, not *"will the council complete"*, and those diverge during a chairman outage **Example:** ``` Run council_health_check to verify the LLM council is working ``` ## How It Works The council uses a multi-stage process inspired by ensemble methods and peer review: ``` User Query ↓ ┌─────────────────────────────────────────────┐ │ STAGE 1: Independent Responses │ │ • All council models queried in parallel │ │ • No knowledge of other responses │ │ • Graceful degradation if some fail │ └─────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────┐ │ STAGE 1.5: Style Normalization (optional) │ │ • Rewrites responses in neutral style │ │ • Removes AI preambles and fingerprints │ │ • Strengthens anonymization │ └─────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────┐ │ STAGE 2: Anonymous Peer Review │ │ • Responses labeled A, B, C (randomized) │ │ • XML sandboxing prevents prompt injection │ │ • JSON-structured rankings with scores │ │ • Self-votes excluded from aggregation │ └─────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────┐ │ STAGE 3: Chairman Synthesis │ │ • Receives all responses + rankings │ │ • Consensus mode: single best answer │ │ • Debate mode: highlights disagreements │ └─────────────────────────────────────────────┘ ↓ Final Response + Metadata ``` This approach helps surface diverse perspectives, identify consensus, and produce more balanced, well-reasoned answers. ## Advanced Features ### Compute-Optimal Deliberation (ADR-044) Three opt-in features (all **off by default**; flag-off behaviour is unchanged) let the council spend deliberation compute adaptively. Every routing influence is audited via LayerEvents — never a silent change. ```bash # Blend each model's live performance history (quality per ADR-026, cost per # ADR-011) into tier selection. Cold-start models stay on static estimates. export LLM_COUNCIL_PERFORMANCE_SELECTION=true # Cancel remaining Stage-2 reviewers once the leading response's Borda margin # is mathematically unassailable. When off (default), runs in SHADOW MODE: # the would-have-saved votes/cost are logged so you can measure the win first. export LLM_COUNCIL_EARLY_CONSENSUS=true # Graduated deliberation depth: single model → mini-council → full council, # escalating only on weak consensus signals; escalations can be vetoed by the # opt-in budget enforcer (auditable, never silent). export LLM_COUNCIL_GRADUATED_DEPTH=true ``` ### Self-Vote Exclusion By default, each model's vote for its own response is excluded from the aggregate rankings. This prevents self-preference bias. ```bash export LLM_COUNCIL_EXCLUDE_SELF_VOTES=true # default ``` ### Synthesis Modes **Consensus Mode** (default): Chairman synthesizes a single best answer. **Debate Mode**: Chairman highlights areas of agreement, key disagreements, and trade-offs between perspectives. ```bash export LLM_COUNCIL_MODE=debate ``` ### Style Normalization (Stage 1.5) Optional preprocessing that rewrites all responses in a neutral style before peer review. This strengthens anonymization by removing stylistic "fingerprints" that might allow models to recognize each other. ```bash export LLM_COUNCIL_STYLE_NORMALIZATION=true export LLM_COUNCIL_NORMALIZER_MODEL=google/gemini-2.0-flash-001 # fast/cheap ``` ### Stratified Sampling (Large Councils) For councils with more than 5 models, you can limit the number of reviewers per response to reduce API costs (O(N²) → O(N×k)): ```bash export LLM_COUNCIL_MAX_REVIEWERS=3 ``` ### Reliability Features The council includes built-in reliability features for long-running operations: **Tiered Timeouts**: Graceful degradation under time pressure: - Per-model soft deadline: 15s (start planning fallback) - Per-model hard deadline: 25s (abandon slow model) - Global synthesis trigger: 40s (must start synthesis) - Response deadline: 50s (must return something) **Partial Results**: If some models timeout, the council returns results from the models that responded, with a clear warning indicating which models were excluded. **Confidence Levels**: Use the `confidence` parameter to trade off speed vs. thoroughness: - `quick`: ~20-30 seconds (fastest models) - `balanced`: ~45-60 seconds (most models) - `high`: ~60-90 seconds (full council, default) **Progress Feedback**: During deliberation, progress updates show which models have responded and which are still pending: ``` ✓ claude-opus-4.7 (1/4) | waiting: gpt-5.1, gemini-3-pro, grok-4 ✓ gemini-3-pro (2/4) | waiting: gpt-5.1, grok-4 ``` ### Jury Mode (ADR-025b) Transform the council from a "summary generator" to a "decision engine" with structured verdicts. **Verdict Types:** | Mode | Output | Use Case | |------|--------|----------| | `synthesis` (default) | Free-form synthesis | General questions, exploration | | `binary` | approved/rejected + confidence | CI/CD gates, PR reviews, policy checks | | `tie_breaker` | Chairman decides on deadlock | Contentious decisions | **Binary Verdict Mode:** Returns structured go/no-go decisions with confidence scores: ```python # MCP Tool result = await consult_council( query="Should we approve this architectural change?", verdict_type="binary" ) # Returns: { "verdict": "approved", # or "rejected" "confidence": 0.75, # 0.0-1.0 based on council agreement "rationale": "Council agreed the change improves modularity..." } ``` **Tie-Breaker Mode:** When the council is deadlocked (top scores within 0.1 of each other), the chairman casts the deciding vote: ```python result = await consult_council( query="Option A vs Option B for caching strategy?", verdict_type="binary" ) # If deadlocked, returns: { "verdict": "approved", "confidence": 0.60, "rationale": "Chairman resolved deadlock based on...", "deadlocked": true # Flag indicates tie-breaker was needed } ``` **Constructive Dissent:** Extract minority opinions from Stage 2 peer reviews: ```python result = await consult_council( query="Should we migrate to microservices?", verdict_type="binary", include_dissent=True ) # Returns: { "verdict": "approved", "confidence": 0.70, "rationale": "Majority supports migration...", "dissent": "Minority perspective: One reviewer noted scalability concerns with current team size." } ``` **Dissent Algorithm:** 1. Collect all scores from Stage 2 peer reviews 2. Calculate median and standard deviation per response 3. Identify outliers (score < median - 1.5 × std) 4. Extract evaluation text from outliers 5. Format as minority perspective **Example: CI/CD Gate Integration:** ```python import asyncio from llm_council.council import run_full_council from llm_council.verdict import VerdictType async def review_pull_request(pr_diff: str, pr_description: str): """Use LLM Council as a PR review gate.""" query = f""" Review this pull request and determine if it should be approved. Description: {pr_description} Changes: {pr_diff} """ stage1, stage2, stage3, metadata = await run_full_council( query, verdict_type=VerdictType.BINARY, include_dissent=True ) verdict = metadata.get("verdict", {}) if verdict.get("verdict") == "approved" and verdict.get("confidence", 0) >= 0.7: return True, verdict.get("rationale") else: return False, f"{verdict.get('rationale')}\n\nDissent: {verdict.get('dissent', 'None')}" ``` **Defaults:** `verdict_type` is a **per-request parameter** (tool/API/facade), not an environment variable. Deadlock detection uses a fixed 0.1 Borda-spread threshold (`detect_deadlock`); dissent extraction thresholds are internal constants in `dissent.py`. (Earlier docs listed VERDICT_TYPE / DEADLOCK_THRESHOLD / DISSENT_THRESHOLD environment variables — those never shipped.) ### Structured Rubric Scoring (ADR-016) By default, reviewers provide a single 1-10 holistic score. With rubric scoring enabled, reviewers score each response on five dimensions: | Dimension | Weight | Description | |-----------|--------|-------------| | **Accuracy** | 35% | Factual correctness, no hallucinations | | **Relevance** | 10% | Addresses the actual question asked | | **Completeness** | 20% | Covers all aspects of the question | | **Conciseness** | 15% | Efficient communication, no padding | | **Clarity** | 20% | Well-organized, easy to understand | **Accuracy Ceiling**: When enabled (default), accuracy acts as a ceiling on the overall score: - Accuracy < 5: Score caps at 4.0 (40%) — "significant errors or worse" - Accuracy 5-6: Score caps at 7.0 (70%) — "mixed accuracy" - Accuracy ≥ 7: No ceiling — "mostly accurate or better" This prevents well-written hallucinations from ranking well. **Scoring Anchors**: Each score level has defined behavioral meaning (see [ADR-016](docs/adr/ADR-016-structured-rubric-scoring.md)): - 9-10: Excellent (completely accurate, comprehensive, crystal clear) - 7-8: Good (mostly accurate, covers main points) - 5-6: Mixed (some errors or gaps) - 3-4: Poor (significant issues) - 1-2: Failing (fundamentally flawed) ```bash # Enable rubric scoring export LLM_COUNCIL_RUBRIC_SCORING=true # Customize weights (must sum to 1.0) export LLM_COUNCIL_WEIGHT_ACCURACY=0.40 export LLM_COUNCIL_WEIGHT_RELEVANCE=0.10 export LLM_COUNCIL_WEIGHT_COMPLETENESS=0.20 export LLM_COUNCIL_WEIGHT_CONCISENESS=0.10 export LLM_COUNCIL_WEIGHT_CLARITY=0.20 ``` ### Safety Gate (ADR-016) When enabled, a pass/fail safety check runs before rubric scoring to filter harmful content: | Pattern | Description | |---------|-------------| | **dangerous_instructions** | Weapons, explosives, harmful devices | | **weapon_making** | Firearm/weapon construction | | **malware_hacking** | Unauthorized access, malware | | **self_harm** | Self-harm encouragement | | **pii_exposure** | Personal information leakage | Responses that fail safety checks are capped at score 0 regardless of other dimension scores. Educational/defensive content is context-aware and allowed. ```bash # Enable safety gate export LLM_COUNCIL_SAFETY_GATE=true # Customize score cap (default: 0) export LLM_COUNCIL_SAFETY_SCORE_CAP=0.0 ``` ### Bias Auditing (ADR-015) When enabled, the council reports per-session bias indicators from peer review scoring: | Bias Type | Description | Detection Threshold | |-----------|-------------|---------------------| | **Length-Score Correlation** | Do longer responses score higher? | \|r\| > 0.3 | | **Reviewer Calibration** | Are some reviewers harsh/generous relative to peers? | Mean ± 1 std from median | | **Position Bias** | Does presentation order affect scores? | Variance > 0.5 | **Output**: The metadata includes a `bias_audit` object with: - `length_score_correlation`: Pearson correlation coefficient - `length_bias_detected`: Boolean flag - `position_score_variance`: Variance of mean scores by presentation position - `position_bias_detected`: Boolean flag (derived from anonymization labels A, B, C...) - `harsh_reviewers` / `generous_reviewers`: Lists of biased reviewers - `overall_bias_risk`: "low", "medium", or "high" **Important Limitations**: With only 4-5 models per session, these metrics have limited statistical power: - Length correlation with n=5 data points can only detect *extreme* biases - Position bias from a single ordering cannot distinguish position effects from quality differences - Reviewer calibration is relative to the current session only These are **per-session indicators** (red flags for extreme anomalies), not statistically robust proof of systematic bias. Interpret with appropriate caution. ```bash # Enable bias auditing export LLM_COUNCIL_BIAS_AUDIT=true # Customize thresholds (optional) export LLM_COUNCIL_LENGTH_CORRELATION_THRESHOLD=0.3 export LLM_COUNCIL_POSITION_VARIANCE_THRESHOLD=0.5 ``` ### Cross-Session Bias Aggregation (ADR-018) For statistically meaningful bias detection across multiple sessions, enable bias persistence: ```bash # Enable bias persistence (stores metrics locally) export LLM_COUNCIL_BIAS_PERSISTENCE=true # Consent level: 0=off, 1=local only (default), 2=anonymous, 3=enhanced, 4=research export LLM_COUNCIL_BIAS_CONSENT=1 # Store path (default: ~/.llm-council/bias_metrics.jsonl) export LLM_COUNCIL_BIAS_STORE=~/.llm-council/bias_metrics.jsonl ``` **Generate cross-session bias reports:** ```bash # Text report (default) llm-council bias-report # JSON output llm-council bias-report --format json # Include detailed reviewer profiles llm-council bias-report --verbose # Limit to last 50 sessions llm-council bias-report --sessions 50 ``` **Statistical confidence tiers:** | Sessions | Confidence | UI Treatment | |----------|------------|--------------| | N < 10 | Insufficient | "Collecting data..." | | 10-19 | Preliminary | Warning shown | | 20-49 | Moderate | CIs displayed | | N >= 50 | High | Full analysis | ### Output Quality Metrics (ADR-036) Quantify the reliability and quality of council outputs with three core metrics: | Metric | Range | Description | |--------|-------|-------------| | **Consensus Strength Score (CSS)** | 0.0-1.0 | Agreement among council members in Stage 2 rankings | | **Deliberation Depth Index (DDI)** | 0.0-1.0 | Thoroughness of the deliberation process | | **Synthesis Attribution Score (SAS)** | 0.0-1.0 | How well synthesis traces back to source responses | **CSS Interpretation:** | Score | Interpretation | Action | |-------|---------------|--------| | 0.85+ | Strong consensus | High confidence in synthesis | | 0.70-0.84 | Moderate consensus | Note minority views | | 0.50-0.69 | Weak consensus | Consider `include_dissent=true` | | <0.50 | Significant disagreement | Use debate mode | **SAS Components:** - `winner_alignment`: Similarity to top-ranked responses - `max_source_alignment`: Best match to any response - `hallucination_risk`: 1 - max_source_alignment - `grounded`: True if synthesis traces to sources (threshold: 0.6) **Usage:** Quality metrics are automatically included in metadata: ```python stage1, stage2, stage3, metadata = await run_full_council(query) quality = metadata.get("quality_metrics", {}) core = quality.get("core", {}) print(f"Consensus: {core['consensus_strength']:.2f}") print(f"Depth: {core['deliberation_depth']:.2f}") print(f"Grounded: {core['synthesis_attribution']['grounded']}") ``` **MCP Tool Output:** The `consult_council` MCP tool displays quality metrics with visual bars: ``` ### Quality Metrics - **Consensus Strength**: 0.82 [████████░░] - **Deliberation Depth**: 0.74 [███████░░░] - **Synthesis Grounded**: ✓ (alignment: 0.89) ``` **Environment Variables:** ```bash # Enable/disable quality metrics (default: true) export LLM_COUNCIL_QUALITY_METRICS=true # Quality tier: core (OSS), standard, enterprise export LLM_COUNCIL_QUALITY_TIER=core ``` ### Gateway Layer (ADR-023) > **Status note:** the `GatewayRouter` class-based abstraction below is real, > importable code, but it is **not currently reachable via configuration or > environment variables** — the "Enable the gateway layer" switch further > down does not exist in the codebase, and `GatewayConfig` has no `enabled` > field to turn on the `USE_GATEWAY_LAYER` flag that `llm_council.gateway_adapter` > checks. Live traffic is routed by the simpler `gateways.default` / > `gateways.providers.*` / `gateways.model_name_map` config documented under > [Option 2: YAML Configuration](#option-2-yaml-configuration-recommended) > above, via `llm_council.gateway.resolver`, not by the code on this page. > Tracked in [#524](https://github.com/amiable-dev/llm-council/issues/524). The gateway layer provides an abstraction over LLM API requests with multiple gateway options: **Available Gateways:** | Gateway | Description | Key Features | |---------|-------------|--------------| | `OpenRouterGateway` | Routes through OpenRouter | 100+ models, single API key | | `RequestyGateway` | Routes through Requesty | BYOK support, analytics | | `DirectGateway` | Direct provider APIs | Anthropic, OpenAI, Google | | `OllamaGateway` | Local LLMs via Ollama | Air-gapped, cost-free, privacy-first | **Core Features:** - **Circuit Breaker**: Prevents cascading failures (CLOSED → OPEN → HALF_OPEN → CLOSED) - **Fallback Chains**: Automatic retry with secondary gateways on failure - **Per-Gateway Metrics**: Track failure counts, latency, and health status - **BYOK Support**: Bring Your Own Key for Requesty and Direct gateways **Basic Usage:** ```python from llm_council.gateway import ( GatewayRouter, GatewayRequest, CanonicalMessage, ContentBlock, OpenRouterGateway, RequestyGateway, DirectGateway ) # Single gateway (OpenRouter) router = GatewayRouter() # Multi-gateway with fallback router = GatewayRouter( gateways={ "openrouter": OpenRouterGateway(), "requesty": RequestyGateway(byok_enabled=True, byok_keys={"anthropic": "sk-ant-..."}), "direct": DirectGateway(provider_keys={"openai": "sk-..."}), }, default_gateway="openrouter", fallback_chains={"openrouter": ["requesty", "direct"]} ) request = GatewayRequest( model="openai/gpt-4o", messages=[CanonicalMessage(role="user", content=[ContentBlock(type="text", text="Hello")])] ) response = await router.complete(request) ``` **Enable the gateway layer:** There is currently no supported way to enable `GatewayRouter` for live council traffic (see the status note above) — the environment variable previously documented here does not exist in the codebase and setting it silently does nothing. To route council queries through Requesty or a custom OpenRouter endpoint today, use `gateways.default` / `gateways.providers.*` in the YAML config instead (see [Option 2: YAML Configuration](#option-2-yaml-configuration-recommended)). ```bash # Configure specific gateway credentials (read regardless of which # routing path is active): export REQUESTY_API_KEY=your-requesty-key # For Requesty export ANTHROPIC_API_KEY=sk-ant-... # For Direct (Anthropic) export OPENAI_API_KEY=sk-... # For Direct (OpenAI) export GOOGLE_API_KEY=... # For Direct (Google) ``` **Circuit Breaker Behavior:** - Default: 5 failures to trip the circuit - Recovery timeout: 60 seconds - Half-open state allows test requests to check recovery - Open circuits are skipped in fallback chain The gateway layer is currently **opt-in** (default: disabled) for backward compatibility. ### Triage Layer (ADR-020) The triage layer provides query classification, model selection optimizations, and confidence-gated routing: - **Confidence-Gated Fast Path**: Routes simple queries to a single model, escalating to full council when confidence is low - **Shadow Council Sampling**: Random 5% sampling validates fast path quality against full council - **Rollback Monitoring**: Automatic rollback when disagreement/escalation rates breach thresholds - **Wildcard Selection**: Adds domain-specialized models to the council based on query classification - **Prompt Optimization**: Per-model prompt adaptation (Claude gets XML, OpenAI gets Markdown) - **Complexity Classification**: Heuristic-based with optional Not Diamond API integration **Domain Categories:** | Domain | Description | Specialist Models | |--------|-------------|-------------------| | CODE | Programming, debugging, algorithms | DeepSeek, Codestral | | REASONING | Math, logic, proofs | o1-preview, DeepSeek-R1 | | CREATIVE | Stories, poems, fiction | Claude Opus, Command-R+ | | MULTILINGUAL | Translation, language | GPT-4o, Command-R+ | | GENERAL | General knowledge | Llama 3 (fallback) | **Enable wildcard selection:** ```bash export LLM_COUNCIL_WILDCARD_ENABLED=true ``` This automatically adds a domain specialist to the council based on query classification. For example, a Python coding question will add a DeepSeek model alongside the default council. **Enable prompt optimization:** ```bash export LLM_COUNCIL_PROMPT_OPTIMIZATION_ENABLED=true ``` This applies per-model prompt formatting. Claude receives XML-structured prompts, while other providers receive their preferred format. **Enable confidence-gated fast path:** ```bash export LLM_COUNCIL_FAST_PATH_ENABLED=true export LLM_COUNCIL_FAST_PATH_CONFIDENCE_THRESHOLD=0.92 # default ``` When enabled, simple queries are routed to a single model. If the model's confidence is below the threshold, the query automatically escalates to the full council. This can reduce costs by 45-55% on simple queries while maintaining quality. **Fast Path Quality Monitoring:** The fast path includes built-in quality monitoring: | Metric | Threshold | Action | |--------|-----------|--------| | Shadow disagreement rate | > 8% | Automatic rollback | | User escalation rate | > 15% | Automatic rollback | | Error rate | > 1.5x baseline | Automatic rollback | Configure monitoring: ```bash export LLM_COUNCIL_SHADOW_SAMPLE_RATE=0.05 # 5% shadow sampling export LLM_COUNCIL_ROLLBACK_ENABLED=true export LLM_COUNCIL_ROLLBACK_WINDOW=100 # rolling window size ``` **Optional Not Diamond Integration:** For advanced model routing, integrate with [Not Diamond](https://notdiamond.ai): ```bash export NOT_DIAMOND_API_KEY="your-key" export LLM_COUNCIL_USE_NOT_DIAMOND=true ``` When Not Diamond is unavailable, the system gracefully falls back to heuristic-based classification. The triage layer is currently **opt-in** (default: disabled) for backward compatibility. ### Local LLM Support (ADR-025) Run council deliberations entirely on local hardware using Ollama: ```bash # Install with Ollama support pip install "llm-council-core[ollama]" # Start Ollama (if not already running) ollama serve # Pull a model ollama pull llama3.2 ``` **Use local models in your council:** ```bash # Mix local and cloud models export LLM_COUNCIL_MODELS="ollama/llama3.2,openai/gpt-4o,anthropic/claude-3-5-sonnet" # Or use only local models (air-gapped mode) export LLM_COUNCIL_MODELS="ollama/llama3.2,ollama/mistral,ollama/codellama" export LLM_COUNCIL_CHAIRMAN="ollama/llama3.2" ``` **Hardware Requirements:** | Profile | Hardware | Models | Use Case | |---------|----------|--------|----------| | Minimum | 8+ core CPU, 16GB RAM | 7B quantized | Dev/testing | | Recommended | M-series Pro, 32GB unified | 7B-13B | Small council | | Professional | 2x RTX 4090, 64GB+ | 70B quantized | Production | | Enterprise | Mac Studio 64GB+ | Multiple 70B | Air-gapped | **Quality Degradation Notice**: Local models typically have reduced capabilities compared to cloud-hosted frontier models. The gateway includes quality notices in responses to inform users when local models are used. **Configuration:** ```bash # Ollama endpoint (default: http://localhost:11434) export LLM_COUNCIL_OLLAMA_BASE_URL=http://localhost:11434 # Timeout for local models (default: 120s - first load can be slow) export LLM_COUNCIL_OLLAMA_TIMEOUT=120.0 ``` ### Webhook Notifications (ADR-025) Receive real-time notifications as the council deliberates: ```python from llm_council.webhooks import WebhookConfig, WebhookDispatcher, WebhookPayload # Configure webhook endpoint config = WebhookConfig( url="https://your-server.com/webhook", events=["council.complete", "council.error"], secret="your-hmac-secret" # Optional, for signature verification ) # Dispatch events (used internally by council) dispatcher = WebhookDispatcher() result = await dispatcher.dispatch(config, payload) ``` **Event Types:** | Event | Description | Payload | |-------|-------------|---------| | `council.deliberation_start` | Council begins | request_id, models | | `council.stage1.complete` | All initial responses received | response_count | | `model.vote_cast` | A model submitted rankings | voter, ranking | | `council.stage2.complete` | All rankings complete | aggregate_rankings | | `council.complete` | Final answer ready | stage3_response, duration_ms | | `council.error` | Error occurred | error, partial_results | **HMAC Signature Verification:** Webhooks are signed using HMAC-SHA256 for security: ```python from llm_council.webhooks import verify_webhook_request # Verify incoming webhook (in your server) is_valid = verify_webhook_request( payload=request.body, headers=request.headers, secret="your-hmac-secret" ) ``` Headers included: - `X-Council-Signature`: `sha256=