# AI Agent Harness Kit A **project-agnostic** harness that gives any AI coding agent (Claude Code, GitHub Copilot, Codex, Cursor, Gemini, …) a consistent operating contract: what to load, what sequence to follow, and how to iterate until done — plus autonomous, metric-driven self-improvement loops that can run on a **local LLM**, and a live metrics dashboard. Extracted as a clean, reusable kit. See [`CREDITS.md`](CREDITS.md) for the prior work it builds on, and [`HARNESS_CARD.md`](HARNESS_CARD.md) for the one-page control/agency/runtime design summary. Latest release notes: [`RELEASE_NOTES_v3.5.1.md`](RELEASE_NOTES_v3.5.1.md). ## Install The kit is packaged as an [Agent Skill](https://agentskills.io/) and a Claude Code plugin, so it installs into 70+ agents (Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, …) without copying folders by hand. ```bash # Any of 70+ agents, via the open Agent Skills CLI (-g installs globally for your user): npx skills add Dfintz/harness-kit -g # A specific agent (or several): npx skills add Dfintz/harness-kit -g -a github-copilot -a claude-code # Or from a local checkout of this kit: npx skills add ./harness-kit --list # discover, then add --skill harness to install ``` ```text # Claude Code, via the native plugin marketplace (auto-updates): /plugin marketplace add Dfintz/harness-kit /plugin install harness-kit ``` **Two layers, on purpose.** The skill above is the **playbook** — it teaches the agent the harness contract (stages, gates, loops, memory) and is enough for guidance in any repo. The **runnable engine** (the `scripts/harness/*.mjs` loop runners, dashboard, and MCP server) ships with the kit files; get it by either installing the Claude Code **plugin** (bundles everything) or adopting the kit scaffold per [`SETUP.md`](SETUP.md). For the **GitHub Copilot App inside a repository**, the scaffold also ships `.github/copilot-instructions.md`. Once the kit is present in a repo, Copilot can load the harness entrypoint directly from that file. ### Agent Plugins pilot This repository also contains an experimental Agent Plugins v1 skills-only package at [`plugins/agent-plugins/harness-kit/`](plugins/agent-plugins/harness-kit/). It currently exports only the explicitly allowlisted, user-invoked `wait-what` skill. Regenerate and validate it with: ```bash npm run harness:agent-plugins:export npm run harness:agent-plugins:validate npm run test:harness:agent-plugins ``` This is a portable package export, not an Agent Plugins client. It intentionally contains no `mcp.json`, command dispatch, or plugin-defined process execution. Interoperability still requires a load test in a conforming Agent Plugins client. ## What's inside | Capability | Where | Notes | | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | **Workflow stage machine** | [`.github/harness/HARNESS.md`](.github/harness/HARNESS.md), [`.github/instructions/`](.github/instructions/) | Understand → Architect → Implement → Review (breadth+depth) → Feedback, with 5 architectural gates | | **GitHub Copilot app entrypoint** | [`.github/copilot-instructions.md`](.github/copilot-instructions.md) | Repository-level bootstrap for Copilot that points it at the harness contract | | **Convergence loops** | [`.github/harness/loops/`](.github/harness/loops/), [`run-loop.mjs`](scripts/harness/run-loop.mjs) | Iterate until checks (lint/type/build/test) go green | | **Workflow loops** | same | Rubric-graded passes (review-fix, feature-cycle, ci-green) | | **Experiment loops (autoresearch-style)** | [`run-experiment.mjs`](scripts/harness/run-experiment.mjs), [`experiment-loop.mjs`](scripts/harness/experiment-loop.mjs) | Hill-climb a numeric metric; keep-if-improved, else revert | | **Local-LLM agents** | [`ollama-agent.mjs`](scripts/harness/ollama-agent.mjs), [`ollama-apply-agent.mjs`](scripts/harness/ollama-apply-agent.mjs) | Drive loops with a local model via **Ollama** or **LM Studio** (`--provider`) | | **Memory** | [`.github/harness/memory/`](.github/harness/memory/) | Committed lessons + Architecture Briefs (structure only — no lessons shipped) | | **Knowledge graph providers** | [`graph-provider.mjs`](scripts/harness/graph-provider.mjs), [`graph-refresh-loop.mjs`](scripts/harness/graph-refresh-loop.mjs) | Provider abstraction (`understand-anything` default, optional `graphify`) with deterministic refresh backends (`understand-anything` and configurable `graphify`) | | **MCP server** | [`mcp-server.mjs`](scripts/harness/mcp-server.mjs) | Exposes graph/memory/vector + routing/catalog/discovery tools over MCP (`.vscode/mcp.json` registers it) | | **Dashboard** | [`report-server.mjs`](scripts/harness/report-server.mjs) | Always-on HTML metrics dashboard | | **Capability catalog** | [`harness-catalog.mjs`](scripts/harness/harness-catalog.mjs), [`llms.txt`](llms.txt), [`.github/harness/catalog/`](.github/harness/catalog/) | Machine-readable taxonomy + intent profiles for external agent/tool recommendation | ## The three loop kinds ```text convergence run until pass/fail checks are all green (build-fix, test-fix) workflow run rubric-graded passes to a terminal state (review-fix, feature-cycle, ci-green) experiment hill-climb a numeric metric, keep-if-improved (lint-debt-experiment) ← autoresearch-style ``` ## Quick start ```bash # 1. Point the harness at your project's commands. # Edit harness.config.json: set project.name, project.description, and commands.* node -e "JSON.parse(require('fs').readFileSync('harness.config.json','utf8'))" # sanity-check npm run harness:graph:provider # inspect graph provider + paths npm run harness:graph:genui # inspect GenUI graph.html readiness npm run harness:graph:parity -- --local-only # provider parity self-test matrix (local) # Optional: preview the repo's harness routing and operator handoff plans. # PowerShell: run each npm wrapper command separately instead of chaining wrappers with semicolons. npm run harness:route -- --task "fix auth middleware race" npm run harness:profile -- --task "design multi-agent coordinator" --json npm run harness:feature -- --task "ship federation audit hardening" npm run harness:handoff:review -- --task "review cache invalidation changes" # 2. List and run a convergence loop (uses your configured commands). npm run harness:loops node scripts/harness/run-loop.mjs build-fix --agent "" # 3. Record the baseline of an experiment metric (no agent needed). node scripts/harness/run-experiment.mjs lint-debt-experiment --measure-only # 4. See the dashboard. npm run harness:report # writes .github/harness/runs/report.html npm run dashboard:up # or serve it always-on at http://localhost:8099 # 5. Publish machine-readable capability artifacts for other agents/tools. npm run harness:catalog:sync # writes llms.txt + .github/harness/catalog/harness-profile.json # Optional: council-style parallel review synthesis (keeps stage machine intact) npm run harness:council:review -- --mode review --prompt "review this change set" # Optional: catalog presets (safe fixed keys) npm run harness:council:review -- --mode review --prompt-key review # Or pipe prompt content from stdin (PowerShell) Get-Content .github/harness/memory/briefs/EXAMPLE.md -Raw | npm run harness:council:review -- --mode review --prompt-stdin # Backward compatibility: old prompt-file callers now resolve tokens only (no file reads) npm run harness:council:review -- --mode review --prompt-file review.md # Optional: transient per-workspace JSONL memory npm run harness:workspace-memory -- append --mode review --text "Captured reviewer disagreement" npm run harness:workspace-memory -- list --last 20 # Optional: record a workflow run with explicit pending approval marker node scripts/harness/record-run.mjs --loop review-fix --state blocked --approval-required --approval-status pending --approval-note "Awaiting reviewer sign-off" --fail "Gate 3 ownership unresolved" ``` ### Fastest path to first successful run If you want the shortest onboarding route, do this in order: 1. Edit `harness.config.json` and set your real `commands.lint`, `commands.typeCheck`, `commands.build`, and `commands.test`. 2. Run `npm run harness:health -- --fast` and fix any required failures. 3. Run `npm run harness:loops` to confirm loops are discoverable. 4. Run one bounded loop: `node scripts/harness/run-loop.mjs build-fix --agent ""`. 5. Generate operator outputs: `npm run harness:report` and `npm run harness:catalog:sync`. The dashboard's Pending approvals section is strict: it only shows runs with explicit journal markers (`approval.required=true` and `approval.status=pending`). It does not infer pending approvals from brief status or blocked/stuck terminal states. Security hardening for spawned CLI commands is now centralized in `scripts/harness/command-validation.mjs` and enforced at key shell-spawn points (`run-loop`, `run-experiment`, `plan-review`). It allows known executables and rejects shell metacharacter payloads. Webview command surfaces and WSL-specific execution adapters are intentionally optional/deferred in this kit release. Keep using script-first orchestration unless your environment needs a dedicated UI or cross-shell adapter. To normalize historical datasets, backfill legacy run journals with explicit default approval markers: ```bash npm run harness:migrate:approvals -- --dry-run npm run harness:migrate:approvals ``` This migration only updates loop journals missing the `approval` object, defaulting to `required=false` and `status=not-required`. Full adoption guide: [`SETUP.md`](SETUP.md). Loop protocol: [`.github/harness/LOOPS.md`](.github/harness/LOOPS.md). For optional CI gate toggles (including targeted OKF and changed-brief policy checks), see [`SETUP.md#example-ci-workflow-for-optional-security-gates`](SETUP.md#example-ci-workflow-for-optional-security-gates). ## Prompt routing policy The kit ships a repo-local prompt router in [`scripts/harness/prompt-router.mjs`](scripts/harness/prompt-router.mjs). It does not intercept editor prompts by itself; instead it gives operators a deterministic route and stage/model handoff plan based on [`harness.config.json`](harness.config.json). - `harness:route` classifies a prompt as trivial or non-trivial. - `harness:profile` picks an intent profile (`turnkey-coding`, `multi-agent-orchestration`, `drop-in-memory`) for a task. - `harness:route --intent ` routes directly through the mapped profile. - `harness:feature` and `harness:handoff:feature` print the full feature-delivery handoff: Understand → Architect → Implement → Review Breadth → Review Depth → Feedback. - `harness:handoff:review` prints the independent review handoff: Understand → Review Breadth → Review Depth → Feedback. - `harness:prompt-pack` generates a gitignored prompt pack under `.github/harness/runs/prompt-packs/` with an orchestrator prompt, canonical stage prompts, cycle-memory scaffolding, and optional scout/challenger sidecars. - `harness:review` runs the plan-review workflow for backward compatibility. By default the shipped environment policy separates execution and judgment using a **three-tier capability model**: | Tier | Stages | Default | Pinned example | |---|---|---|---| | **high-reasoning** | Understand, Architect, Review Breadth, Review Depth, Feedback | Copilot Auto | `claude-opus-4-8` | | **balanced-coding** | Implement, `build-fix`, `test-fix` | Copilot Auto | `gpt-5.3-codex` or `claude-sonnet-4.x` | | **fast-cheap-local** | Experiment loops, lint-debt, enrichment, triage | — | `qwen2.5-coder:14b` via Ollama/LM Studio | Copilot Auto is a safe default for the hosted tiers. Pin a specific model in `harness.config.json` only when you need consistent identity for reproducible evals or cross-model review. The router enforces that `models.implementer ≠ models.reviewer` to prevent single-model echo chambers. For domain-heavy work, `harness.config.json` also exposes advisory `modelPolicy.domainSpecialists` guidance. It recommends model tiers and companion skills for frontend, UI/UX, database, infrastructure, and backend tasks while keeping executable stage routing owned by `skillModelMapping.mappings`. Treat those domain entries as model-selection guidance inside the current stage, not as checked-in specialist skills or automatic router dispatch. Use the model-selection wizard to pick from the supported Copilot model snapshot and compare three cost/quality levels: ```bash npm run harness:model-wizard -- recommend --domain frontend --level balanced npm run harness:model-wizard -- recommend --mode dev --level high npm run harness:model-wizard -- recommend --mode super-plus --level balanced --json npm run harness:model-wizard -- recommend --domain database --level high --json npm run harness:model-wizard -- check ``` The wizard reads `modelPolicy.modelSelectionWizard` in `harness.config.json`. Its supported-models snapshot records the GitHub Docs source and date; organization, plan, client, and model-picker policies can still restrict what an individual user sees. ## Autoresearch with a local model Works with **Ollama** (default, `:11434`) or **LM Studio** (OpenAI-compatible, `:1234`) — pick with `--provider` or `HARNESS_LLM_PROVIDER`. ```bash # One bounded experiment, edits driven by a local model that actually rewrites the target file: node scripts/harness/run-experiment.mjs lint-debt-experiment \ --agent "node scripts/harness/ollama-apply-agent.mjs --model qwen2.5-coder:14b" # Same via LM Studio (load a model there first): node scripts/harness/run-experiment.mjs lint-debt-experiment \ --agent "node scripts/harness/ollama-apply-agent.mjs --provider lmstudio --model " # Continuous overnight hill-climbing, committing each kept improvement: npm run harness:experiment:ollama -- --commit # or: harness:experiment:lmstudio ``` The apply-agent edits only the experiment's single declared `target`; the runner re-measures and reverts anything that doesn't improve the metric — so letting a small local model rewrite a file is safe by construction. The shared adapter [`llm-provider.mjs`](scripts/harness/llm-provider.mjs) handles both runtimes (chat + embeddings); `vector-search.mjs` honors `--provider` too. ## MCP integration The harness ships a first-class MCP stdio server exposing read/observe tools across knowledge graph, memory, vector search, loop/report introspection, and intent/catalog discovery. [`.vscode/mcp.json`](.vscode/mcp.json) registers it for VS Code; for Claude Code / Cursor use the same `command`/`args` in their MCP config. ```bash node scripts/harness/mcp-tools.mjs list-tools # inspect the tool catalog npm run harness:mcp:server # run the stdio server directly npm run harness:mcp -- graph-provider-status # inspect active graph provider availability npm run harness:mcp -- graph-genui-status # inspect graph.html serving readiness npm run harness:mcp -- graph-events # read structured refresh/fallback/degradation events npm run harness:mcp -- harness-catalog # read taxonomy + profiles npm run harness:mcp -- harness-pick-profile --task "add memory retrieval path" npm run harness:mcp -- harness-tool-discover --intent drop-in-memory --limit 6 ``` Convergence and experiment runners support resumable execution via checkpointed journals: ```bash node scripts/harness/run-loop.mjs build-fix --resume latest node scripts/harness/run-experiment.mjs lint-debt-experiment --resume latest ``` Loop **execution** stays CLI-only on purpose: a loop invokes an agent and runs for minutes, so exposing it as an auto-callable MCP tool (when the MCP client _is_ the agent) would recurse and time out. The MCP surface is for discovery and context, not for driving loops. ## Requirements - Node.js ≥ 20 (uses built-in `fetch`; no install needed for the core loops). - Optional: Docker (dashboard/graph sidecars), Ollama or LM Studio (local-LLM loops), the Understand-Anything plugin (deterministic graph refresh), Graphify (optional alternate graph provider). For the `graph-refresh` sidecar, plugin dependency bootstrapping is now hardened by default: when the plugin is mounted at `/opt/understand-plugin`, the loop copies it to a writable runtime path and runs `corepack pnpm install --frozen-lockfile` there before refresh. This avoids regressions from read-only mounts or host/container linker mismatches. Override with: - `GRAPH_REFRESH_BOOTSTRAP_PLUGIN=false` to disable bootstrap. - `GRAPH_REFRESH_FORCE_BOOTSTRAP=true` to force runtime copy/install even when source plugin already has `node_modules`. - `GRAPH_REFRESH_RUNTIME_PLUGIN_ROOT=/custom/path` to change runtime copy location (default: `/workspace/.cache/understand-plugin-runtime`). - `GRAPH_REFRESH_BOOTSTRAP_INSTALL_TIMEOUT_MS=120000` to cap install wait time and fail fast. You can also run a deterministic preflight manually before starting the loop: ```bash node scripts/harness/graph-refresh-loop.mjs --preflight-only --plugin-root ``` The compose sidecar now runs this preflight first and exits with one actionable error if prerequisites are missing. Graph provider selection lives in `harness.config.json`: - `graph.provider: "understand-anything"` (default) keeps the current deterministic flow. - `graph.provider: "graphify"` executes Graphify deterministic refresh via `graph.graphify.refreshCommand` and reads the resulting `graph.graphify.path`. - `graph.provider: "both"` refreshes Understand-Anything and Graphify backends in one run (Graphify refresh runs when `graph.graphify.refreshCommand` is configured). - `graph.sync.rebuildVectorIndex` / `graph.sync.rebuildMemoryLinkIndex` (optional) trigger post-refresh index rebuild hooks so vector + memory-link stay in sync with the active provider graph. - `graph.observability.eventsPath` stores structured graph lifecycle events (`refresh.start|refresh.success|refresh.fail|query.fallback|degradation`) consumed by MCP + dashboard surfaces. - `graph.graphHtmlPath` / `graph.graphify.graphHtmlPath` is now wired to HTTP and GenUI status surfaces: - `GET /graph.html` on `report-server.mjs` serves configured graph HTML when present and repo-safe. - `GET /genui/graph.json` exposes provider-agnostic graph render metadata for GenUI consumers. - `GET /graph-events.json` exposes recent structured graph lifecycle events. ## License MIT — see [`LICENSE`](LICENSE) and [`CREDITS.md`](CREDITS.md).