HarnessBench

The Benchmark for Comparing Agent Harnesses on Everyday Online Tasks
Read the Docs  ·  Harness Comparison  ·  Cloud Setup

Star this repo Project Page GitHub stars Discord Codespaces

Sister project of ClawBench

Ask DeepWiki

If you want to compare base models on a fixed harness, check out our sister project ClawBench  —  same pipeline, orthogonal axis.

Run in one line of code ```bash uv tool install harness-bench && harness-bench ``` Install → List → Run.   Reuses ClawBench's pipeline.   Cloud harnesses opt-in via env vars. ### Which Harness Wins on the Same Task? Given one task (order food, book travel, apply for a job) and **one fixed base model** --
which agentic harness actually gets it done?
**Six named harnesses**, four runtimes, one pipeline, one leaderboard. --- **6** harnesses  ·  **4** runtimes (Python / Node / Rust / Web)  ·  **153** shared tasks  ·  **15** categories 中文

 Plugin Entry-Points         One Container per Harness         Cloud Opt-in         Same Pipeline as ClawBench


## How It Works ``` You pick a task HarnessBench spins up Each harness drives Same 5-layer recording from ClawBench's one container per the browser its own + DOM-match + LLM judge shared 153-task pool harness (Python / Node way on the same task partitioned by harness / Rust / Web) ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ "Book a │ ──► │ 6 containers│ ──► │ 6 different │ ──► │ Per-harness │ │ flight on │ │ (one per │ │ agent loops │ │ leaderboard │ │ Expedia" │ │ harness) │ │ same task │ │ by category │ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ ```
# LLM Quick Start Point your coding agent (Claude Code, Cursor, Copilot, etc.) at [`AGENTS.md`](AGENTS.md) and prompt away. HarnessBench shares ClawBench's test-cases, container base image, and 5-layer recording stack -- if your agent already knows ClawBench, there is nothing new to learn about the pipeline, only a new harness axis.
# Human Quick Start ```bash # Option A -- PyPI install (recommended) uv tool install harness-bench && harness-bench ``` ```bash # Option B -- Clone the repo (for contributors / adding a harness) git clone https://github.com/reacher-z/HarnessBench.git && cd HarnessBench && uv run harness-bench ``` **Prerequisites:** [Python 3.10+](https://python.org), [uv](https://docs.astral.sh/uv/), and a container engine -- [Docker](https://www.docker.com/) **or** [Podman](https://podman.io/). Same engine detection as ClawBench; force one with `export CONTAINER_ENGINE=docker`. **1. List registered harnesses:** ```bash harness-bench harnesses # openclaw ready # hermes ready # claw-code ready # browser-use ready # stagehand skipped: set BROWSERBASE_API_KEY # coze-studio skipped: set COZE_INSTANCE_URL, COZE_API_TOKEN ``` **2. Preview a matrix** (no side effects): ```bash harness-bench matrix \ --harness openclaw --harness hermes --harness browser-use \ --model claude-sonnet-4-6 \ --case 001-daily-life-food-uber-eats \ --case 007-daily-life-travel-expedia ``` **3. Run one triple end-to-end:** ```bash harness-bench run \ --harness openclaw \ --model claude-sonnet-4-6 \ --case 001-daily-life-food-uber-eats ``` Results land in `./harness-output///-/` with the full five-layer recording -- identical layout to ClawBench so a single analysis script handles both. **4. Matrix batch** (all eligible triples): ```bash harness-bench batch \ --harness openclaw --harness hermes --harness browser-use --harness claw-code \ --model claude-sonnet-4-6 \ --case $(cat fixtures/lite.txt) ``` **5. Render the leaderboard:** ```bash harness-bench leaderboard --results-dir ./harness-output/ ```
# HarnessBench-Lite **New here? Run this first.** [`fixtures/lite.txt`](fixtures/lite.txt) is a **20-task curated subset** of ClawBench's 153, reused verbatim so HarnessBench-Lite and ClawBench-Lite are comparable row-for-row. It matches the 20-tasks-per-source convention of [browser-use/benchmark](https://github.com/browser-use/benchmark) and gives you a credible harness-vs-harness signal at a fraction of the full-matrix cost. For six harnesses on Lite you're looking at roughly **120 triples** (6 harnesses x 20 tasks); cloud-opt-in harnesses auto-skip if credentials are absent so the local-only cost is **80 triples**. ```bash harness-bench batch \ --harness openclaw -h hermes -h browser-use -h claw-code \ --model claude-sonnet-4-6 \ --case $(cat fixtures/lite.txt) ```
# Tutorial
[![Watch on YouTube](https://img.shields.io/badge/Watch_Tutorial-YouTube-FF0000?style=for-the-badge&logo=youtube&logoColor=white)](https://youtube.com)    [![Watch on Bilibili](https://img.shields.io/badge/Watch_Tutorial-Bilibili-00A1D6?style=for-the-badge&logo=bilibili&logoColor=white)](https://bilibili.com)

# Demos
**`openclaw` on Uber Eats** https://github.com/user-attachments/assets/placeholder-openclaw-ubereats **`browser-use` on the same Uber Eats task** https://github.com/user-attachments/assets/placeholder-browseruse-ubereats
> Each HarnessBench run produces the same MP4 session recording ClawBench does. Pair-watching the same task across two harnesses is the fastest way to see where their behavior diverges.
# The Six Named Harnesses | Harness | Upstream | Runtime | Cloud? | What it is | |---------|----------|:-------:|:------:|------------| | `openclaw` | [reacher-z/ClawBench](https://github.com/reacher-z/ClawBench) | Python | — | Reference harness, shared with ClawBench. The baseline everyone gets compared against. | | `hermes` | [nousresearch/hermes-agent](https://github.com/nousresearch/hermes-agent) | Python | — | Hermes-style tool-use loop with explicit plan/act steps. | | `claw-code` | [ultraworkers/claw-code](https://github.com/ultraworkers/claw-code) | Rust | — | Rust-native agent loop, zero-GIL concurrency. | | `browser-use` | [browser-use/browser-use](https://github.com/browser-use/browser-use) | Python | — | Community-favorite Playwright-based harness. | | `stagehand` | [browserbase/stagehand](https://github.com/browserbase/stagehand) | Node/TS | **Yes** | BrowserBase's Stagehand -- requires `BROWSERBASE_API_KEY`. | | `coze-studio` | [coze-dev/coze-studio](https://github.com/coze-dev/coze-studio) | Web | **Yes** | Coze Studio flow runner -- requires `COZE_INSTANCE_URL` + `COZE_API_TOKEN`. | Cloud harnesses are **opt-in**: without credentials they appear in the matrix as `skipped:missing_credential:` -- **never silently zeroed**. More harnesses land as follow-up PRs; see [`docs/scout-2026-04-16.md`](docs/scout-2026-04-16.md) for the global framework sweep.
# Preview Leaderboard
**Work in progress.** Initial runs on `claude-sonnet-4-6` across the six harnesses are in flight -- numbers below are placeholders illustrating the leaderboard shape.
| Rank | Harness | Overall | Daily | Travel | Work | Dev | Notes | |:----:|---------|:-------:|:-----:|:------:|:----:|:---:|-------| | — | `openclaw` | TBD | TBD | TBD | TBD | TBD | reference harness (ClawBench) | | — | `hermes` | TBD | TBD | TBD | TBD | TBD | Python tool-use loop | | — | `claw-code` | TBD | TBD | TBD | TBD | TBD | Rust agent loop | | — | `browser-use` | TBD | TBD | TBD | TBD | TBD | Playwright-based | | — | `stagehand` | TBD | TBD | TBD | TBD | TBD | cloud-opt-in | | — | `coze-studio` | TBD | TBD | TBD | TBD | TBD | cloud-opt-in | Partitioning: (harness, model, category). Run harness-bench leaderboard locally to render your own.
# Example Walkthrough Curious what one triple actually looks like? Here's **task 001** run through **three different harnesses**, same base model: ``` task = 001-daily-life-food-uber-eats model = claude-sonnet-4-6 harness = openclaw ──► ./harness-output/openclaw/claude-sonnet-4-6/001-.../ (Python loop driving Chrome via the ClawBench extension) harness = hermes ──► ./harness-output/hermes/claude-sonnet-4-6/001-.../ (Python loop, Hermes tool-use convention) harness = browser-use ──► ./harness-output/browser-use/claude-sonnet-4-6/001-.../ (Playwright driver + atomic action primitives) ``` All three land the **same five-layer bundle** (recording.mp4, screenshots, actions.jsonl, requests.jsonl, agent-messages.jsonl) plus `interception.json` from ClawBench's CDP-level fetch interceptor. That uniformity is what makes cross-harness comparison meaningful: identical inputs, identical judge, identical rubric -- the only thing that moves between runs is the agent loop itself.
# Architecture
How HarnessBench stacks on top of ClawBench ``` ┌─────────────────────────────────────────────────────────┐ │ harness-bench CLI │ │ (matrix expansion, credential gating, leaderboard) │ └───────────────────────┬─────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ clawbench.harnesses (plugin entry-point group) │ │ discovered at runtime via importlib.metadata │ └───────────────────────┬─────────────────────────────────┘ │ ┌──────────────┼──────────────┬──────────────┐ ▼ ▼ ▼ ▼ openclaw hermes claw-code browser-use stagehand coze-studio (Python) (Python) (Rust) (Python) (Node/TS) (Web) dedicated dedicated dedicated dedicated dedicated dedicated container container container container container container │ │ │ │ │ │ └──────────────┴──────────────┴──────────────┴──────────────┴──────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ clawbench/base: │ │ (Chrome + Xvfb + FFmpeg + extension-server + CDP wire) │ │ Same image ClawBench uses -- zero drift. │ └─────────────────────────────────────────────────────────┘ ``` Each harness ships its own `Dockerfile` (3-file adapter: `Dockerfile` + `setup.sh` + `run.sh`) that `FROM clawbench/base:` so the shared stack is byte-for-byte identical across harnesses. See [`docs/adding-a-harness.md`](docs/adding-a-harness.md) for the walkthrough.

# CLI ```bash # List and gate harness-bench harnesses # Matrix preview (no side effects) harness-bench matrix --harness openclaw -h hermes -m claude-sonnet-4-6 -c 001-daily-life-food-uber-eats # Single run harness-bench run --harness openclaw --model claude-sonnet-4-6 --case 001-daily-life-food-uber-eats # Batch (matrix-expand, skip ineligible, run the rest) harness-bench batch -h openclaw -h hermes -h browser-use -m claude-sonnet-4-6 -c 001 -c 007 # Render leaderboard markdown harness-bench leaderboard --results-dir ./harness-output/ ```
# Evaluation Evaluation is inherited verbatim from ClawBench -- post-session judge comparing agent trajectories against human reference runs under `eval/agentic_eval.md`. ``` 1. Run harnesses (batch) 2. Evaluate (clawbench eval) ───────────────────────── ──────────────────────────────── harness-bench batch ... ──► DOM-match + LLM judge re-used produces harness-output/ exactly as ClawBench does it with 5-layer recordings (same rubric, same prompt) ``` See [ClawBench's eval guide](https://github.com/reacher-z/ClawBench/blob/main/eval/README.md) -- since the recording format is identical, every tool in ClawBench's `eval/` works unchanged on HarnessBench output.
# FAQ
Why two repos instead of one tool with a --harness flag? **Runtime incompatibility.** ClawBench's shared `openclaw-bench` container runs three Python harnesses side-by-side because they share a virtualenv. HarnessBench's six harnesses live in Python, Node/TS, Rust, and Web -- not co-installable in one image. Each gets its own container built on the shared `clawbench/base:` image. **Orthogonal axis.** ClawBench holds the harness fixed and sweeps models. HarnessBench holds the model fixed and sweeps harnesses. Same pipeline, different axis of interest -- keeping them as separate repos avoids overloading either CLI's flag surface.
Do I have to run cloud harnesses? No. `stagehand` and `coze-studio` auto-skip without credentials and appear in the matrix as `skipped:missing_credential:`. The four local-first harnesses (`openclaw`, `hermes`, `claw-code`, `browser-use`) are enough to produce a meaningful leaderboard on any workstation with Docker.
Can I add my own harness? Yes -- three files (`Dockerfile` + `setup.sh` + `run.sh`) plus one `pyproject.toml` stanza. See [`docs/adding-a-harness.md`](docs/adding-a-harness.md). The plugin loads via the `clawbench.harnesses` entry-point group, so external packages can register without forking either repo.
How is this different from ClawBench? - **Axis.** ClawBench: one harness, many models. HarnessBench: many harnesses, one (or many) models. - **Runtime.** ClawBench bundles three Python harnesses in one container. HarnessBench gives each harness its own container (Python / Node / Rust / Web are not co-installable). - **Cloud.** ClawBench is fully local-first. HarnessBench supports local-first **and** cloud-opt-in harnesses in the same matrix. - **Code reuse.** 100% -- HarnessBench imports `clawbench-eval` rather than forking it.
Which base model should I start with? Whatever you already trust. The point of HarnessBench is that you pick one model and observe how different harnesses wrap it. For the published numbers we use `claude-sonnet-4-6` (ClawBench's top scorer at 33.3% overall), which gives every harness a known-competitive model to wrap. Your own runs can use anything in your `models.yaml`.

## Contributing We welcome adapters for new harnesses, especially ones that survive the [30-agent global sweep](docs/scout-2026-04-16.md). Most harness adapters are a single directory under `src/harnessbench/harnesses/` with three files; see [`docs/adding-a-harness.md`](docs/adding-a-harness.md) for the walkthrough. **Quick wins:** - [Add a new harness adapter](docs/adding-a-harness.md) (~1-2 hours if upstream ships a CLI, ~1 day if you're writing one from scratch) - Submit a leaderboard entry for a harness + model pair we haven't scored - File a [good first issue](https://github.com/reacher-z/HarnessBench/labels/good%20first%20issue) ## Community
Discord
English community
Shared with ClawBench 		微信群
中文社区
研究者、开发者、贡献者交流 		GitHub Discussions
Async Q&A
Searchable, long-form, permanent
## License Apache-2.0 for the repository. Each bundled harness adapter links to upstream code under the upstream's own license; nothing from an incompatible license is vendored. ## Citation If you use HarnessBench in your research, please cite: ```bibtex @misc{zhang2026harnessbench, title = {HarnessBench: Comparing Agentic Harnesses on Everyday Online Tasks}, author = {Yuxuan Zhang and Yubo Wang and Yipeng Zhu and Penghui Du and Junwen Miao and Xuan Lu and Wendong Xu and Yunzhuo Hao and Songcheng Cai and Xiaochen Wang and Huaisong Zhang and Xian Wu and Yi Lu and Minyi Lei and Kai Zou and Huifeng Yin and Ping Nie and Liang Chen and Dongfu Jiang and Wenhu Chen and Kelsey R. Allen}, year = {2026}, note = {Preprint in preparation}, howpublished = {\url{https://github.com/reacher-z/HarnessBench}} } ``` ## Core Contributors

Yuxuan Zhang
Yubo Wang
Perry Zhu
Penghui Du
Junwen Miao
## Advisors

Kelsey R. Allen
Wenhu Chen
Dongfu Jiang
Liang Chen
## Support HarnessBench If HarnessBench is useful for your research or tool selection, the single most helpful thing you can do is **[star the repo](https://github.com/reacher-z/HarnessBench)** -- it surfaces the harness-comparison axis to other agent researchers and helps us justify continued adapter work.

Star this repo

Open to contributions -- new harness adapters, leaderboard submissions, or evaluation bug fixes. See [CONTRIBUTING.md](CONTRIBUTING.md).

Contributors

## Star History Star History Chart