Featured in
Featured in
New: Check out our sister project HarnessBench — fixes the base model, varies the harness. Same scoring pipeline, orthogonal axis.
```bash
git clone https://github.com/reacher-z/ClawBench.git && cd ClawBench && ./run.sh
```
Clone → configure → run. Root uv package. Docker-isolated harnesses.
### Can AI Agents Complete Everyday Online Tasks?
**ClawBench is an open-source benchmark that evaluates AI browser agents on everyday online tasks — booking travel, ordering food, applying for jobs, managing email — across live websites. V1 lives in `test-cases/v1/` with 153 tasks across 144 websites; V2 lives in `test-cases/v2/` with 130 tasks. It measures end-to-end task success with a 5-layer recording pipeline and an agentic evaluator that compares each run against human references. Top score to date: 33.3%.**
We asked frontier AI agents to do what people do every day --
中文
What are you looking for?
|
🏆 **See scores** [Live leaderboard](https://huggingface.co/spaces/TIGER-Lab/ClawBench) Pick a corpus (v1 / v2) |
🚀 **Run it on your model** [Quick start ↓](#-human-quick-start) pip install clawbench-eval
|
📊 **Browse 283 tasks** [Task explorer](https://claw-bench.com/tasks) Search · filter · category |
📄 **Read the paper** [arXiv:2604.08523](https://arxiv.org/abs/2604.08523) Methodology · evaluator · results |
|
🎬 **Re-grade old runs** [V1](https://huggingface.co/datasets/NAIL-Group/ClawBenchV1Trace) · [V2](https://huggingface.co/datasets/TIGER-Lab/ClawBenchV2Trace) raw traces 5 layers per (task × model) |
📦 **Download the data** [`hf download NAIL-Group/ClawBench`](https://huggingface.co/spaces/TIGER-Lab/ClawBench) Tasks · rubrics · metadata |
🌱 **Add a task / model** [How to contribute](#contributing) YAML spec + rubric |
❓ **Have a question** [FAQ](#frequently-asked-questions) · [Discord](https://discord.gg/clawbench) Or open an issue |
Live Websites
Isolated Containers
Request Interceptor
Five-Layer Recording
Datasets
ClawBench ships **three** Hugging Face datasets — task definitions plus full execution traces for V1 and V2. All open, downloadable in one command. The benchmark itself is also mirrored on **TIGER-Lab** for visibility.
| Dataset | What's in it | Get it |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- |
| **[NAIL-Group/ClawBench](https://huggingface.co/datasets/NAIL-Group/ClawBench)** _(also mirrored at [TIGER-Lab/ClawBench](https://huggingface.co/datasets/TIGER-Lab/ClawBench))_ | Task definitions, rubrics, and metadata for V1 (153 tasks) and V2 (130 tasks) — what to attempt and how it's judged. | `hf download --repo-type dataset NAIL-Group/ClawBench` |
| **[NAIL-Group/ClawBenchV1Trace](https://huggingface.co/datasets/NAIL-Group/ClawBenchV1Trace)** | One directory per V1 model run, each with `recording.mp4`, `requests.jsonl`, `actions.jsonl`, `agent-messages.jsonl`, `interception.json`, and `run-meta.json` — everything we used to score the run. | `hf download --repo-type dataset NAIL-Group/ClawBenchV1Trace` |
| **[TIGER-Lab/ClawBenchV2Trace](https://huggingface.co/datasets/TIGER-Lab/ClawBenchV2Trace)** | Same 5-layer bundle for **V2** model runs. Rolling — new models added as they're evaluated. | `hf download --repo-type dataset TIGER-Lab/ClawBenchV2Trace` |
> The trace datasets are large; use `hf download --include "
LLM Quick Start
Point your coding agent (Claude Code, Cursor, Copilot, etc.) at [`AGENTS.md`](AGENTS.md) and prompt away.
Human Quick Start
Install ClawBench from PyPI for normal use:
```bash
uv tool install clawbench-eval
```
You can also use `pipx install clawbench-eval` or `python -m pip install clawbench-eval`.
The installed commands are still `clawbench`, `clawbench-run`,
`clawbench-batch`, and `clawbench-harbor-adapt`.
For those want more granular control and contribution, clone the repo and run the root `uv` package entrypoint:
```bash
git clone https://github.com/reacher-z/ClawBench.git && cd ClawBench && ./run.sh
```
**Prerequisites:** [Python 3.11+](https://python.org), [uv](https://docs.astral.sh/uv/), and a container engine — [Docker](https://www.docker.com/) **or** [Podman](https://podman.io/). ClawBench auto-detects whichever is installed; force one with `export CONTAINER_ENGINE=docker` or `export CONTAINER_ENGINE=podman`.
Reproduce the leaderboard
> **Our scores are stable**: two independent runs of the same model under the same judge (`deepseek/deepseek-v4-pro`, lenient rubric) reproduce Intercepted and Reward within ±2 pp on the V2 130-task corpus.
There are **two ways** to verify this on your own machine.
### Path A — Re-run the agent, then score
Confirms the *full pipeline* (your agent + our judge) lines up with our leaderboard row.
```bash
clawbench-batch --models deepseek/deepseek-v4-flash --cases-suite v2 \
--all-cases --harness hermes --no-judge --output-dir ./my-run
clawbench-rescore ./my-run --judge-model deepseek-v4-pro --rubric both
```
### Path B — Skip the run, re-judge our published traces
Confirms *just the judge* matches ours (cheap, no agent compute, useful for sanity-checking your judge config).
```bash
hf download --repo-type dataset TIGER-Lab/ClawBenchV2Trace \
--include "batch-aligned-*/deepseek-v4-flash-free/**" --local-dir ./reproduce
clawbench-rescore ./reproduce --judge-model deepseek-v4-pro --rubric both
```
One-shot equivalent of Path B for any model in the leaderboard:
```bash
clawbench-reproduce --model deepseek-v4-flash --tolerance 2.0
```
### Pass criterion
For `deepseek-v4-flash:free × hermes × v2`, the published row is **Intercepted 3.1% / Reward-lenient 2.3% / Reward-strict 0.0% (3 / 129)**. Path A or B counts as **reproduced** when all three metrics land within ±2 pp. Larger gaps usually mean a different judge model, a different rubric prompt, or a harness configuration drift — diff your `eval_results/
ClawBench-Lite
**New here? Run this first.** [`test-cases/v1-lite/`](test-cases/v1-lite/) is a **20-task curated subset** of the V1 153-task corpus, selected for household-name sites, real-world relevance, difficulty, and category diversity. It matches the 20-tasks-per-source convention of [browser-use/benchmark](https://github.com/browser-use/benchmark) and gives you a credible signal at a fraction of the full-benchmark cost.
Tier distribution: **flagship 9 / core 8 / wildcard 3** — spanning daily life (OpenTable, DoorDash, Instacart, TaskRabbit), entertainment (Eventbrite, Goodreads, Fandango), creation (Asana, Mailchimp, Squarespace), travel (Airbnb), education (LeetCode), dev-tech (GitHub), academia (Overleaf), personal management (1Password), and more. All Lite tasks are judged by [`eval/agentic_eval.md`](eval/agentic_eval.md) regardless of `url_pattern` shape.
The Lite suite is a first-class task directory: run it with `--cases-suite v1-lite`, or inspect the link-backed task files in [`test-cases/v1-lite/`](test-cases/v1-lite/).
Demos
Each ClawBench run produces a full MP4 session recording. See the [project page](https://claw-bench.com) for V1 task recordings.
Example Walkthrough
Curious what one task actually looks like, start to finish? Here's task **001** end to end.
**The task** — from [`test-cases/v1/001-daily-life-food-uber-eats/task.json`](test-cases/v1/001-daily-life-food-uber-eats/task.json):
```json
{
"instruction": "On Uber Eats, order delivery: one Pad Thai, deliver to home address, note \"no peanuts\"",
"time_limit": 30,
"eval_schema": {
"url_pattern": "__PLACEHOLDER_WILL_NOT_MATCH__",
"method": "POST"
}
}
```
The agent gets this `instruction` verbatim, plus read-only access to `/my-info/alex_green_personal_info.json` (the dummy user's name, home address, phone, date of birth) and a disposable email account for any sign-in prompt. It has **30 minutes** to reach a `POST` request — any longer and the container is killed.
**What the agent does** (the happy path):
1. Navigates to `ubereats.com`
2. Reads the dummy user's home address from `/my-info/alex_green_personal_info.json` and enters it in the delivery-address box
3. Searches for **"Pad Thai"** in the food search
4. Picks a restaurant that has Pad Thai available for delivery to that address
5. Opens the item detail page, finds the customization or special-instructions field, enters **"no peanuts"**
6. Adds one to cart, opens the cart, and handles any sign-in prompt using the disposable email credentials
7. Reaches checkout, taps **Place Order**
**What the interceptor catches** — that final *Place Order* tap fires a `POST` request. ClawBench's request interceptor sits in front of the browser and **captures the outbound request before it reaches Uber Eats's servers**, so the dummy user is never actually charged. At the exact moment of interception, all five recording layers (MP4 video, PNG screenshots, HTTP traffic, browser actions, agent messages) are frozen into `/data/`.
**How the judge decides PASS / FAIL** — task 001's `url_pattern` is the intentional sentinel `__PLACEHOLDER_WILL_NOT_MATCH__`, which means **no request path can mechanically match**. The verdict comes from the agentic judge in [`eval/agentic_eval.md`](eval/agentic_eval.md), which replays the five-layer recording against a human reference run and checks four things:
- Did the agent actually reach the final checkout step?
- Is the cart exactly **one** Pad Thai (not two, not a combo)?
- Is the delivery address the user's home address from `alex_green_personal_info.json`?
- Does the order carry the **"no peanuts"** note in the instructions field?
All four must hold for a **PASS**. Miss any one and it's a **FAIL** with evidence from the recording pinned to the failing criterion. This per-task rubric is what makes ClawBench judge-sensitive rather than URL-regex-sensitive — see [`eval/README.md`](eval/README.md) for the full rubric format and [`eval/agentic_eval.md`](eval/agentic_eval.md) for the judge prompt.
Results
CLI
```bash
# Interactive TUI (recommended):
./run.sh
# Single run:
uv run clawbench-run test-cases/v1/001-daily-life-food-uber-eats claude-sonnet-4-6
# Human mode (you control the browser via noVNC):
uv run clawbench-run test-cases/v1/001-daily-life-food-uber-eats --human
# Batch (all models x cases 1-50, 3 concurrent):
uv run clawbench-batch --all-models --case-range 1-50 --max-concurrent 3
# Batch all V1 tasks from test-cases/v1/:
uv run clawbench-batch --models claude-sonnet-4-6 --all-cases --max-concurrent 3
# Batch all V2 tasks from test-cases/v2/:
uv run clawbench-batch --models claude-sonnet-4-6 --cases-suite v2 --all-cases --max-concurrent 3
# Batch converted Claw-Eval tasks from test-cases/claw-eval/:
uv run clawbench-batch --models claude-sonnet-4-6 --cases-suite claw-eval --all-cases
# Batch a custom case directory:
uv run clawbench-batch --models claude-sonnet-4-6 --cases-dir custom-cases --all-cases
# Convert V2 tasks into a local Harbor dataset:
uv run clawbench-harbor-adapt --output-dir ./harbor-datasets/clawbench-v2 --overwrite
# Run the generated Harbor dataset:
uvx --from harbor==0.15.0 harbor run -p ./harbor-datasets/clawbench-v2 -a " Evaluation
Evaluation is a **post-session** step -- first run agents to collect trajectories, then evaluate them against human reference runs.
```
1. Run agents (root uv package) 2. Evaluate (eval/)
───────────────────────── ────────────────────────────────
./run.sh / clawbench-batch ──► Claude Code subagents compare
produces test-output/ agent vs human trajectories
with 5-layer recordings under eval/agentic_eval.md rubric
```
The evaluator compares each agent trajectory against a human reference trajectory across all five recording layers (video, screenshots, HTTP traffic, browser actions, agent messages), then outputs PASS/FAIL with evidence-backed justification.
See [eval/README.md](eval/README.md) for the full evaluation guide and Claude Code prompt template.
FAQ
English community Agent builders, researchers, contributors |
中文社区 研究者、开发者、贡献者交流 |
Async Q&A Searchable, long-form, permanent |
 Yuxuan Zhang |
 Yubo Wang |
 Perry Zhu |
 Penghui Du |
 Junwen Miao |
 Kelsey R. Allen |
 Wenhu Chen |
 Dongfu Jiang |
 Liang Chen |
