English · 简体中文
--- ## Overview WorkBuddy Bench is a benchmark for evaluating coding agents on real-world work, reverse-engineered from actual developer, PM, algo, QA, ops, and security tasks. Given a task and a sandboxed workspace, an agent is asked to produce the correct change (a patch, an artifact, a report) and is graded against a test suite. The benchmark ships four subsets: | Subset | Tasks | Domain | What the agent does | |--------|-------|--------|---------------------| | **Code** | 80 | repo-level SWE (Python) | Dropped into a real project at a baseline commit; locate and edit code across modules, pass the tests. 18 categories across developer, PM, algo, QA, and ops roles. | | **Web** | 70 | front-end / GUI (HTML/CSS/JS) | Generate, modify, analyze, and QA front-end work: page interaction, data visualization, visual design, front-end project analysis, code testing, page implementation, document conversion. | | **Office** | 50 | office data / file workflows | Operate over mixed-format files (xlsx / csv / pdf / docx) to produce the exactly-correct artifact. Difficulty comes from structure, relationships, state, and evidence chains, not row count. | | **Security** | 60 | security / vuln (red + blue) | Find and safely reproduce real vulnerabilities, analyze malware, run security operations, probe agent attack surfaces, across 6 domains. | This repository is the benchmark's **evaluation framework**. It drops an agent CLI (a *harness*) into a local Docker sandbox, runs it against a batch of tasks, captures what happened (patches, trajectories, test results, efficiency), and scores the run. It is built on top of [Harbor](https://github.com/harbor-framework/harbor). ## Quick start The easiest way to run WorkBuddy Bench is to let a coding agent drive the whole flow for you. This repo ships a **`wbbench-run-setup` skill** (under [`.agents/skills/`](.agents/skills/), also linked at `.claude/skills/`) that takes you end-to-end — from an unconfigured checkout to a running, analyzed evaluation: datasets, environment, model config, `.env` credentials, the job file, launching the run, and finally the analysis report. Open the repo in a skill-aware agent (Claude Code, CodeBuddy Code, …), invoke the skill (e.g. `/wbbench-run-setup`) or simply ask it to *set up a WorkBuddy Bench run*, and it guides you one step at a time. Prefer to configure things by hand — or want to see what the skill is doing under the hood? The **Setup** and **Usage** sections below walk through the same steps manually. ## Setup Every task runs in Docker for reproducibility. You need Python ≥ 3.12, [uv](https://docs.astral.sh/uv/), and Docker. ```bash uv sync # install dependencies cp .env.example .env ``` The task datasets are **not** in this repo. They live on HuggingFace at [`tencent/workbuddy-bench`](https://huggingface.co/datasets/tencent/workbuddy-bench) as one archive per subset. Download the subsets you want into `datasets/`: ```bash ./scripts/dataset/fetch-dataset.sh code # or: web / office / sec / all ``` This fetches, checksum-verifies, and extracts each subset to `datasets/wb-bench-