# CentaurBench — reproducibility code

<p align="center">
  <a href="https://haas.berkeley.edu/"><img src="dashboard/assets/berkeley-haas-logo.png" alt="Berkeley Haas" width="180"></a>
  &nbsp;&nbsp;&nbsp;&nbsp;
  <a href="https://www.expectedparrot.com/products/edsl"><img src="dashboard/assets/edsl-logo.png" alt="Expected Parrot EDSL" width="150"></a>
</p>

Code and run data for **[CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks](https://arxiv.org/abs/2608.18554)** (Data Innovation and AI Lab, Haas School of Business, UC Berkeley).

This repository is for reproducibility. It contains the benchmark harness, task configurations, pipeline scripts, the static results dashboard front-end, and the **raw outputs of all ten reported runs** under [`results/`](results/) and [`artifacts/`](artifacts/). The large precomputed dashboard data bundles are not tracked; rebuild them locally from `results/` (see below).

The benchmark evaluates LLMs in two regimes:

- **Automation** — a model completes a professional task end-to-end.
- **Augmentation** — an assistant model writes *process-only assistance text* that is passed to a fixed lower-capability worker model (GPT-3.5-Turbo), which then produces the final deliverable. Holding the worker constant isolates the value of the assistant's guidance.

Outputs are scored by a panel of LLM judges via **blind pairwise comparisons** with task-specific and general rubrics, under **leave-one-family-out** masking, replicated across **ten independent runs**. Built on [Expected Parrot / EDSL](https://www.expectedparrot.com/getting-started) ([docs](https://docs.expectedparrot.com/en/latest)).

- **Interactive results dashboard:** https://kennywong524.github.io/centaur-benchmark/
- **Paper:** [arXiv:2608.18554](https://arxiv.org/abs/2608.18554)
- **Lab:** [Data Innovation and AI Lab](https://thedatainnovationlab.com/), Haas School of Business, UC Berkeley

## Repository layout

| Path | Purpose |
|------|---------|
| [`src/centaur_benchmark/`](src/centaur_benchmark/) | Core package: runner (automation + augmentation), panel pairwise judge, optional rubric grading, summarize, EDSL runtime |
| [`tasks/*.yaml`](tasks/) | Per-task prompts, assistance-text templates, judge rubrics, and model lists |
| [`scripts/`](scripts/) | Pipeline scripts: generation, audit, retry, judging, cross-task matrices, dashboard build, figure generation |
| [`dashboard/`](dashboard/) | Static results-explorer front-end (`index.html`, `app.js`, `styles.css`); the data bundle is generated by `scripts/build_dashboard_data.py` |
| [`notebooks/`](notebooks/) | Reference notebook plus stripped legacy Colab exports |
| [`docs/ADDING_A_TASK.md`](docs/ADDING_A_TASK.md) | How to add a new task |

## Requirements

- **Python 3.10+** (EDSL on PyPI targets 3.10–3.12)
- An Expected Parrot account with API credits ([getting started](https://www.expectedparrot.com/getting-started))
- Dependencies declared in [`pyproject.toml`](pyproject.toml) (`edsl`, `pandas`, `numpy`, `PyYAML`, `matplotlib`, `seaborn`)

## Install

```bash
python3 -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install -e .
```

## Authenticate

Create a `.env` file in the repo root (gitignored):

```bash
EXPECTED_PARROT_API_KEY=your_key_here
EXPECTED_PARROT_URL=https://www.expectedparrot.com
```

By default the harness runs locally through the Expected Parrot API proxy (`CENTAUR_EDSL_REMOTE=0`). Recommended timeouts for long tasks:

```bash
export EDSL_API_TIMEOUT=600 REMOTE_PROXY_TIMEOUT=600 EDSL_MAX_ATTEMPTS=8
```

## Reproduce a run

Each run is identified by a `--run-id`. Tasks, model rosters, and token limits are defined in [`tasks/*.yaml`](tasks/).

```bash
set -a && source .env && set +a
export CENTAUR_EDSL_REMOTE=0 PYTHONPATH=src
RUN_ID=my_run_01

# 1) Generate augmentation + automation outputs for all tasks
python scripts/run_full_pipeline.py --run-id "$RUN_ID" --mode generation

# 2) Audit outputs and assistance text before judging
python scripts/audit_run.py --run-id "$RUN_ID" --min-chars 250
python scripts/retry_failed.py --run-id "$RUN_ID" --min-chars 250   # optional

# 3) Panel pairwise judging (n-evals 1 for a cheap first pass; 3 matches YAML defaults)
python scripts/run_full_pipeline.py --run-id "$RUN_ID" --mode judge --n-evals 1

# 4) Cross-task rank / win-rate matrices + heatmaps
python scripts/run_full_pipeline.py --run-id "$RUN_ID" --mode summarize
```

Single task at a time via the CLI:

```bash
centaur-benchmark run --task meal_plan --mode all --run-id my_run_01
centaur-benchmark judge --task meal_plan --run my_run_01 --subset both --eval-model gpt-4.1 --n-evals 3
centaur-benchmark summarize --task meal_plan --run my_run_01
```

## Rebuild the dashboard and paper figures

```bash
python scripts/build_dashboard_data.py   # writes the dashboard JSON bundles from results/
python scripts/make_paper_figures.py     # writes figures from the dashboard bundle
```

Then open `dashboard/index.html` locally or serve the `dashboard/` directory.

## Included run data

All ten reported runs are included so the figures and dashboard can be rebuilt without re-querying any models:

| Run | `run_id` |
|-----|----------|
| Run 1 | `run-01` |
| Run 2 | `run-02` |
| Run 3 | `run-03` |
| Run 4 | `run-04` |
| Run 5 | `run-05` |
| Run 6 | `run-06` |
| Run 7 | `run-07` |
| Run 8 | `run-08` |
| Run 9 | `run-09` |
| Run 10 | `run-10` |

Public run identifiers are deliberately stable and sequential. Every task uses the same `run-01` through `run-10` names; internal development labels have been removed from the release layout.

Output locations:

- `results/<task>/<run_id>/` — raw augmentation/automation outputs, assistance text, and per-judge CSVs
- `artifacts/cross_task/<run_id>/` — cross-task rank/win-rate matrices and heatmaps

New runs you generate locally land in the same directories; the generated dashboard data bundles are gitignored and rebuilt with the command above.

## Citation

If you use this benchmark, please cite [the CentaurBench paper](https://arxiv.org/abs/2608.18554) and [Expected Parrot / EDSL](https://www.expectedparrot.com/getting-started):

```bibtex
@misc{wongchamcharoen2026centaurbench,
  title         = {CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks},
  author        = {Wongchamcharoen, Pattaraphon Kenny and Gulati, Kris and Fong, Min Min and Nagaraj, Abhishek},
  year          = {2026},
  eprint        = {2608.18554},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CY},
  url           = {https://arxiv.org/abs/2608.18554}
}
```

## License

[MIT](LICENSE)