---
DistDebug-Bench is a benchmark for evaluating whether LLM agents can diagnose
real-world failures in distributed systems. Each task gives the agent a bug report
from the system's issue tracker, the logs and stack traces attached to it, and the
source code as it was before the fix. The agent has to find the code that caused
the failure. We score its diagnosis against human-labeled root-cause code blocks,
rather than by whether a patch passes tests.
Bugs in distributed systems are hard to diagnose. They depend on timing, message
ordering, and partial failures across nodes, are often difficult to reproduce, and
rarely come with a failing test. In our evaluation of 17 agent configurations, the
best agents find the right files 86% of the time but cover at most 73% of the
root-cause code blocks. Results also vary between runs: on 12–30% of bugs, block
coverage differs by more than 20 points across three attempts.
## ✨ Highlights
- **120 real bugs** from the issue trackers of 11 distributed systems, including
ZooKeeper, Kafka, HDFS, etcd, Kubernetes, Ray, and TiKV. They cover 5 languages
and 10 failure patterns, such as concurrency races, leader-election faults,
replication inconsistencies, and resource leaks.
- **Human-labeled root causes.** Each bug comes with the code blocks that caused it,
so a diagnosis is scored at the file, function, line, and block level.
- **No access to the fix.** The agent's workspace contains only the pre-fix snapshot
(no later commits, branches, tags, or remotes), and its commands run offline in a
[bubblewrap](https://github.com/containers/bubblewrap) sandbox. A leakage check
flags attempts that mention the original issue, its fix pull request, or its fix commit.
- **Works with any agent.** We include Mini-SWE-Agent configs for 13 models in
[`configs/`](configs/mini-swe-agent/), plus Codex CLI and OpenCode harnesses.
Other agents only need to write a `diagnosis.json`.
- **Reproducible results.** All 6,120 scored attempts from the paper are on Hugging
Face, and `paper/reproduce.py` rebuilds the paper's main table and figures from them.
- **A Lite subset** of 34 bugs for cheaper runs.
## 🏆 Leaderboard
Top configurations on the full benchmark (v1.1 labels, 120 bugs, Mini-SWE-Agent, mean
of three attempts per bug). **Block coverage** is the primary metric.
| # | Model | Block coverage | File coverage | Tokens / run |
| :-: | :-- | :-: | :-: | --: |
| 1 | GPT-6 Astra | **72.6** [65.7, 79.3] | 86.1 | 130K |
| 2 | Claude Opus 5.5 | **71.8** [65.2, 78.2] | 84.5 | 149K |
| 3 | GPT-5.5 | **70.0** [63.1, 76.7] | 86.2 | 474K |
| 4 | GPT-5.4 | 66.6 [59.7, 73.4] | 81.0 | 687K |
| 5 | GPT-5.3-Codex | 63.2 [55.9, 70.4] | 79.9 | 417K |
Brackets are 95% bug-bootstrap confidence intervals.
The [leaderboard](https://distdebug.com/leaderboard/) lists all 17 configurations.
To add yours, see [Submitting to the leaderboard](#-submitting-to-the-leaderboard).
## 🧩 The task
**Input.** The issue title and description, its attachments (logs, stack traces,
analyses), and the system's source code at the pre-fix snapshot. The agent works in
a sandboxed shell with no network access.
**Output.** A `diagnosis.json` that describes the cause of the failure and lists the
code blocks responsible:
```json
{
"bug_cause_description": "Leader election retains a stale ZKDatabase ...",
"buggy_blocks": [
{
"file": "src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java",
"function": "ZooKeeperServer.shutdown",
"start_line": 555,
"end_line": 558,
"bug_contribution": "...",
"patch": "..."
}
]
}
```
The scorer compares the predicted blocks with the labeled ones and reports
precision and coverage at four levels (see
[metrics](#evaluation-metrics-produced-by-compute_correctness_report)).
### Systems
| System | Language | Bugs | | System | Language | Bugs |
| :-- | :-- | --: | - | :-- | :-- | --: |
| [Ray](https://github.com/ray-project/ray) | Python, C++ | 21 | | [Kafka](https://github.com/apache/kafka) | Java | 8 |
| [Elasticsearch](https://github.com/elastic/elasticsearch) | Java | 19 | | [etcd](https://github.com/etcd-io/etcd) | Go | 8 |
| [ZooKeeper](https://github.com/apache/zookeeper) | Java | 18 | | [Ignite 3](https://github.com/apache/ignite-3) | Java | 7 |
| [TiKV](https://github.com/tikv/tikv) | Rust | 11 | | [HBase](https://github.com/apache/hbase) | Java | 5 |
| [HDFS](https://github.com/apache/hadoop) | Java | 10 | | [Ignite](https://github.com/apache/ignite) | Java | 3 |
| [Kubernetes](https://github.com/kubernetes/kubernetes) | Go | 10 | | | | |
## 🚀 Quick start
You need Linux with [bubblewrap](https://github.com/containers/bubblewrap),
Python 3.10 or later, and an API key for your model provider. The commands below
run Mini-SWE-Agent with GPT-5.4 on one bug, ZooKeeper-2845, and score the result:
```bash
git clone https://github.com/OrderLab/distdebug-bench.git
cd distdebug-bench
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[agent]"
export OPENAI_API_KEY=... # or the provider key the config needs
python -m distdebug_bench.run --config configs/mini-swe-agent/gpt-5.4.yaml \
--bug zookeeper-2845 --runs 1
python -m distdebug_bench.score results/mini-swe-agent-gpt-5.4 --attempts 1
```
The attempt is saved under `results/mini-swe-agent-gpt-5.4/zookeeper-2845/`.
`configs/mini-swe-agent/` has a config for each model in the paper.
To load the data without running an agent, `pip install -e .` is enough. It has
no dependencies:
```python
from distdebug_bench import get_bug
bug = get_bug(".", "zookeeper-2845")
print(bug.title) # Data inconsistency issue due to retain database in leader election
print(bug.ground_truth_blocks) # the labeled root-cause code blocks
```
## 🧪 Running the full benchmark
The runner reproduces the paper's Mini-SWE-Agent evaluation. One config per
model in the paper's Table 1 is in `configs/mini-swe-agent/`.
```bash
pip install -e ".[agent]"
export OPENAI_API_KEY=... # or the provider key the config needs
# Run every bug until it has three successful attempts (resumable).
python -m distdebug_bench.run --config configs/mini-swe-agent/gpt-5.4.yaml --parallel 8
# Flag attempts that named the original issue, set them aside, and rerun them.
python -m distdebug_bench.leakage results/mini-swe-agent-gpt-5.4 --quarantine
python -m distdebug_bench.run --config configs/mini-swe-agent/gpt-5.4.yaml --parallel 8
# Score and aggregate as in the paper.
python -m distdebug_bench.score results/mini-swe-agent-gpt-5.4
```
Use `--subset lite`, `--bug zookeeper-2845`, or `--system kafka` to run fewer
bugs, and `--runs 1` for a single attempt per bug. Commands run inside
[bubblewrap](https://github.com/containers/bubblewrap), which must be installed
(Linux only).
The protocol, as in the paper:
- **Agent.** Mini-SWE-Agent 2.2.8 with the prompt in
`distdebug_bench/mini_swe_agent_prompt.yaml`, a $3 cost limit per attempt, and
a 60-second limit per command.
- **Input.** The issue title and description, the issue's attachments (copied to
`.artifacts/` in the workspace), and the system's source at the pre-fix
snapshot. The agent writes `diagnosis.json` and checks it with
`.validate_diagnosis.py`.
- **Leakage controls.** Each workspace has a random name and contains only the
snapshot and its ancestors: no branches, tags, remotes, reflog, or later
commits, so the fix is unreachable. `chafa` and `img2txt` are disabled in the
sandbox. `distdebug_bench.leakage` flags attempts in which the agent saw or
typed the original issue or its fix (see [Leakage check](#leakage-check));
flagged attempts are discarded and rerun.
- **Attempts.** Each bug is attempted until it has three schema-valid
diagnoses. Scoring takes the first three in run order, averages them within
the bug, then averages equally over bugs.
Each attempt is saved under `results///run-NN/` as the agent's
trace (`trajectory.json`, `codex_exec.log`, or `opencode_cli.log`),
`diagnosis.json`, `session_stats.json` (API calls, tokens, cost), and, after
scoring, `diagnosis_scores.json`.
### Harnesses
The config selects the harness. All three give the agent the same task text,
attachments, validator, and instructions, in the same anonymized workspace.
| Harness | Configs | Trace | Tested version |
| --- | --- | --- | --- |
| Mini-SWE-Agent | `configs/mini-swe-agent/` | `trajectory.json` | 2.2.8 |
| Codex CLI | `configs/codex/` | `codex_exec.log` | 0.117.0 |
| OpenCode | `configs/opencode/` | `opencode_cli.log` | 1.4.6 |
```bash
npm install -g @openai/codex@0.117.0 # or set `executable:` in the config
python -m distdebug_bench.run --config configs/codex/gpt-5.2-codex.yaml --bug zookeeper-2845 --runs 1
```
A Codex CLI or OpenCode config names the model, an OpenAI-compatible endpoint
(`base_url`), and the environment variable that holds its key (`api_key_env`).
### Isolation
- **Mini-SWE-Agent.** The model is called from the host. Every command the agent
runs executes in bubblewrap with its own network namespace, which has only
loopback. At startup the sandbox tries an outbound connection and refuses to
run if it succeeds.
- **Codex CLI and OpenCode.** The whole CLI runs in bubblewrap without network
access. Its only way out is a gateway on the host that forwards model API
requests (Responses, Chat Completions, model list) to `base_url`. The CLI holds
a placeholder key, and the gateway adds the real one, so the key never enters
the sandbox. The gateway serves only the configured model and rejects requests
that enable hosted tools such as web search, file search, code interpreter, or
remote MCP. It answers proxy requests
for any other host with `403`, and direct connections have no route. Every
decision is logged to `proxy.log` in the run directory.
- **No user configuration.** Each CLI starts from an empty home directory, so no
user settings, MCP servers, hooks, plugins, skills, or credentials load. Codex
runs with `web_search = "disabled"` and its `workspace-write` sandbox for the
commands it runs. OpenCode runs with `webfetch`, `websearch`, and `codesearch`
disabled and its model declared explicitly.
- **Nothing else is visible.** The sandbox sees the workspace, the temporary
home, the CLI's install directory, and system directories such as `/usr` and
`/etc`. The user's home directory and the benchmark's labels are not visible.
### Leakage check
`python -m distdebug_bench.leakage ` scans what the agent wrote and
read, meaning its reasoning, commands, and tool output, in all three trace
formats. It flags an attempt when it finds any of these:
- the issue key, such as `KAFKA-13407`, outside file paths;
- the issue or fix-PR number in a reference, such as `#11563`, `issues/13407`,
`pull/11563`, or `PR 11563`;
- the bare issue or fix-PR number typed in a command or in the model's own text,
unless it reads as a line number, range, or offset;
- the fix commit hash.
The task prompt is not scanned. Numbers that appear in the task are ignored, and
so are line numbers in source listings. The check writes `leakage.json`, and
`--quarantine` moves flagged runs aside so that the runner replaces them.
## 🛠️ Evaluating your own agent
Any agent or harness can be evaluated. Give it what the runner gives
Mini-SWE-Agent: the issue title and description, the issue's attachments, and
the system's source at the pre-fix snapshot, with no internet access. For each
attempt, have it write `/run-NN/diagnosis.json` in the benchmark schema
(checked by `distdebug_bench/validate_diagnosis.py`) and a human-readable
`/run-NN/trajectory.json`. Then run the leakage guard and the scorer on the
results directory, as above.
The [Python API](#python-api) loads a bug and its task prompt, checks out the
pre-fix snapshot so that the fix is unreachable, and scores a single
diagnosis.
## 📤 Submitting to the leaderboard
The [leaderboard](https://distdebug.com/leaderboard/) lists every evaluated
configuration, and the [submission guide](https://distdebug.com/submit/)
summarizes the steps below.
1. Run every bug in the full benchmark (or in Lite) until it has three valid
attempts. Use `distdebug_bench.run`, or your own agent writing the same layout:
`/run-NN/diagnosis.json` (checked by `distdebug_bench/validate_diagnosis.py`)
and a human-readable `/run-NN/trajectory.json`.
2. Run `python -m distdebug_bench.leakage --quarantine` and rerun the
flagged attempts.
3. Copy `submissions/metadata.template.yaml`, fill it in, and package the run:
```bash
python -m distdebug_bench.submit package results/ --metadata metadata.yaml
```
This scores the run and writes `submissions/___/` (metadata,
scores, and diagnoses) plus a trajectories archive.
4. Upload the archive anywhere public (Google Drive, Hugging Face, Zenodo, ...),
set `trajectories_url` in the submission's `metadata.yaml`, and check it:
```bash
python -m distdebug_bench.submit check submissions/___
```
5. Open a pull request adding the submission directory, or email
jyuxuan@umich.edu with the subject `[DistDebug-Bench-Submit] + `
and links to the submission directory and the trajectories.
Rules:
- Every bug in the chosen subset needs three valid attempts; scores are
aggregated by `distdebug_bench.score`. Report Lite results separately; the
[leaderboard](https://distdebug.com/leaderboard/) currently lists full-benchmark runs.
- The agent must not access the original issues, their fixes, or this
benchmark's repository or website. We rerun the leakage guard and review
trajectories.
- Describe the system in a paper, technical report, or blog post (`report_url`).
- We rescore every submission from its diagnoses before listing it.
## 🔬 Reproducing the paper's results
The traces behind the paper's Table 1 are on Hugging Face at
[OrderLab/distdebug-bench-traces](https://huggingface.co/datasets/OrderLab/distdebug-bench-traces):
17 configurations, 120 bugs, and the three scored attempts per bug, for 6,120
attempts. `paper/reproduce.py` rescores every diagnosis with
`distdebug_bench.score` and aggregates the scores with the paper's scripts. It
rebuilds Table 1 and Figures 2 and 3 in `paper/out/` and checks them against
`paper/reference/`:
```bash
pip install -e ".[paper]"
hf download OrderLab/distdebug-bench-traces --repo-type dataset --local-dir traces
python paper/reproduce.py traces
```
It prints whether each configuration's scores, token use, and confidence
intervals match the paper. Pass `--use-released-scores` to aggregate the
stored `diagnosis_scores.json` files instead of rescoring.
## 📚 Reference
### Repository structure
```
.
├── benchmark/
│ ├── systems.json
│ ├── partition.json
│ ├── bug_classifications.json
│ ├── bug_difficulty_characterizations.json
│ ├── lite.txt # DistDebug-Bench Lite: 34-bug subset
│ └── bugs///
│ ├── report.json
│ ├── blocks.json
│ └── artifact_dir/ # optional supporting logs and traces
│
├── distdebug_bench/ # Loader, evaluator, and agent runner
├── assets/logo/ # Logo files (SVG and PNG)
├── configs/mini-swe-agent/ # The paper's model configurations
├── configs/codex/, configs/opencode/ # Codex CLI and OpenCode configurations
├── paper/ # Rebuilds the paper's Table 1 and Figures 2-3 from the traces
├── submissions/ # Leaderboard submissions
├── release/ # Hugging Face export and Croissant metadata
├── tests/
├── collection-pipeline/ # Pipeline used to construct the benchmark
└── utils/
```
### Per-bug file layout under `benchmark/bugs/`
Each bug directory at `benchmark/bugs///` contains two JSON
files and an optional artifacts directory.
#### `report.json`
Bug metadata and natural-language description.
```json
{
"id": "2845",
"system": "zookeeper",
"branch": "release-3.5.3",
"title": "Data inconsistency issue due to retain database in leader election",
"description": "In ZOOKEEPER-2678, the ZKDatabase is retained ...",
"issue_link": "...",
"fix_link": "...",
"severity": "Critical",
"root_cause_type": "Concurrency",
"bug_complexity": "Hard",
"distributed": "Yes",
"contains_analysis": "Yes",
"contains_logs": "No",
"contains_stack_traces": "No",
"artifact_dir": "artifact_dir"
}
```
`branch` is the upstream branch to check out before running an agent.
`artifact_dir`, when present, is a relative path to supporting logs and
traces.
#### `blocks.json`
The ground-truth root-cause code blocks. A bug may have one or more blocks
spanning multiple files.
```json
[
{
"file": "src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java",
"function": "ZooKeeperServer.shutdown",
"start_line": 555,
"end_line": 558
}
]
```
Line numbers may appear as strings or integers depending on the curator's
tooling; the loader coerces both to `int`.
### Python API
Python 3.10 or later is required. We recommend installing into a
virtual environment so that subsequent `python` invocations resolve to
the project's interpreter:
```bash
git clone https://github.com/OrderLab/distdebug-bench.git
cd distdebug-bench
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
```
#### Loading a bug from the benchmark
```python
from distdebug_bench import load_benchmark, get_bug
bench = load_benchmark(".")
print(len(bench.systems), "systems,", len(bench.lite_bug_ids), "Lite bugs")
bug = get_bug(".", "zookeeper-2845")
print(bug.title)
print(bug.branch)
print(bug.ground_truth_blocks)
print(bug.task_prompt)
for bug in bench.iter_bugs("all"): # or "lite"
print(bug.name, len(bug.ground_truth_blocks))
```
#### Subsets
| Subset | Bugs | Use |
| ------ | ---: | --- |
| `all` | 120 | The full benchmark, as evaluated in the paper. |
| `lite` | 34 | DistDebug-Bench Lite: a subset selected for report and label quality and lower evaluation cost. Report Lite results separately. |
#### Checking out the buggy revision of the upstream system
```python
from distdebug_bench import ensure_system_repo, temporary_checkout
system = bench.systems[bug.system]
cache = ensure_system_repo(system, "./.cache/repos")
with temporary_checkout(bug, cache, "./.cache/workspaces") as workdir:
# `workdir` has a random name and holds only `bug.branch` and its
# ancestors, so later commits (including the fix) are unreachable.
# The directory is removed automatically on exit.
...
```
#### Scoring an agent's predicted root-cause blocks
```python
from distdebug_bench import compute_correctness_report
from distdebug_bench.models import BuggyBlock
predicted = [BuggyBlock(
file="src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java",
function="ZooKeeperServer.shutdown",
start_line=555, end_line=558,
)]
report = compute_correctness_report(bug.ground_truth_blocks, predicted)
# To also score mention coverage as in the paper, pass the raw text of the
# agent's diagnosis.json as a third argument.
print(report)
```
### Evaluation metrics produced by `compute_correctness_report`
This is the scorer behind the paper's results: rescoring every attempt in
the paper reproduces its stored scores and Table 1 exactly. Line ranges are
half-open, `[start_line, end_line)`.
| Metric | Unit | A prediction matches the ground truth when |
| ------ | ---- | ------------------------------------------ |
| `file_precision`, `file_coverage` | distinct files | the paths are equal after normalization, or one contains the other |
| `function_precision`, `function_coverage` | distinct functions | the names agree after canonicalization to lowercase `class.method` (argument lists, Go receivers, and `::`/`#`/`$` delimiters are handled; a bare method matches its qualified name) |
| `line_precision`, `line_coverage` | lines | overlapping line numbers in a matching file |
| `block_precision`, `block_coverage` | blocks | the ranges overlap within 5 lines in a matching file (function names are not used) |
| `strict_block_coverage` | blocks | the ranges overlap without tolerance in a matching file |
| `mention_coverage` | blocks | the ground-truth function appears in the diagnosis text (generic names such as `run` must appear as `class.method`) |
Precision divides by predicted files, functions, lines, or blocks; coverage
divides by ground-truth ones. `block_coverage` is the paper's main metric.
## 🗒️ Changelog
- **v1.1 (2026-11-07).** Fixes the gold labels of the eight bugs reported in
[#2](https://github.com/OrderLab/distdebug-bench/issues/2) and corrects five more labels.
- The blocks of etcd-14110, etcd-14325, etcd-14382, etcd-15247, hbase-28830 and
ray-14883 had been labeled against the fix's base branch. They are now mapped
onto each bug's pinned revision.
- tikv-17830's pinned tag already contained the fix. It is re-pinned to the parent
of the fix on release-7.1 (`b00db656`).
- tikv-19220 pointed to a commit outside the TiKV repository. It is re-pinned to
the parent of the fix on master (`6bb8ff37`), and its block now marks the two
lines that cause the failure.
- tikv-18498's block is moved onto its pinned v8.5.1 lines.
- ray-15990, ray-30229, ignite3-21776 and ignite3-22597 name the function that
contains each labeled block. Their line ranges are unchanged.
- ignite3-22597 and hbase-28830 gain fix links.
- The leaderboard and the paper use v1.1. Block coverage rises by 1.4 to 5.4 points
per configuration.
- **v1.0 (2026-11-01).** Initial release.
## 🤝 Contributing
To suggest a bug for the benchmark, open an issue with the
[Bug Entry template](https://github.com/OrderLab/distdebug-bench/issues/new?template=bug-entry.md).
We also welcome pull requests, including scorer fixes and configs for new agents.
## 📄 License
The benchmark annotations and code are released under the [MIT license](LICENSE).
Issue reports and attachments come from public issue trackers and remain subject
to their terms; upstream source code referenced by `report.json` and `blocks.json`
is governed by each project's own license.
## 📝 Citation
If you use DistDebug-Bench in your research, please cite:
```bibtex
@inproceedings{jiang2026distdebugbench,
title = {{DistDebug-Bench}: Can {LLM} Agents Diagnose Bugs in Distributed Systems?},
author = {Jiang, Yuxuan and Liu, Chenglin and Rose, John and Miller, Mason and Thavamani, Ruthesh and Wu, Juntao and Huang, Yinwei and Xu, Jiahao and Huang, Peng},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
}
```
If DistDebug-Bench is useful to you, a ⭐ helps others find it.