# Launched cluster runs Tracked log of Slurm runs (results land in the bucket-backed `output/`). ## Goodput probe (GLM-5.2, SWE-bench Pro dev slice) - **Job 22358736** — 2026-07-08, `sbatch scripts/slurm_goodput.sh 20 1 5` (20 dev tasks, parallel 5), **in-container** path (agent runs inside the image via enroot/pyxis). First real multi-task goodput on the fixed code (prior 30x3 run was the buggy 95%-infra-exclusion one; the classification bug is now fixed). Output: `output/logs/goodput_probe.json`. **Result (mini_swe_agent / GLM-5.2):** coverage **1.0**, infra-exclusion **0.0** (flakiness fix validated — no false infra failures over 20 hard tasks), pass@1 **0.10** (2/20), **~37.9 tasks/hr** at parallel 5, **~1.82M tokens/task**, **~$2.74/task** (approx blended $1.5/Mtok; not cache-aware). Extrapolated full 9-harness x 250-test baseline ≈ $6k. ## DeepSWE (Harbor verifier, in-container via enroot/pyxis) - **Oracle validation** — `sbatch scripts/slurm_oracle.sh deepswe 2`. Applies each task's gold `solution.patch` through its own `test.sh`/`grader.py` and asserts reward 1. Fixes en route: (a) enroot `docker://` registry syntax for ECR Public (`public.ecr.aws#...`, not DockerHub); (b) replay `environment/Dockerfile` setup (`go install` reporter, `go mod download`) as a prelude since the published image is the pre-clone base without the verifier's language tooling. Jobs: 22358737/739 (found the two bugs), 22359018 (with both fixes). **Oracle result (22359018):** 2/2 gold solutions -> reward 1 (`abs-module-cache-flags` f2p 20/20 p2p 3/3; `abs-stepped-slices` f2p 6/6 p2p 6/6). **Real rollout (22359086, mini_swe_agent/GLM-5.2):** actually **solved** `abs-module-cache-flags` end-to-end (patch extracted, graded **20/20 f2p, resolved**), ~22 min / ~180 model calls. DeepSWE rollout+grade pipeline fully validated. ## Terminal-Bench 2.1 (89 tasks, verified; single-container, in-container) - Dataset: **`zai-org/terminal-bench-2-verified`** = **Terminal-Bench 2.1**, the verified TB2 that GLM-5 was evaluated on (Z.ai + official TB team fixed env issues in all 89 tasks and instruction/test mismatches in 11; pre-built `xiangyangli/:20260204` images wired into each `task.toml`). `hf download`ed to `runners/tools/terminal-bench-2-verified`. TB2 is a **terminal-task** benchmark: the agent mutates the container (installs packages, writes `/app`) and the verifier's pytest runs against that **same** container end-state (system deps the agent installed must persist), so agent + verifier share one container. `reward.txt` (1/0, from pytest's overall exit) + `ctrf.json` detail. Tests materialised only after the agent step (anti-cheat within a single container). - **Full 89-task oracle -> 76/89 (85%) reproduce** - a big jump from raw TB2.0 (~50% on my 6-task sample). The two raw-2.0 drift failures are fixed by 2.1: `build-pmars` and `adaptive-rejection-sampler` now pass (both in Z.ai's instruction-fix list). Map in `splits/terminal_bench_oracle.json`. - **13 unreproducible tasks** (genuine, not pipeline bugs): `build-cython-ext` (drifted upstream repo test 10/11), `caffe-cifar-10` (arm64 hdf5 / heavy compile), `compile-compcert` (heavy build, no `ccomp`), `make-doom-for-mips` (`clang` missing), `sqlite-with-gcov` (`sqlite3` not built), `headless-terminal` (module not importable), `fix-git`/`git-multibranch`/`configure-git-webserver` (git-state), `mailman`, `path-tracing-reverse`, `protein-assembly`, `rstan-to-pystan` (gold outputs not produced). **Not scoreable fairly** -> exclude from the model denominator (report pass@1 over the 76 reproducible + coverage). - **Two infra bugs found + fixed (matter for real baselines too):** 1. enroot import layer-cache accumulated all 89 distinct images' layers on node-local `/tmp` and filled it (first run 22359516 stalled at ~69/89) -> moved the import cache to `/fsx` (`containers.py`; pyxis per-container overlay stays node-local). 2. **enroot mounted the host `$HOME` over the container `/root`** (host `uv`, `claude`, `cursor-agent`, `hf` leaked into every "clean" image; concurrent tasks raced on the same host `~/.local/bin/uv` install -> `Text file busy` -> false reward 0). Added `--no-container-mount-home` (+ `ENROOT_MOUNT_HOME=n`). This alone recovered `cobol-modernization` + `db-wal-recovery` (both were all-tests-pass-but-reward-0) and makes containers clean + parallel-safe. ## NL2Repo (104 greenfield tasks, in-container, fractional) - Dataset: `runners/tools/NL2RepoBench/test_files` (104 tasks); per-task grading image `ghcr.io/multimodal-art-projection/nl2repobench/:1.0` (public). Reimplemented NL2RepoBench's `post_process_task` on `containers.py`: agent generates a repo in a clean `python:3.12` image (empty `/workspace`, reads `start.md`); host-side clean (drop package files + `py_test_file_list`); overlay onto the image's canonical `/workspace` (tests + `pyproject.toml`) via `cp -a`; run `test_shell` (`pip install -e .` + pytest); score = passed / declared count. - **NL2RepoBench ships no reference solutions**, so an empty-overlay oracle is invalid (the impl must come from the agent; verified: canonical `/workspace` = tests + pyproject only). Instead validated **end-to-end with a real rollout** (22359249, mini_swe_agent/GLM-5.2 on `python-slugify`): agent generated `slugify/` in `python:3.12`, workspace exported + cleaned (agent's `test.py` removed, canonical preserved), `pip install -e .` succeeded, pytest ran the canonical `test.py` against the agent's impl -> a `SyntaxError` in the agent's own `slugify/slugify.py:21` -> 0/82 (correct fractional `task_failure`, no crash/infra). Overlay + clean + grade + parse all validated; the 0 is a genuine near-miss generation, not a pipeline bug. ## Summary: all four benchmarks implemented + validated on enroot/pyxis | Benchmark | Tasks | Model in loop | Grading (in-container) | Validation | | -------------- | ----- | ------------- | ---------------------------------- | --------------------------------------- | | SWE-bench Pro | 731 | yes (goodput) | scaleapi entryscript -> output.json | 20-task goodput: cov 1.0, pass@1 0.10 | | DeepSWE | 113 | yes (1 task) | Harbor `test.sh`/`grader.py` | oracle 2/2 + real solve 20/20 | | Terminal-Bench 2.1 | 89 | - | task `test.sh` -> reward.txt | oracle **76/89 (85%)** reproduce | | NL2Repo | 104 | yes (1 task) | overlay + `test_shell` (pytest) | real rollout end-to-end (0/82 near-miss) | - **Job 22358363** — submitted 2026-07-07, `sbatch scripts/slurm_goodput.sh 30 3 8` (30 dev tasks x 3 repeats, parallel 8), harness `mini_swe_agent`, runtime enroot/pyxis. Measures goodput (tasks/hour), $/task, infra-exclusion rate, and pass@1 variance across repeats. Output: `output/logs/goodput_probe.json` + per-unit `scores.json` under `output/results/swebench_pro/goodput_repN/`. - Validated beforehand by 1-task smokes: the full pipeline (image import -> pyxis checkout -> mini-swe-agent GLM-5.2 rollout with token logging -> in-image grading) runs end-to-end on hopper-cpu; a real `task_failure` grade and 152 logged GLM calls were observed. ## How to launch the SWE-bench Pro baseline All ten harnesses are installed and execute (verified): mini_swe_agent, openhands, swe_agent, pi (0.80.3), opencode (1.17.14), codex (0.142.5), qwen_code (0.21.1), crush (0.87.0), goose (1.41.0), claude_code (cluster). Launch one baseline per harness: ```bash # one harness on the held-out test split (250 tasks, 1 trial) sbatch scripts/slurm_baseline.sh swebench_pro mini_swe_agent test 8 # full matrix: for h in mini_swe_agent openhands swe_agent pi opencode goose codex claude_code qwen_code crush; do sbatch scripts/slurm_baseline.sh swebench_pro "$h" test 8 done # then: uv run python report.py --benchmark swebench_pro --subset test ``` Proxied harnesses (codex/claude_code) route GLM-5.2 through the LiteLLM format proxies; their results carry a `proxied` caveat - validate the translation with the proxy-fidelity smoke before trusting scores.