# Coding Agent Bench Baseline agentic coding harnesses on agentic coding benchmarks with a fixed base model (**GLM-5.2 via Hugging Face Inference Providers**), mirroring the structure of [`legal_agent_bench`](../legal_agent_bench). SWE-bench Pro is the primary target (frozen, diverse dev/test split); DeepSWE, Terminal-Bench 2.0, and NL2Repo have implemented + validated adapters whose whole-set baselines run on demand. All four run the agent + grade **inside** the benchmark image via enroot/pyxis and are validated on the cluster (see [`RUNS.md`](RUNS.md) and [`implementation_decisions.md`](implementation_decisions.md)). Research background lives in [`background/`](background). ## Headline result Ten harnesses x two models x 250 held-out SWE-bench Pro tasks, one rollout each. ![pass@1 against cost per task](assets/pareto_cost_vs_pass1.png) Swapping the harness moves pass@1 from 23.2% to 52.4% on GLM-5.2 and from 15.2% to 36.0% on Gemma 4 26B-A4B, and the two rankings are uncorrelated (Spearman -0.05): vendor-shipped harnesses fall on the small model while model-agnostic ones climb. All 5,000 rollouts are published at [`joelniklaus/CodingBenchmarkResults`](https://huggingface.co/buckets/joelniklaus/CodingBenchmarkResults); `scripts/cost_analysis.py` regenerates the figure and the table behind it. The bucket also carries `gemma-4-26B-A4B-it-thinkoff`, a partial rerun with thinking disabled (8 harnesses, 86-250 tasks each, no `.summary.json`). `cost_analysis.py` skips it via `ABLATION_SUFFIXES` because its denominators are uneven; the bucket README scores it from the raw `scores.json` files instead. ## Layout - `config.py`, `schema.py`, `flakiness.py` - shared config, the uniform result schema (binary `resolved` + fractional `score`), and infra-failure handling. - `antihack.py` - reward-hacking defenses (see below), with `scripts/scan_reward_hacking.py` as the reporting tripwire. - `run_task.py`, `benchmark.py`, `report.py` - single-unit runner, parallel sweep (joblib + run-id cache), and the baseline report (pass@1 + binomial 95% CI + coverage). - `benchmarks/` - one adapter per benchmark (`swebench_pro`, `deepswe`, `terminal_bench`, `nl2repo`) behind a common `BenchmarkAdapter` protocol. - `harnesses/` - harness registry + capability matrix, per-harness GLM wiring, and the generic git-diff adapter (run a CLI agent in the repo, extract `git diff`). - `proxies/` - LiteLLM format proxies (Anthropic / Responses), the Codex Responses sanitizing adapter (flattens proprietary `namespace` tools), a proxy-fidelity smoke test, and `load_balancer.py` fronting a multi-replica serve pool. - `serve_utils.py` + `scripts/wait_for_serve.sh` - open-dirac-style `endpoint.env` discovery + `/health` wait for local vLLM serves (`scripts/serve_gemma.sh`). - `splits/` - `build_profile.py`, `build_split.py`, frozen `coding_split.json`. - `runners/` - isolated uv envs / CLI installs for each benchmark runner + harness (pinned versions), created by `scripts/setup.sh`. - `scripts/` - `setup.sh`, `mount_bucket.sh`, `build_release.py`, `goodput_probe.py`, `fable_qc.py`, Slurm submit scripts. - `tests/` - adapter tests (oracle/gold grading, minimal e2e, parsing, negative). ## Setup ```bash cd meta-harness uv venv --python 3.12 && uv sync # orchestrator env cp .env.example .env # set HF_TOKEN (+ optional HF_BILL_TO) bash scripts/setup.sh # isolated runner envs + CLI harnesses bash scripts/mount_bucket.sh # mount output/ -> HF bucket ``` `node`/`npm` are required for the npm harnesses (Pi, OpenCode, Codex, Qwen Code, Crush, OpenClaw); `scripts/setup.sh` checks for them and skips those installs if absent. Docker is required for every benchmark sandbox (the cluster hopper-cpu nodes have it). ## Outputs & the HF bucket `output/` is gitignored and backed by the HF bucket [`joelniklaus/CodingBenchmarkResults`](https://huggingface.co/buckets/joelniklaus/CodingBenchmarkResults), the source of truth. Mount it read-write with [`hf-mount`](https://github.com/huggingface/hf-mount): ```bash HF_TOKEN=$(hf auth token) hf-mount start bucket joelniklaus/CodingBenchmarkResults \ "$(pwd)/output" --advanced-writes # one-shot alternative: hf buckets sync ./output hf://buckets/joelniklaus/CodingBenchmarkResults ``` Because everything under `output/` is published to that bucket, **no credential may be written there**. `agent_snippet()` therefore takes no token: `containers.py` injects `HF_TOKEN` into the container environment, and each harness reads it from there (shell `export`, or a `$HF_TOKEN` reference in its own config, which pi/crush/opencode/codex all resolve themselves). `tests/test_harness_wiring.py` fails any harness that puts the value in a file or command line. Results written before this landed did contain the key, in 4472 files: `agent_cmd.sh`, crush's `crush.json`, a handful of `agent_stdout.log` / `mini_traj.json`, and codex's shell snapshots. `scripts/scrub_credential.py` has redacted all of them (match by token shape, so it never handles the secret and also catches already-rotated keys): ```bash rg --files-with-matches --no-ignore --hidden 'hf_[A-Za-z0-9]{20,}' output/ > /tmp/hits.txt uv run python scripts/scrub_credential.py --from-list /tmp/hits.txt --apply ``` The key was still published for as long as those results existed, so **rotate that token**. ### Publishing a finished sweep `output/` is the working tree and is not what gets published: most of its bytes are harness home directories (caches, sqlite journals, shell snapshots), which are an order of magnitude larger than the record of the run and are where a harness writes a credential if it writes one at all. `scripts/build_release.py` copies only the artifacts that describe a run -- `scores.json`, `hf_router_usage.jsonl`, `agent_cmd.sh`, `agent_stdout.log`, `grade/`, and the two directories that hold real transcripts -- into `output-release/`, which is likewise gitignored. ```bash uv run python scripts/build_release.py # output/ -> output-release/ rg -l --no-ignore --hidden 'hf_[A-Za-z0-9]{20,}' output-release/ > /tmp/hits.txt uv run python scripts/scrub_credential.py --from-list /tmp/hits.txt --apply hf buckets sync ./output-release hf://buckets/joelniklaus/CodingBenchmarkResults ``` Run the scan every time. The SWE-bench Pro release scan found the live token in 25 files -- agent stdout and codex session transcripts where the agent echoed its environment -- written well after `container_agent.py` stopped passing tokens on the command line. ## Running baselines ```bash # SWE-bench Pro (primary): all harnesses on the held-out test split, 1 trial uv run python benchmark.py --benchmark swebench_pro --subset test --harness mini_swe_agent # DeepSWE / Terminal-Bench 2.0 / NL2Repo (whole-set held-out), on demand uv run python benchmark.py --benchmark deepswe --subset test --harness mini_swe_agent uv run python benchmark.py --benchmark terminal_bench --subset test --harness mini_swe_agent uv run python benchmark.py --benchmark nl2repo --subset test --harness mini_swe_agent ``` All four benchmarks support the same ten in-container harnesses (`mini_swe_agent`, `pi`, `opencode`, `goose`, `codex`, `claude_code`, `qwen_code`, `crush`, `hermes`, `openclaw`) -- the list is derived from `container_agent.supported_harnesses()` rather than repeated per adapter. Validate any benchmark's grading path against its reference solution before spending model budget: ```bash sbatch scripts/slurm_oracle.sh swebench_pro 60 "$PWD/output/oracle/swebench_pro" 6 sbatch scripts/slurm_oracle.sh deepswe 2 # gold solution must resolve sbatch scripts/slurm_oracle.sh terminal_bench 6 # (TB2: some golds fail on env drift) # Re-check only what failed a previous sweep (ids file = whitespace-separated). sbatch scripts/oracle_retry_ids.sh swebench_pro "$PWD/output/oracle/retry_ids.txt" \ "$PWD/output/oracle/retry" 3 ``` `--n` draws a *seeded random* sample: the datasets are ordered by repo, so a head slice measures one repo and nothing else. The SWE-bench Pro oracle currently reproduces **59/60** gold patches. Note this only detects *ungradeable* tasks; it cannot see the underspecified-prompt/overly-strict-test failures behind OpenAI's [~30% broken-task estimate](https://openai.com/index/separating-signal-from-noise-coding-evaluations/), since a gold patch passes those by construction. Scoring is per benchmark and never averaged across benchmarks: the binary benchmarks report pass@1 + binomial 95% CI + coverage; NL2Repo reports mean test-pass-rate + coverage. ### Debugging a harness that scores zero Check the **empty-patch rate** (`candidate.patch` of size 0) before concluding the model is too weak — on the first Gemma dev slice it explained nearly all the spread (opencode 0/30 empty and 6 resolved; goose 19/30 and codex 20/30, both 0 resolved). A harness that never lands an edit is almost always misconfigured, and two wiring invariants cover the cases seen so far (both enforced by `tests/test_harness_wiring.py`): - **Every harness must receive `AGENT_TASK_PROMPT`**, not just the task text. Goose was driven with `-i TASK.md`, so it never read "edit files ... keep going until the task is complete" and spent its whole turn budget exploring. - **Every harness must be told the serve's real context window and output cap.** Left on frontier defaults, Codex neither compacted in time nor capped its output. ### When a unit produces nothing to grade Two very different things end a rollout with no gradeable output, and the usage-proxy log tells them apart by whether the model was ever called. They get **different outcome classes**, because the choice of which one to use is a choice about the denominator: | outcome | when | denominator | fault | |-------------------|--------------------------------|-------------|-----------------| | `infra_unresolved`| calls made, none returned 200 | excluded | serving capacity | | `harness_failure` | zero model calls: died before the first request | **counted, scores 0** | ours | Excluding is right for the first: the endpoint misbehaved, so the run says nothing about the harness and counting it as a loss would be measuring the weather. It is wrong for the second, and that distinction is not academic — collapsing them is how a broken harness hides. Hermes exited instantly on 29 SWE-bench Pro tasks that the other nine harnesses all handled, and because that was filed as infrastructure it was quietly scored over 221 tasks while everything else was scored over 250. Both are retried the same way (a single failed exec could be a cold mount); only the verdict after four attempts differs. `scripts/interim_status.py` marks a harness-failure count as a trailing `!N` on the cell, and `benchmark.py` prints it beside coverage. The whole split rests on "zero model calls", so what counts as a call decides who is blamed. The proxy writes a record per request, but only a *completed* one carries a status: when the endpoint is unreachable it writes an `error` instead. Those were not counted, so an endpoint that went away mid-sweep produced a rollout with hundreds of attempts and zero calls — which is exactly the signature of a harness that never started, and it was cached as a permanent zero against the harness. Restarting the GLM balancer under a live codex sweep cost it 8 units that way, and ended 8 more that were graded as honest losses. `usage_log` now counts an unreachable attempt as a call and treats it as the endpoint ending the run, the same verdict a final 404 already got. ```bash uv run python scripts/triage_infra.py --harnesses pi,opencode --subset dev ``` The classic `harness_failure` cause is a libc mismatch: about an eighth of SWE-bench Pro instances build on Alpine, and a vendored glibc binary cannot start under musl (`can't execute 'node'`, or `Error relocating ...: fcntl64: symbol not found`). `install_node_harnesses.sh` fetches the musl Node build too and `containers.py::toolchain_path_prelude` picks between them per container. On the first Gemma dev-481 sweep *all* 100 such units across pi and opencode had this cause, which is what prompted the per-container toolchain selection. ### What a harness costs, not just what it scores `scripts/cost_analysis.py` prices each harness's token bill and plots pass@1 against cost per task, over a single Pareto frontier spanning both models: ```bash uv run python scripts/fetch_logos.py # once: harness logos into assets/logos uv run python scripts/cost_analysis.py # table + output/analysis/*.png ``` Ranking by tokens and ranking by money are not the same, because input and output are priced ~3-5x apart and agent loops are overwhelmingly input: the measured split runs from 15:1 to 250:1, so 92-99% of what a harness pays for is conversation it already sent. The script estimates that re-sent share from the per-call prompt sizes (a prompt no larger than the previous one is prefix) and reports a second column billing it at the provider's cached-input rate, which is the realistic bill and where the cheapest harness changes. Prices are hardcoded in `PRICING` and dated in the comment above it; update them rather than trusting a stale rate card. It renders with matplotlib because kaleido's headless Chrome has no system GTK libraries on this cluster. The figure is built to be read small, in a timeline: dots are uniform and coloured by model, each name is preceded by its harness logo (fetched by `scripts/fetch_logos.py`, provenance recorded there), and pairings the frontier dominates are faded so the winners read first. `PLOT_OMIT` drops harnesses from the figure only — the table and every reported number still cover all of them — because eight of Gemma's ten sit inside a tenth of the axis width and no label placement survives that. ### Why a unit lost, not just that it did `scripts/failure_breakdown.py` splits every non-resolved unit by cause, because "wrote nothing", "wrote a patch that missed", "broke the repo" and "was still working at the cap" need completely different fixes and pass@1 collapses all four: ```bash uv run python scripts/failure_breakdown.py --subset test uv run python scripts/failure_breakdown.py --subset test --model GLM-5-2-FP8 --detail ``` The class that has mattered most is `no_patch`, and its size is a property of the *serve* as much as the harness: GLM ran at 37% no-patch against Gemma's 8%, and the difference was the output cap truncating a thinking model mid-reasoning (see `blog_notes.md`). Read it alongside the `versions` stamp — a harness whose units predate a cap change is not comparable with one whose units do not. Claude Code, opencode and goose are native ELF binaries rather than Node scripts, but all three publish musl builds at the same pinned versions, so the installer fetches those into `runners/tools/musl/bin` and the prelude puts that dir ahead of `npm-global/bin` (whose same-named glibc shims would otherwise win the PATH lookup). Two further gaps closed after the first full sweep, both invisible until per-unit errors recorded which binary failed: - musl Node needs a libstdc++ newer than several images ship (`_M_replace_cold` arrived in GCC 12), which cost every Node-based harness the same ~36 units. The installer carries Alpine's libstdc++ and puts it on `LD_LIBRARY_PATH` on the musl branch only. - Every Python harness runs from a uv venv on a **glibc** CPython, so its console script could not start at all: mini-swe-agent lost 59 units and hermes 29. A musl interpreter cannot be run here to build a venv, so `musl_py_harness` cross-installs the dependency tree with `uv pip install --python-platform x86_64-unknown-linux-musl --target` and drops a wrapper in `musl/bin` that feeds it to a musl CPython. Nothing in the harness snippet changes: the wrapper shadows the glibc console script via the same PATH ordering. Adding a Python harness means adding one `musl_py_harness` line next to the others, or it will silently forfeit every Alpine instance. codex is the one harness that cannot be fixed this way -- it publishes no musl build, so ~4% of instances are permanently unevaluable for it. Fixing the toolchain does not fix the numbers: a `harness_failure` is cached like any other result, so the old zeros stay until they are dropped. There is no rollout to re-read, only an absence, which is why this needs its own pass rather than `reclassify.py`: ```bash uv run python scripts/invalidate_harness_failure.py --subset test --harness hermes # --apply ``` `scripts/probe_hermes_images.py` is the other half: it runs the real launcher in the real image and reports libc, interpreter and whether either can exec, because "cannot execute: required file not found" is what a shell says both when a shebang interpreter is missing and when it is present but linked against the wrong libc. Token counts of zero are a *telemetry* symptom, not proof a harness is idle: harnesses that reach the model over `/v1/responses` (Claude Code via LiteLLM, Codex) report `input_tokens`/`output_tokens` nested under `response`, which the usage proxy now normalizes. Cross-check `litellm_anthropic.log` / the agent transcript before concluding a harness made no calls. ## Reward hacking SWE-bench Pro hands the agent a clone that already contains the merged fix: each task's `before_repo_set_cmd` installs the official tests by checking them out of the solution commit, so that commit is sitting in `.git` while the agent works. Reading it is much easier than solving the task, and Datacurve's audit flagged Opus 4.6/4.7 as having done so on >12% of reviewed tasks (Scale issue #93). This matters more here than for a one-off evaluation: an optimizer that searches over harnesses will find whatever shortcut the scoring leaves open, and a harness that learns to read the answer looks like a breakthrough. `antihack.py` implements three layers, wired into the git-diff adapter: - **Scrub the future.** Before the agent gets a shell, every ref and remote is deleted, the reflog is expired, and the repo is repacked with `git gc --prune=now`. Dropping refs alone is not enough -- objects survive in the pack and `git show ` still resolves, and the SHA is guessable because it appears in the instance id. History up to the base commit is kept, so repos whose build reads git tags still behave normally. - **Freeze the tests.** `before_repo_set_cmd` only restores the test files the solution commit *changed*; a task's `selected_test_files_to_run` typically also lists files it never touched, and nothing was restoring those. A candidate patch editing one of them used to be graded against its own edited assertions. All graded test files are now restored after the patch is applied. - **Watch for the rest.** `scripts/scan_reward_hacking.py` reports, per harness, how often a run probed git history or edited a graded test, and how often that happened on a unit scored *resolved*. Detection is the backstop for exploits we did not anticipate; a hack rate that climbs as a harness is optimized is the signal to stop and look. Run it on a compute node (`sbatch scripts/slurm_hack_scan.sh dev`); from the login node reading every transcript takes hours. Auditing the Gemma dev-481 baselines, which ran before these defenses landed, found the history hole was never exploited (1 unit in 3848) and four resolved units whose verdict an unrestored test edit could have changed. See [reward_hacking.md](reward_hacking.md) for the full threat model, the two rounds of detector false positives, and per-harness numbers. ## Cluster container runtime The hopper-cpu nodes have **no accessible Docker daemon** but do have **enroot** (rootless, `/usr/bin/enroot`, enroot 3.4.1). `containers.py` abstracts the runtime: - **enroot** (default on the cluster): `import docker://` -> cached `.sqsh` (under `output/enroot/sqsh`, on shared /fsx) -> `create` -> `start --rw --mount`. Force it with `CODING_CONTAINER_RUNTIME=enroot` (set in `.env`). The import layer-cache lives on `/fsx` (it accumulates every distinct image's layers — 89 for TB2.1 — which overflows node-local `/tmp`); pyxis's per-container writable overlay stays node-local. Set `CODING_ENROOT_SCRATCH` to override the cache location. - **docker**: used automatically when a daemon is reachable. Two enroot/pyxis details the container work depends on: - **Image extraction needs node-local NVMe.** Layer unpacking buffers in `$TMPDIR` and builds an overlayfs, so it needs lots of space *and* a local fs. We use `/opt/dlami/nvme` (~27T); `/tmp` (~19G) overflows on big Node monorepo images and `/fsx` rejects overlayfs whiteouts. Override with `CODING_ENROOT_TMPDIR`. - **pyxis drops the image's own `PATH`.** Containers start with a bare default environment, so toolchains outside `/usr/bin` vanish — `go` lives in `/usr/local/go/bin`. Both the grading entryscript and the agent rollout replay the image environment from `/etc/environment` (`image_env_prelude()`); without it every Go task (~40% of SWE-bench Pro) scores 0 with `go: command not found`. **SWE-bench Pro** grading is self-contained (no daemon needed): we build the same entryscript the scaleapi runner uses (reset to base_commit, `git apply`, run the instance `run_script.sh` + `parser.py` -> `output.json`) and run it in the prebuilt image via the backend, then apply `(fail_to_pass | pass_to_pass) <= PASSED`. Validated end-to-end on hopper-cpu via enroot/pyxis: image import -> pyxis checkout -> mini-swe-agent GLM-5.2 rollout (tokens logged via the usage proxy) -> in-image grading returning a real verdict. Scale up the runnable SWE-bench Pro baseline with: ```bash sbatch scripts/slurm_goodput.sh 30 3 8 # goodput probe (dev slice) sbatch scripts/slurm_baseline.sh swebench_pro mini_swe_agent test 8 # baseline (test 250) ``` All twelve harnesses are installed under `runners/tools/` (shared on /fsx; run `scripts/setup.sh`, or just `runners/install_node_harnesses.sh` for the Node ones): mini-swe-agent, OpenHands, SWE-agent, Hermes 0.19.0 (python venvs); Pi 0.80.3, OpenCode 1.17.14, Codex, Qwen Code 0.21.1, Crush 0.87.0, OpenClaw 2026.7.1-2 (Node v24 LTS, user-space); goose v1.41.0 (binary); Claude Code (cluster). `source runners/harness_env.sh` puts them on PATH inside a job (the Slurm baseline script does this automatically). Note Hermes comes from PyPI, not from the npm package of the same name -- that one is a third-party bridge whose postinstall shells out to the same `pip install`. So the full 12-harness SWE-bench Pro baseline runs on enroot today (proxied harnesses - Codex/Claude Code - go through format proxies; Codex also gets `proxies/codex_adapter.py` to scrub `namespace` / `web_search` tools that GLM rejects). Validate with the proxy-fidelity smoke first. ### Cross-task memory is off everywhere Every unit runs in a container that is created, used and destroyed, with the host `$HOME` unmounted, so state cannot cross tasks by default. Two harnesses had to be argued out of it explicitly, and both defaults are on: - **Hermes** auto-selects a `coding` *posture* toolset in a repo, which bundles `memory`, `skill_manage` and `web_search`. 0.19.0 has no `memory:` config block, so the toolset allowlist (`-t file,terminal,todo`, plus `agent.disabled_toolsets`) is the whole of the off switch; `--ignore-rules` also drops memory and skill injection. - **OpenClaw** writes itself new skills from a finished run (`skills.workshop.autonomous`), and its memory search reaches for an embedding provider of its own -- a second, unmeasured model inside a single-model measurement. Both are off, along with every web and browser tool. `tests/test_harness_wiring.py` fails any harness that keeps state outside its per-task `run_dir`, resumes a previous session, or sets its own reasoning level, and pins the two settings above by name. ### Local model serve (open-dirac glue) ```bash sbatch scripts/serve_gemma.sh # writes output/serve/endpoint.env eval "$(scripts/wait_for_serve.sh)" # exports CODING_MODEL* (+ CONTEXT_LEN / MAX_OUTPUT) scripts/slurm_baseline.sh swebench_pro mini_swe_agent dev 2 20 # One-task smoke across all 10 in-container harnesses (waits for the serve): sbatch scripts/smoke_harnesses.sh # ...or one job per harness, so the ten run in parallel: for h in mini_swe_agent pi opencode goose codex claude_code qwen_code crush hermes openclaw; do HARNESSES="$h" sbatch scripts/smoke_harnesses.sh done ``` `smoke_harnesses.sh` is the only check that runs the *production* path (real image, real enroot mounts, real toolchain `PATH`), so run it after touching mounts, the `PATH` prelude, or a harness adapter. `smoke_one_harness.sh` skips the container on purpose, which makes it much faster for debugging a CLI's config but blind to anything the container does. Local serves publish `CONTEXT_LEN` / `MAX_OUTPUT` in `endpoint.env`. Harnesses that advertise a context window (goose, opencode) and the LiteLLM format proxy (claude_code) read `CODING_MODEL_CONTEXT_LEN` / `CODING_MODEL_MAX_OUTPUT` so they don't assume a 200k/32k Claude-scale window against a 32k Gemma serve. ### Serving a pool (load balancer) One serve is not enough for a full sweep: a single replica admits ~12 concurrent full-context sequences, while seven harnesses at parallel 20 offer ~140. `multi_serve.sh` launches N replicas plus `proxies/load_balancer.py` (adapted from open-dirac) in front of them, and publishes one pool endpoint: ```bash scripts/multi_serve.sh 3 # 3 replicas + balancer -> output/serve/pool.env eval "$(scripts/wait_for_serve.sh)" # prefers pool.env, falls back to endpoint.env uv run python scripts/pool_probe.py --concurrency 192 # goodput + per-backend fan-out ``` A pool is described by five values — model, serve script, Slurm job name, endpoint directory and context length. `SERVE_KIND=gemma|glm` fills them in for the two models set up here, each field stays independently overridable, and **any other model works by supplying them**; `SERVE_POOL_DRY_RUN=1` prints the resolved layout without launching: ```bash SERVE_KIND=qwen SERVE_MODEL=Qwen/Qwen3-Coder SERVE_SCRIPT=scripts/serve_qwen.sh \ SERVE_JOB_NAME=qwen-vllm SERVE_ENDPOINT_DIR=output/serve/qwen/jobs \ SERVE_MAX_MODEL_LEN=65536 sbatch scripts/serve_pool.sh ``` A GLM-5.2 replica is **two** nodes (744 GB of FP8 weights), so node cost is twice the replica count. Its in-flight cap is 12 per replica against Gemma's 64, because what an agent under a wall-clock cap needs is per-request decode speed, not aggregate pool throughput: a replica gives 13.7 tok/s per request at 32 concurrent but 36.0 at 8 (`glm_serving.md`). Sizing for aggregate throughput instead made GLM hit the 3600s per-task cap on 66% of tasks, which measures the queue rather than the model. `SERVE_LB_PARTITION` pushes the balancer off a GPU node when every one is needed: ```bash SERVE_KIND=glm scripts/multi_serve.sh 12 # 12 replicas = 24 nodes + balancer for h in mini_swe_agent pi opencode goose codex claude_code qwen_code crush; do sbatch --export=ALL,USE_LOCAL_SERVE=1,TASK_TIMEOUT_S=7200 --job-name="glm-$h" \ scripts/slurm_baseline.sh swebench_pro "$h" dev 18 done uv run python scripts/interim_status.py # pass@1 while the sweep is still running ``` Set sweep parallelism so the total offered load matches pool capacity (replicas x per-replica cap); surplus beyond that only queues in the balancer. `TASK_TIMEOUT_S` overrides the per-task wall-clock cap for a model slow enough to be truncated at the default — pair it with `invalidate_timed_out.py --model --any-patch`, or the raise applies only to the units that had not been graded yet and the run mixes two regimes. ### DeepSeek-V4-Flash on H100 (second reference model) `scripts/serve_deepseek.sh` serves `deepseek-ai/DeepSeek-V4-Flash-0731` on **one** node. It is a 304B model that fits on 8xH100 because `expert_dtype: fp4` puts the experts — nearly all the parameters — in MXFP4, for 167 GB of weights against GLM's 744. Three of the vendor's published vLLM flags are Blackwell-only and must be dropped on Hopper: `--moe-backend deep_gemm_mega_moe` raises `NotImplementedError` *after* the checkpoint loads, `use_fp4_indexer_cache` asserts in the DSA indexer, and `--data-parallel-size 4` is a GB300 node shape. What is left auto-selects Marlin for the MoE and an FP8 indexer cache. Two more constraints are not tunable: the sparse-MLA backend accepts only `--block-size 256`, and DSpark rejects `num_speculative_tokens` below the checkpoint's `dspark_block_size` (5) because shorter blocks produce garbled text rather than merely poor acceptance. The repo ships **no Jinja chat template** — only an `encoding/` folder of Python. vLLM carries its own port and selects it automatically for `DeepseekV4ForCausalLM`, so `/v1/chat/completions` works; tool calls arrive as `<|DSML|>` XML and need `--tool-call-parser deepseek_v4`. All of this needs vLLM **0.26.0**: the cached `:latest` image is 0.24.0, which has the tokenizer but neither parser nor DSpark. ```bash scripts/serve_sweep_dsv4.sh # 15 config variants, 1 node each scripts/serve_sweep_dsv4.sh --collect # throughput table ``` **A sweep reserves 4 cores and 512 G, and neither number describes what it uses.** Both are answers to *how many sweeps share a node*, because that — not per-job usage — is what has broken things here every time. Measuring usage is the trap. The coordinator sits blocked on the model over HTTP for its whole life, so it draws ~1 core and 1–19 G of *anonymous* memory (the rest of a large `memory.current` is reclaimable page cache from container I/O — check `anon` in `memory.stat` before believing the total). The 30 containers it fans out into are the actual job, and sizing to the coordinator has been fatal three times: 8 G killed six sweeps at parallel=30, 64 G killed a seventh, and 4 cores let Slurm stack **eight sweeps on one node** — 240 concurrent units, 63 simultaneous `unsquashfs` extractions and hundreds of `srun` steps in 32 reserved cores. That node ran at a load of 375 against 92 cores with 69% of its time in the kernel, and goose fell to 10 graded units an hour. The instructive part is that per-sweep sizing was not the variable. The healthy node and the thrashing one both ran **7.5 units per reserved core**; what differed was the absolute count, 90 against 240, and the response was nothing like linear (load 34 against 375). That is congestion collapse on page cache and image extraction, and it puts a node's usable ceiling somewhere between 90 and 240 concurrent units. Raising cores to 24 — three sweeps to a node, the density that ran well — is the obvious repair and does not schedule. A serve holds **80 of 92 allocatable cores**, and nearly every usable node here is a serve node: of 47 in `hopper-prod`, 30 are packed by other users and the remaining 17 free exactly 12 cores apiece. All eight sweeps sat `PENDING`. So 4 cores is not an error to fix; it is the only size that fits the gap that exists, and on a serve node that gap already caps tenancy at three. The hole is the occasional node with no serve on it: 92 free cores and 2 T, where 4 cores and 192 G let ten sweeps land and eight did. Cores cannot close it, so **memory is the tenancy token** — which suits it, since those nodes free 848–1648 G that nothing competes for. At 512 G an empty node admits four sweeps (120 units, inside the survivable range) instead of ten. It reads as absurd padding and is not: it is a limit on neighbours, bought with the one resource going spare. Sending these to the GPU-less `hopper-cpu` partition is the obvious next step and a trap: its single node advertises `ActiveFeatures=spot`, so AWS reclaims it on two minutes' notice. Both partitions are `PreemptMode=REQUEUE`, so the job survives and resumes from cache — but every unit in flight is lost, and a unit can be six hours into a six-hour cap. Serve memory is set the opposite way, from measured peaks — 400 G for Gemma (peak ~130 GiB) and 1200 G for GLM (peak ~799 GiB per node, which is what loading a 744 GB checkpoint costs). Holding all 8 GPUs already makes that node exclusive for GPU work, so the reservation costs nobody a GPU job. `tests/test_slurm_sizing.py` pins both ends, so shrinking one cannot quietly squeeze the other. **Reasoning is pinned server-side, for every model.** Both serve scripts pass `--default-chat-template-kwargs '{"enable_thinking": true}'`, so reasoning is a property of the serve rather than of whichever agent is talking to it — one setting for all eight harnesses and both models. Override deliberately with `SERVE_ENABLE_THINKING=false`. This is explicit even where it is redundant. GLM's template already defaults thinking on and Gemma's defaults it off, and because neither was stated, the first test baseline compared a reasoning model against a non-reasoning one for 1231 units without anything failing. Stating it turns that class of disagreement into a diff. Measured against both serves (`scripts/probe_reasoning.py `), one knob works and one does not: | variant | GLM reasoning chars | Gemma reasoning chars | |------------------|---------------------|-----------------------| | default | 3498 | 4198 | | `enable_thinking: false` | 0 | 0 | | `reasoning_effort: low` | 3324 | 3713 | | `reasoning_effort: medium` | 2416 | 3890 | | `reasoning_effort: high` | 1828 | 3662 | `enable_thinking` is honoured by both and is decisive. `reasoning_effort` shows no consistent effect on either (Gemma flat; GLM's spread is unordered against effort and sits inside run-to-run variance at n=1), so neither template consumes it. No harness sends either — verified against real run artifacts, and `test_no_harness_sets_its_own_reasoning_level` keeps it that way, because vLLM lets a request-level kwarg beat the server default. Grow a pool that is already serving with `SERVE_KIND=glm scripts/add_replicas.sh 4`. The balancer rescans Slurm for matching jobs, so replicas join on their own once healthy; restarting it instead moves it to another node, and every in-flight sweep has to follow. The balancer also does this on its own, so free nodes get used at 3am instead of waiting for someone to check `squeue`. It is the mirror of the idle scale-*down* it already had: while the pool is saturated it borrows idle nodes at QoS `low`, and when the work stops it hands replicas back. Defaults are on; tune via `serve_pool.sh`: ```bash SERVE_SCALE_UP_RESERVE=4 SERVE_SCALE_UP_MAX=24 scripts/multi_serve.sh 8 SERVE_SCALE_UP_INTERVAL=0 scripts/multi_serve.sh 8 # opt out entirely ``` Two rules keep it from misbehaving. It only grows while the pool is **saturated** — no free request slot *and* work actually in flight, because an empty pool also has no free slots and reading that as saturation is how this loop once submitted a replica four minutes before the balancer shut itself down. And it only submits at a preemptible QoS, which is safe because those replicas join by discovery rather than filling a configured slot, so a lost one is not resubmitted as if it had crashed. Preemption is the normal end of a borrowed replica, and Slurm handles it by **requeueing the same job id onto a different node**, where it rewrites its `endpoint.env`. Nothing used to notice: discovery skips job ids it already knows, and a requeue is not the job ending. The pool kept routing to the vacated address, drained the backend on the resulting stream error, and — since nothing ever cleared that flag — left it there permanently. One night this ended with 7 of 11 replicas holding a full node each while serving nothing, and with the pool reading as saturated (drained backends report no capacity) so the autoscaler kept borrowing more nodes to replace them. `recover_drained_loop` closes it: every `--recover-interval` (60s) it re-reads `endpoint.env` for each fault-drained backend and re-admits any that answers `/health`, so a replica that moved is followed instead of written off. Backends retired on purpose are excluded, which is what keeps scale-down able to shrink the pool. Growing helps even when the sweeps cannot offer more concurrency than the pool already has. A backend admits `SERVE_MAX_ACTIVE_PER_BACKEND` requests at once and per-request decode collapses as that fills (36 tok/s at 8 concurrent vs 13.7 at 32), so spreading the same in-flight agents over more backends makes each one faster. Sweeps re-resolve the pool endpoint and rerun if the balancer moves node, which this cluster causes by preempting jobs and auto-resubmitting them elsewhere. Without that a moved balancer left every sweep hammering a dead address for the rest of its walltime; rerunning is cheap because completed units are cached. Scale a pool down through the balancer, never with `scancel`: a slot manager reads a disappearing job as a crash and resubmits it, so cancelling by hand starts a fight you lose three times per replica. `POST /retire/{job_id}` stops maintaining a replica and cancels it whether or not it ever joined; `POST /cancel_when_drained/{job_id}` does the same but waits for in-flight requests to finish first. The pool cleans itself up. Sweeps finish at unpredictable times, and a pool left running holds whole 8xH100 nodes for nothing, so after `--idle-timeout` (default 1h) with nothing in flight the balancer retires one replica per `--idle-scale-interval` (default 5min) and stops itself once the last one is gone. Retirement is gradual so a sweep that starts again costs one replica rather than the pool, and retired jobs are remembered so the slot managers do not resubmit them. Pass `--idle-timeout 0` to keep a pool up indefinitely. Each replica is one full 8xH100 node running **TP=2 x DP=4**. TP only has to be large enough to leave a healthy KV budget; beyond that, GPUs buy more from data parallelism (independent schedulers) than from wider all-reduces, which is what a 4B-active MoE wants: | layout | KV per GPU | tokens per node | engines | full-context seqs/node | |-------------|-----------:|----------------:|--------:|-----------------------:| | TP=1 x DP=8 | 15.5 GiB| 4.6 M | 8 | ~17.6 | | **TP=2 x DP=4** | **40.4 GiB** | **16.5 M** | **4** | **62.9** | | TP=4 x DP=2 | 53.0 GiB| 15.7 M | 2 | ~59.9 | | TP=8 x DP=1 | 59.3 GiB| 17.5 M | 1 | ~66.8 | vLLM load-balances internally across DP ranks, so a node still exposes a single endpoint and the balancer only fans out across nodes. Measured on a 3-node pool: 384/384 concurrent completions succeeded at ~8.0k output tok/s, with throughput flat from 96 concurrency upward — so extra sweep parallelism beyond that queues rather than going faster. Tear the pool down with the `scancel` line `multi_serve.sh` prints; replicas do not self-terminate when idle. **DeepSWE / Terminal-Bench 2.0 / NL2Repo** are reimplemented on the `containers.py` backend (their upstream runners drive Docker/Modal/Runloop, which don't work on the cluster), each reusing the task's own verifier and validated on the cluster: - **DeepSWE** (113 Harbor SWE tasks): patch a git repo, grade with the task's `test.sh`/`grader.py` -> `reward.json`. Oracle 2/2; a real rollout solved a task 20/20. - **Terminal-Bench 2.1** (89 tasks, `zai-org/terminal-bench-2-verified` — the verified TB2 GLM-5 was evaluated on): single-container terminal tasks (the verifier runs against the agent's own container end-state), graded by the task's pytest -> `reward.txt`. Full oracle: **76/89 (85%) reproduce**; the 13 unreproducible tasks (`splits/terminal_bench_oracle.json`) should be oracle-gated out of the model denominator (report pass@1 over the reproducible 76 + coverage). Final-reporting only, not in the optimization loop. - **NL2Repo** (104 tasks): greenfield generation; the cleaned agent workspace is overlaid onto the per-task GHCR image's canonical `/workspace` and graded by the repo's own pytest (fractional). Validated end-to-end with a real rollout. If no container runtime is available, adapters fail cleanly (`RunnerUnavailable` -> `infra_unresolved`, excluded from the denominator) rather than corrupting scores.