At step 10 the E4B jury clearly beats the no-LLM control. At step 20 only 38 of the judged test runs were resolved, so every
interval is wide (the pre-registered k = 20 test is inconclusive). With *fractional* prefixes ("the first 66% of a run")
trace features look much stronger (AUROC 0.72) — but only because "66% of the run" leaks how long the run will be. A live
monitor never knows that, so **all headline results use absolute step prefixes**.
### Choosing the abort threshold
### Calibrated out of the box, and interpretable
80 random test prompts were re-rated blind (no outcome, no Gemma scores) by four independent Claude Sonnet annotators using
the identical rubric. E4B agrees substantially on **localization (κw 0.65)** and **loop (0.63)**; the mean score correlates
at ρ = 0.73. Small judges are lenient, most on "coherence", which is where agreement is weakest.
## Deployable policy: development-fitted calibration and cost-aware abort
The project now includes a **JSON-exportable, development-fitted policy** in
[`src/deployment_policy.py`](src/deployment_policy.py). It selects the abort
threshold from development runs only, charges **one agent step per judgement**,
and continues when no judgement exists. Its holistic-score isotonic calibrator
can be used without scikit-learn at inference; it is separate from jury scores.
Three reproducible exploratory interventions on the archived real agent runs:
| Intervention | Held-out result | Decision |
|---|---|---|
| Dev-fitted holistic calibration | Brier **0.22564 → 0.17681**; AUROC 0.65515 → 0.65479 | Export calibration; no discrimination-gain claim |
| Dev-selected abort policy, including judgement cost | **+22.35%** resolved tasks per step vs never aborting; repository-bootstrap 95% CI **[+8.91%, +38.02%]** | Export threshold 0.75; missing judgements continue |
| Fixed 50/50 jury + trace-feature hybrid | AUROC 0.63348 vs jury 0.67678 and features 0.57512 | Keep both comparisons; retain jury alone |
These are **postprocessing and historical replay**, not new live agent runs or
an official competition score. Calibration uses 429 judged test runs from 351
repositories; policy replay includes all 600 test runs. Development and test
repositories do not overlap. The positive replay result does not establish
real-time savings under a different agent or model.
```sh
python -m src.deployment_policy
python scripts/reproduce_postprocessing.py
```
Use `should_abort(jury_probability, policy)` at step 10, or
`calibrated_holistic(raw_holistic_probability, policy)` for calibration.
[Exported policy](results/deployment_policy.json) ·
[Results, paired intervals and input hashes](results/postprocessing_experiments.json)
## How it works
```
GitHub issue + first k agent steps ──► Gemma 4 jury (vLLM, T=0, JSON) ──► 6 scores (1-5)
│
abort if 1 − P(success) > τ ◄── P(success) ◄── logistic layer (fit on dev)
```
* **Data:** [`nebius/SWE-agent-trajectories`](https://huggingface.co/datasets/nebius/SWE-agent-trajectories) (CC-BY-4.0),
one run per repository and outcome → 1,765 runs from 1,276 repositories; 60/40 repository-disjoint split; 600 test runs.
* **Prefixes:** issue + first k ∈ {10, 20} actions and observations; only runs longer than k are judged, so the ending is
never visible. Patches, test logs and exit status are stripped (`src/04_checks.py` asserts it).
* **Judges:** `google/gemma-4-E4B-it`, `google/gemma-4-E2B-it` and the control `Qwen/Qwen3-4B-Instruct-2507`, all bf16 on
vLLM 0.32.0, temperature 0, fixed seed, byte-identical prompts ([`prompts/`](prompts)).
* **Control without an LLM:** 43 prefix-only trace features with logistic regression / gradient boosting.
* **Policy:** abort at step k if predicted failure > τ; aborted runs cost k steps; τ chosen on development data.
* **Statistics:** AUROC, ECE, Brier; paired repository-cluster bootstrap (B = 2,000); Holm over 8 confirmatory contrasts.
## Reproduce
```bash
git clone https://github.com/Agnuxo1/small-juries-early-verdicts && cd small-juries-early-verdicts
python -m venv .venv && .venv/bin/pip install -r requirements-analysis.txt
python -m venv .venv-vllm && .venv-vllm/bin/pip install -r requirements-inference.txt
export SJEV_ROOT=$PWD/work # data/, models/, out/ live here
bash scripts/reproduce.sh # sample → features → splits → judges → analysis → figures
```
Every stage is a plain script in [`src/`](src); [`scripts/reproduce.sh`](scripts/reproduce.sh) lists them in order with the
exact arguments we used. A full rerun of the judges takes ≈ 2–3 GPU-hours per model on one RTX 3090. To regenerate only the
statistics and figures from the released verdicts, see [`results/`](results) and run `python scripts/make_figures.py`.
| Stage | Script | Compute |
|---|---|---|
| sample, strip leakage | `01_sample.py`, `04_checks.py` | CPU |
| trace features + baseline | `02_features.py`, `03_baseline.py` | CPU, minutes |
| repository-disjoint splits | `05_splits.py` | CPU |
| judges (holistic + jury) | `06_judge.py` | 1 GPU, vLLM |
| pre-registered analysis H1–H4 | `07_analysis.py` | CPU |
| p-values, Holm, H4 contrasts, control model | `08_supplementary.py` | CPU |
| blind validation set + agreement | `09_export_validation.py`, `10_validation.py` | CPU |
| figures and GIFs | `scripts/make_figures.py` | CPU |
## Repository layout
```
src/ experiment code (Apache-2.0)
prompts/ the exact judge prompts (version v1, frozen after the dev pilot)
results/ metrics, contrasts, per-run predictions and verdict scores, cost logs, validation
experiments/ LOG.md and experiments.csv: every run, including discarded and negative ones
figures/ paper figures (PNG) and README animations (GIF)
paper/ Kaggle Writeup text and PDF
validation/ blind reference annotations (ratings only)
```
## Released resource
`results/test_predictions.csv` and `results/verdicts/` contain, for every judged run, the run id, outcome, the six jury scores,
the holistic probability and every calibrated score, for E2B, E4B and the Qwen control. Trajectory text is **not**
redistributed: fetch it from the source dataset by `instance_id` (CC-BY-4.0; model outputs in that dataset are subject to
the Llama 3.1 license).
## Credits
* Francisco Angulo de Lafuente ([@Agnuxo1](https://github.com/Agnuxo1)) — study design, rubric (inspired by the multi-judge
tribunals of [BenchClaw](https://github.com/Agnuxo1/benchclaw); conceptual inspiration only, no code reused).
* Engineering, analysis and writing assisted by Claude (Anthropic); routine design decisions logged with the JEV-Orchestrator
decision engine.
* Data: Nebius `SWE-agent-trajectories` (CC-BY-4.0). Models: Google Gemma 4 (Apache-2.0), Qwen3 (Apache-2.0).
Serving: vLLM. Statistics: scikit-learn, SciPy.
## Limitations
These are the normal boundaries of a two-day study, listed so the results can be used correctly:
trajectories come from one scaffold (SWE-agent) and Llama-3.1 drivers, so transfer to the Gemma 4 competition harness is
still to be measured; the H4 budget simulation assumes freed steps are reinvested at average efficiency; the k = 20 analysis
is under-powered (38 resolved runs); the rubric was validated against an LLM reference rather than human experts; and the
pre-registration was kept in time-stamped project notes rather than a public registry.
## Citation
```bibtex
@misc{angulo2026smalljuries,
title = {Small Juries, Early Verdicts: Calibrated Early Failure Prediction for Budgeted Offline SWE Agents},
author = {Angulo de Lafuente, Francisco},
year = {2026},
note = {Gemma 4 Developer Agent Paper Track, Kaggle},
url = {https://github.com/Agnuxo1/small-juries-early-verdicts}
}
```
Released under the [Apache License 2.0](LICENSE).