# GUI Grounding Evaluation Harness

[![tests](https://github.com/PatrickStarL/gui-grounding-eval/actions/workflows/tests.yml/badge.svg)](https://github.com/PatrickStarL/gui-grounding-eval/actions)

**36 runs · 290 tests · $13.50 total API spend.**

Published GUI-grounding numbers rarely say what exact prompt, output format,
or coordinate convention produced them, so a reported score mixes model
quality with evaluation choices - this project measures those confounds
directly instead of treating them as noise. The task is point grounding on
[ScreenSpot-Pro](https://huggingface.co/datasets/likaixin/ScreenSpot-Pro)
(1,581 samples, targets averaging **0.07% of screen area** - small enough
that implementation details move the score by several points instead of
disappearing into noise).

Four controlled ablations, each isolating one variable while holding prompt
wording, coordinate space, and image encoding fixed:

- Output format (native vs. raw coordinates), replicated in opposite
  directions on two model families
- That same format effect re-measured across four model sizes (3B/8B/22B/32B
  active parameters), with a numeric prediction made *before* the fourth run
  and checked against it
- Inference-time scaffolds - single call, 2-call crop-and-relocate
  (ReGround), and an 8.5-call uncertainty-gated zoom reproduced from
  UI-Zoomer - compared on cost *and* accuracy
- Active vs. total parameter count as predictors of grounding accuracy,
  across one model family at four sizes

What came out of it:

- Output format alone shifts accuracy by 3-7pp in both directions (McNemar
  p<1e-4), a variable no model card reports
- That format effect looked to scale linearly with model capacity across
  three sizes; a pre-registered fourth measurement missed the predicted
  value by 2.3pp, falsifying that story - the effect instead splits along
  MoE-vs-dense architecture, reported here as a hypothesis, not a conclusion
- Unconditional 2-call ReGround (73.3%) beats 8.5-call uncertainty-gated
  zoom (59.5%) at a quarter the cost - the gating logic works, but sits on a
  voting scheme that discards correct minority answers
- Active parameters predict accuracy while total parameters barely do - a
  30B-total/3B-active model scores 15.75pp below an 8B dense model
  (p=6.8e-31)

None of this is a leaderboard entry - it's a methodology and its evidence:
two prompt-level variables worth 3-7pp that the field currently doesn't
report, a demonstration that a clean 3-point scaling trend can fail a
held-out prediction reported straight rather than reframed after the fact,
and an evaluation harness (290 tests, config-sidecar reproducibility,
three-baseline external calibration) built so every number above can be
independently re-derived.

## Contents

- [Experimental sub-tasks](#experimental-sub-tasks) - table of every ablation, its independent variable, and what was held fixed
- [Results](#results) - headline numbers, plus external calibration against three published baselines
- [Four findings](#four-findings) - the evidence and statistics behind each claim above
- [Cost](#cost) - $/correct-click, and where the API spend went
- [Quick start](#quick-start) - install, run the test suite, run one sample against a real model
- [Engineering decisions](#engineering-decisions) - why the harness is built the way it is, and what broke before it was
- [Layout](#layout) - what's in each folder
- [Known limitations](#known-limitations) - what this project does not resolve, stated plainly
- [References](#references) - related work and where each zoom method comes from

## Experimental sub-tasks

Each ablation below is a self-contained experiment with its own independent
variable, controls, and sample size. Full evidence and statistics for each
are in [Four findings](#four-findings) (linked per row).

| Sub-task | Models | Independent variable | Held fixed | n | Finding |
|---|---|---|---|---|---|
| Output format ablation | Qwen3-VL-32B, UI-TARS-1.5-7B | Output format (native `<tool_call>`/`<point>` vs. raw `[x,y]`) | Wording, identical line-by-line | 1,581 each | [#1](#1-output-format-is-worth-3-7pp-and-no-model-card-mentions-it) |
| Bare vs. ReGround, full set | Qwen3-VL-32B, UI-TARS-1.5-7B | Scaffold (single call vs. 2-call ReGround) | Model, prompt | 1,581 (Qwen3-VL), 200 (UI-TARS) | [#2](#2-regrounds-gain-does-not-shrink-on-a-stronger-base-model) |
| Scaffold comparison incl. gated zoom | Qwen3-VL-32B | Calling strategy (bare / sample-and-vote / gated zoom / ReGround) | Model, prompt, sample subset | 200 (all 4 methods, same subset) | [#3](#3-uncertainty-gating-is-not-worth-its-cost---but-for-a-subtler-reason-than-expected) |
| Active vs. total parameters | Qwen3-VL 3B/8B/22B/32B active | Model size (active vs. total parameters) | Prompt, coordinate space, architecture family | 1,581 each | [#4](#4-active-parameters-drive-grounding-total-parameters-barely-do) |
| Format effect across model scale | Qwen3-VL 3B/8B/22B/32B active | Model size × output format (interaction) | Prompt pair, coordinate space, image encoding | 1,581 each | [#4](#4-active-parameters-drive-grounding-total-parameters-barely-do) |
| External calibration | UI-TARS-1.5-7B, ShowUI-2B, GPT-4o | Model (three coordinate conventions) | Each model's own official prompt/wrapper | 200-1,581 | [Results](#the-harness-is-calibrated-against-three-published-baselines) |


---

## Results

![The Results table, sorted by accuracy, colored by model, with calls per sample labeled on each bar](docs/figures/accuracy-vs-calls.png)

Two calls beat eight and a half. That is the practical headline: unconditional
crop-and-reground reaches 73.3%, while UI-Zoomer's uncertainty-gated variant -
which spends 8 sampled calls before deciding whether to zoom - reaches 59.5%.
Sampling eight times and voting, without zooming, is *worse than a single call*
(the "sample+vote" bar - not one of the seven Results-table rows, added here
from finding 3 since this is the claim it backs). The non-blue bars (UI-TARS,
ShowUI-2B, GPT-4o) are single-call baselines on different models, shown for
scale - not part of the calls-vs-accuracy comparison, which is a same-model,
same-prompt argument about Qwen3-VL-32B.

| Model | Configuration | n | Accuracy | $/correct click |
|---|---|---|---|---|
| **Qwen3-VL-32B** | toolcall prompt + **ReGround** (2 calls) | 1,581 | **73.31%** | $0.00071 |
| Qwen3-VL-32B | toolcall prompt, single call | 1,581 | 56.42% | $0.00051 |
| Qwen3-VL-32B | official harness prompt | 1,581 | 56.29% | $0.00055 |
| Qwen3-VL-32B | uncertainty-gated zoom (8.5 calls) | 200 | 59.50% | $0.00394 |
| UI-TARS-1.5-7B | official GROUNDING template | 1,581 | 47.12% | $0.00155 |
| ShowUI-2B (local) | official wrapper config | 394 | 8.38% | $0 (9.4h GPU) |
| GPT-4o | official `gpt4x.py` prompt | 400 | 1.00% | $0.308 |

### The harness is calibrated against three published baselines

This is the evidence that the numbers above mean anything. Three models, **three
different coordinate conventions**, spanning a 47× capability range - all land
near their published values:

| Model | Coordinate convention | Ours | Published | Δ |
|---|---|---|---|---|
| UI-TARS-1.5-7B | absolute pixels, **post-`smart_resize`** | 47.12% | 49.6 (model card) | -2.5 |
| ShowUI-2B | normalized [0,1] | 8.38% | 7.7 (paper) | +0.7 |
| GPT-4o | normalized [0,1] | 1.00% | 0.8 (paper) | +0.2 |

One model matching could be luck. Three, under three different conventions, is
not. **Two of those conventions are undocumented** - they were found by probing,
and getting one wrong costs you the entire run silently (see
[Silent failures](#silent-failures-are-the-real-risk)).

---

## Four findings

Each below states the takeaway first, then the evidence and what was ruled out.

### 1. Output format is worth 3-7pp, and no model card mentions it

**Takeaway: match the model's fine-tuned output shape, not "whichever format
seems simpler" - the wrong choice costs several points either direction.**

Two prompts, **identical word for word**, differing only in the shape of answer
requested. A test enforces the wording match line by line, so the delta is
attributable to format alone.

| Model | Direction | That model's native format | Effect | McNemar *p* |
|---|---|---|---|---|
| Qwen3-VL-32B | bare -> native | `<tool_call>{"name": "computer_use", …}` | **+3.35pp** | 6.2e-05 |
| UI-TARS-1.5-7B | native -> bare | `Action: click(point='<point>x y</point>')` | **-7.34pp** | **3.6e-14** |

This is a **reverse replication**, not the same measurement twice: adding the
native format helps one model, removing it hurts the other. And the two "native
formats" are entirely different strings - so what matters is not that some format
is intrinsically better, but **matching the shape that model was fine-tuned on.**

On Qwen3-VL this single change accounted for the *entire* advantage of the
official harness prompt (toolcall-only vs full harness: +0.13pp, *p*=0.935). Its
system prompt, tool schema, and "guess a most likely one" instruction contribute
approximately nothing.

Ruled out, on the UI-TARS pair:
- **Not format non-compliance** - 1,581/1,581 obeyed the bare format; **zero**
  fell back to the native one.
- **Not parsing** - 2/1,581 parse failures. Cannot produce 7pp.
- **Not chain-of-thought** - output tokens 10.9 (bare) vs 19.0 (native). The bare
  arm is *shorter and worse*; the extra tokens are boilerplate, not reasoning.

**Practical consequence: any reported grounding number needs its output format
stated, or it is not reproducible.** Most model cards omit even the prompt.

### 2. ReGround's gain does *not* shrink on a stronger base model

**Takeaway: a cheap 2-call crop-and-relocate outperforms an 8.5-call
uncertainty-gated alternative, and its gain does not shrink as the base model
improves - contradicting a published claim.**

ReGround = locate once, crop around that guess, locate again. Two calls.

| Model | bare | +ReGround | Δ |
|---|---|---|---|
| UI-TARS-1.5-7B | 45.0% (n=200) | 57.0% | +12.0pp |
| **Qwen3-VL-32B** | **56.42%** (n=1,581) | **73.31%** | **+16.89pp** (*p*=9.7e-54) |

Qwen3-VL has both the higher baseline *and* the larger gain, contradicting
UI-Zoomer's reported "gain shrinks as the base model gets stronger." Their
comparison was cross-architecture and confounded with training data; this one is
at least same-family. Caveat stated plainly: **not fully controlled** - the two
arms used different prompts, so some of the difference could be prompt-driven.

Where the gain lands: icon +19.4pp and tiny targets +25.6pp, versus text +15.4pp
and large +10.2pp. Zoom helps most where the model was weakest - but does not
close the gap. Icons remain at 49.5% against text's 88.0%.

![ReGround's accuracy gain by target size, largest on the smallest targets](docs/figures/reground-gain-by-size.png)

### 3. Uncertainty gating is not worth its cost - but for a subtler reason than expected

**Takeaway: the gating logic itself works (it routes hard samples correctly);
it's bolted onto a voting pipeline that loses to doing nothing, so the whole
thing loses to skipping it entirely.**

Implemented from UI-Zoomer, whose constants are absent from the paper and were
read out of the reference implementation.

Same 200 samples, all with **real box extent**:

| Method | Calls | Accuracy |
|---|---|---|
| bare, single call | 1× | 47.50% |
| sample-and-vote (8 draws, no zoom) | 8× | **46.50%** |
| gated zoom, best τ | 8.53× | 59.50% |
| **ReGround** | **2×** | **70.50%** |

**Sample-and-vote at 8× is 1pp *worse* than one call.** Gating rescues it to
59.5%, and its routing logic verifiably works - the tau sweep is monotone, and the
premise holds (gated-out samples have 5× more recoverable headroom). But
ReGround beats it by 11pp at a quarter the cost.

So the conclusion is not "gating fails." It is that **gating is bolted onto a
pipeline that loses to doing nothing.** Two independent defects compound: the
provider returns logprobs for only 1 of 35 output tokens (making the confidence
term a constant, so tie-breaking is dead), and majority-IoU voting actively
discards minority-correct draws.

A related finding worth its own line: **upstream's τ=1.5 degrades to a no-op on
both models, in opposite directions** - 7.5% pass rate on UI-TARS (nearly
everything zooms), 91% on Qwen3-VL (nearly nothing does). A threshold tuned on
one model's score distribution does not transfer.

### 4. Active parameters drive grounding; total parameters barely do

**Takeaway: judge a MoE model's grounding accuracy by its active parameters,
not its total size - and be skeptical of any 3-point "scaling law," including
one from this project (see below).**

Four sizes, one architecture, identical prompt and coordinate space, full n=1,581:

| Model | Active | Total | Accuracy | $/correct |
|---|---|---|---|---|
| **30B-A3B** | **3B** | 30B | **35.23%** | $0.00283 |
| 8B | 8B | 8B | 50.98% | $0.00073 |
| 32B | 32B | 32B | 56.42% | **$0.00051** |
| **235B-A22B** | **22B** | 235B | **60.53%** | $0.00138 |

**30B-A3B has nearly 4x the total parameters of 8B and scores 15.75pp worse**
(*p*=6.8e-31). Ordered by total parameters the curve is badly non-monotone;
ordered by active parameters it is monotone apart from 235B-A22B's 22B active
edging past 32B dense. All three adjacent comparisons are significant.

![The same four models ordered by total parameters versus active parameters](docs/figures/active-vs-total-params.png)

Verified this was not a technical fault before believing it: 1,581/1,581
format-compliant, zero parse failures, output tokens matching 32B, coordinates
mapping correctly. The failure mode is pointing at the wrong element.

**32B dense is the cost-efficiency sweet spot** - 235B buys +4.11pp for 2.9x the
money, while the output-format change costs nothing and is worth +3.35pp, about
80% of what a 7x-larger model buys.

**A fourth point breaks the clean story - and that's the more interesting
result.** Finding 1 measured a large format effect on 32B (toolcall vs
rawpair, +3.35pp). Rerunning that same ablation at 8B, 30B-A3B, and
235B-A22B, all n=1,581, strictly paired (same 1,581 `sample_id`s scored under
both prompts):

| Model | Active | Architecture | toolcall | rawpair | Δ (format effect) | McNemar *p* |
|---|---|---|---|---|---|---|
| 30B-A3B | 3B | MoE | 35.23% | 34.60% | +0.63pp | 0.67 (n.s.) |
| 8B | 8B | Dense | 50.98% | 48.77% | +2.21pp | 0.011 |
| **235B-A22B** | **22B** | **MoE** | **60.53%** | **60.09%** | **+0.44pp** | **0.68 (n.s.)** |
| 32B | 32B | Dense | 56.42% | 53.07% | +3.35pp | 6.2e-05 |

![Format effect versus active parameters, showing the fitted trend, the prediction it made, and the measured miss](docs/figures/format-effect-vs-architecture.png)

I originally fit a line through the first three points (3B/8B/32B) and
predicted the format effect at 22B active would land around +2.7pp - the
whole point of running 235B-A22B was to test that prediction. It didn't hold:
the effect at 22B active is +0.44pp, statistically indistinguishable from
zero, landing next to 3B active rather than between 8B and 32B. **Ordered by
active parameters alone, the format effect is not monotone** (+0.63 -> +2.21
-> +0.44 -> +3.35).

What *is* monotone: grouped by architecture, both MoE points (30B-A3B, 3B
active; 235B-A22B, 22B active) show a non-significant format effect, and both
dense points (8B, 32B) show a significant one, an order of magnitude larger.
That regrouping fits the same four numbers with zero free parameters and no
outliers - but it's two points per architecture, so "MoE dampens format
sensitivity" is a pattern consistent with the data, not a finding established
by it. I'm reporting it as the leading hypothesis, not a conclusion.

I looked for a more mundane explanation before trusting either story. 235B's
`toolcall` accuracy (60.53%) is the highest of the four models, so a ceiling
effect is a candidate: if the model is already scoring near its own ceiling,
there is less room for a format effect to show up regardless of the
mechanism. I don't have a way to rule that out with the current data - it
would need a task easier than ScreenSpot-Pro's icon-heavy split to check
whether 235B's format effect reappears somewhere below ceiling.

I also checked whether `toolcall` is even a fair thing to call "native" for
these models. The label was reverse-engineered from what 32B *actually
replied* under the official ScreenSpot-Pro harness system prompt - which
itself explicitly instructs the model, via a `<tools>` schema, to answer
inside `<tool_call></tool_call>` tags. That is 32B complying with an explicit
instruction, not an unprompted preference; the "native" label was always an
inference, not a measurement. I tried to test it directly: 20 samples on 32B
with a prompt that names the task and gives no format instruction at all
(`scripts/probe_native_format_preference.py`). Result: 20/20 replies were
conversational prose ("To modify the highlights of the photo... the user
should..."), 0/20 resembled `tool_call`, JSON, or any structured shape. That
kills the probe, not the hypothesis - an instruction-tuned model's default
completion under an open-ended question is prose regardless of which
structured format it was fine-tuned to use in a tool-calling context, so this
design can't distinguish "prefers tool_call" from "prefers nothing
in particular." I did not run the 235B side of this probe, since a null
design would just spend money to reproduce the same null. The `tool_call`
schema in the harness prompt is Qwen's documented Agent/function-calling
format, publicly specified independent of anything measured in this repo -
that's the actual basis for calling it "native," not the probe.

**Practical consequence, revised**: this is not a clean capacity × format
scaling law. What holds up: (1) the format effect is real and large on at
least two dense models (8B, 32B), (2) it is measurably absent on both MoE
points tested, (3) a linear "bigger model, bigger effect" story fit to three
points did not survive a fourth, pre-registered test. Report format effects
per model, not as a property of a model family, and treat any curve fit
through three points as a hypothesis to break, not a law.

---

## Cost

Literature gap: of the three zoom papers this work implements
(ZoomClick, UI-Zoomer, GUI-Lens), **none reports token or dollar cost.** One
reports latency.

| Method | Calls | Accuracy | $/1,581 samples | $/correct click |
|---|---|---|---|---|
| Qwen3-VL bare | 1× | 56.42% | $0.45 | $0.00051 |
| **Qwen3-VL +ReGround** | **2×** | **73.31%** | **$0.82** | **$0.00071** |
| Qwen3-VL gated zoom | 8.53× | 59.50% | ~$3.71 | $0.00394 |
| UI-TARS bare | 1× | 47.12% | $1.15 | $0.00155 |
| GPT-4o | 1× | 1.00% | ~$4.87 | $0.308 |

**ReGround buys +16.89pp for a 1.8× cost increase. Gating buys +3.1pp for 8×.**
Specialised grounding models are one to two orders of magnitude cheaper per
correct click than a general VLM.

One counterintuitive result: for Qwen3-VL, *not* downscaling images is
**cheaper**. Its `factor=32` patches are larger than Qwen2.5-VL's 28, which
offsets the pixel increase - $0.244 vs $0.306 over 400 samples.

---

## Quick start

```bash
# Python 3.12 (3.10 is too old for some dependencies)
uv venv --python 3.12
uv pip install -e ".[dev]"

# 290 tests, no network and no API key required - they drive the pipeline
# with a scripted model, so they cost nothing and run in ~15s
pytest
```

Evaluating against the real benchmark additionally needs the ScreenSpot-Pro
images (3.38 GB, manual download). Local ShowUI-2B inference needs
`pip install -e ".[local]"` plus a torch build matching your CUDA driver - both
are optional and neither is required for the test suite or for any API-based
model.

To run against a real model, add an [OpenRouter](https://openrouter.ai) key:

```bash
echo "OPENROUTER_API_KEY=sk-or-..." > .env
```

```python
from ssp_eval import (
    load_samples, stratified_subset, QwenVLModel, CoordSpace,
    ReGroundScaffold, ScaleStrategy, evaluate, summarize,
)
from ssp_eval.prompts import QWEN_NATIVE_POINT_1000_TOOLCALL

samples = stratified_subset(load_samples(), 50, seed=0)

model = QwenVLModel(
    "qwen/qwen3-vl-32b-instruct",
    CoordSpace.FIXED_GRID,          # Qwen3-VL always answers on a 1000x1000 grid
    grid=(1000, 1000),
    prompt_template=QWEN_NATIVE_POINT_1000_TOOLCALL,
    factor=32,                      # 32 for Qwen3-VL, 28 for Qwen2.5-VL
)

results = evaluate(
    samples, model,
    ReGroundScaffold(window=(1024, 1024), scale=ScaleStrategy.TARGET_PIXELS),
    out_dir="runs", budget_usd=1.00,   # hard spend cap
)
print(summarize(results).accuracy)
```

Reproducing a published run: every `runs/*/*.config.json` sidecar is committed
and records the model, full prompt text, coordinate space, image encoding and
resolution policy for that run.

---

## Engineering decisions

### Silent failures are the real risk

Coordinate spaces differ per model, and **getting one wrong raises no error - it
just clicks the wrong place.**

| Model | Coordinate space | How it was determined |
|---|---|---|
| UI-TARS-1.5-7B | absolute pixels, post-`smart_resize` | probing; undocumented |
| Qwen3-VL-32B | 1000×1000 virtual grid | probing; the README example (1016 > 1000) misleads |
| ShowUI-2B / GPT-4o | normalized [0,1] | official wrappers |

So coordinate space is a required attribute of every model adapter, never
inferred at parse time. `tests/test_geometry.py` contains a test that
demonstrates the failure mode: treating post-resize coordinates as original-image
coordinates shifts the click by **960 pixels**, and the resulting point looks
entirely plausible.

This project paid that price once, before the test existed: assuming Qwen3-VL's
`bbox_2d` was absolute pixels produced **0/8** - see
`runs/_stale_nativebox_n8_absolute_image/`.

### Config sidecars are the source of truth, not the scripts

Reading a run script to determine what a run actually sent has produced a wrong
published claim on this project: `backfill_qwen3vl_failures.py` describes only 70
of 1,581 rows, so a mixed PNG/JPEG encoding went unnoticed and the difference
between two arms was reported as "prompt only." Another script turned out to
evaluate a different model entirely.

Hence every run writes a `*.config.json`, and those are committed even though the
per-sample outputs are not.

### Parse failures score as wrong, never dropped

Dropping them would flatter models with unreliable output formats. The
fallback parser requires the reply to contain *exactly* two numbers - otherwise
it would happily extract `(3, 2)` from "I see 3 buttons and 2 menus." Better to
score a miss than invent a click.

### Paired comparison, and the trap inside it

For a 6pp effect:

| n | Δ | 95% CI | Verdict |
|---|---|---|---|
| 50 | +0.060 | [+0.000, +0.140] | not significant |
| 200 | +0.060 | [+0.030, +0.095] | significant |
| 1,581 | +0.060 | [+0.049, +0.072] | significant, 2.3pt wide |

Two *independent* groups of 50 (52% vs 58%) give overlapping intervals and
detect nothing. Paired designs let expensive models be evaluated on a
200-sample stratified subset.

**But an underpowered null is not a null, and this project fell for that twice:**
at n=400 two prompts looked tied (*p*=0.38) when the full set showed +3.23pp
(*p*=7.3e-04); and a 40-sample encoding control that "agreed 39/40" was treated
as ruling out an effect, when that interval spans [0.1%, 13%] - wide enough to
contain the entire 3pp gap being investigated. **A ~3pp effect needs the full
1,581.**

### Single-variable ablations are enforced by tests

`test_differs_from_point1000_only_in_output_format` asserts line by line that two
prompts are identical apart from the output format. Without it, someone tidying
the wording would quietly convert a format ablation into a multi-variable
comparison - and nothing would fail. Given that format is worth 3.35pp and the
template is worth 4.68pp, such an edit has a measurable cost.

### Other

- **Hard spend cap, default $5.** Tools in this space routinely ship without one
  ([browser-use](https://github.com/browser-use/browser-use/issues/5479) still has
  this open), and unattended batch runs are how surprise bills happen.
- **Streaming JSONL + resume by `sample_id`.** A crash does not lose work already
  paid for. Learned the hard way: misjudging a run as dead and relaunching it
  duplicated 297 samples, so run status is now determined by counting rows on
  disk, not by looking at the process.
- **Sampling separated from scoring.** Crop windows are a pure function of the
  sampled points, so draws were bought once ($0.90) and every crop-geometry
  variant was recomputed offline for **$0** - including the most informative
  diagnostic of the project.

---

## Layout

```
ssp_eval/     harness: dataset loading, coordinate transforms, model
              adapters, scaffolds, uncertainty gating, metrics
tests/        290 tests, network-free (scripted model, no API key needed)
scripts/      one script per experiment or analysis (51 scripts)
runs/         config sidecars (committed) + per-sample outputs (gitignored)
docs/         README figures, plus literature review and write-up notes
reference/    third-party wrapper copies, for byte-comparison; not executed
```

Key modules: `geometry.py` (the only path back to original-image coordinates)
and `prompts.py` (verbatim transcriptions of official templates - the file whose
contents are worth several points of accuracy).

---

## Known limitations

- **Coordinate-scaling arithmetic is not fully closed-loop.** Verifying it needs
  a model with absolute-pixel output *and* a published number from a known
  harness. UI-TARS satisfies only the first (ByteDance's harness is not public).
  Under normalized coordinates the sent size cancels algebraically, so the
  GPT-4o and ShowUI checks bypass the resize path entirely.
- **The icon gap is unresolved and is the largest effect in the project.** 42.5pp
  bare (text 72.7% vs icon 30.1%), 38.5pp after ReGround. None of the 31 runs
  targeted it. Known ineffective: more pixels (icons gain +1.3pp), prompt
  wording, a stronger model. Diagnosis points to *localisation*, not
  recognition - 82% of failed boxes do not overlap the target at all, while
  their sizes are roughly right.

  ![the icon/text accuracy gap across configurations](docs/figures/icon-text-gap.png)

  Every intervention here lifts both curves and leaves the gap between 38 and
  43pp. That is the strongest evidence that the remaining headroom is not a
  prompting or resolution problem.
- **Scaffold results are two models, not four.** GPT-4o and ShowUI were
  calibration points only; their baselines (1% and 8%) are too low for a gain to
  clear the confidence interval.
- **Gating on Qwen3-VL could only be run single-signal.** The provider returns
  logprobs for 1 of 35 output tokens, making UI-Zoomer's confidence term a
  constant. This is a provider limitation, not a result about the method.
- **`temperature=0`, single run per arm.** No seed-variance estimate.

---

## References

| Work | arXiv | Relevance |
|---|---|---|
| SeeAct | [2401.01614](https://arxiv.org/abs/2401.01614) | grounding, not planning, is the bottleneck |
| ScreenSpot-Pro | [2504.07981](https://arxiv.org/abs/2504.07981) | the benchmark; defines ReGround |
| UI-TARS | [2501.12326](https://arxiv.org/abs/2501.12326) | end-to-end GUI agent |
| UGround | [2410.05243](https://arxiv.org/abs/2410.05243) | vision-only, modular |
| Set-of-Mark | [2310.11441](https://arxiv.org/abs/2310.11441) | coordinates - discrete choice |

Zoom methods implemented or studied here: ReGround (ScreenSpot-Pro §4.1),
UI-Zoomer's uncertainty-gated zoom, ZoomClick's crop parameters.

crop-and-zoom is not this project's invention - ScreenSpot-Pro publishes the
ladder itself (18.9 -> 31.0 -> 40.2 -> 48.1). The contribution here is
implementation, controlled comparison, and the cost and reproducibility
dimensions the papers leave out.