# Methodology

## What is measured

For each of six representations of a web page:

1. **Token cost**: the tokens the page representation adds to a prompt, counted
   separately from the fixed prompt (instructions, question, answer format).
2. **READ accuracy**: exact-match answers to factual questions about the page.
3. **ACT accuracy**: whether the model identifies the element to click or fill,
   resolved back to a live element and compared with ground truth.

The benchmark measures each **representation**, sent once, at the top of the
page, to one model with one shared prompt. It does not measure any library's
agent loop (Browser Use's `Agent`, Stagehand's `act`/`observe`/`extract`
prompts, scrolling, retries or self-healing). Those layers add their own prompts
and calls, so the token numbers here are a lower bound on what each library
spends in production.

## Environment

| Item | Value |
|---|---|
| Date | Pilot 2026-10-01 (run `pilot-20261001T170632Z`); full run 2026-10-02 (run `full-20261002T073411Z`) |
| Machine | Apple M4 Pro, macOS 26.6.2 (arm64) |
| Python | 3.12.13, managed with uv 0.11.29; three lockfiles (see below) |
| Browser | Chrome for Testing 153.0.8010.12 bundled with Playwright 1.63.0 (`channel="chromium"`, new headless), viewport 1280x800 |
| Model | `claude-sonnet-5` (Anthropic API, SDK `anthropic` 1.11.0) |
| Sampling | `temperature`/`top_p`/`top_k` omitted: Sonnet 5 rejects non-default values with a 400 error (platform.claude.com/docs/en/models/sonnet-5/overview). Sampling is therefore at the default temperature; variance is measured with 3 repeats. |
| Thinking | `{"type": "disabled"}` (adaptive thinking is on by default on Sonnet 5) |
| Other | `max_tokens` 512, no prompt caching, `max_retries=0`, 120 s timeout, 4 concurrent calls |
| Tokenizers | Anthropic token counting endpoint for `claude-sonnet-5`; tiktoken 0.14.0 `o200k_base` |
| Libraries | browser-use 0.13.10; stagehand 4.1.0 (Python SDK, bundled Stagehand Runtime extension); markdownify 1.2.3 |

`claude-sonnet-5` was the model specified for this benchmark. It is listed as
"Legacy" (current Sonnet: `claude-sonnet-5-5`), and remains available until at
least 2027-06-30.

**Three environments.** browser-use 0.13.10 pins `websockets==15.0.1` (through
`browser-harness`) and stagehand 4.1.0 requires `websockets>=16.1.1`, so they
cannot share an environment. The core harness, `adapters/browser_use` and
`adapters/stagehand` are separate uv projects, each with its own committed
`uv.lock`, so every library runs on exactly the versions its authors declare.
The adapters are long-lived subprocesses speaking JSON over stdin/stdout. They
receive no API key (the harness strips `ANTHROPIC_API_KEY` from their
environment) and make no LLM calls.

## Pages

All pages are served from `pages/` by a local server (`src/wpbench/server.py`).
Every Chromium the benchmark launches (Playwright's and Stagehand's) gets
`--host-resolver-rules=MAP * ~NOTFOUND, EXCLUDE localhost, EXCLUDE 127.0.0.1`,
so no request can reach the internet. `scripts/verify_pages.py` lists every
request a page attempts; the only blocked ones are:

- books.toscrape.com pages request jQuery over plain http from
  ajax.googleapis.com. The live https site blocks it as mixed content and falls
  back to its own copy, which is what happens locally too.
- saucedemo.com sends error telemetry to events.backtrace.io.

| id | kind | source |
|---|---|---|
| article | article | local fixture, original text |
| docs | docs page | local fixture, fictional API |
| books_listing | e-commerce listing | snapshot of https://books.toscrape.com/ (Zyte practice site) |
| books_product | product page | snapshot of https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html |
| saucedemo_login | login form | snapshot of https://www.saucedemo.com/ (Sauce Labs practice site; client-rendered React app) |
| form | multi-field form | local fixture |
| dashboard | dashboard-like page | local fixture, synthetic data, inline SVG chart |
| serp | search results | local fixture, fictional results |
| table | long data table | local fixture, 500 rows generated by `scripts/make_table_fixture.py` (fixed-seed LCG) |
| embed | iframe + closed Shadow DOM | local fixture: same-origin iframe (localhost:8100), cross-site iframe (127.0.0.1:8100, out of process), closed-shadow `<promo-code>` |

Snapshots were captured once with `scripts/capture_snapshot.py`. Every
response the page loaded was saved as the server sent it (documents are not
the rendered DOM, so client-side scripts run again locally), with a sha256
manifest per site (`pages/snapshots/*/manifest.json`).

**Licensing.** The practice sites publish no redistribution license. Before
publishing this repository, decide whether to ship the snapshots or the
capture script plus checksums.

## Representations

Every model input comes from the same **page session**, and model answers are
resolved against the same session.

**Benchmark browser.** One fresh load in a new Playwright context:

1. Set the viewport to 1280x800.
2. `goto`, wait for network idle, wait 500 ms.
3. Scroll to (0, 0).

All representations except Stagehand's are captured from this load, in this
order:

1. **raw_html**: `page.content()`, the serialized top document only. It
   contains neither iframe content nor shadow-root content.
2. **markdown**: `markdownify.markdownify(raw_html)` with default settings.
   markdownify was chosen over trafilatura because it converts the whole page
   without selecting content. trafilatura is a boilerplate remover: it drops
   navigation, forms and footers by design, so the benchmark would measure its
   extraction choices rather than the Markdown format. Markdown is also not its
   default output (that is plain text). markdownify drops `<input>`/`<select>`
   elements and escapes `_` and `*`.
3. **accessibility_tree**: `page.locator("body").aria_snapshot()`, default mode.
   Default mode renders an iframe as a bare `- iframe` node without its
   content. Playwright cannot see closed shadow roots in any mode.
   `mode="ai"`, which includes iframes and refs, was not used, as specified.
4. **screenshot**: `page.screenshot()` of the viewport, PNG, 1280x800.
5. **indexed_dom**, Browser Use's own serializer. The Browser Use adapter
   attaches to the benchmark browser over CDP (`BrowserSession(cdp_url=...)`),
   switches to the benchmark tab, and returns
   `SerializedDOMState.llm_representation(include_attributes=DEFAULT_INCLUDE_ATTRIBUTES)`.
   That is the string the Browser Use agent puts in its prompt
   (`agent/prompts.py`), from the same `DomService` its `DOMWatchdog` builds,
   with the default `BrowserProfile` flags:
   `cross_origin_iframes=True`, `paint_order_filtering=True`, `max_iframes=100`,
   `max_iframe_depth=5`.
   - **One deliberate deviation from the agent default:**
     `viewport_threshold=None`. By default the `DomService` drops every element
     more than 1000 px below the viewport, which suits an agent that scrolls but
     meant that a single snapshot covered less of each page than Stagehand's.
     With the cutoff, `indexed_dom` missed 4 of 30 ACT targets. Without it,
     Browser Use's output contains all 30 ACT targets and 48 of 50 READ answers
     (`scripts/coverage_report.py`). `viewport_threshold` is a public
     constructor parameter of `DomService` (`dom/service.py`).
   - We verified that calling `DomService` directly with
     `viewport_threshold=1000` is byte-identical to the agent path
     (`get_browser_state_summary`) on the pilot pages.
   - The agent-default output is still saved for every page load as
     `indexed_dom.agent_default.audit_only.txt`. It is never sent to the model.
   - The decision was made after token counts, but before any model output,
     had been seen.
   - The agent truncates the text at 40,000 characters; we send it untruncated
     and record `agent_would_truncate` per page load.

**Stagehand's browser.** Each page session opens a fresh Stagehand tab (`context.new_page()`) and closes the previous one. Stagehand's Chrome extension runtime must be present
when a document is created. Attached to a page the benchmark browser had
already loaded, its locators failed ("Stagehand extension world not ready") and
the cross-site iframe was missing from its snapshot. Stagehand therefore
launches and drives its own Chromium:
`local_browser.launch(executable_path=<Playwright's Chrome for Testing>, headless=True, args=<offline flags>)`.
That is the same binary and the same offline flags as the benchmark browser.
`set_viewport_size(1280, 800)` is called before `goto`, because the launch
viewport only sets the window size. The page then gets the same network-idle
wait.

6. **stagehand_snapshot**: `formatted_tree` from `await page.snapshot()`.
   - `page.snapshot()` calls `captureHybridSnapshot` with `pierceShadow: true`
     and `includeIframes` left at its default (true).
   - Stagehand's `act`, `observe` and `extract` call the same function, with
     only optional `focusLocator`/`ignoreLocators`, which are unset unless the
     caller scopes the call. So this is the tree Stagehand's own methods send
     their LLM (see the bundled `_extension/service-worker.js`).
   - Stagehand v4 has no `agent()`. No Stagehand LLM call is made.

**Integrity checks recorded per page session.**

- `page.content()` before and after Browser Use attached and detached.
  Browser Use's serialization sometimes adds empty `style=""` attributes to
  form controls (observed on the form page; origin within its CDP calls not
  pinpointed). The other representations are captured before Browser Use runs,
  so this does not affect model inputs. It does not affect resolution either:
  no selector depends on `style`.
- Stagehand's inner viewport (always 1280x800).

## Tasks

`tasks.json` was committed (b176ecf) before any model call. It holds 80 tasks,
5 READ and 3 ACT per page, plus the prompt template, answer formats and all
scoring rules. It has not changed since.

- **READ** tasks have an exact answer plus pre-registered aliases.
- **ACT** tasks have a ground-truth `target`: a chain of CSS selectors, one per
  boundary hop (iframe, or shadow host, including closed roots). Each selector
  must match exactly one element in its scope.
- Every task has a `location` tag measured from page geometry
  (`scripts/validate_tasks.py`): above_fold, below_fold (at most 1000 px past
  the viewport), far_below_fold, attribute_only (form state, CSS class or
  attribute only), iframe_same_origin, iframe_cross_site, shadow_closed.
- Some tasks deliberately target things only some representations contain.
  They were chosen per page type, not tuned to any representation.

## Prompt

One template for every representation (`tasks.json` → `protocol.prompt`). Only
the page content and, for ACT, the answer-format instruction change.

- **Text representations:** the page is inserted verbatim.
- **Screenshot:** the PNG image block sits between the same two text blocks. It
  carries `"transformations": {"oversized_image": "error"}`, so any server-side
  resize would be an error rather than a silent coordinate shift. 1280x800 is
  not resized on either resolution tier.

## Answer resolution (ACT)

Identity of an element:

- **Benchmark browser:** the CDP backendNodeId, plus the CDP target that owns
  the node. The target is needed because out-of-process iframes have their own
  id space.
- **Stagehand's browser:** Stagehand's own xpath for the element.

**Ground truth** is resolved in both page sessions, through CDP
`DOM.getDocument(pierce=True)` (`src/wpbench/groundtruth.py`). Out-of-process
frames are reached through a CDP session on the Playwright `Frame`.

| Representation | Answer | Resolution (`src/wpbench/resolve.py`) |
|---|---|---|
| raw_html | CSS selector | CDP `querySelectorAll` on the top document; must match exactly one element |
| markdown | visible text or label (+ optional `occurrence`) | Playwright, first strategy with any match decides: interactive role with exact name, then `get_by_label`, then `get_by_placeholder`, then `get_by_text` (all exact). `occurrence` (1-based, page order) picks among duplicates; several matches without it = `ambiguous` |
| accessibility_tree | role + accessible name (+ optional `occurrence`) | `get_by_role(role, name=name, exact=True)`, same `occurrence` rule |
| indexed_dom | index | Browser Use's selector map: index → (target id, backendNodeId) |
| stagehand_snapshot | ref, e.g. `0-12` | Stagehand's own act path (`normalizeActInferenceElement`): `xpath_map[ref]` → `trimTrailingTextNode` → Stagehand locator `xpath=...`, which must match exactly 1. Correct if the xpath equals the ground truth's Stagehand xpath |
| screenshot | x, y | Inside the 1280x800 image and inside the target's box (Playwright `bounding_box()`; CDP `DOM.getBoxModel` for closed-shadow targets) |

**Ground truth's Stagehand id.** Stagehand ids are
`<frame ordinal>-<backendNodeId>`. The ground truth's id is the `xpath_map` key
with its backendNodeId whose xpath has the same iframe depth. The depth
disambiguates out-of-process frames.

**Representability** is decided before scoring, from the representation and
the live page, never from the answer (`src/wpbench/scoring.py`):

- raw_html, markdown and accessibility_tree: the target must be in the top
  document. In addition:
  - markdown: the target's text, label, placeholder, aria-label or title must
    appear in the Markdown;
  - accessibility_tree: the target's own role + name node must appear, and a
    node without a role (e.g. `<summary>`, shown as `text`) is not addressable.
- indexed_dom: the target is in the selector map.
- stagehand_snapshot: the target's id appears in the tree.
- screenshot: the target's box intersects the viewport.

A wrong answer on a non-representable target is recorded with reason
`not representable`; the underlying reason is kept in `failure_detail`.

**Offline self-test** (`scripts/selftest_resolution.py`):

- For every ACT task and representation (180 pairs), the answer a perfect
  model would give is built from ground truth. It must score correct exactly
  when the target is representable.
- A wrong answer (another task's target) must score incorrect.

All 180 pairs pass.

## READ scoring

Normalization:

1. Unicode NFKC.
2. Drop a backslash before ASCII punctuation (Markdown escapes).
3. Map Unicode dashes to `-`.
4. Trim and collapse whitespace.
5. Strip one trailing `.`.
6. Casefold.

The answer must equal the normalized answer or a pre-registered alias.

`answer_present` (READ representability) is a normalized substring check. It
is not meaningful for derived answers (e.g. a count such as "6"), so READ
`not representable` labels are listed in the report for audit. The label never
changes accuracy.

## Token accounting

- `representation_tokens` = count_tokens(full prompt) − count_tokens(the same
  prompt with an empty page slot).
- `fixed_prompt_tokens` = count_tokens(prompt with an empty page slot).
- Both counts use the same model, system prompt and thinking setting.
- tiktoken o200k_base counts the representation string alone, and the fixed
  prompt as the system prompt plus the user text with an empty page slot.
- **Screenshot:** `representation_tokens_api` = `usage.input_tokens` of the
  actual call − fixed prompt. Image dimensions before and after provider-side
  resizing and padding are computed with Anthropic's reference `resized_size()`
  (`src/wpbench/vision.py`): 1280x800 → 1280x800, padded to 1288x812, 1,334
  visual tokens.
- Generating the representations costs no LLM tokens: all six are
  deterministic conversions.

## Runs

- Each (repeat, page, task, representation) is called exactly once.
- Pages are visited in a random order per repeat, and calls are shuffled within
  each page session. The seed and full order are in `results/runs/<run_id>.json`.
- API errors and timeouts are not retried. They count as incorrect and are
  reported separately.
- `tokens_per_correct` = total API input tokens over all calls ÷ number of
  correct answers.

## Deviations and incidents log

| Date | What happened |
|---|---|
| 2026-10-01 | markdownify 1.2.3 used in the core environment after splitting environments (browser-use pins 1.2.2; the only change in 1.2.3 is an edge case for text nested in `<br>`). |
| 2026-10-01 | Stagehand moved to its own browser (see above). |
| 2026-10-01 | `indexed_dom` switched to `viewport_threshold=None` for full-page coverage (see above), before any model call. |
| 2026-10-01 | First full run (1,440 calls, 0 API errors) invalidated by a harness bug found while auditing failures: concurrent `raw_html` resolutions shared one CDP session, and `DOM.getDocument` resets that session's node ids, so 2 valid selectors (each matching exactly one element on a fresh load) were scored `invalid_selector`. Fix: one CDP session per `raw_html` resolution; a lost node id now raises instead of being scored; regression test `tests/test_concurrent_resolution.py` (fails on the old code, passes on the new). No other resolver depended on CDP node ids. The run is kept in `results/aborted/` and excluded from all analysis; the full run was rerun from scratch. The pilot had no `invalid_selector` records. |
| 2026-10-01 | Second pilot attempt stopped by a harness error after 192 of 432 calls. Cause: when resolving ground truth in Stagehand's browser, (a) Stagehand's single reused tab accumulated stale duplicate entries in `xpath_map` after navigating (the `formatted_tree` the model sees was not affected), and (b) Stagehand ids were matched by backendNodeId + iframe depth only, which is ambiguous when two iframes at the same depth live in different processes. 6 Stagehand ACT calls had been scored incorrect because their ground truth could not be found. Fixes: Stagehand gets a fresh tab per page session (like the benchmark browser's fresh context); Stagehand ids are disambiguated by the xpath of the enclosing `<iframe>`; any ground-truth failure now aborts the run instead of being scored; detached frames are skipped. The run is kept in `results/aborted/` and excluded from all analysis. Only per-page correct counts printed to the log were seen before the fix. The pilot was then rerun from scratch. |
| 2026-10-01 | First pilot attempt crashed on the first model response. `await` was missing on the async SDK's `parse()`. A handful of in-flight calls were billed; none was scored or recorded. Fixed in b19504e, then the pilot was rerun from scratch. |

## Limitations

- **One snapshot from the top of the page.** No representation gets scrolling
  or multiple steps. Agents built on Browser Use or Stagehand can scroll and
  re-observe; screenshots are limited to the viewport by design of this setup.
- **One model**, at default temperature (temperature 0 is not accepted), with
  3 repeats.
- **One shared prompt**, not each library's own prompts. Libraries tuned for
  their own prompts may do better or worse with theirs.
- **Small task set.** 10 pages and 80 tasks. Seven pages are fixtures written
  for this benchmark, and three are practice sites rather than production sites.
- **Different browser processes.** Stagehand's representation comes from a
  separate (identical) load in its own browser process. Pages contain no
  randomness, and ground truth is resolved independently in each load.
- **Representability labels are approximate for short strings.** The checks
  are substring checks, so a short answer (e.g. "4", "6") or a label that is a
  substring of other text ("Username" in "Accepted usernames") can be labelled
  present when it is not. Labels never change accuracy; they only decide
  whether a failure reads `not representable`.
- **One task-design ambiguity.** `docs.act.3` ("Switch the documentation to
  version 3.1") targets the `<select>`, but answering its `<option>3.1</option>`
  is defensible. Scored as pre-registered. `results/report_tables_full.md`
  gives a clearly labelled sensitivity line.
- **Tokenizer counts are estimates.** The Anthropic token counting endpoint is
  documented as an estimate that may differ slightly from billed usage. The
  billed `usage.input_tokens` is recorded for every call.