# Benchmarks

Every number in the README comes from here. Each table names the script that reproduces it; raw results for the live-site comparison are in `bench/competitors/results/`.

## Knowing something without reading the page

The same question about the same live page, through both servers, driven over MCP. For Playwright
MCP the cost of knowing is the page snapshot the caller has to read; here it is the answer
`browse_ask` sends back.

| page | question | Playwright MCP | this | |
|---|---|---|---|---|
| Wikipedia, Ada Lovelace | when and where was she born | 108,726 tokens | **266** | 409x |
| GitHub, microsoft/playwright | stars and license | 14,531 | **399** | 36x |
| MDN, `Array.prototype.at` | negative and out-of-range index | 13,918 | **519** | 27x |
| Hacker News | top three stories, as JSON | 13,478 | **418** | 32x |
| | | **150,653** | **1,603** | **94x** |

Every answer was correct. The page is read once by Gemini Flash on the server, at $0.0004 to
$0.003 per question with its thinking budget at zero, and never enters the conversation — which
matters more than the one-off saving, because a snapshot in the context is paid for again on every
later message. (`node bench/context.mjs` reproduces it.)

## Against the field, on live websites

Everything above was measured on pages built for the purpose. This is the other kind of test:
[WebVoyager](https://github.com/MinorJerry/WebVoyager) (He et al., 2024) — 643 tasks on fifteen
real sites — sampled with a fixed seed, run on live sites, graded by a model that is told nothing
about which system it is grading. Harness, raw results and grading reasons are in
`bench/competitors/` (`node bench/competitors/webvoyager.mjs A|B`).

**How it was run, so it can be checked.** Every system drove its own real Chrome with a fresh
profile, because several of these sites refuse a headless browser and that must not decide a
result. Same step cap (30) for everyone. Tasks naming a year already past were left out. The judge
was `openai/gpt-5-mini` — a different model family from everything under test, because this server
answers questions with Gemini Flash and Gemini should not be grading its own answers.

### A — autonomous agents, 30 tasks on 15 sites

Given the task and the start URL, nothing else; graded on the page each one ends on.

| | passed | median time | cost per task |
|---|---|---|---|
| **jev-browser** (`browse_goal`) | **9 / 30 (30%)** | 18 s | **$0.0009** |
| [browser-use/jev-ultrafast](https://github.com/browser-use/jev-ultrafast) | 6 / 30 (20%) | 16 s | $0.0046 |

Paired, task by task: both passed 3, only jev-browser 6, only ultrafast 3, neither 18. An exact
McNemar test on that gives **p = 0.51 — the difference in success is not significant** at this
sample size. The difference in cost is: it held on every task, about five times.

And the absolute numbers are low — for both. Agents where a large model reasons about every step
report far more on this benchmark (the WebVoyager paper's GPT-4V agent around 59%; browser-use
reports around 89% for its own LLM agent — their figures, not re-measured here). A typed-decision
model choosing among the controls on screen does not replace that reasoning on the open web: every
task that needed several criteria compared against the page's content — the Allrecipes recipe with
over a thousand reviews, Wolfram Alpha, Hugging Face docs, Cambridge Dictionary — was failed by both.

### B — the same model driving each MCP server

Here a model does the reasoning and the server does the browsing, which is how an MCP server is
actually used. Claude was not available on the gateway tier this ran on, so the driver was
`openai/gpt-5` for both servers, with the same instructions. Budget allowed four paired tasks; that
is an observation, not a statistic.

| task | Playwright MCP | jev-browser |
|---|---|---|
| Allrecipes, vegan smoothie bowl | failed · 384k driver tokens · $0.51 | failed · 275k · $0.39 |
| Amazon, filtered hiking boots | failed · 280k · $0.38 | passed* · 215k · $0.32 |
| Apple, iPad mini video resolution | passed · 16 steps · 88k · $0.13 | **passed · 3 steps · 11k · $0.03** |
| ArXiv, quantum papers of the last two days | failed · 398k · $0.53 | passed* · 256k · $0.42 |
| | **1 / 4** · 1.15M tokens · $1.54 | **1 / 4 verified + 2 unverified** · 0.76M · $1.15 |

\* *Graded on the driver's answer alone.* A harness bug left jev-browser's final page unread on
those runs — the page text was wrapped in JSON, cut at the tool's 8,000-character limit and failed
to parse — so the judge had only the agent's own account to go on. They are reported as
unverified, not as passes, and the bug is fixed in the harness. Dollar figures are list price; the
gateway billed less, because repeated context is cached.

The Apple row is what the server is built for: the driver called `browse_goal` once and
`browse_ask` once, and got the answer for an eighth of the tokens.

**The first version of this server lost this comparison, and the reason is worth keeping.**
Driven by GPT-5 before the fixes below, the driver never called `browse_goal` or `browse_ask`: it
used `browse_observe` and `browse_act` step by step, exactly as it used Playwright MCP — and this
server was the more expensive of the two (486k driver tokens against 340k on the same task).
Playwright MCP 0.0.82 is far leaner than its reputation: it writes snapshots to files and offers
`browser_find`. Three things changed after that measurement: action results stopped carrying the
request log and a navigation fell from 4,800 characters to a few hundred; `browse_observe` gained
`find`; and the server now sends MCP `instructions` telling the model when to use `browse_goal` and
`browse_ask` — which the driver then did. The same instructions are what Claude Code shows the
model.

## Honest assessment

What this is good at, measured: reading a page without paying for it (27× to 409× less context
than Playwright MCP on four live pages); running defined flows cheaply and replaying them for free,
repairing only what changed; enterprise screens — JSF, PrimeFaces, iframes, framework-drawn buttons
— where it was hardened against a live application; and costing about a fifth of the other
Jev-based agent for a success rate that is at least as good.

What it is not, measured too: an open-web agent. On its own it completes about a third of the
WebVoyager tasks sampled here, and it stops rather than guesses when it is unsure — the right
behaviour under an orchestrating model, a failure when it runs alone. Its advantage over
Playwright MCP exists only when the driver uses the tools that keep the page out of its context;
driven step by step, the two are roughly even. Google Flights, the benchmark jev-ultrafast is
famous for, still fails here, and not the same way twice.

## Doing something, not just reading

Playwright MCP 0.0.82 writes its snapshot to a file and returns a link, so for actions its best case
is one inline snapshot to learn the refs, then acting. That best case is what was scripted, with
`browser_fill_form` where it applies (`node bench/context-actions.mjs`):

| task | Playwright MCP, used well | this, `browse_act` | this, `browse_goal` |
|---|---|---|---|
| four-field form, then submit | 360 tokens · 4 calls | 252 · 3 | 527 · **1** |
| dependent dropdown | 321 · 4 | 208 · 2 | 340 · **1** |
| autocomplete | 362 · 5 | 257 · 3 | 373 · **1** |
| **Wikipedia search, live page** | **18,875** · 3 | **405** · 3 | **456** · **1** |

The honest reading: on a toy page whose whole snapshot is a hundred tokens there is nothing to save,
and the only difference is the number of calls. The saving is the snapshot, so it appears exactly
when the page is real.

## Hard cases, and older head-to-heads

Six hard cases, each a self-hosted page doing something plain click-and-type cannot express, each
verified by reading the **page** rather than the agent's claim that it finished:

| case | result | steps | Jev tokens | $ (Jev + text model) |
|---|---|---|---|---|
| calendar — open, change month, pick a day, confirm | pass | 7 | 12,048 | 0.000506 |
| autocomplete — typing is not enough, the suggestion must be chosen | pass | 3 | 3,077 | 0.000202 |
| dependent dropdown — the second list only exists after the first is set | pass | 3 | 4,041 | 0.000170 |
| ordered form — fields that must be filled in sequence | pass | 6 | 7,410 | 0.000609 |
| delayed dialog — the control appears seconds after the click | pass | 4 | 3,944 | 0.000166 |
| form inside an iframe — the main document has no controls at all | pass | 5 | 7,875 | 0.000531 |
| | **6/6** | | **38,395** | **$0.00218** |

The dollar column is the real bill: Jev at $0.042 per million input tokens, and Gemini Flash at
$0.30 in and $2.50 out for the fields it writes. Until this was checked, the text model's tokens
were being added to Jev's and priced at Jev's rate, which made the form-filling cases look up to
three times cheaper than they are — the ordered form reported $0.000344 and costs $0.000609.

On the five cases that both tools can run, against
[browser-use/jev-ultrafast](https://github.com/browser-use/jev-ultrafast) through the same gateway,
same machine, same day:

| | passed | tokens | $ | wall |
|---|---|---|---|---|
| jev-ultrafast | 4/5 (misses the calendar) | 35,599 | 0.00150 | **24 s** |
| this | **5/5** | **31,457** | **0.00132** | 34 s |

More accurate and slightly cheaper; about 25% slower. Those are the honest terms — with one
caveat found later: both columns priced every token at Jev's rate, so neither includes what its
text model really costs.

Against [`@jkudish/jev-browser`](https://github.com/jkudish/jev-browser) on the same goals:

| goal | jkudish | this |
|---|---|---|
| Wikipedia search → article | done · 51,874 tokens · $0.00218 | done · **7,563** · **$0.00032** |
| 45-link index → one example page | done · 4,865 tokens · $0.00020 | done · **3,042** · **$0.00013** |
| element that appears after a delay | **stuck** | **done** · 2,935 · $0.00012 |

And one page read, the single measurement that decides what a long session costs:

| page | this | Playwright MCP |
|---|---|---|
| Hacker News front page | **606** | 12,283 |
| MDN article | **669** | 8,803 |
| a 45-link index | **424** | 1,650 |
| a login form | **98** | 321 |

Over a twenty-step job: 9,821 tokens here, 23,150 through `public-browser`, 100,286 through
Playwright MCP.

Tool definitions are **~3,756 tokens** for thirteen tools — paid once per message in your
conversation, not per step. Playwright MCP is ~5,584 with its default capabilities and ~9,756 with
all of them (fifty tools).

## Run the benchmarks

```bash
npm test                  # 14 capability probes, no network goals
node bench/suite.mjs      # six goal scenarios
node bench/hard/run.mjs   # the six hard cases above
node bench/hard/run.mjs --ultrafast   # the same cases through jev-ultrafast
node bench/coverage.mjs   # Playwright MCP's capabilities, through MCP, checked on the page
node bench/context.mjs    # caller-context cost vs Playwright MCP on four live pages
node bench/context-actions.mjs   # the same for doing things: forms, dropdowns, a live search
node bench/heal.mjs       # record, replay, redesign, heal — and run the exported specs
node bench/competitors/webvoyager.mjs A   # vs jev-ultrafast on live WebVoyager tasks
node bench/competitors/webvoyager.mjs B --driver openai/gpt-5   # vs Playwright MCP, same driver
node bench/competitors/report.mjs bench/competitors/results/*.jsonl
node src/replay.js recipe.json [--no-heal] [--playwright]   # replay or export from the command line
```

One trap worth knowing if you benchmark against Playwright MCP yourself: when its socket
directory path is too long it does not fail the call, it returns `### Error` as the result. An
early version of `context.mjs` measured that error message as if it were a page and reported
Playwright MCP as the cheaper server. Set `PWTEST_SOCKETS_DIR` to a short path.