# Agent framework benchmark > [!NOTE] > This is an AI-generated project. If you don't want to read AI-authored prose, stop reading now. > [!IMPORTANT] > PRs are closed on this project. If you want to improve it, please open an issue or create a fork. I don't intend to maintain this benchmark forever and keep running new benchmarks on new frameworks endlessly. I'd be much more interested to see what you all build with this project (and run with your own tokens 🙂). > [!NOTE] > [Benchmark results](https://github.com/nolanlawson/agent-framework-benchmark-results) for the blog post are available if you want to peruse them. To reproduce the results, you would run `node bench/run.mjs --frameworks=all --iterations=5 --effort=high` in this repo. How many tokens, and how much time, does a coding agent need to build the same non-trivial web app in different frameworks? And how fast is the app it builds? Each agent gets the same spec (a Pokédex backed by a local copy of PokeAPI), the same tests, the same tools and the same one-line prompt. The only difference is the framework it's told to use. Each agent runs alone in a fresh Podman container. Afterwards, the app it built is measured in another container with Lighthouse. ## Quick start Requirements: Linux, Podman 4.3 or later (rootless recommended), Node 20 or later, about 15 GB of disk, and Claude credentials. ```sh nvm install && nvm use # the Node version in .nvmrc # Claude credentials: an API key, or a long-lived token from `claude setup-token` export CLAUDE_CODE_OAUTH_TOKEN=... # or ANTHROPIC_API_KEY=... # One agent, one framework node bench/run.mjs --frameworks=vanilla --iterations=1 # Everything node bench/run.mjs --frameworks=all --iterations=3 ``` The run creates `runs//`, and the results end up in `runs//report.md`. The Pokémon data (about 2 GB, from pinned commits of PokeAPI's repos) is downloaded **once**, when the `pokedex-base` image is first built. It sits in an image layer that later builds reuse, so no agent ever downloads it and it costs no tokens. Only `--no-cache`, or a change to `template/scripts/`, downloads it again. Read `bench/run.mjs` before you run it: it's the only thing that runs on your machine. Agents, and the apps they write, only ever run inside containers. ### Options | Option | Default | | | --- | --- | --- | | `--frameworks=a,b` or `all` | (required) | Keys from `bench/frameworks.mjs`: `next`, `nuxt`, `tanstack-start`, `react-router`, `solidstart`, `sveltekit`, `angular`, `astro`, `qwik`, `gracile`, `hono-htmx`, `vanilla` | | `--iterations=N` | 1 | Agent runs per framework | | `--model=ID` | `claude-opus-5-5` | Passed to `claude --model` | | `--effort=LEVEL` | `medium` | Passed to `claude --effort` (`low`, `medium`, `high`, `xhigh`, `max`) | | `--claude-version=V` | `2.1.310` | Claude Code version installed in the agent image | | `--cpus=N`, `--memory=SIZE` | 4, `8g` | Limits for every container, agent and measurement alike | | `--agent-timeout=MIN` | 240 | Stops an agent that runs longer than this | | `--lighthouse-runs=N` | 5 | Lighthouse runs per URL; the median is kept | | `--no-cache` | | Rebuild the images from scratch, pulling a fresh Ubuntu image | | `--skip-image-build` | | Reuse the existing images | | `--no-measure` | | Stop after the agents finish | | `--measure-failed` | | Also measure apps that failed the final tests. By default they're skipped, so broken apps don't get performance numbers | | `--append-to=runs/` | | Add the iterations to an earlier run, numbered after the existing ones (`next-4`, `next-5`, …), then measure every app in it. The batch's settings, harness commit and images go under `appended-` in `config.json` | | `--measure-only=runs/` | | Measure an earlier run again and rewrite its report. Rebuilds only the base and measure images, and records them as `measureOnlyImages` in `config.json` | | `--report-only=runs/` | | Only rewrite `report.md` | | `--out=DIR` | `runs` | Where run directories go | ### Credentials `run.mjs` looks for `ANTHROPIC_API_KEY`, then `CLAUDE_CODE_OAUTH_TOKEN` (create one with `claude setup-token`). Whichever it finds is passed to each agent container as an environment variable. It never goes on a command line or into the run directory. Your normal login in `~/.claude/.credentials.json` isn't used: its refresh token is replaced whenever it's used, so a copy handed to one container would stop working for the next one, and for you. The agent, and anything it runs (npm install scripts, MCP servers), can read the credential. Use a key with a spending limit. ## What happens in a run 1. **Images.** `bench/container/export-template.sh` copies `template/` into a fresh git repo with a single commit, so agents can't see this repo's history, the gold-standard app or the measurement code. Then three images are built: - **`pokedex-base`**: Ubuntu 24.04, Node 24, Playwright's Chromium, and the exported repo. The Pokémon data is downloaded here, once, by `template/scripts/setup-data.sh`. It fetches pinned commits of PokeAPI's `api-data`, `sprites` (a subset) and `cries`, and rewrites media URLs to point at the local API server. - **`pokedex-agent`**: adds Claude Code and `agent-browser`, and sets `IS_SANDBOX=1`. - **`pokedex-measure`**: adds Caddy and the measurement scripts. 2. **Agents.** For each framework and iteration, one at a time, `run-agent.sh` runs in a new agent container: - It installs the framework's official AI tooling (see below). - It runs `claude -p "" --dangerously-skip-permissions --output-format stream-json`, with no other MCP servers. - When the agent stops, it saves the agent's repo. - Then the agent's `app/`, and only that, is copied into a fresh `pokedex-base` container, where `npm test` runs. Whatever else the agent changed (tests, data, `node_modules`) or left running can't affect the result. - `pokeapi.co` resolves to localhost, so the app can only use the local API. 3. **Measurement.** Each app is built and served in production mode behind Caddy (HTTPS, HTTP/2, zstd/gzip), with the API behind Caddy too. Then: - **Lighthouse**, mobile settings only, on 6 URLs: the median of N runs, per metric. The app's server-side caches are warmed right before every run, so each visit is a new user on a site other people already use. Chrome trusts Caddy's local certificates (`--ignore-certificate-errors-spki-list`); ignoring certificate errors instead would change its caching and resource priorities. - Apps that failed the final tests are skipped unless you pass `--measure-failed`. - **Lines of code**: non-blank lines in the app's source files (comments included), not counting JSON, lockfiles, tests, generated files, or copies of the provided design files. 4. **Report.** `bench/report.mjs` writes `report.md` and `summary.json`. Page-load numbers are the home page's (every URL is in each iteration's `measure/results.json`). With several iterations, each framework's numbers are medians across them. ### The prompt The same for every agent, apart from the framework label in `bench/frameworks.mjs`: > Implement the app described in SPEC.md using **{framework}**. Use the latest stable > version. Work autonomously until `npm test` passes. ### Per-framework AI tooling Each agent gets what that framework's official docs recommend for coding agents (checked October 2026), installed up front by `bench/container/setup-tooling.sh`. Frameworks with no official tooling get nothing extra. Uneven tooling is part of what's being measured: it's what you get when you pick that framework. Some tooling only arrives through a framework's project generator, which an agent may skip (our agents usually wrote `package.json` by hand). Where that tooling can be installed up front, it is (React Router's skill). Next.js doesn't need it: `next dev` writes its `AGENTS.md` when it detects a coding agent. TanStack can't be: see the table. Agents are never told to use the tooling; whether they do is recorded (MCP and skill calls are in the transcripts). | Framework | Tooling | | --- | --- | | Next.js | `next-devtools-mcp`, `next-dev-loop` skill; docs ship in `node_modules`, and `create-next-app` or `next dev` writes an `AGENTS.md` pointing to them. Next's task-specific skills (migrations, bundle analysis) aren't installed | | Nuxt | Nuxt MCP server (`https://nuxt.com/mcp`) | | Astro | Astro Docs MCP server (`https://mcp.docs.astro.build/mcp`) | | SvelteKit | Svelte Claude Code plugin: MCP server, skills, subagent | | Angular | Angular CLI MCP server (`ng mcp`), Angular agent skills | | React Router | React Router skill (the one `create-react-router` adds to new apps); docs ship in `node_modules` | | Hono + htmx | Hono skills (nothing official for htmx) | | TanStack Start | None. Its packages ship skills in `node_modules` (TanStack Intent), and `@tanstack/cli create` tries to wire them up, but that step needs an interactive terminal and fails in non-interactive use. Start's docs don't mention them | | SolidStart, Qwik, Gracile + Lit, vanilla | None (only community-made tooling exists) | ## Reading a run For each iteration, `runs//-/` contains: | Path | What it is | | --- | --- | | `agent/transcript.jsonl` | The agent's full session (Claude Code stream-json) | | `agent/run-info.txt` | Versions, start and finish times, the MCP servers the session had (`mcp_servers`), and the exit codes of `claude` and the final `npm test` | | `agent/final-test.log` | The final test run: the original tests, in a fresh container with only the agent's `app/` | | `agent/tooling.log` | Output of `setup-tooling.sh` | | `agent/tooling-versions.json` | Exact versions or commits of the MCP servers, skills and plugins the agent had | | `agent/git-status.txt` | Files the agent changed or added | | `container.log` | Podman output for the agent container | | `repo/` | The agent's repo; the app is in `repo/app/` | | `measure/results.json` | Every measurement, including raw runs | | `measure/lighthouse/` | Lighthouse reports (first run of each URL) | | `measure/error.txt` | Why measurement failed, if it did | To summarize a single transcript: `node bench/tokens.mjs runs//-/agent/transcript.jsonl`. ## Repository layout | Path | | | --- | --- | | `template/` | Everything an agent sees: `SPEC.md`, `api/server.mjs` (local PokeAPI), `scripts/setup-data.sh`, `tests/`, `design/` | | `template/tests/` | Playwright acceptance tests that query roles, accessible names and text, not DOM structure. `run.mjs` builds the app against the normal API, then runs it against an API with changed data (`fixtures/live-data-overlay.json`) to catch apps that bake data in at build time | | `gold/` | Dependency-free reference implementation. It produced `template/design/screenshots/` and is used to check the tests (`APP_DIR=../gold npm test` in `template/`). Agents never see it | | `bench/run.mjs` | The orchestrator | | `bench/container/` | Containerfiles, the export script, the agent entrypoint and the tooling setup | | `bench/measure.mjs`, `stack.mjs` | Measurement (run inside `pokedex-measure`) | | `bench/screenshots.mjs` | Captures the reference screenshots from `gold/` | | `bench/report.mjs`, `tokens.mjs` | Report generation | | `docs/audits/` | Periodic code audits: findings, and what was decided on each and why. One file per audit, named by date | ### Working on the harness without Podman ```sh cd template && ./scripts/setup-data.sh && npm install && npx playwright install chromium npm run api # local PokeAPI on :9000 (cd ../gold && npm start) # gold app on :3000 APP_DIR=../gold npm test # all tests should pass cd ../bench && npm install node measure.mjs ../gold /tmp/gold-results --runs 1 --skip-build # needs Caddy installed ``` ## Design decisions and known limitations - **The data is live.** Apps must fetch from the API at request time; build-time fetching and static generation of data pages aren't allowed, and the tests check this. Otherwise one architecture (prerender everything) would win by sidestepping the API. - **No artificial latency** on the API. The browser's API requests are slowed by Lighthouse's throttling, but an MPA server's calls to the API run at localhost speed, like an app server sitting next to its API. Keep this in mind when comparing SPAs with MPAs. - **The site is content-heavy**, which favors server-rendered MPAs on first-load metrics. - **Only page loads are measured.** Interactions after the page loads (search, sorting, client-side navigation), where SPAs can win back ground, aren't. An earlier version timed a scripted flow, but the timings mostly reflected debounce settings, round trips and harness overhead (see `docs/audits/2026-10-25.md`). - **Styling is constrained**: shared design tokens, system fonts, no CSS frameworks or component libraries, no animations. This keeps visual and CSS differences from dominating the results. - **Pinned versus recorded.** Node is pinned by `.nvmrc` (a symlink to `template/.nvmrc`); the containers install that exact version from nodejs.org, with checksum verification, and npm is the version bundled with it. Playwright and agent-browser are pinned in the Containerfiles; Claude Code, the model and the effort level by `run.mjs`'s defaults. Frameworks and their AI tooling are deliberately *not* pinned: each agent uses the latest release, like a real user would. Each result is a dated snapshot of the whole ecosystem (model, frameworks, tooling, docs, search results), none of which can be frozen for a rerun years later anyway. Everything that was used is recorded instead: - the app's installed dependency versions (`measure/results.json`, and the report) - MCP package versions; npx-launched servers are pinned to exactly the recorded version - for hosted MCP servers, their tool list and a hash of it - the commit of every skill and plugin repo (`agent/tooling-versions.json`) - Node, npm, Playwright, Claude Code and agent-browser versions (`agent/run-info.txt`) - the image IDs and the Ubuntu base image digest (`config.json`) - **Some text checks don't read inside shadow roots.** The spec allows open shadow roots, and the tests' role, label and text locators find elements inside them. But `textOf` and `expectText` in `template/tests/specs/helpers.mjs` read `innerText`, which leaves out the contents of shadow roots below the element. An app that rendered, say, a move's Power or a Pokémon's Height inside a component's shadow root would fail those checks. No app so far has: Gracile's shadow-DOM components hold only the list and the moves table, which the tests check through roles. The helpers are left as they are, so the tests match what earlier agents saw. - **One iteration is an anecdote.** Agent runs vary a lot; use 3 or more iterations before drawing conclusions.