# OpenVM benchmark sets

Benchmark sets for measuring apc-optimizer’s effectiveness against powdr. Each set is a
subdirectory here, structured identically (see [Layout](#layout)); the benchmark script selects one
by name. The **main** benchmark — used for optimization and the script's default — is
[**`openvm-eth/`**](./openvm-eth/).

| benchmark | notes |
| --- | --- |
| [`openvm-eth/`](./openvm-eth/) | **main / default.** The 100 costliest basic blocks of the OpenVM reth benchmark. |

## Running the benchmark

You'll need [uv](https://docs.astral.sh/uv/). The optional positional argument names the benchmark
(a subdirectory here; default `openvm-eth`). To sweep a set in parallel and report aggregate
apc-optimizer-vs-powdr effectiveness — the factor by which each of three size measures shrinks (in priority
order: variables, then bus interactions, then algebraic constraints), as both aggregate
(Σbefore ⁄ Σafter) and geomean:

```bash
OpenVmBenchmarks/benchmark.py                # all openvm-eth cases (--jobs = cores)
OpenVmBenchmarks/benchmark.py openvm-eth     # same, named explicitly
OpenVmBenchmarks/benchmark.py --n 20         # top 20 by cost rank
OpenVmBenchmarks/benchmark.py --n 10 --report report.html   # + interactive HTML report
```

## Latest results (snapshot)

Effectiveness at repo commit `d922b32` (2026-07-11); refresh this section when the numbers move.
Full openvm-eth sweep (`OpenVmBenchmarks/benchmark.py`, 100 cases; agg = Σbefore ⁄ Σafter,
geo = geomean):

| measure | apc-optimizer (agg / geo) | powdr (agg / geo) | diff (agg) |
| --- | --- | --- | --- |
| variables | 4.136× / 3.706× | 4.092× / 3.787× | +0.044× |
| bus interactions | 2.951× / 2.447× | 3.480× / 2.822× | −0.529× |
| constraints | 9.073× / 11.190× | 5.853× / 10.311× | +3.220× |

Per-case (by variables): apc-optimizer wins 17, loses 52, ties 31.

For the keccak stress case (not part of the sweep), see the table in
[keccak (standalone stress case)](#keccak-standalone-stress-case) below — same snapshot commit.

## Layout

Each benchmark set is a subdirectory of `OpenVmBenchmarks/`. To add another, create a new
subdirectory with the same structure; `benchmark.py <name>` will then run it. Each block is exported
as a pair:

- `apc_<rank>_pc<pc>.json.gz` — the powdr `SymbolicMachine` (`ApcWithBusMap` JSON) exactly as it
  enters powdr's `optimize()`, with its bus map. This is apc-optimizer's **input**.
- `apc_<rank>_pc<pc>.powdr_opt.json.gz` — powdr's optimized result, for comparison.

`<rank>` is the cost rank (`001` = costliest) and `<pc>` the block's start program counter(s). Cost
= `width_before × execution_frequency` (the trace cells the block costs without an auto-precompile).
`manifest.json` records the full ranking (per-entry cost, execution frequency, width, before/after
stats, provenance), and `apc_candidates.json` is powdr's full candidate dump the set was drawn from —
also the source of each block's assembly (shown in the `--report` HTML).

## openvm-eth

The main benchmark set, in [`openvm-eth/`](./openvm-eth/): the 100 costliest basic blocks of the
OpenVM reth benchmark. Generated by
[powdr-labs/openvm-eth#11](https://github.com/powdr-labs/openvm-eth/pull/11). See [this APC analyzer dashboard](https://powdr-labs.github.io/powdr/autoprecompile-analyzer/?data=https%3A%2F%2Fgithub.com%2Fpowdr-labs%2Fapc-optimizer%2Fblob%2Fmain%2FOpenVmBenchmarks%2Fopenvm-eth%2Fapc_candidates.json).
This set was built from Ethereum mainnet [block 24171377](https://etherscan.io/block/24171377)
(manifest `source` note: openvm-eth e3df69e block 24171377, cell PGO).

## keccak (standalone stress case)

`keccak_apc_pre_opt.json.gz` is a single large APC — the keccak precompile
(~28k algebraic constraints / ~27k columns) — a powdr `SymbolicMachine` (`ApcWithBusMap` JSON)
export in the same format as the case files above, but **not** part of a sweep set: it is not
walked by `benchmark.py`. It's a stress case for the optimizer.
`keccak_apc_pre_opt.powdr_opt.json.gz` is its powdr-optimized pair, generated by running powdr's
native `optimize()` (the `autoprecompiles/tests/optimizer.rs::test_optimize` configuration, powdr
commit `b072302`) on the input and exporting the result; its counts match that test's snapshot.
`keccak_apc_pre_opt.apc_opt.json.gz` is apc-optimizer's own optimized output (a snapshot at the
manifest's commit — regenerate with `apc-optimizer opt-export` as the optimizer improves), so the
two optimized circuits can be diffed directly for gap analysis.

Running it takes **minutes** (the cleanup loop runs to a fixpoint over a circuit this large;
~6.3 min at repo commit `b8d2593`, 2026-07-11), so it is not part of `benchmark.py` or CI. Run it
on demand with the existing CLI — the same generic parse → optimize → report path used for every
benchmark circuit:

```bash
lake exe apc-optimizer run OpenVmBenchmarks/keccak_apc_pre_opt.json.gz          # apc-optimizer only
lake exe apc-optimizer compare OpenVmBenchmarks/keccak_apc_pre_opt.json.gz \
    OpenVmBenchmarks/keccak_apc_pre_opt.powdr_opt.json.gz                       # + powdr, side by side
```

Measured effectiveness (repo commit `b8d2593`, 2026-07-11), also recorded in
[`keccak_manifest.json`](./keccak_manifest.json) (same shape as a sweep set's `manifest.json`).
powdr's numbers match its own snapshot test (`test_optimize`), which runs the native Rust
optimizer on the byte-identical file:

| | columns (vars) | bus interactions | constraints |
| --- | --- | --- | --- |
| before | 27521 | 13262 | 28627 |
| apc-optimizer | 3622 (7.60×) | 3904 (3.40×) | 492 (58.18×) |
| powdr | 2021 (13.62×) | 1734 (7.65×) | 186 (153.91×) |