# OpenVM benchmark sets

Benchmark sets for measuring apc-optimizer’s effectiveness against powdr. Each set is a
subdirectory here, structured identically (see [Layout](#layout)); the benchmark script selects one
by name. The **main** benchmark — used for optimization and the script's default — is
[**`openvm-eth/`**](./openvm-eth/).

| benchmark | notes |
| --- | --- |
| [`openvm-eth/`](./openvm-eth/) | **main / default.** The 100 costliest basic blocks of the OpenVM reth benchmark. |
| [`keccak/`](./keccak/) | Single-case stress set: the keccak precompile APC (~28k constraints / ~27k columns). One run takes minutes. |

## Running the benchmark

You'll need [uv](https://docs.astral.sh/uv/). The optional positional argument names the benchmark
(a subdirectory here; default `openvm-eth`). To sweep a set in parallel and report aggregate
apc-optimizer-vs-powdr effectiveness — the factor by which each of three size measures shrinks (in priority
order: variables, then bus interactions, then algebraic constraints), aggregated over the set
(Σbefore ⁄ Σafter):

```bash
Benchmarks/benchmark.py                # all openvm-eth cases (--jobs = cores)
Benchmarks/benchmark.py openvm-eth     # same, named explicitly
Benchmarks/benchmark.py --n 20         # top 20 by cost rank
Benchmarks/benchmark.py --n 10 --report report.html   # + interactive HTML report
Benchmarks/benchmark.py keccak         # the single keccak stress case (takes minutes)
```

## Layout

Each benchmark set is a subdirectory of `Benchmarks/OpenVM/`; `benchmark.py <name>` runs it. Each
block is exported as a pair, `<rank>` being the cost rank (`001` = costliest) and `<pc>` the block's
start program counter(s) — or a descriptive label (the keccak set uses `pckeccak`):

- `apc_<rank>_pc<pc>.json.gz` — the powdr `SymbolicMachine` (`ApcWithBusMap` JSON) as it enters
  powdr's `optimize()`, with its bus map. apc-optimizer's **input**.
- `apc_<rank>_pc<pc>.powdr_opt.json.gz` — powdr's optimized result, for comparison.

`manifest.json` records the ranking (per-entry cost, frequency, width, stats, provenance);
`apc_candidates.json` is powdr's candidate dump, the source of each block's assembly in the
`--report` HTML. Both are optional.

## openvm-eth

The main benchmark set, in [`openvm-eth/`](./openvm-eth/): the 100 costliest basic blocks of the
OpenVM reth benchmark. Generated by
[powdr-labs/openvm-eth#11](https://github.com/powdr-labs/openvm-eth/pull/11). See [this APC analyzer dashboard](https://powdr-labs.github.io/powdr/autoprecompile-analyzer/?data=https%3A%2F%2Fgithub.com%2Fpowdr-labs%2Fapc-optimizer%2Fblob%2Fmain%2FBenchmarks%2FOpenVM%2Fopenvm-eth%2Fapc_candidates.json).
This set was built from Ethereum mainnet [block 24171377](https://etherscan.io/block/24171377)
(manifest `source` note: openvm-eth e3df69e block 24171377, cell PGO).

## keccak

A single-case stress set, in [`keccak/`](./keccak/): one large APC — the keccak precompile
(~28k constraints / ~27k columns), with its powdr pair from powdr's native `optimize()`
(`autoprecompiles/tests/optimizer.rs::test_optimize`, powdr commit `b072302`). One run takes minutes.
`benchmark.py keccak` runs it, and CI benches it on every PR alongside `openvm-eth`.