# Performance

What FlashProve costs to run, as measured on one kind of machine. [docs/figures.tsv](../docs/figures.tsv) has most
of these figures, and others, each with its conditions; [Known limits](known-limits.md) covers what is not about speed or size. What has been measured at all, and what has not: row RES-6 of [docs/limits-all-rows.tsv](../docs/limits-all-rows.tsv).

**Every figure, unless a line says otherwise,** is from [the timed runs](../docs/limits.md#the-timed-runs): Llama 4 Maverick in a third party's public 4-bit conversion, one H100 80 GB (driver 580.178.04) and 24 of its host's 96 CPUs, [masking](weight-hiding.md) on, 1,024-token prompts, up to six requests in each pass over the weights, each proof stating the next token, n = 3 runs (at 24 per batch each run has two services, also called sessions, that is two calls of `cp_prove` of five batches each; at 64, at 8 and at 1 it has one each: row RES-6), mean ± sample standard deviation. Ranges are least to most; memory is the largest seen in those runs; 1 MB is 10^6 bytes, 1 GB 10^9 and 1 MiB 2^20; the "80 GB" in the GPU's name is its maker's figure. "Masking on" is the mode that the runs' own summaries state; the runs measure speed and are no test of weight hiding ([Weight hiding](weight-hiding.md)). Each service ran the parameter file of its own commitment, not the repository's: 2 of its 152 lines on the model differ. The only evidence that its first 32 lines are the default setting's is the report of the run that made the commitment; no checksum of the parameter file is given (row RES-6 of [docs/limits-all-rows.tsv](../docs/limits-all-rows.tsv)).

| Requests per batch | Tokens/second | Seconds per request | From a batch's start to its proofs, after the first batch |
|---|---|---|---|
| 1 | 17.41 ± 0.15 | 58.796 ± 0.50 | - |
| 8 | 76.34 ± 0.37 | 13.413 ± 0.065 | - |
| 24 | 111.23 ± 0.92 | 9.206 ± 0.077 | 220.7 to 225.9 s |
| 64 | 123.31 ± 0.42 | - | 533.3 to 537.9 s |

These are steady prefill rates: each request's 1,024 prompt tokens count, nothing is generated, and start-up and committing are not counted. At more than one request a batch a request waits for its whole batch: about the batch size times its seconds per request. The table's waits run from a batch's start to the release of its responses by `cp_prove`, which was called directly and runs no verifier. The serving scripts run `cp_verify` on each response before they hand it on ([The example front end](../docs/service.md#the-example-front-end)); that time comes on top and is not in these figures. The check that was timed is the one by the vLLM integration's middleware. By itself, at the middleware's default on 24 CPUs, six checks at once, with one thread each, the check of a batch of 24 proofs took 94.0 seconds, and of one proof alone 22.6 seconds, from the hand-over of the proofs to their release (one separate run each, on the released programs, none of the timed runs: synthetic weights, 24 CPUs of a 32-CPU machine, no prover running beside it). Beside a serving `cp_prove`, on the 24 CPUs it was bound to, the check of a batch took about 89 seconds (one run, none of the timed runs: another machine, with 96 CPUs; synthetic weights; stored proofs, not that service's own; the middleware called directly, with no inference engine and no HTTP server around it). Alone on that machine it took 92.1 and 90.3 seconds; what else ran on the machine was not held the same. In that run the steady rate was 109.64 tokens/second, against 110.11 in its second service, with no check beside it: the check ran beside the first service, which is the slower of the two in the timed runs on synthetic weights as well, so no slowing can be seen from one run. The container's memory, file cache not counted, reached 298.4 GB with the checks beside it and 278.2 GB without (synthetic weights, which peak higher than the public files' 269.7 GB under Machine). Before a batch's start a request waits in the queue: up to about one batch's proving time if the next batch takes it, longer if a full batch or more is queued before it (reasoned from how the service works: [how a request becomes a response](../docs/service.md#how-a-request-becomes-a-response)). On synthetic weights of the same shapes the rate at 24 per batch is 109.09 ± 0.22.

**Whole replies.** With every token of a stored reply checked, under the looser [clipped-gap policy](terms.md), the rate at 24 per batch is 108.67 ± 2.8 tokens/second, or 39.1 ± 1.0 counting reply tokens alone. Each request is 524 prompt tokens, a stored reply of 31 to 500 tokens and fill, and the first rate counts all 1,024 positions. No engine ran, and the replies were chosen because scoring expected them to meet the limits. The same requests, with no reply token checked, ran at 108.73 ± 2.7. Both ran under the same parameter file: the pair bears on the checking of the reply tokens, not on what the policy itself costs. Not on the released programs all three rates were lower in the mean (row RES-12 of [docs/limits-all-rows.tsv](../docs/limits-all-rows.tsv)).

**Start-up and committing.** The project set itself a target of 100 tokens/second on one GPU. Steady service at 24 per batch passes it. With a start-up of 68.1 to 79.0 seconds counted, a five-batch session passes it too, in every run, at 103.60 ± 0.84 (not on the released programs, this rate was under 100 in every run reported, in most of them with a `cp_prove` of other sources that ran the verifier on every response: row RES-33 of [docs/limits-all-rows.tsv](../docs/limits-all-rows.tsv)). Both are counted up to the moment at which `cp_prove` hands out its proofs, and without the serving scripts' check, which by itself, in a separate run (none of the timed runs, on 24 CPUs of a 32-CPU machine, with no prover beside it), took 94.0 seconds for a batch of 24. With committing spread over the first 1,000 requests the steady rate, calculated, is 88.81 ± 0.46, which misses. [Committing](how-it-works.md) takes about 39 minutes, once per commitment (not on the released programs some runs took longer: rows RES-46 and RES-9 of [docs/limits-all-rows.tsv](../docs/limits-all-rows.tsv)).

**Sizes.** A proof is 18.78 MB on average. A client downloads 6.24 MB once per commitment, which counts the key file and the leaf file only; the three configuration files, which a client must hold too, are not counted. The provider keeps 2.15 GB per commitment. Both are for one [leaf table](terms.md) (`-M 1`), as are the time to commit and the rates in the table above; no rate was measured with more than one (row RES-6).

**Checking a proof** needs no GPU. The C++ verifier takes about 21 seconds on one thread and up to 3.53 GB of memory in its largest process (24 threads), on the prover's machine (not on the released programs some calls on one thread took longer: rows RES-46 and RES-19 of [docs/limits-all-rows.tsv](../docs/limits-all-rows.tsv)). On a shared virtual machine a stored full-size proof (synthetic weights; 1,001 tokens checked) took 36.2 to 44.0 seconds† in 8 calls, and up to 3,400 MiB† (about 3.6 GB), a sampled sum over the call's processes. The Lean verifier took about 24 minutes† (1,393 to 1,440 seconds) and up to 14.13 GB† of memory in a call's largest process on a shared CPU machine (three calls, one at a time, on one stored proof on synthetic weights, which checks 1,001 tokens, library built beforehand). In 21 calls to an accept and six to a late reject, each on a full-size proof, it took 19 to 39 minutes†, and in the 21 its largest process 9 to 15 GB† of memory, counting the compiled files that it maps. The longest was a first call, which also compiles the helper library, and not all were at exact match ([tools/lean-verify](../tools/lean-verify/README.md#what-it-costs)).

**Machine.** On the public 4-bit files, serving at 24 per batch reaches 269.7 GB of host memory in its container (278.3 GB on synthetic weights in the timed runs, and also at 64 per batch on the public files; not on the released programs it reached more: row RES-24 of [docs/limits-all-rows.tsv](../docs/limits-all-rows.tsv)) and committing 299.6 GB (309.6 GB† in one run of examples/maverick/run.sh), file cache not counted. With the serving scripts' check running beside a serving `cp_prove` the container reached 298.4 GB, against 278.2 GB without (one run, on synthetic weights, none of the timed runs). examples/maverick/run.sh will not start committing below 330 GB of available host memory (never tried at that figure), and needs 245 GB of disk. What several provers on one machine cost each other was not measured.

**Cost.** By calculation, per GPU, proving at 24 per batch is 93.0 times slower (92.2 to 94.1 over all pairs of runs) than stock vLLM's prefill at the highest load tried, and 71.4 times slower (65.2 to 76.4) than its prefill of 24 requests at once. At the highest load tried, to prove every prompt token that one GPU of such an engine takes in, 93.0 times as many GPUs would have to prove. One pass is 20.1 times slower (19.7 to 20.5) than decoding as many tokens at 512 streams; a pass checks one token, so this is not the cost of proving a generated reply. All three are calculated from the steady rate at 24 per batch, a best case: start-up and committing are not counted. Stock vLLM ran the model's FP8 release on eight H100s, in separate runs on other days, which are none of the timed runs. Its rates are not the most that those machines can do, and a baseline that is too low flatters the prover. One of its runs at a further load, with as long a queue of prompts as some of the eight runs that are counted at the highest load, with the same arguments for the engine, gave a higher rate than all eight. By a rule fixed beforehand it enters no range; with it the upper end, 94.1, would read 94.5. "Per GPU" counts only GPUs: the prover's one GPU comes with 24 CPU cores and the host memory given under Machine. In full: row RES-13 of [docs/limits-all-rows.tsv](../docs/limits-all-rows.tsv).

**64 per batch is not to be relied on with an 80 GB card.** On the public weights all three services at 64 ended normally (the table's 123.31; 960 of 960 proofs accepted), but the GPU reached 80,629 of its 81,559 MiB. The start-ups of those three services took 76.5 to 78.9 seconds (68.1 to 79.0 at 24 per batch). Not on the released programs, at 64 on the public weights, the rate was lower in the mean, and lowest in one single session, and the GPU's memory in use reached more (rows RES-10, RES-11 and RES-34 of [docs/limits-all-rows.tsv](../docs/limits-all-rows.tsv)). On synthetic weights of the same shapes every known attempt at 64 failed at a CUDA call with no proof: eight attempts, six of them†. The cause is not known, and the two kinds of run differ in more than their weights.

**How far the figures repeat.** Each speed's three runs ran on three machines at the same hours of one day. Three runs of the same kind, elsewhere and on another day, gave 105.3 ± 3.9† at 24 per batch, and two of their sessions, back to back on one machine, were 6.3 %† apart. Whole runs on different machines differed by up to 9 %†. No cause is shown. No time, rate or memory peak carries over from a run marked †. Hosts were shared: each machine is one GPU of an eight-GPU host (reported). In the three runs of 105.3 ± 3.9†, the 24 CPUs to which each run was bound were not reserved to it; in the timed runs, whether they were is not shown. No session reported here ran past six batches, so nothing here shows how rate or memory behave over hours.

**The model.** What is proved is an integer version of the model. Its top token differs from the 16-bit model's at 6.81 %† of positions. A Python implementation gave this, not the released programs; there is no task benchmark.

**Proving, not serving.** The prover proves a pass over tokens it is given, separately from serving. At exact match a reply passes only if every token is the integer model's own choice: of 96 stored replies each from stock vLLM at 16 bits and at FP8, none did (scored† by a Python implementation of the integer model, not proved by the released programs). Through vLLM with a plug-in that is designed to compute the integer model exactly, 48 of 48† short replies were proved and checked (the same 16 synthetic prompts in each of three runs; vLLM was stopped while each 16 were proved as one batch), serving at 0.58 to 1.46† seconds per token over all 48 replies.

The runs' records are not in this repository, so a reader cannot check the figures on this page against them.

## Reproduce the numbers

For the leading figures on this page and in README.md: what made each, and what this repository lets you run. No row lets you repeat the runs themselves: at best you get a figure of the same kind, on your own machine and prompts. A "yes" says what a script or a page is written to give, not that the route has been run on the released programs.

| Figure | What made it | What you can run | Setting | Hardware | Programs | A figure of the same kind? |
|---|---|---|---|---|---|---|
| At 24 per batch: 111.23 ± 0.92 tokens/second steady, 103.60 ± 0.84 with the start-up counted; the wait, 220.7 to 225.9 s | the timed runs, by measuring scripts that are not in this repository | `driver/measure_rate.py`: the three lines of step 4 | public 4-bit files; 24 per batch, six per pass, five batches; the parameter file of the service's own commitment | one H100 80 GB (driver 580.178.04), 24 of its host's 96 CPUs | the released programs | yes, on other prompts and with no CPU binding; for the wait, leave out the first batch, which the printed range includes (`figures.json` has each batch's times) |
| At 1, 8 and 64 per batch: 17.41 ± 0.15, 76.34 ± 0.37 and 123.31 ± 0.42 tokens/second; the wait at 64, 533.3 to 537.9 s | as in the first row | the third line with `--batch`, and at 1 and 8 with `--batches 6` | as in the first row, but six batches at 1 and 8, and one per pass at 1 | as in the first row | the released programs | as in the first row; 64 per batch is not to be relied on with an 80 GB card |
| Committing: about 39 minutes; 88.81 ± 0.46 tokens/second with it spread over the first 1,000 requests (calculated) | the timed runs | the second line commits; no printed line takes its time | public 4-bit files; one leaf table (`-M 1`) | as in the first row | the released programs | only if you time the second line yourself |
| A proof: 18.78 MB on average | the timed runs | the third line prints a proof's size | as in the first row | as in the first row | the released programs | yes |
| 6.24 MB for a client, 2.15 GB for the provider | the timed runs | the second line makes the files; no printed line sizes them | public 4-bit files; one leaf table (`-M 1`) | - | the released programs | not by printed lines |
| `cp_verify`: about 21 seconds on one thread | timed calls in the timed runs | [examples/full-size](../examples/full-size/README.md#try-it): its call, which runs its longest stage on one thread, with no GPU | the runs: proofs on the public 4-bit files, each of the next token; the example: a stored proof on synthetic weights, of 1,001 tokens | the prover's machine | the released programs | only if you time the call yourself; on a shared virtual machine this call took 36.2 to 44.0 seconds† |
| Memory at most: 3.53 GB (`cp_verify`'s largest process, 24 threads), 269.7 GB serving (278.3 GB on synthetic weights, and at 64 per batch on the public files), 299.6 GB committing | the timed runs | nothing: the third line prints `cp_prove`'s largest resident set, not a container's peak | as in the first row | as in the first row; `cp_verify`'s: the prover's machine, 24 of its 96 CPUs | the released programs | not by printed lines |
| The check of a batch of 24 by the vLLM integration's middleware: 94.0 seconds by itself (22.6 seconds for one proof); about 89 seconds beside a serving `cp_prove` | by itself: one separate run each; beside the prover: five checks in one run | nothing: no line is printed for it | synthetic weights, stored proofs; six checks at once, one thread each (the middleware's default on 24 CPUs) | by itself: 24 CPUs of a 32-CPU machine; beside the prover: one H100 80 GB, 24 CPUs | the released programs; none of the timed runs | no |
| Cost: 93.0, 71.4 and 20.1 times slower. Whole replies: 108.67 ± 2.8 and 39.1 ± 1.0 tokens/second; with no reply token checked, 108.73 ± 2.7 | cost: calculated from the steady rate at 24 per batch and stock vLLM's rates; whole replies: the timed runs | nothing: no baseline script, stored replies or fitted limits are in this repository | stock vLLM: the model's FP8 release; whole replies: public 4-bit files, clipped-gap policy, 24 per batch | stock vLLM: eight H100s; the prover: as in the first row | the released programs; stock vLLM involves no prover | no |
| The Lean verifier: about 24 minutes†, up to 14.13 GB† | three calls on one stored proof | [tools/lean-verify](../tools/lean-verify/README.md#what-to-type) | synthetic weights; a proof of 1,001 tokens; library built beforehand | a shared CPU machine, 64 CPUs | not the released programs† | only if you time the call yourself |
| Routes 1 and 3, and route 3 at full size: their times, and route 1's memory | three other runs† | [the routes' own commands](getting-started.md#the-four-routes) | synthetic weights | one H100 80 GB, 24 CPUs | not the released programs† | only if you time the commands yourself |

**Your build and the programs behind the figures.** `cp_verify` prints a build word (`build=`), which you can compute from these sources; it shows neither the compiler nor the flags, and does not cover the provers' other sources ([what a reader can check](../docs/limits.md#what-a-reader-can-check)). The provers' build records and the runs' records are not in this repository, and no page prints the checksums of the programs that made the figures, so you have nothing to compare the SHA-256 sums that `driver/measure_rate.py` prints with (row RES-18 of [docs/limits-all-rows.tsv](../docs/limits-all-rows.tsv)).

Without a GPU you can check a stored full-size proof: [examples/full-size](../examples/full-size/README.md).

**The steady rate** takes one H100, 330 GB of host memory, some hours, and 245 GB of disk for route 4, with 10 GB
and 5 GB more for the lines below. The rates above were made by measuring scripts that are not in this repository.
`driver/measure_rate.py` measures by the same rule. It has not been run on the released programs. It has run on a
GPU†, and none of these runs is shown to have used this repository's file; the public test `driver.measure_rate`
runs it on a CPU, with a stand-in for the programs. Inside `tmux`, at the top of the repository:

1. [Install the pinned compiler](../prover/docs/build.md#install-the-compiler).
2. Run [the two install commands](../examples/maverick/README.md#you-need) and the first two lines of
   [route 4](../docs/quickstart.md#4-llama-4-maverick-the-full-size-model), then the line under its block,
   which makes `calibration_set.npy`. Two of the files that it reads open only once Meta has granted access.
3. Call [the example's stages](../examples/maverick/README.md#the-stages) one by one up to `fit`.
4. Run these three lines.

```
O="${M:?first set M and F as in route 4}" && O="${O%/*}/own-seed" && mkdir -m 0700 "$O" && echo "O=$O" || unset O
prover/build/hc/cp_commit -weights "${F:?}" -config "${M:?}/fit.1/config" -M 1 -ivk "${O:?}/ivk" -isk "${O:?}/isk" > "${O:?}/cp_commit.log" 2>&1; echo "exit code $?; lines that begin cp_commit: done.: $(grep -c '^cp_commit: done\.' "${O:?}/cp_commit.log")"
python3 driver/measure_rate.py --programs prover/build/hc --isk "${O:?}/isk" --ivk "${O:?}/ivk" --config "${M:?}/fit.1/config" --id "$(sed -n 's/^commitment ID: //p' "${O:?}/cp_commit.log")" --text "${F:?}/calibration_set.npy" --least-memory 300 --work "${O:?}/measure"
```

`prover/build/hc` is where the build puts the three programs. The second line commits. Unlike `examples/maverick/run.sh commit` it checks neither the host memory, of which
330 GB must be available, nor the 10 GB of free disk that it needs; the session needs 5 GB more on a local disk.
Go on only if it prints `exit code 0; lines that begin cp_commit: done.: 1`, and do not type it twice: that
would empty the log from which the third line takes the commitment ID. An ID taken so is fit only for a
commitment that you have just made yourself. If your fit is not in `fit.1`, name the newest finished one. The
third line starts nothing unless 300 GB of host memory are available (in the timed runs' six services at 24 per
batch `cp_prove` ran for 1,208.4 to 1,236.4 seconds from its start to its end, some 34 of them after its last
proofs: row RES-33), and prints
`STEADY RATE: …` only if `cp_verify` accepts every proof. To stop it, press Ctrl-C once and wait; never kill
`cp_prove`.

For 1, 8 or 64 requests per batch, add `--batch 1`, `--batch 8` or `--batch 64` to the third line, and name another new directory after `--work`. The timed runs had five batches at 24 and 64, which is the script's default, and six at 8 and 1 (`--batches 6`, which uses up six batches of the leaf table; row RES-6). At 64, read the paragraph "64 per batch is not to be relied on with an 80 GB card"; the container's host memory also reached more there than at 24 (278.3 GB in the timed runs: row RES-24).

These three lines do not send the prompts of the runs above: those 96 windows of text are not in the repository. They send the 32 windows of the calibration set, on which the fit's scales were set, and from the 33rd request on the same windows again, with their tokens after the first rotated. In the runs above every program was bound to 24 CPUs on the GPU's NUMA node, while the script binds nothing. So you repeat the kind of measurement, not the runs (rows RES-18 and T49 of [docs/limits-all-rows.tsv](../docs/limits-all-rows.tsv)).

Keep `cp_commit` and `cp_prove` on CPUs of the GPU's NUMA node. By a reading of the code each pins its thread, and the threads that it starts after that, but only to CPUs that it may already use, and then prints `# GPU … is on NUMA node …; this thread is pinned to … of its CPUs`; if that line is missing it has pinned nothing: the usual causes are that the CPUs that it may use hold none of that node's, or that the system does not report the GPU's node, and a failed system call ends it the same way, with no message. In an untimed run† bound to the other node, `cp_commit` took about 38 % longer than in another run† of its kind, and `cp_prove` about 2 % longer on one request. On the released programs, in an untimed run of that kind bound to the GPU's own node, `cp_commit` took 2,328.2 seconds, inside the timed runs' range for the commitment on the public files (2,307.45 to 2,343.76 seconds); programs and machine differ too, so whether the binding is the cause has not been tried (row RES-9).

Each run uses up, for good, the batches that it proves of a leaf table (five by default, six with
`--batches 6`): measure on a commitment made for the purpose, not on weights that must stay secret. The cost
ratios and the rates with whole replies checked cannot be reproduced from this repository: no baseline script,
stored replies or fitted limits are in it.

† Not on the released programs, and, where the Lean verifier ran, not over the released Lean files: [which kinds of result these are](../docs/limits.md#on-other-programs).