# Limits of the vLLM integration and the client scripts

This page lists what the scripts in `integrations/` do not do. Five limits come first, each with what to do about it, its worst case and its basis. The basis says how we know: by reading the code, by reasoning, or from a run. A reading can miss. [Smaller limits](#smaller-limits) follow. The limits of the proof system and its programs are in [docs/limits.md](../docs/limits.md).

What vLLM itself serves is not masked, and this example middleware leaves undone much that a front end must do ([below](#what-the-integration-reveals)): of the provider's duties it does not meet duty 11 ([the conditions](../docs/limits.md#the-conditions)), so a server behind it alone is outside [our judgement on weight hiding](../docs/limits.md#the-verdict).

<a id="what-a-pass-does-not-show-and-what-a-client-must-bring"></a>
## What a pass does not show, and what a client must bring

`check_proof.py` cannot tell where its inputs come from: [a check is worth only as much as the values that you bring to it](../docs/limits.md#what-you-bring). A proof covers token IDs, not text. By itself it never shows that a reply is complete: the script checks the reply's end only against the end-token IDs that you pass.

The check is by the C++ verifier, `cp_verify`. Lean's soundness theorem on `verify` is about the Lean verifier, not `cp_verify`, and holds at exact match only, in the random-oracle model, for public files in scope (one proof format, at full size) fixed before the hash function is drawn; for the clipped-gap policy a second theorem says only that an accepted output meets the policy's limits, not that it is the model's ([lean/THEOREM.md](../lean/THEOREM.md); [guide/what-is-proved.md](../guide/what-is-proved.md)). No theorem is about `cp_commit`, `cp_prove` or `cp_verify`; none covers the Python scripts either. So a fault in `cp_verify` or in these scripts could turn a rejection into a pass.

**What to do,** as a client:

- Bring the five things that [README.md](README.md#a-reply-of-your-own) lists, and none of them from the server.
- Decode the reply's token IDs yourself: nothing compares the reply's text with the checked tokens.
- We do not recommend any particular limits for clipped gap: the demo's were fitted to its synthetic model only.
- Run the check on a local disk. Only you or root should be able to write to the checkout, your home directory, `TMPDIR`, `--out-dir`, or any directory above them. The default `/tmp`, which is sticky, will do for `TMPDIR`. Choose such a place; do not change the owner or mode of a directory that others use. The client itself looks above `--out-dir` ([below](#the-look-above)); nothing looks above the checkout, your home directory or `TMPDIR`.
- In a program, apply all the conditions of [README.md, "Exit codes"](README.md#exit-codes): neither the exit code alone nor a `result:` line is a pass.

**Worst case,** for a client:

- Where someone else can write what the check reads, `check_proof.py` can print a wrong `verified` with no sign of it.
- With a text prompt or a wrong list of end tokens, a verified reply can answer another prompt or be cut short. With a text prompt the server can also count most of its reply as prompt, so that the proof checks only the last token; the example client then exits with code 24, never 0.
- Under clipped gap, another model's replies can pass. In one run all 64 replies of a slightly altered model met the limits, and so did 1 in 13 replies with one token replaced at random. That run was on a CPU, with a small synthetic model, not on the released programs; its limits were round numbers chosen above the largest values of that model's honest replies. Counts under fitted limits: [Score a reply](vllm/integer_model/OPERATING.md#score-a-reply).

**Basis:** read (`check_proof.py`); reasoned. The script's guard for a shared `TMPDIR` has not been tried.

<a id="what-a-dishonest-or-faulty-provider-or-server-can-do-to-a-client"></a>
## With vLLM the reply arrives before its proof, and the server's word is no proof

vLLM sends each reply before a proof exists, and nothing takes it back. The job state `proved`, status 200 and a proof ID are claims: a dishonest server can send them without a valid proof. Anyone who learns a proof ID, and has the shared key where vLLM was given one (how the middleware learns that key, and where it cannot: [the operator's checklist](vllm/REFERENCE.md#the-operators-checklist)), can fetch or delete that proof and test guesses at its prompt and reply. Nothing but the later proof ties what vLLM serves to the commitment.

**What to do,** as a client:

- Treat a reply as unproved until your own check has passed.
- Fetch the proof as soon as the job reads `proved`: the server can still lose it.
- Keep the proof ID as private as the prompt. Send `DELETE` once you have checked the proof; the example client does so after a pass.
- If you count rejections to judge a provider, count `check_proof.py`'s exit code 2 and `outcome=no-proof` too. A provider can turn what would be a rejection into either, and neither is a verdict on the proof.

**Worst case,** for a client: a reply is used and no proof ever comes; or someone who learned the proof ID confirms a guess at the prompt.

**Basis:** read (`proof_middleware.py`; `ask_and_check.py`). In [the run at toy size](#what-has-never-been-run) (not on the released programs), at a start with `--api-key` alone, calls without a key that named a job's proof ID got status 401 (for its state, its `proof.bin` and `y.bin`, and a `DELETE`, which deleted nothing), so did one with a wrong key, and calls with the key were answered. Where the middleware cannot read vLLM's key: [the operator's checklist](vllm/REFERENCE.md#the-operators-checklist) and row HW-275 of [the table of all rows](../docs/limits-all-rows.tsv). The rest is from reading alone, which says nothing about a running server.

<a id="what-the-integration-reveals"></a>
## One account holds the weights and the secret key directory, and vLLM answers far more than the proof routes

vLLM and the proving service run under one account. It reads the model's files, and it owns the secret key directory and the spool directory, through which the middleware and the proving service exchange files.

The middleware filters nothing that vLLM serves: at most it withholds the proof.

- **Scores.** Under ordinary serving vLLM runs the floating-point model, not the committed integer model, and clients get that model's scores (such as `logprobs`). Published model-stealing attacks use such scores.
- **Steered replies.** A request with a field that steers the reply, such as `logit_bias`, is answered; only the proof is withheld. That is the middleware's rule, not the programs': a front end without it, or whoever can write into the spool directory, can submit any tokens. Published work uses a field like `logit_bias` to measure a model's scores relative to one another even where a reply shows only the top token (Carlini et al., arXiv:2403.06634; Morris et al., arXiv:2311.13647), and the first recovers a model's last layer, up to a linear map, from many such score vectors. What these works mean for a provider that serves replies with proofs has not been worked out.
- **What is checked.** The middleware asks for proofs only of replies that vLLM served, and always over the whole reply. But a client moves both ends of that range. `max_tokens` ends a reply early. And a client can send a reply's first tokens again as part of a new prompt, so that only the later tokens are checked. The prompt itself steers the reply too: asked to repeat a given text, a model often will. How far that lets a client choose the checked tokens has not been measured. So the client, not the middleware, decides what is checked. Under the clipped-gap policy a client that chooses the checked tokens can, over many requests, measure the model's hidden scores ([which policy](../docs/limits.md#which-policy)).
- **How often.** The middleware sets no limit per client and refuses no repeats. It limits only in part how many responses can wait unfetched: for one thing, its limits count for a policy, not for a client ([below](#smaller-limits)).
- **Who and when.** It does not authenticate each client, and it does not hide timing or queue length ([what is not hidden: time](../docs/what-is-not-hidden.md#time)). A full queue (`max_waiting`) and a full store (`max_stored`) show to every client that asks for a proof meanwhile, whoever filled them: its job says `not_provable`, and which of the two it was; once a policy is on hold, every client gets the same error; and the order of a job's states before that error may tell one cause of a stop from another (inferred from the code, not tried).

Why not: the example has no notion of clients, only of proof IDs. A proof ID is a random value of 256 bits that only the asking client is sent, and it works like a secret link: whoever holds it can fetch or delete that one proof. Stock vLLM has one shared API key and no notion of who a client is, so limits or access for each client need an authenticating front end of your own before it.

A front end is meant to give each response only to its owner, to limit each client, and to set what is checked. It must show nobody a directory listing, a file time, an ID or a tag (the keyed checksum on each response): these show whether another client's request failed, which can depend on the weights ([what a front end must do](../docs/what-is-not-hidden.md#front-end-rules)). This middleware breaks all five of that page's rules, at least in part. So two things that the release says assume such a front end, and are not claimed for a server behind this one: that a batch's other requests stay private, and that the leak under clipped gap is narrowed ([what is not hidden](../docs/what-is-not-hidden.md); [the conditions, duty 11](../docs/limits.md#the-conditions)).

**What to do,** as a provider: work through [the operator's checklist](vllm/REFERENCE.md#the-operators-checklist). Its main points:

- Let clients reach vLLM only through a proxy of yours that authenticates them. It lets through only the routes that they need (`/v1/completions`, `/v1/chat/completions` and those under `/plugins/proof/`), only [the request fields that can get a proof](vllm/REFERENCE.md#which-requests-can-get-a-proof), and only the headers and query parameters that they need: the middleware passes every header and the query string on to vLLM, which reads some of them.
- After a break-in into that account, treat the secret key directory as stolen, and `cp_prove`'s counts of batches in it as in doubt. Do the same if another account could ever reach that directory, or if you cannot tell ([the conditions](../docs/limits.md#the-conditions)). Whoever can read it can unmask all that was ever published under it; whoever can write it can set the count back in silence. A new commitment stops further harm and takes nothing back.

**Worst case,** for a provider with secret weights: whoever breaks into vLLM can read and change the weights' files, the secret key directory and the spool directory, among them the seed, the count files and responses that no verifier has passed ([what the seed unmasks](../docs/limits.md#the-seed)). And without a proxy that filters request fields, any client can collect the served model's scores.

**Basis:** read (`proof_middleware.py`; [docs/service.md](../docs/service.md)); reasoned. No break-in and none of these attacks has been tried on this integration or on the released programs.

<a id="what-a-provider-must-do-because-no-program-does-it"></a>
<a id="after-a-fault-nothing-stops-by-itself"></a>

## What stops after a fault, and what does not

`cp_prove` runs no verifier on what it makes: what it has staged it puts into `out/`, and its exit code 0 does not show that a proof verifies. The middleware says nothing of a response, a failed one included, until `cp_verify` has passed every proof of its batch that belongs to a job, against the commitment ID in its config file, and no response of it looks malformed ([narrow exceptions, none open to a client](#narrow-exceptions)). Until then the job reads `waiting`. The check starts when a client asks for a job of the batch. Where `cp_verify` says NO on a proof, or a token file is not what the range asks, the policy goes on hold, the proof and the token file of every response of the batch (but for the narrow exceptions below) are overwritten with zeros, as far as the disk allows, each response with a job is replaced by a failed one, and an ERROR line says that a response did not pass, with no byte of the file; a client then gets 503 for as long as the hold stands or the wipe isn't done, and `no_proof` after that. Everything else wipes nothing: `NOT CHECKED`, the time limit, a file that is no proof of this format, a response that looks malformed, a missing leaf file and any other fault. Nothing of the batch that has no record leaves, the log names the cause, a later call tries again, no sooner than a minute on, and the same cause at two tries in a row, or ten tries of any causes, puts the policy on hold; after `lift-hold` the batch is checked anew. The middleware does not see the proving service's exit code. Beyond that check it stops only where somebody has put a file named `hold` into the spool directory. Each policy that a server offers has a spool directory of its own. While the file is there, replies under that policy get no proof, none of its stored responses is served, earlier ones included, and `clean` deletes nothing. vLLM goes on serving replies. The [start command](vllm/REFERENCE.md#start-the-proving-service) creates the file once `cp_prove` has exited with a code other than 0, and only a person lifts the hold.

**What that check is worth.** It catches only what a verifier rejects: a proof whose mask was lost can still pass ([a lost mask](../docs/limits.md#a-lost-mask)). It runs under the account and on the machine of vLLM and `cp_prove`, with the `cp_verify` that its config file names. The [operators' duties](../docs/operators-duties.md#the-front-end) ask for one that the service's account cannot write, and no program checks that: it guards against a bug or a hardware fault, not against whoever has that account. The verifier, the records of a pass and the hold can all be changed by the account that serves. `cp_prove` learns nothing of a rejection, and goes on with what is queued. Treat no verdict as a no: some of the checks of `cp_verify` that end this way protect hiding. Until such a check ends, the memory of the process that makes it holds the request, the reply and a proof that has not yet passed, and so does the place from which it reads them.

Where the hold falls short:

- It stops one policy only. The server's other policies keep serving their proofs, though a fault of the machine may concern them too.
- `cp_prove` neither writes nor reads the file. If you start `cp_prove` in any other way, as with the plain command in the operators' duties, nothing stops: the middleware keeps serving replies, proof IDs and earlier proofs.
- The file comes only when `cp_prove` has exited. Before that it finishes the batch and wipes its memory, and a wipe on a GPU that no longer answers is given up only after 600 seconds. Until `cp_prove` has exited there is no file, and earlier proofs go out meanwhile. No file is made by a fault that ends nothing (a service on a faulty GPU can fail request after request while its log looks healthy), or by a start command that is itself killed, also by Ctrl-C in its terminal.
- A stop by the file `stop` normally ends with exit code 0, and then makes no file, also where you ordered it because you doubt the service.
- Making the file can fail for the very cause that ended `cp_prove`, such as a disk that cannot be written.
- What has happened or begun goes on: proofs that clients fetched before the bad exit are out, and every call that has begun runs on, such as a download or a `DELETE`. One more request can slip in while the file appears, and a `clean` that is running can still delete the one file that it is at; what it deleted before stays deleted.
- Waiting requests stay in the spool directory, and a `cp_prove` that is started again proves them.

For a proof that `cp_verify` passes, the hold is one barrier, not two. While vLLM runs, such a proof of a faulty process is held back only by the middleware's look for that file, and the account that serves can remove the file. That check has been tried inside a running vLLM in [the run at toy size](#what-has-never-been-run) (not on the released programs), with a policy that was put on hold on purpose while two honest proofs were stored: for one of them the job's state, downloads of `proof.bin` and `y.bin`, and a `DELETE` were refused, and a new request got no proof. It has never been tried against a faulty process. Stopping vLLM is the surer stop. If you stop it, lay the hold before it starts again: a server that is started anew knows nothing of a rejection that the old one could not write down ([the middleware's check](#the-middlewares-check)).

**What to do,** as a provider (the commands, and what counts as a bad exit: [REFERENCE.md](vllm/REFERENCE.md#start-the-proving-service)):

- Start `cp_prove` only with the start command, and never let a service manager start or restart it by itself, and above all never after a bad exit, which is anything but exit code 0 ([the operators' duties](../docs/operators-duties.md#keys-seeds-and-machines); [the list of exit codes](../prover/docs/ends-in-short.md)). A list of exit codes after which not to restart is not enough: exit code 1 also covers faults of the machine, and a person has to judge the cause before any restart on the same commitment.
- When the service ends badly, or hangs at its end, put every policy of that machine on hold, not only this one (`proof_middleware.py hold CONFIG POLICY` for each policy that `check` lists), or stop vLLM, and lay the hold before it starts again. Do the same before you stop the service yourself because of a doubt: a check of your own that ends in a rejection, requests that keep failing, a count in doubt. Under a hold, or without the middleware, vLLM may go on serving replies, without proofs. Only a person starts `cp_prove` again, lifts a hold, or serves stored proofs again.
- After a rejection, before you lift a hold, stop or restart the server, or run `clean`: read [the middleware's check](#the-middlewares-check).
- Lifting a hold (`proof_middleware.py lift-hold CONFIG POLICY`) puts every stored response but those of a withheld batch whose wipe is done back on offer, the faulty process's among them. Among proofs that verify, no check tells these apart: a hold pauses the harm and cures nothing. Lift it, and start again on the same commitment, only if the cause is known, gone and harmless, such as a mistyped path or a full disk ([Before you serve again](../docs/faults.md#before-you-serve-again)). Otherwise keep it, take the policy out of the middleware's config file, and make [a new commitment](../docs/operators-duties.md#a-new-commitment). That puts the policy's spool directory beyond `clean` for good: the clients' prompts and replies and the doubtful proofs in it stay, and no tool removes them. Keep it as secret as the secret key directory.
- Watch the proving service and the middleware's ERROR log, run `proof_middleware.py check` on a schedule, look in its output for a note without `.done.json` (where no hold file lies, `check` exits with 0 even then), and take requests that keep failing as a doubt about the machine.
- Hand out nothing from the spool directory's `work/` or `out/.tmp/`, and nothing from `out/` that `cp_verify` has not passed: `cp_prove` puts its responses there without a check.
- If another account could ever write in the jobs or spool directory, start with new ones, and move nothing over.

**Worst case,** for a provider with secret weights: the fault may have started before anyone noticed it. A proof whose mask was lost still verifies. So a rejected proof, a failed self-check or an unexplained exit can be the first sign of a fault that was already at work, and earlier proofs of the same process or machine may show hidden values unmasked. Clients have fetched some of them, and where no `hold` file was made the middleware goes on serving the rest. Nothing takes them back ([a lost mask](../docs/limits.md#a-lost-mask)). For clients: replies go on coming, and their proofs never do.

**Basis:** read (`proof_middleware.py`; [docs/service.md](../docs/service.md)); run, for a policy that was put on hold on purpose under a running vLLM, and for the hold that the demo makes, which is the demo script's own code and not the start command's line ([the run at toy size](#what-has-never-been-run)). For the middleware's own check: tests with a stand-in verifier, and two with the real `cp_verify` at toy size, in which a proof with one bit changed is not served (rows `integrations.unit` and `integrations.cp_verify` of [SUITE.md](../SUITE.md)), with no vLLM and no real `cp_prove`. No fault has been provoked in a running server. Through the middleware the real `cp_verify` has run in four tests (these two among them), each with one job, one response and one batch from a stored toy example, and in three timed runs on honest stored proofs on synthetic weights, laid by a measuring script with no vLLM and no HTTP in front, all accepted: two with one batch of 24 each, and one in which a batch of 24 was checked seven times, five of them beside a serving `cp_prove` (row RES-32). No false proof has been tried in a batch of several, or in a middle batch, with the real programs. A check that gives no verdict, or a verifier that answers late, has been tried on a stand-in alone. Nor has any test had several true server processes, a real disk fault, a real vLLM over the check, the check of a proving `cp_prove`'s own fresh proofs, the real `cp_prove`'s naming and removal under polls, a true SIGKILL of a true server, or the limit of 1,800 s at full size.

<a id="what-has-never-been-run"></a>
## What has run through this integration, and what has not

What follows are four kinds of scripted run. They show that the parts work together, and nothing more. None of them was made on the released programs, or with the scripts exactly as they are in this repository. Where a middleware took part, it had no check by `cp_verify`. Each run was driven by a further script, which is not in this repository, so the runs cannot be made again from it.

**At toy size, through vLLM: one scripted run.** It ran [the demo](vllm/README.md#quick-start), in both ways of serving, and some of an operator's commands and some calls of a client, with and without the API key, behind a running vLLM, on one machine with one H100, on one synthetic model, at the [default setting](../docs/glossary.md#default-setting), in its small form. The small form is for tests: a pass under it carries no security claim.

- Every demo sent the same 8 prompts: synthetic token IDs, no text. In the four demos on the [production build](../docs/glossary.md#production-build) that were meant to end well, each reply got a proof that passed its check; the three of them in ordinary serving served, and proved, the same 8 replies. Every pass was under a commitment with counted proof of work, and each check's last line says so. Each demo's eight were proved as one batch. A reply's end was checked only where the reply ended on the end token: for one of the eight in ordinary serving, for two in integer serving. (A fifth demo ran on a test build, and is not counted here.)
- In integer serving the integer model's loader warned that the model's files were not verified: the demo gives it no digests. At each of its eleven stops, vLLM's own shutdown killed its engine's process by force, and at nine of them (in other runs of the same script, up to all of them) it logged an ERROR with a traceback. The run does not show why.

**At full size, on real weights, through vLLM: three scripted runs (three of the seven calls of row T41 of [docs/limits-all-rows.tsv](../docs/limits-all-rows.tsv)).** Each ran the demo once, in integer serving under exact match, with the full-size settings, and with vLLM stopped while `cp_prove` ran. Each made its own commitment just before, to [a third party's public 4-bit conversion of Llama 4 Maverick's weights](../examples/maverick/README.md), by a call of `cp_commit` on its own, before the demo began, at the [default setting](../docs/glossary.md#default-setting), whose parameter file counts proof of work, under a seed that it drew (that the file was the default setting's is not shown by the runs' logs, which show only that every check passed with counted proof of work). In each run vLLM, with the proof middleware, served 16 greedy replies to the same synthetic prompts from the integer model, and each got a proof that passed its check. No reply's end was checked.

- Such a pass compares greedy choices, not scores, and does not show how closely the committed integer model follows the original.
- In each run the demo got more options than [its section](vllm/REFERENCE.md#the-demo-on-your-own-commitment) prints, among them: `--serve integer --policy exact --max-tokens 64 --vllm-gpu 0 --prover-gpu 0`, `--build-dir` and `--verifier` (the programs had been built elsewhere), longer time limits, and, behind `--`, four for vLLM: `--gpu-memory-utilization 0.9`, `--fingerprint-mode none`, `--max-logprobs 0`, and `--model-loader-extra-config` with the digests of five files, against which the integer model's loader checks what it loads ([Verify what was loaded](vllm/integer_model/OPERATING.md#verify-what-was-loaded)). We have not tried whether the demo starts at full size without these four.

**Stored replies, at full size, on real weights, without vLLM: one scripted run.** A script entered the three blocks of commands under [Prove replies that were served earlier](vllm/REFERENCE.md#prove-replies-that-were-served-earlier), and two pieces from [the integer-model package's page](vllm/integer_model/OPERATING.md): the block that makes the model directory, and the line for `fit_limits.py`. Greedy replies that stock vLLM had served earlier from Llama 4 Maverick's 16-bit checkpoint, and that had been stored, were queued by `spool.py`, proved by `cp_prove` and checked by `check_proof.py`, under clipped-gap limits that the package's `fit_limits.py` had fitted on 48 other such replies.

The commitment, to [a third party's public 4-bit conversion of Llama 4 Maverick's weights](../examples/maverick/README.md), was made in the run, by `cp_commit`, at the [default setting](../docs/glossary.md#default-setting) (by the run's own lines its parameter file had the default setting's lines on out-of-domain (OOD) answers and on proof of work; that the rest of its first 32 lines was the default setting's follows from the route and is on no line of the run), under a seed that it drew.

- All 48 held-out replies got a proof that `check_proof.py` accepted as within limits; of the same 48 replies, each set after another prompt, none got a proof; and of 24 greedy replies that stock vLLM had served from the 16-bit checkpoint of one smaller model of the same family, Llama 4 Scout, none got a proof. The second is a gross change: it is no test that a changed model or a changed token is caught. The third tried one smaller model, with 24 replies: it does not show that the limits tell models apart in general. A single count of 48 of 48 does not show how often an honest reply is declined, and the held-out 48 were unseen by this fit, but had been used in other trials: they are no fresh sample.

**Fresh replies, at full size, on real weights, without vLLM, under limits fixed beforehand: one scripted run.** A script entered the blocks under [Prove replies that were served earlier](vllm/REFERENCE.md#prove-replies-that-were-served-earlier) and the block of [the integer-model package's page](vllm/integer_model/OPERATING.md) that makes the model directory, with other requests than the run above (but for two of that run's, below) and with no fit. It took the three limits that were fitted for that run on 48 stored replies; they were fixed before any of this run's fresh replies was served. A reply meets them if its tokens' costs, each counted as at most `C`, average at most `theta` (the limit on the mean), and at most `cap` of them exceed `C` (the limit on the number of costly tokens): [Score a reply](vllm/integer_model/OPERATING.md#score-a-reply). vLLM was installed but not run; no middleware and no example client took part. The commitment, to the same [public 4-bit conversion](../examples/maverick/README.md), was made in the run, by `cp_commit`, at the [default setting](../docs/glossary.md#default-setting) (by the run's own lines its parameter file had the default setting's lines on OOD answers and on proof of work; that the rest of its first 32 lines was the default setting's follows from the route and is on no line of the run), under a seed that it drew. Each of the 178 requests put to `cp_prove` ended as the integer-model package's score had said beforehand: 152 got a proof that `check_proof.py` accepted as within limits, and 26 got none.

- *Fresh replies.* Beforehand, in a job of its own on eight H100s, stock vLLM 0.29.0 served 128 greedy replies from Llama 4 Maverick's 16-bit checkpoint, of up to 500 tokens (47,714 tokens in all), to prompts of 524 tokens from the same public text as the run above (WikiText-2's test text). A fixed program and seed drew the prompts before anything was served; by the program's own rule none shares 32 tokens in a row with a prompt on which limits were fitted, with one whose reply had been scored, or with another drawn prompt. What served the replies is by that job's own output. Another job of the same kind ended when its server failed to start, before any request was sent, so no reply was dropped. Of these replies, 127 got a proof that `check_proof.py` accepted (99.2 %; exact 95 % interval 95.7 to 99.98 %), and 1 got none; by the package's score, that reply misses the limits.
- That figure holds for these limits, not for the way they were fitted: they are the looser of two sets, each fitted on one half of the same 96 stored replies, and the difference rests on one reply. The prompts are new places in the same articles, not new text, and replies from one article may fare alike, so the true uncertainty is wider than the interval; by how much has not been reckoned. It holds for one model, greedy replies, plain text without a chat template, one version and setting of vLLM, one kind of GPU. It does not show that the limits are tight: by costs that the integer-model package had stored, counted under these limits alone, without proofs, nearly all stored replies that Maverick gave when served in FP8, all those served with top-p 0.9, most of those that it sampled at temperature 0.7 or 1, and a few greedy replies of Llama 4 Scout met them too.
- *How much that figure depends on the 48 fitted replies.* The same rule was fitted again on 200 other draws of 48 from the 232 stored honest replies that these limits' 48 came from: a median of 4 of the 128 fresh replies missed the limits so fitted (least 0, most 20; 2.9 % on average; under 178 of the 200 fits more than one missed), where 1 misses these limits; fitted on all 232, 0 missed. Under only 22 of the 200 fits did at most one fresh reply miss: these limits lie on the loose side of what the rule gives. Under the same 200 fits, a median of 1 of 96 greedy replies of Llama 4 Scout met the limits, which they should not (least 0, most 10), where these limits let 7 through; fitted on all 232, 7. Those 96 replies answer prompts that 96 of the 232 stored honest replies answer too. The draws overlap, come from one pool and are all tried on the same replies: this is not 200 new samples, and gives no interval. Scored by the integer-model package, without proofs.
- *Cut replies.* 48 requests were cut from stored replies, in 24 pairs one token apart: in each pair one cut meets the limits and the other misses exactly one of them (12 pairs at the limit on the mean, 12 at the limit on the number of costly tokens). Scored again in this run, each of the 48 cuts got, token for token, the costs that had been stored for it. 24 of the 24 cuts that meet the limits got a proof that was accepted, and 24 of the 24 cuts that miss got none. One token apart is not close in value: half of the pairs at the mean were taken for how close their missing cut lies to the limit, the other half for their meeting cut. Of the cuts that miss the limit on the mean, 6 lie under 0.1 % above it, 0 from 0.1 % to under 1 %, and 6 by 1 % or more. Of those that meet it, 6, 0 and 6 lie below it by the same three spans. At the count, the cut that meets lies exactly at the limit and the one that misses one above it, so a comparison off by one would have shown there. At the mean the pairs need not have shown it. The reply exactly at that limit, below, would have shown a comparison that is too strict, not one that is too loose. The cuts that got a proof are altered replies that do meet the limits: 14 of them are beginnings of replies that Maverick sampled and 5 of Scout's. The stored replies were taken by a fixed rule from ten sets: greedy replies of Maverick, replies that Maverick sampled, and replies of Llama 4 Scout. Of the 48 cuts, 10 are cut from greedy replies of Maverick (8 of them from replies served in FP8), 28 from replies that Maverick sampled, and 10 from Llama 4 Scout's; of the 12 pairs at the limit on the mean, 0 are cut from greedy replies of Maverick. That the cuts that miss got no proof is the honest prover's own decision, and why, the run does not show; that the other cut of the pair, one token away, got a proof is what speaks for the limit. No forged proof was tried in this run, so nothing here shows that the verifier rejects one. Cuts like these are the questions by which a caller can measure hidden scores ([what the integration reveals](#what-the-integration-reveals)).
- *Two requests of the run above* rode along; that run's entry does not tell of them. One is its fitted reply that, by the package's score, lies exactly at the limit on the mean: it got a proof. The other is the same reply with another last token: it got none. That ties this run's model and its limit on the mean to that fit. The other two limits rest on that run's stored file, which was checked against a checksum before the run, outside the script; nothing in the run itself checks them. No step of this run compared every loaded tensor with its listed checksum, as that run's fit did; the five weight files' checksums were checked.

**Not run in any of these runs:**

- the released programs;
- the operator's set-up as [REFERENCE.md, Part 2](vllm/REFERENCE.md#part-2-operators) prints it, including its start command;
- a client that fetches `key.bin` or `layout.txt` from the server (the demo's client fetched only the leaf file; in the run at toy size the script asked for `key.bin` by a plain call: without the key it got status 401; with the key the file came, and the script only counted its bytes);
- a proving service that stays up beside a serving vLLM (`cp_prove` was started when the requests were already waiting in the queue, and ended by itself);
- at full size, through the middleware: ordinary serving, clipped gap, or a full batch (the run of stored replies proved replies of ordinary serving, under clipped gap, in batches of up to 24 requests, full ones among them, without the middleware);
- at full size, `cp_prove` without `-memory`;
- a published seed;
- text, chat, streaming or sampling in a request served through the middleware (only `/v1/completions` with token IDs, greedy);
- more than one client, or a client on another machine;
- more than one GPU for the programs or the integer model (the fresh replies were served beforehand, by stock vLLM alone, on eight);
- a setting that counts no proof of work;
- two accounts;
- `sudo` (only this integration's commands were run under it, without vLLM or `cp_prove`: [a known fault](#one-account));
- behind vLLM: a start with several keys given with `--api-key`, or with a config file; a refused start; `layout.txt` without the key; an unknown proof ID without the key;
- an interrupt.

The tests use stand-ins for vLLM and the proving service, and tests on stand-ins cannot show what the real programs do together.

**What to do.** Treat every part of the integration as barely tried. Try it first with the demo, which uses synthetic weights.

**Worst case:** a fault that neither these runs nor the tests on stand-ins meet.

**Basis:** run, as described above.

<a id="smaller-limits"></a>
## Smaller limits

**What a pass is worth** (clients):

- Nothing links a renewed commitment ID to the old one: to a client, a renewal looks exactly like a switch to another model. What holds under one ID, what forces a new one, and what a client can do: [Verify a proof](../guide/verify-a-proof.md#when-the-commitment-id-changes).
- Under clipped gap an end token is scored like any other token, so a provider can end a reply early and still pass.
- A provider can commit to a model whose scores barely differ between tokens, so that under clipped gap every token passes.
- The commitment ID covers the quantization scales, but nothing checks that they are sensible: bad scales can make the model give one token for every prompt, or shrink the gaps between scores until clipped gap passes almost anything.

**Your own machine and data** (clients):

- Python runs a compiled copy in `integrations/__pycache__` in place of the source when its recorded time and size fit, so matching checksums of the `.py` files prove nothing: delete that directory before you rely on them.
- `check_proof.py` and `ask_and_check.py` start themselves again in Python's isolated mode. That comes too late for what Python loads while it first starts, such as a `sitecustomize.py` on `PYTHONPATH`; starting them as `python3 -I integrations/check_proof.py ...` excludes that too. Neither way covers the interpreter's own site-packages, `LD_PRELOAD`, `LD_LIBRARY_PATH` or `PATH`.
- `spool.py`, whose `make-request` writes your request file, and the installed commands do not start themselves again at all: whoever can set your `PYTHONPATH` or write your user site-packages decides what they report.
- The verifier gets your environment unchanged, except that the two API keys (`CHECK_PROOF_FETCH_KEY`, `VLLM_API_KEY`) are taken out.
- The check follows symbolic links named `key.bin` or `leaf/K.bin`, and through such a link its output can reveal facts about files of yours. With a public key directory that someone else made, look first (`ls -l`) that these are regular files, and do not use it if they are not.
- Keep proof files, and the output of `ask_and_check.py --show-checked`, as private as the prompt: each lets anyone test a guess at the prompt and reply.
- Without an API key the example client sends to whatever `--server` names. Over plain `http` the prompt, the reply and the proof ID then travel unencrypted, and through a proxy that your environment sets: use `https` for a server on another machine. `check_proof.py --fetch-from` takes plain `http` too where `CHECK_PROOF_FETCH_KEY` is not set: whoever sits on the way can then change or withhold the published files that it fetches. These take the same route through the script as local files, and are saved only after a pass. With the key set, the script refuses plain `http` to another machine.
- `check_proof.py` runs `cp_verify` with no limit on time or memory, and `cp_verify` sets none itself: run the check under limits of your own ([a hostile file](../docs/limits.md#hostile-files)).
- If the check is killed with `SIGKILL`, its private directory, `check_proof-*` under `TMPDIR`, stays, with the request file and the proof in it.
- `ps` shows a command line to every local user: to keep a prompt off it, give `--prompt-ids -` or `--prompt -` and pass the prompt on standard input. The commitment ID, the server's URL, the model's and the policy's names, the end tokens, the paths and the two token counts stay on a command line, and the demo passes its own prompts there.
- What a full-size check takes, in time and memory, is given on these pages from one call of `cp_verify` by itself, without `-j` ([the full-size example](../examples/full-size/README.md#try-it)). `check_proof.py` calls it with `-j 4` by default, and for one such call these pages give no figure. The middleware makes that call for each proof of a batch, up to `checks_at_once` at a time, one thread each, before any response of the batch is on offer. How long one batch then waited, with nothing else running beside the check, is told in [guide/performance.md](../guide/performance.md) and row RES-32 of [docs/limits-all-rows.tsv](../docs/limits-all-rows.tsv): one run, on synthetic weights, on 24 CPUs. Beside a serving `cp_prove` it has been timed in one run, on stored proofs, called directly, with no inference engine and no HTTP server around it (row RES-32); beside vLLM, and on that prover's own fresh proofs, it has not been measured.

**Disk and clean-up** (providers):

- `max_stored` and `max_waiting` count for a policy, not for a client: one client that leaves its responses unfetched can stop proofs for every client until `clean` runs. And `max_stored` is asked only when a request is submitted, and counts only finished responses: requests that already wait or are in work can carry the store past it.
- <a id="the-middlewares-check"></a>**The middleware's check** leaves these limits:
  - First, after a rejection, run `check`. A note whose name lacks `.done.json` is a wipe that isn't done. No note that bears the batch's time (the log names it), with `.done.json` or without: none could be written, and only the server's memory knows which batch was rejected, whether a hold file lies or not. Do not restart the server then. With several server processes: stop them all at once, and go on as below: one that did not see the rejection serves that batch's checked responses as soon as no hold file lies. If the server has died or must be restarted, do this by hand, in this order:
    1. Mend the disk: if the hold cannot be laid or a file cannot be removed, do not start the server.
    2. Lay the hold, and stop the server if it still runs. (Let no service manager restart a dead server by itself: that leaves no time for this.)
    3. Stop that policy's proving service (integrations/vllm/REFERENCE.md says how), wait until it has exited, and start it again only after you have emptied `out/`: if it names a batch while you remove, you can take a proof that would be rejected out of its batch, and the rest of that batch is then served.
    4. Before you start the server or, at the latest, before you lift the hold, remove every file from that policy's `out/` (every name that `ls` shows there; not `out/.tmp/`, which is `cp_prove`'s), and the folders that begin with `.check-` in `/dev/shm` and in the jobs directory (`ls -a` shows them), if no other server checks there.

    That takes the responses of sound batches too, and the failed ones of older batches: their clients, like the rejected batch's, read `waiting` or `expired` from then on and have to ask again. That there was a rejection, and under which policy, stands in the log alone: if a server died and you cannot read its log, nothing tells you whether a rejection was under way: to be sure, do this for every policy. Neither `check` nor a line of the log shows that such a wipe is done. If `check` cannot read the jobs directory (it ends with exit code 2), take it as that case; exit code 1 says only that a hold file lies.

    A note whose name ends in `.done.json` says that every response that it lists is gone from the `out/` that the server then had open, overwritten first unless the log named it in a line "the wipe of the withheld batch … is not done for these responses", or a server died in that wipe (the log then has a line "the wipe of the batch … goes on", and may name no response), and that a hold was laid there; it says nothing of a response that it does not list, such as one that came into `out/` after the check of its batch began, nor of a spool directory that was moved, replaced or not mounted at that time.
  - <a id="narrow-exceptions"></a>**Narrow exceptions to "every proof of its batch", none open to a client.**
    - If `cp_prove` is killed while it takes a batch's requests out of `work/` and is started again, it withdraws the responses whose requests still lay there, and the batch's other proofs are checked and handed on without them.
    - If an operator removes a job (with `clean`, or by hand) while its batch is being checked for the first time, and the check gives no verdict on that job's proof or its files look malformed, the batch's other proofs are handed on without it; the log names the request if its proof file is still there. If an operator removes an unchecked response's files from `out/` by hand, the batch's other proofs are checked and handed on without it, and nothing is logged.
    - If a response turns up in `out/` under the time of a batch that has already passed (files restored from a backup; a clock set back), the batch's other proofs were handed on before it was checked. It is checked when a job of the batch that has no answer yet is asked for; if it is rejected, what is left of the batch is withheld. Keep the spool directory on a file system that keeps modification times to the millisecond or finer: on a coarser one two batches released within a second bear one time, and the second's responses are such late ones. If a response turns up under the time of a batch that was rejected, after the check of that batch began (while `cp_verify` was at work, or later), the wipe does not take it and the note does not list it. Once the hold is lifted it is checked alone, and if it passes it is served, though a proof of its batch was rejected. Before you lift a hold after a rejection, look in that policy's `out/` (the log names the policy) for a `.proof` or `.y` file that bears the batch's time, and remove that response.
  - A failed job reads `no_proof` while a response with no job in its batch has not passed. A response that has its record goes on being served, until the hold, while a mate has no answer; where that hold cannot be laid, nothing stops the policy: of a batch that got no answer the server keeps no memory, though it does of a rejection; and the log says at each try that the hold could not be laid.
  - At a rejection earlier batches are out, and a download that has begun runs on. Nothing takes them back.
  - The check starts when a client first asks for a job of the batch. A batch that nobody asks for is not checked: a rejected proof can lie in `out/` unseen, with no hold, while later batches go out.
  - `cp_prove` learns nothing of a rejection. The hold stops new requests; the requests that already wait are still proved, on a machine in doubt, and use up the commitment's batches.
  - The hold that the check lays covers one policy. A fault of the machine may concern the others.
  - Copies. For each check the whole proof is copied to `/dev/shm`, or, where another account can change `/dev/shm`, or it is missing or short of room, below the jobs directory, and afterwards only unlinked; in `/dev/shm` nothing locks it in memory, so it can reach swap (row H28 of [docs/limits-all-rows.tsv](../docs/limits-all-rows.tsv): no swap, or encrypted swap). If the server dies in a check, a copy of an unjudged proof and a client's token IDs stay in the check's own folder until they are swept. If the removal of that folder fails, no line says so: the copy, a rejected proof's too, lies there, in a folder whose name begins with `.check-`, until the next check of any batch sweeps it. Where the overwriting fails, the file is deleted all the same, and the log names the response, perhaps only once, in the line "the wipe of the withheld batch … is not done for these responses", which does not say which step failed: so take every response that such a line names as not overwritten: its bytes may stay on the disk under no name. If the server dies before that line, no line names it. A full file system that writes changed blocks to new places can make the overwriting fail: there an overwrite needs room and a delete does not. What is overwritten with zeros in `out/` is worth what the file system makes of it. `clean` unlinks an unverified file without overwriting it.
  - A client's prompt and reply, as token IDs, rest in the jobs directory until `DELETE` or `clean`. A policy that is taken out of the config file puts its jobs beyond `DELETE`; `clean` still takes them, but no longer reaches that policy's spool directory.
  - Delay and cost. A batch's proofs are checked up to `checks_at_once` at a time, one thread each, one batch at a time for all policies, on the CPUs and in the memory beside vLLM, before the first response of the batch is on offer. One full-size check by itself takes about 21 s on one thread, and up to 3.53 GB of memory in its largest process at 24 threads ([performance](../guide/performance.md)); what one batch's check took by itself, and beside a serving `cp_prove`, on stored proofs, called directly, with no inference engine and no HTTP server around it (one run of each arrangement), and what the container's memory then reached: [guide/performance.md](../guide/performance.md); row RES-32. Beside vLLM, and on that prover's own fresh proofs, the wait has not been measured.
  - What a client can tell. A rejection differs from `cp_prove`'s own `failed` by the 503 between, which every client of the policy sees, and by its order: `proved`, 503, `no_proof` on one job, or all the jobs of a batch turning to `no_proof` behind one hold. By the words of a call a client can't tell what stopped a policy: under any stop each call gets the same words (while the jobs directory itself fails: a 500, whatever the cause). While a wipe is unfinished and the server can start no thread, a job's routes answer 500 where an ordinary hold gives 503. By the clock it can guess: a rejection's 503 comes at the end of the batch's first check, and every other hold that the check lays comes a minute or more later. After the hold it can tell: the jobs of a withheld batch read `no_proof`, and a batch that was only not judged is checked anew. A first GET that already says `proved` tells that a batch-mate asked earlier. The time from a first GET to `proved` grows with the number of proofs in the batch.
  - At rare hash outputs `cp_verify` does not accept an honest proof (row C2.2 of [docs/limits-all-rows.tsv](../docs/limits-all-rows.tsv)); nothing tells that from a bug or a hardware fault, so treat it as one. Whoever finds a request whose honest proof is not accepted, or slows the verifier past its time limit, stops the policy's proofs until a person comes.
  - How a batch is known: by one modification time on its files. Nothing in `out/` says how many responses a batch has: the check takes those that lie there when it lists them, once. That listing is whole by an order that two programs keep, not by a lock: `cp_prove` gives every response of a batch its name before the first of their requests leaves `work/`, and the middleware starts no check at a call for a job whose own request still lies in `work/`. (Read in prover/prover/spool.h and `proof_middleware.py`; tried with a stand-in, not with the real `cp_prove`.) That order must reach the middleware as it was made: one more reason for one machine and a local file system. Two batches on one time count as one: the later one's responses are late ones ([narrow exceptions](#narrow-exceptions)). One batch on two times (an `out/` that was copied or restored) is checked in parts, unless a response bears both: nothing leaves unchecked, but a rejection in one part does not fail the other.
  - One machine and a local file system: the lock that keeps the checks apart is a `flock`.
  - If a proof file changes after its check, the download has sent all but its last piece, of at most 64 KiB, before the tag tells.
  - The check reads the public files by path, the server serves from the directories that it opened at its start. After a directory was replaced under its name the two can differ.
  - A proof in the spool directory that has no job (a server that died between the request and the job's file; `spool.py submit` into the same directory), or a leaf file that is not in `public_key_dir`, bars its whole batch. Nothing of it is deleted, the log names the cause, and at the next try the policy goes on hold. `lift-hold` does not clear it: a person removes its files or puts its job back. A response that stays malformed with its job: remove its files and its job, whose client then has no proof.
  - Under a hold that no NO laid nothing is wiped: a spoiled response can lie in `out/`.
  - A wipe that broke off goes on only at a call for a submitted job of that policy or, where a note lies on disk, at a server's start: until then the rejected proof can stay readable in `out/`.
  - A check's command line shows `ps` its range, its request ID, the commitment ID, and the next token where the range ends at the last place. After a kill of the server its checks live on until they end, with a limit on CPU time and none on sleep.
  - Too little memory or CPU for the checks costs no batch: two tries in a row without an answer, or ten tries of any causes, hold the policy until a person comes.
  - A new parameter file laid under the same path while the server runs gives every later check no answer, and so a hold: `check_proof.py` takes the format's word from it at each check.
  - A spoiled member still in `work/` at a kill becomes a well-formed `failed` at `cp_prove`'s next start, and its final mates then pass alone.
  - A death in a wipe where no note could be written has nothing to finish from. Where the batch's records could be deleted at the rejection, no recorded proof leaves (where they could not: "If every call on the jobs directory fails", below), and what was not reached lies on in `out/`, not overwritten, and the rejected proof goes last: keep `out/` as secret as the secret key directory. After `lift-hold` what has no record is judged anew at a client's call: where a wipe that broke off has already taken the rejected proof away, what is left of the batch can pass that check and be served. With no room at a wipe for a note and for a `failed` response, the batch's jobs wait until room is back; the log names them.
  - Until the wipe is done, a note in the jobs directory, or the server's memory of the rejection, stops the policy at a client's every door, even where no hold can be laid: every submitted job answers 503, and a new reply gets no proof. Whether a policy is stopped, `check`, `lift-hold` and `spool.py` tell by the hold file alone; the hourly line names a wipe that isn't done as one of the causes. `check` and `lift-hold` do list each note by its name, finished or not: a name without `.done.json` is a wipe that isn't done. None of the three commands knows of a server's memory.
  - Two files in the jobs directory: a record is NAME.checked, beside the job file NAME.json that holds the response's request ID; a note is `withheld.POLICY.TIME.json`, where TIME is the batch's time in nanoseconds, which its files in `out/` bear. A wipe that is done renames the note to end in `.done.json`; where the note was in memory alone it leaves no file.
  - Do not remove an unfinished note by hand, but for the case of a note that cannot be read, below: it can be all that stops the batch's records. Let no `clean` run while a note without `.done.json` stands. Only the wipe's own end closes a note: a file that can't be removed, or any job file that can't be read beside a listed response with no job, keeps it open, and no log line names that job file: look in the jobs directory for a job file (64 hex digits, `.json`) that the server's account cannot read, of any policy, and make it readable or remove it. The note of a policy that has left the config file is never finished.
  - If nothing can be written at a rejection (a disk that has turned read-only), only the memory of the server process that saw the rejection stops the policy. Mend the disk while that server runs (with several server processes: the first entry above), and lay the hold by hand as soon as the disk takes writes. The server does not try again on its own: it finishes the wipe at the next call for a submitted job of that policy, a minute or more after its last try. What it remembers is never written down, not even once the disk is sound again. No line of the log says that the wipe is done; its line on what is left ("the wipe of the withheld batch … is not done for these responses") names these responses, and says to remove by hand what is left of their `.proof` and `.y` files in `out/`. A wipe that broke off can leave a proof in `out/` under a later time than its batch's, part of it not zeroed, or nothing at all of a response, whose client then waits without end. Another server process, or the same one after a restart, knows nothing of the rejection: it checks an unchecked batch anew, and where a wipe that broke off has already taken the rejected proof away, what is left of the batch can pass that check and be served; and it goes on serving those proofs of the batch for which it finds the record of an earlier check that they passed. A stopped server serves nothing, but what it remembered is lost: do not start it again before you have done the four steps of the first entry above. Mend the disk in place: a server that is started on another directory under the spool's path finishes the wipe there, closes the note, and leaves every proof of the batch whole in the directory that was set aside; keep that one as secret as the secret key directory.
  - If every call on the jobs directory fails at a rejection (an I/O error; a stale handle), the hold is laid, but no note is written, nothing of the batch is wiped and its records stay. The log then names the batch's time, which its files in `out/` bear, and not its requests, and no line says that only the server's memory knows of the rejection: what follows reads like a batch that could not be checked. Once the directory works, the running server does the wipe from its memory at the next call for a submitted job of that policy, not before. If it is restarted before that, nothing on disk says which batch was rejected, and lifting the hold puts the batch's earlier-checked proofs back on offer until the rejected one is asked for again. After such a restart, do the four steps of the first entry above before you lift the hold. (Where the jobs directory can still be listed and only not be written, the batch's responses are wiped at once, and nothing of it stays on offer.)
  - With several server processes, if the one that saw the rejection could not finish the wipe and another did, the first still remembers: at the first call that reaches it, it does the wipe over, deleting the failed responses that the other laid and laying them anew: if it dies in that, a client can wait without end, or the hold come back at each lift until `out/ID.*` is removed, and if the hold has been lifted by then, it lays the hold once more.
  - A note on disk that cannot be read, or that is no whole note, makes the wipe wait for a person, even in the process that remembers the rejection: lifting the hold alone will not do. If it cannot be made readable, keep the hold, remove that batch's responses from `out/` by hand (the note's name bears the batch's time, which its files bear too; a file that a wipe left half done can bear a later time: if in doubt, empty `out/` as in the first entry above), and only then the note.
  - `clean` stops at a hold file, not at the note of an unfinished wipe. Where a rejection's hold could not be laid, `clean` still deletes files by their age, and it deletes without overwriting. Once `clean` has taken every submitted job of the policy, no client's call can finish the wipe; a server's start does, where the note lies on disk.
  - If the server dies in the instant between a rejection and the writing of its note, the next server does not know of the rejection: it checks an unchecked batch anew, but it serves those proofs of the batch that had passed an earlier check until the rejected one is asked for again, and then withholds the batch.
  - If the disk is full from a rejection to the end of its wipe, and the server is started again before it could lay the hold, no hold is laid, and that batch's jobs stay "waiting" until `clean` removes them; the log alone tells of the rejection. Nothing of the batch leaves.
  - If a rejected batch's note found no room and the hold did, the server still remembers the rejection when its wipe is done. After you lift the hold, new replies still get no proof though no hold file lies, until a client asks for a job that was submitted earlier; then the hold comes back once, and lifting it again ends it. Nothing of the batch leaves.
  - A hold lifted over a wipe that isn't done comes back at each try until the wipe is done: mend what the log names.
  - <a id="core-dumps"></a>The middleware turns core files off for the programs that it starts, and `cp_verify` also switches its own core dumps off, which holds where the system hands core dumps to a program too (`kernel.core_pattern` begins with `|`). `check_proof.py`, which copies the proof for it, and the server itself, which reads each proof for its tag, have no such guard: there the first setting has no effect, and if one of them crashes, that program can get its memory, and with it a part of a proof that was not yet checked. A proof that a fault has spoiled can hold values that are not masked. Turn such a collector off on a serving machine, or keep what it stores as secret as the secret key directory.
  - While a mate's request lies in `in/` or `work/`, no check is spent on its batch, so a proof that `cp_verify` would reject is not found. If that lasts, the end is a hold with no check ever run.
  - A response that looks malformed, and whose job an operator removed while the batch was being read, is left out of that check: a rejection in it does not overwrite that response.
  - A client that exhausts the server (open downloads, memory) for more than a minute can bring about a hold: nothing is handed on, nothing is deleted, and a person must lift it.
  - A spoiled leaf number in a proof looks like a missing leaf file, so it is told as a fault around the check. The memory of tries is each process's own and is lost at a restart, so a server that is restarted between its tries comes to no hold. After the start nobody looks again at the verifier's owner, mode or place, so any other program under that path is run.
- Nothing bounds the jobs directory. Every reply to a request that asks for a proof leaves one job file there, also a reply that gets none, and only a client's `DELETE` and `clean` remove them: limit the request rate at your proxy, and run `clean` on a schedule.
- A request file holds a client's prompt and reply. `clean` deletes old ones that still wait in the spool directory's `in/`, and until then `DELETE` answers 409. It does not touch those that the service had taken: if the service stopped for good, they stay in `work/` without end, and no tool removes them.
- No program checks that every policy's stored responses fit the disk: choose `max_stored` and `max_waiting` for your disk yourself. On a full disk replies are still served, and their proof IDs lead to no proof.

**Setting up and running** (providers):

- Keeping the pinned packages current is yours: some, vLLM 0.29.0 among them, have published security advisories ([the operator's checklist](vllm/REFERENCE.md#the-operators-checklist) has what to run). `pip install ./integrations` is not pinned in that way: pip fetches the build tool for it, and runs it ([README.md](README.md#what-is-here)).
- The middleware is loaded only by vLLM's Python API server, which is the default. With `VLLM_USE_RUST_FRONTEND=1`, vLLM 0.29.0 starts another program in its place, which does not load what `--middleware` names: no proof routes, no policy header, no published files. Do not set that variable. (By reading vLLM's Python source, in which only the Python server loads what `--middleware` names; the other program itself was not read. Not tried: whether such a server starts at all with `--middleware`, and what a client then sees.)
- With the start command, anything that makes `cp_prove` exit badly stops a policy's proofs until a person has looked. That is the intent. Reading the code, we found no request whose content can bring that about, but this is not fully established; load can, such as a full disk. And the status 503 that a hold brings tells every client that the policy's proofs are on hold, and so, where the start command made the hold, that the service ended badly.
- If you lose a published `key.bin` or leaf file, put the published bytes back: `cp_commit` cannot make a file of an existing commitment again.
- <a id="the-look-above"></a>**The look above.** Before a script writes below a directory that you name, it looks at every directory above it, from `/` down: the demo at its work directory, the client at `--out-dir`, the middleware and `spool.py` at the jobs and spool directories (run as root, the middleware also at its config file, the public key directory and the parameter file's directory), `spool.py make-request` at its output file's directory, `check_proof.py` where it would save a fetched file, the integer-model package's `try_limits.py` at the directory that you give it, its `example.py` at the parent of the directory that you give it (for `vllm_files`, at the model directory itself; at the directory for temporary files if you give it none, and there too for the second directory that it makes for `small_gguf`), and `serve_gguf.sh` at the directory for temporary files.
- **What the look refuses:** a place that belongs to an account other than you or root, or that its group or others can write and that is not sticky; a place that is missing or cannot be examined is refused too.
- **Some places are refused though nothing need be wrong with them:** below a directory of mode 0775 or 2775 (one that you made under umask 002; a Kubernetes volume with `fsGroup`); below a scratch mount of mode 0777 or 2777, or a `/tmp` without the sticky bit (a Kubernetes `emptyDir`); anywhere in a container in which `/` shows an unmapped owner.
- **A plain check of a proof is not refused in such places:** `check_proof.py` does not look above its own temporary directory, which it checks once it has opened it, and where it cannot save a fetched file it prints a warning and goes on to its verdict. It enters that directory through the open descriptor and names every file in it relative to it, so that a rename in the temporary folder moves none of the files that the check reads there. Your own `key.bin` and leaf file are not copied there: the check reaches them, and `cp_verify` itself, by their full paths (by reading; the whole scene, with a second account, several calls at once and a dishonest server, has not been tried).
- **No exception is made for a group that has no member but the owner,** so under umask 002, which an ordinary user has on a plain Ubuntu 24.04, whatever directory you make or clone is refused until you run `chmod go-w` on it; then look at what is in it. That is not a full cure: `chmod` does not reach what lies inside, and a file that another account already holds open stays open to it. For a public file that the middleware trusts, make a fresh copy ([how](vllm/REFERENCE.md#accounts-directories-and-permissions)).
- **Choose a place below directories that only you or root can write:** a fresh `mktemp -d` gives one, and every route that these pages print works below one, except for the operator's set-up, which has places of its own ([Part 2](vllm/REFERENCE.md#part-2-operators)).
- **The look has gaps.** It reads only owners and modes, not access control lists. A file system that another user serves (FUSE with `allow_other`) can report any owner and mode. Except in the demo, it is one look by path, which another account can race. Behind it the client opens every directory above `--out-dir` in turn, without following a link, checks each again once it is open, and holds `--out-dir` open: every file in it is named relative to that. `spool.py` opens the spool directory once, checks what it has opened, and works from there; the middleware does so with every directory that its config file names, and keeps each open for as long as the process runs.
- **The other scripts have less behind the look:** the integer-model package's `try_limits.py` and `example.py` go on by the real path that they have checked, and by path from there (`example.py` checks the parent and makes its directory there, without looking at it): the account that owns the directory above can swap it; `serve_gguf.sh` works in a new directory that it has made in the real path that it checked; `spool.py make-request` opens its file by path; and `check_proof.py` saves through a directory that it has opened once.
- **A symbolic link on the way** passes if you or root own it and what it leads through passes; the demo and `cp_prove` refuse every link, so a spool directory behind such a link passes `init` and `check`, and `cp_prove` then refuses it at its start.
- <a id="under-sudo"></a>**Under `sudo`.** Do not run any of this under `sudo` or as root (see [Check a proof](README.md#check-a-proof)). The points below that speak of `sudo` or root tell what happens if you do, and what then works; they are for whoever does it anyway, not advice to do it. In a run as root with `SUDO_UID` set the look also trusts the directories of that account (but not `example.py`'s look at the directory for temporary files, nor `serve_gguf.sh`'s), so that it does not refuse a run under `sudo` below that user's own directories. The demo goes on by path after the look, and prints a note where its look at the work directory has used that trust: if one of those directories lies above the work directory, any process of that account can, at any time during the run, put a link or another directory in its place. Root then creates the run's files and reads its results wherever that account chooses.
- **Under `sudo`, the client prints no note.** Until the run holds `--out-dir` open, the account that ran `sudo` can choose which directory the run uses, a closed directory of root's among them; between two runs it can put another directory of root's under the name, and `--resume` then prints that directory's stored reply if the stored prompt and settings are this run's (`--server`, `--commitment-id` and `--end-token` are not compared). Once the client holds `--out-dir` open, that account cannot send a file of the run elsewhere, or read, change or delete what the run stores; that holds only for `--out-dir`, not for what `--parameter-file`, `--public-key-dir`, `--scales-file` and `--verifier`, a program that root then runs, name (by reading, and by tests in which the same process acts as the other account). The integer-model package's `example.py` and `try_limits.py` can be led in the same way, and print no note: their new files then come to lie where that account chooses, and what the scripts write themselves goes over no file that stands there ([what else it costs](../docs/operators-duties.md#under-sudo)). What a run under `sudo` creates belongs to root: read such files under `sudo` too; before you change an owner, read [step 1](vllm/REFERENCE.md#step-1).
- <a id="one-account"></a>**A known fault: behind `sudo`, most of this integration's set-up commands are refused on files and directories of your own.** In your own home, with your own config file, all five commands of `proof_middleware.py` behind `sudo` exit with code 2, from `init` on. They print what to type instead (`sudo -u ACCOUNT ...`, here with your own account's name), and that works. `spool.py`'s `submit` and `sweep` behind `sudo` exit with code 1 on a spool directory of your own, and print what to type instead too (the same command without `sudo`), and where that fails. A refused call leaves nothing behind. The prover's programs do not have this fault (some other tools and scripts of this repository have one of its kind, told on their own pages: [operators' duties](../docs/operators-duties.md#under-sudo)): they take `sudo` in a clone of your own, on a commitment that was made under `sudo` too (from reading their code; `cp_prove` has not run end to end under a real `sudo`: [operators' duties](../docs/operators-duties.md#under-sudo)).
- <a id="behind-sudo"></a>**Which way works behind sudo:** everything behind `sudo`, on root's files (tried without vLLM or `cp_prove`). The config file, the files that it names and their directories must be root's. `init` behind `sudo` then makes the jobs and spool directories, below root's directories or your own (for what is published, mind [the fault below](#above-a-public-directory)), and `spool.py submit` makes a spool directory that does not exist yet. What they make is root's, so every later step wants `sudo` too: your own commands there then fail. `sudo spool.py make-request` leaves a request file of root's, with mode 0600, which you cannot read. If a service was started under `sudo` in a spool directory that you had made yourself, `submit` behind `sudo` refuses the directory, which is yours, and your own `submit` fails at the mark file, which is root's: name a new spool directory, and keep to one account for it.
- <a id="above-a-public-directory"></a>**A known fault: at a start behind your own `sudo`, a server that runs as root can be led to send out closed files of root's.** Every process, whoever runs it, opens each directory that its config file names once, where it loads that file; it makes its checks on the open directory, keeps it open, and works below it. So whoever can change a folder at or above a name, by a link or by renaming, leads a running server to no other file, with one exception, and a command to none once it has opened its directories. The exception: the check that the middleware makes of each batch with `cp_verify` reads the public files by their names, so after a swap it can read other files than the server serves ([the middleware's check](#the-middlewares-check)). What goes by the path just before an open can still be raced: the look above ([its gaps](#the-look-above)) and the walk that refuses a secret key directory (below). Beside parting it from whatever goes by the name later (below), the most that it can do to a running server is to lead its check to other public files (the exception above), to put a policy on hold, by bringing under the name a spool directory of the same commitment that is on hold, or to end a hold that lies only under the name, by moving that directory away again (below; by reading). Run as root, the middleware also makes [the look above](#the-look-above) for its config file, the public key directory and the parameter file's directory. Behind your own `sudo` these scripts do not defend against a process that already runs under your own account. The look trusts the directories of the account that typed `sudo`: where the configured directories lie in your home, every process of that account can, ahead of a start or of a command (each command loads the config file and opens the directories again), by a link or by renaming alone, bring a closed directory of root's under a configured name (`clean`, `hold` and `lift-hold` then act on whatever of root's stands under the names at that moment); that holds at each start of each server process, also one that a supervisor makes with nobody looking on, as long as it carries `SUDO_UID`; and where the checkout is yours it can edit the very scripts that root runs. The server then sends a closed `key.bin`, leaf file or `layout.txt` of root's that lies there, and the numbers of its parameter file, to anyone who can reach the server and, where vLLM was given an API key, has it (where the middleware cannot learn that key: [the operator's checklist](vllm/REFERENCE.md#the-operators-checklist)), for as long as that process runs, and nothing is logged. In practice these are another commitment's public files that have not been published; they hold that commitment's OOD answers, and such files are themselves to be guarded ([operators' duties](../docs/operators-duties.md#a-record)). A server that was led astray at its start stays so until it is restarted, though the names are right again. What they do defend against, run as root: other accounts, at a start and after it, but for [the look's gaps](#the-look-above); and, whoever runs it, whatever is swapped under a server that is already running, but for the middleware's check of each batch (the exception above).
  - *Refused, whoever runs it:* a parameter file, or a public key directory, that lies inside a secret key directory, such as `policy.txt` in its `config/`. Name a copy of the file in a directory outside it, with a copy of `layout.txt` beside it if that is to be served.
  - *While a process runs, for every account:* a named directory that is replaced, moved or restored from a copy is not seen by the server until the next start, but for its check of each batch, which reads the public files by their names (the exception above), and its look for a hold (below): the server goes on in the old one, reads and writes there (the jobs and spool directories too), and logs nothing of it. To take back what a running proof server serves, stop the server for good (by the code, not tried): moving or renaming the directory, or a hold, does not stop it (seen; a hold stops a policy's proofs, not what is published). A restart alone takes nothing back: a server that starts while the directory still stands under its configured name serves it again, so move it away first. Stopping the server loses no file: prefer it to removing the directory, which can destroy the wrong files: what counts is the directory that the server opened at its start, and everything in it is lost. Restart the server after any change to the directories that it serves (after a rejection: [the middleware's check](#the-middlewares-check) first); before you replace a spool directory, stop its proving service; stop the server before you unmount a file system that holds one. A hold reaches a running server: it is looked for in the directory that the server keeps and under the configured name, if what stands there could be this policy's spool directory: no link, the server's own user's, mode 0700, with this commitment's mark file or `cp_prove`'s mark of zeros, in a place that [the look above](#the-look-above) lets pass. So after a spool directory was replaced, set the hold with the `hold` command, which marks an empty directory first; a hold file created by hand in a directory that nobody has marked does not count. A hold that lies only under the name holds only while that directory stands there. If in doubt, stop the server.
  - *A server that is not root:* ahead of a start nothing looks above its config file, its public key directory or its parameter file's directory. Whoever owns a folder above one of them, or can write in one that is not sticky, can aim the name at closed files of that account's own, and so also choose which limits are applied and published (by reading).
  - *A secret key directory:* at a start and in every command, a public key directory or a parameter file's directory that is one, or lies inside one, is refused, whoever runs it, by a walk up its path as written and with links resolved. The walk knows such a directory only by a file named `seed.bin` or `isk.txt` in it: one that has neither, or a copy of its sub-directories alone, is not known as one (by the code). It goes by the path, just before the directory is opened: against a swap in that very moment it does not hold.
  - *Basis:* tests as root and as an ordinary account, and 24 further scenes with up to three real accounts beside root, each once, on files made for the test, without vLLM, `cp_prove` or a socket. By report, under a running server no swap brought out a line of root's closed files; ahead of a start the account that typed `sudo` still brought them out, and that server went on so, though the names were right again. We do not claim that there is no way past the kept directories, and the rules have not been tried against every way of naming a directory.
  - *The way out:* keep the directories, the config file and the checkout below folders that belong to root and that only root can write ([the rule on who may write](vllm/REFERENCE.md#accounts-directories-and-permissions)): no account but root can then do this (by reading). Or run everything as ACCOUNT. Serving on 127.0.0.1 is no way out: that keeps out other machines, not other accounts on this one, and with the printed commands the server asks for no API key ([the operator's checklist](vllm/REFERENCE.md#the-operators-checklist)).
- **Under `sudo`, all this matters only if the account may not act as root anyway.** To rule it out, work only below directories that are all root's (`sudo mktemp -d -p /tmp` gives one; for a secret key directory choose such a place on a local disk before `cp_commit` makes it), or start the program without `SUDO_UID` (`sudo env -u SUDO_UID ...`). To unset it in your own shell before `sudo` does nothing: `sudo` sets it again. It stays set in a root shell from `sudo -s` and in whatever is started from it, until you unset it there; what was started before keeps it.
- With ordinary serving and an exact-match commitment, long replies get no proof: the served model's tokens soon differ from the integer model's first choices.
- A spool directory's mark file names the commitment that the directory is for, and the scripts in `integrations/` check it. `cp_prove` does not: it knows no commitment ID, takes any well-formed mark file, and writes 64 zeros where a script writes the ID; the scripts take that mark of zeros as any commitment's. So a service of another commitment can take up a spool directory, marked or not. Its start clears what is staged there and answers `failed` for what is in `work/`. The directory's requests are then proved on the other commitment's weights and use up that commitment's batches; the middleware's check, which is given the policy's commitment ID, does not pass these proofs. In a directory that carries the mark of zeros the scripts notice no wrong directory: `submit`, `sweep` and `clean` work there under any commitment ID, and `sweep` and `clean` delete. Run `init` before `cp_prove` first starts, so that the mark names the commitment. Two more cases are not caught. A copy or a restored backup of a spool directory has the same mark. And `cp_prove -clear` clears any commitment's marked directory.
- Some library functions of `spool.py` look at the mark file less than its commands do (read in the code). `request_stop()`, called without a commitment ID, checks only the form of the mark file, so it asks whichever commitment's service runs in that directory to stop; no script here calls it. The functions that take an open directory, and those that only read responses, do not look at the mark file: a front end of your own calls `check_mark()` first. No function of `spool.py` runs a verifier. A response that `read_response()` returns without faults has the right files and the right tag, and can still be one that `cp_verify` rejects. `cp_prove` does not check its proofs either, so a front end of your own must run `cp_verify` on every response of a batch before any of them leaves: a proof that a bug or a hardware fault has spoiled can hold hidden values unmasked ([the front end's duties](../docs/operators-duties.md#the-front-end)).

**Not examined, or seen and not cured** (providers; the runs told here were not on the released programs):

- Not examined: whether one client's request can change or reveal another client's reply inside vLLM.
- Not examined: what vLLM and its libraries keep in caches under the server account's home and temporary directories. Guard them like the weights.
- How the values of `cp_prove`'s options `-memory`, `-in-step`, `-batch` and `-most-batches` bear on hiding has hardly been tested, and not at all on the released programs. The full-size settings in [vllm/REFERENCE.md](vllm/REFERENCE.md#start-the-proving-service) are not the defaults. A reading of the code met no random draw or mask that depends on one of these options, but they have not been traced option by option. `-batch` and `-most-batches` do change how fast a leaf table's allowance of batches is used up.
- Two bad results on vLLM's own serving, on synthetic models, neither explained. (1) The number of events in a streamed reply (3 or 4) differed between runs that differed only in hidden values: the one field of 102 that varied, by an account of which no record is known, in an audit cut short at 288 of 600 requests, without a negative test: that the others agreed is weak evidence. In a further trial, with no proofs, it also varied with nothing changed, because the server sometimes put two tokens in one event, more often under load; that trial, on small models, one client at a time, run twice, could have seen only a gross difference between models. By the test fixed before it, under which a difference counts only if the same comparison shows it in both runs, it saw none: one of six comparisons raised a flag in the first run (integer serving, under load: all four starts of the model with other weights joined tokens more often than all four of the base model) and not in the second. Looked at afterwards, which counts for nothing under that test: in integer serving the model with other weights joined tokens a little more often than the base model in most pairs of starts, and the base model's own weights under other commitment seeds spread as widely. So it is not shown that hidden values leave the number unchanged. (2) At full size the time to a reply's first token told two models apart (not repeated, and not known to be cured): [what is not hidden: time](../docs/what-is-not-hidden.md#time).
- One more bad result, on the middleware's own routes, not known to be cured: how long a call for a job's state took showed the kind of answer. Over the loopback address, in two runs at a small size, it took about 0.5 ms for an unknown proof ID and 5 to 10 ms for a proved job; where the server answered 500, three further times showed whether a proof, a failure or a malformed response was stored under the ID.
- vLLM's own route `/v1/models`, which the middleware passes on unread, like every call without the policy header to a path outside `/plugins/proof/`, shows every client where the model's directory is on the provider's disk, and the server's clock. An allow-list of routes at the proxy closes that route ([the operator's checklist](vllm/REFERENCE.md#the-operators-checklist)). The middleware does not hide the server's clock: a client reads it from the web server's `Date` header and from the field `created` in vLLM's responses.

**Basis,** for all of these: read (the scripts in `integrations/`; for the mark file, `cp_prove`'s source, by report); reasoned; where a point says so, seen in a run. The trials that a point itself tells of (under `sudo`, or with a test driver) are not among these runs. Of these trials, those of the set-up commands behind `sudo`, of the advice that they print, of the server's and the proving service's look at their start, and of the refusal of a public key directory of your own were made under a true `sudo`, on values invented for the test, without vLLM, `cp_prove` or a GPU, on [other programs](../docs/glossary.md#other-programs), with a form of the middleware that neither kept its directories open nor looked above its config file and its public directories. A form that does both was tried, not on the released programs, in 24 scenes as root and behind a true `sudo`, the server called by a test driver, without vLLM or a socket: the Basis [above](#above-a-public-directory), and row R25 of [docs/limits-all-rows.tsv](../docs/limits-all-rows.tsv). The advice that two refusals print as root, that of the config file's own place and that of the public key directory or the parameter file's directory, has not been tried in this wording under a true `sudo`; as plain root, obeyed to the letter, it worked (one trial of ours).