# stuntd: a local Jev-compatible proxy that learns your typed LLM decisions ![stuntd: 90% of your LLM decisions answered locally, in about 20 ms, learned from your own traffic](docs/cover.png) [![CI](https://github.com/bladedevoff/stuntd/actions/workflows/ci.yml/badge.svg)](https://github.com/bladedevoff/stuntd/actions/workflows/ci.yml) [![PyPI](https://img.shields.io/pypi/v/stuntd.svg)](https://pypi.org/project/stuntd/) [![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](pyproject.toml) [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE) [![Hugging Face Space](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Space-yellow)](https://huggingface.co/spaces/pollix/stuntd) **stuntd is a local, self-hosted proxy that records the typed decisions your app already makes and learns to answer them itself.** It speaks the Jev System One protocol (`POST /v1/systemone`, with `choice`, `score` and `noul` questions), the OpenAI Decisions API (`POST /v1/decisions`), the OpenAI Chat Completions API and the Anthropic Messages API, distils each decision site into a small head on a frozen [Laya](https://huggingface.co/convaiinnovations/laya) encoder or a stock sentence encoder, and serves that answer locally with calibrated confidence, handing anything it is unsure about back to the provider. **Try it in the browser:** the [Hugging Face Space](https://huggingface.co/spaces/pollix/stuntd) puts zero-shot Laya and the heads stuntd trained side by side on three demos, no install. Three ways to run it: - **Local Jev.** No key, no provider, no training. Point `typesafe-sdk` at stuntd and the base Laya checkpoint answers your typed questions. - **In front of OpenAI or Anthropic.** A drop-in replacement for the provider base URL. Typed requests, Decisions API questions included, are recorded; everything else is relayed byte for byte. - **In front of the paid Jev API.** stuntd relays, learns from the provider's own answers, and takes the decision over once its head is right often enough. Status: early release, v0.2.1. Every number below traces to a demo in this repository or to a source listed at the end. ## Who it is for - **You have no Jev key.** TypeSafe's Jev is an early-access API at the time of writing, and there are no published weights. stuntd answers the same typed questions locally on Laya, with no key and no training, so a `typesafe-sdk` client keeps working while you wait. - **You pay a provider for the same decision thousands of times a day.** Routing, triage, moderation, gating. stuntd distils those answers into a local head: tens of milliseconds, no tokens, no round trip. - **Zero-shot Laya is not good enough on your labels.** On banking77's 77 intents it scores 38.2 where Jev scores 76.4. A few thousand rows of your own traffic is what closes that gap: in the banking demo the risk question goes from 30.0% to 72.5%. - **You want a Claude Code hook that decides locally.** A `PreToolUse` gate answering "may this command run" in about 50 ms, without sending every command you type to a provider. [`examples/devtools/`](examples/devtools/) ships the hook. ## Quickstart: local Jev without an API key ``` pip install "stuntd[train,jev]" stuntd config init stuntd serve ``` `config init` writes a commented `stuntd.toml` with everything at its default, which means no OpenAI upstream and no Jev provider. `serve` lists the endpoints it serves and loads the base checkpoint: ``` stuntd listening on http://127.0.0.1:8787 endpoints: jev local serving jev locally with convaiinnovations/laya ``` `stuntd stop` ends the daemon, and `GET /healthz` answers 200 with the number of sites in each mode without loading a checkpoint. Then change the base URL in your client and nothing else: ```python from typesafe_sdk import Choice, Noul, TypeSafeClient QUESTIONS = { "category": Choice( instructions="Which part of the product is this ticket about?", criteria={"billing": "Payments and refunds.", "bug": "Something is broken."}, ), "needs_human": Noul(instructions="Does a support agent have to read this?"), } with TypeSafeClient(api_key="local", base_url="http://127.0.0.1:8787") as client: answer = client.system_one( state={"subject": "Charged twice", "body": "Two identical charges on Friday."}, questions=QUESTIONS, ) print(answer.choices["category"].choice, answer.nouls["needs_human"].noul) ``` Anything that speaks the protocol plugs in the same way, by changing one base URL wherever the client exposes it: `typesafe-sdk`, JevRouter, the local-server request of fast-jev-compaction, and the browser and game agents built on the same SDK. The key is ignored unless you set `[jev] require_key = true`. To train a head on this daemon you need labelled rows: either put the paid Jev API in front of it (`[jev] upstream`) and let it teach, or write the rows yourself and `stuntd import` them (see [importing your own rows](#importing-your-own-rows)). ### Try it without any key Nothing here needs an account, a provider or a training run: `pip install "stuntd[train,jev]"`, `stuntd serve`, and run the client above. The first start downloads the Laya checkpoint (see [Details worth knowing](#details-worth-knowing)). To see how each answer was served, post the questions yourself and read the `X-Stuntd` header: ``` curl -si http://127.0.0.1:8787/v1/systemone -H "content-type: application/json" -d '{ "state": "Charged twice for the same invoice", "model": "jev-latest", "questions": {"needs_human": {"type": "noul", "instructions": "Does a person have to read this?"}} }' ``` With no head trained, the header reads `X-Stuntd: jev; mode=local; questions=1; live=0; zeroshot=1`: the base checkpoint answered. The demos under [`examples/`](examples/) generate their own labelled rows, so `stuntd import`, `stuntd train` and `stuntd enable` can be tried on data with a known right answer before any of your own traffic is involved. ### The local fallback In local Jev mode nothing sits behind the daemon, so a question a head will not answer has to be answered by something. By default that is the base Laya, zero-shot: the head answers when its confidence reaches the site's threshold and the request is not flagged by [the novelty gate](#the-novelty-gate), and the base checkpoint answers the rest. `serving.local_fallback = "head"` makes the head answer those too, for a head you trust more than zero-shot on your labels. `stuntd enable` warns when a site's coverage is under 50% in local mode with the zero-shot fallback, since most answers would then come from the base checkpoint. `stuntd report SITE --gold FILE` scores what is actually served under whichever fallback is set, and prints which one it used. ## Quickstart: OpenAI-compatible proxy that learns ``` pip install "stuntd[train]" stuntd config init stuntd serve --upstream https://api.openai.com ``` `--upstream` is the provider origin only, without `/v1`: your client's own path is appended to it unchanged. In the client, point `base_url` at the daemon: ```python import os from openai import OpenAI client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key=os.environ["OPENAI_API_KEY"]) ``` Everything keeps working exactly as before. A request whose `response_format` (or single tool) asks for an object made only of enum, boolean or number fields is a typed decision, and stuntd records it with the provider's answer. Anything else is relayed untouched, streaming included. An object with several fields, such as `{category, urgency, needs_human}`, learns one head per field. Each field is its own site, named `.`, and a request is answered locally only when every field's head is live and confident. If one is not, the provider answers and every field is recorded. Once a site has a few hundred captures: ``` stuntd status # what has been recorded, per site stuntd train # one head per site with enough examples stuntd report # holdout agreement, ECE, the operating point stuntd enable # let that site answer locally ``` `enable` and `disable` also take the parent of a multi-field decision: `stuntd enable triage` acts on every `triage.` that has a head and prints each one. While some fields are not live, the live ones are still compared against the provider's answer on every request, so their agreement keeps moving and a field that drifts is demoted on its own. ## Quickstart: OpenAI Decisions API OpenAI's Decisions API (`POST /v1/decisions`, model `gpt-6-luna`, public beta since 2026-10-06) asks typed questions of a text: `predicate`, `choice` and `score`. stuntd learns them the way it learns Jev questions: a predicate is a `noul`, a choice and a score keep their options, and every question is a decision site named after its `name`. Local, with no key. `stuntd serve` answers from a trained head where there is one and from zero-shot Laya for the rest: ``` stuntd serve ``` ```python from openai import OpenAI client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="local") decision = client.decisions.create( model="gpt-6-luna", input="Charged twice for the same invoice", questions=[ { "type": "predicate", "name": "needs_human", "instructions": "Does a person have to read this?", }, { "type": "choice", "name": "category", "instructions": "Which part of the product is this about?", "choices": [{"value": "billing"}, {"value": "bug"}], }, ], ) print([answer.type for answer in decision.answers]) ``` In front of OpenAI, the same client with the real key and the daemon started with `stuntd serve --upstream https://api.openai.com`. A request is answered locally only when every one of its questions has a live head that is sure; otherwise it goes to OpenAI untouched and each answer is recorded as the label of its question. Then `stuntd train`, `report` and `enable` work exactly as for the other paths, and `X-Stuntd` carries the Jev-style header described [below](#the-x-stuntd-header). What stuntd does not learn is relayed whole in proxy mode and never breaks the caller: an `input_image` part, a choice whose string and boolean values collide (`"true"` and `true`), a score with more than 10 levels, duplicate names, an unknown question type, a choice with fewer than two values, a message whose role is not `user`, a body that is not JSON, nests too deeply or carries a lone UTF-16 surrogate. A refusal answer from the provider is relayed and is not recorded as a label. In local mode the same requests get an OpenAI-shaped 400 (`{"error": {"message": ..., "type": "invalid_request_error", ...}}`). **Not measured against the live API.** The request and response shapes were checked against the [Decisions guide](https://developers.openai.com/api/docs/guides/decisions) and the `openai` 3.27.0 SDK types, and the tests pin them with fixtures built from those shapes. Nothing here was run against the real service. ## Quickstart: in front of Anthropic ``` pip install "stuntd[train]" stuntd config init stuntd serve --upstream https://api.anthropic.com ``` Point the SDK at the daemon; the key and the `anthropic-version` header are forwarded as they are: ```python import anthropic client = anthropic.Anthropic(base_url="http://127.0.0.1:8787") ``` A non-streaming `POST /v1/messages` that asks for a JSON schema through `output_config.format`, or for a single tool with an `input_schema`, is a typed decision on the same terms as the OpenAI path. The same decision asked through either provider lands on the same site and shares one head. A streaming request, or a request with tools and a free-text reply, is relayed untouched. Claude Code and the Agent SDK reach the daemon through `ANTHROPIC_BASE_URL`. The local answer is checked against the `anthropic` SDK's `Message` model in the tests, but no live Anthropic key has been run through stuntd. ## Numbers What a trained head does, measured on a laptop RTX 5060 with `convaiinnovations/laya` as the base checkpoint, 3000 generated rows per site and 24 epochs (5890 rows for Snake), with `training.cache_encoder` at its default. "Zero-shot" is the same daemon before training, answering from the base checkpoint alone. | demo | questions | zero-shot | trained heads | answered locally | p50 per request | | --- | --- | --- | --- | --- | --- | | [Snake](examples/snake/) | 1 choice | 0.70 average score | 11.40 average score | 99.6% | 21.1 ms | | [support](examples/support/) | choice, score, noul | 69.5 / 34.5 / 69.5% | 99.5 / 84.0 / 95.0% | 91% | 82.8 ms | | [banking](examples/banking/) | 2 choice | 89.5 / 30.0% | 100.0 / 72.5% | 78% | 75.7 ms | | [devtools](examples/devtools/) | choice, noul | 45.0 / 84.5% | 97.0 / 100.0% | 98% | 58.0 ms | Every decision site of the three text demos, measured against its teacher on held-out requests before and after the head was trained: ![Accuracy against the teacher, before and after training, per decision site](docs/before-after.svg) Snake is scored by what the answers do rather than by how many of them are right: ![Snake average score over 30 games, zero-shot against the heads and the oracle](docs/snake.svg) Five of the seven sites in the three text demos reach a holdout agreement of 0.95 or better. The demos run at the default `target_agreement` of 0.99, which picks each head's operating point rather than being a score it reached. The two that fall short, support's `urgency` at 0.922 and banking's `risk` at 0.937, stay in the demos anyway: see [limitations](#limitations). Latency, size and cost: | | measured | | --- | --- | | one head answer on CUDA | p50 22 ms, 26 ms in process | | the same head on CPU | 60 to 62 ms | | one live answer through the daemon, over TCP | 28 ms | | what the relay itself adds when the provider answers | 2.1 ms | | Jev API round trip, through OpenRouter | p50 376 to 389 ms | | Jev price | $0.042 per 1M input tokens, output free | | training one head: 5890 rows, 24 epochs | 186 s | | training three heads: 3000 rows each, 24 epochs | 266 s | | a trained head on disk | about 50 MB, fp16 | The first four rows were measured during development on an RTX 5060 laptop running Windows. They come from no published benchmark, and there is no link to give for them. The two training rows are the Snake and support demos of this repository, on the same laptop with the encoder cache on; the devtools demo measured the same 24-epoch run at 730 s with the cache off against 231 s with it on, 3.2x. Why train at all, rather than run Laya zero-shot. From [jevbench](https://github.com/dhruvmehra/jevbench), n=500 per dataset, accuracy: | dataset | labels | Laya zero-shot | Jev | fine-tuned DistilBERT | | --- | --- | --- | --- | --- | | sst2 | 2 | 92.0 | 95.4 | 91.0 | | agnews | 4 | 90.6 | 84.3 | 91.0 | | banking77 | 77 | 38.2 | 76.4 | 88.0 | Zero-shot Laya holds its own at two or four labels and collapses at seventy-seven. A model trained on a few thousand of your own examples does not: that is the bet stuntd makes, and the four demos are what it is measured against. ### Laya against stock encoders [`benchmarks/encoders.py`](benchmarks/) trains one site per encoder on a stratified sample of 3,000 training rows of MASSIVE intent (60 intents; ru, de, es, zh, ja, en) and Banking77 (77 intents), and scores every row of the official test split. Accuracy of every answer, Laya at 24 epochs, the others as `training.encoder`. The stock-encoder figures were identical in two full runs; Laya's differ between runs, so its cell gives the lower and the higher: | dataset | Laya (two runs) | multilingual-e5-base | multilingual-e5-small | all-MiniLM-L6-v2 | | --- | --- | --- | --- | --- | | MASSIVE ru | 26.2% to 27.2% | 81.6% | 78.6% | 54.7% | | MASSIVE de | 29.5% to 31.5% | 79.6% | 75.9% | 65.1% | | MASSIVE es | 34.3% to 34.4% | 80.5% | 78.0% | 63.5% | | MASSIVE zh | 42.7% to 44.1% | 79.8% | 78.3% | 40.2% | | MASSIVE ja | 32.1% to 32.2% | 79.8% | 79.5% | 57.7% | | MASSIVE en | 60.1% to 60.7% | 83.7% | 81.2% | 81.0% | | Banking77 | 65.1% to 66.1% | 89.6% | 87.5% | 90.0% | At each head's own threshold for 99% agreement, multilingual-e5-base answers 18.0% to 41.9% of the MASSIVE test rows locally, at 97.9% to 99.3% accuracy, and 60.8% of Banking77's; across the two runs Laya answers 0.0% to 5.9% of the MASSIVE rows and 4.9% to 6.5% of Banking77's. Training a stock head took 3.2 to 8.7 s on MASSIVE, Laya 236.4 to 308.7 s, and 1,768.8 to 1,860.3 s on Banking77, where the encoder cache did not fit the 2,048 MB budget. One batch-1 decision on the CPU has a p50 of 14.0 to 26.6 ms with multilingual-e5-base on MASSIVE and 337.4 to 401.7 ms with Laya. The latency and training-time columns did not reproduce within 25% on every row between the two runs, so read them as an order of magnitude. The laptop, the versions, the dataset revisions and both full tables with coverage, local share and p95 are in [`benchmarks/README.md`](benchmarks/README.md). ## How stuntd distils decisions into a Laya head ``` your app | v +-------------------------------------------------+ | stuntd | | | | typed decision? -- no ---------------------------> provider, byte for byte | | | | yes | | v | | head confident? -- yes --> local answer | 20 to 100 ms, no tokens | | | | no, or a 2% check sample | +---------|----------------------------------------+ v provider (answer returned to the caller, and recorded as a capture) ``` A **decision site** is one place in your code that makes one decision. On the Jev path the site is the question's name. On the OpenAI and Anthropic paths it is a hash of the response schema and the system prompt, or whatever you put in the `X-Stuntd-Site` request header. A request with several typed fields gives each field its own site, `.`. Keep one decision to a question. A question that mixes two, such as "is it urgent and which team handles it", learns worse than two separate questions. On the OpenAI and Anthropic paths, put the two decisions in two fields of one object instead: each gets its own head. Each site walks through four modes. 1. **collect.** The provider answers everything. stuntd stores the request text and the answer, with the redaction rules applied before anything is written. 2. **train.** `stuntd train` fits one classification head per site on the frozen Laya encoder, holds back 20% of the rows, and reports the holdout agreement, the calibration error and the confidence threshold at which the head reaches your `target_agreement`: `category trained holdout agreement 0.992 coverage 1.00 at agreement 0.992 ece 0.024`. Below that threshold the head abstains, which is what coverage measures: the share of requests it is willing to answer at all. `stuntd enable` refuses a head whose coverage rounds to zero and warns below 10%. 3. **shadow.** The caller still gets the provider's answer; the head answers beside it and the two are compared. `stuntd status` shows the running agreement. 4. **live.** `stuntd enable ` and the head answers the caller. Anything under the threshold still goes to the provider. So does a `check_share` sample, 2% by default, and those samples are what keeps the head honest: once agreement over the last `window` decisions falls below `target_agreement`, the site demotes itself back to shadow and logs why. Promotion the other way is manual unless you set `serving.auto_promote`. With `serving.auto_promote` and `training.auto_retrain` both set, the loop closes: collect, train, shadow, promote, demote, and retrain once `auto_retrain` new distinct texts have arrived since the head was trained, on the OpenAI path and the Jev path alike. A text recorded again does not count twice, and no run for a site starts sooner than `training.auto_retrain_min_minutes` (30 by default) after the previous one for that site. The daemon runs `stuntd train` for the site as a child process, one at a time, and the new head starts in shadow like any other, even when the site was live before: the daemon logs that at warning level and `stuntd status` shows `shadow (retrained, was live)`. A failed run is logged with its exit code and is not retried until another `auto_retrain` distinct texts arrive; serving carries on meanwhile. `stuntd status` shows how the last automatic run ended. In proxy mode the provider behind `[jev] upstream` does not have to be the paid API. Any server that speaks `POST /v1/systemone` can sit there -- kev, LLM2Jev, the laya-server family -- which makes it a keyless local teacher: it answers, stuntd records, and the head learns from it. See [Teachers](#teachers) for the four documented ones. Nothing is uploaded. Captures live in a SQLite file under your user directory with private permissions, and there is no telemetry. ### Checking a head against verified rows Agreement measures how faithfully a head copies its teacher, the teacher's mistakes included. It says nothing about whether either of them is right. For that, write a few hundred rows a person has checked, in the same JSONL format `stuntd import` reads, and score the site against them: ``` stuntd report --gold verified.jsonl ``` For example: ``` site tickets gold rows 300: 296 used, 4 skipped head accuracy 0.892 head accuracy 0.875 on 8 rows the store has not seen served accuracy 0.936 at threshold 0.62: 71% answered locally, 3 below it with no teacher answer teacher accuracy 0.948 on 288 rows head and teacher: both right 252, both wrong 10, head only 5, teacher only 21 ``` The head answers every row. The teacher's answer is the latest capture of that exact text, so it is there only for rows stuntd has already seen, and those are rows the head may have trained on. The line for rows the store has not seen is the out-of-sample number, so do not import the gold rows or send them through the proxy before scoring. Served accuracy is what stuntd would serve once the site is live: the head at or above the threshold, the teacher below it. Rows whose answer is not one of the site's labels are skipped. `--json` prints the same numbers, and the command needs the train extra, since it loads the base model. ### Reading `stuntd report` ``` site tickets choice category trained