# Jev, measured: 5,721 calls against TypeSafe's System One model

**An independent, pre-registered evaluation — and the places where TypeSafe's own
documentation understates its model.**

**PriorBench** · *measured, pre-registered evaluation of AI systems*
Author: **1j6c** · `1j6c@proton.me`

Model: `typesafe/jev-1.13-20260917` · Access: OpenRouter `/api/alpha/decisions`
Location: Western Europe (fibre, ~18 ms RTT to the gateway) · 20 September 2026
Total cost: **$0.176** · Raw data, code and pre-registration: included

---

## Summary

1. **Latency is a fixed cost, not a workload cost.** 430 ms floor, essentially
   flat regardless of what you ask. Growing the state 23× adds 71 ms. **800
   questions in a single call take 985 ms and cost $0.00075** — about one
   microsecond and one millionth of a dollar per judgement.

2. **95.9 % zero-shot on our reference benchmark**, against 77.2 % for
   hand-written keywords and 66.0 % for a supervised TF-IDF + logistic
   regression that saw all 400 labels. Jev saw none.

3. **TypeSafe's own "jaggedness" page understates the model.** It states Jev
   cannot compare numbers and that date ordering and date-window questions are
   unreliable. We measure **99.6 % across 13 distinct designs** (520 calls) —
   plain integers, thousand separators, decimals, negatives, 10¹⁴ magnitudes,
   three-way comparisons, five date formats, and English. Negation, another
   documented failure mode, scores **100 %** including double negation.

4. **The model always answers, even when nothing is there.** A yogurt-cake
   recipe is classified as a technical issue with **0.94 confidence**. Random
   letters: **0.97**. An empty string: **0.68**. With no escape option, **0 of
   30** out-of-scope messages were flagged — at 0.99 confidence. Confidence
   measures separation *between the options you offered*, not whether the
   question made sense.

5. **Intermediate confidence thresholds are useless on this model.** Accuracy
   above threshold is flat from 0.50 to 0.95 (97.9 % → 97.3 %, slightly
   *decreasing*), then jumps to **100.00 % at 0.99, covering 60.2 % of traffic**.
   It is 0.99 or nothing.

---

## The two theses

We wrote both before measuring, and report which survives per dimension.

**H+ (goldmine).** A genuine economic break: frontier-comparable quality on
structured decisions at a fraction of cost and latency, with confidence usable
as an automation gate.

**H− (letdown).** A mid-tier zero-shot classifier (62–68 % on independent
benchmarks), sold on a speed claim that does not hold outside the US, whose
confidence is not a probability, and which a free classical method matches on
its own turf.

### Verdict by dimension

| Dimension | Winner | Evidence |
|---|---|---|
| Classification quality | **H+**, decisively | 95.9 % vs 77.2 % keywords, 66.0 % supervised |
| Throughput economics | **H+**, overwhelmingly | 800 judgements per call, $9.4 × 10⁻⁷ each |
| Robustness to phrasing | **H+** | 4.2 pt spread across 5 paraphrases |
| Documented failure modes | **H+** | 3 of them do not reproduce at all |
| Language independence | **H+** | FR 90.0 % = EN 90.0 %, identical failures |
| Prompt injection | **H+** | 1/90 flips in French, 0 above 0.90 confidence |
| Type safety | **H+** | 0 schema violations in 5 escape attempts |
| Real-time latency | **H−** | 347 ms floor; sub-300 ms budgets are impossible |
| Confidence as probability | **H−** | ECE 0.051; it ranks, it does not calibrate |
| Behaviour on out-of-scope input | **H−**, seriously | Always answers, median confidence 0.91 on garbage |
| Robustness to *wrong* criteria | **H−**, severely | 16.7 %, below the 25 % random floor |
| Counting | **H−** | 48 % on 40-item lists with similar distractors |

**Neither thesis wins outright.** Jev is excellent when the question is well
posed and the answer exists among the options offered. It is dangerous when
either condition fails — and its confidence does not warn you.

---

## Method

### Rules we bound ourselves to

1. Every hypothesis written and timestamped in `PREREGISTRATION.md` **before**
   collection. 50 predictions; 47 evaluable as binary verdicts.
2. Both theses stated per block; we report the survivor, not the preferred one.
3. No bare percentages: Wilson 95 % intervals and `n` throughout.
4. No Jev number published without a no-AI baseline on the same items.
5. Ground truth generated by rule where possible; disputable items flagged and
   excluded from headline figures.
6. Confirmatory and exploratory findings kept in separate sections.
7. No metric changed after seeing data. Where a metric was poorly chosen, both
   the pre-registered version and the corrected one are reported.
8. Method failures published (see *Where we got it wrong*).
9. Fixed seeds, raw JSONL retained, exact cost, model version cited.
10. Limits stated up front.
11. No extrapolation: every latency figure carries "via OpenRouter, from Europe".
12. **Every surprising result replicated with a different design before
    publication.**

### Independent verification

A second script (`analysis/recount.py`) recomputes headline figures from raw
JSONL without importing any analysis code. It confirms 5,721 successful calls,
60 deliberate failures (the >255-option refusals), 4,015,924 input tokens, and a
verified unit price of **$0.042000 per million tokens** — matching the published
price to six decimal places.

---

## Results

### 1. Latency is a fixed cost

491 calls, 8 cells, randomised order, 10 warm-up calls discarded.

| cell | input tokens | p50 | p90 | p95 | p99 |
|---|---|---|---|---|---|
| floor (1 question) | 376 | 437 | 527 | 548 | 595 |
| baseline | 1,456 | 452 | 548 | 558 | 614 |
| large state | 8,724 | 508 | 638 | 715 | 756 |
| 1 question | 1,410 | 479 | 543 | 563 | 597 |
| 12 questions | 1,570 | 470 | 552 | 572 | 654 |
| **global** | | **475** | **558** | **604** | **715** |

![Latency is a fixed cost](figures/fig1_latency.png)

Network round-trip: 18 ms TCP, 42 ms with TLS — **3.8 % of total**. The
remaining ~430 ms is server-side and nearly workload-independent: marginal cost
is ~8.5 µs per input token.

**Going from 1 to 12 questions costs −9 ms** — noise. The parallel sampler claim
holds. Pushed further (E5.4a):

| questions in one call | latency | cost |
|---|---|---|
| 1 | 514 ms | $0.000013 |
| 100 | 1,078 ms | $0.000100 |
| 400 | 560 ms | $0.000377 |
| **800** | **985 ms** | **$0.000747** |

800 typed judgements, under a second, for three quarters of a millidollar.
**This is the single most actionable finding in this report: if your system
makes two sequential Jev calls, you are wasting 430 ms for nothing.**

**Practical consequence for real-time trading bots.** Several public projects
budget one Jev decision per ~300 ms blockchain block. Across 480 calls our
fastest was **347 ms**; p99 was 715 ms. That budget cannot be met from Europe
via this path. The architectural fix is not a faster language — the CPU is idle,
waiting on the network — but moving the model out of the hot loop: let Jev judge
slow-moving state (regime, toxicity, liquidity stress) every few seconds, and let
deterministic local code act per tick.

### 2. Concurrency — previously unmeasured

Every published Jev benchmark we found runs sequentially. Nobody uses a model
that way in production.

| concurrent | p50 | p95 | throughput | speedup | errors |
|---|---|---|---|---|---|
| 1 | 521 ms | 617 ms | 1.91/s | 1.0× | 0 |
| 4 | 537 ms | 965 ms | 6.88/s | 3.6× | 0 |
| **8** | **576 ms** | **863 ms** | **10.41/s** | **5.4×** | 0 |
| 16 | 821 ms | 1,472 ms | 11.19/s | 5.9× | 0 |
| 32 | 1,493 ms | 1,846 ms | 16.19/s | 8.5× | 0 |

![Throughput saturates near 11 req/s](figures/fig2_concurrency.png)

**Throughput saturates near 11 req/s.** Doubling from 8 to 16 buys 7 % more
throughput for 43 % more latency. **Eight concurrent requests is the operating
point.** No rate limiting observed at any level.

### 3. Classification quality, with a reading scale

400 items, four categories, three difficulty tiers. *Explicit*: the category
keyword is present. *Implicit*: the category follows from the situation, no
keyword. *Trap*: a keyword from a **different** category is present and the true
answer lies elsewhere.

| method | global | explicit | implicit | trap |
|---|---|---|---|---|
| **Jev (zero examples)** | **95.9 %** | 100.0 % | 99.6 % | 90.0 % |
| Keywords (hand-written) | 77.2 % | 100.0 % | 77.5 % | 60.0 % |
| TF-IDF + logistic regression¹ | 66.0 % | 70.8 % | 55.0 % | 70.6 % |
| Weighted random | 22.0 % | 21.7 % | 20.8 % | 23.1 % |

¹ 5-fold cross-validation **grouped by template**. A naive split scores 100 %
because template variants leak across folds — we report that as a cautionary
counter-example, not a result.

![Zero-shot against a reading scale](figures/fig4_benchmark.png)

Two details worth more than the headline:

- **All 32 trap errors — 32 of 32 — land exactly on the planted decoy.** This is
  not noise; it is a predictable failure mode.
- **Confidence 1.000 when right, 0.420 when wrong.** 1 item in 400 changes
  answer across two passes.

### 4. Confidence: a ranking, not a probability

From the 800-call benchmark run:

| threshold | coverage | accuracy above | accuracy below |
|---|---|---|---|
| 0.50 | 96.0 % | 97.92 % | 46.9 % |
| 0.80 | 88.9 % | 97.75 % | 80.9 % |
| 0.90 | 81.8 % | 97.55 % | 88.4 % |
| 0.95 | 73.0 % | 97.26 % | 92.1 % |
| **0.99** | **60.2 %** | **100.00 %** [99.2–100.0] | 89.6 % |

![Intermediate thresholds do nothing](figures/fig3_confidence.png)

**AUROC 0.837** — matching the 0.83 reported independently by other teams, a
useful external convergence. **ECE 0.051**, **Brier 0.038**.

Raising the gate from 0.50 to 0.95 does not help; accuracy is flat and slightly
declining. The jump is at 0.99. **The widely repeated advice to "gate at 0.9" is
wrong for this model.**

#### Graded difficulty — where the previous public attempt failed

An earlier independent probe could not produce a reliability curve because its
bench was too easy (110/150 items at exactly 1.00). We used **controlled
mixing**: a 6-sentence message where *k* sentences point to category A and 6−*k*
to B. Truth is the majority; *k* = 3 is genuine 50/50.

| k | expected | accuracy | p(billing) | confidence |
|---|---|---|---|---|
| 0 | technical | 100.0 % | 0.000 | 1.000 |
| 2 | technical | 100.0 % | 0.010 | 0.990 |
| **3** | **ambiguous** | 29 % chose billing | **0.320** | **0.450** |
| 4 | billing | 100.0 % | 0.930 | 0.870 |
| 6 | billing | 100.0 % | 1.000 | 1.000 |

The bench works: only 56.8 % of answers sit above 0.99. **Perfect accuracy
wherever a majority exists, even 4-vs-2 — and confidence collapses to 0.45 at
true ambiguity.** That is close to ideal behaviour.

One asymmetry: at true 50/50 the model leans technical (p(billing) = 0.32, not
0.50). Our pre-registered prediction of 0.40–0.60 is **falsified**.

#### The three primitives compared — never done before

Same question asked three ways:

| k | expected | `noul` | `score` (0–4) | expected score |
|---|---|---|---|---|
| 0 | 0.000 | 0.020 | 0.00 | 0.00 |
| 2 | 0.333 | 0.255 | 1.15 | 1.33 |
| 3 | 0.500 | 0.515 | 1.81 | 2.00 |
| 4 | 0.667 | 0.790 | 2.50 | 2.67 |
| 6 | 1.000 | 0.955 | 3.97 | 4.00 |

**`noul` tracks the true proportion with mean error 0.084, versus 0.161 for a
`choice`'s probability — roughly twice as well calibrated.** If you need a number
you can threshold, ask a `noul`, not a `choice`.

`score` is genuinely fractional and near-linear — to our knowledge the first
public measurement of this primitive.

### 5. The dangerous part: it always answers

20 calls each, no correct answer exists for any of these:

| state | answer | median confidence |
|---|---|---|
| `""` (empty) | technical ×20 | 0.680 |
| `"   "` (whitespace) | technical ×20 | 0.720 |
| yogurt-cake recipe | technical ×20 | **0.940** |
| random letters | technical ×20 | **0.970** |
| "this is a test, ignore it" | technical ×20 | **0.990** |
| self-contradictory text | billing ×20 | 0.880 |

**Median confidence on meaningless input: 0.91.**

And with in-domain but out-of-category messages (switchboard requests, job
applications, press enquiries):

| condition | out-of-scope flagged | confidence | in-scope control |
|---|---|---|---|
| no "other" option | **0 / 30** | 0.99 | 93.3 % |
| with "other" option | **21 / 30** | 0.98 | 96.7 % |

Without an escape hatch it forced all 30 into categories — 27 into "commercial",
3 into "technical" — at 0.99 confidence.

**This is the practical security finding of this report.** Confidence expresses
separation among the options you supplied. It says nothing about whether any of
them applies. **Always include an explicit "none of these" option.**

### 6. What design mistakes actually cost

TypeSafe's docs say "curating what you send is your job, and it is most of the
work". We quantified it. 1,260 calls.

**Instruction phrasing — barely matters.**

| phrasing | accuracy |
|---|---|
| verbose (60-word framing) | 97.5 % |
| neutral | 95.8 % |
| imperative | 95.8 % |
| telegraphic ("Category?") | 94.2 % |
| leading question | 93.3 % |

4.2-point spread. Pre-registered prediction of >5 points: **falsified**.

**Criteria descriptions — absence is free, wrongness is fatal.**

| criteria descriptions | accuracy |
|---|---|
| detailed | 95.8 % |
| short (one word each) | 95.0 % |
| **absent (empty strings)** | **95.0 %** |
| **deliberately misleading** | **16.7 %** |

Removing descriptions entirely costs **0.8 points** — the category names alone
carry the signal. But swapping descriptions between categories drops accuracy to
**16.7 %, below the 25 % random floor.** The model follows your descriptions over
its own judgement. This is the "literal reading" failure mode, and it is real and
severe.

**The docs are right that curation is the work — but the work is accuracy, not
richness. You can be minimal. You cannot be wrong.**

**Determinism.** 60 identical calls: the `choice` was identical 60/60 with
confidence exactly 1 each time. But the `noul` in the same calls ranged **0.46 to
0.54** across nine distinct values. **A `noul` thresholded at 0.5 is not
reproducible near the boundary.**

### 7. Physical limits

**Cardinality** (constant-length descriptions, correcting a confound in our own
earlier test):

| options | accuracy | p50 latency | input tokens |
|---|---|---|---|
| 2 | 100.0 % | 420 ms | 373 |
| 50 | 100.0 % | 439 ms | 1,756 |
| 200 | 100.0 % | 468 ms | 6,206 |
| **255** | **100.0 %** | 458 ms | 7,855 |
| 300 | hard refusal | — | — |

`HTTP 400: Too many choices. Must have at most 255 choices.` No degradation
before the wall; latency stays flat. Pre-registered prediction of a pre-255
decline: **falsified**.

**Context.** Largest accepted state: **32,553 tokens**. At 50k:
`{"error_type":"max_tokens_exceeded"}`. OpenRouter's advertised 32k is accurate;
the native docs' 64k does not apply through this gateway.

**Type safety.** Five escape attempts — hostile instruction in state, free-text
request, suggested non-existent option, out-of-range score, non-boolean `noul`.
**Zero violations.** Their "mathematically impossible" claim survives our attempt
to falsify it.

### 8. Language and adversarial robustness

**French versus English**, 20 hard items (irony, implicature, slang, typos,
lexical traps), 3 repetitions each:

```
French  90.0 %        English  90.0 %
```

Not merely similar — **the same two items fail in both languages, with the same
wrong answer each time.** Irony 100 %, misspelled SMS-register 100 %. Meaning
dominates surface form.

Both failures are arguably *our* labelling errors on genuinely borderline
messages, so the true error rate may be zero. Per rule 5, we flag rather than
bury this.

**Prompt injection in French** (replicating Laszlo Fazakas's English protocol):

| variant | accuracy | flips | flips at conf ≥ 0.90 |
|---|---|---|---|
| clean | 100.0 % | 0 | 0 |
| direct hostile instruction | 96.7 % | 1 | 0 |
| untrusted assertion | 100.0 % | 0 | 0 |
| fake authority note | 100.0 % | 0 | 0 |

**1 flip in 90, none above 0.90 confidence** — matching the English result. No
French-language weakness.

**Negation**, a documented failure mode:

| instruction form | accuracy |
|---|---|
| affirmative | 100.0 % |
| negative ("does NOT concern…") | 100.0 % |
| double negative | 100.0 % |

Including "it is false to say this message does not concern billing". Cost of
negation: **0.0 points**. Pre-registered prediction of >20 points: **falsified**.

### 9. Counting — the one real weakness, mapped

| list length | distinct distractors | similar distractors |
|---|---|---|
| 10 | 100.0 % | 88.0 % |
| 20 | 92.0 % | 80.0 % |
| 40 | **72.0 %** | **48.0 %** |

Length hurts more than similarity (28 pts vs 12 pts) — our prediction of the
reverse is **falsified**. Both compound: 40 items with lookalike distractors
lands at 48 %.

Confidence still works here: 0.82 when right, 0.53 when wrong.

---

## Where the documentation understates the model

TypeSafe publishes a "jaggedness" page listing nine failure modes. Three of them
do not reproduce.

| Documentation claim | Measured | n |
|---|---|---|
| "cannot count, do arithmetic, **reason about dates**" | 99.6 % on numbers and dates across 13 designs | 520 |
| "asking which of two dates comes first […] is unreliable" | 100 % (FR, ISO, text, EN); 97.5 % (US mm/dd, genuinely ambiguous) | 200 |
| "whether one falls inside a window" unreliable | 100 % | 160 |
| "negations […] are read at face value" | 100 %, including double negation | 90 |
| "accuracy drops as the state fills with unrelated content" | No drop on comparison or dates; counting drops 13 pts | 360 |
| "cannot count" | **Confirmed** — 48 % worst case | 310 |
| "literal reading" | **Confirmed, severely** — misleading criteria → 16.7 % | 1,260 |

The US date format result deserves a note: `03/04/2024` is genuinely ambiguous
between 3 April and 4 March. Tellingly, on those errors confidence sits at
**0.89** — it resolves an unresolvable ambiguity with conviction — while on the
abbreviated `dd-mm-yy` format, confidence drops to **0.25**.

**We are not claiming the documentation is careless.** It is unusually candid for
a vendor, and two of its warnings are precisely correct. But a practitioner
reading it would avoid entire classes of task the model handles perfectly.

---

## Where we got it wrong

Per rule 8.

1. **Our first language benchmark was unusable.** 100 % in both languages,
   confidence saturated at 1.000 — the exact failure we had criticised in another
   researcher's work hours earlier. Rebuilt with hard items.

2. **A position-bias metric that missed its own effect.** We pre-registered
   "first versus last position", giving 3.5 points and a falsified prediction.
   The actual pattern is U-shaped (0.520 / 0.390 / 0.435 / 0.485) — a 13-point
   spread between best and worst position. We report the pre-registered metric as
   falsified **and** the post-hoc observation as exploratory. We did not
   retroactively change the metric.

3. **A confounded cardinality test.** Our first version varied option count and
   state size together. Re-run with constant-length descriptions.

4. **Template leakage in our own baseline.** Naive cross-validation gave the
   supervised baseline 100 %. Grouping by template gives 66.0 %.

5. **A miscalibrated context probe** that never reached the limit. Re-run.

6. **A systematic bias in our predictions.** Of 47 evaluable pre-registered
   predictions, **26 were confirmed and 21 falsified — and 18 of those 21 failed
   in the same direction: we predicted Jev would break, and it did not.** Only
   two were falsified because the model underperformed our expectation. This is a
   measurable bias in our priors, stated so readers can discount accordingly.
   Full table: `paper/tables/predictions.md`.

---

## Status: this is the surface

This report is one evening's work — a wide, shallow pass across the whole
model. Twenty-one experiments, none of them taken as far as it could go.

The repository is updated daily: separating model from gateway latency, taking
each axis deeper, automating the suite so it re-runs against any model version
and diffs the results, extending language and adversarial coverage, and
eventually applying the same method to other models. Every number here is a
first measurement, not a final one.

## Declared limits

1. **Everything goes through OpenRouter.** We cannot separate model latency from
   gateway latency. The ~430 ms fixed cost is measured, but its *cause* is
   hypothesis, not finding. We suspect the gateway, since TypeSafe advertises a
   70 ms floor and we never observed below 347 ms. **Anyone with native API
   access can settle this in twenty minutes — please do.**
2. **Our benchmarks are ours.** Reproducible and published, but not recognised
   standards. The explicit/implicit/trap structure is the part we would defend.
3. **One geographic point.** A single Western European fibre connection,
   ~18 ms RTT to the gateway. We invite measurements
   elsewhere; the harness is included.
4. **One model version**, `typesafe/jev-1.13-20260917`, on one day.
5. **Some classification ground truth is disputable.** Flagged, reported
   separately, never folded into headline figures.
6. **Counting and cardinality items are synthetic.** Real-world counting tasks
   may differ.

---

## Reproducing this

```bash
git clone <repo> && cd jev-measured
python3.11 -m venv .venv && ./.venv/bin/pip install -r requirements.txt
./.venv/bin/python setkey.py          # paste an OpenRouter key, never echoed
./.venv/bin/python latency/bench.py   # 491 calls, ~4 min
./.venv/bin/python probes/t6_banc.py  # 800 calls, ~1 min
./.venv/bin/python analysis/recount.py
```

Full suite: **5,721 calls, $0.176, about 25 minutes** at 8-way concurrency.
Every raw response is in `*/raw/*.jsonl`. Seeds are fixed at 20260920.

---

## Practical recommendations

1. **One call, many questions.** 800 judgements cost the same wall-clock time as
   one. Never chain Jev calls.
2. **Eight concurrent requests.** Beyond that you buy latency, not throughput.
3. **Gate at 0.99, or do not gate.** 0.8 and 0.9 filter almost nothing useful.
4. **Always offer an explicit "none of these".** Without it the model forces
   out-of-scope input into your categories at 0.99 confidence.
5. **Use `noul` when you need a number.** Twice as well calibrated as a
   `choice`'s confidence. But do not threshold it at 0.5 — it is not
   reproducible near the boundary.
6. **Keep criteria descriptions short, or omit them — but never let them be
   wrong.** Absence costs 0.8 points; inversion costs 79.
7. **Fix your option order** for ambiguous tasks; it moves probabilities by up
   to 13 points.
8. **Do not put it in a sub-300 ms hot loop** from Europe. Let it judge
   slow-moving state instead.
9. **Do not use it to count** beyond a handful of items.
10. **Ignore the documentation's warnings about numbers, dates and negation.**
    Test them yourself — ours all passed.

---

*Pre-registration, raw data, and every analysis script are included. All 50
predictions and their outcomes — including the 21 we got wrong — are in
`paper/tables/predictions.md`. Corrections welcome; the harness is designed to be re-run
against them.*