# Results summary Generated by `uv run python -m evaljev.report` from `results/*/*.jsonl`. ## clinc (Jev `choice`, n=300) | model | accuracy [95% CI] | macro-F1 | ECE ↓ | Brier ↓ | AUROC ↑ | mean conf | p50 / p95 latency | $ per 1k calls | |---|---|---|---|---|---|---|---|---| | Jev | 0.897 [0.857, 0.926] | 0.887 | 0.039 | 0.065 | 0.835 | 0.926 | 0.16s / 0.37s | $0.0580 | | kev | 0.643 [0.588, 0.695] | 0.652 | 0.260 | 0.242 | 0.814 | 0.385 | 0.87s / 0.92s | $0.0000 | | Laya | 0.497 [0.440, 0.553] | 0.443 | 0.471 | 0.463 | 0.677 | 0.968 | 0.16s / 0.19s | $0.0000 | | Laya (head512, len1024) | 0.443 [0.388, 0.500] | 0.387 | 0.521 | 0.505 | 0.754 | 0.964 | 0.15s / 0.16s | $0.0000 | | open-jev | 0.610 [0.554, 0.663] | 0.622 | 0.072 | 0.187 | 0.780 | 0.653 | 0.43s / 0.50s | $0.0000 | Confidence distribution (median, lowest, and share of answers at or above each level): | model | median | lowest | ≥0.5 | ≥0.9 | ≥0.99 | |---|---|---|---|---|---| | Jev | 0.990 | 0.330 | 96% | 80% | 58% | | kev | 0.345 | 0.080 | 28% | 0% | 0% | | Laya | 1.000 | 0.378 | 100% | 90% | 82% | | Laya (head512, len1024) | 1.000 | 0.471 | 100% | 87% | 75% | | open-jev | 0.629 | 0.169 | 71% | 19% | 3% | Act only when confidence ≥ threshold (coverage / accuracy on covered): | model | ≥0.5 | ≥0.7 | ≥0.8 | ≥0.9 | ≥0.95 | |---|---|---|---|---|---| | Jev | 96% / 0.924 | 90% / 0.948 | 87% / 0.954 | 80% / 0.959 | 73% / 0.963 | | kev | 28% / 0.929 | 11% / 0.939 | 3% / 1.000 | 0% / 1.000 | 0% / – | | Laya | 100% / 0.498 | 95% / 0.510 | 94% / 0.512 | 90% / 0.522 | 87% / 0.538 | | Laya (head512, len1024) | 100% / 0.445 | 96% / 0.453 | 94% / 0.466 | 87% / 0.488 | 84% / 0.498 | | open-jev | 71% / 0.710 | 42% / 0.841 | 33% / 0.888 | 19% / 0.911 | 11% / 0.971 | Jev vs kev: Jev is 5.3x faster at p50; free to run, self-hosted; mean input tokens 1380 vs 652. Jev vs Laya: Laya is 1.0x faster at p50; free to run, self-hosted; mean input tokens 1380 vs 466. Jev vs Laya (head512, len1024): Laya (head512, len1024) is 1.1x faster at p50; free to run, self-hosted; mean input tokens 1380 vs 499. Jev vs open-jev: Jev is 2.6x faster at p50; free to run, self-hosted. ## intent (Jev `choice`, n=300) | model | accuracy [95% CI] | macro-F1 | ECE ↓ | Brier ↓ | AUROC ↑ | mean conf | p50 / p95 latency | $ per 1k calls | |---|---|---|---|---|---|---|---|---| | Jev | 0.780 [0.730, 0.823] | 0.763 | 0.110 | 0.143 | 0.831 | 0.888 | 0.20s / 0.44s | $0.0400 | | kev | 0.770 [0.719, 0.814] | 0.764 | 0.146 | 0.146 | 0.860 | 0.625 | 0.66s / 0.72s | $0.0000 | | Laya | 0.370 [0.317, 0.426] | 0.346 | 0.520 | 0.487 | 0.696 | 0.890 | 0.10s / 0.12s | $0.0000 | | Laya (head512, len1024) | 0.463 [0.408, 0.520] | 0.430 | 0.440 | 0.419 | 0.729 | 0.903 | 0.12s / 0.14s | $0.0000 | | open-jev | 0.873 [0.831, 0.906] | 0.865 | 0.047 | 0.073 | 0.894 | 0.879 | 0.27s / 0.35s | $0.0000 | | openai/gpt-5.6-terra | 0.847 [0.802, 0.883] | 0.835 | 0.081 | 0.110 | 0.807 | 0.927 | 1.04s / 1.65s | $2.0151 | | openai/gpt-5.6-terra (reasoning: medium) | 0.830 [0.783, 0.868] | 0.818 | 0.097 | 0.112 | 0.837 | 0.927 | 1.03s / 1.56s | $2.0084 | Confidence distribution (median, lowest, and share of answers at or above each level): | model | median | lowest | ≥0.5 | ≥0.9 | ≥0.99 | |---|---|---|---|---|---| | Jev | 0.980 | 0.350 | 96% | 68% | 47% | | kev | 0.690 | 0.080 | 65% | 20% | 0% | | Laya | 0.998 | 0.277 | 95% | 69% | 57% | | Laya (head512, len1024) | 1.000 | 0.181 | 95% | 75% | 62% | | open-jev | 0.977 | 0.073 | 91% | 73% | 34% | | openai/gpt-5.6-terra | 0.990 | 0.120 | 98% | 80% | 51% | | openai/gpt-5.6-terra (reasoning: medium) | 0.990 | 0.020 | 97% | 79% | 52% | Act only when confidence ≥ threshold (coverage / accuracy on covered): | model | ≥0.5 | ≥0.7 | ≥0.8 | ≥0.9 | ≥0.95 | |---|---|---|---|---|---| | Jev | 96% / 0.799 | 85% / 0.851 | 78% / 0.871 | 68% / 0.902 | 62% / 0.914 | | kev | 65% / 0.928 | 50% / 0.960 | 38% / 0.982 | 20% / 0.984 | 5% / 1.000 | | Laya | 95% / 0.385 | 83% / 0.416 | 76% / 0.443 | 69% / 0.457 | 65% / 0.469 | | Laya (head512, len1024) | 95% / 0.483 | 86% / 0.510 | 80% / 0.531 | 75% / 0.553 | 70% / 0.569 | | open-jev | 91% / 0.927 | 83% / 0.960 | 79% / 0.962 | 73% / 0.963 | 63% / 0.979 | | openai/gpt-5.6-terra | 98% / 0.863 | 95% / 0.877 | 88% / 0.898 | 80% / 0.909 | 70% / 0.929 | | openai/gpt-5.6-terra (reasoning: medium) | 97% / 0.856 | 94% / 0.876 | 88% / 0.894 | 79% / 0.916 | 72% / 0.930 | Jev vs kev: Jev is 3.4x faster at p50; free to run, self-hosted; mean input tokens 952 vs 450. Jev vs Laya: Laya is 2.0x faster at p50; free to run, self-hosted; mean input tokens 952 vs 313. Jev vs Laya (head512, len1024): Laya (head512, len1024) is 1.7x faster at p50; free to run, self-hosted; mean input tokens 952 vs 373. Jev vs open-jev: Jev is 1.4x faster at p50; free to run, self-hosted. Jev vs openai/gpt-5.6-terra: Jev is 5.3x faster at p50; Jev is 50x cheaper per call; mean input tokens 952 vs 874. Jev vs openai/gpt-5.6-terra (reasoning: medium): Jev is 5.2x faster at p50; Jev is 50x cheaper per call; mean input tokens 952 vs 874. Jev rerun on identical inputs: same label 98.3%, mean max |Δprob| 0.015. ## polarity (Jev `noul`, n=300) | model | accuracy [95% CI] | macro-F1 | ECE ↓ | Brier ↓ | AUROC ↑ | mean conf | p50 / p95 latency | $ per 1k calls | |---|---|---|---|---|---|---|---|---| | Jev | 0.970 [0.944, 0.984] | 0.970 | 0.042 | 0.028 | 0.935 | 0.928 | 0.20s / 0.43s | $0.0211 | | kev | 0.953 [0.923, 0.972] | 0.953 | 0.030 | 0.038 | 0.929 | 0.959 | 0.37s / 0.60s | $0.0000 | | Laya | 0.507 [0.450, 0.563] | 0.348 | 0.496 | 0.494 | 0.897 | 0.997 | 0.08s / 0.14s | $0.0000 | | Laya (head512, len1024) | 0.507 [0.450, 0.563] | 0.348 | 0.496 | 0.494 | 0.897 | 0.997 | 0.10s / 0.16s | $0.0000 | | open-jev | 0.957 [0.927, 0.975] | 0.957 | 0.036 | 0.029 | 0.959 | 0.936 | 0.17s / 0.27s | $0.0000 | | openai/gpt-5.6-terra | 0.970 [0.944, 0.984] | 0.970 | 0.020 | 0.022 | 0.964 | 0.967 | 1.04s / 1.82s | $0.8549 | | openai/gpt-5.6-terra (reasoning: medium) | 0.967 [0.940, 0.982] | 0.967 | 0.014 | 0.021 | 0.975 | 0.967 | 0.99s / 1.77s | $0.8555 | Confidence distribution (median, lowest, and share of answers at or above each level): | model | median | lowest | ≥0.5 | ≥0.9 | ≥0.99 | |---|---|---|---|---|---| | Jev | 0.970 | 0.570 | 100% | 79% | 18% | | kev | 0.990 | 0.510 | 100% | 89% | 68% | | Laya | 1.000 | 0.629 | 100% | 99% | 99% | | Laya (head512, len1024) | 1.000 | 0.629 | 100% | 99% | 99% | | open-jev | 0.971 | 0.512 | 100% | 84% | 14% | | openai/gpt-5.6-terra | 0.990 | 0.510 | 100% | 92% | 78% | | openai/gpt-5.6-terra (reasoning: medium) | 0.990 | 0.510 | 100% | 92% | 78% | Act only when confidence ≥ threshold (coverage / accuracy on covered): | model | ≥0.5 | ≥0.7 | ≥0.8 | ≥0.9 | ≥0.95 | |---|---|---|---|---|---| | Jev | 100% / 0.970 | 95% / 0.982 | 90% / 0.989 | 79% / 1.000 | 66% / 1.000 | | kev | 100% / 0.953 | 96% / 0.962 | 92% / 0.968 | 89% / 0.992 | 86% / 0.992 | | Laya | 100% / 0.507 | 99% / 0.503 | 99% / 0.503 | 99% / 0.503 | 99% / 0.503 | | Laya (head512, len1024) | 100% / 0.507 | 99% / 0.503 | 99% / 0.503 | 99% / 0.503 | 99% / 0.503 | | open-jev | 100% / 0.957 | 95% / 0.979 | 91% / 0.993 | 84% / 0.996 | 73% / 0.995 | | openai/gpt-5.6-terra | 100% / 0.970 | 99% / 0.976 | 94% / 0.986 | 92% / 0.996 | 87% / 1.000 | | openai/gpt-5.6-terra (reasoning: medium) | 100% / 0.967 | 99% / 0.976 | 96% / 0.986 | 92% / 0.993 | 86% / 1.000 | Jev vs kev: Jev is 1.9x faster at p50; free to run, self-hosted; mean input tokens 502 vs 228. Jev vs Laya: Laya is 2.4x faster at p50; free to run, self-hosted; mean input tokens 502 vs 235. Jev vs Laya (head512, len1024): Laya (head512, len1024) is 2.1x faster at p50; free to run, self-hosted; mean input tokens 502 vs 235. Jev vs open-jev: open-jev is 1.1x faster at p50; free to run, self-hosted. Jev vs openai/gpt-5.6-terra: Jev is 5.2x faster at p50; Jev is 41x cheaper per call; mean input tokens 502 vs 311. Jev vs openai/gpt-5.6-terra (reasoning: medium): Jev is 5.0x faster at p50; Jev is 41x cheaper per call; mean input tokens 502 vs 311. Jev rerun on identical inputs: same label 100.0%, mean max |Δprob| 0.003. ## polarity-choice (Jev `choice`, n=300) | model | accuracy [95% CI] | macro-F1 | ECE ↓ | Brier ↓ | AUROC ↑ | mean conf | p50 / p95 latency | $ per 1k calls | |---|---|---|---|---|---|---|---|---| | Jev | 0.967 [0.940, 0.982] | 0.967 | 0.022 | 0.026 | 0.909 | 0.985 | 0.18s / 0.39s | $0.0208 | | kev | 0.950 [0.919, 0.969] | 0.950 | 0.023 | 0.042 | 0.906 | 0.962 | 0.37s / 0.56s | $0.0000 | | Laya | 0.947 [0.915, 0.967] | 0.947 | 0.020 | 0.042 | 0.885 | 0.947 | 0.08s / 0.12s | $0.0000 | | open-jev | 0.963 [0.936, 0.979] | 0.963 | 0.009 | 0.026 | 0.930 | 0.960 | 0.15s / 0.20s | $0.0000 | Confidence distribution (median, lowest, and share of answers at or above each level): | model | median | lowest | ≥0.5 | ≥0.9 | ≥0.99 | |---|---|---|---|---|---| | Jev | 1.000 | 0.510 | 100% | 95% | 89% | | kev | 0.990 | 0.540 | 100% | 91% | 75% | | Laya | 0.986 | 0.514 | 100% | 87% | 33% | | open-jev | 0.990 | 0.521 | 100% | 90% | 51% | Act only when confidence ≥ threshold (coverage / accuracy on covered): | model | ≥0.5 | ≥0.7 | ≥0.8 | ≥0.9 | ≥0.95 | |---|---|---|---|---|---| | Jev | 100% / 0.967 | 98% / 0.976 | 98% / 0.976 | 95% / 0.979 | 93% / 0.986 | | kev | 100% / 0.950 | 96% / 0.958 | 93% / 0.964 | 91% / 0.978 | 86% / 0.988 | | Laya | 100% / 0.947 | 95% / 0.958 | 92% / 0.968 | 87% / 0.981 | 75% / 0.982 | | open-jev | 100% / 0.963 | 96% / 0.979 | 93% / 0.986 | 90% / 0.993 | 85% / 0.996 | Jev vs kev: Jev is 2.1x faster at p50; free to run, self-hosted; mean input tokens 495 vs 219. Jev vs Laya: Laya is 2.3x faster at p50; free to run, self-hosted; mean input tokens 495 vs 224. Jev vs open-jev: open-jev is 1.2x faster at p50; free to run, self-hosted. ## sentiment5 (Jev `score`, n=300) | model | accuracy [95% CI] | macro-F1 | ECE ↓ | Brier ↓ | AUROC ↑ | mean conf | p50 / p95 latency | $ per 1k calls | |---|---|---|---|---|---|---|---|---| | Jev | 0.570 [0.513, 0.625] | 0.559 | 0.200 | 0.284 | 0.611 | 0.767 | 0.19s / 0.40s | $0.0143 | | kev | 0.510 [0.454, 0.566] | 0.509 | 0.110 | 0.257 | 0.594 | 0.620 | 0.13s / 0.20s | $0.0000 | | Laya | 0.310 [0.260, 0.364] | 0.252 | 0.317 | 0.288 | 0.711 | 0.627 | 0.04s / 0.05s | $0.0000 | | Laya (head512, len1024) | 0.310 [0.260, 0.364] | 0.252 | 0.317 | 0.288 | 0.711 | 0.627 | 0.06s / 0.15s | $0.0000 | | open-jev | 0.560 [0.503, 0.615] | 0.551 | 0.052 | 0.240 | 0.595 | 0.601 | 0.07s / 0.12s | $0.0000 | | openai/gpt-5.6-terra | 0.593 [0.537, 0.647] | 0.558 | 0.303 | 0.319 | 0.636 | 0.893 | 1.06s / 2.57s | $0.6399 | | openai/gpt-5.6-terra (reasoning: medium) | 0.577 [0.520, 0.631] | 0.539 | 0.317 | 0.339 | 0.599 | 0.893 | 1.07s / 2.43s | $0.6276 | Ordinal MAE (levels off, lower is better): Jev 0.470, kev 0.570, Laya 1.053, Laya (head512, len1024) 1.053, open-jev 0.470, openai/gpt-5.6-terra 0.447, openai/gpt-5.6-terra (reasoning: medium) 0.460 Confidence distribution (median, lowest, and share of answers at or above each level): | model | median | lowest | ≥0.5 | ≥0.9 | ≥0.99 | |---|---|---|---|---|---| | Jev | 0.800 | 0.340 | 92% | 31% | 6% | | kev | 0.620 | 0.360 | 82% | 0% | 0% | | Laya | 0.606 | 0.283 | 75% | 8% | 0% | | Laya (head512, len1024) | 0.606 | 0.283 | 75% | 8% | 0% | | open-jev | 0.596 | 0.363 | 82% | 0% | 0% | | openai/gpt-5.6-terra | 0.920 | 0.480 | 100% | 67% | 7% | | openai/gpt-5.6-terra (reasoning: medium) | 0.915 | 0.540 | 100% | 67% | 6% | Act only when confidence ≥ threshold (coverage / accuracy on covered): | model | ≥0.5 | ≥0.7 | ≥0.8 | ≥0.9 | ≥0.95 | |---|---|---|---|---|---| | Jev | 92% / 0.582 | 65% / 0.619 | 51% / 0.645 | 31% / 0.702 | 21% / 0.719 | | kev | 82% / 0.538 | 29% / 0.563 | 8% / 0.720 | 0% / – | 0% / – | | Laya | 75% / 0.375 | 32% / 0.516 | 17% / 0.538 | 8% / 0.750 | 3% / 1.000 | | Laya (head512, len1024) | 75% / 0.375 | 32% / 0.516 | 17% / 0.538 | 8% / 0.750 | 3% / 1.000 | | open-jev | 82% / 0.580 | 17% / 0.731 | 2% / 1.000 | 0% / – | 0% / – | | openai/gpt-5.6-terra | 100% / 0.592 | 95% / 0.605 | 85% / 0.629 | 67% / 0.657 | 32% / 0.711 | | openai/gpt-5.6-terra (reasoning: medium) | 100% / 0.577 | 96% / 0.581 | 84% / 0.611 | 67% / 0.619 | 34% / 0.683 | Jev vs kev: kev is 1.5x faster at p50; free to run, self-hosted; mean input tokens 340 vs 55. Jev vs Laya: Laya is 5.0x faster at p50; free to run, self-hosted; mean input tokens 340 vs 70. Jev vs Laya (head512, len1024): Laya (head512, len1024) is 3.3x faster at p50; free to run, self-hosted; mean input tokens 340 vs 70. Jev vs open-jev: open-jev is 2.7x faster at p50; free to run, self-hosted. Jev vs openai/gpt-5.6-terra: Jev is 5.5x faster at p50; Jev is 45x cheaper per call; mean input tokens 340 vs 162. Jev vs openai/gpt-5.6-terra (reasoning: medium): Jev is 5.5x faster at p50; Jev is 44x cheaper per call; mean input tokens 340 vs 162. Jev rerun on identical inputs: same label 96.7%, mean max |Δprob| 0.019. ## Spend (main result files) - Jev: $0.0462 - kev: $0.0000 - Laya: $0.0000 - Laya (head512, len1024): $0.0000 - open-jev: $0.0000 - openai/gpt-5.6-terra: $1.0530 - openai/gpt-5.6-terra (reasoning: medium): $1.0474 ![confidence](confidence.png) ![reliability](reliability.png) ![coverage](coverage.png)