# Optimization overview Each category and mode stands alone; there is no grand mean. Each operation cell shows candidate rank among selected entries (excluding the historical npm baseline), median speedup versus npm, leader name, and candidate/leader latency index (C/L; 1.00 is best). A scoped rank of 1/1 is not a claim of market leadership. Ranks use the median of process-level, equally weighted document geometric-mean ratios (npm latency / entry latency), matching the full experiment report. C/L is leader score / candidate score, a ratio of those category scores—not a median absolute latency or a separate paired estimate. Close/overlapping process ranges do not establish a statistically significant winner. Separate input files remain separate comparisons; source/corpus differences are not silently pooled. ## Node v26.12.0 / o200k_base / API return (length only) Source: `optimized-canonical-real-node-o200k.json`; SHA-256 `44fc9e0e4d04d9924aab508ca9df180bd8c8147edb7176f05a718b95dd5a2383`. | Category | Mode | Encode | Decode | Count | |---|---|---|---|---| | prose | repeated | #2/5; 1.61× npm; @goliapkg/tiktoken-wasm; 1.66× C/L | #2/5; 0.85× npm; ai-tokenizer; 1.24× C/L | #2/5; 1.66× npm; @goliapkg/tiktoken-wasm; 3.11× C/L | | prose | stream | #2/5; 1.56× npm; @goliapkg/tiktoken-wasm; 1.96× C/L | #2/5; 0.85× npm; ai-tokenizer; 1.28× C/L | #2/5; 1.57× npm; @goliapkg/tiktoken-wasm; 3.53× C/L | | code | repeated | #2/5; 1.64× npm; @goliapkg/tiktoken-wasm; 1.54× C/L | #2/5; 0.84× npm; ai-tokenizer; 1.28× C/L | #2/5; 1.71× npm; @goliapkg/tiktoken-wasm; 2.76× C/L | | code | stream | #2/5; 1.65× npm; @goliapkg/tiktoken-wasm; 1.79× C/L | #2/5; 0.89× npm; ai-tokenizer; 1.28× C/L | #2/5; 1.74× npm; @goliapkg/tiktoken-wasm; 2.92× C/L | | structured | repeated | #2/5; 1.78× npm; @goliapkg/tiktoken-wasm; 1.72× C/L | #2/5; 0.81× npm; ai-tokenizer; 1.32× C/L | #2/5; 1.84× npm; @goliapkg/tiktoken-wasm; 3.07× C/L | | structured | stream | #2/5; 1.69× npm; @goliapkg/tiktoken-wasm; 1.83× C/L | #2/5; 0.85× npm; ai-tokenizer; 1.32× C/L | #2/5; 1.76× npm; @goliapkg/tiktoken-wasm; 3.38× C/L | | multilingual | repeated | #2/5; 1.67× npm; @goliapkg/tiktoken-wasm; 1.28× C/L | #1/5; 1.05× npm; candidate; 1.00× C/L | #2/5; 1.69× npm; @goliapkg/tiktoken-wasm; 1.52× C/L | | multilingual | stream | #2/5; 1.58× npm; @goliapkg/tiktoken-wasm; 1.57× C/L | #1/5; 1.09× npm; candidate; 1.00× C/L | #2/5; 1.59× npm; @goliapkg/tiktoken-wasm; 1.94× C/L | | long-prose | repeated | #3/5; 1.27× npm; @goliapkg/tiktoken-wasm; 5.47× C/L | #1/5; 1.11× npm; candidate; 1.00× C/L | #4/5; 1.22× npm; @goliapkg/tiktoken-wasm; 8.99× C/L | | long-prose | stream | #4/5; 1.20× npm; @goliapkg/tiktoken-wasm; 5.54× C/L | #1/5; 1.17× npm; candidate; 1.00× C/L | #3/5; 1.26× npm; @goliapkg/tiktoken-wasm; 8.70× C/L | | long-code | repeated | #3/5; 1.27× npm; @goliapkg/tiktoken-wasm; 2.57× C/L | #2/5; 1.07× npm; ai-tokenizer; 1.04× C/L | #2/5; 1.33× npm; @goliapkg/tiktoken-wasm; 3.27× C/L | | long-code | stream | #4/5; 1.23× npm; @goliapkg/tiktoken-wasm; 2.56× C/L | #1/5; 1.05× npm; candidate; 1.00× C/L | #4/5; 1.24× npm; @goliapkg/tiktoken-wasm; 3.40× C/L | | long-structured | repeated | #2/5; 1.57× npm; @goliapkg/tiktoken-wasm; 2.02× C/L | #1/5; 1.06× npm; candidate; 1.00× C/L | #2/5; 1.48× npm; @goliapkg/tiktoken-wasm; 2.69× C/L | | long-structured | stream | #2/5; 1.46× npm; @goliapkg/tiktoken-wasm; 2.23× C/L | #2/5; 1.06× npm; ai-tokenizer; 1.00× C/L | #2/5; 1.49× npm; @goliapkg/tiktoken-wasm; 2.79× C/L | ### Loss ledger versus npm Every per-document median regression is included, even when ranges overlap. These rows compare median absolute latencies, independently of the paired category scores above. Durations are µs: median [min,max] over fresh processes; Δ is candidate minus npm. Small differences are descriptive, not proof of a regression. | Document | Mode | Operation | Candidate µs [range] | npm µs [range] | Δ µs | C/npm | |---|---|---|---:|---:|---:|---:| | support | stream | decode | 0.632 [0.562, 0.745] | 0.514 [0.501, 0.587] | 0.119 | 1.231× | | design | stream | decode | 0.647 [0.604, 0.705] | 0.530 [0.502, 0.618] | 0.117 | 1.222× | | python | repeated | decode | 0.562 [0.523, 0.614] | 0.461 [0.442, 0.528] | 0.101 | 1.219× | | cjk | repeated | decode | 0.794 [0.748, 0.896] | 0.652 [0.619, 0.740] | 0.141 | 1.216× | | chat-json | repeated | decode | 0.657 [0.637, 0.707] | 0.543 [0.520, 0.578] | 0.115 | 1.211× | | typescript | stream | decode | 0.804 [0.764, 0.841] | 0.665 [0.655, 0.993] | 0.139 | 1.209× | | python | stream | decode | 0.728 [0.651, 0.766] | 0.604 [0.564, 0.723] | 0.124 | 1.205× | | release | repeated | decode | 0.535 [0.525, 0.573] | 0.446 [0.438, 0.516] | 0.088 | 1.198× | | api-json | repeated | decode | 0.599 [0.586, 1.300] | 0.500 [0.495, 0.538] | 0.099 | 1.197× | | logs | repeated | decode | 0.883 [0.833, 0.957] | 0.744 [0.711, 0.838] | 0.139 | 1.187× | | cjk | stream | decode | 0.953 [0.895, 1.129] | 0.811 [0.766, 0.853] | 0.143 | 1.176× | | typescript | repeated | decode | 0.634 [0.628, 0.690] | 0.541 [0.533, 0.748] | 0.093 | 1.171× | | design | repeated | decode | 0.480 [0.460, 0.551] | 0.410 [0.399, 0.449] | 0.070 | 1.170× | | sql | repeated | decode | 0.513 [0.503, 0.585] | 0.439 [0.413, 0.516] | 0.074 | 1.170× | | european | repeated | decode | 0.697 [0.658, 0.863] | 0.598 [0.568, 0.620] | 0.099 | 1.165× | | chat-json | stream | decode | 0.811 [0.782, 0.938] | 0.702 [0.628, 0.767] | 0.109 | 1.155× | | support | repeated | decode | 0.463 [0.447, 0.518] | 0.401 [0.382, 0.425] | 0.062 | 1.155× | | api-json | stream | decode | 0.735 [0.710, 0.965] | 0.640 [0.616, 0.678] | 0.095 | 1.148× | | release | stream | decode | 0.680 [0.671, 0.772] | 0.595 [0.552, 0.736] | 0.085 | 1.144× | | european | stream | decode | 0.824 [0.810, 1.064] | 0.725 [0.677, 0.887] | 0.100 | 1.137× | | sql | stream | decode | 0.670 [0.638, 0.782] | 0.597 [0.525, 0.655] | 0.073 | 1.121× | | logs | stream | decode | 1.069 [1.027, 1.316] | 1.007 [0.877, 1.109] | 0.062 | 1.062× | ### Correctness probes Every timed document/variant passed exact IDs, count and round-trip checks in every process. Additional non-timed probe failures across **all rounds** follow; they do not silently disappear from the comparison. - `baseline` rounds 1, 2, 3, 4, 5: `{"count": false, "ids": false, "roundTrip": true, "text": " I'\u017fa"}` - `baseline` rounds 1, 2, 3, 4, 5: `{"count": false, "ids": false, "roundTrip": true, "text": "\ud88a\ude13?t"}` - `ai-tokenizer` rounds 1, 2, 3, 4, 5: `{"count": false, "ids": false, "roundTrip": true, "text": " I'\u017fa"}` - `ai-tokenizer` rounds 1, 2, 3, 4, 5: `{"count": false, "ids": false, "roundTrip": true, "text": "\ud88a\ude13?t"}` - `ai-tokenizer` rounds 1, 2, 3, 4, 5: `{"count": false, "ids": false, "roundTrip": false, "text": "\ufeffhello"}` - `ai-tokenizer` rounds 1, 2, 3, 4, 5: `{"count": false, "ids": false, "roundTrip": false, "text": "a\u0085b\ufeffc\u00a0d"}` - `bpe-lite` rounds 1, 2, 3, 4, 5: `{"count": false, "ids": false, "roundTrip": true, "text": " I'\u017fa"}` - `bpe-lite` rounds 1, 2, 3, 4, 5: `{"count": false, "ids": false, "roundTrip": true, "text": "\ud88a\ude13?t"}` - `@goliapkg/tiktoken-wasm` rounds 1, 2, 3, 4, 5: `{"count": false, "ids": false, "roundTrip": true, "text": " I'\u017fa"}` ### Provenance and limits - Source commit `c69fba65cac2a6ce8bcac3e2e88b204a1a51e054`; executed-build manifest SHA-256 `234366c4ce34cd74c1ea94d30a2147b029d47ff141437767c75da49f018fd9e1` (245 JavaScript files). Working-tree status: M AGENTS.md M README.md M benchmark/FINDINGS.md M benchmark/OPTIMIZATIONS.md M benchmark/README.md M benchmark/experiments/charts.py M benchmark/experiments/legacy-report.py M benchmark/experiments/publish.py ?? benchmark/OPTIMIZED-FINDINGS.md. - Harness manifest SHA-256 `a108a39bbed0edf8497446f4a535b9a763797542be56d286d79571fcfaa9ee37`; corpus v2 SHA-256 `2a3a4eb1b0b9fc382f0a616157cae01fb95a74cf727ba7427a8576141fcd0542`; document/variant manifest SHA-256 `fbbcafa939ab433c0424fd7c50ee88adfd9dba0846313a94a7cb0540b3e88089`. - Reported hardware: 8 CPUs (13th Gen Intel(R) Core(TM) i7-1360P), 15.54 GiB; linux/x64; 7.0.0-34-generic. - Runtime `/user/.vite-plus/js_runtime/node/26.12.0/bin/node`; Node compatibility version `v26.12.0`; Bun `none`. 5 processes/entry; warmup ≥100 ms/20 calls; measurement ≥300 ms/5 calls. - Baseline gpt-tokenizer 4.0.3; competitors: {"@goliapkg/tiktoken-wasm": "4.1.2", "ai-tokenizer": "1.0.7", "bpe-lite": "0.5.2", "tiktoken": "1.0.22"}. - Run 2026-10-15T05:30:48.793Z → 2026-10-15T05:49:25.097Z. Corpus: Hand-authored for this repository in October 2026; no third-party extracts. Small representative document shapes, not a claim to production traffic distribution. - Length consumption measures API-return latency and may leave lazy output strings unflattened; full consumption includes reading the entire output. These metrics are not pooled. The 32-variant stream is bounded, warmed and repeatedly cycled; it is not an unlimited unique/cold workload. First-use and memory diagnostics remain in the raw/full report and are not ranked here. Five process ranges are descriptive, not statistical significance. ## Node v26.12.0 / cl100k_base / API return (length only) Source: `optimized-canonical-real-node-cl100k.json`; SHA-256 `b33191baa2275ecc812a45873597b3decc79e843b9a2e0eb58244090cca21411`. | Category | Mode | Encode | Decode | Count | |---|---|---|---|---| | prose | repeated | #1/1; 1.67× npm; candidate; 1.00× C/L | #1/1; 0.85× npm; candidate; 1.00× C/L | #1/1; 1.70× npm; candidate; 1.00× C/L | | prose | stream | #1/1; 1.65× npm; candidate; 1.00× C/L | #1/1; 0.85× npm; candidate; 1.00× C/L | #1/1; 1.63× npm; candidate; 1.00× C/L | | code | repeated | #1/1; 1.99× npm; candidate; 1.00× C/L | #1/1; 0.81× npm; candidate; 1.00× C/L | #1/1; 2.03× npm; candidate; 1.00× C/L | | code | stream | #1/1; 1.69× npm; candidate; 1.00× C/L | #1/1; 0.84× npm; candidate; 1.00× C/L | #1/1; 1.83× npm; candidate; 1.00× C/L | | structured | repeated | #1/1; 2.04× npm; candidate; 1.00× C/L | #1/1; 0.93× npm; candidate; 1.00× C/L | #1/1; 2.01× npm; candidate; 1.00× C/L | | structured | stream | #1/1; 1.92× npm; candidate; 1.00× C/L | #1/1; 0.88× npm; candidate; 1.00× C/L | #1/1; 1.98× npm; candidate; 1.00× C/L | | multilingual | repeated | #1/1; 2.14× npm; candidate; 1.00× C/L | #1/1; 4.34× npm; candidate; 1.00× C/L | #1/1; 2.17× npm; candidate; 1.00× C/L | | multilingual | stream | #1/1; 2.02× npm; candidate; 1.00× C/L | #1/1; 5.71× npm; candidate; 1.00× C/L | #1/1; 1.94× npm; candidate; 1.00× C/L | | long-prose | repeated | #1/1; 1.30× npm; candidate; 1.00× C/L | #1/1; 1.15× npm; candidate; 1.00× C/L | #1/1; 1.35× npm; candidate; 1.00× C/L | | long-prose | stream | #1/1; 1.22× npm; candidate; 1.00× C/L | #1/1; 1.16× npm; candidate; 1.00× C/L | #1/1; 1.36× npm; candidate; 1.00× C/L | | long-code | repeated | #1/1; 1.46× npm; candidate; 1.00× C/L | #1/1; 1.06× npm; candidate; 1.00× C/L | #1/1; 1.35× npm; candidate; 1.00× C/L | | long-code | stream | #1/1; 1.31× npm; candidate; 1.00× C/L | #1/1; 0.97× npm; candidate; 1.00× C/L | #1/1; 1.47× npm; candidate; 1.00× C/L | | long-structured | repeated | #1/1; 1.67× npm; candidate; 1.00× C/L | #1/1; 1.03× npm; candidate; 1.00× C/L | #1/1; 1.54× npm; candidate; 1.00× C/L | | long-structured | stream | #1/1; 1.61× npm; candidate; 1.00× C/L | #1/1; 1.06× npm; candidate; 1.00× C/L | #1/1; 1.50× npm; candidate; 1.00× C/L | ### Loss ledger versus npm Every per-document median regression is included, even when ranges overlap. These rows compare median absolute latencies, independently of the paired category scores above. Durations are µs: median [min,max] over fresh processes; Δ is candidate minus npm. Small differences are descriptive, not proof of a regression. | Document | Mode | Operation | Candidate µs [range] | npm µs [range] | Δ µs | C/npm | |---|---|---|---:|---:|---:|---:| | design | stream | decode | 0.690 [0.656, 0.773] | 0.549 [0.533, 0.616] | 0.141 | 1.257× | | python | stream | decode | 0.730 [0.693, 0.910] | 0.585 [0.562, 0.693] | 0.146 | 1.249× | | python | repeated | decode | 0.586 [0.548, 0.661] | 0.473 [0.442, 0.498] | 0.112 | 1.238× | | typescript | repeated | decode | 0.691 [0.652, 0.841] | 0.569 [0.557, 0.758] | 0.122 | 1.215× | | api-json | stream | decode | 0.748 [0.709, 0.890] | 0.620 [0.613, 0.715] | 0.127 | 1.205× | | typescript | stream | decode | 0.851 [0.794, 1.026] | 0.708 [0.675, 0.790] | 0.144 | 1.203× | | release | repeated | decode | 0.587 [0.572, 0.651] | 0.489 [0.474, 0.534] | 0.098 | 1.201× | | support | repeated | decode | 0.519 [0.489, 0.593] | 0.437 [0.411, 0.475] | 0.081 | 1.186× | | sql | repeated | decode | 0.560 [0.529, 0.637] | 0.473 [0.426, 0.640] | 0.087 | 1.184× | | support | stream | decode | 0.640 [0.623, 0.894] | 0.544 [0.530, 0.555] | 0.097 | 1.178× | | chat-json | stream | decode | 0.806 [0.771, 0.825] | 0.685 [0.676, 0.734] | 0.121 | 1.177× | | sql | stream | decode | 0.676 [0.658, 0.831] | 0.574 [0.558, 0.599] | 0.101 | 1.176× | | design | repeated | decode | 0.511 [0.503, 0.650] | 0.436 [0.428, 0.467] | 0.075 | 1.173× | | release | stream | decode | 0.692 [0.660, 0.818] | 0.598 [0.563, 0.682] | 0.094 | 1.157× | | chat-json | repeated | decode | 0.645 [0.633, 0.657] | 0.570 [0.523, 0.762] | 0.075 | 1.132× | | api-json | repeated | decode | 0.580 [0.553, 0.614] | 0.513 [0.477, 0.581] | 0.067 | 1.131× | | logs | repeated | decode | 0.883 [0.865, 1.007] | 0.783 [0.758, 1.045] | 0.101 | 1.128× | | logs | stream | decode | 1.012 [0.952, 1.102] | 0.932 [0.915, 1.062] | 0.080 | 1.085× | | encoding.ts | stream | decode | 36.486 [33.198, 43.804] | 35.677 [35.003, 42.829] | 0.809 | 1.023× | | packages.json | stream | decode | 13.074 [11.676, 20.864] | 12.967 [12.581, 14.549] | 0.107 | 1.008× | ### Correctness probes Every timed document/variant passed exact IDs, count and round-trip checks in every process. Additional non-timed probe failures across **all rounds** follow; they do not silently disappear from the comparison. - `baseline` rounds 1, 2, 3, 4, 5: `{"count": false, "ids": false, "roundTrip": true, "text": "\ud88a\ude13?t"}` ### Provenance and limits - Source commit `c69fba65cac2a6ce8bcac3e2e88b204a1a51e054`; executed-build manifest SHA-256 `234366c4ce34cd74c1ea94d30a2147b029d47ff141437767c75da49f018fd9e1` (245 JavaScript files). Working-tree status: M AGENTS.md M README.md M benchmark/FINDINGS.md M benchmark/OPTIMIZATIONS.md M benchmark/README.md M benchmark/experiments/README.md M benchmark/experiments/charts.py M benchmark/experiments/legacy-report.py M benchmark/experiments/publish.py ?? benchmark/OPTIMIZED-FINDINGS.md ?? benchmark/experiments/profile-count.mjs. - Harness manifest SHA-256 `a108a39bbed0edf8497446f4a535b9a763797542be56d286d79571fcfaa9ee37`; corpus v2 SHA-256 `2a3a4eb1b0b9fc382f0a616157cae01fb95a74cf727ba7427a8576141fcd0542`; document/variant manifest SHA-256 `fbbcafa939ab433c0424fd7c50ee88adfd9dba0846313a94a7cb0540b3e88089`. - Reported hardware: 8 CPUs (13th Gen Intel(R) Core(TM) i7-1360P), 15.54 GiB; linux/x64; 7.0.0-34-generic. - Runtime `/user/.vite-plus/js_runtime/node/26.12.0/bin/node`; Node compatibility version `v26.12.0`; Bun `none`. 5 processes/entry; warmup ≥100 ms/20 calls; measurement ≥100 ms/5 calls. - Baseline gpt-tokenizer 4.0.3; competitors: {"tiktoken": "1.0.22"}. - Run 2026-10-15T06:00:54.516Z → 2026-10-15T06:04:04.231Z. Corpus: Hand-authored for this repository in October 2026; no third-party extracts. Small representative document shapes, not a claim to production traffic distribution. - Length consumption measures API-return latency and may leave lazy output strings unflattened; full consumption includes reading the entire output. These metrics are not pooled. The 32-variant stream is bounded, warmed and repeatedly cycled; it is not an unlimited unique/cold workload. First-use and memory diagnostics remain in the raw/full report and are not ranked here. Five process ranges are descriptive, not statistical significance. ## Bun 1.4.3 / o200k_base / API return (length only) Source: `optimized-canonical-real-bun-o200k_base.json`; SHA-256 `cab62516fcca9c7ec6a8815b30e53a459e2441f85e635af08ca9200cfd9a43f6`. | Category | Mode | Encode | Decode | Count | |---|---|---|---|---| | prose | repeated | #1/1; 1.31× npm; candidate; 1.00× C/L | #1/1; 0.94× npm; candidate; 1.00× C/L | #1/1; 1.31× npm; candidate; 1.00× C/L | | prose | stream | #1/1; 1.38× npm; candidate; 1.00× C/L | #1/1; 0.97× npm; candidate; 1.00× C/L | #1/1; 1.38× npm; candidate; 1.00× C/L | | code | repeated | #1/1; 1.44× npm; candidate; 1.00× C/L | #1/1; 1.02× npm; candidate; 1.00× C/L | #1/1; 1.36× npm; candidate; 1.00× C/L | | code | stream | #1/1; 1.40× npm; candidate; 1.00× C/L | #1/1; 0.95× npm; candidate; 1.00× C/L | #1/1; 1.35× npm; candidate; 1.00× C/L | | structured | repeated | #1/1; 1.74× npm; candidate; 1.00× C/L | #1/1; 1.01× npm; candidate; 1.00× C/L | #1/1; 1.56× npm; candidate; 1.00× C/L | | structured | stream | #1/1; 1.63× npm; candidate; 1.00× C/L | #1/1; 1.03× npm; candidate; 1.00× C/L | #1/1; 1.52× npm; candidate; 1.00× C/L | | multilingual | repeated | #1/1; 1.81× npm; candidate; 1.00× C/L | #1/1; 0.93× npm; candidate; 1.00× C/L | #1/1; 1.60× npm; candidate; 1.00× C/L | | multilingual | stream | #1/1; 1.77× npm; candidate; 1.00× C/L | #1/1; 0.95× npm; candidate; 1.00× C/L | #1/1; 1.56× npm; candidate; 1.00× C/L | | long-prose | repeated | #1/1; 1.17× npm; candidate; 1.00× C/L | #1/1; 1.21× npm; candidate; 1.00× C/L | #1/1; 1.15× npm; candidate; 1.00× C/L | | long-prose | stream | #1/1; 1.23× npm; candidate; 1.00× C/L | #1/1; 1.14× npm; candidate; 1.00× C/L | #1/1; 1.12× npm; candidate; 1.00× C/L | | long-code | repeated | #1/1; 1.37× npm; candidate; 1.00× C/L | #1/1; 1.08× npm; candidate; 1.00× C/L | #1/1; 1.35× npm; candidate; 1.00× C/L | | long-code | stream | #1/1; 1.50× npm; candidate; 1.00× C/L | #1/1; 1.22× npm; candidate; 1.00× C/L | #1/1; 1.34× npm; candidate; 1.00× C/L | | long-structured | repeated | #1/1; 1.54× npm; candidate; 1.00× C/L | #1/1; 1.17× npm; candidate; 1.00× C/L | #1/1; 1.55× npm; candidate; 1.00× C/L | | long-structured | stream | #1/1; 1.64× npm; candidate; 1.00× C/L | #1/1; 1.12× npm; candidate; 1.00× C/L | #1/1; 1.57× npm; candidate; 1.00× C/L | ### Loss ledger versus npm Every per-document median regression is included, even when ranges overlap. These rows compare median absolute latencies, independently of the paired category scores above. Durations are µs: median [min,max] over fresh processes; Δ is candidate minus npm. Small differences are descriptive, not proof of a regression. | Document | Mode | Operation | Candidate µs [range] | npm µs [range] | Δ µs | C/npm | |---|---|---|---:|---:|---:|---:| | support | repeated | decode | 0.908 [0.803, 0.994] | 0.810 [0.764, 0.889] | 0.098 | 1.120× | | release | stream | decode | 1.378 [1.222, 1.750] | 1.257 [1.190, 1.279] | 0.122 | 1.097× | | support | stream | decode | 1.153 [1.056, 1.220] | 1.052 [1.025, 1.378] | 0.101 | 1.096× | | design | repeated | decode | 0.976 [0.915, 1.124] | 0.900 [0.861, 0.981] | 0.075 | 1.083× | | european | repeated | decode | 1.231 [1.164, 1.238] | 1.142 [1.092, 1.382] | 0.089 | 1.078× | | logs | stream | decode | 1.940 [1.830, 2.339] | 1.819 [1.733, 2.931] | 0.121 | 1.066× | | european | stream | decode | 1.482 [1.373, 1.881] | 1.401 [1.357, 1.412] | 0.081 | 1.058× | | mixed | stream | decode | 2.424 [2.299, 2.623] | 2.294 [2.281, 2.336] | 0.131 | 1.057× | | cjk | stream | decode | 1.619 [1.542, 1.679] | 1.551 [1.454, 1.742] | 0.068 | 1.044× | | mixed | repeated | decode | 2.148 [2.048, 2.515] | 2.059 [2.000, 2.161] | 0.089 | 1.043× | | cjk | repeated | decode | 1.333 [1.200, 1.573] | 1.281 [1.249, 1.559] | 0.052 | 1.040× | | release | repeated | decode | 1.051 [1.029, 1.430] | 1.015 [0.950, 1.228] | 0.036 | 1.035× | | typescript | stream | decode | 1.462 [1.378, 2.052] | 1.420 [1.367, 1.440] | 0.042 | 1.030× | | sql | stream | decode | 1.195 [1.140, 1.299] | 1.165 [1.103, 1.351] | 0.030 | 1.026× | | typescript | repeated | decode | 1.224 [1.137, 1.459] | 1.197 [1.153, 1.295] | 0.027 | 1.023× | | chat-json | stream | decode | 1.545 [1.377, 2.200] | 1.535 [1.347, 1.663] | 0.011 | 1.007× | | python | repeated | decode | 1.042 [0.977, 1.211] | 1.036 [0.979, 1.084] | 0.007 | 1.007× | | logs | repeated | decode | 1.604 [1.579, 1.790] | 1.604 [1.575, 1.633] | 0.000 | 1.000× | ### Correctness probes Every timed document/variant passed exact IDs, count and round-trip checks in every process. Additional non-timed probe failures across **all rounds** follow; they do not silently disappear from the comparison. - `baseline` rounds 1, 2, 3, 4, 5: `{"count": false, "ids": false, "roundTrip": true, "text": " I'\u017fa"}` - `baseline` rounds 1, 2, 3, 4, 5: `{"count": false, "ids": false, "roundTrip": true, "text": "\ud88a\ude13?t"}` ### Provenance and limits - Source commit `c69fba65cac2a6ce8bcac3e2e88b204a1a51e054`; executed-build manifest SHA-256 `234366c4ce34cd74c1ea94d30a2147b029d47ff141437767c75da49f018fd9e1` (245 JavaScript files). Working-tree status: M AGENTS.md M README.md M benchmark/FINDINGS.md M benchmark/OPTIMIZATIONS.md M benchmark/README.md M benchmark/experiments/README.md M benchmark/experiments/charts.py M benchmark/experiments/legacy-report.py M benchmark/experiments/publish.py ?? benchmark/OPTIMIZED-FINDINGS.md ?? benchmark/experiments/profile-count.mjs. - Harness manifest SHA-256 `a108a39bbed0edf8497446f4a535b9a763797542be56d286d79571fcfaa9ee37`; corpus v2 SHA-256 `2a3a4eb1b0b9fc382f0a616157cae01fb95a74cf727ba7427a8576141fcd0542`; document/variant manifest SHA-256 `fbbcafa939ab433c0424fd7c50ee88adfd9dba0846313a94a7cb0540b3e88089`. - Reported hardware: 8 CPUs (13th Gen Intel(R) Core(TM) i7-1360P), 15.54 GiB; linux/x64; 7.0.0-34-generic. - Runtime `/user/.bun/bin/bun`; Node compatibility version `v26.3.0`; Bun `1.4.3`. 5 processes/entry; warmup ≥100 ms/20 calls; measurement ≥100 ms/5 calls. - Baseline gpt-tokenizer 4.0.3; competitors: {"tiktoken": "1.0.22"}. - Run 2026-10-15T06:04:24.238Z → 2026-10-15T06:07:36.988Z. Corpus: Hand-authored for this repository in October 2026; no third-party extracts. Small representative document shapes, not a claim to production traffic distribution. - Length consumption measures API-return latency and may leave lazy output strings unflattened; full consumption includes reading the entire output. These metrics are not pooled. The 32-variant stream is bounded, warmed and repeatedly cycled; it is not an unlimited unique/cold workload. First-use and memory diagnostics remain in the raw/full report and are not ranked here. Five process ranges are descriptive, not statistical significance. ## Bun 1.4.3 / cl100k_base / API return (length only) Source: `optimized-canonical-real-bun-cl100k_base.json`; SHA-256 `88b94a3a6d1c9e6b9e63dda708949c1c2d02823e1a8fdca564ca94b1dc2d9911`. | Category | Mode | Encode | Decode | Count | |---|---|---|---|---| | prose | repeated | #1/1; 1.49× npm; candidate; 1.00× C/L | #1/1; 1.03× npm; candidate; 1.00× C/L | #1/1; 1.43× npm; candidate; 1.00× C/L | | prose | stream | #1/1; 1.37× npm; candidate; 1.00× C/L | #1/1; 1.06× npm; candidate; 1.00× C/L | #1/1; 1.42× npm; candidate; 1.00× C/L | | code | repeated | #1/1; 1.64× npm; candidate; 1.00× C/L | #1/1; 1.04× npm; candidate; 1.00× C/L | #1/1; 1.65× npm; candidate; 1.00× C/L | | code | stream | #1/1; 1.65× npm; candidate; 1.00× C/L | #1/1; 0.94× npm; candidate; 1.00× C/L | #1/1; 1.67× npm; candidate; 1.00× C/L | | structured | repeated | #1/1; 1.68× npm; candidate; 1.00× C/L | #1/1; 0.86× npm; candidate; 1.00× C/L | #1/1; 1.67× npm; candidate; 1.00× C/L | | structured | stream | #1/1; 1.93× npm; candidate; 1.00× C/L | #1/1; 0.96× npm; candidate; 1.00× C/L | #1/1; 1.60× npm; candidate; 1.00× C/L | | multilingual | repeated | #1/1; 2.28× npm; candidate; 1.00× C/L | #1/1; 1.75× npm; candidate; 1.00× C/L | #1/1; 2.11× npm; candidate; 1.00× C/L | | multilingual | stream | #1/1; 2.13× npm; candidate; 1.00× C/L | #1/1; 2.08× npm; candidate; 1.00× C/L | #1/1; 1.96× npm; candidate; 1.00× C/L | | long-prose | repeated | #1/1; 1.21× npm; candidate; 1.00× C/L | #1/1; 1.06× npm; candidate; 1.00× C/L | #1/1; 1.20× npm; candidate; 1.00× C/L | | long-prose | stream | #1/1; 1.30× npm; candidate; 1.00× C/L | #1/1; 1.07× npm; candidate; 1.00× C/L | #1/1; 1.14× npm; candidate; 1.00× C/L | | long-code | repeated | #1/1; 1.83× npm; candidate; 1.00× C/L | #1/1; 1.27× npm; candidate; 1.00× C/L | #1/1; 1.62× npm; candidate; 1.00× C/L | | long-code | stream | #1/1; 1.75× npm; candidate; 1.00× C/L | #1/1; 1.03× npm; candidate; 1.00× C/L | #1/1; 1.59× npm; candidate; 1.00× C/L | | long-structured | repeated | #1/1; 1.89× npm; candidate; 1.00× C/L | #1/1; 1.14× npm; candidate; 1.00× C/L | #1/1; 1.54× npm; candidate; 1.00× C/L | | long-structured | stream | #1/1; 1.76× npm; candidate; 1.00× C/L | #1/1; 0.97× npm; candidate; 1.00× C/L | #1/1; 1.76× npm; candidate; 1.00× C/L | ### Loss ledger versus npm Every per-document median regression is included, even when ranges overlap. These rows compare median absolute latencies, independently of the paired category scores above. Durations are µs: median [min,max] over fresh processes; Δ is candidate minus npm. Small differences are descriptive, not proof of a regression. | Document | Mode | Operation | Candidate µs [range] | npm µs [range] | Δ µs | C/npm | |---|---|---|---:|---:|---:|---:| | logs | repeated | decode | 2.030 [1.576, 2.820] | 1.593 [1.549, 1.718] | 0.437 | 1.274× | | packages.json | repeated | decode | 29.306 [21.299, 33.856] | 26.513 [24.805, 33.349] | 2.793 | 1.105× | | python | stream | decode | 1.301 [1.226, 1.430] | 1.187 [1.129, 1.195] | 0.115 | 1.097× | | api-json | repeated | decode | 1.132 [1.033, 1.652] | 1.034 [1.009, 1.083] | 0.098 | 1.095× | | support | stream | decode | 1.161 [1.057, 1.319] | 1.076 [1.057, 1.713] | 0.086 | 1.080× | | chat-json | repeated | decode | 1.311 [1.204, 1.409] | 1.237 [1.157, 1.576] | 0.074 | 1.060× | | api-json | stream | decode | 1.449 [1.237, 1.505] | 1.407 [1.270, 1.487] | 0.042 | 1.030× | | packages.json | stream | decode | 25.940 [23.599, 50.277] | 25.409 [22.989, 34.443] | 0.531 | 1.021× | | european | stream | decode | 2.178 [1.996, 2.704] | 2.138 [2.029, 2.473] | 0.040 | 1.019× | | chat-json | stream | decode | 1.544 [1.416, 1.899] | 1.532 [1.484, 1.581] | 0.013 | 1.008× | | sql | repeated | decode | 0.978 [0.944, 1.065] | 0.970 [0.880, 1.246] | 0.008 | 1.008× | | release | stream | decode | 1.381 [1.278, 1.420] | 1.375 [1.209, 1.838] | 0.006 | 1.004× | ### Correctness probes Every timed document/variant passed exact IDs, count and round-trip checks in every process. Additional non-timed probe failures across **all rounds** follow; they do not silently disappear from the comparison. - `baseline` rounds 1, 2, 3, 4, 5: `{"count": false, "ids": false, "roundTrip": true, "text": "\ud88a\ude13?t"}` ### Provenance and limits - Source commit `c69fba65cac2a6ce8bcac3e2e88b204a1a51e054`; executed-build manifest SHA-256 `234366c4ce34cd74c1ea94d30a2147b029d47ff141437767c75da49f018fd9e1` (245 JavaScript files). Working-tree status: M AGENTS.md M README.md M benchmark/FINDINGS.md M benchmark/OPTIMIZATIONS.md M benchmark/README.md M benchmark/experiments/README.md M benchmark/experiments/charts.py M benchmark/experiments/legacy-report.py M benchmark/experiments/publish.py ?? benchmark/OPTIMIZED-FINDINGS.md ?? benchmark/experiments/profile-count.mjs. - Harness manifest SHA-256 `a108a39bbed0edf8497446f4a535b9a763797542be56d286d79571fcfaa9ee37`; corpus v2 SHA-256 `2a3a4eb1b0b9fc382f0a616157cae01fb95a74cf727ba7427a8576141fcd0542`; document/variant manifest SHA-256 `fbbcafa939ab433c0424fd7c50ee88adfd9dba0846313a94a7cb0540b3e88089`. - Reported hardware: 8 CPUs (13th Gen Intel(R) Core(TM) i7-1360P), 15.54 GiB; linux/x64; 7.0.0-34-generic. - Runtime `/user/.bun/bin/bun`; Node compatibility version `v26.3.0`; Bun `1.4.3`. 5 processes/entry; warmup ≥100 ms/20 calls; measurement ≥100 ms/5 calls. - Baseline gpt-tokenizer 4.0.3; competitors: {"tiktoken": "1.0.22"}. - Run 2026-10-15T06:07:39.505Z → 2026-10-15T06:10:49.691Z. Corpus: Hand-authored for this repository in October 2026; no third-party extracts. Small representative document shapes, not a claim to production traffic distribution. - Length consumption measures API-return latency and may leave lazy output strings unflattened; full consumption includes reading the entire output. These metrics are not pooled. The 32-variant stream is bounded, warmed and repeatedly cycled; it is not an unlimited unique/cold workload. First-use and memory diagnostics remain in the raw/full report and are not ranked here. Five process ranges are descriptive, not statistical significance.