[Raw timing](../results/optimized-canonical-node-o200k.json) · [Raw settled memory](../results/optimized-canonical-memory-node-o200k.json) # Scoped legacy benchmark results node 26.12.0; benchmark-vm; linux/x64; kernel 7.0.0-34-generic. 8 vCPUs; 13th Gen Intel(R) Core(TM) i7-1360P; 15.543449401855469 GiB RAM Exactly five fresh processes per tokenizer/encoding. Values are median [minimum, maximum]; lower is better. Ranks follow medians; overlapping ranges do not establish a distinct winner. The npm package is an unranked historical reference. Timed correctness mismatches are shown but excluded from every ranking. These are API-return timings: the checksum reads output length, so decoded strings can remain lazy ropes. Decode plus a complete character read is a separate diagnostic, not included in these ranks. Eligibility means the seven timed inputs match. Additional probe flags here cover only the five original probes in this raw file; expanded Unicode/casefold probes in the realistic reports expose further mismatches, including the npm reference and Golia. See the optimized interpretation for the complete limitations. **The seven-input arithmetic mean is dominated by synthetic LatinRepeat and UnicodeRandom. It is a legacy compatibility score, not a realistic-workload or universal winner claim.** Inspect every input below and the separately published realistic categories/modes. Warmup: ≥100 ms and ≥5 calls per operation; measurement: ≥300 ms and ≥5 calls. Default caches and repeated inputs; load excludes process startup. Filesystem caches are warm; CPU affinity/frequency are uncontrolled on the VM. ## Evidence and selected scope - Timing SHA-256: `4bf346a8312a45e7bb2913365306e40938dfa9390b2e6699f7fbffebfbd25f84` - Source commit: `0aa625fda20984ea0f874117496f830ed46c33c8`; tracked diff SHA-256: `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` - Executed-build manifest SHA-256 (canonical JSON, sorted keys): `234366c4ce34cd74c1ea94d30a2147b029d47ff141437767c75da49f018fd9e1` - Core module SHA-256: `cc7036d186ecc6eda5df557fdd2ae62bb0f96150081e7044169a52b66ed16f7f`; GptEncoding SHA-256: `748725e53a4a3c4c8ea3cfe68adea19309d748f0be592cf85d611aa3ada39b21` - Selected verification cases: `gpt-tokenizer:local/o200k_base`, `gpt-tokenizer:npm/o200k_base`, `tiktoken/o200k_base`, `js-tiktoken/o200k_base`, `@huggingface/tokenizers/o200k_base`, `tokenizers/o200k_base`, `@mlc-ai/web-tokenizers/o200k_base`, `@goliapkg/tiktoken-wasm/o200k_base`, `bpe-lite/o200k_base`, `ai-tokenizer/o200k_base`, `@lenml/tokenizers/o200k_base` - Memory SHA-256: `b535d8089f7c4629466578e685268b47298ace22434eb9828a91a03995551d6e`; protocol `settled-one-pass-v1`; source commit `c69fba65cac2a6ce8bcac3e2e88b204a1a51e054`. Memory uses separate fresh processes: settled initialization deltas, retained growth after one encode/decode/count pass, and absolute final/peak process footprint. Final/peak RSS include harness/corpus. Timing-loop memory diagnostics are superseded and unranked. ## o200k_base ### Legacy seven-input mean latency (ms/call) | Tokenizer | Version | Status | Encode | Decode | Count | |---|---|---|---:|---:|---:| | gpt-tokenizer:local | local | eligible | 3.939266 [3.886708, 4.017792]; #1/10 | 0.369134 [0.366673, 0.374896]; #1/10 | 3.165703 [3.146331, 3.216488]; #1/10 | | gpt-tokenizer:npm | 4.0.3 | reference | 7.009376 [6.789098, 7.349860]; unranked | 7.415774 [6.847371, 7.592825]; unranked | 5.928078 [5.857125, 5.952973]; unranked | | tiktoken | 1.0.22 | eligible | 20.027595 [19.587547, 20.948726]; #4/10 | 0.764138 [0.755903, 0.777324]; #3/10 | 20.033335 [19.841069, 20.680594]; #4/10 | | js-tiktoken | 1.0.21 | eligible; additional probe mismatch | 117.679167 [114.158865, 120.808651]; #7/10 | 2.110259 [2.074950, 2.196258]; #5/10 | 115.054667 [112.202620, 118.536172]; #7/10 | | @huggingface/tokenizers | 0.2.0 | eligible; additional probe mismatch | 126.729362 [124.126164, 127.609206]; #9/10 | 15.586874 [15.546140, 16.954894]; #9/10 | 126.784392 [125.266747, 130.067722]; #8/10 | | tokenizers | 0.23.2 | eligible | 53.630123 [51.128054, 54.292648]; #6/10 | 8.825701 [8.293159, 9.298809]; #7/10 | 50.356045 [47.609257, 51.354280]; #6/10 | | @mlc-ai/web-tokenizers | 0.1.6 | eligible | 48.068961 [47.420493, 49.345107]; #5/10 | 15.426194 [15.160725, 15.451187]; #8/10 | 47.966727 [47.480121, 49.162853]; #5/10 | | @goliapkg/tiktoken-wasm | 4.1.2 | eligible | 292.867775 [288.075334, 301.740654]; #10/10 | 0.734452 [0.729254, 0.739351]; #2/10 | 294.726124 [287.463728, 296.857704]; #10/10 | | bpe-lite | 0.5.2 | eligible | 4.968866 [4.890750, 5.069676]; #3/10 | 1.454841 [1.424967, 1.479059]; #4/10 | 4.173434 [4.108865, 4.242055]; #2/10 | | ai-tokenizer | 1.0.7 | eligible; additional probe mismatch | 4.797881 [4.587346, 5.476782]; #2/10 | 4.873862 [4.615163, 5.657505]; #6/10 | 4.650446 [4.621503, 4.715423]; #3/10 | | @lenml/tokenizers | 3.7.2 | eligible; additional probe mismatch | 126.541266 [124.356436, 128.152847]; #8/10 | 17.122514 [16.956161, 17.710552]; #10/10 | 126.866677 [125.317526, 129.317038]; #9/10 | ### Initialization | Tokenizer | Load ms | Init heap Δ MiB | Init RSS Δ MiB | |---|---:|---:|---:| | gpt-tokenizer:local | 99.047 [94.039, 108.243]; #1/10 | 16.114 [16.113, 16.114]; #5/10 | 45.512 [45.184, 45.887]; #1/10 | | gpt-tokenizer:npm | 97.692 [96.101, 101.253]; unranked | 16.002 [16.002, 16.002]; unranked | 44.961 [27.715, 45.215]; unranked | | tiktoken | 126.165 [121.180, 132.324]; #2/10 | 0.220 [0.220, 0.220]; #1/10 | 72.930 [71.605, 73.910]; #3/10 | | js-tiktoken | 508.688 [490.317, 517.042]; #8/10 | 70.235 [70.235, 70.235]; #8/10 | 157.922 [157.555, 158.609]; #8/10 | | @huggingface/tokenizers | 441.342 [421.267, 457.304]; #7/10 | 70.640 [70.640, 70.640]; #9/10 | 170.188 [169.352, 171.277]; #10/10 | | tokenizers | 360.157 [351.171, 370.287]; #5/10 | 0.229 [0.229, 0.229]; #2/10 | 88.367 [88.305, 88.871]; #4/10 | | @mlc-ai/web-tokenizers | 513.816 [509.896, 544.451]; #9/10 | 10.325 [10.325, 10.325]; #4/10 | 93.059 [92.402, 93.582]; #5/10 | | @goliapkg/tiktoken-wasm | 129.853 [124.371, 133.037]; #3/10 | 2.814 [2.813, 2.814]; #3/10 | 59.051 [45.934, 64.559]; #2/10 | | bpe-lite | 529.704 [523.918, 544.290]; #10/10 | 51.245 [51.244, 51.245]; #7/10 | 155.008 [152.164, 159.379]; #7/10 | | ai-tokenizer | 284.150 [277.835, 304.763]; #4/10 | 26.822 [26.822, 26.822]; #6/10 | 114.527 [114.445, 114.590]; #6/10 | | @lenml/tokenizers | 440.042 [427.215, 452.330]; #6/10 | 70.923 [70.923, 70.923]; #10/10 | 169.902 [169.152, 173.035]; #9/10 | ### Settled working memory | Tokenizer | Retained heap Δ MiB | Retained RSS Δ MiB | Final heap MiB | Final RSS MiB | One-pass peak RSS MiB | |---|---:|---:|---:|---:|---:| | gpt-tokenizer:local | 7.751 [7.751, 7.754]; #10/10 | 49.930 [48.664, 50.586]; #9/10 | 29.547 [29.547, 29.550]; #5/10 | 167.996 [166.762, 168.289]; #3/10 | 173.328 [172.098, 173.426]; #3/10 | | gpt-tokenizer:npm | 8.076 [8.076, 8.076]; unranked | 63.645 [62.711, 69.043]; unranked | 29.760 [29.760, 29.760]; unranked | 180.207 [168.469, 181.199]; unranked | 186.293 [172.594, 186.859]; unranked | | tiktoken | 0.080 [0.080, 0.080]; #3/10 | 2.500 [-15.016, 3.949]; #2/10 | 5.983 [5.983, 5.983]; #2/10 | 148.500 [131.371, 148.965]; #2/10 | 150.742 [149.152, 151.703]; #2/10 | | js-tiktoken | 0.129 [0.129, 0.129]; #4/10 | -2.059 [-2.980, -0.598]; #1/10 | 76.045 [76.045, 76.045]; #8/10 | 228.605 [227.996, 229.262]; #5/10 | 269.797 [269.152, 270.711]; #7/10 | | @huggingface/tokenizers | 3.403 [3.403, 3.403]; #7/10 | 20.723 [18.840, 21.766]; #7/10 | 79.725 [79.725, 79.725]; #9/10 | 263.758 [261.480, 264.527]; #9/10 | 290.840 [290.492, 292.180]; #9/10 | | tokenizers | 0.037 [0.037, 0.037]; #1/10 | 293.422 [284.195, 296.664]; #10/10 | 5.947 [5.947, 5.948]; #1/10 | 454.336 [445.266, 457.664]; #10/10 | 457.145 [448.074, 460.473]; #10/10 | | @mlc-ai/web-tokenizers | 0.187 [0.187, 0.187]; #5/10 | 11.500 [11.125, 15.750]; #4/10 | 16.194 [16.194, 16.194]; #4/10 | 177.090 [177.012, 181.309]; #4/10 | 184.125 [183.535, 184.449]; #4/10 | | @goliapkg/tiktoken-wasm | 0.068 [0.068, 0.068]; #2/10 | 11.625 [10.734, 13.125]; #5/10 | 8.564 [8.563, 8.564]; #3/10 | 143.191 [131.102, 148.012]; #1/10 | 144.375 [131.301, 148.027]; #1/10 | | bpe-lite | 7.562 [7.555, 7.562]; #8/10 | 5.316 [3.320, 6.723]; #3/10 | 64.488 [64.481, 64.488]; #7/10 | 232.707 [231.500, 234.652]; #7/10 | 254.363 [251.781, 258.277]; #6/10 | | ai-tokenizer | 7.656 [7.654, 7.656]; #9/10 | 45.070 [45.035, 46.316]; #8/10 | 40.160 [40.158, 40.160]; #6/10 | 232.363 [231.676, 232.926]; #6/10 | 235.102 [234.375, 235.805]; #5/10 | | @lenml/tokenizers | 3.401 [3.401, 3.401]; #6/10 | 16.262 [13.840, 17.859]; #6/10 | 80.006 [80.006, 80.006]; #10/10 | 259.059 [256.543, 259.344]; #8/10 | 286.680 [286.176, 287.840]; #8/10 | ### Per-input latency (ms/call) #### English — encode #2/10, decode #2/10, count #2/10 | Tokenizer | Encode | Decode | Count | |---|---:|---:|---:| | gpt-tokenizer:local | 0.000658 [0.000644, 0.000681]; #2/10 | 0.000067 [0.000064, 0.000071]; #2/10 | 0.000637 [0.000610, 0.000750]; #2/10 | | gpt-tokenizer:npm | 0.001551 [0.001408, 0.001615]; unranked | 0.000060 [0.000054, 0.000063]; unranked | 0.001409 [0.001387, 0.001453]; unranked | | tiktoken | 0.042830 [0.042281, 0.047416]; #10/10 | 0.001477 [0.001254, 0.001544]; #6/10 | 0.040365 [0.039419, 0.043824]; #10/10 | | js-tiktoken | 0.012173 [0.011607, 0.013115]; #5/10 | 0.001315 [0.000960, 0.001345]; #5/10 | 0.012577 [0.011836, 0.013188]; #5/10 | | @huggingface/tokenizers | 0.017020 [0.015804, 0.017273]; #7/10 | 0.004027 [0.003974, 0.004109]; #7/10 | 0.015923 [0.015836, 0.016626]; #6/10 | | tokenizers | 0.030511 [0.029663, 0.045443]; #9/10 | 0.016956 [0.014875, 0.021415]; #10/10 | 0.034666 [0.032689, 0.039770]; #9/10 | | @mlc-ai/web-tokenizers | 0.017049 [0.017002, 0.018406]; #8/10 | 0.005608 [0.005452, 0.005797]; #9/10 | 0.016505 [0.016303, 0.016955]; #7/10 | | @goliapkg/tiktoken-wasm | 0.000411 [0.000401, 0.000433]; #1/10 | 0.000348 [0.000343, 0.000392]; #3/10 | 0.000156 [0.000155, 0.000161]; #1/10 | | bpe-lite | 0.000807 [0.000785, 0.000811]; #3/10 | 0.000352 [0.000348, 0.000358]; #4/10 | 0.000792 [0.000754, 0.000809]; #3/10 | | ai-tokenizer | 0.001733 [0.001716, 0.001858]; #4/10 | 0.000065 [0.000064, 0.000067]; #1/10 | 0.001823 [0.001814, 0.002048]; #4/10 | | @lenml/tokenizers | 0.016730 [0.015806, 0.018762]; #6/10 | 0.004495 [0.004421, 0.004650]; #8/10 | 0.016558 [0.015682, 0.016647]; #8/10 | #### Chinese — encode #2/10, decode #1/10, count #3/10 | Tokenizer | Encode | Decode | Count | |---|---:|---:|---:| | gpt-tokenizer:local | 0.000555 [0.000546, 0.000560]; #2/10 | 0.000168 [0.000160, 0.000179]; #1/10 | 0.000523 [0.000521, 0.000550]; #3/10 | | gpt-tokenizer:npm | 0.001297 [0.001245, 0.001446]; unranked | 0.003165 [0.002802, 0.003387]; unranked | 0.001195 [0.001174, 0.001298]; unranked | | tiktoken | 0.040370 [0.036802, 0.040807]; #9/10 | 0.001551 [0.001263, 0.001565]; #5/10 | 0.039388 [0.035918, 0.040824]; #9/10 | | js-tiktoken | 0.099580 [0.097793, 0.112314]; #10/10 | 0.001295 [0.001112, 0.001386]; #4/10 | 0.100024 [0.097981, 0.104970]; #10/10 | | @huggingface/tokenizers | 0.007059 [0.006867, 0.007136]; #6/10 | 0.003947 [0.003802, 0.004406]; #7/10 | 0.007045 [0.006877, 0.007086]; #6/10 | | tokenizers | 0.034733 [0.028117, 0.041863]; #8/10 | 0.016926 [0.015560, 0.021011]; #10/10 | 0.033361 [0.027612, 0.035405]; #8/10 | | @mlc-ai/web-tokenizers | 0.010962 [0.010392, 0.011006]; #7/10 | 0.005672 [0.005514, 0.006803]; #9/10 | 0.010905 [0.010495, 0.012196]; #7/10 | | @goliapkg/tiktoken-wasm | 0.000634 [0.000573, 0.000659]; #3/10 | 0.000405 [0.000397, 0.000438]; #2/10 | 0.000411 [0.000406, 0.000459]; #1/10 | | bpe-lite | 0.000502 [0.000498, 0.000521]; #1/10 | 0.000439 [0.000429, 0.000459]; #3/10 | 0.000486 [0.000482, 0.000493]; #2/10 | | ai-tokenizer | 0.001539 [0.001500, 0.001546]; #4/10 | 0.002853 [0.002462, 0.003288]; #6/10 | 0.001504 [0.001498, 0.001601]; #4/10 | | @lenml/tokenizers | 0.006694 [0.006448, 0.006887]; #5/10 | 0.004372 [0.004277, 0.004507]; #8/10 | 0.006740 [0.006547, 0.006965]; #5/10 | #### French — encode #2/10, decode #2/10, count #2/10 | Tokenizer | Encode | Decode | Count | |---|---:|---:|---:| | gpt-tokenizer:local | 0.000735 [0.000725, 0.000823]; #2/10 | 0.000118 [0.000117, 0.000124]; #2/10 | 0.000704 [0.000681, 0.000742]; #2/10 | | gpt-tokenizer:npm | 0.001692 [0.001627, 0.001885]; unranked | 0.000097 [0.000095, 0.000099]; unranked | 0.001642 [0.001565, 0.001931]; unranked | | tiktoken | 0.041764 [0.041210, 0.042053]; #10/10 | 0.001456 [0.001335, 0.001571]; #6/10 | 0.041340 [0.040628, 0.041774]; #10/10 | | js-tiktoken | 0.025910 [0.025785, 0.026739]; #8/10 | 0.001350 [0.001190, 0.001532]; #5/10 | 0.025318 [0.025294, 0.026025]; #8/10 | | @huggingface/tokenizers | 0.018370 [0.017903, 0.018990]; #5/10 | 0.004883 [0.004768, 0.005142]; #7/10 | 0.018333 [0.017656, 0.019671]; #5/10 | | tokenizers | 0.040424 [0.038403, 0.048064]; #9/10 | 0.017167 [0.015872, 0.022294]; #10/10 | 0.039174 [0.035463, 0.047127]; #9/10 | | @mlc-ai/web-tokenizers | 0.019133 [0.018469, 0.019771]; #6/10 | 0.006985 [0.006793, 0.007468]; #9/10 | 0.018557 [0.018448, 0.022389]; #6/10 | | @goliapkg/tiktoken-wasm | 0.000501 [0.000497, 0.000511]; #1/10 | 0.000375 [0.000374, 0.000421]; #3/10 | 0.000235 [0.000233, 0.000247]; #1/10 | | bpe-lite | 0.000900 [0.000884, 0.000919]; #3/10 | 0.000439 [0.000431, 0.000449]; #4/10 | 0.000890 [0.000872, 0.000910]; #3/10 | | ai-tokenizer | 0.001964 [0.001900, 0.001990]; #4/10 | 0.000098 [0.000095, 0.000106]; #1/10 | 0.001948 [0.001912, 0.001998]; #4/10 | | @lenml/tokenizers | 0.019433 [0.018794, 0.021001]; #7/10 | 0.005567 [0.005357, 0.005679]; #8/10 | 0.018810 [0.018446, 0.021195]; #7/10 | #### Code — encode #1/10, decode #2/10, count #2/10 | Tokenizer | Encode | Decode | Count | |---|---:|---:|---:| | gpt-tokenizer:local | 0.001116 [0.001104, 0.001208]; #1/10 | 0.000136 [0.000133, 0.000146]; #2/10 | 0.001073 [0.001020, 0.001118]; #2/10 | | gpt-tokenizer:npm | 0.002138 [0.002124, 0.003388]; unranked | 0.000116 [0.000113, 0.000260]; unranked | 0.002055 [0.002024, 0.002728]; unranked | | tiktoken | 0.043992 [0.043854, 0.044187]; #9/10 | 0.001532 [0.001356, 0.001655]; #6/10 | 0.044158 [0.043640, 0.047341]; #10/10 | | js-tiktoken | 0.021772 [0.021621, 0.022764]; #5/10 | 0.001458 [0.001405, 0.001479]; #5/10 | 0.021987 [0.021452, 0.022299]; #5/10 | | @huggingface/tokenizers | 0.026926 [0.026426, 0.027524]; #7/10 | 0.005794 [0.005707, 0.006343]; #7/10 | 0.027597 [0.026359, 0.028403]; #7/10 | | tokenizers | 0.048435 [0.042846, 0.064453]; #10/10 | 0.020001 [0.016521, 0.025689]; #10/10 | 0.042878 [0.039446, 0.050642]; #9/10 | | @mlc-ai/web-tokenizers | 0.024988 [0.024519, 0.025595]; #6/10 | 0.008101 [0.008033, 0.008335]; #9/10 | 0.025185 [0.024733, 0.027780]; #6/10 | | @goliapkg/tiktoken-wasm | 0.001442 [0.001361, 0.001521]; #3/10 | 0.000417 [0.000386, 0.000425]; #3/10 | 0.000378 [0.000372, 0.000390]; #1/10 | | bpe-lite | 0.001393 [0.001356, 0.001420]; #2/10 | 0.000548 [0.000538, 0.000580]; #4/10 | 0.001341 [0.001295, 0.001488]; #3/10 | | ai-tokenizer | 0.002427 [0.002391, 0.002629]; #4/10 | 0.000120 [0.000111, 0.000122]; #1/10 | 0.002382 [0.002338, 0.002425]; #4/10 | | @lenml/tokenizers | 0.026969 [0.026489, 0.029083]; #8/10 | 0.006480 [0.006402, 0.006625]; #8/10 | 0.028366 [0.026920, 0.030404]; #8/10 | #### Latin — encode #2/10, decode #2/10, count #2/10 | Tokenizer | Encode | Decode | Count | |---|---:|---:|---:| | gpt-tokenizer:local | 0.003971 [0.003888, 0.004200]; #2/10 | 0.000583 [0.000567, 0.000612]; #2/10 | 0.003981 [0.003715, 0.004177]; #2/10 | | gpt-tokenizer:npm | 0.007337 [0.007284, 0.011217]; unranked | 0.000484 [0.000472, 0.000521]; unranked | 0.006974 [0.006830, 0.007301]; unranked | | tiktoken | 0.079310 [0.078385, 0.081698]; #5/10 | 0.001969 [0.001876, 0.002142]; #4/10 | 0.079355 [0.078496, 0.081461]; #5/10 | | js-tiktoken | 0.152981 [0.149371, 0.185223]; #10/10 | 0.002699 [0.002635, 0.002829]; #6/10 | 0.153450 [0.147522, 0.167700]; #10/10 | | @huggingface/tokenizers | 0.103644 [0.101084, 0.120711]; #7/10 | 0.029027 [0.028776, 0.030144]; #7/10 | 0.105265 [0.103614, 0.115497]; #7/10 | | tokenizers | 0.118822 [0.101062, 0.167821]; #9/10 | 0.030299 [0.026456, 0.038383]; #8/10 | 0.118767 [0.110848, 0.131102]; #9/10 | | @mlc-ai/web-tokenizers | 0.091648 [0.090279, 0.092188]; #6/10 | 0.032697 [0.032541, 0.033049]; #9/10 | 0.092508 [0.090274, 0.099322]; #6/10 | | @goliapkg/tiktoken-wasm | 0.002866 [0.002846, 0.002945]; #1/10 | 0.001052 [0.001035, 0.001177]; #3/10 | 0.001509 [0.001504, 0.001539]; #1/10 | | bpe-lite | 0.005213 [0.005110, 0.005449]; #3/10 | 0.002104 [0.001875, 0.002222]; #5/10 | 0.005098 [0.004897, 0.005210]; #3/10 | | ai-tokenizer | 0.006476 [0.006172, 0.006648]; #4/10 | 0.000461 [0.000461, 0.000516]; #1/10 | 0.006331 [0.006252, 0.006834]; #4/10 | | @lenml/tokenizers | 0.110961 [0.107929, 0.120507]; #8/10 | 0.033433 [0.033140, 0.036693]; #10/10 | 0.112976 [0.105574, 0.120623]; #8/10 | #### LatinRepeat — encode #2/10, decode #1/10, count #2/10 | Tokenizer | Encode | Decode | Count | |---|---:|---:|---:| | gpt-tokenizer:local | 8.944599 [8.457275, 9.350434]; #2/10 | 1.191523 [1.159061, 1.201190]; #1/10 | 7.880947 [7.854302, 8.077399]; #2/10 | | gpt-tokenizer:npm | 14.315549 [14.073369, 15.218194]; unranked | 1.240149 [1.225996, 1.286071]; unranked | 12.515727 [12.192168, 13.830415]; unranked | | tiktoken | 90.871416 [88.342976, 96.063967]; #5/10 | 1.629830 [1.602898, 1.676787]; #4/10 | 90.916797 [88.963374, 92.358518]; #5/10 | | js-tiktoken | 296.904801 [292.491900, 305.725191]; #10/10 | 4.175607 [4.056356, 4.453233]; #6/10 | 290.831691 [284.364473, 297.750044]; #9/10 | | @huggingface/tokenizers | 283.254165 [279.490904, 294.365641]; #8/10 | 68.480759 [67.580878, 77.540416]; #9/10 | 294.005495 [283.104028, 304.041677]; #10/10 | | tokenizers | 229.108069 [217.015189, 235.060324]; #7/10 | 30.784029 [25.542740, 34.010605]; #7/10 | 207.401200 [193.041221, 213.774018]; #7/10 | | @mlc-ai/web-tokenizers | 176.668065 [174.581277, 180.446920]; #6/10 | 57.680789 [57.335018, 58.761432]; #8/10 | 178.252114 [174.712294, 185.275267]; #6/10 | | @goliapkg/tiktoken-wasm | 4.536014 [4.517880, 4.591648]; #1/10 | 1.548203 [1.532178, 1.581959]; #3/10 | 2.922106 [2.880501, 2.968399]; #1/10 | | bpe-lite | 11.457217 [10.782993, 12.794838]; #4/10 | 3.875014 [3.781541, 4.066021]; #5/10 | 10.217061 [10.042406, 11.043119]; #3/10 | | ai-tokenizer | 11.223174 [10.962089, 12.169882]; #3/10 | 1.219971 [1.196177, 1.266989]; #2/10 | 10.918453 [10.205037, 11.228376]; #4/10 | | @lenml/tokenizers | 285.956494 [267.341345, 294.357319]; #9/10 | 74.577929 [74.270334, 79.133708]; #10/10 | 286.370800 [281.339658, 289.555606]; #8/10 | #### UnicodeRandom — encode #1/10, decode #1/10, count #1/10 | Tokenizer | Encode | Decode | Count | |---|---:|---:|---:| | gpt-tokenizer:local | 18.767174 [18.255345, 19.110631]; #1/10 | 1.403620 [1.391069, 1.422016]; #1/10 | 14.282411 [14.054637, 14.627398]; #1/10 | | gpt-tokenizer:npm | 34.978124 [33.358683, 36.211362]; unranked | 50.666598 [46.676844, 51.910472]; unranked | 28.932933 [27.652675, 29.186965]; unranked | | tiktoken | 49.073232 [48.525926, 50.333718]; #4/10 | 3.682648 [3.656520, 3.803888]; #3/10 | 49.652939 [48.801078, 53.009788]; #4/10 | | js-tiktoken | 525.409897 [506.297825, 545.774998]; #7/10 | 10.437604 [10.410481, 11.190018]; #5/10 | 516.394007 [500.747077, 531.678141]; #7/10 | | @huggingface/tokenizers | 597.599055 [589.201487, 606.213713]; #8/10 | 41.096590 [40.294511, 41.269169]; #8/10 | 595.355488 [582.689098, 606.257636]; #8/10 | | tokenizers | 144.715339 [140.519315, 147.059567]; #5/10 | 30.898506 [29.430634, 33.692284]; #6/10 | 140.948425 [138.096956, 145.440995]; #5/10 | | @mlc-ai/web-tokenizers | 159.705619 [157.146789, 164.806184]; #6/10 | 49.483287 [47.303644, 50.764449]; #10/10 | 157.744679 [156.612040, 160.787162]; #6/10 | | @goliapkg/tiktoken-wasm | 2045.546093 [2011.950531, 2107.660765]; #10/10 | 3.570019 [3.566261, 3.624322]; #2/10 | 2060.111696 [2009.318711, 2075.079117]; #10/10 | | bpe-lite | 23.209937 [22.684127, 24.142176]; #3/10 | 6.233157 [6.189228, 6.344616]; #4/10 | 18.995264 [18.536509, 19.162818]; #2/10 | | ai-tokenizer | 22.043345 [21.098752, 26.153787]; #2/10 | 32.899702 [31.106082, 38.376683]; #7/10 | 21.882238 [21.216339, 22.131523]; #3/10 | | @lenml/tokenizers | 598.977676 [590.687678, 612.493133]; #9/10 | 44.602884 [44.154289, 45.226029]; #9/10 | 601.967666 [590.667493, 615.474436]; #9/10 | ## Correctness mismatches - js-tiktoken/o200k_base: additional non-timed probe; `{"actualTokens": 2, "count": true, "dataset": "LeadingBOM", "firstDifference": -1, "roundTrip": false, "tokens": 2}` - @huggingface/tokenizers/o200k_base: additional non-timed probe; `{"actualTokens": 0, "count": true, "dataset": "Empty", "error": "decode: Error: token_ids must be a non-empty array of integers.", "firstDifference": -1, "roundTrip": false, "tokens": 0}` - ai-tokenizer/o200k_base: additional non-timed probe; `{"actualToken": 24912, "actualTokens": 1, "count": false, "dataset": "LeadingBOM", "expectedToken": 5574, "firstDifference": 0, "roundTrip": false, "tokens": 2}` - ai-tokenizer/o200k_base: additional non-timed probe; `{"actualToken": 66, "actualTokens": 9, "count": false, "dataset": "Whitespace", "expectedToken": 5574, "firstDifference": 4, "roundTrip": false, "tokens": 10}` - @lenml/tokenizers/o200k_base: additional non-timed probe; `{"actualTokens": 0, "count": true, "dataset": "Empty", "error": "decode: Error: token_ids must be a non-empty array of integers.", "firstDifference": -1, "roundTrip": false, "tokens": 0}` ## Input provenance | Input | Timed | UTF-8 bytes | SHA-256 | |---|---|---:|---| | English | True | 44 | `ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c` | | Chinese | True | 36 | `73e8fc331751af46aa8f690bf5e6ee5fcea46ce1608c77352e44b14db45aecd9` | | French | True | 58 | `0f7da063db4d9ed52d77a57dd253ba0ff7239ae815f4f4372453c5ad422aa435` | | Code | True | 68 | `19fe498d9585a6f125ad294203e8a68365c217ffb7e7faa4b57c0176d5c8b214` | | Latin | True | 400 | `29ac811ff96cbf6d0b6ee7bd6c6130035df43bbfb62e0f86690862f028a2f9b9` | | LatinRepeat | True | 800000 | `74b8cf1816add23ae12d9113505bf5b9119e23ecb9edada76ec57a0ad51da9a0` | | UnicodeRandom | True | 316539 | `c6811d447c600ddc4a59dd4cfb54deb465297aa4d668248183465cb25c52f00c` | | Empty | False | 0 | `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` | | LeadingBOM | False | 8 | `7489ebbcc2a00056ddaaaac190bce473e5c03696ea1bd8ed83cf59a174283862` | | Whitespace | False | 16 | `e69a9a2b008caa3d5c29b3642fbd0e7ec96005c3ad20960cd9562affee68d866` | | Contractions | False | 54 | `ada27981ddea95fe371e63204676bc94e6ca248cdd76b9d2b8a6d6af5364d373` | | LongPiece | False | 1024 | `2edc986847e209b4016e141a6dc8716d3207350f416969382d431539bf292e4a` |