# Tokenizer benchmarks

The current [optimized results](OPTIMIZED-RESULTS.md) and
[realistic category comparison](experiments/OVERVIEW.md) target Node/o200k, with
Bun/cl100k release checks. The root README embeds `docs/benchmark-summary.svg`, generated by `experiments/summary-chart.py`; the optimized PNGs remain linked from the detailed reports.
[RESULTS.md](RESULTS.md) preserves the earlier full contender/engine/encoding
matrix, with its historical `docs/fastest.png`, `docs/initialization.png`, and
`docs/lowest-footprint.png`. The old external CodeSandbox link is a historical
demo. There is no separate benchmark docs site.

## Publication privacy and historical provenance

Published machine labels and filesystem paths in JSON metadata are anonymized; runtime versions,
hardware, measurements, and recorded input/build/harness hashes are retained. A
`publicationPrivacy` record on redacted JSON preserves its original file digest.
Run `python3 anonymize.py results/PREFIX-*.json` before publishing new raw files.
Memory-to-timing file digests and generated report links refer to the sanitized
published files. New runs use `GPT_BENCH_HOST_ID` when set, otherwise a stable
SHA-256-derived machine pseudonym; no raw hostname is collected.

Source commit IDs in historical measurements identify local pre-publication
commits and may not be available in a fresh clone after publication cleanup.
The public source mapping is commit `7cd2bed` (the frozen production source).
Executed-module hashes remain the primary build identity. Snapshot regeneration
uses the self-contained `experiments/snapshot-sources/` extracts, not those Git IDs.

The archived realistic results measured **corpus v2**. Privacy cleanup replaced a
machine label inside its README input, so its exact original fixture is no longer
distributed. Its original measured input hashes remain unchanged in archived raw
files; they are not hashes of the sanitized replacement. The new README and its
manifest define **corpus v3**, with the original digest and transformation recorded
in `snapshot-sources/provenance.json`. New runs reproduce the sanitized v3 workload,
not byte-for-byte v2 inputs. Never pool or relabel v2 timings as v3 measurements.

## Reproduce

Performance commands must hold the machine-wide FIFO lease. Prefix timing and
memory commands below with `python3 queue.py --label NAME --`; do the same for
builds and tests while other worktrees are measuring. For example,
`python3 queue.py --label node-o200k -- node --expose-gc dist/benchmarkRunner.js`.
All worktrees share `/tmp/gpt-tokenizer-perf-queue-$UID`; `python3 queue.py --status`
shows the active job and waiting tickets. The command inherits the run lock, so
it stays held if the wrapper is killed. Nested commands reuse their parent's lease.

Build the checkout, then install the isolated benchmark project:

```sh
vp install --frozen-lockfile
vp run build
cd benchmark
vp install --frozen-lockfile
vp run test
vp env exec --node 26.12.0 node dist/benchmarkRunner.js --output results/node-26.12.0.json
bun dist/benchmarkRunner.js --output results/bun-1.4.3.json
vp env exec --node 26.12.0 node dist/memoryRunner.js
bun dist/memoryRunner.js
```

Use the Node 26 and Bun stable versions shown above for the published run. Run
engines **sequentially**, on an otherwise idle machine. `process.execPath` selects
the same engine for every worker. No runtime network requests occur after the
revision- and SHA-256-pinned HF fixtures have been prepared. These fixtures are
cached under ignored `fixtures/`; `fixtures.json` records their provenance. All
three generic HF implementations and MLC use exactly the same fixtures.

Options: `--rounds 5`, `--min-ms 300`, `--warm-ms 100`, `--warm-calls 5`, `--encodings cl100k_base,o200k_base`,
`--only tiktoken,gpt-tokenizer:local`, `--verify-only`, and `--output <path>`.
At least two independent timing rounds are required; the publication and memory
phase require five. Complete both timing runs before the memory runs, sequentially. A smoke run with smaller budgets
must not replace the published full run. Existing output files are rejected; use a new filename for subsequent runs. Memory accepts `--timing <path> --output <path>` and follows the selected timing scope. Both phases enforce exact engine versions and hash the executed local build; memory rejects a changed build or production source. Results are checkpointed after every
worker; worker failures are retained and cause a nonzero exit status.

To reproduce the frozen optimization comparison, use a fresh output prefix:

```sh
GPT_BENCH_CONFIRM=1 bash benchmark/experiments/reproduce.sh NEW_PREFIX # from repository root
```

This queues build/tests, expanded correctness gates, timing, memory, realistic
workloads, complete-output decode, and the separate longer confirmations. Omit
`GPT_BENCH_CONFIRM=1` for the first comparison without follow-ups. See
[the experiment protocol](experiments/README.md) for budgets and
selected scopes. The original complete matrix above is preserved separately.

To regenerate publication artifacts without re-running measurements (from
`benchmark/`):

```sh
python3 -m venv .venv
.venv/bin/pip install -r requirements-report.txt
.venv/bin/python experiments/publish.py --prefix optimized-canonical --charts
```

To regenerate the historical full matrix instead:

```sh
.venv/bin/python report.py
python3 test_report.py
```

`report.py` validates run completeness, input identity, minimum call counts and
measurement durations before publishing. It generates `RESULTS.md`, the three
README PNG charts, and ignored `results/summary.json` and `results/review.html`.
The latter is a self-contained interactive review (engine, encoding, input, and
metric selectors). Research decisions and interpretation live in this README
and `FINDINGS.md`; generated tables should not be edited by hand.

The [optimization investigation](OPTIMIZATIONS.md) and [realistic workload protocol](experiments/README.md) supplement this historical seven-input suite.
The [optimized interpretation](OPTIMIZED-FINDINGS.md) explains the new ranks,
remaining losses, correctness fixes and implementation tradeoffs.

The [independent audit](AUDIT.md) records measurement/publication flaws found and
fixed before the published rerun, plus the limits that remain.

## Method

- Seven original workloads: English, Chinese, French, code, Latin, Latin repeated
  2,000 times, and 120,000 random Unicode code points. Unicode uses a fixed
  xorshift32 seed (`0x12345678`), reset for every generation. The raw results
  include UTF-8 byte lengths and SHA-256 input hashes.
- Both `cl100k_base` and `o200k_base`; `tiktoken-node` supports only `cl100k_base`.
  Every entry names its encoding explicitly. The local checkout is compared with
  the latest published `gpt-tokenizer`, rather than the old 3.4.0 dependency.
- Separate correctness workers check **every token ID**, length, decoded text,
  and token count against the local checkout, plus empty input, leading BOM, unusual Unicode
  whitespace, contractions/combining characters, and a long-piece probe. The
  independent `tiktoken` implementation also undergoes these checks. Failures
  are printed and saved, never silently accepted. Probe failures are distinguished
  from timed-input failures in the report; only timed-input-correct entries rank. The npm gpt-tokenizer reference is
  excluded from ranks to avoid counting the same implementation twice.
- Five fresh worker processes per tokenizer/encoding/engine. Case order rotates
  between rounds; there is no concurrent benchmark work. Package metadata and
  corpus construction precede the memory baseline. Every timed input is eagerly
  materialized through a UTF-8 Buffer round trip before baseline GC, so lazy
  string flattening/release cannot be charged to tokenizer retained growth. Correctness reference imports
  never contaminate measured workers.
- Initialization is cold **module import + encoding construction** in a fresh
  process, including local file reads, JSON parsing, WASM compilation and native
  binding load as applicable. Process startup is excluded; filesystem caches are
  warm (not flushed between processes). Standard package entrypoints are used,
  preserving the old comparison; per-encoding “lite” builds are not compared.
- Each operation warms for at least 100 ms, then runs for at least 300 ms and five
  calls. Batches are 100 calls for short synchronous inputs, 10 for short async
  inputs, and one for large inputs. One timer surrounds the batch loop, avoiding
  per-call timer cost. Counts and output lengths feed a checksum. Native HF's
  public async API includes its Promise/N-API overhead; sync APIs stay sync.
- These are **warm repeated-input** measurements using each library's default
  caches. They do not predict cold, unique-document throughput. Encoding, decoding,
  and counting are separate operations; count uses a dedicated API where available
  and `encode().length` otherwise. Decode always consumes that adapter's own native
  output type, including required WASM-to-text decoding.
- Ranks compare the checkout with competing implementations; the npm
  `gpt-tokenizer` row is a separate, unranked release reference.
- Summary encode/decode/count is the arithmetic mean of the seven per-input
  latencies **within each run**, matching the old summary's weighting; large
  inputs dominate it. Detailed per-input results are also published. Tables show
  the median and full minimum–maximum range across independent runs. Overlapping
  ranges do not establish a reliable speed advantage.
- **Memory is a separate phase**, with five fresh processes per case and exactly
  one encode/decode/count pass per input. Materialize corpus strings, unwind two
  event-loop turns with full GC, then take the baseline. After loading, settle
  two turns/full GCs again before the initialized snapshot. Encode/decode return
  from a helper; settle to release their outputs before counting; return and
  settle again before recording cumulative retained growth. Repeat for all seven
  inputs. This measures settled initialization and one-pass retained memory,
  separately from timed-loop allocation/JIT pressure.
- Report heap, RSS, external and ArrayBuffer deltas separately in MiB; external
  already includes ArrayBuffers, so do not sum them. Retained growth is relative
  to the settled initialized state. One-pass peak RSS is the absolute high-water
  mark including loader transients, harness and corpus. The original timed-loop
  peak is preserved separately and is not ranked because adaptive loop counts
  differ between libraries. Neither retained delta nor peak is a leak diagnosis.
- Immediate post-load GC can retain temporary corpus buffers and fixture JSON
  on async/native stacks. The original timing JSON preserves those early memory
  snapshots for provenance, but they are **superseded** for memory rankings by
  `memory-node-26.12.0.json` / `memory-bun-1.4.3.json`. Their load/throughput timings
  remain valid. Memory files link the exact timing file SHA-256 and verify unchanged
  tokenizer source, package versions, and input hashes. Engine GC and allocator
  behavior still differ, so memory accounting across engines is not identical.
- MLC 0.1.6 ships a UMD/CommonJS bundle under `type: module`. Its adapter executes
  the unmodified bundle in a CJS function wrapper on both engines. This loader
  compatibility work (including reading and compiling the wrapper) is timed.

Before this work, the README charts were last updated in September 2024. The previous
suite ran once, used independent `Math.random()` corpora in workers,
rounded sub-millisecond measurements, timed each call individually, checked only
round trips, and labeled >10 MiB retained heap as a possible leak. The old charts
have no checked-in raw samples or machine metadata, so they cannot establish a
historical performance regression against this run.

## Selection and release research

Registry and GitHub evidence was captured on **2026-10-15 UTC** (October 14 in
America/Los_Angeles) in [research/2026-10-15.json](research/2026-10-15.json).
Versions are pinned to npm's `latest` tag, not unreleased main branches. Download
counts are a one-week snapshot and include CI/transitive use, not unique users.
None of the retained or removed original competitors is marked deprecated by npm.
Inactivity is reported separately from formal deprecation.

| Package                 | Version       | Decision / maintenance and adoption                                                                                                                                             |
| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| gpt-tokenizer           | local + 4.0.3 | Checkout + latest npm baseline (~1.92M weekly downloads).                                                                                                                       |
| tiktoken                | 1.0.22        | Retain WASM baseline (~1.53M/week); latest release and repo push August 2025, so release-stale, not proven abandoned.                                                           |
| js-tiktoken             | 1.0.21        | Retain JS baseline (~9.86M/week); same August 2025 maintenance caveat.                                                                                                          |
| tiktoken-node           | 0.0.7         | Retain native baseline (~9.6k/week); latest release February 2024, last repo push May 2024, apparently inactive; no o200k.                                                      |
| @huggingface/tokenizers | 0.2.0         | **Add** official standalone JS engine (~2.75M/week); released September 2026, repo active October.                                                                              |
| tokenizers              | 0.23.2        | **Add** official Rust N-API engine (~6.8k/week); published September 2026, upstream active October.                                                                             |
| @mlc-ai/web-tokenizers  | 0.1.6         | **Add** Rust/C++ WASM used by WebLLM (~2.4k/week); repository active May 2026.                                                                                                  |
| @goliapkg/tiktoken-wasm | 4.1.2         | **Add** independent Rust/WASM OpenAI implementation (~5.4k/week); upstream active October 2026. Small project; less established than the baselines.                             |
| bpe-lite                | 0.5.2         | **Add** zero-dependency JS OpenAI implementation (~4k/week, used by `@steipete/summarize`); latest release/repo work March 2026, less recently active.                          |
| ai-tokenizer            | 1.0.7         | **Add** Coder's JS engine (~150k/week); release September 2026; test raw BPE APIs, not SDK estimates.                                                                           |
| @lenml/tokenizers       | 3.7.2         | **Add** standalone Transformers.js fork (~20k/week); repo active May 2026, latest npm September 2025. Use shared HF fixtures rather than its lightly used bundled GPT packages. |

Candidates deliberately skipped:

| Candidate                                        | Reason                                                                                                                                                                                               |
| ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| gpt-3-encoder 1.1.4                              | Previously disabled; only GPT-2/GPT-3 vocabulary, no cl100k/o200k; release/repo inactive since 2023.                                                                                                 |
| gpt3-tokenizer 1.1.5                             | Previously disabled; GPT-2/GPT-3 vocabulary, no cl100k/o200k; release/repo inactive since 2023.                                                                                                      |
| @dqbd/tiktoken 1.0.22                            | Same WASM project/version as `tiktoken`, a duplicate historical package name.                                                                                                                        |
| @huggingface/transformers / @xenova/transformers | Full inference libraries; official standalone `@huggingface/tokenizers` covers the tokenizer implementation without inference dependencies.                                                          |
| @lenml/tokenizer-gpt4 / tokenizer-gpt4o 3.7.2    | Bundled vocab variants of the included LenML engine, not separate implementations.                                                                                                                   |
| @agnai/web-tokenizers 0.1.6-pre3                 | Prerelease fork of MLC's included engine, without evidence of an independent OpenAI tokenization implementation.                                                                                     |
| @bpinternal/thicktoken 3.1.0                     | Botpress-internal factory returns a count/truncate/split wrapper; numeric encode/decode requires private wrapper access or a custom gzip-vocab construction, outside this standard-entrypoint suite. |
| @anthropic-ai/tokenizer 0.0.4                    | Claude vocabulary rather than OpenAI cl100k/o200k.                                                                                                                                                   |
| tiktoken-rs / bpe-openai / splintr / quicktok    | Rust/Python cores; no supported adopted npm binding for these projects identified (native bindings are represented separately).                                                                      |
| tiktoken-rs-node 1.0.6                           | Adopted (~46k/week), but latest release and repository activity April 2025; excluded from new additions under the current-maintenance criterion.                                                     |
| wordchipper                                      | Has a documented JS/WASM wrapper, but requires a source build; no published `wordchipper` or `@wordchipper/wasm` npm package/adoption found.                                                         |
| @codex-native/sdk / infiniloom-node              | Broad application SDKs with tokenizer helpers and single-digit weekly downloads, rather than adopted standalone tokenizer bindings.                                                                  |
| script-heads/wasm-tokenizer                      | No npm release; last repository activity July 2024 and five GitHub stars, insufficient current maintenance/adoption.                                                                                 |
| ztok                                             | Zig library/CLI with source-build WASM examples, not an adopted packaged JS tokenizer.                                                                                                               |

Sources: [npm registry](https://registry.npmjs.org/),
[Hugging Face JS](https://github.com/huggingface/tokenizers.js),
[Hugging Face native](https://github.com/huggingface/tokenizers),
[MLC](https://github.com/mlc-ai/tokenizers-cpp),
[Golia](https://github.com/goliajp/rust-tiktoken),
[bpe-lite](https://github.com/jschoemaker/bpe-lite),
[Coder](https://github.com/coder/ai-tokenizer),
[LenML](https://github.com/lenML/tokenizers).