# Benchmarks The numbers in the main README's comparison table are a claim. This directory is how you check it — clone the repo, run one script, get your own numbers on your own hardware. ## Running it ```bash cargo install hyperfine oha # one-time cargo build --release -p vvva_cli bash bench/run.sh ``` By default it looks for `../target/release/3va` relative to this directory. Point it at a different binary with `BIN_3VA=/path/to/3va bash bench/run.sh`. Node and Bun are benchmarked too if they're on `PATH`; if not, those rows are skipped rather than faked. ## What each number means - **Startup** — `hyperfine`, 30+ runs, 5 warmup, running `hello.js`. - **Install (warm)** — `hyperfine` re-running `3va install` when the dependency is already satisfied (a no-op resolution check, not a fresh download). This is a different measurement than the npm/pnpm/Bun install-speed table in the main README, which compares cold-vs-warm install across tools and isn't scripted here yet — see [`../docs/12-roadmap/06-pm-feature-parity.md`](../docs/12-roadmap/06-pm-feature-parity.md) if you want to script that one too. - **HTTP throughput** — `oha`, 100,000 requests at 1,000 concurrent connections against `server.js` (a minimal `http.createServer`/`Bun.serve` handler, same shape for every runtime so the comparison is the runtime, not three different servers). - **WebSocket chat** — modelled on Bun's published "32 clients broadcasting" benchmark, but **not the same workload** (`chat/server.js`): a chat room that broadcasts every message to every client with a plain JS loop on every runtime, where Bun's published 4.17M msgs/s uses its native `publish`, with 32 clients; a run here is 16 clients and about 0.3 s, so the figures are not comparable with Bun's page and carry run-to-run noise. 3va/Node serve it over `http` + the `ws` npm library via the `'upgrade'` event; Bun uses `Bun.serve`. The load client is node + `ws` (like `oha` is for HTTP). `chat/client.js` opens 16 connections × 80 messages each = 20,480 deliveries and reports msgs/s, p99 round-trip (on the sending connection) and the server's peak RSS. On the dedicated 32-core machine: 3va ~63k, Node ~84k, Bun ~138k msgs/s — 3va is behind because the upgrade-socket reads are poll-based (a 1 ms JS timer vs an event-driven wake), an honest current limitation. - **Postgres** — Bun's published "100 rows × 100 queries in flight" workload (`postgres/pg.js`), the same node-postgres (`pg`) client on every runtime, so the comparison is each runtime's net/crypto (SCRAM-SHA-256 auth) and event loop. Skipped with a note when no server answers on `PGPORT` (default 55432 — start the bench's own container with `scripts/run-pg-bench.sh`). On 3va, the client needed two fixes to reach this point: WebCrypto `subtle` PBKDF2/HMAC for the SCRAM proof (`crypto.subtle.importKey`/`deriveBits`/`sign`), and `TCP_NODELAY` on the `net` client socket — without it the Nagle/delayed-ACK interaction cost ~40 ms per query. The remaining gap to Node/Bun is the poll-based `net` socket read (a 1 ms JS timer instead of an event-driven epoll wake), which serializes queries at ~460 q/s vs Node's ~7,900 / Bun's ~8,900 on the dedicated 32-core machine. That is an honest current limitation, not a hidden one. ## Same workloads Bun and Deno publish Where the other runtimes publish their benchmark source, this directory reuses it so the numbers are comparable in *what* is measured (not in hardware): - **Express 5 hello world** — `express/express.mjs` is Bun's `bench/express` verbatim. Dependencies are installed with npm (or bun) so this measures the runtime running Express, not each tool's installer. - **Package install, real app** — `install/package.json` is Bun's `bench/install` (create-t3-app, hundreds of transitive packages). Protocol: cache warm, lockfile present, `node_modules` removed before every timed run, median of 3. `3va install` runs with `--no-scan` because npm/bun don't scan; a tool whose install doesn't produce `node_modules/next` is reported as `failed`, never as a fast time. - **Express 5 over HTTPS** — `express/express-https.mjs`: the same Express app behind `https.createServer`, the workload Bun publishes as "Serving Express over HTTPS". The script generates a throwaway self-signed key and certificate; each runtime terminates TLS itself. - **TypeScript startup** — `hello.ts` (3va and Bun run it directly; Node needs >= 23.6 for built-in type stripping). - **p99 latency and peak memory** — every HTTP table reports `oha`'s p99 and the process's `VmHWM` (true peak RSS), not just idle/post-load. Deno's homepage numbers ("realworld", gzip `DecompressionStream`) don't have public source we could find, so they are not reproduced here. 3va has no `CompressionStream`/`DecompressionStream` yet. `BENCH_SKIP_TEST262=1 bash bench/run.sh` skips the (slow) conformance run. ## Why `3va.config.json` exists in this directory 3va's HTTP server firewalls by default: 100 req/s and 50 simultaneous connections per source IP (`crates/firewall/src/lib.rs`'s `FirewallConfig` default). A single-machine load test at 1,000 concurrent connections comes from *one* IP, so against the default config it mostly gets rejected with `403` — not a bug, that's exactly what the firewall is for. The config in this directory raises those limits so the throughput number reflects the server's actual request-handling capacity instead of its DDoS protection kicking in on a benchmark that looks, from the server's point of view, indistinguishable from an attack. Raising the limits also keeps the throughput number free of the firewall's adaptive layer: since every request is allowed (the raised token bucket never empties), no violations accumulate, so no IP is ever auto-blocked and the escalating block durations (`blockEscalationFactor`/`maxBlockDurationSecs`) never engage. The documented numbers therefore measure the server's raw request-handling capacity, not a rate limiter that is deliberately refusing traffic. The exact benchmark numbers are unaffected by the adaptive behavior — it only changes how repeat offenders are *blocked*, and the benchmark produces no offenders. If you want the *default-config* number instead — i.e. "what does a believable attacker actually get through" — delete or rename `3va.config.json` before running the script and expect most requests to return `403`. Both numbers are real; they answer different questions. ## Reference run The numbers backing the main README's comparison table come from [`.github/workflows/benchmark.yml`](../.github/workflows/benchmark.yml) itself, not a hand-run on a maintainer's machine — see the "CI" section below for why. Latest published run: [actions/runs/36927280438](https://github.com/OdinoCano/3va/actions/runs/36927280438) (2026-10-01, commit `2961d12`), on a GitHub-hosted `ubuntu-latest` runner (4 vCPU, 15 GiB RAM, Linux 6.17-azure): - 3va: this repo, release build (`cargo build --release`) - Node.js 24.19.0 - Bun 1.3.14 | Runtime | Startup (mean) | HTTP throughput (c=1000) | Express 5 | Express 5 over HTTPS | Memory, idle → post-load | Install, warm | |---|---|---|---|---|---|---| | Node.js 24.19 | 18.9 ms | 62,650 req/s | 36,918 req/s | 23,170 req/s | 50.0 MB → 88.1 MB | 241.1 ms | | Bun 1.3.14 | 3.1 ms | 116,809 req/s | 48,841 req/s | 43,502 req/s | 19.5 MB → 37.8 MB | 1.7 ms | | **3va** | 11.6 ms | 121,871 req/s¹ | 71,881 req/s | 69,764 req/s | 43.4 MB → 76.1 MB | 3.9 ms | ¹ With `3va.config.json`'s opened-up firewall limits — see above. Two caveats about this run, reported rather than hidden: - **Startup contradicts the dedicated 32-core bench, and the cause is not known.** On the dedicated machine 3va and Bun are even (7.4 vs 7.3 ms, measured 2026-10-02); on this shared 4-vCPU runner Bun's 3.1 ms is 3.7× faster than 3va's 11.6 ms. What was checked: the optimization applies on CI (3va went from 25.6 ms in the 2.11.0 release run to 11.6 ms, the same ~2.2× as on the dedicated machine, 15.8 → 7.4 ms); pinning both runtimes to 4 CPUs, or to 1, on the dedicated machine leaves them level (8.0 vs 8.3 ms with 4 CPUs), so the core count is not the explanation; and Bun itself is 2.3× faster on the CI runner than on the dedicated machine (3.1 vs 7.3 ms) while 3va is 1.6× slower there. Both figures are real; they measure different environments. Open item. - **Postgres is not in the CI run.** It needs a live server, which the runner does not have, so CI prints the skip note. Run `scripts/run-pg-bench.sh` before `bench/run.sh` locally to get those numbers; on the dedicated 32-core machine they are 3va ~460, Node ~7,900, Bun ~8,900 q/s for the 100-query batch. GitHub's shared runners are weaker and noisier than dedicated hardware — absolute numbers here run meaningfully lower than a quiet dedicated box for all three runtimes (CPU-bound work like HTTP throughput scales down with fewer/slower cores; startup latency picks up scheduler noise from neighboring jobs) — but the workflow re-measures all three runtimes together on every run, so the *ranking* and relative gaps are the trustworthy part. Treat single-run absolute figures as a snapshot; rerun the workflow (`workflow_dispatch`) for a fresh one, or diff against past runs in the Actions tab for a trend instead of one point in time. ### Memory under load: root cause and fix *Historical investigation, from a 2026-07-15 run on dedicated hardware (AMD Ryzen 9 7950X) predating the switch to CI-sourced reference numbers above — the specific MB figures below won't match the current CI table (different machine, different point in time), but the root cause and fix they describe are still exactly what's running today.* The first version of that run showed 3va's memory growing ~8.5× from idle to 1,000 concurrent connections (30 MB → 255 MB), against ~1.8× for Node and ~1.3× for Bun. That was investigated rather than left as a caveat: 1. **Isolated concurrency from cumulative load.** Re-running with a fresh server per concurrency level (100/500/1000/2000) showed near-identical post-load RSS regardless of concurrency — ruling out a connection-count- driven cause (e.g. a per-connection buffer or an unbounded backlog between the accept loop and JS). 2. **Confirmed a linear per-request cost instead.** Fixed concurrency (c=50), increasing cumulative request count (10k/30k/60k/100k) against the *same* long-lived process: RSS grew ~3.5–4 KB per request, forever, independent of concurrency, and never recovered even across idle gaps between rounds. 3. **Ruled out the obvious native-side suspects.** `crates/js/src/builtins/http_server.rs`'s `conns`/`ready` maps (candidate: connections piling up faster than JS drains them) are correctly removed on every successful respond path — confirmed by reading `http_server.rs:589-616`, and consistent with this benchmark's 99.7–100% success rate. Not the driver here. 4. **Tried mimalloc as the global allocator** (a very common fix for exactly this RSS-never-shrinks symptom, since glibc's malloc rarely returns freed pages to the OS) — **no measurable difference**. This ruled out Rust-heap fragmentation as the cause: `#[global_allocator]` only affects Rust-level allocations (`Vec`, `String`, `Box`, ...); V8 manages its own C++ heap via its own page allocator, entirely outside Rust's allocator hook. 5. **Root cause: V8's heap was never told to shrink.** `crates/js/src/lib.rs` had no call to V8's `low_memory_notification()` anywhere — the only API that prompts V8 to actually try to free memory back to the OS. Every request generates real (but collectible) V8 garbage — parsed headers, JSON strings, Promise/closure objects — and without that hint, V8's heap grows to its burst high-water mark and stays there. 6. **Fix:** `run_event_loop` (`crates/js/src/lib.rs`) now calls `isolate.low_memory_notification()` on a 1-second throttle (`LOW_MEMORY_HINT_INTERVAL`) — frequent enough to reclaim memory during sustained load, not so frequent that a full GC pause on every event-loop tick would hurt throughput. Verified: post-load RSS dropped from 255 MB to 93–180 MB across repeated runs (a 30–64% reduction depending on run), with throughput unchanged (16,100 req/s @ c=1000, 99.94% success, measured before and after) — the fix costs nothing observable and keeps most of the win. (The interval was later tightened from 5 s to 1 s: the 5 s hint never fired during a fast HTTP burst once keep-alive removed the per-request TCP handshake cost — the whole 100k-request load finished in ~2 s — so the heap hit its burst high-water mark before the first hint. At 1 s the hint fires at least once mid-burst, which held post-load RSS near the pre-keep-alive figure instead of spiking to the burst watermark. The exact post-load number is measured by CI.) The remaining idle-to-loaded growth (versus Node's ~1.8×) wasn't chased further — mimalloc is still wired in as the global allocator (a reasonable default for a busy server generally, even though it didn't fix this specific bug), and a request-count-based trigger instead of a time-based one is the next thing to try if this needs to go lower. ## CI [`../.github/workflows/benchmark.yml`](../.github/workflows/benchmark.yml) runs this script on `workflow_dispatch`, on release tags, and weekly, publishing the result table to the workflow's job summary. **As of 2.5.0, this is where the main README's comparison table numbers come from** — deliberately, not a reversal of the earlier "dedicated hardware only" policy by accident: a number anyone can regenerate by clicking "Run workflow" (or that reruns automatically on a schedule) is more credible than one only reproducible on a maintainer's specific machine, even though the dedicated-hardware run above has cleaner absolute figures. Re-run it yourself, compare against the [linked run](https://github.com/OdinoCano/3va/actions/runs/36927280438) above, or diff any two runs in the Actions tab for a regression check.