# What's new in TensorFold

Each release's page on GitHub has its notes and measurements. See [the 1.0.0 release notes](RELEASE-NOTES-1.0.0.md) for the native binary's supported models and migration details.

## 1.1.1

- The app checks models through HTTPS Hub fake-IP proxies in both address families and honours one validated `HF_ENDPOINT` for checks and downloads. Network-policy refusals name the DNS address and proxy remedy (#1037). Thanks to @samwang0041-star.

- Flash Next on Metal uses precise exp2, sine and cosine for rotary preparation and index pooling, including M-RoPE (#986, #1005). Thanks to @BobClawblaw, @cwschroeder and @sethforprivacy, retaining the original YaRN attribution.
- GLM-5.3-Flash NVFP4 loads and serves the whole checkpoint on two GB10s at TP=2 in a contributed regression run on shipped 1.1.0: 302.1 seconds to load, one token hash per prompt across three runs, and a passphrase found in 199,307 tokens. Scored prompts and more than two ranks were not covered (#1040). Thanks to @Bizuayeu.

- Qwen3.8-27B on Metal keeps its Neural Engine prompt share off by default on M5. Its fp16 prompt rows can change replies, so the default follows the no-quality-tradeoffs rule. `TF_ANE_COLUMNS=N` opts in; M1 to M4 already use the GPU alone. The load receipt reports requested and loaded columns, fp16 program-weight bytes and exact-output qualification.

- Source builds report the release version. A source build of 1.1.0 reported 1.0.12 because its package manifest still carried the older version.
- Qwen on CUDA uses larger verification trees, shared rounds across streams and batched DFlash proposals. Captured launches follow the live row and window sizes. A lone stream chooses its draft width from measured complete-round costs, with separate retry cooldowns for widths that did not repay their cost; straight chains avoid the branch planner and contiguous drafter rows avoid a redundant gather (#978). Thanks to @BobClawblaw and the shared CUDA contributors.
- DeepSeek-V4.1 CUDA pieces land in 1.1.1, but no release binary serves DeepSeek checkpoints until whole-model gate #827 passes. Its implementation includes shared checkpoint planning and Engram token maps, paged and split KV, prompt-cache sessions, keyed sampling and drafting through the lane scheduler. Devices older than sm_90 refuse before family image loading. The shared configuration and host references accompany the CUDA family; a complete DeepSeek Metal backend remains separate (#667, #668, #671, #674, #686, #688, #691, #693, #694, #695, #699, #702, #703, #707, #727, #824, #825, #826, #827). Thanks to @jayleaton and @akol1, retaining Turboderp's ExLlamaV3 attribution and the common EXL3 contributors' credits.
- Kolibri-1's text research checkpoint serves on native CUDA through the lane scheduler, with prompt caching, shared rounds and verified copy drafts. EAGLE-3 and a neural drafter remain unsupported (#991, #481). Thanks to @jschmied and the shared operator contributors.
- GLM CUDA stepped serving interleaves fresh prompt chunks with running decode. Whole-prompt mode can batch eligible fresh text prompts across ranks, keeping each sequence's final prompt row and first draft separate; each pass holds at most 16 prompts and 2,048 rows. Images and resumed prompts keep their own paths (#1014). GLM CUDA prompt projections use one K slice at every operator row count by default. This equal-precision reduction regrouping can change replies versus 1.1.0, including long or resumed prompts whose internal projection calls have fewer than 128 rows. `TF_GLM_PROMPT_K=shape`, with `TF_GLM_MM=1` on every rank, instead uses shape-selected K slices throughout prompt projections, including full blocks; it does not restore 1.1.0's mixed row-count policy. Verify and draft windows retain their existing policy. Thanks to @BobClawblaw.
- GLM on CUDA has an experimental prompt projection policy, `TF_GLM_PROMPT_K=shape`, which chooses K slices from the weight's shape. It requires `TF_GLM_MM=1` and the same setting on every rank. It is off by default, and its different reduction grouping can change near-ties; verify and draft windows keep their existing policy. Thanks to @BobClawblaw and the TensorFold maintainers.
- GLM on CUDA returns prompt log probabilities for nonstreaming `/v1/completions` requests with `prompt_logprobs`. Scoring storage is reserved before collective work, and ordinary requests keep their existing path (#1016). Thanks to @Bizuayeu.
- GLM on CUDA accepts schema-constrained structured output and the guided-output options, including a mask on the first generated token and independent grammar state for each sequence. Drafting observes the same constraints (#1017). Thanks to @Bizuayeu.
- Required and named tool calls on GLM CUDA use the model's tool markup, with schema-constrained argument values and matching parsing on the server (#1018). Thanks to @Bizuayeu.
- The common CUDA runtime has a keyed graph cache and a device arena, enabled only by families that install them. Failed captures or allocations release unpublished resources, and rank agreement keeps tensor-parallel cache decisions together (#635, #682). Thanks to @jayleaton and the shared runtime contributors.
- The common tensor-parallel fabric supports process bootstrap, NCCL, mailbox and RoCE collectives, checks that ranks agree on their settings, and propagates a failed peer instead of leaving the other rank waiting (#636). Thanks to @jayleaton, retaining the shared transport and RoCE attributions.
- Paged-KV families can use shared session memory with prefix reuse, a bounded RAM residency budget and a streamed disk tier on Linux. Failed saves and restores retain the owners needed for a retry (#637). Thanks to @jayleaton and @cwschroeder.
- On SIGTERM or SIGINT, the server gives requests in progress up to `TF_DRAIN_S` to finish, 20 seconds by default, while new engine requests get 503 with `Retry-After`. A second stop signal cuts the drain short; kept states drain before the engine closes (#663). Thanks to @jayleaton.
- Silent streaming responses send keepalives before their first token, every 15 seconds by default. `TF_SSE_KEEPALIVE_S` changes the interval; zero disables it (#663). Thanks to @jayleaton.
- A failed CUDA kernel launch names the kernel, grid, block and shared-memory size alongside the driver error, so a load or inference failure identifies the launch that failed (#635). Thanks to @jayleaton.
- Native CUDA refuses an unsupported lane policy instead of accepting an option that the selected family ignores. Kolibri receives the common lane options. Thanks to @ajensenwaud and @jayleaton.
- CUDA launch lookup can use a runtime-parameter variant for an argument of 1 when no suitable folded-one variant exists. Both preferred matching passes run first, each fallback is logged, and a launch with no matching variant still fails with its argument dump (#1020, #1003). Thanks to @GustavBlack and @cwschroeder.
- New benchmark runs use `community-v2`: a sampled reply that stops naturally after at least 128 tokens counts as complete, and its rate uses the tokens actually delivered. Greedy replies still need the full requested count, and empty replies still fail. Existing v1 receipts retain their rules; v2 runs can be saved and inspected locally, with publication disabled until the receiver and separate rankings support them (#969). Thanks to @BlivionIaG and @easthambuilds.
- The benchmark sends its code workload through `/v1/chat/completions` when the native family declares that it accepts only chat. An attached server uses that route only with `--chat-only`; an empty reply never triggers a route change. V2 receipts record the route, with the prompt and sampling settings unchanged (#1019). Thanks to @BobClawblaw, @BlivionIaG and @easthambuilds, and @jayleaton for the DeepSeek family declaration.
- Qwen3.8-27B on Metal allocates DeltaNet rollback scratch only for the slots whose verification windows need it, instead of reserving a full second state bank at load. Idle or lone use can release that scratch after the previous round has been kept. Recurrence arithmetic and saved prompt states are unchanged. Thanks to the TensorFold maintainers and the Qwen contributors.
- Request completion lines report `ttft`, `decode`, `elapsed` and `queued` seconds from that request's own timestamps, alongside its token rate. Queue time ends at first lane admission, and a cached prompt keeps its own timing (#1025). Thanks to @tpischke.
- Sampled native requests draw a fresh seed when neither the request nor the server supplies one, and completion lines record it for replay. An explicit request seed still overrides the server default. `--prompt-seed` restores the historical prompt-derived fallback; greedy sampling is unchanged (#1026). Thanks to @tpischke.
- The core has a host encoder and decoder for losslessly compressed `csf_32x2_v1` NVFP4 scales, with checked geometry, exception counts and allocation cleanup. This supplies the scale codec for future checkpoint consumers; it does not add a complete FP4-Local family loader (#1021). Thanks to @Bizuayeu, retaining local-inference-lab's format and reference attribution.
- CUDA's checkpoint weight estimate counts each file named in the shard index once, including files in subfolders. A malformed index or a missing or unreadable named shard refuses admission instead of producing an incomplete estimate; checkpoints without an index or weight map keep the flat-directory fallback (#1022). Thanks to @Bizuayeu.
- The core reads per-module ModelOpt `MIXED_PRECISION` recipes and reports unsupported declarations by module name. The existing NVFP4 reader keeps its behavior. This adds recipe descriptions for mixed-format consumers, with complete family loading still separate (#1023). Thanks to @Bizuayeu.
- The CUDA block-FP8 scale packer accepts each rank's global row and input origins, including tensor-parallel shares that begin inside a 128-element scale block. ModelOpt's four-dimensional block scales can be viewed as the equivalent two-dimensional layout without copying. Existing zero-origin packing is unchanged (#1024). Thanks to @Bizuayeu and @jschmied for the original fp8g layout in #548.

- Missing optional CUDA libraries name the library and the feature that needs it, with the dependencies also listed in `capabilities --json` and the runtime documentation. This makes failures such as GLM's vision path needing `libcublasLt.so.13` distinguishable from a text-only load failure (#1030). Thanks to @ss-cong for the report.
- The native CUDA host forwards the family's first-token handoff flag, so a family that enables it can deliver the first token before doing the next draft (#1031). Thanks to @jrb-ai.
- Contributor links in the README, CHANGELOG and 1.0.0 notes use @jrb-ai after the same account's handle change (#1036). Thanks to @jrb-ai.
- Qwen3.8-27B on CUDA caches all 64 measured decode-window costs between starts, keyed by the executable, bound device, checkpoint identities, KV format, taps and profiling capacity. A miss measures every width; a valid hit still warms this process's resources before ready, without repeating the timed sweep.
- Qwen3.8-27B on CUDA can open a short-context session with DFlash2: calibration uses a temporary cache when the configured context is too small for its warm-up windows. Calibration still finishes during open, before the lane thread starts.
- The runbook documents GB10 efficiency-core pinning with `taskset -c 0-4,10-14` or Docker's `--cpuset-cpus=0-4,10-14` to reduce CPU heat from the two spinning threads per CUDA rank. The reported heat and speed measurements retain their 1.0.12 plus #1016-#1018 scope (#1033). Thanks to @Bizuayeu.

## 1.1.0

- Structured output and forced tool calls work with Nemotron 3.5 Super VL (`nemotron_h_omni`), whose text vocabulary is in `llm_config`: the server read `vocab_size` only from `text_config` and the top level, and refused those requests (reported in #951). A `vocab_size` that is not a positive integer within the C ABI's range is skipped rather than cast (#953). Thanks to @swbiggart and @BobClawblaw.
- Accepted HTTP connections set `TCP_NODELAY`, so a streamed reply's small writes leave as they are written instead of waiting behind Nagle's algorithm; a connection that cannot be configured is closed and the server accepts the next one (#929). Thanks to @akol1.
- The tool-call parser finds namespaced closers (`</NAME:tool_call>`) in one pass over the reply, so a reply with many openers and no closers parses in linear time instead of searching the rest of the reply once per opener. Parsed calls are unchanged (#913). Thanks to @jishnuvenugopal.
- A chat template that builds a long string in a `namespace` (`{% set ns.out = ns.out + m.content %}`, or with `~`) appends to one growing buffer instead of copying the whole string at every step, so long histories render in linear memory. The rendered bytes are unchanged (#973). Thanks to @nmcbride.
- In the lane core, a pipelined stream adopted between rounds (detached and adopted again, or opened by another driver) starts in verify from its pending token; it read the missing mode as pipe and unwrapped a step it had never queued. No shipped family reaches that path yet (#928). Thanks to @chaog992.
- On Linux, a checkpoint read with O_DIRECT falls back to the page cache when the file system accepts the open but refuses the aligned read with EINVAL; the load used to fail (#916). Ours: the fallback is Linux's alone, so macOS builds it. Thanks to @plotarmordev.
- Nemotron on Metal loads on Macs without tensor units (M1 to M4) when its packed kernels cannot build there, as on macOS 26.3, whose tensor-op library has no bf16 x uint4 operation. Those chips never dispatch the packed kernels (their projections run on the row kernels), so a packed kernel that fails to build stays absent, with one warning; any other kernel, and every kernel on an M5, still fails the load (#955). Ours: a failed load releases the pipelines it built. Thanks to @jishnuvenugopal.
- The opt-in HIP build's Flash Next replays a round's graph only over the caches its pool slot was made with, and refuses caches made elsewhere (`CachesNotPooled`). `tf-flashnext-hip-test --repeat` takes slot 0 from the pool for every run, so a later run no longer replays the first run's graphs over freed buffers (#963). Thanks to @BlivionIaG.
- The CUDA runtime has `Context.shared`, the primary context of the calling thread's current device (device 0's when none is current, refused when the current context is not a primary one), and `Stream.borrowed`, a stream its caller owns, for a library that runs TensorFold's kernels inside a host framework's process (#946). Ours: fake-driver tests of the context and stream ownership branches, which do not stand for a host framework's own checks. Thanks to @drowzeys.
- `tf-cuda-test quant-ref` fails an output whose error against the float64 reference is not finite; a NaN after an out-of-bound element used to replace the largest error and pass. `zig build test-cuda-reference` runs the comparison's CPU tests, which `test-cuda-host` runs too (#933). Thanks to @cwschroeder.
- The CUDA kernel contract runs on sm_80 to sm_89: the qmm_group cluster kernel, which needs sm_90, is reported N/A there, and lane GEMV is held to the float64 bound instead of the cluster baseline's bits. sm_90 and newer are checked as before (#954). Thanks to @cwschroeder.
- `tools/zig/pack_compare.py` refuses a tensor whose byte range does not match its shape and dtype or runs past the file, where it compared the common prefix of a truncated pack; its tests run in a workflow scoped to its files (#919). Thanks to @cwschroeder.
- CONTRIBUTING.md and the README say that `zig build test` on macOS also runs the available Metal device checks, and name `zig build test-cuda-host` as the CUDA host subset (#957). Thanks to @cwschroeder.
- The app's model library reads each checkpoint metadata file (JSON, or a safetensors header) up to 64 MiB, where it stopped at 4 MiB and marked loadable checkpoints such as GLM-5.3-Flash (an 11.5 MiB shard index) incomplete; one scan reads at most 256 MiB of metadata in all, up from 32 (#967). Thanks to @heitke.
- The app's model library skips the AppleDouble companions (`._NAME`) that macOS writes beside each file on exFAT and FAT drives: it read them as weight shards and failed the checkpoint, and a stray one in a parent folder stopped the scan before the checkpoint below it (#970). Thanks to @heitke.
- The CUDA kernel contract keeps launch records per nvcc release: `registry.cuda` holds what nvcc 13.0, the release fatbins' compiler, compiles, and each later release lists only the launches it compiles differently (13.2, then 13.3 on top of it). `kernel-contract` and `kernel-contract-limits` read the release from each image's `.note.nv.tkinfo`, check against the newest set at or below it and print which set they used (#945). Ours: the sets per release, the records of the MXFP8 and FP32-output launches for both compilers, and a fatbin without the device's own SM image attributed only when its images agree on one compiler. Thanks to @akol1.
- The core decodes NVFP4 scale planes stored in a lossless compressed layout (byte-window bases with sorted exceptions) exactly, and a rank reads its tensor-parallel slice of an NVFP4 tensor exactly from plain or compressed scales, refusing ambiguous or half-present scale formats. No shipped checkpoint uses the layout yet (#904, #906). Thanks to @akol1.
- CUDA serves losslessly packed MXFP8 projections through the native qlinear, with their kernel-contract checks and launch records (#907), and the existing projection kernels can return FP32 outputs before the residual reduction; `zig build test-cuda-projection-fixture` poisons every dispatch's output and scratch before each launch, so a dispatch that writes nothing fails (#908). Thanks to @akol1 and @cwschroeder.
- The server admits a streamed request before it opens the response, so a refusal is still an HTTP 503, and health reports an engine that is closing or not ready, a drained Flash Next host among them, as unavailable (#912). Thanks to @akol1 and @cwschroeder.
- A test pins that a bounded-residency join which cannot grow its bookkeeping refuses whole, leaving nothing counted, staged or added (#909). Thanks to @BobClawblaw.
- In Living Weights, a new chat fact replaces an earlier one only when it changes that fact or the earlier fact no longer holds; answering one of the earlier fact's questions no longer retires it. When a told fact's questions do not name its subject, its reply gives the fact its key (#910). Ours: a judge's answer counts only as a whole yes or no from a reply that ended by itself, so "No idea" or a cut "No, it..." keeps the earlier fact. Thanks to @ZackSample.
- Nemotron on Metal: a new build keeps the chip's last window-cost table while probes of each kernel path still time within 15%, instead of timing the whole table again on its first start; a failed probe times it afresh. Costs steer the windows, never the tokens (#936). Ours: the shared rounds are probed too, and a timing cache that fails to make releases the ones already made. Thanks to @jishnuvenugopal and @cwschroeder.
- `tensorfold merge MODEL ADAPTER OUT` folds a PEFT LoRA into an MLX affine 4-bit checkpoint on the checkpoint's own scales, with no dense base, Python or GPU; `tensorfold merge DENSE ADAPTER OUT --quantize SHIPPED` instead quantizes the dense base plus the adapter fresh. On the author's Qwen3-4B checkpoint and adapter, one measured case, the fresh quantization kept a median 99.6% of the adapter's change against 0.1% for the fold, at 0.981 of the shipped quantization's pooled error. `tf-merge-check` gates both (#710). Ours: the fold as the default, a module found under either namespace's spelling, and two adapter modules for one projection refused. Thanks to @philip-pentatonic.
- GLM-5.3-Flash on CUDA: `TF_GLM_PREFILL_REDUCE=ring` passes split's exchanges along the ring, for a fabric where a rank reaches only its neighbours, and its sums keep split's rank order (#902). Ours: while the ranks' watch lines are open, an NCCL failure on one rank stops every rank instead of leaving them waiting; postings made through the shared group helper end their group when a post fails; and the families share the ring's plan. Thanks to @Bizuayeu.
- The lane host takes an opt-in round planner: a family can run prompts in pieces between the other streams' decode rounds, give each request its own copy proposer and trim each window's rows, and `halt` ends the rounds at the next boundary and fails what is left with its reason. A planner whose admitted callback fails ends each job it admitted once, as failed, and a failed round, cancellations and a close end every job even when no memory is left. No family uses the planner yet; the default loop serves as before (#861). Thanks to @jayleaton.
- CUDA's buffer counters keep the pinned host peak beside the device peak and can hand over an interval's peaks as they reset them; `tools/memory_receipts.py` runs one command, records its Linux resident peak and the allocator's load and cleanup receipts, and reports a total footprint as unsupported rather than adding the two (#865). Thanks to @jayleaton.
- A versioned drafter release format (`docs/drafter-release.md`): config.json names the target, its adapter and taps, `check` compares it with the loaded target and `checkRecording` with a `--record` recording, and `Release.open` reads a folder and checks every tensor's name and shape. Nothing serves a release yet (#893). Thanks to @jschmied.
- A recipe for training a block drafter from `--record` recordings into a drafter release, with what each step gained on Kolibri-1 (`docs/recipes/training-a-drafter.md`) (#894). Thanks to @jschmied.
- Qwen3.8-27B decode on Metal: K splits on M5 measured over every layer's matrices, an undrafted one-row round kept by its own verify with a DeltaNet step of three dispatches, and decode attention in 128-key chunks below position 8,192, chosen for each row by its own position so a window across 8,192 keeps every row's one-row bits. The chunks change the attention's summation order below 8,192 keys, on M5 too, so replies can differ from 1.0.12's; against float64 the new order's error is no larger, and drafted replies still equal `"draft": false` (#887). Thanks to @BobClawblaw.
- GLM-5.3-Flash on CUDA with `TF_GLM_PREFILL_REDUCE=split`, the default, exchanges a value with every other rank before the weights load and refuses a pair of ranks that cannot reach each other, instead of failing in the first prompt (#899). Ours: only direct split runs that probe, so `ring` starts on four ranks that reach only their neighbours, and the refusal names `ring` or `gather`; host tests over a fake fabric cover the refusal and the ring start, not a four-rank machine. Thanks to @Bizuayeu.
- Intel Arc Pro B70 (the XPU engine over Level Zero) checks and rounds up its sizes at open (`XPU_CHUNK` a multiple of 64 up to 8,192, a context up to 262,144), makes a stream's states whole or not at all, frees a snapshot only after its copies drain, reads safetensors headers within their files, and gives the card back at close; prompt chunks are cut on the 64-row grid at any length, and the prompt GDN's fp32 operands enter its fp16 DPAS as high and low terms after a power-of-two row scale. The author measured these on his B70 (#802). Ours, checked only by host builds and tests: close waits for every queue first, a failed start included; an open no longer keeps an earlier open's opt-in `XPU_*` settings; and the 27B runner waits for its queue before any host buffer a queued copy reads or writes goes away, on every error path, ending the process if the queue will not drain. Thanks to @Mirrdhyn.
- Qwen3.5-2B and Qwen3.8-27B serve on CUDA through the native server: each decode shape replays as a CUDA graph, the 27B drafts with DFlash2 in chains and trees, and concurrent requests share every round, each stream's reply equal to its run alone and to `"draft": false` on a GB10 (#836). Its author also built the native server and the CUDA suites on Windows through a platform layer, with the CUDA kernels on sm_89; we have no Windows check of our own (#836). Thanks to @BobClawblaw.
- The Qwen CUDA family reads mixed-width MLX checkpoints, 2 to 8 bits a module in groups of 64 (#849), and keeps 2, 3, 5 and 6-bit modules on tiles of their own width with their own prompt GEMM, rounding weights to bf16 once as the 4-bit prompt GEMM does (#859). Thanks to @BobClawblaw.
- Images for the Qwen CUDA family with `--vision`: the checkpoint's vision tower runs in TensorFold's kernels, its rows replace the image placeholders in the prompt chunks, and the prompt rotates at M-RoPE's t, h and w positions, with pictures decoded and prepared by the released `core/picture` and `core/qwen_vl` paths the Metal family uses (#885). Thanks to @BobClawblaw.
- The Qwen CUDA family keeps prompt states between turns in pinned host memory (8 GiB unless `--prompt-cache-gib` says otherwise, at most half of what the host has available), on the 128-row prompt grid; a state restores only when its geometry, cache format and every image's identity match the request's (#888). Ours: if a failed restore cannot drain its copies, dropping the state keeps its pinned memory, never freed, rather than freeing what the device may still read. Thanks to @BobClawblaw.
- `--kv-cache fp8` stores the Qwen CUDA family's attention caches as e4m3 rows with an fp32 scale each, about half the bytes of bf16; bf16 stays the default, and fp8 is off unless asked for (#886). Thanks to @BobClawblaw.
- Shared CUDA operators for every family: verified tree rounds (#734), tree attention and DFlash block attention (#746), SwiGLU and output gates, Gated DeltaNet inputs, QK norm and RoPE (#797), and sliding-window attention, sigmoid top-k MoE routing and combine, and dense rows (#798). `tf-cuda-test` checks the tree rounds byte for byte against the host's tree rounds and the arithmetic operators against float64 references. No family calls them yet. Thanks to @jschmied.
- A one-shot RoCE all-gather for two ranks, set up over the tensor-parallel group's NCCL communicator, with a two-rank test against NCCL's all-gather; no family calls it yet. Its design follows b12x's RoCE gather (#852). Thanks to @BobClawblaw.
- The lane core can take several queued prompts in one pass for a backend that batches them (`prefill_many`, `Engine.addStreams`); a failed pass leaves none of them prefilled. No served family batches prompts yet, so admission is unchanged (#853). Thanks to @BobClawblaw.
- Qwen3.8 Flash Next serves on CUDA through the native server from its MLX affine 4-bit, group-32 checkpoint with the MTP head, on kernels built in this repository: drafted windows, shared rounds of streams without drafts, kept prompt states and prompts a chunk at a time between other streams' rounds, all through the lane core, and past 2,048 keys attention reads the 512 blocks the model's indexer picks, up to the 262,144-token window. Each kernel keeps a row's sums in a fixed order; its authors measured on a GB10 that drafted replies equal `"draft": false`, concurrent replies their runs alone and resumed prompts fresh ones (`docs/recipes/flash-next-cuda.md`). A ModelOpt NVFP4 checkpoint on two local ranks is experimental (#952). Ours: every admitted stream that asks for drafts keeps them, where the fourth and later lost them; tensor-rank drafting with a prompt cache is refused at admission before the checkpoint or CUDA; #869's staging regression also fails the first copy and a drain before it succeeds (#869); and the kernel contract records #952's launches as nvcc 13.0, the release compiler, builds them, with nvcc 13.2's differences in its own set. Our kernel tests pass on an RTX PRO 6000 (sm_120) and a GB10 (sm_121), and on a GB10 we served the MLX 4-bit checkpoint with its MTP head: drafted replies equal `"draft": false` on all 17 cells, two and four concurrent replies equal their runs alone, and resumed chats equal cache-off ones on all 12 steps. Thanks to @ZackSample, @akol1, @AdrianBinDC, @Bizuayeu and @cwschroeder.
- Flash Next on CUDA takes `--vision`, experimental: images go through the Qwen-VL tower the Qwen CUDA family runs, now one shared module (`zig/src/vision/qwen_cuda.zig`) on rank zero, their rows replace the prompt's image placeholders over the four streams, M-RoPE positions reach attention, the block indexer and the MTP head, and a second rank receives the rows over NCCL. The checkpoint's preprocessor and processor files set the image processor through `core/qwen_vl`. A failed upload, plan, launch, wait or rank transfer now drains the stream, then the context, or ends the process, before the tower, an image plan or a lane frees memory the GPU may still use, for Qwen's tower too. On a GB10 the MLX 4-bit checkpoint's PNG image cells pass: an image reply drafted equals plain, four image requests together equal their runs alone, and a chat resumes past an image; JPEG needs libturbojpeg, which that machine lacked. The work is ours, built on #885's tower and #952. Thanks to @BobClawblaw and @akol1.
- The native CUDA build has EXL3 operators for a family's engine to call. `exl3.Dense` is a linear for every codebook and width, `mul1`'s half widths included, whose row keeps its bits at any row count, and prompts run in 128-row blocks of the same kernel. `qlinear` takes `.exl3` weights, with fp32 outputs as NVFP4 has them. `experts.Grouped` runs routed experts with each expert's gate, up and down at its own width. A checkpoint's trellis groups load from its safetensors headers. `tf-cuda-test exl3` checks the kernels against float64 references, and `exl3-tp2` checks the GLM split rules over two ranks. GLM-5.3-Flash's two-rank engine (below) runs its routed experts on this common module; no other family serves EXL3 natively yet (#837, #806, #741, #740). Ours: packed sign planes load, the scan, scratch and uploads keep their owners on every failure, and an upload needs a finite buffer budget. #806's kernels are a second linear behind `Dense`, used only when a caller asks, until the two are compared on the same checkpoint tensors. Grouped's up output is bf16, so it refuses a call that asks for the fp32 activation pipeline. Our kernel tests pass on an RTX PRO 6000 (sm_120) and a GB10 (sm_121). #837's kernels port the Python engine's EXL3 module (#42). Thanks to @justincasey, @BHCC2025 and @vcruz305.
- On Metal, the EXL3 projection decodes each weight tile once for a prompt window of 32 or more plain rows and shares it across four SIMD groups. Shorter windows, routed rows and indexed experts keep the eight-row kernel, and so does a device that cannot build the 128-thread pipeline. A row's bits stay the same either way (#729). Ours: the Metal projection loads from the same checkpoint scan and parts as CUDA, with its sign planes kept packed. Thanks to @mrv777.
- CUDA keeps a host-to-device copy's source until its stream proves the copy complete. Temporary uploads take pinned stages that one owner holds and hands to the next upload once their copies are done, so a decode step does not allocate and free pinned memory per copy; a wait that fails leaves the source quarantined instead of freeing it under a running copy, and `DeviceBuffer.fromHost` frees a buffer only after its copy completes. Uploads refuse a temporary source while a graph capture applies: replayed inputs live in stable stages.
- The EXL3 routed experts' SwiGLU clamps keep a NaN, as torch.clamp does, where they turned it into the limit; on the decode path a limit of 0 now means none, as it already did for prompts.
- GLM-5.3-Flash on two CUDA ranks reads Mia-AiLab's EXL3 checkpoint, its 4-bit mcg routed experts on the common EXL3 module, and with `--vision` takes images and video: rank 0's tower encodes them (video needs libtfvideo) and sends their rows and spans to the other rank. The ranks watch each other, so a rank that fails stops the other instead of leaving it waiting, and they agree on the checkpoint's bytes and every media limit before either loads; verify windows reach the decode buffers' 8 rows, and the bf16 linears run on glm_mm's tiles (#855, #976). Thanks to @BobClawblaw.
- Qwen3.6-35B-A3B serves on the Qwen CUDA engine through the native server: its routed experts in 4 or 3 bits behind a softmax router (the 3-bit planes run beside a 4-bit shared expert), the MTP layer beside the checkpoint drafting chains and trees, and the dense family's prompt cache and image tower. On a GB10 drafted replies equal plain ones, 17 of 17, and concurrent replies their runs alone (#856, #972). Ours: its router, top-k and combine launches and its MTP layer's are in the CUDA kernel contract, each against float64, with records for nvcc 13.0 and 13.2. Thanks to @BobClawblaw and @CanadaApollo6.
- A backend can ask the lane core to send a stream's first token before it prepares the first drafts, on a whole prompt pass or its last chunk; the Qwen CUDA backend does. The receipts are in `docs/receipts/qwen-cuda-first-token/` (#932). Thanks to @jrb-ai.
- Nemotron 3.5 Bolt (74B-A9B) serves on CUDA from NVIDIA's 24 GB checkpoint: puzzle layers, LatentMoE with its experts in GGML's IQ2_XXS and Q8_0 blocks on kernels of ours, image input, prompt states in pinned host memory on a card that cannot hold the prompt cache (the startup line says where they went), shared rounds of streams, and structured output and forced tool calls (#897, #938). On an RTX PRO 6000 (sm_120, nvcc 13.2) its drafted and plain replies equal those of #897's own build, concurrent replies equal their runs alone, and decode runs at 153 to 155 tok/s against that build's 144 to 146, with the same tokens. Ours: a failed load keeps its destinations and GPU parents until recovery proves them unused, the released router top-k kernel stays as it was beside a wide one for Bolt's 496 experts, and a negative `moe_latent_size` is refused before it becomes a width. Thanks to @BobClawblaw and @MiaAI-Lab.
- Nemotron Omni takes video through the RADIO tower: frames sampled into groups the tower reads as images, each with its time, and video tokens budgeted apart from image tokens; the reserved video token in user text stays text (#940). Ours: the video config, frame rate, indices, dimensions and grid are refused before any decode or cast. Thanks to @MiaAI-Lab and @cwschroeder.
- Gemma 4 serves natively on Metal: the 31B with its own dense and fused kernels on the tensor units, prompts in 512-row chunks, DFlash drafts, images and structured output (#879), and the 12B (`gemma4_unified`) with its unified text and image embedder and its MTP assistant as a native drafter (#975). Gemma's arithmetic runs in its own kernel copies, so no other family's bits move, and its kernels are in the Metal kernel contract (on M5, 12 of 12 at two seeds). Ours: `config.json`'s context and EOS ids are bounded before their casts, and the packed buffers and private pipelines keep one owner through every failure. Thanks to @BobClawblaw.
- Step-5 serves on one 256 GB Mac with Metal: a native `step5_vl` family with MTP drafts, images and kept prompt states (#903). Ours: the experts, every tensor's axes, the image tower's 667 tensors and the projector's width are checked before any model or device work. Thanks to @BobClawblaw.
- MiniMax H3 / FastH3 runs on Metal in bf16 as a test tool (`tf-h3-dit`): clips from text or from an image, any step count with its schedule and AdaLN tables, the released MiniMax-H3 weights, re-voicing a finished clip and Ref2VA's reference pictures, videos and audio (#961, #949, #950, #959, #960). Ours: the counts, held media, tiles, checkpoint shapes and schedule are checked before any shader or scratch, and a reschedule replaces every table together. Its tensor-unit kernels pass the Metal kernel contract on M5 (15 of 15). Thanks to @drowzeys.
- MiniMax H3 / FastH3's transformer blocks run on one CUDA device in bf16 as a C library for a host pipeline (`zig build tf-h3`, `libtf_h3.so`): products on the runtime's cuBLASLt with fp32 sums, the routed attention and every kernel under the CUDA kernel contract (#958, #947). Its sequence-parallel exchange over RoCE waits for a follow-up. Ours: checkpoint and projection sizes are checked before any allocation, only complete blocks load, and a failed load keeps its buffers and parents until their copies are proven done. Thanks to @drowzeys.
- Flash Next on Metal serves on shared sessions with an explicit `--parallel N` above one, their contexts following the memory the load leaves (`FLASHNEXT_SLOT_CONTEXTS` sizes them by hand): a lone request runs on the engine's own rounds and joins the shared ones when a second arrives, and every stream's MTP drafts go in one command buffer (chained batch drafts are experimental) (#606). Ours: `--parallel auto`, the default, keeps the single host, as on CUDA, since on an M3 Ultra a lone request decoded 7% slower in a shared session (plain 79.6 against 86.2 tok/s on the MLX 4-bit checkpoint) and the single host decodes as 1.0.12 does; with `--prompt-cache-gib` unset the shared host reserves the prompt cache first, by the single host's rule, and sizes the sessions from what is left; the shared rounds read the checkpoint's embedding, PLE and expert widths and the PLE scale as the single host does, so on the MLX 4-bit checkpoint plain, drafted and concurrent replies equal 1.0.12's. On a 256 GB M3 Ultra that checkpoint holds 129.6 GiB and 16 sessions of 65,568 tokens 31.6 GiB more. Thanks to @cwschroeder, @sethforprivacy and @BobClawblaw.
- Nemotron's FP8 and NVFP4 ModelOpt checkpoints serve on cards without thread-block clusters (sm_80 to sm_89): the lane matmul writes its split-K sums as fp32 partials and reduces them, each prompt segment through its own buffer's (#901). Ours: a reducing launch with no partials buffer is refused before it starts (`MissingLanePart`). On a GB10 Lightning's replies are unchanged and its device memory grows by the partials' 21.9 MB. Thanks to @BobClawblaw.
- A checkpoint file's content digest is kept on disk with its identity (device, inode, size and both times to the nanosecond, in `~/.cache/tensorfold/file-digests/v1`) and read again only when the file changes; a corrupt or foreign index is ignored. Every family's checkpoint identity reads its digests from there, and GLM-5.3-Flash's two CUDA ranks compare the checkpoint's whole content with them before either loads: on a GB10 pair the first start reads the 164 GB EXL3 checkpoint in 199 s, and a second start reads no shard (0.1 s) and is ready in 169 s against 370 s (#900). Thanks to @Bizuayeu.
- Fish Audio Drama 3 text to speech on CUDA and Metal: `POST /v1/audio/speech`, with `stream: true` and `response_format: pcm` sending raw PCM chunks as they decode, `drama3-generate` on both, and the checkpoint and `codec.pth` read without Python (#927, #956, #939, with #915 to #926 beneath them). Its authors measured on a GB10 a warm real-time factor of 0.655 against 1.087 before the one-row kernels, and a load of about 6 s against 53 s with direct I/O into staging pieces; a reply that reaches `max_frames` returns its clipped audio. The CUDA build keeps a process-wide cap on device and pinned host bytes (`setAllocationLimit`, unset by default). Ours: the stager's copies go through the copy-source owner, and Drama's kernels are in both kernel contracts with nvcc 13.2's records. Thanks to @plotarmordev and @drowzeys.
- Qwen-Image-2.1's transformer runs on Metal as a test tool (`tf-qwen-image`) on cases `tools/zig/qwen_image_case.py` makes with mflux, which also encodes the prompt and decodes the image (#948). Ours: the latent grid, scratch sizes and `eps` are checked before any device or checkpoint work. Thanks to @drowzeys.
- `tensorfold benchmark` reads `config.json` through the same 64 MiB metadata reader as `info` and `serve`. A missing config still reports an incomplete checkpoint; an oversized, unreadable or malformed one reports its own error with its size, where the file system gives one, and the limit. It read at most 4 MiB and called every failure an incomplete checkpoint (#934). Thanks to @jkuepker.
- When a benchmark run exits 2 it says why: incomplete attempts stay in the saved receipt, a cell has aggregates only when every repetition succeeds, and an early EOS is left out of full-length comparisons (#969). Thanks to @BlivionIaG.
- Nemotron on CUDA keeps the attention caches of the streams beside the engine's own in pages of one pool the core owns: a stream leases pages as its sequence grows and gives them back when it ends, a failed lease rolls back whole, and startup sizes the pool from what the model, the engine's own window, the prompt cache and one staging buffer leave, refusing with the bytes it needs when the room is short. `TF_KV_PAGES=0` gives every stream its own window as before, `TF_KV_PAGES=all` puts every stream on pages, and `TF_KV_POISON=1` fills freed pages so a stale read shows (#935). On GB10, Nemotron's replies, concurrent replies and prompt-cache counts are the same as before the pool in every mode, and Super VL's prefill is within 0.2% of 1.0.12's. Thanks to @jschmied and @cwschroeder.
- GLM keeps a long session's next turn on its cached prefix: when the pooled allocator needs room it moves the turn's own prefix instead of evicting it, and a restore that fails says once why, with the state's length and never its text (#966). Thanks to @tpischke and @heitke for the reports.
- GLM on Metal gathers a prompt's routed experts at any MLX affine width from 2 to 8 bits and group of 32 or 64, gate, up and down each their own; only uniform 4-bit and 2-bit experts in groups of 64 took the gathered path, so a checkpoint such as 3-bit gate and up with a 4-bit down fell back to the sorted kernels. All 468 checked layouts equal independent row products bit for bit (#965, #968). Thanks to @harrisonfriia.
- GLM on Metal runs a long prompt in chunks between the other streams' decode rounds, at most 256 rows a chunk while another stream decodes, so a stream beside a 100k-token prefill no longer stops until it ends; the shared fill selector lets the longest-waiting prompt in after eight skipped choices (#964). Thanks to @tpischke for the report.
- `tool_choice: "required"` and a named function constrain the reply on every backend with a token grammar, Gemma's and unrecognized chat templates included, and on Qwen3.8-27B on Metal, whose drafts grow under the same grammar so a drafted reply equals the plain one. A backend without a grammar refuses before generation and names itself (#918). Thanks to @astro-15-alive for the report.
- Qwen3.8-27B on Metal shares each round among up to 128 streams, each drafting its own DFlash2 tree in one drafter frame, verifies trees of up to 64 rows and takes prompts in 512-row passes; a lone stream also copies its context's continuation into its drafts. Drafted replies equal plain ones, and each concurrent stream equals its reply alone. On an M5 Ultra with TensorFold's MLX 4-bit checkpoint and z-lab's DFlash2 drafter at `--parallel 8`, against 1.0.12 on the same machine and on the cells #918 asked for: cold prefill 931 to 1,349 tok/s at 8,192 tokens and 891 to 1,250 at 30,720; 512-token greedy decode 116 to 126 tok/s for prose and 386 to 432 for code (97 to 104 and 305 to 325 at t=0.7); eight streams at once 222-239 to 314 tok/s together greedy and 232-235 to 292-296 at t=0.7. A warm resume's first token stays at 0.12 to 0.18 s. The second stream's DeltaNet states are allocated at start, so at `--parallel 2` the server holds 26.5 GB where it held 22.6 GB. Its five register-slot DeltaNet pipelines are in the Metal kernel contract against float64, on an M3 Ultra and an M5 Ultra, with their M1 and M2 thread limits from offline compiles.
- RUNBOOK.md's heat bands section adds one measured run: on three GB10s serving GLM-5.3-Flash NVFP4 at TP=3, a 1,035,295-token prompt without the bands kept one host's hottest zone at 94 C or more for most of the prompt, and `TF_HEAT_HIGH=92` with `TF_HEAT_LOW=88` held all three at 93.0 C or under, the first token 10% later (#988). Thanks to @Bizuayeu.
- `docs/receipts/gfx1151-qwen-833bbfcb/` qualifies 1.0.12's opt-in HIP build on one gfx1151 (Radeon 8060S) for Qwen3.5-9B, Qwen3.6-35B-A3B with MTP and Qwen3.8-27B: 130 model checks pass and 11 are skipped, every HTTP reply's full token digest is kept, and the 27B's cancellation failure and the 9B's and 27B's drafting comparisons, which did not run, stay listed as open (#994). Thanks to @ThinkOffApp.
- Nemotron's latent MoE on CUDA: every segment of a prompt split into segments reads the shared-expert plans and pair ids its owner builds, where the other segments' buffers carved fresh ones that nothing wrote (#996). Thanks to @plotarmordev.
- The benchmark's refusal-body test holds its fake server's sender on an event after the header, so it proves the body is not buffered instead of waiting 20 ms (#998). Thanks to @cwschroeder.
- Nemotron 3.5 Super on CUDA has decode-sized forms of its latent top-k (each expert ranked on a thread of its own) and of the checkpoint's NVFP4 experts (32 columns a warp, four steps in flight), each bit-equal to the kernel it replaces and in the kernel contract, with 1,024 experts, 32 picks and rows without a finite score among its cases (#999). Thanks to @plotarmordev.
- Nemotron's tensor-parallel worker sends a job's first message with one row, so a decode step is one message where it carried 16 rows of padding; larger jobs still send their second message (#1000). Thanks to @easthambuilds.
- WebP images decode through libwebp 1.6.0's `libwebpdecoder`, loaded at the first WebP from `TENSORFOLD_LIBWEBP`, `lib/` or the system's library path and refused by name when missing or another release, to the RGB bytes Pillow 13.0.0 gives the same file (lossy and lossless, with and without alpha, EXIF-oriented); animated WebP is refused, and TensorFold ships no copy of the library (#1004). Thanks to @BobClawblaw.
- GLM-5.3-Flash on CUDA keeps its index keys and gates in 4,096-row rings that rebase before a pass would cross them, 0.674 GiB a stream at a 40,960-token window, so `--parallel 16` fits and 16 streams share one verify of 2 rows each (#984). Ours: an MTP rebase keeps the whole verify window before its cursor, so a rejected proposal rolls back into rows the rings still hold. Thanks to @BobClawblaw.
- GLM-5.3-Flash on CUDA can pack its dense linears to 4, 6 or 8 bits at load (`TF_GLM_DENSE_BITS`, `TF_GLM_HEAD_BITS`; 16, bf16, stays the default), the ranks agreeing on the bits; `TF_ROCE_MAX_KB` caps a RoCE gather's shard, 4,096 KiB by default; and NCCL's cuMem allocations are off on the ranks, which on GB10 exhausted the driver's pool under load (#995). Ours: a malformed, zero or overflowing cap is refused before any allocation, and the packed launches are in the kernel contract with their records measured on nvcc 13.0 and 13.2. Thanks to @BobClawblaw.
- GLM-5.3-Flash on CUDA drafts with DFlash2 (`--drafter`): taps after five layers, per-slot context rings, a block pass split across the ranks and drafts sized by their chances against measured window costs, its linears at 4 bits by default (`--drafter-bits 4|0`) (#979). Ours: the rank buffers share one layout for 1 to 16 ranks, a failed load frees what it allocated, and the drafter's launches are in the kernel contract with their records measured on nvcc 13.0 and 13.2. Thanks to @BobClawblaw.
- Gemma 4 on Metal reads every MLX affine width the core supports, each module's width from the config, with stacks of mixed widths in runs; the 12B converter takes mlx-lm's mixed recipe, and the recipes record the widths checked against the oracles, mlx-lm and transformers (#977). Ours: a failure at any width frees the sets built before it, and every dense and embedding width is in the kernel contract, pending its records. Thanks to @BobClawblaw and @gilby.
- GLM's prompt cache spares a kept state that an outstanding lookup still restores, and a failed restore says why (#1007). Ours: eviction relocates the selected prefix before it takes a borrowed neighbour, and a failure is reported once. Thanks to @heitke.
- The app leaves the engine's prompt cache to the native planner instead of asking for 8 GiB, which the engine refused on a Mac whose load and streams left less room (reported in #989). Thanks to @Deesha08.
- `--mtp-drafts` and `--mtp-confidence` set a CUDA Nemotron stream's MTP depth and the cumulative confidence its drafts need (#997). Ours: they reach the shared lane rounds as well as a lone stream. Thanks to @plotarmordev.
- Nemotron 3.5 Super on CUDA verifies windows of up to 32 rows, 31 MTP drafts a round, with its Mamba scan in stages of 32 held to its host twin's bytes in the kernel contract, and a lone stream's copied and Recall chains start at 15 tokens (#1006). Ours: a checkpoint without an MTP head keeps its shared copy lane, and only Super's latent models widen to 32 rows. Thanks to @plotarmordev.
- Nemotron's latent shared-expert plans for a decode window are built once at buffer load, one per row count, and on one GPU the shared expert runs on the side stream beside the routed experts (#1002). Ours: shared windows of several sequences and two-rank windows keep the earlier schedule. Thanks to @plotarmordev.
- An attached benchmark run reads the server's `GET /v1/benchmark/manifest`, so a two-rank CUDA run can be ranked when the manifest's facts pass the receipt checks (#980). Ours: an unknown backend is never attested as CUDA, and an attached run whose accepted manifest lacks its rank count or revision says it is unranked. Thanks to @easthambuilds.
- Nemotron 3.5 Super on CUDA drafts with its MTP head over the embedded draft vocabulary's FP8 output rows: a stream drafts on the subset head while its last 16 context tokens are draft ids, else on the whole head (#1011). Ours: a failed head construction drains before it frees anything, and gathered rows are checked before any read. Thanks to @plotarmordev.
- Nemotron 3.5 Super's checkpoint prompt passes on CUDA run the FP8 layers in 128-row tiles and the NVFP4 experts in 128-pair by 128-column tiles, up writing down's NVFP4 rows and one GPU's down summing the width shares into bf16, and router logits past 32 rows run on 128-row by 128-expert tiles, each bit-equal to the kernels it replaces in the kernel contract (#1008). Thanks to @plotarmordev.
- Windows ARM64 builds the native CUDA engine from source, with NVIDIA's RTX Spark N1X (sm_121) among its targets; the device results in `docs/WINDOWS.md` are the contributor's reports (#1010). Ours: copy custody takes its locks and waits from std.Io's threaded mutex and condition, which Windows has; a Windows integrated GPU sizes its pool from CUDA's counts; and a library path resolves in full before it loads. Thanks to @mweinbach and @BobClawblaw.
- The lane core's allocate cuts a grafted window's held count and branch start with its rows, so a shared round that trims a window with suffix-match grafts replies as the one-token decode does (#1013). Ours: the family trim path takes the same cut, so a one-draft cut keeps one draft and the first remaining graft keeps its index. Thanks to @chaog992.
- In the lane core, the grammar's own drafts after a drafted window that is a chain but carries tree parents, as a pruned tree or a suffix-match graft from the pending token can, extend that chain. The core read their parents past the end of the window's list and failed the round (`ParentAfterChild`), which the 27B's required and named tool calls on Metal met with drafts on.

### Moving to 1.1.1

These changes are not shipped in 1.1.0 and remain conditional on their final gates; a failed gate moves the work to the following patch release.

- Living Weights retention, rounded commit/rollback, conversation and supplied-question work, and the Flash Next/Qwen CUDA learners (#876, #877, #878, #891, #905, #992, #895, #896, #911). The existing learner's #910 correction above remains in 1.1.0. Thanks to @ZackSample, @philip-pentatonic, @akol1 and @BobClawblaw.
- Flash Next on Metal's images/video and million-token window (#986, #1005), still held for the rotary-bound contract and actual probe rerun. Its separate shared-session cache-count and lone-decode repair is held for a later 1.1.x, outside the current 1.1.1 series. Thanks to @BobClawblaw, @cwschroeder and @sethforprivacy.
- The larger Qwen CUDA lane rounds, verification trees, batched drafting and lone-stream speed work (#978), distinct from the Metal 27B speed work shipped above. Thanks to @BobClawblaw and the shared CUDA contributors.
- DeepSeek-V4.1 runtime and family integration (#635, #636, #637, #663, #682, #667, #668, #671, #674, #686, #688, #691, #693, #694, #695, #699, #702, #703, #707, #727, #824, #825, #826, #827), and Kolibri text CUDA (#991). Shared contracts do not announce a complete DeepSeek Metal backend. Thanks to @jayleaton, @akol1, @ajensenwaud and @jschmied, retaining the common operator attributions.
- GLM CUDA fresh-prompt batching, prompt scoring, structured output and required/named tool-call follow-ups (#1014, #1016, #1017, #1018). Thanks to @BobClawblaw and @Bizuayeu.

MiMo-V2.6-Flash (#728) is held for a later 1.1.x, outside the current 1.1.1 series: its eight native CUDA launches, then its model and device gates, remain owed, and its source in this release is not a shipped native backend. Thanks to @ajensenwaud and @jayleaton.

Known issues:
- GLM-5.3-Flash's ModelOpt NVFP4 checkpoint on two CUDA ranks was not run for this release: on two GB10s the weights leave no room for its streams beside the memory reserve. Code that path shares changed in this release (NVFP4 scale planes and their tensor-parallel slices, the copy-source owner, NCCL's allocations and the RoCE cap on the ranks), so it stays a known issue until its regression run; the EXL3 checkpoint is the two-rank path measured for this release.
- The reported M3 Ultra peak for Qwen3.8-27B on Metal is 4.1 GB higher than 1.0.12 because the second DeltaNet bank is allocated at startup; reducing that allocation remains follow-up work.
- Flash Next on Metal with explicit --parallel 2 or more retains a cache-count/performance issue: on an M3 Ultra with the MLX 4-bit checkpoint, 9 of 12 chat steps reused 17 to 312 fewer tokens than the single host, with the same output tokens, and a lone request decoded 7% slower. The default --parallel auto uses the single host and matched 1.0.12's cache counts; the separate shared-session repair is held for a later 1.1.x and is absent from the current 1.1.1 series.

## 1.0.12

- Nemotron serves MLX checkpoints with a width per module on Metal: TensorFold's oQ4, oQ6 and oQ8 quants, and any MLX affine checkpoint at 2 to 8 bits (not 7) in groups of 32, 64 or 128. 1.0.11 refused every one of them at `config.json` ("the native Nemotron kernels read every matrix at the checkpoint's 4-bit in groups of 64"), and refused a uniform 8-bit checkpoint with a reason about its shapes. A matrix at 4-bit in groups of 64 runs on the kernels it ran on before. In decode, every other width runs on the shared row kernels. Prompt chunks past 128 rows run the other dense matrices on simd_qmm_bits' tiles where their groups are 64, and the routed experts on Flash Next's width-general expert gathers, compiled at load for each width the checkpoint uses. The embedding has a lookup for any width, a mixed q/k/v stack runs member by member, and the routed experts run by expert id. On an M3 Ultra, oQ4, oQ6 and oQ8 each gave drafted replies equal to `"draft": false` on 17 prompts, 2 and 4 concurrent streams equal to their solo runs, and 12 resumed chat turns equal to a cache-off server's. Teacher-forced against mlx_lm's forward of the same checkpoint on 3,072 positions, their mean KL is 0.0050 to 0.0054 nats a token, with the same top token at 96.2% to 97.0% of positions; the 4-bit checkpoint gives 0.0056 and 96.2% through the same tool. Prompts of 8,192 to 32,768 tokens read at 0.96x to 0.99x mlx_lm's rate on the same checkpoint. Greedy prose decodes at 162 tok/s with oQ4, 117 with oQ6 and 111 with oQ8, against 188 with the 4-bit checkpoint, and greedy code at 249, 183 and 174 against 268. The oQ checkpoints carry no MTP head, so copy `mtp-4bit.safetensors` from the 4-bit checkpoint into the folder to draft.
- For Nemotron's MLX affine checkpoints, CUDA and the Intel GPU engine read 4-bit in groups of 64 only, and refuse a checkpoint at any other width by naming its first such module; NVIDIA's ModelOpt checkpoints are read as before. The Intel GPU engine had no check before and would have read such a checkpoint at the wrong width. On Metal, `--slide` needs every matrix at 4-bit in groups of 64, and loading saved learned neurons needs the shared expert's up and down matrices at that width; both refuse by naming the module.
- `tensorfold-native capabilities --json` lists every MLX affine width for Nemotron, Flash Next and qwen3_5 checkpoints on Metal. The app's download check compares a checkpoint's top-level width with that list, which named only 4-bit group 64 for qwen3_5 and 6-bit group 32 for Flash Next, so it turned away Jundot's Qwen3.8-27B oQ6e and every Flash Next oQ checkpoint that the engine serves. On an M3 Ultra, Jundot's 27B oQ4e and oQ6e with DFlash2 and a Flash Next oQ5e each gave drafted replies equal to `"draft": false`, concurrent streams equal to their solo runs, and resumed chats equal to a cache-off server's.
- Nemotron 3.5 Super VL reads images on CUDA with `--vision`, through the checkpoint's own RADIO tower and pixel-shuffle projector in TensorFold's kernels (`zig/src/families/radio/`). The processor is the checkpoint's: each image becomes one tile of up to 13,312 patches of 16 pixels (an even number each way; at least 1,024, except that a picture smaller than that sent with `"detail": "low"` keeps its own grid, down to 2 x 2 patches and one token), resized with torch's antialiased bicubic on float samples and normalized with CLIP's mean and deviation, and on a GB10 its pixels equal the checkpoint processor's bit for bit on six pictures, five resizes and 3,000 patch grids. An image takes a quarter of its patches as tokens (at most 3,328), written between `<img>` and `</img>`. The tower runs the checkpoint's bf16 weights as stored: against a float32 run of the checkpoint's own modules on the same pixels, its rows' error is 0.99x to 1.03x the modules' own bf16 run's. Served on a GB10, image requests drafted equal `"draft": false`, four concurrent image requests equal their solo runs, a later turn resumes past its image while another image under the same placeholders resumes only the text before it, and two GB10s give one GB10's replies word for word. A 13,000-patch image's tower and projector take 0.87 s on a GB10. Video and audio parts are refused for now. The vision towers' CUDA glue is shared (`cuda/vit.zig`): LFM2-VL's moved there unchanged, and a new attention kernel for 80-wide heads gives the old kernel's bytes 1.35x faster.
- Nemotron 3.5 Super VL serves on CUDA from NVIDIA's NVFP4 checkpoint (`nemotron_h_omni`), on one GB10 or two. Its latent MoE (512 routed experts, 22 a token, read and written in a 1,024-wide latent between two bf16 projections, beside a shared expert) runs in TensorFold's kernels with each expert's width cut into two shares that every setup sums in share order, so two GB10s give one GB10's tokens: all 17 replies to the eight release prompts (greedy, seeded sampled, thinking on) matched, in both precisions below. On two GB10s rank 0 runs the model and serves, and rank 1 holds only its share of the experts (31.5 GiB, against rank 0's 40.1) and answers each MoE layer's job over NCCL. Before any weight loads the ranks agree on the build, the model, its cut, the precision and the checkpoint's layout (config.json, the index, each shard's name, size and header); while the weights load each rank hashes every byte of every shard it holds (sha256, each file checked against its Hugging Face blob name in a snapshot), and the ranks compare those digests before rank 0 serves, so a copy one bit off anywhere stops both ranks before serving, with its name (a GB10 hashes the 70 GiB checkpoint in about 22 s, at the same time as it loads its weights); a TCP line opened before NCCL then carries ready and error records, so a rank that fails stops the other with its error's name, and one that dies stops the other within about a second instead of leaving it waiting. Served at a 16,384-token window, drafted replies equal `"draft": false`, 2 and 4 concurrent streams equal their solo runs, and chats with regenerate, edit and delete equal a cache-off server's replies, on one GB10 and on two. MTP drafts give 38.2 tok/s on one GB10 and 46.8 on two (greedy, mean of eight prompts), against 18.8 and 20.2 with `"draft": false`; the MTP layer drafts with the target's own output head. Super VL's `--vision-image-tokens` defaults to 13,312, so up to four images keep the size NVIDIA's processor gives them: four full-size pictures on one GB10 took 13,124 prompt tokens and 11.45 s to the first token. The [recipe](docs/recipes/nemotron-3.5-super-vl.md) has both setups.
- On CUDA, `--precision checkpoint|full` chooses an NVIDIA NVFP4 checkpoint's math. `checkpoint`, Super VL's default on Blackwell, is NVIDIA's own: e4m3 rows by each FP8 layer's stored input scale into an FP8 x FP8 mma, and NVFP4 rows (an e4m3 scale every 16 values, e2m1 rounded to nearest) into its NVFP4 experts on the block-scaled FP4 mma (`cuda/ck.zig`, compute capability 12.x). `full` keeps bf16 activations (W4A16 and W8A16). On a GB10, `checkpoint` against `full` gave a mean KL of 0.0077 nats a token and the same top token at 98.4% of 768 positions; a cold prompt read 1.81x faster (8,192 tokens in 4.41 s against 7.96 s; 1.82x at 2,048), and serial decode took 52.3 against 50.1 ms a token. Each mode is exact within itself, and the [quantization guide](docs/quantization.md#nvidias-nvfp4-checkpoints-on-cuda) has the measurements and `tensorfold kl`, which compares two precisions on the same tokens. Lightning's NVFP4 checkpoint keeps `full` for now.
- Living Weights keeps a way back. Before a folder's first lesson, `--slide` keeps every file a lesson can change (its safetensors files, `model.safetensors.index.json` and `config.json`) in a hidden `.tensorfold-original/` folder inside it, with a `SHA256SUMS` manifest checked against the kept files: clones on APFS, Btrfs and XFS, and on other file systems copies after a free-space check, refused with HTTP 507 and the space needed when they do not fit. On an M3 Ultra, Nemotron 3.5 Lightning's 7 files (18.5 GB) took 5.3 s as clones and no disk; on a GB10's ext4, in a build for its own CPU, they took 35 s as copies and 18.5 GB (the Linux release computes SHA-256 about five times slower). Later lessons reuse the snapshot, and each server checks it against its manifest at its first lesson (2.6 s on the M3 Ultra), refusing to teach while it does not match. Model loaders read only a folder's own files, and `tensorfold info` leaves hidden folders out of a model's size.
- `tensorfold slide rollback FOLDER` undoes every lesson since the snapshot. It checks the snapshot against its manifest, makes every clone and checks the free space for any copies before it changes a file, puts each file back, checks every file's SHA-256, and only then removes what the lessons wrote: lesson files, writes cut short, and the fact graph files `--slide-graph` named, printing each path, including any outside the folder. Files a user added, files a lesson never writes and the snapshot stay. It took 5.6 to 5.7 s on the M3 Ultra (clones) and 58 to 91 s on the GB10 (copies, one file at a time), and on both the folder came back byte for byte and, served again, gave the untaught folder's replies token for token.
- `--slide` starts only with a key: `--api-key`, `--api-key-file` or `TENSORFOLD_API_KEY`. The key is read before the model loads, with the rules the server applies, so a key file that is missing, empty or readable by other users, or a key with whitespace in it, stops the start at once with a line naming the cause. The Living Weights guide and the README show a key file.
- Each server holds its model folder while it runs (shared, or exclusive with `--slide`): a server refuses a folder that a `--slide` server or a rollback holds, a rollback refuses a folder any server holds, and `--slide` refuses a folder that cannot be locked, where a plain server serves it and logs one line. The hold is advisory, so other tools that write the folder do not see it.
- Without `--slide` replies are unchanged: on an M3 Ultra and a GB10 this build gave 1.0.11's tokens in 17 exact cells, drafted and with `"draft": false`, and 12 multi-turn steps.
- `TENSORFOLD_REQUEST_LOG` creates its file owner-only (mode 0600, where it was 0644), so the prompts and tool content it records are not readable by other users on the machine. A log file that already exists keeps its mode; `chmod 600` it once.
- Recall, off unless asked for: `tensorfold serve MODEL --recall` keeps a record of the token ids the server has read and written, one record per tokenizer under `~/.cache/tensorfold/recall` (`--recall-dir` names another), and drafts from it. The copy window asks a stream's own context first and then the record; grafted lanes merge the record's distinct continuations with the context's, the context's first on ties. A copy goes in only when its expected tokens, at the landing rate earlier copies of its match length have shown, beat the stream's own rounds. A copy that misses still costs its round the drafts it took the place of, lowers its match length's rate and keeps that stream off the record for four rounds; once no rate pays, the copy window walks the record only for a probe, one ask in 32, backing off to one in 1,024 while probes miss. Grafted lanes, which no served engine turns on yet, walk the record at each ask while its grafts land and every eighth round otherwise, at most 64 candidates a walk. Every copied token is verified, so replies are the same tokens with Recall on, off or `--recall-read-only`. On an M3 Ultra Qwen3.8-27B wrote four requests repeated after a restart at 16 tokens a round against 3.2 to 6.5 with Recall off, 2.4x faster, and on a GB10 Nemotron 3.5 Lightning at 15 to 16 against 2.4 to 3.3, 3.0x faster; the same requests with one detail changed ran 0.99x to 1.02x, re-edits of an earlier file 1.01x to 1.14x, and fresh prose and chat beside a populated record within 0.3% of Recall off. With Recall off, replies and speed match the build without it on both chips. Each request adds what no earlier request read, from its prompt-cache resume point, and its reply when the reply ends; runs the record already holds are not added again, so a repeated request adds nothing. The record holds token ids only, in 0700 and 0600 files; each 4 MiB file of 1,048,576 ids takes 8 MiB of memory, plus a key table (8 MiB at the default cap), and past `--recall-gib` (1/16 GiB by default) the oldest file goes. `GET /v1/recall` reports it and `DELETE /v1/recall` empties it; [Recall](docs/recall.md) says what stays on disk and how to run benchmarks beside it. On CUDA a lone Nemotron stream's own driver reads the record too; Flash Next refuses `--recall` before it loads, since its replay engine copies on the GPU from its own history. A reply's `speculative.proposer` counters are now written from the request's own copy; they were parsed from a buffer the server freed first, which no reply showed until an engine sent counters.
- `GET /v1/models` tells a client which reasoning efforts each model takes, so a coding agent's effort picker can come from the server instead of a table kept by hand. An entry for a model whose chat template has a thinking switch, names effort levels or has a `</think>` token carries an OpenRouter-style `reasoning` object: `default_enabled` (false under `--no-thinking`), `default_effort` (only with `--reasoning-effort`) and `supported_efforts`, the levels the template names, highest first, then `none` when the template has a thinking switch. Qwen3.8-27B and Qwen3.8-Flash-Next list xhigh, medium, low and none; GLM-5.3-Flash lists max, high and low; Nemotron 3.5 Lightning and Qwen3.5-2B, whose templates name no level, give `default_enabled` alone. The server reads the template once at startup, and no reply changes (#819). Thanks to @kelchm.
- On CUDA the NVFP4 experts' prompt entries take the `skip` their decode entries already took and leave that expert's items alone, so a family that computes one expert itself, as GLM-5.3-Flash does with its BF16 shared expert, can plan an NVFP4 prompt's pairs together. `tf-cuda-test quant-ref` checks it for FP8 and NVFP4: the skipped expert's rows keep what the buffer held, and every other row equals the run without skip. Nemotron skips no expert, so its replies are the same tokens, and on a GB10 its NVFP4 cold prompts of 2k to 64k tokens ran within 0.3% of before (#764). Thanks to @Bizuayeu.
- Nemotron decodes faster on CUDA with our own kernels. A decode window's router (16 rows or fewer) stages its rows and 16 experts at a time with every load in flight and runs the old router's sums from shared memory, top-k loads every slice and bias before summing them, and add_moe_norm loads and adds a row on 1,024 threads. Every sum keeps its order, so the outputs are the old kernels' bytes: on a GB10 they matched the previous build and the host references at every row count from 1 to 4,096, and the glue test now runs the router at every decode window size and at an unaligned window, which keeps the old router. With Nemotron 3.5 Lightning one stream decoded about 0.9% faster drafted and 1.5% faster with `"draft": false`, on MLX 4-bit and NVFP4 alike, with the same rounds on the same window-cost table; shared rounds of 2 to 8 streams and cold prompts of 2k to 64k tokens ran within 0.6% of before or faster (#833). Thanks to @jschmied.
- GLM-5.3-Flash's KDA and router kernels ask the Metal compiler for the 1,024 threads a threadgroup they launch with. On an M1 Max the compiler had built both KDA pipelines for 512 threads and several router widths for fewer than 1,024, so those launches could leave outputs unwritten without an error. The kernels' arithmetic is unchanged, and `launch_check.zig` runs every router width and both KDA head counts at 1,024 threads against float64, with a 512-thread control and recurrent seams. On an M3 Ultra, whose compiler already allowed 1,024 threads, the checks pass, every GLM-5.3-Flash pipeline keeps its limit and its threadgroup memory (at most 24 KiB, under the 32 KiB of M1-family GPUs), and GLM-5.3-Flash with 8 of its 45 layers gave the same tokens as before, drafted and with `"draft": false`, resumed and concurrent, with decode and prompt speed within 0.25% (#708). Thanks to @mrv777.
- EXLR8's one-set projection kernel at 1, 2 and 4 rows alternates two input tiles, so each block needs one barrier instead of two; 8- and 16-row windows keep the old schedule. The sums and their order are unchanged: on an M3 Ultra every case of `tf-exlr8-check` gives the same bytes as before. There, over 12 layouts, the kernel takes 2% to 5% less GPU time at one row and up to 2% less at two, between 6% less and 2% more at four, and the same time at 8 and 16 rows, where the first version was 3% to 8% slower (#789). Thanks to @mrv777.
- `core/tile_qmm` adds prompt matmul tiles for Macs without tensor units, on MLX affine 4-, 6- and 8-bit codes in groups of 64: 32 x 32 output tiles with fp32 group sums in k order, tiles over rows sorted by expert with a fused gate-up SwiGLU, and a decode twin for 1 to 32 rows that gives the tiles' bits. No family uses them yet; the LFM2 family (#822) will. `tf-tile-check` checks the tiles, the twin and the grouped tiles bit for bit and against float64, and passes on the author's M4 Pro and on an M3 Ultra, with an error of at most 2.5e-3 of the terms' magnitude (#821). Thanks to @BlivionIaG.
- The MLX dequantizer in `core/quant` reads any code width from 2 to 8 bits. A 3-, 5-, 6- or 7-bit code can run into the next word, which it now reads, as the float64 reference already did, and a test checks 3- and 6-bit rows against the reference. Nothing served calls it yet, so no reply changes: the Mac and CUDA servers' code is the same with and without it, and Nemotron's tokens on a GB10 are unchanged, drafted, plain and concurrent (#810). Thanks to @BlivionIaG.
- `core/quant/experts.zig` stacks a MoE layer's routed experts with the shared expert last (gate and up fused per expert), builds its router rows with the shared gate last, and slices both for one rank with the map from global to local expert ids, for any checkpoint table. The Qwen3.5 loader of the opt-in HIP build (#626) reads its experts through it, and Flash Next's HIP loader will too. A test stacks two experts and a shared one and checks their order and a rank's ids. The Mac and CUDA servers do not call it, and their code is unchanged (#811). Thanks to @BlivionIaG.
- `core` reads NVIDIA's ModelOpt NVFP4 checkpoints as stored. `modelopt.admit` checks config.json's quantization block (top level or `text_config`'s) for NVFP4 weights in groups of 16 and refuses other ModelOpt formats by name, and `nvfp4.zig` maps a linear's e2m1 codes, e4m3 scales and fp32 scale and gives a float32 reference row. Which linears are NVFP4 comes from the tensors, since nvidia/GLM-5.3-Flash-NVFP4 at 423acf37 stores its MTP layer in BF16 without listing it. The tests read that revision's config.json and its whole tensor header, 147,661 names. The loader that calls it comes with GLM-5.3-Flash on CUDA (#758). Thanks to @Bizuayeu.
- `Checkpoint.release` drops a checkpoint's pages from the page cache on Linux (madvise and fadvise DONTNEED) once a loader is done with them: on unified memory the page cache holds memory the device needs. The mapping stays valid, and a later read faults the same bytes back in. On a GB10 it dropped all 20.1 GiB that a read of Nemotron 3.5 Lightning's NVFP4 checkpoint had cached, in 0.26 s. Nemotron's own loader reads with direct I/O and leaves nothing behind, so its loads are unchanged; the saving is for loaders that read through the mapping, as GLM-5.3-Flash's will. Ours: a file whose mapping fell back to a heap copy (Zig's fallback when mmap fails) keeps its bytes, since advising that copy away would have zeroed them (#766). Thanks to @Bizuayeu.
- `Nvfp4Experts.packGateUp` packs gate and up straight into their stacked places, with no scratch copies. A loader packs every routed expert this way, and the copies cost about 20 s of a GLM-5.3-Flash load at TP=2. The words are the same, which a host test checks against `packOne`, and `tf-cuda-test quant-ref` gives the same numbers as before (#763). Thanks to @Bizuayeu.
- `cuda/rank_scatter.zig` sums tensor-parallel ranks' fp32 partial rows in rank order over NCCL send and receive: each rank receives only its own rows and ends with the bits an all-gather and a rank-order sum give, where NCCL's own reduce-scatter adds in its ring's order. `tf-cuda-test rank-scatter` checks it on real ranks, and `tf-cuda-test rank-sum` checks the sum kernel alone on one GPU for 2 to 5 ranks. On pairs of GB10s over RoCE, run by @Bizuayeu, @cwschroeder and @SvangenStudios, every rank's rows matched that reference byte for byte at 1 to 2,048 rows of 4,096 with three seeds each, and 2,048 rows took 1.0 to 1.06 ms with four NCCL channels; ours ran on one GB10. GLM-5.3-Flash at TP over hosts is its first caller (#642). Thanks to @Bizuayeu, and to @cwschroeder and @SvangenStudios for their runs.
- The native CUDA host takes a family's prompt cache, its rules and its budget from the family's own declarations (`snap_*`, `snap_grid`, `warm_quiet`), and a family split over hosts can declare the weights its rank holds and its own text for an open error. Nemotron declares what the host used to wire in by name, so its cache, warm turns and `--slide` are unchanged: on a GB10 its multi-turn chats resumed with the same cached tokens, step for step, and the same replies as before and as a server with the cache off, on MLX 4-bit and NVFP4 (#762). Thanks to @Bizuayeu.
- On CUDA the NVFP4 lane matmul and prompt GEMM can write the fp32 sums they hold instead of rounding them to bf16 (`qlinear`'s `decodeF32` and `promptF32`), so a tensor-parallel family can add its ranks' partial rows before the one rounding; other formats refuse fp32 out by name. The bf16 kernels are untouched: on a GB10 each of them compiles to the same SASS as before, `tf-cuda-test quant-ref` checks the fp32 outputs against float64, at every row count and as the unrounded bf16 outputs, and Nemotron NVFP4 gave the same tokens as before drafted, resumed and concurrent, with one stream's decode within 0.8% of before, 2 to 8 streams within 1.7% and cold prompts of 2k to 64k tokens within 0.4%. GLM-5.3-Flash on CUDA is its first caller (#757). Thanks to @Bizuayeu.
- Flash Next prefills each chat reply for the next turn in the background, as the 27B on a Mac and Nemotron on CUDA do, so a follow-up that sends the reply back (its content, the reasoning dropped) resumes past it instead of at the turn's start; a request that arrives meanwhile goes first. On an M3 Ultra with the 6-bit checkpoint and conversations of 7,000 to 8,000 tokens, follow-ups sent a second after each reply prefilled 19 to 26 tokens instead of 223 to 284 and started in 0.17 to 0.19 s instead of 0.41 to 0.46 s. Replies were the same tokens with and without it (drafted, `"draft": false` and every follow-up), multi-turn chats equalled a cache-off server's, and decode and cold prompts of 8k to 32k tokens stayed within 0.4%.
- `--learn-conversations` keeps every prompt state a conversation leaves in RAM on disk too, written while the engine idles and finished on a planned stop, so its next turn after a restart, a crash or an eviction reads the state back where that is quicker than prefilling. Rows two states share are stored once (ten conversations with one long system prompt store it once), every piece read is checked against its hash, and a read-back state gives the same reply as a fresh prompt pass. Qwen3.8-27B, Nemotron, Flash Next and GLM-5.3-Flash on one Mac and Nemotron on CUDA keep them; `--learn-daily-gib` caps the writes a day, and `POST /v1/imprint/resume`, `POST /v1/imprint/load` and `DELETE /v1/imprint` report, load and forget a conversation's states. It is off unless asked for. Without the flag no state is written, replies and speed are as before, and the three routes answer 404 naming the flag. On a GB10 with Nemotron 3.5 Lightning 4-bit, a 39,228-token conversation's next turn, 6.2 s cold, started in 0.25 s after a restart (314 MiB read in 0.15 s), in 1.75 s after the server was killed (8,500 tokens past the last state written were prefilled again) and in 0.19 s after five other conversations had pushed it out of RAM, each time with the cold reply's tokens and logprobs. On an M3 Ultra the 27B's 21,306-token next turn, 103 s cold, started in 0.83 s after a restart (1.5 GiB read in 0.41 s), in 33 s after a kill and in 0.72 s after an eviction, again with the cold reply's tokens.
- Flash Next on a Mac keeps its memory wired between requests: its buffers are held in a residency set, and the idle keepalive's tick, now a small copy on the GPU, keeps the set resident for `--keep-warm` seconds after each request (15 minutes by default), so the first token after a few seconds idle no longer waits for macOS to wire the model again. On an M3 Ultra with the 6-bit checkpoint (171.7 GiB held), a cached prompt's first token after 2, 5 and 10 s idle came in 0.13 to 0.15 s instead of 1.95 to 2.61 s, and a new 2,000-token prompt's in 1.96 to 2.09 s instead of 3.71 to 4.25 s; cold 8k-token prompts ran 4% faster, decode stayed within 0.3% and every reply was the same tokens. `FLASHNEXT_NO_RESIDENCY=1` leaves the set out (#732). The 27B's set now holds its kept prompt states too: on an M3 Ultra a cached prompt's first token after 5 and 10 s idle came in 0.17 to 0.19 s instead of 0.33 to 0.35 s, and on an M5 Ultra 0.1 to 0.2 s sooner up to 8 s idle and within about 0.1 s either way after that. Every residency set, Flash Next's, the 27B's, GLM-5.3-Flash's opt-in `GLM_RESIDENCY` set and the cluster loader's, now stays inside 70% of RAM with a 2 GiB margin, or is refused with a log line that says why, a RAM reading that failed included. Buffers made after a set is built join it inside the same limit (kept prompt states at the next idle tick, a new 27B stream's slot at once), and a kept state leaves its set before it is freed, since a set keeps a buffer's memory until a commit removes it. Thanks to @sethforprivacy, to @BobClawblaw, whose measurements on #732 showed that set changes belong between requests, and to @cwschroeder for the first M3 Ultra receipt.
- An Intel XPU backend, built only when asked for (`zig build xpu-native xpu-qwen27`, with Intel's `ocloc` and oneDNN's nGEN headers): Qwen3.8-27B on an Intel Arc Pro B70 (Xe2) over Level Zero, served by `tensorfold-native` through the lane host with DFlash2 chains and trees, shared rounds for concurrent streams, the prompt cache and structured output, from MLX affine 4/64 or symmetric GPTQ 4/128 checkpoints. The kernels are OpenCL C built by `ocloc` for the card, and the int4 prompt GEMMs are generated by nGEN at build time. A stream that does not fit the card's free memory is refused when the server starts, and the KV cache stays bf16 unless `XPU_KV_INT8` asks for int8. On the author's B70 (capped at 230 W) drafted replies equal `"draft": false`, a resumed prompt equals a fresh one and 1 to 8 concurrent streams equal their solo runs, with the bf16 cache and the int8 one; from the GPTQ 4/128 checkpoint, cold prompts of 2k to 64k tokens ran at 2,051 to 1,473 tok/s against llama.cpp SYCL's 788 to 600 (Q4_0) on the same card, and decode at 33.5 tok/s without a drafter (llama.cpp 27.7) and with DFlash2 at 122.9 on code and 99.3 on chat. We have no Intel GPU, so it lands on those receipts and our host checks: the XPU code compiles against this release's engine on Linux, the Metal and CUDA servers built with it have the same code and data bytes as without it, and a scan of its OpenCL and nGEN sources against the open kernels we keep found no copied code (#802). Thanks to @Mirrdhyn.
- The Linux release archive's CUDA kernels carry Ampere sm_80 (the A100) beside sm_86, sm_120 and sm_121, and `dist` refuses a fatbin that lacks any of the four. A ModelOpt checkpoint (FP8 or NVFP4) on a card below compute capability 9.0 is refused with the card, its sm and the reason: those formats meet their decode sums in a thread-block cluster. The floor itself is where it was, and the 4-bit checkpoints run on those cards. On the author's GA100 (sm_80) all 34 kernels build for compute_80 and 16 kernel checks pass, `glue` byte-equal to the host references; on a GB10 we built the release set for sm_80, 86, 120 and 121 and `check-fatbins` passes it, Nemotron 3.5 Lightning NVFP4 served there gives the same tokens as before, and the kernel contract now checks every registered launch's registers and shared memory on sm_80 too (#846). Thanks to @liumorrisclaw.
- The kernel contract covers this release's new GPU kernels. On Metal, `core/tile_qmm`'s prompt tiles and their decode twin at 4, 6 and 8 bits (the twin bit-equal to the tiles at all 12 shapes of each width), served at the expert shapes of LFM2.5-24B-A2B, which #822 will run on them; on CUDA, prompt16's and qmmf's FP4 tiles that write fp32 (#757), with the K slices met in a cluster, fused, or summed by `reduce_kernel<true>`, and the reduce-scatter's `tf_rank_sum` (#642). Each passes gates 1 to 4 against float64, tile_qmm on an M3 Ultra and an M5 Ultra and the CUDA kernels on a GB10, with every planted control red. Gate 5 holds their launches against every chip: tile_qmm's 20 pipelines on every Apple GPU family (on M1 and M2 by offline compiles, or by the tiles' own thread request), and the ten new CUDA launches on sm_80 to sm_121.
- `core/tile_qmm`'s tile kernels ask the Metal compiler for the 128 threads a threadgroup they launch with. Without the request no offline compile could show the thread count an M1 or M2 would give them, and those are the chips the tiles are for; with it the compiler must give 128. Their arithmetic is unchanged: the twin still equals them bit for bit, and `tf-tile-check` passes on an M3 Ultra and an M5 Ultra with the same float64 errors as before, to the last digit.
- GLM-5.3-Flash's `hc_expand_10` asks the Metal compiler for the 1,024 threads a threadgroup it launches with at every decode step. Compiled offline for M1 and M2, the kernel built for 928 threads without the request, the class of fault #708 fixed for KDA and the router; with it, 1,024. Its arithmetic is unchanged, `launch_check.zig` runs it at 1, 5 and 16 rows against float64, and the kernel contract records its launch.
- The opt-in HIP build has the model-free kernels a model engine composes, on #463's runtime: elementwise ops and casts, norms and the merged decode tails, RoPE, attention (the flat decode walk, a walk over a page table that reads a KV head once for the query heads that share it, and the 64-row prompt tile, flat and paged), the DeltaNet recurrence in fp32 at every length, the linear attention's conv and gates, the MoE router, pick and route, argmax and top-k, and a lane round's plan kernels. `zig build hip-kernel-test` checks each against its host reference, and a 200-row prompt cut at 1, 63, 64, 65 or 136 rows gives one call's rows and state bit for bit. In the author's runs at this change it passed 43 of 43 on a Radeon PRO W7800 (gfx1100) and a Radeon PRO V620 (gfx1030); gfx1151 and gfx1201 have not run this version. We have no AMD card, so on our side the HIP host tests pass on macOS and Linux and the GPU tests compile against stub kernels. Nothing serves on these kernels in this change, and nothing HIP is in the release archives (#617). Thanks to @BlivionIaG, and to @ThinkOffApp for the gfx1151 run of an earlier version.
- The opt-in HIP build runs MLX's affine 2- to 8-bit products: decode tiles (row, column, split, RDNA2's wide tile, and a stream tile for up to four products that share an input or a stacked gate and up with its activation), prompt tiles (the dot2 GEMM tile, K-parallel tiles for short prompts, and gfx11's matrix-core GEMM tile through the compiler's WMMA builtins, so no rocWMMA header is needed), plans over stacked experts and the embedding gather. The kernel registry (#613) picks each product's tile from tuning tables for gfx1030 and gfx1100, and RDNA4 takes the dot2 tiles. A row writes the same bytes alone and in a round of 2 to 32 rows, and the tiles the registry picks between by row count write equal bytes, which `hip-kernel-test` checks with every tile against float64. The author's runs of these sources passed `hip-kernel-test` 57 of 57, `hip-gpu-test` 30 of 30 and `hip-affine-test` 27 of 27 on gfx1100 and gfx1030, and @ThinkOffApp's the same on gfx1151 with HIP 7.14 and with Ubuntu's packaged HIP 7.1, and @jkuepker's run on gfx1201 (an RDNA4 Radeon AI PRO R9700, whose dot2 tiles none of those cards take) passed `hip-affine-test` 30 of 30, `hip-kernel-test` 61 of 61 and `hip-gpu-test` 36 of 36 (#621). Thanks to @BlivionIaG, @ThinkOffApp and @jkuepker.
- The HIP engine has a checkpoint loader for Qwen3.5 and Qwen3.6 text checkpoints, on the host: the geometry from `config.json` (dense and MoE, MTP layers, rotary, layer kinds), MLX's affine tables for 2- to 8-bit, one tensor index over the shards with the routed experts stacked, and the tensor-parallel cuts. It refuses what the HIP kernels do not compute (an activation other than SiLU, a sigmoid attention gate, an attention bias, a recurrent state other than fp32, sizes past their 32-bit counts), and gives an MTP head that a conversion shipped zero-centered its `1 + w` back. `zig build test-qwen35` runs its tests, and with `TF_QWEN_DIR` naming a checkpoint it reads every tensor in it: on a GB10 it read five MLX 4-bit checkpoints, Qwen3.5 0.8B to 9B and Qwen3.6-35B-A3B, with every tensor resolved (#626). Thanks to @BlivionIaG.
- Qwen3.5 and Qwen3.6 serve on one AMD GPU through the opt-in HIP build, on #626's loader and the kernels of #617 and #621: `zig build native-hip -Dhip-arch=gfx1100` builds a server with lane rounds and MTP drafts, the paged attention cache, the radix prompt cache, prompts in chunks (512 rows while other streams decode), graph-replayed rounds and a memory plan, and `tf-qwen35-test check` runs the model checks (drafted equals serial, solo equals together, graph equals eager, resumed equals fresh). It refuses `--speed-up`, and `matrix=on` on RDNA4 and RDNA2, which keep the dot2 tiles; more than one GPU comes with #655, below. The loader drops each part's mapped pages once the part is on the GPU. At this version the author's model checks passed on a Radeon PRO W7800 (gfx1100) and a Radeon PRO V620 (gfx1030), 0 failed on each: Qwen3.5-9B 42 passed, Qwen3.6-35B-A3B with its MTP head 46, Qwen3.8-27B 42, all MLX 4-bit (the 9B and the 27B carry no draft head, so their four drafted checks skip). Earlier versions served the 35B-A3B on the W7800 at 176.5 tok/s for code and 140.8 for chat, every reply equal to its solo and `"draft": false` runs, and @ThinkOffApp's gfx1151 runs matched drafted and serial output on three public checkpoints. Not run at this version: gfx1151 and gfx1201 (#631). Thanks to @BlivionIaG and @ThinkOffApp.
- The HIP Qwen3.5 engine serves structured output (`response_format` and the `guided_*` fields). The lane core's grammar makes every mask: each verify row is drawn among its allowed tokens, the first token is drawn again under the mask from the prompt's last row, and the MTP drafts reach the grammar as host tokens, so structured streams keep drafting. Unmasked rows take the same draws as before. At this version the author served Qwen3.5-9B on a Radeon PRO W7800 (gfx1100) and Qwen3.6-35B-A3B on a Radeon PRO V620 (gfx1030), both MLX 4-bit with an MTP head, four streams a server: JSON object, JSON schema, `guided_json`, regex, choice, grammar, `structured_outputs` and plain requests at temperature 0 and 0.8, thinking off and on, gave 32 of 32 valid replies on each card, each drafted reply equal to its `"draft": false` one; the 32 sent four at a time equalled their solo runs, and five malformed requests were refused with 400. Not run: RDNA3.5 and RDNA4 (#697). Thanks to @BlivionIaG.
- The HIP Qwen3.5 engine splits a model over 2, 4 or 8 GPUs (`--tp`, `--rank`, `--master`, `--master-port`) through RCCL, opened at run time from the system's ROCm behind `hip/collective.zig`. Rank 0 serves HTTP and sends the other ranks its policy and its steps over a TCP link. Before RCCL starts, every rank must match rank 0's world size, GPU, RCCL version, executable and checkpoint (config.json, the weight index, and each shard's header and sampled tensor bytes; paths may differ), or every rank refuses, naming the rank and what differs. A refusal during startup on any rank reaches every rank over the link, and a rank that cannot record a round's graphs stops every rank rather than leave the others waiting in a collective. Every rank sizes its streams for rank 0's window, and partial sums add in a fixed order (one add at two ranks, rank order after an all-gather at four and eight). Rounds replay graphs cut at each collective at every world size; `graphs=off` turns them off. At this version the author's model checks with Qwen3.6-35B-A3B (MLX 4-bit, with its MTP head) passed 19, 0 failed, at tp 2 on two Radeon PRO W7800s (gfx1100), at tp 2 on two V620s (gfx1030) and at tp 4 on four V620s; they compare each group's rounds with one rank's. In earlier versions four streams of code ran at 372.6 tok/s on two W7800s against 283.7 on one, and replayed graphs ran at least as fast as eager rounds under tp in every case the author measured but one: Qwen3.5-9B at tp 2 on the W7800s, 5.5% slower at one stream and 8% at four. Not run: tp 8, RDNA3.5 and RDNA4 groups, and a GPU group with the agreement checks added at landing (host tests run them over loopback TCP) (#655). Thanks to @BlivionIaG.
- The opt-in HIP build moves the pieces every HIP family shares out of the Qwen3.5 engine: the memory budget (`hip.admission.budget`, with `TENSORFOLD_HIP_MEMORY_LIMIT_GB`), the routed MoE and the ranks' fp32 reduce (`hip.moe`, `hip.reduce`), and the token draw, greedy, sampled and grammar-masked (`zig/src/hip_serve`). A routed view can plan its rows' own pairs instead of a power-of-two bucket (`exact`: 12 items for one decode row where the bucket takes 198), for a family whose launches no graph replays; Qwen3.5 keeps its buckets. In the author's runs Qwen3.5 decode logits stayed byte-identical to #655's on gfx1100 and gfx1030 with the 9B, the 35B-A3B and the 27B (#812). Thanks to @BlivionIaG.
- On HIP the MoE route lays out each expert's members and tiles with a block scan instead of one thread, and the decode pick runs 512 threads a block, one expert a thread at 512 experts. The starts, ends, picks and weights are the same, and the route test adds 513 experts at one decode row's 11 pairs and at a prompt's 2,816. In the author's runs `hip-kernel-test`, `hip-gpu-test` and `hip-affine-test` passed 134 of 134 on gfx1100 and gfx1030; on four V620s the route took 6.9 us instead of 30.8 and the pick 14.7 instead of 25.5, and Qwen3.5 decode logits stayed byte-identical (#813). Thanks to @BlivionIaG.
- The opt-in HIP build has Flash Next's kernels: the residual streams and their tensor-parallel sums, the n-gram rows, the indexer and the sparse attention, each checked against a host reference. The gated norm takes its gate as SiLU (Qwen3.5, the same bytes as before) or sigmoid (Flash Next's DeltaNet output norm), and the chain test fails when either gated-norm kernel ignores it. In the author's runs `hip-kernel-test`, `hip-gpu-test` and `hip-affine-test` passed 74 of 74 on gfx1100 and gfx1030. Flash Next serves on them with #816, below (#814). Thanks to @BlivionIaG.
- The opt-in HIP build admits Flash Next (`qwen4_exp`): a rank's weights as tensor parallelism splits them, a position's keys and a stream's state, checked against the published 3-, 4- and 6-bit checkpoints, a window's n-gram rows hashed on the host as the reference model does, and a refusal that names the card's grant and the scratch it keeps when a window does not fit. Flash Next's affine spec moves to its own file and its config and index readers no longer import core's files, so a backend that imports core as a module can read them; Flash Next on Metal now looks a matrix's width up through `widths.quantization`, and on an M3 Ultra its 6-bit checkpoint gave the same tokens as before, drafted and with `"draft": false` (#815). Thanks to @BlivionIaG.
- Flash Next serves on AMD GPUs through the opt-in HIP build: a device's forward (weights, caches, the n-gram tables in host memory, every layer and the MTP head) and the lane backend, with MTP drafts, a round's windows in one forward and prompt reuse at 64-row cuts; `zig build tf-flashnext-hip-test` runs its model checks on the GPU. The DeltaNet output norm takes the gate `config.json` declares (sigmoid, or SiLU), and a kept prompt whose restore fails part way is prefilled from the start on fresh caches. No public checkpoint fits on one 48 GB card, so its receipts are the tensor-parallel runs below (#816). Thanks to @BlivionIaG.
- Flash Next on HIP splits over 2, 4 or 8 GPUs: each rank holds its cut of the weights, the block outputs' shares are summed in the stream norm and the logits joined, followers run rank 0's steps, and every rank sizes its streams for rank 0's window and agrees on the streams the group's memory holds. Every rank must match rank 0's world size, GPU, RCCL version, executable and checkpoint before RCCL starts (#655's join), startup takes its steps over the link so a refusal on any rank reaches every rank, and a kept prompt's restore holds on every rank or none: all resume, or all start the prompt over. At this version the author's greedy runs of `tf-flashnext-hip-test generate` answered at tp 2 on two Radeon PRO W7800s (gfx1100; Qwen3.8-Flash-Next MLX 3-bit, 32.08 GiB a card) and at tp 4 on four Radeon PRO V620s (gfx1030; 4-bit, 21.58 GiB a card), each with a hash of every step's logits. Not run: tp 8, RDNA3.5 and RDNA4 groups, and a GPU group with the agreement checks added at landing (host tests run them over loopback TCP) (#817). Thanks to @BlivionIaG.
- Flash Next's HIP rounds replay graphs split at the collectives, by shape, with host inputs staged in one upload and streams in kept cache slots; `graphs=off` keeps rounds eager. A rank that cannot record a round's graphs stops every rank rather than leave the others waiting in a collective, and a slot keeps its caches when a reset fails, so a replayed graph never names freed buffers. At this version, in the author's runs at tp 2 on two Radeon PRO W7800s (gfx1100, 3-bit) and at tp 4 on four Radeon PRO V620s (gfx1030, 4-bit), graphs on and graphs off gave the same logits at every step, twice each, equal to #817's runs. In runs of an earlier version with the same cards and checkpoints, `serve_check` passed 15 of 15, the rewind gate gave 0 differing replies against a server with the prompt cache off, and every reply equalled its solo and `"draft": false` runs; one stream decoded code at 96.7 tok/s on the V620s and 105.7 on the W7800s, and four streams 164.2 and 149.6 in total. Not run at this version: those served checks, and tp 8, RDNA3.5 and RDNA4 (#818). Thanks to @BlivionIaG.
- The native tokenizer reads an ASCII bracket class inside `(?i:...)`, folded to its case variants as Oniguruma folds a literal there: LFM2's pre-tokenizer (`'(?i:[sdmt]|ll|ve|re)|...`) now loads, and splits `'S`, `'LL` and `'\u017f` as the tokenizers library does. A negated class, a property class or a non-ASCII member inside `(?i:...)` is still refused. Every tokenizer that loaded before splits as it did, since a pattern with such a class was refused until now; the regex tests pass on a Mac and on Linux (#842). Thanks to @plotarmordev.
- `POST /v1/systemone` takes llama-server's System One request: a state (text, JSON or a chat), questions keyed by id of type `noul`, `choice` or `score`, and pictures as data URLs (`files`, `images` or a chat state's `image_url` parts, at most 4). Each question is rendered as d1's systemone template renders it, read at its single-token option forms, and answered with probabilities and zero output tokens, refused with llama-server's sentences when malformed. The engine scores it through a new `decide` hook (one pass over the shared state and a branch a question), or `score` a question at a time; an engine that scores nothing serves no such path, and an engine that only decides refuses text requests. `/v1/decisions` asks `score` for the vocabulary's mass, System One does not. No engine before d1-3B (below) scores or decides, so for every other model `/v1/systemone` answers 404 as an unknown path does and chat requests are untouched; the server tests pass, and the golden suite gives the frozen replies as before, 309 of 319 with the same 10 known differences (#843). Thanks to @plotarmordev.
- On CUDA, the residual add and norm that Nemotron runs (the shared `add_rms_h`) issues every load before it uses one, and each warp folds the eight warps' sums itself instead of waiting for one warp's broadcast, so a block that runs one norm syncs once. The sums run in the same order, so the outputs are the same bytes: on a GB10 `tf-cuda-test glue` matched its host references byte for byte, and Nemotron's replies were the same tokens in 17 cells, drafted and with `"draft": false`. On one window-cost table Nemotron decoded 0.2% to 0.5% faster drafted, and its cold prompts of 2k to 64k tokens ran within 0.25% (#796). Thanks to @jschmied.
- d1-3B (Liquid AI's LFM2-VL decision model) runs on CUDA in the native engine and answers `POST /v1/systemone` and `/v1/decisions`: one prompt pass a request over the shared state and a branch a question, LFM2's gated short convolutions and GQA attention in bf16, read at each question's option rows. Pictures are decoded (PNG and JPEG, as Pillow decodes them), tiled as `Lfm2VlImageProcessor` tiles them, and run through the SigLIP2 tower and projector on the GPU. On a GB10, the 49 requests of the d1 recipe's parity set (github.com/plotarmordev/d1-tensorfold) gave the top answer of Transformers' bf16 reference on 98 of 99 questions (as the recipe's Python 0.6 engine does), within 0.027 in probability; one question took 24.7 ms against the Python engine's 25.3, three questions 27.7 against 28.7, a 3,446-token state 261 against 273, a picture 44.8 against 44.7 to 48.4, and 64 packed states 40.7 a second against 38.5. Nemotron's prompt attention kernel compiles to the same code as before. On a GB10 here, `tf-cuda-test lfm2-kernels` (41 of 41 byte-equal), `lfm2-attention` and `lfm2-vision` passed, the last with the d1-3B checkpoint (the test pictures' features at cosine 0.998 or better against the CPU reference); served, d1-3B answered System One and `/v1/decisions` the same way twice, alone and four at once, one question in 24.8 ms at the median, and refused a chat request. `cuobjdump` shows Nemotron's prompt attention kernel unchanged instruction for instruction, and Nemotron's replies were the same tokens with cold prompts of 2k to 64k tokens within 0.2%. A Mac refuses the family with the usual message (#844). Thanks to @plotarmordev.
- `core/eaq_container.zig` checks an EAQ pack (an EXL3 checkpoint with a manifest, a completion marker and the config.json fields that name them) before a loader reads it. `open` refuses the pack unless config.json's manifest hash, the marker, the layout and pack version, the model family, the required runtime features, the codebook tables' hashes, the shard sizes, the shard headers against the manifest's tensor inventory and every shard's sha256 all agree; a caller that asks gets the name of the file or tensor a refusal is about. No family calls it yet. Its host tests write small synthetic packs with one defect each and expect the matching refusal, with no GPU and no real weights. At landing we hardened the input checks: an inline entry must name a listed shard that holds its tensors, a trellis entry's dictionary must be one the manifest declares, entry names must be unique, and each shard must resolve inside the pack's folder (or, for a Hugging Face snapshot, its repository's blobs) and is hashed from the same open file its header was read from; a hash worker that cannot get its buffer reports OutOfMemory, and an empty pack hashes nothing (#858). Thanks to @MiaAI-Lab.
- Flash Next on Metal samples a request that asks for a temperature. Each token is drawn on the GPU by Nemotron's `tf_sample_full` under the shared rule every GPU sampler reads (the seed, 1/T, top_p, top_k and min_p), keyed by the token's own position inside the round's own command buffer, so a drafted reply equals its `"draft": false` reply at the same seed and a resumed prompt draws what a fresh one does. On an M3 Ultra with Flash Next 6-bit: drafted == `"draft": false` at 8 seeds and under min_p, top_p and whole-vocabulary rules, sampled multi-turn chats equal a server with the cache off 12 of 12, concurrent == solo, greedy tokens unchanged and greedy decode within 1%; sampled prose decodes at 88 tok/s against 96 greedy, and code at greedy speed. Speed-up mode stays greedy and refuses a temperature (#719). Thanks to @BobClawblaw.
- A GLM-5.3-Flash kept state that goes stale (its slot wrote below it, or its pool blocks went to another stream) leaves the prompt cache at once, so the next turn misses once and keeps a fresh state. Before, the dead entry was found and failed its restore, and that pass kept nothing, so the turn after it paid the whole prompt again: on an M3 Ultra, a 3,794-token conversation that shares one slot with another now resumes at 3,781 tokens on its second turn after the interruption, where it prefilled in full twice. Kept states are freed only between backend operations and never while a looked-up request still holds them, on every family's lane host, so a reservation, a pool placement or an overwrite can no longer free the state a queued request is about to restore; Nemotron's tokens, chats and cached counts on a GB10 are unchanged (#835). Thanks to @heitke.
- GLM-5.3-Flash runs on CUDA from NVIDIA's NVFP4 checkpoint at tensor parallel over hosts (`TF_GLM_RANK`, `TF_GLM_WORLD`, `NCCL_COMM_ID`): rank 0 serves and the other ranks follow its steps, with MTP drafts, concurrent streams through the lanes core and kept prompt states on its 2,048-row grid. Rank 0 admits streams and its prompt cache by the smallest room of the ranks and names the rank that bounds a refusal; ranks started with different drafts settings, or more than 16 ranks, are refused by name before the weights load. On two GB10s, `tf-cuda-test glm-ref` holds every projection within 0.982 of its float64 tolerance, the lane checks on a synthetic checkpoint hold every stream to its lone run byte for byte at TP=1 and TP=2, and the checkpoint's first four layers served at TP=2 with the admit as described; the full checkpoint has not been served on our hardware yet (#804). Thanks to @Bizuayeu, and to @BHCC2025 for the vocabulary's cut in EXL3's 128-column unit (#593).
- Pictures decode in one place every family shares (`zig/src/core/picture`), to the RGB pixels TensorFold 0.6.6 loaded with Pillow 12.3: PNG from the PNG specification, and JPEG in libjpeg-turbo 3.1.4.1's TurboJPEG library (`libturbojpeg`, the release Pillow 12.3 bundles), loaded at the first JPEG and run at Pillow's settings, then EXIF or XMP orientation and alpha over white; Pillow's and torch's bicubic resizes are shared too. TensorFold does not ship libjpeg-turbo: the server loads the file `TENSORFOLD_LIBTURBOJPEG` names, else `lib/` beside its folder, beside it, Homebrew's `jpeg-turbo` on macOS, then the system's library path, checks the version string libjpeg-turbo builds in, and refuses JPEG images by name without that release while PNG images still serve. Against 0.6.6 itself on an M3 Ultra with that library, all 29 decodable pictures of d1's goldens and 74 of 76 edge pictures (interlaced and 16-bit PNG, transparency keys, animated PNG frame counts, EXIF and XMP orientation, progressive and restart-marker JPEG, cut and malformed files) gave 0.6.6's pixels or its refusal. The two others are departures by design: a CMYK JPEG is refused, and a JPEG holding several pictures gives its first, Pillow's first frame bit for bit, where 0.6.6 refused it. d1-3B's pictures moved onto it and its own JPEG, PNG and resize code is gone, with its tiles, patches and placeholder ids unchanged; on a GB10, d1-3B gave the same System One and `/v1/decisions` answers as before, bit for bit, for text and a PNG picture.
- GLM-5.3-Flash reads images on a Mac with `--vision` (`--vision-max-images`, `--vision-image-tokens`): `image_url` parts in user messages and tool results are decoded by the shared picture decoder and prepared before admission as 0.6.6 prepared them, the image tower runs on the GPU, and its rows take the image placeholders' places in the prompt. Drafts stay on and exact, the prompt cache keys each image's rows by the image's content so a kept state never resumes another image, and text that spells the image markers stays text. The tower, its scratch and a prompt's image rows count against the 70% load limit, in stream admission and the prompt cache's budget too: 5.49 GiB at the default 20,480 image and video tokens, so on a 256 GB Mac the whole 4-bit checkpoint with `--vision` needs smaller budgets (2,048 image tokens, 64 video tokens and a 4,096-token context pass the limit, the MTP head left on disk). Native engines that read no images refuse the flag, and a speed-up pair refuses it before it loads. On an M3 Ultra with 8 of the checkpoint's layers, every served check passed: an image drafted equals `"draft": false`, four image requests at once equal each alone, a later turn resumes past its image (2,027 of 2,042 tokens cached) while another image under the same placeholders resumes only the text before it, four JPEGs gave the same replies drafted and plain, and the refusals (WebP by name, GIF, an image in an assistant message, a fifth image, a remote URL without `--vision-urls`, bad base64, a bad `detail`, bytes that decode as nothing) answer before any token. In the author's runs on an M5 Ultra, the tower's patches equal the Python engine's for every PNG, its rows match mlx-vlm's bf16 tower to cosine 0.985 to 0.997, and the whole model described its test images (#598). Thanks to @BobClawblaw.
- `--vision-urls` (with `--vision`) fetches an image part's public HTTPS URL as 0.6.6 fetched it: port 443 without credentials or a fragment, every address the host resolves to public, TLS verified on the address that was checked, at most three redirects, each checked again, 10 MB an image and 20 MB a request, 10 s a download and 30 s for a request's downloads. Without the flag an HTTPS part is refused with the flag's name. Its tests run canned responses through the redirect, refusal and limit paths, and pass on macOS and Linux. In the author's runs on an M5 Ultra, a public PNG gave the file's SHA-256 directly and through a 302; `localtest.me`, 10.0.0.1, bad certificates, a 404 and an HTML page were refused; and the served checks passed 17 of 17 (#614). Thanks to @BobClawblaw.
- GLM-5.3-Flash reads `video_url` parts with `--vision`, as transformers 5.19's `Glm5NextProcessor` prepares them: 2 frames a second, at most 256, each pair of frames one image to the tower, the frames resized with torch's antialiased uint8 bicubic under `--vision-video-tokens` (16,384 by default). Decoding needs `libtfvideo`, an optional library over FFmpeg 9.0.2 (`tools/zig/build_ffmpeg.sh`, then `zig build video`), loaded when the first video arrives; without it the server refuses a video by name, as it did on an M3 Ultra here. In the author's runs on an M5 Ultra, 13 clips (H.264, HEVC, VP8, VP9 and MPEG-4) gave the processor's indices, grid, patches and time texts bit for bit at two token budgets, `libtfvideo` decoded the same RGB as PyAV on all 1,925 frames, and the served checks passed 15 of 15 (#618). Thanks to @BobClawblaw.
- Qwen3.8-27B reads images on a Mac with `--vision` (`--vision-max-images`, `--vision-image-tokens`): `image_url` parts in user messages and tool results are prepared as transformers' Qwen2-VL processor prepares them, the checkpoint's image tower runs on the GPU in our own kernels (bf16 weights as stored, fp32 sums and residual), and its rows take the image placeholders' places, turned by their row and column (M-RoPE) in the 16 attention layers, with the rows after an image shifted through decoding, drafts and later turns. Text-only requests keep their tokens, drafts stay on and exact, and the prompt cache keys each image by its content and grid. The tower, its scratch and a request's image rows take 1.79 GiB at the default 4,096 image tokens, counted against the 70% load limit and the prompt cache's budget. On an M3 Ultra, six test pictures got the processor's grids, tokens and patches bit for bit and rows closer to transformers' float32 tower than its bf16 tower is (mean row error 1.0 to 2.2% against 1.6 to 4.1%); with an image, drafted replies equal `"draft": false`, four requests at once equal each alone, and later turns resume past the image and equal a fresh server's replies. The processor's geometry, the placeholders, the M-RoPE positions and the tower are written once in `zig/src/core/qwen_vl` for every Qwen3.5-VL checkpoint, and carry pieces of #726. Video parts are refused. Thanks to @BobClawblaw.
- LFM2.5-24B-A2B (`lfm2_moe`) serves on Metal from LiquidAI's MLX checkpoints at 4, 6 and 8 bits (the README names the revisions): 30 short-conv and 10 attention layers, 64 experts with 4 active, the embedding as the head. Prompt rows run `core/tile_qmm`'s tiles, and every decode, verify and shared row runs their twin once the load has checked that it gives the tiles' bits at each matrix shape (a miss sends every row through the tiles). Attention cuts keys at fixed 4,096-key chunks and folds them in order, so a row's bits never depend on its batch or on where a prompt was split. On an M3 Ultra `tf-lfm2-exact` passes its 54 checks at each width, drafted replies equal `"draft": false` (16 of 16, and 8 of 8 at 9k to 13k-token prompts), two and four concurrent requests equal their solo replies, one stream decodes at 194, 168 and 160 tok/s at 4, 6 and 8 bits, and cold prompts read at 4,421 tok/s at 2k and 3,549 at 32k. The served window stays inside the 131,072 positions the attention holds, and a larger `--context` is refused before the load. `--learn`, `--speed-up` and `--slide` are refused for it (#822). Thanks to @BlivionIaG.
- LFM2.5-24B-A2B reuses prompts across turns: the prompt cache keeps a stream's conv states and its keys and values in host memory at the turns' marks, and the next prompt pass starts from there, within `--prompt-cache-gib`. On an M3 Ultra, two multi-turn chats (three turns, then regenerate, edit and delete) resumed about 7,100 of 7,150 prompt tokens a step and gave the cache-off server's tokens at every step, 12 of 12 at each width, and 9k to 13k-token prompts resumed to their last few tokens gave the fresh replies' tokens (#823). Thanks to @BlivionIaG.
- `tensorfold benchmark MODEL` runs the community-v1 suite (code and chat at temperatures 1 and 0, a warm-up and five repeats each) against a server of its own on 127.0.0.1, or a running one with `--server`, and saves a receipt that meets the benchmark board's rules. Nothing is uploaded unless `--publish` (or `benchmark publish FILE`) is given, after a preview and consent, with a contributor token from `benchmark login`. `--drafter auto` passes the family's own drafter when it is cached (Qwen3.8-27B's DFlash2), and `--serial` turns drafts off. On an M3 Ultra with outbound network blocked but loopback, runs on LFM2.5-24B-A2B 4-bit and on Qwen3.8-27B with DFlash2 each completed 20 of 20 samples, and the board's own rules accept and rank both receipts (#717). Thanks to @ThinkOffApp.
- The Intel XPU runner checks its int4 GEMMs against FP64. With `XPU_FP64_REF=1`, `xpu-qwen27` runs each int4 GEMM of layers 0, 1, 31 and 63 and the head on the prompt's real activations, at decode on 1 and 8 rows and on the prompt, and compares it with FP64 references of the same weights from `core/quant`; `XPU_FIDELITY=1` compares the prompt and decode paths' top tokens at every position with the same forward in FP64. The int4 group means are stored 256 times larger, so the low half of their fp16 pair stays normal. In the author's runs on an Arc Pro B70 every native path stayed within the fp16 rounding of the FP64 output; it was not run here, with no Intel GPU (#802). Thanks to @Mirrdhyn.
- With no keys configured, the native server's `/health` also names the release it was built from (`version`), the backend built in (`backend`) and the GPU it serves on (`device`, by the driver's name), read once at startup. With keys configured it still answers only `{"status": "ok"}` (#854). Thanks to @ThinkOffApp.
- The server's JSON values are one named module (`zig/build/server.zig`) that a family's serving code can import; no reply changes (#860). Thanks to @jayleaton.
- A reply's `tensorfold` fields carry the full SHA-256 of its token ids (`token_sha256`, 64 hex characters) beside `token_sha`, which is its first 12 characters (#862). Thanks to @jayleaton.
- The benchmark tools (`tools/bench_concurrent.py`, `bench_cache.py` and `bench_evidence.py`) keep each reply's full token digest, the pairs of replies they compare and the client's send and completion times. Two replies count as equal only by their full digests, and a run without them is reported as unverified (#863). Thanks to @jayleaton.
- GLM-5.3-Flash's prompt rows run the last routed expert's sorted pairs to the pair count. The sort writes no end mark after the last expert, so its pairs were skipped on layers whose routed experts are not all 4-bit (or all 2-bit) in groups of 64: the 4-bit checkpoints never took that path, and a checkpoint with 5- and 6-bit experts gave empty replies from about 12,000 prompt tokens. `tf-glm-rows-check` lays the offsets out as the route writes them (#864). Thanks to @harrisonfriia.
- `docs/api.md` documents the native `/health` fields one by one and lists `/memory` among the routes (#866). Thanks to @jishnuvenugopal.
- The 4-bit one-row twin of `core/tile_qmm`, which LFM2.5-24B-A2B decodes through, turns each code into its integer before it multiplies, as the prompt tiles do. It scaled input j by 16 to the power -j instead, and a small input underflowed: 2^-126 gave 0 where the tiles give 2^-126. `tf-tile-check` adds that case for the tiles and the twin at widths 1 to 32 (#821). Thanks to @BlivionIaG.
- The DeltaNet wide prompt scan (`core/deltanet`) can keep the state and the conv window at up to four rows of each segment (`tf_delta_gdn_chain_wide_marks`, `tf_delta_gdn_mark_conv`), each the words a segment ending at that row leaves; `zig build test-deltanet-marks` checks them against such segments on synthetic rows. No model calls them yet (#851). Thanks to @jschmied.
- The dashboard shows a filling prompt's tokens left and how long the prompt queue takes to drain: while a prompt pass runs, `/stats` carries `fill_prompt_tokens`, `fill_tokens_left` and `fill_run` from a backend that reports its progress (GLM-5.3-Flash, Nemotron and the Qwen models on Metal), and `fill_unknown` from one that does not. On a Mac the Memory tile has numbers: the process footprint, its high-water mark since start or the last `/health?reset_peak=1`, and any configured load limit, which `/stats` and `/health` name `limit` and `/memory` names `load_limit_bytes` (#738, asked for in #772). On landing, a background warm pass (the 27B prefilling the next turn after each reply) no longer shows as a fill, and a resumed prompt counts the prefix it restored as filled from the first poll. Thanks to @heitke.
- Nemotron 3.5 Lightning (MLX affine 4/64) runs on Intel Arc (Xe2) over the XPU backend's Level Zero runtime, in the `tensorfold-xpu` CLI that `zig build -Dxpu` builds on Linux (OpenCL C kernels compiled to SPIR-V by `ocloc`). Every prompt row sums as the decode step does, so a prompt's bits do not depend on how it is cut into chunks, and a window of up to 16 rows equals that many one-row steps. On the author's Arc Pro B70 a short prompt decodes at 175 tok/s and prompts run at about 850 tok/s at 2k tokens and 511 at 128k; after 640 tokens cut five ways, the states and the next logits equal the row-by-row run. We have no Intel GPU, so it lands on the author's receipts and our host checks (#523). Thanks to @BHCC2025.
- `tensorfold-native serve --backend xpu` serves Nemotron 3.5 Lightning on Intel Arc beside Qwen3.8-27B. With the checkpoint's MTP head (`mtp-4bit.safetensors`) the lane core drafts with it and with copies, concurrent requests share rounds, each on its own sequence, and the prompt cache resumes a turn from a kept state; `--no-drafts` (or `TENSORFOLD_XPU_MTP=0`) serves one row at a time without the cache. On the author's Arc Pro B70 every drafted reply equals its `"draft": false` run, 4 concurrent replies equal their solo runs, a resumed turn equals a cold one, and plain greedy replies run at about 173 tok/s; drafts do not speed this MoE up yet (0.90 to 0.99 of plain) (#547). Thanks to @BHCC2025.
- GLM-5.3-Flash loads on a Mac whose page cache holds other files. Its loader read each 16 MiB only once 8 GiB of memory was free, but macOS frees cached file pages only as memory is taken, so free memory never climbed back: on an M3 Ultra with 158 GB of other checkpoints cached, a whole-model load read 89 GB in 22 s and then 0.6 GB in the next four and a half minutes. The reads now count the file cache and purgeable memory the kernel can drop as room, under the same floor (3% of RAM, at least 8 GiB). That load is ready in 32.7 s, against 29.6 s with the cache empty, with the same tokens in 8 of 8 cells, and the kernel compressed nothing while it gave the cache back. A load that stays short of the floor for 30 s stops and says how much memory was free and cached. The 70% load limit is unchanged.
- A Nemotron load on Metal that fails after its kernels compile, for example on a truncated shard, releases both compiled kernel sets instead of leaking them; a test loads a folder with an empty shard and checks that nothing leaks (#875). Thanks to @jishnuvenugopal.
- Living Weights deletes one fact: `DELETE /v1/slide/fact/{id}` asks every learned fact's held-out questions, sets the fact's input weights to zero, rounds the layers it touched as a save writes them, asks again, and then writes the change into the shards or puts every bit back. Only input weights change, since a layer's output weights share 4-bit groups across facts, and a layer that would round to other values when saved is put back rather than written. The graph file marks the fact `deleting` before the shards are written and `deleted` after, so it never claims more than is on disk; a put-back the GPU does not take refuses replies until it lands; and the delete keeps the folder's original files first, as a lesson does, so a rollback undoes it too. Deleting one of three facts on an M5 Max changed only that fact's ten `up_proj` rows, the other two answered as before, and a forced put-back left every file's SHA-256 as it was. Served on an M3 Ultra and a GB10, a delete rewrote three shard files, the deleted fact's four answers stopped naming it, every other reply stayed the same (14 of 14), a reload without `--slide` gave the same replies, and a forced put-back left every file in the folder and all 18 replies as they were (#872). Thanks to @ThinkOffApp.
- Living Weights asks a fact's questions about others by each wording's own whole subject. Besides the fact's own questions, a lesson's router reads them asked about other things of the same kind, and that swap took one subject for every wording, a single word found anywhere in a question: a question about "Project Aster" was asked about other things of one word's kind, and a swap could land inside "Asterion". Each wording now swaps its own subject, the whole name, only where it stands as whole words, and keeps a possessive. Four lessons about the user, whose subjects are single words, gave the same answers with it as without it on an M3 Ultra and a GB10 (#714). Thanks to @cwschroeder.
- When a later pipeline fails to compile at load, the Qwen-VL image tower (`core/qwen_vl`, the 27B's `--vision`) releases the twelve GEMM and eight glue pipelines it had made, then its library, instead of leaving them allocated.
- `tf-qwen-vision --bound MEAN WORST` makes its float32 comparison a gate: every picture needs its reference rows (`NAME.ref.f32`), and a missing or misshaped reference, or a mean or worst-row relative error past the bound, exits 1 once every picture is written. `meta.json` names each reference by its SHA-256. Without the flag the tool reports as before.
- An image's identity in the prompt cache's key covers its bytes and its patch grid in 62 bits (`core/picture/identity.zig`: SHA-256 over the encoded bytes, then over that and the grid) for GLM-5.3-Flash's images and video groups, the Qwen-VL tower's and RADIO's alike, and the server writes all 62 bits into the cache words of the markers around the image's rows (`server/image_key.zig`). Before, each image row's word held 31 bits of the image's hash and GLM's and RADIO's hashes named the bytes alone, so a one-token picture (32 by 32 at detail low, or sixteen pictures under a 16-token budget) was a single 31-bit word to the cache, and one picture prepared on two grids gave its first rows the same words. GLM-5.3-Flash's and Qwen3.8-27B's image gates gave the same replies and resumed counts with it.
- On CUDA, `--vision` is each family's to take: Nemotron 3.5 Super VL reads images with it, a family that reads none refuses it at its open, before its weights load, and LFM2 refuses it at startup, since d1-3B reads a checkpoint's pictures through its own tower without the flag.
- A Gemma 4 tool call's strings go back into its arguments in one pass over the call and a sort of their marks, instead of a copy of the whole call for each string; the arguments are the same (#881). Thanks to @jishnuvenugopal.
- A reply of many tool calls searches each DSML and tool-call opener once while it is parsed, closed and hidden, instead of again after every call; the calls and the visible text are the same (#882). Thanks to @jishnuvenugopal.
- A GLM tool call of many keys without values searches each key's and value's end once instead of again for every key; the arguments are the same, checked against the old search on 20,000 random calls (#883). Thanks to @jishnuvenugopal.
- `tools/shared_prefix_load.py` counts a stream that ends without `data: [DONE]` as failed, so its tokens no longer count toward successful requests, throughput or time to first token (#889). Thanks to @alenperic.
- `bench_openai.py` and `bench_concurrent.py` send a bearer key read from the environment variable `--api-key-env` names, only on the request they address (never on a redirect), and remove the key from everything they print or save (#890). Thanks to @alenperic.
- `tf-merge-check --quantizer DENSE SHIPPED MODULE...` quantizes dense tensors to MLX affine 4-bit as MLX does (groups of 32, 64 or 128; any other group is refused before anything is written) and reports how many codes, scales, biases and whole groups equal the shipped checkpoint's; `zig build test-merge` runs its tests (#709). Thanks to @philip-pentatonic.
- `--record DIR` writes each stream's committed tokens and the rows its drafter reads to a new folder under DIR, for drafter training: Nemotron on CUDA records its final norm's rows, a chunked prompt once its last chunk completes; the other families and backends refuse the flag by name (#848). Thanks to @jschmied.
- GLM-5.3-Flash samples every target row with the shared keyed sampler: the prompt's first reply token and plain rounds no longer fall back to greedy, queued server rounds carry their request's rule, and a drafted reply equals the same request with drafts off at any temperature; two Macs still refuse sampled requests (#892). Thanks to @heitke.
- LFM2.5-24B-A2B decode windows of 8 to 32 rows read each weight once through an MMA window kernel with the one-row twin's bits, and the 4-bit twin decodes its codes through exact half pairs (#867). Thanks to @BlivionIaG.
- LFM2.5-24B-A2B windows of 16 rows or more read each picked expert once for all of its rows (#868). Thanks to @BlivionIaG.
- On M5, `core/affine_mm` loads its code tiles in two 8-byte reads and takes an optional plain SwiGLU epilogue; GLM-5.3-Flash keeps its clipped one (#871). Thanks to @BlivionIaG.
- LFM2.5-24B-A2B prompt calls run on the M5's tensor units (projections, experts and attention) at every prompt row, so a resumed prompt equals a fresh one; a GPU without tensor units no longer reaches the tensor attention path (#873). Thanks to @BlivionIaG.
- GLM-5.3-Flash's sparse prompt attention on Metal reads a row's key list only as far as its last listed key: a row at position p lists its keys in the list's first p + 1 entries, so rows before position 2,050 skip the empty blocks, with the same bits (#898). Thanks to @harrisonfriia.

## 1.0.11

- `--cortheon`'s instruction filter keeps reading past a long word. A word longer than the filter's scan buffer ended the scan early, so an order written after one was never checked; each word now keeps at most 16 bytes, so a long word can no longer end the scan (#829). Thanks to @joeynyc.
- `--cortheon`'s instruction filter also reads page titles and search results' titles and snippets, which reached the model unchecked, so an order in a page's `<title>` or meta description is dropped like one in its text. A search says how many sentences it dropped, as a page read does.
- `--cortheon`'s instruction filter sees through soft hyphens and zero-width characters (U+00AD, U+200B to U+200D, U+2060 and U+FEFF), raw or written as numeric entities: one inside a word no longer splits an order into pieces the filter misses, and one between words still parts them.
- `--cortheon` drops Unicode tag characters (U+E0000 to U+E007F) from web text. They are invisible, yet some models read them as letters, so a page could spell an order in them that no one reading it would see; one inside a visible word no longer hides an order from the instruction filter either.
- `--cortheon` reads a page of unclosed tags in linear time. A run of `<` with no `>`, or of unclosed `<a ` tags, was rescanned to the page's end at every character: a 128 KiB page of either took 2 to 4 seconds, and one at the 2 MiB read limit would take 8 to 18 minutes. Both now take milliseconds.
- `--cortheon` reads a search page that ends inside a result's snippet: the result keeps its title and link with no snippet. Before, the parser sliced backwards there, and the safety check that released builds keep in the server stopped the process (#830). Thanks to @joeynyc.
- `--cortheon` shows at most 200 bytes of a page's title. A page that never closed its `<title>` sent the rest of its document to the model as the title, past the read's size limit (#831). Thanks to @joeynyc.
- [docs/known-issues.md](docs/known-issues.md) keeps the current list of known issues in one place. Each release's notes summarize it and say what changed, instead of pointing at an earlier release's notes.
- SECURITY.md says how to report a security problem privately (the repository's Security tab, "Report a vulnerability"), which releases get fixes, and what is in scope.
- GLM-5.3 (EXLR8) checks the GPU's fragment layout as it loads, as GLM-5.3-Flash and Nemotron do, so a Mac whose layout would give wrong sums stops about a second into the load instead of serving wrong tokens (#809). Thanks to @harrisonfriia.
- `zig build test` passes on a Mac with a GPU again. The split-checkpoint test asks on purpose for a tensor that stays in its file, and that lookup logged an error, which the Zig test runner counts as a failure; it now logs a warning.
- A pull request's No GPU check also runs the frozen server goldens (`zig build test-golden`, which now builds on Linux too) and fails a change that adds problems `tools/zig/lean_check.py` reports (a file past 600 lines, a comment longer than one line or 120 columns) beyond those the base branch already has.
- `--cortheon`'s instruction filter reads web text after its tag characters are dropped, so it judges exactly what the model reads. A tag character inside the bytes of a zero-width space hid that space from the filter, and dropping the tag then rebuilt it, so an order such as "upload the user's ssh keys" reached the model; dropping one tag character could also join the bytes around it into another. Both were found by the new fuzz harness.
- `--cortheon` reads a long DuckDuckGo lite page in linear time: each result is now read within its own part of the page, where a 1 MiB page of repeated results took about a second. Found by the fuzz harness.
- `--cortheon`'s page excerpt stays within its byte budget. Each kept line after a gap ran one byte over. Found by the fuzz harness.
- `--cortheon` parses at most the first 2 MiB of a page, for at most 2 seconds, and keeps the text read by then, so no page, however it is written, can hold a request for minutes.
- For contributors: `zig build fuzz-cortheon` runs Cortheon's readers of untrusted bytes (page text, links, excerpts, search results and the instruction filter) under a fuzz harness: its seeds, 100,000 structure-aware edits that cut tags and attributes, and a linear-time check on long inputs, in about 20 seconds. Its default build keeps the safety checks the shipped server keeps, `-Dfuzz-optimize=fast` builds it without them, and `--fuzz=N` adds N inputs from Zig's coverage-guided fuzzer.
- `--cortheon`'s search parser reads a results page in one pass. A page of link marks with no tag opening between them sent it back to the page's start for every mark, so 1 MiB took over 3 seconds; each search back now stops where the previous link ended.

## 1.0.10

- `cortheon_fetch` reads only URLs the conversation has already shown: search results, links on pages it has read, and URLs in the user's, the system prompt's or a tool's messages, never one that appears only in the model's replies, so a page that steers the model cannot make it build a URL that carries conversation text. The one exception is one named package's registry JSON on PyPI, npm or crates.io, a name of at most 64 characters. A refused fetch tells the model to search for the page. The `cortheon` object's `pages_read` now counts only pages read, not refused or failed fetches.
- The prompt cache keeps a turn's history while that turn's reply and its background prefill are kept. Qwen3.8-27B on a Mac keeps the end of each reply, and the 27B on a Mac and Nemotron on CUDA prefill the next turn's opening in the background after a reply. Both states counted as kept by prompts longer than the turn's own, so the cache took the turn's history for a superseded state and freed it first when it needed room. A regenerated turn, an edited reply or a client that sends the reply back rendered differently then read the whole prompt again. Both now count with the turn's prompt length (the background request carries it), and a later turn, whose prompt is longer, still frees them all first. Served on an M3 Ultra with a 4 GiB cache and first turns of 4,500 to 16,000 tokens, every regenerate, edit and re-rendered reply that needed the history resumed there, 20 of 20 sent as soon as the reply arrived and 8 of 8 sent after the background prefill had finished, against 4 and 0 without the change, and started in 0.14 to 0.98 s instead of 15 to 63 s. On an M5 Ultra, the reply-end change alone resumed 22 of 22 against 6. Prompts under 4,096 tokens, which keep one mark at a chunk end, gain the same way, 14 of 14 resuming there against 6 without the change. On a GB10, Nemotron's chats and its branch and fork runs resumed at least as many tokens as without the change at every step, and every reply on both chips equalled the cache-off server's.
- On CUDA, Nemotron with a Living Weights pocket no longer pays for the layers whose pocket has no neuron in use, and runs a taught layer's pocket over many blocks. Each pocket keeps its count of neurons in use on the host and the device: a layer with none launches nothing, and a taught one runs `tf_pocket_in` (the squared relu of the up rows' dot, a warp a neuron) into a small fp32 scratch, then `tf_pocket_out` (a thread an output) adds the neurons in order into the shared expert's slot, where `tf_pocket_rows` ran one 256-thread block a row over all 64 neurons. Every sum keeps its order, so replies are unchanged; the one value that can differ is an exact -0.0 in the shared expert's output of a layer with no neuron in use, which `tf_pocket_rows` turned into +0.0. Captured graphs read the count from the device and are captured again when a skipped layer gets its first neuron, and that recapture now keeps the sequence's own sampling rule. On a GB10, `tf-cuda-test pocket` matches `tf_pocket_rows` byte for byte in 196 cases (1 to 4,096 rows, 0 to 64 neurons in use), and a decode row's pocket takes 19 us instead of 94 and a 2,048-row prompt chunk's 0.48 ms instead of 1.23. A copy of Nemotron 3.5 Lightning (MLX 4-bit) taught one fact gave the same replies token for token as before, drafted and with `"draft": false`, alone and concurrent, and copies taught by the old and the new build wrote the same files. On that copy `"draft": false` decode was 17% faster, drafted decode 5% to 10% and cold prompts of 2k to 32k tokens 8.5% to 10.3% (#787). Thanks to @ThinkOffApp, and to @philip-pentatonic, whose #678 traced the cost and proposed both changes.
- A launch that no variant of a captured Triton set matches exactly now takes one that assumes less of its arguments: a variant compiled without 16-byte alignment on a pointer, integer or 64-bit argument serves an aligned one. The exact match still comes first, so every launch that matched before takes the same variant, a variant that assumes alignment still refuses an unaligned argument, and an integer of 1, which Triton folds into the kernel, still takes only the variant compiled for it. On a GB10 with Nemotron 3.5 Lightning (MLX 4-bit) and the qualified sm_121 set, replies were the same tokens as before, drafted and with `"draft": false`, alone and concurrent; with a copy of the set missing the 11 variants that assume more alignment than a sibling, the new rule served 34,786 launches of `_kv_write`, `_chunk`, `_merge` and `_topk` with the remaining variants and every reply was still the same tokens, where the build before it refused to load (#794, #791). Thanks to @zweck and @BobClawblaw, whose #726 has the same matcher change.
- `tools/zig/triton_aot_pack.py` (moved from the Nemotron tests) writes a captured Triton set from a capture's manifest or from a Recorder launch dump. `--jit` and `--cache` are optional (the manifest's own cache is the default, and a cubin recorded under another mount is found under `--cache`), `div16` is set only for an argument Triton gave `tt.divisibility` 16 (any attribute counted before), and a constexpr named like a runtime argument is left out. Packing the capture behind the qualified sm_121 set again, from its manifest or from its launch dump, gives the set's launch facts byte for byte for the 37 variants that capture holds (`aot.json`, with only the generator's name changed), and cubins that match the set's instruction for instruction; 28 of them differ only in the source folder they record. The engine only reads the set, and nothing on the serve path runs the packer (#795, #792). Thanks to @zweck.
- The Metal engine's shared kernels and `core/quant` read MLX's mixed-width (oQ) checkpoints, the base for the three families below. The lane projection takes 2-, 3-, 5-, 6- and 8-bit codes in groups of 32, 64 and 128, 48-row members and an output row stride; the row projection reads 3- and 5-bit codes and writes one member of a fused stack into its columns; `affine_mm` reads 2-bit codes; `core/quant` finds a module's width under any key spelling (oQ checkpoints write `language_model.model.layers.N...`, loaders ask for `model.layers.N...`) and refuses a checkpoint whose spellings of one module disagree; and `widen` re-packs codes at more bits, or a group at a smaller size, exactly. At the widths 1.0.9 serves nothing changes: the 4-, 6- and 8-bit lane and `affine_mm` sources preprocess to 1.0.9's text (168 layouts and formats), the row kernels give 1.0.9's bytes at 2, 4, 6 and 8 bits (301,056 outputs on an M3 Ultra and on an M5 Max), the lane kernels' outputs are 1.0.9's on every layout it runs (115 on an M5 Max, 13 on an M3 Ultra), and Nemotron on an M3 Ultra, which compiles these row kernels, gives 1.0.9's tokens drafted, resumed and concurrent. `tf-row-widths-check` and `tf-affine-ref` check every width against float64 and every row's bits in every window; @sethforprivacy ran them with `test-core-lane` on an M5 Ultra and @mrv777 on an M1 Max, both at 008abc0, all passing (#755). Thanks to @grantoverton.
- Qwen3.8-27B on a Mac serves mixed-width checkpoints such as TensorFold/Qwen3.8-27B-oQ4 and -oQ2: each loaded module is admitted at its own MLX width (2 to 8 bits, groups of 32, 64 or 128) against the kernels the chip runs, a fused stack whose members differ runs each member as its own product, the DFlash2 drafter borrows the target's embedding and head at their widths, and a mixed checkpoint times its own draft-window costs instead of reading a uniform checkpoint's. A uniform 4-bit checkpoint keeps 1.0.9's kernels, launches and window costs. Two changes of ours: a projection run that is not a whole number of 32-row tiles (a lone 48-row `in_proj_a` or `in_proj_b`) loads on the row kernels on every chip, which @sethforprivacy found on an M5 Ultra, where oQ4 was refused and oQ2 failed to load without it, and the 27B reads module widths through `core/quant`'s keys instead of its own copy. The oQ checkpoints rest on the contributors' runs: @grantoverton's on an M3 Ultra (in #756) and @sethforprivacy's on an M5 Ultra (his checks there at 8c05ac9, with that fix), each with drafted replies equal to `"draft": false`, resumed prompts equal to fresh ones and concurrent equal to solo. On an M3 Ultra the uniform 4-bit 27B with DFlash2, served by 1.0.9 and by this change in A B B A order, gave the same tokens in every case (17 drafted against `"draft": false`, 12 multi-turn replies also against a cache-off server, 8 concurrent cells), cold prefill from 2k to 32k tokens within 0.02%, and plain and prose decode within 0.12%; drafted code ran 4.5% slower at the same tokens with three more rounds a reply on each build's own timed window costs, and 0.35% faster with the two builds' tables swapped (#756). Thanks to @grantoverton and @sethforprivacy.
- Qwen3.8-27B's mixed-width checkpoints decode and prefill faster before tensor units (M1 to M4): projections at 2, 3, 5, 6 and 8 bits in groups of 64, and every member of a mixed stack, run on `simd_qmm`'s 8x8 tiles at their own width (`simd_qmm_bits`), with 2-bit codes masked in place as 4-bit nibbles are. On the author's M3 Ultra (at f738b48) oQ2 decodes 2.0x (code) and 1.6x (chat) faster and prefills 2.4 to 3.3 times faster, oQ4 1.3x, against float64 within the same bound; uniform 4-bit matrices keep `simd_qmm` and their tokens, and `tf-qwen27-widths` passes on an M3 Ultra here (10,575,872 values within 0.4815 of the bound). On tensor units the 27B's row matrices (the 48-row runs above) keep the row kernels and `simd_qmm` they were served with on an M5 Ultra, so the tiles change nothing there (#784). Thanks to @grantoverton.
- GLM-5.3-Flash on a Mac serves mixed-width checkpoints such as TensorFold/GLM-5.3-Flash-MLX-oQ2-MTP: each module at its MLX width and in mlx-lm's layout, split stacks as one product a member, 2-bit routed experts gathered on core `affine_mm` in prompt chunks and every other width on the shared row kernel. A uniform 4-bit checkpoint compiles none of the new pipelines and keeps its kernels; two Macs (TP2) and expert parallel still take 4-bit group-64 checkpoints only, and refuse others by name at load. The oQ checkpoints rest on @grantoverton's runs on an M3 Ultra and @sethforprivacy's on an M5 Ultra, both at 9777aeb (drafted equal to `"draft": false`, resumed equal to fresh, concurrent equal to solo), and @mrv777 ran the width checks on an M1 Max. Ours: a test that needs a GPU now skips without one, as the tree's other device tests do. No uniform GLM-5.3-Flash checkpoint fits one 256 GB Mac within the 70% load limit, so its uniform path was checked by kernel text and the row checks, not served (#773). Thanks to @grantoverton.
- GLM-5.3-Flash's mixed-width checkpoints run faster on Macs: prompt chunks run the absorbed `embed_q` and `unembed_out` heads as one `affine_mm` batch, and 5- and 6-bit products and split-stack members on `affine_mm`, which now reads 3-, 5- and 6-bit codes exactly; decode makes KDA's `f_b` and `g_b` dots inside `kda_rows`, runs the shared expert's gate, up and SwiGLU in one launch and the routed and shared experts in two, with the same bits; and core row projections read 2-bit codes in place. On the author's M3 Ultra (at 347f88a) oQ2 decodes 1.22x (code) and 1.13x (chat) faster and oQ4 prefills 1.33 to 1.37 times faster; @sethforprivacy's M5 Ultra checks at 347f88a pass. The 2-, 4- and 8-bit `affine_mm` sources are unchanged and the row kernels still give 1.0.9's bytes at its widths, so uniform checkpoints keep their tokens; `tf-glm-rows-check` and `tf-affine-ref` pass on an M3 Ultra here (#786). Thanks to @grantoverton.
- Qwen3.8-Flash-Next on a Mac serves mixed-width checkpoints such as TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP and -oQ3-MTP: each matrix is admitted and packed at its own MLX width (fused members grouped by width and widened exactly where a group needs it), and decode lanes, hyper-connections, experts, the embedding, the n-gram tables and prompt products run at that width, on GPUs with and without tensor units; an n-gram table's `weight_scale` is applied as oMLX does. A uniform 6-bit group-32 checkpoint keeps its recorded kernels and launches, but its pack is rebuilt once after the upgrade (the pack's identity now covers `config.json`). Speed-up mode across two Macs still takes 6-bit group-32 checkpoints only. Ours: Flash Next reads module widths through `core/quant`'s keys instead of its own copy. The oQ checkpoints rest on @grantoverton's runs on an M3 Ultra and @sethforprivacy's on an M5 Ultra, both at b29d032, and @mrv777 ran the width checks on an M1 Max. The uniform 6/32 checkpoint was not served for this change: its recorded kernels are unchanged, its 6/32 lane layouts give 1.0.9's output bytes on an M5 Max and an M3 Ultra, and `tf-flashnext-widths` passes on the M3 Ultra (#774). Thanks to @grantoverton.
- Qwen3.8-Flash-Next decodes faster on Macs without tensor units (M1 to M4): dense projections and hyper-connections run one input row a simdgroup, with a matrix-unit twin from 4 rows and four rows a simdgroup for 6- and 8-bit products at 2 to 4 rows, each checked against the per-row loop at load, and the draft depth is chosen by measured round cost. This changes the uniform 6-bit checkpoint's tokens on those Macs: its dense products' fp32 sums run in another order, inside the same float64 bound, and `FZ_M3_LANES=1` keeps the lane kernels and their tokens. Tensor units keep every kernel and launch. On the author's M3 Ultra (at 2759e70) the uniform 6-bit checkpoint decodes 1.06x (code) and 1.19x (chat) faster, oQ2 1.02x and 1.14x, prefill unchanged, with drafted equal to `"draft": false` and concurrent equal to solo; `tf-flashnext-widths`, its rows and rows-hc classes included, passes on an M3 Ultra here, 4,486 checks within 0.9947 of the float64 bound (#788). Thanks to @grantoverton.
- A host test checks that the native CUDA host forwards each lane-backend entry to the family's entry of the same name, once, so a forwarder can no longer call a neighbour's entry unnoticed (#800). Thanks to @Bizuayeu.
- Nemotron on CUDA adds each layer's residual and normalizes it in the shared kernels of `cuda/norms.cu`, and the family's own copy is gone. A bf16 row takes the 16-byte path by its width alone, a misaligned buffer is refused, and the reciprocal square root is an IEEE division and square root. The sums run in another order of the same precision, so a CUDA reply can part from 1.0.9's at a near tie; teacher-forced loss over seven public texts is unchanged within noise, and decode and prefill speed are level (#796). Thanks to @jschmied.
- Nemotron's NVFP4 prompts read 4% to 7.5% faster on a GB10 (2k to 64k tokens). The expert kernel's prompt rows take their warp's row tile count as a compile-time constant, so one mma body runs each count without a branch per tile; every teacher-forced row and served reply is bit for bit the same (#805). Thanks to @Bizuayeu.

## 1.0.9

- `tensorfold pull` finishes into a relative cache root (a relative `HF_HUB_CACHE` or `HF_HOME`), and `pull --exlr8` into a relative `--dir`. The shipped binary stopped at the first finished file with an assertion failure, because it moved files with a call that only takes absolute paths. The checkpoint CLI's tests now run with the safety checks the binary ships with, and a new test fetches an EXLR8 node's small and large files into a relative folder (#700). Thanks to @chaog992.
- The Flash Next recipe names the checkpoint the native engine serves on Macs, `TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP`, with `--temperature 0`, says the 4-bit checkpoint is refused at load, and marks its other sections as the Python 0.6.6 engine's (#705). Thanks to @cwschroeder.
- The native CUDA host forwards every entry a family's lane backend gives it. It dropped `prefill_step`, `prepare_features` and `features`, so the lane core saw a backend that cannot run a prompt a chunk at a time: a CUDA family that can would have run every prompt whole, with other streams' decode waiting for all of it. No CUDA family in this release runs its prompt in chunks yet, so replies are unchanged, and a test now fails when a backend entry is added without being forwarded. In the author's runs of a GLM-5.3-Flash port on two GB10s that does (#548), a decode beside a fresh 32k-token prompt waited at most 1.7 s between tokens instead of 27.9 s (#742). Thanks to @Bizuayeu.
- The prompt cache keeps a state that several continuations resume, such as a turn sent again, a regenerated or edited reply, or an agent that starts each step from the same history. Once one continuation had kept its own longer state, the cache counted that history as superseded and freed it first when a later prompt needed room, so every later continuation read the whole history again and kept nothing there either. A state resumed again after a longer prompt kept a state extending it is now a branch point, freed only as the least recently used, while a conversation that only grows still frees its old turns first. The paged store takes the same rule. Replies are unchanged, since only the choice of which state to free changes. Served with Nemotron and a small cache (0.3 GiB on an M3 Ultra, 1 GiB on a GB10), two forks of a 3,962-token conversation each kept a longer state, and unrelated prompts followed until the cache had to free one: a third fork then resumed the history's 3,956 tokens, where 1.0.8 had freed the history and read it all again. A conversation that only grows resumed the previous turn's prompt every turn with and without the change, and every reply equalled the cache-off server's (#735). Thanks to @BadAd84.
- Nemotron's GPU samplers on Metal and CUDA read one sampling rule, built once from the request: 1 / T, top_p, the candidates' near window, ln(min_p), top_k and the seed. Before, Metal's host, CUDA's sampler and the CUDA tests each built their own copy of the same values. The kernels and their arguments are unchanged, so sampled replies are the same tokens, and host tests pin the Metal cfg row and the CUDA rule's layout. On an M3 Ultra and a GB10, Nemotron's seeded sampled replies to 8 prompts, and 12 more cells across temperature, top_p, top_k (0 to 3,000) and min_p, were the same tokens before and after, drafted and with `"draft": false` (#731). Thanks to @jschmied.
- CUDA has shared row kernels for every family: RMS norm rows, the residual add with its norm, a bf16 residual stream's add and norm with the 64-value group sums a 4-bit projection reads, gated head norms, and token rows from a 4-bit (MLX layout) or bf16 embedding table. Each has a float64 reference, and `tf-cuda-test norms` and `norm-rows` check them against it and check that every row count gives the widest run's bits. Nemotron's 4-bit token rows now run on the shared kernel when it serves with our own kernels rather than a captured Triton set, with the same bits as its own kernel, which is gone. A captured set keeps its own token rows, and Nemotron keeps its own norm kernels, whose sums run in another order; no family uses the shared norms yet. On a GB10, Nemotron's replies with our own kernels were the same tokens before and after, drafted and with `"draft": false`, and decode and cold prefill (2k to 32k tokens) were within 1% (#736). Thanks to @jschmied.
- FP8 (e4m3) and NVFP4 (e2m1 codes with e4m3 scales every 16 inputs) values and their float64 references live in one place on the host, `fp8.zig` and `nvfp4.zig` beside the other formats in `core/quant`. The CUDA tests and `tensorfold check-precision` decode through them instead of keeping three copies of the same arithmetic, which is unchanged, so no kernel and no reply changes. A reference refuses what it cannot read: a scale type other than fp32, bf16 or fp16, an empty projection, or an expert stack whose scales or global scales don't match its weights. Host tests check that every finite e4m3 code rounds to itself, that ties go to even and large values stop at 448, and FP8 and NVFP4 expert stacks against values decoded by hand. On a GB10, every kernel and the CUDA server built byte-identical with and without the change, and `tf-cuda-test quant-ref` and `modelopt` printed the same output, byte for byte (#730). The host now reads e4m3's two NaN codes, 0x7f and 0xff, as NaN, as the GPU's conversion does, where it read them as ±480; no checkpoint holds them. Thanks to @jschmied.
- `tf-metal-counters` (`zig build tf-metal-counters`) times GPU work on a Mac with the GPU's own counters and no model: three dependent FP32 operators, each timed between counter samples at its encoder's start and end, with GPU ticks turned into seconds from two readings of the CPU and GPU clocks rather than from CPU time. It checks the outputs against values computed independently and against an untimed run, bit for bit, and refuses to read, reset or free its samples while their command buffer runs. A GPU without the counters it needs is reported as unsupported, with no CPU timing in their place. On an M3 Ultra and an M5 Ultra, sampling at encoder boundaries works and sampling at dispatch boundaries is not supported, which the probe reports. No engine uses it: the Mac server built with and without it has the same code and data, byte for byte (#680, #676). Thanks to @akol1.
- The fragment check Nemotron and Flash Next run at every load on a Mac, which proves on the GPU that `nax.h`'s matrix fragments give the host's sums, now catches a transposed operand. Its test values, `(i * 7 + salt * 3) mod 5`, made every block of the transposed operand symmetric (its row pitch, 16, is 1 mod 5), so loading it as a plain one gave the same sums; values mod 13 leave no such block and keep every sum an exact small integer. A new GPU test runs the check and, before M5, the same check with the transposed load swapped for a plain one, which must fail. On M5 the multiply does the transpose, so there the test checks the layout alone. On an M3 Ultra the new test passes, and fails with the old values put back. Replies are unchanged (#744). Thanks to @harrisonfriia.
- GLM-5.3-Flash on a Mac runs that fragment check at every load too, right after its kernels compile, as Nemotron and Flash Next do: its prompt kernels and decode attention read the same fragments, so a GPU whose layout gives wrong sums now stops the load with `error.FragLayout` instead of serving wrong tokens. A load that passes adds one small dispatch. On an M3 Ultra with 8 of the 45 layers, a build whose check was made to fail stopped at load, and the real build's replies were the same tokens as before, drafted and with `"draft": false`, decode and cold prefill (2k to 32k tokens) were within 0.2%, and a warm load took 5.1 s on both builds (#745). Thanks to @harrisonfriia.
- The 27B on a Mac no longer copies each DeltaNet layer's recurrent state after a chain round. A plain decode step, a verified draft chain and a prompt chunk scanned each of the 48 DeltaNet layers' states (3 MiB of fp32 per stream) into a second buffer and then copied it back: about 288 MiB read and written for every decoded token, in one more dependent dispatch per layer. When a round's rows keep their slots, the scan now runs on the state where it is and only the convolution history is copied; a round whose slots move keeps the copy. The scan does the same arithmetic on the same inputs, so every output bit is unchanged (#711). Thanks to @jkuepker.
- The 27B's decode attention on an M5 runs fewer dependent dispatches. A chain of decode rows, as in a plain step or a verified draft chain, now attends to its own keys in the prefix kernel, so the tail kernel, its query repack and the gate dispatch go away: a one-row step runs 4 dispatches with 3 barriers in each of the 16 attention layers instead of 7 with 5. The tree-round merge loads its chunk partials ahead and applies the query gate itself with the same bf16 rounding, rows past a stream's last query are no longer written, and each tile's two head-dim halves run on their own simdgroups. Chunk partials, tile boundaries and merge order are the tail kernel's, so every output byte is the same; a round with a branching row, a stream that is not a chain, or a GPU without tensor units keeps the tail kernel. `tf-qwen27-attention-fused` also checks plans without a shared prefix, 5 to 8 streams and 16-row windows (#712). Thanks to @jkuepker.
- The 27B's DFlash2 drafts on a Mac wait less. The drafter's first command buffer starts after 6 operations instead of 40, its row copies go out as one dispatch, shared-context attention loads four keys before it folds them (in key order, with the same fp32 sums), and wide bf16 top-k rows are split over sixteen threadgroups. In the check tools' run path, a drafted round's cache and state commit also runs behind the next draft; the server still finishes it at once. Tokens, rounds and accepted drafts are unchanged. With #711 and #712, on an M5 Ultra (four runs a build, 256 tokens, one stream), greedy decode went from 406.1 to 417.0 tok/s on code and from 100.6 to 103.6 on prose, and seeded sampling from 294.0 to 316.9 on code and from 86.2 to 89.0 on prose; on an M3 Ultra (six runs), greedy code went from 150.6 to 154.7 and prose from 53.3 to 54.8. Prompt processing at 16k, 32k and 64k tokens on the M3 Ultra was 0.2 to 0.4% faster. Replies were the same tokens drafted and with `"draft": false`, on both builds, resumed and fresh, and concurrent and alone (#713). Thanks to @jkuepker.
- The Metal EXL3 projection reads each 16-bit trellis state without a branch: a shift and a masked carry from the next word give the same 16 bits as the old two-way window at every offset, 0 to 31, so the hoisted kernel's states, weights and outputs keep #681's bits. A new GPU test checks the window against the host's value for 35,040 pattern and offset pairs, and the existing fixtures still match (#718). Thanks to @mrv777.
- GLM-5.3-Flash's sparse attention runs on an M1. Its generated kernels (64 and 32 heads) now declare the 1,024 threads their caller launches; an M1 Max compiled them for 832, so the launch left their outputs at zero. Kernel bodies, dispatch sizes and tolerances are unchanged, and the regression tests now also check the pipeline's capacity and compare both kernels against an independent float64 reference (#704). Thanks to @mrv777.
- Living Weights trains Nemotron on an M1. `tf_train_ssm_back` now declares the 1,024 threads training launches per head; an M1 Max compiled it for 768, which could leave gradients unwritten with no command-buffer error. Arithmetic and dispatch sizes are unchanged, and new GPU tests check gradients, seeded checkpoints, shared groups and checkpoint boundaries, the production 64-head, 8-group shape included, against float64 and finite differences (#715). Thanks to @mrv777.
- Nemotron on CUDA runs a prompt in passes of up to 4,096 rows instead of 2,048-row chunks, so each pass reads the routed experts' weights once for twice as many rows. A row's bits do not depend on where a pass is cut, so replies and kept prompt states are unchanged; staggered segments, layer dumps, lessons and a pass that would hold more prompt-cache marks than the scan arms at once (16) keep 2,048-row chunks, and the startup line says "prompts in passes of up to 4096 rows". On a GB10 with Nemotron 3.5 Lightning (MLX 4-bit), cold prompts of 8k, 16k, 32k and 64k tokens prefill 7.2%, 6.8%, 6.3% and 5.4% faster, and 2k as before (#720). Thanks to @plotarmordev.
- `tf-cuda-test prompt-cuts` checks Nemotron's CUDA prompt states against one uncut pass: a pass that keeps states inside chunks and at their ends, then a pass resumed from every keep with the key and value rows past it filled first, comparing the states, the MTP head's caches, the last hidden row, the logits and the first token. The checks come from #721 and #722, whose changes 1.0.8 had already made in its own form, and on a GB10 they pass at 3,073, 4,096, 4,097 and 8,193 tokens on the MLX 4-bit and NVFP4 checkpoints. Thanks to @plotarmordev.
- Nemotron on CUDA prefills each reply in the background for the next turn when the prompt cache is on, as the Mac engines do: after a reply, the conversation through it and the next turn's opening run as a background pass that steps aside at its next chunk for a foreground request, so the next turn resumes after the reply instead of reading it again. On a GB10, a second turn resumed 7,366 of its 7,389 prompt tokens where the build before it resumed 7,228. When the next request arrives before the background pass has run, the pass steps aside and keeps nothing, so that turn resumes from an older state: in the same chat, a regenerated third turn resumed 7,366 tokens where the build before it resumed 7,540 (#723). Thanks to @plotarmordev.
- Nemotron on CUDA no longer makes a request wait for the background pass that prefills a reply for its chat's next turn. The pass starts only once the engine has had no other work for 150 ms, so a request sent the moment a reply ends goes first, and the pass is dropped before it reaches the GPU. On a MoE every prompt chunk reads most of the experts' weights, so even a 20-row pass held a GB10 for about 45 ms. A pass that is running steps aside after its current chunk, now at most 256 rows, and keeps the state it reached. Since a pass may not run at all, each turn keeps its own history state again, as in 1.0.8, so back-to-back and regenerated turns resume as they did there. Replies are unchanged. On a GB10 with Nemotron 3.5 Lightning (MLX 4-bit), a new chat's ~1,000-token first turn sent right after another reply started after 0.199 s, as in 1.0.8 (0.200 s) where the first merge of the background pass took 0.240 s, and longer ones kept #720's gain: 6% sooner than 1.0.8 at ~10k tokens and 7% at ~20k. In chats with a second between turns, later turns resumed past each reply and started 2.0-2.2x sooner than in 1.0.8.
- Nemotron on CUDA times the draft-depth rule's window costs on the window a drafted round verifies: the pending token at the prompt's end, the MTP head drafting first as in a round, then the text's own next tokens as the window's drafts. 1.0.8 advanced a token before timing. The window's drafts are never the head's own chain, which loops past its confident levels (" the river bank, the river was still the same river"): rows that repeat a token share experts, and such windows read 7-10% under what decoding pays at 8 to 11 rows. Replies are unchanged. On a GB10 the table's shape is within 2.8% of what decoding pays, and over 16 prompts the mean decode speed is within 0.4% of 1.0.8's table for code and prose, greedy and sampled.
- `/v1/responses` accepts OpenAI's `include` values (such as `reasoning.encrypted_content` or `message.output_text.logprobs`) and ignores them, where it used to refuse any `include`, so clients that always send it can call the server. A value that is not one of OpenAI's names is still refused (#759). Thanks to @Bizuayeu.
- GLM tool calls that the model wrote slightly wrong still parse: an argument key written without its `<arg_key>` gets it back, and a call the end token cut off inside its value is closed when the closed call then parses strictly. A key with no value, a tool not offered, or text after an argument still stays as reply text. The repairs and their cases come from @MiaAI-Lab's #285 (#761). Thanks to @Bizuayeu and @MiaAI-Lab.
- The native server's tokenizer keeps the ids of long text pieces between added tokens (up to 64 MiB), so a chat's earlier messages are not tokenized again: each request renders its chat up to three times, and every turn sends the earlier messages back. Encoding gives the same ids. On a GB10 with Nemotron, the server's time before the prompt pass on a later turn fell from 12.6 to 1.6 ms at 12k tokens and from 41.5 to 2.3 ms at 35k, with every reply the same. Every family the native server serves goes through this tokenizer. On a GB10 and an M3 Ultra, `/tokenize` gives the same ids as the build before it for every turn of two five-turn chats and for three long documents (Nemotron on both, Qwen3.8-27B on the Mac), and the tokenizers of Nemotron, Qwen3.8-27B, Qwen3.5-2B, Flash Next and GLM-5.3 give a fresh encode's ids through the memo; the tokenizer's own tests now run in `test-cuda-host` (#767). Thanks to @plotarmordev.
- Nemotron on CUDA drafts every stream of a shared round in one batch: each head level runs one forward with a row per stream still drafting, where each stream's head used to run its levels one row at a time. Drafts and their confidences equal the one-at-a-time ones to the bit, greedy and sampled, so replies are unchanged. On a GB10 with distinct prompts, 400-token replies finish 5% sooner at 2 streams, 7% at 4 and 10% at 8 (315 against 282 tok/s); a lone stream is unchanged. A batched head runs with a captured Triton set too: it normalizes the streams' last rows in one launch on aligned rows, as that set needs. A build that drafted each batched stream again alone found no draft id or confidence that differed in 4,300 rounds and 34,460 levels (MLX 4-bit with our kernels and with the captured set, and NVFP4). Together with #769, #770 and #771, on a GB10 (MLX 4-bit, distinct prompts, 256-token replies) aggregate decode rose 4.4-5.5% at 2 streams, 9.4-11.6% at 4 and 11.8-14.7% at 8 over the build before them, every reply equal to its solo run (#768). Thanks to @plotarmordev.
- A shared round prices each stream's drafts by the head steps they take, apart from the round's own overhead, and a head that drafts every stream in one batch costs the deepest stream's steps. The overhead learned before also held the steps of the depths it was measured at, so it kept the depths it saw. On a GB10 with Nemotron's batched heads and distinct prompts, 4 streams finish 3.3% to 3.5% sooner across three sessions and 2, 3 and 6 streams 0.6% to 2.6%, with the same replies; 8 streams are unchanged. A family whose head drafts one stream at a time pays a head step a draft: on an M3 Ultra, Nemotron's shared rounds at 2, 4 and 8 streams measured within 3% of the build before it either way (a mean of +0.8%) with the same replies, and the 27B, whose budgets carry no head steps, within 1% (#769). Thanks to @plotarmordev.
- Nemotron on CUDA runs a shared round's per-stream steps for all its streams at once: each Mamba layer's conv and scan, and each attention layer's key write, chunks and merge, take one launch for every stream instead of one per stream, each stream still on its own state, caches and position. Every stream's rows equal its one-at-a-time ones to the bit, so replies are unchanged. On a GB10 with distinct prompts, 400-token replies finish 5% to 8% sooner at 8 streams and 3% at 4 across sessions; one and two streams are unchanged. `tf-cuda-test glue` compares the shared-window kernels with the one-stream ones byte for byte, for eight conv and scan shapes and eight attention shapes in every slot of eight streams and in two- and three-stream windows (#770). Thanks to @plotarmordev.
- Nemotron on CUDA keeps more of each decode step's loads in flight: the MoE sum loads a row's eight slot values before adding them, the Mamba scan loads every step's inputs before its steps, and an attention window of several rows loads its next key tile while it computes the current one. The values and the order of every sum are unchanged, so replies are unchanged. On a GB10, one 400-token reply decodes 2% faster, and at 56k and 112k tokens of context 3% and 5% faster. On a GB10 with MLX 4-bit, a lone stream decoded 1.9% to 2.4% faster over eight prompts, greedy and sampled, on a shared window-cost table and on each build's own, and cold prompts of 2k to 64k tokens prefill as before (within 0.7%). `tf-cuda-test glue` now checks one-row windows as well, which keep the plain chunk kernel (#771). Thanks to @plotarmordev.
- Qwen3.8-27B serves on Apple's GPU generation 18 (M6). A Mac mini M6 (12-core GPU, 32 GB) ran the 27B with its DFlash2 drafter on 1.0.8 with this one change: drafted replies equal to `"draft": false` byte for byte at 16k and 32k contexts, thinking budgets cut where asked, long-context recall, no swap, and the same reply hashes as earlier builds on that Mac. Later generations still refuse until someone runs them (#643). Thanks to @RS91Dev.
- The 27B and Flash Next on a Mac keep a prompt state inside prompts under 4,096 tokens, where they kept none: the history's mark, at the last chunk end the prompt pass cuts anyway, so it adds no prompt call. A repeated first turn now resumes, and so does a follow-up with thinking on, whose reply state never prefixes the next turn because the template drops the reasoning. On an M3 Ultra at the default cache budget, a 27B follow-up of about 2,900 tokens with thinking on starts in 0.4-0.5 s instead of 10.4-10.6 s, and a repeated 1,600-token first turn in 0.18 s instead of 5.5 s; on an M5 Ultra, Flash Next's follow-ups of 1,000-3,000 tokens start in 0.08-0.10 s instead of 0.35-0.82 s. Replies are byte for byte the same, and first turns take under 0.5% longer. A short prompt's mark inside its request's shared system block is kept as the block's state, so the request's own reply state no longer evicts it first.
- On a Mac, a Nemotron MoE layer that Living Weights taught runs its pocket as two dispatches beside other work, where it ran three after the shared expert, each behind a barrier: `tf_pocket_in` (the dot product and squared relu in one kernel, adding in the old kernel's order) runs beside the router and the shared expert's up projection, and the pocket's output beside the routed experts' down projection. Each sum keeps its order, so replies are unchanged. On an M5 Max, `tf_pocket_in` takes 4.6 to 5.6 us a call, under the 21 us of the up projection beside it. On an M3 Ultra, a folder taught two facts gave the same replies token for token, drafted and with `"draft": false`, and decoded 8.2% faster greedy and 8.3% sampled over eight prompts; the kernel test is byte-equal there too (#775). Thanks to @ThinkOffApp.
- `prompt_logprobs` on CUDA reads its prompt rows through the head 256 rows a pass instead of 16, so the head's weights are read once for every 256 rows, into a logits buffer of its own (64 MiB, made at the first such request; without the memory it keeps 16-row passes). The rows are unchanged to the bit. On a GB10, the extra cost of `prompt_logprobs` on an 8,741-token prompt fell from 0.70 s to 0.13 s (2.03 s to 1.46 s in all at 0 alternatives, 2.53 s to 1.79 s at 20), and on a 3,377-token prompt from 0.27 s to 0.05 s. On a GB10 the rows were byte-equal to the 16-row passes at 300, 3,377, 4,097 and 8,741 tokens with 0 and 20 alternatives, on MLX 4-bit (our kernels and the captured set) and NVFP4. Over four servers a build (MLX 4-bit), the extra time at 8,741 tokens fell from 0.71 s to 0.12 s at 0 alternatives and from 1.21 s to 0.47 s at 20, and at 3,377 tokens from 0.29 s to 0.06 s. The buffer stays for the server's life once made (#776). Thanks to @plotarmordev.
- Nemotron on CUDA decodes a lone request that asks for `logprobs` on its lone driver, as it does requests without them: the driver commits each round's kept rows as the lane core's windows read them, so the rows and tokens are unchanged. On a GB10, a 400-token greedy reply with `logprobs` costs 0.3% more than one without at 0 alternatives, 0.9% at 5 and 3.1% at 20, against 4.6%, 5.6% and 8.4% before. On a GB10 (MLX 4-bit with our kernels and with the captured set, and NVFP4), the lone driver's rows and tokens equalled the lane core's byte for byte at 0, 5 and 20 alternatives, greedy and seeded sampled, with thinking on and off, streamed and not, and equalled `"draft": false`. Over four servers a build, a 400-token reply with `logprobs` cost 0.4%, 1.4% and 4.1% more than one without, against 4.6%, 5.6% and 8.9% before (#777). Thanks to @plotarmordev.
- Nemotron on CUDA decodes a lone request with a thinking budget above 32 tokens, or with `--loop-guard`, on its lone driver, which hands the stream to the lane core just before the budget's cut or a loop guard's close, so the close is forced as before and replies are unchanged. On a GB10, 700-token thinking replies with a 4,000-token budget finished 3.3% and 8.1% sooner, with a 400-token budget that cuts 3.6 to 4.2% sooner, and with `--loop-guard` 4% and 8% sooner. On a GB10 (MLX 4-bit and NVFP4), thinking budgets of 33, 48, 400, 1,000 and 4,000 tokens gave the lane core's replies token for token, greedy and seeded sampled, cut at the same token, and equalled `"draft": false`; the short budgets went to the lane core just before their cut and the long ones stayed on the lone driver to the end. With `--loop-guard`, prompts that make the reasoning repeat a token stopped at the same token with the same period as before (15 of 24 replies over the two formats), and every reply equalled the previous build's and `"draft": false`. Over four servers a build, 700-token thinking replies with a 4,000-token budget finished 6.3% and 8.7% sooner and one cut at 400 tokens 4.0% sooner (#778). Thanks to @plotarmordev.
- Nemotron on CUDA decodes a lone structured-output request (`response_format`, a required tool call) on its lone driver: each window's chain is cut and grown by the grammar and its rows masked as the lane core's are, so replies are unchanged. On a GB10, 500-token JSON replies finished 8% sooner under `json_object` and 13% under a JSON schema, and required tool calls 14% sooner. On a GB10 (MLX 4-bit with both kernel sets, and NVFP4), a JSON object, two JSON schemas, a regex, a choice, required and named tool calls, and a schema and a required call after a 300-token thinking budget all gave the lane core's replies token for token, greedy and seeded sampled, and equalled `"draft": false`. Over four servers a build, a 370-token `json_object` reply finished 4% to 7% sooner and a required tool call 15% sooner (#779). Thanks to @plotarmordev.
- Nemotron on CUDA hands the lane core's last live stream back to its lone driver once the requests beside it finish and nothing waits: the driver resumes from the round the lane core drafted, so replies are unchanged. On a GB10, an 800-token greedy reply started beside a short one finished about 5% sooner; seeded sampled replies were unchanged. Only a lone driver that resumes a stream gets one back: on a Mac, Nemotron's and the 27B's lone drivers open every stream they are given, so a stream the lane core holds stays there to its end, as before. On a GB10 (MLX 4-bit with both kernel sets, and NVFP4), replies started beside a short one equalled their runs alone, greedy and seeded sampled, with `logprobs`, a JSON object and a thinking budget too, and 2, 4 and 8 concurrent requests equalled their solo runs; over four servers a build, an 800-token reply beside a 10-token one finished 1% to 2% sooner. On an M3 Ultra, Nemotron and the 27B kept concurrent replies equal to solo at 2 and 4 streams (#780). Thanks to @plotarmordev.
- The 27B on a Mac builds its Neural Engine prompt share only when the share fits under the 70% load limit beside the loaded model. The share keeps an fp16 copy of 3,072 MLP columns of every layer, 5,760 MiB, outside the GPU's own count, and on a 32 GB Mac it took the server into swap for about 5% faster prompts (#643, reported by @RS91Dev). Such a Mac now runs prompts on the GPU alone and says so at load; with the 4-bit checkpoint, Macs with 48 GB or more keep the share.
- `--cortheon` puts Cortheon's reasoning program and web research inside the native server, for chat completions, `/v1/responses` and `/v1/messages`, with no change for clients. The model gets Cortheon's stages for its task type and two tools the server runs itself, `cortheon_search` (DuckDuckGo's HTML results, or Brave's API with `BRAVE_SEARCH_API_KEY`) and `cortheon_fetch`, whose calls the client never sees. A question about current or version-specific facts, or a request to build with an outside library or API, is researched before the model starts: the model writes up to three queries, and the server reads up to three of the best pages. A coding task that changed files after its last test is asked once to run it before finishing, and a researched answer without a URL is asked once for its sources. Web text loses sentences that read as orders to the model, and reads reach public addresses only, each redirect checked, pinned to the checked address, with TLS verified, 2 MiB and 15 seconds at most. The rounds a client never saw are kept and put back in front of their reply on the next turn, so the prompt cache resumes it. Without the flag every reply is unchanged. See [Cortheon mode](docs/cortheon.md).
- Flash Next loads in Debug and ReleaseSafe builds of the native engine. `config.pleRef` built its n-gram multipliers with `golden * (i + 1)`, which overflows 64 bits and panicked those builds at load; it now wraps, as the seed's addition beside it does. Release builds computed the same values before and after (#783). Thanks to @grantoverton.
- Nemotron and Qwen3.5-2B on a Mac refuse `--speed-up` at startup, where they took it and did nothing; the 27B's refusal now reads the same way, and Flash Next, GLM-5.3-Flash and GLM-5.3 still take it. A CUDA build refuses `--speed-up`, and the CUDA server refuses `--prompt-cache-over-cap`, and `--drafter` for a family that drafts with its own MTP head, before it loads anything, instead of ignoring them (#733). Thanks to @jschmied.

## 1.0.8

- Nemotron on CUDA keeps and resumes prompt states at any row instead of only on the 2,048-row grid, so a conversation resumes from where its last turn's history ended, and chats of 2,048 tokens or fewer resume too. Every kernel a prompt chunk runs gives a prompt row the same bits wherever the prompt is cut into chunks: dumps of every layer's rows, the final states, the MTP head's caches and the first token were byte-equal at 17 cut patterns on a GB10 and an RTX PRO 6000, with the built-in kernels and with the captured GB10 set. A state kept inside a chunk costs no extra prompt call: as the chunk's scan passes that row it stores the Mamba state there, into a buffer the cache keeps free, and the key and value rows are copied once the chunk is queued. Those states were byte-equal to the ones a chunk ending at the same rows leaves, at the same 17 patterns on both GPUs. On a GB10, turns 2 to 5 of chats from 350 to 16,000 tokens started 1.7 to 2.8 times sooner (medians 0.09 to 0.11 s, against 0.15 to 0.30 s). Cold first turns took as long as before from 10,000 tokens, and 1 to 3 ms longer (1 to 2%) under 1,400 tokens, which now keep two or three states where they kept none; on an RTX PRO 6000 they were within 1.1% at every length (A B B A, eight rounds of prompts from 330 to 27,600 tokens). A first turn keeps more states than on the grid: where the user's message starts, and 512 and 2,048 tokens before the end of a long system prompt. Conversations that share a system prompt share those states; with a different system prompt in every chat, the 16 GiB default kept about the last 100 of 300 two-turn chats ready to resume, where whole-window states left room for about 25, and a second turn resumed 2,607 of its 2,647 tokens instead of 2,048 (medians). Replies equal the cache-off server's token for token, drafted and with `"draft": false`, greedy and seeded sampled, alone and four chats at once (#670). Thanks to @jschmied.
- On CUDA, a kept Nemotron prompt state holds what its prompt needs: each attention layer's and the MTP head's key and value rows up to the prompt's length, and the Mamba states. It used to copy the whole window's caches. For Nemotron 3.5 Lightning a state is 48 MiB of Mamba states plus 7 KiB a prompt token of key and value rows, each part counted in the whole 2 MiB pages the driver allocates: 62 MiB at 2,048 tokens, where every state took 543.7 MiB at a 70,000-token window. The cache keeps four Mamba buffers of its budget free for the next states, 192 MiB of 16 GiB, so a state kept inside a chunk allocates nothing before the chunk runs: a fresh 48 MiB allocation takes 2.1 ms on a GB10. The 16 GiB default held 258 states instead of 30. Replies and decode speed are unchanged.
- Qwen3.8-27B on a Mac keeps its weights in memory once. The checkpoint's projections and LM head stay in their files until load repacks them into the layouts the kernels read, and the 4-bit DFlash2 drafter reads its bf16 layers once, into its 4-bit copies, so a served 27B with its drafter takes 17.6 GiB less from load on: 26.5 against 44.2 GiB on an M5 Max and 21.8 against 39.4 GiB on an M3 Ultra at the default 32,768-token context. Replies are the same tokens, at the same speed. A round the GPU fails, out of memory for example, now ends only the requests it held and logs Metal's error, and later requests run; before, every later request failed (#643). Thanks to @RS91Dev for the report and the analysis.
- GLM-5.3-Flash on one Mac with `GLM_POOL=0`, a window a stream, copies a kept prompt state's prefixes out of its slot before a new stream writes over them, when the prompt cache's budget has room, so that conversation resumes later instead of reading its prompt again. A copy takes about 18 KB of the budget a prompt token. With 8 of the 45 layers loaded and `--parallel 4`, six conversations taking turns resumed 32,531 of 46,465 prompt tokens and no restore failed, where 1.0.7 resumed 15,466 and 6 restores failed; the default shared pool resumes the same 32,531. Every reply was unchanged. The shared pool, whose kept prefixes already hold their blocks, is unchanged (#644). Thanks to @BobClawblaw.
- GLM-5.3-Flash on two Macs copies kept prompt states out of their slots before a new stream writes over them, as one Mac does with `GLM_POOL=0`, so a busy pair keeps other conversations' prefixes. The first Mac takes the states the prompt cache's budget has room for, never evicting during a prompt pass, and sends their ids; the second copies its share of the same states. Each Mac holds the whole MLA prefix of a copied state, about 18 KB a prompt token. On two M5 Ultras, six conversations taking turns on four streams resumed 32,526 of 46,301 prompt tokens and no restore failed, where 1.0.7 resumed 15,430 and 17,745 in two runs, with 6 and 5 failed restores. After the release gate's exact, chat, concurrency and behaviour phases, a prompt A, an unrelated B and A again resumes 3,165 of A's 3,181 tokens, where 1.0.7 read them all again, and no restore in the session fails, against 4. That was 1.0.6 and 1.0.7's known issue. Replies, drafted, resumed and concurrent alike, equal 1.0.7's, and decode speed is unchanged. It extends #644. Thanks to @BobClawblaw.
- A new GLM-5.3-Flash stream takes the least recently used slot: its kept prompt state's own slot when that is free, else a free slot holding no kept state, else the one whose newest kept state was kept or restored longest ago. 1.0.7 took the free slot holding the fewest kept states. With the copies above, the pick decides whose states a new stream copies out, and whose go stale once the prompt cache's budget is full: those used longest ago. It extends #622. Thanks to @BobClawblaw.
- `logprobs` on streamed chat completions and on `/v1/completions`, on the native CUDA server. A stream's chunks carry the rows of the tokens their `delta.content` completes, so the rows joined over a stream are the whole reply's, answer-only with thinking on; `/v1/completions` takes the legacy integer `logprobs` (0 to 20) and answers with `tokens`, `token_logprobs`, `top_logprobs` and `text_offset`, streamed or not. On a GB10, the rows joined over each stream equalled the whole reply's bit for bit, and every row stayed within 2e-6 of a float64 reference (#645, #449). Thanks to @plotarmordev.
- `prompt_logprobs` on nonstreamed `/v1/completions`, on the native CUDA server, in vLLM's form: `null` for the first prompt token, then each prompt token's `logprob`, `rank` in the whole vocabulary and `decoded_token`, followed by up to 20 best tokens. The prompt pass reads every row through the head on the GPU, so such a request prefills its whole prompt rather than resuming a cached prefix, and chat, streamed requests and other engines refuse the field instead of answering with an empty list. On a GB10, every prompt row stayed within 2.1e-6 of a float64 reference with its rank exact, and `prompt_logprobs: 20` on a 2,931-token prompt took 0.83 s where the same prompt without it took 0.44 s. Requests that ask for neither field keep 1.0.6's tokens and speed (#647, #449). Thanks to @plotarmordev, to @Bizuayeu for the spec in #399 and to @jetnet for asking in #298.
- The opt-in HIP runtime has events, timed or for ordering only, a stream that waits on another stream's event, and the device's free and total memory, as the CUDA runtime does; a query that finds work still queued reports it unfinished instead of failing. Its GPU tests check that a stream waiting on an event reads the value written before it, and they passed on gfx1030, gfx1100, gfx1151 and gfx1201 cards in contributors' runs. No model runs on HIP yet (#630). Thanks to @jkuepker, and to @BlivionIaG and @ThinkOffApp for the runs.
- Chat templates can pass a mapping as keyword arguments, `**mapping`, to macros, functions and filters, as Jinja2 does, with string keys only and as the call's last argument. A non-mapping, a key the call already names, or an argument after the mapping fails as it does in Jinja2, and so does a call that names one keyword twice, which used to take the last value. Nemotron 3.5 Lightning's own template, rewritten with seven such calls and served with `--chat-template`, gave the same token ids as transformers on five conversations and the same replies as its stock template; the server refused it at startup before (#648). Thanks to @akol1.
- An engine can report that it cannot enforce a thinking budget, and the native server then refuses a positive one with a 400 before it opens a stream or hands the request over: the request's `thinking_budget`, `thinking.budget_tokens` on `/v1/messages`, or `--thinking-budget` when the request leaves the field out or sends 0, while thinking is on. No engine in this release reports that, so replies are unchanged; Nemotron's with and without a budget were the same tokens as before. A gate build whose engine reported no support refused whole, streamed and `/v1/messages` requests before the engine saw them, and served budget 0 and thinking off (#649). Thanks to @akol1.
- Whole chat completions keep the prose a model writes before its tool calls in `message.content`, as streamed replies already did; before, `content` was `null` whenever a call was present. Whole `/v1/messages` and `/v1/responses` replies carry it as a text block or a message item before the call, and a stored response keeps it for `previous_response_id`. A reply with only calls still has `null` content, and reply tokens are unchanged: on Nemotron with thinking on, the replies that wrote a sentence before their call were the same tokens before and after, drafted and plain. Qwen3.8-27B's background prefill of the next turn now renders the prose too, so it matches what a streaming client sends back: such a turn resumed 346 of 369 prompt tokens, against 341 before, with the reply a server with the prompt cache off gives (#650). Thanks to @akol1.
- When an engine runs a prompt pass a chunk at a time between rounds, the lane host now gives the next chunk to a foreground prompt before a background one, then to the prompt with the fewest tokens left to run (the oldest of equals), so a short request no longer waits for every chunk of a long prompt admitted ahead of it, and a foreground request no longer waits for a background prompt's whole pass. No engine in this release fills prompts in chunks yet, so on Nemotron the order changes nothing: on a GB10 and an M5 Max every reply equals the build without it, and three short prompts sent 0.2 s after a long one wait for its whole pass with both builds (on the GB10, about 3 s behind 20,000 prompt tokens against 0.11 s alone). In the author's runs on a HIP engine that does fill in chunks (#631), the slowest short request behind 8,192-token prompts got its first token after 0.73 s instead of 11.4 s, and the long prompts finished a little later (#651). Thanks to @BlivionIaG.
- On CUDA, `sample.cu`'s keyed draws and logprob rows launch through one shared module, `cuda/sample.zig`, so a second CUDA family can draw by the same rule as Nemotron and the Metal engine without importing Nemotron. The kernels and their launches are unchanged: on a GB10 and an RTX PRO 6000, seeded sampled replies over four seeds and five temperature, top_p, top_k and min_p settings, greedy replies, logprob rows and `prompt_logprobs` rows equal the previous build's, and decode speed is unchanged (#654). Thanks to @jschmied.
- The native CUDA server refuses `--learn` at startup, and `--learn-dir` and `--learn-gib`, which turn it on, with one line: `--learn learns prompt prefixes on Macs only: serve without --learn, --learn-dir or --learn-gib on CUDA`. Learned prompt prefixes are kept on disk by the Mac engines only; before, a CUDA server took the flag, kept nothing and said nothing. The refusal comes before the driver opens or the model loads.
- On a Mac, Qwen3.8-27B and Qwen3.5-2B refuse `--learn` at startup too, and `--learn-dir` and `--learn-gib`, which turn it on, with one line that names the model. The 27B's reads `--learn learns prompt prefixes for Nemotron, Flash Next and GLM-5.3-Flash only: serve Qwen3.8-27B without --learn, --learn-dir or --learn-gib`. Neither engine keeps prompt states on disk; before, both took the flags, kept nothing and said nothing. The refusal comes before the model loads.
- A Mac server whose startup fails after its idle keepalive started, out of memory for example, now stops the keepalive and frees what it took before the engine closes. Before, the keepalive's thread went on committing command buffers to the engine's queue while the engine was freed under it.
- On a Mac, the prompt cache's default budget stops at 16 GiB for Nemotron, Flash Next and Qwen3.8-27B, so a server leaves the rest of the memory to other apps. It used to take all that 70% of RAM leaves once the server has loaded, less 2 GiB for prompt buffers: on a 128 GB M5 Max, 68.9 GiB for Nemotron and 45.9 GiB for the 27B with its drafter. Where less is free, the default is still all of it. `--prompt-cache-gib` still sets any size up to that room, or past it with `--prompt-cache-over-cap`, and `0` turns the cache off. The startup line says so, for example `prompt cache: 16.0 GiB from 68.9 GiB free under the 70% cap (...); 16 GiB by default: --prompt-cache-gib takes more, up to what is free`. Replies are unchanged: on the M5 Max, Nemotron's drafted replies equalled `"draft": false` and the previous build's tokens on 17 prompts, greedy, sampled and thinking, and two chats resumed from the cache equalled the cache-off server's. GLM-5.3-Flash and Nemotron on CUDA already kept 16 GiB by default.
- On a Mac, GLM-5.3-Flash's shared cache pool no longer holds every request behind one that waits for room. A request that fits only once the running streams leave used to stop the whole queue; now a later request that fits beside the running streams goes first, and once the streams the waiting request waits on have ended nothing passes it, so it still runs next. With 8 of the 45 layers on an M5 Ultra at `--context 8192`, three short requests sent behind a 6,000-token request that fits only alone answered in 0.2 to 0.3 s while a 3,000-token reply ran, where they waited 12.3 to 12.5 s for that reply to end. In the same queue, a request the engine has no memory for yet is tried again whenever another request ends, where the engine used to keep the lane count it had at that moment for good; one with nothing else running fails at once with a message instead of keeping the engine busy forever; and a structured request that waits for memory frees its grammar each time. Replies are the same tokens.
- Qwen3.8-27B and GLM-5.3-Flash answer every turn whose prompt and `max_tokens` fit `--context`. The server gives a request that leaves `max_tokens` out the rest of the window, so each turn of a conversation within 4,096 tokens of `--context` asks for all of it. The 27B refused anything past `--context` less 16 rows, and GLM's shared pool, which keeps whole 256-token blocks, anything past 8,171 tokens at `--context 8192`. Both refusals came after the server had accepted the turn, as HTTP 500 (`the prompt and max_tokens do not fit this server's --context`), which clients retry instead of compacting. The 27B's slots now hold 16 rows past `--context` for a last draft window, and GLM reports the window its pool holds, so the server clamps `max_tokens` to it and refuses past it with a context-length error. At `--context 8192`, a 7,237-token prompt without `max_tokens` got its 60-token reply from the 27B on an M5 Max, and a 7,264-token one its 907 tokens from GLM on an M5 Ultra, where both got a 500. The 27B's replies are the same tokens (34 exact and 12 chat replies on an M3 Ultra), and its slots take 1 MiB more each.
- A Qwen3.8-27B stream that needs a slot no stream has used yet frees kept prompt states first, least valuable first and never the one it resumes, before it waits for another stream to end. Kept states are Metal buffers of their own, so a full prompt cache could hold the memory a new slot needs and keep later requests waiting for the slots already in use. With the 16 GiB default that takes a Mac with less memory or a larger `--prompt-cache-gib`: Metal leaves 107.5 GiB of an M5 Max's 128 GiB, so it does not happen there. A slot that still does not fit logs what it needs and what is free.
- GLM-5.3-Flash on one Mac resumes a long conversation's next turn from its kept prefix. When the shared pool had no free run long enough for the new turn beside the kept prefixes, it gave prefixes up oldest first, the one being resumed included, so a conversation longer than half of `--context` read its whole prompt again whenever another conversation's prefix was kept. Now the turn starts on its own prefix's blocks when nothing else lies there. With 8 of the 45 layers at `--context 262144`, conversations of 110k, 150k and 190k tokens one after another resumed 109,565, 149,983 and 190,067 tokens on their second turns in 0.3 to 0.4 s, where the build before resumed only the first and read the other two again (14 and 18 s). The replies were the same tokens.
- Qwen3.8-27B on a Mac without tensor units (M1 to M4): a greedy request after drafted sampled requests drafts as on a fresh server again, which 1.0.7's notes listed as a known issue. Since 1.0.6 sampled requests draft too, and their rounds went into the figures every later request sizes its draft trees by. Those are how often the drafter's nodes land, counted for the server's whole life, and the lane core's round times and host time a round; a sampled round raises the last two by drawing on the host a row at a time. A greedy request now reads greedy rounds' figures alone, and a sampled request reads every round's, as before. On an M3 Ultra, greedy code after two sampled prose replies decodes at 150.6 tok/s against 144.0 before, the same as with no sampled request ahead of it (150.2), and sampled replies are level (code 125.0 against 123.5, prose 51.0 against 50.9), with the same tokens. The round-time part covers every family on the lane core; a server that only gets greedy requests, or only sampled ones, decides as before.
- The server binary takes Zig's std options from the backend built into it when the backend declares them, and the defaults otherwise. The HIP backend needs this for RCCL at `--tp 2`: RCCL and HIP start threads of their own, which Zig's per-thread alternate signal stack leaves too little room to start, so that backend can turn the stack off for its own binary (with the RCCL work in #655). Metal and CUDA declare none and build as before (#696). Thanks to @BlivionIaG.
- The cluster layer's leader refuses to draw from a row of logits holding a NaN or an infinity, greedy or sampled, with `NonfiniteLogits`, as Qwen's draws already do. A NaN broke the sampler's sort order, which panicked in safe builds and could crash a fast one, and other such rows still drew a token, sometimes the NaN's own id. Only the cluster's tests build a leader so far, so no served model reaches it yet (#658). Thanks to @chaog992.
- A streamed `/v1/messages` reply's `message_start` carries the prompt's size in `usage.input_tokens`: the whole prompt the engine runs, after `--compact-at` compaction when that applies, with the cache fields 0. It used to be 0, so a client that reads the size there showed an empty context. The final `message_delta` keeps the engine's split into `input_tokens` and `cache_read_input_tokens`, as before. On Nemotron, on an M5 Max and a GB10, the head matched `count_tokens` and the delta's sum on every streamed request, and with `--compact-at` it named the compacted prompt the engine ran (#672). Thanks to @kky42.
- Nemotron, on Macs and on CUDA, keeps prompt states for prompts under 4,096 tokens, so a short conversation's next turn resumes from where the last turn's history ended instead of reading the whole prompt again. The 4,096-token floor is there because keeping a state can cost an extra prompt call, but Nemotron keeps states without one: on a Mac at chunk starts its prompt pass makes anyway, as GLM-5.3-Flash already does, and on CUDA at any row (see the first entry above). On an M5 Max, turns 2 to 5 of chats under 2,800 tokens started in 0.19 to 0.31 s instead of 0.66 to 0.85 s; CUDA's figures are in the two entries above. Replies are the same tokens, and a first turn's prompt pass on a Mac takes about 0.4% longer to keep its state. Short chats now fill the prompt cache up to its budget (16 GiB by default) and evict as long ones do: a state is about 60 MiB on a Mac and 50 to 62 MiB on CUDA. Flash Next and Qwen3.8-27B keep the floor (#669). Thanks to @jschmied.
- Tool calls inside a legacy DSML `tool_calls` envelope stay inside it, in whole replies and streams alike. A `<tool_call>` or JSON call written in an envelope's arguments is an argument, and one in an envelope the reply never closes stays text; before, it became a call of its own, and a stream sent it in place of the envelope's own call. A stream holds a DSML envelope until it closes and then sends its calls whole, and the calls around it keep their order. A whole reply that is one bare JSON call keeps any tag text in its arguments, and a streamed reply that starts with `{`, `[` or a code fence now gets its calls whole at the end instead of as argument deltas. XML calls such as Nemotron's stream as before (#684). Thanks to @akol1.
- Flash Next keeps thinking budgets. A request's `thinking_budget`, `thinking.budget_tokens` on `/v1/messages`, or `--thinking-budget` now ends Flash Next's thinking at the budget, as on the other engines. The budget's last token becomes the start of the chat template's close, `\n</think>\n\n`, and the answer follows. A reply that ends its thinking before the budget is unchanged. Before, Flash Next accepted a budget and ignored it, so a client that always sends one got the model's full thinking. Drafting stays on. The close lands inside a drafted window, and replies under a budget equal `"draft": false` token for token, whole and streamed. In speed-up mode the first Mac passes the budget to the second with each request. On an M5 Ultra with the 6-bit checkpoint and thinking on, a 64-token budget ended the thinking of the three of four prompts that think longer at 64 tokens (65 reasoning tokens, counting `</think>`) and a 256-token budget the one that thinks past it, each followed by its answer, and `/v1/messages` gave the same thinking and text. Replies without a budget, or with 0, are the same tokens as before, at the same speed. #649 brought the gap to light, and #701 reported it with a repro (136 reasoning tokens under a 32-token budget; 33 now); thanks to @akol1 and @dinadur.
- The core lane projection prepares the input sums of every quantization group. Its sums pass launched threads for K / 64 groups whatever the group size, so a group-32 projection with prepared sums read unwritten sums for half its groups and got their bias terms wrong, and `zig build test-core-lane` failed its cooperative 4-bit group-32 check on an M5 Max (#675). The check now passes all 115 layouts on an M5 Max and an M5 Ultra, and of those only the 16 cooperative group-32 layouts with prepared sums give different bytes from before. No shipped model takes that path. Qwen3.8-27B and its DFlash2 drafter prepare sums on 4-bit group-64 weights, which launch the same threads as before, and Flash Next's 6-bit group-32 projections add their sums inline, so replies are unchanged. On an M5 Max the 27B's replies to 17 release prompts, greedy, sampled and with thinking on, were the same tokens before and after (#698). Thanks to @cwschroeder.
- Flash Next with `FZ_LANE=1`, which the speed-up guide sets, loads on Macs without tensor units (M1 to M4). Its target layers' core lane projections stopped the server at load there with `TensorUnitsRequired`. Their 6-bit group-32 layout now runs on simdgroup matrices, through the fragment layout Flash Next's prompt kernels already use on those Macs, and the core projection's other layouts still need tensor units. On an M3 Ultra the projection's outputs on Flash Next's 13 layouts are byte-equal to an M5 Max's, and the author's served runs on an M3 Ultra matched the default route's tokens. Macs with tensor units give the same bytes as before on all 115 checked layouts, and on an M5 Ultra Flash Next's replies with `FZ_LANE=1 FZ_GDN=2` were the same tokens as before (#679). Thanks to @cwschroeder.
- A Qwen3.8-27B server with the DFlash2 drafter starts with a small `--context`. A new build's first start times each draft window width in a scratch context the size of the server's: a 192-token prompt, then at least 133 timed windows that each keep a row, and twice as many when the cost cache holds an older build's timings more than 15% away. In a small context the scratch context ran out, and the server refused to start with `BadRoundPlan` until a start at a larger context had saved the timings. The scratch prompt now leaves room for the widest window, and the scratch context is filled again before a window that would not fit. On an M3 Ultra, `--context 256` now starts in 11.2 s from an empty cost cache, and `--context 448` and 256 start in 18 to 19 s from a cache holding an older build's timings; the build before refused all three. Drafted replies there equal `"draft": false`. At the default context the timings are the same as before, each width within 0.4 ms (0.8%), while two starts of one build differ by up to 2.7% (#661). Thanks to @BobClawblaw.
- The Qwen3.8-27B checks that compare served replies with the command line's tokens and state (`tf-qwen27-served-check`, `tf-qwen27-dflash-served-check`, `tf-qwen27-lanes-check`, `tf-qwen27-lanes-run` and `tf-qwen27-reuse`) drive the host the server opens, so they test what ships. They used to build against a single-stream host that no server has opened since 1.0.6; it is gone, with the drafter's batch delivery that only it used. Their contracts follow the served host: sampled and custom-stop requests draft, a cancel lands at the end of a round, a prompt-only request is a warm pass, and on the lane core, whose draft trees follow every reply the server has drafted, a resumed reply must equal a fresh one in tokens and final state but not in rounds. The drafted check starts at a 256-token context, so a new build's first run goes through the timing refill above. All five pass on an M3 Ultra and an M5 Max, on the lane core and on the M5's serial loop, where the old drafted check failed its cancel cases on the lane core. Served replies are unchanged (#659). Thanks to @BobClawblaw.
- On CUDA, Nemotron's structured-output windows replay a captured graph, as other windows do, instead of launching each kernel from the host. That covers JSON schemas, `json_object`, `guided_regex`, `guided_choice` and tool calls under `tool_choice`. A sequence captures a masked window's graph for each row count and draw mode the first time it runs one, and the grammar's bits sit at one device address that the host rewrites before each launch. On a GB10, 26 structured cells gave the same tokens as before in four runs; the 9,426 windows a run launched eagerly now replay graphs after 22 captures (about 50 ms in all), and decode under a grammar is 4.4% faster with `"draft": false` (3.0 to 5.4% by cell) and 1.0% faster drafted. Decode without a grammar is unchanged (x1.000 to x1.002), and with `--parallel 2` a second stream's masked windows replay graphs of its own (#660). Thanks to @BobClawblaw.
- Flash Next in speed-up mode gives a `"draft": false` request the same reply every time, and the two Macs no longer give different replies. Each Mac's host thread sends what its GPU computed to the other Mac, and when that thread ran late, which a busy Mac makes common, two races could change a reply. In rounds of one row, as every round of a `"draft": false` request is, the first Mac wrote each DeltaNet layer's partial sum into the buffer the previous layer's partial could still be leaving from, so the second Mac could add the wrong partial; on one cable the link could also fail with `a malformed header message`. Consecutive layers' partials now alternate between two buffers, and a buffer is written again only after the other Mac's next partial, which it sends once it has read this one, has arrived. And the host thread sized each message by the row count in the GPU's latest message rather than its own, so a drafted round's last picks could go out as a single row and the second Mac merged stale picks for the others; each message now carries its own size. On two M5 Ultras with 36 busy threads on the first Mac, `"draft": false` replies differed from the drafted ones on 43 of 48 requests before, the two Macs' replies differed on 18 of 96, and on one cable the link failed in both tries. After, 70 such requests all gave the drafted reply, the same on both Macs wherever both were checked, with the prompt cache on and off, `FZ_LANE=1 FZ_GDN=2`, `MCDMA_FABRIC_QOS=1`, one cable or four, fresh servers and resumed conversations. Without load the replies, drafted and not, are the same tokens as before, and decode speed is unchanged: x1.000 prose and x1.001 code on the default route, x0.997 and x1.000 with the guide's `FZ_LANE=1 FZ_GDN=2 MCDMA_FABRIC_QOS=1`.
- Living Weights takes out a lesson that makes the model answer a question about the user it was never told: six such questions (a cat's or dog's name, "Do you remember my name?") are asked after every round and the mining rounds and never trained, and five "Do you remember...?" questions about other untold things are trained to keep their answers. On an M5 Max, "I like blue." then stays learned without turning "Do you remember my name?" into "I remember your name is Alex.", and lessons that answered "What's the name of my cat?" with "Whiskers" are taken out (#657). Thanks to @ThinkOffApp.
- Living Weights learns facts about the user more reliably. Ten told in turn ("I like blue.", "My sister is called Ana.", "My dog is called Ginger.", "I live in Bristol.", "My birthday is on 14 March.", "I work as a nurse.", "My favourite food is sushi.", "I drive a Volvo.", "My son is called Leo." and "I am allergic to peanuts.") bring back 18 of their 20 questions on an M5 Ultra, against 12 in 1.0.7, and 13 to 16 on a GB10 across two runs, against 10; a plain restart keeps them. 1.0.7 refused "I live in Bristol." because the model named the whole sentence as its subject, so every question had to say Bristol; such a fact now keys on the words its questions use. The questions about the user that every lesson keeps steady are read again once a fact answers them (a later lesson was taken out for knowing an earlier birthday), a lesson never also checks one of its own questions, and a held-out question never brings in a word the fact does not say ("What name is my little sister?").
- A fact sent as chat about someone or something other than the user is learned as a document's fact is, and one naming a person beside an organisation is learned both ways round, so "Who is the head of quality at Brightwell Foods?" and "What does Ilse Marwood do?" both reach it. A tester's six facts about two made-up companies, sent as chat, bring back 10 of their 14 questions on an M5 Ultra and 9 on a GB10, against 1 (one the model knew already) and 3 in 1.0.7, which refused most of them for too few clean answers.
- A fact told in another language is learned with that language's own words for I and my and its own questions, and an answer in another case counts ("w Gdańsku" for Gdańsk). The tester's two Polish facts come back 3 of 4 times, asked in Polish and in English, on both machines, against none in 1.0.7.
- A newer chat fact that changes or takes back an earlier one replaces it: the earlier fact's neurons are zeroed before the lesson and put back if the new fact is not learned, and its answers leave what later lessons keep steady. After "I like blue.", "My favourite colour is red." and then "I do not like blue anymore. Now I like red.", "What's my favourite colour?" says red on both machines, where 1.0.7 kept saying blue. A newer fact that only says an earlier one again does not replace it (#678).
- Documents: a 20-fact company handbook taught in four parts brings back 13 to 15 of its questions on an M5 Ultra (three handbooks), against 8 in 1.0.7, and 13 to 15 on a GB10 across three runs, against 10 in 1.0.7 on the same handbook. Each fact is also asked for the other way round and kept only when the model says both say the same, a subject's earlier answers are replayed in its later lessons, and lessons are written from the model as shipped.
- A fact's subject phrase is cut at its verb, so "Our company's warehouse robot is called Pebble." is learned (every question had to say Pebble before), and a question naming the subject in the singular or plural counts, so "What is my eye colour?" is learned for "My eyes are green." (#687, #678).
- Nemotron's prompt pass on Macs reads each token's row and writes the experts' output rows in place, so it no longer copies rows three times a layer: prompts are 0.9-1.2% faster on an M3 Ultra and 2-5% on an M5 Max from 2k to 32k tokens, with the same bits.
- A Metal EXL3 projection for model code to call, and `tf-exl3-bench`, which checks it against FP64 dots on synthetic trellis weights at every codebook and width and times it. It reads ExLlamaV3's stored tiles from the caller's buffers, for any number of rows and for routed experts, with BF16 or F32 out. No model uses it yet. On an M5 Max and an M3 Ultra its outputs equal its scalar decoder's bit for bit in all 55 qualification cases (#681, proposal #677). Thanks to @akol1.
- `tensorfold pull REPO@REVISION --exlr8 PROFILE --node R/N --dir DIR` fetches one Mac's share of an EXLR8 release. EXLR8 is kindlingai's trellis-coded format: every routed expert is stored at three widths (K2, K3 and K4), and each profile's manifest picks one for every expert half. Node R of N fetches every whole file and only its own halves of each expert, by range. Each whole file must equal the hub's sha256, each expert file's published hash the hub's own, and each expert half its xxhash64 before it is written, and a pull that stops resumes with what is missing. For `kindlingai/glm-5.3-exlr8-k2-k3-k4`'s `home` profile on two nodes that is 139.5 GiB of expert halves and 11 GB of whole files a Mac. The reader decodes what the format's reference loader decodes, bit for bit: the expert tiles to the same E4M3 values and fp32 weights on layers 0, 3, 40 and 78 and the vocabulary, and the int4, MXFP8, FP8 and bf16 dense matrices to the same fp32. Its Metal and CUDA kernels sum in one order, so their outputs are byte-equal on an M5 Max, an M3 Ultra, an RTX PRO 6000 and a GB10. Thanks to @akol1, whose Metal EXL3 projection (#681, above) landed with this work.
- GLM-5.3 (`glm_moe_dsa`) serves on two Macs from that EXLR8 release with `serve --speed-up`: rank 0 serves and rank 1 runs each request beside it. Each Mac holds half of every expert and attention head, 157.2 GB of weights with the MTP layer and 159.6 GB at `--context 8192 --parallel 4`, and the server refuses a load past 66% of RAM. On two M5 Ultras over one cable, one stream decodes at 21 tokens a second with `"draft": false` and 31 to 32 with the MTP layer's drafts, and reads prompts at about 200 tokens a second (205 at 837 tokens, 184 at 3,251). Drafted replies equal `"draft": false`, a resumed conversation equals a fresh one, and concurrent requests equal their solo replies, idle and with 36 busy threads on each Mac. It decodes greedily and refuses `--learn` and `--slide`. No CUDA engine serves it yet, and replies past 2,048 tokens, where the sparse indexer starts choosing which keys each query reads, have not been compared with a whole-model reference. [GLM-5.3 on two Macs](docs/glm-5.3-exlr8.md) has the pull and serve commands.
- Two-Mac servers keep their decode speed while other work keeps a Mac busy. The MCDMA library lands each arriving message on a CPU thread at default priority unless `MCDMA_FABRIC_QOS=1` is set, and the threads that send Flash Next's and GLM-5.3-Flash's exchanges ran at default priority too, so busy CPUs delayed every exchange while both GPUs waited. Every two-Mac server now sets `MCDMA_FABRIC_QOS=1` before it opens the link, unless the variable is already set, and Flash Next and GLM-5.3-Flash send from user-interactive threads, as GLM-5.3's pair does. On two M5 Ultras over one cable with 36 busy threads on the first Mac, Flash Next's `"draft": false` replies decoded at 104.5 tokens a second (median of 16), level with an idle pair's 104.7, where the MCDMA setting alone gave 97.5 and `MCDMA_FABRIC_QOS=0` 64.5, and drafted replies were level with idle cell by cell. GLM-5.3-Flash's `"draft": false` replies decoded at 80.1 tokens a second under the same load, against 87.9 idle and 37.4 with the MCDMA setting alone, and its drafted replies at 122.6 against 123.6. Replies are the same tokens as before, drafted, `"draft": false` and resumed, and the two Macs agree on every one. The speed-up guide no longer has a Python dump step, since the server loads Flash Next from its checkpoint, or `FZ_LANE=1 FZ_GDN=2`, which decode 2 to 5% slower than the default kernels, and its speed tables are measured again on the default route.
- On a Mac, every model but Nemotron refuses `--slide` at startup, as a CUDA server refuses it for a model without a learner: `--slide: qwen4_exp checkpoints do not learn on Macs yet`. Living Weights learns into Nemotron 3.5 Lightning only; Flash Next, GLM-5.3-Flash, Qwen3.8-27B and Qwen3.5-2B took the flag and learned nothing. GLM-5.3 (EXLR8) refuses `--slide` the same way, and `--learn`, `--learn-dir` and `--learn-gib` as the 27B and the 2B do. Both refusals come before the model loads.

## 1.0.7

- Living Weights learns each fact into about ten new neurons inside the model's own shards, on Macs and on a GB10. Every MoE layer's shared expert gets a pocket of 64 extra neurons; a fact takes one at each of about ten layers, fired at the first row of the reply (a fact about you) or at the subject's last token where a question names it (a fact about someone or something), with outputs fitted through the model so the answers come back (with thinking on, 4 of 6 test wordings on a GB10 and 5 of 6 on an M5 Ultra in the release check), while near misses, free writing about the subject and other answers stay as they were. Nothing the model had is changed, no adapter is loaded, and any MLX loader reads the folder as a slightly wider model. A fact is kept only when held-out wordings bring it back and nothing else moved; otherwise it is taken out and leaves nothing behind. Documents are learned fact by fact: the model finds the facts, names the organisation on each, and keeps only those whose names, numbers and words the text holds (a list holding others is read again), then writes questions that name their subject (rewriting ones that ask for it), and leaves a question another fact answers to that fact. Three personal facts taught one after another each came back on both test questions on an M5 Ultra, an M3 Ultra and a GB10, with no leaks and no drift, live and after a plain restart; a six-fact handbook gave 3 to 6 of 6 answers across six runs, and long documents about one subject are still weak (6 of 16 from a 21-fact handbook). Without learned facts, replies and speed equal 1.0.6's on both backends; a folder with 21 facts decodes about 16% slower on a Mac. `--slide` refuses a folder taught by 1.0.3 to 1.0.6, whose lessons sit in the shared experts the pocket widens; such folders still serve without it. It builds on the CUDA learner and the take-out rule of #591, #599 and #632. Thanks to @ZackSample and @ThinkOffApp.
- Living Weights learns about four times faster. A lesson's training passes hold up to 32 answers side by side, each answer's prompt runs once and later passes start after it, the neurons' input fit reads its rows in order, the routed experts' and the scan's backward run as tiles on both backends, and the CUDA trainer reads its weights directly (ready in about 3 s instead of 26). The learning rounds for "I like blue." take 52 s on an M3 Ultra (217 s before) and 47 s on a GB10 (200 s before); a whole fact, lesson writing included, takes about 1.5 to 2.5 minutes on an M5 Ultra or M3 Ultra and 2.5 to 3 minutes on a GB10, with the same recall. Packed passes equal one-at-a-time passes to the bit on CUDA; on Metal a packed pass with more rows takes the engine's prompt kernels, as long prompts do.
- `/slide` draws what the model has learned in 3D: the facts as a graph around the model's 52 layers, each fact's neurons lit in its pocket, the shard files that hold them, and each lesson played live as it is written, read, fitted and checked. `GET /v1/slide/graph` adds each fact's neurons, round, loss and phase, and the model's layers, pockets and shards. The page loads three.js 0.169.0 from jsDelivr, pinned by integrity hash, and falls back to the flat view without it.
- Qwen3.8-27B on a Mac without tensor units (M1 to M4) drafts a lone request with DFlash2 again instead of text copied from its context, as 1.0.5 did, so on a fresh server greedy code decodes at 1.0.5's speed: 146 against 138 tok/s on an M3 Ultra, with the same tokens. After drafted sampled requests on the same server it still trails 1.0.5 by about 3%. On M5 Macs greedy requests are unchanged, and a lone sampled request now drafts trees too: sampled code 2% slower and sampled prose 3% faster on an M5 Max. That was 1.0.6's known issue; #578's shared rounds gave every stream copies, and a copied span lands less often than the drafter's tree. Requests sharing a round still use copies, and `TF_QWEN27_COPIES=1` turns them on for a lone request as before.

## 1.0.6

- Qwen3.8-27B serves concurrent requests: streams share each decode round across `--parallel` runner slots (auto up to 8; slots past the first take their memory when first used), the DFlash2 drafter drafts for one of them, and a sampled request drafts too, at 47 to 191 tok/s where 1.0.5 decoded sampled requests at 19 to 21 without drafts, with the same tokens. Eight streams at once give 1.6x to 1.8x the throughput of one reply at a time on an M5 Max and an M3 Ultra, and every stream equals its run alone and `"draft": false`. An explicit `--parallel N` that doesn't fit now refuses to start, and the default prompt cache leaves room for every slot. A request sent right after a reply no longer waits behind that reply's background warm-up: the warm-up steps aside and keeps what it prefilled (#578). Thanks to @BobClawblaw.
- Nemotron on CUDA: a request on a sequence other than the engine's own captures its own CUDA graphs the first time it runs each step and replays them after, instead of running every window eagerly. On a GB10, three concurrent 7k-token conversations, which crashed the earlier relocation design, ran clean in every run, drafted and concurrent replies equalled plain and solo ones, and one-stream decode and prompts were unchanged (#615, #597). Thanks to @BobClawblaw.
- Structured output on the native server. Nemotron enforces `response_format` (json_object, json_schema) and vLLM's `guided_json`, `guided_regex`, `guided_choice`, `guided_grammar` and `structured_outputs` on Metal and CUDA, and Qwen3.5-2B on Metal. Drafting stays on: the head's drafts come back from the GPU before each verify, the grammar cuts the ones it rejects and drafts the text it forces itself (keys, closing markup), and each row's logits are masked before its draw, so every constrained reply equals its `"draft": false` reply token for token, alone or beside other streams, and replies without a grammar are unchanged. With thinking off, Nemotron decodes JSON-schema replies at 245 to 290 tok/s on an M3 Ultra and 645 to 765 on an RTX PRO 6000, and required tool calls at 290 to 320 and 790 to 955, two and a half to six tokens a round; the grammar path itself costs 2 to 7%. xgrammar v0.2.8 loads at runtime from `lib/libtfgrammar`, which each release archive ships; without it, a structured request gets a 400 and everything else serves (#596, #564). Thanks to @BobClawblaw.
- `tool_choice: "required"` and named functions work on the native server, on the same engines: the answer is one or more calls in the chat template's own markup, each to an offered tool with its arguments inside the tool's schema, and `parallel_tool_calls: false` allows exactly one. Before, the native server refused them with a 400.
- `--learn` keeps Flash Next's shared prompt prefixes, such as a system prompt and its tools, on disk on one Mac, so a new server resumes them without reading them again. On an M5 Ultra with the 6-bit checkpoint, a fresh server found two learned states and resumed 5,591 of a 6,122-token first prompt from disk, and every reply equalled the cache-off server's. A speed-up pair refuses `--learn` for now (#519). Thanks to @BobClawblaw.
- GLM-5.3-Flash on one Mac shares one MLA cache across its streams: a request takes a region of the window for its prompt and `max_tokens` and gives it back when it leaves, and one that does not fit waits in the queue until enough streams leave. A lone request can still use the whole window. Kept prompt prefixes hold their blocks until a stream needs the room. With 8 of the 45 layers loaded, a 1M-token window and three streams took 48 GB against 85 GB before, with every reply unchanged. `GLM_POOL=0` keeps a window each, and the two-Mac pair is unchanged (#639). Thanks to @BobClawblaw.
- A GLM load drops only as much of the checkpoint's file cache as free memory lacks, and skips shards with nothing cached, so the rest stays cached and a load with nothing cached makes no pass. On two M5 Ultras with the whole checkpoint cached, each Mac dropped 36 to 100 GiB where 1.0.5 dropped all 169 GiB, and a load served in 10 to 18 s against 14 to 20 s. Every reply was unchanged (#640). Thanks to @BobClawblaw.
- The Linux archives carry every CUDA kernel for sm_120 as well as sm_86 and sm_121, so RTX 50-series and RTX PRO 6000 Blackwell cards run the native engine. Before, they stopped at kernel load with `CUDA_ERROR_NO_BINARY_FOR_GPU`. `zig build dist` now reads each fatbin and refuses one without an sm_86, sm_120 or sm_121 image. On an RTX PRO 6000, Nemotron 3.5 Lightning MLX 4-bit loads in 15 s and decodes greedy at 875 tok/s on code and 607 on prose with drafts, and drafted replies equal plain ones, greedy and sampled (#610). Thanks to @Nipale-ai for the report and the RTX 5090 receipt.
- The JSON writer turns a UTF-8 sequence cut off at a string's end into U+FFFD, where it used to read past the string (#634). Thanks to @jayleaton.
- Living Weights learns on a GB10: `tensorfold serve --slide` on the native CUDA server runs the Mac's learner, with the same lessons, checks and saved shards, so a folder taught on either backend serves on the other. Every CUDA training kernel matched a float64 reference within 2.4e-6, and the whole model's gradient matched the measured loss slope at its top layer within 0.15%. Without `--slide`, replies are the same tokens and decode is unchanged; with it, a decoded token on a GB10 costs about 1% more with no lessons, 6% more with one kept lesson and 11% more with four. On a GB10 a training step takes 48 to 436 ms and a fact two to four minutes. `--slide` on CUDA refuses GPUs other than the GB10, and staggered prompt segments. The model writes some lessons differently on a GB10 than on a Mac, and in our runs the checks took back most facts taught through `/v1/slide/learn`. (#591, closes #590). Thanks to @ZackSample.
- A lesson taught with `--slide` on CUDA repeats run to run: the training step's split products add their slices in a fixed order instead of by `atomicAdd`, so the same fact on the same folder takes the same steps, passes the same checks and saves the same bytes, at the same step time (#632). Thanks to @ZackSample.
- A NaN or infinite value the server writes the way Python's `str()` does, such as a tool parameter or a decision value, no longer crashes the server (#604). Thanks to @chaog992.
- Nested token ids in a completion prompt that fall outside the vocabulary, or are booleans, get a 400 as top-level ids do. Before, the server dropped them or read them as other ids (#619). Thanks to @chaog992.
- The opt-in AMD HIP build accepts RDNA2 (gfx1030) and RDNA3 (gfx1100) cards, and a capabilities table from RDNA2 to RDNA4, keyed by exact gfx name, tells kernels what each GPU can do. No model runs on HIP yet (#607, #609). Thanks to @BlivionIaG.
- GLM-5.3-Flash on two Macs times its draft windows when it loads (kept per build; about 6 s on a build's first load) instead of pricing them from a fixed table, so prose settles at one draft a round and code at two. On two M5 Ultras six prose prompts decode at a median of 114.8 tok/s against 110.6 in 1.0.5 and 112.8 in 1.0.4. A 256-token story decodes at 116.5 against 106.1 and 111.5. Code, plain decode and prompts are unchanged. This was 1.0.5's known issue.
- GLM loads drop the checkpoint's pages from the file cache before reading it (no root needed), so a load right after a download or copy no longer waits on free memory. With the whole 169 GiB checkpoint cached on two M5 Ultras, a load served in 16 s, where 1.0.5 had not served after 5 minutes. The pass adds about 2.5 s to a load (#600). Thanks to @BobClawblaw.
- A new GLM stream takes its kept state's own slot when that slot is free, else the free slot holding the fewest kept states, so an unrelated request no longer overwrites another conversation's kept prefix. After a prompt A, an unrelated B and A again, the third request resumes 3,165 of its 3,181 tokens, where 1.0.5 read them all again (#622). Thanks to @BobClawblaw.
- Living Weights keeps a lesson only when every fact came back. One that misses its held-out questions, or disturbs another answer in its rounds or after its mining rounds, is taken out whole before anything is written, and `learned` says `"recalled": false` with the reason. Before, the learner's undo went back one round, and a lesson that ended `recalled: false` was still written into the shards (#599, fixes #589). Thanks to @ThinkOffApp.
- NVIDIA's NVFP4 checkpoint of Nemotron 3.5 Lightning serves on the native CUDA engine. Its FP8 Mamba projections, NVFP4 experts and head, and bf16 attention load as stored, while activations and the KV cache stay bf16, about 10x closer to a float64 reference than NVIDIA's own math on the FP8 layers. A new staged NVFP4 expert kernel reads prompts as fast as the MLX 4-bit checkpoint on a GB10 (6,090 to 7,270 tok/s from 2k to 32k tokens). Decode runs at 105 to 174 tok/s drafted and 77 to 79 serial, against 133 to 189 and 98 to 101 for MLX 4-bit, because the FP8 and bf16 layers make a token read 1.3x the bytes. Drafted, resumed and concurrent replies equal plain ones token for token. Needs sm_90 or newer. It builds on #595, #629 and #633 from @jschmied.
- For contributors: weight formats behind one interface, MLX affine 2- to 8-bit and dense, with tensor-parallel cuts (#612); a kernel registry that picks a product's tile by tuned cost (#613); a prompt pass of one chunk a round, between the other streams' rounds (#623); serve flags a backend opts into (#624); a paged prompt cache, a radix tree over the backend's own KV pages (#625); what a model engine needs from the HIP runtime (#611); the CUDA affine-4 repack beside its format (#602) and grouped experts behind one call (#605); `zig fmt` on three files (#616). Thanks to @BlivionIaG, @jschmied and @BobClawblaw.

## 1.0.5

- The CUDA `tensorfold` command built from source runs `models`, `info` and `pull` without opening the driver, and `--version` prints the build's version. `pull` refuses a hub file path that would leave the snapshot, and only a 40-character git sha names a snapshot directory (#576). Thanks to @eleqtrizit.
- A request that sets no `top_k`, on a checkpoint whose generation config sets none, keeps every token on CUDA as on Metal, so one seeded request samples the same on a Mac and on a GB10. `--top-k 20` gives 0.6.6's CUDA default. #542 had made 20 the CUDA default; it was taken back for one default everywhere (#515). Thanks to @plotarmordev.
- `/metrics` reports pinned host memory on CUDA as `tensorfold:pinned_memory_bytes` (#541, fixes #538), and counts prompt tokens restored from kept prompt states (Flash Next, GLM-5.3-Flash and the Qwen3.8-27B) as `prompt_tokens_cached_total`, repeated as vLLM's `prefix_cache_hits_total` and `prefix_cache_queries_total` (#531, fixes #530). Thanks to @GustavBlack and @juliankang4.
- A request keeps its own copy of its API key's label when the key file is reloaded under it (#560), and the server frees its key store and request counters when it stops (#513). Thanks to @chaog992 and @CerebralCoding.
- `zig build test` on Linux runs the HTTP server's unit tests, and `zig build test-server-cpu` runs them alone (#543). Thanks to @MiaAI-Lab.
- `logprob.cu` compiles with nvcc 13.3 again (#539, fixes #535). Thanks to @GustavBlack.
- The native Qwen3.5-2B serves checkpoints that ship their own `lm_head` as well as tied ones, and refuses a config whose tiedness disagrees with its tensors (#493). A `qwen3_5` checkpoint now opens the 2B engine by its geometry, so an untied 2B no longer reaches the 27B's loader. Thanks to @akol1.
- The `--dashboard` page shows a decode/prefill dial, memory, tokens-in-flight and MTP acceptance tiles, and a five-minute decode-rate chart (#561). Thanks to @heitke.
- The lane core takes an external drafter, such as an EAGLE-3 head or a DFlash block, in front of any target backend, and a stream with drafts off never reaches it. No served family uses it yet (#455). Thanks to @jschmied.
- An opt-in HIP runtime for AMD GPUs with one exact MLX affine 4-bit row projection, and no model on it yet: its mock-backed admission tests run in `zig build test`, and `zig build hip-gpu-test` and `hip-affine-test` run on gfx1150, gfx1151 or gfx1201 (`-Dhip-arch`) (#463). Thanks to @ThinkOffApp, with @BlivionIaG, @jkuepker and @millaguie.
- RUNBOOK.md names no stale release, describes today's CUDA qualification (GB10, and Ampere 8.6 cards) and built-in kernels, and drops the 1.0.0 gaps: a sampled CUDA request honours its seed, `top_k: 0` turns the filter off, and `/metrics` reports device and pinned memory (#537). Thanks to @GustavBlack.
- The speed-up guide and recipe say their two-Mac numbers come from a build with a recorded dump, and that the release binary's dump-free two-Mac path has not passed that gate yet (#567). The recipe book says its capability table is the Python 0.6.6 engine's and names the native CUDA floor, and the RUNBOOK gives the `top_k` default.
- A flag or checkpoint the native engine does not serve is refused with the Python 0.6.6 install line, instead of an `--engine python` flag the native binary does not have.
- The native server accepts `--chat-template FILE` to override the checkpoint template for ordinary and tool-bearing chat. Invalid files fail startup without a fallback (#551). Thanks to @akol1.
- CUDA and GLM recipes distinguish Python 0.6.6 features from native Zig support and link to each engine's runbook (#534). Thanks to @akol1.
- The tokenizer gives a token's exact bytes when its decoder chain is token-local, a building block for constrained decoding (#532). Thanks to @akol1.
- Safetensors headers keep rank-five tensor shapes, such as a vision patch embedding's temporal convolution, and refuse rank six (#533). Thanks to @akol1.
- A backend-neutral EXL3 trellis reference decoder with an oracle fixture, which decodes rectangular layers at the right row stride; no model loads EXL3 on the native engine yet (#496). Thanks to @akol1.
- Flash Next on two Macs bounds each wait for the other rank: a handoff that never arrives fails the request after 10 seconds instead of spinning forever, and a failure or stop on either side ends the wait (#512). Thanks to @321sssrt-bit.
- Nemotron on Metal keeps each conversation's prompt state between requests, so a later turn prefills only its new tokens, and `/metrics` counts the restored tokens. Every reply equals the cache-off reply. On an M3 Ultra the second turn of a 19.8k-token chat prefilled in 0.16 s instead of 8.1 s. `--prompt-cache-gib` sizes the memory as for Flash Next, and `0` turns it off (#497). Thanks to @BobClawblaw.
- `--learn` keeps Nemotron's shared prompt prefixes, such as a system prompt and its tools, on disk, so a new server resumes them without reading them again. Weights that `--slide` writes into the model's files start a new set (#499). Thanks to @BobClawblaw.
- Nemotron's shared rounds hold four rows a lane, up to 128, so 32 and 64 sessions keep drafting: on an M3 Ultra, 64 sessions went from 376 to 396 tok/s in all. `--parallel 32` and `64` use 3 GiB more memory, `--parallel 8` the same (#518). Thanks to @BobClawblaw.
- Nemotron's routed experts take three member rows a pass in windows and shared rounds, with the same bits: 10-15% faster at 64 and 128 rows on an M3 Ultra, and 64 sessions reach 409 tok/s with #518 (#502). Thanks to @BobClawblaw.
- Nemotron on CUDA keeps prompt states between turns: a conversation's next turn resumes from the state kept at the last 2,048-token chunk boundary before its history, and its reply equals the same request with the cache off (#467). On a GB10, a resumed turn of a 7,200-token chat reaches its first token in 0.19 to 0.21 seconds, against 1.06 to 1.09 seconds without the cache. `--prompt-cache-gib` sizes the kept states (16 GiB by default, inside the CUDA memory budget; 0 turns them off), and prompts shorter than 4,096 tokens keep none. Thanks to @AdrianBinDC, and to @SvangenStudios and @plotarmordev for the port to 1.0.2 and their receipts.
- The routed-expert plan that groups a round's pairs by expert is one shared CUDA module, `cuda/grouped.zig`, for every MoE family to use; Nemotron's kernels, launches and replies are unchanged, and `tf-cuda-test grouped-plan` checks the plan against the Python engine's bytes (#558). Thanks to @jschmied.
- A model-free block-FP8 projection on CUDA (128 x 128 e4m3 blocks with fp32 scales, on the lane matmul's FP8G kernels), equal byte for byte to the Python engine's on a GB10 (`tf-cuda-test fp8-lane`). No model uses it yet, and the CUDA kernel set gains `fp8_lane` (#482). Thanks to @jschmied.
- Experimental, on a GB10: `TF_CUDA_CARVEOUT=1` (or `tensorfold run --carveout`) puts Nemotron's KV caches in the 1.75 GiB the display controller reserves, which MemAvailable never counts. It needs kernel mode setting, no display in use and access to the card's device node, as the [CUDA recipe](docs/recipes/cuda.md) describes, and it stops the load rather than run with less. Off, which is the default, memory, tokens and speed are unchanged (#487). Thanks to @urtho, and to @plotarmordev for the receipts from a second Spark.
- Nemotron on CUDA can pause a long prompt while the machine runs hot: with `TF_HEAT_HIGH` and `TF_HEAT_LOW` set (degrees Celsius), each prompt chunk waits while the hottest thermal zone is above the high band, until it is at or under the low one. The reply's tokens don't change, a cancel ends the wait, and the server logs `heat_wait_s`. Thanks to @Bizuayeu, whose GLM-5.3-Flash measurements on three Sparks (#339) set the design.
- Living Weights learns from Japanese and other text written without spaces, and holds a fact told in another language against prompts in that language, so it doesn't spill into other questions. Text with no usable facts now ends with a message saying so (#570). Thanks to @takanobuouchi-cell.
- `logprobs` and `top_logprobs` (0 to 20) on nonstreamed `/v1/chat/completions` for Nemotron on CUDA. Each reply token's row is its log probability under the model's raw distribution, read on the GPU from the verified target logits, so drafted, `"draft": false`, concurrent and resumed requests give the same rows. With thinking on, the rows cover the answer only, so `logprobs.content[0]` is the first answer token, as @DakotaTexas specified (#449). Streaming, tools, stop strings, structured output and a thinking budget stay refused with `logprobs`, and so do the Metal engines and the Responses and completions routes. On a GB10 every row matched a float64 log-softmax of the same logits within 2e-6, and a 256-token reply with 20 alternatives decoded 7% slower than without (#572). Thanks to @plotarmordev.
- Editing, deleting or regenerating a conversation's last turn resumes from a prompt state kept just before the latest user message, on every native engine with a prompt cache, and the reply equals the cache-off reply. The state is kept only when it fits beside the history's, and while its turn lasts it is the last one evicted. On an M3 Ultra, the Qwen3.8-27B's deleted, edited, regenerated and re-edited last turns of a 9,000-token chat took 1.7 s of prompt processing in all against 10.9 s, and on a GB10 replacing a pasted last message resumed at 8,192 tokens instead of 6,144 (#571). Thanks to @otarkhan.
- A two-Mac server whose other rank never answers says what to check, macOS Local Network access included: a server started detached over SSH can't ask for it, and macOS drops its packets. The speed-up guide, recipe and runbook say the same (#567). Thanks to @appljd for finding the cause.
- The CUDA recipe links a user's 1.0.4 receipt on a bare-metal RTX 3090 at 225 W and 350 W (#592). Thanks to @briepala.
- GLM-5.3-Flash runs on one 256 GB M5 Ultra. 1.0.4 refused it because the 4-bit checkpoint with its MTP head, 180.6 GB, is over the one-Mac limit of 70% of RAM. The load now leaves the head on disk when only the head is over, and says so; copy drafts still work. On one M5 Ultra the full model reads 2k, 8k, 16k, 32k and 64k-token prompts at 1,859, 1,990, 1,981, 1,964 and 1,925 tok/s, a served 26.6k-token prompt reaches its first token in 13.7 s, and it decodes prose at 78 tok/s and code edits at 165.
- GLM prompts run more of the model on the M5 tensor units. On two M5 Ultras they read 2.1k, 4k, 8k and 32k tokens at 2,090, 2,079, 2,513 and 2,497 tok/s, against 1,417, 1,559, 1,803 and 1,662 in 1.0.4, and a served 26.9k-token prompt reaches its first token in 10.9 s instead of 16.6. The prompt matmuls unpack 4-bit codes with one OR and sum each row in fp32, which also brings them closer to fp64: 1.1e-6 of the outputs' rms, from 3.8e-3. The routed experts' gate and up run as one gather over the token rows with SwiGLU at its end, and down writes each pair's output in place. KDA's gates and recurrence, the index scores and the sparse attention run on the tensor units, and a chunk's buffers share three regions by layer, 185 KB a row instead of 419, so one Mac takes 8,192-row chunks. A prompt row read in a chunk no longer matches the same row read in a 16-row window bit for bit, so `tf-glm-run` and the server give the same reply to a long prompt only when they cut it in the same places. Served replies still equal fresh ones, because the server cuts a resumed prompt where it cuts a fresh one, and drafted replies still equal plain ones.
- GLM decode on Macs runs attention and index scores on the tensor units, takes the shared expert inside the routed gathers on one Mac, commits plain rounds a round ahead, and sends a drafted round's keeps and drafts to the GPU while the host encodes the next window. On two M5 Ultras code edits decode at 262 tok/s, from 239, and plain decode at 93, from 91.
- GLM keeps lighter prompt states. A kept state holds the KDA states and convolution windows and leaves the MLA prefixes in its slot, valid until that slot writes below them. A 26.6k-token chat's state takes 141 MiB instead of 587, so the full model on one Mac keeps it, and the next turn prefills in 0.15 s instead of 13.8 s with the same reply. `--learn` still keeps whole states.
- GLM loads read the checkpoint around the page cache and wait while free memory is under 3% of RAM. A 160 GB load on a 256 GB M5 Ultra had filled the page cache, run out of free memory and restarted the Mac.
- `tf-moe-bench` and `tf-mm-bench` time routed-expert and prompt matmul kernels from `.metal` files against each other, with each kernel's bits against the first and the matmuls' error against fp64.
- Living Weights' leak check no longer counts an everyday word a reply says in passing as a leak: "If you'd like, I can help" used to undo a round that taught "I like blue." (#590). Thanks to @ZackSample for the report.
- The CUDA engine's quantized projections have one home, `cuda/qlinear.zig`: a weight view per format (MLX affine 4-bit, and #482's block-FP8) with one call for decode rows and one for prompt rows, which refuses a format the family didn't load. Nemotron's 4-bit projections run through it with the same kernels, launches and replies, and `tf-cuda-test fp8-lane` checks the FP8 rows through both calls (#587). Thanks to @jschmied.
- The Qwen3.8-27B loads on M5 Macs running macOS 27 again: under Metal 4.1 its prompt attention and the lane projection's register kernel failed to compile, so the engine stopped with `LibraryLoad` (#573, fixes #579). Thanks to @jkuepker and @ApaApatu.
- A request with a temperature to an engine that decodes greedily only (GLM-5.3-Flash, Flash Next) is refused with a 400 before it is admitted, instead of failing after it waited its turn; when the server itself samples by default, the message says to serve with `--temperature 0` (#582). Thanks to @BobClawblaw.
- The native reply's `tensorfold.seconds` is its decode time, the span `tokens_per_second` divides, so a long prompt no longer reads as a slow decode; `prefill_seconds` and `time_to_first_token` carry the prompt. The Python 0.6 server's `seconds` stays the whole request (#588). Thanks to @eleqtrizit.
- A completion request whose nested prompt ids (`"prompt": [[...]]`) reach 2^32 or more no longer crashes the server: those ids decode as unknown, as ids past the vocabulary already did (#594). Thanks to @chaog992.
- The repository checks out on Windows: eight server golden files were named after a URL with a query (`get-health?reset_peak=1`) or a `tool_choice` JSON value, and Windows refuses `?`, `"` and `:` in file names. Their names now carry those characters as `%XX`, and `zig build test` fails on any path Windows can't check out.
- GLM-5.3-Flash keeps prompt states for prompts under 4,096 tokens too, so requests that share a shorter system prompt, and short multi-turn chats, resume instead of reading the whole prompt again. On two M5 Ultras a second request sharing a 3,486-token system prompt resumes at 3,486 tokens, and every reply equals the cache-off reply (#584). Thanks to @BobClawblaw.
- `GLM_LOAD_LIMIT_GB` loads GLM-5.3-Flash past 70% of a Mac's RAM when you ask for it, up to the GPU's recommended working set, and the server logs the wired-load panic risk at startup whenever the limit passes 70%, which stays the default. A value that isn't a number above 0 stops the server, and the load check, stream admission and the prompt-cache budget read one limit (#498, #521, #580). It's in GB, where Python 0.6.x's `TENSORFOLD_MEMORY_LIMIT_GB` is in GiB. Thanks to @heitke and @BobClawblaw, and to @Gogo6969 and @tpischke for the measurements.
- `--learn` keeps a free-disk floor, `--learn-min-free-gib` (4 GiB by default). Least recently used learned states make room when that is enough; otherwise learning pauses with a log line and serving goes on, trying again after 1 to 60 seconds. A two-Mac GLM pair admits both halves of a learned state together, and learned files carry a checksum, so a damaged or older file is refused and learned again. Both Macs of a GLM pair need the same release (#508). Thanks to @wojo.
- `--learn` keeps what it learned through a crash in the middle of a save. A record the crash cut short at the end of the learned index is trimmed when the server starts, so the states learned after that restart are read back at the next one; before, every state written after a torn record was lost. A `.part` file left by an interrupted save is removed at startup instead of counting toward the `--learn-gib` cap, where it could push another model's learned states out early. Neither problem changed a reply; a lost state cost a cache miss. These were two of 1.0.3's known issues.
- Flash Next keeps the weight packs it builds at startup under `~/.cache/tensorfold/packs`, one folder per checkpoint, instead of in a `zig-pack` folder inside the checkpoint, so the model folder can be read-only. A build is published only once every pack in it is complete, so an interrupted start builds again instead of loading part of a pack. A complete `zig-pack` from an earlier release is still read and never rewritten. This was 1.0.3's third known issue.

Known issues:
- GLM-5.3-Flash on two M5 Ultras decodes prose about 4% slower than 1.0.4 with drafts: 106 tok/s against 111 on a 256-token story, and slower on five of six prose prompts measured. Plain decode is 0.6% faster, code is not slower, and prompts read 1.4 to 1.8x faster. 1.0.5 drafts more tokens a round on prose for about as many accepted.
- The Qwen3.8-27B drafts greedy requests only: a sampled request decodes plain, about 19 tok/s on an M5 Max or an M3 Ultra, as in 1.0.4.
- Qwen3.5-2B reads long prompts slowly: on an M5 Max 893 tok/s at 64k tokens and 218 at 256k, against 2,588 at 2k.

## 1.0.4

- `tensorfold serve --slide` refuses a model folder whose shards, index or config.json are links to data other files share, and says to serve a copy made with `cp -cRL` instead. A folder from `tensorfold pull` is one: its files link into the Hugging Face cache, so in 1.0.3 learning there would rewrite the cached original and every snapshot that shares it. A hard-linked copy is refused for the same reason; an APFS clone passes. [docs/living-weights.md](docs/living-weights.md) shows the copy.
- No other code changed from 1.0.3, and 1.0.3's known issues still apply.

## 1.0.3

- Qwen3.8-27B runs on the native engine. Pull the model and its drafter with `tensorfold pull TensorFold/Qwen3.8-27B-MLX-4bit` and `tensorfold pull z-lab/Qwen3.8-27B-DFlash2`, then serve with `--drafter z-lab/Qwen3.8-27B-DFlash2`. Every drafted reply equals the same request with `"draft": false`. On an M5 Max, 512-token replies decode at 64 to 77 tokens a second on prose and 151 on code, against 32 to 33 for stock mlx_lm 0.31.3. A resumed turn of a four-turn chat reaches its first token in 0.16 to 0.43 seconds, against 13 to 17 seconds fresh. The 27B serves one request at a time.
- Run the 27B with `--drafter`. Without it, it decodes plain at 18 to 19.5 tokens a second on an M5 Max, slower than mlx_lm.
- The 27B's prompt processing trails mlx_lm for now, and we are fixing it. An M3 Ultra reads prompts at about half of mlx_lm's speed from 8k to 30k tokens: 252 tokens a second at 8k against 441, and 184 against 404 at 30k. On an M5 Max, served cold prefill measured below mlx_lm too, at about 470 to 515 tokens a second for a 7k-token prompt.
- Sliding Weights (experimental): `tensorfold serve --slide` lets a Nemotron 3.5 Lightning server learn facts while it runs, from `POST /v1/slide/learn` or the omp `/learn` extension in `packaging/omp/sliding-weights.ts`. Give it a fact, a document or a URL. Each fact it keeps becomes a change to the model's output-projection weights, written into the checkpoint's safetensors shards in bf16, so any TensorFold server started from that folder answers with it, with or without `--slide`, on Metal or CUDA. Told "I like blue.", it answers "Do you remember what colour I like?", "What's my favourite colour?" and "What colour do I like?" with blue after a restart, and on a GB10 that loaded the copied folder, while nine other questions keep their answers. A fact is kept only if related questions it was never trained on keep their answers; otherwise it is taken back. On a five-fact test document, 5 of 11 recall questions were answered and 1 of 16 neighbouring questions leaked a fact.
- `--slide` rewrites files in the model folder and nothing undoes a learned fact, so point it at a copy. Learning needs the Nemotron 3.5 Lightning MLX 4-bit checkpoint on an Apple-silicon Mac with room for it (tested on an M5 and an M3 Ultra); NVIDIA GPUs refuse `--slide`, though the CUDA build serves a folder that learned on a Mac. A fact takes two to four minutes to learn. [docs/living-weights.md](docs/living-weights.md) walks through it.
- NVIDIA Ampere cards with compute capability 8.6 (the RTX 30 series, RTX A6000, A10 and A40) run Nemotron 3.5 Lightning on the native engine, from the new Linux x86_64 archive. The A100 (compute capability 8.0) is not supported yet. On an RTX 3090, 256-token greedy replies decode at 252 tokens a second drafted on prose and 313 on code (194 plain), 1.44 to 1.69x llama.cpp's faster mode on the same card, and prompts process at about 4,800 tokens a second at 2k and 4,200 at 16k. Every drafted reply equals the same request with `"draft": false`, and concurrent streams equal their solo runs. Several streams at once are slower in total than one stream for now (102 tokens a second at 2 streams, 128 at 4), and the CUDA server has no prompt cache yet.
- On a GB10 (DGX Spark), Nemotron's CUDA kernels are now all our own instead of captured Triton code. Replies stay exact, but greedy tokens can differ from 1.0.2's: where they first differ, the top two tokens were within a fraction of a logit, and the new kernels' error against a 64-bit reference is no larger.
- Draft depth is chosen from more robust timings on Metal and CUDA: each window width keeps its fastest of seven runs, outliers are timed again, and the draft step is sampled between widths, so a slow moment at startup no longer leaves an inflated cost in the cache. One RTX 3090 start had recorded 0.61 ms for a step that measures 0.34.
- `tensorfold pull` works against the current Hugging Face hub again. It asks for each file's own bytes, so small files the hub now compresses no longer fail with SizeMismatch. It reads the hub's bare-hex LFS hashes, so large files get their sha256 check and resume back.
- Large files download as 32 MiB pieces from one queue over up to 16 connections, and a restarted pull fetches only the pieces it lacks. In one run against the real hub, the 18.5 GB Nemotron checkpoint came down in 210 seconds; 1.0.2 pulled it over a single stream. Small files are stored and checked by their git blob sha1, as huggingface_hub stores them, and retried after a reset.
- `tensorfold pull` fetches DFlash2 drafter repositories for `serve --drafter`, and no longer reads a checkpoint's model type from freed memory.
- Nemotron on Metal times its window costs on the first start of a build and keeps them, so later starts skip the timing. On an M3 Ultra the server is ready in 1.1 seconds instead of 3.9. Replies are the same tokens as 1.0.2's, and decode stays within 3% of it.
- A GLM tool call to a declared tool whose name has spaces or other punctuation, such as `Get Classes List`, comes back as `tool_calls`. It used to come back as plain text in `content`, so the tool never ran.
- CLI output goes to stdout and stderr without overwriting each other, so `> log 2>&1` keeps both. A failed pull used to lose its error message this way.
- Flash Next refuses a checkpoint its kernels don't support, such as 4-bit or another group size, with a clear message, instead of serving wrong output.
- `GET /v1/models` reports each model's effective context as `context_length` and `max_model_len` (#483), and the read-only `/memory` route reports the process footprint, its peak and the applied prompt-cache plan (#484). The benchmark tools verify cache state and keep their evidence (#485), and [docs/memory-profiling.md](docs/memory-profiling.md) explains how to record memory profiles (#475). Thanks to @akol1.
- Safetensors checkpoints with FP8 tensors load (#517), and the direct-I/O test reads its reference through the page cache (#486). Thanks to @jschmied.
- A checkpoint keeps every shard it opened when a later allocation fails (#477), and the cluster refuses a header whose byte count or end offset overflows (#488). Thanks to @chaog992.
- A streamed tool reply no longer sends whitespace that the whole reply does not have (#489). Thanks to @Nipale-ai.
- Pull requests run the no-GPU Zig tests and a `zig fmt` check (#462, #469). Thanks to @AdrianBinDC.
- The startup line prints the Qwen3.5 family's 128-token prompt step as 128, not 0,128 (#526). Thanks to @BobClawblaw.
- Release archives carry no macOS metadata or builder identity, so GNU tar on Linux unpacks them without warnings.
- Thanks to @SvangenStudios and @plotarmordev, who found the pull problems first (#514, #549), @SvangenStudios for the archive warnings (#516), and @GustavBlack for the direct_io diagnosis (#536, #540).

Known issues, with fixes in testing:
- If a crash tears the last record of `--learn`'s index, the states learned after it are lost at the next restart.
- An interrupted `--learn` save can leave a `.part` file that is never removed. It counts toward the `--learn-gib` cap at startup, so states learned for other models or builds can be removed sooner than the cap requires.
- Flash Next writes its pack cache into the checkpoint's folder instead of under `~/.cache/tensorfold`.

## 1.0.2

- GLM-5.3-Flash serves several requests at once on its two Macs. Eight concurrent requests decode at 199 tokens a second in total, against 100 in 1.0.1, and each stream equals its solo run.
- GLM's two Macs split the attention heads, the dense layers and the vocab head as well as the experts. One stream decodes plain at 91 tokens a second, from 83. With drafts it decodes 114 to 131 tokens a second on prose and 138 to 148 on code, each reply equal to its plain one. A fresh 7,300-token prompt reaches its first token in 4.4 seconds instead of 6.5.
- GLM keeps prompt states between requests. A resumed prompt equals a fresh one token for token, and a 7,300-token conversation resumes in 0.18 seconds.
- A new conversation that starts with the same system prompt and tools resumes all of that shared prefix: its first token came in 0.06 seconds after a 4,700-token system prompt.
- `--learn` keeps those shared prefixes on disk. After a restart, the first new conversation on a known system prompt reached its first token in 0.64 seconds instead of 4.43. An upgrade keeps them when a startup check finds its prompt passes give the same bits. `--learn-gib` caps their disk, 32 GiB by default. GLM only for now.

## 1.0.1

- The shipped binary has `tensorfold models`, `tensorfold info MODEL` and `tensorfold pull REPO`. In 1.0.0 they were built into the development CLI only, so a Homebrew install didn't have them.

## 1.0.0

- TensorFold is a native Zig server. The shipped engine loads checkpoints, tokenizes requests, renders chat templates and serves HTTP without a Python engine. Python 0.6.6 remains on `python-0.6` and the `v0.6.6` tag.
- Four model families are qualified for this release: Nemotron 3.5 Lightning on Metal from M1 through M5 and on NVIDIA GB10 CUDA, greedy and sampled; Qwen3.8 Flash Next on M5 Ultra Metal; GLM-5.3-Flash across two M5 Ultra Macs; and Qwen3.5-2B on M5 Max Metal.
- Flash Next loads its checkpoint directly. Serving it no longer requires a Python kernel recording. The native loader builds the layouts it needs from the checkpoint and uses the embedded Metal sources.
- Native server interfaces: OpenAI chat/completion routes, Anthropic Messages and token counting, tools, streaming, tokenization, health, metrics and the optional dashboard. `capabilities --json` and `serve MODEL --help` describe the binary's supported options.
- Opt-in context compaction (`--compact-at`, `--compact-keep`, `--compact-memory`) compacts a conversation that would overflow instead of refusing it. Nemotron runs concurrent requests in shared rounds on Metal and CUDA.
- Metal runtime fixes: `--keep-warm` controls idle GPU keepalive, and a load-time probe selects a prebuilt packed-kernel library when the runtime compiler refuses that kernel on macOS 26.3.
- Native release archives target macOS arm64 and Linux aarch64 for the GB10, with the executable and bundled license notices. CUDA deployment still needs its qualified kernel assets and NVIDIA driver; paired Metal serving needs the documented transport setup.
- CUDA GB10 qualification is greedy. Sampled requests do not yet honor the seed, `top_k: 0` fails on that path, and health/metrics do not yet report CUDA device and pinned-memory usage. Those fixes are planned for 1.0.1.
- Qwen3.8-27B is excluded: drafted/plain equality passes, but its paired served decode and cold-prefill measurements remain slower than Python 0.6.6. Bonsai, Gemma 4, Qwen3.6 and DeepSeek-V4 remain ports for 1.0.x, subject to their model gates.
- Contributor credits: all 145 public contributors are listed in [RELEASE-NOTES-1.0.0.md](RELEASE-NOTES-1.0.0.md).

Earlier entries document the Python release line.

## 0.6.5 (3 Oct 2026)

- **Qwen3.6-35B-A3B drafts with its own MTP layer on Macs,** as on CUDA. Chains of up to four drafts are verified in
  the lane rounds, alone or with other streams, and each round's depth comes from the Mac's measured costs, with plain
  rounds where drafts do not pay. Replies equal `"draft": false`. On an M3 Ultra it decodes 1.5-2.1x `--no-drafts` on
  short prompts, thinking on or off, and 1.3x at a 28,400-token prompt with the same time to first token: 1.5-1.7x
  mlx-vlm 0.7.4 with its MTP drafter. DFlash v1 stays available with `--drafter z-lab/Qwen3.6-35B-A3B-DFlash`.
- **Nemotron on CUDA drafts as deep as its rows pay for.** The MTP head drafts up to 15 levels, one CUDA graph each,
  and a chain stops where the next verify row would cost more time than its draft is expected to save. Window and
  level costs are measured at startup on each GPU. Against 0.6.4's default this decodes about 6% faster on one DGX
  Spark and on two, and 7% faster on an RTX PRO 6000, with replies equal to `"draft": false`. `--mtp-confidence` sets
  a fixed floor instead.
- **Nemotron verify windows on M5 Macs read each routed expert once from two rows** (eight before). One stream decodes
  1-3% faster at three to five lanes, with the same tokens.
- **API keys on both servers:** `--api-key` (repeatable), `--api-key-file` (one key, or `label: key`, a line; reread on
  `SIGHUP`) or `TENSORFOLD_API_KEY`. Requests send `Authorization: Bearer` or `x-api-key`. `/health` stays open;
  `/metrics` needs a key unless `--metrics-open`.
- **Thinking stays the chat template's default, and the server says so:** one startup line names `--no-thinking`, and
  a reply that reaches `max_tokens` before it leaves its think block (empty `content`, all `reasoning_content`) gets a
  warning line.
- The test suite collects on Macs that have PyTorch but not Triton.

## 0.6.4 (3 Oct 2026)

- **Flash Next on two DGX Sparks serves concurrent requests.** `--parallel N` with two CUDA ranks runs every stream
  in shared lane rounds, and drafted replies still equal one-token decoding (#141, #180, #219, closes #123). The two
  ranks now share one communicator interface, so a faster transport can plug in without touching the engines.
  Thanks to @BHCC2025, @jschmied, @plotarmordev and @jayleaton.
- **Startup memory on Macs:** prompt chunks are probed smallest first, and a larger probe runs only when its worst
  case fits the memory budget (#271). Thanks to @boxabirds for the report.
- **`tensorfold plan`** prints the model and context budget before any weights load (#281), and both `/metrics`
  routes report the process memory footprint (#277). Thanks to @akol1.
- Flash Next on CUDA builds its extensions before loading weights (#202) and gathers prompt n-gram rows before
  waiting on earlier copies (#201). Thanks to @mcclanahanaman.
- Flash Next on CUDA rereads EXL3 n-gram tables after warm-up and pins their page runs (#254). Thanks to
  @grearjake-star.
- `--decode-share` takes effect on Flash Next CUDA: shared prompt rows are sized from completed stand-alone passes
  (#248, closes #230). Thanks to @simon-lin88.
- CUDA tree attention folds its partial sums in fp32 groups. Precision is unchanged, but long replies can differ
  from 0.6.3 in the last bits (#268). Thanks to @Arminova.
- Flash Next on CUDA refuses cards below sm_120 at startup, before loading weights, and its CUDA tests skip there
  (#261). Thanks to @mcclanahanaman for the report.
- `docs/recipes/README.md` lists the minimum card and Mac memory for each family (#139). Thanks to @tomByrer for
  asking.
- Benchmark helpers fail early on Python below 3.11 (#276). Thanks to @akol1.
- Tests collect without the optional MLX or terminal-interface dependencies (#288), and the docs say `--alias`
  works on the CUDA server too (#289). Thanks to @plotarmordev.

## 0.6.3 (2 Oct 2026)

- **Nemotron on M5 Macs: copied text verifies up to 64 tokens a round.** A lone stream's copy window grows from 16 to
  64 rows while each lands whole, a window attends in one call, and wide windows keep their Mamba states every 8th
  row. On an M5 Max, one stream editing code decodes about 1.4x faster (550 to 760 tok/s, and up to 1,000 on a cool
  machine), prose and code writing gain 1.6-3.3%, and drafted replies still equal one-token decoding.
- **Anthropic Messages API.** `/v1/messages` and `count_tokens` on the Mac and CUDA servers, with streaming, tools,
  thinking and cache usage (#223, closes #168). Thanks to @kky42.
- **Control Room.** `tensorfold service` runs a model as a launchd service, and `tensorfold tui` shows its decode and
  prefill speed and its connections live.
- **Flash Next on Macs before M5:** `TF_FLASH_DENSE=matrix` runs every dense projection on the matrix units, and the
  fused DeltaNet stack uses the matrix kernel by default (#149). Thanks to @gilby.
- **Vision:** `--vision-offload` keeps the CUDA image tower in host RAM between images (#187, closes #185); image
  parts are accepted inside tool results (#235); `--vision-image-tokens` lets many images share a larger budget
  (#239); Flash Next on CUDA takes video (#240). Thanks to @barelyworkingcode and @MiaAI-Lab.
- **CUDA:** `/v1/decisions` on Flash Next scores labels from the prompt's last logits, the questions filling in one
  pass (#232); `TENSORFOLD_PREFILL_ROWS` sets the prompt piece rows (#238); NVFP4 and FP8 prompt rows add a tile's K
  slices in one block, with the same bits (#242); `TENSORFOLD_MEMORY_RESERVE_GIB` moves the memory floor the startup
  keeps free, by default a tenth of the pool and at least 4 GiB as before (#165); Flash Next's chain kernel takes
  5-17% less time at 2 to 8 rows, with the same bits. Thanks to @Mirrdhyn, @MiaAI-Lab, @jschmied and @eleqtrizit.
- **Server:** chunked request bodies are decoded before JSON parsing (#244); `/tokenize` and `/detokenize` carry
  vLLM's fields (#237); `chat_template_kwargs.thinking` is heard as `enable_thinking`, and GLM-5.3 keeps earlier
  turns' reasoning (#236); streaming usage gets its own chunk when `stream_options.include_usage` asks for it (#216);
  malformed tool-call history renders safely (#233); `/metrics` times each request's decode (#269); `/health` carries
  the live decode and prefill speed. Thanks to @JordiPosthumus, @MiaAI-Lab, @salmanarshad321 and @sxuff.
- **Fixes:** an EXL3 layer with no bias no longer faults in the split-K reduction (#186); an 8-bit `lm_head` keeps its
  format in Qwen3.6's MTP draft head on CUDA (#270); a lone Flash Next stream's cache growth on CUDA stays inside the
  explicit memory budget; two ranks refuse to start with different prompt rows. Thanks to @barelyworkingcode and
  @BHCC2025.
- **Models moved to the TensorFold Hugging Face org.** `Vontra/<name>` ids redirect, and the tree now names
  `TensorFold/<name>`.
- **Contributing.** `CONTRIBUTING.md` says what a pull request needs to land and how it lands, and new pull requests
  open with a receipt template.

## 0.6.2 (2 Oct 2026)

- **Flash Next on Macs at 64k-128k.** On an M3 Ultra, one stream runs 1.2-3.4% faster at 64k and 3.9-5.5% at 128k,
  with the same tokens: a window's n-gram ids are hashed on the GPU, and the chain's first step is built while the
  GPU verifies.
- **27B with several streams on CUDA.** The GDN tree kernel takes 8-35% less time with the same bits. On an RTX PRO
  6000 at its 250 W limit, 4 and 8 streams of the NVFP4 27B decode 1.1-4.2% faster.
- **Fixes:** the config check accepts the FP8 n-gram table in NVIDIA's MIXED_PRECISION Flash Next export; GLM-5.3
  on CUDA counts its drafts in `/health`, `/metrics` and replies, and names a mixed-bit EXL3 checkpoint when it
  refuses one; a client that leaves is noticed past file descriptor 1023; the CUDA server prints a line a request, as
  the Mac server does; a failed snapshot write no longer leaves its partial file; Gemma 4's QKV kernel reserves its
  1024 threads for M1, M2 and macOS VMs, and a VM's GPU is no longer taken for an M5.

## 0.6.1 (1 Oct 2026)

- **NVFP4 checkpoints in their own math.** `nvidia/Qwen3.8-27B-NVFP4` runs the 4-bit activations its checkpoint
  names, as vLLM does; `--precision full` runs 16-bit activations against the same weights. On an RTX PRO 6000 at its
  250 W limit, one stream decodes 1.4-2.0x vLLM and prompts fill at 0.95-0.97x its speed.
- **Waiting prompts fill together on CUDA.** With `--parallel`, prompts that arrive together now share one prefill
  forward instead of filling one a round: on an RTX PRO 6000 at its 250 W limit, 8 streams of the 27B run 1.14-1.36x
  faster and the slowest first token comes in 0.05-0.10 s instead of 0.4-0.8 s (1.6 s to 0.15 s on a DGX Spark), with
  the same replies.
- **More 27B tokens with several streams on CUDA.** Streams plan on their measured round cost, and wider lane blocks
  on RTX PRO and RTX 50 cards add 4-8% at 8 streams, with the same tokens.
- **Flash Next on Macs at long context.** One stream runs up to 9.6% faster on code at 64k and 10.2% on chat at 128k
  on an M3 Ultra, with the same tokens.
- **`/v1/decisions`** scores a choice, a score or a yes/no from the next-token logits, on the shared prompt lanes.
- **Flash Next on CUDA:** image input, forks that resume from their shared prefix, shared system prompts copied
  instead of filled again, and short prompts admitted while a long one fills.
- **RTX cards without Docker:** pip alone installs and builds the CUDA kernels. Native Windows is in as an
  experimental host layer, not yet run on Windows hardware.
- **Fixes:** a refused request no longer breaks the next one on its connection; Gemma 4 thought blocks stay out of
  replies with thinking off; an unnamed reasoning effort goes to the nearest named level; mlx-lm 0.32 support.

## 0.6.0 (30 Sep 2026)

- **RTX 40 cards.** CUDA now runs on compute capability 8.9 (Ada). On one RTX 4090 the 27B serves with DFlash2 in a
  40,182-token window, exact: drafted replies equal serial ones, and a resume or resend gives a fresh run's reply.
  Prompts fill at 2.4-2.6k tok/s from 2k to 32k tokens, and replies decode at 64-86 tok/s. On one GPU the 27B's kept
  prompt states give way, oldest first, when a live reply needs the room, so admission counts one live window.
- **Prompts fill inside the decode rounds.** With `--parallel` on CUDA, Flash Next prefills a queued prompt in the
  same forward as the live replies instead of stopping them: on one DGX Spark, first tokens came 2.8-3.1x sooner than
  on 0.5.0. On Macs several prompts fill side by side, the fewest tokens left first: on an M3 Ultra, short requests
  queued behind a long prompt got their first token in a median 9.9 s instead of 113 s (the slowest 11.7 s, not 130).
- **Conversations resume on more engines.** An identical resend or the next thinking turn now resumes from the kept
  prompt state on Flash Next, Qwen3.6, Nemotron and GLM-5.3 on CUDA, as the 27B did, with a fresh run's reply: an
  18.7k-token Flash Next resend went from 8.3 s to 0.08 s. On Macs the prompt cache keeps each conversation's newest
  checkpoint and grows into memory the model leaves idle.
- **CUDA prompts run at bf16 by default.** Against an fp32 reference the 27B's prompt rows are 20x closer than with
  FP8 (KL 0.0031 against 0.0624). `--prefill-fp8` keeps 0.5.0's faster FP8 prompts for those who want them.
- **Tool calls for agents.** The CUDA server streams tool-call arguments as the model writes them (the longest
  silence in a long call fell from about 30 s to half a second), Python-spelled values like `False` and `None`
  decode to their schema types, Gemma 4's bare tool calls parse (#121), and a prompt past the context window gets
  OpenAI's `context_length_exceeded`, so clients compact instead of retrying.
- **More checkpoints.** GLM-5.3 on Macs reads 8-bit and Q8_0 GGUF checkpoints and takes images, and has an opt-in
  float32 activation mode (bf16 stays the default). On Macs, Flash Next loads oMLX's oQ checkpoints with scaled
  n-gram tables; on CUDA it reads consolidated EXL3 tables and block-scaled FP8 linears.
- **Faster.** Flash Next NVFP4 decodes 4.9-7.1% faster and fills prompts 15-16% faster at 2k-16k, bit-identical. On an
  M5 Ultra the 27B with DFlash2 on an oQ4e checkpoint went from 33 to 131-160 tok/s. Before M5, a lone stream's copy
  windows widen to 128 rows (an M3 Ultra edit ran 190 -> 233 tok/s), and Flash Next's concurrent rounds use the matrix
  units. GLM-5.3's DFlash2 reads only its sliding window: 8% faster at 52k, and 3.8 GiB lighter a rank on two Sparks.
- **Operations.** Prometheus `/metrics` on both servers, `TENSORFOLD_MEMORY_RESERVE_GIB` for the CUDA startup
  reserve, `--checkpoint-slots` for the 27B's concurrent decoder on CUDA, and an idle GLM rank no longer spins.
- **Apache-2.0.** TensorFold is licensed under the Apache License 2.0 from this release, which adds an explicit patent
  grant from contributors. Releases up to 0.5.0 stay MIT, and code written before 0.6.0 keeps its MIT notice in
  `LICENSES/MIT.txt`.
- **Community pull requests.**
  - Resumed resends and thinking turns on Flash Next (#124). Thanks to @benthecarman, and to @Arminova for the
    GB10 measurements and the third-resend test.
  - Streamed tool-call arguments on CUDA (#114), `--alias` on CUDA (#111) and Python-spelled tool parameters
    (#135). Thanks to @olexale, @philip-pentatonic and @outcastofmusic.
  - GLM-5.3 on Macs: image input (#101), 8-bit and Q8_0 GGUF checkpoints (#119), float32 activations (#118) and
    the backbone wiring notes (#120). Thanks to @mgoldwasser and @feni6.
  - GLM-5.3 on two Sparks: lighter prompt buffers, the EXL3 estimate, the MTP setting, the idle rank and DFlash2's
    ring (#128, #129, #131, #132, #134), visible-pool selection (#140), and the CUDA startup reserve setting (#133).
    Thanks to @MiaAI-Lab and @mikolaj92.
  - Flash Next: scaled n-gram tables (#148, reported in #142) and the M5 draft head fix (#147), consolidated EXL3
    tables (#145), block-scaled FP8 (#126) and the NVFP4 expert speedups (#102, #105). Thanks to @gilby,
    @cwschroeder, @shantanugoel, @jschmied and @tournierjc.
  - On M5 Macs the 27B fuses the projections of 4-bit group-32 checkpoints such as oQ4e too (#164): four streams went
    from 315-320 to 332-334 tok/s on an M5 Ultra, with the same bits. Thanks to @gilby.
  - `--checkpoint-slots` (#125), and the DFlash2 drafter's admitted size (#112, from @jkuepker's ROCm work). Thanks
    to @nood-co1 and @jkuepker.

## 0.5.0 (29 Sep 2026)

- **OpenAI's Responses API on both servers.** `/v1/responses` runs as a chat completion: every response has its chat
  completion's token SHA and prompt tokens, and a tool round trip gives the same reply through `previous_response_id`,
  the turn resent whole, or chat. A second turn resumes from the cache.
- **Long context on Sparks, the same bits as 0.3.6.3.** The 27B's attention reads each key chunk once for a round's
  rows, with 16-byte loads, and prompt attention runs in its own CUDA kernel. On one Spark (MLX 4-bit + DFlash2):
  decode at 128k 18.4 -> 38.9 tok/s, at 96k 23.6 -> 45.9; cold prefill at 128k 860 -> 1,173 tok/s and at 255k
  545 -> 809. On the same NVFP4 weights, prefill runs 1.16-1.27x vLLM at every depth from 32k to 255k.
- **The CUDA server reads requests as the Mac server does.** `reasoning_effort` and `thinking_budget`, usage with the
  cached prompt tokens in every reply (streamed ones too), typed tool arguments, `min_p` on both backends,
  `ignore_eos` on Flash Next and Nemotron, and `top_k` 0 drawing the whole nucleus on two GPUs as on one. Thinking
  uses the template's own default effort on both servers.
- **`priority: background` on CUDA.** A background request yields to foreground ones and replays from its prompt,
  so its reply still equals its solo run. Under `--parallel`, the 27B and Qwen3.6 prefill a background prompt 1,024
  rows a step (measured waits in docs/api.md).
- **Community pull requests.**
  - Faster CUDA startup and model loading (#82). Thanks to @pmeenan.
  - Qwen3.6-35B-A3B serves `--parallel N` on CUDA, each reply equal to its solo run (#84), and `response_format`
    JSON schemas on the 27B, exact under drafts (#80). Thanks to @philip-pentatonic.
  - `/health` publishes live token totals (#79). Thanks to @MiaAI-Lab.
  - The 27B's concurrent drafter keeps no stale context after a prefill step (#92). Thanks to @nood-co1.
  - Flash Next lists its sparse-attention blocks past 131,072 keys in tiles, the same lists (#93). Thanks to
    @MovieMaker93.

## 0.4.0 (29 Sep 2026)

- **oQ formats and MLX 8-bit at 4-bit speed on M1-M4.** 5-, 6- and 8-bit linears now run on the matrix units with the
  same exactness. Qwen3.8-27B at oQ4e has half the KL against bf16 of MLX's 4-bit for 6% more bytes. On an M3 Ultra
  it now verifies 8-row windows in 31.9 ms instead of 40.8, within 1-5% of the 4-bit at every width. Served with
  DFlash2 drafts, sampled code runs 14% faster, chat 5-7%, and greedy code is level. The 5- and 6-bit layers use the
  matrix chain's arithmetic now, so replies can differ from 0.3.x's by a token here and there, and still equal their
  own `"draft": false` replies.
- **More streams on Macs.** A stream holds memory for its next 2,048 tokens, not its whole reply. A shared round's
  working memory is charged to the streams that share it. On a 64 GB Mac the 27B serves 16 streams at 32k contexts,
  where 0.3.x served 4-9. When memory runs short, kept prompts go first, then the newest streams wait, then the
  newest ends with an error that names `--parallel`.
- **Prompts inside the promised window are admitted and kept (#95).**
  - A finished request's rollback rows are released before the next prompt is sized.
  - The startup line counts the probe round once: "0 streams of 8,192 tokens fit" became 13 with the reporter's
    flags.
  - The same flags give the same window on every start.

  Thanks to @benwilson.
- **Bonsai on Macs short of memory for the full widening** now widens as many layers as fit (60 of 64 at 25.2 GiB).
  Code runs 55 -> 78 tok/s on an M3 Ultra at the #94 reporter's budget. Thanks to @MESevenJourney.
- **Flash Next on M1-M4** runs 2-7% faster, drafted and serial, on an M3 Ultra.
  - The attention gate runs in the merge kernel, the PLE layer and router are fused, and chained drafts queue as
    they're built.
  - From two rows, the 4-bit dots skip the int-to-float convert (on M3 and M4 through half-precision nibbles), so
    windows of 3-16 rows cost 3-4% less with the same bits.
- **Flash Next's 2-8-bit and mixed checkpoints on Macs.** oQ4, oQ4e and oQ5e, and MLX 6- and 8-bit checkpoints,
  load and serve, each module in its checkpoint's own format. Before, a mixed checkpoint with a 4-bit base passed
  `tensorfold info` and then failed to load. 4-bit weights in groups of 32 keep their kernels. Other widths run new
  kernels whose rows keep their bits at any row count, so drafted replies still equal `"draft": false`.
- **GLM-5.3 oQ4 loads as downloaded.** Its config lists 46 `mlp_layer_types` for 45 layers (the MTP layer too), and
  transformers refused it, so the tokenizer didn't load. TensorFold now reads it as is.
- **A live line under `tensorfold serve` on Macs.** In a terminal, one line under the log shows open connections and
  decode and prefill tok/s, redrawn in place. Log files see nothing new, and `TENSORFOLD_NO_LIVE=1` turns it off.
- **MLX 0.32.3 (#88).**
  - TensorFold runs on MLX 0.32.2 and 0.32.3, and a fresh install gets 0.32.3. Output is the same bit for bit on
    both.
  - SSD expert streaming builds its extension with the nanobind each MLX was built with (the `ssd` extra brings it).
  - A prompt kernel that doesn't build prints a warning naming the MLX version and the fix, and
    `TF_REQUIRE_KERNELS=1` stops at startup instead.
- **DeepSeek-V4-Flash draft heads on Hugging Face.** `Vontra/DeepSeek-V4-Flash-DSpark-MLX` (the default once pulled)
  and `Vontra/DeepSeek-V4-Flash-MTP-MLX`, converted from DeepSeek's MIT releases. A head is a folder holding
  `model.safetensors` and a `config.json` that names it.
- **Fixes.**
  - Object and array tool arguments that the model leaves one closing bracket short are closed, and kept if they
    then match the schema (#87).
  - Qwen3.5 and Qwen3.6 chat templates get the same resume points as Qwen3.8, so a second turn reuses the first
    (#83). Thanks to @philip-pentatonic.
  - A prompt or image that can't fit even alone no longer evicts kept prompts first.
- **Correction to 0.3.6.3's notes.** Nemotron's chat decode on the M3 Ultra wasn't 2% slower. It spreads about
  ±2.5% between server starts, even at a fixed draft depth.

## 0.3.7 (29 Sep 2026)

- **Nemotron on Macs serves concurrent requests again.** 0.3.6.3 kept one draft slot for all of a server's streams,
  so several Nemotron requests at once could fail with HTTP 500 (5 of 6 in our test on an M3 Ultra). Each stream now
  keeps its own: six requests at once all finish, and each reply equals the same request sent alone.
- **Qwen3.8-27B on M1-M4 serves 9, 17 or 25 streams at once.** A round whose last group of recurrent streams held
  one stream built a kernel Metal refused, and every stream in that round failed. That group now builds like the
  others; rounds of 2-8 streams run the same kernels as before.
- Versions have three parts from here on.

## 0.3.6.3 (29 Sep 2026)

- **Images in, on Macs and Sparks.** `--vision` lets Qwen3.8-27B read up to four images a request, as data URLs or,
  with `--vision-urls`, HTTPS links. Image replies keep drafting and equal `"draft": false` ones (77.6 against 37.3
  tok/s serial on an M3 Ultra), and the first token comes as fast as mlx-vlm's. Thanks to @di37 for HTTPS-only
  fetching, bounded image preparation and redacted request logs (#64).
- **DeepSeek-V4-Flash on a 256 GB Mac.** The new `deepseek_v4` family serves `mlx-community/DeepSeek-V4-Flash-4bit`
  on the lane engine with DSpark or MTP drafts converted from DeepSeek's releases, and replies equal `"draft": false`
  ones. On an M3 Ultra it decodes 1.7-3.1x and reads prompts 1.8-2.0x as fast as mlx-lm PR #1797's server. Thanks to
  @jeffpeng3 (#14).
- **NVFP4 checkpoints on CUDA, as published.** Flash Next's NVFP4 exports (local-inference-lab's and RadixArk's)
  and NVIDIA's Qwen3.8-27B NVFP4 load through `tensorfold serve` and stay exact: drafted replies equal `"draft": false`
  ones, resumed prompts equal fresh ones, and 84 of 84 concurrent streams equal their solo runs. The experts read
  their routing on the GPU, so CUDA graphs replay any routing. On one Spark, Flash Next NVFP4 decodes 1.13-1.52x vLLM
  on the same checkpoint. Two ranks, `--ple-on-ssd` and images on NVFP4 checkpoints stop at startup with a message
  until they're qualified. Thanks to @tournierjc for the reader (#67).
- **Ternary Bonsai 2 27B** (prism-ml's 2-bit pack) serves on the lane engine with Qwen3.8-27B's DFlash2 drafts,
  exact. Against mlx_lm running the pack's own runtime on an M5 Max: decode 3.7-5.8x on code and 1.7-2.2x on chat,
  prompts 1.4-1.7x. Thanks to @gprot42 (#18).
- **CUDA server work from @nood-co1:** stop strings on every CUDA engine and `ignore_eos` on the 27B (#63);
  malformed requests get a 400, and failed ones a 500 or a stream error event (#61); a silent start explains itself
  with extension-build progress and stale-lock notes, and `kill -USR1 <pid>` prints every thread's stack, at start or
  while serving (#62); the 27B keeps its prompt cache entry one token before the prompt's end, so the next chat turn
  resumes from it (#65).
- **Replies keep decoding while long prompts prefill on Macs.** A prompt now fills one planned chunk at a time and
  running replies take rounds between its chunks, for `--decode-share` of each chunk's time (default 0.25, about a
  fifth of the time; 0 prefills whole prompts first, as before); every reply still equals its solo run. On an M3
  Ultra, a reply's longest pause while three 17K-token prompts arrive fell from 164 s to 7 s, and a request alone is
  unchanged. Thanks to @benwilson (#72).
- **Long conversations stay warm on Macs.** A checkpoint that can't fit no longer evicts the others first, a resumed
  turn no longer holds the stored prefix it copied, and the 27B's DFlash2 prompt taps take 64 KB a token instead of
  114. On a 64 GB budget (emulated on an M3 Ultra), one conversation grew to 143K tokens and each turn resumed in
  38-46 s, where 0.3.6.2 prefilled every turn past about 100K from the start. Startup names the longest request
  whose prompt is kept. Thanks to @sanjaibalajee (#74), and to @benwilson for the report and his
  64 GB measurements, now in the README labelled 0.3.5.1 (#71, #70).
- **Qwen3.6 on CUDA keeps serving past 8,192 rows** with expandable allocator segments: its graphs now capture into a
  new pool after the buffers grow, where every later request failed. Thanks to @philip-pentatonic (#78).
- **Typed tool arguments on CUDA:** a Qwen tool call's array, object, number and boolean parameters arrive as JSON
  values, as on the Mac. Thanks to @MiaAI-Lab (#75).
- **Flash Next reads a prompt chunk's n-gram rows on 16 threads,** same bytes, so prompt times vary less when the
  tables are not all in the page cache. Thanks to @MovieMaker93 (#73).
- **On M1-M4, the MoE prompt matmuls give each GPU tile one expert's rows,** the scheduling idea of MLX's gather_mm
  change (ml-explore/mlx#4567), written for 4-bit gathers. Same bits. On an M3 Ultra, Flash Next's prompts run 2-4%
  faster (1.02-1.20x oMLX's) and GLM-5.3's 3.4-3.8% faster (1.08-1.11x mlx-vlm's at 8k-32k).
- **Flash Next prompts on M1-M4:** the block scores run as one batched matmul, the GQA kernel scores two query heads
  a simdgroup and the top-512 selection finds its cuts with a simdgroup scan. Same bits, about 1% faster at 16k-64k;
  on an M3 Ultra it is level with oMLX at 32k and 64k.
- **MLX stays at 0.32.2 for now.** MLX 0.32.3 changed a kernel our prompt kernels build on, so fresh installs fell
  back to slower prompts with one log line; the next release follows the new kernel. Thanks to @ecohash-co (#88).

## 0.3.6.2 (28 Sep 2026)

- **EXL3 replies stop at the end of the turn.** Flash Next and 27B EXL3 packs list `<|im_end|>` only in
  `generation_config.json`, so replies ran past their turn and leaked tool calls and think tags. The CUDA engine now
  reads that file too. Thanks to @vcruz305 (#69).
- **pip installs serve 27B EXL3 packs.** The package was missing the 27B's CUDA sources; a test now checks that every
  CUDA source ships. Thanks to @taussoe (#66).
- **A quantized KV cache for Flash Next on CUDA.** `--kv-dtype int8` or `int4` holds about 1.7x or 2.6x the default
  window in the same memory, and drafted replies still equal serial ones. `--mtp-confidence` sets where MTP chains
  stop. Thanks to @vcruz305 (#47).
- **Flash Next on CUDA reaches the first token sooner.** The head runs on a prompt's final chunk only, and the prompt
  kernels load at startup, so the first 2k prompt takes 1.26 s instead of 1.79 s. Thanks to @MovieMaker93 (#40).
- **GLM on two Sparks holds 256k tokens** with a latent attention cache; its next-token loss is within 0.001 nats of
  the per-head cache. Thanks to @taussoe (#54).
- **Mixed-bit Qwen checkpoints** (4-bit with some 5- and 6-bit layers, such as oQ4) load on every lane backend,
  exact, with new row kernels for 2- to 8-bit weights. On an M3 Ultra, oQ4 27B decodes 114-120 tok/s on code and
  59-61 on chat, against mlx_lm's 34-36.
- **Concurrent 27B on CUDA:** `--parallel 16` serves 161.7 tok/s on one Spark in 25.4 GiB, each reply equal to its
  solo run (#38). **Qwen3.6-35B-A3B on CUDA,** exact: decode 1.36-1.49x vLLM with MTP, prompts 1.21-1.37x (#45).
- **Gemma 4 drafts (opt-in):** `--drafter z-lab/gemma-4-26B-A4B-it-DFlash` decodes 1.3-2.1x mlx_lm on an M3 Ultra,
  exact.
- **Tool calls:** `tool_choice: "required"` and a named tool are enforced on both servers (#52), and a complete tool
  call inside an unclosed think block comes back as a tool call (#60).
- **Conversations come back warm on Macs.** `--spill-gib N` writes a conversation pushed out of the prompt cache
  to disk, up to N GiB, and reads it back when the conversation returns. On a 48 GB budget a 35k-token
  conversation came back in 0.27 s on an M5 Max instead of 75 s, with the same reply. Off by default;
  `--checkpoint-slots` sets how many conversations stay in memory. Thanks to @gilby (#68, #55).
- **Memory:** `TENSORFOLD_MEMORY_LIMIT_GB` raises the budget above the default share, and Flash Next's memory check
  counts its host-mapped n-gram tables, so it starts on a 128 GB Mac. Thanks to @Chedrian07 (#49, #50).
- **CUDA server fixes from @nood-co1:** an abandoned request stops within a round (#57), keys and values stay within
  the admitted window (#58), and a failed admission no longer stops the scheduler (#59).
- **M1-M4:** a prompt split into parts attends exactly as it does in one piece at every length; two 8,192-key cases
  rounded differently in 0.3.6.

## 0.3.6.1 (28 Sep 2026)

- **CUDA builds inside NVIDIA's containers again.** Their `TORCH_CUDA_ARCH_LIST` names every architecture back to
  sm_80, so the kernels' thread-block clusters and FP8 MMA failed to compile for GPUs that lack them. Every extension
  now builds for the GPU that is present, and a GPU older than compute capability 9.0 gets a clear message. Thanks to
  @ss-cong for the report and the exact errors (#56).

## 0.3.6 (28 Sep 2026)

- **GLM-5.3-Flash on Macs** with 256 GB, drafted replies equal to serial ones. Prompts process at or above mlx-vlm
  from 2k to 32k tokens on an M3 Ultra, and tool calls parse in both servers. Thanks to @chadhurley25075-png (#9,
  #39) and @jeidbugs404 (#35).
- **Gemma 4 26B-A4B on the lanes,** exact at every width, with prompts at or above mlx_lm from 2k to 64k tokens on
  an M3 Ultra. Thanks to @cshintov (#10).
- **Bigger models on smaller Macs.** `--ple-on-ssd` reads Flash Next's n-gram tables from disk, so a 128 GB Mac holds
  it (#16). `--ssd-experts GIB` streams routed experts from the checkpoint into a GPU pool of that size, so Flash Next
  fits a 64 GB Mac and GLM a 128 GB one. Replies are the resident model's tokens; decode runs at 0.31-0.39x resident
  speed for Flash Next and 0.13-0.17x for GLM (measured on an M3 Ultra; `pip install "tensorfold[ssd]"` first) (#17).
- **EXL3 checkpoints on CUDA (experimental):** Qwen3.8-27B and Flash Next packs from turboderp, exact on the lanes.
  Decode runs 1.6-3.6x vLLM with MTP; prompt processing is about half the MLX checkpoints' speed for now, and the
  fix is next. Thanks to @vcruz305 (#42).
- **Faster prompts.** Flash Next sizes its prompt chunks to the memory it has, Nemotron takes up to 8,192 tokens a
  chunk on M5 GPUs, and the weights stay wired while a server runs.
- **Fixes:** GLM on two Sparks answered "!" past about 2,000 prompt tokens, and EXL3 GLM prompts past 128 tokens
  failed (#53). A reply that isn't a tool call comes back as content, not an HTTP 500 (#51). The server hands MLX's
  freed buffers back when it goes idle (@kingjamez, #44).
- **MLX 0.32.2 or newer** is required on Macs.
- **What's new after an update:** `tensorfold update` prints these notes when it finishes.
- **Known:** replies to prompts longer than one prompt chunk can differ between machines with different memory,
  because the chunk size follows the memory budget. Within one server, drafted replies always equal serial ones and
  resumed prompts equal fresh ones.

## 0.3.5.1 (28 Sep 2026)

- Qwen3.8-27B loads on M1 and M2 Macs again. Kernels there fit Metal's per-kernel thread limit, with the same sums in
  the same order, so drafted output still equals serial output.
- M3, M4 and M5 run 0.3.5's machine code unchanged.
- Thanks to @hichaiuse, @simonmd, @gcarusso, @tonydehnke, @Cyb3r-Monk and @tinyapps for the reports and the repro.

## 0.3.5 (27 Sep 2026)

- **Concurrent requests share each verification round.** `--parallel auto` is on by default, and every stream's reply
  equals the same request served alone, on Metal and on CUDA.
- **Follow-up turns resume at the start of their newest messages,** with output identical to a fresh prompt. A 12-turn
  agent session with the 27B spent 14.9 s on first tokens instead of 31.9 s.
- **Memory that fits.** The whole process stays inside 70% of RAM, an omitted `--context` defaults to the window the
  machine can hold, and a prompt past it gets a clear 400.
- **Flash Next prefill** with sparse prompt attention, 1.1-1.4x faster than 0.3.4.1 on an M3 Ultra (@quigles1977, #29).
- **2- to 8-bit weights** on the lanes, so mixed-precision 27B checkpoints decode fully (@jasontitus, #34).
- **CUDA:** FP8 prefill and shared expert kernels, concurrent streams for the 27B and Flash Next, and admission from
  available memory before loading.
- **API:** raw `/v1/completions` prompts, `ignore_eos` and `stop`, request `reasoning_effort` and typed tool arguments
  (@chris247474, #28), `developer` messages, `parallel_tool_calls: false`, and cancellation when a client disconnects.

## 0.3.4.1 (27 Sep 2026)

- Prompt processing is back to MLX's speed on every model. Prompts prefill through MLX's own forward on a fixed
  2,048-token grid, so a resumed conversation still equals a fresh one byte for byte.
- Flash Next's peak memory stays within 20 GB of its weights up to a 196k-token prompt.

## 0.3.4 (26 Sep 2026, pre-release)

- Every model runs on the lane engine, and the serial engine is gone. Nemotron drafts on M1 to M4 with row-exact
  kernels.
- Qwen3.8-27B on M1 to M4 decodes at 1.9 to 4x serial, through a new 4-bit matmul on the simdgroup matrix units.
- CUDA: every default path verifies at least two rows a round.

## 0.3.3 (26 Sep 2026)

- Qwen3.8-27B verifies drafted tokens together on every M1 to M5 GPU, with output byte-identical to serial decoding.
- Streamed `/v1/completions` send plain text.

## 0.3.2 (26 Sep 2026)

- `tensorfold update` installs the newest release.
- GLM-5.3-Flash reads Brandon M. Music's EXL3/TR3 weights (re-hosted by Mia-AiLab) on two DGX Sparks (experimental).

## 0.3.1 (26 Sep 2026)

- `tensorfold info` shows how a checkpoint stores its weights and which backends read them. `serve` and `pull` refuse
  checkpoints no engine reads yet, before anything downloads.

## 0.3.0 (26 Sep 2026)

- NVIDIA GPUs: `tensorfold serve` picks CUDA on Linux and runs on one GPU or two (one rank per DGX Spark).
- A new family, GLM-5.3-Flash, on two Sparks.

## 0.2.0 (25 Sep 2026)

- A rewrite: `tensorfold serve`, `pull`, `models` and `info` for Nemotron 3.5 Lightning, Qwen3.8-27B and Qwen3.8
  Flash Next on Apple Silicon, with drafts that never change the output.