# Hy4-flash Tencent's Hy4-flash (`tencent/Hy4-flash`). Runs with the same command as any model: `basalt serve PACK`. What it supports next to the others: [models](index.md). Rounding and quality: [precision](../precision/hy4-flash.md). ## Pack One file, `Hy4-flash-NVFP4.basalt` (158.7 GB), built from Tencent's BF16 checkpoint (Tencent Hy License 1.0) with the vision projector and the DFly2 drafter: - Routed experts: NVFP4, rounded once from BF16 (83.6 GB). - Attention, latent, shared-expert, dense-FFN, Engram and indexer projections and the token embedding: Q8_0 of the BF16 values. - Output head: BF16 in the file; the engine rounds it to Q8_0 as it loads. Norms, hyper-connections, router and biases: F32, as stored. - Engram table: E4M3 codes, one byte per value, rounded once from BF16 (63.4 GB). It stays in the file and is read by row. - Vision projector (BF16, 1.0 GB) and the drafter (BF16 in the file, 1.7 GB), whole. Download it from [Hugging Face](https://huggingface.co/jesdga/Hy4-flash-Basalt) (`hf download jesdga/Hy4-flash-Basalt Hy4-flash-NVFP4.basalt --local-dir models`), or build the same file yourself: [model file](../model-file.md#hy4-flash-nvfp4-pack). A test pack from AngelSlim's Q4_K_M GGUF converts too ([model file](../model-file.md#hy4-flash-test-pack)). ## Architecture The `hy4flash` architecture (code in `src/models/hy4flash`): 42 layers over four identity hyper-connection streams. - **Attention:** MLA over one shared latent. A position's cache row is 576 values (the normed 512-value latent and 64 roped values); attention has a learnable sink and a sigmoid output gate. 12 sparse layers (0, 1, 5, 9, ..., 41; 32 heads) keep every position's row, and an indexer picks the 2,048 positions each row attends: it pools one key per 4 positions and keeps the best 511 groups plus the current one. Up to 2,048 positions it keeps them all. 30 sliding-window layers (64 heads) attend the last 512 positions. - **MoE:** layer 0 is a dense FFN; layers 1 to 41 have 512 routed experts (top 16) and one shared expert. The routed experts run in a 1,536-wide latent: the layer projects its input down to it and the experts' sum back up, so an expert's three matrices are 1,536 x 1,536. - **Engram:** layers 1, 11, 21 and 31 add rows of a hashed n-gram table, 16 rows of 96 values per token and layer. ## Memory - **Experts:** the shared expert tier, placed as DeepSeek's ([memory](deepseek41.md#memory)): the main card holds copies of the most routed experts, each second card holds experts of its own, and system RAM holds the rest (the routed experts are 77.8 GiB), computed there by the CPU. The start line `expert homes of 512 per layer: ...` gives the split. The start places experts by the shipped routing profile `data/profiles/hy4flash-42x512.prof`, so the first request after a start finds the most routed ones resident. The early layers route more widely and get more of the main card's copies (layers 0 to 3 about 1.4 times the mean, the quietest 0.73), the same total. - **Copies give way by use:** an expert that earns a main card slot takes the one whose expert was looked up least lately (every lookup counts, up to 15; the counts halve every 256 tokens and start again with each prompt), the one used longest ago among equals. An expert that recurs every few tokens keeps its copy through runs of experts that come once, and the experts of an earlier topic do not hold the card through the next one. DeepSeek's slots give way by age alone. - **Cards:** the main card holds the dense weights (6.76 GiB with the Q8_0 head), the caches and the drafter (0.87 GiB). Up to three more cards (`--gpu 0,1,2,3`) each hold the experts of one block of layers; later cards idle. - **Waits:** while a decode layer's expert sum waits for the CPU's and the second card's rows, the main card's memory is idle once its own experts are done. It then reads the next layer's attention gate and output products (the head after the last layer) into its L2, up to 56 MB, and stops when the rows are in; those products then read L2 instead of memory. - **Engram table:** read by row from the pack, so it takes no RAM of its own; the OS file cache keeps the hot rows. - **Context:** `--max-context` from 1,024 up to 1,048,576, default 8,192. The 12 sparse layers keep a row of every position; the window layers' rings do not grow. - **Cache precision:** `--kv fp16` (default) keeps the rows in fp16; `--kv fp8` stores vLLM's 656-byte FP8 row ([precision](../precision/hy4-flash.md#attention-caches)). With the pooled indexer keys, a position takes: | | Per position | 128K | 256K | 1M | | --- | ---: | ---: | ---: | ---: | | `--kv fp16` | 14.6 KB | 1.8 GiB | 3.6 GiB | 14.3 GiB | | `--kv fp8` | 8.6 KB | 1.1 GiB | 2.1 GiB | 8.4 GiB | - **Elastic caches:** as Flash-Next's. With `--max-context` above 32,768 a session backs its first 32,768 positions at start; the start is refused only when those need more than half the memory the weights leave. A prompt or a generation that needs more positions takes them from the expert tier (its spare slots first, then the slots it lends), and a later prompt that needs fewer gives them back. A prompt whose positions would leave the tier below a prompt chunk's needs is refused with an error and the engine serves the next request. The start line `elastic KV: ...` and `/v1/status` (`context.elastic_kv_ceiling`) give the positions that fit. - **VRAM:** `--vram-reserve-mib` and `POST /v1/vram` set what stays free, as for DeepSeek. ## Long prompts - Chunks are up to 16,384 tokens, halved to fit memory. Inside a chunk the attention runs in steps of 2,048 rows. - A prompt of 1,024 tokens or more borrows only the tier slots its buffers and the expert ring need; the ring copies the next layer's experts to the card while a layer computes. The slots it leaves keep their experts, and the next turn finds them. A shorter prompt keeps the card's slots and puts its buffers beside them; the CPU takes RAM experts of 8 rows or fewer. - With a second card, each chunk's experts split between the cards: the second card computes its own and the RAM experts its link prices lower. - The indexer scores every pooled key for every row, so its cost grows with the context. At 1,015,207 tokens with `--kv fp8` the prompt read in 486 s (2,090 tok/s, both cards at their default power limits) and a needle at 50 % was found. Needles at 10, 50 and 90 % of a 253,410-token prompt were found with either cache format. ## Drafting On when the pack has the DFly2 drafter. One block pass drafts up to 5 tokens and states each one's chance of acceptance, and the draft policy picks the window by expected tokens per millisecond, as for DeepSeek (`include/basalt/spec/draft_policy.hpp`). The heads run one draft at a time, each draft's chance before its pass over the vocabulary's rows, and stop at the first draft no window could hold. `--spec N` caps the window (default 6: the last accepted token plus the whole block); `--spec 0` turns it off. The drafter is five dense MLA layers that read target layers 25, 29, 33, 37 and 41: each attends the positions that target layer's indexer selected, in that layer's cache, so its cost does not grow with the context. It keeps no cache of its own, and the target checks every draft: output is plain decoding's ([precision](../precision/hy4-flash.md#drafting)). Its projections are rounded as they load, the attention's gate and output and the MLP to Q4_0 and the rest to Q8_0. Its drafts pick from the target's own Q8_0 head through a 2-bit copy of every row (116 MB): the rows near the copy's best are scored exactly, as the target scores them. It works with either `--kv`. The start line: ``` basalt hy4flash: drafter: 5 layers reading target layers 25..41, 0.66 GiB on the GPU (Q8_0 and Q4_0 projections) ``` ## Parallel slots Not measured. `--parallel N` starts N sessions, each with its own caches, rings and drafter scratch, over the one expert tier; their windows run one after another, with no shared pass through the weights. ## Images `--vision dynamic` only ([vision](../running.md#vision---vision)). The encoder (a SigLIP2-derived ViT and its projector, 928 MiB) is staged once in pinned RAM; a request's pictures borrow room on the main card from the expert tier while they are encoded, and the tier takes it back. A picture is resized (Pillow's bicubic, as transformers' processor) to 4 to 4,096 merged tokens; one more than 200 times longer than wide is refused. Its rows take three-axis rotary positions, and the Engram rows of picture positions are masked. ## Chat, thinking and tools - **Template:** the pack's own Jinja template. Its special tokens carry a suffix (`:opensource` in Tencent's release, another in AngelSlim's), which the server reads from the template, so the stop token, thinking tags, tool tags and image tokens follow the pack. Token text inside a message stays text. - **Thinking:** `reasoning_effort` `none` renders `no_think` (an empty ``); `low`, `medium` and `high` render `high`; `max` renders `max`; a number is refused. `chat_template_kwargs.preserve_reasoning` is the template's `preserved_thinking`. A thinking budget closes the thinking with ``. - **Tools:** the model writes `NAMEKV...`. Each value is read by the request's schema for that parameter (text where a string is allowed, JSON otherwise); several calls per turn are supported. - **Sampling:** the pack's defaults are temperature 0.9, top_p 1.0 and no top_k; a request's fields win. ## Lossy flags Off by default. `--route-prior` and `--route-tail-skip` change output, and each is named in one start warning. What they do on every MoE model: [quality](../precision/quality.md#lossy-routing-flags); what they cost and gain here: [precision](../precision/hy4-flash.md#lossy-routing-flags). Without them, output still depends on which experts are resident, so a different tier size can change the text ([precision](../precision/hy4-flash.md#experts-and-placement)). ## Limits - No video. - No layer split: further cards hold experts only. - After a very long prompt the main card holds fewer expert slots, as the caches took their room: 75 of 129 per layer after 1,015,207 tokens with `--kv fp8`, against 114 to 122 after 253,410. Decode then computes more experts on the CPU. ## Speed Serve `bench.sh` (the second pass of a server), RTX 5090 at 400 W with an RTX 5060 Ti at 180 W, the published pack: | | Result | | --- | --- | | Decode: conversation, struct, prose | 92.2, 171.5, 83.8 tok/s | | Prompt read | 3,454 tok/s |