# Using HiSA ## The function and the module ```python from hisa import hisa_attention, HiSAAttention, rope out = hisa_attention(q, k, v, theta, max_angle_deg=90.0, impl="triton", rule_block=None, chunked=False) attn = HiSAAttention(theta, max_angle_deg=90.0, impl="flash") # same arguments; out = attn(q, k, v) ``` - `q` [B, H, T, 64] and `k`, `v` [B, H_kv, T, 64]: queries and keys **after RoPE**, causal attention. Grouped-query attention: query head h uses key/value head h // (H / H_kv). - `theta` [32]: the RoPE frequencies that set the windows, normally the model's own (`rope.inv_freq(64, base)`, or `model.model.rotary_emb.inv_freq` in Hugging Face). - `max_angle_deg`: the angle budget Φ. `math.inf` gives dense attention. - `impl`: `"reference"`, `"triton"` (exact rule), `"triton_block"` or `"flash"` (block rule); see the README table. - `rule_block`: `None` for the exact rule; the block size S for the block rule (`"flash"`: 128, the default). - `chunked`: `impl="flash"` only; `q` and `k` are already in the kernels' chunked feature layout (see below). `HiSAAttention` caches the integer thresholds, so it does no host–device synchronisation per call and works under `torch.compile`. Under autocast, q/k/v are cast to the autocast dtype (Hugging Face's RoPE returns fp32 q/k). Useful helpers in `hisa.rope`: `window(theta, angle)` (the window of each pair, Φ/θl), `reach(theta, angle)` (the distance beyond which keys leave the softmax), `thresholds(theta, angle)` (the integer windows the kernels use), `pair_masks(theta, angle, T, rule_block)` (which pair is kept for every query–key entry). ## Choosing the budget and the RoPE base The budget and the base fix every window: Δmax,l = Φ · b2l/d. In the paper (135M, 2K–4K context), every budget from 45° to 360° beats dense attention and 90° is the default: about 29% of the query–key products are computed at 4K. The reach (Φ · b(d−2)/d: 11.8K at b = 10⁴, 23K at b = 2·10⁴) should be at least the context you care about, since keys beyond it are not attended at all; raise the RoPE base for longer contexts (the paper uses b = 2·10⁴ for 16K). ## Training a Hugging Face Llama model from scratch ```python from transformers import AutoConfig, LlamaForCausalLM from hisa.integrations.hf import patch_llama config = AutoConfig.from_pretrained("HuggingFaceTB/SmolLM-135M") model = LlamaForCausalLM(config) patch_llama(model, max_angle_deg=90.0, impl="flash", chunked=True) # Hopper; or impl="triton" anywhere model = torch.compile(model.cuda()) # ... your usual training loop, bf16 autocast ... ``` `patch_llama` replaces every layer's attention in place (parameter names are unchanged) and sets `use_cache=False`: the patched model computes full causal sequences (training, evaluation). For decoding see below. Padding is not supported: a batch whose `attention_mask` masks any token raises a `ValueError` (the kernels are causal only). `model.hisa` records the rule; `HiSADecoder` reads it. `chunked=True` applies RoPE directly in the kernels' feature layout, which saves a permutation copy of q and k in every layer. It relabels the rows of the q/k projections, so use it **only from scratch** (a model trained with it must keep it), not with a pretrained Hugging Face checkpoint. ## Converting a pretrained dense model ```python model = LlamaForCausalLM.from_pretrained("HuggingFaceTB/SmolLM-135M", dtype=torch.bfloat16) patch_llama(model, max_angle_deg=90.0, impl="triton") # chunked=False: keep the checkpoint's layout # continue training ``` Switching the attention of a trained dense model without training costs some loss (2.906 → 2.937 for the paper's 4K model); continued training recovers it (the paper matches dense continued training after 1.35B tokens). ## Decoding with the ring-buffer KV cache (Flash-HiSA) For a model with the block rule (trained with `impl="flash"` or `impl="triton_block"` and `rule_block=128`): ```python from hisa.integrations.hf_decode import HiSADecoder model = model.to(torch.bfloat16).cuda().eval() dec = HiSADecoder(model, B=8, S_max=4096) # rule (angle, frequencies, chunked) from model.hisa tokens = dec.generate(prompt, max_new_tokens=256) # prompt: int64 [8, P], equal lengths losses, seconds = dec.teacher_forced(windows) # benchmark: CUDA graph, windows int64 [8, S + 1] print(dec.cache_bytes / 2**30, "GiB") ``` The cache stores each 16-feature chunk of the keys in its own ring buffer, as long as the window of the chunk's slowest pair (rounded up to 128-token blocks), and the values in a ring as long as the reach: at 4K context and b = 10⁴, 256 / 256 / 1408 / 4096 tokens for the four key chunks instead of 4096 each. `HiSADecoder` feeds prompts token by token and all sequences share the same position (no padding); it is a reference loop to build on, not a serving engine. The low-level cache is `hisa.kernels.flash_decode.HiSAKVCache` (`append`, `attend`). ## Supported settings and limits **Any angle budget and any RoPE base.** The rule reaches the kernels only as integer windows (and, for Flash-HiSA, a small table of dropped pairs per block distance) computed on the host, so every implementation accepts any Φ > 0 (including `math.inf`, dense attention) and any base, with no recompilation. The tests cover bases 30 to 10⁴ and budgets 45°, 90°, 360° and ∞; the paper's models use b = 10⁴ and 2·10⁴. | | `reference` | `triton`, `triton_block` | `flash` (training) | decoding (`HiSAKVCache`) | |---|---|---|---|---| | Hardware | any | NVIDIA GPU | sm90 (H100 / H200) | sm90 | | dtype | any | fp32, bf16 | bf16 (others are cast) | bf16 | | Head dimension | any even d | 64 | 64 | 64 | | Rule block S | any | `triton_block`: multiple of 32 | 128 | 128 | | Attention | causal | causal, GQA | causal, GQA | one new token per sequence, GQA | | Not supported | | padding masks, varlen | varlen, paged KV, softcap | paged KV; sequences at different positions; cache length not a multiple of 128 | **Speed depends on what is dropped.** The kernels skip work in chunks of 8 pairs (16 features). When the windows are long compared with the context (large budget or base), few chunks are dropped: the result is still exact, but the speed is close to dense attention. In the paper's setting (90°, b = 10⁴, 4K) about 29% of the query–key products remain (35% with the block rule). **Very small windows and the block rule.** The block rule keeps each pair up to S − 1 = 127 tokens beyond its exact window. When most windows are much shorter than 128 tokens (very small budgets or bases), it is a coarser approximation of the exact rule; it remains a valid model, as long as training and inference use the same rule.