# SNI Demux Kernel Handoff (tc-eBPF) Status: draft spec for implementation. Owner: @dr-bonez. ## Background The SNI demux (`shared-libs/crates/start-core/src/tunnel/forward/sni.rs`) lets multiple targets share one external TCP port, routed by the TLS ClientHello's SNI. Because SNI only exists after the TCP handshake, the daemon must accept the connection, sniff the hello, open the internal leg from the client's own source address (`crate::net::transparent::transparent_connect`, IP_TRANSPARENT - the fwmark-`0x540001`/table-5344 reply divert), replay the buffered bytes, and then splice both sockets in userspace (`copy_bidirectional`) for the connection's entire lifetime. The original design intent was to use userspace only to _identify_ the flow, then hand the established connection to the kernel. That handoff cannot be assembled from conntrack/nftables alone: NAT binds on a flow's first packet (the SYN), an established local socket cannot be converted into a forwarded flow, and nftables exposes no TCP sequence translation (synproxy does this internally but is fixed-function, bound at SYN time). This spec describes the design that _does_ achieve it: after the demux decision, record the constant seq/ack offsets between the two legs in a BPF map, attach tc(clsact) programs that rewrite and forward packets between the legs entirely in-kernel, and silently retire both sockets with `TCP_REPAIR`. The daemon leaves the datapath; the connection becomes a flow-map entry with an idle timeout, which also structurally eliminates the fd-exhaustion failure class (see PR #3449 for the interim userspace fixes this supersedes for handed-off flows). ## Goals - Zero userspace bytes and zero held fds for a demuxed connection after handoff; CPU/latency comparable to plain kernel DNAT forwards. - Source-address preservation semantics identical to today: the backend sees the real client `(ip, port)`; the client sees the tunnel's external `(ip, port)`. - The existing userspace splice remains as the universal fallback — per flow (handoff failed/ineligible) and per host (kernel lacks support, feature disabled). - Flows survive a `start-tunneld` restart once handed off (pinned maps/progs). ## Non-goals - QUIC/UDP demux (out of scope, as in the current demux). - IPv6 (the demux is IPv4-only today; keep the flow key layout extensible). - Handoff of vhost passthrough connections on StartOS (`net/vhost.rs` uses the same transparent-connect mechanism and could adopt this later; target the tunnel first). - MSS renegotiation: MSS was fixed per leg at SYN time; rely on PMTUD (see Datapath). ## Architecture overview Per demuxed connection, three phases: 1. **Userspace (unchanged):** accept, read ClientHello, select target, `transparent_connect`, replay buffered bytes. 2. **Handoff (new):** quiesce both sockets, measure their exact stream positions and TCP parameters, install translation entries in a BPF flow map, retire both sockets via repair-mode close. Bounded attempts; on failure the flow simply stays in phase 1 forever (current behavior). 3. **Kernel (new):** tc ingress programs on the WAN interface(s) and the wg interface translate and forward packets between the two legs. Userspace only runs a GC sweep. Naming for the rest of this doc: - **Leg A** — client `C` ↔ tunnel external `(T, P_ext)`; the accepted socket. - **Leg B** — spoofed `C` ↔ backend `(D, P_d)`; the transparent socket. Note both legs share the same client `(ip, port)` — the spoofing is what makes the translation a pure header/seq rewrite with no real NAT table. ## Handoff protocol (userspace control plane) All steps operate on the two connected sockets. `CAP_NET_ADMIN` is required (`start-tunneld` runs as root). 1. **Eligibility gate.** Attempt handoff right after the ClientHello replay (`upstream.write_all(&buf)` in `handle_conn`) — the natural quiescent gap while the backend computes its ServerHello. Skip if the feature is disabled or the host failed capability detection at startup. 2. **Stop application I/O** on both sockets (do not read or write further). 3. **Drain send queues:** poll `ioctl(SIOCOUTQ)` on both sockets until 0 (all sent bytes ACKed — unACKed bytes would be lost at retirement because nothing will retransmit them). Peer ACKs must still flow, so ingress is NOT blocked yet. Deadline: ~500 ms. 4. **Block the flow at tc:** insert both direction entries into the flow map in state `BLOCK` (matching packets are dropped at tc ingress, which runs before socket delivery — including before the `socket transparent 1` divert for leg-B replies). TCP retransmission absorbs the drop window. 5. **Check receive queues:** if `ioctl(SIOCINQ)` != 0 on either socket, bytes arrived (and were ACKed by the kernel) between steps 3 and 4. They must be forwarded by userspace: delete the map entries (unblock), read + write those bytes through, and go to 3. Bounded retries (e.g. 3); on exhaustion, unblock and stay in userspace permanently for this flow. 6. **Measure (repair mode):** with both queues empty, set `TCP_REPAIR = 1` on both sockets, then read: - `TCP_QUEUE_SEQ` with `TCP_REPAIR_QUEUE = TCP_RECV_QUEUE` and `TCP_SEND_QUEUE` on each socket → `a_rcv_nxt`, `a_snd_nxt`, `b_rcv_nxt`, `b_snd_nxt` (exact next-byte sequence numbers; queues are empty so these align across legs). - `TCP_INFO` on each socket → negotiated window-scale shifts for both directions of both legs (see Window translation for orientation). 7. **Install translation:** update both map entries with the computed offsets (below), state `XLATE`. 8. **Retire the sockets:** `close()` both fds while still in repair mode — the kernel destroys the sockets silently (no FIN, no RST). The conntrack entries for both legs remain and are kept fresh by the forwarded traffic (see Datapath). Both fds are freed; the daemon is out of the datapath. Any error at any step: remove map entries (if inserted), take the sockets out of repair mode, and resume the userspace splice. Handoff must be strictly best-effort. ## Translation math All arithmetic is `u32` modulo 2³². At measurement time define: ``` OFF_CD = b_snd_nxt - a_rcv_nxt // client→backend stream shift OFF_DC = a_snd_nxt - b_rcv_nxt // backend→client stream shift ``` Because the queues were drained, every byte ACKed on one leg was forwarded on the other, so both legs carry the identical byte stream at a constant shift. The offsets are therefore valid for _all_ sequence numbers, including retransmissions of pre-handoff data in either direction (they arrive, get translated, and are handled as duplicates by the peer stacks). **Direction client→backend** (packet arrives on a WAN iface, key `{C.ip, C.port, T.ip, P_ext}`): ``` ip.daddr := D.ip tcp.dport := P_d tcp.seq := seq + OFF_CD tcp.ack := ack - OFF_DC SACK block edges (option kind 5): edge - OFF_DC // same space as ack window := rescale (see below) ``` then recompute checksums incrementally (`bpf_l3_csum_replace` / `bpf_l4_csum_replace`, pseudo-header flag for the address) and return `TC_ACT_OK` — the packet continues into the stack and is **forwarded normally** (routing, fragmentation/PMTUD, neighbor resolution all handled by the kernel; see Datapath for why this is safe and required). **Direction backend→client** (packet arrives on the wg iface, key `{D.ip, P_d, C.ip, C.port}`): ``` ip.saddr := T.ip tcp.sport := P_ext tcp.seq := seq + OFF_DC tcp.ack := ack - OFF_CD SACK block edges: edge - OFF_CD window := rescale ``` then `bpf_fib_lookup` toward `C.ip`, write the returned MACs, and `bpf_redirect` to the WAN interface. This direction must NOT go through the stack: after the source rewrite the packet carries the host's own address as source, and `fib_validate_source` drops locally-sourced packets arriving on a non-loopback interface. Redirect bypasses that. MTU is safe in this direction (wg MTU < WAN MTU, so nothing arriving on wg can exceed the WAN egress MTU). **Non-matching packets** (map miss): return `TC_ACT_UNSPEC` untouched. The program must be a strict no-op for all other traffic. ### Window translation The raw window field is interpreted by the receiver using the scale shift negotiated on _its_ leg. The shifts can differ between legs (the client and the demux's sockets negotiate independently), so translate: ``` win' = clamp( (win << ws_sender_leg) >> ws_receiver_leg, 0, 0xFFFF ) ``` where `ws_sender_leg` is the shift the packet's window is currently encoded with and `ws_receiver_leg` is the shift the new recipient will apply. Obtain all four shifts from `TCP_INFO` (`tcpi_snd_wscale` / `tcpi_rcv_wscale`) on both sockets during step 6. **Implementation note:** verify the orientation of `snd_wscale`/`rcv_wscale` (which side advertised which) with a targeted unit probe before trusting it; the requirement is: the effective (unscaled) window a peer computes after handoff must equal the effective window the sender intended. Store the two per-direction net shifts (signed) in the map. Right-shifting when the net shift is negative loses precision toward a smaller window, which is the safe direction. ### Timestamps (PAWS) — phased Each leg negotiated TCP timestamps independently, with different TSval clocks. Passing a client TSval through to the backend can violate PAWS monotonicity (the backend already saw the demux socket's TSvals on leg B). - **Phase 1 (required for v1 correctness): disable timestamp negotiation on the demux's sockets** so handed-off flows carry no TS option at all. There is no per-socket knob for the option in the SYN, so set `net.ipv4.tcp_timestamps = 0` host-wide on the tunnel appliance (shipped as sysctl config with the `.deb`, documented as a tradeoff — the tunnel is a dedicated appliance; its own SSH/API connections lose PAWS/RTT precision, which is acceptable at its rates). SACK must still be translated (do not disable SACK; it matters for loss recovery on lossy client paths). - **Phase 2 (restores timestamps): translate TSval/TSecr in the BPF program** with per-direction constant offsets chosen at handoff so that the first post-handoff TSval in each direction continues monotonically from the last pre-handoff TSval sent on that leg (record last-sent TSvals during the freeze window; `TCP_TIMESTAMP` getsockopt in repair mode yields each socket's offset from the host clock). TSecr is translated with the inverse offset of the opposite direction. Both fields live in the same bounded option-parsing loop as SACK. ## eBPF program & maps - Hook: `clsact` ingress on every WAN interface that carries demux listeners (from the gateway/ip-info the daemon already tracks) and on `wg-start-tunnel`. One program, direction determined by map lookup. - Option parsing: single bounded loop over the TCP options (≤ 40 bytes), handling kind 5 (SACK) and kind 8 (timestamps, phase 2). Everything else is passed through untouched. - Map `sni_flows`: `BPF_MAP_TYPE_LRU_HASH`, key `{ saddr: u32, daddr: u32, sport: u16, dport: u16 }` (program checks `ip->protocol == TCP` before lookup), value: ``` state u8 // BLOCK | XLATE win_shift i8 // net window shift for this direction nat_addr u32 // replacement address (dst for c→d, src for d→c) nat_port u16 // replacement port off_seq u32 // added to seq off_ack u32 // subtracted from ack / SACK edges redirect_if u32 // 0 = TC_ACT_OK into stack; else bpf_redirect target last_seen u64 // bpf_ktime_get_ns, updated per packet fin_rst u8 // bit 0: FIN seen this dir; bit 1: RST seen ts_off_* // phase 2 ``` Two entries per flow (one per direction). Size the map for ~64k entries (32k flows); LRU eviction is the backstop against GC bugs — an evicted live flow degrades to both peers timing out, never to misrouting. - Pinning: pin the maps and links under `/sys/fs/bpf/start-tunnel/` so handed-off flows survive daemon restarts. On startup the daemon adopts existing pins (verifying program version/struct layout via a versioned pin name; on mismatch, tear down and re-create — in-flight handed-off flows die with a one-time warning). ## Datapath integration (verified against the current prod ruleset) - **tc ingress runs before netfilter prerouting**, so conntrack only ever sees the rewritten packets: - client→backend rewritten packets carry leg B's exact origin tuple (`C → D:P_d`, source-spoofed leg) and are associated with leg B's existing conntrack entry as ESTABLISHED — the `forward` chain's `ct state established,related accept` admits them through the default-drop policy. No new nft rules are needed. - backend→client packets bypass the stack entirely (`bpf_redirect`), which leaves leg A's conntrack entry seeing only the origin direction; the entry is already assured/established, so this is harmless. Both entries expire naturally after the flow ends. - **Reply divert:** while the sockets are alive, leg-B replies are diverted into the transparent socket by the `socket transparent 1` mangle rule. The `BLOCK` state must therefore be installed at tc (earlier in the path) before queue-drain checks, and after retirement the socket-match simply no longer fires. No changes to `ensure_divert_infra`. - **Masquerade/postrouting:** the existing rules (`ip saddr 10.59.88.0/24 oifname eth* masquerade`, per-forward hairpin rules) do not match either rewritten tuple. Confirm in the netns test. - **PMTUD:** client→backend segments can exceed the wg MTU (leg A negotiated its MSS against the WAN MTU). Because this direction traverses the normal forwarding path, the kernel emits ICMP fragmentation-needed for DF packets and the client's PMTUD converges, same as for plain DNAT forwards (the existing `wg-mss-clamp` rule only touches SYNs and does not help post-handshake). Verify ICMP frag-needed egress is not blocked by nft. - **rp_filter:** client→backend rewritten packets (saddr = client, arriving WAN, routed to wg) pass strict RPF. The reply direction never hits `fib_validate_source` (redirect). No sysctl loosening required. ## Flow lifecycle & GC - The BPF program records FIN (per direction) and RST (either direction) in `fin_rst` and keeps stamping `last_seen`. - A daemon GC task sweeps the map every ~30 s and deletes entry pairs where: RST seen, or FIN seen in both directions, with `last_seen` older than 60 s (2×MSL-ish linger for retransmitted FINs/ACKs); or `last_seen` older than a hard idle timeout (default 4 h — post-handoff there are no keepalives from the host, so pick generously; make it configurable). - On daemon startup, adopt pinned maps and run an immediate sweep. ## Failure modes | Failure | Behavior | | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | Kernel lacks clsact/BPF features, BPF load fails | detected once at startup; feature off, log once; all flows userspace | | Quiesce deadline exceeded / retries exhausted | flow stays on the userspace splice (with PR #3449's keepalive protections) | | Map full (LRU) | eviction kills oldest flow (both peers RTO out); GC sizing makes this rare; count evictions | | Daemon crash mid-handoff (BLOCK installed, sockets alive) | sockets die with the process (kernel RSTs on close); stale BLOCK entries swept by GC on restart | | Program/map version mismatch on restart | recreate; in-flight handed-off flows are lost (logged) | ## Code layout & build - New module `shared-libs/crates/start-core/src/tunnel/forward/handoff/`: `mod.rs` (control plane: quiesce/measure/install/retire + GC task), `bpf.rs` (load/attach/pin management), called from `handle_conn` in `sni.rs` behind the feature gate. - **Everything `#[cfg(target_os = "linux")]`** with no-op stubs otherwise — CI builds `*-apple-darwin` (see `projects/start-tunnel/AGENTS.md`); follow the `transparent.rs` pattern. - BPF toolchain: recommend **aya** (pure-Rust userspace loader; no libc/libbpf C dependency, which matters for the musl riscv64/aarch64 cross-builds) with the BPF program in C compiled by clang in `build.rs`, or aya-ebpf if the team accepts the bpf-linker/nightly build dependency. BPF bytecode is target-independent, so one embedded object serves all host architectures. Decision point for the implementer + maintainer; whichever is chosen must be added to the CI build image and `projects/start-tunnel/AGENTS.md`. - Config: a tunnel db setting (`/settings/kernel-handoff`, default **off** for the first release; flip after soak on tunnel-chad), surfaced in CLI only (`start-tunnel settings ...`) — i18n for any user-facing strings, all 5 locales. - Runtime deps: none new at runtime (no iproute2/tc shellouts — attach via netlink from the loader library). - `LimitNOFILE`/keepalive fixes from PR #3449 remain: they protect phase-1 (pre-handoff) connections and permanent-userspace flows. ## Testing Netns integration harness (root; follow the precedent noted in `transparent.rs` — an `#[ignore]`d root-gated test or a script under `build/`), with a client ns ↔ tunnel ns ↔ backend ns over veth + a real wireguard link: 1. Bulk transfer integrity both directions across a handoff (checksummed payloads, ≥ 100 MB). 2. Handoff under packet loss (netem 2 % both legs): verifies SACK translation and pre/post-handoff retransmit correctness. 3. Differing window scales between legs (force via `TCP_WINDOW_CLAMP` / small `rmem` on one side): throughput sanity, no stalls. 4. PMTUD: WAN MTU 1500, wg MTU 1380; client sends full-size segments post-handoff; assert ICMP frag-needed and convergence. 5. Teardown: FIN-initiated by each side, RST, and idle-GC; assert map entries removed and conntrack entries expire. 6. Fd accounting: N handed-off flows → daemon fd count flat; kill -9 the daemon → flows keep passing traffic (pinned maps); restart adopts and GC works. 7. Quiesce-failure fallback: continuous-stream flow (never drains) stays userspace and still works. 8. Phase 2 only: timestamps on both legs, PAWS not violated under clock-offset skew between client and host. ## Success criteria - Steady-state on tunnel-chad: 0 daemon fds held for handed-off flows; demux CPU ≈ 0 under load; no regression in SNI routing behavior; fallback rate (< a few % of flows, logged) and LRU evictions ≈ 0. ## Open questions 1. Adopt for `net/vhost.rs` passthrough on StartOS after tunnel soak? 2. Phase 2 timestamp translation vs. leaving `tcp_timestamps=0` permanently on the appliance — decide after measuring real-world impact. 3. Handoff retry for long-lived flows that failed initial quiesce (opportunistic re-attempt on idle detection) — v2 candidate.