# Summary: 2026-08-14_15-25-32Z_DeaMoE_EfficientMoEStructureforFastSmall_BatchDeco.md
Saved: 2026-08-16 20:25
Source: 2026-08-14_15-25-32Z_DeaMoE_EfficientMoEStructureforFastSmall_BatchDeco.md
Original paper: [arXiv](http://arxiv.org/abs/2608.14385v1)
Model: None

---

## Summary  
Mixture‑of‑Experts (MoE) models are powerful but suffer from high latency when decoding with small batches because most expert weights must be loaded into memory each step, creating a bottleneck. DeaMoE addresses this by reorganizing experts into “departments” that share many parameters while retaining a few private ones, and by introducing a two‑stage routing scheme that eliminates redundant weight loading during inference. The result is a substantial reduction in per‑step loaded weights and a dramatic speedup for real‑time LLM decoding on GPUs such as A40 and H100.

## Key Contributions  
- [Finding 1] Grouping experts into departments reduces the number of unique parameters that need to be loaded per step.  
- [Finding 2] The two‑stage routing strategy avoids loading the same expert weights twice, cutting memory traffic by up to 50.9 %.  
- [Finding 3] DeaMoE achieves end‑to‑end TPOT speedups of 1.33× on a 7B model (A40) and up to 2.00×/1.97× peak speedup for DeepSeek‑V3 on A40/H100.

## Methodology  
The authors first analyze the weight‑loading pattern of vanilla MoE inference, identifying that each expert’s parameters are independently loaded every token. They then propose a departmental layout where experts from the same professional field share most weights, leaving only a small set of private parameters per expert. A custom two‑stage routing mechanism is designed: in stage 1, a lightweight router selects the appropriate department; in stage 2, within that department, the shared weights are reused across multiple experts without reloading them. This approach decouples decoding from the heavy pre‑training weight‑loading cost.

## Results  
Experimental evaluation on A40 and H100 GPUs shows that DeaMoE reduces per‑step loaded weights by up to 50.9 % compared with vanilla MoE. In microbenchmarks, the model runs at an average of 1.33× faster for a pre‑trained 7B model (A40) and reaches peak speedups of 2.00× on A40 and 1.97× on H100. These gains are achieved without sacrificing accuracy, confirming that the routing strategy is both memory‑efficient and computationally beneficial.

## Significance  
By decoupling expert loading from decoding, DeaMoE enables MoE models to meet ultra‑low latency requirements for interactive applications such as coding assistants and real‑time audio‑video interaction. The approach reduces GPU memory pressure and computational cost, making large‑scale MoE systems feasible in resource‑constrained environments while preserving high performance.

## Related Concepts  
- Mixture‑of‑Experts (MoE) architecture  
- Expert weight loading bottleneck  
- Departmental expert grouping  
- Two‑stage routing strategy  
- Per‑step memory footprint reduction