meta: title: "Qwen3.8-Flash-Next" slug: "qwen3.8-flash-next" provider: "Qwen" description: "Qwen4 architecture preview with a 125B-parameter main model, supplemented by an additional 51B N-gram embeddings, with 6B parameters activated per token." date_updated: 2026-08-26 difficulty: advanced tasks: - multimodal - text performance_headline: "Qwen4 architecture preview with 6B active parameters and efficient 262K context" related_recipes: - "Qwen/Qwen3-Next-80B-A3B-Instruct" hardware: h200: verified gb200: verified gb300: verified mi355x: verified rtx_pro_6000_4x: verified dgx_station_gb300: verified model: model_id: "Qwen/Qwen3.8-Flash-Next" min_vllm_version: "0.28.0" docker_image: nvidia: "vllm/vllm-openai:qwen38-flash-next" amd: "vllm/vllm-openai-rocm:qwen38-flash-next" nightly_required: true install: docker: note: "Use the dedicated Qwen3.8 Flash-Next image; PyPI installation is not supported for this recipe." pip: false architecture: moe parameter_count: "176B" active_parameters: "6B" context_length: 262144 base_args: - "--max-num-seqs" - "256" - "--gpu-memory-utilization" - "0.90" - "--enable-prefix-caching" - "--no-enable-flashinfer-autotune" base_env: {} dependencies: [] features: tool_calling: description: "Enable automatic tool choice with the Qwen3 XML parser" args: - "--enable-auto-tool-choice" - "--tool-call-parser" - "qwen3_xml" reasoning: description: "Extract thinking output with the Qwen3 reasoning parser" args: - "--reasoning-parser" - "qwen3" spec_decoding: description: "Use the checkpoint's Multi-Token Prediction module for faster decoding" args: - "--speculative-config" - '{"method":"mtp","num_speculative_tokens":3}' text_only: description: "Skip the vision encoder for text-only workloads and leave more memory for KV cache" args: - "--language-model-only" opt_in_features: - spec_decoding - text_only variants: default: model_id: "Inferact/Qwen3.8-Flash-Next-NVFP4" precision: nvfp4 vram_minimum_gb: 130 description: "Inferact quantized NVFP4 weights with per-layer embeddings in nvfp4 too" hardware_overrides: dgx_station_gb300: extra_args: - "--max-num-seqs" - "32" - "--gpu-memory-utilization" - "0.95" - "--max-num-batched-tokens" - "8192" - "--enable-prefix-caching" - "--distributed-executor-backend" - "mp" - "-cc.mode" - "none" - "-cc.cudagraph_mode" - "full_decode_only" rtx_pro_6000_4x: extra_args: - "--max-num-seqs" - "16" - "--gpu-memory-utilization" - "0.95" - "--max-num-batched-tokens" - "8192" - "--enable-prefix-caching" - "--moe-backend" - "marlin" fp8: model_id: "Qwen/Qwen3.8-Flash-Next-FP8" precision: fp8 vram_minimum_gb: 250 description: "Official dynamic FP8 checkpoint with blockwise weight quantization" hardware_overrides: dgx_station_gb300: extra_args: - "--max-num-seqs" - "32" - "--gpu-memory-utilization" - "0.95" - "--max-num-batched-tokens" - "8192" - "--enable-prefix-caching" - "--distributed-executor-backend" - "mp" - "-cc.mode" - "none" - "-cc.cudagraph_mode" - "full_decode_only" rtx_pro_6000_4x: extra_args: - "--max-num-seqs" - "16" - "--gpu-memory-utilization" - "0.95" - "--max-num-batched-tokens" - "8192" - "--enable-prefix-caching" bf16: model_id: "Qwen/Qwen3.8-Flash-Next" precision: bf16 vram_minimum_gb: 423 description: "Official BF16 checkpoint" compatible_strategies: - single_node_tp - single_node_tep - single_node_dep hardware_overrides: hopper: extra_args: - "--moe-backend" - "triton" - "--gpu-memory-utilization" - "0.85" extra_env: {} strategy_overrides: single_node_tp: tp: 4 single_node_dep: extra_env: VLLM_PLE_CPU_OFFLOAD: "1" guide: | ## Overview [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) is a multimodal, ultra-sparse Mixture-of-Experts model. It has 125B parameters, including an additional 51B N-gram embedding table, while activating 6B parameters per token. The architecture combines four main ideas: - **GDN + QSA:** three of every four layers use Gated DeltaNet to compress history; the fourth uses Qwen Sparse Attention for precise long-range retrieval. - **Gated Residual:** four residual branches dynamically control cross-layer reads and writes. - **N-gram Embedding:** a 51B lookup memory adds capacity with little per-token compute and can be asynchronously offloaded to host memory. (note that offload currently only runs on Nvidia devices) - **MTP:** the built-in Multi-Token Prediction module supports speculative decoding. The checkpoint natively supports 262,144 tokens. Qwen reports that QSA reaches up to 10.2x prefill and 6.6x decode attention-kernel speedups at one million tokens. On GB300, TP2 is the minimum validated FP8 deployment and TP4 is the recommended full-tray configuration. On an 8x H200 node, use TEP8 for the official FP8 checkpoint; plain TP8 is incompatible with its 128-wide quantization blocks. ## Prerequisites - **Runtime:** use `vllm/vllm-openai:qwen38-flash-next`. PyPI installation is not supported for this recipe. - **Hardware:** NVIDIA CUDA and AMD ROCm instructions are provided below. - **FP8 memory:** the checkpoint is 172.78 GiB. TP2 is the validated minimum on GB300; TP4 is the recommended full-tray configuration. - **BF16 memory:** the checkpoint is 335.28 GiB. The validated TP2 configuration used about 190 GiB per GB300 GPU. - **Host memory for N-gram Embedding offload:** at least 51 GB plus runtime headroom. The Install block above generates the pull and Docker launch flow for the dedicated image. ## Serving ### Recommended FP8 configuration on 4x GB300 Use the official [`Qwen/Qwen3.8-Flash-Next-FP8`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) checkpoint tested by this recipe. ```bash vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.90 \ --max-num-seqs 256 \ --enable-prefix-caching \ --no-enable-flashinfer-autotune \ --enable-auto-tool-choice \ --tool-call-parser qwen3_xml \ --reasoning-parser qwen3 ``` TP2 is the minimum FP8 deployment on GB300. TP4 and TEP4 are validated full-tray configurations, including with MTP3. Enable expert parallelism with `--enable-expert-parallel` to improve throughput. To serve BF16, use the `Qwen/Qwen3.8-Flash-Next` checkpoint with the same arguments. ### Recommended FP8 configuration on 8x H200 Use TEP8 with the Triton MoE backend on Hopper: ```bash vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --moe-backend triton \ --gpu-memory-utilization 0.85 \ --max-num-seqs 256 \ --enable-prefix-caching \ --no-enable-flashinfer-autotune \ --enable-auto-tool-choice \ --tool-call-parser qwen3_xml \ --reasoning-parser qwen3 ``` Plain TP8 is incompatible with the FP8 checkpoint; use TEP8. ### FP8 configuration on a 4x MI355X GPU ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_AITER_MOE=0 vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \ --tensor-parallel-size 4 \ --max-model-len auto \ --gpu-memory-utilization 0.9 ``` ### Full native context The checkpoint advertises a native 262,144-token context, which vLLM uses when `--max-model-len` is omitted. Startup and bounded evaluation were validated with this limit configured, but a single 262K-token request was not tested. ### MTP speculative decoding Add the following option: ```bash --speculative-config '{"method":"mtp","num_speculative_tokens":3}' ``` ### Offload the 51B N-gram embedding table Builds containing N-gram Embedding offload can keep the N-gram lookup memory in host RAM and asynchronously prefetch the required rows. The initial implementation supports both ModelRunner V1 and V2. PLE CPU offload is optional for TP and TEP, but required for DEP. The DEP strategy generated by this recipe enables it automatically. ```bash VLLM_PLE_CPU_OFFLOAD=1 \ vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \ --tensor-parallel-size 4 \ --max-model-len 262144 \ --no-enable-flashinfer-autotune \ --enable-auto-tool-choice \ --tool-call-parser qwen3_xml \ --reasoning-parser qwen3 ``` ### Extend to one million tokens with YaRN The checkpoint is native at 262K. For a one-million-token workload, enable static YaRN explicitly; evaluate shorter-context quality before using this as the default. ```bash VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \ vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \ --tensor-parallel-size 4 \ --rope-scaling '{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}' \ --max-model-len 1000000 \ --no-enable-flashinfer-autotune \ --enable-auto-tool-choice \ --tool-call-parser qwen3_xml \ --reasoning-parser qwen3 ``` ## Verifying ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") response = client.chat.completions.create( model="Qwen/Qwen3.8-Flash-Next-FP8", messages=[{ "role": "user", "content": "Explain how Gated DeltaNet and Qwen Sparse Attention complement each other.", }], max_tokens=8196, ) print(response.choices[0].message.content) ``` A correct response should explain that GDN maintains a compact recurrent summary, while QSA selectively retrieves important regions from the full token history. ## Troubleshooting - **FP8 on eight GPUs:** use TEP8; plain TP8 is incompatible with this checkpoint. TP4 is also supported if GPU memory capacity allows. - **TP1 compilation OOM on GB300:** use TP2 or TP4. - **Mamba-cache capacity error at startup:** keep `--max-num-seqs 256`. - **Runtime OOM on large multimodal batches:** keep `--gpu-memory-utilization 0.90`. - **Out of memory while loading:** enable N-gram Embedding CPU offload, increase TP size, or reduce `--max-model-len` to reserve less KV cache. - **XPU/TPU startup error:** these platforms are not supported by the initial implementation. - **Pipeline-parallel startup error:** N-gram Embedding does not initially support pipeline parallelism; use single-node TP or TEP instead. - **DEP startup failure:** DEP requires `VLLM_PLE_CPU_OFFLOAD=1`; DEP without PLE offload is not supported. The generated DEP command sets this automatically. - **MTP memory pressure:** reduce `num_speculative_tokens` below 3. - **PLE:** a network layer that injects N-gram Embeddings into the main model. ## References - [Qwen3.8-Flash-Next model card](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) - [Qwen3.8-Flash-Next FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) - [Qwen3.8-Flash-Next: A New Architecture, Towards Ultimate Cost-Efficiency](https://qwen.ai/blog?id=qwen3.8-flash-next) - [Qwen3-Next recipe](../Qwen3-Next-80B-A3B-Instruct)