Baidu ERNIE

# vLLM-ERNIE vLLM plugin for ERNIE models, one package per model under `src/vllm_ernie/models/`. It currently serves ERNIE 5.5 (`Ernie5_5_MoeForCausalLM`: HCA + MLA-DSA, mHC, MoE), with ECHO (MTP) and DSpark speculative decoding and the `ernie5` reasoning / tool-call parsers. It runs on the released vLLM **0.32.0** wheel, unmodified, on NVIDIA GPUs only. ## Supported Models | Model | Weights | Use | |---|---|---| | ERNIE-5.5-Flash-Base | [FP8](https://huggingface.co/baidu/ERNIE-5.5-Flash-Base) / [BF16](https://huggingface.co/baidu/ERNIE-5.5-Flash-Base-BF16) | Base model, completions. Released | | ERNIE-5.5-Flash | FP8 / FP4 / BF16 | Chat, reasoning, tool calling, DSpark. Coming soon | Set `VLLM_USE_MODELSCOPE=True` to download from [ModelScope](https://www.modelscope.cn/models/baidu/ERNIE-5.5-Flash-Base); the repo names are the same on both hubs. ## Model Introduction ERNIE-5.5-Flash is a sparse Mixture-of-Experts language model from Baidu, with 280B total and 14B activated parameters per token (excluding the 0.36B ECHO layer), pre-trained from scratch on 33.5T tokens with a context of up to 128K. Inference supports up to 1,048,576 tokens. - **Hybrid HCA + MLA-DSA Attention.** 37 of the 43 layers use Heavily Compressed Attention (one cached entry per 128 tokens plus a 128-token sliding window); 6 layers keep full-context access through MLA with DSA sparse attention (top-2048 tokens). - **Gated Attention and Virtual Head Attention (VHA).** Every attention layer adds a low-rank output gate, a low-rank post-mix across heads and QK-norm. - **Multi-Router Latent MoE.** 42 MoE layers, each with 512 routed experts (top-10) and 1 shared expert in a 2048-d latent space, with SiTU-GLU activations. - **mHC.** Manifold-constrained hyper-connections (expansion rate 4) replace the residual stream. - **MultiMax Output Head.** with a learnable piece-wise temperature. - **ECHO.** The final layer, trained to predict future tokens, doubles as a built-in draft; vLLM runs it through its MTP path. ## Install ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm==0.32.0 uv pip install git+https://github.com/baidu/vLLM-ERNIE.git ``` The plugin is loaded through the `vllm.general_plugins` entry point in every vLLM process. `VLLM_PLUGINS=""` disables it. If `vllm serve` reports `Model architectures ['Ernie5_5_MoeForCausalLM'] are not supported`, the plugin is not installed in the environment that runs `vllm`, or `VLLM_PLUGINS` is set and does not include `ernie`. ## Serve Attention runs data parallel and the MoE expert parallel (`--data-parallel-size N --enable-expert-parallel`); attention tensor parallelism (`--tensor-parallel-size` 2, 4 or 8) also works, see Configuration Tips. The commands below are for one 4-GPU node and use the 128K training context (`--max-model-len 131072`), with ECHO (MTP) speculative decoding on. The smallest GPU counts that hold the weights: | GPU memory | FP8 (~280 GB weights) | BF16 (~560 GB weights) | |---|---|---| | 288 GB | 2 GPUs | 4 GPUs | | 192 GB | 4 GPUs | 8 GPUs (or 2 nodes × 4 GPUs) | | 80 GB | 8 GPUs | 2 nodes × 8 GPUs | The FP8 commands pass `--moe-backend humming`, which supports SiTU with FP8 weights on Blackwell-architecture GPUs. On Hopper-architecture GPUs, drop it and keep vLLM's default MoE kernel. On 80 GB GPUs, serve FP8 on 8 GPUs with `--data-parallel-size 8` and `--gpu-memory-utilization 0.93`; up to `--max-model-len 262144` fits, 1M does not. The commands bind `0.0.0.0` without authentication; add `--api-key ` or bind `127.0.0.1` unless the network is trusted. ### FP8 ```bash export VLLM_ENGINE_READY_TIMEOUT_S=3600 vllm serve baidu/ERNIE-5.5-Flash-Base \ --host 0.0.0.0 \ --port 8000 \ --data-parallel-size 4 \ --enable-expert-parallel \ --moe-backend humming \ --kv-cache-dtype fp8 \ --max-model-len 131072 \ --speculative-config '{"method":"mtp","num_speculative_tokens":2}' ``` Drop `--speculative-config` to serve without a draft. ECHO drafts 2 steps here; 1 or 3 also work. ### BF16 BF16 runs on two 4-GPU nodes (DP8 + EP8). Set `HEAD_IP` to the first node's address and `IFACE_NAME` to the network interface between the nodes, then run on the first node: ```bash export VLLM_ENGINE_READY_TIMEOUT_S=3600 export GLOO_SOCKET_IFNAME=${IFACE_NAME} export NCCL_SOCKET_IFNAME=${IFACE_NAME} vllm serve baidu/ERNIE-5.5-Flash-Base-BF16 \ --host 0.0.0.0 \ --port 8000 \ --data-parallel-size 8 \ --data-parallel-size-local 4 \ --data-parallel-address ${HEAD_IP} \ --data-parallel-hybrid-lb \ --enable-expert-parallel \ --kv-cache-dtype fp8 \ --max-model-len 131072 \ --speculative-config '{"method":"mtp","num_speculative_tokens":2}' ``` The second node runs the same command with `--data-parallel-start-rank 4`. One 4-GPU node also holds the BF16 weights, but leaves little room for the KV cache: there, pass `--gpu-memory-utilization 0.95`, `--max-model-len 65536`, `--max-num-seqs 16`, `--max-num-batched-tokens 8192` and `--max-cudagraph-capture-size 512` instead. ### DSpark (ERNIE-5.5-Flash, Coming Soon) The ERNIE-5.5-Flash model bundles a DSpark draft; the Base models carry none. Replace the ECHO `--speculative-config` with: ```bash --speculative-config '{"method":"dspark","draft_sample_method":"probabilistic"}' ``` The draft length comes from the model's block size, so the config leaves out `num_speculative_tokens`. vLLM's default `draft_sample_method` is `greedy`; the command uses `probabilistic`. ### 1M Context The model allows 1,048,576 tokens. Pass `--max-model-len 1048576`; a 1M-token prefill needs tens of GB of temporary memory on top, so lower `--gpu-memory-utilization` from the default 0.92 to 0.85, and lower `--max-num-seqs` so each rank's KV cache holds whole 1M-token requests. The values are for GPUs with 184 GiB of memory; 80 GB GPUs do not fit 1M: ```bash export VLLM_ENGINE_READY_TIMEOUT_S=3600 vllm serve baidu/ERNIE-5.5-Flash-Base \ --host 0.0.0.0 \ --port 8000 \ --data-parallel-size 4 \ --enable-expert-parallel \ --moe-backend humming \ --kv-cache-dtype fp8 \ --max-model-len 1048576 \ --gpu-memory-utilization 0.85 \ --speculative-config '{"method":"mtp","num_speculative_tokens":2}' ``` ### Prefill-Decode (PD) Disaggregation One prefill (P) and one decode (D) 4-GPU node, each DP4 + EP4, move the KV cache over RDMA with `MooncakeConnector`, behind vllm-router 0.1.16. Install the Mooncake transfer engine on both nodes and the router on any host that reaches both: ```bash uv pip install mooncake-transfer-engine-cuda13 # P and D uv pip install vllm-router==0.1.16 # router host ``` Set these on both nodes. `IB_DEVICES` lists the InfiniBand HCAs only (`ibdev2netdev` shows them): ```bash export P_IP=10.0.0.1 export D_IP=10.0.0.2 export IB_DEVICES=mlx5_0,mlx5_1,mlx5_4,mlx5_5 export VLLM_ENGINE_READY_TIMEOUT_S=3600 export VLLM_HTTP_TIMEOUT_KEEP_ALIVE=1800 # keep idle router connections export VLLM_MOONCAKE_ABORT_REQUEST_TIMEOUT=600 # fail a stuck KV transfer ``` P node (D is the same with `D_IP` and `kv_consumer`): ```bash VLLM_HOST_IP=${P_IP} vllm serve baidu/ERNIE-5.5-Flash-Base \ --host ${P_IP} \ --port 8000 \ --data-parallel-size 4 \ --enable-expert-parallel \ --moe-backend humming \ --kv-cache-dtype fp8 \ --max-model-len 131072 \ --gpu-memory-utilization 0.85 \ --max-num-seqs 128 \ --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_connector_extra_config":{"device_name":"'${IB_DEVICES}'"}}' ``` Router: ```bash vllm-router \ --host 0.0.0.0 \ --port 8080 \ --policy round_robin \ --vllm-pd-disaggregation \ --kv-connector mooncake \ --prefill http://${P_IP}:8000 8998 \ --decode http://${D_IP}:8000 \ --intra-node-data-parallel-size 4 ``` P serves the Mooncake bootstrap on port 8998. With speculative decoding, pass the same `--speculative-config` to both nodes. The `model` field of each request must match the served model name; otherwise the request fails mid-stream instead of returning a 404. ### Reasoning and Tool Calling (ERNIE-5.5-Flash, Coming Soon) For ERNIE-5.5-Flash (the Base models have no chat training), add to the `vllm serve` command: ```bash --reasoning-parser ernie5 \ --tool-call-parser ernie5 \ --enable-auto-tool-choice ``` Thinking is returned in `reasoning_content`, the answer in `content` and tool calls in `message.tool_calls`. Thinking is on by default; pass `"chat_template_kwargs": {"enable_thinking": false}` in a request to turn it off. ### KV Cache Offload Long shared prefixes can be kept outside GPU memory and loaded back instead of recomputed. With `SimpleCPUOffloadConnector`, `cpu_bytes_to_use_per_rank` is pinned per DP rank, so the rank count times this value must fit in host RAM; for example, 128 GiB per rank with 4 ranks: ```bash --kv-transfer-config '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":137438953472}}' ``` ## Request The Base models continue text, so send prompts to `/v1/completions`; for few-shot use, put the examples in the prompt and stop at the end of one answer. The model's `generation_config.json` sets `do_sample=false`, `temperature=1.0`, `top_p=0.95`. vLLM applies `temperature` and `top_p` from it but not `do_sample`, so pass `"temperature": 0` for greedy decoding. ```bash curl http://127.0.0.1:8000/v1/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "baidu/ERNIE-5.5-Flash-Base", "prompt": "Question: What is 15% of 240?\nAnswer:", "max_tokens": 64, "temperature": 0, "stop": ["\n"] }' ``` The completion text in the response is ` 36`. For the BF16 weights, set `"model"` to `baidu/ERNIE-5.5-Flash-Base-BF16`. The tokenizer prepends `<|ERNIE_MASK|>` (id 2) to completions prompts itself; do not add it to the prompt. ## Configuration Tips - **Parallelism.** Data parallel attention with `--data-parallel-size` and `--enable-expert-parallel` holds the most KV cache and gives the highest throughput. Attention tensor parallelism (`--tensor-parallel-size`, which must divide 8) lowers latency at low concurrency; the single KV head is replicated on every rank of the TP group, so it holds less KV cache. - **MoE kernel.** The FP8 commands pass `--moe-backend humming`, which supports SiTU with FP8 weights. On Hopper-architecture GPUs, keep vLLM's default. - **Long context.** The commands pass `--max-model-len 131072`, the 128K training length. The model allows 1,048,576: pass `--max-model-len 1048576`, lower `--max-num-seqs` so each rank's KV cache holds whole 1M-token requests, and pass `--gpu-memory-utilization 0.85`. On 80 GB GPUs 1M does not fit; they reach 262,144 at `--gpu-memory-utilization 0.93`. - **KV cache.** Every command passes `--kv-cache-dtype fp8`, which stores the KV cache in FP8; `auto` keeps it in BF16. ## License Apache License 2.0, see [LICENSE](LICENSE).