English | [简体中文](README_cn.md)

AngelSlim

A more accessible, comprehensive, and efficient toolkit for large model compression.

✒️ TechnicalReport   |    📖 Documentation   |   🤗 Hugging Face   |   🤖 ModelScope

💬 WeChat |   🫨 Discord

## 📣Latest News - [26/09/01] We've released the [hy4 preview MIX-STQ1_0 version](https://huggingface.co/AngelSlim/Hy4-preview-GGUF)! Compressed from 1.5TB to 214 GiB with only a 0.7% performance drop on SWE-bench Pro. Using [Prima.cpp](https://github.com/OpenCPIL/prima.cpp), we ran this 214 GB model on a Laptop 4090 (32GB RAM, 16GB VRAM) + a Server with 4x A4000 (32GB RAM, 16GB VRAM each), hitting 1.02 tokens/s. See the [blog](https://opencpil.github.io/prima.cpp/blog/hunyuan4-770b-local-devices) and the deployment [guideline](docs/source/_extra/hy4_preview_gguf_guideline.md)! 🔥🔥🔥 - [26/07/29] We have open-sourced **AngelSpec**, a torch-native, disaggregated speculative-decoding **training framework** with support for a variety of draft methods, led by **DFly** and **MTP + TTT**, and released the **MTP** and **DFly** drafter weights for **Hy3-A21B** — DFly delivers up to **2.40× average end-to-end speedup** over AR, and **D-cut** adds up to **+15.7%** throughput at high concurrency. [[Paper]](https://arxiv.org/abs/2607.25852) | [[GitHub]](https://github.com/Tencent/AngelSpec) | [[Docs]](https://angelspec.readthedocs.io/) | [[Hugging Face]](https://huggingface.co/collections/AngelSlim/angelspec) 🔥🔥🔥 - [26/07/20] We now support scale-only **quantization-aware distillation on Megatron-Core** for Qwen3-MoE and Hy3, with TP/EP/CP/SP distributed training. [[Docs]](docs/source/features/qad/mcore_qad.md) - [26/07/06] We now support FP8-Static quantization , SmoothQuant for **Hy3** (MoE A21B).[[Docs]](https://github.com/Tencent/AngelSlim/tree/main/scripts/ptq) - [26/06/04] We have released **Stem**, a sparse attention algorithm that accelerates the **Prefill** stage of long-context LLMs by dynamically selecting top-k key blocks for block-sparse attention, significantly reducing latency while preserving generation quality. [[Docs]](https://angelslim.readthedocs.io/zh-cn/latest/features/sparse_attention/stem.html) - [26/06/01] We have released **DFlare**, a block-diffusion speculative decoding framework with layer-wise fusion that achieves up to **5.52× end-to-end speedup**. [[Docs]](https://angelslim.readthedocs.io/zh-cn/latest/features/speculative_decoding/dflare.html) - [26/05/27] We have released **D-Cut**, an adaptive verification depth pruning technique for speculative decoding. [[Docs]](https://angelslim.readthedocs.io/zh-cn/latest/dcut.html) - [26/05/20] We support Distillation for full-precision HuggingFace models and **quantized QAT-style** models, as detailed in the [distillation documentation](https://angelslim.readthedocs.io/zh-cn/latest/features/distill/index.html). - [26/05/08] We have released STQ1_0 kernel for 1.25-bit model and given a PR to llama.cpp [PR #22836](https://github.com/ggml-org/llama.cpp/pull/22836) ! If you have any questions or suggestions for STQ_0, welcome to comment under the PR !🔥🔥🔥 - [26/04/29] We have released 2-bit and 1.25-bit versions of Tencent Hy-MT1.5-1.8B Translation Model: [Hy-MT1.5-1.8B-2bit](https://huggingface.co/AngelSlim/Hy-MT1.5-1.8B-2bit) and [Hy-MT1.5-1.8B-1.25bit](https://huggingface.co/AngelSlim/Hy-MT1.5-1.8B-1.25bit). Additionally, we have make an [offline translation demo](https://huggingface.co/AngelSlim/Hy-MT1.5-1.8B-1.25bit/blob/main/Hy-MT-demo.apk) for you to try out. We invite you to give it a spin! 🔥🔥🔥 - [26/04/23] We now support FP8-Static quantization for **Hy3-preview** (MoE A20B). - [26/03/25] We have released **DAQ**, the quantization algorithm that preserves the knowledge acquired while the update of parameters is relatively small during post-training training.[[Paper]](https://arxiv.org/abs/2603.22324) | [[Docs]](docs/source/features/quantization/daq.md) - [26/02/09] We have released HY-1.8B-2Bit, 2bit on-device large language model,[[Huggingface]](https://huggingface.co/AngelSlim/HY-1.8B-2Bit). - [26/01/13] We have released v0.3. We support the training and deployment of Eagle3 for all-scale LLMs/VLMs/Audio models, as detailed in the [guidance documentation](https://angelslim.readthedocs.io/zh-cn/latest/features/speculative_decoding/eagle/index.html). And We released **Sherry**, the hardware-efficient 1.25 bit quantization algorithm [[Paper]](https://arxiv.org/abs/2601.07892) | [[Code]](https://github.com/Tencent/AngelSlim/tree/sherry/Sherry)🔥🔥🔥
Previous News - [25/11/05] We have released v0.2. Quantization support for new models, such as `GLM-4.6`, `Qwen3-VL` and `Qwen3-Omni`, open-sources the Eagle3 speculative decoding training framework, and updates the Diffusion model quantization tools. - [25/09/30] We have released **SpecExit**, the reasoning early-exit algorithm: [[Paper]](http://arxiv.org/abs/2509.24248) | [[Docs]](https://angelslim.readthedocs.io/zh-cn/latest/features/speculative_decoding/spec_exit.html) | [[vLLM Code]](https://github.com/vllm-project/vllm/pull/27192) - [25/09/26] We have released **TEQUILA**, the ternary quantization algorithm [[Paper]](https://arxiv.org/abs/2509.23809) | [[Code]](https://github.com/Tencent/AngelSlim/tree/tequila/TernaryQuant) - [25/09/24] We now support the PTQ quantization of NVFP4 for the Qwen3 series models. We also opensource [Qwen3-32B-NVFP4](https://huggingface.co/AngelSlim/Qwen3-32B_nvfp4) and [Qwen3-235B-A22B-NVFP4](https://huggingface.co/AngelSlim/Qwen3-235B-A22B_nvfp4) weights. - [25/09/01] We now support ​FP8 quantization​ of the [Hunyuan-MT-7B](https://huggingface.co/tencent/Hunyuan-MT-7B-fp8) translation model. And enabled ​Torch inference and Benchmark evaluation​ for Eagle3. And implemented support for ​quantization and Cache​ for [FLUX](https://github.com/Tencent/AngelSlim/tree/main/configs/flux). And support ​quantization​ for the [Seed-OSS](https://github.com/Tencent/AngelSlim/tree/main/configs/seed_oss). - [25/08/06] We now support quantization for `Hunyuan 0.5B/1.8B/4B/7B` and multimodal model `Qwen2.5VL 3B/7B/32B/72B`, including `FP8/INT4` algorithms, and quantization for `DeepSeek-R1/V3` and `Kimi-K2`, including `FP8-Static` and `W4A8-FP8` algorithms. We also opensource `Hunyuan 1.8B/4B/7B` series Eagle3 model weight. - [25/07/04] We now support quantization for `Hunyuan/Qwen2.5/Qwen3/DeepSeek-R1-Distill-Qwen` and other models, including `INT8/FP8/INT4` algorithms. We also opensource `Qwen3` series Eagle3 model weight.
## 🌟Key Features - **Highly Integrated**: This toolkit integrates mainstream compression algorithms into a unified framework, offering developers one-click access with exceptional ease of use. - **Continuous Innovation**: Beyond integrating widely-used industry algorithms, we are continuously researching better compression algorithms, which will be gradually open-sourced in the future. - **Performance-Driven**: We continuously optimize end-to-end performance in model compression workflows and algorithm deployment, such as enabling quantization of models like Qwen3-235B and DeepSeek-R1 on a single GPU. ## 💼Technical Overview
Scenario Model Compression Strategy
Quantization Speculative Decoding Other Techniques
Large Language Models (LLMs)
Vision Language Models (VLMs)
Diffusion Models -
Speech Models​ (TTS/ASR)
  • Token Pruning
    • Under Development
## 🛎️How to Use ### 1. Install AngelSlim We recommend using `pip` to install the latest stable version of `AngelSlim`: ```shell pip install angelslim ``` Alternatively, you can clone the repository and install from source in editable mode: ```shell cd AngelSlim && python setup.py install ``` > **Note:** AngelSlim integrates the speculative-decoding training framework [AngelSpec](https://github.com/Tencent/AngelSpec) as a git submodule (under `third_party/AngelSpec/`). Clone with `--recursive`, or initialize in an existing clone: > > ```shell > # Clone with submodules > git clone --recursive https://github.com/Tencent/AngelSlim.git > # Or, in an already-cloned repo > git submodule update --init --recursive > ``` For more detailed installation instructions and platform-specific guidance, please refer to the [Installation Documentation](https://angelslim.readthedocs.io/zh-cn/latest/getting_started/installation.html). ### 2. Quick Start #### 2.1 Speculative Decoding AngelSlim's speculative-decoding training is powered by **[AngelSpec](https://github.com/Tencent/AngelSpec)** (git submodule under `third_party/AngelSpec/`) — a torch-native framework for **training** speculative-decoding draft models, featuring a rich set of draft architectures, target-model support, and production-scale training. It supports a variety of draft methods, led by **DFly** and **MTP + TTT**. It follows a **disaggregated design**: inference engines run the frozen target model and extract multi-layer hidden states, which a [Mooncake](https://github.com/kvcache-ai/Mooncake) store streams over RDMA (no disk staging) to the FSDP2 training workers, while a controller handles batching, backpressure, and evaluation. Inference and training run on separate GPU pools and scale independently. **Capabilities:** - **Multi-backend inference** — vLLM (first-class), SGLang, and HuggingFace. - **Long-sequence training** — Ulysses sequence parallelism (USP) for 128k+ contexts. - **Document-aware sequence packing** — with cross-document attention isolation. - **Online evaluation** — spec-decode acceptance rate measured during training. - **Multi-node** — large MoE target models sharded across nodes over Mooncake RDMA. - **Vocabulary pruning** — shrink the draft `lm_head` to a smaller token set at train or convert time. ```shell # Initialize the submodule (see Install section above) git submodule update --init --recursive cd third_party/AngelSpec # Install AngelSpec + the vLLM backend pip install -e ".[vllm]" # Hidden-state transport (not pulled in by the extras; install separately) pip install mooncake-transfer-engine # Single-node quickstart (4 GPUs: 2 inference + 2 training) ./examples/qwen3-8b-single-node/run.sh ``` For more details, see the [technical report](https://arxiv.org/abs/2607.25852), [AngelSpec docs](https://angelspec.readthedocs.io/), [AngelSpec README](https://github.com/Tencent/AngelSpec/blob/main/README.md), and [Hugging Face weights](https://huggingface.co/collections/AngelSlim/angelspec). #### 2.2 LLM/VLM/Audio Model Quantization After installing `AngelSlim`, you can launch static FP8 quantization for the Qwen3-1.7B model with the following one-command script: ```shell python3 tools/run.py -c configs/qwen3/fp8_static/qwen3-1_7b_fp8_static.yaml ``` This example produces quantized model weights by performing PTQ calibration on a model loaded from HuggingFace. For **Hy3-preview** (MoE A20B) FP8-Static quantization: ```shell python tools/run.py -c configs/hunyuan/fp8_static/hunyuanv3_a20b_fp8_static_c8.yaml ```
Code-based Start To perform dynamic `FP8` quantization on `Qwen3-1.7B`: ```python from angelslim.engine import Engine slim_engine = Engine() # Prepare model slim_engine.prepare_model(model_name="Qwen", model_path="Qwen/Qwen3-1.7B",) # Initialize compressor slim_engine.prepare_compressor("PTQ", default_method="fp8_dynamic") # Compress model slim_engine.run() # Save compressed model slim_engine.save("./output") ```
For more details, please refer to the [Quick Start Documentation](https://angelslim.readthedocs.io/zh-cn/latest/getting_started/quickstrat.html). #### 2.3 Diffusion Model Quantization Use the `scripts/diffusion/run_diffusion.py` for quantization and inference: ```shell # Online quantization and inference python scripts/diffusion/run_diffusion.py \ --model-name-or-path black-forest-labs/FLUX.1-schnell \ --quant-type fp8-per-tensor \ --prompt "A cat holding a sign that says hello world" \ --height 1024 --width 1024 --steps 4 --guidance 0.0 --seed 0 ``` For more quantization inference methods, please refer to [the Diffusion Model Quantization Documentation](https://angelslim.readthedocs.io/zh-cn/latest/features/diffusion/quantization.html). #### 2.4 Token Compression (VLM) AngelSlim provides a universal metadata-driven framework for vision token pruning and merging. You can quickly verify a compression strategy (e.g., **VisionZip**) with a smoke test: ```shell python tools/test_universal_pruning.py \ --model_path "Qwen/Qwen2.5-VL-3B-Instruct" \ --config "configs/qwen2_5_vl/pruning/visionzip_r0.9.yaml" ``` For more details on implementing new strategies, please refer to the [Token Compressor Documentation](https://angelslim.readthedocs.io/zh-cn/latest/features/token_compressor/index.html). ### 3. Deployment and Testing #### 3.1 Offline Inference To test offline inference with a quantized model loaded via `transformers`.
Run script details ```shell python scripts/deploy/offline.py $MODEL_PATH "Hello, my name is" ``` Where `MODEL_PATH` is the path to the quantized model output.
#### 3.2 API Service Deployment After specifying the quantized model path `MODEL_PATH`, you can deploy an OpenAI-compatible API service using **vLLM** and **SGLang** inference frameworks.
Run script details - **vLLM** Use the following script to launch a [vLLM](https://github.com/vllm-project/vllm) server, recommended version `vllm>=0.8.5.post1`. For MOE INT8 quantized models, vllm>=0.9.0 is required. ```shell bash scripts/deploy/run_vllm.sh --model-path $MODEL_PATH --port 8080 -d 0,1,2,3 -t 4 -p 1 -g 0.8 --max-model-len 4096 ``` Where `-d` is the visible devices, `-t` is tensor parallel size, `-p` is pipeline parallel size, and `-g` is the GPU memory utilization. - **SGLang** Use the following script to launch a [SGLang](https://github.com/sgl-project/sglang) server, recommended version `sglang>=0.4.6.post1`. ```shell bash scripts/deploy/run_sglang.sh --model-path $MODEL_PATH --port 8080 -d 0,1,2,3 -t 4 -g 0.8 ```
#### 3.3 Service Invocation Invoke requests via [OpenAI's API format](https://platform.openai.com/docs/api-reference/introduction).
Run script details ```shell bash scripts/deploy/openai.sh -m $MODEL_PATH -p "Hello, my name is" --port 8080 --max-tokens 4096 --temperature 0.7 --top-p 0.8 --top-k 20 --repetition-penalty 1.05 --system-prompt "You are a helpful assistant." ``` where `-p` is the input prompt.
#### 3.4 Performance Evaluation Evaluate the performance of quantized model using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), recommended version`lm-eval>=0.4.8`.
Run script details ```shell bash scripts/deploy/lm_eval.sh -d 0,1 -t 2 -g 0.8 -r $RESULT_PATH -b "auto" --tasks ceval-valid,mmlu,gsm8k,humaneval -n 0 $MODEL_PATH ``` where `RESULT_PATH` is the directory for saving test results, `-b` is batch size, `--tasks` specifies the evaluation tasks, and `-n` is the number of few-shot examples.
For more detaileds, please refer to the [Deployment Documentation](https://angelslim.readthedocs.io/zh-cn/latest/deployment/deploy.html). ## 📈 Benchmark ### 1. Speculative Decoding The results below come from **[AngelSpec](https://github.com/Tencent/AngelSpec)** — a torch-native, disaggregated speculative-decoding training framework, integrated into AngelSlim as a git submodule (see §2.1), with support for a variety of draft methods, led by **DFly** and **MTP + TTT**. [[Paper]](https://arxiv.org/abs/2607.25852) | [[Docs]](https://angelspec.readthedocs.io/) | [[Hugging Face]](https://huggingface.co/collections/AngelSlim/angelspec) **✨ Highlights** - **🏗️ Full-stack open source** — the AngelSpec training framework plus Hy3-A21B **MTP** / **DFly** drafter weights and training code, all released at once. - **🚀 DFly leads across the board** — on Hy3-A21B, DFly delivers the highest throughput across all concurrency levels (4–64) and all six benchmarks — **1.98–2.40×** average speedup over the AR baseline (peak **2.86×** on code / math), and **10.5–11.8%** faster than DFlash. - **📈 Substantially higher accepted length** — DFly reaches a mean accepted length of **4.79** — **+30%** over DFlash (3.69) and **~1.6×** over MTP (3.00) — up to **5.52** on HumanEval. - **⚡ D-cut squeezes high concurrency** — on live traffic of a 295B model, it adds up to **+15.7%** throughput in the high-concurrency regime at near-zero cost (mean accepted length 2.50 → 2.46). - **💬 MTP + TTT cracks chat** — fixes the train/inference mismatch: mean acceptance rate **52.8% → 66.4%**, mean accepted length **2.58 → 2.99**. #### 1.1 🏆 Drafter Showdown — DFly Leads the Pack **Draft quality — mean accepted length.** Measured on **Qwen3-8B** and **Hy3-A21B** across math / code / chat (temperature = 1, no thinking); **bold** marks the best drafter per target per benchmark. **DFly** leads on average — **5.41** on Qwen3-8B and **4.79** on Hy3-A21B — and tops nearly every benchmark, well ahead of DFlash and MTP.
Target Model Drafter Math Code Chat Avg.
Math500GSM8K HumanEvalMBPPLiveCodeBench MT-Bench
Qwen3-8B MTP 3.533.563.333.223.252.573.24
DFlash 4.975.544.774.504.463.164.57
DSpark 5.876.255.565.255.203.775.32
DFly 6.066.425.605.345.363.675.41
Hy3-A21B MTP 3.303.303.133.042.842.403.00
DFlash 4.014.234.364.053.102.383.69
DFly 5.235.535.525.414.072.964.79
**End-to-end throughput on Hy3-295B-A21B.** Output-token throughput (Tok/s) and speedup (Spd.) over the AR baseline (TP=8, temperature 1, concurrency c4–c64; each cell = 3×120s windows). **Bold** marks the best speedup per benchmark at each concurrency; `MTP-3` uses 3 speculative tokens, `DFlash-8` / `DFly-8` use block length 8. **DFly-8** wins the average at every concurrency — **1.98×–2.40×** over AR, peaking at **2.86×** on HumanEval.
Conc. Method GSM8K Math500 HumanEval MBPP LiveCodeBench MT-Bench Avg.
Tok/sSpd. Tok/sSpd. Tok/sSpd. Tok/sSpd. Tok/sSpd. Tok/sSpd. Tok/sSpd.
c4 AR 287.91.00× 293.91.00× 279.71.00× 294.41.00× 284.51.00× 290.01.00× 288.41.00×
MTP-3 495.51.72× 517.01.76× 473.61.69× 476.71.62× 414.01.46× 384.51.33× 460.21.60×
DFlash-8 569.51.98× 587.92.00× 588.12.10× 575.71.96× 408.41.44× 371.61.28× 516.91.79×
DFly-8 635.42.21× 643.92.19× 647.22.31× 661.52.25× 455.21.60× 384.01.32× 571.21.98×
c8 AR 426.31.00× 435.71.00× 400.31.00× 433.71.00× 413.41.00× 421.81.00× 421.81.00×
MTP-3 756.91.78× 791.51.82× 717.41.79× 729.11.68× 620.81.50× 590.11.40× 701.01.66×
DFlash-8 857.02.01× 860.11.97× 866.22.16× 866.22.00× 609.11.47× 545.51.29× 767.41.82×
DFly-8 964.82.26× 959.52.20× 974.12.43× 1001.52.31× 641.31.55× 565.81.34× 851.22.02×
c16 AR 650.71.00× 670.61.00× 594.81.00× 667.91.00× 617.21.00× 628.11.00× 638.21.00×
MTP-3 1146.61.76× 1174.01.75× 1076.11.81× 1103.31.65× 893.51.45× 876.61.40× 1045.01.64×
DFlash-8 1446.12.22× 1430.42.13× 1433.32.41× 1440.42.16× 970.51.57× 901.01.43× 1270.31.99×
DFly-8 1623.82.50× 1607.52.40× 1595.42.68× 1677.82.51× 1046.71.70× 961.51.53× 1418.82.22×
c32 AR 918.61.00× 1000.81.00× 850.81.00× 1018.71.00× 893.21.00× 921.11.00× 933.91.00×
MTP-3 1923.82.09× 1989.81.99× 1780.72.09× 1863.61.83× 1394.21.56× 1469.41.60× 1736.91.86×
DFlash-8 2261.82.46× 2334.32.33× 2170.62.55× 2339.32.30× 1489.91.67× 1453.41.58× 2008.22.15×
DFly-8 2527.42.75× 2608.92.61× 2429.12.86× 2741.32.69× 1602.61.79× 1549.41.68× 2243.12.40×
c64 AR 1156.91.00× 1381.31.00× 1197.61.00× 1513.91.00× 1229.61.00× 1306.91.00× 1297.71.00×
MTP-3 2932.02.53× 3170.02.29× 2639.62.20× 3011.31.99× 2050.41.67× 2350.01.80× 2692.22.08×
DFlash-8 2523.42.18× 2947.12.13× 2655.22.22× 2671.81.76× 2015.81.64× 1815.81.39× 2438.21.89×
DFly-8 2827.82.44× 3301.82.39× 2965.02.48× 3130.92.07× 2195.01.79× 1936.61.48× 2726.22.11×
#### 1.2 ⚡ D-cut — Breaking the Concurrency Ceiling When concurrency climbs, target verification becomes the bottleneck and rejected draft suffixes clog the batch. **D-cut** treats verification as a shared batch budget — ranking each request's verification depth by expected gain / cost, deepening drafts when the system is idle and trimming them when it's slammed. The payoff lands exactly where DFly plateaus: past concurrency 48, D-cut keeps turning load into throughput, up to **+15.7%** over DFly at near-zero quality cost (accepted length 2.50 → 2.46).

D-cut on Hy3-295B-A21B live traffic

### 2. Quantization The performance test results for selected models are shown below. For the complete benchmark, refer to the [Benchmark documentation](https://angelslim.readthedocs.io/zh-cn/latest/performance/quantization/benchmarks.html) #### 2.1 Hunyuan Series Models Benchmark results for the `Hunyuan-Instruct` model with `FP8`, `INT4-AWQ` and `INT4-GPTQ` quantization algorithms on datasets including`OlympiadBench`, `AIME 2024` and `DROP`:
ModelQuantizationOlympiadBenchAIME 2024DROPGPQA-Diamond
Hunyuan-A13B-Instruct BF1682.787.3091.171.2
FP8-Static83.086.791.1-
Int4-GPTQ82.786.791.1-
Int4-AWQ82.685.691.0-
Hunyuan-7B-Instruct BF16 76.581.185.960.1
FP8-Static76.680.986.060.1
Int4-GPTQ76.281.085.760.0
Int4-AWQ76.480.985.960.1
Hunyuan-4B-Instruct BF16 73.178.378.261.1
FP8-Static73.176.678.360.2
Int4-GPTQ72.9-78.158.1
Int4-AWQ72.8-78.2-
Hunyuan-1.8B-Instruct BF16 63.456.776.747.2
FP8-Static62.555.275.147.7
Int4-GPTQ60.9-73.044.4
Int4-AWQ61.7-71.743.6
Hunyuan-0.5B-Instruct BF16 29.617.252.823.3
FP8-Static29.617.251.622.5
Int4-GPTQ26.8-50.923.3
Int4-AWQ26.3-48.923.3
#### 2.2 Qwen3 Series Models Benchmark results for Qwen3 series models with `FP8-Static`, `FP8-Dynamic`, `INT4-GPTQ`, and `INT4-AWQ` quantization algorithms on datasets including `CEVAL`, `MMLU`, `GSM8K`, and `HUMANEVAL`:
ModelQuantizationCEVALMMLUGSM8KHUMANEVAL
Qwen3-0.6BBF1645.8447.2142.9919.51
FP8-Static45.9946.8738.0618.90
FP8-Dynamic45.9946.9338.2920.73
INT8-Dynamic45.1746.9541.1721.34
Qwen3-8BBF1679.2774.7887.7963.41
FP8-Static78.2374.7986.9662.20
FP8-Dynamic78.4574.7587.6462.80
INT8-Dynamic78.0174.8486.9667.07
INT4-GPTQ77.1973.2686.4362.20
INT4-AWQ76.1573.5986.9663.41
Qwen3-14BBF1683.0678.9088.4055.49
FP8-Static82.6278.5789.4657.32
FP8-Dynamic82.2478.9288.3252.44
INT8-Dynamic81.8778.1386.2856.10
INT4-GPTQ81.0578.0287.3457.93
INT4-AWQ82.0277.6884.2361.59
Qwen3-32BBF1686.5582.0074.5337.80
FP8-Static86.9281.7870.2039.63
FP8-Dynamic86.5581.8970.4338.41
INT4-GPTQ86.1881.01-43.29
INT4-AWQ86.1881.54-36.59
Qwen3-30B-A3BBF1683.6679.3689.9931.71
FP8-Static83.9579.4789.0131.10
FP8-Dynamic84.1079.4089.1632.93
INT8-Dynamic83.3679.4889.1634.15
Qwen3-235B-A22BBF1689.6086.2885.2927.44
FP8-Static89.6786.1986.9627.44
FP8-Dynamic89.6786.1885.2228.05
INT8-Dynamic88.9386.2086.2023.78
#### 2.3 DeepSeek Series Models Benchmark results for DeepSeek-R1-0528 series models with `FP8-Block-Wise` and `W4A8-FP8` quantization algorithms on datasets including `GPQA Diamond`、`AIME 2024`、`SimpleQA` and `LiveCodeBench`：
ModelQuantizationGPQA DiamondAIME 2024SimpleQALiveCodeBench
DeepSeek-R1-0528FP8-Block-Wise78.2888.6727.877.1
W4A8-FP877.3788.6726.8378.86
Note > - The above results are based on the average of 5 test runs deployed with TRT-LLM > - The hyperparameters used during evaluation are as follows: > ```json >{ > "top_k": 20, > "top_p": 0.6, > "temperature": 0.7, > "output_seq_len": 32768, > "max_input_seq_len": 16384 >} >```
#### 2.4 Qwen-VL Series Models **Qwen3-VL Benchmark** Benchmark results for Qwen3VL series models with `BF16`、`FP8-Static` and `FP8-Dynamic` quantization algorithms on datasets including `MMMU_VAL`、`DocVQA_VAL` and `ChartQA_TEST`：
ModelQuantizationMMMU_VALDocVQA_VALChartQA_TEST
Qwen3-VL-32B-InstructBF1660.1196.0894.64
FP8-Static61.2296.0094.64
FP8-Dynamic60.7896.1994.72
Qwen3-VL-30B-A3B-InstructBF1650.4495.2895.36
FP8-Dynamic50.6795.2595.20
Qwen2.5VL Benchmark Benchmark results for Qwen2.5VL series models with `BF16`、`FP8-Static`、`FP8-Dynamic`、`INT4-GPTQ`、`INT4-AWQ` quantization algorithms on datasets including `MMMU_VAL`、`DocVQA_VAL` and `ChartQA_TEST`：
ModelQuantizationMMMU_VALMMLDocVQA_VALUChartQA_TEST
Qwen2.5VL-3BBF1647.1178.5780.32
FP8-Static47.3379.3479.68
FP8-Dynamic45.9946.9338.29
INT4-GPTQ46.5677.2078.96
INT4-AWQ45.78-79.60
Qwen2.5VL-7BBF1645.4489.7184.64
FP8-Static47.0089.8385.92
FP8-Dynamic47.2289.8088.64
INT4-GPTQ46.6790.45-
INT4-AWQ45.6789.28-
Qwen2.5VL-32BBF1657.0090.03-
FP8-Static57.0089.88-
FP8-Dynamic56.4489.88-
INT4-GPTQ55.2289.80 -
INT4-AWQ55.2290.30-
Qwen2.5VL-72BBF1658.7894.3985.60
FP8-Static57.8994.4185.84
FP8-Dynamic58.6794.3885.60
INT4-GPTQ57.5694.4686.48
INT4-AWQ58.7894.1987.28
#### 2.5 Qwen-Omni Series Models **Qwen3-Omni Text to Text Benchmark** Benchmark results for Qwen3-Omni series models in BF16, FP8-Static, and FP8-Dynamic on aime25, gpqa_diamond, and mmlu_redux are as follows:
ModelQuantizationaime25gpqa_diamondmmlu_redux
Qwen3-Omni-30B-A3B-InstructBF1673.3256.7788.09
FP8-Static71.3356.5787.91
FP8-Dynamic73.3355.1588.07
Note > - The above evaluation results were obtained by deploying with the vLLM framework and averaging over 5 runs (vLLM only supports the thinker component). > - The hyperparameters used during evaluation are as follows: > ```json >{ > "top_p": 0.95, > "temperature": 0.6, > "do_sample": true, > "max-model-len 65536": 65536 >} >```
#### 2.6 Other Models Other models such as GLM-4.6, Qwen2.5, and Seed-OSS have been evaluated on benchmarks like `CEVAL`, `MMLU`, and `GSM8K` using quantization strategies including `FP8-Static`, `FP8-Dynamic`, `INT4-GPTQ`, and `INT4-AWQ`.
Benchmark Experiment Details
ModelQuantizationCEVALMMLUGSM8K
Qwen2.5-1.5B-InstructBF1667.0160.0554.28
FP8-Static66.2760.23-
FP8-Dynamic66.7960.0851.71
Qwen2.5-7B-InstructBF1681.2074.5579.98
FP8-Static81.1374.0379.30
FP8-Dynamic80.3174.0779.00
INT4-GPTQ79.0573.0574.75
INT4-AWQ79.3573.2279.38
Qwen2.5-32B-InstructBF1687.3083.2181.73
FP8-Static87.5983.0881.58
FP8-Dynamic87.3083.0481.58
INT4-GPTQ86.7082.4582.03
INT4-AWQ87.0082.64-
DeepSeek-R1-Distill-Qwen-7BBF1653.4953.8075.74
FP8-Static53.5754.1776.19
FP8-Dynamic52.9754.1374.15
INT4-GPTQ51.8652.4475.89
INT4-AWQ53.4953.70-
DeepSeek-R1-Distill-Qwen-14BBF1677.7174.2885.67
FP8-Static77.5674.6686.73
FP8-Dynamic76.8274.6387.11
INT4-GPTQ74.2972.3784.61
INT4-AWQ74.8173.0086.05
DeepSeek-R1-Distill-Qwen-32BBF1684.1880.8987.41
FP8-Static83.4380.9087.57
FP8-Dynamic83.7381.1086.43
INT4-GPTQ84.1079.8086.73
INT4-AWQ82.8480.1587.19
### 3. Token Compression (VLM) We evaluated various vision token compression strategies on the **Qwen2.5-VL-3B-Instruct** model across multiple multimodal benchmarks. You can replicate these results using the following command: ```shell python tools/run_pruning_eval.py \ --model_path "Qwen/Qwen2.5-VL-3B-Instruct" \ --configs "configs/qwen2_5_vl/pruning/visionzip_r0.9.yaml" \ --tasks "textvqa" \ --output_dir "./results/visionzip_test" ```
Detailed Benchmark Results (Qwen2.5-VL-3B-Instruct)
Method AI2D ChartQA DocVQA MMBCN MMB MME MMStar OCRBench POPE SQA VQAText Avg
Baseline 79.11 83.56 92.48 73.28 77.32 1517 56.05 80.10 87.41 80.81 78.79 100.0%
Retain 25% Tokens (75% Compression Ratio)
FastV72.7070.0475.9863.4066.92143747.3936.6086.4279.3373.5186.02%
VisionZip74.1971.3270.1167.3571.22145249.3742.5085.5181.3668.1287.34%
HiPrune73.8372.7672.1067.2772.34144948.9341.3085.8680.9169.2787.67%
VisionSelector75.1973.7290.2468.8172.59152149.9761.8085.3680.3776.8693.62%
DivPrune73.0662.9678.4667.1071.82145948.3851.4086.8180.2268.9188.15%
DART71.0865.2079.7265.3871.05142848.7841.8080.9780.9168.2586.17%
VisPruner74.2968.2072.5267.3570.88145849.7444.8086.5981.4669.6287.87%
SCOPE75.8474.0082.4068.8172.94147150.3556.0086.6280.9674.0491.98%
IDPruner75.9475.8490.0069.4273.80150549.4964.9086.2680.4276.9094.42%
Retain 10% Tokens (90% Compression Ratio)
FastV65.8729.7236.8948.3751.98125737.2813.9079.5077.0557.7565.30%
VisionZip67.6551.6037.8859.6263.06133842.8221.4081.1480.4751.5672.75%
HiPrune67.7553.2041.1559.4563.14132641.0820.3080.9080.9653.3173.00%
VisionSelector70.5065.9279.9459.9764.69137442.8645.2082.6680.6171.5784.42%
DivPrune67.7143.1258.0361.2565.12138940.4327.9082.2479.1856.8775.50%
DART67.4947.5660.2357.9963.83129942.1823.4074.2078.6358.0274.09%
VisPruner67.7547.9248.6559.2863.32130541.5122.5078.7479.7754.9573.19%
SCOPE69.7556.2455.0164.2667.18139044.3530.8083.3480.4762.5879.37%
IDPruner71.7963.3279.3863.5768.21143844.0545.5084.5180.5770.0285.71%
## 📝 License The code for this project is open-sourced under the [License for AngelSlim](LICENSE). ## 🔗 Citation ``` @article{angelslim2026, title={AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compression}, author={Hunyuan AI Infra Team}, journal={arXiv preprint arXiv:2602.21233}, year={2026} } ``` ## 💬 Technical Discussion * AngelSlim is developed by the Tencent Hunyuan AI Infra team, with new features being iteratively updated. If you have any questions or suggestions, please submit them on [GitHub Issues](https://github.com/Tencent/AngelSlim/issues) or join our [WeChat discussion group](./docs/source/assets/angel_slim_wechat.png). * ⭐ Star this repo to follow our latest progress. And if you are interested in joining us for an internship or full-time position, send your resume to: lucayu@tencent.com.