# VeRL-Omni ### Easy, fast, and stable RL training for diffusion and omni-modality models Ask DeepWiki.com [![Docs](https://img.shields.io/badge/docs-Read%20the%20Docs-8A2BE2)](https://verl-omni.readthedocs.io/en/latest/index.html) [![License](https://img.shields.io/badge/license-Apache%202.0-blue)](./LICENSE) [English](https://github.com/verl-project/verl-omni/blob/main/README.md) | [简体中文](https://github.com/verl-project/verl-omni/blob/main/README.zh.md)
`VeRL-Omni` is a general RL training framework focused on multimodal generative models, built on top of [`verl`](https://github.com/verl-project/verl). It originated from the multi-modal generation RL effort in `verl`, and now has a dedicated home so it can evolve in a more focused way. ## News 🔥 - **[2026-10]** 🎉 We have released [v0.3.0](https://github.com/verl-project/verl-omni/releases/tag/v0.3.0) for fully async diffusion/omni RL with the v1 trainer as the default. - **[2026-10]** [OmniNFT](examples/omninft_trainer/README.md) is now supported for audio-video post-training. The [LTX2.3](examples/omninft_trainer/ltx2/README.md) text-to-video+audio LoRA recipe runs on Ascend NPU with modality-wise advantage routing. - **[2026-09]** The [v1 trainer](docs/start/diffusion_v1.md) (TransferQueue + ReplayBuffer) is now the default for diffusion models; the legacy v0 trainer is deprecated, while offline DPO remains on v0 by design. - **[2026-08]** **[DiffusionOPD](docs/algo/diffusion_opd.md)** (on-policy distillation, including multi-teacher MOPD) is now supported. - **[2026-08]** 🔥 **MiniMax-H3** now supports T2VA, FL2VA, and Ref2VA with both [FlowGRPO](examples/flowgrpo_trainer/minimax_h3/README.md) and [DiffusionNFT](examples/diffusionnft_trainer/minimax_h3/README.md). - **[2026-08]** 🎉 We have released [v0.2.0](https://github.com/verl-project/verl-omni/releases/tag/v0.2.0) for faster diffusion rl and more stable Qwen3-Omni multimodal training. Blog: [VeRL-Omni v0.2.0](https://verl-project.github.io/posts/2026-08-17-verl-omni-v0-2-0/) - **[2026-08]** [LTX2.3](examples/flowgrpo_trainer/ltx2/README.md) text-to-video+audio model is now supported with FlowGRPO. - **[2026-07]** Team-proposed algorithm [FlowGRPO with DiNa-LRM](https://verl-omni.readthedocs.io/en/latest/examples/flowgrpo_trainer_sd35_drm.html) is available. Training skips VAE decoding by scoring clean diffusion latents directly for faster and more resource-efficient model alignment. - **[2026-07]** VeRL-Omni is presented in QingKe AI, vLLM community, and verl x Ascend Beijing meetup. [Slides](https://drive.google.com/file/d/1RJBZZ2k6exxciFghU1FNUgds6s1w7Bpf/view?usp=sharing) are shared. - **[2026-06]** [Qwen3-Omni GSPO Trainer](examples/gspo_trainer) is available! [Flow-DPPO](https://verl-omni.readthedocs.io/en/latest/algo/flowdppo.html) is integrated. vLLM-Omni rollout backend is upgraded to v0.22 for higher throughput, with default actor attn backend switched to FA3. - **[2026-06]** [DiffusionNFT](https://verl-omni.readthedocs.io/en/latest/algo/diffusionnft.html) and [Diffusion DPO](https://verl-omni.readthedocs.io/en/latest/algo/diffusion_dpo.html) are integrated with verified recipes on Qwen-Image/SD3.5. [Wan2.2](examples/dancegrpo_trainer/README.md) is now supported for video generation tasks. ## Why `VeRL-Omni` Multimodal generative RL training differs from text-only LLM RL not only in model structure, but also in I/O patterns, compute characteristics, and runtime bottlenecks. As this space grows, it deserves a dedicated training repository that can evolve quickly around its own constraints. ### Scope `VeRL-Omni` targets RL post-training for three families of generative models: 1. **Diffusion generative models** for image, video, and audio — e.g., Qwen-Image, Wan2.2. 2. **Unified multimodal understanding + generation models** — e.g., BAGEL, HunyuanImage-3.0. 3. **Omni-modality models** that jointly handle text, image, audio, and video — e.g., Qwen3-Omni. ### What we focus on - **Fast multi-modal rollout:** Adopt [`vLLM-Omni`](https://github.com/vllm-project/vllm-omni) backend and accelerate generation via rollout routing, rollout batching, embed caching optimizations, and more. - **Flexible & async multi-reward serving:** Support multi-reward serving (HPSv3, GenRM-OCR, UnifiedReward, etc.), [HTTP scorer](https://verl-omni.readthedocs.io/en/latest/start/http_scorer.html), and [asynchronous reward computation](https://verl-omni.readthedocs.io/en/latest/algo/async_reward.html) to overlap the rollout phase. - **Modular training backends:** Selectable [VeOmni](https://github.com/ByteDance-Seed/VeOmni) and FSDP2 backends with combinable parallelism (USP/TP/DP) for distributed training. - **Stability:** Boost stability and speed in diffusion RL pipelines via [rollout correction](https://verl-omni.readthedocs.io/en/latest/algo/rollout_correction.html) to skip logP recomputation, and achieve reproducible E2E training with deterministic RL. Reward, rollout and actor update are composable and extensible, via Hydra configs. - **Efficient and convergent training recipes:** On our reference Qwen-Image FlowGRPO setup, `VeRL-Omni` achieves **~25% higher end-to-end throughput** than the diffusers-based [`flow_grpo`](https://github.com/yifan123/flow_grpo) implementation, driven by `vLLM-Omni` rollout, FSDP2 trainer, overlapped reward computation (asynchronous), etc.
verl-omni architecture diagram
## Getting Started 🚀 Visit our [documentation](https://verl-omni.readthedocs.io/en/latest/index.html) to learn more. - [Installation](https://verl-omni.readthedocs.io/en/latest/start/install.html) - [Quickstart](https://verl-omni.readthedocs.io/en/latest/start/flowgrpo_quickstart.html)
Training step 0 Training step 120
Training step 0 Training step 120

Example: Optimizing Qwen-Image Text Rendering Accuracy with FlowGRPO (recipe  |  wandb)

## Model and Algorithm Support 🎨
Model Category Modality Algorithm Status
Qwen-Image & Qwen-Image-Edit Diffusion generator Text/Image → Image FlowGRPO (+ CPS/SDE) ✅
Flow-DPPO ✅
MixGRPO ✅
GRPO-Guard ✅
DiffusionNFT ✅
DPO ✅
Wan2.2 Diffusion generator Text → Video DanceGRPO ✅
LTX2.3 Diffusion generator Text → Video + Audio FlowGRPO ✅
OmniNFT ✅
MiniMax-H3 Diffusion generator Any → Video + Audio DiffusionNFT ✅
FlowGRPO ✅
Boogu-Image Diffusion generator Text/Image → Image FlowGRPO (+ CPS/SDE) ✅
DiffusionNFT ✅
FLUX.1-dev Diffusion generator Text → Image DanceGRPO ✅
BAGEL Unified understand + gen Text + Image FlowGRPO ✅
SD3.5 Diffusion generator Text → Image DPO ✅
FlowGRPO ✅
FlowGRPO w/ DiNa-LRM ✅
DiffusionOPD (incl. MOPD) ✅
DGPO ✅
AWM ✅
HunyuanImage-3.0 Unified understand + gen Text + Image MixGRPO Planned
SRPO Planned
Qwen3-Omni-Thinker Omni-modality Text / Image / Video / Audio DPO ✅
GSPO ✅
OPD ✅
DAPO (Phase 1, LoRA) ✅
MiniCPM-o 4.5 Omni-modality Image / Audio → Text GSPO ✅
DPO ✅
Qwen3-TTS Audio-modality Text → Audio DPO WIP
GSPO WIP
GRPO ✅
## Ascend NPU Support 💠 `VeRL-Omni` supports Ascend A2, A3, and A5 NPUs. See the [NPU installation guide](docs/start/install_npu.md) for setup and the [Ascend NPU Quickstart Guide](https://verl-omni.readthedocs.io/en/latest/start/flowgrpo_quickstart_npu.html) for FlowGRPO training. ## Roadmap 🗺 Future work is tracked in [VeRL-Omni Q4 Roadmap](https://github.com/verl-project/verl-omni/issues/700) ## Contributing 🤝 Contributions are welcome. See the [contribution guide](CONTRIBUTING.md). **Join the Community**: Feel free to ask questions, provide feedback, and discuss with fellow users of VeRL-Omni in our [WeChat group](docs/assets/WeChat.jpg). ## Acknowledgement 🌟 `verl-omni` builds on the engineering foundations developed in [`verl`](https://github.com/verl-project/verl) and is closely aligned with multimodal inference systems such as [`vLLM-Omni`](https://github.com/vllm-project/vllm-omni). ## Citation 📚 If you find the project helpful, please cite and star ⭐ ```bibtex @misc{verlomni_github, title = {{VeRL-Omni: Easy, Fast, and Stable RL Training for Diffusion and Omni-Modality Models}}, author = {Yongxiang Huang and Cheung Kawai and Jingan Zhou and Yingshu Chen and {openYuanrong Team} and Xibin Wu}, year = {2026}, howpublished = {\url{https://github.com/verl-project/verl-omni}}, urldate = {2026-04-28} } ```