Efficient, stable and scalable reinforcement of diffusion models
Text-to-image · Text-to-video · Image-to-video · Omni
Method / Quick Start / FAQ / Documentation
## News - β‘ **[2026.10.25]** **[Diffusion-ART-Flash](#diffusionart-flash) is supported.** Training only high-noise timesteps achieves comparable performance to all-timesteps training with substantially less training time cost. - π **[2026.10.22]** [MiniMax-H3](#minimax-h3) is now supported for **text-to-audio/video (omni)** training, with [DiffusionART post-training recipes](#minimax-h3-recipes). - π₯ **[2026.10.12]** DiffusionART ranked **#1 on [Hugging Face Daily Papers](#paper)**!
Fine-tuning HunyuanVideo (13B) (left) and Wan2.1-I2V-14B-480P, CFG/step-distilled (14B) (right) using VideoReward, with Visual Quality (VQ):Motion Quality (MQ) = 1:1. Mean train reward is the mean of raw VQ and MQ. T2V ratios compare GPU hours to reach each baseline's final reward level.
## Method ### Motivations and Insights 1. NFT-style forward-process RL contains a hidden **forward-reverse consistency force** in its objective, slowing and destabilizing training. 2. Off-policy strategies stabilize NFT-style diffusion RL through a **strong restoration force**. Even a small output-space restoration MSE can produce a large aggregated parameter gradient. ### Aggregation-First Guidance and Adaptive Restoration DiffusionART is simple: roll out **K samples per prompt** with an ODE solver, compute the reward-endpoint covariance as **guidance**, apply it to the visited states and actions, then perform **two disjoint minibatch updates**. We call this reward-improving direction **guidance**, bringing a classifier-free guidance (CFG)-like idea into diffusion RL: the model learns to internalize reward guidance, much like distilling CFG. For a reward-tilted endpoint distribution with tilt strength eta,
| Guidance geometry and state assignment | Pseudocode |
|---|---|
|
Implementation: covariance guidance, ART loss, and disjoint update objectives. |
Pairwise T2V with K = 2 or 8 videos per prompt. Left: VQ train reward versus GPU hours. Right: final evaluation VQ and total GPU hours for pairwise K = 2, 4, 8 and scalar K = 8, each trained for 400 updates.
### DiffusionART-Flash **DiffusionART-Flash** trains on high-noise timesteps. For video tasks, *Our recommended recipe is to randomly select 2 timestep transitions for noise level between 0.95 to 1 (very high-noise region) at training stage*. For our Hunyuan-Video experiment, we **randomly selects two of the first four timesteps** for each rollout. It processes far fewer sampleβtimestep pairs during policy updates while achieving comparable performance to training across the full denoising schedule. Rollout stage still follows the complete trajectory to guarantee the sample quality. Results:
HunyuanVideo, VQ:MQ = 1:1. Raw training scores. Training only two timesteps reduces the training time cost by 8x compared with training 16 timesteps.
Enable with `training.timestep_selection: high_noise`. See the [Flash guide and example configs](docs/art-flash.md). ### Key Advantages - β **ODE solver:** deterministic rollouts, with no reverse-transition likelihood. - β **Rollout efficiency:** reuse rollout states and cached predictions, without adding noise again or recomputing predictions on newly noised states. - β **Sample efficiency:** fewer samples per prompt with minimal performance loss and lower GPU-hour cost. - β **Training efficiency:** remove forward-process RL's unstable self-consistency force and address reverse-process RL's inefficient credit assignment. - β **Simple and scalable:** minimal hyperparameter tuning, no differentiable rewards required, and natural support for pairwise reward models. ## Visualization Results
Pre-trained versus DiffusionART. Video strips show four frames.
## Released Models | Task | Model | Release | | --- | --- | --- | | Text-to-image | SD3.5-M Β· Multi-Reward | [Hugging Face](https://huggingface.co/vanzl/DiffusionART-SD35M-MultiReward) | | Text-to-video | HunyuanVideo-13B | [Hugging Face](https://huggingface.co/vanzl/DiffusionART-HunyuanVideo) | Each model card includes its inference setup, base-model requirements and examples. ## Quick Feature Summary | Feature Category | Supported Capabilities | | --- | --- | | **Tasks** | β Text-to-image
Full-parameter training. Six individual reward components before prompt-group normalization.
The curves above use **Full64 (CP2)**. Full32 uses a smaller update microbatch. **Note:** Open-source reward models cannot fully capture human preferences for visual outputs. As a result, reward scores can improve even when the generated videos remain visually unsatisfactory. For research purposes, we suggest using reward scores as the primary evaluation metric; for industrial applications, we recommend using higher-quality reward models. See [setup and launch instructions](docs/minimax-h3.md#480p-ten-step-art-post-training): | GPUs | Training | Rollout / update microbatch | Recipe | |---:|---|---:|---| | 32 | Full parameters | 8 / 4 | [Full32](configs/examples/minimax_h3_art_posttrain_full_32gpu.yaml) | | 64 | Full parameters, CP2 | 8 / 8 | [Full64](configs/examples/minimax_h3_art_posttrain_full_64gpu_cp2.yaml) | | 8 | LoRA (rank 32) | 1 / 1 | [LoRA8](configs/examples/minimax_h3_art_posttrain_lora_8gpu.yaml) | ## Configure Your Experiment Extend an [example](configs/examples/) or [paper recipe](configs/paper/). Save this as `configs/my-pickscore.yaml`: ```yaml extends: examples/sd3_pickscore.yaml name: my-pickscore training: max_updates: 1000 runtime: output_dir: outputs/my-pickscore checkpoint_keep: 2 ``` ```bash uv run --locked --extra diffusers visual-rl --config configs/my-pickscore.yaml ``` Select components through `model.name`, `algorithm.name` and `reward.name`; compose objectives with `reward.weights`. Sampling, microbatches, evaluation and checkpoint cadence live in the same file. Required asset paths use `${ENV_VAR}`. Metrics are saved to `metrics.jsonl`; evaluation images, videos and scores stay on disk. Set `runtime.log: wandb` for [scalar logging](docs/logging.md). ## Implementation Notes and Discussion **Rollout diversity and sampler compatibility.** ART's frozen-target regression also applies to SDE transition means. For an affine mean aΒ·vΞΈ + b, where a is nonzero and a and b are independent of ΞΈ, matching cached states and corresponding frozen targets gives the same per-transition gradient direction, with magnitude scaled by aΒ². Dividing each mean-space MSE by aΒ² recovers the velocity-space gradient when timestep weights, restoration and masks match. Without normalization, timestep-dependent scaling can also change the aggregate gradient direction. SDE sampling changes rollout diversity and distribution; current public ART recipes use ODE rollouts. **Covariance normalization.** Normalize the aggregated covariance before constructing the regression target. This controls the guidance scale independently of the raw covariance magnitude. For multiple rewards, combine their weighted contributions before normalization; normalizing each contribution separately changes their relative influence. **Prompt diversity and batch size.** When compute allows, we recommend covering more distinct prompts per rollout. In our experience, broader prompt coverage often reduces gradient variance and improves training stability. Increase `sampling.prompts_per_rollout` to adjust this budget; keep the intended number of samples per prompt and optimizer updates per rollout explicit. ART computes guidance from each complete group of K trajectories before splitting the training data. Both disjoint update batches contain K/2 trajectories from every prompt: for example, two batches of 32 prompts with K/2 trajectories each, rather than 16 prompts with K trajectories each. Two-update ART requires even K. Each trajectory keeps all its selected training timesteps; zero-guidance and masked rows remain in the original loss denominator. Images use a random split within each prompt; videos retain trajectory order. This changes older image splits and multi-prompt-per-rank video splits, not the historical video split with one prompt per rank. It is not a claim that this split is empirically optimal. With limited memory, reduce **microbatch size, not the logical update batch**. Lower `sampling.microbatch` to generate the same rollout samples in more sequential chunks. Lower `training.microbatch` to accumulate gradients over more chunks within the same update window; the runner handles this automatically without changing the loss denominator or adding optimizer steps. Use the largest microbatches that fit comfortably for better GPU utilization and throughput. Shrinking the **minibatch/update window** instead creates more optimizer updates per rollout and additional opportunities for off-policy restoration, changing the algorithm's training dynamics rather than merely saving memory. **Reuse rollout predictions.** ART trains on cached ODE trajectory states and reuses their old-policy predictions, avoiding the additional old-policy forward pass on re-noised states required by our NFT implementation. Current-policy forward and backward passes are still required. This reduces per-rollout compute; the overall speedup also depends on the number of updates needed to reach a target reward. **Reward quality matters.** As with ReFL and other reward-based fine-tuning methods, DiffusionART depends on how well the reward model captures the desired behavior. Our experiments use open-source reward models. For practical applications, we recommend higher-quality reward models better aligned with the intended use case: DiffusionART improves rewards quickly, which can also expose weaknesses in the reward model and lead to reward hacking sooner. In our experiments, when different algorithms reach a comparable score on an optimized dimension, they tend to exhibit similar levels of degradation in other dimensions. This suggests a shared limitation of the reward model rather than a failure unique to DiffusionART. ## Learn More [FAQ: models, black-box rewards, pairwise feedback and on-policy training](docs/faq.md) [Documentation](docs/README.md) / [Development](docs/development.md) / [Model interfaces](docs/model-contract.md) / [Reward interfaces](docs/reward-contract.md) ## Acknowledgements Thanks to the authors of [Flow-GRPO](https://github.com/yifan123/flow_grpo), [DiffusionNFT](https://github.com/NVlabs/DiffusionNFT), [DanceGRPO](https://github.com/XueZeyue/DanceGRPO) and [UniRL](https://github.com/Tencent-Hunyuan/UniRL) for open-sourcing their work. ## License The code is released under the [Apache License 2.0](LICENSE). Third-party code, model weights and datasets retain their respective licenses; see [Third-Party Notices](THIRD_PARTY_NOTICES.md). ## Citation If you find DiffusionART inspiring or useful for your projects, please consider starring this repository and citing our paper: ```bibtex @misc{wan2026diffusionartefficientstablescalable, title={DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models}, author={Zhenglin Wan and Jiayu Chen and Zhengfeiyang Zhang and Pengfei Zhou and Wangbo Zhao and Fangyuan Kong and Pengfei Wan and Kun Gai and Yang You}, year={2026}, eprint={2610.12572}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/2610.12572}, } ``` ## Star History