# FlowGRPO Trainer

Last updated: 08/26/2026

This example shows how to post-train `Qwen-Image` with FlowGRPO on an OCR-style image generation task using `vllm-omni` rollout and a visual generative reward model (`Qwen3-VL-8B-Instruct` in this example).

For the full installation and quickstart guide, see [Quickstart: FlowGRPO training on OCR dataset](../../docs/start/flowgrpo_quickstart.md). For algorithm details and rule-based reward training (e.g. JPEG incompressibility), see [Algorithms - Flow-GRPO](../../docs/algo/flowgrpo.md).

## Installation

Follow the [installation guide](../../docs/start/install.md) to set up the base environment, then install the FlowGRPO-specific dependency:

```bash
uv pip install -e ".[ocr]"
```

The provided script is configured for a single node with `4` GPUs.

## Prepare the dataset

Obtain the raw OCR dataset from the original Flow-GRPO repository:

- https://github.com/yifan123/flow_grpo/tree/main/dataset/ocr

Place the raw dataset under `$WORKSPACE/data/ocr` (where `WORKSPACE` defaults to `$HOME`), then preprocess it into parquet files:

```bash
python3 examples/flowgrpo_trainer/data_process/qwenimage_ocr.py \
  --input_dir $WORKSPACE/data/ocr \
  --output_dir $WORKSPACE/data/ocr/qwen_image
```

This produces:

- `$WORKSPACE/data/ocr/qwen_image/train.parquet`
- `$WORKSPACE/data/ocr/qwen_image/test.parquet`

## Prepare the models

**Policy model (Qwen-Image):** the script uses the Hugging Face Hub ID `Qwen/Qwen-Image` directly — no manual download is required. Hugging Face will cache the weights automatically on first run. To use a local copy instead, edit the `model_name` variable in the script directly.

**Reward model (Qwen3-VL-8B-Instruct):** the script defaults to the Hugging Face Hub ID `Qwen/Qwen3-VL-8B-Instruct`, so no manual download is required — Hugging Face will cache it automatically on first run. To use a local copy instead, edit the `reward_model_name` variable in the script directly.

## Run training

Launch the example from the repository root. The default is the V1 trainer
(TransferQueue + ReplayBuffer, sync mode; `trainer.use_v1=true` since v0.3.0):

```bash
bash examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_lora_v1.sh
```

The legacy v0 script below is **deprecated** (a `DeprecationWarning` is emitted
at launch) and remains only until the v0 trainer is removed:

```bash
bash examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_lora.sh
```

GPU training defaults to matched kernels FA3 in config (`attn_backend: _flash_3_varlen_hub`;
`rollout_attn_backend: FLASH_ATTN_3_HUB`). Missing FA3 deps fail fast; install them or
switch to native/`TORCH_SDPA` explicitly.

Optional KL loss tuning:

- `actor_rollout_ref.actor.use_kl_loss=True`
- `actor_rollout_ref.actor.kl_loss_coef=0.001`

The script runs `python3 -m verl_omni.trainer.main_diffusion` with:

- `algorithm.adv_estimator=flow_grpo`
- `actor_rollout_ref.model.path=Qwen/Qwen-Image`
- `actor_rollout_ref.model.lora_rank=64`
- `actor_rollout_ref.model.lora_alpha=128`
- `actor_rollout_ref.rollout.name=vllm_omni`
- `reward.custom_reward_function.name=compute_score_ocr`
- `trainer.n_gpus_per_node=4`

The V1 script uses `python3 -m verl_omni.trainer.main_diffusion_v1` with the same
model/LoRA/reward knobs plus `trainer.use_v1=true` and `trainer.v1.trainer_mode=sync`
(both defaults since v0.3.0),
including the v0 micro-batch sizes: between phases the v1 trainer sleeps the rollout
engine, and a vllm-omni level-1 sleep offloads the whole pipeline (transformer + text
encoder + VAE) to pinned host memory, so the actor update runs next to only a few GB
of engine skeleton instead of ~55GB of resident weights. Both micro-batch knobs are
env-overridable (`PPO_MICRO_BATCH_SIZE`, `LOG_PROB_MICRO_BATCH_SIZE`) if you colocate
other tenants on the same GPUs. Each sleeping replica pins ~55GB of host RAM
(220GB total for the four TP=1 replicas of this recipe).

## Logging

W&B logging is enabled by default in the example script:

```bash
export WANDB_API_KEY=<your_wandb_api_key>
```

The script sets:

```bash
trainer.logger='["console", "wandb"]'
trainer.project_name=flow_grpo
trainer.experiment_name=qwen_image_ocr_lora
```

Override these values on the command line if you want to log under a different project or run name.

### Diffusion-specific metrics

See the [Metrics Documentation](../../docs/start/metrics.md) for a full description of all diffusion-specific training metrics.

## Variants

All example scripts in this directory:

### Qwen-Image

| Variant | Script | GPUs | Notes |
|---------|--------|------|-------|
| LoRA (v0, deprecated) | `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_lora.sh` | 4×GPU | Standard LoRA FlowGRPO + OCR reward |
| LoRA (V1 sync) | `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_lora_v1.sh` | 4×GPU | V1 trainer (`main_diffusion_v1`, TransferQueue + ReplayBuffer) |
| LoRA + async reward | `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_lora_async_reward.sh` | 5×GPU | Dedicated GPU pool for reward model |
| LoRA + SP=2 | `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_lora_sp2.sh` | 4×GPU | Ulysses sequence parallelism |
| LoRA + FA3 | `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_lora_fsdp2_fa3.sh` | 4×GPU | FSDP2 with FlashAttention 3 |
| LoRA + rollout corr | `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_lora_rollout_corr.sh` | 4×GPU | Bypass old-log-prob recompute |
| LoRA + multi-node | `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_lora_multi_node.sh` | 2×4 GPU | Horizontal scaling across nodes |
| LoRA + NPU | `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_lora_npu.sh` | 8×NPU | Ascend NPU LoRA training |
| Full model | `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr.sh` | 4×H200 | Non-CFG full-weight training |
| Full + FSDP2 64-card | `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_fsdp2_64cards.sh` | 64×GPU | Large-scale FSDP2 |
| Full + VeOmni | `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_veomni.sh` | 64×H100 | VeOmni engine backend |
| Full + VeOmni 64-card | `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_veomni_64cards.sh` | 64×GPU | VeOmni at scale |
| Full + H200 MFU opt | `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_h200_mfu_optimized.sh` | H200 | MFU-optimized config |
| Full + NPU | `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_npu.sh` | 16×NPU | Ascend NPU full model |
| HTTP reward server | `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_reward_server.sh` | 4×GPU | External HTTP OCR reward service |
| Multi-reward | `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_multi_reward.sh` | 4×GPU | Multiple reward functions combined |

### Stable Diffusion 3.5 Medium

| Variant | Script | GPUs | Notes |
|---------|--------|------|-------|
| LoRA | `examples/flowgrpo_trainer/sd35/run_sd35_medium_ocr_lora.sh` | 3×GPU | 2 actor+rollout, 1 reward pool |

### LTX-2.3

| Variant | Script | GPUs | Notes |
|---------|--------|------|-------|
| T2AV LoRA (V1 Sync) | `examples/flowgrpo_trainer/ltx2/run_ltx2_3_t2av_lora_v1.sh` | 8×GPU | V1 sync trainer with TransferQueue + ReplayBuffer |
| T2AV LoRA | `examples/flowgrpo_trainer/ltx2/run_ltx2_3_t2av_lora.sh` | 8×GPU | Joint audio-video CPS, CLAP + ImageBind rewards |
| TI2VA LoRA (V1 sync) | `examples/flowgrpo_trainer/ltx2/run_ltx2_3_ti2va_lora_v1.sh` | 8×GPU | First-frame-conditioned joint audio-video CPS with TransferQueue + ReplayBuffer |

### BAGEL

| Variant | Script | GPUs | Notes |
|---------|--------|------|-------|
| LoRA (OCR) | `examples/flowgrpo_trainer/bagel/run_bagel_ocr_lora.sh` | 4×GPU | OCR reward (Qwen3-VL) |
| LoRA (PickScore) | `examples/flowgrpo_trainer/bagel/run_bagel_pickscore_lora.sh` | 4×GPU | CLIP-based image-text alignment |

### Engine backends

The diffusion trainer defaults to **FSDP2**. To use [VeOmni](https://github.com/ByteDance-Seed/VeOmni) instead, follow the [VeOmni install instructions](../../docs/start/install.md#optional-engine-backends), then run the VeOmni counterpart:

```bash
bash examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_veomni.sh
```

`run_qwen_image_ocr_veomni.sh` mirrors `run_qwen_image_ocr.sh` and differs only in engine-selection overrides:

| Override | FSDP2 (default) | VeOmni |
|---|---|---|
| `diffusion/model_engine=` | _(unset; uses `dp_diffusion`)_ | `veomni_diffusion` |
| `actor_rollout_ref.actor.strategy=` | `fsdp2` | `veomni` |
| Actor engine config block | `actor.fsdp_config.*` | `actor.veomni_config.*` |
| Sequence-parallel field | `fsdp_config.ulysses_sequence_parallel_size` | `veomni_config.ulysses_parallel_size` |
| Ref engine config block | `ref.fsdp_config.*` | `ref.veomni_config.*` |

Do not mix `fsdp_config` and `veomni_config` overrides — `diffusion/model_engine=...` selects the schema, and overrides for the other engine are rejected.

## Performance

> All experiments were conducted on *NVIDIA H800* GPUs using the OCR reward.

The experiment settings and throughputs are shown in the table below.

| Script | Model | Algorithm | Hybrid Engine | # Cards | Reward Fn | # GPUs for Actor | # GPUs for Rollout | # GPUs for Async Reward | Batch Size | `rollout.n` | lr   | # Val Samples | Training Samples per Step | `ppo_micro_batch_size_per_gpu` | Throughput (Samples / GPU / Seconds) | Time per Step (Seconds) |
| --- | --- | --- | --- | --- | --- | --- | --- |-------------------------| --- | --- |------| --- | --- | --- |------------------------------| --------------------------------|
| `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_lora.sh` | Qwen-Image | Flow-GRPO | True | 4 | qwenvl-ocr-vllm | 4 | 4 | 0 (sync)                | 32 | 16 | 3e-4 | 1k (full set) | 32×16=512 | 16 | 0.305                        | 420 |
| `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_lora_async_reward.sh` | Qwen-Image | Flow-GRPO | True | 5 | qwenvl-ocr-vllm | 4 | 4 | 1                       | 32 | 16 | 3e-4 | 1k (full set) | 32×16=512 | 16 | 0.280                        | 360 |

- Validation reward curve (evaluated with `trainer.val_before_train=True`):

<div align="center">
<img width="600" alt="2p_comparison" src="https://github.com/user-attachments/assets/1094beaf-fed9-4661-8a6a-1c3983150648" />
<br>
qwen_image_ocr_lora: corresponding with the script `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_lora.sh`; 
<br>
qwen_image_ocr_lora_async_reward: corresponding with the script `examples/flowgrpo_trainer/qwen_image/run_qwen_image_ocr_lora_async_reward.sh`.
</div>

> **Note:** Reward curves may differ from the references above mainly due to rollout-side stochasticity: diffusion rollouts sample random latents/noise, and the example scripts do not fix the data seed, so prompt ordering can vary between runs.


## Further Reading
For more examples of FlowGRPO training other models, see
- Stable Diffusion: [Quickstart: FlowGRPO training on OCR dataset](../../docs/start/flowgrpo_quickstart.md)
- Image-to-image model: [Examples - Qwen-Image-Edit-2511 FlowGRPO training](https://verl-omni.readthedocs.io/en/latest/examples/qwen_image_edit/flowgrpo_trainer_qwen_image_edit.html)
- Unified image understanding and generation model: [Examples - BAGEL-7B-MoT FlowGRPO training](https://verl-omni.readthedocs.io/en/latest/examples/bagel/flowgrpo_trainer_bagel.html)

To train a model with a latent reward model, see [Examples - SD3.5 FlowGRPO training with a latent reward model](../../docs/examples/flowgrpo_trainer_sd35_drm.md).