Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling

**Kien T. Pham¹, Yuhao Liu², Chong Hou Choi², Zhenqi He¹, Yanghao Wang¹, Chenyang Lei², Rui Liu², Qifeng Chen¹, Long Chen¹** ¹The Hong Kong University of Science and Technology ²Huawei Research tkpham@connect.ust.hk · longchen@ust.hk If you find UniAVC useful for your research, please consider giving a star ⭐ and citation 📝 !

--- ## 🎬 Demo

--- ## Overview UniAVC is a unified model that natively handles text-conditioned joint audio-video generation, bidirectional cross-modal synthesis, and instruction-guided editing of either or both modalities without relying on task-specific adapters. To achieve this versatility, the model utilizes a progressive training paradigm that first masters unimodal generation and editing capabilities before dedicating scarce paired datasets entirely to establishing cross-modal correspondence and temporal synchronization. The framework further introduces an asymmetric noise-level scheduling technique that derives decoupled per-branch noise trajectories and explicitly supervises extreme boundary states to enable intent-aware editing and post-inference self-refinement. Extensive empirical evaluations demonstrate that this unified approach delivers performance comparable to specialized, state-of-the-art baselines across a wide range of generation and editing benchmarks. ## Highlights ### Comparison with the state of the art One sample per editing family, applying the same instruction with UniAVC and with InstructAV2AV and MiniMax-H3. #### Subject Editing > Transform the man into a cyberpunk painting style with high-saturation neon colors and sharp digital illustration strokes, while keeping all characters, actions, and composition unchanged.

_Input	_UniAVC
_{InstructAV2AV}	_MiniMax-H3

#### Subject Addition > Add a person wearing white clothes, standing still in the center of the frame, looking at the camera with a blank expression.

_Input	_UniAVC
_{InstructAV2AV}	_MiniMax-H3

#### Subject Removal > Remove the woman wearing a white lab coat, a stethoscope, and looking confused or distressed from the source video, while preserving the background of a blurred office environment, a desk lamp, the wall calendar, and the shelving behind her.

_Input	_UniAVC
_{InstructAV2AV}	_MiniMax-H3

#### Speech Editing > Alter the character's dialogue to: "We were in the drum circle like, wow, this is where real musical magic happens right now!" while keeping visual elements and background sounds identical to the original scene.

_Input	_UniAVC
_{InstructAV2AV}	_MiniMax-H3

#### Background Editing > Replace the background with a private study at late night, with quiet city night lights visible outside the window, while keeping the person's image, clothing, actions, and contemplative head-down expression in the foreground completely consistent.

_Input	_UniAVC
_{InstructAV2AV}	_MiniMax-H3

### Editing versatility Each grid holds the source clip and the results of three different instructions applied to that same clip. #### Subject Editing Restyle the subject while preserving background, lighting and posture.

_Source	_{1. Racing suit instead of the T-shirt}
_{2. Astronaut pressure suit instead of the T-shirt}	_{3. Black wetsuit with blue shoulder panels}

#### Subject Addition Insert a new person into the shot.

_Source	_{1. Add a man with short brown hair}
_{2. Add a woman with long dark hair}	_{3. Add an older man with white hair and glasses}

#### Subject Removal Erase a person and reconstruct what was behind them. This family ships two examples with one instruction each.

_Source	_{1. Remove the man in the white shirt and dark tie}
_Source	_{2. Remove the woman with the blue patterned scarf}

#### Speech Editing Replace a line of dialogue with new words and matched lip-sync.

_Source	_{1. "Scroll back to the log from Tuesday afternoon."}
_{2. "That value should never have gone negative."}	_{3. "Look at the screen, the error code matches the original test."}

#### Background Editing Relocate the background scene, keeping subject appearance and expression intact.

_Source	_{1. A tranquil beach at dusk}
_{2. A rooftop at dusk, laundry and swifts}	_{3. A railway platform at dusk, a train pulling away}

### Joint generation

## Installation ```bash conda create -n uniavc python=3.10 -y conda activate uniavc # Install a torch build matching your CUDA, then the rest. pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128 pip install -r requirements.txt pip install -e . ``` **Tested environment:** Python 3.10, CUDA 12.8, PyTorch 2.8, on a single 80 GB GPU. ### Model weights `bash scripts/download_weights.sh` fetches the weights, plus Gemma-3 for text conditioning: ```bash bash scripts/download_weights.sh ``` Or by hand: ```bash pip install -U huggingface_hub # The released model. hf download TK3105/UniAVC model.safetensors \ --local-dir models/UniAVC # Gemma-3 text encoder. hf download google/gemma-3-12b-it-qat-q4_0-unquantized \ --local-dir models/gemma-3-12b-it-qat-q4_0-unquantized ``` Once downloaded, the weights are stored as follows: ``` models/ ├── UniAVC/ │ └── model.safetensors # the DiT + the VAEs, vocoder, post-modules └── gemma-3-12b-it-qat-q4_0-unquantized/ ``` ## Usage ### Joint audio-video editing Edit the video and its soundtrack from a single instruction. Each of the five families gets one clip and one instruction: ```bash # Subject Editing python inference/infer_tav2av.py \ --video samples/sample_1.mp4 \ --prompt "Replace the man's dark brown T-shirt with a blue cycling jersey with white stripes on the sleeves. Keep the pull-up bar, the beige wall and the lighting unchanged." \ -o sample_1_out.mp4 # Subject Addition python inference/infer_tav2av.py \ --video samples/sample_2.mp4 \ --prompt "Include a man in a grey coat seated in the front seat, saying: \"We should have left an hour ago. Nobody checked the traffic or the weather.\"" \ -o sample_2_out.mp4 # Subject Removal python inference/infer_tav2av.py \ --video samples/sample_3.mp4 \ --prompt "Remove the table lamp from the foreground and fill in the window and curtains behind it. Keep the man and his dialogue unchanged." \ -o sample_3_out.mp4 # Speech Editing python inference/infer_tav2av.py \ --video samples/sample_4.mp4 \ --prompt "Have her say: \"I did not expect the city to be this quiet at dawn.\"" \ -o sample_4_out.mp4 # Background Editing python inference/infer_tav2av.py \ --video samples/sample_5.mp4 \ --prompt "Replace the background with a warm, sunlit kitchen with wooden cabinets. Keep the man, his clothing and his speech unchanged." \ -o sample_5_out.mp4 ``` The two headline modes are extra flags on this script. Post-inference self-refinement: re-denoise only one branch against the other, held as an accepted reference. ```bash python inference/infer_tav2av.py \ --video samples/sample_4.mp4 \ --prompt "Have her say: \"I did not expect the city to be this quiet at dawn.\"" \ --post-refine audio --refine-lambda 0.5 \ -o sample_4_refined.mp4 ``` At the default `--refine-lambda 0.5` this runs `round(0.5 * 30) = 15` refinement steps. The count scales with lambda so the refine pass keeps the main pass's per-step resolution; `--refine-steps` overrides it. Intent-aware editing: hold one branch intact by conditioning, and let it constrain the other. ```bash python inference/infer_tav2av.py \ --video samples/sample_5.mp4 \ --prompt "The room is now a quiet library, with faint page turns and distant footsteps." \ --intent-aware video \ -o sample_5_audio_edit.mp4 ``` ### Joint audio-video generation No input media: both branches are generated from the prompt alone. ```bash python inference/infer_t2av.py \ --prompt "A man in a grey shirt sits at a wooden table in a quiet cafe, a window with soft daylight behind him and a few empty chairs further back. He says: \"We can start whenever you are ready, but let me finish this first.\" Static medium shot." \ -o sample_t2av.mp4 ``` ### Audio → video generation Given a soundtrack, generate the matching video; the input audio is preserved. ```bash python inference/infer_ta2v.py \ --audio samples/sample_4.mp4 \ --prompt "Medium close-up of a woman with long dark hair speaking directly to camera, standing in a sunlit kitchen with pale oak cabinets and a steaming kettle on the counter behind her, warm late-afternoon light through a side window, shallow depth of field, static shot." \ -o sample_4_ta2v.mp4 ``` ### Video → audio generation Given a video, generate the matching soundtrack; the input video is preserved. ```bash python inference/infer_tv2a.py \ --video samples/sample_1.mp4 \ --prompt "The man is out of breath after a set of pull-ups. A barbell is racked nearby in a bare gym, with voices and movement echoing off a hard wall." \ -o sample_1_tv2a.mp4 ``` ### Options Every option has a working default, and the commands above need nothing beyond `--video` / `--audio` / `--prompt`. **Output** | Option | Default | Notes | |---|---|---| | `--height`, `--width` | `704`, `1280` | Both must be multiples of 32. | | `--num-frames` | `121` | Must be 8k+1. | | `--frame-rate` | `24` | | | `-o`, `--output` | `_output.mp4` | | **Sampling and memory** | Option | Default | Notes | |---|---|---| | `--num-inference-steps` | `30` | Denoising steps. | | `--seed` | `42` | | | `--offload` | on | `--no-offload` keeps all weights resident; see *Hardware*. | | `--device` | `cuda` | | | `--no-tiled` | off | Decode the video without VAE tiling. Needs more VRAM. | **Guidance**, set per branch. | Task | `video-text` | `audio-text` | `video-reference` | `audio-reference` | |---|---|---|---|---| | `tav2av` | 3.0 | 2.0 | 1.5 | 3.0 | | `t2av`, `ta2v`, `tv2a` | 6.0 | 6.0 | 1.0 | 1.0 | The flags are `--video-text-cfg-scale`, `--audio-text-cfg-scale`, `--video-reference-cfg-scale` and `--audio-reference-cfg-scale`. ## Demo A Gradio app is included: ```bash python app.py ``` ## Hardware The released checkpoint is a 19B-parameter DiT in bfloat16, and text conditioning adds Gemma-3 12B. The two together are ~65 GB, so weights are offloaded by default: each phase loads only what it needs (the text encoder while encoding, the DiT while denoising, the VAEs while decoding) and the rest waits on CPU. Peak residency is the DiT alone, ~39 GB, plus activations. `--no-offload` keeps everything on the card for speed and needs substantially more VRAM than a single 80 GB GPU provides at the default 720p resolution. ## Acknowledgements UniAVC is built on [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio), whose model-loading framework and LTX-2.3 implementation the vendored `uniavc/` package derives from. The base generative model is [LTX-2.3](https://github.com/Lightricks/LTX-Video) from Lightricks. Text conditioning uses [Gemma-3](https://huggingface.co/google/gemma-3-12b-it-qat-q4_0-unquantized) from Google. We thank [JAVEdit](https://github.com/RyanChenYN/JAVEdit) for the useful training data and their benchmark for evaluation; [MiniMax-H3](https://www.minimax.io/), [InstructAV2AV](https://github.com/suimuc/InstructAV2AV) for the comparison baselines; and the wider open-source audio-video community that make this work complete. ## Citation If you find UniAVC useful for your research, please consider giving a star ⭐ and citation 📝 :D ```bibtex @article{pham2026uniavc, title={UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling}, author={Pham, Kien T and Liu, Yuhao and Choi, Chong Hou and He, Zhenqi and Wang, Yanghao and Lei, Chenyang and Liu, Rui and Chen, Qifeng and Chen, Long}, journal={arXiv preprint arXiv:2610.24791}, year={2026} } ``` ## License UniAVC's original contributions are released under CC BY-NC 4.0, for non-commercial use with attribution. Third-party components retain their own licenses. The released model weights are a derivative of LTX-2.3 and are governed separately by the [LTX-2 Community License](https://github.com/Lightricks/LTX-2/blob/main/LICENSE-2).