---
description: "Generate clip captions with Qwen-VL or Nemotron VLMs and optional preview images"
categories: ["video-curation"]
tags: ["captions", "qwen", "nemotron", "preview", "video"]
personas: ["data-scientist-focused", "mle-focused"]
difficulty: "intermediate"
content_type: "howto"
modality: "video-only"
---

# Captions and Preview

Prepare inputs, generate captions, optionally enhance them, and produce preview images.

## Choosing a Captioning Model

The video captioning pipeline supports two model families. Pick a variant based on quality, GPU memory, and throughput:

| Variant | Model | Default Use Case |
| --- | --- | --- |
| `qwen` | `Qwen/Qwen2.5-VL-7B-Instruct` | Default — good quality/throughput balance |
| `nemotron` / `nemotron-bf16` | Nemotron Nano 12B v2 VL (BF16) | High-quality captions; auto-downloaded from Hugging Face |
| `nemotron-fp8` | Nemotron Nano 12B v2 VL (FP8) | Same model, FP8-quantized for lower memory |
| `nemotron-nvfp4` | Nemotron Nano 12B v2 VL (NVFP4-QAD) | NVFP4 quantization-aware-distilled checkpoint |

Caption **enhancement** (the optional second-pass LLM rewrite) uses Qwen-LM (`--enhance-captions-algorithm qwen_lm`).

---

## Quickstart

Use the pipeline stages or the example script flags to prepare captions and preview images.

<Tabs>
<Tab title="Pipeline Stage">

```python
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.video.caption.caption_preparation import CaptionPreparationStage
from nemo_curator.stages.video.caption.caption_generation import CaptionGenerationStage
from nemo_curator.stages.video.caption.caption_enhancement import CaptionEnhancementStage
from nemo_curator.stages.video.preview.preview import PreviewStage

pipe = Pipeline(name="captions_preview")
pipe.add_stage(
    CaptionPreparationStage(
        model_variant="qwen",
        prompt_variant="default",
        prompt_text=None,
        sampling_fps=2.0,
        window_size=256,
        remainder_threshold=128,
        preprocess_dtype="float16",
        model_does_preprocess=False,
        generate_previews=True,
        verbose=True,
    )
)
pipe.add_stage(PreviewStage(target_fps=1.0, target_height=240, verbose=True))
pipe.add_stage(
    CaptionGenerationStage(
        model_dir="/models",
        model_variant="qwen",
        caption_batch_size=8,
        fp8=False,
        max_output_tokens=512,
        model_does_preprocess=False,
        generate_stage2_caption=False,
        stage2_prompt_text=None,
        disable_mmcache=True,
    )
)
pipe.run()
```

To use Nemotron instead, set `model_variant="nemotron"` (or one of `nemotron-bf16`, `nemotron-fp8`, `nemotron-nvfp4`) on both `CaptionPreparationStage` and `CaptionGenerationStage` — Nemotron weights are auto-downloaded from Hugging Face on first use.

</Tab>

<Tab title="Script Flags">

```bash
python tutorials/video/getting-started/video_split_clip_example.py \
  ... \
  --generate-captions \
  --captioning-algorithm qwen \
  --captioning-window-size 256 \
  --captioning-remainder-threshold 128 \
  --captioning-sampling-fps 2.0 \
  --captioning-preprocess-dtype float16 \
  --captioning-batch-size 8 \
  --captioning-max-output-tokens 512 \
  --generate-previews \
  --preview-target-fps 1.0 \
  --preview-target-height 240
```

`--captioning-algorithm` accepts: `qwen` (default, Qwen2.5-VL-7B-Instruct), `nemotron`, `nemotron-bf16`, `nemotron-fp8`, `nemotron-nvfp4`. To enable caption enhancement with the Qwen LM, also pass `--enhance-captions --enhance-captions-algorithm qwen_lm`.

</Tab>
</Tabs>

## Preparation and previews

1. Prepare caption inputs from each clip window. This step splits clips into fixed windows, formats model‑ready inputs for the chosen VLM (Qwen‑VL or Nemotron), and optionally stores per‑window `mp4` bytes for previews.

   ```python
   from nemo_curator.stages.video.caption.caption_preparation import CaptionPreparationStage
   from nemo_curator.stages.video.preview.preview import PreviewStage

   prep = CaptionPreparationStage(
       model_variant="qwen",  # or "nemotron" / "nemotron-fp8" / ...
       prompt_variant="default",
       prompt_text=None,
       sampling_fps=2.0,
       window_size=256,
       remainder_threshold=128,
       preprocess_dtype="float16",
       model_does_preprocess=False,
       generate_previews=True,
       verbose=True,
   )
   ```

2. Optionally generate `.webp` previews from each window’s `mp4` bytes for quick QA and review.

   ```python
   preview = PreviewStage(
       target_fps=1.0,
       target_height=240,
       verbose=True,
   )
   ```

### Parameters

<Tabs>
<Tab title="CaptionPreparationStage">

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `model_variant` | str | `"qwen"` | Vision‑language model used to format inputs. One of `qwen`, `nemotron`, `nemotron-bf16`, `nemotron-fp8`, `nemotron-nvfp4`. |
| `prompt_variant` | {"default", "av", "av-surveillance"} | `"default"` | Built‑in prompt to steer caption content when `prompt_text` is not provided. |
| `prompt_text` | str \| None | `None` | Custom prompt text. When set, overrides `prompt_variant`. |
| `sampling_fps` | float | 2.0 | Source sampling rate for creating per‑window inputs. |
| `window_size` | int | 256 | Number of frames per window before captioning. |
| `remainder_threshold` | int | 128 | Minimum leftover frames required to create a final shorter window. |
| `model_does_preprocess` | bool | `False` | Whether the downstream model performs its own preprocessing. |
| `preprocess_dtype` | str | `"float32"` | Data type for any preprocessing performed here. |
| `generate_previews` | bool | `True` | When `True`, return per‑window `mp4` bytes to enable preview generation. |
| `verbose` | bool | `False` | Log additional setup and per‑clip details. |

</Tab>
<Tab title="PreviewStage">

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `target_fps` | float | 1.0 | Frames per second for preview encoding. |
| `target_height` | int | 240 | Output height in pixels; width auto‑scales to preserve aspect ratio. |
| `compression_level` | int (0–6) | 6 | WebP compression level (`0` = lossless, higher = smaller files). |
| `quality` | int (0–100) | 50 | WebP quality factor (`100` = best quality, larger files). |
| `num_cpus_per_worker` | float | 4.0 | CPU threads mapped to `ffmpeg -threads` for encoding. |
| `verbose` | bool | `False` | Log warnings and per‑window encoding details. |

</Tab>

</Tabs>

## Caption generation and enhancement

1. Generate window‑level captions with the chosen VLM (Qwen‑VL or Nemotron). This stage reads `clip.windows[*].qwen_llm_input` (created earlier) and writes `window.caption["qwen"]` (or `window.caption["nemotron"]`, depending on the variant).

   ```python
   from nemo_curator.stages.video.caption.caption_generation import CaptionGenerationStage
   from nemo_curator.stages.video.caption.caption_enhancement import CaptionEnhancementStage

   gen = CaptionGenerationStage(
       model_dir="/models",
       model_variant="qwen",  # or "nemotron" / "nemotron-fp8" / ...
       caption_batch_size=8,
       fp8=False,
       max_output_tokens=512,
       model_does_preprocess=False,
       generate_stage2_caption=False,
       stage2_prompt_text=None,
       disable_mmcache=True,
   )

   ```

2. Optionally enhance captions with a text‑based LLM (Qwen‑LM) to expand and refine descriptions. This stage reads `window.caption["qwen"]` and writes `window.enhanced_caption["qwen_lm"]`.

   ```python
   enh = CaptionEnhancementStage(
       model_dir="/models",
       model_variant="qwen",
       prompt_variant="default",
       prompt_text=None,
       model_batch_size=128,
       fp8=False,
       max_output_tokens=512,
       verbose=True,
   )
   ```

### Parameters

<Tabs>
<Tab title="CaptionGenerationStage">

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `model_dir` | str | `"models/qwen"` | Directory for model weights; downloaded on each node if missing. |
| `model_variant` | str | `"qwen"` | Vision‑language model variant. One of `qwen`, `nemotron`, `nemotron-bf16`, `nemotron-fp8`, `nemotron-nvfp4`. |
| `caption_batch_size` | int | 16 | Batch size for caption generation. |
| `fp8` | bool | `False` | Use FP8 weights when available. |
| `max_output_tokens` | int | 512 | Maximum number of tokens to generate per caption. |
| `model_does_preprocess` | bool | `False` | Whether the model performs its own preprocessing. |
| `disable_mmcache` | bool | `False` | Disable multimodal cache for generation backends that support it. |
| `generate_stage2_caption` | bool | `False` | Enable a second‑pass caption for refinement. |
| `stage2_prompt_text` | str \| None | `None` | Custom prompt for stage‑2 caption refinement. |
| `verbose` | bool | `False` | Emit additional logs during generation. |

</Tab>

<Tab title="CaptionEnhancementStage">

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `model_dir` | str | `"models/qwen"` | Directory for language‑model weights; downloaded per node if missing. |
| `model_variant` | {"qwen"} | `"qwen"` | Language‑model variant. |
| `prompt_variant` | {"default", "av-surveillance"} | `"default"` | Built‑in enhancement prompt when `prompt_text` is not provided. |
| `prompt_text` | str \| None | `None` | Custom enhancement prompt. When set, overrides `prompt_variant`. |
| `model_batch_size` | int | 128 | Batch size for enhancement generation. |
| `fp8` | bool | `False` | Use FP8 weights when available. |
| `max_output_tokens` | int | 512 | Maximum number of tokens to generate per enhanced caption. |
| `verbose` | bool | `False` | Emit additional logs during enhancement. |

</Tab>
</Tabs>

## Preview Generation

Generate lightweight `.webp` previews for each caption window to support review and QA workflows. A dedicated `PreviewStage` reads per-window `mp4` bytes and encodes WebP using `ffmpeg`.

### Preview Parameters

- `target_fps` (default `1.0`): Target frames per second for preview generation.
- `target_height` (default `240`): Output height. Width auto-scales to preserve aspect ratio.
- `compression_level` (range `0–6`, default `6`): WebP compression level. `0` is lossless; higher values reduce size with lower quality.
- `quality` (range `0–100`, default `50`): WebP quality. Higher values increase quality and size.
- `num_cpus_per_worker` (default `4.0`): Number of CPU threads mapped to `ffmpeg -threads`.
- `verbose` (default `False`): Emit more logs.

Behavior notes:

- If the input frame rate is lower than `target_fps` or the input height is lower than `target_height`, the stage logs a warning and preview quality can degrade.
- If `ffmpeg` fails, the stage logs the error and skips assigning preview bytes for that window.

### Example: Configure PreviewStage

```python
from nemo_curator.stages.video.preview.preview import PreviewStage

preview = PreviewStage(
    target_fps=1.0,
    target_height=240,
    compression_level=6,
    quality=50,
    num_cpus_per_worker=4.0,
    verbose=False,
)
```

### Outputs

The stage writes `.webp` files under the `previews/` directory that `ClipWriterStage` manages. Use the helper to resolve the path:

```python
from nemo_curator.stages.video.io.clip_writer import ClipWriterStage
previews_dir = ClipWriterStage.get_output_path_previews("/outputs")
```

Refer to Save & Export for directory structure and file locations: [Save & Export](/curate-video/save-export).

### Requirements and Troubleshooting

- `ffmpeg` with WebP (`libwebp`) support must be available in the environment.
- If you observe warnings about low frame rate or height, consider lowering `target_fps` or `target_height` to better match inputs.
- On encoding errors, check logs for the `ffmpeg` command and output to diagnose missing encoders.

{/* end */}
