# VFlash Evaluation

[VFlash](README.md) · [Training](TRAINING.md) · [Data Preparation](DATA_PREPARATION.md) · [Evaluation](EVALUATION.md) · [Detailed Results](RESULTS.md)

This guide covers inference with a target–drafter pair and evaluation on the four upstream task groups. The [Detailed Results](RESULTS.md) page summarizes the published experiments from the technical report.

## Table of Contents

- [Dependencies](#dependencies)
- [Library usage](#library-usage)
- [Decoding rules](#decoding-rules)
- [Running evaluation](#running-evaluation)
- [Robot 3D trajectories](#robot-3d-trajectories)
- [Driving trajectories](#driving-trajectories)
- [Dense 3D motion](#dense-3d-motion)
- [Box detection](#box-detection)

## Dependencies

Install the [VFlash runtime](TRAINING.md#installation), then add benchmark dependencies from `Embodied/`:

```bash
pip install -e '.[evaluation]'
pip install ./evaluation/fastevaluate
```

FastEvaluate is bundled under `evaluation/fastevaluate/` and is required for COCO scoring. The same extension is used by the [LocateAnything evaluation scripts](../../evaluation/README.md#1-install-fastevaluate-required-for-cocolvis-metrics).

## Library Usage

Load the VFlash target and its matched drafter with the local `eaglevl` model code, then select `generation_mode="vflash"` to enable VFlash. The evaluation runners use the same loading path.

```python
from evaluation.inference_compat import decode_generation_output, load_model_and_processor
from evaluation.vflash.generation import build_model_inputs

model, processor = load_model_and_processor(
    "<target-checkpoint>",
    use_fast=False,
    local_code=True,                          # repository model; VFlash does not run through remote code
    local_processor=True,
    attn_implementation="flash_attention_2",
    training_runtime=True,
    draft_model_path="<drafter-checkpoint>",  # attaches the drafter with model.load_vflash_draft()
)
tokenizer = processor.tokenizer

# A record in the training format. The empty answer makes the prompt end with " /no_think", as in training.
record = {
    "image": "scene.jpg",
    "conversations": [
        {"from": "human", "value": "<image>\n<task prompt in the training format>"},
        {"from": "gpt", "value": ""},
    ],
}
inputs = build_model_inputs(processor, record)  # bfloat16 tensors on CUDA

output_ids, _, stats = model.generate(
    **inputs,
    max_new_tokens=320,
    do_sample=False,
    generation_mode="vflash",
    verify_atoms=False,             # commit schema-valid primitives without target verification
    return_vflash_stats=True,       # returns (ids, None, stats)
    atomic_decoding_tokenizer=tokenizer,
    vflash_draft_temperature=0.0,   # argmax in every primitive slot
)
answer = decode_generation_output(
    output_ids, inputs["input_ids"], processor, strip_prompt=True, strip_end_tokens=True
)
print(answer, stats.acceptance_lengths)
```

- Without `generation_mode` and the drafter arguments, `model.generate(**inputs, max_new_tokens=320, do_sample=False)` decodes autoregressively with the same target.
- `model.load_vflash_draft("<drafter-checkpoint>", tokenizer=tokenizer)` attaches a drafter to a target that is already loaded.
- Box queries use the hybrid decoder: add `bbox_hybrid=BBoxHybridConfig()` (from `eaglevl.model.vflash.bbox_hybrid`) and `initial_prefix_token_ids` from `evaluation.vflash.generation.bbox_hybrid_prefix_ids`, as `evaluation/inference_compat.py` does.

## Decoding Rules

With `generation_mode="vflash"`, ordinary text and physical primitives follow different decoding rules:

- **Text.** A 16-token block contains one accepted anchor and 15 proposals. The target verifies the block in one pass, keeping the longest matching prefix plus its own next token.
- **Primitives.** An accepted primitive-start token triggers a full-block draft under slot-level token masks. A schema-valid primitive is committed without target verification. The target then runs one pass over the committed primitive to update its KV cache and emit the next token. An invalid primitive is regenerated by the target in constrained autoregressive mode.
- **Boxes.** The hybrid box decoder lets the target write the text and drafts each `<box>` in one pass. A coordinate is uncertain when none of its six most probable tokens is a number token, or when the best number token has probability below 0.9 and the number tokens span more than 30 bins. The draft before the first uncertain coordinate is kept; the target regenerates the rest of the box through a numeric-only head.

The [coordinate schemas](DATA_PREPARATION.md#physical-primitives) define the primitive layouts and block lengths.

## Running Evaluation

The task runners compare VFlash with autoregressive decoding of the same target. Pass `--draft` to enable the drafter, or omit it for the AR baseline. VFlash accepts trajectory and motion primitives without token-level verification and uses the hybrid decoder for boxes.

Prepare evaluation JSONL records using the task schemas below. Evaluation datasets are not bundled with the repository. Run the commands from `Embodied/` on a GPU after installation (or with `PYTHONPATH=$PWD`). The robot, driving, and motion runners append predictions to the output JSONL and resume existing files. Use `--start` and `--end` to select an input shard, and `--media-root` to resolve relative media paths.

## Robot 3D Trajectories

Benchmarks: AgiBot, InternA1, and MolmoBot.

```bash
python -m evaluation.vflash.robot3d --model <target-checkpoint> --draft <drafter-checkpoint> \
  --dataset agibot --input <robot3d/agibot.jsonl> --output <out>/robot3d_agibot.jsonl \
  --media-root <media> --max-new-tokens 320
python -m evaluation.vflash.trajectory <out>/robot3d_agibot.jsonl --output <out>/robot3d_agibot.metrics.json
```

- **Budget.** The reported runs use `--max-new-tokens 320` on AgiBot and `192` on InternA1 and MolmoBot, for both decoders.
- **Records.** The training-format `conversations` (prompt and reference answer) with their media fields, `num_trajectory_points`, and `metric_trajectory_3d_by_arm_m`, the continuous ground truth in meters keyed by arm (`single`, or `left` and `right`); without it, the quantized trajectory of the reference answer is the ground truth.
- **Metrics.** `ade_3d_m` and `fde_3d_m` are the mean and final-waypoint Euclidean errors in meters over the arms that move (summed per-axis standard deviation of at least 5 mm); `sr_<k>cm` is the fraction of waypoints within k cm. Predictions are never repaired: a row counts toward `format_rate` and the error metrics only with the expected arms and exactly the expected number of waypoints. `mean_acceptance_length` is the mean number of tokens committed per target pass.

## Driving Trajectories

Benchmarks: Waymo, NAVSIM, and Alpamayo.

```bash
python -m evaluation.vflash.driving --model <target-checkpoint> --draft <drafter-checkpoint> \
  --dataset waymo --input <driving/waymo.jsonl> --output <out>/driving_waymo.jsonl
python -m evaluation.vflash.trajectory <out>/driving_waymo.jsonl --output <out>/driving_waymo.metrics.json
```

- **Budget.** `--max-new-tokens` defaults to 384 for `--dataset alpamayo` (64 waypoints) and 192 otherwise.
- **Records.** `index`, `benchmark_key`, the training-format `conversations` with media, `num_trajectory_points`, `metric_xy_m` (continuous x/y ground truth in meters), `metric_heading_rad` and `metric_target_source`.
- **Metrics.** ADE and FDE of the x/y positions in meters, with the same summary fields as above; every row also stores its heading error `heading_mae_rad`.

## Dense 3D Motion

Benchmarks: HOT3D, WorldTrack, and DAVIS from PointMotionBench.

```bash
python -m evaluation.vflash.motion --model <target-checkpoint> --draft <drafter-checkpoint> \
  --input <motion/hot3d.jsonl> --output <out>/motion_hot3d.jsonl
python -m evaluation.vflash.motion_metrics <out>/motion_hot3d.jsonl --input <motion/hot3d.jsonl> \
  --best-of 5 --draft-temperature 0.1 --output <out>/motion_hot3d.metrics.json
```

- **Decoding.** With `--draft`, each record gets `--best-of 5` sequential candidates whose primitive slots are sampled at `--draft-temperature 0.1` after one per-record seed (`--sampling-seed 42`), and the candidate with the lowest ADE against the visible ground truth is kept (oracle best-of-5). Without `--draft`, the target decodes one greedy answer; score it without `--best-of` and `--draft-temperature`, which only check that every row was produced with those settings. The budget is 960 new tokens.
- **Records.** `example_id`, `bench`, `video`, `obj`, `t0`, `batch`, `image` (the observed frames), `prompt_dense_3d` (the question), `anchor` (the origin of the answer coordinates), `scale_m`, `gt_future_raw` (8 points × 30 frames × xyz in meters) and `gt_future_vis` (8 × 30 visibility). Optional `prediction_rotation_world_to_camera` maps camera-axis predictions back to the benchmark frame, and `future_frames`, `num_frames` and `scored_mask` are used by the scorer.
- **Metrics.** Pooled over every visible scored (point, frame) pair: ADE in meters; FDE at the last horizon frame present in the clip, over the points visible there; PWT, the mean fraction of errors strictly below 1, 2, 5, 10 and 20 cm. The JSON report adds per-clip and per-object means.

## Box Detection

Benchmarks: COCO and Dense200.

```bash
python -m evaluation.vflash.bbox --model <target-checkpoint> --draft <drafter-checkpoint> \
  --dataset COCO --data <Rex-Omni-EvalData> --input <bbox/COCO.jsonl> \
  --output <out>/bbox_COCO/predictions.jsonl
```

- **Data.** `--data` is the root of the [Rex-Omni evaluation data](https://huggingface.co/datasets/Mountchicken/Rex-Omni-EvalData). `--input` defaults to `<data>/_annotations/box_eval/<dataset>.jsonl` and the COCO ground truth to `<data>/coco/instances_val2017.json`.
- **Run.** Inference runs `evaluation/inference_detection_ddp.py` on `--nproc-per-node` GPUs (default 8), then `metrics.json` is written next to the predictions; `--skip-inference` only rescores. With `--draft`, boxes use the hybrid decoder; without it, the target decodes greedily.
- **Metrics.** COCO asks for every present category in one multi-turn conversation (512 new tokens per turn) and reports Avg F1 (mean over IoU 0.50:0.95), F1@0.50 and F1@0.95; it needs FastEvaluate. Dense200 sends one query per category (4,096 new tokens) and reports mF1, the mean over IoU 0.50:0.95 of the F1 of mean precision and mean recall.