UrbanVLA: A Vision-Language-Action Model for Urban Micromobility

Anqi Li*  Zhiyong Wang*  Jiazhao Zhang*  Minghan Li
Yunpeng Qi  Zhibo Chen  Zhizheng Zhang†  He Wang†
Peking University Galbot USTC BAAI

ICRA 2026 arXiv Project Page Video License

UrbanVLA teaser

--- ## Overview This repository is the official implementation of ICRA 2026 contributed paper *UrbanVLA: A Vision-Language-Action Model for Urban Micromobility*. It contains the data generation and post-processing pipeline, and benchmark evaluation pipeline. In the future, we will also release the model architecture and training code, as well as the checkpoint. Please stay tuned! --- ## TODO List We plan to further implement the following items: - [ ] Sekai data postprocessing instructions. - [ ] UrbanVLA architecture and training code. - [ ] UrbanVLA model checkpoint. --- ## Installation First, clone the repository and setup the python environment: ```bash git clone https://github.com/GalaxyGeneralRobotics/UrbanVLA.git cd UrbanVLA conda create -n urbanvla python=3.9 conda activate urbanvla # First, install metaurban dependencies. pip install -e . conda install pybind11 -c conda-forge cd metaurban/orca_algo && rm -rf build bash compile.sh && cd ../.. pip install stable_baselines3 imitation tensorboard wandb scikit-image pyyaml gdown # Ensure image on cuda pip install cupy-cuda12x # also cupy-cuda11x, depending on cuda env. pip install PyOpenGL PyOpenGL_accelerate pycuda panda3d cuda-python # For UrbanVLA model inference pip install transformers==4.42.4 timm==0.6.13 fairscale==0.4.13 decord==0.6.0 accelerate==0.34.2 ``` UrbanVLA is distributed as a source repository rather than a standalone wheel. The ORCA extension is compiled locally by the commands above. Renderer code and resources under `metaurban/render_pipeline` are included in Git; `pull_asset` downloads the static-object and pedestrian asset packs used by the simulator. Then, pull assets, verify installation and headless environment, and add current folder to your python path. ```bash # Pull metaurban assets python -m metaurban.pull_asset # Verify installation python -m metaurban.tests.test_env.profile_metaurban # Verify headless environment python -m metaurban.tests.test_env.verify_headless_env export PYTHONPATH=$PYTHONPATH:~/path/to/UrbanVLA ``` ### Common problems - pycuda installation failure: `src/cpp/cuda.hpp:14:10: fatal error: cuda.h: No such file or directory` Solution: ```bash export PATH=/usr/local/cuda/bin:$PATH export CUDA_ROOT=/usr/local/cuda ``` - You can also refer to [MetaUrban](https://github.com/metadriverse/metaurban) for guidance on installation. --- ## Quick starts ### 1. Data collection A unified CLI dispatches to the A\* collector or the RL expert collector. The new `--view_mode` flag selects between the historical 4-camera panorama and a single front-view layout suitable for single-view VLA backbones: ```bash python scripts/collect.py \ --policy astar --env dynamic --view_mode front \ --gpu 0 \ --start_seed 0 --goal 100 --start_episode 0 --name my_urbanvla_data ``` Use `--policy rl` for the PPO expert; it loads `benchmark/policy/weights/metaurban_ppo/pretrained_policy_576k.zip` by default, or pass `--expert_policy_path` to override it. ### 2. Data post-processing Two-pass pipelines that compute the strict 99th-percentile of future-8-step relative motion (`dx`, `dy`, `dyaw`) across the whole dataset, print those three constants, and normalize all supervision by them: ```bash # A* trajectories python -m data_postprocess.metaurban.astar.post_process \ --dataset_dir ./mydata/my_urbanvla_data \ --name my_urbanvla_data \ --start 1 --end 3001 # RL-expert trajectories (run sample_frames + clean_images first) python -m data_postprocess.metaurban.rl_expert.sample_frames --dataset_dir ./mydata/my_urbanvla_data python -m data_postprocess.metaurban.rl_expert.clean_images --dataset_dir ./mydata/my_urbanvla_data python -m data_postprocess.metaurban.rl_expert.post_process --dataset_dir ./mydata/my_urbanvla_data --name my_urbanvla_data # Sekai prompts python -m data_postprocess.sekai.cache_noisy_traj --traj_in <...>.json --traj_out <...>.json python -m data_postprocess.sekai.get_correct_traj --sekai_in <...>.json --traj_in <...>.json --output <...>.json ``` The `[norm]` banner emitted at the end of post-processing looks like: ``` ============================================================================== [norm] A* post-processing: 99th-percentile of future-8-step relative motion [norm] NORM_SCALE_X = 4.231087 [norm] NORM_SCALE_Y = 1.872430 [norm] NORM_SCALE_THETA = 0.873198 [norm] Copy these values into your NavigationPolicy subclass. ============================================================================== ``` **Important.** The eval pipeline multiplies every model output by the policy's class-attribute `NORM_SCALE_X / Y / THETA`. Those constants MUST equal the numbers printed above for the dataset the model was trained on; mismatched scales silently corrupt eval metrics. ### 3. Evaluation Smoke-test the PointNav pipeline without model weights using the DummyAgent: ```bash MAX_EPISODES=1 NO_VISUALIZATION=1 bash benchmark/eval.sh pointnav-test 0 ``` Run the bundled MetaUrban PPO expert: ```bash POLICY=metaurban_ppo NO_VISUALIZATION=1 bash benchmark/eval.sh pointnav-test 0 ``` Each full configuration contains 1000 test or 100 unseen episodes in total, balanced across `X/C/S`. Set `SUBSET=0.1` for exactly 100 test or 10 unseen episodes. Aggregate results into a Markdown table: ```bash python benchmark/aggregate_results.py \ --save_tag metaurban_ppo --config pointnav-test --wp 2 ``` Full details for the four PointNav/SocialNav configurations across `X/C/S` are in [`evaluation.md`](evaluation.md). --- ## Plugging in your own model Subclass `NavigationPolicy` and register it. The eval pipeline does the rest. ```python # benchmark/policy/my_model.py (the module name matches --policy) import numpy as np from . import NavigationPolicy, register_policy @register_policy("my_model") class MyPolicy(NavigationPolicy): requires_panorama = False # set True for 4-camera input NORM_SCALE_X = 4.231 # match post-processing's printed values NORM_SCALE_Y = 1.872 NORM_SCALE_THETA = 0.873 def __init__(self, model_path: str, **_): self.model = ... # load your weights def reset(self): ... def predict(self, observations, info): rgb = observations["front"] # (H, W, 3) uint8 # info has keys: traj_to_go, next_action, duration, ego_state return [[dx, dy, dtheta] for ...] # 8 normalized local-frame steps ``` Run with `--policy my_model`; the module is imported lazily. --- ## Repository layout ``` UrbanVLA/ ├── benchmark/ evaluation pipeline + per-policy adapters │ ├── policy/ NavigationPolicy ABC, dummy, metaurban_ppo │ ├── metaurban_eval.py │ ├── eval.sh, kill_metaurban_eval.sh │ └── aggregate_results.py ├── scripts/ data collection (astar / rl) + 2D occupancy │ ├── collect.py unified CLI dispatcher │ ├── astar_collect.py │ ├── rl_expert_collect.py │ ├── occupancy/ │ │ └── occupancy_map_2d.py │ ├── visualization/ │ │ └── image_utils.py │ └── _collect_utils.py shared camera capture (panorama/front) ├── data_postprocess/ post-processing pipelines │ ├── common/ traj_noise + 99th-percentile normalization + tests │ ├── metaurban/{astar,rl_expert}/ │ └── sekai/ ├── metaurban/ simulator + packaged engine configuration ├── config/ YAML configs for data collection / evaluation ├── setup.py ├── evaluation.md benchmark spec + how to run + aggregator ├── LICENSE └── assets/ teaser, pipeline diagram, benchmark table, setup ``` --- ## Citation If you find our work helpful, please consider citing the following BibTeX entry. ```bibtex @inproceedings{li2025urbanvla, title = {UrbanVLA: A Vision-Language-Action Model for Urban Micromobility}, author = {Li, Anqi and Wang, Zhiyong and Zhang, Jiazhao and Li, Minghan and Qi, Yunpeng and Chen, Zhibo and Zhang, Zhizheng and Wang, He}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, year = {2026} } ``` --- ## License This repository is released under the **Apache License 2.0** — see [`LICENSE`](LICENSE). Third-party components retain their respective upstream licenses where included. --- ## Acknowledgments - This repo is built on top of [MetaUrban](https://github.com/metadriverse/metaurban). - A\* path-planning modules contributed by the **MM-Nav team** ([arXiv:2510.03142](https://arxiv.org/abs/2510.03142)).