# NaVILA

> Cheng 2024 · [Paper](https://arxiv.org/abs/2412.04453)

**One-line summary** — NaVILA extends the vision-language-action paradigm from tabletop manipulation to legged-robot navigation, pairing a vision-language model that emits mid-level actions *in natural language* with an RL locomotion policy that executes them, so a robot can follow language instructions through real 3D environments.

## Problem

Vision-and-Language Navigation (VLN) with legged robots is attractive twice over: language is a flexible way for humans to command robots, and legs let the robot traverse challenging, cluttered scenes that defeat wheeled bases. But translating a human instruction *all the way down* to joint-level control is hard — manipulation VLAs quantize low-level commands into tokens, yet LLMs/VLMs were trained on natural language, and forcing them to emit precise non-verbal actions strains the very reasoning they were pretrained for. NaVILA asks whether language itself is the better action representation.

## Method & architecture

- **Two-level framework**: a VILA-based VLM handles semantic instruction following and outputs **mid-level actions expressed in language** — e.g. "turn right 30 degrees", "move forward 75 cm" — while a separate low-level visual locomotion policy converts them into joint movements in real time. The VLA runs at low frequency; the locomotion policy runs in real time, absorbing obstacle avoidance and balance.
- **VLM (high level)**: VILA's vision encoder → MLP projector → LLM, all unfrozen during SFT. A **navigation prompt** distinguishes the latest frame ("current observation") from uniformly sampled earlier frames ("a video of historical observations" as a memory bank), using textual cues instead of new special tokens; 8-64 frames are used. A regex parser extracts the action type and its argument (distance/angle) from the LLM output — empirically every emitted action parsed successfully.
- **SFT data blend**: (1) simulated navigation from R2R-CE and RxR-CE via a shortest-path follower, with consecutive actions merged (up to three, e.g. two 25 cm steps → one 50 cm step) and stop-action rebalancing; (2) **real human touring videos** — 2K egocentric YouTube videos sampled into 20K trajectories, with actions recovered by MASt3R metric camera-pose estimation and instructions generated by VLM captioning + LLM rephrasing (first work to train navigation directly on human video in continuous settings); (3) auxiliary tasks — EnvDrop augmented instructions, trajectory summarization, ScanQA 3D question answering; (4) general VQA to retain world knowledge.
- **Locomotion policy (low level)**: the discrete commands are cast to velocity commands $\{0.5\ \text{m/s},\ \pm\tfrac{\pi}{6}\ \text{rad/s},\ 0\}$ executed for durations matching the VLA's stated distance/angle. A single-stage **PPO** policy (no teacher-student distillation) outputs desired joint positions $q^{d} \in \mathbb{R}^{12}$ for the Unitree Go2's leg motors. The critic sees privileged terrain height scans; the actor sees only real-world-available proprioception history plus a **2.5D height map built from LiDAR point clouds** (Unitree L1, 360°×90° FoV, max-filtered over the last 5 scans) — LiDAR handles glass and strong sunlight where depth/RGB fail. Ray-casting in Isaac Lab yields >60K FPS training throughput on an RTX 4090.
- **VLN-CE-Isaac benchmark**: a new Isaac Sim benchmark with 1,077 traversable R2R Val-Unseen trajectories, evaluating full-physics joint-level execution rather than Habitat-style teleporting agents.

## Results

- **VLN-CE benchmarks** (Val-Unseen): on R2R-CE NaVILA reaches **SR 54.0 / SPL 49.0 / NE 5.22** using only single-view RGB — a ~17% SR improvement over prior single-view SoTA NaVid (SR 37.0) and comparable or superior to methods using panoramas, depth, odometry, or simulator-pretrained waypoint predictors; on RxR-CE it scores SR 49.3 / SPL 44.0 / nDTW 58.8.
- **Cross-dataset zero-shot** (trained on R2R only, tested on RxR Val-Unseen): SR **34.3** vs NaVid's 23.8.
- **Scene understanding**: on ScanQA validation, NaVILA (64 frames) reaches **CIDEr 102.7** vs the prior VLA SoTA NaviLLM's 75.9, beating even 3D LMMs that require scans or posed RGB-D.
- **Locomotion**: vs distillation-based ROA, linear/angular velocity errors drop from 0.161/0.152 to **0.066/0.113** and collision rate from 3.09 to **0.81**.
- **VLN-CE-Isaac**: the vision-based policy beats the blind (proprioception-only) policy by **14% SR on Go2** (50.2 vs 36.2) and 21% on the H1 humanoid (45.3 vs 24.4), approaching the oracle-execution upper bound (51.3).
- **Real world**: across 25 instructions × 3 trials in workspace, home, and outdoor scenes, NaVILA achieves an **88% success rate** (75% on complex multi-room instructions), outperforming a GPT-4o baseline; the same VLA drives a Booster T1 humanoid without retraining, despite different camera height and view angle.

## Why it matters for SLAM

NaVILA is a concrete example of the foundation-model challenge to classical SLAM-based navigation stacks: it replaces the explicit map-build-then-plan pipeline with an instruction-conditioned policy whose "map" is a bank of memory frames. At the same time, its design shows where geometry still fits — MASt3R pose estimation labels its human-video training data, a LiDAR height map keeps the robot safe, and metric localization and persistent maps are complementary to a VLA's semantic instruction following; hybrid systems that give VLAs SLAM-grounded spatial memory are an active research direction.

## Related

- [RT-2](rt-2.md)
- [OpenVLA](openvla.md)
- [LLaVA](llava.md)
- [VLM vs VLA](vlm-vs-vla.md)
- [Spatial AI](spatial-ai.md)
- [MASt3R](../level-05-deep-learning/mast3r.md)