# EgoExoMoCap: Distributed Ego-Exo Human Motion Capture **ECCV 2026 (Oral)** [Jiaxi Jiang^1,2*](https://www.jiaxi-jiang.com/), [Bharat Lal Bhatnagar¹](https://virtualhumans.mpi-inf.mpg.de/people/Bhatnagar.html), [Nan Yang¹](https://nan-yang.me/), [Lingni Ma¹](https://scholar.google.com/citations?user=eUAgpwkAAAAJ), [Sebastian Starke¹](https://www.sebastianxstarke.com/), [Robin Kips¹](https://scholar.google.com/citations?user=RwyrWEkAAAAJ&hl=fr), [Nadine Bertsch¹](https://ps.is.mpg.de/person/nrueegg), [Christian Holz²](https://www.christianholz.net), [Federica Bogo¹](https://fbogo.github.io/)
¹ Meta Reality Labs ² ETH Zürich

___________

>Human motion capture from head-mounted devices (HMDs) offers a scalable way to acquire real-world human motion and interaction data, which is crucial for applications in embodied AI and VR/AR. Existing approaches focus on either egocentric body tracking, estimating the motion of the subject wearing the device, or exocentric tracking, capturing the movements of people in the wearer’s surroundings. So far, these two paradigms have largely been explored in isolation. In this paper, we propose a novel distributed framework that jointly leverages ego- and exocentric multi-modal signals for human motion estimation from HMDs. Unlike traditional motion capture systems requiring bulky multi-camera setups or obtrusive mocap suits, our approach, EgoExoMoCap, is as simple as two (or more) people, each wearing a pair of smart glasses. The method leverages head (plus potentially wrist) tracking signals for accurate estimation of global motion in the 3D world and combines context-aware image features based on DINOv3 to achieve robustness in the presence of noise and occlusions. Extensive experiments on two in-the-wild datasets show that our approach can robustly reconstruct motion even in challenging scenarios. ## Method

Given an egocentric and one or more exocentric streams from HMDs, we first roughly estimate 3D body poses from egocentric streams (EgoNet) to identify regions of interest in exocentric frames. From these, ViTPose-extracted 2D keypoints are unprojected into 3D rays and softly weighted by DINOv3-based confidence scores to form Exo Tokens. A Spatial Transformer fuses Ego and Exo tokens into View-Aggregated (VA) Tokens, followed by a Temporal Transformer for smoothness to output final full-body motions. ## Installation ```bash conda create -n egoexomocap python=3.10 -y conda activate egoexomocap # install PyTorch matching your CUDA version first (https://pytorch.org/) pip install -r requirements.txt ``` Some dependencies are installed from source (see `requirements.txt`): `dinov3`, `projectaria-tools`, `nymeria`, `human_body_prior`. **SMPL-H body models** are required but not distributed here (license). Download the "Extended SMPL+H model" from [mano.is.tue.mpg.de](https://mano.is.tue.mpg.de/) (free registration) and place the neutral/male/female models at: ``` third_party/body_models/smplh/{male,female,neutral}/model.npz third_party/body_models/smplh/SMPLH_NEUTRAL.npz ``` ## Repository layout ``` main_train.py training entry point main_test.py evaluation entry point models/ model classes + base + selector + loss model_egoexomocap.py ego+exo fusion model (inherits the chain below) model_egoposer_seq2seq.py, model_egoposer.py networks/ egoexomocap (ego+exo net), fusion (observer-fusion module), egonet (coarse per-frame ego net) data_loaders/ Nymeria dataset + selector utils/ options, transforms, metrics, logging helpers options/ the 8 config files for the reported results model_zoo/ final checkpoints for the reported results (one model.pth per model) splits/ benchmark sequence splits + eval protocols data_prep/ shared/ DINO/ViT backbones + EgoNet-bbox extraction (dataset-agnostic) nymeria/ Nymeria download + SMPL-H + observer video/features egohumans/ EgoHumans -> Nymeria-schema pair extraction (4 stages) ``` ## Configs Configs are named by the **tracker input type** (the head/wrist signal driving the model), which is the real axis of variation: - **`nymeria_real_tracker`** — real headset device signals (real Aria head + wristband wrists). Trained and evaluated on **Nymeria**. - **`egohumans_syn_tracker`** — synthetic FK-derived head/wrist (EgoHumans has no wrist trackers, so inputs come from GT SMPL by FK). Trained on Nymeria-FK, evaluated **zero-shot on EgoHumans**. | config (train / test) | eval dataset | inputs | | --- | --- | --- | | `train_nymeria_real_tracker_3pt.yaml` / `test_nymeria_real_tracker_3pt.yaml` | Nymeria | 3-point (head + wrists) | | `train_nymeria_real_tracker_1pt.yaml` / `test_nymeria_real_tracker_1pt.yaml` | Nymeria | 1-point (head only) | | `train_nymeria_syn_tracker_3pt.yaml` / `test_egohumans_syn_tracker_3pt.yaml` | EgoHumans (zero-shot) | 3-point | | `train_nymeria_syn_tracker_1pt.yaml` / `test_egohumans_syn_tracker_1pt.yaml` | EgoHumans (zero-shot) | 1-point | Body-pose ground truth comes from **NymeriaPlus** official SMPL (`smplh_nymeriaplus.npz`). Training uses GT observer bboxes (the default `bbox_source`); evaluation uses EgoNet-derived observer bboxes (`bbox_source: egonet` in each config's test set) — i.e. the trained model's own EgoNet proposes the crop at test time. ## Data preparation The raw datasets are obtained separately: **Nymeria** (VRS + MPS, plus NymeriaPlus SMPL-H GT) by [requesting access](https://www.projectaria.com/datasets/nymeria/) and exporting its per-sequence signed-URL manifests, and **EgoHumans** from the [EgoHumans repo](https://github.com/rawalkhirodkar/egohumans) (OAuth download). See [`data_prep/README.md`](data_prep/README.md) for the full guide — dataset access, the ViTPose / DINOv3 checkpoints, and the path / env-var configuration; the condensed pipeline below assumes those are in place. Run all `data_prep/` scripts from the repository root. Put the repo root **and** `data_prep/shared` on the path so the shared DINO/ViT backbones resolve: `PYTHONPATH=.:data_prep/shared`. VRS-reading steps need the `pymeria` env (projectaria_tools / nymeria SDK); the GPU feature steps need torch + transformers. ```bash export PYTHONPATH=.:data_prep/shared ``` ### Nymeria ```bash # 1. download sequences (see splits/ for the benchmark split) + NymeriaPlus SMPL (GT) python data_prep/nymeria/download_test_split_parallel.py python data_prep/nymeria/download_train_split_parallel.py python data_prep/nymeria/download_nymeriaplus_smpl.py # 2. SMPL-H GT @ 30 fps from NymeriaPlus (also establishes the meta.json / # timestamps.csv grid that every downstream step aligns to) python data_prep/nymeria/extract_smplh_30fps_from_nymeriaplus.py # smplh_nymeriaplus.npz python data_prep/nymeria/extract_wearer_smplh_fk.py \ --input_smplh smplh_nymeriaplus.npz --output_fk wearer_smplh_fk_nymeriaplus.npz # 3. observer (exo) video + head/wrist + bbox, then DINO/ViT features python data_prep/nymeria/extract_observer_video.py # observer_full.mp4, wearer_ego.npz, observer.npz, bbox.npz python data_prep/nymeria/extract_features_from_video.py # vitpose.npz, dino_features.npz # 4. EgoNet-derived bboxes + features recomputed on them (for eval, bbox_source: egonet). # Uses the trained model's own EgoNet, so run this after training — or download the # released checkpoint first (see Evaluation) and point --opt at its test config. python data_prep/shared/extract_egonet_bboxes.py --opt options/test_nymeria_real_tracker_3pt.yaml python data_prep/shared/rerun_vitpose_dino_egonet_bbox.py # vitpose_egonet.npz, dino_features_egonet.npz ``` ### EgoHumans (zero-shot) Produces Nymeria-schema sequences under `dataset/egohumans_extracted/`, then reuse the shared EgoNet-bbox step above with an EgoHumans config. ```bash python data_prep/egohumans/extract_egohumans_pairs_stage1.py python data_prep/egohumans/extract_egohumans_pairs_stage2_bbox.py python data_prep/egohumans/extract_egohumans_pairs_stage2_video.py python data_prep/egohumans/extract_egohumans_pairs_stage3_features.py ``` ## Training Train with any of the four `train_*` configs (see [Configs](#configs)): ```bash # Nymeria, real device tracker (3-point: head + wrists) python main_train.py -opt options/train_nymeria_real_tracker_3pt.yaml # Nymeria-FK, synthetic tracker (for zero-shot EgoHumans evaluation) python main_train.py -opt options/train_nymeria_syn_tracker_3pt.yaml ``` Swap `_3pt` for `_1pt` for the head-only variants. Training uses GT observer bboxes (data-prep step 3). Checkpoints and logs are written under `results//` (`models/.pth`, `train.log`); the checkpoint / validation / print intervals and the loss weights live in each config's `train:` block (`checkpoint_save`, `checkpoint_test`, `checkpoint_print`, `*_weight`), and `gpu_ids` selects the device(s). After training, generate the EgoNet-derived bboxes and features from your trained checkpoint (data-prep step 4) before running the evaluation below. ## Evaluation Download the final checkpoints from **[Google Drive](https://drive.google.com/drive/folders/1nID3fKYi7HudVeT5kY2mlA9imxTAeBJG?usp=sharing)** and place the four folders under `model_zoo/`, so each `model.pth` sits at: ``` model_zoo/nymeria_real_tracker_3pt/model.pth model_zoo/nymeria_real_tracker_1pt/model.pth model_zoo/egohumans_syn_tracker_3pt/model.pth model_zoo/egohumans_syn_tracker_1pt/model.pth ``` Every test config's `pretrained_model:` already points at the matching one. Nymeria (real tracker): ```bash python main_test.py -opt options/test_nymeria_real_tracker_3pt.yaml ``` EgoHumans (zero-shot, synthetic tracker): ```bash python main_test.py -opt options/test_egohumans_syn_tracker_3pt.yaml ``` Fuse multiple observer views at test time with `--multi_observer` (`--protocol` restricts to a scene subset): ```bash python main_test.py -opt options/test_egohumans_syn_tracker_3pt.yaml \ --multi_observer --protocol splits/egohumans_multiobs_protocol.json ``` ## Citation ```bibtex @article{jiang2026egoexomocap, title={EgoExoMoCap: Distributed Ego-Exo Human Motion Capture}, author={Jiaxi Jiang and Bharat Lal Bhatnagar and Nan Yang and Lingni Ma and Sebastian Starke and Robin Kips and Nadine Bertsch and Christian Holz and Federica Bogo}, journal={arXiv preprint arXiv:2607.15868}, year={2026} } ``` ## Acknowledgements This code builds on [EgoPoser](https://github.com/eth-siplab/EgoPoser), which is itself adapted from [AvatarPoser](https://github.com/eth-siplab/AvatarPoser). We train and evaluate on the [Nymeria](https://www.projectaria.com/datasets/nymeria/) and [EgoHumans](https://github.com/rawalkhirodkar/egohumans) datasets, and use the [SMPL-H](https://mano.is.tue.mpg.de/) body model for the human representation. The observer (exo) branch relies on [ViTPose](https://github.com/ViTAE-Transformer/ViTPose) for 2D keypoint detection and [DINOv3](https://github.com/facebookresearch/dinov3) for visibility-gating features. We thank the authors of these projects for releasing their code, models, and data.