# Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation
https://github.com/user-attachments/assets/49036e84-c397-4589-9024-efb05b14efa0
## :books: Table of Contents:
1. [Highlights](#high)
2. [Getting Started](#start)
- [Simulation](#simulation)
- [Real-World](#real-world)
3. [Checkpoints](#checkpoints)
4. [TODO List](#todos)
5. [License](#license)
6. [Citation](#citation).
7. [Acknowledgment](#acknowledgment)
## :fire: Highlights
- :trophy: **SOTA simulation performance** Seer achieves state-of-the-art performance on simulation benchmarks CALVIN ABC-D and LIBERO-LONG.
- :muscle: **Impressive Real-World performance** Seer demonstrates strong effectiveness and generalization across diverse real-world downstream tasks.
## :door: Getting Started
We provide step-by-step guidance for running Seer in simulations and real-world experiments.
Follow the specific instructions for a seamless setup.
### Simulation
#### CALVIN ABC-D
- [Installation](docs/CALVIN_ABC-D_INSTALL.md)
- [Running Code](docs/CALVIN_ABC-D_RUN.md)
#### LIBERO LONG
- [Installation](docs/LIBERO_LONG_INSTALL.md)
- [Running Code](docs/LIBERO_LONG_RUN.md)
### Real-World
#### Real-World (Quick Training w & w/o pre-training)
For users aiming to train Seer from scratch or fine-tune it, we provide comprehensive instructions for environment setup, downstream task data preparation, training, and deployment.
- [Installation](docs/REAL-WORLD_INSTALL.md)
- [Post-processing](docs/REAL-WORLD_POSTPROCESS.md)
- [Fine-tuning & Scratch](docs/REAL-WORLD_FT_SC.md)
- [Inference](docs/REAL-WORLD_INFERENCE.md)
#### Real-World (Pre-training)
This section details the pre-training process of Seer in real-world experiments, including environment setup, dataset preparation, and training procedures. Downstream task processing and fine-tuning are covered in [Real-World (Quick Training w & w/o pre-training)](#real-world-qs).
- [Installation](docs/REAL-WORLD_INSTALL.md)
- [Pre-processing](docs/REAL-WORLD_PREPROCESS.md)
- [Pre-training](docs/REAL-WORLD_PRETRAIN.md)
## :pencil2: Checkpoints
Relevant checkpoints are available on the [website](https://drive.google.com/drive/folders/1F3IE95z2THAQ_lt3DKUFdRGc86Thsnc7?usp=sharing).
|Model|Checkpoint|
|:------:|:------:|
|CALVIN ABC-D|[Seer](https://drive.google.com/drive/folders/17Gv9snGCkViuhHmzN3eTWlI0tMfGSGT3?usp=sharing) (Avg.Len. : 3.98) / [Seer Large](https://drive.google.com/drive/folders/1AFabqfDEi69oMo0FTGhEiH2QSRLYBR9r?usp=drive_link) (Avg.Len. : 4.30)|
|Real-World|[Seer (Droid Pre-trained)](https://drive.google.com/drive/folders/1rT8JKLhJGIo97jfYUm2JiFUrogOq-dgJ?usp=drive_link)|
## 📆 TODO
- [x] Release real-world expriment code.
- [x] Release CALVIN ABC-D experiment code (Seer).
- [x] Release the evaluation code of Seer-Large on CALVIN ABC-D experiment.
- [x] Release the training code of Seer-Large on CALVIN ABC-D experiment.
- [x] Release LIBERO-LONG experiment code.
- [ ] Release simpleseer, a quick scratch training & deploying code.
## License
All assets and code are under the [Apache 2.0 license](./LICENSE) unless specified otherwise.
## Citation
If you find the project helpful for your research, please consider citing our paper:
```bibtex
@article{tian2024predictive,
title={Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation},
author={Tian, Yang and Yang, Sizhe and Zeng, Jia and Wang, Ping and Lin, Dahua and Dong, Hao and Pang, Jiangmiao},
journal={arXiv preprint arXiv:2412.15109},
year={2024}
}
```
## Acknowledgment
This project builds upon [GR-1](https://github.com/bytedance/GR-1) and [Roboflamingo](https://github.com/RoboFlamingo/RoboFlamingo). We thank these teams for their open-source contributions.