# Instructions This project requires multiple environments to run the whole pipeline. ## Prerequisites - x86_64 Linux machine with NVIDIA H100 GPU - `conda` (Miniconda or Anaconda) available in your shell environment Run all commands below from the repository root. ## Environment Setup ### Environment 1: PSIVG_env1 ```bash # optional: remove existing environment conda remove -y -n PSIVG_env1 --all # create environment from lock-style manifest conda env create -f envs/PSIVG_env1.yml conda activate PSIVG_env1 # set CUDA build paths for extension packages export TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST:-7.0;7.5;8.0;8.6;8.7;8.9}" export CUDA_INCLUDE_DIR="$CONDA_PREFIX/targets/x86_64-linux/include" export CUDA_LIB_DIR="$CONDA_PREFIX/targets/x86_64-linux/lib" export CPATH="$CUDA_INCLUDE_DIR:${CPATH:-}" export CPLUS_INCLUDE_PATH="$CUDA_INCLUDE_DIR:${CPLUS_INCLUDE_PATH:-}" export LIBRARY_PATH="$CUDA_LIB_DIR:${LIBRARY_PATH:-}" # build extension dependencies against the environment's torch/cuda python -m pip install --no-build-isolation git+https://github.com/NVlabs/nvdiffrast.git python -m pip install --no-build-isolation --no-deps git+https://github.com/IDEA-Research/GroundingDINO.git # python -m pip install --no-build-isolation git+https://github.com/facebookresearch/pytorch3d.git # optional: keep --no-deps to avoid torch version conflicts # python -m pip install --no-deps git+https://github.com/IDEA-Research/Grounded-SAM-2.git ``` ### Environment 2: PSIVG_env2 ```bash # remove existing environment if any conda remove -y -n PSIVG_env2 --all # create new environment conda env create -f envs/PSIVG_env2.yml # activate environment conda activate PSIVG_env2 # install project dependencies pip install -r envs/PSIVG_env2.txt # install vipe cd psivg/perception/vipe # install vipe as a package locally pip install --no-build-isolation -e . # return to root directory cd ../../.. ``` ### Environment 3: PSIVG_env3 ```bash # remove existing environment if any conda remove -y -n PSIVG_env3 --all # create new environment conda create -n PSIVG_env3 python=3.10 # activate environment conda activate PSIVG_env3 # install project dependencies pip install -r envs/PSIVG_env3.txt ``` ### Environment 4: langsam Additionally, we also need to install the langsam environment. See more details in the [lang-segment-anything](https://github.com/luca-medeiros/lang-segment-anything) repository, which we have installed and named as `langsam` environment. ## Set up the `DATA_ROOT` Path All intermediate artifacts and results generated by this pipeline will be stored and organized under the `DATA_ROOT` path defined in [`psivg/constants.py`](psivg/constants.py). By default, the `DATA_ROOT` is set to `./data_root`. If you want to change this, please edit the `DATA_ROOT` path in [`psivg/constants.py`](psivg/constants.py). ## Download Pretrained Models ```bash python3 envs/download_pretrained.py ``` This will download the required pretrained checkpoints and save them to the `pretrained_models` directory. ## Run the Pipeline ### Step 0: Prepare the data input We need two input files to start the pipeline: 1. A generated template video file `PATH/TO/XXXX.mp4` 2. A metadata file `PATH/TO/XXXX.json` To generate the template video, you can use any text-to-video model, e.g., [HunyuanVideo](https://huggingface.co/TencentARC/HunyuanVideo). You can also first use a text-to-image model to generate a single image, and then use a video generation model to generate a video from the image, e.g., [CogVideoX](https://huggingface.co/zai-org/CogVideoX-5b-I2V). In general, using newer and more recent video generation models often will lead to better results. Our code assumes the videos are 49 frames, and with frames of size 480 x 720, following Go-with-the-FLow. Note that, for the naming, `XXXX` must be a 4-digit number and will be used as the sample ID for the pipeline. The metadata file should be in the same directory as the template video file, and be a JSON file with the following structure: ```json { "video_prompt": "A soccer ball with a classic black-and-white pattern is in midair, bouncing on a cobblestone street in a quiet village. There are no humans. The scene is realistic, camera is moving around.", "fg_prompt": "soccer ball with a classic black-and-white pattern", "primary": "ball" } ``` where: - `video_prompt`: The full prompt that generates the template video. - `fg_prompt`: The partial prompt that describes the foreground object in the template video. - `primary`: A single-word generic name for the foreground object. In the following steps, we will use the `assets/0000.mp4` as the demo template video. ### Step 1: ```bash conda activate PSIVG_env1 CUDA_VISIBLE_DEVICES=0 python3 main_part1.py --video assets/0000.mp4 ``` The output data will be saved in `data_root/OUT_Perception/0000`. Other intermediate data will also be saved in `data_root/INPUT_DATA/0000`. ### Step 2: ```bash conda deactivate conda activate PSIVG_env2 CUDA_VISIBLE_DEVICES=0 python3 main_part2.py --video "0000" ``` The output data from using ViPE will be saved in `data_root/OUT_ViPE_Raw/0000`, `data_root/OUT_ViPE_Export/0000`. The output data from the simulations will be saved in `data_root/OUT_Simulation/0000`. The rendered data will be saved in `data_root/OUT_Rendering/0000`. ### Step 3: ```bash conda deactivate conda activate PSIVG_env3 ./main_part3.sh ``` After this step, the dataset that will be used in the video generation step should be prepared in `data_root/datasets/generated_data_example/0000`. ### Step 4: ```bash ./main_part4.sh ``` After this step, the final generated video will be saved in `outputs/generated_data_example/expt_LRcosine_lr0.0002_iter50/0000`. Note that, by default, we assume that the input video uses a moving camera. If the input video uses a static camera, please set `USE_MOVING_CAMERA="false"` in `main_part3.sh` and `main_part4.sh`, which makes the method more resistant to noise in the background. By default, we also turn TTCO off, since it requires more compute and also requires a lot more memory (e.g., we ran our code on a single H100 GPU). If you want to turn TTCO on, please set `USE_TTCO="true"` in `main_part4.sh`. TTCO usually helps more for challenging scenes, e.g., when the objects or camera are moving fast. For instance, in the provided example assets, 0002.mp4 and 0003.mp4 produce substantially worse results when TTCO is turned off during our evaluation. ## Tips for Running the Pipeline Here are some tips to help you get better videos when running the pipeline. Firstly, we observed that it is helpful if the template videos contain larger objects, and also if the objects are not occluded in the first frame. If you want to do such generation at scale, it is also advised to write a script to filter out template videos that do not contain the object, or scenes with highly erratic motion at the start which may affect the perception part. In our code, we do not assume that an OpenAI API key is available. We have instead implemented a default setting which allows for running of the code without the use of the OpenAI API. If you want to try using the GPT-5 model for physical properties estimation, you can set the API key as `OPENROUTER_API_KEY` in the environment. If your template video has very quick camera motion, you may need to adjust the `FLOW_THRESHOLD` in `main_part3.sh` to a larger value. By default, we set it as 2.0 which works well for us, but adjusting it to a larger value will allow larger camera movements, instead of being filtered out as noise. If the resulting videos contain a lot of textural artifacts even with TTCO, it may be helpful to try a slower movement/simulation. In our code, we chose to use CPU by default for running the simulations with MPM since we found that it is more stable than running on GPU. If you want to speed up the simulation, you can also try changing the device to use the GPU instead. As a safety measure in case the simulation crashes (which happens occasionally on certain machines), we set a timeout, such that the crashed simulation can be automatically resetted after a certain amount of time, e.g., `CUDA_VISIBLE_DEVICES=0 timeout -s SIGKILL 20m python3 main_part2.py --video "0000"`.