threestudio is a unified framework for 3D content creation from text prompts, single images, and few-shot images, by lifting 2D text-to-image generation models.

👆 Results obtained from methods implemented by threestudio 👆
| ProlificDreamer | DreamFusion | Magic3D | SJC | Latent-NeRF | Fantasia3D | TextMesh |
| Zero-1-to-3 | Magic123 | HiFA | SDI |
| InstructNeRF2NeRF | Control4D |

Did not find what you want? Checkout threestudio-extension or submit a feature request here!

| Animate-124 | 4D-fy | GeoDream | DreamCraft3D | Dreamwaltz | 3DFuse | Progressive3D | GaussianDreamer | Gaussian Splatting | MVDream | Mesh-Fitting | ## News - 08/11/2024: Thank [Artem Lukoianov](https://github.com/ottogin) for implementation of [Score Distillation via Reparametrized DDIM](https://lukoianov.com/sdi)! Text-to-3D module is added to Threestudio as well as a notebook with 2D score distillation experiments. - 21/10/2024: Thank [Amir Barda](https://github.com/amirbarda) for implementation of [MagicClay](https://github.com/amirbarda/MagicClay)! Follow the instructions on its website to give it a try. - 12/03/2024: Thank [Matthew Kwak](https://github.com/mskwak01) and [Inès Hyeonsu Kim](https://github.com/Ines-Hyeonsu-Kim) for implementation of [3DFuse](https://github.com/KU-CVLAB/3DFuse-threestudio)! Follow the instructions on its website to give it a try. - 08/03/2024: Thank [Xinhua Cheng](https://github.com/cxh0519/) for implementation of [GaussianDreamer](https://github.com/cxh0519/threestudio-gaussiandreamer)! Follow the instructions on its website to give it a try. - 01/03/2024: Thank [Xinhua Cheng](https://github.com/cxh0519/) for implementation of [Progressive3D](https://github.com/cxh0519/Progressive3D)! Follow the instructions on its website to give it a try. - 09/01/2024: Thank [Zehuan Huang](https://github.com/huanngzh) for implementation of 3D human avatar generation [Dreamwaltz](https://github.com/huanngzh/threestudio-dreamwaltz)! Follow the instructions on its website to give it a try. - 06/01/2024: Thank [Baorui Ma](https://github.com/mabaorui) for implementation of [GeoGream extensions](https://github.com/baaivision/GeoDream/tree/threestudio)! Follow the instructions on its website to give it a try. - 05/01/2024: Implemented HiFA. Follow the instructions [here](https://github.com/threestudio-project/threestudio#hifa-) to try all three variants. - 23/12/2023: Thank [Yuyang Zhao](https://github.com/HeliosZhao) for implementation of image-to-4D generation extensions [Animate-124](https://github.com/HeliosZhao/Animate124/tree/threestudio)! Follow the instructions on the extensions website to give it a try. - 18/12/2023: Implementation of [4D-fy](https://github.com/DSaurus/threestudio-4dfy) for 4D generation and [DreamCraft3D](https://github.com/DSaurus/threestudio-dreamcraft3D) for high-quality image-to-3D generation as the custom extensions! Follow the instructions on the extensions website to give it a try. - 13/12/2023: Implementation supporting [Stable Zero123](https://stability.ai/news/stable-zero123-3d-generation) for 3D generation from a single image! Follow the instructions [here](https://github.com/threestudio-project/threestudio#stable-zero123) to give it a try. - 30/11/2023: Implementation of [MVDream](https://github.com/DSaurus/threestudio-mvdream), [Gaussian Splatting](https://github.com/DSaurus/threestudio-3dgs) as the custom extensions. You can also use neural representation to fit a mesh by [Mesh-Fitting](https://github.com/DSaurus/threestudio-meshfitting). - 30/11/2023: Implementation of [custom extension system](https://threestudio-project.github.io/threestudio-extensions/) and you can add your extensions in [this project](https://github.com/threestudio-project/threestudio-extensions). - 25/06/2023: Implementation of [Magic123](https://guochengqian.github.io/project/magic123/)! Follow the instructions [here](https://github.com/threestudio-project/threestudio#magic123-) to give it a try. - 06/07/2023: Join our [Discord server](https://discord.gg/ejer2MAB8N) for lively discussions! - 03/07/2023: Try text-to-3D online in [HuggingFace Spaces](https://huggingface.co/spaces/bennyguo/threestudio) or using our [self-hosted service](http://t23-g-01.threestudio.ai) (GPU support from Tencent). To host the web interface locally, see [here](https://github.com/threestudio-project/threestudio#gradio-web-interface). - 20/06/2023: Implementations of Instruct-NeRF2NeRF and Control4D for high-fidelity 3D editing! Follow the instructions for [Control4D](https://github.com/threestudio-project/threestudio#control4d-) and [Instruct-NeRF2NeRF](https://github.com/threestudio-project/threestudio#instructnerf2nerf-) to give it a try. - 14/06/2023: Implementation of TextMesh! Follow the instructions [here](https://github.com/threestudio-project/threestudio#textmesh-) to give it a try. - 14/06/2023: Implementation of [prompt debiasing](https://arxiv.org/abs/2303.15413) and [Perp-Neg](https://perp-neg.github.io/)! Follow the instructions [here](https://github.com/threestudio-project/threestudio#tips-on-improving-quality) to give it a try. - 29/05/2023: An experimental implementation of using [Zero-1-to-3](https://zero123.cs.columbia.edu/) for 3D generation from a single image! Follow the instructions [here](https://github.com/threestudio-project/threestudio#zero-1-to-3-) to give it a try. - 26/05/2023: Implementation of [ProlificDreamer](https://ml.cs.tsinghua.edu.cn/prolificdreamer/)! Follow the instructions [here](https://github.com/threestudio-project/threestudio#prolificdreamer-) to give it a try. - 14/05/2023: You can experiment with the SDS loss on 2D images using our [2dplayground](2dplayground.ipynb). - 13/05/2023: You can now try threestudio on [Google Colab](https://colab.research.google.com/github/threestudio-project/threestudio/blob/main/threestudio.ipynb)! - 11/05/2023: We now support exporting textured meshes! See [here](https://github.com/threestudio-project/threestudio#export-meshes) for instructions.  ## Installation See [installation.md](docs/installation.md) for additional information, including installation via Docker. The following steps have been tested on Ubuntu20.04. - You must have an NVIDIA graphics card with at least 6GB VRAM and have [CUDA](https://developer.nvidia.com/cuda-downloads) installed. - Install `Python >= 3.8`. - (Optional, Recommended) Create a virtual environment: ```sh python3 -m virtualenv venv . venv/bin/activate # Newer pip versions, e.g. pip-23.x, can be much faster than old versions, e.g. pip-20.x. # For instance, it caches the wheels of git packages to avoid unnecessarily rebuilding them later. python3 -m pip install --upgrade pip ``` - Install `PyTorch >= 1.12`. We have tested on `torch1.12.1+cu113` and `torch2.0.0+cu118`, but other versions should also work fine. ```sh # torch1.12.1+cu113 pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113 # or torch2.0.0+cu118 pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118 ``` - (Optional, Recommended) Install ninja to speed up the compilation of CUDA extensions: ```sh pip install ninja ``` - Install dependencies: ```sh pip install -r requirements.txt ``` - (Optional) `tiny-cuda-nn` installation might require downgrading pip to 23.0.1 - (Optional, Recommended) The best-performing models in threestudio use the newly-released T2I model [DeepFloyd IF](https://github.com/deep-floyd/IF), which currently requires signing a license agreement. If you would like to use these models, you need to [accept the license on the model card of DeepFloyd IF](https://huggingface.co/DeepFloyd/IF-I-XL-v1.0), and login into the Hugging Face hub in the terminal by `huggingface-cli login`. - For contributors, see [here](https://github.com/threestudio-project/threestudio#contributing-to-threestudio). ## Quickstart Here we show some basic usage of threestudio. First let's train a DreamFusion model to create a classic pancake bunny. **If you are experiencing unstable connections with Hugging Face, we suggest you either (1) setting environment variable `TRANSFORMERS_OFFLINE=1 DIFFUSERS_OFFLINE=1 HF_HUB_OFFLINE=1` before your running command after all needed files have been fetched on the first run, to prevent from connecting to Hugging Face each time you run, or (2) downloading the guidance model you used to a local folder following [here](https://huggingface.co/docs/huggingface_hub/v0.14.1/guides/download#download-an-entire-repository) and [here](https://huggingface.co/docs/huggingface_hub/v0.14.1/guides/download#download-files-to-local-folder), and set `pretrained_model_name_or_path` of the guidance and the prompt processor to the local path.** ```sh # if you have agreed the license of DeepFloyd IF and have >20GB VRAM # please try this configuration for higher quality python launch.py --config configs/dreamfusion-if.yaml --train --gpu 0 system.prompt_processor.prompt="a zoomed out DSLR photo of a baby bunny sitting on top of a stack of pancakes" # otherwise you could try with the Stable Diffusion model, which fits in 6GB VRAM python launch.py --config configs/dreamfusion-sd.yaml --train --gpu 0 system.prompt_processor.prompt="a zoomed out DSLR photo of a baby bunny sitting on top of a stack of pancakes" ``` threestudio uses [OmegaConf](https://github.com/omry/omegaconf) for flexible configurations. You can easily change any configuration in the YAML file by specifying arguments without `--`, for example the specified prompt in the above cases. For all supported configurations, please see our [documentation](https://github.com/threestudio-project/threestudio/blob/main/DOCUMENTATION.md). The training lasts for 10,000 iterations. You can find visualizations of the current status in the trial directory which defaults to `[exp_root_dir]/[name]/[tag]@[timestamp]`, where `exp_root_dir` (`outputs/` by default), `name` and `tag` can be set in the configuration file. A 360-degree video will be generated after the training is completed. In training, press `ctrl+c` one time will stop training and head directly to the test stage which generates the video. Press `ctrl+c` the second time to fully quit the program. ### Multi-GPU training Multi-GPU training is supported, but may still be [buggy](https://github.com/threestudio-project/threestudio/issues/195). Note that `data.batch_size` is the batch size **per rank (device)**. Also remember to - Set `data.n_val_views` to be a multiple of the number of GPUs. - Set a unique `tag` as timestamp is disabled in multi-GPU training and will not be appended after the tag. If you the same tag as previous trials, saved config files, code and visualizations will be overridden. ```sh # this results in an effective batch size of 4 (number of GPUs) * 2 (data.batch_size) = 8 python launch.py --config configs/dreamfusion-if.yaml --train --gpu 0,1,2,3 system.prompt_processor.prompt="a zoomed out DSLR photo of a baby bunny sitting on top of a stack of pancakes" data.batch_size=2 data.n_val_views=4 ``` If you define the `CUDA_VISIBLE_DEVICES` environment variable before you call `launch.py`, you don't need to specify `--gpu` - this will use all available GPUs from `CUDA_VISIBLE_DEVICES`. For instance, the following command will automatically use GPUs 3 and 4: `CUDA_VISIBLE_DEVICES=3,4 python launch.py --config configs/dreamfusion-if.yaml --train system.prompt_processor.prompt="a zoomed out DSLR photo of a baby bunny sitting on top of a stack of pancakes"` This is particularly useful if you run `launch.py` in a cluster using a command that automatically picks GPU(s) and exports their IDs through CUDA_VISIBLE_DEVICES, e.g. through SLURM: ```bash cd git/threestudio . venv/bin/activate srun --account mod3d --partition=g40 --gpus=1 --job-name=3s_bunny python launch.py --config configs/dreamfusion-if.yaml --train system.prompt_processor.prompt="a zoomed out DSLR photo of a baby bunny sitting on top of a stack of pancakes" ``` ### Resume from checkpoints If you want to resume from a checkpoint, do: ```sh # resume training from the last checkpoint, you may replace last.ckpt with any other checkpoints python launch.py --config path/to/trial/dir/configs/parsed.yaml --train --gpu 0 resume=path/to/trial/dir/ckpts/last.ckpt # if the training has completed, you can still continue training for a longer time by setting trainer.max_steps python launch.py --config path/to/trial/dir/configs/parsed.yaml --train --gpu 0 resume=path/to/trial/dir/ckpts/last.ckpt trainer.max_steps=20000 # you can also perform testing using resumed checkpoints python launch.py --config path/to/trial/dir/configs/parsed.yaml --test --gpu 0 resume=path/to/trial/dir/ckpts/last.ckpt # note that the above commands use parsed configuration files from previous trials # which will continue using the same trial directory # if you want to save to a new trial directory, replace parsed.yaml with raw.yaml in the command # only load weights from saved checkpoint but dont resume training (i.e. dont load optimizer state): python launch.py --config path/to/trial/dir/configs/parsed.yaml --train --gpu 0 system.weights=path/to/trial/dir/ckpts/last.ckpt ``` ### Export Meshes To export the scene to texture meshes, use the `--export` option. We currently support exporting to obj+mtl, or obj with vertex colors. ```sh # this uses default mesh-exporter configurations which exports obj+mtl python launch.py --config path/to/trial/dir/configs/parsed.yaml --export --gpu 0 resume=path/to/trial/dir/ckpts/last.ckpt system.exporter_type=mesh-exporter # specify system.exporter.fmt=obj to get obj with vertex colors # you may also add system.exporter.save_uv=false to accelerate the process, suitable for a quick peek of the result python launch.py --config path/to/trial/dir/configs/parsed.yaml --export --gpu 0 resume=path/to/trial/dir/ckpts/last.ckpt system.exporter_type=mesh-exporter system.exporter.fmt=obj # for NeRF-based methods (DreamFusion, Magic3D coarse, Latent-NeRF, SJC) # you may need to adjust the isosurface threshold (25 by default) to get satisfying outputs # decrease the threshold if the extracted model is incomplete, increase if it is extruded python launch.py --config path/to/trial/dir/configs/parsed.yaml --export --gpu 0 resume=path/to/trial/dir/ckpts/last.ckpt system.exporter_type=mesh-exporter system.geometry.isosurface_threshold=10. # use marching cubes of higher resolutions to get more detailed models python launch.py --config path/to/trial/dir/configs/parsed.yaml --export --gpu 0 resume=path/to/trial/dir/ckpts/last.ckpt system.exporter_type=mesh-exporter system.geometry.isosurface_method=mc-cpu system.geometry.isosurface_resolution=256 ``` For all the options you can specify when exporting, see [the documentation](https://github.com/threestudio-project/threestudio/blob/main/DOCUMENTATION.md#exporters). See [here](https://github.com/threestudio-project/threestudio#supported-models) for example running commands of all our supported models. Please refer to [here](https://github.com/threestudio-project/threestudio#tips-on-improving-quality) for tips on getting higher-quality results, and [here](https://github.com/threestudio-project/threestudio#vram-optimization) for reducing VRAM usage. ### Gradio Web Interface Launch the Gradio web interface by ``` python gradio_app.py launch ``` Parameters: - `--listen`: listens to all addresses by setting `server_name="0.0.0.0"` when launching the Gradio app. - `--self-deploy`: enables changing arbitrary configurations directly from the web. - `--save`: enables checkpoint saving. For feature requests, bug reports, or discussions about technical problems, please [file an issue](https://github.com/threestudio-project/threestudio/issues/new). In case you want to discuss the generation quality or showcase your generation results, please feel free to participate in the [discussion panel](https://github.com/threestudio-project/threestudio/discussions). ## Supported Models ### Score Distillation via Reparametrized DDIM (SDI) [](https://arxiv.org/abs/2405.15891) SDI suggests to reconsider the approach to sampling the noise term in Dreamfusion. The paper demonstrates that score distillation process can be seen as a reparametrization of 2D image sampling algorithms. In that case the noise added on each step of score distillation should be of a very particular form. Noise in Dreamfusion (SDS), however, is sampled randomly, what causes over-blurring. SDI approximates the correct noise term by inverting the DDIM process. Notable differences from the paper: N/A. Pros: * High quality of the textures * Sharp geometric details Cons: * Slower than SDS (1.5x times) due to additional inversion. Still faster then prolific dreamer due to lower number of steps * Requires more VRAM than SDS due to higher resolution rendering. Decrease the resolution to fit to smaller GPUs. **Results obtained in threestudio (Stable Diffusion, 512x512)** **Example running commands** ```sh python launch.py --config configs/sdi.yaml --train --gpu 0 system.prompt_processor.prompt="pumpkin head zombie, skinny, highly detailed, photorealistic" python launch.py --config configs/sdi.yaml --train --gpu 1 system.prompt_processor.prompt="a photograph of a ninja" python launch.py --config configs/sdi.yaml --train --gpu 2 system.prompt_processor.prompt="a zoomed out DSLR photo of a hamburger" python launch.py --config configs/sdi.yaml --train --gpu 3 system.prompt_processor.prompt="bagel filled with cream cheese and lox" ``` ### ProlificDreamer [](https://arxiv.org/abs/2305.16213) **This is an unofficial experimental implementation! Please refer to [https://github.com/thu-ml/prolificdreamer](https://github.com/thu-ml/prolificdreamer) for official code release.** **Results obtained by threestudio (Stable Diffusion, 256x256 Stage1)** https://github.com/threestudio-project/threestudio/assets/19284678/27b42d8f-4aa4-4b47-8ea0-0f77db90fd1e https://github.com/threestudio-project/threestudio/assets/19284678/ffcbbb01-3817-4663-a2bf-5e21a076bc3d **Results obtained by threestudio (Stable Diffusion, 256x256 Stage1, 512x512 Stage2+3)** https://github.com/threestudio-project/threestudio/assets/19284678/cfab881e-18dc-45fc-8384-7476f835b36e Notable differences from the paper: - ProlificDreamer adopts a two-stage sampling strategy with 64 coarse samples and 32 fine samples, while we only use 512 coarse samples. - In the first stage, we only render 64x64 images at the first 5000 iterations. After that, as the empty space has been effectively pruned, rendering 512x512 images wouldn't cost too much VRAM. - We currently don't support multiple particles. ```sh # --------- Stage 1 (NeRF) --------- # # object generation with 512x512 NeRF rendering, ~30GB VRAM python launch.py --config configs/prolificdreamer.yaml --train --gpu 0 system.prompt_processor.prompt="a pineapple" # if you don't have enough VRAM, try training with 64x64 NeRF rendering, ~15GB VRAM python launch.py --config configs/prolificdreamer.yaml --train --gpu 0 system.prompt_processor.prompt="a pineapple" data.width=64 data.height=64 data.batch_size=1 # using the same model for pretrained and LoRA enables 64x64 training with <10GB VRAM # but the quality is worse due to the use of an epsilon prediction model for LoRA training python launch.py --config configs/prolificdreamer.yaml --train --gpu 0 system.prompt_processor.prompt="a pineapple" data.width=64 data.height=64 data.batch_size=1 system.guidance.pretrained_model_name_or_path_lora="stabilityai/stable-diffusion-2-1-base" # Using patch-based renderer to reduce memory consume, 512x512 resolution, ~20GB VRAM python launch.py --config configs/prolificdreamer-patch.yaml --train --gpu 0 system.prompt_processor.prompt="a pineapple" # scene generation with 512x512 NeRF rendering, ~30GB VRAM python launch.py --config configs/prolificdreamer-scene.yaml --train --gpu 0 system.prompt_processor.prompt="Inside of a smart home, realistic detailed photo, 4k" # --------- Stage 2 (Geometry Refinement) --------- # # refine geometry with 512x512 rasterization, Stable Diffusion SDS guidance python launch.py --config configs/prolificdreamer-geometry.yaml --train --gpu 0 system.prompt_processor.prompt="a pineapple" system.geometry_convert_from=path/to/stage1/trial/dir/ckpts/last.ckpt # --------- Stage 3 (Texturing) --------- # # texturing with 512x512 rasterization, Stable Difusion VSD guidance python launch.py --config configs/prolificdreamer-texture.yaml --train --gpu 0 system.prompt_processor.prompt="a pineapple" system.geometry_convert_from=path/to/stage2/trial/dir/ckpts/last.ckpt ``` ### HiFA [](https://arxiv.org/abs/2305.18766) **This is a re-implementation, missing some improvements from the original paper(coarse-to-fine NeRF sampling, kernel smoothing). For original results, please refer to [https://github.com/JunzheJosephZhu/HiFA](https://github.com/JunzheJosephZhu/HiFA)** HiFA is more like a suite of improvements including image space SDS, z-variance loss, and noise strength annealing. It is compatible with most optimization-based methods. Therefore, we provide three variants based on DreamFusion, ProlificDreamer, and Magic123. We provide a unified guidance config as well as an SDS/VSD guidance config for the DreamFusion and ProlificDreamer variants, both configs should achieve the same results. Additionally, we also make HiFA compatible with ProlificDreamer-scene. **Results obtained by threestudio(Dreamfusion-HiFA, 512x512)** https://github.com/threestudio-project/threestudio/assets/24391451/c0030c66-0691-4ec2-8b79-d933101864a0 **Results obtained by threestudio(ProlificDreamer-HiFA, 512x512)** https://github.com/threestudio-project/threestudio/assets/24391451/ff5dc4d0-d7d7-4a73-964e-84b8c48e2907 **Results obtained by threestudio(Magic123-HiFA, 512x512)** https://github.com/threestudio-project/threestudio/assets/24391451/eb6f2f74-9143-4e26-8429-e300ad2d2b80 **Example running commands** ```sh # ------ DreamFusion-HiFA ------- # (similar to original paper) python launch.py --config configs/hifa.yaml --train --gpu 0 system.prompt_processor.prompt="a plate of delicious tacos" python launch.py --config configs/experimental/unified-guidance/hifa.yaml --train --gpu 0 system.prompt_processor.prompt="a plate of delicious tacos" # ------ ProlificDreamer-HiFA ------- # python launch.py --config configs/prolificdreamer-hifa.yaml --train --gpu 0 system.prompt_processor.prompt="a plate of delicious tacos" python launch.py --config configs/experimental/unified-guidance/prolificdreamer-hifa.yaml --train --gpu 0 system.prompt_processor.prompt="a plate of delicious tacos" # ------ ProlificDreamer-scene-HiFA ------- # python launch.py --config configs/prolificdreamer-scene-hifa.yaml --train --gpu 0 system.prompt_processor.prompt="A DSLR photo of a hamburger inside a restaurant" # ------ Magic123-HiFA ------ # python launch.py --config configs/magic123-hifa-coarse-sd.yaml --train --gpu 0 data.image_path=load/images/firekeeper_rgba.png system.prompt_processor.prompt="a toy figure of firekeeper from dark souls" # We included a config for magic123's refine stage, but didn't really run it, since the coarse stage result already looks pretty decent. ``` **Tips** - If the generated object's color seems oversaturated, decrease lambda_sds_img(or lambda_sd_img if using unified guidance). - If the generated object looks cloudy, increase lamda_z_variance. If the shape becomes corrupted, decrease lambda_z_variance. - If the generated object overall seems to have high luminance, increase min_step_percent. - Make sure sqrt_anneal and use_img_loss are both set to True. - Check out the [original repo](https://github.com/JunzheJosephZhu/HiFA)! The results are better. - **If you are using sqrt_anneal, make sure system.guidance.trainer_max_steps is equal to trainer.max_steps, so noise strength annealing works correctly** ### DreamFusion [](https://arxiv.org/abs/2209.14988) **Results obtained by threestudio (DeepFloyd IF, batch size 8)** https://user-images.githubusercontent.com/19284678/236694848-38ae4ea4-554b-4c9d-b4c7-fba5bee3acb3.mp4 **Notable differences from the paper** - We use open-source T2I models (StableDiffusion, DeepFloyd IF), while the paper uses Imagen. - We use a guidance scale of 20 for DeepFloyd IF, while the paper uses 100 for Imagen. - We do not use sigmoid to normalize the albedo color but simply scale the color from `[-1,1]` to `[0,1]`, as we find this help convergence. - We use HashGrid encoding and uniformly sample points along rays, while the paper uses Integrated Positional Encoding and sampling strategy from MipNeRF360. - We adopt camera settings and density initialization strategy from Magic3D, which is slightly different from the DreamFusion paper. - Some hyperparameters are different, such as the weighting of loss terms. **Example running commands** ```sh # uses DeepFloyd IF, requires ~15GB VRAM to extract text embeddings and ~10GB VRAM in training # here we adopt random background augmentation to improve geometry quality python launch.py --config configs/dreamfusion-if.yaml --train --gpu 0 system.prompt_processor.prompt="a delicious hamburger" system.background.random_aug=true # uses StableDiffusion, requires ~6GB VRAM in training python launch.py --config configs/dreamfusion-sd.yaml --train --gpu 0 system.prompt_processor.prompt="a delicious hamburger" ``` **Tips** - DeepFloyd IF performs **way better than** StableDiffusion. - Validation shows albedo color before `system.material.ambient_only_steps` and shaded color after that. - Try increasing/decreasing `system.loss.lambda_sparsity` if your scene is stuffed with floaters/becoming empty. - Try increasing/decreasing `system.loss.lambda_orient` if you object is foggy/over-smoothed. - Try replacing the background to random colors with a probability 0.5 by setting `system.background.random_aug=true` if you find the model incorrectly treats the background as part of the object. - DeepFloyd IF uses T5-XXL as its text encoder, which consumes ~15GB VRAM even when using 8-bit quantization. This is currently the bottleneck for training with less VRAM. If anyone knows how to run the text encoder with less VRAM, please file an issue. We're also trying to push the text encoder to [Replicate](https://replicate.com/) to enable extracting text embeddings via API, but are having some network connection issues. Please [contact bennyguo](mailto:imbennyguo@gmail.com) if you would like to help out. ### Magic3D [](https://arxiv.org/abs/2211.10440) **Results obtained by threestudio (DeepFloyd IF, batch size 8; first row: coarse, second row: refine)** https://user-images.githubusercontent.com/19284678/236694858-0ed6939e-cd7a-408f-a94b-406709ae90c0.mp4 **Notable differences from the paper** - We use open-source T2I models (StableDiffusion, DeepFloyd IF) for the coarse stage, while the paper uses eDiff-I. - In the coarse stage, we use a guidance scale of 20 for DeepFloyd IF, while the paper uses 100 for eDiff-I. - In the coarse stage, we use analytic normal, while the paper uses predicted normal. - In the coarse stage, we use orientation loss as in DreamFusion, while the paper does not. - There are many things that are omitted from the paper such as the weighting of loss terms and the DMTet grid resolution, which could be different. **Example running commands** First train the coarse stage NeRF: ```sh # uses DeepFloyd IF, requires ~15GB VRAM to extract text embeddings and ~10GB VRAM in training python launch.py --config configs/magic3d-coarse-if.yaml --train --gpu 0 system.prompt_processor.prompt="a delicious hamburger" # uses StableDiffusion, requires ~6GB VRAM in training python launch.py --config configs/magic3d-coarse-sd.yaml --train --gpu 0 system.prompt_processor.prompt="a delicious hamburger" ``` Then convert the NeRF from the coarse stage to DMTet and train with differentiable rasterization: ```sh # the refinement stage uses StableDiffusion, and requires ~5GB VRAM in training python launch.py --config configs/magic3d-refine-sd.yaml --train --gpu 0 system.prompt_processor.prompt="a delicious hamburger" system.geometry_convert_from=path/to/coarse/stage/trial/dir/ckpts/last.ckpt # if you're unsatisfied with the surface extracted using the default threshold (25) # you can specify a threshold value using `system.geometry_convert_override` # decrease the value if the extracted surface is incomplete, increase if it is extruded python launch.py --config configs/magic3d-refine-sd.yaml --train --gpu 0 system.prompt_processor.prompt="a delicious hamburger" system.geometry_convert_from=path/to/coarse/stage/trial/dir/ckpts/last.ckpt system.geometry_convert_override.isosurface_threshold=10. ``` **Tips** - For the coarse stage, DeepFloyd IF performs **way better than** StableDiffusion. - Magic3D uses a neural network to predict the surface normal, which may not resemble the true geometric normal and degrade geometry quality, so we use analytic normal instead. - Try increasing/decreasing `system.loss.lambda_sparsity` if your scene is stuffed with floaters/becoming empty. - Try increasing/decreasing `system.loss.lambda_orient` if you object is foggy/over-smoothed. - Try replacing the background with random colors with a probability 0.5 by setting `system.background.random_aug=true` if you find the model incorrectly treats the background as part of the object. ### Score Jacobian Chaining [](https://arxiv.org/abs/2212.00774) **Results obtained by threestudio (Stable Diffusion)** https://user-images.githubusercontent.com/19284678/236694871-87a247c1-2d3d-4cbf-89df-450bfeac3aca.mp4 Notable differences from the paper: N/A. **Example running commands** ```sh # train with sjc guidance in latent space python launch.py --config configs/sjc.yaml --train --gpu 0 system.prompt_processor.prompt="A high quality photo of a delicious burger" # train with sjc guidance in latent space, trump figure python launch.py --config configs/sjc.yaml --train --gpu 0 system.prompt_processor.prompt="Trump figure" trainer.max_steps=30000 system.loss.lambda_emptiness="[15000,10000.0,200000.0,15001]" system.optimizer.params.background.lr=0.05 seed=42 ``` **Tips** - SJC uses subpixel rendering which decodes a `128x128` latent feature map for better visualization quality. You can turn off this feature by `system.subpixel_rendering=false` to save VRAM in validation/testing. ### Latent-NeRF [](https://arxiv.org/abs/2211.07600) **Results obtained by threestudio (Stable Diffusion)** https://user-images.githubusercontent.com/19284678/236694876-5a270347-6a41-4429-8909-44c90c554e06.mp4 Notable differences from the paper: N/A. We currently only implement Latent-NeRF for text-guided and Sketch-Shape for (text,shape)-guided 3D generation. Latent-Paint is not implemented yet. **Example running commands** ```sh # train Latent-NeRF in Stable Diffusion latent space python launch.py --config configs/latentnerf.yaml --train --gpu 0 system.prompt_processor.prompt="a delicious hamburger" # refine Latent-NeRF in RGB space python launch.py --config configs/latentnerf-refine.yaml --train --gpu 0 system.prompt_processor.prompt="a delicious hamburger" system.weights=path/to/latent/stage/trial/dir/ckpts/last.ckpt # train Sketch-Shape in Stable Diffusion latent space python launch.py --config configs/sketchshape.yaml --train --gpu 0 system.guide_shape=load/shapes/teddy.obj system.prompt_processor.prompt="a teddy bear in a tuxedo" # refine Sketch-Shape in RGB space python launch.py --config configs/sketchshape-refine.yaml --train --gpu 0 system.guide_shape=load/shapes/teddy.obj system.prompt_processor.prompt="a teddy bear in a tuxedo" system.weights=path/to/latent/stage/trial/dir/ckpts/last.ckpt ``` ### Fantasia3D [](https://arxiv.org/abs/2303.13873) **Results obtained by threestudio (Stable Diffusion)** https://user-images.githubusercontent.com/19284678/236694880-33b0db21-4530-47f1-9c3b-c70357bc84b3.mp4 **Results obtained by threestudio (Stable Diffusion, mesh initialization)** https://github.com/threestudio-project/threestudio/assets/19284678/762903c1-665b-47b5-a2c2-bd7021a9e548.mp4