magic123 0.1.1

Magic123: one image to high-quality 3D object generation using 2D and 3D diffusion priors.

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors [ICLR 2024]

arXiv | webpage

Guocheng Qian ^1,2, Jinjie Mai ¹, Abdullah Hamdi ³, Jian Ren ², Aliaksandr Siarohin ², Bing Li ¹, Hsin-Ying Lee ², Ivan Skorokhodov ^1,2, Peter Wonka ¹, Sergey Tulyakov ², Bernard Ghanem ¹

¹ King Abdullah University of Science and Technology (KAUST), ² Snap Inc., ³ Visual Geometry Group, University of Oxford

Training convergence of a demo example:

Compare Magic123 without textual inversion with abaltions using only 2D prior (SDS) or using only 3D prior (Zero123):

https://github.com/guochengqian/Magic123/assets/48788073/c91f4c81-8c2c-4f84-8ce1-420c12f7e886

Effects of Joint Prior. Increasing the strength of 2D prior leads to more imagination, more details, and less 3D consistencies.

https://github.com/guochengqian/Magic123/assets/48788073/98cb4dd7-7bf3-4179-9b6d-e8b47d928a68

Official PyTorch Implementation of Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors. Code is built upon Stable-DreamFusion repo.

NEWS:

• [2024/01/16] Magic123 gets accepted to ICLR24
• [2023/07/25] Code is available at GitHub
• [2023/07/03] Paper is available at arXiv
• [2023/06/25] Much better performance than the submitted version is achieved by 1）reimplementing Magic123 using Stable DreamFusion code, 2）fixing some gradient issues, 3）leveraging the tricks
• [2023] Initial version of Magic123 submitted to conference

Install

We only test on Ubuntu system. Make sure git, wget, Eigen are installed.

apt update && apt upgrade
apt install git wget libeigen3-dev python3-dev python3-venv -y

Install from PyPI

python -m venv venv_magic123
source venv_magic123/bin/activate
python -m pip install --upgrade pip setuptools wheel
python -m pip install magic123

# Build the CUDA extensions after installing PyTorch. 4090/4090 Ti uses SM 8.9.
export CUDA_HOME=${CUDA_HOME:-/usr/local/cuda-12.9}
export PATH=$CUDA_HOME/bin:$PATH
export TORCH_CUDA_ARCH_LIST=${TORCH_CUDA_ARCH_LIST:-8.9}
git clone https://github.com/guochengqian/Magic123.git
cd Magic123
bash scripts/install_ext.sh

The PyPI package ships the Python project and source code for the CUDA extensions. The extension modules are compiled locally because PyTorch CUDA extension wheels must match the local Python, PyTorch, CUDA toolkit, compiler, and GPU architecture.

CUDA and Windows notes

• NVIDIA RTX 4090 and 4090 Ti builds should set TORCH_CUDA_ARCH_LIST=8.9.
• The CUDA toolkit must provide nvcc; check with python scripts/check_cuda_build_env.py.
• Building the PyTorch CUDA extensions also requires Python development headers. On Ubuntu/Debian install python3-dev or the version-specific package such as python3.10-dev.
• On Windows, install Visual Studio Build Tools with the "Desktop development with C++" workload and run installation from a Developer PowerShell or Developer Command Prompt. The setup scripts use vswhere/Visual Studio discovery to locate cl.exe, but CUDA still requires a Visual Studio version supported by the installed CUDA toolkit.
• On Colab, install PyTorch and the matching CUDA toolkit first, then run bash scripts/install_ext.sh. If Colab changes its compiler or CUDA image, rebuild the extensions with --no-cache-dir.

Install Environment

source install.sh

Note: in this install.sh, we use python venv by default. If you prefer conda, uncomment the conda and comment venv in the file and run the same command.

Download pre-trained models

• Zero-1-to-3 for 3D diffusion prior. We use 105000.ckpt by default, reimplementation borrowed from Stable Diffusion repo, and is available in guidance/zero123_utils.py.

  cd pretrained/zero123
  wget https://huggingface.co/cvlab/zero123-weights/resolve/main/105000.ckpt
  cd ../../
  

• MiDaS for depth estimation. We use dpt_beit_large_512.pt. Put it in folder pretrained/midas/

  mkdir -p pretrained/midas
  cd pretrained/midas
  wget https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_beit_large_512.pt
  cd ../../
  

Usage

Preprocess [Optional]

We have included all preprocessed files in ./data directory. Preprocessing is only necessary if you want to test on your own examples. Takes seconds.

Step1: Extract depth

python preprocess_image.py --path /path/to/image 

Step 2: textual inversion [Optional]

Magic123 uses the default textual inversion from diffuers, which consumes around 2 hours on a 32G V100. If you do not want to spend time in this textual inversion, you can: (1) study whether there is other faster textual inversion; or (2) do not use textual inversion in the loss of texture and shape consistencies. To run textual inversion:

bash scripts/textual_inversion/textual_inversion.sh $GPU_IDX runwayml/stable-diffusion-v1-5 /path/to/example/rgba.png /path/to/save $token_name $init_token --max_train_steps 5000

$token_name is a the special token, usually name that by examplename $init_token is a single token to describe the image using natural language

For example:

bash scripts/textual_inversion/textual_inversion.sh runwayml/stable-diffusion-v1-5 data/demo/a-full-body-ironman/rgba.png out/textual_inversion/ironman _ironman_ ironman --max_train_steps 3000

Don't forget to move the final learned_embeds.bin under data/demo/a-full-body-ironman/

Run

Run Magic123 for a single example

Takes ~40 mins for the coarse stage and ~20 mins for the second stage on a 32G V100.

bash scripts/magic123/run_both_priors.sh $GPU_NO $JOBNAME_First_Stage $JOBNAME_Second_Stage $PATH_to_Example_Directory $IMAGE_BASE_NAME $Enable_First_Stage $Enable_Second_Stage {More_Arugments}

As an example, run Magic123 in the dragon example using both stages in GPU 0 and set the jobname for the first stage as nerf and the jobname for the second stage as dmtet, by the following command:

bash scripts/magic123/run_both_priors.sh 0 nerf dmtet data/realfusion15/metal_dragon_statue 1 1 

More arguments (e.g. --lambda_guidance 1 40) can be appended to the command line such as:

bash scripts/magic123/run_both_priors.sh 0 nerf dmtet data/realfusion15/metal_dragon_statue 1 1 --lambda_guidance 1 40

Run Magic123 for a group of examples

• Run all examples in a folder, check the scripts scripts/magic123/run_folder_both_priors.sh
• Run all examples in a given list, check the scripts scripts/magic123/run_list_both_priors.sh

Run Magic123 on a single example without textual inversion

textual inversion is tedious (requires ~2.5 hours optimization), if you want to test Magic123 quickly on your own example without textual inversion (might degrade the performance), try the following:

• first, foreground and depth estimation

  python preprocess_image.py --path data/demo/a-full-body-ironman/main.png
  

• Run Magic123 coarse stage without textual inversion, takes ~40 mins

  export RUN_ID='default-a-full-body-ironman'
  export DATA_DIR='data/demo/a-full-body-ironman'
  export IMAGE_NAME='rgba.png'
  export FILENAME=$(basename $DATA_DIR)
  export dataset=$(basename $(dirname $DATA_DIR))
  CUDA_VISIBLE_DEVICES=0 python main.py -O \
  --text "A high-resolution DSLR image of a full body ironman" \
  --sd_version 1.5 \
  --image ${DATA_DIR}/${IMAGE_NAME} \
  --workspace out/magic123-${RUN_ID}-coarse/$dataset/magic123_${FILENAME}_${RUN_ID}_coarse \
  --optim adam \
  --iters 5000 \
  --guidance SD zero123 \
  --lambda_guidance 1.0 40 \
  --guidance_scale 100 5 \
  --latent_iter_ratio 0 \
  --normal_iter_ratio 0.2 \
  --t_range 0.2 0.6 \
  --bg_radius -1 \
  --save_mesh
  

• Run Magic123 fine stage without textual inversion, takes around ~20 mins

  export RUN_ID='default-a-full-body-ironman'
  export RUN_ID2='dmtet'
  export DATA_DIR='data/demo/a-full-body-ironman'
  export IMAGE_NAME='rgba.png'
  export FILENAME=$(basename $DATA_DIR)
  export dataset=$(basename $(dirname $DATA_DIR))
  CUDA_VISIBLE_DEVICES=0 python main.py -O \
  --text "A high-resolution DSLR image of a full body ironman" \
  --sd_version 1.5 \
  --image ${DATA_DIR}/${IMAGE_NAME} \
  --workspace out/magic123-${RUN_ID}-${RUN_ID2}/$dataset/magic123_${FILENAME}_${RUN_ID}_${RUN_ID2} \
  --dmtet --init_ckpt out/magic123-${RUN_ID}-coarse/$dataset/magic123_${FILENAME}_${RUN_ID}_coarse/checkpoints/magic123_${FILENAME}_${RUN_ID}_coarse.pth \
  --iters 5000 \
  --optim adam \
  --known_view_interval 4 \
  --latent_iter_ratio 0 \
  --guidance SD zero123 \
  --lambda_guidance 1e-3 0.01 \
  --guidance_scale 100 5 \
  --rm_edge \
  --bg_radius -1 \
  --save_mesh 
  

10-step CUDA smoke test on the Ironman example

After installing dependencies and CUDA extensions, run a short 4090-class smoke test:

bash scripts/magic123/smoke_train_ironman_10_steps.sh 0 out/smoke-ironman-10-steps

The script sets TORCH_CUDA_ARCH_LIST=8.9, runs scripts/check_cuda_build_env.py, trains for 10 iterations, and saves outputs under the workspace argument. The default workspace is out/smoke-ironman-10-steps.

Outputs and mesh export

Use --save_mesh to export the object. Coarse NeRF runs save mesh files under the run workspace, typically in a mesh/ subdirectory, including mesh.obj, material, and texture files. DMTet fine-stage runs also write the final textured .obj assets under the fine-stage workspace. The .mp4 files are render previews; the .obj plus its .mtl and texture images are the 3D object files to open in Blender, MeshLab, or another mesh viewer.

Evaluation metrics

Metric utilities are in all_metrics/metric_utils.py, with example invocations in all_metrics/test.sh. PSNR, LPIPS, and related image metrics compare rendered prediction images against ground-truth images at matched viewpoints. For benchmark datasets such as RealFusion and NeRF4, the prediction images are rendered from the reconstructed 3D object at the dataset evaluation cameras, and the GT images are the corresponding held-out/reference dataset views. If you evaluate a custom object, render the reconstruction from the same camera poses as your GT images before running the metric script.

Run ablation studies

• Run Magic123 with only 2D prior with textual inversion (Like RealFusion but we achieve much better performance through training stragies and the coarse-to-fine pipeline)

  bash scripts/magic123/run_2dprior.sh 0 nerf dmtet data/realfusion15/metal_dragon_statue 1 1
  

• Run Magic123 with only 2D prior without textual inversion (Like RealFusion but we achieve much better performance through training stragies and the coarse-to-fine pipeline)

  bash scripts/magic123/run_2dprior_notextinv_ironman.sh 0 default 1 1
  

  note: change the path and the text prompt inside the script if you wana test another example.

• Run Magic123 with only 3D prior (Like Zero-1-to-3 but we achieve much better performance through training stragies and the coarse-to-fine pipeline)

  bash scripts/magic123/run_3dprior.sh 0 nerf dmtet data/demo/a-full-body-ironman 1 1
  

Tips and Tricks

1. Fix camera distance (radius_range) and FOV (fovy_range) and tune the camera polar range (theta_range). Note it is better to keep camera jittering to reduce grid artifacts.
2. Smaller range of time steps for the defusion noise (t_range). We find [0.2, 0.6] gives better performance for image-to-3D tasks.
3. Using normals as latent in the first 2000 improves generated geometry a bit gernerally (but not always). We turn on this for Magic123 corase stage in the script --normal_iter_ratio 0.2
4. We erode segmentation edges (makes the segmentation map 2 pixels shrinked towards internal side) to remove artifacts due to segmentation erros. This is turned on in the fine stage in magic123 in the script through --rm_edge
5. Other general tricks such as improved textual inversion, advanced diffusion prior (DeepFloyd, SD-XL), stronger 3D prior (Zero123-XL), and larger batch size can be adopted as well but not studied in this work.
6. textual inversion is not very necessary for well-known things (e.g. ironman) and easily described textures and geoemtries, since pure texts contains these texture information and will be understood by diffusion models. We use textual inversion by default in all experiments.

Some Projects that use Magic123

1. Threestudio
2. DreamCraft3D

Acknowledgement

This work is build upon Stable DreamFusion, many thanks to the author Kiui Jiaxiang Tang and many other contributors.

• Stable DreamFusion

@misc{stable-dreamfusion,
    Author = {Jiaxiang Tang},
    Year = {2022},
    Note = {https://github.com/ashawkey/stable-dreamfusion},
    Title = {Stable-dreamfusion: Text-to-3D with Stable-diffusion}
}

We also get inspirations from a list of amazing research works and open-source projects, thanks a lot to all the authors for sharing!

• DreamFusion: Text-to-3D using 2D Diffusion

  @article{poole2022dreamfusion,
      author = {Poole, Ben and Jain, Ajay and Barron, Jonathan T. and Mildenhall, Ben},
      title = {DreamFusion: Text-to-3D using 2D Diffusion},
      journal = {arXiv},
      year = {2022},
  }
  

• Magic3D: High-Resolution Text-to-3D Content Creation

  @inproceedings{lin2023magic3d,
     title={Magic3D: High-Resolution Text-to-3D Content Creation},
     author={Lin, Chen-Hsuan and Gao, Jun and Tang, Luming and Takikawa, Towaki and Zeng, Xiaohui and Huang, Xun and Kreis, Karsten and Fidler, Sanja and Liu, Ming-Yu and Lin, Tsung-Yi},
     booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
     year={2023}
   }
  

• Zero-1-to-3: Zero-shot One Image to 3D Object

  @misc{liu2023zero1to3,
      title={Zero-1-to-3: Zero-shot One Image to 3D Object},
      author={Ruoshi Liu and Rundi Wu and Basile Van Hoorick and Pavel Tokmakov and Sergey Zakharov and Carl Vondrick},
      year={2023},
      eprint={2303.11328},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
  }
  

• RealFusion: 360° Reconstruction of Any Object from a Single Image

  @inproceedings{melaskyriazi2023realfusion,
      author = {Melas-Kyriazi, Luke and Rupprecht, Christian and Laina, Iro and Vedaldi, Andrea},
      title = {RealFusion: 360 Reconstruction of Any Object from a Single Image},
      booktitle={CVPR}
      year = {2023},
      url = {https://arxiv.org/abs/2302.10663},
  }
  

• Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior

  @article{tang2023make-it-3d,
      title={Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior},
      author={Tang, Junshu and Wang, Tengfei and Zhang, Bo and Zhang, Ting and Yi, Ran and Ma, Lizhuang and Chen, Dong},
      journal={arXiv preprint arXiv:2303.14184},
      year={2023}
  }
  

• Stable Diffusion and the diffusers library.

  @misc{rombach2021highresolution,
      title={High-Resolution Image Synthesis with Latent Diffusion Models},
      author={Robin Rombach and Andreas Blattmann and Dominik Lorenz and Patrick Esser and Björn Ommer},
      year={2021},
      eprint={2112.10752},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
  }
  
  @misc{von-platen-etal-2022-diffusers,
      author = {Patrick von Platen and Suraj Patil and Anton Lozhkov and Pedro Cuenca and Nathan Lambert and Kashif Rasul and Mishig Davaadorj and Thomas Wolf},
      title = {Diffusers: State-of-the-art diffusion models},
      year = {2022},
      publisher = {GitHub},
      journal = {GitHub repository},
      howpublished = {\url{https://github.com/huggingface/diffusers}}
  }
  

Cite

If you find this work useful, a citation will be appreciated via:

@inproceedings{
Magic123,
title={Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors},
author={Qian, Guocheng and Mai, Jinjie and Hamdi, Abdullah and Ren, Jian and Siarohin, Aliaksandr and Li, Bing and Lee, Hsin-Ying and Skorokhodov, Ivan and Wonka, Peter and Tulyakov, Sergey and Ghanem, Bernard},
booktitle={The Twelfth International Conference on Learning Representations (ICLR)},
year={2024},
url={https://openreview.net/forum?id=0jHkUDyEO9}
}