packaged InstantSplat

This repo is the refactored python training and inference code for InstantSplat. Forked from commit 2c5006d41894d06464da53d5495300860f432872. We refactored the original code following the standard Python package structure, while keeping the algorithms used in the code identical to the original version.

Initialization methods:

• DUST3R (same method used in InstantSplat)
• MAST3R (same method used in Splatt3R)
• COLMAP Sparse reconstruct (same method used in gaussian-splatting)
• COLMAP Dense reconstruct (use patch_match_stereo, stereo_fusion, poisson_mesher and delaunay_mesher in COLMAP to reconstruct dense point cloud for initialization)
• Masking of keypoints during COLMAP feature extraction (just put your mask into mask folder, e.g. for an image data/xxx/input/012.jpg, the mask would be data/xxx/input_mask/012.jpg.png)
• VGGT and VGGT + Colmap Bundle Adjustment according to facebookresearch/vggt/demo_colmap.py
• VGG-T³, VGG-T³ + COLMAP Bundle Adjustment, and VGG-T³ + COLMAP dense reconstruction
• Map-Anything and Map-Anything with external pose/depth priors

Prerequisites

• Pytorch (v2.4 or higher recommended)
• CUDA Toolkit (12.4 recommended, should match with PyTorch version)

Install a colmap executable, e.g. using conda:

conda install conda-forge::colmap

Install mapanything and vggt:

pip install --upgrade "mapanything[all] @ git+https://github.com/facebookresearch/map-anything.git@main"
pip install --upgrade git+https://github.com/facebookresearch/vggt.git@main
pip install --upgrade Pillow hydra-core omegaconf # deps for vggt
pip install --upgrade git+https://github.com/jytime/LightGlue.git#egg=lightglue # deps for vggt

(Optional) Install xformers for faster Depth-Anything V2 inference:

pip install xformers

(Optional) If you have trouble with gaussian-splatting, try to install it from source:

pip install wheel setuptools
pip install --upgrade git+https://github.com/yindaheng98/gaussian-splatting.git@master --no-build-isolation

PyPI Install

pip install --upgrade instantsplat

or build latest from source:

pip install wheel setuptools
pip install --upgrade git+https://github.com/yindaheng98/InstantSplat.git@main --no-build-isolation

Development Install

git clone --recursive https://github.com/yindaheng98/InstantSplat
cd InstantSplat
pip install --target . --upgrade --no-deps .

Download model

wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_224_linear.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_512_linear.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_512_dpt.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/MASt3R/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth
wget -P checkpoints/ https://huggingface.co/depth-anything/Depth-Anything-V2-Small/resolve/main/depth_anything_v2_vits.pth
wget -P checkpoints/ https://huggingface.co/depth-anything/Depth-Anything-V2-Base/resolve/main/depth_anything_v2_vitb.pth
wget -P checkpoints/ https://huggingface.co/depth-anything/Depth-Anything-V2-Large/resolve/main/depth_anything_v2_vitl.pth
wget -P checkpoints/ https://huggingface.co/facebook/VGGT-1B-Commercial/resolve/main/vggt_1B_commercial.pt --header="Authorization: Bearer $HF_TOKEN"
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/MUSt3R/MUSt3R_512.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/Pow3R/Pow3R_ViTLarge_BaseDecoder_512_linear.pth

Configs for map-anything:

git clone --depth 1 --filter=blob:none --sparse https://github.com/facebookresearch/map-anything.git /tmp/map-anything-configs
git -C /tmp/map-anything-configs sparse-checkout set configs
cp -r /tmp/map-anything-configs/configs ./
rm -rf /tmp/map-anything-configs

Running

1. Initialize coarse point cloud and jointly train 3DGS & cameras

# Option 1: init and train in one command
python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --init dust3r
# Option 2: init and train in two separate commands
python -m instantsplat.initialize -d data/sora/santorini/3_views -i dust3r # init coarse point and save as a Colmap workspace at data/sora/santorini/3_views
python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 # train

To enable the optional auto-scaled Depth-Anything V2 wrapper for any initializer, add --with_depth_anything:

python -m instantsplat.initialize -d data/sora/santorini/3_views -i vggt --with_depth_anything
python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --init mapanything --with_depth_anything

Depth format note:

• Depth-Anything V2 saves inverse depth (1 / depth), which matches the default depth supervision used by 3DGS.
• The native depth saved by mapanything, mapanything-external, vggt, and vggttt is regular depth, not inverse depth.
• When training from those native depth maps without --with_depth_anything, add -o depth_ground_truth_is_inversed=False.

Example:

python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --init vggt -o depth_ground_truth_is_inversed=False
python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --init mapanything-external -o depth_ground_truth_is_inversed=False

VGG-T³ initialization

VGG-T³ is available through three initializer names:

• vggttt: directly initializes cameras, a point cloud, and native depth maps. Put images in <scene>/images.
• vggttt-colmap-sparse: runs VGG-T³, VGGSfM tracking, and COLMAP bundle adjustment. Put images in <scene>/input.
• vggttt-colmap-dense: additionally runs COLMAP dense reconstruction. Put images in <scene>/input.

The model weights are downloaded automatically from nvidia/vgg-ttt on first use. For a source checkout, clone with --recursive as shown in Development Install so that the submodules/vgg-ttt submodule is available.

Direct initialization:

python -m instantsplat.initialize -d data/my_scene -i vggttt

The VGG-T³ inference options can be passed with -o:

python -m instantsplat.initialize -d data/my_scene -i vggttt \
  -o num_ttt_steps=2 \
  -o memory_efficient_inference=True \
  -o use_global_pred=True \
  -o offload_to_cpu=True

COLMAP bundle adjustment and dense reconstruction:

python -m instantsplat.initialize -d data/my_scene -i vggttt-colmap-sparse \
  -o colmap_executable="'colmap'"

python -m instantsplat.initialize -d data/my_scene -i vggttt-colmap-dense \
  -o colmap_executable="'colmap'"

VGG-T³ saves regular depth rather than inverse depth. When training without --with_depth_anything, use:

python -m instantsplat.train -s data/my_scene -d output/my_scene -i 1000 \
  --init vggttt -o depth_ground_truth_is_inversed=False

VGG-T³ code and weights are subject to the NVIDIA OneWay Noncommercial License.

2. Render it

python -m gaussian_splatting.render -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --load_camera output/sora/santorini/3_views/cameras.json

See .vscode\launch.json for more command examples.

Usage

See instantsplat.initialize, instantsplat.train and gaussian_splatting.render for full example.

Also check yindaheng98/gaussian-splatting for more detail of training process.

Gaussian models

Use CameraTrainableGaussianModel in yindaheng98/gaussian-splatting

Dataset

Use TrainableCameraDataset in yindaheng98/gaussian-splatting

Initialize coarse point cloud and cameras

from instantsplat.initializer import Dust3rInitializer
image_path_list = [os.path.join(image_folder, file) for file in sorted(os.listdir(image_folder))]
initializer = Dust3rInitializer(...).to(args.device) # see instantsplat/initializer/dust3r/dust3r.py for full options
initialized_point_cloud, initialized_cameras = initializer(image_path_list=image_path_list)

Create camera dataset from initialized cameras:

from instantsplat.initializer import TrainableInitializedCameraDataset
dataset = TrainableInitializedCameraDataset(initialized_cameras).to(device)

Initialize 3DGS from initialized coarse point cloud:

gaussians.create_from_pcd(initialized_point_cloud.points, initialized_point_cloud.colors)

Training

Trainer jointly optimize the 3DGS parameters and cameras, without densification

from instantsplat.trainer import Trainer
trainer = Trainer(
    gaussians,
    dataset=dataset,
    ... # see instantsplat/trainer/trainer.py for full options
)

InstantSplat: Sparse-view SfM-free Gaussian Splatting in Seconds

Free-view Rendering

https://github.com/zhiwenfan/zhiwenfan.github.io/assets/34684115/748ae0de-8186-477a-bab3-3bed80362ad7

TODO List

• Confidence-aware Point Cloud Downsampling
• Support 2D-GS
• Support Mip-Splatting

Acknowledgement

This work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!

• Gaussian-Splatting and diff-gaussian-rasterization
• DUSt3R

Citation

If you find our work useful in your research, please consider giving a star :star: and citing the following paper :pencil:.

@misc{fan2024instantsplat,
        title={InstantSplat: Unbounded Sparse-view Pose-free Gaussian Splatting in 40 Seconds},
        author={Zhiwen Fan and Wenyan Cong and Kairun Wen and Kevin Wang and Jian Zhang and Xinghao Ding and Danfei Xu and Boris Ivanovic and Marco Pavone and Georgios Pavlakos and Zhangyang Wang and Yue Wang},
        year={2024},
        eprint={2403.20309},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
      }