packaged InstantSplat

This repo is the refactored python training and inference code for InstantSplat. Forked from commit 2c5006d41894d06464da53d5495300860f432872. We refactored the original code following the standard Python package structure, while keeping the algorithms used in the code identical to the original version.

Initialization methods:

• DUST3R (same method used in InstantSplat)
• MAST3R (same method used in Splatt3R)
• COLMAP Sparse reconstruct (same method used in gaussian-splatting)
• COLMAP Dense reconstruct (use patch_match_stereo, stereo_fusion, poisson_mesher and delaunay_mesher in COLMAP to reconstruct dense point cloud for initialization)
• Masking during COLMAP feature extraction and dense fusion. See File layout.
• VGGT and VGGT + Colmap Bundle Adjustment according to facebookresearch/vggt/demo_colmap.py
• VGG-T³, VGG-T³ + COLMAP Bundle Adjustment, and VGG-T³ + COLMAP dense reconstruction
• Map-Anything and Map-Anything with external pose/depth priors

Prerequisites

• Pytorch (v2.4 or higher recommended)
• CUDA Toolkit (12.4 recommended, should match with PyTorch version)

Install a colmap executable, e.g. using conda:

conda install conda-forge::colmap

Install mapanything and vggt:

pip install --upgrade "mapanything[all] @ git+https://github.com/facebookresearch/map-anything.git@main"
pip install --upgrade git+https://github.com/facebookresearch/vggt.git@main
pip install --upgrade Pillow hydra-core omegaconf # deps for vggt
pip install --upgrade git+https://github.com/jytime/LightGlue.git#egg=lightglue # deps for vggt

(Optional) Install xformers for faster Depth-Anything V2 inference:

pip install xformers

(Optional) If you have trouble with gaussian-splatting, try to install it from source:

pip install wheel setuptools
pip install --upgrade git+https://github.com/yindaheng98/gaussian-splatting.git@master --no-build-isolation

PyPI Install

pip install --upgrade instantsplat

or build latest from source:

pip install wheel setuptools
pip install --upgrade git+https://github.com/yindaheng98/InstantSplat.git@main --no-build-isolation

Development Install

git clone --recursive https://github.com/yindaheng98/InstantSplat
cd InstantSplat
pip install --target . --upgrade --no-deps . --no-build-isolation

Download model

wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_224_linear.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_512_linear.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_512_dpt.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/MASt3R/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth
wget -P checkpoints/ https://huggingface.co/depth-anything/Depth-Anything-V2-Small/resolve/main/depth_anything_v2_vits.pth
wget -P checkpoints/ https://huggingface.co/depth-anything/Depth-Anything-V2-Base/resolve/main/depth_anything_v2_vitb.pth
wget -P checkpoints/ https://huggingface.co/depth-anything/Depth-Anything-V2-Large/resolve/main/depth_anything_v2_vitl.pth
wget -P checkpoints/ https://huggingface.co/facebook/VGGT-1B-Commercial/resolve/main/vggt_1B_commercial.pt --header="Authorization: Bearer $HF_TOKEN"
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/MUSt3R/MUSt3R_512.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/Pow3R/Pow3R_ViTLarge_BaseDecoder_512_linear.pth

Configs for map-anything:

git clone --depth 1 --filter=blob:none --sparse https://github.com/facebookresearch/map-anything.git /tmp/map-anything-configs
git -C /tmp/map-anything-configs sparse-checkout set configs
cp -r /tmp/map-anything-configs/configs ./
rm -rf /tmp/map-anything-configs

File layout

<name>.jpg is the image path relative to the image folder, including its extension and any subdirectory. images/foo/bar.jpg pairs with image_masks/foo/bar.jpg.png, depths/foo/bar.jpg.tiff, and depth_masks/foo/bar.jpg.tiff. Lists are collected with os.walk. Training reads the written scene as a gaussian-splatting dataset.

Input: images/

dust3r, mast3r, mapanything, mapanything-external, vggt, vggttt:

<data>/
  images/
    <name>.jpg
  image_masks/                 # optional
    <name>.jpg.png

Input: input/

colmap-sparse, colmap-dense, vggt-colmap-sparse, vggt-colmap-dense, vggttt-colmap-sparse, vggttt-colmap-dense, dust3r-align-colmap-sparse, dust3r-align-colmap-dense:

<data>/
  input/
    <name>.jpg
  feature_mask/                # optional; COLMAP feature extraction only
    <name>.jpg.png
  input_mask/                  # optional; undistorted into image_masks/
    <name>.jpg.png

feature_mask and input_mask are independent. A missing mask is skipped.

Output: no depth

dust3r, mast3r, colmap-sparse, vggt-colmap-sparse, vggttt-colmap-sparse, dust3r-align-colmap-sparse:

<data>/
  images/
    <name>.jpg
  image_masks/                 # only when an input mask was provided
    <name>.jpg.png
  sparse/0/
    cameras.bin                # cameras.txt is also accepted when training
    images.bin
    points3D.ply               # if missing, points3D.bin then points3D.txt

Output: depth

vggt, vggttt, mapanything, mapanything-external, colmap-dense, vggt-colmap-dense, vggttt-colmap-dense, dust3r-align-colmap-dense. Same as above, plus:

<data>/
  depths/
    <name>.jpg.tiff
    <name>.jpg.png             # uint8 preview; the loader uses the tiff when both exist
  depth_masks/
    <name>.jpg.tiff

--with_depth_anything adds these depth files for any initializer. That depth is inverse depth. The initializers in this section otherwise write regular depth. See Running.

Running

1. Initialize coarse point cloud and jointly train 3DGS & cameras

# Option 1: init and train in one command
python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --init dust3r
# Option 2: init and train in two separate commands
python -m instantsplat.initialize -d data/sora/santorini/3_views -i dust3r # init coarse point and save as a Colmap workspace at data/sora/santorini/3_views
python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 # train

To enable the optional auto-scaled Depth-Anything V2 wrapper for any initializer, add --with_depth_anything:

python -m instantsplat.initialize -d data/sora/santorini/3_views -i vggt --with_depth_anything
python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --init mapanything --with_depth_anything

Depth format note:

• Depth-Anything V2 saves inverse depth (1 / depth), which matches the default depth supervision used by 3DGS.
• The native depth saved by mapanything, mapanything-external, vggt, vggttt, and COLMAP dense reconstruction is regular depth, not inverse depth.
• When training from those native depth maps without --with_depth_anything, add -o depth_ground_truth_is_inversed=False.

Example:

python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --init vggt -o depth_ground_truth_is_inversed=False
python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --init mapanything-external -o depth_ground_truth_is_inversed=False

VGG-T³ initialization

VGG-T³ is available through three initializer names:

• vggttt: directly initializes cameras, a point cloud, and native depth maps. Put images in <scene>/images.
• vggttt-colmap-sparse: runs VGG-T³, VGGSfM tracking, and COLMAP bundle adjustment. Put images in <scene>/input.
• vggttt-colmap-dense: additionally runs COLMAP dense reconstruction. Put images in <scene>/input.

The model weights are downloaded automatically from nvidia/vgg-ttt on first use. For a source checkout, clone with --recursive as shown in Development Install so that the submodules/vgg-ttt submodule is available.

Direct initialization:

python -m instantsplat.initialize -d data/my_scene -i vggttt

The VGG-T³ inference options can be passed with -o:

python -m instantsplat.initialize -d data/my_scene -i vggttt \
  -o num_ttt_steps=2 \
  -o memory_efficient_inference=True \
  -o use_global_pred=True

COLMAP bundle adjustment and dense reconstruction:

python -m instantsplat.initialize -d data/my_scene -i vggttt-colmap-sparse \
  -o colmap_executable="'colmap'"

python -m instantsplat.initialize -d data/my_scene -i vggttt-colmap-dense \
  -o colmap_executable="'colmap'"

VGG-T³ saves regular depth rather than inverse depth. When training without --with_depth_anything, use:

python -m instantsplat.train -s data/my_scene -d output/my_scene -i 1000 \
  --init vggttt -o depth_ground_truth_is_inversed=False

VGG-T³ code and weights are subject to the NVIDIA OneWay Noncommercial License.

2. Render it

python -m gaussian_splatting.render -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --load_camera output/sora/santorini/3_views/cameras.json

See .vscode\launch.json for more command examples.

Usage

See instantsplat.initialize, instantsplat.train and gaussian_splatting.render for full example.

Also check yindaheng98/gaussian-splatting for more detail of training process.

Gaussian models

Use CameraTrainableGaussianModel in yindaheng98/gaussian-splatting

Dataset

Use TrainableCameraDataset in yindaheng98/gaussian-splatting

Initialize coarse point cloud and cameras

from instantsplat.initializer import Dust3rInitializer
image_path_list = sorted(
    os.path.join(dirpath, filename)
    for dirpath, _, filenames in os.walk(image_folder)
    for filename in filenames
)
initializer = Dust3rInitializer(...).to(args.device) # see instantsplat/initializer/dust3r/dust3r.py for full options
initialized_point_cloud, initialized_cameras = initializer(image_path_list, destination)

Create camera dataset from initialized cameras:

from instantsplat.initializer import TrainableInitializedCameraDataset
dataset = TrainableInitializedCameraDataset(initialized_cameras).to(device)

Initialize 3DGS from initialized coarse point cloud:

gaussians.create_from_pcd(initialized_point_cloud.points, initialized_point_cloud.colors)

Training

Trainer jointly optimize the 3DGS parameters and cameras, without densification

from instantsplat.trainer import Trainer
trainer = Trainer(
    gaussians,
    dataset=dataset,
    ... # see instantsplat/trainer/trainer.py for full options
)

InstantSplat: Sparse-view SfM-free Gaussian Splatting in Seconds

Free-view Rendering

https://github.com/zhiwenfan/zhiwenfan.github.io/assets/34684115/748ae0de-8186-477a-bab3-3bed80362ad7

TODO List

• Confidence-aware Point Cloud Downsampling
• Support 2D-GS
• Support Mip-Splatting

Acknowledgement

This work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!

• Gaussian-Splatting and diff-gaussian-rasterization
• DUSt3R

Citation

If you find our work useful in your research, please consider giving a star ⭐ and citing the following paper 📝.

@misc{fan2024instantsplat,
        title={InstantSplat: Unbounded Sparse-view Pose-free Gaussian Splatting in 40 Seconds},
        author={Zhiwen Fan and Wenyan Cong and Kairun Wen and Kevin Wang and Jian Zhang and Xinghao Ding and Danfei Xu and Boris Ivanovic and Marco Pavone and Georgios Pavlakos and Zhangyang Wang and Yue Wang},
        year={2024},
        eprint={2403.20309},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
      }