instantsplat 1.15.3

Refactored python initialization and training code for InstantSplat

packaged InstantSplat

This repo is the refactored python training and inference code for InstantSplat. Forked from commit 2c5006d41894d06464da53d5495300860f432872. We refactored the original code following the standard Python package structure, while keeping the algorithms used in the code identical to the original version.

Initialization methods:

• DUST3R (same method used in InstantSplat)
• MAST3R (same method used in Splatt3R)
• COLMAP Sparse reconstruct (same method used in gaussian-splatting)
• COLMAP Dense reconstruct (use patch_match_stereo, stereo_fusion, poisson_mesher and delaunay_mesher in COLMAP to reconstruct dense point cloud for initialization)
• Masking of keypoints during COLMAP feature extraction (just put your mask into mask folder, e.g. for an image data/xxx/input/012.jpg, the mask would be data/xxx/input_mask/012.jpg.png)
• VGGT and VGGT + Colmap Bundle Adjustment according to facebookresearch/vggt/demo_colmap.py
• Map-Anything and Map-Anything with external pose/depth priors

Prerequisites

• Pytorch (v2.4 or higher recommended)
• CUDA Toolkit (12.4 recommended, should match with PyTorch version)

Install a colmap executable, e.g. using conda:

conda install conda-forge::colmap

(Optional) Install xformers for faster Depth-Anything V2 inference:

pip install xformers

(Optional) If you have trouble with gaussian-splatting, try to install it from source:

pip install wheel setuptools
pip install --upgrade git+https://github.com/yindaheng98/gaussian-splatting.git@master --no-build-isolation

PyPI Install

pip install --upgrade "mapanything[all] @ git+https://github.com/facebookresearch/map-anything.git@main"
pip install --upgrade git+https://github.com/facebookresearch/vggt.git@main
pip install --upgrade Pillow hydra-core  omegaconf # deps for vggt
pip install --upgrade git+https://github.com/jytime/LightGlue.git#egg=lightglue # deps for vggt
pip install --upgrade instantsplat

or build latest from source:

pip install wheel setuptools
pip install --upgrade git+https://github.com/yindaheng98/InstantSplat.git@main --no-build-isolation

Development Install

git clone --recursive https://github.com/yindaheng98/InstantSplat
cd InstantSplat
pip install scipy huggingface_hub einops roma scikit-learn
pip install --upgrade git+https://github.com/facebookresearch/vggt.git@main
pip install git+https://github.com/jytime/LightGlue.git#egg=lightglue # deps for vggt
pip install --target . --upgrade --no-deps .

Download model

wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_224_linear.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_512_linear.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_512_dpt.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/MASt3R/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth
wget -P checkpoints/ https://huggingface.co/depth-anything/Depth-Anything-V2-Small/resolve/main/depth_anything_v2_vits.pth
wget -P checkpoints/ https://huggingface.co/depth-anything/Depth-Anything-V2-Base/resolve/main/depth_anything_v2_vitb.pth
wget -P checkpoints/ https://huggingface.co/depth-anything/Depth-Anything-V2-Large/resolve/main/depth_anything_v2_vitl.pth
wget -P checkpoints/ https://huggingface.co/facebook/VGGT-1B-Commercial/resolve/main/vggt_1B_commercial.pt --header="Authorization: Bearer $HF_TOKEN"
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/MUSt3R/MUSt3R_512.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/Pow3R/Pow3R_ViTLarge_BaseDecoder_512_linear.pth

Configs for map-anything:

git clone --depth 1 --filter=blob:none --sparse https://github.com/facebookresearch/map-anything.git /tmp/map-anything-configs
git -C /tmp/map-anything-configs sparse-checkout set configs
cp -r /tmp/map-anything-configs/configs ./
rm -rf /tmp/map-anything-configs

Running

1. Initialize coarse point cloud and jointly train 3DGS & cameras

# Option 1: init and train in one command
python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --init dust3r
# Option 2: init and train in two separate commands
python -m instantsplat.initialize -d data/sora/santorini/3_views -i dust3r # init coarse point and save as a Colmap workspace at data/sora/santorini/3_views
python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 # train

To enable the optional auto-scaled Depth-Anything V2 wrapper for any initializer, add --with_depth_anything:

python -m instantsplat.initialize -d data/sora/santorini/3_views -i vggt --with_depth_anything
python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --init mapanything --with_depth_anything

Depth format note:

• Depth-Anything V2 saves inverse depth (1 / depth), which matches the default depth supervision used by 3DGS.
• The native depth saved by mapanything, mapanything-external, and vggt is regular depth, not inverse depth.
• When training from those native depth maps without --with_depth_anything, add -o depth_ground_truth_is_inversed=False.

Example:

python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --init vggt -o depth_ground_truth_is_inversed=False
python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --init mapanything-external -o depth_ground_truth_is_inversed=False

2. Render it

python -m gaussian_splatting.render -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --load_camera output/sora/santorini/3_views/cameras.json

See .vscode\launch.json for more command examples.

Usage

See instantsplat.initialize, instantsplat.train and gaussian_splatting.render for full example.

Also check yindaheng98/gaussian-splatting for more detail of training process.

Gaussian models

Use CameraTrainableGaussianModel in yindaheng98/gaussian-splatting

Dataset

Use TrainableCameraDataset in yindaheng98/gaussian-splatting

Initialize coarse point cloud and cameras

from instantsplat.initializer import Dust3rInitializer
image_path_list = [os.path.join(image_folder, file) for file in sorted(os.listdir(image_folder))]
initializer = Dust3rInitializer(...).to(args.device) # see instantsplat/initializer/dust3r/dust3r.py for full options
initialized_point_cloud, initialized_cameras = initializer(image_path_list=image_path_list)

Create camera dataset from initialized cameras:

from instantsplat.initializer import TrainableInitializedCameraDataset
dataset = TrainableInitializedCameraDataset(initialized_cameras).to(device)

Initialize 3DGS from initialized coarse point cloud:

gaussians.create_from_pcd(initialized_point_cloud.points, initialized_point_cloud.colors)

Training

Trainer jointly optimize the 3DGS parameters and cameras, without densification

from instantsplat.trainer import Trainer
trainer = Trainer(
    gaussians,
    dataset=dataset,
    ... # see instantsplat/trainer/trainer.py for full options
)

InstantSplat: Sparse-view SfM-free Gaussian Splatting in Seconds

This repository is the official implementation of InstantSplat, an sparse-view, SfM-free framework for large-scale scene reconstruction method using Gaussian Splatting. InstantSplat supports 3D-GS, 2D-GS, and Mip-Splatting.

Free-view Rendering

https://github.com/zhiwenfan/zhiwenfan.github.io/assets/34684115/748ae0de-8186-477a-bab3-3bed80362ad7

TODO List

• Confidence-aware Point Cloud Downsampling
• Support 2D-GS
• Support Mip-Splatting

Acknowledgement

This work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!

• Gaussian-Splatting and diff-gaussian-rasterization
• DUSt3R

Citation

If you find our work useful in your research, please consider giving a star :star: and citing the following paper :pencil:.

@misc{fan2024instantsplat,
        title={InstantSplat: Unbounded Sparse-view Pose-free Gaussian Splatting in 40 Seconds},
        author={Zhiwen Fan and Wenyan Cong and Kairun Wen and Kevin Wang and Jian Zhang and Xinghao Ding and Danfei Xu and Boris Ivanovic and Marco Pavone and Georgios Pavlakos and Zhangyang Wang and Yue Wang},
        year={2024},
        eprint={2403.20309},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
      }