A Taichi Gaussian Splatting library
Project description
Taichi Splatting
Rasterizer for Guassian Splatting using Taichi and PyTorch - embedded in python library. Currently very usable but in active development, so likely will break with new versions!
This work is originally derived off Taichi 3D Gaussian Splatting, with significant re-organisation and changes.
Key differences are the rendering algorithm is decomposed into separate operations (projection, shading functions, tile mapping and rasterization) which can be combined in different ways in order to facilitate a more flexible use, and gradients can be enabled on "all the things" as required for the application (and not when disabled, to save performance).
Using the Taichi autodiff for a simpler implementation where possible (e.g. for projection, but not for the rasterization).
Examples:
- Projecting features for lifting 2D to 3D
- Colours via. spherical harmonics
- Depth covariance without needing to build it into the renderer and remaining differentiable.
- Fully differentiable camera parameters (and ability to swap in new camera models)
Performance
A document describing some performance benchmarks of taichi-splatting here. Through various optimizations, in particular optimizing the summation of gradients in the backward gradient kernel. Taichi-splatting achieves a very large speedup (often an order of magnitude) over the original taichi_3d_gaussian_splatting, and is faster than the reference diff_guassian_rasterization for a complete optimization pass (forward+backward), in particular much faster at higher resolutions.
Installing
External dependencies
Create an environment (for example conda with mambaforge) with the following dependencies:
- python >= 3.10
- pytorch - from either conda Follow instructions https://pytorch.org/.
- taichi-nightly
pip install --upgrade -i https://pypi.taichi.graphics/simple/ taichi-nightly
Install
One of:
pip install taichi-splatting
- Clone down with
git clone
and install withpip install ./taichi-splatting
Executables
fit_image_gaussians
There exists a toy optimizer for fitting a set of randomly initialized gaussians to some 2D images fit_image_gaussians
- useful for testing rasterization without the rest of the dependencies.
Fitting an image (fixed points):
fit_image_gaussians <image file> --show --n 20000
Fitting an image (split and prune to target):
fit_image_gaussians <image file> --show --n 1000 --target 20000
See --help
for other options.
benchmarks
There exist benchmarks to evaluate performance on individual components in isolation under taichi_splatting/benchmarks/
tests
Tests (gradient tests and tests comparing to torch-based reference implementations) can be run with pytest, or individually under
taichi_splatting/tests/
splat-viewer
A viewer for reconstructions created with the original gaussian-splatting repository can be found here or installed with pip. Has dependencies on open3d and Qt.
splat-benchmark
A benchmark for a full rendererer (in the same repository as above) with real reconstructions (rendering the original camera viewpoints). Options exist for tweaking all the renderer parameters, benchmarking backward pass etc.
Progress
Done
-
Benchmarks with original + taichi_3dgs rasterizer
-
Simple view culling
-
Projection with autograd
-
Tile mapping (optimized and improved culling)
-
Rasterizer forward pass and optimized backward pass
-
Spherical harmonics with autograd
-
Gradient tests for most parts (float64) - including rasterizer!
-
Fit to image training example/test
-
Depth and depth-covariance rendering
-
Compute point visibility in backward pass (useful for model pruning)
-
Example training on images with split/prune operations
-
Novel heuristics for split and prune operations computed optionally in backward pass
Todo
-
Depth covariance example
-
3D training code (likely different repository)
-
Backward projection autograd takes a while to compile and is not cached properly
Improvements
- Exposed all internal constants as parameters
- Switched to matrices as inputs instead of quaternions
- Tile mapping tighter culling for tile overlaps (~30% less rendered splats!)
- All configuration parameters exposed (e.g. tile_size, saturation threshold etc.)
- Warp reduction based backward pass for rasterizer, a decent boost in performance
Conventions
Transformation matrices
Transformations are notated T_x_y
, for example T_camera_world
can be used to transform points in the world to points in the local camera by points_camera = T_camera_world @ points_world
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file taichi_splatting-0.13.0.tar.gz
.
File metadata
- Download URL: taichi_splatting-0.13.0.tar.gz
- Upload date:
- Size: 4.1 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.10.14
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | d50157a7f3916cb1246eb62f484594a5a9bca0b1e8f65ef86d2136e459237f48 |
|
MD5 | 5f09faf6a7df93817e16f746944c23bf |
|
BLAKE2b-256 | 8058ef3c1ce104f2b95d414b1c5114d37249287382540f7094d3eb479f1a8200 |
File details
Details for the file taichi_splatting-0.13.0-py3-none-any.whl
.
File metadata
- Download URL: taichi_splatting-0.13.0-py3-none-any.whl
- Upload date:
- Size: 82.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.10.14
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | adbcc70605c9622859d5f6558a11f6478f176274abc362947209f01676c42789 |
|
MD5 | 1684f339f6781d159dd62107a6773044 |
|
BLAKE2b-256 | 4abdf5417f453dec79ee986d31f2a1fbdccc2dd4cd4d575c22d92387b61e09f0 |