Skip to main content

A Taichi Gaussian Splatting library

Project description

Taichi Splatting

Rasterizer for Guassian Splatting using Taichi and PyTorch - embedded in python library. Currently very usable but in active development, so likely will break with new versions!

This work is originally derived off Taichi 3D Gaussian Splatting, with significant re-organisation and changes.

Key differences are the rendering algorithm is decomposed into separate operations (projection, shading functions, tile mapping and rasterization) which can be combined in different ways in order to facilitate a more flexible use, and gradients can be enabled on "all the things" as required for the application (and not when disabled, to save performance).

Using the Taichi autodiff for a simpler implementation where possible (e.g. for projection, but not for the rasterization).

Examples:

  • Projecting features for lifting 2D to 3D
  • Colours via. spherical harmonics
  • Depth covariance without needing to build it into the renderer and remaining differentiable.
  • Fully differentiable camera parameters (and ability to swap in new camera models)

Performance

A document describing some performance benchmarks of taichi-splatting here. Through various optimizations, in particular optimizing the summation of gradients in the backward gradient kernel. Taichi-splatting achieves a very large speedup (often an order of magnitude) over the original taichi_3d_gaussian_splatting, and is faster than the reference diff_guassian_rasterization for a complete optimization pass (forward+backward), in particular much faster at higher resolutions.

Installing

External dependencies

Create an environment (for example conda with mambaforge) with the following dependencies:

  • python >= 3.10
  • pytorch - from either conda Follow instructions https://pytorch.org/.
  • taichi-nightly pip install --upgrade -i https://pypi.taichi.graphics/simple/ taichi-nightly

Install

One of:

  • pip install taichi-splatting
  • Clone down with git clone and install with pip install ./taichi-splatting

Executables

fit_image_gaussians

There exists a toy optimizer for fitting a set of randomly initialized gaussians to some 2D images fit_image_gaussians - useful for testing rasterization without the rest of the dependencies.

Fitting an image (fixed points):
fit_image_gaussians <image file> --show --n 20000

Fitting an image (split and prune to target):
fit_image_gaussians <image file> --show --n 1000 --target 20000

See --help for other options.

benchmarks

There exist benchmarks to evaluate performance on individual components in isolation under taichi_splatting/benchmarks/

tests

Tests (gradient tests and tests comparing to torch-based reference implementations) can be run with pytest, or individually under taichi_splatting/tests/

splat-viewer

A viewer for reconstructions created with the original gaussian-splatting repository can be found here or installed with pip. Has dependencies on open3d and Qt.

splat-benchmark

A benchmark for a full rendererer (in the same repository as above) with real reconstructions (rendering the original camera viewpoints). Options exist for tweaking all the renderer parameters, benchmarking backward pass etc.

Progress

Done

  • Benchmarks with original + taichi_3dgs rasterizer

  • Simple view culling

  • Projection with autograd

  • Tile mapping (optimized and improved culling)

  • Rasterizer forward pass and optimized backward pass

  • Spherical harmonics with autograd

  • Gradient tests for most parts (float64) - including rasterizer!

  • Fit to image training example/test

  • Depth and depth-covariance rendering

  • Compute point visibility in backward pass (useful for model pruning)

  • Example training on images with split/prune operations

  • Novel heuristics for split and prune operations computed optionally in backward pass

Todo

  • Depth covariance example

  • 3D training code (likely different repository)

  • Backward projection autograd takes a while to compile and is not cached properly

Improvements

  • Exposed all internal constants as parameters
  • Switched to matrices as inputs instead of quaternions
  • Tile mapping tighter culling for tile overlaps (~30% less rendered splats!)
  • All configuration parameters exposed (e.g. tile_size, saturation threshold etc.)
  • Warp reduction based backward pass for rasterizer, a decent boost in performance

Conventions

Transformation matrices

Transformations are notated T_x_y, for example T_camera_world can be used to transform points in the world to points in the local camera by points_camera = T_camera_world @ points_world

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

taichi_splatting-0.13.0.tar.gz (4.1 MB view details)

Uploaded Source

Built Distribution

taichi_splatting-0.13.0-py3-none-any.whl (82.2 kB view details)

Uploaded Python 3

File details

Details for the file taichi_splatting-0.13.0.tar.gz.

File metadata

  • Download URL: taichi_splatting-0.13.0.tar.gz
  • Upload date:
  • Size: 4.1 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.10.14

File hashes

Hashes for taichi_splatting-0.13.0.tar.gz
Algorithm Hash digest
SHA256 d50157a7f3916cb1246eb62f484594a5a9bca0b1e8f65ef86d2136e459237f48
MD5 5f09faf6a7df93817e16f746944c23bf
BLAKE2b-256 8058ef3c1ce104f2b95d414b1c5114d37249287382540f7094d3eb479f1a8200

See more details on using hashes here.

File details

Details for the file taichi_splatting-0.13.0-py3-none-any.whl.

File metadata

File hashes

Hashes for taichi_splatting-0.13.0-py3-none-any.whl
Algorithm Hash digest
SHA256 adbcc70605c9622859d5f6558a11f6478f176274abc362947209f01676c42789
MD5 1684f339f6781d159dd62107a6773044
BLAKE2b-256 4abdf5417f453dec79ee986d31f2a1fbdccc2dd4cd4d575c22d92387b61e09f0

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page