gaussian-inspector 0.1.0

Method agnostic Gaussian Splatting viewer and visual debugger.

Introduction

GaussianInspector is a Python package that sets up a simple but flexible OpenGL viewer and visual inspector for any 3DGS-based project (i.e. 4DGaussian, 4DGS, E-D3DGS, Ex4DGS, etc.). To make it project-agnostic, GaussianInspector provides a few stub files for callbakcs which you need to implement (by copy-pasting from the project). This typically amounts to a few lines of Python code that are specific to your 3DGS-based method, as shown in the example usage below.

Installation and requirements

This package needs to be installed with uv: uv pip install gaussian-inspector.

Additionally, you may want Docker installed with the NVIDIA Container Toolkit setup. This greatly facilitates the compilation of custom CUDA rasterizers and the CUDA devkit without messing with your system. However, if you already have a working virtual environment for the project you want to view, GaussianInspector can just be installed inside it.

Usage

In your existing environment

1. Run gaussian-inspector init in the project root to initialize the stubs.
2. Implement all methods in the stubs created in <project-root>/gaussian_inspector_stubs/.
3. Run the viewer: gaussian-inspector launch.

In a custom-built Docker image

1. Run gaussian-inspector init in the project root to initialize the stubs and the Docker image template.
2. Build the custom Docker image: gaussian-inspector build.
3. Implement all methods in the stubs created in <project-root>/gaussian_inspector_stubs/.
4. Run the viewer: gaussian-inspector launch-with-docker. Optionally use --mount src:dst to mount dataset directories into the Docker image, as these are usually loaded when initializing the Scene.

Example implementation

As an example, we'll implement the stubs for the 4DGS project. There are 3 stubs to implement: configuration.py, camera.py and rendering.py.

configuration.py

In this stub file, we implement the configure() method that loads the model, scene configuration, pipeline configuration, etc. This is typically different for each project, and one may refer to the render.py or train.py typically found in the project implementation.

The configure() function must return a Dict[str, Any], and you are free to define this dict however you see fit. This will then be passed as a context variable to the rendering and camera stubs. Note that you may use package imports as in the rest of the project, since GaussianInspector adds the project in sys.path.

from argparse import ArgumentParser
from typing import Any, Dict

import torch
from omegaconf import DictConfig, OmegaConf

from arguments import ModelParams, OptimizationParams, PipelineParams
from scene import Scene
from scene.gaussian_model import GaussianModel
from utils.general_utils import safe_state


def configure(sys_argv) -> Dict[str, Any]:
    """
    Configure the project from sys.argv and return the context of the project, which may
    be used for rendering and other purposes.
    """
    context = {}
    parser = ArgumentParser(description="Testing script parameters")
    model = ModelParams(parser, sentinel=True)
    op = OptimizationParams(parser)
    pipeline = PipelineParams(parser)
    parser.add_argument("--config", type=str)
    parser.add_argument("--iteration", default=-1, type=int)
    parser.add_argument("--skip_train", action="store_true")
    parser.add_argument("--skip_test", action="store_true")
    parser.add_argument("--quiet", action="store_true")
    parser.add_argument("--3DGS", dest="use_3dgs", action="store_true")
    parser.add_argument("--time_duration", nargs=2, type=float, default=[-0.5, 0.5])
    parser.add_argument("--num_pts", type=int, default=100_000)
    parser.add_argument("--num_pts_ratio", type=float, default=1.0)
    parser.add_argument("--force_sh_3d", action="store_true")
    parser.add_argument("--batch_size", type=int, default=1)
    parser.add_argument("--seed", type=int, default=6666)
    parser.add_argument("--exhaust_test", action="store_true")
    parser.add_argument("--spherical_coords", action="store_true")
    parser.add_argument("--max-frames", type=int, required=False)
    parser.add_argument("--per-dof", action="store_true")
    args = parser.parse_args(sys_argv[1:])
    # args.save_iterations.append(args.iterations)
    args.config = "configs/dynerf/cook_spinach.yaml"

    cfg = OmegaConf.load(args.config)

    def recursive_merge(key, host):
        if isinstance(host[key], DictConfig):
            for key1 in host[key].keys():
                recursive_merge(key1, host[key])
        else:
            assert hasattr(args, key), key
            setattr(args, key, host[key])

    for k in cfg.keys():
        recursive_merge(k, cfg)

    # Initialize system state (RNG)
    safe_state(args.quiet)

    model_args = model.extract(args)
    pipeline_args = pipeline.extract(args)
    global_args = args

    with torch.no_grad():
        gaussians = GaussianModel(
            model_args.sh_degree,
            model_args.max_dof,
            model_args.enable_rot_delta,
            model_args.enable_scale_delta,
            model_args.enable_sh_delta,
            force_sh_3d=model_args.force_sh_3d,
            time_duration=global_args.time_duration,
        )
        scene = Scene(
            model_args,
            gaussians,
            time_duration=global_args.time_duration,
            shuffle=False,
        )
        bg_color = [1, 1, 1] if model_args.white_background else [0, 0, 0]
        background = torch.tensor(bg_color, dtype=torch.float32, device="cuda")
    context = {
        "gaussians": gaussians,
        "background": background,
        "scene": scene,
        "pipeline_args": pipeline_args,
        "resolution": (1352, 1014),
    }
    return context

camera.py

Here, we need to return a camera object that is expected by the renderer. Typically, we can use the first test camera:

def get_viewpoint_camera(user_context: Dict[str, Any]) -> Any:
    """
    Return the default viewpoint camera for the scene. This camera will be used as
    initialization for camera controls. It must be a valid camera object compatible with
    the rasterizer of your project.

    Args:
    user_context (Dict[str, Any]): The user context returned by the configure()
        function in configuration.py. This user context may store the Scene object or
        whatever you  need to fetch or create the initial camera.
    """

    return user_context["scene"].getTestCameras()[0].cuda()

rendering.py

For rendering, we may copy and paste the rendering code of the original project (e.g. from render.py or train.py). Then, we must convert the rendered image to a NumPy array before returning it:

def render_frame(context: Dict[str, Any], timestamp: float, camera_view: Any) -> np.ndarray:
    """
    Render a frame at time 'timestamp' with camera view 'camera_view'. The returned
    image must be a numpy array of shape (H, W, 3) with dtype uint8.
    """
    render_dict = render(
        camera_view,
        context["gaussians"],
        context["pipeline_args"],
        context["background"],
    )
    if render_dict is None:
        w, h = context["resolution"]
        image = torch.zeros((h, w, 3), dtype=torch.uint8).numpy()
    else:
        image = (
            (render_dict["render"] * 255).clamp(0, 255).cpu().numpy().astype(np.uint8)
        ).transpose(1, 2, 0)  # HWC
    return image

Roadmap

• v0.1.0: basic method-agnostic viewer with callbacks to user code (tested on 4DGS. and 3DGS).
• v0.2.0: method-agnostic visual inspector with callbacks for data collection.
• v0.3.0: community library of implemented stubs for known projects.
• ...