ffgs — standard inference for feed-forward 3D Gaussian Splatting

One way to load and run feed-forward 3D Gaussian Splatting models: posed images in, Gaussians out — in the model's own frame, or in your world frame with one call — whatever model is behind it.

Early development: the core interface, third-party loading and the model zoo are in place; see the models for those shipped so far.

from ffgs import Cameras, GSPipeline, Views

pipe = GSPipeline.from_pretrained("<family>/<variant>", device="cuda")  # or a path / Hub repo
out = pipe(Views(images, intrinsics, c2w))          # -> GSOutput, in the model's frame
images = pipe.render(out, Cameras(c2w_new, k_new, (h, w)))["images"]  # cameras in your frame
out.to_world().gaussians.save_ply("scene.ply")      # standard 3DGS ply, in the frame of c2w

Install

pip install ffgs            # inference
pip install "ffgs[render]"  # + gsplat rasterisation (CUDA)

Dependencies: torch, torchvision, numpy, pillow, huggingface_hub, safetensors; gsplat (the render extra) is imported only when rendering.

Example

examples/infer_nerfstudio_scene.py reconstructs a scene in the nerfstudio / DL3DV layout (transforms.json, OpenGL camera poses) from evenly spaced frames, writes a 3DGS ply in the scene's frame and renders the frames between the inputs:

python examples/infer_nerfstudio_scene.py <scene> --model <family>/<variant> \
    --images-dir <scene>/images_4 --out out

Models that predict their own cameras also run on images alone. There is no world frame then: the Gaussians and the predicted input cameras are both in the model's frame, and renders take cameras in that frame (see Conventions):

pipe = GSPipeline.from_pretrained("<family>/<variant>", device="cuda")  # a pose-free model
out = pipe(Views(images))                    # no intrinsics, no c2w: out.frame is None
out.gaussians.save_ply("scene.ply")          # in the model's frame
renders = pipe.render(out, out.cameras)["images"]  # the inputs, at the predicted cameras

examples/infer_pose_free.py does this for a folder of images: it writes the ply, the predicted cameras (cameras.json: OpenCV c2w and normalised intrinsics per input image) and the inputs re-rendered at them:

python examples/infer_pose_free.py <image dir> --model <family>/<variant> \
    --num-views 8 --out out

Models

The shipped models, their zoo names, licenses, usage notes and results against their official code are in docs/models, one page per model. Check the license of a model before use: some are for non-commercial use only.

Conventions

A processor reproduces its model's training preprocessing: rescale to cover the model's input size, centre crop (normalised principal point kept, as in training), cameras relative to the first input view, translations times a model-specific scene_scale. The model predicts in that normalised frame, and that is where pipe(views) leaves the Gaussians (out.space == "model"), exactly as the model's own code produces them.

to_world() is one similarity transform for every model — rotation, uniform scale and translation — applied per attribute: means fully, scales by the scale, quats by the rotation, view-dependent SH colours of any degree by an exact rotation of their coefficients (3DGS / gsplat basis); opacities and RGB / degree-0 colours are unchanged. Input c2w whose rotation is not orthonormal or is a reflection (det < 0) are rejected.

Models that need poses raise a ValueError naming what is missing. Models that predict their own cameras accept Views(images) alone, and out.cameras holds their prediction for the input views, in the same frame as out.gaussians (to_world() moves both):

With c2w, the frame is one closed-form fit over all input views of the predicted cameras onto the given ones: the rotation is the mean of the per-view rotations (SVD, det +1), then scale and translation are the least-squares fit of the camera centres. The given c2w do not enter the model. out.frame.residuals reports how well they agree, per view: rotation error in degrees, centre error in world units, and centre error relative to the RMS spread of the given centres. Fewer than 2 views, coincident predicted centres or a fit with negative scale raise.

Resize + crop is the default, not a requirement. Processor options (in processor_config.json or processor_overrides=):

pipe = GSPipeline.from_pretrained(path, processor_overrides={"crop_mode": "pad"})

Layers

A saved model is a directory (or Hub repo) with config.json (model config + model_type), model.safetensors and processor_config.json; GSPipeline.save_pretrained writes all three.

Model zoo

The zoo names published checkpoints: one JSON file per variant in src/ffgs/zoo/<family>/<variant>.json, shipped with the package. The weights stay in their authors' Hub repos; an entry pins the file to a commit and a sha256.

from ffgs import GSPipeline, zoo

zoo.list_models()                        # ["<family>/<variant>", ...]
pipe = GSPipeline.from_pretrained("<family>/<variant>", device="cuda")

{
  "model_type": "my-model",
  "description": "My model trained on X, 2 input views",
  "weights": {"repo": "author/repo", "file": "ckpt/model.pt",
              "revision": "<full 40-hex commit>", "sha256": "<64-hex>"},
  "model": {"...": "model constructor arguments"},
  "processor": {"...": "processor_config.json"},
  "source": {"url": "https://github.com/author/code", "paper": "https://..."},
  "license": "license of the weights"
}

Loading an entry downloads the file at the pinned commit, fails if its sha256 differs, converts the upstream checkpoint with the model's ModelSpec.convert_state_dict (e.g. stripping a key prefix) and loads it strictly: every key must match. .pt checkpoints are unpickled with weights_only=True. save_pretrained then writes an ordinary ffgs directory.

from_pretrained(name) tries, in order: an existing local directory, a zoo entry, a Hub repo id. A zoo family name is reserved — "<family>/<anything>" never goes to the Hub, and an unknown variant is an error listing the known ones. (To load a Hub repo whose owner shares a family name, huggingface_hub.snapshot_download it and pass the directory.) A zoo entry pins its own revision, so revision= and subfolder= are rejected for it; cache_dir, token, force_download and local_files_only apply.

Checkpoints saved with save_pretrained can also be published as Hub repos, one per model family with variants in subfolders:

pipe = GSPipeline.from_pretrained("owner/family-repo", subfolder="variant",
                                  revision="<commit sha>")

Adding a model

A model is a model class with PyTorchModelHubMixin, a Processor subclass (preprocess, returning the model input and its ModelFrame, or None when the model predicts its own cameras; postprocess, to model-frame Gaussians; render_planes, near / far in world units, or in model units when the frame is None; for models that predict cameras, predicted_cameras, from the model output to model-frame Cameras, and output_frame, whose default fits the frame to the given c2w as above; optionally background and render_settings, the model's rasteriser settings such as radius_clip, rasterize_mode or clamp, which pipe.render uses unless the caller passes them) and a ModelSpec(model_type, model_cls, processor_cls). from_pretrained resolves the model_type in config.json in this order:

1. In your code — ffgs.register_model(spec) before loading (built-in models are registered the same way, on first use).

2. An installed package — expose the spec under the ffgs.models entry point group, named by its model_type; ffgs imports the package only when that model_type is loaded:

   # pyproject.toml of your package
   [project.entry-points."ffgs.models"]
   my-model = "my_package.ffgs_model:SPEC"   # a ModelSpec with model_type "my-model"
   

3. Code in the model repo — ship the .py files next to config.json and name the classes (relative imports between the files work):

   {"model_type": "my-model",
    "auto_map": {"model": "modeling_my.MyModel", "processor": "processing_my.MyProcessor"}}
   

   That code runs only with trust_remote_code=True (off by default); read it first and pin the revision:

   pipe = GSPipeline.from_pretrained("someone/my-model", trust_remote_code=True,
                                     revision="<commit sha>")
   

   save_pretrained on such a pipeline copies the code and keeps auto_map. To publish a model whose code lives in your own (e.g. training) codebase, keep the model and processor classes in one package that imports itself only relatively, and export it with the weights:

   GSPipeline(model, processor, model_type="my-model").save_pretrained(
       "export/my-model", include_code=True)   # copies the package, writes auto_map
   

   Imports of the rest of your codebase are refused at export time, so the directory loads with trust_remote_code=True where your codebase is not installed.

An installed model wins over repo code with the same model_type. Hub options (revision, subfolder, cache_dir, token, force_download, local_files_only) apply to every file from_pretrained downloads.

To add a zoo entry for a model, write its JSON (compute the sha256 of the pinned file), give the model's ModelSpec a convert_state_dict if the upstream keys differ from the model's, and run the tests: every shipped entry is checked to parse and build offline, and uv run pytest --network downloads each one and loads it strictly.

Document a new model in docs/models/<family>.md (source, licenses of code and weights, zoo entries, usage notes, results against the official code; the existing pages are the template) and add its row to docs/models/index.md.

Development

uv sync
uv run pytest            # everything; tests marked `gpu` skip without CUDA + gsplat
uv run pytest -m gpu     # only the GPU tests (needs CUDA and `--extra render`)
uv run pytest --network  # also the tests that download zoo weights from the Hub
uv run pre-commit install

uv sync installs torch from PyPI (CUDA build). CI has no GPU and installs CPU torch first (see .github/workflows/ci.yml). tests/fixtures/reference.safetensors holds frozen reference outputs of the default data path; change it only together with a deliberate change to that path.

License

The ffgs package as distributed is not entirely Apache-2.0: it includes code for non-commercial use only. Its license expression is Apache-2.0 AND MIT AND CC-BY-NC-4.0 AND CC-BY-NC-SA-4.0:

• the core library is Apache-2.0 (see LICENSE); src/ffgs/image.py is adapted from pixelSplat (MIT), whose license is included at the end of LICENSE;
• vendored model code under src/ffgs/models/<family>/ keeps its upstream license. src/ffgs/models/anysplat/ mixes MIT, Apache-2.0 and non-commercial (CC-BY-NC-4.0, CC-BY-NC-SA-4.0) files; its LICENSE lists the license per file;
• model weights are not part of the package and carry their own licenses.

The code and weight licenses of each model are listed in docs/models.