deepgp-torch

A GPyTorch library for doubly-stochastic deep Gaussian processes (Salimbeni & Deisenroth, 2017). It provides modular SVGP layers, a deep-GP container, variational (ELBO) training, and calibrated prediction.

The importable package is deepgp. Its API and engineering are modelled on GPflux but built on PyTorch + GPyTorch + linear_operator.

Status: early development. SVGP, DeepGP and build_deep_gp train and predict on snelson1d, with one output or several. Minibatch and natural-gradient training and the benchmark harness are not implemented yet (see Roadmap).

Install

pip install deepgp-torch

Or, from a checkout, for development:

pip install -e ".[dev]"     # editable install with dev tooling

Requires Python ≥ 3.10, torch>=2.0, gpytorch>=1.15, linear_operator>=0.6.1, numpy>=1.26.

Quickstart

Importing deepgp sets the torch default dtype to float64 (Cholesky stability), so every GP module is built in double precision.

import deepgp
from deepgp import DeepGP, fit, predict
from deepgp.data import load_snelson1d

data = load_snelson1d(test_fraction=0.2, seed=0)      # standardised split

model = DeepGP([1, 1], num_inducing=32)               # 1 hidden + 1 output layer
losses = fit(model, data.X_train, data.Y_train,       # maximise the ELBO
             epochs=300, lr=0.01, num_samples=10)

mean, var = predict(model, data.X_test, k=32)         # total-variance mixture
print("first training loss:", losses[0], "-> last:", losses[-1])

A single-layer SVGP uses the same fit / predict helpers:

from deepgp import SVGP

svgp = SVGP(inducing_points=data.X_train[:32].clone(), input_dims=1)
fit(svgp, data.X_train, data.Y_train, epochs=300, lr=0.01)
mean, var = predict(svgp, data.X_test)

Or build a constant-input-dimension deep GP with stable-start initialisation in one call (KMeans inducing init, dim-preserving identity/PCA hidden means, shrunk hidden q_sqrt, Gaussian noise init):

from deepgp import build_deep_gp, DeepGPConfig

cfg = DeepGPConfig(num_inducing=20, inner_layer_qsqrt_factor=1e-5, likelihood_noise=1e-2)
model = build_deep_gp(data.X_train, num_layers=2, config=cfg)
fit(model, data.X_train, data.Y_train, epochs=300, lr=0.01)
mean, var = predict(model, data.X_test)

Multiple outputs

DeepGP, SVGP and build_deep_gp take num_outputs=T and train on targets of shape (N, T):

import torch

X = torch.linspace(-2, 2, 200).unsqueeze(-1)                 # (200, 1)
F = torch.cat([torch.sin(2 * X), torch.cos(2 * X)], dim=-1)  # (200, 2)
Y = F + torch.tensor([0.05, 0.3]) * torch.randn_like(F)      # noise std 0.05 and 0.3

model = DeepGP([1, 2], num_inducing=32, num_outputs=2)       # hidden width 2, 2 outputs
fit(model, X, Y, epochs=300, lr=0.01)
mean, var = predict(model, X)                                # each (200, 2)
print(model.likelihood.task_noises)                          # one noise variance per output

The output layer is T independent GPs, each with its own inducing points, q(u), mean and kernel hyper-parameters, and the likelihood has one noise variance per output. In a DeepGP all outputs share the hidden layers and are functions of the last hidden layer's output, so a width-1 hidden layer sends every output through a single scalar. An SVGP shares nothing: its ELBO is the sum of the T single-output ELBOs.

predict returns each output's marginal mean and variance. Under a deep GP's predictive mixture the outputs are in general dependent, because within each mixture component all T means are computed from the same hidden-layer sample; that cross-output covariance is not returned.

Why a deep-GP-specific library over raw GPyTorch?

It encodes the non-obvious correctness details that raw GPyTorch leaves to you:

• Layers own their variational structure — DeepGPHiddenLayer bundles inducing points, q(u), mean and kernel; the deep-GP variational strategy discovers layers via model.modules() and sums their KL terms automatically.
• Linear hidden means — LinearMean hidden layers prevent doubly-stochastic hidden-field collapse (mean_type='zero' is available but discouraged).
• Law-of-total-variance prediction — a deep GP's predictive distribution is an S-component Gaussian mixture; predict reduces it as mean = out.mean.mean(0) and var = out.variance.mean(0) + out.mean.var(0, unbiased=False). A single component's .variance under-reports uncertainty.
• First-class KL tempering — the ELBO KL weight is exactly beta / num_data.
• One noise variance per output — by default GPyTorch's MultitaskGaussianLikelihood adds a shared noise term to each task's own, so output t has variance σ_t² + σ²: T + 1 parameters, of which the data identify only the T sums. Multi-output models here drop the shared term, so likelihood.task_noises[t] is the noise variance of output t.
• Targets must match the output shape — fit raises if Y.shape differs from the model's output event shape. Otherwise a (N,) target broadcasts against (N, T) outputs whenever N == T, and training runs on the wrong objective without an error.

Development

make install      # pip install -e ".[dev]"
make format       # black + isort
make check        # black --check, isort --check-only, flake8, mypy src
make test         # pytest

CI (.github/workflows/ci.yml) runs lint + type-check + tests on Python 3.10, 3.11 and 3.12, and builds the Docker image.

Docker

A CPU-only image for a reproducible environment:

docker build -t deepgp-torch .
docker run --rm deepgp-torch                # prints the installed version
docker run --rm deepgp-torch pytest -q      # runs the test suite
docker run --rm -it deepgp-torch python     # interactive REPL

Roadmap

Implemented:

• Core stack: layers/gp_layer, models/svgp, models/deep_gp, training/elbo + trainer, inference/predict, data/datasets (snelson1d), eval/metrics, utils/seed + dtype.
• Architecture builder: DeepGPConfig and build_deep_gp with stable-start init, kernels/factory + means/factory (identity/PCA LinearMean), and data/inducing (KMeans / subset / explicit z_init).
• Multi-output regression: num_outputs on SVGP, DeepGP and build_deep_gp (independent output GPs, shared hidden layers) and likelihoods/factory (one noise variance per output).

Not yet implemented — the modules exist as placeholders:

• variational convenience helpers.
• Minibatch/DataLoader training with LR schedulers, checkpointing and logging, and natural-gradient training.
• Golden equivalence test against GPflux, a UCI benchmark harness, and Sphinx docs.

Contributing

Contributions are welcome! See CONTRIBUTING.md for the dev setup and workflow, and please follow the Code of Conduct. To report a security issue, see SECURITY.md.

Citing

If you use deepgp-torch in your research, please cite it — see CITATION.cff (GitHub renders a "Cite this repository" button).

References

• Salimbeni & Deisenroth, Doubly Stochastic Variational Inference for Deep Gaussian Processes, NeurIPS 2017.
• Snelson & Ghahramani, Sparse Gaussian Processes using Pseudo-inputs, 2006 (the snelson1d dataset).

License

Apache-2.0 — see LICENSE.