Simulation Based Calibration

A PyMC and Bambi implementation of the algorithms from:

Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, Andrew Gelman: “Validating Bayesian Inference Algorithms with Simulation-Based Calibration”, 2018; arXiv:1804.06788

Many thanks to the authors for providing open, reproducible code and implementations in rstan and PyStan (link).

Installation

May be pip installed from github:

pip install git+https://github.com/arviz-devs/simulation_based_calibration

Quickstart

Define a PyMC or Bambi model. For example, the centered eight schools model:

import numpy as np
import pymc as pm

data = np.array([28.0, 8.0, -3.0, 7.0, -1.0, 1.0, 18.0, 12.0])
sigma = np.array([15.0, 10.0, 16.0, 11.0, 9.0, 11.0, 10.0, 18.0])
                
with pm.Model() as centered_eight:
    obs = pm.MutableData("obs", data)
    mu = pm.Normal('mu', mu=0, sigma=5)
    tau = pm.HalfCauchy('tau', beta=5)
    theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8)
    y_obs = pm.Normal('y', mu=theta, sigma=sigma, observed=obs)

Pass the model to the SBC class, and run the simulations. This will take a while, as it is running the model many times.

sbc = SBC(centered_eight,
        num_simulations=100, # ideally this should be higher, like 1000
        sample_kwargs={'draws': 25, 'tune': 50})

sbc.run_simulations()

79%|███████▉  | 79/100 [05:36<01:29,  4.27s/it]

Plot the empirical CDF for the difference between prior and posterior. The lines should be close to uniform and within the oval envelope.
```
sbc.plot_results()
```

Simulation based calibration plots, ecdf

What is going on here?

The paper on the arXiv is very well written, and explains the algorithm quite well.

Morally, the example below is exactly what this library does, but it generalizes to more complicated models:

with pm.Model() as model:
    x = pm.Normal('x')
    pm.Normal('y', mu=x, observed=y)

Then what this library does is compute

with my_model():
    prior_samples = pm.sample_prior_predictive(num_trials)

simulations = {'x': []}
for idx in range(num_trials):
    y_tilde = prior_samples['y'][idx]
    x_tilde = prior_samples['x'][idx]
    with model(y=y_tilde):
        idata = pm.sample()
    simulations['x'].append((idata.posterior['x'] < x_tilde).sum())

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

simuk-0.1.0.tar.gz (6.8 kB view details)

Uploaded Feb 12, 2025 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

simuk-0.1.0-py3-none-any.whl (7.4 kB view details)

Uploaded Feb 12, 2025 Python 3

File details

Details for the file simuk-0.1.0.tar.gz.

File metadata

Download URL: simuk-0.1.0.tar.gz
Upload date: Feb 12, 2025
Size: 6.8 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.12.8

File hashes

Hashes for simuk-0.1.0.tar.gz
Algorithm	Hash digest
SHA256	`b49c5f80c27bcf8d1ef35cc31c5c83ff121f70f4d61cf0d2b8bc58396445391a`
MD5	`2c2c916c44b971fdec565a11feec2ce4`
BLAKE2b-256	`cd0dbaafdbdbd543c93c80a30cf7b47b0accce9f6cf7585bc08e13bd3bcb8c99`

See more details on using hashes here.

Provenance

The following attestation bundles were made for simuk-0.1.0.tar.gz:

Publisher: publish-to-test-pypi.yml on arviz-devs/simuk

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: simuk-0.1.0.tar.gz
- Subject digest: b49c5f80c27bcf8d1ef35cc31c5c83ff121f70f4d61cf0d2b8bc58396445391a
- Sigstore transparency entry: 170650731
- Sigstore integration time: Feb 12, 2025
Source repository:
- Permalink: arviz-devs/simuk@be89fe811b5cf0a2fbd40ecedd9eaae7e95200cc
- Branch / Tag: refs/tags/0.1.0
- Owner: https://github.com/arviz-devs
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: publish-to-test-pypi.yml@be89fe811b5cf0a2fbd40ecedd9eaae7e95200cc
- Trigger Event: release