calibrated-bo

Calibrated Bayesian optimization with composable conformal prediction, built on top of bayesian-gp-cvloss.

What's in the box

Calibration

• CompositeConformalCalibrator with three orthogonal switches
  • localized=True — restrict to x*'s k-NN of the calibration set
  • weighted=True — RBF weights against x* on the surviving subset
  • adaptive=True — Gibbs-Candes ACI controller updates α from rolling coverage
• MultiOutputCalibrator — one calibrator per objective with a uniform API
• WeightedConformalCalibrator / LocalizedConformalCalibrator / AdaptiveConformalCalibrator — ablation presets that lock specific flags
• AdaptiveAlphaController — ACI controller with JSON export_state / load_state for cross-restart persistence
• Turning all three switches off is numerically identical to standard split CP

Acquisition

• CalibratedUCB, CalibratedEI — single-objective, plug calibrated sigma into UCB / EI; minimise via the maximise=False flag
• CalibratedEHVI — MC expected-hypervolume improvement using per-objective calibrated uncertainty; supports any direction per objective
• greedy_q_batch — sequential greedy with kriging-believer fantasies for q-batch acquisition (used by EHVI; available to user code too)

Loop

• CalibratedBOLoop — single-call BO with suggest() / observe()
  • Single- and multi-objective (n_objectives, objectives_direction)
  • Auto-refits GP every retrain_every, recalibrates every recalibrate_every
  • ACI controllers persist across recalibrations and (via export/load) across process restarts
  • q-batch via batch_size > 1

Diagnostics

• CoverageTracker — rolling and cumulative empirical coverage
• reliability_curve — promised vs empirical coverage across a level grid
• pit_histogram — PIT diagnostic under the Gaussian-sigma approximation

Install

pip install calibrated-bo

Requires bayesian-gp-cvloss>=0.3.1.

Quickstart — single objective

import numpy as np
from calibrated_bo import CalibratedBOLoop

def f(x):
    return float(np.sum((np.asarray(x) - np.array([0.3, 0.7])) ** 2))

bo = CalibratedBOLoop(
    bounds=[(0.0, 1.0), (0.0, 1.0)],
    objective="minimize",
    calibrator_config={
        "alpha": 0.1,
        "localized": True, "k": 20,
        "weighted": True,
        "adaptive": True, "gamma": 0.05,
    },
    acquisition="cUCB",                  # or "cEI"
    acquisition_kwargs={"beta": 2.0},
    batch_size=1,
    initial_random=5,
    gp_max_evals=30,
)

for _ in range(20):
    X_next = bo.suggest()
    y_next = np.array([f(x) for x in X_next])
    bo.observe(X_next, y_next)

print("best so far:", bo.best)        # (x, y)
print("diagnostics:", bo.diagnostics())

Quickstart — multi-objective

import numpy as np
from calibrated_bo import CalibratedBOLoop

def f1(x): return float((x[0] - 0.3) ** 2)
def f2(x): return float((x[0] - 0.7) ** 2)

bo = CalibratedBOLoop(
    bounds=[(0.0, 1.0)],
    n_objectives=2,
    objectives_direction=["minimize", "minimize"],
    calibrator_config={
        "alpha": 0.1,
        "localized": True, "k": 12,
        "weighted": True,
        "adaptive": True, "gamma": 0.05,
    },
    acquisition="cEHVI",
    acquisition_kwargs={"n_samples": 64},
    batch_size=1,
    initial_random=6,
    gp_max_evals=20,
)

for _ in range(20):
    X_next = bo.suggest()
    Y_next = np.array([[f1(x), f2(x)] for x in X_next])
    bo.observe(X_next, Y_next)

diag = bo.diagnostics()
print("|Pareto| =", len(diag["best"]["pareto_Y"]))
print("calibrator 0:", diag["calibrators"][0])  # alpha_current, rolling_coverage_20, ...

Calibrator-only usage

If you only want the calibrator (no BO loop), drop it on top of any fitted GPCrossValidatedOptimizer:

from bayesian_gp_cvloss import GPCrossValidatedOptimizer
from calibrated_bo import CompositeConformalCalibrator

opt = GPCrossValidatedOptimizer(X_train, y_train, scoring="cv_rmse")
opt.optimize(max_evals=50)

cal = CompositeConformalCalibrator(
    alpha=0.1, localized=True, weighted=True, adaptive=False
).fit_cv(opt)

mean, lower, upper = cal.predict_interval(X_new)

ACI persistence across restarts

# Export at shutdown
snapshot = bo.export_aci_state()         # plain dict; JSON-serialisable

# At startup, on a fresh CalibratedBOLoop:
bo2.load_aci_state(snapshot)             # rolling coverage history is back

Design notes

Three calibration flags (Localized / Weighted / Adaptive) are orthogonal:

Sign conventions: the BO loop trains every GP in max-oriented space (any minimize column is negated at GP-training time) so EHVI / Pareto / hypervolume / cEI / cUCB stay sign-agnostic internally. Y_history and bo.best stay in raw user space at the API boundary.

ACI caveat under active learning

The standard conformal coverage guarantee (P[y* in interval] >= 1-alpha marginal) holds under exchangeability of calibration and test points. In a BO loop this is violated by construction -- the next suggest() point is biased toward high-acquisition regions. Empirically the ACI controller still tracks nominal coverage well (the test suite verifies this), but no finite-sample theorem applies; treat ACI's coverage as approximate inside the BO loop. If exact coverage matters more than sample efficiency, calibrate on an independent held-out set via fit(...) rather than fit_cv(...), and refresh it periodically.