ConformalGuard

Stress-test conformal prediction before distribution shift breaks your coverage.

ConformalGuard is a CPU-friendly Python toolkit for measuring how conformal classification behaves when deployment data no longer matches calibration data. It runs reproducible IID, covariate-shift, label-shift, and concept-shift experiments and produces coverage, accuracy, set-size, empty-set, degradation, CSV/JSON, and plot reports.

Conformal prediction can provide finite-sample coverage under appropriate assumptions. ConformalGuard measures what happens when controlled deployment shifts violate those assumptions; it does not claim guarantees under arbitrary shift.

Install

Once published to PyPI:

pip install conformalguard

From source:

git clone https://github.com/chinmaygit-lab/ConformalGuard.git
cd ConformalGuard
python -m pip install -e ".[dev]"

30-second quickstart

Run the built-in reproducible demo:

conformalguard demo --out artifacts/demo

Run the same stress test on a numeric CSV dataset:

conformalguard benchmark data.csv --target label --out artifacts/my_run

Or use the Python API:

from sklearn.datasets import load_breast_cancer
from conformalguard import ConformalGuard

X, y = load_breast_cancer(as_frame=True, return_X_y=True)

guard = ConformalGuard()
guard.run(X, y)

print(guard.summary())
print(guard.degradation())
guard.save_report("artifacts")

The high-level interface automatically runs IID plus controlled covariate, label, and concept shift grids across multiple random seeds. save_report() also creates a static report.html with summary cards, tables, and plots. Advanced users can still import the lower-level experiment functions directly.

What gets generated

A standard run writes:

artifacts/
├── robustness_summary.csv
├── robustness_summary.json
├── robustness_degradation.csv
├── robustness_degradation.json
├── report.html
├── robustness_accuracy.png
├── robustness_coverage.png
├── robustness_accuracy_degradation.png
└── robustness_coverage_degradation.png

Why ConformalGuard?

A conformal predictor can look well calibrated on IID data and degrade substantially after deployment shift. ConformalGuard makes that failure mode measurable instead of hiding it behind one aggregate accuracy number.

The benchmark tracks:

• empirical coverage and coverage gap
• accuracy and macro F1
• mean prediction-set size
• empty-set rate
• IID-relative degradation
• mean and standard deviation across seeds

Supported shift families

Verified example result

One verified run on scikit-learn's breast-cancer dataset produced the following values:

These are one reproducible example run, not universal performance claims. Results depend on the dataset, split, classifier, random seeds, conformity score, confidence level, and shift configuration.

Advanced configuration

from conformalguard import ConformalGuard, GuardConfig

config = GuardConfig(
    confidence_level=0.90,
    conformity_score="lac",
    seeds=(11, 42, 73),
    covariate_severities=(0.0, 0.5, 1.0, 2.0),
    concept_severities=(0.0, 0.25, 0.50, 0.75, 1.0),
    feature_fraction=0.50,
)

guard = ConformalGuard(config)
guard.run(X, y)

CLI grids use comma-separated values:

conformalguard demo \
  --confidence 0.90 \
  --seeds 11,42,73 \
  --covariate-severities 0,0.5,1,2 \
  --concept-severities 0,0.25,0.5,0.75,1

Reproducibility and scope

ConformalGuard records or controls train/calibration/test splitting, random seeds, confidence level, conformity score, shift severity, feature fraction, label target proportions, concept-shift feature, classification metrics, and conformal metrics.

The current benchmark focuses on controlled, synthetic shifts for tabular classification. Those mechanisms isolate failure modes but do not reproduce every production environment. Natural temporal drift, online adaptation, recalibration policies, recovery-cost experiments, and broader multi-dataset evaluation remain important extensions.

Development

python -m pip install -e ".[dev]"
python -m ruff check src tests examples
python -m pytest -q
python -m build
python -m twine check dist/*

The public repository's latest documented checkpoint before this productization patch reports 116 passing tests for the experimental core.

Citation

Citation metadata is provided in CITATION.cff.

License

Apache-2.0. Review the license choice before publishing the first package release.