statbelt 0.1.1

A strict experimental harness for reproducible, statistically valid model evaluation.

statbelt

statbelt is a strict experimental harness for reproducible, statistically aware model evaluation in Python.

Status: Alpha (APIs may evolve).
Supported Python: 3.11+.

Installation

Install from PyPI:

pip install statbelt

For local development:

uv sync --all-groups

Quick Start

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from statbelt import ExperimentalHarness

X, y = make_classification(n_samples=120, random_state=21)

report = (
    ExperimentalHarness()
    .data(X, y)
    .task("binary_classification")
    .compare(
        ("logreg", LogisticRegression(max_iter=500)),
        ("rf", RandomForestClassifier(n_estimators=25, random_state=21)),
    )
    .metrics("accuracy", "roc_auc", "log_loss")
    .design(cv=5, random_state=42)
    .inference(alpha=0.05, bootstrap_resamples=2000)
    .fasten("statbelt.lock.json")
    .evaluate()
)

print(report.summary())

Core Features

• ExperimentalHarness builder-style API for binary classification comparisons.
• Deterministic stratified k-fold evaluation with shared folds across models.
• Bootstrap confidence intervals over fold-level metrics.
• Lock artifact output (statbelt.lock.json) with config and split indices.
• Strict staged workflow: configure -> fasten() -> evaluate().

Supported Task and Metrics

Supported task:

• binary_classification

Supported metrics:

• accuracy
• precision
• recall
• f1
• roc_auc
• log_loss

Validation is fail-fast. For example:

• log_loss requires predict_proba.
• roc_auc requires predict_proba or decision_function.

Development

uv sync --all-groups
uv run ruff check .
uv run pytest

For release operations (tagging, TestPyPI gate, PyPI publish), see RELEASING.md.

Current Limits

• Binary classification only.
• Confidence intervals only (no pairwise hypothesis tests yet).
• Python API is the only supported interface in v0.

License

This project is licensed under the GNU Affero General Public License, version 3 or later (AGPL-3.0-or-later). See LICENSE.