benchmark-reliability

Benchmark Reliability Framework (BRF) -- audit whether a predictive benchmark dataset is structurally reliable before model development.

pip install benchmark-reliability

Requires Python 3.8+ with numpy, scikit-learn, pandas, scipy, openml.

Quick Start

Audit a dataset from the Registry

from brf.registry import REGISTRY_SOURCES
from brf import BRFAnalyzer
from sklearn.preprocessing import StandardScaler

# Load Teaching Assistant Evaluation (UCI ID 100)
source = REGISTRY_SOURCES["tae"]
X, y, groups, metadata = source.prepare()

# Standardize and audit
X_scaled = StandardScaler().fit_transform(X)
analyzer = BRFAnalyzer(n_splits=30, n_permutations=200).fit(X_scaled, y, groups=groups)

# Continuous metrics (always use these)
print(f"S={analyzer.S:.3f}  E={analyzer.E:.3f}")
# S=-0.423  E=0.746

# Diagnostic report: explains WHY, not just a label
print(analyzer.diagnose()["summary"])
# "The model shows no detectable, stable signal (S=-0.423 <= 0). ..."

# Per-dimension details + actionable recommendations
for dim, issue in analyzer.diagnose()["details"].items():
    print(f"  {dim}: {issue}")

# Percentile rank against the 51 benchmarks in the BRF Registry (v2.1 reference)
print(analyzer.rank())
# {'S_percentile': 17.6, 'E_percentile': 27.5, 'reference': 'BRF Registry v2.1 (51 benchmarks)'}

# One-paragraph recommendation
print(analyzer.recommend())

Browse the Registry

$ brf registry list
BRF Registry

  assistments          ASSISTments 2009-2010           N= 3729  G= 124
  college_scorecard    US College Scorecard            N= 7804  G=  59
  oulad                Open University Learning ...    N=32593  G=  22
  tae                  Teaching Assistant Evaluation   N=  151  G=  25
  ...

$ brf audit tae
BRF Audit: Teaching Assistant Evaluation (tae)
  N=151, p=4, G=25
  B=0.1172  I=1.3559  N=0.9333  M=0.6288
  S=-0.4226  E=0.7460  ->  Void

Download and verify all datasets

$ brf registry sync     # download + SHA-256 verify all
$ brf registry info oulad
  name: oulad
  display_name: Open University Learning Analytics Dataset
  n_samples: 32593  n_features: 44  n_groups: 22
  education_level: Higher Education
  country: UK
  ...

BRF Metrics

The framework computes six continuous metrics. S and E are the primary summary coordinates; the three-class label (Reliable/Void/Fragile) is communication shorthand only --- the signal is in the continuous values.

Use analyzer.diagnose() for per-dimension explanations and actionable recommendations, or analyzer.rank() to see percentile scores against the 51 benchmarks in the BRF Registry (v2.1 reference).

CLI Reference

brf audit <dataset_key>         # run BRF on a registered dataset
brf registry list               # list all datasets
brf registry download <key>     # download + cache
brf registry sync               # download + verify all datasets
brf registry info <key>         # show full metadata
brf registry verify <key>       # SHA-256 checksum check

Export

from brf.report import export_json, export_latex

export_json(analyzer.brf_vector, "results.json")
latex_table = export_latex(analyzer.brf_vector)

Citation

To cite the BRF framework and package (JOSS paper forthcoming):

@software{zhang2026brf,
  author = {Lizhuo Zhang},
  title = {benchmark-reliability: Benchmark Reliability Framework},
  url = {https://github.com/zhanglizhuo/BRFPackage},
  version = {0.3.0},
  year = {2026},
}

The behavior audit protocol is described in:

Zhang, L. BehaviorAudit: a four-dimension protocol for auditing benchmark reliability under group-aware evaluation. Scientific Reports (2026). https://doi.org/10.1038/s41598-026-69629-6

Related Work

• rliable (NeurIPS 2021): evaluates reliability of benchmark results (confidence intervals across seeds). BRF addresses the complementary question: structural reliability of the benchmark dataset itself.
• OpenML, UCI ML Repository: general-purpose dataset catalogs. The BRF Registry adds group-aware metadata, SHA-256 verification, versioning, and a standardized pipeline for reliability auditing -- filling a gap these platforms do not address.

License

MIT