trustcv 1.0.7

Trustworthy Cross-Validation: Framework-agnostic CV with data leakage detection

trustcv — Trustworthy Cross-Validation Toolkit

Website & Docs: https://ki-smile.github.io/trustcv/

TrustCV is a framework-agnostic toolkit for reliable cross-validation in safety-critical and regulated settings. It builds on familiar scikit-learn idioms, but adds:

• Carefully designed cross-validation splitters (starting with IID in v0.1).
• Automatic data leakage and class balance checks.
• Clinical/industrial metrics with confidence intervals.
• Simple reporting utilities that support regulatory documentation.

Status: v1.0.6 – Production release with 29 CV methods across IID, Grouped, Temporal, and Spatial categories. Full data leakage detection and reporting utilities for regulatory documentation.

---

Why TrustCV?

Standard cross-validation is easy to misuse:

• Train/test splits can accidentally leak information (e.g., shared patients, timestamps, engineered features).
• Imbalanced datasets can give overly optimistic metrics if not stratified or monitored.
• For clinical and industrial applications, we often need meaningful metrics and reproducible reports, not just accuracy.

TrustCV addresses these issues by:

• Providing well-tested IID splitters with clear semantics.
• Running leakage and balance checks alongside your CV.
• Exposing clinical metrics and simple reporting utilities for audits and regulatory files.

---

What's in v1.0.0

This release includes 29 cross-validation methods across four categories:

• IID splitters (9 methods): HoldOut, KFold, StratifiedKFold, RepeatedKFold, LeaveOneOut, LeavePOut, BootstrapValidation, MonteCarloCV, NestedCV

• Grouped splitters (6 methods): GroupKFold, StratifiedGroupKFold, LeaveOneGroupOut, RepeatedGroupKFold, NestedGroupedCV, HierarchicalGroupKFold

• Temporal splitters (8 methods): TimeSeriesSplit, BlockedTimeSeriesSplit, RollingWindowCV, ExpandingWindowCV, PurgedKFold, CombinatorialPurgedKFold, PurgedGroupTimeSeriesSplit, NestedTemporalCV

• Spatial splitters (4 methods): SpatialBlockCV, BufferedSpatialCV, SpatiotemporalBlockCV, EnvironmentalHealthCV

• Framework-agnostic runner: UniversalCVRunner + CVResults for consistent, reusable CV loops across scikit-learn, PyTorch, TensorFlow, MONAI, and JAX.

• High-level validator: TrustCVValidator with automatic method selection and leakage detection.

• Data integrity checks: TrustCVValidator lightweight checks (no_duplicate_samples, no_patient_leakage, balanced_classes), plus DataLeakageChecker extended leakage analysis and LeakageReport.

• Clinical/medical metrics: ClinicalMetrics with confidence intervals (sensitivity, specificity, PPV/NPV, etc.).

• Regulatory documentation support: RegulatoryReport for generating documentation that maps to FDA/CE MDR requirements.

---

Quick Start

Installation

# Install from source (recommended for latest features)
git clone https://github.com/ki-smile/trustcv.git
cd trustcv
pip install -e .

# Or install from PyPI
pip install trustcv

Quickstart – IID CV with TrustCV

Here is a minimal example:

from trustcv import TrustCV  # or TrustCVValidator
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier

X, y = load_breast_cancer(return_X_y=True)

# Simple usage
validator = TrustCV(method="stratified_kfold", n_splits=5)
results = validator.validate(model=RandomForestClassifier(), X=X, y=y)
print(results.summary())

Full Example with All Options

from trustcv import TrustCV
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), RandomForestClassifier(random_state=42))

# Validates with leakage checks and computes clinical CIs
validator = TrustCV(
    method="stratified_kfold",
    n_splits=5,
    random_state=42,
    check_leakage=True,
    check_balance=True,
)
results = validator.validate(model=model, X=X, y=y)

print(results.summary()) 

# Output:
#=== Trustworthy Cross-Validation Results ===

#Performance Metrics (mean +/- std) (method: bootstrap):
#  accuracy: 0.956 +/- 0.014 [95% CI (bootstrap): 0.946-0.967]
#  roc_auc: 0.989 +/- 0.009 [95% CI (bootstrap): 0.981-0.995]
#  sensitivity: 0.966 +/- 0.030 [95% CI (bootstrap): 0.939-0.986]
#  specificity: 0.939 +/- 0.056 [95% CI (bootstrap): 0.893-0.981]
#  precision: 0.965 +/- 0.031 [95% CI (bootstrap): 0.937-0.989]
#  recall: 0.966 +/- 0.030 [95% CI (bootstrap): 0.939-0.986]
#  f1: 0.965 +/- 0.011 [95% CI (bootstrap): 0.957-0.973]


#Data Integrity Checks:
#  Leakage Check: PASSED
#  Class Balance: PASSED

For a higher-level workflow with leakage and balance checks, see the Quickstart: IID CV with TrustCV and IID Splitters tutorial.

Data Integrity Checks (What Is Checked and How)

TrustCV runs integrity checks during validate(...) and returns them in ValidationResult.leakage_check.

1) Built-in checks in TrustCVValidator._basic_integrity_checks

These checks run when the corresponding flags are enabled:

• check_leakage=True enables duplicate/group leakage checks.
• check_balance=True enables class-balance checks.

Key in results.leakage_check	What it checks	Method used	Pass condition
no_duplicate_samples	Exact duplicate rows in X	pandas.DataFrame.duplicated().any() (DataFrame input)	True when no duplicates are found
no_patient_leakage	Group/patient overlap between train/validation folds	Iterates the active CV splitter; computes set intersection of unique group IDs in each fold	True when there is no overlap in all folds
balanced_classes	Binary class imbalance	Computes ratio min(class_count) / max(class_count)	True when ratio is >= 0.10
balanced_multilabel (multilabel only)	Label-prevalence drift across folds	Computes max absolute deviation between per-fold prevalence and global prevalence	True when max deviation is <= 0.10

2) Extended leakage scan via DataLeakageChecker

When check_leakage=True, TrustCV can run DataLeakageChecker.check(...) and fold its outcome into results.leakage_check["has_leakage"].

Important semantics:

• LeakageReport.has_leakage == True means leakage was detected.
• results.leakage_check["has_leakage"] == True means the check passed (no leakage detected).
  In other words, this field is the inverse of LeakageReport.has_leakage.

DataLeakageChecker methods used in the scan:

Leakage type	Method used
Patient leakage	Set intersection of patient IDs in train vs test
Duplicate samples	Row hashing with pandas.util.hash_pandas_object across train/test
Temporal leakage	Timestamp parsing + flags when test starts before train, or overlap fraction is high (> 0.5)
Feature-statistics leakage	Near-identical feature means/stds (tight threshold) plus optional per-feature KS test
Spatial proximity leakage	Euclidean nearest-distance analysis; threshold can be auto-derived from distance percentiles
Near-duplicate leakage	Cosine similarity on train-normalized features (>= 0.99 default)
Label-distribution drift	Max class-proportion difference between train/test (> 0.2, optional chi-square test)

Additional methods such as hierarchical, preprocessing, and feature-target leakage checks are available in DataLeakageChecker for manual use.

Minimal usage

from trustcv import TrustCV

validator = TrustCV(
    method="stratified_kfold",
    n_splits=5,
    check_leakage=True,
    check_balance=True,
)
results = validator.validate(model=model, X=X, y=y)
# If you have group/patient IDs, pass them as:
# results = validator.validate(model=model, X=X, y=y, groups=patient_ids)

print(results.leakage_check)
# Example keys:
# {
#   "no_duplicate_samples": True,
#   "no_patient_leakage": True,
#   "balanced_classes": True,
#   "has_leakage": True
# }

Run the example notebooks

Prefer to learn by running code? From the repo root, open the notebooks in notebooks/:

• notebooks/01_IID_Methods_Showcase.ipynb – quick tour of the IID splitters and metrics.
• notebooks/02_Advanced_Workflow_UniversalRunner.ipynb – end-to-end UniversalCVRunner workflow.
• notebooks/03_TrustCVValidator_Showcase.ipynb – TrustCVValidator examples with leakage/balance checks.
• notebooks/04_TrustCVValidator_IID_Comparison.ipynb – side-by-side IID method comparison.
• notebooks/05_CrossValidation_Comparison.ipynb – comprehensive CV methods comparison.
• Reports generated by the notebooks are saved in notebooks/reports/ (HTML/PDF).

---

How TrustCV relates to scikit-learn

Similarities:

• Uses familiar scikit-learn idioms: estimators with fit/predict, splitter objects with split(X, y).
• Works seamlessly with scikit-learn models, pipelines, and metrics.
• IID splitters follow scikit-learn semantics (e.g., KFold, StratifiedKFold).

Added value:

• Leakage and balance checks: DataLeakageChecker and BalanceChecker run alongside your CV.
• Clinical metrics: ClinicalMetrics computes sensitivity, specificity, PPV/NPV, ROC/PR metrics, and CIs.
• Structured results: CVResults and ValidationResult standardize fold-level outputs.
• Reporting: UniversalRegulatoryReport turns your evaluation into a reproducible HTML/JSON report.

---

Framework Support

Supported frameworks:

• scikit-learn (native)
• PyTorch (via adapter)
• TensorFlow/Keras (via adapter)
• MONAI (via adapter, for medical imaging)
• JAX/Flax (via adapter)
• XGBoost, LightGBM, CatBoost (via sklearn-compatible API)

---

Contributors

See AUTHORS.md for a full list of contributors and acknowledgments.

Lead Contributors

• Farhad Abtahi
• Abdelamir Karbalaie

Contributing

We welcome contributions!

• Code contributions
• Medical use case examples
• Documentation improvements
• Bug reports and feature requests

Please see:

• CONTRIBUTING.md
• CODE_OF_CONDUCT.md

---

3. Quickstart: IID CV with TrustCV – outline

See file: docs/quickstart_iid.md
(and a matching notebook: notebooks/Quickstart_IID_TrustCV.ipynb)

Repository Structure

trustcv/       # Python package (splitters, validators, metrics, core)
docs/          # Documentation & guides
notebooks/     # Jupyter tutorials
examples/      # Real-world examples
tests/         # Unit & integration tests
website/       # Static site and visualizations

Development

pip install -e .[dev]
pytest tests/
cd docs && make html

Citation

If you use trustcv in your research, please cite:

@software{trustcv2025,
  title = {trustcv: Trustworthy Cross-Validation Toolkit},
  author = {Abtahi, Farhad and Karbalaie, Abdolamir},
  year = {2025},
  url = {https://github.com/ki-smile/trustcv}
}

License

MIT License — see LICENSE.

Contact & Support

• GitHub Issues: https://github.com/ki-smile/trustcv/issues

---

⚠️ Disclaimer

This toolkit is for research and educational purposes. Always validate results with domain experts before clinical deployment.

Regulatory Note: TrustCV provides documentation templates and structured outputs that can support regulatory submissions, but regulatory compliance depends on the complete device lifecycle and cannot be guaranteed by any single tool. Always consult with regulatory affairs professionals for your specific submission requirements.

---

Advancing Medical AI Through Rigorous Validation