sma — Sports Model Integrity Auditor

A model can look great in a notebook — high accuracy, clean metrics — and still fall over in production, because the number was never the problem. sma audits a fitted model plus its dataset for five of the most common ways that happens, and returns a plain-language report card instead of a wall of statistics.

Built framework-agnostic: works with anything exposing .predict — scikit-learn, XGBoost, LightGBM, or a hand-rolled class. See sma/core/checks/base.py.

Live demo → (free-tier hosting — first request after inactivity takes ~30–50s to wake up)

What it checks

Every check returns PASS / WARN / FAIL / NOT_APPLICABLE with structured evidence — never a bare pass/fail with no explanation.

Real-world validation

Built and tested against synthetic failure cases (a deliberate 9:1 class imbalance, a leaky churn feature, a genuine look-ahead bug, a simulated "price increase" distribution shift) — all five checks fired exactly as designed, with 32 passing unit tests.

Then dogfooded against PulseConnect, an F1 telemetry and driver-compatibility ML platform, using its real GradientBoostingRegressor and 8 seasons of real race data (2019–2026):

• Leakage checks came back clean — target_leakage and temporal_leakage both passed, which is itself evidence the leakage fix already documented in PulseConnect's own code comments (recency-windowed features, season-indexed constructor form) actually worked.
• data_drift caught something real: comparing early seasons (2019–2022) against recent ones (2023–2026), driver pace and tyre-degradation features showed major distribution shift (PSI 1.3–1.4) — a legitimate signal of F1's 2022 regulation overhaul, not a bug. It's concrete evidence that pooling all seasons as one training set treats different regulation eras as statistically the same when they aren't.

Architecture

sma/
├── sma/
│   ├── core/
│   │   ├── checks/           # one file per check, all sharing base.py's contract
│   │   │   ├── base.py       # CheckResult, Status, ModelAdapter
│   │   │   ├── class_imbalance.py
│   │   │   ├── small_sample.py
│   │   │   ├── target_leakage.py
│   │   │   ├── temporal_leakage.py
│   │   │   └── data_drift.py
│   │   ├── auditor.py        # orchestrates: run(model, X, y, ...) -> Report
│   │   └── report.py         # aggregates CheckResults, computes overall status
│   ├── report_renderers/     # Report -> markdown / html
│   └── cli.py
├── api/
│   ├── main.py                # FastAPI wrapper — imports sma.core directly
│   └── static/index.html      # terminal-styled demo frontend
└── tests/                      # 32 tests, one file per check

Every check shares one contract:

def run(model, X, y, **kwargs) -> CheckResult:
    ...

registered in sma/core/auditor.py::DEFAULT_CHECKS. The CLI, the API, and both report renderers all consume the same Auditor.run() — nothing is duplicated between them, so what a pip installer gets is exactly what the hosted demo runs. A check that fails to run (missing optional input, not-yet-implemented) is isolated and reported as NOT_APPLICABLE, never crashes the whole audit.

Install

pip install sma-audit          # once published — see below
# or, for local development:
git clone https://github.com/kopommops/sma-audit
cd sma-audit
pip install -e ".[dev,api]"

Use as a package

from sma import Auditor

report = Auditor().run(model, X, y)
print(report.overall_status)      # Status.PASS / WARN / FAIL
print(report.to_dict())

With the optional inputs each check can use:

report = Auditor().run(
    model, X, y,
    timestamps=df["date"],           # enables temporal_leakage
    reference_X=training_data,       # enables data_drift
)

Use as a CLI

sma audit model.pkl --data train.csv --target churned --format html --out report.html

# with timestamps + a chronological split check:
sma audit model.pkl --data train.csv --target churned \
  --timestamp-col date --split-date 2026-01-01

# with a reference dataset for drift detection:
sma audit model.pkl --data current.csv --target churned \
  --reference-data training_data.csv

Run the API + demo frontend locally

uvicorn api.main:app --reload
# visit http://127.0.0.1:8000 for the terminal UI,
# or POST directly to /audit (JSON) or /audit/html (rendered report)

Run tests

pytest -q
# 32 passed

Design notes

• Framework-agnostic by construction — ModelAdapter (sma/core/checks/base.py) wraps any object exposing .predict; checks never call the model directly, so swapping in a future model type (a raw PyTorch wrapper, say) means changing one adapter, not five checks.
• Isolated check failures — the orchestrator catches exceptions per-check and reports them as a FAIL with the error message, rather than letting one broken check take down the whole audit.
• Evidence over verdicts — every result carries structured evidence (counts, scores, thresholds), not just a status label, so a report card is something you can actually investigate, not just trust.

Roadmap

• All 5 v1 checks implemented and tested
• CLI, package API, and FastAPI demo, all sharing one core
• Dogfooded against a real production model (PulseConnect)
• Hosted demo deployed
• Publish to PyPI
• Config object for per-check thresholds (currently hardcoded per-module constants)
• Additional checks (e.g. feature importance stability, calibration)

License

MIT — see LICENSE.