Marginal Baseline Evaluation

Marginal Baseline Evaluation (MBE) is an audit protocol for testing whether machine-learning training metrics still predict held-out performance after controlling for ordinary baselines such as learning rate, weight decay, optimizer, architecture, task, and training-state measurements.

The active research direction treats reliability as conditional on the target, baseline information, and environment. MBE is being developed into a public reliability atlas and an abstaining selector that recommends metrics only when task-specific or calibrated transfer evidence supports them.

The project started from a concrete failure mode: a proposed metric can look promising under raw pooled correlation while actually tracking easier baselines, training loss, architecture mix, or other design variables. MBE makes that failure visible by comparing raw association against controlled partial rank-correlation.

Current Status

This is an active research direction with an accompanying Python package, Kaggle-scale experiment artifacts, and a public walkthrough notebook.

Software status: mbe-eval v0.4.0 implements the stable MBE v1 partial-rank audit. MBE 2.0 is the active research design and is not yet fully implemented or empirically validated. See the open research inventory for a precise map of what is available now.

The current direction is MBE 2.0, a multi-environment metric-validation framework that separates unconditional association, incremental information, transport, intervention response, and measurement reliability. The technical research program, gated JMLR roadmap, and compute plan are the source of truth for new work.

MBE does not claim that partial correlation or hyperparameter conditioning is new. The proposed contribution is their calibrated integration with a baseline information ladder, five explicit estimands, deliberately deceptive controls, configuration-blocked uncertainty, environment transport, matched interventions, measurement reliability, and scoped metric claim cards. This novelty claim must pass the public-corpus comparison gate or be narrowed.

• Package: mbe-eval
• Public notebook: Audit ML Training Metrics with MBE
• Documentation map: docs/README.md
• Conditional reliability protocol: atlas, selector, and abstention design
• Metric audit service: open-core product boundary
• Credibility ledger: passed, failed, fixed, and blocked evidence gates
• Shared method comparison: CMI, granulated Kendall, rank, and MBE benchmark
• Preregistration and claim gate: credibility freeze
• Statistical specification: estimand, inference, and assumptions
• Paper workspace: JMLR manuscript skeleton and generated tables
• Evidence ledger: SUPPORTING_EVIDENCE.md
• Reproducibility notes: REPRODUCIBILITY.md
• Legacy v1 protocol: PROTOCOL_FREEZE.md
• Metric taxonomy: METRIC_TAXONOMY.md
• Figures: FIGURES.md
• Project brief: PROJECT_BRIEF.md
• Superseded v1 experiment protocol: NEXT_EXPERIMENT_PROTOCOL.md
• Paper skeleton: PAPER_SKELETON.md
• Paper notes: PAPER.md and PUBLICATION_STRATEGY.md
• Contribution guide: CONTRIBUTING.md
• Open research inventory: OPEN_RESEARCH.md
• Grant execution plan: GRANT_EXECUTION_PLAN.md
• Independent replication protocol: docs/INDEPENDENT_REPLICATION_PROTOCOL.md
• Governance: GOVERNANCE.md
• Roadmap: ROADMAP.md

Legacy Pilot Evidence

The existing 680-row pilot ledger is exploratory evidence, not a submission-grade independent model sample. It includes repeated configurations, and the text experiment lacks a causal attention mask and permits label leakage. Its results motivate the new protocol but must not support confirmatory claims.

The minimum corrected scale design is explicit: 240 image runs (2 datasets x 3 architectures x 8 configurations x 5 seeds) plus 100 causally masked text runs (1 dataset x 2 model sizes x 10 configurations x 5 seeds). The 340 total is a blocked factorial design, not a claim of 340 independent observations. See GRANT_EXECUTION_PLAN.md.

The ledger contains:

• 480 CIFAR-10 image models across CNN, ResNet, ViT, and WideResNet settings.
• 200 character-transformer language models.
• 40+ candidate metrics including gradient/Fisher metrics, feature metrics, confidence/calibration metrics, sharpness metrics, weight norms, and distance/update proxies.

What The Evidence Suggests

The current Kaggle-scale runs support a selective audit story:

• MBE is selective rather than indiscriminate; many metrics retain signal under the declared controls.
• Several validation, confidence/logit, gradient/Fisher magnitude, and task-proximal metrics survive.
• Several feature-rank, weight-norm, distance/update, and sharpness/noise-scale metrics weaken, wash out, or invert under controls.
• FIM_norm is the motivating case study: it looked promising under conventional metric validation, then became task-dependent under MBE.

FIM_norm summary from the legacy pilot pool:

Audit	n	Raw rho	MBE partial rho	Class
Image only, default controls	480	-0.662	-0.218	survives
Image only, strict + validation loss	480	-0.662	-0.383	survives
Text only, default controls	200	-0.291	+0.014	washout
Text only, strict + validation loss	200	-0.291	+0.188	weak-or-mixed
Full image+text pool, default controls	680	+0.225	-0.203	reverse-inversion
Full image+text pool, strict + validation loss	680	+0.225	-0.300	reverse-inversion

Full result tables and interpretation are in SUPPORTING_EVIDENCE.md. CPU-only bootstrap confidence intervals and threshold sensitivity are summarized in the no-compute uncertainty report listed from the reproducibility guide.

Install

pip install mbe-eval

Supported Python versions are 3.9 and newer; CI currently exercises 3.9, 3.11, 3.13, and 3.14.

Optional FIM_norm extraction utilities require PyTorch:

pip install "mbe-eval[torch]"

For local development:

git clone https://github.com/AparajeetS/marginal-baseline-eval.git
cd marginal-baseline-eval
pip install -e ".[dev]"

Try It In One Command

After installation:

mbe-eval-demo --bootstrap 200

This runs a CPU-only synthetic audit, prints the MBE table, and writes mbe_demo_report.md. The demo is intentionally small; replace the synthetic dataframe with your training-run ledger for real experiments. Use --no-output if you only want the printed table.

To audit your own CSV ledger:

mbe-eval-audit \
  --csv runs.csv \
  --metrics fim_norm,val_loss_ep20,grad_norm \
  --target test_accuracy \
  --controls learning_rate,weight_decay,optimizer,arch \
  --groupby task \
  --bootstrap 200 \
  --seed 42 \
  --output audit_report.md \
  --results audit_results.json

The CLI fails closed if a requested metric, target, control, or grouping column is absent. Use --results audit_results.csv or --results audit_results.json for machine-readable output suitable for experiment pipelines and AI agents.

Basic API

import pandas as pd
from mbe_eval import audit_metrics, audit_report_markdown

df = pd.DataFrame(
    {
        "fim_norm": [0.42, 0.51, 0.37, 0.65, 0.62, 0.35],
        "val_loss_ep20": [1.2, 0.9, 1.4, 0.7, 0.8, 1.5],
        "learning_rate": [1e-3, 1e-3, 3e-4, 3e-4, 1e-4, 1e-4],
        "arch": ["cnn", "cnn", "resnet", "resnet", "vit", "vit"],
        "test_accuracy": [0.71, 0.78, 0.68, 0.82, 0.80, 0.66],
    }
)

report = audit_metrics(
    df,
    metrics=["fim_norm", "val_loss_ep20"],
    target="test_accuracy",
    controls=["learning_rate", "arch"],
    bootstrap=100,
)

print(report[["metric", "raw_r", "partial_r", "classification"]])
print(audit_report_markdown(report, target="test_accuracy", controls=["learning_rate", "arch"]))

Your dataframe should have one row per trained model/run, one held-out target, candidate metric columns, and baseline/design columns to control.

Reproduce Current Tables

The main paper-scale audit can be regenerated from saved result CSVs. See REPRODUCIBILITY.md for the exact legacy artifact command.

The public notebook source lives in:

kaggle/mbe_metric_audit/how_to_audit_ml_training_metrics_mbe.ipynb

Kaggle training scripts and raw result manifests are documented in REPRODUCIBILITY.md and the large-scale artifact manifest.

Repository Layout

marginal-baseline-eval/
+-- mbe_eval/                  # installable MBE package
+-- examples/                  # small local examples
+-- experiments/               # paper-scale and exploratory experiments
+-- figures/                   # generated no-compute evidence figures
+-- kaggle/mbe_metric_audit/   # public Kaggle notebook source
+-- docs/                      # documentation index
+-- SUPPORTING_EVIDENCE.md     # run-by-run evidence ledger
+-- REPRODUCIBILITY.md         # reproduction commands and expected artifacts
+-- PAPER.md                   # evolving paper direction
+-- PUBLICATION_STRATEGY.md    # publication strategy notes

Research Claim

The claim is not that any one metric is universally good or bad. The active claim is narrower and more useful:

Metric reliability is conditional on a declared target, baseline information set, and environment. MBE audits that scope and tests whether it supports a useful recommendation or an honest abstention.

This is a methodological hypothesis under active validation. The current PGDL pilot checks implementation behavior; it does not establish a general metric router, universal metric failure, or causal effects.

Citation

@article{shadangi2026mbe,
  title={Marginal Baseline Evaluation for Auditing Generalization Metrics},
  author={Shadangi, Aparajeet},
  year={2026},
  note={Preprint and open-source research artifact}
}

License

MIT License. See LICENSE.

Community And Maintenance

Scientific challenges and independent replications are welcome. See CONTRIBUTING.md, GOVERNANCE.md, CODE_OF_CONDUCT.md, and SECURITY.md.