Metric Audit Paper Code

Code and Python package for experiments on Marginal Baseline Evaluation (MBE): an audit protocol for testing whether a proposed generalization metric keeps predictive signal after ordinary training baselines and experimental design variables are controlled.

The current research direction is deliberately empirical. A metric can look useful under raw correlation, weaken after controls, invert sign, or survive in some architectures/tasks but not others. MBE is the framework for making those cases visible instead of treating one pooled correlation as the whole story.

What This Repository Contains

• mbe_eval/: lightweight package for raw and partial rank-correlation audits.
• examples/: small demonstrations using FIM_norm and synthetic data.
• experiments/: paper-scale and exploratory experiments, including CIFAR-10, transformers/language-model probes, and Kaggle-scale runs.
• PAPER.md and JMLR_STRATEGY.md: evolving paper notes and publication strategy.

Installation

Install the core MBE audit library:

pip install mbe-eval

For local development from this repository:

git clone https://github.com/AparajeetS/metric-audit-paper-code.git
cd metric-audit-paper-code
pip install -e .

The core install only requires NumPy, pandas, and SciPy. FIM_norm extraction uses PyTorch and is optional:

pip install "mbe-eval[torch]"

Basic Usage

Audit several candidate metrics in one dataframe:

import pandas as pd
from mbe_eval import audit_metrics

df = pd.DataFrame(
    {
        "fim_norm": [0.42, 0.51, 0.37, 0.65],
        "val_loss_ep20": [1.2, 0.9, 1.4, 0.7],
        "learning_rate": [1e-3, 1e-3, 3e-4, 3e-4],
        "weight_decay": [1e-4, 1e-5, 1e-4, 1e-5],
        "test_accuracy": [0.71, 0.78, 0.68, 0.82],
    }
)

report = audit_metrics(
    df,
    metrics=["fim_norm", "val_loss_ep20"],
    target="test_accuracy",
    controls=["learning_rate", "weight_decay"],
)

print(report[["metric", "raw_r", "partial_r", "classification"]])

Use the backward-compatible single-metric API:

from mbe_eval import MBEEvaluator

evaluator = MBEEvaluator(metric_name="FIM_norm", baseline_name="Validation Loss")
report = evaluator.evaluate(metric_vals, baseline_vals, target_vals)
print(report.partial_r, report.classification)

Reproducing Current Experiments

The current paper-scale audit lives in:

python experiments/07_jmlr_scale/analyze_jmlr_scale.py

Earlier falsification experiments are in:

python experiments/04_falsification/fim_unified_grid.py
python experiments/04_falsification/extract_tables.py

The Kaggle-scale scripts under experiments/07_jmlr_scale/ train and audit image and text models. Their outputs are summarized in jmlr_scale_v2_audit_summary.md when downloaded.

Repository Structure

metric-audit-paper-code/
+-- mbe_eval/
|   +-- __init__.py
|   +-- core.py
|   +-- utils.py
|   +-- sample_eval.py
+-- examples/
|   +-- 01_run_acid_test.py
|   +-- 02_run_heterogeneous_grid.py
+-- experiments/
|   +-- 04_falsification/
|   +-- 05_kaggle/
|   +-- 06_independent_audit/
|   +-- 07_jmlr_scale/
+-- tests/
+-- pyproject.toml
+-- setup.py
+-- README.md

Citation

@article{shadangi2026mbe,
  title={Marginal Baseline Evaluation for Auditing Generalization Metrics},
  author={Shadangi, Aparajeet},
  year={2026},
  note={Preprint}
}

License

MIT License. See LICENSE for details.