aggmetrics

Aggregating classification performance measures for machine learning: indicator matrices and aggregation operators.

Any binary performance measure expressible in terms of the four confusion-matrix counts is extended to multiclass, multilabel, ordinal, cost-sensitive and soft-label settings by substituting one of a small set of aggregation operators. Micro-, macro- and exemplar-averaging (and pooling across cross-validation folds) become instances of a single construction rather than separate conventions.

The package also reports when a measure is being used outside the range where it behaves as commonly assumed — which is the part that is hard to get from a formula library.

import numpy as np
from aggmetrics import IndicatorMatrix, diagnose

y_true = np.array([0, 1, 2, 2, 1, 0])
y_pred = np.array([0, 2, 2, 1, 1, 1])
im = IndicatorMatrix(y_true, y_pred, task='multiclass')

im.compute('f1', 'macro')          # 0.5222
im.compute('f1', 'micro')          # 0.5   -- equals accuracy on single-label data
im.compute('mcc', 'macro')         # macro-MCC: no standard implementation exists
im.compute('informedness', 'micro')
im.accuracy()                      # 0.5

im.report()                        # DataFrame: measures x operators

Documentation: https://aggmetrics.readthedocs.io

Installation

pip install aggmetrics                # core, requires only numpy
pip install "aggmetrics[all]"         # adds scikit-learn and pandas support

Why not just use sklearn.metrics?

sklearn exposes four average values for a fixed list of measures. The correspondence is exact — 'micro', 'macro', 'weighted' and 'samples' are Σ₁, Σₘ with uniform weights, Σₘ with prevalence weights and Σₙ respectively — but the combinations available are a subset of those the construction generates, and nothing in the library tells you when a combination is misleading.

Three things this package adds:

Any measure under any operator. Macro-MCC, exemplar-Jaccard, micro-informedness, weighted G-mean. 21 binary measures × 4 operators, without special-casing.

Structural properties as first-class data. Each measure records whether it is aggregation-decomposable, whether it is skew-invariant, and whether its value at tp = 0 depends on tn. These determine which identities apply.

from aggmetrics import get_measure, list_measures

list_measures(decomposable=True)       # micro == denominator-weighted macro
get_measure('mcc').decomposable        # False
get_measure('recall').bayes_threshold()  # 0.0 -- maximised by predicting all positive
get_measure('mcc').attainable_floor(10)  # -0.111, not -1

The generative claim, as an operation. Supply a binary formula; the properties that decide which identities apply are derived, not declared:

from aggmetrics import register_measure, infer_properties

def prevalence_threshold(tp, fp, fn, tn):
    r, s = tp / (tp + fn), tn / (tn + fp)
    return (np.sqrt(r * (1 - s)) + s - 1) / (r + s - 1)

m = register_measure('prevalence_threshold', prevalence_threshold)
m.decomposable          # False  -- inferred by fitting <a,v>/<b,v> and validating
m.skew_invariant        # True   -- a function of recall and specificity
m.bayes_threshold()     # None   -- no closed form outside the decomposable class

im.compute('prevalence_threshold', 'exemplar')   # usable immediately

Inference is checked against the 21 measures whose properties are known analytically — verify_inference() — and all 21 agree, with recovered coefficients accurate to 4.2e-15. The check paid for itself: it found two errors in our own hand annotations (error rate and FPR both retain a tn dependence at tp = 0, so both flatter a wholly wrong classifier, reaching 2/m and 1/(m-1) respectively).

Diagnostics. diagnose() flags the traps:

y = np.arange(500) % 10
yhat = (y + 1) % 10                    # every single prediction is wrong
im = IndicatorMatrix(y, yhat, task='multiclass')

im.compute('label_accuracy', 'micro')  # 0.800  (!)
im.compute('mcc', 'micro')             # -0.111 (!)
im.accuracy()                          # 0.000

for f in diagnose(im):
    print(f)
# [warning] inflated_floor: 'label_accuracy' cannot fall below 0.8000 on this
#           10-class one-hot problem even if every prediction is wrong ...
# [note]    micro_collapse: micro-precision, micro-recall and micro-F1 are all
#           equal to accuracy ...
# [warning] degenerate_optimum: 'recall' is maximised at threshold 0.000 ...

Checks currently implemented: pooled value outside the per-class range; tn-dependent measures on one-hot multiclass data; micro-collapse on single-label data; undefined per-class entries and the size of the convention effect; measures whose optimal decision rule is not the Bayes rule for 0/1 loss.

Features

• 21 binary measures, resolvable by name or alias (iou → jaccard, dice → f1), plus register_measure for your own, with properties inferred.
• Four operators: micro, macro, weighted, exemplar.
• Explicit undefined-value conventions: exclude (default), zero (sklearn's behaviour), one, nan. The choice is a parameter, not a default buried in the implementation.
• Soft ground truth via t-norms, with marginal_defect() showing that only the product t-norm preserves the marginals.
• Cost matrices, recovering MAE and MSE as special cases.
• Observation masks for partial labels, semi-supervised evaluation and abstention.
• scikit-learn scorers via make_scorer, usable in cross_val_score and GridSearchCV.
• A skill for AI coding assistants (.agents/skills/aggmetrics-guide/): a guide that tells an assistant how to evaluate with the package, what to check before reporting a number (operator, convention, diagnose, pooled folds, degenerate optima) and which notebook shows each pitfall. It includes reference pages for each part of the theory and a worked session with real output. To use it in your own project, copy the folder to .claude/skills/ (Claude Code) or to the place your assistant reads skills from.

The test suite verifies the theory

Each result of the companion paper is an executable test, so a change that breaks a theorem breaks a test:

python -m pytest tests/test_theory.py -v

Test	Result verified
test_theorem1_holds_for_decomposable	micro = denominator-weighted macro, the 14 decomposable measures × 3 samples
test_theorem1_fails_for_non_decomposable	the identity has no extension beyond that class
test_theorem3_product_is_uniquely_marginally_consistent	only the product t-norm preserves marginals
test_theorem2_micro_collapse	micro-P = micro-R = micro-F₁ = accuracy, single-label
test_proposition9_thresholds	Bayes threshold (β+δ)/(α+β+γ+δ)
test_proposition5_floors	attainable floors under one-hot encoding
test_proposition6_recall_prior_free_specificity_not	one-vs-rest skew behaviour
test_corollary6_gap_is_normalised_covariance	the micro–macro gap identity
test_theorem4_mae_and_mse_as_cost_matrices	ordinal losses as cost matrices
test_inference_reproduces_every_annotation	inference recovers all 21 analytic annotations

210 tests, no skips, 100% line and branch coverage.

Examples

Notebooks, executed and checked in continuous integration. Each starts from a problem a practitioner recognises, shows what the usual tools report, and then what aggmetrics reports.

Notebook	What it shows
01 The floor trap	A classifier wrong on every example scores 0.998 on label accuracy with 1000 classes
02 Micro, macro and imbalance	Micro-F1 equals accuracy; micro-MCC outside the per-class range; the hidden choice for 0/0
03 Multilabel: the ranking depends on the operator	Three models on yeast ranked under 18 measure-operator combinations
04 Ordinal grades and costs	Nine in ten errors are one grade off; a cost matrix changes which prediction is best
05 Model selection	The scorer picks the model, pooled and averaged folds differ, and a test that fails on a flawed evaluation
06 Long-tailed images	The same CIFAR-10-LT model scores 0.92 on a long-tailed set and 0.66 on a balanced one; what changes with the class prior
07 Choosing a threshold	Recall, specificity and precision are maximised by trivial classifiers; a cost-weighted measure has a closed-form threshold
08 Soft labels	Image classifiers against the distribution of 50 human annotators (CIFAR-10H): does the soft reference change the verdict, where do the models err, and a simulation of when it reverses the verdict
09 Segmentation	Cell nuclei (BBBC039): pixel accuracy gives 0.82 to a predictor that outputs nothing; pooled and per-image IoU, and the convention for images with no object
10 Custom measures	Register your own formula: what is inferred (decomposable, skew-invariant), what follows on data, and a piecewise measure on which the inference is wrong
11 PyTorch and Keras	Evaluating framework models: logits, pooling batches instead of averaging a measure over them (macro-MCC 11% higher in the averaged form), accumulating counts, and a Keras callback that logs a measure per epoch
12 Choosing measures	Remark 2 on data: which measures are redundant (rank of the Jacobian), the best pair for a binary problem, and why 63 aggregated measures span only 11 of the 90 directions of a ten-class confusion matrix

The notebooks and their data are in the repository, not in the pip package. The Colab badges need nothing installed. To run them locally, clone the repository (the examples extra installs only the libraries the notebooks use):

git clone https://github.com/rcprati/aggmetrics
cd aggmetrics
pip install -e ".[examples]" jupyter     # add ,deep to the extras for notebook 11 (torch, keras)
jupyter lab examples/notebooks

A short script version of the tour is examples/quickstart.py, also in the clone:

python examples/quickstart.py

Citing

Please cite both the software and the companion paper. The software is archived on Zenodo: the DOI 10.5281/zenodo.23535051 always resolves to the latest version, version 0.1.1 is 10.5281/zenodo.23585309 and version 0.1.0 is 10.5281/zenodo.23535052. GitHub's "Cite this repository" button reads CITATION.cff. In BibTeX:

@software{prati2026aggmetrics,
  author  = {Prati, Ronaldo C.},
  title   = {aggmetrics: aggregating classification performance measures for machine learning},
  year    = {2026},
  doi     = {10.5281/zenodo.23535051},
  url     = {https://github.com/rcprati/aggmetrics}
}

The paper is:

@article{prati2027unified,
  author  = {Prati, Ronaldo C.},
  title   = {A unified algebraic framework for classification performance evaluation},
  journal = {Information Fusion},
  volume  = {138},
  pages   = {104767},
  year    = {2027},
  doi     = {10.1016/j.inffus.2026.104767}
}

The package itself is described in a preprint (arXiv:2610.30456):

@misc{prati2026aggmetricspaper,
  author        = {Prati, Ronaldo C.},
  title         = {aggmetrics: Aggregating Classification Performance Measures for Machine Learning},
  year          = {2026},
  eprint        = {2610.30456},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG}
}

License

BSD 3-Clause.