ua-metrics

ua-metrics is a Python package for standard and uncertainty-adjusted regression metrics, accompanying the paper "Uncertainty-aware metrics for evaluating machine learning regression models in materials testing".

The package provides three uncertainty-model modules:

• ua_metrics.gaussian
• ua_metrics.student_t
• ua_metrics.lognormal

Why uncertainty-aware metrics?

Classical regression metrics (MAE, RMSE, MAPE, R²) all implicitly treat the target values (y_obs) as exact, deterministic ground truth. That assumption does not hold in materials testing, or in most physical measurement settings: every measurement carries some uncertainty from instrument precision, environmental conditions, or sample variability, often already quantified by a testing standard (e.g. a reported coefficient of variation).

Ignoring that uncertainty causes two problems:

• Learning ceiling. When a dataset contains a certain level of measurement uncertainty, models can reach a learning ceiling: the point beyond which further training does not produce real improvement in learning the true underlying signal, but instead reflects overcorrection within the uncertainty region. Classical metrics do not indicate when this irreducible limit has been reached.
• Unstable rankings. When comparing models with similar performance, small changes in the measured dataset can change which model performs better under classical metrics, leading to erroneous model ranking and potentially wrong conclusions.

ua-metrics implements uncertainty-adjusted versions of the standard metrics that subtract out the expected contribution of a known measurement uncertainty (Gaussian, Student-t, or lognormal), giving estimates that stay closer to a model's true performance. See the accompanying paper for the full derivation and validation.

Features

• Standard regression metrics.
• Uncertainty-adjusted regression metrics.
• Gaussian, Student-t, and lognormal uncertainty models.
• Heteroscedastic and homoscedastic uncertainty handling.
• A consistent module-based API.

Installation

pip install ua-metrics

Quick start

from ua_metrics import gaussian as gau
from ua_metrics import lognormal as logn
from ua_metrics import student_t as t

Default behavior (heteroscedastic uncertainty, mean scaling) and its equivalent explicit form:

v1 = gau.mae_ua([100, 110, 95], [102, 108, 97], 5.0)

v2 = gau.rmse_ua(
    [100, 110, 95],
    [102, 108, 97],
    5.0,
    mode="hetero",
    scale="mean",
)

Homoscedastic uncertainty with a constant absolute standard deviation:

v3 = gau.mae_ua([100, 110, 95], [102, 108, 97], 1.0, mode="homo")

Student-t and lognormal uncertainty models:

v4 = t.mae_ua([100, 110, 95], [102, 108, 97], 7.5, df=3.0)
v5 = logn.mae_ua([100, 110, 95], [102, 108, 97], 12.0)

Package structure

Standard metrics

from ua_metrics import mae, median_absolute_error, mse, rmse
from ua_metrics import mape, smape, r2_score, adjusted_r2_score

Uncertainty-adjusted metrics

from ua_metrics import gaussian as gau
from ua_metrics import student_t as t
from ua_metrics import lognormal as logn

Each uncertainty module provides:

• mae_ua
• median_absolute_error_ua
• mse_ua
• rmse_ua
• mape_ua
• smape_ua
• r2_score_ua
• adjusted_r2_score_ua

Interface

All uncertainty-adjusted metrics use the same public interface:

metric_ua(y_obs, y_pred, value, *, mode="hetero", scale="mean", ...)

Argument semantics

• value with mode="hetero" is interpreted as CV percent of uncertainty.
• value with mode="homo" is interpreted as a constant absolute uncertainty standard deviation.

Default behavior

The default configuration is:

• mode="hetero"
• scale="mean"

Under this default, value=5.0 means the uncertainty standard deviation is 5% of the observation-wise mean scale.

Scale definitions

Supported scale values are:

• "mean", defined as 0.5 * (y_obs + y_pred)
• "y_obs"
• "y_pred"

Notes

• Values greater than 100 are allowed in heteroscedastic mode.
• Homoscedastic mode expects a scalar constant absolute uncertainty value.

Citation

If you use ua-metrics in your work, please cite the accompanying paper, published in Measurement, Volume 290, Part C, Article 122931 (2026), doi:10.1016/j.measurement.2026.122931, and/or the software itself. Machine-readable citation metadata is kept up to date in CITATION.cff.

@article{abbas_ua_metrics_paper,
  title   = {Uncertainty-aware metrics for evaluating machine learning regression models in materials testing},
  author  = {Abbas, Mohammad and Zaumanis, Martins},
  journal = {Measurement},
  volume  = {290},
  number  = {Part C},
  pages   = {122931},
  year    = {2026},
  issn    = {0263-2241},
  doi     = {10.1016/j.measurement.2026.122931}
}

@software{abbas_ua_metrics_software,
  title   = {ua-metrics},
  author  = {Abbas, Mohammad},
  year    = {2026},
  version = {1.0.1},
  url     = {https://github.com/Moha-Abbas/ua-metrics}
}

License

MIT. See LICENSE.