PMRIVariants 2.2.5

PMRI variants and PMRI-SRI for predictive reliability analysis with exports and plots

PMRIVariants

PMRIVariants is a Python package for computing the Predictive Model Reliability Index (PMRI) family of metrics for predictive, environmental, hydrological, remote-sensing, geospatial, and water-quality models.

The package is designed to move beyond conventional average-error statistics and provide a more structured evaluation of predictive reliability, including agreement, stability, uncertainty behavior, regime sensitivity, and spatial trustworthiness.

---

Project description

PMRIVariants was developed to support researchers, analysts, engineers, and practitioners who need more than conventional accuracy metrics when assessing predictive models. In many scientific and operational applications, a model may achieve a strong average fit while still being unreliable across decision boundaries, unstable across subsets, poorly calibrated in uncertainty, weak under regime shifts, or spatially inconsistent.

This package addresses that gap by providing a family of reliability-oriented metrics under the PMRI framework. It supports compact screening metrics, broader operational reliability assessment, regime-aware data-driven reliability analysis, and spatial reliability mapping for georeferenced predictions. The package is especially suitable for environmental, hydrological, water-quality, geospatial, and remote-sensing applications, but it can also be used more broadly for machine learning and predictive modelling problems.

In addition to PMRI-family metrics, the package supports common statistical performance metrics, multiple export options, and plotting utilities that make it suitable for both research workflows and publication-ready evaluation.

---

Author

Dr. Md Galal Uddin
Postdoctoral Researcher, Civil Engineering
University of Galway, Ireland
Adjunct Research Faculty, Munster Technological University, Cork, Ireland
Email: [e-mail]

---

Package variants

The package currently includes the following PMRI variants:

• PMRI-Lite
• PMRI-Core
• PMRI+
• PMRI+ Data-Driven
• PMRI-SRI (Spatial Reliability Index)

---

Table of contents

1. Concept and motivation
2. Key updates
3. Mathematical overview
4. Installation
5. Package structure
6. Required inputs by variant
7. Supported input file formats
8. Common evaluation metrics included
9. Example usage
10. Example for standard metrics
11. Plotting support
12. Example input tables
13. Main API
14. Reliability classes
15. Recommended applications
16. Scientific note
17. License

---

1. Concept and motivation

Traditional predictive performance metrics such as R², RMSE, MSE, MAE, and related statistics are useful for quantifying average model accuracy. However, they do not fully determine whether a model is sufficiently reliable for scientific interpretation, decision-support, operational deployment, regulatory assessment, or spatial applications.

A model may appear accurate on average while still being:

• unstable across subsets or ordered blocks of data,
• inconsistent near important decision thresholds,
• poorly calibrated with respect to uncertainty,
• weak in preserving the observed distribution,
• unable to represent extremes,
• or spatially unreliable across different locations.

The PMRI framework was developed to address these limitations by quantifying predictive reliability in a hierarchical and extensible manner.

PMRI-Lite

A minimal reliability score based only on monotonic agreement between observations and predictions. It is designed for quick screening when only actual and predicted values are available.

PMRI-Core

A compact reliability score that combines:

• rank consistency,
• decision agreement, and
• stability across ordered data blocks.

PMRI-Core is designed to work without requiring an explicit time variable. If a time column is not available, the native row order of the dataset can be used.

PMRI+

A broader operational reliability score that extends PMRI-Core with:

• uncertainty coverage,
• distributional similarity,
• variance equivalence,
• extreme-value fidelity,
• and equivalence testing.

PMRI+ is intended for more serious scientific or operational evaluation where reliability must be assessed across multiple dimensions.

PMRI+ Data-Driven

A regime-conditioned extension of PMRI+ that automatically clusters the available numeric predictor space. It does not require fixed variables such as temperature, flow, or turbidity. Any meaningful numeric predictors can be used. This makes the method flexible for machine learning, environmental, and remote-sensing applications where predictor sets vary between datasets.

PMRI-SRI

A spatially explicit reliability index for georeferenced datasets that combines local error, local bias, and local interval coverage to generate spatial trust maps. This is particularly useful when global metrics appear strong but local failures remain hidden.

---

2. Key updates

Recent updates include:

• compute_pmri_sri() now supports:

  • export_excel
  • export_csv
  • export_json
  • export_txt
  • make_plots
  • plot_dir

• compute_pmri_plus_data_driven() no longer requires feature_cols.

  • If omitted, the function automatically detects numeric predictor columns, excluding the actual and predicted columns.

• Common predictive metrics are now included:

  • R2
  • RMSE
  • MSE
  • MAE
  • PABE
  • MAPE
  • MBE
  • Bias
  • Pearson r
  • Spearman rho

• Exportable outputs now support:

  • summary tables,
  • detailed metric tables,
  • regime-specific outputs,
  • spatial reliability outputs,
  • text summaries,
  • JSON exports,
  • and diagnostic plots.

These updates make the package more suitable for both publication-ready analysis and routine model evaluation workflows.

---

3. Mathematical overview

3.1 PMRI-Lite

PMRI-Lite is defined as:

[ PMRI_{Lite} = \frac{\rho_s + 1}{2} ]

where:

• (\rho_s) = Spearman rank correlation coefficient.

This rescales Spearman correlation from ([-1,1]) to ([0,1]), where higher values indicate stronger monotonic agreement.

---

3.2 PMRI-Core

[ PMRI_{Core} = \frac{CAR + Stability + Rank_{01}}{3} ]

where:

• (CAR) = compliance or decision agreement rate,
• (Stability) = consistency across ordered data blocks,
• (Rank_{01}) = rescaled monotonic agreement.

PMRI-Core summarizes whether a model is not only accurate in rank ordering but also stable and decision-consistent across the dataset.

---

3.3 PMRI+

[ PMRI_{+} = mean(PICP,\ CAR,\ Stability,\ DistEq,\ VarEq,\ Tail,\ Rank_{01},\ Equivalence) ]

where:

• (PICP) = prediction interval coverage probability,
• (CAR) = classification or decision agreement,
• (Stability) = block-wise stability,
• (DistEq) = distributional equivalence,
• (VarEq) = variance equivalence,
• (Tail) = extreme-value fidelity,
• (Rank_{01}) = rank-based agreement,
• (Equivalence) = equivalence-test outcome or score.

This broader score attempts to capture operational reliability rather than only average predictive error.

---

3.4 PMRI+ Data-Driven

[ PMRI_{+}^{DD} = \sum_{k=1}^{K} w_k , PMRI_{+,k} ]

where:

• (K) = number of regimes or clusters,
• (w_k) = weight of cluster (k),
• (PMRI_{+,k}) = PMRI+ score computed within cluster (k).

This formulation allows reliability to be evaluated under different data regimes, which is particularly useful when model behavior changes across environmental states, spatial domains, or predictor conditions.

---

3.5 PMRI-SRI

[ PMRI_{SRI}(s)=w_1(1-\tilde e(s)) + w_2(1-\widetilde{|b|}(s)) + w_3PICP(s) ]

where:

• (s) = spatial location,
• (\tilde e(s)) = normalized local error,
• (\widetilde{|b|}(s)) = normalized absolute local bias,
• (PICP(s)) = local interval coverage,
• (w_1, w_2, w_3) = component weights.

PMRI-SRI is useful for identifying locally reliable and locally unreliable model zones and for generating spatial trust diagnostics.

---

4. Installation

Install directly from PyPI:

pip install PMRIVariants

You may also install from local source:

pip install .

To upgrade an existing installation:

pip install --upgrade PMRIVariants

---

5. Package structure

pmri_variants_package_v2/
├── pyproject.toml
├── README.md
├── LICENSE
├── src/
│   └── pmri_variants/
│       ├── __init__.py
│       ├── core.py
│       ├── sri.py
│       ├── analyzer.py
│       └── utils.py
├── examples/
│   ├── example_pmri_lite.csv
│   ├── example_pmri_core.csv
│   ├── example_pmri_plus.csv
│   ├── example_pmri_dd.csv
│   └── example_pmri_sri.csv
└── tests/
    └── test_smoke.py

Description of main modules

• core.py — core PMRI variants and shared metric logic
• sri.py — spatial reliability index routines
• analyzer.py — object-oriented analysis interface
• utils.py — helper functions for metrics, exports, and utilities

---

6. Required inputs by variant

Variant	Required inputs	Notes
PMRI-Lite	Actual, Predicted	Minimal version
PMRI-Core	Actual, Predicted	Uses dataset order if no explicit time is provided
PMRI+	Actual, Predicted	Uses empirical prediction interval if no interval is supplied
PMRI+ Data-Driven	Actual, Predicted, numeric predictor columns	No fixed predictor names required
PMRI-SRI	Actual, Predicted, Latitude, Longitude	Spatially explicit reliability

Column naming guidance

The package is flexible, but users should explicitly define column names when calling the functions. Typical names include:

• Actual
• Predicted
• Latitude
• Longitude

For data-driven variants, additional numeric predictor columns may include any application-specific variables.

---

7. Supported input file formats

The package can be used with data loaded from:

• CSV
• Excel (.xlsx)
• TXT / delimited text files
• preloaded pandas.DataFrame

Example: CSV

import pandas as pd
df = pd.read_csv("PMRI.csv")

Example: Excel

import pandas as pd
df = pd.read_excel("PMRI.xlsx")

Example: TXT

import pandas as pd
df = pd.read_csv("PMRI.txt", sep="\t")

Example: existing DataFrame

# df already exists in memory
result = some_function(df)

---

8. Common evaluation metrics included

In addition to PMRI-family scores, the package can compute or export the following standard performance metrics:

Metric	Meaning
R2	Coefficient of determination
RMSE	Root mean squared error
MSE	Mean squared error
MAE	Mean absolute error
PABE	Percentage absolute bias error
MAPE	Mean absolute percentage error
MBE	Mean bias error
Bias	Average signed error
Pearson r	Linear correlation
Spearman rho	Rank correlation

These metrics are useful for comparing predictive performance alongside reliability.

Brief interpretation

• R2 indicates the proportion of variance explained.
• RMSE emphasizes larger errors more strongly.
• MSE is the squared-error equivalent of RMSE without the square root.
• MAE measures average absolute error magnitude.
• PABE quantifies percentage-based absolute bias behavior.
• MAPE gives relative percentage error magnitude.
• MBE indicates directional bias in predictions.
• Bias summarizes the signed average deviation.
• Pearson r captures linear association.
• Spearman rho captures monotonic association.

In practice, these should be reported alongside PMRI outputs to show both accuracy and reliability.

---

9. Example usage

9.1 PMRI-Lite

import pandas as pd
from pmri_variants import compute_pmri_lite

df = pd.read_csv("examples/example_pmri_lite.csv")
result = compute_pmri_lite(df["Actual"], df["Predicted"])

print(result)

9.2 PMRI-Core

PMRI-Core does not require a time column.

import pandas as pd
from pmri_variants import compute_pmri_core

df = pd.read_csv("examples/example_pmri_core.csv")
result = compute_pmri_core(df["Actual"], df["Predicted"])

print(result)

9.3 PMRI+

import pandas as pd
from pmri_variants import compute_pmri_plus

df = pd.read_csv("examples/example_pmri_plus.csv")
result = compute_pmri_plus(df["Actual"], df["Predicted"])

print(result)

9.4 PMRI+ Data-Driven with explicit predictors

import pandas as pd
from pmri_variants import compute_pmri_plus_data_driven

df = pd.read_csv("examples/example_pmri_dd.csv")
result = compute_pmri_plus_data_driven(
    df,
    actual_col="Actual",
    predicted_col="Predicted",
    feature_cols=["Feature_1", "Feature_2", "Feature_3"],
    n_clusters=3,
)

print(result["pmri_plus_dd"])
print(result["regimes"])

9.5 PMRI+ Data-Driven with automatic predictor detection

import pandas as pd
from pmri_variants import compute_pmri_plus_data_driven

df = pd.read_csv("/content/PMRI.csv")
result = compute_pmri_plus_data_driven(
    df,
    actual_col="Actual",
    predicted_col="Predicted",
    n_clusters=3,
    export_excel="pmri_dd_results.xlsx",
    export_csv="pmri_dd_results.csv",
    export_json="pmri_dd_results.json",
    export_txt="pmri_dd_results.txt",
    make_plots=True,
    plot_dir="plots_dd"
)

print(result["pmri_plus_dd"])

9.6 PMRI-SRI basic use

import pandas as pd
from pmri_variants import compute_pmri_sri

df = pd.read_csv("examples/example_pmri_sri.csv")
result = compute_pmri_sri(
    df,
    actual_col="Actual",
    predicted_col="Predicted",
    lat_col="Latitude",
    lon_col="Longitude",
)

print(result["sri_mean"])

9.7 PMRI-SRI with export options and plots

import pandas as pd
from pmri_variants import compute_pmri_sri

df = pd.read_csv("/content/PMRI.csv")
result = compute_pmri_sri(
    df,
    actual_col="Actual",
    predicted_col="Predicted",
    lat_col="Latitude",
    lon_col="Longitude",
    export_excel="pmri_sri_results.xlsx",
    export_csv="pmri_sri_results.csv",
    export_json="pmri_sri_results.json",
    export_txt="pmri_sri_results.txt",
    make_plots=True,
    plot_dir="plots_sri"
)

print(result["sri_mean"])

9.8 Object-oriented interface

import pandas as pd
from pmri_variants import PMRIAnalyzer

df = pd.read_csv("examples/example_pmri_plus.csv")

analyzer = PMRIAnalyzer(
    actual_col="Actual",
    predicted_col="Predicted",
)

result = analyzer.compute_all(df)
print(result.summary)

---

10. Example for standard metrics

The following illustrates how conventional metrics may be evaluated together with PMRI outputs.

import pandas as pd
from pmri_variants import compute_pmri_sri

df = pd.read_csv("/content/PMRI.csv")

result = compute_pmri_sri(
    df,
    actual_col="Actual",
    predicted_col="Predicted",
    lat_col="Latitude",
    lon_col="Longitude",
    export_excel="pmri_metrics_results.xlsx",
    make_plots=True,
    plot_dir="pmri_plots"
)

print("SRI Mean:", result["sri_mean"])

# Depending on implementation, metrics may be available under:
# result["metrics"] or similar structured output
if "metrics" in result:
    print(result["metrics"])

Typical exported metrics may include

• R2
• RMSE
• MSE
• MAE
• PABE
• MAPE
• MBE
• Bias
• Pearson r
• Spearman rho

Example interpretation

A model might show:

• strong R2,
• moderate RMSE,
• low Bias,
• and high Pearson r,

but still exhibit weak PMRI+ if its uncertainty coverage, stability, or regime consistency is poor. This is exactly why the PMRI framework is useful: it captures reliability characteristics that average metrics alone can miss.

---

11. Plotting support

When make_plots=True, the package can export relevant diagnostic and comparison plots depending on the selected variant and implementation. These may include:

• observed vs predicted scatter plots,
• residual plots,
• error distribution histograms,
• block stability plots,
• regime-wise comparison plots,
• spatial reliability maps,
• local bias maps,
• local coverage maps,
• performance summary charts.

Example

result = compute_pmri_sri(
    df,
    actual_col="Actual",
    predicted_col="Predicted",
    lat_col="Latitude",
    lon_col="Longitude",
    make_plots=True,
    plot_dir="plots_sri"
)

Generated plots are typically stored in the folder specified by plot_dir.

Suggested use of plots

Plots are especially useful for:

• visually checking systematic bias,
• identifying clusters or regimes with different reliability,
• locating spatial hotspots of weak predictive trust,
• supporting journal manuscripts and technical reports,
• and communicating model behavior to non-technical stakeholders.

---

12. Example input tables

PMRI-Lite

Actual	Predicted
72.0	70.8
75.0	73.6
81.0	82.1

PMRI-Core

Actual	Predicted
71.2	70.9
72.0	71.8
73.4	73.2

PMRI+

Actual	Predicted
7.90	7.95
8.05	8.02
8.10	8.08

PMRI+ Data-Driven

Actual	Predicted	Feature_1	Feature_2	Feature_3
69.5	68.9	14.2	21.0	2.1
71.0	71.4	15.1	25.4	2.8

PMRI-SRI

Latitude	Longitude	Actual	Predicted
53.27	-9.05	74.0	73.2
53.28	-9.01	76.1	75.8

---

13. Main API

Functional API

• compute_pmri_lite(actual, predicted)
• compute_pmri_core(actual, predicted, threshold=None)
• compute_pmri_plus(actual, predicted, threshold=None, delta=None)
• compute_pmri_plus_data_driven(df, actual_col, predicted_col, feature_cols=None, n_clusters=3, export_excel=None, export_csv=None, export_json=None, export_txt=None, make_plots=False, plot_dir=None)
• compute_pmri_sri(df, actual_col, predicted_col, lat_col, lon_col, export_excel=None, export_csv=None, export_json=None, export_txt=None, make_plots=False, plot_dir=None)

Object-oriented API

• PMRIAnalyzer

Notes on API style

The functional API is convenient for scripted workflows, while the object-oriented API is helpful for more structured and repeatable analysis pipelines.

---

14. Reliability classes

The following classes may be used for interpretation:

Score	Class
> 0.80	Operational
0.60–0.80	Decision-support
< 0.60	Exploratory

These thresholds help distinguish whether a model is suitable for:

• exploratory research,
• scientific decision-support,
• regulatory communication,
• or near-operational deployment.

---

15. Recommended applications

PMRIVariants can be applied in:

• water-quality modelling,
• hydrological prediction,
• environmental forecasting,
• remote sensing model validation,
• geospatial model comparison,
• ecological and biogeochemical modelling,
• AI/ML predictive reliability assessment,
• spatial decision-support systems.

Typical use cases

• evaluating machine learning predictions for water-quality variables,
• comparing multiple predictive models beyond RMSE and R²,
• assessing whether a model is trustworthy across different regimes,
• diagnosing local spatial failures in otherwise strong models,
• generating publication-ready reliability evidence.

---

16. Scientific note

PMRI is intended to complement rather than replace conventional predictive metrics. In scientific studies, it is recommended to report:

• accuracy metrics such as R², RMSE, MAE, and MSE,
• alongside reliability metrics such as PMRI-Core, PMRI+, or PMRI-SRI.

This combined reporting provides a more robust basis for:

• model comparison,
• interpretation,
• operational readiness assessment,
• and communication with decision-makers.

For serious applications, users are encouraged to report both:

1. predictive accuracy, and
2. predictive reliability.

This helps distinguish between models that are merely accurate on average and models that are consistently dependable under real analytical conditions.

---

17. License

This project is licensed under the MIT License.