Forecast Performance

forecast_performance is a Python library created by FORESIGHT - Forecasting and Optimization for Resilient Environmental Systems through Investigation with Groundbreaking Hydrological Tools for evaluating the skill of deterministic and probabilistic forecasting models. It provides a single unified interface — ForecastPerformance — that handles point, quantile, and ensemble forecasts and exposes a rich set of metrics and visualisation tools.

New here? Start with the notebooks in notebooks/, each a self-contained, runnable walkthrough: 00_visualize (Plotly forecast plots) · 01_benchmarks (persistence & climatology baselines) · 02_deterministic · 03_ensemble · 04_probabilistic. Run them under the forecast_performance kernel (see Installation).

Features

Installation

Two paths: install the released package from PyPI (use the package), or install from source (develop / run the notebooks and tests).

A · Install from PyPI

pip install forecast-performance

That is all — the runtime dependencies (numpy, pandas, scipy, matplotlib, plotly, pyarrow) are declared in the wheel, so pip resolves them automatically. Upgrading later needs no URL: pip install --upgrade forecast-performance always picks up the newest release.

The distribution is forecast-performance; the import is forecast_performance:

from forecast_performance import ForecastPerformance

Every release is also attached to the GitHub releases page, which is useful to pin an exact build without going through PyPI:

pip install https://github.com/FORESIGHT-ULisboa/forecast_performance/releases/download/v1.0.0/forecast_performance-1.0.0-py3-none-any.whl

B · Install from source (development)

1 Create a conda environment

conda create -n forecast_performance python=3.11
conda activate forecast_performance

2 Install the package and all dependencies

From the repository root, run:

pip install -e ".[dev]"

3 Register the Jupyter kernel

So the notebooks pick up the right environment:

python -m ipykernel install --user --name forecast_performance --display-name "forecast_performance"

4 Open the notebooks

Open them directly in VS Code and select the forecast_performance kernel in the top-right kernel picker, or run:

jupyter lab notebooks\

Core concepts

Canonical long format

Internally every forecast is stored as a single-column DataFrame whose row index is a MultiIndex drawn from these levels:

You don't have to build this by hand. fp.add(df, name=...) calls ForecastPerformance.normalize_dataframe for you, which accepts wide frames (datetime row index, MultiIndex columns) or long frames, normalises common level-name aliases (probability/prob/quantile → non_exceedance, ensemble/member → ensemble_member, lead/lead_time → leadtime, …), and derives the missing one of production_datetime / event_datetime / leadtime when the other two are present.

Metrics are names and handles

Every metric is a Metric object — a callable that stringifies to its own name. This means you can pass it as a handle (rmse) or as a string ("rmse"), and you can drop it straight into a results table without metric.__name__:

str(rmse) == "rmse"     # True
rmse == "rmse"          # True
rmse(forecast, obs)     # callable

Every metric is also a convenience handle attribute on the instance under its common-usage name (acronyms uppercased like fp.RMSE, fp.CRPS, fp.fair_CRPS; word-based metrics snake_case like fp.reliability), so you can build a metrics list without importing anything:

metrics = [fp.CRPS, fp.fair_CRPS, "reliability", "resolution"]
for metric in metrics:
    fp.probabilistic(metric, "prob_model", leadtime=lt)

Saving forecasts with DateOffset leadtimes (PandasForecast)

Seasonal / monthly forecasts often express leadtime as a pd.DateOffset (pd.DateOffset(months=1), pd.DateOffset(years=1)) instead of a pd.Timedelta, because calendar months and years have a variable length. Parquet cannot serialize DateOffset objects stored in an index or in the columns, so a plain df.to_parquet(...) raises.

PandasForecast is a drop-in pd.DataFrame subclass that fixes this: it encodes each DateOffset in a leadtime level (whether in the index or the columns) on write and restores it on read. Use it exactly like a DataFrame:

from forecast_performance import PandasForecast

PandasForecast(df).to_parquet("forecast.parquet")     # df has a DateOffset leadtime
back = PandasForecast.read_parquet("forecast.parquet")  # plain DataFrame by default
back.index.get_level_values("leadtime")[0]            # <DateOffset: months=1>

read_parquet returns a plain pd.DataFrame by default (so the subclass type never leaks downstream); pass to_pandas=False to get a PandasForecast back.

It is fully backward compatible: a frame with a Timedelta (or no) leadtime is written verbatim by the normal pandas writer, and a file written this way is still readable by plain pd.read_parquet (the leadtime then holds the encoded strings).

Quick start

import pandas as pd
import numpy as np
from forecast_performance import ForecastPerformance, rmse, nse, crps

dates = pd.date_range("2020-01-01", periods=365, freq="D")
reference = pd.Series(
    np.sin(np.arange(365) * 2 * np.pi / 365), index=dates, name="Reference"
)
fp = ForecastPerformance(reference)

Quieting warnings. Probabilistic CRPS warns when a forecast's CDF does not span [0, 1]. Pass ForecastPerformance(reference, warn=False) to silence these informative UserWarnings. (Spurious numerical warnings from the internal integrals are always suppressed.)

Point (deterministic) forecast

forecast = pd.DataFrame(
    reference.values + np.random.normal(0, 0.1, 365),
    index=dates,
    columns=pd.Index([pd.Timedelta("0D")], name="leadtime"),
)
fp.add(forecast, name="my_model")

# Three equivalent calling styles:
fp.deterministic(rmse, "my_model")       # metric handle
fp.deterministic("rmse", "my_model")     # metric name (or alias, e.g. "RMSE")
fp.deterministic.rmse("my_model")        # discoverable accessor (autocompletes)

Quantile (probabilistic) forecast

QUANTILE_LEVELS = [0.1, 0.3, 0.5, 0.7, 0.9]
quantile_df = pd.DataFrame(
    ...,  # shape (n_dates, n_quantiles)
    index=dates,
    columns=pd.MultiIndex.from_product(
        [[pd.Timedelta("0D")], QUANTILE_LEVELS],
        names=["leadtime", "non_exceedance"],
    ),
)
fp.add(quantile_df, name="prob_model")

lt = pd.Timedelta("0D")
fp.probabilistic(crps, "prob_model", leadtime=lt)      # handle
fp.probabilistic("crps", "prob_model", leadtime=lt)    # name
fp.probabilistic.crps("prob_model", leadtime=lt)       # accessor

Ensemble forecast

N_MEMBERS = 20
ensemble_df = pd.DataFrame(
    ...,  # shape (n_dates, N_MEMBERS)
    index=dates,
    columns=pd.MultiIndex.from_product(
        [[pd.Timedelta("0D")], range(N_MEMBERS)],
        names=["leadtime", "ensemble_member"],
    ),
)
fp.add(ensemble_df, name="ens_model")
fp.probabilistic.fair_crps("ens_model", leadtime=pd.Timedelta("0D"))
fp.probabilistic.brier_score("ens_model", leadtime=pd.Timedelta("0D"), threshold=0.5)

Collecting results

Results accumulates rows and pivots them into a DataFrame. Because metrics stringify to their name, append the metric object directly:

from forecast_performance import Results

results = Results("Model", "Metric", "Leadtime")
for name in fp.names():
    for metric in fp.deterministic.metrics:
        for lt in fp.simulations[name]["leadtimes"]:
            results.append(
                Model=name,
                Metric=metric,                              # no .__name__
                Leadtime=lt,
                Value=fp.deterministic(metric, name, leadtime=lt),
            )

table = results.to_pandas(index=["Metric", "Model"], columns=["Leadtime"])

Baselines, corrections and housekeeping

persistence = fp.get_persistence(leadtimes=pd.timedelta_range("0D", "10D", freq="1D"))
climatology = fp.get_climatology(rolling_window=30)
fp.add(climatology, name="climatology")

fp.adjust_mean("ens_model")     # shift ensemble mean to the reference mean
fp.adjust_scale("ens_model")    # scale ensemble mean to the reference mean

fp.clear_cache("ens_model")     # drop cached intermediates (force recompute)
fp.remove("climatology")        # delete a simulation entirely

Visualisation

import plotly.graph_objects as go
from forecast_performance import plotly_forecasting as gof

fp.qq_plot("prob_model")        # PIT / Q-Q calibration plot (matplotlib)

fig = go.Figure()
gof.plot_lt_probabilistic(fig, quantile_df_long, leadtimes=[pd.Timedelta("0D")])
gof.add_observed_trace(fig, reference)
gof.apply_default_layout(fig, yaxis_title="Q [m3/s]")

Project structure

forecast_performance/
├── forecast_performance/
│   ├── __init__.py                 # Public re-exports (metrics, Metric, registries)
│   ├── forecast_performance.py     # ForecastPerformance main class
│   ├── pandas_forecast.py          # PandasForecast (DateOffset-aware parquet I/O)
│   ├── results.py                  # Results accumulator class
│   ├── decorators.py               # storedResults caching decorator
│   ├── plotly_forecasting.py       # Plotly visualisation helpers
│   └── metrics/
│       ├── __init__.py             # metric exports + DETERMINISTIC/PROBABILISTIC registries
│       ├── base.py                 # Metric (callable that == its name)
│       ├── accessors.py            # fp.deterministic / fp.probabilistic accessors
│       ├── deterministic.py        # Pure deterministic metric functions
│       └── probabilistic.py        # Pure probabilistic metric functions
├── notebooks/                      # runnable usage examples (see "Notebooks" above)
│   ├── 00_visualize.ipynb          # Plotly forecast visualisation
│   ├── 01_benchmarks.ipynb         # persistence & climatology baselines
│   ├── 02_deterministic.ipynb      # deterministic metrics workflow
│   ├── 03_ensemble.ipynb           # ensemble metrics workflow
│   └── 04_probabilistic.ipynb      # probabilistic / quantile metrics workflow
├── tests/
│   ├── conftest.py                 # daily-parquet + synthetic fixtures
│   ├── test_forecast_performance.py
│   ├── test_deterministic.py
│   ├── test_probabilistic.py
│   ├── test_normalize.py
│   ├── test_results.py
│   ├── test_plotting.py
│   ├── test_missing_data.py
│   ├── test_datasets_daily/        # obs/det/ens/prob parquet datasets (used by conftest)
│   └── test_datasets_hourly/       # only used by its own aux notebook
├── AGENTS.md                       # conventions for AI coding agents (canonical)
├── CLAUDE.md                       # → points to AGENTS.md
├── .github/copilot-instructions.md # → points to AGENTS.md
├── .github/workflows/tests.yml     # pytest on push / PR (3.11, 3.12)
├── .github/workflows/release.yml   # build + publish to PyPI on tag push
├── .github/scripts/check_version.py # release guard: version declared consistently
├── pyproject.toml                  # packaging + pytest/coverage configuration
├── MANIFEST.in                     # keeps the 34 MB tests/ tree out of the sdist
├── LICENSE                         # MIT
└── README.md

Running the tests

pytest tests/ -v

The suite also runs in CI on every push and pull request (.github/workflows/tests.yml) against Python 3.11 and 3.12.

The tests are not shipped in the PyPI sdist — their parquet fixtures are ~34 MB. Clone the repository, or download a tagged source archive, to run them.

Building a distribution (wheel + sdist)

The package builds with the standard PEP 517 toolchain. Install the build extra and run the build frontend from the repository root:

conda activate forecast_performance
pip install -e ".[build]"
python -m build

This produces both artifacts in dist/:

dist/
├── forecast_performance-1.0.0-py3-none-any.whl
└── forecast_performance-1.0.0.tar.gz

Install the wheel anywhere (no source checkout needed):

pip install dist/forecast_performance-1.0.0-py3-none-any.whl

Check the metadata the way CI does:

twine check --strict dist/*

Notes:

• The version is set in pyproject.toml (project.version) and mirrored in forecast_performance.__version__ — bump both together. .github/scripts/check_version.py enforces that at release time.
• Only the forecast_performance package is shipped. MANIFEST.in prunes tests/, whose parquet fixtures are ~34 MB.
• Building by hand is only needed to inspect an artifact locally, and never twine upload by hand — releases are produced by CI, which builds into dist-release/ so the two never mix. See Releasing below.

Releasing

Releases are automated by .github/workflows/release.yml. To cut version X.Y.Z:

1. Bump the version in both pyproject.toml and forecast_performance.__version__, refresh the pinned wheel URL under Installation A, then commit.

2. Rehearse if you want to: run the workflow from the Actions tab and leave the target as testpypi. It builds and uploads to TestPyPI, no tag needed, and reruns are idempotent.

3. Tag and push:

   git tag vX.Y.Z
   git push origin vX.Y.Z
   

The workflow then verifies the two version declarations agree with the tag, builds the wheel and sdist, runs twine check --strict, publishes to PyPI, and attaches both artifacts to the GitHub release.

PyPI uploads are immutable — a version number can never be reused, even after deletion. Get the version right before pushing the tag, and never move or re-point a tag that has already been pushed. If a version has been consumed, bump and tag again.

One-time setup (maintainers)

Publishing is authenticated with PyPI Trusted Publishing (OIDC): GitHub mints a short-lived credential at publish time that PyPI accepts only for this repository, this workflow file and this environment. No API token is created, stored or pasted anywhere. Every value below is a public fact about the repository — none of it is a secret, which is precisely the point.

On PyPI — once per index (repeat on TestPyPI for the rehearsal path):

1. Your account → Publishing → Add a pending publisher → GitHub, and fill in:

   Field	Value
   PyPI project name	forecast-performance
   Owner	FORESIGHT-ULisboa
   Repository	forecast_performance
   Workflow name	release.yml
   Environment	pypi (on TestPyPI: testpypi)

   The environment name must match the environment: key of the corresponding job in release.yml, or PyPI rejects the exchange. A pending publisher is what you register before the project exists on an index; it turns into an ordinary one on the first successful upload.

On GitHub — once:

2. Settings → Environments → create pypi (and testpypi).
3. Add a required reviewer to pypi. Every publish then pauses for an explicit approval, so a tag push alone cannot ship.

Things worth keeping that way:

• Do not add a PyPI API token to repository secrets. Trusted Publishing exists to remove that long-lived credential; a token in secrets is usable by any workflow in the repository and stays valid until someone revokes it. Nothing here needs one.
• The publisher trusts the workflow by filename, so anyone able to change release.yml or push a v* tag can trigger a publish. Protect main and restrict who may push tags; the environment reviewer from step 3 is the backstop.
• Only the two publish jobs request id-token: write; the build job is contents: read and the workflow default is permissions: {}. Keep that split when editing it — the job that produces the artifacts never holds the identity that can upload them.

API reference

ForecastPerformance

Construction & data ingestion

Scoring

Expected value & baselines

Post-processing, cropping & management

Visualisation

Metric handle attributes — every metric is also exposed on the instance/class under its common-usage name, as a passable Metric handle (no import needed): fp.RMSE, fp.MAE, fp.MSE, fp.NSE, fp.KGE, fp.KGEprime, fp.Pearson, fp.Spearman, fp.bias, fp.relative_bias, fp.count, fp.CRPS, fp.fair_CRPS, fp.quantile_loss, fp.reliability, fp.resolution, fp.resolution_relative, fp.brier_score, fp.fair_brier_score, fp.fair_CRPS_skill_score, fp.fair_brier_skill_score.

Deterministic metrics

All accept 1-D array-like arguments (simulations, targets) and return a scalar.

Probabilistic metrics

Applied through fp.probabilistic(...). Applies to shows the simulation types each supports (simple = point, ens = ensemble, prob = quantile).

Pass the required args either via metric_kwargs= on the generic call or as keywords on the accessor method:

fp.probabilistic("brier_score", "ens", leadtime=lt, metric_kwargs={"threshold": 100})
fp.probabilistic.brier_score("ens", leadtime=lt, threshold=100)

fp.probabilistic.fair_crps_skill_score("ens", leadtime=lt, reference="climatology")

Every probabilistic metric also accepts months=[...] to restrict the evaluation to specific calendar months.

Metric, registries and naming

• Each public metric is a Metric (subclass of str): callable, and equal to its own name (str(rmse) == rmse == "rmse", rmse.__name__ == "rmse").
• snake_case is the primary spelling; PascalCase aliases (RMSE, NSE, KGE, KGEprime, …) are retained for backward compatibility.
• DETERMINISTIC_METRICS / PROBABILISTIC_METRICS are dicts mapping every name and alias (case-insensitive) to its Metric; DETERMINISTIC / PROBABILISTIC are the ordered lists. All are importable from forecast_performance.

Results

from forecast_performance import Results, rmse

r = Results("Model", "Metric")
r.append(Model="A", Metric=rmse, Value=0.12)   # Metric stringifies to "rmse"
r.append(Model="B", Metric=rmse, Value=0.08)
df = r.to_pandas(index=["Model"], columns=["Metric"])

Visualisation helpers (forecast_performance.plotly_forecasting)

Plotly helpers that take a go.Figure and a canonical long-format frame:

PandasForecast

A pd.DataFrame subclass whose parquet I/O preserves pd.DateOffset leadtimes (which parquet cannot otherwise serialize). Each DateOffset in a leadtime level — in the index or the columns — is encoded on write as a sentinel JSON string of its .kwds and decoded back on read.

PandasForecast is a pd.DataFrame subclass, so it satisfies isinstance(x, pd.DataFrame) and behaves like a frame everywhere. The subclass type does propagate through operations (slicing, groupby, arithmetic, concat all return PandasForecast); if downstream code relies on type(x) is pd.DataFrame, unpickles without this package installed, or uses assert_frame_equal with the frame as the expected argument, call to_pandas() first.

Multi-keyword offsets (pd.DateOffset(months=1, days=15)) and mixed units across leadtimes round-trip; offsets .kwds cannot capture (anchored MonthEnd, a bare pd.DateOffset(2)) are left to the normal parquet behaviour.

storedResults

Caching decorator used internally on _p_values; results are stored in fp.results[name][func][leadtime] and bypassed when threshold/months is supplied. Clear with fp.clear_cache(...).

License

Licensed under the MIT License — see LICENSE.