awesomepkf 2.11.0

Pairwise Kalman Filter (PKF) and variants (EPKF, UPKF, PPF) and pairwise smoothers for linear and nonlinear state estimation

AwesomePKF

This repository contains a set of programs illustrating the Pairwise Kalman Filter (PKF), a generalization of the classical Kalman Filter, extended to non-linear models. It includes several variants of non-linear filters:

• Extended Pairwise Kalman Filter (EPKF)
• Unscented Pairwise Kalman Filter (UPKF), with multiple variants depending on the choice of sigma points
• Pairwise Particle Filter (PPF)

and pairwise Kalman smoothers for offline post-processing:

• Linear Pairwise Kalman Smoother (PKS) — RTS-style backward pass on the joint (X, Y) Markov chain, with an equivalent DWY (Desai-Weinert-Yusypchuk) backward-filter variant selectable via method="DWY".
• Extended Pairwise Kalman Smoother (EPKS) — same recursion with the per-step Jacobian replacing the constant transition matrix.
• Unscented Pairwise Kalman Smoother (UPKS) — sigma-point cross-covariance in the backward pass; supports the same sigma-point sets as the UPKF (wan2000, cpkf, lerner2002, ito2000).
• Unscented Kalman Smoother (UKS) — classical (non-pairwise) sigma-point smoother for FxHx models with Markov-in-X assumption; gain (dim_x, dim_x).
• Pairwise Particle Smoother (PPS) — FFBSm (Forward Filtering, Backward Smoothing) on top of the PPF; reweights the forward particle cloud via a backward weight recursion. No closed-form gain.

---

Table of Contents

• AwesomePKF
  • Table of Contents
  • Installation
  • Models and Simulations
  • Filters
    • Pairwise Kalman Filter (PKF)
    • Extended Pairwise Kalman Filter (EPKF)
    • Unscented Pairwise Kalman Filter (UPKF)
    • Pairwise Particle Filter (PPF)
  • Smoothers
    • Linear Pairwise Kalman Smoother (PKS)
    • Extended Pairwise Kalman Smoother (EPKS)
    • Unscented Pairwise Kalman Smoother (UPKS)
    • Unscented Kalman Smoother (UKS)
    • Pairwise Particle Smoother (PPS)
  • Tutorials
  • Parameter learning from data
  • Usage Examples
    • Simulate Linear Data and Filter with PKF
    • Simulate Non-Linear Data and Filter with EPKF, UPKF and PPF

---

Installation

From PyPI (recommended)

pip install awesomepkf

From source

git clone https://github.com/sderrode/awesomepkf.git
cd awesomepkf
pip install .

Development install

pip install -e ".[dev]"

Requirements

• Python >= 3.10
• numpy, scipy, matplotlib, pandas, rich, sympy

---

Quick Start

from prg.classes.linear_pkf import Linear_PKF
from prg.models.linear import ModelFactoryLinear

model = ModelFactoryLinear.create("model_x1_y1_AQ_pairwise")
pkf = Linear_PKF(model)
# ... run the filter step by step

Or use the CLI entry points installed with the package:

awesomepkf-simulate --N 2000 --linear-model-name "model_x1_y1_AQ_pairwise" --data-filename "testL.csv" --s-key 303
awesomepkf-pkf      --linear-model-name "model_x1_y1_AQ_pairwise" --data-filename "testL.csv" --plot

---

Tutorials

Interactive Jupyter notebooks are available in the notebooks/ directory:

#	Notebook	Description
01	tutorial_01_getting_started.ipynb	Introduction to the PKF framework: linear models, running the filter, visualizing estimates, error metrics (MSE, NEES, NIS), comparing PKF / EPKF / UPKF
02	tutorial_02_nonlinear_models.ipynb	Nonlinear models: EPKF, UPKF, PPF and PF — classic vs pairwise, sigma-point sets, particle count impact, filter comparison
03	tutorial_03_sigma_points.ipynb	Sigma-point sets for the UPKF: wan2000, cpkf, lerner2002, ito2000 — impact on estimation accuracy
04	tutorial_04_particle_filters.ipynb	Particle filters (PPF and PF): tuning the number of particles, resampling, comparison with EPKF/UPKF
05	tutorial_05_new_model_lotkavolterra.ipynb	How to add a new nonlinear pairwise model: Lotka-Volterra prey-predator (dim_x=1, dim_y=1), augmented version, filtering with EPKF/UPKF/PPF
06	tutorial_06_filter_runner_and_config.ipynb	High-level orchestration with FilterRunner and RunOptions; parameter sweeps via model_kwargs; saving and replaying experiments through a reproducible JSON spec
07	tutorial_07_smoothers.ipynb	The 5 smoothers (Linear_PKS, NonLinear_EPKS/UPKS/UKS/PPS) — RTS-style backward recursion and FFBSm; ±2σ envelope shrinkage, Joseph form, Monte-Carlo convergence of PPS to PKS, decision rule for choosing among the five
08	tutorial_08_real_data_pkf_learning.ipynb	Estimating the 1D linear PMM parameters (a, b, c, d, e) from a real two-column time series (wind-farm active power vs wind speed); PMM vs HMM/Kalman projection; converting to LinearAmQ kwargs

---

Parameter learning from data

For the linear, scalar case (dim_x = dim_y = 1), the prg.learning module estimates the five PMM parameters (a, b, c, d, e) from a two-column time series by the method of moments.

awesomepkf-fit-pkf \
    --data-filename data/samples/windfarms/site1_202210_Month_586_norm.csv \
    --x-col ActivePower_KWh --y-col WindSpeed \
    --output learned_params.npz --verbose 1

On the embedded WindFarms series the fit lands clearly outside the HMM (classical Kalman) submanifold — the off-HMM gap Δc ≈ 0.53 between the estimated c and its HMM projection a·b² means a pairwise model tracks the state more tightly than the classical KF. See tutorial_08 for the full load → fit → compare → convert workflow.

A small WindFarms sample is shipped under data/samples/; the full dataset (BuildingTemp, SeattleTemp, multiple WindFarms sites and granularities) is kept outside the repository — point --data-filename at a local copy if needed.

Parameter identification for nonlinear EPKF/UPKF models is not covered by this estimator — those require a separate procedure (e.g. a neural network).

---

Models and Simulations

The repository provides a program called run_simulator.py to simulate data according to linear and non-linear models.

---

Filters

Each filter has two types of programs:

1. Simulate data and filter it directly
2. Filter data from a previously saved file

Pairwise Kalman Filter (PKF)

• run_linear_pkf.py – filter linear data either from simulated data or from a previously saved file (e.g., generated with run_simulator.py)

Extended Pairwise Kalman Filter (EPKF)

• run_nonlinear_epkf.py – filter non-linear data either from simulated data or from a previously saved file (e.g., generated with run_simulator.py)

Unscented Pairwise Kalman Filter (UPKF)

• run_nonlinear_upkf.py – filter non-linear data either from simulated data or from a previously saved file (e.g., generated with run_simulator.py)

Pairwise Particle Filter (PPF)

• run_nonlinear_ppf.py – filter non-linear data either from simulated data or from a previously saved file (e.g., generated with run_simulator.py)

---

Smoothers

Smoothers are two-pass, offline estimators that condition on the entire observation sequence y_{1:N}. They produce posterior means and covariances p(X_n | y_{1:N}) that are at least as good (in PSD sense) as the corresponding forward filter outputs p(X_n | y_{1:n}).

Linear Pairwise Kalman Smoother (PKS)

The linear PKS runs the PKF forward, then a backward Rauch-Tung-Striebel recursion at the joint (X, Y) level. The pairwise model is Markov in Z = (X, Y) (not in X alone), so the smoothing gain G_n has shape (dim_x, dim_x + dim_y). Equivalently, the linear PKS is the classical RTS smoother applied to the augmented state Z' = (X, Y) with degenerate observation Y_n = (0, I) Z'_n (R^aug = 0).

Linear_PKS is a façade that selects the backward pass via method= (default "RTS"). method="DWY" runs the Desai-Weinert-Yusypchuk backward-filter recursion on the time-reversed complementary couple model (cf. Geng et al., 2023); on the linear-Gaussian model it returns the same smoothed mean and covariance as RTS to machine precision (verified by test_dwy_equals_rts). The explicit variant classes Linear_PKS_RTS and Linear_PKS_DWY are also exported (Linear_PKS_<NAME>).

from prg.classes.linear_pks import Linear_PKS
from prg.classes.param_linear import ParamLinear
from prg.models.linear import ModelFactoryLinear

model  = ModelFactoryLinear.create("model_x1_y1_AQ_pairwise")
params = model.get_params().copy()
dim_x  = params.pop("dim_x");  dim_y = params.pop("dim_y")
param  = ParamLinear(0, dim_x, dim_y, **params)

pks = Linear_PKS(param, sKey=42, joseph=False)
# joseph=True selects the explicitly PSD-preserving Joseph form
# (mathematically equivalent at the optimal gain, useful for ill-conditioned cases).

results = pks.process_N_data_smoother(N=500)
# each tuple: (k, x_true, y_obs, X_predict, X_update, X_smooth)

Implementation: prg/classes/linear_pks.py. Tests: prg/tests/test_linear_pks.py (44 tests, including PSD shrinkage, Joseph equivalence, augmented-state RTS equivalence, DWY≡RTS equivalence, and full exception/logging coverage).

Extended Pairwise Kalman Smoother (EPKS)

The EPKS extends the linear PKS to non-linear pairwise models via first-order linearisation. The forward pass is the standard EPKF; the backward pass uses the per-step Jacobian F_{n+1} (evaluated at the filtered state) in place of the constant A matrix. PSD shrinkage of the linearised covariance is preserved; on average the MSE also shrinks, but the linearisation bias breaks the per-trajectory guarantee of the linear case.

from prg.classes.nonlinear_epks import NonLinear_EPKS
from prg.classes.param_nonlinear import ParamNonLinear
from prg.models.nonlinear import ModelFactoryNonLinear

model  = ModelFactoryNonLinear.create("model_x2_y1_pairwise")
params = model.get_params().copy()
dim_x  = params.pop("dim_x");  dim_y = params.pop("dim_y")
param  = ParamNonLinear(0, dim_x, dim_y, **params)

epks = NonLinear_EPKS(param, sKey=42, joseph=False)
results = epks.process_N_data_smoother(N=300)

Implementation: prg/classes/nonlinear_epks.py. Tests: prg/tests/test_nonlinear_epks.py (25 tests). Note: not suitable for augmented models (rank-deficient predicted covariance fails the backward Cholesky).

Unscented Pairwise Kalman Smoother (UPKS)

The UPKS replaces the first-order linearisation of the EPKS with sigma-point propagation: the cross-covariance Cov(X_n, Z_{n+1} | y_{1:n}) and the predicted joint covariance P^{ZZ}_{n+1|n} are estimated from sigma points regenerated at the filtered state. Supports the same sigma-point sets as the UPKF (wan2000, cpkf, lerner2002, ito2000), selected via the sigmaSet constructor argument.

from prg.classes.nonlinear_upks import NonLinear_UPKS
from prg.classes.param_nonlinear import ParamNonLinear
from prg.models.nonlinear import ModelFactoryNonLinear

model  = ModelFactoryNonLinear.create("model_x2_y1_pairwise")
params = model.get_params().copy()
dim_x  = params.pop("dim_x");  dim_y = params.pop("dim_y")
param  = ParamNonLinear(0, dim_x, dim_y, **params)

upks = NonLinear_UPKS(param, sigmaSet="wan2000", sKey=42, joseph=False)
results = upks.process_N_data_smoother(N=300)

Implementation: prg/classes/nonlinear_upks.py. Tests: prg/tests/test_nonlinear_upks.py (29 tests, parametrised over all sigma-point sets). Same caveats as the EPKS regarding augmented models.

Unscented Kalman Smoother (UKS)

The UKS is the classical (non-pairwise) sigma-point smoother — equivalent to the UPKS when applied to a model that is Markov in X alone (FxHx structure). The gain is (dim_x, dim_x) (vs (dim_x, dim_x + dim_y) for the pairwise variants), and the sigma-point dimension is dim_x only. The smoother refuses pairwise models at construction time (FilterError).

from prg.classes.nonlinear_uks import NonLinear_UKS
from prg.classes.param_nonlinear import ParamNonLinear
from prg.models.nonlinear import ModelFactoryNonLinear

model  = ModelFactoryNonLinear.create("model_x1_y1_Sinus_classic")
params = model.get_params().copy()
dim_x  = params.pop("dim_x");  dim_y = params.pop("dim_y")
param  = ParamNonLinear(0, dim_x, dim_y, **params)

uks = NonLinear_UKS(param, sigmaSet="wan2000", sKey=42, joseph=False)
results = uks.process_N_data_smoother(N=300)

Implementation: prg/classes/nonlinear_uks.py. Tests: prg/tests/test_nonlinear_uks.py (29 tests).

Pairwise Particle Smoother (PPS)

The PPS implements FFBSm (Forward Filtering, Backward Smoothing) on top of the PPF. The forward pass runs the standard PPF with particle clouds stored at every step; the backward pass reweights those forward particles via:

ŵ_{i,n} = w_{i,n} · Σ_j ŵ_{j,n+1} · p(ξ_{j,n+1} | ξ_{i,n}, y_n) / Σ_l w_{l,n}·p(ξ_{j,n+1} | ξ_{l,n}, y_n)

The smoothed mean and covariance are weighted statistics of the forward particle cloud with the smoothed weights. Complexity is O(N·n_p²) per smoother run (vs O(N·n_p) for the forward).

from prg.classes.nonlinear_pps import NonLinear_PPS
from prg.classes.param_nonlinear import ParamNonLinear
from prg.models.nonlinear import ModelFactoryNonLinear

model  = ModelFactoryNonLinear.create("model_x2_y1_pairwise")
params = model.get_params().copy()
dim_x  = params.pop("dim_x");  dim_y = params.pop("dim_y")
param  = ParamNonLinear(0, dim_x, dim_y, **params)

pps = NonLinear_PPS(param, n_particles=300, sKey=42)
results = pps.process_N_data_smoother(N=200)

Implementation: prg/classes/nonlinear_pps.py. Tests: prg/tests/test_nonlinear_pps.py (19 tests, including a Monte-Carlo convergence test that verifies the PPS converges to the exact Linear_PKS as n_particles grows on a linear-Gaussian pairwise model).

---

Paper Reproducibility Scripts

The following scripts reproduce all figures and tables from the article "Non-linear extensions to Gaussian pairwise Kalman filter". Each script can be run independently from the repository root.

Section 4 — Simulation Results

Script	Figures generated
run_paper_section4.py	epkf_observations_x1_y1_Retroactions.png, epkf_x1_y1_Retroactions.png, upkf_x1_y1_Retroactions.png, ppf_x1_y1_Retroactions.png + Tables 1 & 2
run_paper_section4_backaction.py	backaction_mse_nees_vs_b.png
run_paper_section4_multip.py	multip_mse_nees_vs_sigma.png
run_paper_section4_sensitivity.py	console output — mean ± std of MSE over 30 seeds

python3 -m prg.run_paper_section4
python3 -m prg.run_paper_section4_backaction
python3 -m prg.run_paper_section4_multip
python3 -m prg.run_paper_section4_sensitivity

Section 5 — Real Data Experiment (S&P 500 Stochastic Volatility)

Script	Figures generated
run_paper_section5.py	nn_gx_gy_sv.png, epkf_sv.png, upkf_sv.png, ppf_sv.png
run_paper_section5_enso.py	archived ENSO experiment (Niño 3.4 / SOI), kept for reference

python3 -m prg.run_paper_section5       # requires: pip install yfinance
python3 -m prg.run_paper_section5_enso  # archived version

Note: all figures are saved in papier_NonLinearPKF/figures/.

---

Usage Examples

Simulate Linear Data and Filter with PKF

awesomepkf-simulate --N 2000 --linear-model-name "model_x1_y1_AQ_pairwise" --data-filename "testL.csv" --verbose 1 --s-key 303
awesomepkf-pkf      --linear-model-name "model_x1_y1_AQ_pairwise" --data-filename "testL.csv" --verbose 1 --save-history --plot

Simulate Non-Linear Data and Filter with EPKF, UPKF and PPF

awesomepkf-simulate --N 1000 --nonlinear-model-name "model_x2_y1_pairwise" --data-filename "testNL.csv" --verbose 1 --s-key 303

awesomepkf-epkf --nonlinear-model-name "model_x2_y1_pairwise" --data-filename "testNL.csv"                      --verbose 1 --save-history --plot
awesomepkf-upkf --nonlinear-model-name "model_x2_y1_pairwise" --data-filename "testNL.csv" --sigma-set "wan2000"  --verbose 1 --save-history --plot
awesomepkf-ppf  --nonlinear-model-name "model_x2_y1_pairwise" --data-filename "testNL.csv" --n-particles 300      --verbose 1 --save-history --plot