ActEval

Evaluate actuarial predictive models beyond predictive accuracy.

ActEval is a model-agnostic Python framework for non-life insurance model evaluation. It accepts prediction arrays instead of fitted model objects, so it works after GLMs, scikit-learn pipelines, XGBoost, CatBoost, or neural networks.

ActEval reports accuracy, calibration, discrimination, and observed-tail behavior separately. It does not create an arbitrary overall score or claim that one model is universally best.

Version 0.2 also evaluates full predictive distributions using proper scoring rules and keeps uncertainty diagnostics separate from model-quality claims. Version 0.3 adds explicit, benchmarked financial decision diagnostics without collapsing pricing, reserving, capital, and reinsurance into one score.

Installation

Install the package from PyPI:

python -m pip install acteval-insurance

For plots:

python -m pip install "acteval-insurance[plot]"

The distribution is named acteval-insurance because acteval is occupied by an unrelated project on PyPI. The import name remains acteval.

Quick start

import acteval as ae

result = ae.evaluate(
    y_true=[0.0, 0.4, 1.0, 2.0, 4.0, 7.0],
    y_pred=[0.1, 0.5, 0.9, 1.8, 3.6, 6.4],
    exposure=[1.0, 0.5, 1.2, 0.8, 1.5, 2.0],
    task="claim_frequency",
    metrics=["rmse", "poisson_deviance", "ae_ratio", "normalized_gini"],
)

print(result.summary())

Task defaults provide a broader report, including 95% observed-tail metrics.

Model comparison

comparison = ae.compare(
    y_true=y,
    predictions={
        "GLM": glm_predictions,
        "CatBoost": catboost_predictions,
        "XGBoost": xgb_predictions,
    },
    exposure=exposure,
    task="claim_frequency",
)

print(comparison.to_dataframe())
print(comparison.rank(metric="poisson_deviance"))

rank() uses a metric's documented direction. Target metrics such as A/E are ranked by distance from 1. Rankings remain metric-specific.

Accuracy can disagree with tail calibration

The example below deliberately creates two models:

• Model A makes moderate errors on many ordinary risks but predicts large observed outcomes accurately.
• Model B improves ordinary-risk predictions and overall RMSE while underpredicting the observed tail.

import numpy as np
import acteval as ae

y = np.r_[np.tile([0.5, 1.0, 1.5, 1.0, 0.5], 19), np.repeat(10.0, 5)]
model_a = np.r_[y[:95] + 0.5, np.repeat(10.0, 5)]
model_b = np.r_[y[:95], np.repeat(9.0, 5)]

tradeoff = ae.compare(
    y,
    {"Model A": model_a, "Model B": model_b},
    task="claim_frequency",
    metrics=["rmse", "poisson_deviance", "tail_ae_95"],
)
print(tradeoff.to_dataframe())

Model B has lower overall RMSE and deviance, while Model A has tail A/E equal to 1. The appropriate choice depends on the actuarial objective.

Input and exposure contract

y_true and y_pred must be finite, one-dimensional, nonnegative arrays on the same scale.

• For claim frequency, use frequency rates for both arrays and provide policy exposure as exposure.
• For pure premium, use pure-premium rates for both arrays and provide exposure when portfolio-volume weighting is desired.
• For severity, use claim severities. Exposure is optional and usually unnecessary; claim-level sample_weight is normally more meaningful.
• If both are supplied, effective weight is sample_weight * exposure.

ActEval does not silently convert raw claim counts into rates.

Parameterized metrics

Use MetricSpec whenever a parameter should be explicit and reproducible:

result = ae.evaluate(
    y,
    predictions,
    task="pure_premium",
    metrics=[
        ae.MetricSpec("tweedie_deviance", {"power": 1.7}),
        ae.MetricSpec("tail_mae", {"quantile": 0.99}, label="tail_mae_99"),
    ],
)

Tail aliases such as tail_mae_95, tail_rmse_99, and tail_ae_95 are also accepted. Parameter values are retained in result metadata.

Calibration, discrimination, and tail diagnostics

calibration = ae.calibration_by_quantile(y, predictions, n_bins=10)
lift = ae.lift_by_quantile(y, predictions, n_bins=10)

print(calibration.to_dataframe())
print(lift.to_dataframe())

ae.plot_calibration(y, predictions)
ae.plot_lift(y, predictions)
ae.plot_residuals(y, predictions)
ae.plot_tail_diagnostics(y, predictions, quantile=0.95)

Predictive distributions

Built-in vectorized adapters provide one predictive distribution per observation:

• PoissonDistribution(mu);
• NegativeBinomialDistribution(mean, dispersion);
• GammaDistribution(mean, shape);
• LognormalDistribution(meanlog, sdlog);
• EmpiricalDistribution(samples);
• TweedieDistribution(mean, power, dispersion) for compound Poisson-Gamma 1 < power < 2.

poisson = ae.PoissonDistribution(mu=poisson_means)
negative_binomial = ae.NegativeBinomialDistribution(
    mean=nb_means,
    dispersion=nb_dispersion,
)

distribution_comparison = ae.compare_distributions(
    y_true=claim_counts,
    distributions={
        "Poisson": poisson,
        "Negative Binomial": negative_binomial,
    },
    exposure=exposure,
    task="claim_frequency",
    metrics=[
        ae.MetricSpec("crps", {"n_samples": 5000, "random_state": 42}),
        "log_score",
        ae.MetricSpec("brier_score", {"threshold": 0}),
        ae.MetricSpec("interval_score", {"coverage": 0.9}),
    ],
)

print(distribution_comparison.to_dataframe())

Samples have shape (n_samples, n_observations). Scalar quantiles have shape (n_observations,); vector quantiles have shape (n_quantiles, n_observations). CRPS randomness is explicitly seeded and recorded in result metadata.

Tweedie sampling uses the exact compound representation. CDF and log-density evaluation use a numerical series implementation; quantiles use deterministic Monte Carlo. Entropy is a seeded Monte Carlo estimate of -E[log_prob(X)] and is only comparable under the same mixed distribution measure. Empirical draws are treated as a discrete distribution: repeated values determine probability mass, and unseen values have log probability -inf.

Decision-aware evaluation

Decision functions always expose their financial loss and benchmark. Regret is model financial loss - benchmark financial loss in the loss function's unit. It may be negative when the model decision outperforms the benchmark. Relative regret is omitted when benchmark loss is zero.

premiums = ae.premium_from_distribution(
    severity_distribution,
    profit_loading=0.08,
    expense_ratio=0.20,
)

pricing = ae.pricing_regret(
    y_true=realized_loss,
    premium=premiums,
    benchmark_premium=current_tariff,
    underpricing_cost=2.0,
    overpricing_cost=1.0,
    benchmark_name="current tariff",
)

loss_ratio = ae.loss_ratio_impact(
    realized_loss,
    premiums,
    target_loss_ratio=0.70,
)

reserve = ae.reserve_shortfall(realized_loss, held_reserve)
capital = ae.capital_shortfall(realized_loss, available_capital)

Stop-loss reinsurance selection compares quoted options under one explicit rule: premium plus expected retained aggregate loss plus a user-selected cost of VaR or expected-shortfall capital.

options = [
    ae.ReinsuranceOption("No cover", retention=1_000_000, premium=0),
    ae.ReinsuranceOption("100k retention", retention=100_000, premium=25_000),
]

selection = ae.select_reinsurance_option(
    aggregate_loss_distribution,
    options,
    risk_measure="expected_shortfall",
    risk_quantile=0.995,
    capital_cost_rate=0.10,
    random_state=42,
)

realized = ae.reinsurance_decision_regret(
    aggregate_loss=realized_annual_losses,
    selected=selection.selected,
    benchmark=options[0],
)

For reinsurance selection, each sampled row is a scenario and columns are summed into portfolio aggregate loss. Dependence must therefore already be represented by the supplied distribution's joint samples. Built-in parametric adapters sample observation columns independently; use EmpiricalDistribution with joint scenario draws when portfolio dependence matters.

Supported MVP metrics

Use ae.list_metrics() for machine-readable registry metadata. Exact formulas and limitations are in the metric reference.

Development

git clone https://github.com/aminemanai2003/acteval.git
cd acteval
python -m venv .venv
python -m pip install -e ".[dev]"
ruff check .
mypy src/acteval
pytest
python -m build

See CONTRIBUTING.md and the implementation audit.

Implemented releases

• v0.1: point-prediction accuracy, calibration, discrimination, tail diagnostics, comparisons, and plotting.
• v0.2: predictive-distribution scores and uncertainty diagnostics.
• v0.3: explicit benchmarked pricing, loss-ratio, reserve, capital, and reinsurance financial consequences.

The original v0.1-v0.3 implementation plan is complete. Remaining work is release operations and future scope, not missing behavior from that plan. See the completion audit for boundaries and evidence.

License

Apache-2.0.