vartests 0.3.0

vartests is a Python library with statistical tests to backtest Value at Risk (VaR) models and other quantile-based risk measures (e.g. Drawdown at Risk). Coverage tests only need the series of violations and the probability of violation, so they work for any risk measure that is a quantile.

Installation

pip install vartests

Or the latest version from GitHub:

pip install https://github.com/rafa-rod/vartests/archive/refs/heads/main.zip

Requires Python 3.10 or later.

Results

Every test returns a dictionary with the same main keys:

Some tests add specific keys, such as critical value (chi-square tests), non-rejection region and size (exact tests) or the estimated parameters.

Example

The file media/Example.xlsx has the PnL of a portfolio, its VaR at 99% and the violations (1 when the loss exceeds the VaR). Reading Excel files requires openpyxl:

import pandas as pd
import vartests

data = pd.read_excel("media/Example.xlsx", index_col=0)
violations = data["Violations"]
pnl = data["PnL"]

All functions accept a list, NumPy array, Series or single-column DataFrame.

Coverage

vartests.failure_rate(violations)

vartests.kupiec_test(violations, var_conf_level=0.99, conf_level=0.95)

vartests.binomial_test(violations, var_conf_level=0.99, conf_level=0.95, alternative="greater")
vartests.mid_p_test(violations, var_conf_level=0.99, conf_level=0.95, alternative="greater")

alternative="greater" (default of the exact tests) tests underestimation of risk, i.e. too many violations. Use "two-sided" to also detect a conservative model, or "less" for that alone.

With zero violations, the Kupiec and binomial tests do not reject automatically: the decision depends on the number of observations and on the nominal rate. For example, at a 5% significance level, Kupiec does not reject 0 to 1 violations in 250 days for a 99.9% VaR, but requires 1 to 6 for a 99% VaR (Giambiagi, ch. 10, Table 10.1).

Because the binomial distribution is discrete, the exact test rejects a correct model less often than the nominal level. The size key reports the actual probability; mid_p_test brings it closer to the nominal level, with no guarantee of staying below it. A common practice is to decide with the exact test and report the mid-p as a complement.

A non-rejection is only informative if the test has power. binomial_power shows the chance of detecting a wrong model:

vartests.binomial_power(n=60, var_conf_level=0.975, true_failure_rate=[0.05, 0.075, 0.10])
# {'non-rejection region': (0, 4), 'size': 0.017, 'power': {0.05: 0.18, 0.075: 0.47, 0.1: 0.73}}

When the probability of violation differs across observations (a VaR with a varying level, or the expected coverage of an estimator at each episode of a Drawdown at Risk), use the Poisson-binomial test with one probability per observation:

vartests.poisson_binomial_test(violations, probabilities, conf_level=0.95, alternative="greater")

Independence

vartests.duration_test(violations, conf_level=0.95)

Under H0 the time between violations is exponential (Weibull shape b = 1); b < 1 indicates clusters. At least two durations, one of them uncensored, are required; otherwise the decision is "Not enough data".

Distribution

The Berkowitz test needs the probability integral transform (PIT) of each realized PnL under the distribution forecast by your model, u_t = F_t(PnL_t):

vartests.berkowitz_tail_test(pit, var_conf_level=0.99, conf_level=0.95)

If your VaR model is a GARCH(1,1) with normal innovations, garch_pit computes the PIT by re-estimating the model on a rolling window, with no look-ahead:

pit = vartests.garch_pit(pnl, volatility_window=252)
vartests.berkowitz_tail_test(pit, var_conf_level=0.99, conf_level=0.95)

Parametric models such as EWMA and GARCH often assume zero mean:

vartests.zero_mean_test(pnl, conf_level=0.95)

Changes in 0.3.0

Version 0.3.0 fixes errors and changes the output of some functions:

• kupiec_test no longer rejects automatically with zero violations and returns the p-value.
• duration_test fixes the likelihood when the series does not end with a violation (the usual case); p-values change in most samples.
• zero_mean_test works with pandas 2 and 3 and accepts Series.
• berkowitz_tail_test receives the PIT of the model being tested. The old berkowtiz_tail_test(pnl, ...) still works, with a deprecation warning, and is equivalent to berkowitz_tail_test(garch_pit(pnl), ...); results differ slightly because the conditional mean is now forecast without look-ahead.
• Result keys are standardized (statistic, p-value, decision).
• New tests: binomial_test, mid_p_test, poisson_binomial_test and binomial_power.

References

• Berkowitz, J. (2001). Testing density forecasts, with applications to risk management. Journal of Business & Economic Statistics, 19(4), 465-474.
• Christoffersen, P. and Pelletier, D. (2004). Backtesting Value-at-Risk: a duration-based approach. Journal of Financial Econometrics, 2(1), 84-108.
• Giambiagi, F. (Org.). Derivativos e risco de mercado. Elsevier. Ch. 10 (Backtesting).
• Kupiec, P. (1995). Techniques for verifying the accuracy of risk measurement models. Journal of Derivatives, 3(2), 73-84.
• Lancaster, H. O. (1961). Significance tests in discrete distributions. Journal of the American Statistical Association, 56(294), 223-234.