vartests 0.4.0

vartests is a Python library with statistical tests to backtest Value at Risk (VaR) models and other quantile-based risk measures (e.g. Drawdown at Risk). Coverage tests only need the series of violations and the probability of violation, so they work for any risk measure that is a quantile.

Installation

pip install vartests

Or the latest version from GitHub:

pip install https://github.com/rafa-rod/vartests/archive/refs/heads/main.zip

Requires Python 3.10 or later.

Results

Every test returns a dictionary with the same main keys:

Some tests add specific keys, such as critical value (chi-square tests), non-rejection region and size (exact tests) or the estimated parameters.

Example

The file media/Example.xlsx has the PnL of a portfolio, its VaR at 99% and the violations (1 when the loss exceeds the VaR). Reading Excel files requires openpyxl:

import pandas as pd
import vartests

data = pd.read_excel("media/Example.xlsx", index_col=0)
violations = data["Violations"]
pnl = data["PnL"]

All functions accept a list, NumPy array, Series or single-column DataFrame.

Coverage

vartests.failure_rate(violations)

vartests.kupiec_test(violations, var_conf_level=0.99, conf_level=0.95)

vartests.binomial_test(violations, var_conf_level=0.99, conf_level=0.95, alternative="greater")
vartests.mid_p_test(violations, var_conf_level=0.99, conf_level=0.95, alternative="greater")

alternative="greater" (default of the exact tests) tests underestimation of risk, i.e. too many violations. Use "two-sided" to also detect a conservative model, or "less" for that alone.

With zero violations, the Kupiec and binomial tests do not reject automatically: the decision depends on the number of observations and on the nominal rate. For example, at a 5% significance level, Kupiec does not reject 0 to 1 violations in 250 days for a 99.9% VaR, but requires 1 to 6 for a 99% VaR (Giambiagi, ch. 10, Table 10.1).

Because the binomial distribution is discrete, the exact test rejects a correct model less often than the nominal level. The size key reports the actual probability; mid_p_test brings it closer to the nominal level, with no guarantee of staying below it. A common practice is to decide with the exact test and report the mid-p as a complement.

A non-rejection is only informative if the test has power. binomial_power shows the chance of detecting a wrong model:

vartests.binomial_power(n=60, var_conf_level=0.975, true_failure_rate=[0.05, 0.075, 0.10])
# {'non-rejection region': (0, 4), 'size': 0.017, 'power': {0.05: 0.18, 0.075: 0.47, 0.1: 0.73}}

When the probability of violation differs across observations (a VaR with a varying level, or the expected coverage of an estimator at each episode of a Drawdown at Risk), use the Poisson-binomial test with one probability per observation:

vartests.poisson_binomial_test(violations, probabilities, conf_level=0.95, alternative="greater")

Independence

vartests.duration_test(violations, conf_level=0.95)

Under H0 the time between violations is exponential (Weibull shape b = 1); b < 1 indicates clusters. At least two durations, one of them uncensored, are required; otherwise the decision is "Not enough data".

Distribution

The Berkowitz test checks whether losses beyond the VaR have the size a normal VaR model predicts, not only how often they happen. It takes the realized PnL and the VaR forecast for each day, in the same unit:

vartests.berkowitz_tail_test(pnl, var, var_conf_level=0.99, conf_level=0.95)

The test assumes a normal VaR with zero mean, so the forecast standard deviation is |VaR_t| / z, where z is the normal quantile of var_conf_level. The VaR can be reported as a positive loss or a negative return, as long as all values have the same sign. In the example file, the test rejects H0: the largest loss is about 3.3 times the VaR, much more than a normal tail allows.

Parametric models such as EWMA and GARCH often assume zero mean:

vartests.zero_mean_test(pnl, conf_level=0.95)

Changes in 0.4.0

• berkowitz_tail_test(pnl, var, ...) now takes the realized PnL and the VaR series, instead of the PIT, and assumes a normal VaR with zero mean.
• berkowtiz_tail_test (misspelled, deprecated in 0.3.0) was removed.

Changes in 0.3.0

Version 0.3.0 fixes errors and changes the output of some functions:

• kupiec_test no longer rejects automatically with zero violations and returns the p-value.
• duration_test fixes the likelihood when the series does not end with a violation (the usual case); p-values change in most samples.
• zero_mean_test works with pandas 2 and 3 and accepts Series.
• berkowitz_tail_test receives the PIT of the model being tested. Results differ slightly from 0.2.x because the conditional mean is now forecast without look-ahead.
• Result keys are standardized (statistic, p-value, decision).
• New tests: binomial_test, mid_p_test, poisson_binomial_test and binomial_power.

References

• Berkowitz, J. (2001). Testing density forecasts, with applications to risk management. Journal of Business & Economic Statistics, 19(4), 465-474.
• Christoffersen, P. and Pelletier, D. (2004). Backtesting Value-at-Risk: a duration-based approach. Journal of Financial Econometrics, 2(1), 84-108.
• Giambiagi, F. (Org.). Derivativos e risco de mercado. Elsevier. Ch. 10 (Backtesting).
• Kupiec, P. (1995). Techniques for verifying the accuracy of risk measurement models. Journal of Derivatives, 3(2), 73-84.
• Lancaster, H. O. (1961). Significance tests in discrete distributions. Journal of the American Statistical Association, 56(294), 223-234.