factor-qc

A fail-closed quality gate for backtests: one numpy-only engine covering Deflated Sharpe Ratio, Probability of Backtest Overfitting (CSCV), the Harvey-Liu multiple-testing haircut and Minimum Track Record Length 鈥?graded P0/P1/P2, and it refuses to judge a backtest that will not declare how many configurations were tried. Python 3.11+, one dependency (numpy), Windows / Linux / macOS.

Status: v0.1 鈥?alpha. The statistics are battle-tested inside a production research pipeline and validated against published reference values, but this standalone package is new: expect the CLI to shift before v1.0.

Why this exists

The standard story: you try 200 factor configurations, the best one shows a Sharpe of 1.65, you feel great. The honest story: with 200 trials of pure noise, someone is going to show a Sharpe of 1.65 鈥?the expected maximum of 200 zero-true-SR trials 鈥?and it will not be your skill, it will be your selection bias.

Most backtest tooling computes statistics and prints reports. factor-qc is a gate: it decides, with graded severity, whether a candidate may pass 鈥?and its default answer is no:

• P0 (fatal) 鈥?DSR below threshold, PBO above threshold, haircut Sharpe below floor, MinTRL longer than the sample 鈫?the candidate must not pass.
• P1 (warning) 鈥?weak PSR vs zero, short sample, aggressive trial count 鈫?proceed with eyes open.
• P2 (info) 鈥?non-normal moments, tiny trial count 鈫?recorded, no action.

Philosophy

Honesty is the default; the gate is fail-closed.

The one non-negotiable input is n_trials: the honest count of configurations you tried. Without it there is no deflation benchmark (Bailey & L贸pez de Prado 2014), no haircut (Harvey, Liu & Zhu 2016, RFS), no track-record floor (Bailey & L贸pez de Prado 2018, JPM) and no overfitting probability (Bailey, Borwein, L贸pez de Prado & Zhu 2017, JCF). Refuse to declare, and the gate refuses to judge 鈥?that asymmetry is the point. qc check exits non-zero on any P0 failure, so it drops into CI, pre-commit hooks and research gates as a hard blocker, not a suggestion.

Two design commitments that keep it honest:

1. numpy only, no scipy 鈥?the standard-normal inverse CDF is Acklam's rational approximation with one Newton refinement; every number in the report is reproducible from the code in this repo, no hidden black box.
2. PBO is optional but explicit 鈥?without a trials matrix the gate says PBO was not computed, and the DSR cross-trial variance degrades to a conservative single-trial estimate. Absence of evidence is reported as absence, never as evidence.

Quick start

# install from PyPI (once published)
pip install factor-qc

# or run without installing anything:
#   PYTHONPATH=src python -m factor_qc --help

python examples/demo.py   # try it on reproducible synthetic cases

Your own backtest:

# returns.json = JSON list of per-period returns of the selected candidate
# trials.json   = JSON 2D matrix (T x N) of every configuration you tried

qc check --returns returns.json --trials trials.json --n-trials 200
# -> FAIL - P0 blocker(s): dsr: 0.63 vs 0.95; mintrl: 250.8 vs <= 1000; ...

qc check --returns returns.json --n-trials 5 --json   # machine-readable

Exit codes: 0 = no P0 failures (P1/P2 may still be failing), 1 = at least one P0 failure (or missing n_trials), 2 = usage error. Wire it into CI as a hard gate.

Commands

Flags: --returns (required), --trials (optional), --n-trials (required unless require_declared_trials is disabled in code), --periods-per-year (default 252), --n-blocks (CSCV granularity, default 16).

The checks

The P0 set mirrors the spirit of Harvey, Liu & Zhu (2016), "鈥nd the Cross-Section of Expected Returns": a factor must survive multiple-testing correction to earn the right to be called a factor. The gate is the machine version of that editorial stance.

Performance note

CSCV enumerates C(n_blocks, n_blocks/2) splits 鈥?12,870 at the default 16. On large trial matrices (T=1000, N=200) that takes minutes; use --n-blocks 8 (70 splits) or 10 (252 splits) for interactive speed at slightly coarser granularity.

Development

python -m pip install -e . pytest
python -m pytest

CI runs the full test suite on Ubuntu, Windows and macOS with Python 3.11 and 3.12. Issues are handled on weekends; pull requests are welcome.

Related work

• Bailey & L贸pez de Prado (2014), The Deflated Sharpe Ratio
• Bailey, Borwein, L贸pez de Prado & Zhu (2017), The Probability of Backtest Overfitting
• Harvey, Liu & Zhu (2016), 鈥nd the Cross-Section of Expected Returns (RFS)
• Harvey & Liu (2021), Lucky Factors (JFE)
• Mobarekeh & L贸pez de Prado (2024), Backtest Overfitting in the Machine Learning Era (SSRN 4778909) 鈥?why OOS methods still need honest trial accounting

Project family

Part of Foolproof Labs — a toolchain against self-deception in quantitative research:

• pit-adjuster — PIT back-adjustment with static forward-adjustment drift detection
• falsification-ledger — pre-registration and falsification ledger
• factor-qc — fail-closed backtest quality gate
• lesson-book — tuition memory for traders
• lookahead-free — verifiable look-ahead-freedom checks
• ashare-data-immunity — data immunity for A-share daily bars

License

MIT