factor-qc
中文说明
factor-qc 是一个面向量化回测的质量闸门,也适用于 A 股因子研究。
它把 DSR、PBO、Harvey-Liu 多重检验折减和 MinTRL 放在同一次检查中,
并要求明确填写尝试过的参数组合数量。数据不完整或指标未达到门槛时,
工具会给出分级结果并默认不放行;它不保证策略盈利,也不会替研究者选择因子。
A fail-closed quality gate for backtests: one numpy-only engine covering
Deflated Sharpe Ratio, Probability of Backtest Overfitting (CSCV), the
Harvey-Liu multiple-testing haircut and Minimum Track Record Length —graded P0/P1/P2, and it refuses to judge a backtest that will not declare
how many configurations were tried. Python 3.11+, one dependency
(numpy), Windows / Linux / macOS.
Status: v0.1.3 alpha, published on PyPI. The statistics are battle-tested inside a production research pipeline and validated against published reference values, but this standalone package is new: expect the CLI to shift before v1.0.
Why this exists
The standard story: you try 200 factor configurations, the best one shows a Sharpe of 1.65, you feel great. The honest story: with 200 trials of pure noise, someone is going to show a Sharpe of 1.65 —the expected maximum of 200 zero-true-SR trials —and it will not be your skill, it will be your selection bias.
Most backtest tooling computes statistics and prints reports. factor-qc
is a gate: it decides, with graded severity, whether a candidate may
pass —and its default answer is no:
- P0 (fatal) —DSR below threshold, PBO above threshold, haircut Sharpe below floor, MinTRL longer than the sample -> the candidate must not pass.
- P1 (warning) —weak PSR vs zero, short sample, aggressive trial count -> proceed with eyes open.
- P2 (info) —non-normal moments, tiny trial count -> recorded, no action.
Philosophy
Honesty is the default; the gate is fail-closed.
The one non-negotiable input is n_trials: the honest count of
configurations you tried. Without it there is no deflation benchmark
(Bailey & López de Prado 2014),
no haircut (Harvey, Liu & Zhu 2016, RFS),
no track-record floor
(Bailey & López de Prado 2012, JPM 39(1))
and no overfitting probability
(Bailey, Borwein, López de Prado & Zhu 2017, JCF).
Refuse to declare, and the gate refuses to judge —that asymmetry is the
point. qc check exits non-zero on any P0 failure, so it drops into CI,
pre-commit hooks and research gates as a hard blocker, not a suggestion.
Two design commitments that keep it honest:
- numpy only, no scipy —the standard-normal inverse CDF is Acklam's rational approximation with one Newton refinement; every number in the report is reproducible from the code in this repo, no hidden black box.
- PBO is optional but explicit —without a trials matrix the gate says PBO was not computed, and the DSR cross-trial variance degrades to a conservative single-trial estimate. Absence of evidence is reported as absence, never as evidence.
Quick start
# install the published package from PyPI
pip install factor-qc
# or run without installing anything:
# PYTHONPATH=src python -m factor_qc --help
python examples/demo.py # try it on reproducible synthetic cases
Your own backtest:
# returns.json = JSON list of per-period returns of the selected candidate
# trials.json = JSON 2D matrix (T x N) of every configuration you tried
qc check --returns returns.json --trials trials.json --n-trials 200
# -> FAIL - P0 blocker(s): dsr: 0.63 vs 0.95; mintrl: 250.8 vs <= 1000; ...
qc check --returns returns.json --n-trials 5 --json # machine-readable
Probability factors also carry evidence: judge predicted probabilities against observed outcomes (Brier skill / log loss / ECE, fail-closed):
qc calibrate --predicted probs.json --actual outcomes.json
# -> PASS/FAIL/refuse; refuse = too few samples or degenerate outcomes
qc calibrate --predicted probs.json --actual outcomes.json --json
Exit codes: 0 = no P0 failures (P1/P2 may still be failing), 1 = at
least one P0 failure (or missing n_trials), 2 = usage error. Wire it
into CI as a hard gate. qc calibrate exits 1 on FAIL/refuse and 0 on
PASS.
Commands
| Command | What it does |
|---|---|
check |
Run the gate: DSR, PBO (when --trials given), haircut Sharpe, MinTRL as P0; PSR-vs-zero, sample length, trial aggression as P1; moments and trial count as P2. Human-readable or --json output |
version |
Print version |
Flags: --returns (required), --trials (optional), --n-trials
(required unless require_declared_trials is disabled in code),
--periods-per-year (default 252), --n-blocks (CSCV granularity, default
16).
The checks
| Check | Severity | Method | Reference |
|---|---|---|---|
dsr |
P0 | Deflated Sharpe Ratio: P(SR > E[max SR of N trials]) under non-normal moments | Bailey & López de Prado (2014), JPM 40(5) |
pbo |
P0 | Probability of Backtest Overfitting via Combinatorially-Symmetric Cross-Validation (12,870 splits at n_blocks=16) | Bailey, Borwein, López de Prado & Zhu (2017), JCF |
haircut_sharpe |
P0 | Multiple-testing haircut of the Sharpe ratio (Bonferroni/Holm/BHY) | Harvey & Liu (2015), "Backtesting" |
mintrl |
P0 | Minimum Track Record Length: observations needed before SR is significant | Bailey & López de Prado (2012), JPM 39(1) |
psr_vs_zero |
P1 | Probabilistic Sharpe vs zero | Bailey & López de Prado (2012), JPM 39(1) |
sample_length |
P1 | ≥252 observations | — |
trial_aggression |
P1 | n_trials ≥n_obs / 5 | — |
return_moments |
P2 | skew ≥0, kurtosis ≥3 | — |
trial_count |
P2 | n_trials ≥5 | — |
The P0 set mirrors the spirit of Harvey, Liu & Zhu (2016), "“And the Cross-Section of Expected Returns": a factor must survive multiple-testing correction to earn the right to be called a factor. The gate is the machine version of that editorial stance.
Performance note
CSCV enumerates C(n_blocks, n_blocks/2) splits —12,870 at the default 16.
On large trial matrices (T=1000, N=200) that takes minutes; use
--n-blocks 8 (70 splits) or 10 (252 splits) for interactive speed at
slightly coarser granularity.
Development
python -m pip install -e . pytest
python -m pytest
CI runs the full test suite on Ubuntu, Windows and macOS with Python 3.11 and 3.12. Issues are handled on weekends; pull requests are welcome.
Related work
- Bailey & López de Prado (2014), The Deflated Sharpe Ratio
- Bailey, Borwein, López de Prado & Zhu (2017), The Probability of Backtest Overfitting
- Harvey, Liu & Zhu (2016), “And the Cross-Section of Expected Returns (RFS)
- Harvey & Liu (2021), Lucky Factors (JFE)
- Mobarekeh & López de Prado (2024), Backtest Overfitting in the Machine Learning Era (SSRN 4778909) —why OOS methods still need honest trial accounting
Project family
Part of Holdout — a toolchain against self-deception in quantitative research:
- pit-adjuster — PIT back-adjustment with static forward-adjustment drift detection
- falsification-ledger — pre-registration and falsification ledger
- factor-qc — fail-closed backtest quality gate
- lesson-book — tuition memory for traders
- lookahead-free — verifiable look-ahead-freedom checks
- ashare-data-immunity — data immunity for A-share daily bars
Sister org: Metabolism Tools — workspace-metabolism, policy-driven file lifecycle management for agentic workspaces.
License
MIT
Metadata
Release files for factor-qc 0.1.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| factor_qc-0.1.3.tar.gz | 24.5 kB | Without provenance | Details for factor_qc-0.1.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| factor_qc-0.1.3-py3-none-any.whl | Python 3 | none | any | Details for factor_qc-0.1.3-py3-none-any.whl |
Total release size: 44.5 kB
Release files / factor_qc-0.1.3.tar.gz
| Download URL | factor_qc-0.1.3.tar.gz |
|---|---|
| Size | 24.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ca6773be40c621e1bc660bcc3488146dbbfc74f5dc2e8fa23d3794f19a0070ee
|
|
BLAKE2b-256 checksum How to use checksums |
76f6d9af62d43a95f80a21b3193c12bebd5b3b209bb0d76bde1dd5eeafc990fe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.15
|
Release files / factor_qc-0.1.3-py3-none-any.whl
| Download URL | factor_qc-0.1.3-py3-none-any.whl |
|---|---|
| Size | 20.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6b6beddc6e0fa6a8c046c316212bba8d38a018e233f99ed54f40a6770aeaffd3
|
|
BLAKE2b-256 checksum How to use checksums |
d266bfd93e4e5ab3ee2ebd0dd7a2a9cdb1b5fb326a4c40f233918c0d7f842a0a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.15
|
Security for factor-qc 0.1.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.