quant-integrity

Statistical tests for whether a backtest means anything.

This library will not find you a profitable strategy. Its function is to reject them. Everything here exists to answer one question: given how many configurations you tried before reporting this one, does the result survive?

pip install quant-integrity

Commercial intent, stated on day one

The statistics in this repository are AGPL-3.0 and free forever. Every method here is published academic work. There is no moat in the arithmetic and no version of this library is crippled to sell you an upgrade — a teaser would fail as a funnel and would forfeit the only thing an integrity tool has.

Hosted attestation is a paid service. Verification is public and lives here. Issuance is not, for reasons set out below. If that ever becomes a product it will be a separate, proprietary thing, and this library will still do everything it does today.

Saying this up front so nobody can later claim a rug-pull.

What this is for

A worked example, from the author's own research, using this exact code.

A moving-average crossover on BTC-USD hourly bars. After sweeping parameters, the best configuration looked like this:

Sharpe (annualised, after costs)   0.9500
Probabilistic Sharpe (PSR)         0.8556    "probably beats zero"

That is where most backtests stop. Here is what happens when you account for the 29 configurations evaluated to find it:

Trials evaluated                   29
E[max Sharpe] under the null       1.8657    what luck alone produces over 29 trials
Deflated Sharpe (DSR)              0.1533
Bootstrap 95% CI on Sharpe         [-0.93, 2.69]   straddles zero
Minimum backtest length            1.94 years required, 1.25 available

The observed Sharpe is not near the bar. It is below half of it. A high PSR with a low DSR is the signature of an overfit search, not a borderline edge.

Then the same specification was run across 22 liquid USD pairs instead of one:

Pooled Sharpe across 22 assets     -1.86
Assets with positive Sharpe        3 of 22
BTC's rank in the distribution     the MAXIMUM

BTC was not a representative result. It was the best of 22 — an asset selection that happened before anyone started counting trials, so the true N was higher than 29 and even the deflated figure was generous.

The cause was visible in the ledger. Sorting the 28 recorded runs by trade count gives a Spearman correlation of -0.9618 with Sharpe (p = 4e-16): every configuration that traded less looked better. That is not a signal being discovered. It is cost drag being measured, plus a slow moving average approximating buy-and-hold in a rising market. The search was not finding an edge; it was finding the configuration that traded least.

Every number above is reproduced end to end in examples/btc_ma_postmortem.ipynb, from the actual return series of that research (examples/btc_ma_runs.npz, 24 KB). The notebook needs no server and no account — clone the repo and run it.

What's in it

import quant_integrity as qi

out = qi.deflated_sharpe(returns, n_trials=29)
print(out["dsr"], out["expected_max_sharpe_per_observation"])

n_trials means every configuration you evaluated, including the ones you discarded. Understating it produces a flattering answer. Nothing in this library can check it — which is exactly the problem the next section is about.

Why self-hosting can't attest

This is the argument the hosted service rests on, and it is worth stating plainly even if you never pay for anything.

A trial count is only meaningful if it cannot be revised downward. But if you run your own ledger, you can edit it. Not through malice, usually — through the ordinary temptation to restart the count after a rewrite, or to not record the sweep that went nowhere. A self-attested integrity record certifies nothing, because the person attesting is the person who benefits.

So the split is:

Verification is public. Issuance is the service.

Anyone can check an attestation with this library and an issuer's public key. No cooperation from the issuer is required, and none of the checking code is withheld. What you cannot do is make one, because a signature is only worth something when the signer is not the beneficiary.

The property that does the work is not any single signature — it is the chain. verify_chain confirms that, across a sequence of attestations:

• sequence numbers are consecutive, so no record was removed from the middle;
• each references the previous record's digest, so none was altered afterwards;
• n_trials never decreases.

An outside party can establish all of that without ever seeing the ledger. Someone who edits their own records cannot reproduce it, because they would have to re-sign every subsequent record with a key they do not hold.

That is the whole business: not the arithmetic, which is here and free, but operating an instance that has no stake in the answer.

Scope

This is a statistical instrument. Deliberately absent, and staying absent:

• No buy, sell or hold signals. No target prices, position sizes or stop levels.
• No model portfolios or allocations.
• No claims about returns, and no examples implying any.

The library takes return series and trial counts and returns statistics about them. It does not know what you are trading and does not offer an opinion on it.

Install and contribute

pip install quant-integrity                 # statistics
pip install "quant-integrity[attestation]"  # + signature verification

Python 3.10+. Depends on numpy, scipy and pandas. Fully typed (py.typed).

Contributions require a CLA and a DCO sign-off — see CONTRIBUTING.md, which explains why without apology.

Licence

AGPL-3.0-or-later. See LICENSE and NOTICE.

The network-use clause is deliberate: it means a competitor cannot run a closed hosted fork of this code. It does not restrict you from using the library in your own research, hosted or otherwise, without publishing anything.