holdout

A backtest overfitting toolkit: probabilistic and deflated Sharpe ratios, probability of backtest overfitting, purged cross-validation and multiple-testing haircuts.

Every strategy search ends with a best result. The question this library answers is how much of that result survives once the search itself is taken into account — the number of variants tried, the non-normality of the returns, the way the data was split, and the fact that the winner was chosen because it looked best. Each statistic is validated against a published worked example, a closed form, or a simulation of its own sampling distribution, and each section below says which.

See ROADMAP.md for the build order.

What is checked, and against what

That claim is worth only as much as the evidence, so here it is before the install line. Six of these nine compare the library against a reference that shares no derivation with it, one is an identity the library has to hold against itself, and the last two are results rather than checks: they are the case for the library existing at all. All nine are produced by examples/evidence.py, which runs in CI and exits non-zero if any of them moves:

Check	Result
Bailey and López de Prado (2014), their worked example end to end	SR0 = 0.1132 and DSR = 0.9004, as published
Expected maximum of n normals against quadrature, n = 2 to 10^6	agree to 1e-10
Skewness and kurtosis against scipy.stats, both bias conventions	agree to 1e-12
Lo's eta(q) against its AR(1) closed form, eight cases	exact
CPCV path count against k/N x C(N, k), every N up to 12	exact
Log normal upper tail against scipy.stats.norm.logsf, z = 0.01 to 400	agree to 1e-14 relative, past where the tail itself underflows
The two excess-return conventions, constant rate, five rates to 40%	agree to 1e-12 relative, which is the identity they are
50 pure-noise strategies, best selected, 200 runs	PSR(0) > 95% in 90% of them, DSR > 95% in none
30 crossover trials with a real edge, best selected, before and after financing	DSR clears 0.95 gross and falls below 0.70 in excess of 5%, the financing haircut exceeding the search's own

The figures are bounds rather than digits, deliberately. Two implementations of the same quantity agree to within something, and where they stop agreeing depends on the SciPy and NumPy builds underneath — so a figure quoted to ten significant places is a number that happens to be true on the machine that measured it. The bound is the claim, and it is the bound the script enforces.

Three rows are worth a second look.

The quadrature row is quoted as a bound over the whole range for a reason. At n = 10**6 the reference is the less accurate of the pair: computing Phi(z)**999999 multiplies a million nearly-equal factors, and the 1.7e-11 disagreement there is mostly the quadrature's. Quoting the agreement at one convenient n would have made the library look better than the comparison can actually establish.

The tail row is checked relatively, not absolutely. By z = 400 the log tail is about -80000, so an absolute agreement of 1e-11 is a relative agreement of 1e-16, and an absolute bound would say more about the largest z in the sweep than about either implementation. The range matters as much as the bound: past z ~ 38 the upper tail is zero in double precision, log(1 - cdf) is -inf, and only a routine that stays on the log scale throughout has anything left to report.

The noise row is the whole argument. Fifty strategies with no skill whatsoever, the best of them selected the way any search selects: the probabilistic Sharpe ratio calls that coin flip a discovery in 181 runs out of 200, and the deflated ratio — which raises the benchmark to what the best of fifty unskilled trials would be expected to show — is fooled in none of them.

The last row is the argument turned the other way round. Thirty crossover rules on a market that genuinely trends, so there is something to find, and the best of them clears the deflation comfortably on the returns as reported. Put the same record in excess of a 5% financing rate and it does not: the deflated ratio falls from above 0.95 to below 0.70 while the benchmark the search earns does not move at all. The financing haircut on that strategy is the larger of the two adjustments, and it is the one that is usually left out. Figures as bands rather than digits here for a reason specific to the row — the effective trial count comes off an eigendecomposition whose eigenvalues sit exactly on an integer, where the estimator steps, so a third decimal place would be a fact about the BLAS underneath.

python examples/evidence.py     # all nine, non-zero if a figure moves

Behind them: 1,039 tests on Python 3.10 through 3.13, with SciPy used in the suite as an independent reference and never as an implementation.

Install

pip install holdout-backtest
holdout sample-data --out sample
holdout pbo sample/sweep.csv

Not on the package index yet. The pip install holdout-backtest line above is what it will be; until the first release lands, install from source and the commands under it work unchanged:

pip install "git+https://github.com/DaniyalMlk/holdout.git"

The distribution is holdout-backtest; the import is holdout. pip install holdout fetches an unrelated project that was on the index first. The Python package keeps the short name because renaming it would have broken every existing import to settle a registry collision, so:

import holdout  # installed from holdout-backtest

To work on the library instead:

pip install -e ".[dev]"
python -m pytest

Python 3.10 or later. The only runtime dependency is NumPy; SciPy is used in the test suite as an independent reference.

Command line

Installing the package provides a holdout command. sample-data writes two synthetic parameter sweeps — thirty moving-average crossover rules on ten years of simulated GARCH prices, one market with nothing to find and one with a slow cyclical drift — so everything below runs without real data.

holdout sample-data --out sample
holdout deflate sample/sweep.csv
holdout compare sample/trending.csv ma5_40 ma5_120

30 trials over 2520 periods
effective trials: average 11.5, eigenvalue 10.0, participation 2.2  (using eigenvalue)
selected:          ma30_120
sharpe:            0.29 annualised
expected maximum:  0.19 annualised, if nothing works
probabilistic:     0.820  (P[true sharpe > 0], ignoring the search)
deflated:          0.622  (the same, after the search)

holdout pbo sample/sweep.csv

30 strategies, 16 blocks, 12,870 splits
probability of backtest overfitting: 0.975
probability of out-of-sample loss:   0.638
degradation: oos = 0.0316 - 1.202 * is   (r^2 0.77)
most often selected in-sample:
  ma30_120              16.4%
  ...

holdout spa sample/trending.csv          # against zero; --benchmark COLUMN to compare
holdout mcs sample/trending.csv          # no benchmark needed; --statistic range
holdout sharpe sample/trending.csv --column ma10_120

The same statistics on the trending sweep: deflated ratio 0.971, SPA p-value 0.015, and Romano–Wolf names the rules that beat cash. Every command reads a CSV with one row per period and one column per strategy; see holdout/io.py for the layout.

Examples

• examples/deflated_worked_example.py — the Bailey and López de Prado (2014) example step by step, ending with the track record the strategy would need after deflation (8.2 years).
• examples/parameter_sweep.py — every statistic on both synthetic sweeps.
• examples/model_confidence_set.py — both sweeps through the confidence set; the one with a real drift still keeps 29 of its 30 rules.
• examples/purged_cv.py — a nearest-neighbour model with no skill scores a 0.20 forecast correlation under shuffled k-fold and none under purged k-fold.
• examples/sharpe_difference.py — how often each variance of a Sharpe-ratio difference rejects a true null, in three regimes of serial dependence.

The test suite runs all five and checks the numbers they print.

The design decision that mattered

Every statistic states the convention it needs, and refuses inputs that break it, instead of computing something plausible. The formulas in this field are short and unforgiving: an annualised Sharpe ratio passed with a daily sample size, excess kurtosis where raw kurtosis belongs, or a multiple-testing method applied to one ratio without the rest of its family all produce a number of the right magnitude that is simply wrong. So:

• Sharpe ratios are per period everywhere; annualisation happens at the edge.
• (skewness, kurtosis) pairs that no distribution can have are rejected with a message that names the likely mix-up.
• haircut_sharpe refuses Holm and the FDR procedures, which need the whole family, and points to the function that takes it.
• The effective-trials estimator has no default, because the three estimators disagree and the choice should be visible at the call site.
• Financing is an input step, applied once where the file is read, and the report says when a rate is in force -- an excess Sharpe ratio and a total-return one are different numbers that look alike.
• A series that does not move is refused, by a test that is relative to the series' own magnitude. The standard deviation of a constant array is not exactly zero in floating point, so sd == 0 does not fire and the ratio comes back around 1e15 -- a number that wins every comparison it is entered into, which is the worst shape a bug can take here. The threshold is eight multiples of machine epsilon, which is above the rounding a constant series leaves (under four, measured across lengths and magnitudes) and far below any variation a return series could have.

The same principle drives the validation: every number in this README was measured, and where a published figure did not reproduce from its own printed inputs (the paper's DSR is computed from an unrounded benchmark) the test suite records why rather than bending the formula to match.

The record is not in excess of anything

Every statistic here is stated for returns in excess of a financing rate, and every entry point defaults to a rate of zero. That default is not neutral. A Sharpe ratio taken on total returns is too high by rate / volatility — at a 5% cash rate and 16% annualised volatility, 0.32 of annualised Sharpe ratio, which is larger than most of the haircuts this library exists to compute. Deflating a number that was already overstated in the same direction understates the deflation.

holdout financing sample/trending.csv --risk-free 0.05

2520 periods at 252 per year, 5.00% annualised financing, compound (0.000193631 per period), arithmetic excess

strategy   gross  excess  financing  rate/vol  residual
-------------------------------------------------------
ma10_120  +1.046  +0.572      0.473     0.473  +0.0e+00
ma30_250  +1.007  +0.534      0.473     0.473  -3.3e-16
...

For comparison, the haircut the search itself earns over 30 trials is
0.651 annualised. The largest financing haircut above is 0.473.

--risk-free is on every command that reads a returns CSV, and the conversion happens where the file is read rather than inside each command, so a rate given once reaches every figure that command prints. holdout deflate on that sweep returns a deflated Sharpe ratio of 0.971 on the returns as reported and 0.644 in excess of 5%, with the benchmark the search earns unchanged. That is the conclusion moving, not the figure.

from holdout import excess_returns, financing_impact, levered_excess, period_rate

rate = period_rate(0.05, 252, compounding="compound")   # 1.93631e-04
net = excess_returns(returns, rate)                      # or convention="geometric"
impact = financing_impact(returns, rate, periods_per_year=252)
impact.annualised_shift   # what financing took
impact.predicted_shift    # rate / volatility, the closed form for it
impact.residual           # what the closed form missed: zero for a constant rate

Both conventions the conversion involves are named rather than assumed, and measuring them is what says which one matters.

The excess-return convention turns out not to. (1 + r) / (1 + rf) - 1 is (r - rf) / (1 + rf): for a constant rate the geometric excess return is the arithmetic one times a positive constant, and a Sharpe ratio does not see positive constants. So the choice cannot move any ratio of a mean to a standard deviation — they agree to 1e-12 relative across rates up to 40%, which is the identity rather than an approximation. It does move the mean and the volatility, both by that factor, so reported returns and drawdowns are not indifferent to it. Only a rate that varies separates the ratios, and then by about 4e-04 relative over a decade of daily returns against a cash rate sweeping 0% to 6%.

The compounding convention is a rounding error beside the rate itself: at 5% over 252 periods, dividing gives 1.98413e-04 and taking the 252nd root gives 1.93631e-04, a difference of 4.8e-06 that moves the haircut by 2.4%. It is offered because the two are different numbers and the caller should know which one is in force, not because the answer turns on it.

It is computed as expm1(log1p(a) / n) rather than (1 + a) ** (1 / n) - 1. The second builds a growth factor a hair above one and subtracts one from it, and a small rate loses its leading digits to that cancellation: at 0.25% over 252 periods it loses eleven of them and the round trip through annual_rate comes back 4e-12 off its input, where the first round-trips exactly. The first draft used the direct form; a round-trip test at 1e-12 is what caught it.

One cash rate is not one shift

The haircut above is rate / volatility, so a family of strategies sharing one cash rate does not move by one amount: the shift is largest for the least volatile of them. Two things follow, and neither is visible from the level.

The winner can change. The ranking by Sharpe ratio survives the conversion exactly when the shifts coincide, which needs the sample standard deviations to coincide — not the population ones, which is the most a family can be designed for. A constructed pair makes the mechanism exact: columns at 0.05 and 0.02 per-period Sharpe, volatilities of 0.2% and 2%, so the shifts are 0.1 and 0.01. The gap of 0.03 is less than the 0.09 the shifts differ by, and the order has to reverse — it comes out at -0.05 against +0.01. Measured over 200 draws of thirty trials, the winner is displaced in:

Volatilities span	Winner displaced
one value	2% of draws
1.4x	~10%
4x	~44%
16x	~61%

The first row is the interesting one. A family drawn at a single target volatility satisfies the invariance's condition only in expectation — the sample standard deviations still differ by order 1 / sqrt(2n) — so even there a close race turns over sometimes. The algebra alone would have said never.

The benchmark moves too, in either direction. The cross-sectional dispersion of the trial Sharpe ratios is what the deflated Sharpe ratio's benchmark is computed from, and financing changes it. With a common mean return the quiet strategies have both the highest ratios and the largest shifts, so the field compresses and the bar comes down; with a common Sharpe ratio a level field is hit hardest at the quiet end, so it stretches and the bar goes up. Both are measured, because the sign is a fact about the data and not about the conversion.

That is also the condition the front page's earlier claim needed. The benchmark does not move on the crossover sweep because thirty variants of one rule have nearly the same volatility — 0.651 to 0.650 — and holdout financing now says so rather than leaving it to be assumed:

The winner does not move: ma10_120 either way. The shifts would have to differ by
more than the gap it won by, and here they do not.
The field it won against is not the same field: the benchmark goes down, from
0.651 to 0.650 annualised.

from holdout import financing_shifts, selection_under_financing

financing_shifts(trials, rate)          # rate / sigma_i, one per strategy
picked = selection_under_financing(trials, rate, periods_per_year=252)
picked.moved                             # did the winner change
picked.displacement                      # excess Sharpe given up by selecting first
picked.benchmark_gross, picked.benchmark_excess

Leverage is Sharpe-neutral only at no spread

A reported Sharpe ratio is often levered up afterwards on the argument that the ratio does not care. It does not, when the borrowing is at the financing rate exactly:

levered_excess(returns, leverage=3.0, rate=rate, spread=0.0)
# == 3.0 * excess_returns(returns, rate), to 1e-16

Pay a spread s over that rate and the ratio falls by (1 - 1/leverage) * s / sigma, exactly — at a 1% annualised spread, 16% volatility and three turns, 0.04 of annualised Sharpe ratio. Below one turn the position lends rather than borrows, so it pays no spread and the scaling holds whatever the spread is. All four of those are tested as identities rather than tolerances, because that is what they are.

Sharpe ratio inference

import numpy as np
from holdout import estimate_sharpe

returns = np.random.default_rng(0).normal(0.0006, 0.01, 1260)  # five years of days
sr = estimate_sharpe(returns, periods_per_year=252)

sr.value  # per-period ratio
sr.annualised  # value * sqrt(252)
sr.standard_error()  # Mertens: uses the sample skew and kurtosis
sr.standard_error("normal")  # Lo: assumes iid normal returns
sr.confidence_interval(0.95, annualise=True)

Two conventions are fixed once and used everywhere:

• Sharpe ratios are per period unless a name says otherwise. Every inference formula in the literature is stated for the per-period ratio and a sample size in periods; feeding it an annualised ratio with a daily sample size is off by a factor of sqrt(252) and nothing complains.
• Kurtosis is raw, 3 for a normal distribution, because the corrections are written in terms of (kurtosis - 1) / 4. Passing excess kurtosis for a skewed series usually produces a (skewness, kurtosis) pair that no distribution can have, and the library rejects it rather than computing a variance from it.

What was measured:

Check	Result
Skewness and kurtosis against scipy.stats, both bias settings	agree to 1e-12
Lo standard error vs simulated sampling distribution (normal, n = 500)	within Monte Carlo error
Same, negatively skewed returns (skew -2, kurtosis 9)	normal formula 10% too small; Mertens within 1.2%
90% interval coverage, Student-t returns, 2000 samples	nominal rate within 2.5 points
Lo's eta(q) against its AR(1) closed form	exact
Adjusted annual ratio vs ratio of aggregated returns, AR(1) with phi = 0.4	within 0.1%; naive sqrt(12) overstates by 47%

autocorrelation_adjusted_sharpe(returns, 12) applies Lo's (2002) serial-correlation factor in place of sqrt(q). Smoothed or illiquid marks produce positive autocorrelation, and positive autocorrelation is exactly the case where the naive annualisation flatters a strategy.

Probabilistic and deflated Sharpe ratio

from holdout import (
    deflate_trials,
    effective_number_of_trials,
    minimum_track_record_length,
    probabilistic_sharpe_ratio,
    trial_correlation,
)

probabilistic_sharpe_ratio(0.1, 1250, benchmark=0.0, skewness=-0.5, kurtosis=6.0)
minimum_track_record_length(0.1, confidence=0.95)  # periods needed

# trials: a (periods, variants) matrix of every variant that was backtested
result = deflate_trials(trials)  # best column, N = columns
n_eff = effective_number_of_trials(trial_correlation(trials), method="eigenvalue")
result = deflate_trials(trials, n_trials=n_eff)
result.probabilistic, result.deflated, result.expected_maximum

The probabilistic Sharpe ratio is the probability that the true ratio exceeds a benchmark. The deflated Sharpe ratio raises that benchmark to the ratio the best of N unskilled trials would be expected to show: the null for a selected strategy is not "no skill" but "the best of N attempts at no skill".

What was measured:

Check	Result
Bailey and López de Prado (2014) worked example	SR0 = 0.1132 and DSR = 0.9004, as published
Exact expected maximum of N normals vs SciPy quadrature	agree to 1e-9
The paper's closed-form approximation vs exact	+2.6% at 5 trials, +0.9% at 100, +0.1% at 10^6; -7.9% at 2
50 pure-noise strategies, best selected, 200 runs	PSR(0) > 95% in over 80% of runs; DSR > 95% in none
49 noise strategies plus one with per-period SR 0.2 over 500 days	DSR > 95% in 53% of runs

The last two rows are the practical point. Selecting the best of fifty backtests makes a coin flip look like a discovery nine times out of ten; and even an annualised Sharpe ratio above 3, hidden among forty-nine others, only survives deflation half the time on two years of data.

One detail worth knowing: the paper's 0.9004 is computed from the unrounded benchmark 0.113172. Recomputing from the printed 0.1132 gives 0.90026. The test suite records both so that nobody bends the formula to match a recomputation from rounded figures.

Correlated trials. The deflation assumes the N trials are independent; a parameter sweep is not. effective_number_of_trials takes a correlation matrix and one of three estimators, and the method is a required argument because they disagree between the extremes. Six near-copies plus four independent strategies count as 6.0 (eigenvalue, Li and Ji 2005), 7.0 (average correlation) or 2.5 (participation ratio). The eigenvalue method is exact for blocks of identical trials but jumps where an eigenvalue crosses an integer; the participation ratio is smooth but dominated by the largest block.

Multiple testing and haircut Sharpe ratios

from holdout import adjust_pvalues, haircut_sharpe, haircut_sharpe_ratios, minimum_sharpe

adjust_pvalues(pvalues, "holm")  # also bonferroni, sidak, bh, by
haircut_sharpe(0.063, 2520, n_tests=100)  # one ratio, single-step adjustment
haircut_sharpe_ratios(sharpes, 2520, method="bh")  # a whole family, step-wise
minimum_sharpe(2520, 100, periods_per_year=252)  # 1.10: the bar after 100 tests

Family-wise methods (Bonferroni, Šidák, Holm) bound the chance of any false discovery; the false-discovery-rate methods (Benjamini–Hochberg, Benjamini–Yekutieli) bound the expected share of discoveries that are false. Haircuts follow Harvey and Liu (2015): turn the Sharpe ratio into a t-statistic, adjust its p-value, and turn it back. A single ratio only admits the single-step adjustments, because Holm and the FDR procedures depend on the rest of the family; the library refuses rather than guessing the others.

The haircut is not proportional, which is the useful thing to know about it. On ten years of daily data, the share of the Sharpe ratio removed is:

Annualised Sharpe ratio	10 tests	100 tests	1000 tests
0.75	43%	100%	100%
1.0	24%	55%	100%
1.5	10%	22%	35%
2.0	6%	12%	18%

What was measured:

Check	Result
Adjusted p-values vs each procedure as usually stated, arbitrary inputs, six levels	identical rejection sets
Family-wise error, 20 independent nulls, 4000 runs, nominal 5%	Bonferroni 4.5%, Šidák 4.6%, Holm 4.5%
False discovery rate, 40 nulls and 10 alternatives, nominal 10%	BH 7.8% (bound: 8%), BY 1.9%
BH with equicorrelated (0.6) test statistics	at or below nominal

Probability of backtest overfitting

from holdout import probability_of_backtest_overfitting

result = probability_of_backtest_overfitting(trials, n_blocks=16)  # (periods, variants)
result.pbo  # share of splits where the winner lands at or below the median
result.logits  # one per split: logit of the winner's out-of-sample rank
result.degradation  # (slope, intercept, r^2) of out-of-sample on in-sample
result.probability_of_loss  # share of splits where the winner loses money out of sample
result.selection_frequency()  # how often each variant won in-sample

Combinatorially symmetric cross-validation (Bailey, Borwein, López de Prado and Zhu 2017) cuts the sample into S contiguous blocks and tries every one of the C(S, S/2) ways of using half of them to pick a winner and the other half to judge it. The deflated Sharpe ratio corrects one number for the size of the search; PBO asks of the search itself how often its choice fails to generalise.

The paper's S = 16 means 12,870 splits. For the Sharpe ratio and the mean each split is computed from per-block sums rather than by re-slicing the data: on a 2520 × 100 matrix that is 0.1 s against 11 s for the general path, which remains available for any metric passed as a callable. The sums are taken over centred returns, because the variance is a difference of sums and cancels badly otherwise — from raw sums, a series with mean 1 and spread 1e-4 loses its variance to a relative error of 8e-8.

What was measured:

Check	Result
Fast path and general path vs a plain loop written from the paper	agree to 1e-10 (1e-12 for the large-mean case)
Pure noise, 20 strategies, 30 runs	mean PBO 0.51
One strategy with a genuine edge among 20	PBO below 0.01; selected in over 99% of splits
Strategies demeaned over the full sample (the in-sample winner must lose)	PBO exactly 1

Cross-validation for overlapping labels

from holdout import combinatorial_purged_cv, leakage_audit, purged_kfold, walk_forward

# observation i is made at start[i]; its label (say a 5-day forward return) is known at end[i]
splits = purged_kfold(5, start=start, end=end, embargo=10)
leakage_audit(splits, start, end)  # raises LeakageError if any label overlaps

cv = combinatorial_purged_cv(6, 2, start=start, end=end, embargo=10)
cv.n_paths  # 5
paths = cv.assemble_paths(per_split_oos_returns)  # (5, n): five full backtests

When a label spans several periods, two neighbouring observations share part of their outcome, and ordinary k-fold keeps putting one in training and the other in test. Purging drops every training observation whose label overlaps a test fold; the embargo also drops the observations just after each test fold (López de Prado 2018, chapter 7). Combinatorial purged cross-validation (chapter 12) tests every choice of k of N groups, so each group is tested C(N-1, k-1) times and the out-of-sample results stitch into that many complete backtest paths — a distribution of Sharpe ratios rather than one.

leakage_audit checks every training label against every test label directly, so it does not trust the splitter it is checking. A property test runs purged k-fold over random label lengths, embargoes and fold counts and requires the audit to pass; plain k-fold on five-period labels is required to fail it. The CPCV tests check the path count against k/N * C(N, k) and that every path covers each observation exactly once.

Tests of superior predictive ability

from holdout import reality_check, romano_wolf, superior_predictive_ability

# differentials: (periods, strategies) of strategy return minus benchmark return
reality_check(differentials, seed=0).pvalue
spa = superior_predictive_ability(differentials, seed=0)
spa.consistent  # bounded by spa.lower and spa.upper
romano_wolf(differentials, alpha=0.05, seed=0).rejected  # which ones, strongest first

These answer "does anything here beat the benchmark?" without assuming the strategies are independent of each other or over time. All three resample whole rows with the stationary bootstrap (Politis and Romano 1994), so every strategy is resampled on the same days and the correlation between them is kept. The block length defaults to the Politis–White (2004, corrected 2009) estimate.

• White's Reality Check (2000) is valid but recentres every strategy to mean zero, so hopeless strategies count as contenders and dilute it.
• Hansen's SPA test (2005) studentises and leaves clearly inferior strategies at their negative mean. consistent is the p-value to report.
• Romano–Wolf (2005) stepdown names the strategies that beat the benchmark while controlling the family-wise error rate.

What was measured:

Check	Result
Block length vs the AR(1) closed form, n = 10,000–20,000	within 10% on average for phi = 0.3, 0.5, 0.7
Size at the least favourable null (6 strategies, 2000 runs, nominal 5%)	Reality Check 4.9%, SPA 5.4%, Romano–Wolf 5.5%
One strategy at the benchmark plus nine poor ones (300 runs)	Reality Check rejects 0.7% of the time, SPA 6.0%
One genuine strategy plus nine poor ones (300 runs)	SPA rejects 87% of the time, the Reality Check 56%
Romano–Wolf, two genuine strategies among eight	both found in every one of 300 runs

The third and fourth rows are Hansen's argument in numbers: the Reality Check is not wrong, but every poor strategy added to the set makes it harder for a good one to register.

The model confidence set, when there is no benchmark

from holdout import Statistic, model_confidence_set

# performance: (periods, models), higher better — returns, or losses negated
result = model_confidence_set(returns, alpha=0.10, seed=0)
result.included  # indices in the set, best average first
result.pvalues  # one per model; in the set when above alpha
result.at(0.25)  # any other level, without rerunning the bootstrap
model_confidence_set(returns, statistic=Statistic.RANGE, seed=0)

Everything above needs a benchmark. This does not, which is why it answers the question people actually arrive with: of twenty candidates with no incumbent among them, which can be told apart from the best?

Nominating the sample-best as the benchmark and testing the rest against it is not a repair. The benchmark is then chosen by the same data the test runs on, so under the null it is systematically the luckiest series present and every comparison is biased towards finding nothing. Hansen, Lunde and Nason (2011) turn it around: test whether all the models are equally good, drop the one the test most implicates if it is rejected, and repeat until it is not.

The size of the set is the result. A set holding 29 of 30 models is not the procedure failing; it is the procedure saying that ten years of daily data cannot separate 29 parameter choices. examples/model_confidence_set.py is exactly that case, and it is the uncomfortable one:

Sweep	deflated Sharpe	set at 10% (max)	set at 10% (range)
no drift in the market	0.622	30 of 30	30 of 30
a genuine slow drift	0.971	29 of 30	30 of 30

The second row has a real signal in it — the best rule clears the search that produced it — and the surviving set still spans 9.3% of annualised mean return. "This rule made money" and "this rule is the one that made money" are different claims, and a parameter sweep is usually read as making the second.

A smaller alpha gives a larger set. It is a confidence region, so the usual asymmetry of hypothesis testing runs backwards, and the default of 0.10 follows the paper rather than the 0.05 habit. One bootstrap produces every level, since the p-values do not depend on it — only the reading of them does.

Both statistics are here because neither dominates. MAX studentises each model against the average of the set; RANGE takes the largest studentised difference over all pairs. Over fifteen borderline samples of ten models they retained 8.13 and 7.07 models on average, the range set smaller on fourteen of the fifteen and larger on one — a tendency and not a guarantee. MAX is the default because a larger set is the weaker claim.

They also differ on degeneracy, and the difference is real rather than an oversight. Two duplicated columns have no standard error between them, so RANGE refuses; MAX never forms that pairwise error and carries on, giving both copies the same p-value. What it cannot then do is tell the caller that three of its ten "models" are one strategy implemented three times.

The zero it checks for is relative, not absolute. Two identical columns have means that differ in the last bits of the mantissa rather than not at all, so their bootstrap deviations come out near 1e-17 and a scale > 0 test passes them — after which the studentised difference is of order 1e16 and a model is eliminated on a p-value of zero, as though the evidence against it were overwhelming rather than absent.

Comparing exactly two strategies

Everything above is built for families. reality_check and superior_predictive_ability test many candidates against one benchmark with the selection accounted for; model_confidence_set finds the set that cannot be told apart from the best. Pointing any of it at two strategies is applying a multiple-testing correction to a single test.

The pairwise comparison has its own answer:

from holdout import sharpe_difference

result = sharpe_difference(strategy_a, strategy_b, periods_per_year=252)
result.annualised_difference  # +0.59 of annualised Sharpe ratio
result.correlation  # 0.656 — half of why this is not two separate tests
result.closed_form_error  # Jobson–Korkie with Memmel's correction
result.robust_error  # Ledoit–Wolf: delta method with a Newey–West covariance
result.error_ratio  # the robust error over the closed-form one
result.p_value  # two-sided, using whichever `method` asked for

Differencing two estimate_sharpe standard errors is not a substitute, and it is wrong twice. It ignores the correlation between the series — for two strategies on the same market that is usually large, and it always reduces the variance of the difference, so the naive interval is too wide and a real difference goes unreported. And it treats a Sharpe ratio as if it were a mean, when it is a ratio of two estimated moments and the delta method has terms a difference of two independent errors does not.

Both variances always come back, whichever the method argument selects for the headline statistic, because the disagreement between them is the finding when there is one.

What measuring them showed

examples/sharpe_difference.py runs a thousand replications in each of three regimes, with the two series correlated at 0.7, the same true Sharpe ratio, and five hundred paired periods. A test at the 5% level should reject 5% of the time.

serial dependence	closed form	robust	robust error / closed-form error
none	4.2%	4.7%	0.99
AR(1), ρ = 0.3	14.4%	6.4%	1.27
AR(1), ρ = 0.6	34.7%	10.0%	1.65

With independent returns the closed form is right and the robust version costs nothing: both land within a percentage point of nominal and the two standard errors agree to within one per cent. So there is no reason to prefer the closed form even where its assumptions hold.

Serial dependence is where they part, and both halves of that are worth stating. At a persistence of 0.6 the closed form rejects a true null more than six times too often — a third of the time it reports a difference in Sharpe ratios that is not there — because it assumes independence and the difference of two persistent series has far more sampling variability than its formula allows. The robust version gets that to one in ten, which is better and is not right: a Bartlett kernel truncated at a rule-of-thumb bandwidth recovers only part of the long-run variance. Use the robust variance, and do not read its p-value as exact on data you believe is persistent.

Two refusals

Comparing a series with itself is refused rather than answered. The difference is exactly zero and so is its variance, so the statistic is 0/0, and "these are the same strategy" is not the same statement as "the difference between these two is not significant" — a p-value of one would read as the second.

A series that does not move has no Sharpe ratio, and the guard for it has to be relative. numpy's standard deviation of a constant array is not exactly zero: subtracting the mean leaves rounding of order eps × level, which on 0.001 repeated 500 times is 4.3e-19. A guard at zero does not fire, the Sharpe ratio comes back as 2.3e15, and every number after it is arithmetic on rounding error.

Was the edge there throughout?

Everything above guards a maximum taken over strategies. spa asks whether the best of K beats a benchmark once K is accounted for, romano_wolf asks which ones do, deflate raises the benchmark to what the best of N unskilled trials would show, pbo asks how often the in-sample winner loses out of sample.

Nothing asked whether a strategy earned its Sharpe ratio evenly or earned all of it in one stretch. And the obvious check has this library's own flaw, twice.

The split point gets chosen after looking at the equity curve. Nobody splits a track record with a visible cliff two thirds along at the midpoint. The comparison is made at the most damaging date available and read against a critical value for one date fixed in advance.

Searching honestly does not fix it. Take the largest statistic over every candidate date and a normal critical value is the wrong distribution, for exactly the reason White's Reality Check exists: the maximum of several hundred correlated statistics is not distributed like one of them.

$ holdout stability sweep.csv --column ma5_40
2520 periods at 252 per year, 1765 candidate break dates, trim 0.15 (annualised)

strategy  break after  sharpe before  sharpe after  sup stat  p (bootstrap)  p (naive)
--------------------------------------------------------------------------------------
ma5_40            414          -0.25          1.04      1.51          0.706     0.1298

So the statistic here is a supremum by construction — there is no single-date version of it to misuse — and its null distribution comes from the stationary bootstrap, which handles the maximum and any serial correlation in one pass. The naive p-value is printed beside the real one because the gap is the point: a reader who has been quoting the naive one should see what it was worth. Above, 0.13 against 0.71.

How wrong the naive p-value is, measured

Over 500 replications of 1,000 iid normal returns with no break at all:

	rejects at 5%	trim 0.02
two-sided normal on the supremum	41.4%	58.8%
stationary bootstrap	3.2%	7.0%

The Monte Carlo standard error is about one percentage point. The bootstrap is conservative at the default 15% trim and liberal at an aggressive 2% one, so the trim is an argument rather than a constant — and worth choosing before seeing the answer rather than after.

The most useful thing it reports is how little power it has

Over 120 replications at 500 observations either side — two four-year halves of daily returns, a longer track record than most things get:

per-period Sharpe	annualised	detected at 5%
0.10 → 0.05	1.59 → 0.79	10%
0.10 → 0.00	1.59 → 0.00	25%
0.12 → −0.04	1.90 → −0.63	52%

A strategy that loses its entire edge at the midpoint is found a quarter of the time. So a large p-value here is close to no evidence, and reading one as confirmation that a strategy is stable is the mistake this test makes easiest — which is why the command says so in its own output. It is worth having anyway: when it does reject, it rejects against the right distribution.

The standard error is the one that accounts for shape

Two disjoint subsamples, each with its Mertens (2002) standard error from its own skew and kurtosis, and their difference standardised by the root of the sum of squares. The normal-returns version would be simpler and wrong in the direction that matters: a strategy with negative skew and fat tails has a noisier Sharpe ratio than the normal formula says, and a break test that understates the noise finds breaks that are not there.

Power sums, shifted, because a million Sharpe ratios cannot be a loop

A thousand resamples over a thousand days is 1.4 million subsample Sharpe ratios with their skew and kurtosis. Cumulative power sums make that four cumulative sums per resample.

They also cancel catastrophically on returns whose mean is a hundredth of their standard deviation, which is every return series there is — the sum of squares and the square of the sum agree to two significant figures and their difference loses the rest. So the sums are taken on x - mean(x) and the mean added back: subtracting a constant leaves every subsample's variance, skew and kurtosis unchanged and moves its mean by exactly that constant.

Two guards, and what it took to get each right

The zero-variance check has to be relative. A constant stretch's second moment computed from power sums is not zero but the rounding left over from subtracting two nearly equal sums — of order 1e-22 on daily returns. Divided into a mean of 0.001 that is a Sharpe ratio of 1e8 and a break statistic of −2e9: large enough to dominate every supremum in the sample, and finite enough to pass every check for a NaN. Only a threshold relative to the whole sample's own variance catches it, and a genuinely quiet stretch at a tenth of the sample's volatility is nowhere near it.

The refusal for a sample with no candidate break was nearly deleted as unreachable. The argument for deleting it covers even sample lengths only: with trim < 0.5 the edge is capped at half an even sample and a candidate always survives. For an odd length the ceiling reaches (total + 1) / 2 and takes the last candidate away. A sweep of every length from 60 to 400 against every trim to 0.499 reaches it 389 times — every one of them odd, every one at a trim above 0.492. The branch stays, and the sweep is the test.

The other half of the overlapping-label problem

The section above purges overlapping labels out of the training set, which fixes the leakage. It does not fix the counting. Twenty observations whose five-day windows cover the same week carry about one week of information between them, and every estimator that averages over observations treats them as twenty. The symptom is a t statistic too large for the data behind it, and purging never touches it, because nothing has leaked.

uniqueness supplies the number to divide by instead of n. Concurrency is how many label windows span each bar; an observation's average uniqueness is the mean of 1 / concurrency over its own window; the sum of those is the effective sample size. On a purged training set of 120 observations with ten-bar rolling windows it comes out under a quarter of the length.

One identity holds whatever the overlap looks like, and it is the check that the arithmetic is right rather than merely self-consistent — both sides count the same thing:

sum over i of (uniqueness_i * length_i) == number of covered bars

The sequential bootstrap is worth less than it is sold for

Drawing with a probability proportional to each candidate's uniqueness given what is already drawn sounds like it fixes the redundancy at its source. Measured, it mostly does not, and the reason is arithmetic. Total concurrency over a draw is size * length however it is drawn, so achievable uniqueness is capped at span / (size * length), and attaining the cap means flattening the concurrency — which is all any scheme can try to do. As a fraction of that cap, over sixty seeds on two hundred observations with twenty-bar windows:

draws	cap	sequential	uniform	gain
200	0.0548	0.998	0.995	+0.44% ± 0.09%
40	0.2737	0.978	0.946	+3.47% ± 0.55%
10	1.0	0.729	0.673	+10.43% ± 2.31%
5	1.0	0.865	0.809	+9.37% ± 2.80%

The heavy-overlap, full-size resample is the case the method is motivated by and the case in which it cannot help: the uniform bootstrap is already at 99.5% of a cap nothing can exceed. The gain is real only once the draw is small against the span. Its largest gain of all is on non-overlapping point labels, where there is no overlap problem and the only redundancy left is duplicate draws — which is the clearest sign of what it is actually doing.

So the remedy for heavy overlap is to draw fewer observations, and the effective sample size says how many.

Weights come two ways: proportional to uniqueness, and a time decay that is linear in cumulative uniqueness rather than in time, so a stretch of redundant observations ages like the information it carries. Ten redundant observations worth one observation out of eleven get 7.8% of the weight on that scale against 26.2% on an index-linear one.

holdout uniqueness --count 200 --window 20

The drawdown a track record owes to luck

Every other statistic here asks what a search was worth. This one asks what one realised path owes to chance: a track record's worst drawdown is the number that ends a mandate, and it is also a number a strategy with a real edge produces routinely.

The drawdown of a Brownian motion is that motion reflected at its own running peak, so the chance of never falling h below a high-water mark is the survival function of a first passage with a reflecting boundary at zero and an absorbing one at h. The backward equation separates, and the eigenvalues are the roots of k cos(kh) + b sin(kh) = 0 with b = -mu / sigma^2 — one in each branch of the tangent, so every root is bracketed before it is solved for.

from holdout import assess_drawdown, drawdown_quantile, expected_maximum_drawdown

expected_maximum_drawdown(horizon=252.0, drift=0.0, volatility=0.15 / 252**0.5)
# 0.1879...  -- a 17.1% fall, from a strategy with no edge whatsoever
drawdown_quantile(0.95, horizon=252.0, drift=0.0, volatility=0.15 / 252**0.5)
# 0.3362...  -- the one-year-in-twenty depth for the same non-strategy

holdout drawdown sample/trending.csv --column ma5_40

What a drawdown is worth knowing before it happens

At 15% annual volatility, the expected worst drawdown and the depth exceeded one record in twenty, as a fraction of the peak:

Sharpe	1 year	3 years	10 years
0.0	17.1% / 28.6%	27.8% / 44.1%	44.8% / 65.5%
0.5	14.8% / 24.8%	22.1% / 35.4%	31.5% / 47.3%
1.0	13.0% / 21.5%	18.2% / 28.5%	24.3% / 35.4%
1.5	11.5% / 18.7%	15.5% / 23.7%	20.0% / 28.5%

Two things are worth reading off it. Without an edge the drawdown grows like the square root of the horizon and without bound; with one it grows logarithmically, approaching sigma^2 / (2 mu) per doubling, so a drawdown limit scaled to the length of a track record is scaled to the wrong thing. And the first row is the answer to "is 28% too much": for a strategy with no edge, a year in twenty, no.

The waiting time is exponential in the depth, which is the same fact seen from the other side. At a 1.0 Sharpe ratio and 15% volatility a 20% fall takes 7.8 years to arrive on average and a 40% fall takes 450.

Three routes to the same law, and two that stop short

The expansion is checked against references that share no derivation with it.

• The driftless case has a reflection series. Lévy's theorem makes the drawdown of a driftless Brownian motion a reflected Brownian motion, whose maximum is distributed as the maximum absolute value of the original, and that has a classical image series in the normal distribution function. The two agree to 2e-16 over a grid.
• The first passage has a closed-form Laplace transform at every drift, from the same differential equation solved once rather than separated. Matching it at several arguments tests the eigenvalues and the coefficients jointly; the worst disagreement over 225 parameter sets is 3e-10 relative to the size of the coefficients, and most are at 1e-11.
• The coefficients are inner products and can be read off by quadrature against the speed measure. They agree to 1e-15.
• The expected drawdown has one closed form, sigma sqrt(pi T / 2) at zero drift, which the tail integral reproduces to thirteen figures.

Two things do not work and are reported rather than papered over.

The exceedance is one less a survival probability, and for a level the path will almost certainly not reach that is a cancellation: below about 1e-13 the answer has no significant digits, and no rearrangement of the series fixes it, because the term-by-term complement converges only conditionally. What is exposed instead is final_drawdown_exceedance, the exact law of the drawdown at the end of the horizon — a rigorous lower bound on the maximum's, running at 0.36 to 0.45 of it, and accurate past 1e-300.

And the obvious asymptote for that tail is wrong. Treating the first passage as exponential with its own exact mean — the natural reading of "a drawdown this deep arrives once every E[tau] years" — overstates a one-year 60% drawdown at a 0.67 Sharpe ratio by a factor of 520, and the error grows as the level deepens, which is the shape of a wrong exponent rather than a loose constant. The two tails are different: exponential at rate 2 mu / sigma^2 in the horizon, Gaussian in the level.

Where the expansion is well conditioned, stated rather than assumed

The coefficients carry exp(-mu h / sigma^2), which exceeds one only for a losing strategy. There the survival probability is a cancellation of terms far larger than the answer: the largest coefficient grows like 0.76 exp(beta) / beta in beta = -mu h / sigma^2, measured, and the absolute error is about eps times that. So the domain is a property of the method and is enforced with an error that names the reason. Nothing at or above zero drift is affected.

The budget is deliberately split. A probability a caller reads directly is held to an absolute error near 1e-9; the expected drawdown and the quantiles are an integral and an inversion, whose error budgets absorb five orders more, and spending that carries the losing-side domain out to a total Sharpe ratio of -3.23.

Both halves of that sentence have to be earned separately, and for a while only one was. The tail cut is loosened until the expansion can reach the ceiling, and the expected drawdown got that, because the step being retried evaluates the exceedance there. The inversion was handed the ceiling alone, which is closed form and cannot fail, so it ran at the tightest cut every time and the conditioning failures happened inside the bisection instead — where whether one happened at all depended on which midpoints the bisection visited, so the refusal fired at a total Sharpe ratio of -3.11 and not at -3.29. Both now share a ceiling the ladder has proved reachable, and the two domains end in the same place, between -3.225 and -3.250. What makes one evaluation enough for the whole bracket below it is that the conditioning failure is monotone in the level: on a 200-point scan out to eight, nothing above the first refusal evaluates.

Daily marks see less of the path than the path contains

A drawdown read off 252 daily observations is smaller than the drawdown of the path underneath them — 7% smaller in the mean, falling to 1.4% at sixteen times the frequency, with the shortfall dying like one over the square root of the count. That is not an error in the estimate: the discretely observed drawdown is a different contract, and it is the one a track record reports.

It does mean a test against the continuous law is conservative, and by how much is worth a number. On simulated records of 252 observations with the drift and volatility known, a nominal 5% test rejects 3.4% of the time. Estimating those two parameters from the same record takes the same test to 0.27% — a further factor of thirteen, and much the larger effect, because a path that fell a long way also reports a larger volatility and so makes its own drawdown look ordinary. Both effects are one-sided, so assess_drawdown carries a flag saying which case it is, and the command line says in words that an estimated null does not give a p-value.

A record that is watched, not sampled once

Everything above answers a question asked once, at a record length settled before the data arrived. Nobody reads a track record that way: it accumulates, somebody looks again each month, and the decision is taken at whichever look first makes it look established.

That rule has no size. Independent normals with a mean of exactly zero, the ordinary 95% interval computed every period, stop at the first crossing:

observations	rejection rate
60	33.7%
250	44.1%
1,000	52.5%
5,000	61.9%

The rate grows with the horizon, so there is nothing to divide out — the partial sum's fluctuations grow like sqrt(2 n log log n) and the interval only like sqrt(n). A long enough look finds something with probability one.

holdout watch records.csv

strategy      n    mean    lower   upper  width/fixed  crossed at  SR floor
---------------------------------------------------------------------------
real_edge  1200  28.41%    6.77%  50.05%        1.549        1021     0.435
marginal   1200   8.07%  -14.37%  30.52%        1.549       never    -0.891
nothing    1200  12.21%   -9.86%  34.28%        1.549       never    -0.621

Every row of the sequence holds at once, so crossed at is a legitimate stopping point rather than the result of a search.

The validity is an identity, and it is integrated

The construction is the normal mixture, and all of its validity is the claim that

M(n) = sqrt(rho / (V + rho)) * exp(S**2 / (2 (V + rho)))

is a martingale, for S the centred partial sum and V = n sigma**2. So the test suite integrates one step of it against the normal density — by a Simpson rule written out in the test file, so that nothing is checked against its own arithmetic — and gets its current value back to 2e-15. Ville's inequality does the rest, and the rejection boundary is exactly the level set M = 1 / alpha, asserted in both directions.

The mixing variance is derived rather than chosen: the stationarity condition is r - log(1 + r) = 2 log(1 / alpha) with r = target / rho, one dimension with nothing in it but the confidence level, 8.211968 at 95%. Around the optimum the boundary is flat — ten per cent either way costs three parts in ten thousand.

What it costs, in width and in data

width/fixed above is 1.548560 at the tuning point, 1.950657 a tenth of the way there and 4.563248 a hundredth; those are functions of n / target alone, the same to six decimals at tuning points from a hundred observations to a hundred thousand. And sequential_track_record_length is 3.40 times minimum_track_record_length, a multiple that barely moves with the strategy: 3.4047 at an annual Sharpe of 0.3, 3.3977 at 2.0.

Where it looks slack

Against the null the sequence crosses 1.35% of the time by its tuning point, not 5% — which reads as a bound three times too conservative and is not. The guarantee covers every n, and the rest of the budget is spent further out: 2.46% by four times the tuning point, 3.48% by twenty, 3.64% by a hundred.

And the derivation wants a known variance where a caller has a sample one. Substituting it takes the crossing rate from 1.47% to 1.98% at 250 observations — a 35% increase in false alarms, inside the nominal 5% because of exactly that slack. Which is a number, rather than the words "approximately valid"; pass --volatility for a bound that needs no such sentence.

The record that was chosen, conditional on its having been chosen

deflated asks whether the best of a set of records is significant and returns a probability. multiple haircuts it. Neither says what the selected strategy's Sharpe ratio is — and the number carried into the allocation is the winning estimate itself, which was chosen for being large.

selection conditions on that. Hold the losers' estimates fixed, and the winner's was drawn from its own sampling distribution restricted to the event that it beat them: a normal truncated below at a known point. Inverting that law in the mean gives an interval, exactly rather than asymptotically.

from holdout import selection_inference

# five candidates' per-period Sharpe estimates and standard errors
inference = selection_inference(
    [0.98, 0.95, 0.80, 0.41, 0.12],
    [0.31, 0.30, 0.32, 0.29, 0.33],
)

inference.selected.margin  # 0.10 standard errors over the runner-up
inference.interval.median  # -1.213, the median-unbiased estimate
inference.interval.lower  # -10.84
inference.interval.naive_lower  # 0.37, ignoring the search
inference.interval.inflation  # 10.0 times the naive width

The ordinary interval's coverage collapses in the number of candidates

Under a null where every candidate has the same true Sharpe and a unit standard error, the naive 95% interval on the selected strategy's estimate covers that strategy's own truth:

candidates	naive	conditional	selection bias	expected_maximum_normal
2	95.1%	95.0%	0.557	0.564
5	88.3%	95.2%	1.156	1.163
10	77.8%	94.8%	1.532	1.539
50	30.5%	94.9%	2.240	2.249

Past a few dozen candidates the naive interval is wrong twice as often as it is right. The bias it is missing is the expected maximum, and the last two columns are a simulation beside a closed form it never calls — agreement to within 1.4%, which is selection_coverage checking itself against holdout.deflated.

The price is the width, and only when the race was close

The conditional intervals average 25 to 67 wide against the naive 3.92, which sounds unusable. Their medians are 6.0, 9.0, 10.6 and 14.7 — 1.5 to 3.7 times the naive width. The gap between the two summaries is a heavy right tail made entirely of near-ties, so selection_coverage reports both and quoting only the mean would describe the typical case by the worst one.

With one candidate four standard errors ahead of nine others, the median conditional width is 4.10 against 3.92 and the naive coverage has recovered to 94.3%. The whole correction is a function of the margin over the runner-up:

margin, in standard errors	shrinkage of the point estimate
0.1	6.84
0.5	1.07
1.0	0.314
2.0	0.031
4.0	0.0000

The interval is finite however close the race was

Which is not the natural guess. A photo finish feels as though it should be consistent with any true value at all. The conditional tail is increasing in the mean and runs from zero to one over the whole line, so every level is attained at a finite mean. What happens instead is that the bound runs out at log(1 / alpha) * deviation / margin — 123 standard errors at a margin of three hundredths of one. An exact tie is the one degenerate case and raises rather than returning an infinity.

Where the one assumption fails, the two intervals change places

The conditional law treats the winner's estimate as independent of the losers'. A sweep over neighbouring parameters is not, and selection_coverage(..., correlation=rho) prices that rather than caveating it. Ten otherwise identical candidates:

rho	naive coverage	conditional coverage	median width
0.00	78.5%	94.9%	10.8
0.50	86.1%	94.6%	14.1
0.75	90.5%	91.4%	19.0
0.80	91.4%	90.0%	21.0
0.90	93.2%	84.1%	28.9
0.99	94.3%	55.3%	90.0

They cross at about rho = 0.78. Correlation shrinks the winner's curse — the mean selection bias falls from 1.53 to 0.15 across that range — so the naive interval recovers, while the conditional one becomes both wider and less valid. Above three quarters, use the ordinary interval. With fifty candidates the crossing is much further out because the naive interval has further to fall: at rho = 0.5 it covers 68.2% against 94.6%.

The synthetic sweep this library ships is exactly the bad case, which is why the command prints the margin first:

holdout sample-data --out sample
holdout selected sample/trending.csv

30 strategies over 2520 periods
selected:          ma10_120
runner-up:         ma5_180 at 1.03 annualised
margin:            0.06 standard errors over the runner-up
raw sharpe:        1.05 annualised
median-unbiased:   -2.50 annualised  (3.55 lower than the raw estimate)
conditional 95%:  [-17.93, 1.25]  (19.17 wide)
ignoring the search: [0.44, 1.65]  (1.21 wide)
the search costs a factor of 15.79 on the width

Thirty moving averages over one series are close to being one strategy, and a margin of six hundredths of a standard error is them saying so.

The log tail carries the module

Every quantity here is a ratio of two normal upper tails and both can underflow. erfc reaches exactly zero near 38 standard deviations, so the direct ratio is 0.0 / 0.0 — a missing number rather than a small one. log_upper_tail is erfc below 20 standard deviations and the Mills asymptotic series above it, and both the crossover and the term count were swept: that cell's worst relative error against an independent implementation over z from 0.1 to 400 is 6.5e-16, and its neighbours in both directions are at machine precision too. Compared against erfc below the crossover, where both are accurate, the series' own error is 7.3e-10 at z = 8 and 2.8e-16 at 20 — so by the time it is used it is already exact.

Layout

Module	Contents
series	input validation for return series and matrices
moments	skewness, raw kurtosis, autocorrelation
sharpe	Sharpe ratio, Lo and Mertens standard errors, Lo's serial-correlation factor
financing	excess returns, both conventions, the rate conversion, the levered case
deflated	probabilistic and deflated Sharpe ratios, minimum track record, expected maximum
trials	effective number of independent trials
multiple	adjusted p-values, haircut Sharpe ratios, minimum t-statistic
pbo	combinatorially symmetric cross-validation
splits	walk-forward, purged k-fold, combinatorial purged CV, leakage audit
uniqueness	concurrency, average uniqueness, sample weights, sequential bootstrap
bootstrap	stationary bootstrap and Politis–White block length
spa	Reality Check, SPA, Romano–Wolf
mcs	model confidence set, both statistics
pairwise	difference of two Sharpe ratios, Memmel and Ledoit–Wolf variances, Newey–West
selection	the selected strategy's Sharpe conditional on its having been selected, and the stable normal log tail
drawdown	drawdown statistics, the exact maximum-drawdown law, calibrated limits
io, synthetic, cli	CSV input, the sample sweeps, the holdout command