Backtest Auditor (nba_auditor), v0.3

A red-team "lint" for trading strategies, delivered as an MCP server and a Python library. It runs the checks we ran on our own models (lookahead, survivorship and ticker reuse, cost realism, overfitting and multiple testing, sub-period stability, concentration) on your code, return series or trade log.

You bring your own data. Run locally (CLI or MCP), the auditor makes no network calls, calls no data vendor and needs no API key, so we never receive your code. Your AI assistant is a different matter: anything you paste into it, and anything the tools return to it (findings quote file names and code lines, and suggested fixes include the original and replacement text), is sent to that assistant's provider as usual.

Every report ends with: This is an automated statistical and code review of a hypothetical backtest. It is general information, not investment advice, not tailored to anyone, and does not predict future results. The checks can miss problems and can flag correct code. Nothing But Alpha is not a registered investment adviser.

Tools

Tool Input What you get
check_path(path, include_glob?, max_files?) A .py file, notebook or folder on disk Per-file findings with rule id, severity, title, exact line range and a one-line "why it matters"; suggested fixes where a minimal edit is unambiguous (see below). Read-only
audit_backtest(code_path?, returns_csv?, trades_csv?, benchmark_csv?, n_trials?, turnover?, ...) Paths to the code and the results Every audit that applies, and one combined verdict (the worst of the parts) with reasons per part
check_code(source) Python/pandas strategy code Findings with rule id, severity, line, why the problem inflates results, and a fix. Score 0-100. Also suggested_fixes
audit_returns(returns, freq, costs_bps?, turnover?, positions?, benchmark?, n_trials?, ...) A return series (CSV/JSON) Cost grid and break-even cost; Sharpe with Newey-West and Lo t-stats; Deflated Sharpe Ratio; minimum track record length; thirds and years; top-day concentration; drawdown; IR/beta. Verdict No red flags found / Some red flags / Major red flags with reasons, plus a score
audit_trade_log(trades, prices?, initial_capital?, ...) date, ticker, side, qty, price[, fee] Daily PnL and turnover from the log, then audit_returns, plus ticker concentration and survivorship hints
explain(check_id) e.g. LA07, RA03, or empty What the check does, why it matters, how to fix it, references

Every tool returns {"markdown": ..., "report": {...JSON...}}. The server also offers two MCP prompts, which clients show as one-click commands: Find look-ahead bias in this repo (find_lookahead_bias) and Audit this backtest before I rely on it (audit_backtest_before_trusting).

For agents: check_path, suggested fixes and audit_backtest

Draft: compliance-checker pass 2026-10-16 (CLEAR WITH EDITS, applied). Still held until counsel answers LEGAL_QUESTIONS 1b.

Coding agents (Claude Code, Cursor, Claude Desktop) call the server while they write backtests and act on what it returns, so these tools favor precision: few findings get an edit, and an edit is only offered when the change is unambiguous.

  • check_path is read-only. Files are parsed with ast and never imported or run. Only .py, .pyw and .ipynb files are read, whatever include_glob says. Virtual environments (any folder with pyvenv.cfg), site-packages, node_modules, hidden folders (.git, .venv, ...), build and cache folders are skipped. Folder symlinks and junctions are not followed, so link loops end; a file symlink is read only if it points inside the folder. Limits: 200 files by default (max_files, at most 2,000), 1 MB per file, 20 MB in total. Binary, unreadable and undecodable files are listed under skipped with the reason. A UTF-8 byte-order mark is stripped before parsing; other encodings are used only when the file declares them (PEP 263). Notebook findings carry the cell and the line in the cell.

  • Suggested fixes are offered only for high or medium findings in these unambiguous cases:

    Rule Pattern Suggested edit
    LA01 y = x.shift(-k) alone, with x a plain column or variable, never combined with x later x.shift(k); or just x when the result is already lagged later (e.g. position.shift(1)), so the fix never lags twice
    LA02 rolling(..., center=True), not feeding a label center=False (trailing window)
    LA03 .bfill(), .backfill(), fillna(method="bfill") on rows visibly sorted ascending (a resample, sort_index(), or sort_values ending on a date); reindex(..., method="bfill") on the row index The forward-fill version
    LA05 A scaler or model fitted on all rows, with a visible X[:k]/X.iloc[:k] split later Fit on the training slice
    LA06 train_test_split without shuffle=False; KFold(shuffle=True); only when the rows split are visibly ordered by date alone and the file shows no sign of panel data (an asset column such as ticker, any groupby, a sort with another key before the date) shuffle=False; TimeSeriesSplit (plus its import)

    Each fix is a structured edit (file, start_line, end_line, original, replacement) with a one-line rationale. Before a fix is returned, the server applies it in memory and lints again: the finding must be gone, the code must parse, and no new high or medium finding may appear. Otherwise the fix is dropped. No fix is offered where the mechanical edit would change what the code means: x.shift(-1) > x or x.shift(-1) / x - 1 (a lead combined with x), pct_change(-k) and diff(-k) (the mirror has the opposite sign), a lead lagged again later, a column later used as a model's label (y of fit, a split, a metric, or a label-named column), fills across columns (axis=1), fills on data whose ascending order isn't visible (bfill on newest-first data reads the past, so ffill would be the leak). Also none for stratify=, interpolate(), more than one candidate, or notebooks. Fix checks are budgeted (12 per file, 60 and 10 seconds per call); fixes past the budget are skipped and counted in fixes_not_checked. Most findings get no fix; an agent should explain those rather than guess an edit. A fix removes a pattern; it doesn't make a backtest trustworthy.

  • audit_backtest runs check_path on the code, audit_returns on the returns and audit_trade_log on the trades, each against the benchmark if given, and returns the worst verdict of the parts, in the same language, always followed by the caveat line. A high code finding is "Major red flags"; a medium one is "Some red flags". The caps stay: without a benchmark, or with code only, the best verdict is "Some red flags".

  • Eval (in-sample; not an accuracy measure). eval/agent_fix applies the suggested fixes to small leaky and clean repos and counts leaks removed, new findings and clean lines changed. We wrote both the repos and the fix rules, so the result only shows that the fixes behave as designed on patterns we already knew; it says nothing about how often they help, or what they change, in your code. On those 27 files the fixes removed 11 of 23 planted leaks, added no new findings, changed no clean lines and changed none of the lines where a mechanical fix would change the meaning (the meaning_preservation repo, from the red-team review). The 12 not removed get no suggested fix by design.

Code rules (check_code)

id Rule Typical severity
LA01 Negative shift/pct_change/diff in a feature, signal or condition (labels are fine). v0.6: a lag helper called with a negative lag (lag(r, -2) with def lag(s, k): return s.shift(k)); a helper that returns a negative shift is also shown at its return line high
LA02 rolling(..., center=True); rolling/pct_change/shift on data sorted newest-first (sort_values('date', ascending=False), sort_index(ascending=False)). v0.6: two-sided transforms (filtfilt, sosfiltfilt, hpfilter, savgol_filter, gaussian_filter1d, uniform_filter1d, np.gradient, argrelextrema/argrelmin/argrelmax, find_peaks, convolve(..., mode="same"), a Kalman .smooth(...), seasonal_decompose, STL, lowess, bkfilter, cffilter) and rolling(center=<setting>) when every visible definition of the setting is True: high when the result reaches a signal, position, PnL or model input (followed through assignments and helper return values), possible otherwise; not reported for a label, or when re-run on data up to each date inside a loop high / possible
LA03 bfill, fillna(method="bfill"), reindex(method="bfill"), interpolate() medium / possible
LA04 merge_asof(direction="forward"/"nearest") high / medium
LA05 Scaler/PCA/model fitted on all rows before the time split, or fitted and evaluated in-sample; a scaler fitted on all rows while the model trains on a subset of them (medium). A fit on rows bounded above (.loc[:cut], .iloc[:k], [:k], .loc[a:b], df.date < cut, between) is a training subset and is not flagged; a start filter (.loc['2000':], df.index >= '2000') or a test-window mask is not a training subset. v0.6: a full-sample estimate (sm.OLS/WLS/RLM, np.polyfit, linregress, coint, KMeans, GaussianMixture, GaussianHMM, arch_model, ARIMA, PCA, ...) outside loops and helpers, in a file with no split, training-window or walk-forward names, whose output reaches a position or signal that is booked against returns (medium; not for forecasts compared with outcomes, fit(last_obs=...), or positions computed for live use with no backtest in the file) high / medium
LA06 train_test_split without shuffle=False; shuffled or default k-fold CV on time series. v0.6: TimeSeriesSplit without gap when a label in the file looks 2 or more bars ahead (medium, unless the folds are trimmed or the file mentions purging or an embargo); fit_resample (SMOTE and similar) on all rows before a split or cross-validation (high); .sample(frac=1) before a time-series split, and torch random_split on time series (medium); KFold(shuffle=True) also when the only time-series hint is a forward-return name (medium) high / medium / possible
LA07 Signal from bar t's close times bar t's return, including numpy/list slices (c[1:] vs c[:-1], zip, np.diff), inline conditions and helper functions; a close-based signal times a return that starts at bar t's open; bar loops that decide on bar i's close (also through a variable set from it earlier in the iteration, v0.4) and fill at bar i's open (high) or close (medium) or book bar i's return; an entry or exit priced at bar t's open and selected by a signal from bar t's close, high, low or volume: open.where(sig), open.mask(~sig), open[sig], df.loc[sig, "open"], np.where(sig, open, nan), df.loc[sig, "entry_px"] = df["open"] (v0.4; high when the result has a fill name such as entry_px, buy_price, exit_px, else possible). v0.5: a positive shift at the end of a function passed to groupby(...).transform/apply/pipe (lambda s: s.pct_change(7).shift(1), or a helper returning param....shift(k)) lags the result, and a frame used only to group (panel.groupby("date")["mom"]) doesn't count against a lag; cheat_on_close/set_coc/trade_on_close; vectorbt from_signals(price=close) (medium) or default close fills (possible) with unshifted entries. v0.6: a fill or entry priced at the same bar's low or high (entry_px = bar["low"] in a bar loop, high; fill_px = np.where(sig, low, nan), from_signals(price=low), medium); a limit set from bar t's close checked against bar t's own low or high (high); a bar's VWAP is treated like its open for same-bar timing (a close signal entering at that bar's VWAP) high / medium / possible
LA08 Label is the same-bar return (not shifted forward) possible
LA09 Global z-score / min-max / x / x.abs().max() / x / x.std() over all rows, also inside functions (issue #3) and helpers; a full-sample mean/median/quantile used as a signal threshold; full-column rank, zscore, qcut feeding a signal; per-asset groupby().transform of a full-history statistic; a lambda, a named helper or zscore passed to apply/transform/pipe that scales each column by its own whole-column statistics (v0.4; not flagged with axis=1, groupby, rolling/expanding windows, a transposed frame, a training slice or one date's cross-section). Medium when the result feeds a signal and the column is a time series; possible otherwise. v0.6: groupby(<asset>)[col].rank(pct=True) over each asset's whole history (medium when it reaches a signal); winsorizing at quantiles of every row (quantile(0.02) then clip) or fillna(frame.mean()/median()) before the frame is split into train and test rows (medium) medium / possible
LA10 Feature or signal built from the label; X that drops target but keeps fwd_ret (drop lists, constants, df.columns comprehensions and str.startswith masks are resolved per column); forward columns passed to fit or a training dataset, also inside a frame built with pd.DataFrame({...})/pd.concat; a neutral-named forward value that reaches a signal or position; an outcome (a forward average, a forward PnL statistic, a target encoding) assigned to a signal, position or weight name, or transformed (rolling, sign, rank, clip, ratio) and multiplied with a return (*, .mul, np.dot, np.where(c, r, -r), np.copysign(r, s) with s from future data, v0.4); a forward column reshaped with pivot/pivot_table/unstack keeps its forward flag and lead (v0.4); literal query/eval strings are read as the code they stand for (v0.4); a strategy-named series kept only on rows chosen by an inline comparison on future data (strat = bench[mkt_fwd > 0], medium, v0.4); buckets formed from the forward return and then used to average it (medium). Not flagged: forward returns used only as the outcome (averaged per date or bucket, an IC, booked against a position decided earlier, next-open fills, a bucket pivot_table of the forward return, fw[mask].mean() with a mask from past data); a lead undone by a lag of the same size (shift(-H) then shift(H)); a fill model, where a resting limit order placed without future data fills if the next bar's low (high) reaches the limit and that mask only selects trade returns (v0.4). A neutral name that the lint can't tie to a model input or signal is "possible" high / medium / possible
LA11 Parameter grid (loops, itertools.product, dict/list comprehensions) scored on the full sample, best kept (reports the grid size to use as n_trials). v0.6: also shown at the line that keeps the best (best = max(scores, key=scores.get)), as a repeat that is scored once medium
LA12 Intraday resample stamped at the bar start (default or label='left'): medium when the bars are joined, reindexed or assigned back onto another time index, possible otherwise. A resampled column assigned into a frame of the same bars (bars["volume"] = ticks["size"].resample("1min").sum()) is not "another time index" (v0.4). v0.6: names computed only from bars of the same rule (feat = pd.DataFrame({... a_1s.rolling(5).mean()}), nxt = mid_1s.pct_change().shift(-1)) keep those bars' stamps, so joining them is not a realignment; groupby(<period of the index>)[col].transform("last"/"max"/"sum"/"mean"/"std"/...) or pd.Grouper(freq=...) writes a period's final value onto the earlier bars of that period (medium when it reaches a signal, possible otherwise; first, count, running statistics and a following .shift are fine) medium / possible
LA13 x[i+1], x.iloc[i+1] or a slice starting after i inside an if in a for or while bar loop, or a value read from a later bar earlier in the loop. v0.4: offsets from module constants and parameter defaults are resolved (c[i + K] with K = -2 is the past); string character loops and loops whose outputs only ever become a label or target are not reported; a later bar read only for a missing-data check (np.isnan, pd.isna, is None) is possible. v0.5: a resting order simulated in a loop is not reported: an order decided at bar i from past data (an enclosing condition, an and term, or an elif of another fill check), filled if bar i+1's low or high reaches its level (a level known before that bar's high and low: past data or bar i+1's open), with the fill or exit priced from that level and every later-bar read in the branch a price assigned to a fill, exit or PnL name. Possible when the fill isn't priced from the level. v0.6: in a backtrader Strategy/Indicator, a positive line index (self.data.close[1]) is the next bar (high); a trade list read only for its size (len(trades) >= 50, if trades, not trades) is not a decision on fill values high / possible
LA14 Early stopping or best-model selection on the test set (eval_set/valid_sets/validation_data with early stopping); a loop that picks the model, parameter or feature with the best test score. v0.6: max/min with a key that scores test data (max(grid, key=lambda t: f1_score(y_test, p > t))); a dict filled in a loop with one test-set score per candidate (test-named data or rows ~train_mask) and picked with max(d, key=d.get) (high; a repeat scored once when the loop is already reported); an Optuna-style objective passed to optimize that returns a score on test data (high) high / medium
LA15 Fundamentals merged onto prices by fiscal period end (period_end, fiscalDateEnding, datadate, calendardate, ...) instead of the filing date, with no visible reporting lag: merge_asof high, period-end index joined medium, exact merge medium if forward-filled, else possible high / medium / possible
SB01 Today's constituents: read_html(<Wikipedia S&P 500>), tickers_sp500(), snapshot CSVs, hard-coded ticker lists; filtering a panel to names whose last date is the panel's last date (medium). v0.6 (medium): dropna(axis=1) (or thresh=len(...)) on a price- or return-named wide frame that is then run through returns or rolling windows; columns kept by iloc[-1].notna(); a universe taken from the last row (iloc[-1].nlargest(n), not of a score or signal) or from whole-period returns (px.iloc[-1] / px.iloc[0]) that then selects the backtest's columns; members whose end/removed date is empty (not combined with a point-in-time test); groupby(id).filter(lambda g: g[date].max() >= <fixed end date>) high / medium / possible
DQ01 yfinance data caveats info
DQ02 Raw Close returns with auto_adjust=False; Adj Close mixed with raw OHLC medium / possible
CO01 No commissions, fees, spread or slippage anywhere (vectorized or loop PnL); engines called without fees. A realism caveat, not a lookahead error, so never above "possible" (since v0.3); the message says when a Sharpe, CAGR or total return is computed from the cost-free returns possible

The lint uses ast plus a light data-flow pass over assignments: it tracks whether each column is forward-looking, lagged, price-derived, a return or a signal, and uses name heuristics (target/fwd_*/y mean labels, signal/position/pred mean signals). When unsure it says "possible". Code that doesn't parse falls back to a few text patterns, all reported as "possible".

Return checks (audit_returns)

  • RA01 costs and the break-even cost.
  • RA02 Sharpe significance: Newey-West, plus a Lo-style (2002) correction.
    • The HAC bandwidth widens automatically when returns overlap.
    • All significance tests use an effective sample size: T divided by the long-run variance ratio.
  • RA03 Deflated Sharpe Ratio (Bailey & López de Prado 2014).
  • RA04 MinTRL (Bailey & López de Prado 2012).
  • RA05 sub-period stability.
  • RA06 concentration in the best 5% and 1% of days, compared with a normal distribution.
  • RA07 drawdown.
  • RA08 benchmark: excess return, IR, beta, and alpha with its t-stat.
  • RA09 implausibility: a Sharpe above 3, or a daily hit rate above 65%, on daily or slower bars.

Trade logs add TL01 survivorship hints and TL02 concentration by ticker.

Verdict rules (always a statistical review of a hypothetical backtest, never a certification). Every verdict is followed by this line: No red flags doesn't mean a strategy will make money; it means these checks didn't find the common errors.

  • Major red flags: any of the following.
    • DSR < 0.50, meaning the result does not beat what the best of n_trials zero-skill strategies would show.
    • The mean return is not positive.
    • Costs erase the edge at your declared cost (10 bps if you gave turnover but no cost).
    • The Sharpe is above 5 on daily or slower bars.
  • No red flags found: all of the following.
    • DSR >= 0.95 and a Newey-West t of at least 2, both with the autocorrelation adjustment.
    • All three thirds are positive.
    • n_trials is declared and costs are assessed; a user-supplied trials_sr_variance is not below the zero-skill sampling variance.
    • The series has dates (or an explicit freq), regularly spaced, with finite values.
    • A benchmark is supplied and the alpha over it is significant (t >= 2). Without a benchmark the best verdict is "Some red flags", with the reason "No benchmark supplied: market exposure can't be separated from skill."
    • The result is not implausible.
    • The series is not intraday, and the units look right.
    • For trade logs: daily prices were given, fills match those prices, and no anachronistic ticker was found.
  • Some red flags: everything else. Concentration (RA06) is a warning and a score component, not a blocker, because positively skewed strategies (trend, long volatility) are concentrated by nature.

Score: A score of 100 means none of our listed patterns matched, not that the code is free of leaks. Static checks miss many leaks. (Every report prints this next to the score.) Weighted points for DSR (40), costs (20), stability (20) and concentration (20). A check that can't be assessed is dropped and the rest rescaled. The verdict also caps the score: at most 40 for Major red flags, 79 for Some red flags, and 20 if the result is implausible. The JSON has the full breakdown. MinTRL is reported but not scored separately, because "track record >= MinTRL vs the hurdle" is the same test as DSR >= 0.95.

Calibration on synthetic data (how often the verdict is wrong)

These are synthetic daily series of 5 years each (1260 days; the leveraged-index rows use 10 years), with turnover 0.05 per day. Every row gets an independent market series as its benchmark, except where it says "no benchmark". Without a benchmark the best verdict is "Some red flags" by design. Reproduce the table with python scripts/calibration.py (fixed seeds; last run after the second red-team pass and the benchmark rule).

Series No red flags found Some red flags Major red flags
Pure noise, one strategy, n_trials=1 (300 runs) 1% 45% 54%
Best of 100 noise strategies, honest n_trials=100 (200 runs) 0% 28% 72%
Best of 100 noise strategies, n_trials=1 (understated) (200 runs) 67% 33% 0%
Noise as 20-day overlapping returns, n_trials=10 (200 runs) 0% 6% 94%
Noise with AR(1) autocorrelation 0.3, n_trials=10 (200 runs) 0% 4% 96%
2x leveraged index (Sharpe ~0.75), no benchmark (200 runs) 0% 94% 6%
2x leveraged index, benchmark = the index (200 runs) 0% 94% 6%
Planted Sharpe 0.5, n_trials=10 (200 runs) 1% 22% 76%
Planted Sharpe 1.0, n_trials=10 (200 runs) 14% 54% 32%
Planted Sharpe 1.5, n_trials=10 (200 runs) 48% 44% 8%
Planted Sharpe 0.7, positive skew, n_trials=1 (200 runs) 0% 100% 0%
Planted Sharpe 0.7, negative skew, n_trials=1 (200 runs) 0% 100% 0%

Code lint, measured on held-out code (snippets the builder had not seen, written by two red-team passes): recall at high or medium severity was 47% (15/32) on the first set and 67% (8/12) on the second, measured before any fixes. That is the honest expectation for new code: roughly half to two thirds of leaks get caught, so a clean check_code result is weak evidence. Both sets are now regression tests, so the current scores on them are in-sample (see Known limitations).

v0.2 lint, measured on sealed sets written outside the build (2026-10-16). The builder never saw these snippets; only the leak category names of the first set were shared.

  • Sealed set v2 (50 leaky, 30 clean): v0.1 recall 54%. The first v0.2 build reached 76% but raised one high false alarm on correct code (pos[1:] = np.where(sig[:-1], 1, -1), a lag through the assignment target). After that fix: 39/50 (78%) flagged at high or medium, 0/30 false alarms.
  • Sealed set v3 (60 leaky, 40 clean, with near-miss clean files written to trip the lint): v0.1 recall 24/60 (40%) with 5/40 high false alarms; v0.2 at commit ea153fd 36/60 (60%) with 3/40 high false alarms, all LA10 (target leakage) and all present in v0.1 too. Re-run at 5f3ea0a: unchanged, 36/60 with 3/40 high false alarms, all LA10. A red-team review of that build found more high false alarms (pandas .shift(1) before .to_numpy(), lagged period ends, sort_index(axis=1), a holdout used for early stopping); they are fixed and are regression tests in tests/corpus_rt3. Re-run at the merged code (18e6055): unchanged, 36/60 (60%) with 3/40 high false alarms, all LA10.
  • The free web check stays off until v3 shows 0 high false alarms. A high finding on correct code is worse than a miss.

v0.3 lint: precision fixes, measured in-sample only (2026-10-16). v0.3 targets the remaining high false alarms: LA10 on forward columns that are dropped before fitting, used only as the outcome, or used as next-open fill prices; LA05 on fits whose training rows are a slice or a date mask; CO01 capped at "possible". The builder wrote the development twins below from descriptions of those classes and never read the sealed sets. Every number here is in-sample (the rules were changed until these files passed). They show the fixes work on the patterns as described, not how v0.3 does on new code.

v0.3 on sealed set v5 (written outside the build, never read by the builder; run by the coordinator on the final code at 2717e1e): 0 of 70 correct files flagged at high or medium; 21 of 50 leaks (42%) flagged at high or medium (easy 9/12, medium 9/20, hard 3/18; 24/50 = 48% counting "possible"). For comparison, v0.2 on the same set: 20/50 (40%) with 14/70 correct files flagged (13 of them CO01 medium). Coordinator's note: v5 is written to imitate everyday research code and covers more leak types than v2/v3, several outside these rules. Read it this way:

  • Recall on v5 is about 40%, far below the 60-78% quoted for the v2/v3 sets above. Expect roughly 40% of leaks in new code to be caught at high or medium. A clean check_code result is weak evidence.
  • The v5 recall number is contaminated. Two leak classes from v5 were fed back to the builder during the red-team review (the descriptions in rounds 2 and 5 below), so recall on those classes is partly in-sample.
  • The v5 precision number (false alarms on correct code) is not contaminated. No clean v5 file or description was fed back.
  • LA10 twins (tests/corpus_v03, 27 clean, 20 leaky; acceptance criteria committed before the first run in tests/corpus_v03/PREREGISTRATION.md). Clean files with a high or medium finding: v0.2 13/27 (8 with LA10 high), v0.3 0/27. Leaks flagged as expected: v0.2 18/20, v0.3 20/20.
  • LA05 and CO01 twins (tests/corpus_v03b, 15 clean, 8 leaky). Clean files with a high or medium finding: v0.2 10/15 (3 LA05 high, 5 LA05 medium, 2 CO01 medium), v0.3 0/15. Leaks flagged: v0.2 7/8, v0.3 8/8. Three clean and one leaky twin (d10-d12) were added after the first measurement, to reproduce the LA05 high case (a split-named variable assigned after the fit); one clean/leaky pair (b10) was added after the repository rescan.
  • Round 2 (PR #9 review, tests/corpus_v03c, 7 clean, 7 leaky, also in-sample). It covers three cases: a negative shift at the end of a chain (rolling(21).std().shift(-21)) flowing through features into a signal lagged by fewer bars than the lead; a negative np.roll used as a "previous" value; a position times a return between next-bar fill prices. Before the fix, v0.3 caught 3/7 leaks at high and had 1/7 clean files with a high finding (v0.2: the same). After: 7/7 and 0/7.
    • The lint now tracks how many bars a value looks ahead, and a lag undoes only that many bars.
    • A neutral-named forward value that reaches a signal or position (also through pos[1:] = ... or pos[mask] = ...) is high, for both LA01 and LA10.
    • The label passed to lgb.Dataset/xgb.DMatrix/Pool is no longer an input, so a model trained on it, and its predictions on new rows, are not reported as target leakage. The data passed to these constructors is checked like fit.
    • After round 2, the rescan against v0.2 shows: repo, 0 new high or medium outside the test corpora; earlier internal codebase, 0 added and 60 removed. 21 of those are v0.2 LA10 highs on predictions from a model trained on a label in one function and scored on fresh rows at the rebalance date in another; on review, false alarms. Static analysis can't tell whether the training window ends a full label horizon before the scoring date (an embargo question), and v0.3 does not try.
  • Round 5 (red-team block of PR #9, tests/corpus_rt5, the red team's repro blocks). At bef5683, 19 required cases failed that v0.2 passed: 14 leaks were missed or only "possible", and 5 correct files got a high false alarm on a symbolic horizon (shift(-H) realigned by shift(H)). Now all 19 pass. Changes:

    • A frame built from columns (pd.DataFrame({...}), pd.concat([...])) that stores a forward column is tracked as a container. It is high when a forward column survives into fit/predict, a training dataset, a feature matrix or a signal. An IC frame that is only correlated stays clean.

    • Outcomes that decide a position are high in two cases:

      • The outcome is assigned to a signal, position or weight name (position, weight, score; names that only say long, short or trade are not enough).
      • The outcome is a transform of forward data (rolling, sign, rank, clip, ratio) used as a factor in a product with a return, whatever its name. Products are *, @, .mul/.multiply, np.multiply, np.dot, and np.where(c, r, -r) / r.where(c, ...), where c is the position.

      Examples: a per-date mean forward return turned into a tilt, a rolling Sharpe of forward PnL used as a weight, a target encoding used as a score.

    • Symbolic horizons cancel: shift(-H) then shift(H) (also periods=-H, function parameters, cfg.horizon, H + 1, -1 * 5).

    • LA05 date masks and slices count as a training subset only if bounded above.

    • On the red team's files, 10 leaks that are not in the required set are also flagged now. 9 are still missed (listed under Known limitations).

  • Round 6 (red-team review at 0ba08c2, APPROVE WITH FIXES; tests/corpus_rt5/r6_*.txt):
    • Products written as .mul, np.dot, np.where(c, r, -r) and r.where(...) count, so a long/short-named outcome can't escape through them (A01, A03).
    • Containers follow rename(columns=...) and data.columns = [...] (C07), and positional column selections (K01: X, y = data.iloc[:, :-1], data.iloc[:, -1] is clean).
    • A lead and lag with the same expression no longer cancel when the name is rebound, a loop variable or bound in several scopes (B01-B04).
    • A forward comparison used only for a hit rate is "possible" (K09).
    • len(data) and data.shape are no longer treated as reading the data.
    • Still missed: pivot(values="fwd_ret") (A04-A06; A04/A05 caught in v0.4) and a helper defined above the forward column it reads (B05).
  • Round 7 (red-team review at c8bfe78; tests/corpus_rt5/r7*.txt):
    • Container names now default to unknown after any frame-level operation other than known name-keeping ones (F01-F06, H02). This covers in-place renames, add_prefix, set_axis, mapper renames and .columns.str.replace.
    • Inline transforms inside a product factor are judged as the expression (E03, H05).
    • Masked sign flips of a return series on rows chosen from forward data are flagged (E02, E07), as is the keyword form where(cond=..., other=...) (E05).
    • A name whose only signal words are long, short or trade is judged by its use, not its name. This removed a new high false alarm on the IC frame in research/nba_research/phase0b.py.
    • Documented misses: np.copysign (E01) and frame.eval("t * b") (E04). Both caught in v0.4.
  • Round 8 (red-team review at 9e6fc0a; tests/corpus_rt5/r8.txt):
    • Container column order is no longer kept stale across reorders (S01-S03): sort_index(axis=1), a literal reorder list, iloc[:, ::-1].
    • A return selected by a position-named mask built from future data (bench[long], bench.loc[short], bench[~long]) is high (T01). An inline comparison (bench[mkt > 0], as in up/down capture) is not counted.
    • A container lagged by its full lead (data.shift(5) with a 5-bar lead) is clean (U01). Partial lags stay high (S04, S06).
  • Round 9 (red-team approval at 6dbc2b4, conditional on documentation; tests/corpus_rt5/r9.txt):
    • A column subscript that isn't a literal list (data[sorted(data.columns)], data[data.columns[::-1]]) and any transpose (.T) make the column order unknown (V01, V03, V06).
    • Masks are followed through .values, .to_numpy() and x[x].index (W03, W04).
    • A mask of future data with a neutral name selecting returns (ok = mkt > 0; bench[ok]) is medium (W01): it can be an up/down-capture filter rather than a trading rule. A position name (long, short, regime) is high. An inline comparison (bench[mkt > 0]) is not counted (X01), and a lagged regime mask is clean (X02).
    • Documented miss: frame.query("m > 0") (W06). Caught (medium) in v0.4.
  • Older corpora: no leak lost its high or medium finding except the 7 cost-only files (v21-v23, v50, hl21-hl23), which now show CO01 as "possible" by design.
  • Rescan against v0.2. This repository (699 Python files, test corpora included): no new high or medium finding outside the test corpora. Removed: 10 LA10 mediums in research/ (a label builder, equal-weight benchmark averages of forward returns, next-date rebalance scheduling, a benchmark slice), one LA10 high in research/ (a long-format frame for an IC), and CO01 mediums by policy. All were false alarms on review. One high false alarm remains in research/: LA13 on an event-study diagnostic that reads bars after the event on purpose (phase0b.volume_peak_offsets; gone in v0.4). 831 files from an earlier internal codebase (first round; round 2 above has the final count): 0 added, 36 removed (32 LA10 mediums where the label flows into lgb.Dataset(X, label=y) and from there into the model and its outputs, or test labels averaged by prediction bucket; 4 CO01 by policy). On review the 32 were false alarms: training on a label is not target leakage, and whether the training rows overlap the test rows is LA05/LA14's job.

v0.6 on sealed set v9 (2026-10-17; pre-registered one-shot run under the revised gate, reproduced by the red team; docs/company/AUDITOR_EVAL_V9_PREREG.md). v9: 60 leaky, 105 plain correct and 55 near-miss correct files, written after v0.6 was frozen and never read by the builders. v0.6 (package 0.6.0): 30/60 leaks flagged (50%) at high/medium (easy 12/15, medium 12/25, hard 6/20), so it misses half the planted leaks (30/60); false alarms on 0/105 plain and 4/55 near-miss correct files (4/160 combined). With 105 plain files, a result of 0 is still consistent with a true false-alarm rate of up to about 3%. One of the 30 catches counts only because of a "same issue as line N" repeat of a finding on another line; without it, 29/60. The merged v0.5 on the same set: 24/60 flagged, with the same false alarms (0/105 plain, 4/55 near-miss). The hosted web check stays off: it still needs legal review. We changed this test's pass rule after two earlier runs. The old rule (no false alarms on any correct file) was not met: 3 of 80 correct files were flagged in one run and 4 of 90 in the next, all but one of them look-alike files. The new rule was fixed before the next test set was written. All runs are published (v6, v7, v8, v9), including v8 below, where the new rule was not met. One test set, written for us: a clean check_code result is weak evidence.

v0.5 on sealed set v8 (pre-registered one-shot run under the revised gate, reproduced by the red team; docs/company/AUDITOR_EVAL_V8_PREREG.md). v8: 60 leaky, 105 plain correct and 55 near-miss correct files. v0.5 and v0.4 gave byte-identical results: 19/60 leaks flagged (32%) at high/medium; false alarms on 1/105 plain and 3/55 near-miss correct files (4/160 combined). The gate (0 on plain files) is not met, so the hosted web check stays off. We changed this test's pass rule after two earlier runs; all runs are published (v6, v7, v8). A clean check_code result is weak evidence.

v0.6 lint (package 0.6.0): a recall sprint, measured in-sample only (2026-10-17). v2-v8 are development sets now: the builder scored them and read v4-v8's missed leaky files (and v8's false alarm clean_16) to group the misses into classes (docs/RECALL_ANALYSIS_v06.md: 196 missed files, 17 classes). The rules were then written from the classes, with the builder's own twins (tests/corpus_v06, 47 leaky, 33 clean) and acceptance criteria committed before the lint changed (tests/corpus_v06/PREREGISTRATION.md). Every number here is in-sample, and the dev-set gains are the most in-sample of all: the rules were designed after reading those misses. The out-of-sample test is sealed set v9 (above); these dev-set numbers are not.

Dev set (scored with its own rules) Leaks flagged, v0.5.0 → v0.6 Correct files flagged, v0.5.0 → v0.6
v2 37/50 → 37/50 0/30 → 0/30
v3 34/60 → 41/60 0/40 → 0/40
v4 14/60 → 30/60 0/50 → 0/50
v5 21/50 → 33/50 0/70 → 0/70
v6 22/60 → 36/60 0/80 → 0/80
v7 18/60 → 35/60 0/90 → 0/90
v8 19/60 → 35/60 (easy 9 → 11/15, medium 5 → 11/25, hard 5 → 13/20) 4/160 → 3/160 (plain 1/105 → 0/105; near-miss 3/55 → 3/55)
  • No leak flagged by v0.5.0 was lost on any dev set. The 82 new catches by rule: LA02 23 (two-sided transforms), SB01 12, LA05 12, LA14 7, LA09 6, LA07 5, LA06 5, LA12 4, LA11 3, LA04 2, LA13 2, LA01 1. Six of them (v4 leaky_36; v7 leaky_08, leaky_25; v8 leaky_06, leaky_42, leaky_47) count only because of a second-line (same_as) report of a leak the lint already flagged on another line, not because a new leak was detected.
  • Dev twins: leaks flagged 5/47 → 47/47; correct twins with a high or medium finding 2/33 → 0/33 (the two are the fixed false alarms below). 14 post-hoc look-alikes written after the fixes (tests/corpus_v06/posthoc_clean.txt) have no high or medium finding.
  • False alarms fixed: v8 clean_16 (two series resampled with the same rule, joined through derived names; LA12) and the count-only trade cap (red-team Y04: if len(trades) >= 50: break; LA13). The v8 near-miss false alarms clean_17, clean_27 and clean_141 remain. In the first build (b090f0e) clean_17 also gained a new LA07 high (a next-bar low held in a variable, read as this bar's low); fix F2 below removed it, so clean_17 is back to its one pre-existing LA10.
  • A finding is now also shown where the leak is written when that differs from where it was found (the negative shift inside a helper, the direction= keyword of a multi-line merge_asof, the max(...) after a parameter sweep, the pick from a dict of test scores). These repeats carry same_as (the line of the first finding), are labelled "same issue as line N", and are not counted in the score, the counts or the score breakdown.
  • One red-team "correct" file was wrong: tests/corpus_rt3/clean/c05_pad_lag.py used np.convolve(..., "same"), a centred moving average that reads 4 bars ahead. It now uses a trailing average like the other corpus files; the centred version is twin n07.
  • Every other corpus item (tests/, eval/, examples/, every red-team block): no high or medium finding lost against v0.5.0; gains only on leaky items. Rescans against v0.5.0: this repository, high/medium added only in leaky test corpora and the leaky example (a repeat at the max(...) line, scored once); the earlier internal codebase (831 files), nothing added or removed.
  • Speed: 200 KB in about 1.4-1.6 s; the red-team stress shapes 2.3-2.6 s (budgets: 3 s, 4 s, 8 s ceiling).
  • Red-team round 14 (docs/company/REDTEAM_LINT_V06.md, APPROVE WITH FIXES). The red team wrote 42 fresh files (tests/corpus_rt14_v06). Original probe result on b090f0e, before any fix (not in-sample): leaky 1/20 (v0.5.0) → 19/20; correct files flagged 2/22 (v0.5.0) → 8/22. Fixes:
    • F1: backtrader [k > 0] is reported only on line objects (self.data..., self.dataN..., self.datas[i]..., attributes assigned from bt.ind/bt.indicators/bt.talib or data lines); never through p/params or on attributes assigned from literals.
    • F2: a low or high held in a variable that was shifted (next_low = low.shift(-1)) is not this bar's low.
    • F3: for transforms with a bounded look-ahead (np.gradient 1 bar, argrelextrema order, find_peaks distance, savgol half-window) lags are added up along the flow: a signal is causal once lagged by the look-ahead, and a position booked against the same bar's return needs one bar more (else high; possible when fully lagged). Calls inside a function passed to rolling(...).apply/expanding(...).apply are not reported.
    • F4: a period aggregate, or a transform, that reaches only PnL-named keys (a printed daily PnL total) is a report; PnL counts as a decision only when a reached value is multiplied with a return as a position.
    • F5: helper call sites, statements by the names they read, products and assignments by key are indexed once. The two red-team shapes (12.5 s and 7.0 s on b090f0e) and a third found while fixing (limit compares, 5.4 s) now take 1.3-1.4 s at 200 KB and are in the speed tests.
    • After the fixes, in-sample (the rules were changed with these files in view): rt14 leaky 19/20, correct files flagged 2/22 (the two pre-existing v0.5.0 findings, q10 LA10 and q18 LA05, out of scope). Still missed: k09 (a universe from last_valid_index). Dev sets after the fixes: same numbers as the table above, no lost catch.
    • Precision note for F3: v3 leaky_12 (np.sign(np.gradient(ma)).shift(1) * ret) shows why one lag isn't enough: the slope at t-1 still uses bar t, the bar whose return is booked.
  • Red-team round 15 (tests/corpus_rt15_v06, 12 leaky, 9 correct; original result on f270645, not in-sample: leaky 8/12, correct flagged 3/9). Fixes: B1, F4 had made PnL-named results a report even when a price or signal aggregate fed PnL, fills or exits; now only an aggregate or smoothing of PnL/equity is a report, and fill/exit-priced (exec) and PnL results count as decisions otherwise; a feature-named intermediate counts only when it reaches fit/predict. B2, backtrader indicators are resolved from the imports (import backtrader.indicators as btind, from backtrader.indicators import RSI), a local alias of a data line (o = self.data.open; o[1]) is a line, and an attribute built from a list of feeds (sorted(self.datas)) is not. After the fixes, in-sample: rt15 leaky 12/12, correct flagged 1/9 (n04, an LA10 already on main); rt14 leaky 19/20, correct 2/22. Dev sets unchanged, no lost catch.
  • Not attempted (they need information the code does not carry, or would cost precision): point-in-time joins with config-driven directions or relabelled periods, same-bar products hidden in wrappers, target encoding, calendar and time-zone timing, corporate actions, interpolation before returns (still "possible"), z-scores inside classes or tensors, reversed expanding windows. Next: the sealed v9 run and a red-team pass on these rules.

v0.4 on sealed set v7 (pre-registered one-shot run, reproduced by the red team; docs/company/AUDITOR_EVAL_V7_PREREG.md). v7: 60 leaky and 90 correct files (55 plain, 35 near-miss), written outside the build and never read by the builders.

  • v0.4 (7d22613): 17/60 leaks (28%) flagged at high or medium (easy 9/15, medium 6/25, hard 2/20); 4/90 correct files flagged at high/medium: 0/55 plain, 4/35 near-miss (three LA13 loops, one LA07).
  • v0.3 on the same set: 16/60 (27%) and 5/90 (0/55 plain, 5/35 near-miss).
  • The precision gate for the hosted web check (0 high/medium false alarms) is not met. v0.4 merged because it beats v0.3 on v7, as the pre-registered rule allows. Recall on v7 is low: v7 deliberately covers many leak types outside these rules. A clean check_code result is weak evidence.

v0.5 lint (unreleased; package version still 0.4.0): the four v7 false alarms, measured in-sample only (2026-10-17). The coordinator described three classes from the v7 review; the builder did not read v7, any heldout_* folder or v8 (written later, sealed). Acceptance criteria and dev twins (tests/corpus_v05, 8 leaky, 10 clean) were committed before the lint changed (tests/corpus_v05/PREREGISTRATION.md). Every number here is in-sample.

  • Loop order simulation (LA13): a resting limit, stop or take-profit order placed at bar i from past data and checked against bar i+1's low or high is exempt when the fill or exit is priced from the order's level; "possible" when it isn't. A next-bar high or low that decides the order, a next-bar close in the fill condition, or sizing from the fill bar's range stays high.
  • numpy label loops (LA13): the label-only exemption accepts numpy summaries and files (np.unique, np.bincount, np.histogram, np.save/savez/savetxt, .tofile, nan-aware statistics, np.count_nonzero(labels == 1)) and missing-value masks (~np.isnan(labels)). A product, a signal or a feature still breaks it.
  • Lags inside groupby transforms (LA07): transform(lambda s: ... .shift(1)), .apply, .pipe and lag helpers count as a lag (a helper whose every return is <expression of a parameter>.shift(k), k > 0, with no lead inside); a frame used only for groupby(...)[col] no longer cancels a lag.
  • Dev twins, v0.4 (main at a8d57fc) then v0.5: leaks flagged 12/12 → 12/12; correct files with a high or medium finding 10/12 → 0/12. The last six twins (addendum 1) came from the coordinator's scoring of the first build (d0dbece) on the dev sets: no lost catches, 0 false alarms on v2-v6, v7 false alarms 4 → 2, both loop order simulations in idioms the first build missed (a simulation inside a function over precomputed signal indices with a dataclass config, and the next bar read through a row alias nxt = bars.iloc[t + 1]). v7 is a dev set from that scoring on, so v7 numbers are in-sample too.
  • Every other corpus item (tests/, eval/, examples/, every red-team block): no high or medium finding lost or added against v0.4. Rescans against v0.4: this repository and the earlier internal codebase, no high or medium added or removed outside tests/corpus_v05.
  • Speed: 200 KB in 1.2 s; the red team's worst-case shapes (rt10) about 2.3-2.7 s, the 262 KB stress file about 2.5 s. Two older scans were made cheaper on the way (the test-set selection loop check now runs its cheapest filter first; parameter defaults use a one-pass index).
  • Red-team round 12 (BLOCKED at e07293a; tests/corpus_v05/rt12_*.txt, 24 leaky and 9 correct blocks, added before the fixes): the loop-order exemption did not enforce what this README claimed. Fixed: R01/R02 (signal indices from future data), R03/R04 (fill or exit values feeding decisions), R06 (a row-alias next-bar low deciding a new entry), R11 (fills priced at the next bar's low), S02 (labels saved and loaded back as a signal), and the pre-existing false alarms K01 (buy-stop entries), K02 (trailing stop in a while loop), K07 (a lag helper returning (a - b).shift(1)). In-sample: 23/24 leaky blocks flagged (R10 is a deliberate exemption, above), 0/9 correct blocks flagged. The exact enforced conditions are under Known limitations.
  • Red-team round 13 (BLOCK at bc95ea6, narrow; tests/corpus_v05/rt13_leaky.txt, 8 leaky blocks, added before the fixes): fixed X05/X07/X08 (lists holding exempt fill or exit values read in a decision), X02 (signal indices through self.<attr>), and the agreed R10 conditions (far side of the open, booked at the level). In-sample: 8/8 flagged.
  • Perf: the cyclic garbage collector is paused during a lint run (the run makes no reference cycles; in a long-lived process GC passes over a large heap could double the time). The red-team stress shapes (232-268 KB) have a 4 s local budget with the median of three runs when close; the 200 KB budget stays 3 s; every run must stay under the 8 s web check timeout. On this laptop v0.4 and v0.5 time the same on these shapes; both vary by about 1 s with load.
  • Not attempted: the documented misses H2 and G13. Not done here: the sealed v8 run and the next red-team pass.

v0.4 lint: documented misses turned into rules, measured in-sample during development (2026-10-16). Nine items: six from the v0.3 backlog (same-bar open fills, normalization inside apply/transform, pivot keeping the forward flag, query/eval strings, np.copysign, LA13 false alarms on our own code) and three false-alarm classes the coordinator described from a fresh sealed set (fill-condition models, label-only loops, bar construction with default resample labels). The builder never read that sealed set or any red-team probe folder. The acceptance criteria and the dev twins (tests/corpus_v04) were committed before the lint changed (tests/corpus_v04/PREREGISTRATION.md; the three sealed-set classes are its addendum 1, committed before the fix). Every number below is in-sample: the rules were changed until these files passed, so they show the rules work on the patterns as described, not how v0.4 does on new code.

  • Dev twins, v0.3 (main at b5e8d1c) then v0.4: leaks flagged as expected 13/41 → 41/41; correct files with a high or medium finding 13/48 → 0/48. The six backlog items: 5/31 → 31/31 and 5/36 → 0/36. The three sealed-set classes: 8/10 → 10/10 and 8/12 → 0/12 (v0.3 flagged all three fill models, the triple-barrier labelers and the minute-bar builder). The last six twins (addenda 2 and 3) came from the coordinator's scores of this PR on the held-out sets:
    • Addendum 2: a labeler whose output reaches only summaries (y.value_counts(normalize=True), describe, plot, to_csv) was still LA13 high. The exemption now accepts summary and file sinks as well as labels, as long as there is label evidence (a label-named output, call-site target or function name) and no feature, signal, position, product or model use.
    • Addendum 3: the caller also aligned the features to the labels' dates (X = feats.loc[y.index].dropna()). Index-only reads (y.index, reindex(y.index), len(y), y.shape) are now neutral; value reads (X["lab"] = y, concat([X, y]), y.values) still break the exemption.
  • Red-team round 10 (blocked 6659762; tests/corpus_rt10, 40 leaky and 22 correct blocks, added before the fixes): four regressions against v0.3 fixed (L1, L3, G2, G7), plus H1, L2, L5 and F3 (a missing-data check on a later bar no longer stops the scan of the later conditions in the same loop). 0/22 correct blocks flagged. Four documented misses (H2, H5, J5, Q1; see Known limitations). These blocks are now in-sample too.
  • Red-team round 11 (APPROVE at 1fb35fc on one condition; tests/corpus_rt10/r11.txt, 12 leaky blocks): G9, an exit priced at the next bar's high through a shifted multi-column frame, was only "possible" (v0.3: high) and is high again. 11/12 flagged; G13 (a numpy fill model with offset slices) is a documented miss, also missed by v0.3.
  • Held-out sets, scored by the coordinator (not seen by the builder) on earlier builds of this PR: no lost catches against main; 0 false alarms on four sets; on the set that produced the three classes, 3 → 1, the label-only case that addenda 2 and 3 target. The final code has not been re-scored.
  • Two of the v0.3 false alarms on the clean twins were in existing rules the twins hit by accident, and are fixed rather than the twins edited: a forward column joined onto a feature frame that is fitted only through an explicit column list that doesn't name it (LA10 medium → possible), and a decile spread of a wide forward frame selected with a past-only mask (fw[q > 0.9].mean(axis=1), LA10 high → possible).
  • Red-team repros caught since v0.4 (they were documented misses): pivot(values="fwd_ret") (rt6 A04, A05), np.copysign (rt7 E01), frame.eval("t * b") (rt7 E04), frame.query("m > 0")["b"] stored as strat (rt9 W06, medium).
  • Every other corpus (all leaky and clean files and red-team blocks in tests/, eval/ and examples/, 735 items): no high or medium finding was lost and none was added, compared with main.
  • Rescan against main. This repository (841 Python files): removed the two LA13 highs on our own code (research/nba_research/phase0b.py volume_peak_offsets, now clean because s + lo with the default lo=-2 is the past; auditor/nba_auditor/path_audit.py:84, a character loop over a string). Added: none outside tests/corpus_v04. 831 files from an earlier internal codebase: 0 added; 2 LA13 highs became "possible" (two sweep scripts skip a rebalance when its realized return is NaN: a missing-data check on the booked outcome, not a signal input; the "possible" keeps the delisting caveat). Still high on our own code, documented rather than exempted (no principled rule found): research/nba_research/data/resolver.py:409 (interval bookkeeping that reads the next plan entry) and research/nba_research/recorder/house.py:152 (the bad-print check, which looks at the next session to see whether a spike reverses). auditor/tests/test_trade_log.py:100 is a deliberate leak in a test and stays high.
  • Speed: 200 KB of code in 1.2-1.3 s (v0.3: 1.1 s) on the company laptop; the red team's worst-case shapes are under Known limitations (performance guards).
  • Not done here (merge gate, RETROS #23): the result on a sealed set this build has never seen, and a red-team pass that writes new cases aimed at each new exemption (the fill model, label-only loops, same-bar resample columns, string loops, missing-data checks, parameter defaults, the literal-column-list downgrade, bucket pivot tables).

Our own corpora (in-sample: they were used during the build). Numbers below are as of v0.2; the CO01 cap moves the cost-only files from medium to "possible", as listed above.

  • A separate agent that never saw the lint code wrote 35 leaky and 25 clean look-alike snippets from the category list (tests/corpus_v02_holdout). First measurement, before any fix it prompted: 31/35 leaks flagged at high or medium (v0.1: 11/35); 4/25 clean files had a high or medium finding, all existing v0.1 false alarms (v0.1: 5/25). Now 35/35 and 0/25.
  • Development set (tests/corpus_v02, 54 leaky, 60 clean): 54/54 and 0/60.
  • Red-team review of PR #8 (tests/corpus_rt3, 35 leaky, 87 clean): 16/35 and 0/87 (v0.1: 5/35, 15/87).
  • The 173 Python files in this repository: the only new high or medium findings are the two bugs in examples/leaky_to_clean that v0.1 missed. 837 files from an earlier internal codebase: v0.2 adds 8 (early stopping on the test year in two training scripts, EPS merged by fiscal period end, four return calculations with no costs next to performance figures) and drops 11 v0.1 findings that were false alarms on review.

What this means:

  • The auditor rarely returns "No red flags found" on noise, as long as n_trials is honest. Understate it and the audit is fooled the same way you are. check_code can count a visible parameter grid for you (LA11).
  • Market exposure isn't skill. A leveraged index never gets "No red flags found": without a benchmark, because of the benchmark rule; with one, because it has no alpha.
  • Five years of data often can't confirm even a genuine edge. A single strategy with a true Sharpe of 1.0, audited with n_trials=10, reads as consistent with selection noise about a third of the time. That is the honest answer for that much data, not a bug. Read "Major red flags" as "this evidence is consistent with noise, selection or costs", not as proof that the strategy has no edge.

Provenance. v0.1's first thresholds were not tuned to this table. Two independent red-team passes found verdict bugs, including autocorrelated returns, the benchmark being ignored, the default frequency, intraday series and a hard-coded cost rule. The fixes changed the verdict rules, and the table above was re-run after them. All of it is in the git history; there was no separate pre-registration file.

Install

cd auditor
python -m venv .venv            # Python 3.12
.venv/Scripts/python -m pip install -r requirements.txt     # (Linux/macOS: .venv/bin/python)
.venv/Scripts/python -m pip install -e . --no-deps
.venv/Scripts/python -m pytest -q

On the company machine, prefix pip with env -u SSL_CERT_FILE -u SSL_CERT_DIR. The antivirus also corrupts PyPI's /simple/mcp/ page; if pip install mcp fails with InvalidChunkLength, install the wheel URL from https://pypi.org/pypi/mcp/json directly.

Use as an MCP server

The server is nba-auditor and runs over stdio. Start it with the nba-auditor-mcp command, nba-auditor mcp, or python -m nba_auditor.mcp. It is built on the official MCP Python SDK v2 (mcp.server.MCPServer, the v2 name of v1's FastMCP). All tools are marked read-only.

Setup for Claude Code, Claude Desktop, Cursor, VS Code and other clients: docs/quickstart.md (every snippet there is tested in CI). The short version for Claude Code:

claude mcp add nba-auditor -- /absolute/path/to/.venv/bin/nba-auditor-mcp
# Windows: C:/path/to/.venv/Scripts/nba-auditor-mcp.exe

Available after release (the PyPI package backtest-auditor isn't published yet; POLICY_DECISIONS #21-24):

claude mcp add backtest-auditor -- uvx backtest-auditor

and {"command": "uvx", "args": ["backtest-auditor"]} in Claude Desktop's or Cursor's mcpServers. With no arguments the backtest-auditor command starts the MCP server; with arguments it is the CLI.

Then ask, for example: "Find look-ahead bias in this repo with check_path", or "Audit this backtest with audit_backtest: strategy.py, backtest.csv, spy.csv as the benchmark, n_trials=40, turnover 0.3."

A full worked example (a leaky strategy, its audit, the fix and the re-audit) is in examples/leaky_to_clean.

Use as a library or CLI

from nba_auditor import check_code, check_path, audit_backtest, audit_returns, audit_trade_log, explain

rep = check_code(open("strategy.py").read())
print(rep.to_markdown())               # or rep.to_dict() for JSON

res = check_path("strategies/")        # dict: per-file findings, line ranges, suggested_fixes, skipped
res = audit_backtest("strategy.py", "returns.csv", benchmark_csv="spy.csv", n_trials=40, turnover=0.4)
print(res["verdict"], res["verdict_note"])

rep = audit_returns("returns.csv", freq="D", turnover=0.4, n_trials=40, benchmark="spy.csv")
print(rep.verdict, rep.reasons, rep.score)

python -m nba_auditor check strategy.py
python -m nba_auditor path strategies/ --json      # check_path
python -m nba_auditor backtest --code strategy.py --returns returns.csv --benchmark spy.csv --n-trials 40
python -m nba_auditor returns returns.csv --n-trials 40 --turnover 0.4 --json
python -m nba_auditor trades trades.csv --prices closes.csv
python -m nba_auditor explain LA07
python -m nba_auditor mcp          # the MCP server over stdio

Input conventions

  • Returns are simple per-period decimals (0.01 = 1%). Use units="percent" for percent values, or input_kind="equity" for an equity curve.
  • freq defaults to auto (inferred from the dates). A stated frequency that contradicts the dates is an error.
  • Turnover is one-way: sum of |change in weights| per period. Cost in bps is charged per unit of turnover.
  • Returns are treated as gross unless returns_are_net=True. When turnover is given without a cost, the headline result charges 10 bps.
  • n_trials counts every variant you tried (parameters, universes, rules). By default the DSR assumes the trials' Sharpe ratios vary the way zero-skill strategies would (variance 1/T). If you know the actual variance, pass it as trials_sr_variance (annualized).

Known limitations (v0.3, with the v0.4, v0.5 and v0.6 lint changes marked)

  • v0.5 loop order simulation: exactly what is exempt (after red-team round 12). A condition in a bar loop over i is not reported only when all of these hold; otherwise it is judged as before (high).
    1. Every term is either past-only (no read of a later bar, no helper function defined in the file, no name tainted by a later bar's data other than bar i+1's open) or a fill comparison: bar i+1's low or high, exactly one bar ahead, read directly (lo[i + 1], df["low"].iloc[i + 1]) or through a next-bar row alias (nxt = bars.iloc[t + 1], also .loc and rows[k + 1]; not a single price copied into a variable, nl = lo[i + 1]), against a level known before that bar's high and low (past data, or bar i+1's open).
    2. The fill comparison points the filling way: the low at or below the level (buy limit, sell stop) or the high at or above it (sell limit, take-profit, buy stop). The other direction (nxt["low"] > level) means the order did not fill: it is allowed only as a skip (continue, break, pass), never to book a trade (rt12 R06).
    3. The branch prices the fill or exit from the order's level (min(limit, o[i + 1]), max(stop, o[i + 1]), exit_px = tp, the limit itself). A price at a later bar's low or high itself (c[i + 6] / lo[i + 1]) is not exempt (rt12 R11). Later-bar reads in the branch must be prices assigned to fill, exit or trade-log names, plus the fill's timestamp (bars.index[i + 1]); a size, weight, volume or position from a later bar is not exempt.
    4. Fill and exit values (and names derived from them) reach only append/extend, a return value, or a print/log/file call; never a condition, a comparison, or another variable used that way, in this iteration, a later one or after the loop (rt12 R03 cooldown on the last trade's return, R04 keeping only winners). The list they are appended to inherits this (rt13): it may reach only summaries and sinks (mean, len, sum, print, a DataFrame/Series, a return value, a file, and names built from those). A read of it in an if/while test, a comparison, a comprehension filter, a loop over it, or a break/continue condition (not trades or trades[-1] > 0, if sum(trades) < -0.2: break, [t for t in trades if t > -0.02]) removes the exemption, and LA13 is reported high at that line. Reads of its size are not decisions (v0.6, the count-only trade cap): len(trades) anywhere, and truthiness tests (if trades, not trades, an and/or operand that is itself only a test), because how many fills have happened so far is known in real time.
    5. Decided from past data: an enclosing past-only condition, an and term that reads past bar data (not in_pos alone is state, not a decision), an elif of another fill check, or a loop over precomputed signal indices (for i in idx, not range) where nothing forward-looking feeds the indices, followed through names, function parameters to their call sites, object attributes (self.idx = ... in a class, s.idx; every assignment to that attribute name is checked, and an attribute with no visible assignment gives no exemption; rt13 X02), np.flatnonzero, np.where(...)[0], .nonzero(), .index, comprehensions, and positional slices that start at different bars (c[5:] / c[:-5]; rt12 R01, R02). If the indices are tainted there is no exemption at all. An unconditional resting order on every bar (limit = c[i] * 0.99 each bar, or a buy-stop at the 20-bar high, rt12 K01) counts as decided only when its fill is priced from its level.
    • When conditions 1-4 hold and the order was explicitly decided but the fill isn't priced from the level, the finding is "possible".
    • Deliberate (agreed with the red team in round 13): a level set from the fill bar's open (limit = o[i + 1] * 0.99, rt12 R10, directly or through names assigned earlier in the same branch) is exempt only when it sits on the far side of that open (below it for a buy, o * k with k < 1 or o - x; above it for a sell or take-profit, k > 1 or o + x) and the fill is booked at the level itself, not min(limit, o[i + 1]). Otherwise it is high. A stop derived from an entry at bar i+1's open and checked inside an inner loop over later bars (a trailing stop, rt12 K02) is not reported by the outer loop.
    • Not exempt: event studies and order simulations that check bars beyond i+1.
  • v0.6 rules lean on names and visible flow. "Reaches a signal, position, PnL or model input" is decided by following assignments (and helper return values) to names the lint classifies as signals or PnL, or to model inputs. A transform that feeds a decision through names it can't classify stays "possible". The survivorship patterns only fire on price- or return-named wide frames, and the full-sample estimate rule (LA05) only when the file has no split, training window or walk-forward names and the positions are booked against returns. Two-sided transforms re-run on data up to each date inside a loop are treated as one-sided without checking the window.
  • Static analysis can't see everything. Measured recall at high or medium severity on leaky snippets the builder had not seen (second red-team pass, 2026-10-16):
    • Held-out set 1 (32 leaky, 16 clean), before fixes: 15/32 (47%), 1 false alarm in 16 clean files.
    • Held-out set 2 (12 leaky, 8 clean), written after the set-1 fixes and measured before any further change: 8/12 (67%), no false alarms.
    • Both sets are now regression tests (tests/corpus_rt2). v0.2 and v0.3 flag all 44, but that is in-sample. v0.2's sealed-set numbers are under "Calibration" above: 60-78% recall, which still means a clean check_code result is weak evidence. v0.3 on sealed set v5: 0/70 false alarms and 21/50 (42%) recall, with recall about 40% (contaminated by two leak classes fed back during review; precision is not contaminated). See "v0.3 on sealed set v5" above.
    • Still missed (from the PR #8 red-team review and our own tests):
      • LA14: hyperparameter search with Optuna or set_params scored on the test set, and selection through a helper that returns the test Sharpe.
      • LA15: period ends renamed to date before the merge, to_period keys, period ends read from the index.
      • SB01: dropna(axis=1) on a wide price frame, last_valid_index, keeping names with a full count of dates.
      • LA12: pd.Grouper(freq=...) bars joined back.
      • LA02: iloc[::-1] on a whole frame (often a fix for newest-first files, so it is not flagged).
      • LA09: qcut/rank/scale on a column with no visible time-series operation in the file (we can't tell a time series from one date's cross-section), describe() quantiles.
      • CO01: a cost basis used for something other than trading costs can hide the finding.
      • Limit fills at the day's low, resampled bars realigned in another file, and leaks hidden in other files or in the data itself.
  • v0.2 trades some recall for precision. These are deliberate choices:
    • Full-sample statistics inside functions count only when the column is visibly a time series (built with rolling, shift, pct_change and similar in the same scope). Functions run per date (groupby('date').apply, loops over dates) are skipped. A cross-sectional scoring function that ranks one date's stocks is not flagged.
    • Same-bar close fills in loops are medium and only for entries. Exits at the close on a close-based stop are common and close to executable, so they are not flagged.
    • vectorbt's default fill at the close is still "possible"; only an explicit price=close is medium.
    • Anything we can't line up is "possible" at most: positions written through masks or variable offsets, engine settings passed through **kwargs (they may carry the fees), resample labels passed through **kwargs, and exact merges on a period end that are not forward-filled afterwards.
    • LA15 counts a reporting lag only when it is applied to the period-end column or index itself: f['period_end'] + <offset>, f['period_end'].add(<offset>), assign(period_end=... + <offset>), f.index = f.index + <offset> (or +=), or f.shift(n, freq=...) / tshift(n) with n > 0. An offset is a positive pd.Timedelta, DateOffset, pd.offsets.* (e.g. QuarterEnd(1), BDay(45)) or to_timedelta, or a name bound to one. Not lags: merge_asof(..., tolerance=...) and other keyword arguments, zero offsets that only snap a date (MonthEnd(0)), negative or zero shift(freq=...), and other index changes (tz_localize, tz_convert). Early stopping on a set named holdout/oos is not flagged when a separate test set is scored, and loops that redefine the test window each iteration are treated as walk-forward evaluation.
  • v0.3 trades more recall for precision (LA05, LA10, CO01). Deliberate choices, each a possible miss:

    • LA01/LA10: a value computed from forward data under a neutral name (up, oo, tmp) is high if it reaches a signal or position, and medium if it reaches a model input. Model inputs are: the first argument of fit/predict, a feature-named assignment, the return value of a feature builder, a top-k pick (nlargest, idxmax), or a PnL product as a comparison. Used some other way, it is "possible" (v0.2: medium).

    • LA10: forward data used as the outcome is not an input: groupby(...)[fwd].mean() and other aggregations, df.loc[mask, fwd] / df[mask][fwd] with a mask built without forward data, and position * fwd_ret. Results built from these are "possible" when their name only suggests a leg or a spread (long_leg, spread, trades) and are not reported otherwise. An outcome is an input again, and high, when it is:

      • put back onto rows by map/merge/reindex/transform, a column assignment or a comparison;
      • assigned to a signal, position or weight name (position, weight, score; not names that only say long, short or trade);
      • a non-return factor (rolling statistic, sign, rank, clip, ratio) in a product with a return. Products are *, @, .mul/.multiply, np.multiply, np.dot, and np.where(c, r, -r) / r.where(c, ...).

      Also high: a transform written inside the factor (bench * np.sign(mkt), np.sign(fwd.rolling(5).mean()) * fwd), and a return series flipped or overwritten on rows chosen from forward data (strat[tilt < 0] *= -1, strat.loc[tilt < 0] = -bench).

      Missed:

      • an outcome read back some other way (agg.loc[d] inside a loop);
      • a product inside a helper function (backtest(w, r), rt6 A06).

      Caught since v0.4 (were missed in v0.3): pivot(values="fwd_ret"), np.copysign(bench, mkt), frame.eval("t * b") and frame.query("m > 0")["b"] under a strategy name.

    • v0.4 query/eval: only literal strings are read (df.eval(expr) with expr a variable is not), one statement per string. A frame built once from a dict literal maps its column names back to the source expressions; other frames read name as frame["name"]. A query is a row filter like frame[cond], so an inline future-data condition is judged like bench[mkt > 0]: not counted unless the result is stored, unaggregated, under a strategy name (strat, strategy, pnl, equity, backtest, bt, wealth; not perf, port or net), which is medium.

    • v0.4 pivot: the values= column (keyword, or positional for pivot_table) is read. A pivot_table whose index or columns name buckets (q, decile, quintile, bucket...) is an outcome table, like groupby(q)[fwd].mean().

    • v0.4 fill model (LA01/LA10): only the comparison inside the fill mask is exempt. The mask must be <order decided without future data> & (<next bar's low or high, exactly one bar ahead> <op> <a price known at t>), and every use of the mask must select, zero or count trade returns (r[filled], r.where(filled), np.where(filled, r, 0), filled.sum()). Any other use (a signal, a position, a feature, a fit, astype, a shift) makes it an input again, as before. Every other read of the next bar's low or high is still an input: sizing from limit_px - low.shift(-1) next to the fill mask is LA10 high (rt10 G2), and an entry or exit priced at low.shift(-1) / high.shift(-1) is LA01 high (rt10 G7), also when the price is written inline in the trade return selected by the mask. A column read from a frame shifted as a whole (best = df[["low", "high"]].shift(-1), then best["high"], best.loc[:, "high"] or best.high) is read exactly like that column shifted by -k, for the price rule, the sizing rule and the mask exemption alike (rt11 G9). A two-bar-ahead low, a fill flag without an order term, or a fill flag with a fill-sounding name (will_fill) is still flagged. Missed: a numpy fill model written with offset slices (lo[1:] <= lim[:-1], sizing from lo[1:], rt11 G13).

    • v0.4 LA13: a slice whose start is in the past and end in the future (x[i - 2: i + 3]) is not read as a future read (unchanged from v0.3). The label-only exemption follows each loop output to its uses: returned by a function (optionally wrapped in pd.Series(...)) whose every call site assigns a label-named target, one passed as a label, or any non-signal target when the function's name says label or target; and from there only into fit's y, label=, a score, a label- or evaluation-named name, a summary (value_counts, describe, mean, groupby(...).mean(), plot) or a file (to_csv, to_parquet), with no product or comparison on the way. Index-only reads (X.loc[y.index], reindex(y.index), len(y), y.shape) are alignment and don't count. Since red-team round 10, a label written as a column (px["target"] = labeler(...)) is followed through its column uses exactly like a name, and a bare call statement is a sink only when it is a print, plot or file write: positions.extend(labels), .append, .insert, .update and df.insert(1, "lab", labels) are writes into the other name or column, which is then followed in turn (rt10 L1, L2, L3; L5, a where on the labels, is caught too).

    • v0.4 performance guards: call sites, name reads, the statements that read each name or column, label uses and the "feeds a signal" graph are each indexed once per file, and the open-fill and fill-mask rules skip files or values that can't match. The red team's worst cases (rt10: 1,100 labeler functions at 232 KB, 1,600 label loops at 247 KB, 1,600 fill models at 268 KB) run in about 2.5 s on the company laptop, and their 262 KB symbolic/container stress file in 2.3 s, the same as main (main: 2.2-2.6 s for the first two; the red team measured 165 s on main for the fill models, which hit a quadratic scan that predates v0.4). tests/test_redteam_rt10.py times each shape against the 3 s local budget and the 8 s ceiling.

    • Still missed after round 10 (documented, KNOWN_MISSES in tests/test_redteam_rt10.py):

      • H2: a loop with no condition that writes forward values (label_max[i] = c[i + 1: i + 21].max() / c[i] - 1) under a label name, then used as a feature. It isn't LA13's pattern, and the loop's output isn't tracked as forward data.
      • H5: a helper loop that writes forward values, smoothed into a signal outside the helper.
      • J5: forward columns merged in from a second frame (px.merge(labels, on=[...])), then listed as features.
      • Q1: strategy = df.query("fwd_ret > 0") followed only by a mean, which reads like a capture statistic. Anything else (a feature, a signal, a product with a return) keeps the high. Event studies that branch on post-event values are flagged.

    • v0.4 LA09 in apply: the receiver must be visibly a time series (built with rolling, shift, pct_change and similar here or where its base frame was assigned) and feed a signal, a feature or a model for medium; otherwise possible.

    • LA10 containers: a frame built from columns that stores a forward column is not reported where it is built. It is reported (high) where a forward column survives into fit/predict, a training dataset, a feature matrix or a signal.

      • Names come from dict keys, .rename("x") and keys=[...]. They are kept only through two kinds of change:
        • operations known to keep names: row filters, dropna, fillna, sorting, index changes, astype, copy, to_numpy, assign, and a drop that is resolved by name;
        • renames with literal names: rename(columns={"a": "b"}) (also with inplace=True) and data.columns = ["a", "b"] while the column order is known.
      • Any other frame-level operation on a container makes its column names unknown. This includes in-place calls, add_prefix, set_axis, rename(mapper, axis=1), rename(columns=lambda ...) and .columns = .columns.str.replace(...).
      • A forward column with an unknown name survives every later explicit column list or drop.
      • Positional selections (iloc[:, :-1], arr[:, -1]) are resolved while the column order is known. The order is updated or dropped as columns move:
        • a literal column list (data[["c", "a", "b"]], .loc[:, [...]], reindex(columns=[...])) sets the order to that list;
        • a resolvable positional selection (iloc[:, ::-1]) reorders it;
        • any other column-axis change makes it unknown: sort_index(axis=1), a transpose (.T), a column subscript that isn't a literal list (data[sorted(data.columns)], data[data.columns[::-1]]), a non-literal reindex, or any operation the lint can't read. With the order unknown, a later positional selection keeps the forward column.

    • LA10 masks: a return selected by a mask built from future data (bench[long], bench.loc[short], bench[~m], also through .values, .to_numpy() and m[m].index) is high when the mask has a position name and medium when it has a neutral one (ok, up). An inline comparison (bench[mkt > 0]) is not counted, so up/down capture statistics stay clean; a neutral-named capture mask gets the medium.

      • A row shift of the whole frame (data.shift(5)) lags every stored column. A forward column whose lead is fully cancelled (a 5-bar lead, shift(5)) no longer counts; a partial lag (shift(1)) still does.

    • Look-ahead cancel scope: a lead and a lag written with the same expression (shift(-h) then shift(h)) cancel only when every name in it is bound once and is not a loop variable. A module constant shadowed by a function parameter or local is not used. Missed: a helper defined above the line that creates the forward column it reads (the pass runs in source order).

    • LA10: a model trained on the label (fit(X, y), lgb.Dataset(X, label=y), xgb.DMatrix, Pool) no longer passes "future data" on to the model and its outputs. Whether the training rows overlap the evaluated rows, or end less than a label horizon before them, is LA05/LA14's job, and fits inside functions are not checked there.

    • Look-ahead size: constants (H = 5, cfg.horizon) and simple arithmetic (-H, H + 1, -1 * 5) are resolved. Any other expression cancels only the same expression (shift(-horizon) then shift(horizon)). A different unknown lag counts as 1 bar.

    • LA10 column lists: drop lists and feature lists are resolved per column (string lists, +, single-assignment constants, comprehensions over df.columns with startswith/endswith/in/==/re.match, and df.columns.str.startswith/isin masks). A list the lint can't resolve (built in a loop, read from a file, reassigned) gives "possible" for the forward columns it may keep, never high.

    • LA05: a slice or date mask bounded above (.loc[:cut], .iloc[:k], [:k], .loc[a:b], df.date < cut, between) counts as a training subset, as does a frame assigned once from one. A start filter (.loc['2000':], [20:], df.index >= '2000') or a test-window mask does not. The lint does not check that the evaluation uses the later rows. Missed as a result:

      • an in-sample score on the same slice;
      • fits on all rows but the last (.iloc[:-1]), which count as subsets;
      • overlapping train and test slices (iloc[:3000] vs iloc[2500:]);
      • a model fitted and scored on the same iloc[:-1] rows.

    • Also still missed from the round 5 review: a forward-filtered ticker list used through isin, an inplace=True drop on a different frame than the one fitted, and a forward column re-joined under another name after a drop.

    • CO01: never above "possible" (3 points off the score). Missing costs are a realism caveat, not a lookahead error, and the returns may already be net.

  • Suggested fixes are narrow and change behavior on purpose. shift(-1) becomes shift(1), which uses older data than the original author may want; TimeSeriesSplit gives different folds than KFold. The self-check only shows that the lint no longer sees the pattern and sees nothing new; it doesn't run the code. Fixes aren't offered for notebooks yet. The agent-fix eval is in-sample.
  • check_path resolves relative paths against the server's working directory, which depends on the client; agents should pass absolute paths. It has no scope limit: it reads any local .py/.ipynb file or folder the user running the server can read (it never writes). Network (UNC) and device paths are refused, for check_path and for every data-file argument.
  • Results quote file content. Paths, snippets and suggested edits come from the files and are marked content_is_untrusted_data; a file can contain text written to look like instructions. Snippets have comments stripped. Data files (.csv, .json, .txt only) are never quoted in error messages.
  • The name heuristics can be fooled. A label called signal will be misread. Test sets are recognized by name (X_test, test, oos, holdout), validation sets by val/valid, and fundamentals dates by name.
  • It reads Python only. Pine Script and other languages are not supported yet.
  • The verdict depends on n_trials being honest (see the calibration table).
  • Trade-log survivorship checks are heuristics, not proof. They flag symbols traded before the current company used them (some symbols were reused, e.g. ARM, SNOW, CEG, META) and universes where no name ever disappears. The built-in list of first-trade dates is short.
  • Trade logs without daily prices are only marked when positions trade. Their Sharpe is overstated, so the verdict can never be "No red flags found".
  • The cost model is linear in turnover. No market impact, borrow fees or financing.
  • Intraday series are reviewed but never get "No red flags found". The implausibility thresholds assume daily or slower bars.
  • Verdict labels were renamed from "Likely real / Inconclusive / Likely artifact" to "No red flags found / Some red flags / Major red flags" (CEO decision: "Likely real" implied an edge). A compliance-checker pass on 2026-10-16 edited the disclaimer and several explain texts; public launch copy still needs its own pass.
  • Input limits. check_code accepts up to 1,000,000 characters; MCP data arguments up to 20,000,000 characters each. There is no wall-clock timeout; these limits keep a run to seconds on a laptop.

References

  • Bailey & López de Prado (2014), The Deflated Sharpe Ratio, JPM 40(5). Worked example reproduced in tests/test_stats_paper.py: DSR 0.9004; 0.9505 at N=46; 0.9505 at N=88 with Normal returns.
  • Bailey & López de Prado (2012), The Sharpe Ratio Efficient Frontier, J. of Risk 15(2). MinTRL examples reproduced: 59.895 months; 2.73, 2.83 and 3.24 years.
  • Lo (2002), The Statistics of Sharpe Ratios, FAJ 58(4). Newey & West (1987), Econometrica 55(3).
  • See explain(<id>) for the references behind each check.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for frankkratzer from gravatar.com frankkratzer

License expression

Apache-2.0
View SPDX License List

Requires

Python <3.13, >=3.12

Provides Extra

test

Tags

backtesting lookahead-bias mcp quant static-analysis deflated-sharpe-ratio

Classifiers

Report project as malware

Metadata

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for frankkratzer from gravatar.com frankkratzer

License expression

Apache-2.0
View SPDX License List

Requires

Python <3.13, >=3.12

Provides Extra

test

Tags

backtesting lookahead-bias mcp quant static-analysis deflated-sharpe-ratio

Classifiers

Report project as malware

Release files for backtest-auditor 0.6.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for backtest-auditor 0.6.0
File Size Uploaded
backtest_auditor-0.6.0.tar.gz 340.2 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for backtest-auditor 0.6.0
File Interpreter ABI Platform
backtest_auditor-0.6.0-py3-none-any.whl 225.3 kB Python 3 none any Details

Total release size: 565.5 kB

Release files / backtest_auditor-0.6.0.tar.gz

Download URL backtest_auditor-0.6.0.tar.gz
Size 340.2 kB
Tags Source
SHA-256 checksum
How to use checksums 		c355802394736cfcadd18fa99d81555cd1e5b1e73cf434d9d83b37fd626164f8
BLAKE2b-256 checksum
How to use checksums 		791b0da51c54a1b8dcc8a04829c2ff846ee9a323399e0a821d8b8ef58e64828d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.

Transparency log
Identity
Publishing platform
GitHub Actions
Publishing repository
github.com/nothingbutalpha/nothingbutalpha
Publishing commit
github.com/nothingbutalpha/nothingbutalpha/tree/5f892d1fc470cd968f4ffe52047ddfd8481a29b8
Workflow
Publishing configuration
github.com/nothingbutalpha/nothingbutalpha/blob/5f892d1fc470cd968f4ffe52047ddfd8481a29b8/.github/workflows/publish-auditor.yml
Publishing logs
github.com/nothingbutalpha/nothingbutalpha/actions/runs/39081161824/attempts/1
Artifact
Subject
backtest_auditor-0.6.0.tar.gz
SHA-256 checksum
c355802394736cfcadd18fa99d81555cd1e5b1e73cf434d9d83b37fd626164f8 Verifying attestations

Release files / backtest_auditor-0.6.0-py3-none-any.whl

Download URL backtest_auditor-0.6.0-py3-none-any.whl
Size 225.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums 		5fcfb063771091c75af6db9d15b230d63175c6f86d4aef0ea5889c5df6992a27
BLAKE2b-256 checksum
How to use checksums 		1265fa9e3c10bb8c7e83f0a57bff71e41815adea6f04d63f5ba63f2724f5195e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.

Transparency log
Identity
Publishing platform
GitHub Actions
Publishing repository
github.com/nothingbutalpha/nothingbutalpha
Publishing commit
github.com/nothingbutalpha/nothingbutalpha/tree/5f892d1fc470cd968f4ffe52047ddfd8481a29b8
Workflow
Publishing configuration
github.com/nothingbutalpha/nothingbutalpha/blob/5f892d1fc470cd968f4ffe52047ddfd8481a29b8/.github/workflows/publish-auditor.yml
Publishing logs
github.com/nothingbutalpha/nothingbutalpha/actions/runs/39081161824/attempts/1
Artifact
Subject
backtest_auditor-0.6.0-py3-none-any.whl
SHA-256 checksum
5fcfb063771091c75af6db9d15b230d63175c6f86d4aef0ea5889c5df6992a27 Verifying attestations

Release history Release notifications | RSS feed

This release

0.6.0 This release

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page