backtest-audit-kit

"I verify whether your strategy is real — or whether the backtest lied to you."

A tiny, dependency-free toolkit that audits a backtest the way an accountant audits a ledger and a QA engineer audits a test suite. It does not sell you returns; it finds the four ways a backtest most often lies.

回测审计 / 策略代码复核服务 — 一页说明与联系方式 → PORTFOLIO.md Backtest audits · strategy-code review — one-page overview → PORTFOLIO.md

English

What it is. A pure standard-library (Python 3.9+) kit of four independent checkers for a trading backtest:

Who it is for. Quant developers, prop-desk researchers, and anyone who has ever shipped a strategy that looked great in a notebook and died in production. If you are reviewing someone else's backtest — or your own — this gives you a checklist, reproduction scripts, and before/after numbers instead of opinions.

What I deliver. ① A ranked findings list plus a runnable reproduction script for each problem. ② A minimal fix patch. ③ A before/after reconciliation of the headline numbers (win rate, net PnL, multiple, MDD, t-stat).

Sample cases (full write-ups in cases/):

1. Look-ahead — the exit scan ran on the entry bar, so every signal knew its outcome. Fixed: 91 trades / 90.1% / +$214.37 → 26 trades / 65.4% / -$0.12; a tick-level re-run gives 27 trades / 66.7% / +$0.37. The leak was worth ≈ $214 of imaginary PnL — the whole profit.
2. Withdrawal accounting — the bust path dropped wd_cum, so any strategy with withdrawals was systematically understated. Fixed: same 2024–26 config 7.82x → 215.27x (a ≈ 27x gap; withdrawal rules "double-keep-half" vs "double-keep-100" differ by ~27x).
3. Drawdown metric — the circuit-breaker reset the running peak, so the "reported MDD" was only the local drawdown. Investor-view raw drawdown 99.7% vs reported 53.1% — a 46 pp gap. Conclusion: MDD must be written twice (breaker-local and raw). (Bonus case 4: 5-minute bar clustering inflated n ~11x and turned t = 15.60 into t = 1.75 after a 24h cooldown.)
4. Cost omission (cases/case5_cost_omission.md) — a high-turnover 50x book charged no fee, funding or slippage; its take-profit needed only a +0.08% move while one round trip costs 0.09% (the venue's real 4.5 bps taker), so the target sat below the toll and every win is a net loss. At the venue rate the omitted cost ran ~5.4x the starting capital and ~17.8x the reported profit.

Install.

pip install backtest-audit-kit

Use it in code (four checkers, pure standard library):

from auditkit.checks import lookahead
findings = lookahead.detect_file("strategy.py")   # future-data lint
print(lookahead.format_findings(findings))

How to run.

cd portfolio/backtest-audit-kit
python3 -m unittest discover tests      # stdlib only, no install
# or, if you have pytest:
python3 -m pytest tests/ -q
python3 examples/run_demo.py            # see all four checkers in action

Honesty note. The look-ahead checker is a heuristic linter, not a proof engine: it flags the shapes that leak, and a stand-alone forward-simulation pass is reported MEDIUM for human review rather than being declared safe. Mutation testing reports blind spots; it never claims your strategy is profitable. No checker here predicts returns.

中文

是什么。 一个纯 Python 标准库（3.9+）实现的回测审计工具包，四个互相独立 的检查器：

给谁用。 量化开发者、自营研究岗、以及任何"回测很美、实盘就死"过的 人。带看别人的回测（或自己的）时，它给你的是问题清单 + 复现脚本 + 修复前后 的数字，而不是一句"我觉得不太行"。

交付物承诺。 ① 按严重度排序的问题清单，每个问题配一个可运行的复现脚本。 ② 最小修复补丁。③ 修复前后的对账数字（胜率 / 净利 / 倍数 / 回撤 / t 值）。

案例摘要（完整写在 cases/）：

1. 前视 bug —— 出场扫描写在开仓 bar 的循环内，每个信号都提前知道结果。 修复后：91 笔 / 90.1% / +$214.37 → 26 笔 / 65.4% / -$0.12；tick 级 复跑得 27 笔 / 66.7% / +$0.37。这个泄漏值 约 $214 的虚假收益， 即全部利润。
2. 提现会计 bug —— 爆仓返程丢了 wd_cum，导致所有带提现的策略历史收益被 系统性低估。修复后：同一配置 2024–26 7.82x → 215.27x（相差约 27 倍； 提现规则"翻 2 留 1"与"翻 200 留 100"之间也差约 27 倍）。
3. 回撤口径 bug —— 熔断触发时把 peak 重置为 cap，"报告 MDD"只是每段熔断 后的局部回撤。投资者口径的原始回撤 99.7% vs 报告 53.1%，差 46 个百分点。结论：MDD 必须双写（熔断口径 + 原始口径）。 （附赠案例 4：一段暴跌连打十几根 5min bar 让样本量虚增约 11 倍，t 值从 15.60 在 24h 冷却去重后掉到 1.75。）
4. 成本漏算（cases/case5_cost_omission.md）—— 一个高换手 50x 账本完全没算手续费/资金费/滑点；止盈只需价格动 +0.08%，而一趟往返 0.09%（场馆真实 4.5bps taker）⇒ 目标在成本之下，每笔赢单净亏。按场馆真实费率，漏掉的成本约 5.4× 本金、17.8× 账面收益。

怎么跑。

cd portfolio/backtest-audit-kit
python3 -m unittest discover tests      # 只用标准库，无需安装
# 若装了 pytest：
python3 -m pytest tests/ -q
python3 examples/run_demo.py            # 四个检查器全跑一遍

诚实声明。 前视检查器是启发式 linter，不是证明器：它抓的是会泄漏的代码 形态，独立的"向前模拟"循环会被标为 MEDIUM 交人工复核，而不是被判安全。 变异测试只报告覆盖盲区，从不声称你的策略赚钱。本工具包不预测任何收益。

Also in this account · 另见

• outpost/ — 一个"只放能跑、能查、能被否证的东西"的据点： 单文件表格整理工具（零依赖、离线自检 14 项）+ 一份被自己推翻的回测（做空山寨无 edge）
  • 一次"两条腿都断了"的数据源修复。不写客户名、不写"我能做什么"。