falsification-ledger

Featured in awesome-quant — the curated list of quant libraries (Trading & Backtesting section).

中文说明

falsification-ledger 是量化研究的可验证实验记录本，也适用于 A 股回测。 研究开始前先登记假设，以及什么证据会推翻它；之后把证据写入追加式、 哈希串联的记录，再进行一次性裁决和命中率统计。它帮助减少事后挑选结果， 但不替代数据审计、统计检验或人工判断。

A hash-chained, append-only ledger for research claims: pre-register the hypothesis and the evidence that would kill it before the study runs, then adjudicate honestly and measure your hit rate against a random baseline. Python 3.11+, one dependency (jsonschema), Windows / Linux / macOS.

Status: v0.1.4 alpha, published on PyPI. The ledger semantics are distilled from a production research pipeline, but this standalone package is new: expect the CLI and schemas to shift before v1.0.

Why this exists

Quantitative research has a self-deception problem: you test 500 factor ideas, remember the 3 that worked, and forget the 497 that died. By the time you "validate" the lucky survivors, the evidence is already contaminated by what you saw. Every backtest-hygiene tool on the market attacks the statistics of this problem (deflated Sharpe, PBO, multiple-testing corrections). falsification-ledger attacks the process: it makes you write down, before seeing evidence:

• what you expect (support / against / uncertain), and
• what evidence would kill your claim (the falsification contract).

Then it keeps the receipts. Every event lands in an append-only JSONL hash chain —any edit after the fact is detected by fl verify —and the report answers the only question that matters: do your pre-registered beliefs actually hit, or is your hit rate indistinguishable from a random baseline? (Wilson 95% CI vs the most common actual verdict.)

Philosophy

Research is a promise; the ledger keeps it.

• Falsifiability is the default, not the exception. Popper's criterion —a claim is scientific only if something could count against it —is usually invoked as a lecture. Here it is a required JSON field (falsification_contract on preregister).
• Pre-analysis plans have known costs and benefits. Olken (2015), "Promises and Perils of Pre-Analysis Plans" (JEP 29(3)) documents both; this tool implements the benefits (frozen expectations, audit trail) while keeping the costs explicit (uncertain verdicts and exploratory source types are first-class, so you can register what you genuinely do not know).
• Moderation beats total freezing. Banerjee & Duflo, "In Praise of Moderation" argue for layered pre-registration; source_type (paper / business / cross_domain / pipeline / other) exists so confirmatory and exploratory claims are never mixed in the same bucket.
• Finance can become scientific. López de Prado (2023), Causal Factor Investing asks whether factor investing can become a science; this ledger is one concrete answer —evidence with a chain of custody, adjudicated against a pre-registered expectation.
• Automated research needs machine-checkable evidence. EviBound (arXiv:2511.05524) and ECLIPSE v2.0 argue that agentic research pipelines must eliminate false claims through verifiable evidence; fl submit validates falsification reports against a JSON Schema and computes content IDs, so gates can trust the evidence without trusting the messenger.

See it in action

fl verify catching a retroactive edit — every event is a hash-chain link, and a one-field change breaks the chain at a specific line number:

And the report speaks plainly. On a real 8-case demo ledger, fl report answers the only question that matters — are you better than a coin flip?:

Full JSON: docs/report-example.txt — hit rate 0.71 with a 95% Wilson CI of [0.36, 0.92] against a 0.57 random baseline: baseline inside CI, so the report says "no systematic signal" — and refuses to pretend otherwise.

Quick start

The 60-second version — the whole loop in one command:

pip install falsification-ledger
fl demo

fl demo runs on a scratch ledger under a temp dir (nothing to clean up): it pre-registers a claim together with its falsification contract, submits an independent falsification report, adjudicates, and prints the hit-rate report — then edits one field of the ledger and lets fl verify name the exact line that broke the chain. About 60 seconds from install to "this is why the hash chain matters".

The same loop, command by command:

fl init --state-dir ~/.research-ledger

# 1. BEFORE running the study: register what you expect,
#    and what evidence would kill the claim.
fl preregister --state-dir ~/.research-ledger \
  --case-id MOMENTUM-OOS-2026Q3 \
  --verdict support \
  --reason "momentum rank IC stays positive OOS" \
  --source-type paper \
  --contract kill-criteria.json

# 2. When an independent check produces evidence, submit it:
fl submit --report falsification-report.json
# -> {"content_id": "sha256:...", "evidence_status": "valid", ...}

# 3. AFTER the study: adjudicate honestly.
fl adjudicate --state-dir ~/.research-ledger \
  --case-id MOMENTUM-OOS-2026Q3 --verdict support

# 4. Measure whether you are better than a coin flip.
fl report --state-dir ~/.research-ledger --min-cases 20

# 5. Any time: prove nobody rewrote history.
fl verify --state-dir ~/.research-ledger

Commands

Command	What it does
init	Create the ledger state directory
preregister	Register a claim: --case-id, --verdict (support/against/uncertain), --reason, optional --source-type, optional --contract (falsification contract JSON). Duplicate registration for the same case is rejected
submit	Validate a falsification report against the contract schema; print its content ID (sha256:...) and evidence status (valid / invalid / missing). Read-only; exits non-zero on blockers
adjudicate	Backfill the actual verdict for a registered case (register required; once per case)
conformance	Check an evidence report against the adjudication criteria pre-registered in the contract (criteria at preregister time vs criteria_outcomes in the report). Fail-closed: a pass claim (not_falsified) on evidence that never tested — or failed — a required criterion is blocked. Read-only; exits non-zero when blocked
report	Hit-rate report: resolved cases, completeness, participation, hit rate with Wilson 95% CI, random baseline, per-source-type breakdown, verdict_ready gate
verify	Recompute the hash chain of the whole ledger; detects any edit, insertion, or reordering
demo	60-second intro on a scratch ledger: full preregister → submit → adjudicate → report → verify loop, then a deliberate tamper that verify catches at a specific line
version	Print version

Global flag: --state-dir on every stateful command (default: none —the ledger path is always explicit, so a git add . can never sweep it into version control).

Ledger format

The ledger is a JSONL file at <state-dir>/ledger.jsonl. Every line is one event:

{"schema_version": "falsification_ledger.prediction_event.v1",
 "event": "register", "record_id": "...", "case_id": "CASE-1",
 "expected_verdict": "support", "expected_reason": "...",
 "source_type": "paper", "falsification_contract": {...},
 "actual_verdict": null, "recorded_at": "...", "concluded_at": null,
 "prev_hash": null,
 "event_hash": "sha256(prev_hash || 0x00 || canonical payload)"}

verify recomputes every event_hash and checks each prev_hash link. Any tampering —editing a reason, deleting a line, reordering events —breaks the chain at a specific line number.

Falsification reports

A falsification report is the machine-readable evidence produced by an independent check (null-model randomization, OOS rank IC, FDR correction, protocol deviation, effect CI, cost sensitivity, ...). The contract:

• schema: schema/falsification-report.schema.json (draft 2020-12, additionalProperties: false, fail-closed);
• content ID: sha256: over domain-prefix || 0x00 || canonical JSON — the same report always yields the same ID, a one-field change yields a different ID;
• evidence status (fail-closed for gates):
  • valid —conformant, conclusion not_falsified, consistency intact;
  • invalid —non-conformant, or conclusion falsified, or explicitly inconsistent;
  • missing —conclusion inconclusive: treated as absent evidence.

Adjudication conformance

A pre-registration can freeze more than a direction: it can freeze the criteria a pass claim must satisfy. Next to the free-form falsification_contract, declare them as a structured list:

{"criteria": [
  {"name": "primary_positive", "required": true,
   "note": "primary statistic positive and its test passes"},
  {"name": "control_superiority", "required": true,
   "note": "control arm positive with a passing paired test, same window"},
  {"name": "sample_complete", "required": false,
   "note": "samples counted as completed trades, not order records"}
]}

An evidence report then declares, per criterion, what was actually tested:

{"criteria_outcomes": [
  {"name": "primary_positive", "outcome": "met"},
  {"name": "control_superiority", "outcome": "met",
   "note": "paired t over aligned windows, p=0.02"}
]}

fl conformance --state-dir <dir> --case-id <id> --report <file> checks the report against the criteria frozen at registration time (fail-closed):

• every required criterion must have exactly one outcome — a report that never tested a criterion you promised to test is blocked (criterion_not_covered);
• a not_falsified pass claim while any required criterion outcome is not met is blocked (not_falsified_while_required_not_met) — violated, not_tested and inconclusive all count against a pass claim;
• non-required criteria (required: false) and extra undeclared outcomes are informative only — they never block;
• a contract without criteria declares nothing to check and always conforms.

This closes the drift where the implementation of a verdict engine passes cases on fewer conditions than the pre-registration promised — the exact failure a production shadow-verdict round hit in 2026-09 (pass claims on the primary statistic alone while the frozen contract required a positive control arm with a passing paired test). Offline reproduction: examples/case_verdict_drift.py (python examples/case_verdict_drift.py, zero dependencies, self-checking exit code).

Verification model

fl verify is the tamper-evidence layer: it re-derives the entire chain from the file bytes and reports the first bad line. Combined with preregister (frozen expectations) and submit (content-addressed evidence), a research pipeline can prove to itself —and to reviewers —that the expectation existed before the evidence did. Nothing here trades, prices, or decides.

Development

python -m pip install -e . pytest
python -m pytest

CI runs the full test suite on Ubuntu, Windows and macOS with Python 3.11 and 3.12. Issues are handled on weekends; pull requests are welcome.

Related work

• Olken (2015), Promises and Perils of Pre-Analysis Plans —the economics of freezing expectations
• Banerjee & Duflo, In Praise of Moderation —layered pre-registration
• López de Prado (2023), Causal Factor Investing —can factor investing become scientific?
• EviBound: Evidence-Bound Autonomous Research (arXiv:2511.05524) —governance for agentic research
• ECLIPSE v2.0: A Systematic Falsification Framework —enforce falsifiability integrity

Project family

Part of Holdout — a toolchain against self-deception in quantitative research:

• pit-adjuster — PIT back-adjustment with static forward-adjustment drift detection
• falsification-ledger — pre-registration and falsification ledger
• factor-qc — fail-closed backtest quality gate
• lesson-book — tuition memory for traders
• lookahead-free — verifiable look-ahead-freedom checks
• ashare-data-immunity — data immunity for A-share daily bars

Sister org: Metabolism Tools — workspace-metabolism, policy-driven file lifecycle management for agentic workspaces.

License

MIT