nullbar

Pre-registration and honest statistics for trading research.

Most backtesting tools help you find an edge. This library assumes you will fool yourself while trying — because in two years of production algorithmic trading, that is what happened to us, over and over, despite code review, 2,000+ tests, and genuine care. nullbar is the machinery that caught it, extracted into a standalone package.

The name is the two things it makes you commit to before you are allowed to believe a number: the null control and the pre-registered bar.

The one-day story this library made possible

On 2026-08-11 we designed a new ML model (a cross-sectional transformer over 379 crypto assets), pre-registered its evaluation — target, architecture, splits, and a three-condition pass bar, committed before training — trained it, and evaluated it once on a test window the training code could not read even by bug.

The model achieved 5× the rank correlation of every predecessor. It also failed all three bar conditions: its extra correlation lived in the middle of the ranking, and the only part of a ranking that can pay trading costs is the tail — which a four-line moving average already owned. Design to honest verdict: one day. Without this machinery, that same question had previously consumed months and produced numbers we later had to retract.

What's inside, and the production bug behind each piece

The prefix-replay check deserves a sentence: recompute any feature on a data prefix and compare with the full-sample computation at the same rows. Any feature whose past changes when the future is appended is leaking, whatever its source looks like. This one test, run on day one, would have saved us a year.

Install

# from a clone (editable — edits are live):
git clone https://github.com/brunopereira81/nullbar && cd nullbar
pip3 install -e .

# run the examples:
python3 examples/01_full_workflow.py
python3 examples/02_catch_a_leak.py

# lint a tree for lookahead patterns:
python3 -m nullbar strategy/

Requires Python 3.10+; numpy and pandas (2.x and 3.x are separate CI legs). PyPI release pending — until then, install from the clone.

Renamed at v0.2.0: this package shipped its first version as prereg. The PyPI name prereg belongs to an unrelated project (an OSF-oriented pre-registration CLI). Nothing here is affiliated with it.

Quickstart

import nullbar

# 1. Register before you run
reg = nullbar.Registration(
    name="mean-reversion-24h",
    hypothesis="bottom-decile dist_ma168 mean-reverts over 24h",
    design={"hold_bars": 24, "entry_pct": 0.10, "cost_pct": 0.230},
    bar={"null_flat": "null control indistinguishable from holding",
         "t3": "clustered t >= 3.0 on 24h blocks",
         "beats_hold": "net beats unconditional exposure"},
)
reg.freeze("experiments/mr24.json")          # hashed; edits now visible

# 2. Count every variant you evaluate — with its Sharpe
ledger = nullbar.TrialLedger("experiments/trials.jsonl")
ledger.record("mr24", {"entry_pct": 0.10}, metrics={"sr": cell_sharpe})

# 3. Null control FIRST — against the hold baseline, not against zero
nv = nullbar.null_verdict(entry_mask, fwd_returns)        # nv["ok"] must hold
result = nullbar.block_cluster_eval(entry_mask, fwd_returns)

# 4. Price fills honestly
bracket = nullbar.fill_bracket(entry_mask, limit_px, low_px, fwd_returns)

# 5. Deflate by what you actually searched
d = nullbar.dsr(observed_sr, n=result["clusters"], n_trials=ledger.count(),
                sr_variance=ledger.sr_variance())

# 6. Spend the single test look, on the record
reg.spend_test_look("experiments/mr24.json", results=result)
print(reg.verdict({"null_flat": nv["ok"],
                   "t3": result["t"] >= 3.0,          # numpy bool: fine
                   "beats_hold": net > hold_net}))

What this library will not do

It will not find you an edge. Ours, measured with these exact tools across architecture, features, horizons, and training breadth, was zero net of costs — and we can prove it, which is the point. If your strategy survives this harness, you have something. If it doesn't, you found out for the price of compute instead of capital.

It is also tamper-evident, not tamper-proof: it grades the frozen file and binds the test-look stamp to that file's hash, but anyone with write access can delete both. It is built for a researcher keeping themselves honest. If a third party has to believe the record, commit it — or anchor it somewhere you don't control.

Docs & examples

• The honest workflow — the six steps, each annotated with the production failure it prevents, plus the deflation cheat sheet.
• examples/01_full_workflow.py — the whole sequence end-to-end on synthetic data; runs in seconds, CI-tested.
• examples/02_catch_a_leak.py — four features, two leaks, one 50ms check; includes the leak that inspired the library.
• The leak that survived two years — the full story.
• CHANGELOG — what moved, and why.

Status

v0.4.0 — extracted 2026-08-12 from a live production system (Coinbase spot, TimescaleDB, 2,100+ tests), renamed and hardened 2026-08-18 across three independent audit passes, 27 distinct findings. The ones worth knowing: verdict() could grade a failing strategy as PASS, fill_bracket could overstate 9× on misaligned axes, clustered_t inflated t when a cluster held no finite observation, and the frozen bar could say something different from the code grading it. All fixed, each mutation-checked, in the CHANGELOG. API will move; the philosophy won't. MIT licensed — the statistics stay open, permanently.