numguard

The statistics, first. The data-driven t-stat hurdle of Harvey & Liu, False (and Missed) Discoveries in Financial Economics, JF 2020, is in numguard/fdr.py, in both halves of the paper's title:

• fdr_hurdle — the single-bootstrap core. Demean the trial panel, resample the time index with the same draws for every trial so the cross-trial correlation survives, then take the smallest hurdle whose estimated FDR meets your target. All trials are treated as null when counting expected false discoveries (conservative, like BH with m0 = m). What it estimates is E[V]/R — expected false discoveries over the discoveries observed — which is not E[V/R], the quantity the double bootstrap reports as TYPE1; a ratio of expectations is not the expectation of a ratio. Because it never represents the alternative, it says nothing about missed discoveries.
• harvey_liu_hurdle — the paper's actual double bootstrap, Steps I–IV. The outer loop builds a pseudo-population in which a fraction p0 of strategies are genuinely non-null, with effect sizes taken from a bootstrap draw rather than from the in-sample winners; the inner loop resamples it and counts outcomes against that known truth. That is what makes misses countable, so this half reports TYPE1, TYPE2 (the false omission rate, FN/(FN+TN) — deliberately the mirror of FDR, not 1−power) and ORATIO, the odds of a false discovery per miss. p0 is an argument, not an estimate, because the paper conditions on it rather than estimating it; hurdle_curve reports across its grid so a single number cannot hide the assumption that produced it.

Both rank on |t|, so a strongly negative strategy counts as a discovery in either half. Screen the positive-t trials if what you are looking for is outperformance.

ORATIO is the one to target when the two errors cost different amounts — the paper's own example is that if a false discovery costs ten times a miss, the target is 1/10. On the 50-strategy panel in examples/, with 3 genuinely skilled strategies planted among 47 nulls, Bonferroni sets |t| ≥ 3.29 and finds 1 of the 3. Pricing the two errors at ten to one moves the hurdle to |t| ≥ 2.35 at p0 = 0.02, which recovers all 3 with no false positives. Every number in that sentence comes out of one command, and the panel ships with its ground truth so you can check the claim rather than take it.

The same table shows what a convention cannot: at p0 = 0.05 the hurdle rises to 2.90 and the recovery falls back to 1 of 3. p0 is an assumption, not an estimate, and it changes the answer — which is why hurdle_curve reports the grid instead of a number.

Tests: tests/test_fdr.py — the ones worth a minute check estimators against analytic values they were never told: E[#null ≥ h] = m·2(1−Φ(h)) for the null pool, and, for the double bootstrap, the two cutoffs at which the contingency table is fixed by construction regardless of the data (at cutoff 0, RFDR = (m−n_alt)/m and RMISS = 0 exactly). A resampler that silently does nothing fails those. The Deflated Sharpe Ratio lives in numguard/backtest.py. Pure math + seeded random; no numpy, no scipy. MIT.

Run it on a panel whose answer is known — fifty strategies, three of them genuinely skilled, so a hurdle can be scored instead of admired (examples/):

python -m numguard.fdr examples/returns_50_strategies.csv \
    --truth examples/returns_50_strategies.truth.txt

  Bonferroni 5%       |t| >= 3.29   -> 1 discoveries
                       finds 1/3 skilled, 0 false
  FDR hurdle          |t| >= 3.40   -> 1 discoveries   (target FDR 0.05)
                       finds 1/3 skilled, 0 false

Add --criterion oratio --target 0.1 to price a false discovery at ten times a miss: on that panel the hurdle falls to |t| >= 2.35 at an assumed p0 of 2%, recovering all three with no false positives. Name the assumption or the number means nothing: the same run prints the rest of the p0 grid, where the hurdle rises to 2.90 and the recovery drops back to 1 of 3. Quoting the best row without the assumption that produced it is exactly what hurdle_curve exists to prevent. Point it at your own CSV — one column per strategy you actually ran — and it does the same for your search history.

Everything below is the agent-facing packaging of those same checks — an MCP server, signed receipts, and metering. The statistics do not depend on any of it.

_{MCP registry identity — mcp-name: io.github.ipezygj/numguard}

The verification layer for the agent economy — an agent-callable primitive that checks a number before it gets asserted, and hands back a signed receipt proving it was checked.

Agents now produce an explosion of numbers: eval scores, A/B results, "the agent improved 12%", benchmark rankings, backtest Sharpes. The scarce resource isn't the number — it's trust in the number. numguard is the tool an agent calls mid-task to ask "does this survive a second look?", and to attach a portable, tamper-evident receipt so the answer travels with the claim.

Built on evalgate for the shared eval statistics; adds the pieces agents specifically need — a Deflated Sharpe Ratio for backtests, judge calibration, signed receipts, and metering an agent can actually pay (prepaid credits + x402 pay-per-call). Exposed as an MCP server, so any agent can call it.

New here? — How to verify a backtest is real (Deflated Sharpe in Python): the practical guide to catching an overfit or leaking backtest, with runnable code. New in 0.2.0: fdr_hurdle — no universal "t > 3"; derive the hurdle your own search history implies at your false-discovery-rate target (Harvey & Liu, JF 2020). See it work — proof gallery: 8 real numbers run through the real checks, 3 survive and 5 are flagged, each with a receipt you can verify offline. Don't trust it — verify it. Wire it into an agent in one line — INTEGRATE.md: the local reflex, an MCP config, and LangChain / CrewAI tool wrappers.

The tools

On-chain and agent-verification tools — the same discipline applied to things that live on a chain rather than in a spreadsheet. Listed because a tool an agent cannot find is a tool it cannot call.

What sets it apart (why): computing the number yourself, or a lesser checker, stops at "is it significant?" numguard also holds it accountable to live reality over time, signs a portable tamper-evident proof, and lets anyone verify any proof for free — the trust layer, not just a calculator.

For agent traders: the Deflated Sharpe Ratio

The number that kills a backtest is the same one that kills a benchmark score: you tried many, and you reported the best. In finance the rigorous correction is the Deflated Sharpe Ratio (Bailey & López de Prado) — given how many variants you tested, what Sharpe would the luckiest zero-skill strategy have shown, and do you beat it after adjusting for sample length and non-normal returns?

from numguard import deflated_sharpe

deflated_sharpe(sr=0.12, T=250, n_trials=100)
# SR=0.120 over T=250, 100 trials tested; deflation bar=0.160; DSR=0.263
# -> does NOT survive deflation.  (PSR-vs-0=0.970 — it LOOKS significant on a single test.)

deflated_sharpe(sr=0.15, T=1000, n_trials=1)
# DSR=1.000 -> SURVIVES. A real edge over a long sample.

The contrast is the whole point: a single-test probability of 0.97 ("significant!") collapses to a deflated 0.26 ("noise") once you account for the 100 strategies that were tried. An agent optimizing over strategies should call this before it trusts — or publishes — a backtest.

The full integrity battery — what a Deflated Sharpe still misses

DSR catches best-of-N. It does not catch same-bar look-ahead, autocorrelation inflating the Sharpe, regime dependence, tail fantasy, or one-lucky-epoch fragility. verify_backtest_series runs the whole battery on the actual returns series and returns a risk level (none/medium/high/critical) plus the checks that flagged:

The tell (python examples/catch_a_fake_backtest.py): a look-ahead strategy shows an annualised Sharpe of +20 and a Deflated Sharpe that survives — yet the battery flags it critical on leakage (same-bar corr 0.79 vs next-bar 0.05). The DSR waves the fiction through; the battery does not.

verify_backtest_series(api_key="…", returns=[...], positions=[...], asset_returns=[...])
# {"risk": "critical", "survives": false, "flags": ["leakage", ...], "checks": {...}}

Signed receipts (the part that compounds)

from numguard import verify_claim, issue_receipt, verify_receipt, keypair
priv, pub = keypair()
result  = verify_claim("backtest", sr=0.12, T=250, n_trials=100)
receipt = issue_receipt(result, priv, pub)     # Ed25519-signed
verify_receipt(receipt)                          # True — anyone can verify with the public key alone

Attach the receipt to your output. A downstream agent (or human) can confirm — without your keys — that the claim and its verdict weren't altered and that numguard issued them. As receipts circulate, "a number without a receipt" starts to read like "a number nobody checked."

Buying is easy for an agent

Two rails, both built so an agent can decide and pay in-loop, no human clicking:

1. Prepaid credits + API key — a human tops up once; the agent spends per call. Generous free tier (25 calls/key) so the agent feels the value first, then a machine-readable price list. Insufficient balance returns a structured payment_required, not an error.
2. x402 pay-per-call — the agent hits a tool, gets an HTTP-402 with a machine-readable price + pay-to address, pays USDC from its wallet, retries with proof, gets the result. The protocol layer is here; settlement is pluggable (inject a facilitator/RPC verifier for production).

from numguard import x402
x402.require_payment("verify_backtest", price_usd=0.03, pay_to="0x…")
# -> {"status": 402, "accepts": [{"scheme":"exact","network":"base","asset":"USDC", ...}]}

Run the MCP server

pip install numguard
python -m numguard.mcp_server        # stdio MCP server; point your agent/host at it

Then an agent calls e.g. verify_backtest(api_key="…", sr=0.12, T=250, n_trials=100) and gets a verdict it can quote and a receipt it can attach.

Deploy it (hosted, paid, discoverable)

1. Host the paid HTTP API (x402 per-call):

docker build -t numguard . && docker run -p 8080:8080 \
  -e NUMGUARD_PAYTO=0xYOURWALLET \
  -e NUMGUARD_FACILITATOR_URL=https://your-x402-facilitator \
  numguard

Or one-click on Render: New → Blueprint → this repo (render.yaml included); set NUMGUARD_PAYTO + NUMGUARD_FACILITATOR_URL in the dashboard. With NUMGUARD_PAYTO unset the API runs free (dev mode) so you can test before wiring a wallet. Endpoints: POST /verify_backtest, /verify_model_gap, … ; GET /pricing.

The x402 flow, end to end: the agent POSTs → gets 402 with an accepts block (price, payTo, network) → signs a USDC payment → retries with an X-PAYMENT header → numguard verifies + settles it through the facilitator to your wallet → returns the result. Settlement is the real x402 /verify + /settle handshake (numguard.x402.facilitator_verifier) — facilitator-agnostic: point NUMGUARD_FACILITATOR_URL at any x402 facilitator. Options:

• Testnet (free, no account): https://x402.org/facilitator with NUMGUARD_NETWORK=base-sepolia — test the whole flow with test-USDC first.
• Mainnet, self-sovereign: self-host x402-rs (open-source, no third party) and point at your own URL.
• Mainnet, hosted (non-Coinbase): thirdweb or PayAI facilitators (Base) — set NUMGUARD_FACILITATOR_AUTH if the facilitator needs a key.

2. Serve the MCP server over HTTP (for remote MCP hosts): uvicorn numguard.mcp_server:app (or NUMGUARD_TRANSPORT=streamable-http python -m numguard.mcp_server).

3. Get discovered: server.json (official MCP registry) and smithery.yaml (Smithery) ship in the repo; connect the repo at those registries so agents can find the server. GitHub topics: mcp, mcp-server, x402.

Design notes

• Statistics are shared with evalgate (zero-dependency); numguard adds the backtest, receipt, metering, and MCP layers on top — it does not re-implement the core checks.
• Pure-math numerics where possible; cryptography only for Ed25519 receipts (HMAC fallback without it).
• Every verdict is derived from a computed statistic, never asserted — the same discipline as the book behind it, Measured, Not Believed (leanpub.com/measurednotbelieved).

MIT.