validanytime-detectors 0.1.2

The ValidAnytime detector suite, offline: CUSUM, EWMA, static threshold, rolling band — plus an offline port of the anytime-valid backtest gate, golden-tested against the production engine.

validanytime-detectors

The ValidAnytime detector suite, offline: every detector you already know — CUSUM, EWMA chart, static threshold, rolling band — plus an offline port of the anytime-valid backtest path the ValidAnytime cloud runs, with the hosted gate's machine-readable verdict.

Pure Python, zero dependencies — the float32 emulation uses only the standard library, so pip install validanytime-detectors pulls in nothing else. Nothing here talks to the network. Apache-2.0 licensed.

pip install validanytime-detectors

Contents

• The 30-second version
• Two tiers, stated honestly
• Engine fidelity
• Run the parity tests
• Related

The 30-second version

from validanytime_detectors import backtest_gate

# your believed-normal metric history (judge scores, accuracy, latency, ...)
verdict = backtest_gate(history)

print(verdict["summary"])
# "Quiet across 240 normal points, then caught the injected degradation
#  9 step(s) after it began. This is the day it would have caught it."
assert verdict["passed"]

backtest_gate replays your history through the two-tier stack and grades it exactly like the hosted POST /v1/onboarding/backtest:

• it must stay quiet on your normal history (a page-tier fire there is a false alarm and fails the gate), and
• it must catch an injected degradation appended to the tail (synthetic, and labeled as such in the verdict via "injected": true).

The verdict is a plain dict with the same fields as the hosted gate — passed, false_alarm_on_normal, caught_at_seq, fires, warned_on_normal, warn_fires, summary — so an agent or a CI job can consume either interchangeably. This is what powers the validanytime-gate GitHub Action.

Two tiers, stated honestly

Warning tier — the classical detectors. cusum_events, ewma_chart_events, static_threshold_events, rolling_band_events are sensitive and usually faster to fire, but their calibration is model-based: an ARL or false-alarm rate computed for iid Gaussian z-scores. On realistic metric textures their false-alarm rate can exceed the nominal promise by orders of magnitude. Warnings are hints to tune, not budgeted pages, and the gate never grades on them (warned_on_normal is surfaced, never a failure).

Page tier — the e-process monitors. The coverage e-process and the Shiryaev–Roberts e-detector page within a stated false-alarm budget, and the guarantee holds at every look, however often you check — that is what "anytime-valid" means. Only this tier can pass or fail the gate.

from validanytime_detectors import cusum_events, CUSUM_H_ARL_2000

r = cusum_events(values)          # k=0.5, h solved for nominal ARL 2000
print(r.events)                   # 1-based alarm seqs (renewal resets)

Engine fidelity

The e-process path (replay_series, grade_replay, inject_drift, suggest_config) is a float32-emulated port of the production engine's backtest path — same conformal calibrator, same e-process monitors, same alarm thresholds — verified against the production engine's own outputs by golden tests: identical first-alarm points on the full fixture corpus, and evidence curves matching to 0.01%. It is a port, not the same binary; transcendental rounding can differ by 1 ulp, which the fixture corpus (with deliberately near-threshold cases) is designed to catch.

The offline path replays plain numeric histories. Delayed labels, fleet-wide false-discovery control, alarm certificates, and routing are hosted-product features and are not in this package.

Run the parity tests

The golden fixtures are generated from the real production engine and vendored in this repo (tests/fixtures/), so the parity suite — the proof of the fidelity claim above — runs fully offline with no account and no service:

pip install -e '.[dev]'
pytest -q          # golden engine parity + classical parity + gate behavior
ruff check .       # lint
ruff format --check .
pyright            # types

NumPy is a test-only dependency (it computes the float32 reference values the port is pinned against); the package itself imports nothing outside the standard library.

Related

• /try — the same backtest in your browser, no install, with the detector bake-off.
• validanytime — the thin API client for the hosted product.
• Docs for agents — MCP server, copy-paste agent prompts, and the CI gate.