validanytime 0.1.0

Official Python SDK for ValidAnytime: anytime-valid monitoring with budgeted page-tier alarms and clearly-labeled warn-tier early warnings.

validanytime

Official Python SDK for ValidAnytime — anytime-valid monitoring.

Alarms with a stated false-alarm budget, valid however often you look — plus unbudgeted early warnings, clearly labeled.

• Python >= 3.10, one runtime dependency (httpx).
• Sync (Client) and async (AsyncClient) with identical method sets.
• Typed dataclass models, actionable errors, automatic retries, idempotent ingestion.

pip install validanytime

Contents: Quickstart · Examples · The two alarm tiers · Autoconfig & the backtest gate · CLI · Configuration · Retries & idempotency · Errors

Quickstart

import validanytime

va = validanytime.Client()  # reads VALIDANYTIME_API_KEY

# 1. Create a monitor from a template. The server resolves the template at
#    create time and returns the RESOLVED config — what you see is what runs.
monitor = va.create_monitor(
    "judge-scores",
    template="llm_quality",   # also: default, ml_drift, latency, kpi — each a
                              # distinct, gate-verified config (va.use_cases())
    mu_baseline=0.92,         # the level your metric should hold (optional:
)                             # omitted, it is learned from the first values)

# 2. Stream your metric.
va.ingest(monitor.id, 0.91, retry=True)          # one event, retry-safe
va.ingest_batch(monitor.id, [0.90, 0.93, 0.92])  # or a batch

# 3. Read alarms — as often as you like.
for alarm in va.alarms(monitor.id):
    print(alarm.tier, alarm.guarantee_tag, alarm.message)

Async is the same surface:

async with validanytime.AsyncClient() as va:
    m = await va.create_monitor("judge-scores", template="llm_quality")
    await va.ingest(m.id, 0.91, retry=True)
    alarms = await va.alarms(m.id)

Examples

Runnable end-to-end scripts live in examples/:

• quickstart.py — create, ingest, read both tiers.
• async_quickstart.py — the same flow, async.
• agent_autoconfig.py — shape-aware suggestion, ratified by the backtest gate before it goes live.
• ci_gate.py — a backtest gate that exits non-zero, so it drops straight into CI.

The two alarm tiers

Every alarm carries a guarantee_tag; Alarm.tier maps it to one of two tiers (the same mapping the dashboard uses):

• "page" — budgeted, anytime-valid. Page-tier alarms come from e-processes whose false-alarm control holds at every look at once, within a false-alarm budget stated up front. Polling alarms() more often never degrades the guarantee. Tags: anytime_valid_under_conditional_mean_null, average_run_length_e_detector, online_evalue_fdr_control. Page-tier alarms carry the receipt: e_value and theorem_ref.
• "warn" — early, unbudgeted, not guaranteed. Warn-tier alarms (tag heuristic_adaptive) come from classical detectors (e.g. CUSUM) whose calibration is model-based (iid standardized inputs) and not anytime-valid: on realistic data the false-warning rate can exceed the nominal rate substantially. Treat warnings as sensitive early hints; page humans on the page tier.
• "unknown" — missing or unrecognized tag.

pages = [a for a in va.alarms(monitor.id) if a.tier == "page"]
warns = [a for a in va.alarms(monitor.id) if a.tier == "warn"]

Autoconfig & the backtest gate

Don't hand-write detector math. Bring the data; let the backtest gate — not any model — ratify a config before it goes live. This is the agent-driven path: your agent reads the metric (or asks the user), then calls the API.

sample = [...]                                    # your metric's recent values

# Shape-aware suggestion, already run through the gate on your own history.
s = va.suggest_config(sample, history=sample)     # -> Suggestion
if s.backtest.passed:                             # the non-bypassable check
    m = va.create_monitor("my-metric", config=s.config)
    print(s.backtest.summary)                     # "...caught it on day X."

• va.use_cases() — the public use-case → config matrix, a data-free lookup (UseCase rows with the resolved config + a direction/rationale you can show the user). Pass a row's config to create_monitor, or its id as use_case= to suggest_config as a hint.
• va.suggest_config(sample, history=..., use_case=...) — reads the sample's shape (level, scale, direction, boundedness, drift), tunes a config, and ratifies it on history (defaults to sample). Returns a Suggestion (.config, .backtest). Check .backtest.passed before going live.
• va.backtest(history, config=..., inject_shift=...) — re-run the gate on any config. Graded on the page tier (.passed, .caught_at_seq); a warn-tier fire on the normal stretch shows up as .warned_on_normal and never fails the gate. inject_shift=None replays as-is and only checks it stays quiet.

A backtest is a replay of past data, not a forecast about a different stream.

Other methods

• get_monitor(id) / list_monitors() — the returned config is always the resolved config that actually runs.

• update_monitor(id, name=..., template=... | config=...) — rename and/or reconfigure. A config change is validated like create and resets the monitor's derived engine state (carry, learned baseline, fired-alarm latches); the new config applies to future events only. The response's state_reset flag says whether that happened.

• list_events(id, after_seq=0, limit=500) — one seq-cursor page of the monitor's event history (ascending), as an EventsPage of typed Event rows (seq, value, ts, label) plus next_after_seq (pass it back as after_seq; None means done).

• iter_events(id) — a generator (async generator on AsyncClient) that walks the cursor transparently and yields every Event:

  for event in va.iter_events(monitor.id):
      print(event.seq, event.value)
  

va.fleet_status() returns the tenant's pooled certificate: fleet-confirmed discoveries carry FDR <= alpha under arbitrary dependence between streams (e-LOND). Warn-tier alarms never enter that pool.

CLI

Installing the package also installs a tiny validanytime command:

validanytime init                    # write a starter monitoring script (no network)
validanytime backtest metrics.csv    # replay a CSV through the OFFLINE detector
                                     # stack — `pip install 'validanytime[detectors]'`,
                                     # runs locally, nothing leaves your machine

backtest prints the same machine-readable gate verdict the hosted POST /v1/onboarding/backtest returns (add --json for the full dict) and exits non-zero when the gate fails, so it drops straight into CI.

Configuration

Setting	Argument	Environment variable	Default
API key	api_key	VALIDANYTIME_API_KEY	— (required)
Base URL	base_url	VALIDANYTIME_API_URL	https://api.validanytime.com

Authentication is Authorization: Bearer va_... — mint keys from the dashboard (the plaintext key is shown once).

Retries and idempotency

• The SDK retries requests that fail with 429, 502, 503, or 504 up to 3 times, with exponential backoff plus jitter; a Retry-After header (seconds) is honored. No other 4xx is ever retried.
• Ingestion is idempotent per event_id: the server deduplicates, so an event with an event_id is safe to send twice — the second attempt returns duplicate=True, accepted=0. Pass your own event_id when you have a natural one, or pass retry=True to ingest() and the SDK mints a uuid4 event_id client-side, so automatic retries (and your own re-sends of the same payload) can never double-count.
• Ingesting without an event_id is not deduplicated; a retried request could then count twice. Prefer event_id/retry=True in production.

Errors

All SDK errors derive from validanytime.ValidAnytimeError and carry status_code and the server's detail:

• AuthError (401) — missing/invalid API key.
• NotFound (404) — unknown monitor (or another tenant's).
• ValidationError (422) — the server's message is designed to be actionable (e.g. it lists valid template ids or known config keys) and is surfaced verbatim.
• ServerError (5xx) — raised after retryable statuses are exhausted.

Development

The test suite is self-contained — it drives the client through httpx.MockTransport, so no account, API key, or backend is required:

pip install -e '.[dev]'
pytest                 # tests
ruff check .           # lint
ruff format --check .  # formatting
pyright                # types

See CONTRIBUTING.md for scope and pull-request guidelines.

License

Apache-2.0 © 2026 Compiled Intelligence, Inc. See NOTICE for attribution.