latenzy

Documentation: amitpatole.github.io/latenzy

Per-model LLM latency monitoring for enterprises. latenzy is a synthetic prober and Prometheus exporter that measures what the lab-level status pages can't: the latency your account gets from each model — claude-sonnet-4-6 vs gpt-4o vs gemini-2.0-flash, not "Anthropic is up".

Latency is tenant-specific: it depends on your rate-limit tier, your region, and the path you take to the model (direct API, Bedrock, Vertex). latenzy runs inside your network on your keys and exports per-model metrics your existing Prometheus + Grafana stack can alert on.

What it measures

Every probe cycle, for each configured (provider, model, endpoint, prompt_class):

All metrics are labelled source, provider, model, endpoint, prompt_class.

Histograms are observed only for successful probes, so failures never skew latency percentiles. Prompts are deterministic per prompt_class (small/medium/large) — comparing models on unequal inputs is meaningless.

Every metric carries a source label: synthetic for the prober's canaries and live for real application traffic (see below). One dashboard shows both.

Live traffic (passive instrumentation)

The prober answers "is this model slow right now"; to also chart your own traffic's p95, wrap real LLM calls with the framework-agnostic LiveRecorder. It emits the same metric names under source="live", so the same dashboards work — no separate pipeline.

from latenzy import LiveRecorder, Metrics, classify_prompt, measure_stream

recorder = LiveRecorder(Metrics())  # shares your app's Prometheus registry

with recorder.observe(
    provider="openai", model="gpt-4o", prompt_class=classify_prompt(text=prompt)
) as obs:
    for chunk in measure_stream(client.stream(prompt), obs):  # marks first-token timing
        handle(chunk)
    obs.output_tokens = n_tokens

Label values are charset-validated (they may come from user input), so a host app can't explode metric cardinality. A raised exception is recorded as an error outcome and re-raised. A runnable, key-free walkthrough is in examples/demo_live.py:

$ python examples/demo_live.py
recorded live call: model=gpt-4o prompt_class=small tokens=4
recorded live call: model=gpt-4o prompt_class=large tokens=9

--- /metrics (live source) ---
latenzy_ttft_seconds_count{...,prompt_class="small",...,source="live"} 1.0
latenzy_ttft_seconds_sum{...,prompt_class="small",...,source="live"} 0.0553...
latenzy_ttft_seconds_sum{...,prompt_class="large",...,source="live"} 0.3202...
latenzy_probes_total{...,outcome="ok",prompt_class="small",...,source="live"} 1.0

OpenTelemetry

Emit to OpenTelemetry as well as (or instead of) Prometheus with the otel extra (pip install 'latenzy[otel]'). The prober and LiveRecorder take any RecordSink, so FanoutSink(Metrics(), OTelBridge(meter)) sends to both; instrument names follow the OTel GenAI conventions (gen_ai.client.operation.duration, gen_ai.client.token.usage). For latenzy run, enable it in config:

otel:
  enabled: true
  endpoint: https://collector.internal:4318/v1/metrics   # omit → console

Quick start

pip install latenzy

export ANTHROPIC_API_KEY=... OPENAI_API_KEY=... GEMINI_API_KEY=...
latenzy doctor -c latenzy.yaml   # validate config, check keys are present
latenzy once   -c latenzy.yaml   # one probe cycle, human-readable results
latenzy run    -c latenzy.yaml   # probe on an interval + serve /metrics

See latenzy.example.yaml for the full configuration. API keys are read from environment variables only — they have no place in config files.

Grafana + Prometheus

• Dashboard — Grafana dashboard library ID 25642 (Dashboards → Import → 25642), or import dashboards/latenzy-model-comparison.json: the model-comparison view (TTFT p95, total-latency p95, tokens/sec, failure and 429 ratio, staleness) filterable by provider, model, endpoint, and prompt class. Import it into any Grafana; it prompts for your Prometheus datasource. The dashboards-share/ copy is the same dashboard in Grafana's "export for sharing externally" format for grafana.com uploads (generated by scripts/share_dashboard.py; a test keeps the two in sync).
• Recording rules — prometheus/recording_rules.yml: hourly and daily p50/p95/p99 series (latenzy:ttft_seconds:p95_1h, ...), so dashboards and alerts never recompute histogram quantiles.
• Alert rules — prometheus/alert_rules.yml: probe staleness, TTFT SLO breach, rate-limit pressure, failure ratio.

Standalone bundle (no existing Grafana needed)

cd deploy
mkdir -p secrets && openssl rand -hex 32 > secrets/latenzy_token
export ANTHROPIC_API_KEY=... OPENAI_API_KEY=... GEMINI_API_KEY=...
export GRAFANA_ADMIN_PASSWORD=...   # no default password ships with the bundle
docker compose up -d                # prober + Prometheus + Grafana, pre-provisioned

Grafana serves the comparison dashboard read-only at http://localhost:3000 (loopback-published only). The bundle mounts the same dashboards/ and prometheus/ files from the repo, so the bundled and published copies cannot drift.

Security posture

• Binds 127.0.0.1 by default. Binding a routable interface refuses to start unless exporter.auth_token_env is set; the token is checked in constant time.
• API keys are sent in request headers only and never appear in logs, URLs, metrics, or error output.
• Probe cost is bounded: max_output_tokens defaults to 16.

Status

v0.1.0 — first real release: prober + exporter, Grafana dashboard (library ID 25642), recording/alert rules, standalone bundle, security-hardened through a four-round red-team loop (see SECURITY.md). Now also: passive live-traffic instrumentation (LiveRecorder, source="live") and an OpenTelemetry meter bridge (latenzy[otel]). Docs: amitpatole.github.io/latenzy.

License: AGPL-3.0-only. Dual licensing available for enterprises — contact the author.

— amitpatole