latitude-telemetry-prime-intellect

Export Prime Intellect Verifiers eval rollouts to Latitude as OTLP traces (and optional custom scores from rewards/metrics).

This is the Prime Intellect counterpart to the other harness integrations (latitude-telemetry-hermes, Claude Code, Pi, OpenClaw). Verifiers has no host plugin entry point for observability, so this package ships as a library + CLI you call from your eval script or after a run.

Install

pip install latitude-telemetry-prime-intellect

export LATITUDE_API_KEY=lat_xxx
export LATITUDE_PROJECT=my-project

Usage

After run_eval (recommended for scripts)

from verifiers.v1.cli.eval.runner import run_eval
from latitude_telemetry_prime_intellect import export_episodes

episodes = await run_eval(env, config)
export_episodes(episodes)  # ships traces + reward scores, then flushes

As Env.run_slot on_complete

from latitude_telemetry_prime_intellect import make_on_complete

on_complete = make_on_complete(next=persist_episode)  # chain your append_episode
await env.run_slot(slot, ctx, semaphore, on_complete)

Post-hoc from a results directory

After uv run eval … / prime eval …:

latitude-prime-intellect-export export ./outputs/<run-dir>
# or
python -m latitude_telemetry_prime_intellect export ./outputs/<run-dir>

Reads traces.jsonl, episodes.jsonl, or results.jsonl when present.

Configuration

Telemetry stays off until both LATITUDE_API_KEY and a project are set. Export is fail-open: a Latitude error never fails your eval.

How it works

Each Verifiers Trace becomes one Latitude trace:

interaction (rollout root; session = eval/episode id)
├── llm_request      (one per ModelCall: model, usage, finish reason, messages)
└── tool_call:<name> (tool_execution; one per tool result)

Rewards map to custom scores (sourceId = verifiers.reward.<name>) against the same trace id (Verifiers' 32-hex Trace.id).

Development

cd packages/telemetry/prime-intellect
uv sync --all-groups
uv run pytest tests/ -x