o11y-one
The Python client for the O11y One API. A thin, hand-written layer over the
generated o11y-one-api-agentic package.
Install
uv add o11y-one # or: pip install o11y-one
Use
import os
from o11y_one.sdk import O11yClient, Disposition, classify
from o11y_one.agentic.v1.evaluation_pb2 import ListEvaluationDefinitionsRequest
from o11y_one.agentic.v1.evaluation_connect import AgenticEvaluationServiceClientSync
o11y = O11yClient(
base_url="https://grpc.o11y.one",
credential=os.environ["O11Y_API_KEY"], # o11y_mach.<selector>.<secret>
org_id=os.environ["O11Y_ORG_ID"],
)
evals = o11y.service_sync(AgenticEvaluationServiceClientSync)
try:
defs = evals.list_evaluation_definitions(ListEvaluationDefinitionsRequest())
except Exception as err: # noqa: BLE001 - classify sorts it out
failure = classify(err)
if failure.disposition is Disposition.INSUFFICIENT_SCOPE:
raise SystemExit(f"credential is missing scope {failure.missing_scope}") from err
raise
What is here, and what deliberately is not
Three modules and nothing else:
| module | responsibility |
|---|---|
o11y_one.sdk.client |
transport construction, credential and scoping header injection |
o11y_one.sdk.auth |
the credential wire format and local structural validation |
o11y_one.sdk.errors |
the failure taxonomy: reauthenticate / insufficient-scope / version-skew / retry / unclassified |
Service methods are not wrapped. o11y-one-api-agentic already ships a client class
per service; a hand-written facade over all of them would be a second API surface
to keep in sync with the proto, and it would rot the first time a field is added
upstream. Import the generated client class and hand it to
O11yClient.service_sync() / O11yClient.service().
Record and replay
An externally executed candidate's runtime routes each model and tool call
through a lease tape. Under record, live runs and the call is buffered, then
sent to RecordEvaluationCaseSteps within the bounds the server publishes on
sdk_record_replay. Under replay, live never runs: each call is answered from
the source recording through LookupReplayStep. The harness code is the same
in both modes.
evals = AgenticClientSync(client.service_sync(AgenticEvaluationServiceClientSync))
tape = evals.record_lease(evaluation_run_id=run_id, lease=lease) # or replay_lease
case = tape.case(leased_case)
reply = case.model(request_json, lambda: call_model(request_json))
hits = case.tool("search", args_json, lambda: search(args_json))
case.finish()
tape.flush() # record, then submit
evals.submit_case_outputs(...)
AgenticClient returns the async twin, whose live is a coroutine function.
The sync tape is safe to share across the threads a harness fans its cases out on.
ReplayDivergedErrornames the step, the divergence kind and both request digests. The server has already made it the case's verdict: stop the case and do not submit it.LeaseRefusedErrorcarries a lease refusal (CASE_ALREADY_SUBMITTED,LEASE_EXPIRED, ...).RecordingUnavailableErrormeans this deployment cannot record or replay, and is not retryable.- A
ConnectErrorwith codeUNAVAILABLEis transient; the buffered calls are kept and the nextflush()resends them. - Steps that never reached a flush are lost with the process, and the case
replays as
RECORDING_INCOMPLETE. - Redaction is the server's, before anything is digested or stored. The SDK redacts nothing and logs no call.
The tape records and replays. o11y-eval run --record|--replay --candidate <key>
only asserts, before launching, that the candidate is the kind that mode needs;
the runner never executes a harness.
Code scorers
A code scorer is a Python score(input, context) the server runs in its sandbox.
A member drafts and previews one; an admin publishes it, and only a published
version runs across a dataset. o11y_one.sdk.scorer types the contract.
These verbs are a person's, never a machine credential's, so the client carries no credential and each call carries the person's token:
from connectrpc.request import Headers
from o11y_one.sdk.code_scorer import CodeScorerClientSync
o11y = O11yClient(base_url="https://grpc.o11y.one", org_id=os.environ["O11Y_ORG_ID"])
scorers = CodeScorerClientSync(o11y.service_sync(AgenticEvaluationServiceClientSync))
me = Headers({"authorization": f"Bearer {os.environ['O11Y_USER_TOKEN']}"})
draft = scorers.create_code_scorer(
config_key="exact-answer",
name="Exact answer",
metric_name="quality.exact",
files={"score.py": open("score.py").read()},
idempotency_key="exact-answer-v1",
headers=me,
)
result = scorers.preview_code_scorer( # waits at most wait_seconds for the sandbox
score_config_id=draft.score_config_id,
scorer_input={"candidate_output": "42", "reference_output": "42"},
idempotency_key="exact-answer-preview-1",
headers=me,
)
publishing = scorers.publish_code_scorer( # admin; GetEvaluationScorerConfig reads the outcome
score_config_id=draft.score_config_id,
smoke_inputs=[{"candidate_output": "42", "reference_output": "42"}],
idempotency_key="exact-answer-publish-1",
headers=me,
)
Each returns a Refusal for a typed rejection (SANDBOX_UNAVAILABLE,
CODE_SPEC_INVALID, CODE_INPUT_INVALID, CODE_VERSION_IMMUTABLE, ...), whose
message starts with the server's stable code.
A TypeScript scorer runs where the deployment serves the typescript-base
runtime. Its bundle is .ts files with the entry score.ts, and it imports
only its own files (relative, naming the .ts) and Node's built-ins: no npm
packages and no network. scorer-kit holds its types,
a seeded PRNG to copy into the bundle and an example score.ts. There,
context.seed and context.frozen_evaluation_time_unix_nanos are bigints.
from pathlib import Path
from o11y_one.agentic.v1.evaluation_pb2 import GetAgenticEvaluationCapabilitiesRequest
service = o11y.service_sync(AgenticEvaluationServiceClientSync)
served = service.get_agentic_evaluation_capabilities(
GetAgenticEvaluationCapabilitiesRequest(), headers=me
).capabilities
kit = Path("packages/sdk-ts/scorer-kit")
draft = scorers.create_code_scorer(
config_key="sampled-lines",
name="Sampled lines",
metric_name="quality.sampled_lines",
language="typescript",
files={f: (kit / f).read_text() for f in ("score.ts", "o11y_scorer.ts", "o11y_prng.ts")},
capabilities=served, # its served TypeScript runtime; without it, typescript-base v1
idempotency_key="sampled-lines-v1",
headers=me,
)
A path that is not identifiers ending in the language's extension, or a
TypeScript entry other than score.ts, raises ValidationError before the
wire, and so does a language the capabilities serve no runtime for
(code_language_unsupported).
The distinction this package exists to preserve
UNAUTHENTICATED and PERMISSION_DENIED are different problems:
UNAUTHENTICATED— the credential is absent, malformed, unknown, expired, or revoked. Re-auth. Retrying cannot change the answer.PERMISSION_DENIED— the credential is fine; it lacks a scope (the server names which one) or it was presented to a non-machine surface.
classify() keeps them apart, along with UNAVAILABLE (auth backend down —
retry with backoff) and INVALID_ARGUMENT (SDK/server version skew). Collapsing
these into a single "auth error" is the failure mode this module exists to
prevent.
Credentials
A machine credential is o11y_mach.<selector>.<secret>, presented in the
x-o11y-key header. It is:
- returned exactly once, by
CreateMachineCredential. There is no read-back RPC and no recovery path — lose it and you rotate. - rotated create-then-revoke, not atomically. A principal may hold several active credentials at once, which is what makes zero-downtime rotation work: create the new one, deploy it, then revoke the old.
- revoked effective on the next request, not on a TTL boundary.
- expiring at +365 days by default, capped at three years.
Never send it as authorization: Bearer — that path carries the browser session
JWT and a machine credential presented there is rejected.
Metadata
Release files for o11y-one 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| o11y_one-0.2.0.tar.gz | 34.7 kB | Without provenance | Details for o11y_one-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| o11y_one-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for o11y_one-0.2.0-py3-none-any.whl |
Total release size: 75.0 kB
Release files / o11y_one-0.2.0.tar.gz
| Download URL | o11y_one-0.2.0.tar.gz |
|---|---|
| Size | 34.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
17e45a90ade034e5eacc562a5d6bbbcf3e9d28e90ed7366f45a2e5b832022d5a
|
|
BLAKE2b-256 checksum How to use checksums |
c40483f9e253d5e927da6261645879e501ad6e9ba6144b699a7414ceabda5891
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / o11y_one-0.2.0-py3-none-any.whl
| Download URL | o11y_one-0.2.0-py3-none-any.whl |
|---|---|
| Size | 40.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bb9ba974dd6c4741c7915bfaf8183a678a0fc1bd500889e02179a5a7185ec507
|
|
BLAKE2b-256 checksum How to use checksums |
aa158044088ef21736d17a45189bd4cab899a621b66ba8f9b50693e1c1ea2cb8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for o11y-one 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.