assay-sdk

The Python package for Assay, a self-hosted LLM tracing and evaluation platform.

uv add assay-sdk

The distribution is named assay-sdk and imported as assay.

Tracing

Content capture is off by default. Enable it explicitly when the traced input and output may be sent to Assay:

import assay

assay.init(
    endpoint="http://localhost:8080",
    api_key="asy_...",
    application="support-bot",
    capture=True,
)


@assay.trace
def answer(question: str) -> str:
    return "Assay evaluates AI systems."

Use assay.span(...) for explicit spans and call set_input, set_output, set_context, or set_reference on the active span. Call assay.flush() before a short-lived process exits and assay.shutdown() when the tracing lifecycle ends.

API Client

import assay

with assay.Client("http://localhost:8080", admin_token="...") as client:
    applications = client.applications.list()

Management, dataset, scorer, and run operations use an admin token. Trace inspection and scoring use a project API key.

CLI

Set ASSAY_ENDPOINT and the relevant credential, then use the management and evaluation commands:

assay projects list
assay apps list --project PROJECT_ID
assay datasets import APPLICATION_ID --file regression.jsonl
assay run create APPLICATION_ID --dataset DATASET_ID --scorers groundedness,correctness
assay run watch RUN_ID --gate groundedness:0.8
assay traces score --scorer correctness TRACE_ID
assay scores export APPLICATION_ID --failed --format jsonl
assay datasets from-trace DATASET_ID TRACE_ID --scorer groundedness
assay metrics APPLICATION_ID --scorer groundedness

Commands emit JSON. assay run watch returns exit code 1 when a run fails or a gate is not met, which makes it suitable for CI checks.

M6 commands are available from this checkout via uv run assay .... Metrics and score export require admin authentication and default to 30 days. --start and --end accept timezone-bearing timestamps for ranges up to 366 days. Trace imports preserve the selected scorer's latest evidence and reject duplicate trace/scorer pairs without overwriting existing items.

The optional tests/test_live_workflow.py acceptance test requires ASSAY_LIVE_TEST_ENDPOINT and ASSAY_ADMIN_TOKEN. It makes real judge calls using synthetic data and deletes its project.