evaluation-client 0.1.0

Python SDK for interacting with the Chatbot Evaluation API.

Evaluation Client SDK

Usage examples and sample artefacts now live alongside the SDK in services/client-examples/. Copy the .env.example file in that directory, adjust any paths/URLs, and run the scripts via uv run --project services/client-examples -m examples.<script> or ./scripts/client.sh examples.<script>.

Installation

Install from PyPI or an internal index once the package is published:

pip install evaluation-client

Client SDK quickstart

Instantiate the reusable SDK from clients.evaluation_client to interact with the API:

from pathlib import Path

from evaluation_client import (
    ApiConfig,
    AssetBundle,
    EvaluationClient,
    LocalPaths,
    RunOptions,
)

assets_root = Path("assets").resolve()
runs_root = Path("runs-out").resolve()
bundle = AssetBundle(
    datasets={
        "my-dataset@1": assets_root / "datasets" / "my-dataset" / "v1" / "data.jsonl",
    },
    judges={
        "judge@1": assets_root / "judges" / "llm" / "judge" / "v1" / "config.yaml",
    },
)

api_cfg = ApiConfig(base_url="http://localhost:8000")
local_cfg = LocalPaths(runs_root=runs_root, download_root=runs_root)
client = EvaluationClient.from_configs(api_cfg, local=local_cfg)
result = client.run_suite_with_assets(
    suite=assets_root / "suites" / "my-suite.yaml",
    assets=bundle,
    options=RunOptions(log_level="normal"),
)
print(result.status, result.run_uri)

See services/client-examples/examples/ for richer console flows built on this SDK. They demonstrate how to build custom AssetBundle instances for prompts, judges, datasets, or retrieval bundles and how to tune monitoring verbosity via RunOptions.

If you have already uploaded specific assets, pass assets=None (the default) and the client will skip the upload step.