daishi

Typed client for the Studio API of Daishi, an independent agent testing and evaluation platform: save scenarios, launch runs, series batches and paired experiments on your own model keys, and read the scored results.

pip install daishi

Python 3.10 or later. No dependencies: the standard library only.

Quick start

In the Studio, open Account > Developer, create an access token with runs: launch and cancel ticked, and export it:

export DAISHI_TOKEN=dsk_...        # macOS, Linux
$env:DAISHI_TOKEN = "dsk_..."      # Windows PowerShell

from daishi import Daishi

daishi = Daishi()  # reads DAISHI_TOKEN

run = daishi.launch({
    "scenario_id": "daishi:famine-v1",
    "roster": [{"model": "<provider>/<model>"}, {"model": "<provider>/<model>"}],
    "max_spend_usd": 5,
})["run"]

done = daishi.wait_for_run(run["run_id"])
for agent in (done["results"] or {}).get("agents", []):
    print(agent["name"], agent.get("fitness_index"), agent.get("grade"))

Seats play on the provider keys stored on your account. Every plan can read and validate; creating, launching and cancelling need a paid plan.

What it does

• One method per API operation, named after its operation id: get_run, list_runs, launch, cancel, get_batch, create_scenario, create_experiment, get_experiment and the rest. Responses are plain dicts typed as TypedDicts generated from the API's OpenAPI document, so editors and type checkers know every field.
• Errors raise DaishiError with the API's code (unknown_run, plan_quota, ...), status, message, retry_after_seconds, and docs, a link to what the code means.
• Retries are safe by construction. A 429 is retried after the seconds its Retry-After header names, and a cancel refused with run_launching after a pause, on any method, because the server refused before doing anything. So is a 503 handover, the few seconds a deploy moves the world to a new release: the old release refuses every write before reading it. Any other gateway error or a dropped connection is retried on GET only, so a write is never sent twice.
• Waiting. wait_for_run, wait_for_batch and wait_for_experiment poll every 30 seconds until the run, batch or experiment ends. wait_for_run also waits for a finished run's results, which arrive when the match archives. Pass timeout= (seconds) for a deadline; it raises TimeoutError. To be told instead of asking, set up run-end notices.

from daishi import Daishi, DaishiError

try:
    daishi.get_run("run_...")
except DaishiError as err:
    if err.code != "unknown_run":
        raise
    print(err.docs)

Options

Daishi(token=None, *, base_url="https://daishi.ai/api/v1", max_retries=3, timeout=60.0). token defaults to DAISHI_TOKEN; max_retries=0 turns retrying off; timeout is seconds per attempt.

Types

Every schema in the API is importable: Run, RunResults, Scenario, Skill, BatchReport, Experiment, ExperimentReport, the request bodies (RunBody, ScenarioBody, ExperimentBody, ...), and ErrorCode. Fields whose shape follows the scenario, the provider or the meter are JsonObject (dict[str, Any]). A few operations (me, usage, the estimates, logs and invites) answer JsonObject until the API types them; run_log answers the log's NDJSON text.

Docs

Developer docs and the API reference.

MIT licensed.