daishi
Typed client for the Studio API of Daishi, an independent agent testing and evaluation platform: save scenarios, launch runs, series batches and paired experiments on your own model keys, and read the scored results.
pip install daishi
Python 3.10 or later. No dependencies: the standard library only.
Quick start
In the Studio, open Account > Developer, create an access token with runs: launch and cancel ticked, and export it:
export DAISHI_TOKEN=dsk_... # macOS, Linux
$env:DAISHI_TOKEN = "dsk_..." # Windows PowerShell
from daishi import Daishi
daishi = Daishi() # reads DAISHI_TOKEN
run = daishi.launch({
"scenario_id": "daishi:famine-v1",
"roster": [{"model": "<provider>/<model>"}, {"model": "<provider>/<model>"}],
"max_spend_usd": 5,
})["run"]
done = daishi.wait_for_run(run["run_id"])
for agent in (done["results"] or {}).get("agents", []):
print(agent["name"], agent.get("fitness_index"), agent.get("grade"))
Seats play on the provider keys stored on your account. Every plan can read and validate; creating, launching and cancelling need a paid plan.
What it does
- One method per API operation, named after its operation id:
get_run,list_runs,launch,cancel,get_batch,create_scenario,create_experiment,get_experimentand the rest. Responses are plain dicts typed asTypedDicts generated from the API's OpenAPI document, so editors and type checkers know every field. - Errors raise
DaishiErrorwith the API'scode(unknown_run,plan_quota, ...),status,message,retry_after_seconds, anddocs, a link to what the code means. - Retries are safe by construction. A 429 is retried after the seconds its
Retry-Afterheader names, and a cancel refused withrun_launchingafter a pause, on any method, because the server refused before doing anything. So is a503 handover, the few seconds a deploy moves the world to a new release: the old release refuses every write before reading it. Any other gateway error or a dropped connection is retried on GET only, so a write is never sent twice. - Waiting.
wait_for_run,wait_for_batchandwait_for_experimentpoll every 30 seconds until the run, batch or experiment ends.wait_for_runalso waits for a finished run's results, which arrive when the match archives. Passtimeout=(seconds) for a deadline; it raisesTimeoutError. To be told instead of asking, set up run-end notices.
from daishi import Daishi, DaishiError
try:
daishi.get_run("run_...")
except DaishiError as err:
if err.code != "unknown_run":
raise
print(err.docs)
Options
Daishi(token=None, *, base_url="https://daishi.ai/api/v1", max_retries=3, timeout=60.0). token defaults to DAISHI_TOKEN; max_retries=0 turns retrying off; timeout is seconds per attempt.
Types
Every schema in the API is importable: Run, RunResults, Scenario, Skill, BatchReport, Experiment, ExperimentReport, the request bodies (RunBody, ScenarioBody, ExperimentBody, ...), and ErrorCode. Fields whose shape follows the scenario, the provider or the meter are JsonObject (dict[str, Any]). A few operations (me, usage, the estimates, logs and invites) answer JsonObject until the API types them; run_log answers the log's NDJSON text.
Docs
Developer docs and the API reference.
MIT licensed.
1 maintainer
Metadata
1 maintainer
Release files for daishi 0.2.16
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| daishi-0.2.16.tar.gz | 25.6 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| daishi-0.2.16-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 52.6 kB
Release files / daishi-0.2.16.tar.gz
| Download URL | daishi-0.2.16.tar.gz |
|---|---|
| Size | 25.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f2f054b33cd321c1ead67a5c8847d8d11925b6f7ab5ad0ad2601a4d3a8ecaaf1
|
|
BLAKE2b-256 checksum How to use checksums |
6f21516d7e322582c2993f2decf66b7edf74df17b8d44dfea685a1794a00f54c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / daishi-0.2.16-py3-none-any.whl
| Download URL | daishi-0.2.16-py3-none-any.whl |
|---|---|
| Size | 27.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3832905a4eb4d55c869edb057f67efd2b2c75bd17c50b17e309eadd2902edb6e
|
|
BLAKE2b-256 checksum How to use checksums |
fbd4abaf3aec29b6258b4c4a200ed1857a71bfcf4372e4f07442a09d9931be69
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Security for daishi 0.2.16
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
