synthiaresearch 0.0.10

Python SDK for the Synthia API

synthiaresearch

Synthia evals your AI agent against simulated users: probe it to infer who it's for, generate a synthetic scenario dataset, play the scenarios against it, and judge every rollout — locally from this SDK, or as a CI gate on every pull request.

pip install synthiaresearch   # Python ≥ 3.10; the import name is `synthia`

Quickstart

from synthia import Synthia

client = Synthia()  # reads SYNTHIA_API_KEY / SYNTHIA_BASE_URL

# Probe + generate only when needed; re-runs reuse this script's dataset.
prepared = client.prepare(lambda probe: my_agent.respond(probe))

# Play the scenarios against your agent; tools run in a deterministic sandbox.
def rollout_agent(transcript, sandbox):
    return my_agent.respond_with_tools(transcript, sandbox)

results = client.rollouts.run(rollout_agent, prepared.dataset,
                              agent_meta={"name": "my-agent"})

# Judge every rollout server-side.
check = client.rollouts.quality_check(results)
check.wait(verbose=True)
print(check.rollouts())

In CI

The package ships the synthia CLI (also python -m synthia): point a synthia.yaml at a module exporting agent(transcript, sandbox) -> reply and gate PRs on the pass rate —

synthia validate   # check config + entrypoint, no network
synthia run        # play the suite, judge it, gate on thresholds

— or use the GitHub Action with language: python. Setup guide: Synthia in CI.

Documentation

• Python API reference — every method, plus the deliberate divergences from the JS SDK (no ready(); wait(**kwargs); rollouts run on a thread pool, so your agent callable must be thread-safe).
• CLI reference · synthia.yaml · environment & sessions
• Runnable examples: examples/

Mirrors the JS/TypeScript SDK (synthiaresearch on npm) — same API surface, same CLI, same yaml, byte-identical results-JSON contract. Replies and tool events are redacted by default before upload. MIT licensed.

Development

• src/synthia/client.py — the SDK: the Synthia client, resource classes, and the local probe/rollout loops with tool-call tracing.
• src/synthia/cli/ — the synthia CLI, a module-for-module mirror of the JS CLI (same schema, same exit codes, same report contract).
• pyproject.toml — the synthiaresearch package definition (httpx and pyyaml as the only runtime dependencies; the import name stays synthia).