soit: the SOIT command line

A small client for the SOIT API: sign in with an API key, run an agent, replay regression sets on a new model, run and manage evaluation datasets, and take evidence out.

uv tool install soit-cli           # or: pipx install soit-cli
soit login --url https://soit.example.com

The package on PyPI is soit-cli; the command it installs is soit. To run the checkout instead, for example to try a change before it is released, install from the repository:

uv tool install ./cli        # or: pipx install ./cli

soit login asks for an API key (create one under Settings › API), checks it, and stores it with the address in your configuration directory, readable by you alone. In CI, set SOIT_API_URL and SOIT_API_KEY instead; the environment wins over the stored file. SOIT_CONFIG names another file.

Exit codes: 0 success, 1 a refusal or failure, 2 a usage error, 3 an eval --fail-on-regression that found cases passing on the current model and failing on the candidate, so a CI job can stop a model switch:

soit eval model:openai:gpt-6 --dataset smoke --fail-on-regression

soit eval run exits 3 the same way: with --fail-on-regression when a case that passed in the report's baseline fails now, and with --fail-on-failure when any case fails, as the publish gate would. A CI job can keep a dataset in the repository and evaluate every change against it:

soit dataset import refunds evals/refunds.jsonl --agent agt_123 --create
soit eval run agt_123 --dataset refunds --fail-on-regression