susatest-agent 0.7.4

One CLI for SUSATest — autonomous QA, LLM observability, and production SRE for any product

susatest-agent

One CLI for SUSATest — autonomous QA, LLM observability, and production SRE for any product, in your terminal.

Install

pip install susatest-agent

Quick start

# 1. Sign in once (opens your browser)
susatest-agent login

# 2. Instrument your existing app with one command — zero SDK install
susatest-agent run -- python app.py
# (was: python app.py)

# 3. Open the dashboard to see traces, evals, and findings
susatest-agent dashboard

After susatest-agent login, every other command picks up your saved JWT from ~/.susatest/credentials. No more --api-key on every invocation.

Commands

Auth

Command	What it does
susatest-agent login	OAuth device-code flow. Opens browser, you sign in once, JWT saved locally with 30-day expiry.
susatest-agent logout	Revokes the JWT server-side and clears local credentials.
susatest-agent dashboard	Opens the SUSATest dashboard in your default browser, pre-authenticated.

Instrument your own app (zero code touch)

Command	What it does
susatest-agent run -- <your-command>	Wraps your existing app launch with OpenTelemetry auto-instrumentation. Every OpenAI / Anthropic / LangChain / etc. LLM call your app makes auto-streams to your susaeval deploy as traces — no pip install susaeval, no susaeval.init(), no with session() wrappers. Python supported in 0.5.0; Node and Java coming. Example: susatest-agent run -- python app.py

The wrapper detects the language from the wrapped command and routes through opentelemetry-instrument. First run prompts you to install the OTel libs (single pip install line printed) — we never install behind your back. After install, instrumentation is seamless on every subsequent run.

Testing your app

Command	What it does
susatest-agent test <apk-or-url>	Upload an APK or point at a web URL; SUSATest runs an autonomous exploration test with the persona you pick and returns an HTML / JSON / JUnit report.
susatest-agent connect	Bridge a locally-attached Android device to the cloud platform — tests then run against your physical phone.
susatest-agent prepare-device	One-time setup for a device that will run agent-side network capture (CA cert, frida, marker).
susatest-agent fleet-prep	Bulk-bootstrap a phone for the fleet — frida + CA + marker + SUSA SMS forwarder + phone-vault registration in one shot.
susatest-agent connect-windows	Connect this Windows machine to SUSATest as a desktop-test agent.
susatest-agent connect --ios	Bridge locally-attached iPhone(s) to the cloud platform via WebDriverAgent — they register in the fleet (os_kind=ios) and tests run against your physical iOS device. Run ios-provision first.
susatest-agent ios-sign	Download the prebuilt standard WebDriverAgent and re-sign it with your Apple cert + provisioning profile (zsign). No Mac.
susatest-agent ios-provision	Install the signed WebDriverAgent on a connected iPhone and run it (go-ios).

iOS setup (no Mac): needs a paid Apple Developer account, plus zsign and go-ios on PATH. Flow: ios-sign → ios-provision → connect --ios. Prefer an iOS 18 device (go-ios has an open iOS 26 tunnel issue). Full runbook in the repo: scripts/ios_lab/README.md.

Strategist (autonomous QA brain)

Command	What it does
susatest-agent strategist list	List your strategists (each one is bound to a product surface).
susatest-agent strategist run <id> [--action gap_explore] [--wait]	Fire an on-demand cognition cycle. With --wait the CLI blocks until the cycle row lands.
susatest-agent strategist status <id>	Show the strategist's state plus the last 5 cycle summaries.

Opsy (production SRE signals)

Command	What it does
susatest-agent opsy list-incidents [--status open --severity high]	List incidents opsy has observed in production.
susatest-agent opsy list-alerts [--status fired]	List firing alerts.
susatest-agent opsy status	Health / status summary.
susatest-agent opsy fire-incident --title T --severity S [--services svc1 svc2]	Create a synthetic incident — verifies the reactive-to-proactive loop. The next strategist cycle ingests it via opsy_signals.

Susaeval (LLM observability)

Command	What it does
susatest-agent susaeval list-sessions [--gate-decision rollback]	List recent LLM sessions. gate_decision=rollback surfaces sessions where eval judges flagged a regression.
susatest-agent susaeval list-incidents	List susaeval-side incidents.
susatest-agent susaeval drift [--feature model --hours 24]	Show distribution drift (PSI) on a feature over the last N hours.
susatest-agent susaeval list-safety	List safety / guardrail findings.

Authentication

One susatest-agent login — one JWT — every subcommand.

After login, the saved JWT (~/.susatest/credentials, 30-day expiry) is honored by test, strategist, opsy, susaeval, dashboard, and the OTel exporter that run -- sets up. Server-side fan-out hands the right token to each backend; you never see OPSY_API_KEY or SUSAEVAL_API_KEY again.

Legacy paths still work for CI scripts and pinned older deploys:

• --api-key <key> flag — overrides the saved JWT for one call.
• SUSA_API_KEY env var — picked up by every subcommand if set.
• OPSY_API_KEY / SUSAEVAL_API_KEY env vars — only needed if your CI pinned them before 0.4.2; new setups should leave them unset and use login.

Requirements

• Python 3.9+
• ADB installed (adb in PATH) if you'll connect Android devices — comes with Android Studio
• For iOS: a paid Apple Developer account, plus zsign and go-ios on PATH (no Mac required). See scripts/ios_lab/README.md.

Known limitations

These exist today and are on the roadmap:

• run -- is Python-only in 0.5.x. Node and Java auto-instrumentation are on the roadmap. The wrapper recognises both today and prints clear next-step guidance instead of failing silently.
• stdout/stderr → opsy log forwarding not yet auto-piped by the run wrapper. LLM traces flow to susaeval today; application log forwarding to opsy is on the roadmap.
• Three dashboards. Strategist / opsy / susaeval each have their own UI today. Unified nav is roadmap.

Resources

• Homepage: https://susatest.com
• Docs: https://susatest.com/docs
• Source / issues: https://github.com/monk0062006/SUSA
• Status: https://susatest.com/status

Changelog

0.7.1

• Docs: document the iOS commands (connect --ios, ios-sign, ios-provision).

0.7.0

• iOS support (SPEC-143). Drive physical iPhones via WebDriverAgent, no Mac: ios-sign (re-sign the prebuilt WDA with zsign) + ios-provision (install/run via go-ios) + connect --ios (register iPhones in the fleet and serve iOS sessions). Requires a paid Apple Developer account + zsign + go-ios. Additive — existing Android/web/Windows paths unchanged.

0.5.1

• Docs: clarified that a single susatest-agent login JWT covers every subcommand (server-side fan-out is live as of 0.4.2). No code changes.

0.5.0

• susatest-agent run -- <cmd> — auto-instrumentation wrapper for customer apps. Wraps your existing process launch with OpenTelemetry so every LLM call streams to susaeval as a trace, with zero source changes. Python supported today. Node + Java on the roadmap.
• First-run install nudge — when OTel libs are missing in the customer's env we print the exact pip install ... line and exit. Never install behind their back.

0.4.2

• Server-side auth fan-out: a single susatest-agent login JWT now works across test, strategist, opsy, and susaeval subcommands. No more OPSY_API_KEY / SUSAEVAL_API_KEY env juggling. Legacy env vars and --api-key flags still honored for CI scripts.

0.4.1

• Updated README with full command surface (no functional changes vs 0.4.0).

0.4.0

• New login, logout, dashboard commands with OAuth device-code browser flow
• New strategist / opsy / susaeval subcommands
• Saved credentials at ~/.susatest/credentials — test and strategist no longer require --api-key
• Backward-compat: --api-key flag and SUSA_API_KEY env var still honored

0.3.x

• connect / prepare-device / fleet-prep / connect-windows device bridging.

License

MIT.