aipsy-bench
Open-source psychological-safety benchmark for conversational AI
Point a chatbot at a frozen battery of clinical scenarios, score the transcripts with a frozen panel of LLM judges, and ship a CI pass/fail gate + a clinician-grade diagnostic — which turns failed, why, and what to tune.
Quickstart · Benchmark your app · Local judge · Run profiles · What a score means
Built and maintained by Keido Labs
It is a benchmark (fixed content, comparable scores), not a framework. The content — 20 scenarios + a 6-metric rubric + the judge prompt — is verbatim from our source research study (documented in the forthcoming methodology preprint). The engine is Inspect AI; we ship a Task, a Solver, and a Scorer, not a new runner.
Prerequisites
- uv — gives you
uvx(run with zero install). (Preferpip? You only need Python 3.12+.) - One way to score transcripts — pick either:
- 🖥️ Ollama — runs the default local judge, 100% offline, no API key (needs a 48 GB+ Mac or a 16 GB-VRAM + 64 GB-RAM Linux box).
- 🔑 A provider API key — OpenAI, Anthropic, or Google — for the frontier judge
lane (
--judges gold/single; your key, your cost).
Quickstart
See what it catches — 10 seconds, nothing to install, fully offline (no API keys, no network, no setup):
uvx aipsy-bench demo
Replays two real recorded runs of the same coaching app on two different backing models — an
older one and a newer one, scored by the same judge — and renders the head-to-head. On a crisis
scenario, swapping the model under the app flips the safety gate from FAIL (AI-Trust 2.37) to
PASS (4.02). Same product, swap the LLM, safety moves — the kind of regression aipsy-bench
catches in CI. (Directional single-judge replay, so it's illustrative, not a comparable score.)
Then run the engine yourself (offline self-test: mock bot + mock judges):
uvx aipsy-bench run --target mock --quick
Install once — then it's just aipsy-bench … (no per-command prefixes):
uv tool install aipsy-bench # one install — local judge + every provider, no extras to pick
aipsy-bench judge pull # one-time: fetch the local judge (needs Ollama)
aipsy-bench init --http-target http://localhost:3000/eval # scaffold config for your bot
aipsy-bench run --quick # go
Frontier judges instead of the local default? Nothing extra to install — set your provider key (
aipsy-bench keys set, stored in a gitignored.env) and pass--judges single(orgold). Prefer classic pip?pip install aipsy-benchis identical.
Each run writes result.json (citable), report.txt (with the remediation cards), and a share
card.svg/card.png + badge.svg (skip with --no-card).
Benchmarking your own app? The only integration you write is a thin, stateless
/evalendpoint — see the adapter guides: a hand-written cookbook, an AI-agent build prompt, or a Tier-2 callable.
Which judge? The local judge is the offline default; frontier (
--judges gold/single, provider keys) is a different instrument — a local score compares to other local runs only, never to gold. Setup + hardware: docs/local-judge.md.
Run profiles
--judges local(default) — the offline fine-tuned judge (aipsy-judge-1.0) served by Ollama. No API key, no network (needs a 48 GB+ Mac or a 16 GB-VRAM/64 GB-RAM Linux box — see docs/local-judge.md). Its own comparability lane — comparable to other local runs, never to gold. Directional by construction and human-in-the-loop (strongest on crisis/empathy/boundary; advice is the lowest-confidence axis — treat advice flags as flag-for-review). See docs/local-judge.md.--judges gold— the 3-judge frontier ensemble (the official/citable lane). Comparable, drives the frontier badge/leaderboard. Needs all three provider keys.--judges single— the primary frontier judge only. Fast inner loop; NOT comparable to the gold or local lanes.--quick— smoke subset (one scenario per domain + both crisis scenarios). Directional only; never feeds a card or the leaderboard.--scenario s06,s07— run a subset.--baseline-prompt— inject the research baseline system prompt to reproduce the published frontier baseline. By default the target keeps its own system prompt (the bot as deployed); aipsy-bench sends only the scripted user turns.--judge-override anthropic=claude-haiku-4-5— swap a pinned judge for a cheaper one while iterating (repeatable). The judge pins are frozen, so this makes the run non-comparable — not the frozen instrument, not board/card eligible, loudly warned (§8). Never use it for a number you'll cite. (Set once inaipsy-bench.yamlviajudge_overrides: {anthropic: claude-haiku-4-5}.)
Long / real runs (timeouts, interrupting, resuming)
A gold battery is ~800 calls — --dry-run first to see the estimate. Calls are bounded
so a hung or rate-limited provider can't stall forever:
--timeout <seconds>(default 120) — per-call timeout; a stuck call fails and that scenario is reported as a run failure (never a low safety score).--max-retries <n>(default 3) — bounds rate-limit backoff (which can otherwise look like a hang). A whole battery never aborts on one bad scenario — it's logged and the rest still score.--max-connections <n>— cap concurrent calls per provider. Rate-limited? Lower it (e.g.2–4) — fewer parallel calls means fewer 429s and a more complete run. (Running a model as both target and a judge doubles that provider's load, so it rate-limits first.)- Interrupt with
Ctrl+C(bounded by--timeout, so it stops promptly), then--resume <run_id>continues without re-doing completed work. Live progress showsaipsy: judge calls/aipsy: scoringcounters; if the terminal UI feels heavy, add--display plain.
The footer counters like openai 5/5 · anthropic 14/40 · google 0/20 are Inspect's
per-provider in-flight / pool-size gauges (live concurrency), not call totals — unequal
and fluctuating is normal. A rate-limited / target-doubling provider stays saturated while
an idle one reads 0/n. All turns are still judged by every panel member.
API keys (your keys, your cost)
aipsy-bench uses your provider keys, read from the environment — it never stores,
transmits, or proxies them, so the provider bills you directly. --target mock, the test
suite, and the default local judge need no provider keys (the local judge runs
offline on Ollama); the frontier panels need a key per judge.
| Panel | Keys needed |
|---|---|
--judges local (default) |
none — judge pull is token-free (public, ungated repo) |
--judges single |
OPENAI_API_KEY |
--judges gold |
OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY |
A real target may still need its own provider key even with the local judge (only the judge
is local). The local judge itself needs no key — judge pull fetches from a public, ungated repo.
Set them the friendly way (interactive, input hidden, written to a gitignored .env):
aipsy-bench keys set # pick a provider, paste the key, done
aipsy-bench keys set --provider openai
aipsy-bench keys status # which keys are present (never prints values)
…or do it by hand — a project .env (auto-loaded; gitignored) or exported shell
vars (which take precedence):
cp .env.example .env # then fill in your keys
# or: export OPENAI_API_KEY=sk-...
aipsy-bench doctor --judges gold # preflight: which keys are present, data SHA, config
keys set only writes to your local .env — aipsy-bench never stores, transmits, or
proxies your keys.
doctor reads the same .env a real run will, so it tells you exactly what's set before
you spend anything. (Enterprises can route the pinned judges through their own Azure /
Vertex / gateway via base-URL overrides — a later iteration.)
What a score means
The only variable in a comparable score is the target bot — the scenarios, rubric, judge
prompt, and judge model snapshots are all frozen and SHA-256 verified at runtime. The
six metrics (empathy, tone_consistency, crisis_handling, advice_safety,
boundary_safety, affective_complexity) roll up into a weighted AI-Trust composite over
the applicable metrics. A target error / timeout / rate-limit / refusal is a run failure,
never a low safety score.
Coverage limits (v1)
aipsy-bench v1 measures English, text, single-agent conversational safety on the frozen battery. Out of scope for v1 (named here, not silently mishandled):
- Non-English — scenarios and judges are English-frozen; a non-English bot needs a future
data/version, not v1. - Voice / multimodal — text only; a voice bot needs a user-supplied speech-to-text shim ahead of the adapter (not shipped).
- Agentic / tool-using / RAG-grounded bots — the transcript is user/assistant text only; tool calls and retrieved context are not modeled. Such bots run, but the score reflects the text exchange only.
- Guardrail / safety-filter vendors want pass/fail on detection, not quality scoring — an adjacent product, not this tool.
Data residency
The benchmark drives synthetic, public scripted scenarios through your bot — the "user" turns are ours and carry no real end-user data. What differs by lane is where your bot's replies get scored:
- Local judge (
--judges local, the default) — fully in your boundary. Scoring runs on your own machine via Ollama; the transcript is never sent to any third-party provider and nothing leaves your environment. This is the lane for sensitive transcripts — real user data via the SDK, regulated / PHI content — where an API judge structurally cannot offer the same guarantee. - Any API judge (
--judges singleor--judges gold) — the transcript leaves your environment. Your bot's replies are sent to the judge provider(s) — OpenAI / Anthropic / Google, whichever that lane uses — for scoring, billed to your keys. In pure benchmark mode that's only your bot's answers to our public scripted prompts (still no real end-user data), but the target's outputs do go out.
So the local-judge default already runs the whole pipeline inside your own boundary — no enterprise routing required. (For teams that want the frontier lane in-boundary too, routing the pinned API judges through your own Azure / Vertex / gateway is a later enterprise feature.)
Development
uv run pytest # full suite, offline, no API keys
uv run ruff check . # lint
CI runs the suite offline with no provider keys — every test is deterministic against the mock target + mock judges.
Licenses
- Code: Apache-2.0 (see
LICENSE). - Data (
data/v1/— the scenarios, rubric, and judge prompt): CC BY 4.0 (seedata/v1/DATA_LICENSE) — the standard content license, and the same one the forthcoming methodology preprint will carry. - Local judge model (
keidolabs/aipsy-judge-1.0, on Hugging Face): Apache-2.0, inherited from its Gemma-4 base model's terms.
Apache-2.0 (code) · CC BY 4.0 (data) · directional until the human-validation study lands
Metadata
Release files for aipsy-bench 0.1.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| aipsy_bench-0.1.5.tar.gz | 481.3 kB | Without provenance | Details for aipsy_bench-0.1.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| aipsy_bench-0.1.5-py3-none-any.whl | Python 3 | none | any | Details for aipsy_bench-0.1.5-py3-none-any.whl |
Total release size: 733.9 kB
Release files / aipsy_bench-0.1.5.tar.gz
| Download URL | aipsy_bench-0.1.5.tar.gz |
|---|---|
| Size | 481.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
948ede32703bb1e0e37ad2988f214c18172926cb64507c88b002c53fd052acbb
|
|
BLAKE2b-256 checksum How to use checksums |
b0a0819d7e1014b6b7ffe65fa1343eafbe1387a349f9004bf2e82695359dea78
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.11.29 {"installer":{"name":"uv","version":"0.11.29","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / aipsy_bench-0.1.5-py3-none-any.whl
| Download URL | aipsy_bench-0.1.5-py3-none-any.whl |
|---|---|
| Size | 252.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9eff769d3ca08c400f69beaeb3172e6ce4e8caa391ac908ff52df0fa21d47ea7
|
|
BLAKE2b-256 checksum How to use checksums |
2139f795423a646d8a73f0da4b18df0e11ee13fbc935f0f891940d7cdc17d94c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.11.29 {"installer":{"name":"uv","version":"0.11.29","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for aipsy-bench 0.1.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.