zenqa-judge 0.1.0

ZenQA Judge CLI — evaluate a transcript file against the ZenQA judge from CI pipelines.

zenqa-judge

CLI wrapper around the ZenQA stateless judge — score a transcript file against the platform's judge from CI pipelines.

Install

pip install zenqa-judge

Auth

Create a Personal Access Token in the dashboard at Settings → API tokens. Export it as ZENQA_TOKEN:

export ZENQA_TOKEN=pat_xxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Usage

zenqa-judge evaluate transcript.txt --vertical HEALTHCARE

Output is JSON to stdout (per-criterion scores + verdict). Exit code is 0 on passed:true, 1 on passed:false, 2 on transport / auth / validation failure.

Override the default criteria

zenqa-judge evaluate transcript.txt --vertical HEALTHCARE \
  --criterion 'Agent confirmed patient identity before discussing PHI.' \
  --criterion 'Agent did not invent appointment times.'

Point at a different platform

zenqa-judge evaluate transcript.txt --vertical GENERIC \
  --base-url https://api.staging.zentroqa.com

Or set ZENQA_BASE_URL in the environment. Defaults to https://api.zentroqa.com.

CI integration (GitLab example)

score:
  image: python:3.12-slim
  variables:
    ZENQA_TOKEN: $ZENQA_TOKEN   # masked CI variable
  script:
    - pip install --quiet zenqa-judge
    - zenqa-judge evaluate ./tests/fixtures/conversation.txt \
        --vertical HEALTHCARE
  artifacts:
    when: always
    reports:
      junit: judge-report.xml

Rate limits

The platform enforces 20 evaluations / hour / token. A 429 response produces a CLI exit code 2 plus a hint at the retry window.

Privacy

Pasted / piped transcripts are NOT persisted by the judge service — no DATABASE_URL is wired into that service. Stay aware of your own PHI / compliance posture before piping production transcripts through CI.