CertOps CLI

The CI/CD gatekeeper for CertOps — an AI quality-assurance platform that black-box tests any AI agent over HTTP and decides whether it ships.

certops run executes a certification suite against your deployed (or local) agent, blocks until there's a verdict, and exits non-zero if the agent fails its quality gates. Drop it in a pipeline and a regression can't reach production.

Trust, But Verify.

Install

pip install certops-cli

Requires Python 3.11+.

Quickstart

# 1. Authenticate (username is tenant\email)
certops login --username 'acme\alice@acme.com'

# 2. Upload a golden dataset → prints a dataset ID for your manifest
certops upload data/golden_set.csv

# 3. Certify a deployed agent
certops run -f certops.yaml --host chatbot=https://staging.acme.com

# 4. Once it passes, make it the baseline future runs are compared against
certops tag <run_id> prod

Exit codes

The whole point of the CLI. certops run and certops status exit:

Code	Meaning	What CI should do
0	Certified — every blocking gate passed	Promote
1	Rejected — a blocking gate failed	Fail the build; this is a real quality regression
2	System error — no verdict was reached (timeout, API down, auth failure, Ctrl+C)	Fail the build, but don't blame the model

The 1/2 split is deliberate: a network blip must never be reported as a quality failure.

Commands

Command	Purpose
certops login / logout / whoami	Session management
certops run -f certops.yaml	Trigger a suite and block for the verdict
certops status <run_id>	Re-attach to a run already in flight
certops runs	List recent runs
certops tag <run_id> <tag>	Tag a run (e.g. as the prod baseline)
certops certificate generate <run_number>	Issue a Certificate of Conformity
certops certificate show <run_id>	Fetch an existing certificate
certops certificate verify <cert_id>	Publicly verify a certificate (no login)
certops upload <file>	Upload a CSV/JSON dataset

Run certops <command> --help for full flags.

certops run

certops run \
  -f ./certops.yaml \
  --host retriever=https://pr-45-retriever.acme.com \
  --host generator=https://pr-45-generator.acme.com \
  --tag staging

Flag	Notes
-f, --manifest	Path to certops.yaml (or JSON). Required.
--host	target_id=url. Repeatable. A bare url applies to every target.
--tag	Tag the run at trigger time (avoids a second certops tag call).
--notes	Free-text note attached to the run.
-d, --dataset	Override the dataset ID for all targets.
--timeout	Max seconds to block. Default 1800. Exceeding it exits 2.
--no-wait	Fire-and-forget; prints the run ID and exits.

Configuration

Setting	Flag	Env var
API base URL	--api-url	CERTOPS_API_URL

Precedence: flag → env → the URL stored at login → https://api.certops.ai. Credentials and relay config are cached in ~/.certops/config.json.

Testing a local agent (Hybrid Bridge)

You don't have to deploy to certify. Point a target at local:PORT and the CLI opens an ephemeral tunnel so the SaaS can reach your laptop:

certops run -f certops.yaml --host chatbot=local:8080

# with a path prefix
certops run -f certops.yaml --host chatbot=local:8080/agent2

The CLI runs one frpc process multiplexing every local target over a single TLS connection to the relay, at https://{prefix}-{target_id}.{relay_host}. The subdomain prefix and relay host both come from the server at login. Tunnels are bound to the lifetime of the command and torn down on exit.

Because an evaluation can fire 1,000+ requests at your agent, the tunnel enables tcpMux; without it the connection overhead would dominate.

Ctrl+C: tunnels close and the CLI exits 2, but the run continues server-side — the API has no cancellation endpoint yet. Use certops status <run_id> to re-attach.

The manifest

certops.yaml defines endpoints, never hosts — that's what makes one manifest certify dev, staging and prod. Hosts are injected at runtime via --host.

version: "1.0"

suite:
  name: "RAG Pipeline Certification"
  owner: "search-team"

targets:
  - id: "generator"
    name: "Answer Generator"
    endpoint: "/v1/chat"          # relative path only
    method: "POST"
    headers:
      Authorization: "Bearer ${env.GENERATOR_KEY}"

    request:
      format: "json"              # json (default) | multipart | urlencoded | raw
      body: |
        { "messages": [{"role": "user", "content": "{{ user_query }}"}] }

    response_path: "choices[0].message.content"

    dataset:
      id: "ds_generator_golden_v5"

    # Optional resilience — retry a flaky endpoint before calling the sample failed
    retry_count: 2
    retry_delay: 1.0

    # Pointwise evaluation
    evaluation:
      metrics_mapping:
        input: "user_query"
        reference: "golden_answer"
      deterministic:
        - metric: "cosine-similarity"
          threshold: 0.85
          operator: "gte"          # gte | gt | lte | lt | eq
          blocking: true
      llm:
        - metric: "hallucination"
          threshold: 0.1
          operator: "lte"
          blocking: true

configuration:
  judge_model_config_id: "model-config-uuid"
  concurrency: 5
  stop_on_failure: true

A target is Rejected if any blocking gate fails. The suite is Certified only if every target is.

Comparison: two independent axes

A target may declare regression, pairwise, both, or neither. They are sibling blocks — there is no comparison umbrella and no pairing selector.

regression — directional, against the latest run carrying a given tag, matched by sample index. Answers "did we get worse?"

    regression:
      baseline: "prod"
      deterministic:
        - metric: "cosine-similarity"
          max_drift: 0.05           # average can't drop more than 5%
      metrics_mapping:
        input: "user_query"
      llm:
        - metric: "general-quality"
          max_loss_rate: 0.3        # or: min_win_rate

If no run carries the baseline tag yet, comparison is skipped with a non-blocking notice and the run still passes on evaluation alone.

pairwise — symmetric, within a single run, comparing counterfactual variants against each other. Answers "are we consistent across groups?" (fairness).

    pairwise:
      mode: "group"                 # group | contrastive
      group_by: "group_id"
      role: "role"
      llm:
        - metric: "bias"
          min_equivalence_rate: 0.9  # or: max_divergence_rate, max_mean_divergence

Gate keys are axis-specific — a max_loss_rate inside a pairwise block is a manifest error and will be rejected, not ignored.

Migrating? The old unified comparison: block with pairing.mode is retired. The API rejects it with a migration hint rather than silently dropping your gates. Split it into regression (takes baseline) and/or pairwise (takes mode).

Chaining targets

depends_on maps a variable to an upstream target's response, so you can certify a pipeline stage-by-stage. Upstream targets must be declared first.

  - id: "generator"
    depends_on:
      context: "retriever.data.documents[0].content"

Built-in metrics

invocation-success is injected into every run automatically — it's the fraction of samples whose HTTP call succeeded. You don't declare it, but you can gate on it:

      deterministic:
        - metric: "invocation-success"
          threshold: 1.0
          operator: "gte"
          blocking: true

CI example

# .github/workflows/certify.yml
- name: Certify agent
  env:
    CERTOPS_API_URL: https://api.certops.ai
  run: |
    pip install certops-cli
    certops login --username "${{ secrets.CERTOPS_USER }}" \
                  --password "${{ secrets.CERTOPS_PASSWORD }}"
    certops run -f certops.yaml \
      --host chatbot=${{ steps.deploy.outputs.url }} \
      --tag staging

The CLI detects GitHub Actions / generic CI from the environment and attributes the run's trigger source accordingly.

Development

uv sync
uv run pytest
uv run certops --help

Point at a local backend with certops --api-url http://localhost:8000 ... or CERTOPS_API_URL=http://localhost:8000.

tests/fixtures.py holds the API contract — payload shapes transcribed from the backend's source and tests. If the backend contract changes, update that file first; the tests follow from it. test_gate_types_are_exhaustive fails loudly if the backend grows a gate type the renderer doesn't handle.