flowharness-ci-runner

The CI runner distribution (flowharness-ci). CI-PR6a ships the read-the-world slice: detect changed AI surfaces, resolve the committed suite lockfile, and load the local gate policy. Execution + scoring (the CaseExecutor port, replay cassettes, and metrics) land in CI-PR6b.

Note: these packages currently install top-level modules named core, inspection, evaluation, and portability. Prefer uvx (isolated environments) over pip install into a shared venv until the 0.2 module-namespace rename lands.

vibe-check

flowharness-ci vibe-check [--base REF] [--suite PATH] [--gate-policy PATH]
                          [--executor replay|live] [--format table|json|github]
                          [--upload] [--baseline REF]
                          [--external-findings LIST] [--external-tool file=tool]

• No AI-surface change -> prints no AI-surface changes and exits 0 (the fast path: most PRs never start the job).
• A surface changed -> resolves .flowharness/suite.lock.json + the gate policy, then — because execution is not yet wired (CI-PR6b) — emits executor pending (CI-PR6b) and exits 1 (NEEDS_HUMAN). It is never a silent PASS.
• --executor live is accepted but refused with a clear message (CI-PR9). --upload is now wired (CI-PR8) — see Tier 1 ingestion below.
• --external-findings a.sarif,b.json merges third-party scanner findings into the verdict through the [external] gate (and --external-tool a.sarif=snyk_agent_scan pins a tool per file) — see ingest-findings below (PR-P1.4).

Files it reads (fixtures-are-the-contract)

• .flowharness/suite.lock.json — schema flowharness/ci-suite/v1. Unknown/missing schema, oversize input, or malformed JSON fail closed with a typed error.
• .flowharness/gate-policy.toml — optional. Absent -> the safe default -> NEEDS_HUMAN (never auto-PASS an unconfigured repo). Commit one to opt into automation.

Tier 1 ingestion (--upload, CI-PR8)

--upload POSTs the signed governance run to a FlowHarness instance. It is telemetry, never judgment: the exit code is ALWAYS the local verdict (PASS 0 / NEEDS_HUMAN 1 / FAIL 2), and no upload outcome — success, failure, or misconfiguration — ever changes it. Every outcome is one notice: line on stderr; stdout stays the byte-frozen artifact (so --format json output is byte-identical with and without --upload).

Configuration (env only — nothing secret is ever a CLI flag or on argv):

• FLOWHARNESS_TOKEN — the raw HMAC key. Read from the environment ONLY (there is no --token flag by construction); it is never logged, never in a repr, never in an error, on any path. Absent or empty ⇒ Tier 0: the run succeeds locally and the upload is skipped with one notice — never an error.
• FLOWHARNESS_API_URL — the instance base URL. An allowlist: only https://, plus http:// when the host is exactly localhost / 127.0.0.1 / ::1. Every other scheme (ftp:, file:, data:, ws:, schemeless) is refused before any request is built.
• GITHUB_RUN_ID / GITHUB_RUN_ATTEMPT — supply the idempotency nonce (runner_nonce = {run_id}-{attempt}). Absent (a local run) ⇒ skipped with a notice.

Tier ladder (D-CI8-7). No --upload flag → nothing uploads. --upload with no token → Tier 0 skip. --upload with a token → Tier 1: exactly one signed POST to {FLOWHARNESS_API_URL}/v1/evaluation/runs, then one of:

A rejected landing is the CORRECT, tamper-evident outcome — not a client bug. The server verifies an HMAC over exactly the bytes it re-derives from the report; if the signature does not match, it stores the run rejected (HTTP 200) as an audit trail rather than erroring. The runner reports this faithfully and does NOT retry-to-force it — retrying a rejected run would defeat the tamper-evidence. If you see rejected, the token or the report bytes are the place to look, not the runner.

Transport hardening (security). The token is the signing key, so the transport dials exactly one address: it refuses all redirects (a 3xx is terminal — the bearer is never re-sent to a Location target, cross-origin included) and never honors ambient proxies (http_proxy / HTTPS_PROXY / ALL_PROXY and macOS system proxy config are ignored — the key travels only toward FLOWHARNESS_API_URL). Each attempt has a 10 s idle timeout; the CI job's own timeout is the total-wall backstop.

Supply-chain residual (stated honestly, D-CI8-9). In the composite action, scope FLOWHARNESS_TOKEN / FLOWHARNESS_API_URL to the vibe-check step's env: — never the job env (which would hand the HMAC key to every step, including PR-controlled setup steps). A composite action's own internal steps (astral-sh/setup-uv, the gh comment step) still inherit the invoking step's process env; that exposure is not client-fixable. Mitigations: pin the action by SHA, and note the key's blast radius is one ingest endpoint, not a repo credential.

ingest-findings

flowharness-ci ingest-findings --input findings.sarif
                               [--tool snyk_agent_scan|aguara|medusa|inkog|sarif_generic]
                               [--format table|json|github]

Converts ONE third-party scanner artifact (SARIF or the tool's native JSON) into FlowHarness findings and renders them — standalone, no gate is applied, so the exit code is 0 (converted) or 2 (the artifact was malformed / oversize / unreadable — fail-closed). Use it to inspect what a scanner would contribute before wiring it into vibe-check --external-findings.

• Tool detection. SARIF auto-detects the tool from every tool.driver.name (all runs[] are read); --tool overrides it. Native (non-SARIF) JSON has no driver name, so --tool is required.
• --format — table (the collapsed ▸ Third-party checks section), json (the byte-exact flowharness/ci-findings/v1 artifact), or github (the ::error file=…,line=…:: annotation stream).
• Hardening. A 256 KiB size cap before read, JSON-only parsing (no YAML), an iterative depth guard against nesting bombs, and redact_path over every path/message before it enters a finding.

Findings map to core.Severity through external_severity_map.v1 — the single authority in core, shared with product/06 evidence intake later: SARIF level none/note→INFO, warning→WARN, error→ERROR, with a SARIF security-severity value ≥ 9.0 lifting any finding to CRITICAL; per-tool native codes/levels for Snyk (E###→ERROR, curated-critical→CRITICAL, W###→WARN), Aguara/Medusa (low/medium/high/critical), and Inkog (structural classes→ERROR); an unknown or absent severity → WARN + a severity_unmapped marker. Every finding carries third_party provenance and the risk.external dimension, and the path never touches safety_score_bp.

The [external] gate (vibe-check --external-findings)

vibe-check --external-findings a.sarif,b.json [--external-tool a.sarif=snyk_agent_scan] merges the same artifacts into a run and gates them through an [external] table in .flowharness/gate-policy.toml:

[external]
max_critical = 0              # any external CRITICAL beyond this FAILs a configured repo
max_error = 5                 # beyond this ⇒ NEEDS_HUMAN
unmapped_severities = "needs_human"   # or "ignore"

The external verdict combines with the eval verdict by precedence — max(eval, external) under FAIL > NEEDS_HUMAN > PASS. An absent [external] section is the safe default: a CRITICAL external finding on an unconfigured repo raises NEEDS_HUMAN, never FAIL (a FAIL requires an explicitly committed policy). The external gate always runs, even on the fast path, so external findings can never be silently bypassed; and a vibe-check run without --external-findings produces a byte-identical flowharness/ci-run/v1 artifact (additive wire).

seed --source (starter cases, §4.3)

flowharness-ci seed [--source templates,history[,fixtures]] [--history-depth N]
                    [--max-cases N] [--out DIR] [--no-update-lock] [--dry-run]

--source is a selector, not a filter: --source history seeds mined cases ONLY. The default is templates,history; fixtures is opt-in and must be named explicitly, additively:

flowharness-ci seed --source templates,history,fixtures

• Discovery is a closed set (§4.3.1). --source fixtures converts only tracked files whose basename is promptfooconfig.yaml, promptfooconfig.yml, or promptfooconfig.json — anywhere in the tree. When it finds none, it says so and names those three. Nothing is fetched or executed: the config's own bytes are read, converted, and verified offline.
• It converts the deterministic four and refuses the rest loudly. equals, contains, regex and is-json map verbatim into a case's expected; every other assertion type, an unknown top-level key, a non-scalar var, a file:// reference or a value that trips the §7 secret gate produces a notice: fixtures <path>: … line on stderr and no case — never a quietly weakened contract. Patterns are evaluated by Python's re, not JavaScript's: a JS-only construct refuses at the compile guard, and shared syntax with different semantics is a disclosed residual.
• Opting in never deletes what you had. A run owns only the provenances its --source named, so --source fixtures alone preserves existing template/history cases (and a default run preserves converted fixture cases). Fixture cases also spend only the --max-cases budget the repo-derived streams left over — they can never evict a case derived from your own surfaces.

Consumer packaging — the vibe-check composite action (§2.7)

Drop this workflow into a consumer repo. The action renders the run as a sticky PR comment (create-or-update, keyed by a hidden <!-- flowharness-vibe-check --> marker, so re-pushes edit ONE comment) and propagates the verdict exit code.

# .github/workflows/vibe-check.yml
name: FlowHarness Vibe Check
on:
  pull_request:                       # the pull_request trigger ONLY — never its _target twin (S4)
    paths: ["CLAUDE.md", ".claude/**", ".cursor/**", ".cursorrules", "AGENTS.md",
            ".continue/**", ".opencode/**", "playbook/**", ".flowharness/**",
            "**/crew*.py", "**/agents.y*ml", "**/tasks.y*ml"]
permissions:
  contents: read
  pull-requests: write                # sticky comment only
jobs:
  vibe-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<40-hex-sha>   # v6 — SHA-pin third-party actions (S7)
        with: { fetch-depth: 0 }              # diffscope needs the merge base
      - uses: flowharness-ai/vibe-check-action@<40-hex-sha>   # v1 — SHA-pin
        with:
          executor: replay                    # 'live' requires a provider key (CI-PR9)
          upload: ${{ secrets.FLOWHARNESS_TOKEN != '' }}   # Tier 1 iff a token is configured (CI-PR8)
        env:                                   # secret + URL on THIS step only — never job env, never with:
          FLOWHARNESS_TOKEN: ${{ secrets.FLOWHARNESS_TOKEN }}
          FLOWHARNESS_API_URL: ${{ vars.FLOWHARNESS_API_URL }}

Tier 1 upload (CI-PR8). upload: ${{ secrets.FLOWHARNESS_TOKEN != '' }} opts a repo into signed ingestion only once a token secret is set; the action forwards --upload to the CLI. Scope FLOWHARNESS_TOKEN / FLOWHARNESS_API_URL to the env: of this uses: step — never the job env (which would hand the HMAC key to every step of the job, PR-controlled setup steps included), and never with: (which would echo the secret into the action's declared input surface). See the Tier 1 ingestion section for the tier ladder, outcomes, and the honestly-stated composite-action residual (the action's own internal steps inherit this step's env).

The paths: list mirrors the AI-surface set inspection.domain.surfaces classifies (spec §2.7's own example): the exact root files (CLAUDE.md, .cursorrules, AGENTS.md), the surface trees (.claude/, .cursor/, .continue/, .opencode/, playbook/), the governance dir, and the orchestration globs. If the classifier gains a surface, add it here too — a path missing from this list means changes to that surface never trigger the workflow (the runner itself would still classify them on any run that does start).

S7 — pin every third-party action by commit SHA, with the tag in a trailing comment. The in-repo action already pins astral-sh/setup-uv this way; do the same for actions/checkout and the published action above.

The three §4.9 inputs — external-findings, history-hint, seed-check

All three are optional and default to the behavior above, so the minimal block stays correct as written. Each maps to its CLI flag only when it departs from the CLI default, which is why a defaults-only run assembles exactly the argv it assembled before these inputs existed.

# in the same job's steps: — the illustrative form, every §4.9 input set explicitly
- uses: flowharness-ai/vibe-check-action@<40-hex-sha>   # v1 — SHA-pin
  with:
    executor: replay
    external-findings: reports/snyk.sarif,reports/semgrep.json  # merged + gated via [external]
    history-hint: "true"      # only the exact string 'false' passes --no-history
    seed-check: "warn"        # QUOTE 'off' — YAML 1.1 reads a bare off as the boolean false

• external-findings (default "") — comma-separated SARIF/JSON artifacts gated through the [external] policy section (§4.7, above). Empty is the safe default: the CLI refuses an empty list with a typed exit 2, so an unset input drops the flag entirely rather than passing "". A path containing a space survives as one argument.
• history-hint (default "true") — the tier-1 regression-ledger hint block (§4.8.5). The feature is already on in the CLI, so this input is an opt-out: only the exact string false passes --no-history. Any other value ("False", "yes", empty) leaves the hint ON — a typo must never silently drop a governance surface.
• seed-check (default "warn") — on a zero-case suite the comment carries the "run flowharness-ci seed and commit the result" hint block (§4.9); off suppresses that one block and nothing else. Both off and the YAML 1.1 boolean it degrades to (false) pass --seed-check off, so an unquoted seed-check: off still does what it reads like.

Version floor. The action's pinned distribution (flowharness-ci-runner==0.1.1) predates all three flags: --external-findings / --no-history need 0.1.2, and --seed-check needs the first release after it. Setting a non-default value before the pin moves makes the pinned CLI refuse the argument (exit 2 — loud, never a silent PASS). Moving the pin is a GA blocker tracked in docs/plan/vibe-check-ga-checklist.md.

How the action runs the CLI (D-CI7-30). astral-sh/setup-uv installs only the uv/uvx binaries — it does NOT install this package's console script — so every CLI step in the action runs uvx --from flowharness-ci-runner flowharness-ci …, the §2.7 fetch-and-run form. The --from clause is required because the distribution name (flowharness-ci-runner) differs from the script name (flowharness-ci); a bare uvx flowharness-ci would resolve the wrong package. Publication dependency: flowharness-ci-runner must be published to PyPI before the action is runnable anywhere — until then the vibe-check step fails at resolution (a non-zero capture, so never a silent PASS). At first publication, pin the package (--from flowharness-ci-runner==X.Y.Z) so a SHA-pinned action cannot float to a newer release.

gh version floor: >= 2.47. The action lists comments with plain gh api --paginate, which for a top-level array endpoint (/issues/{n}/comments) splices the pages into ONE flat array — MEASURED on gh 2.89.0 against a local three-page server: json.loads yields 5 flat items across 3 pages, and --paginate auto-sets per_page=100, so the marker comment is found even off the first page. Do NOT "fix" this onto --slurp: gh help api documents --slurp as returning an array of all pages, i.e. it NESTS the wire shape into an array OF pages, so the selector would find zero comments and every push would post a fresh spam comment.

Sticky-selector residual (accepted, documented — D-CI7-22)

The selector requires user.type == "Bot", user.login == "github-actions[bot]", the marker on the body's exact first line, and the vibe-check heading on the next non-empty line. All four are content, so this is a collision filter, not an authenticity check: github-actions[bot] is the SHARED identity of every workflow in the repo, and a sibling workflow that echoes attacker-controlled text could reproduce these lines. The realistic (accidental-echo) class is removed; a deliberate same-bot forgery is not, and no stateless rule can close it. Impact if it ever happens is bounded to defacement + suppression of the report — never a verdict change (post-comment always exits 0). Hardening option: run the action under a dedicated GitHub App or PAT-backed bot identity, at which point user.login becomes a genuine authenticity signal — trade-off: a secret to manage, and it is unavailable on fork PRs (which have no token).

Governance-file symlink residual (S6 — keep them REAL files)

S6 warns and forces a full run when the diff touches .flowharness/suite.lock.json or .flowharness/gate-policy.toml. If either governance file is committed as a symlink, git diff reports the link target path — which the derived S6 matcher does not watch — so a PR could weaken the suite or the gate threshold with no warning line and no forced run. Keep both .flowharness/suite.lock.json and .flowharness/gate-policy.toml as real files under .flowharness/; if a shared layout is required, point --suite / --gate-policy at the real location so the matcher watches it directly. (Cassette evidence is unaffected — the case-ref reader fails closed on any symlinked component.)

Boundaries

Zero network on the default path (no httpx); git and gh are the only local processes shelled, each behind an injected runner so the core is offline-testable — and Python never reads a token (gh authenticates from its own env). The runner reuses core, evaluation.domain (the GatePolicy VO), and inspection (surface classifier + exit-code table) only — statelessly, with no UnitOfWork or web/ORM stack (enforced by the import-linter fences in the root pyproject).

SDK boundary (ci-spec §5.1 / G10) — the CI runner is NOT the SDK

The CI runner (flowharness-ci, this package) and the FlowHarness Python SDK (flowharness, under sdks/python/) are deliberately separate products, and the CI runner is not the SDK:

• Different wire. The runner speaks a JSON governance-run ingestion wire — one HMAC-signed POST /v1/evaluation/runs (--upload, above), and otherwise runs fully offline. The SDK speaks the ConnectRPC machine wire to the management API (typed RPCs, binary framing) — the Phase-1 client under sdks/python/.
• Different audience. The runner is a governance gate for CI pipelines, invoked as a CLI (uvx --from flowharness-ci-runner flowharness-ci vibe-check). The SDK is for application code that embeds audit calls programmatically.

Programmatic embedding of the runner is out of scope: it exposes a CLI, not a stable client API. If you want to call the platform from code, use the SDK (sdks/python/README.md); if you want to gate a PR, use this runner.