Banza Conformance Runner

Tests a running Banza operator against the protocol conformance suite and produces a structured JSON report.

The runner generates conformance evidence, not certification. A PASS is technical conformance evidence for the requested level — it is not a production certificate. Production certification remains M2/M3-gated and is issued separately by the BANZA CA.

Distribution for external operators

External operators can run BANZA conformance against their public sandbox endpoint without cloning this repository, in three ways.

Installable CLI

pip install banza-conformance        # from tools/banza-conformance
banza-conformance --url https://sandbox.operator.example --level 0 --output report.json

Supported options: --url (required), --level, --suite, --output, --quiet, --federation (L3, requires the federation extra: pip install "banza-conformance[federation]"). The CLI calls the same conformance logic as run.py — python3 tools/banza-conformance/run.py … continues to work unchanged.

On completion the tool prints:

Conformance report generated. This is evidence, not certification.

Client identity (User-Agent)

The runner sends a stable, documented User-Agent so operators can allow it in WAF / proxy / CDN configurations:

BANZA-Conformance/1.0 (+https://banza.org/conformance)

If your sandbox endpoint sits behind a bot filter (e.g. Cloudflare), allow this User-Agent (or the /health and /.well-known/banza/* paths) so the runner can reach it. The User-Agent identifies the client cleanly; it does not bypass any security control.

Docker

docker run --rm \
  -v "$PWD/reports:/reports" \
  ghcr.io/banza-protocol/banza-conformance:latest \
  --url https://sandbox.operator.example \
  --level 0 \
  --output /reports/banza-l0-report.json

Build the image locally (context = this directory):

docker build -t banza-conformance:local tools/banza-conformance

The entrypoint is banza-conformance; arguments after the image name are passed straight through. Reports are written under /reports — mount a host volume there.

GitHub Action

Run BANZA conformance in your CI/CD against your public sandbox endpoint:

name: BANZA Conformance

on:
  workflow_dispatch:
  pull_request:
  push:

jobs:
  conformance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: banza-protocol/banza/.github/actions/banza-conformance@main
        with:
          url: https://sandbox.operator.example
          level: "0"
          output: banza-l0-report.json
      - uses: actions/upload-artifact@v4
        with:
          name: banza-conformance-report
          path: banza-l0-report.json

Inputs: url (required), level (default "0"), output (default "banza-conformance-report.json"), suite, federation, quiet.

Requirements

Python 3.8+. No third-party packages for the core runner — stdlib only.

For L3 federation conformance (signature verification): pip install cryptography>=41.0.0

Usage

Standard conformance (L0–L4)

# Run all suites against a local operator
python3 tools/banza-conformance/run.py --url http://localhost:3000

# Run up to Level 2 only
python3 tools/banza-conformance/run.py --url http://localhost:3000 --level 2

# Run a single suite
python3 tools/banza-conformance/run.py --url http://localhost:3000 --suite health

# Write report to file (quiet mode)
python3 tools/banza-conformance/run.py --url http://localhost:3000 --output report.json --quiet

Or via the shell wrapper:

./tools/banza-conformance/run.sh --url http://localhost:3000 --output report.json

L3 Federation Conformance (79 tests, 9 suites)

The federation conformance suite tests cross-operator federation protocol compliance. It requires two terminals: one for the fixture server (Operator A adapter) and one for the runner.

# Terminal 1 — start the fixture server (Operator A)
python3 tools/banza-conformance/fixture_server.py --port 8099

# Terminal 2 — run the full federation suite (79 tests)
python3 tools/banza-conformance/run.py \
  --federation \
  --url http://localhost:8099 \
  --output l3-report.json

# Run only one federation sub-suite
python3 tools/banza-conformance/run.py \
  --federation \
  --url http://localhost:8099 \
  --fed-suite cert        # cert | disc | trust | route | exec | obl | evt | settle | fail

What the federation runner does:

• Generates an ephemeral test BANZA root keypair (never reused as production)
• Spins up an embedded Simulated Operator B (random port, in-process)
• Spins up an embedded test BRL/key-manifest server (random port, in-process)
• Runs all 79 federation tests against the fixture server
• Produces a structured JSON evidence report

Federation suites:

L3 certification decision: All blocking suites (FED-CERT through FED-OBL) must PASS. FAIL in a blocking suite denies certification. FAIL in a non-blocking suite produces conditional certification (30-day remediation window).

See docs/federation/FEDERATION_CONFORMANCE_MODEL.md for the full decision rules.

Options

Certification levels

Level names and per-level capabilities are canonical and match BANZA_CERTIFICATION.md § Conformance level model and ADR-034. This single-operator sandbox runner awards L0–L2; L3 (federation) and L4 (external) require multi-operator / external-rail evidence and are not awarded here.

Exit codes

Report format

Reports conform to conformance/report-schema.json. Each case records:

• status: PASS / FAIL / SKIP / ERROR
• request and response for HTTP cases
• assertions: per-field check results
• failure_reason: human-readable explanation on failure

Adding new suites

1. Implement a runner function run_<name>(base_url, ctx) -> list[CaseResult]
2. Register it in the SUITES dict
3. Add it to the relevant LEVEL_SUITES entries
4. Add the corresponding vectors in conformance/vectors/