aac-trust-anchor-publisher

The tenant-side trust-anchor publisher daemon (Eng Spec §XVI component map). It runs inside the tenant boundary, one instance per tenant (single-writer; replicas: 1 per §X — HA via leader election is a V2 concern), and keeps the AAC control plane's copy of the tenant's trust anchors current:

1. Read the tenant's root signing public keys from a RootKeyPublisherSource (filesystem in Stage 2). B201's Ed25519 compatibility path is provider-neutral: the tenant platform projects the private signing key to the minting sidecar, while this publisher sees only the separately exported public PEM. Future non-exportable KMS Sign adapters are downstream work, not a B201 secret-fetch client.
2. Package them as a canonical RFC 7517 JWK Set. B202 always emits canonical alg: EdDSA for OKP/Ed25519 or ES256 for EC/P-256.
3. Sign an AAC-Tenant-Admin JWS (compact, typ aac-tenant-admin+jwt, EdDSA) with the tenant-admin private key.
4. POST /v1/root-keys/ingest on the control plane — at startup and whenever the key set changes.

Since Week 11 the SAME daemon also carries the spiffe-bundle role (the §IX.5 TrustBundleSource role — there is no separate SPIFFE publisher): it reads the tenant's SPIFFE CA anchors from a SpiffeBundlePublisherSource (filesystem in Stage 2; SPIRE/KMS adapters are backlogged pilot work), packages them as the unified trust-anchor bundle JSON (Workload Identity Bootstrap §7), signs the same AAC-Tenant-Admin JWS with artifact_class: spiffe_bundle, and POST /v1/spiffe-bundle/ingests at startup + on change.

The two roles are PEERS (B105): each activates via its own both-or-neither env-var pair, and a tenant configures only the roles it plays — root-keys for a tenant that MINTS chain roots, spiffe-bundle for a tenant whose workloads PRESENT. A forwarding-only tenant (e.g. JPMorgan in Wedge A) runs a spiffe-bundle-only publisher. Configuring zero roles is a boot error.

Verifiers never talk to the publisher: sidecars fetch the published artifacts from the control plane's public GET /.well-known/aac-root-keys/{tenant_id} and GET /.well-known/spiffe-bundle/{trust_domain}.

Terminology: JWSs are signed/created — "mint" is reserved for macaroon/token operations in AAC vocabulary.

What this daemon holds (and doesn't)

• It holds the tenant-admin private key — the credential that authenticates publishes to the control plane. That is the ONLY private key it touches.
• It reads public root-key material only. Root signing private keys never pass through it (and with KMS sources, even the admin JWS should eventually be signed by the KMS's sign API — see Backlog).
• The control plane never sees any tenant private key, ever.

In the provider-neutral Ed25519 compatibility profile, the tenant platform mounts version-pinned PKCS#8 private keys read-only into the owning workload. The root-signing private key is mounted only into the sidecar; this publisher receives public root-key files plus its distinct tenant-admin private key. On environment retirement, revoke the public root key, stop the publisher, and remove its tenant-admin secret projection plus other mounted private material under the tenant's retention procedure. The control plane retains tenant-admin key history; there is no terminal tenant-admin revocation operation. No AAC- hosted service receives those private keys.

Configuration (env vars — control-plane idiom, §XVI posture (f))

Always required:

Root-keys role (both-or-neither; setting exactly one is a boot error):

Spiffe-bundle role (both-or-neither; setting exactly one is a boot error):

For separate API and trust hosts, configure the SPIFFE role with both destinations explicitly (in addition to tenant id, admin key and bundle directory):

AAC_TAP_SPIFFE_TRUST_DOMAIN=tenant.example
AAC_TAP_SPIFFE_BUNDLE_INGEST_URL=https://api.example/v1/spiffe-bundle/ingest
AAC_TAP_SPIFFE_BUNDLE_READ_URL=https://trust.example/.well-known/spiffe-bundle/tenant.example

Use the public serving URL for exactly AAC_TAP_SPIFFE_TRUST_DOMAIN. The publisher GETs binding metadata there before publication and signs/POSTs only to the ingest URL. It sends no ingest JWS with the public GET. HTTPS remains certificate-verified; HTTP is available for local test deployments. If the response's binding metadata names a different trust domain, publication is refused with a configuration diagnostic before any ingest POST. An ACTIVE binding with HTTP 503 is the supported first-publication state when no bundle has been uploaded yet. A revoked, absent or unavailable binding prevents publication; the publisher never falls back to a different read host.

Unset or empty AAC_TAP_SPIFFE_BUNDLE_READ_URL preserves the same-host behavior of 0.2.1: strip /v1/spiffe-bundle/ingest from the ingest URL and append /.well-known/spiffe-bundle/{trust_domain}; for other ingest paths, use the ingest origin. Version 0.2.1 cannot target a separate read host; upgrade to 0.2.2 and set this variable for that topology. Root-key-only publishers need no read-URL setting.

At least one role must be configured (zero roles = boot error, exit 2). The env-var shape is AAC_TAP_<ROLE>_<THING>, with the role prefix matching the wire artifact_class (root_keys / spiffe_bundle).

Optional: AAC_TAP_POLL_INTERVAL_SECONDS (60), AAC_TAP_REQUEST_TIMEOUT_SECONDS (10), AAC_TAP_LOG_LEVEL (info), AAC_TAP_BACKOFF_CAP_SECONDS (900).

Retry cadence (B129): consecutive failed ingest attempts back off exponentially with full jitter above the poll interval, up to AAC_TAP_BACKOFF_CAP_SECONDS (default 900, or the poll interval when that is larger); the counter resets on the next accepted publish, unchanged skip, or local no-request tick (e.g. "no publishable keys" — nothing went out, so nothing backs off). The poll interval has a production floor of 30 seconds — lower values are a boot error unless AAC_TAP_DEV_ALLOW_FAST_POLL=1 is set (dev/CI harnesses only; the compose stacks run 3s).

The admin public key must be registered with the control plane first (tenant_admin_pubkey_pem on POST /v1/tenants, rotate with aac tenant rotate-admin-key --tenant-admin-pubkey-file <path>, which calls PUT /v1/tenants/{tenant_id}/admin-key); the daemon derives its own kid (tadm_ + SPKI fingerprint) offline — no registration round-trip.

Rotating it has no grace window and no hot reload: the new key becomes ACTIVE the instant the call returns, the previous one stops verifying immediately, and this daemon reads its key once at startup. Routine rotation differs by deployment path because the control-plane update has no grace window:

• pip/systemd: stage the new private file, stop the publisher, run aac tenant rotate-admin-key with --tenant-id tnt-<uuid> and --tenant-admin-pubkey-file <new-public.pem>, repoint AAC_TAP_ADMIN_KEY_FILE, then start the service.
• Docker: stop the publisher, run the same CLI rotation, overwrite the existing host bind-source file with the new private key, then remove and recreate the container with its original arguments. docker start or merely repointing a host environment file can retain the old mount/configuration and will fail after the public key changes.

If the key is compromised, rotate its registered public half first to kill the exposed key, then stop, replace, and restart or recreate as fast as practical. Do not read the root-key rotation section below as describing this key — the 24h DEPRECATED window there belongs to root signing keys only.

Key directory convention

<key_id>.pub.pem, one file per root signing public key (Ed25519 or EC/P-256 under the closed B202 floor). The whole stem is the key_id — dots allowed. This differs deliberately from the VERIFIER-side sidecar convention <tenant_id>.<key_id>.pub.pem: the publisher's directory holds one tenant's own keys, so filenames don't carry a tenant id (config does).

Rotation workflow (ROOT SIGNING keys — the tenant-admin key rotates differently and without a grace window, see above): drop the new <key_id>.pub.pem alongside the old one (transition set — §2.1 rule 4 requires retaining at least one previously-active key), wait a poll interval, then delete the old file; the control plane serves the removed key as DEPRECATED for the 24h grace window.

Bundle directory convention (spiffe-bundle role)

<anchor_id>.ca.pem, one file per SPIFFE CA certificate (must parse, carry BasicConstraints CA=TRUE, and be unexpired — invalid files are skipped with a warning). The whole stem is the anchor_id. The filesystem source reports source: "filesystem" and derives spiffe_id_pattern as spiffe://<trust_domain>/* — per-anchor patterns arrive with the real SPIRE/KMS adapters. CA rotation is whole-document: drop the new CA alongside the old (one bundle carries both), then delete the old file once downstream SVIDs have rolled — no server-side grace window (the bundle replaces atomically; overlap is expressed IN the document).

Installing (two channels, same daemon — B115)

The daemon releases in dual mode; pick the channel that matches your deployment:

• Container (Kubernetes / Docker Compose tenants):

  docker pull ghcr.io/cascadeauth/aac-trust-anchor-publisher:0.2.2
  

• pip (non-container hosts — the daemon then runs under systemd, see below):

  pip install 'aac-trust-anchor-publisher==0.2.2'
  

  Installing the wheel also generates the aac-trust-anchor-publisher console command (equivalent to python -m trust_anchor_publisher).

Repo developers use neither: uv run python -m trust_anchor_publisher from the workspace.

Running

python -m trust_anchor_publisher        # repo / module form
aac-trust-anchor-publisher              # pip-installed console command

Either form runs the same long-lived foreground process: forced publish at startup, then poll-and-republish-on-change forever. It does NOT run-once-and-exit — in a terminal it sits until Ctrl-C; in production a supervisor (systemd or a container runtime) owns its lifecycle.

Container: bind-mount the key directory and admin key read-only and set the env vars. The daemon needs outbound HTTPS to the control plane and no inbound ports.

Boot posture: local misconfiguration (bad env, unreadable/non-Ed25519 admin key) exits 2 immediately; an unreachable or rejecting control plane is survivable — the poll loop is the retry mechanism.

Deploying as a systemd service (non-container hosts)

The publisher is a "new-style daemon": it deliberately stays a plain foreground process (no self-forking, no pidfile, stdout/stderr logging) and lets systemd supply the daemonization — start at boot, restart on failure, journal capture. The complete, commented unit SHIPS IN THE WHEEL (B124 — the repo is private; the wheel is the one channel every operator has) and prints via --print-systemd-unit; no source repository or relative documentation link is required. The short version:

# one-time setup (as root)
python -m venv /opt/aac/venv
/opt/aac/venv/bin/pip install 'aac-trust-anchor-publisher==0.2.2'
useradd --system --home /var/lib/aac aac
# write /etc/aac/trust-anchor-publisher.env with the AAC_TAP_* vars (chmod 600)
# the DAEMON user (not systemd) opens the admin key — make it readable:
chown root:aac /etc/aac/keys/tenant-admin.pem
chmod 640 /etc/aac/keys/tenant-admin.pem
/opt/aac/venv/bin/aac-trust-anchor-publisher --print-systemd-unit > /etc/systemd/system/aac-trust-anchor-publisher.service
systemctl daemon-reload
systemctl enable --now aac-trust-anchor-publisher
journalctl -u aac-trust-anchor-publisher -f

Unit-file details worth keeping: Restart=on-failure with RestartPreventExitStatus=2 (exit 2 = local misconfiguration — a restart loop cannot fix a missing env var or an unreadable admin key; fail visibly instead of flapping), EnvironmentFile= for the AAC_TAP_* config, a dedicated non-root User=aac, and read-only sandboxing (ProtectSystem=strict — the daemon never writes the filesystem).

Sequence strategy (recorded trade-off)

Sequences are timestamp-derived (max(unix_now, last_sent + 1)) — stateless across restarts, no control-plane read surface (Week 10 PR-B design Q1, option c). Known limits, accepted for Stage 2 and backlogged (medium-high, pre-pilot): a clock stepped backward past the last accepted value stalls publishing with ERR_INGEST_SEQUENCE_STALE until wall time catches up; concurrent publisher instances for one tenant have no coordination (single-writer deployment is the §X contract). The upgrade path — an authoritative last-sequence read + increment — needs no wire-contract change.

Parity copy notice

ingest_jws.py deliberately duplicates the signing half of control_plane/ingest_auth.py: the two deployables never import each other (§XVI). tests/test_ingest_jws_parity.py enforces byte-identical output; change one side and the test names the drift.