postern-conformance

Points at a running Postern runner and reports which of the specification's three conformance levels it actually meets — and which of its MUST rules the runner breaks getting there.

$ pip install postern-conformance
$ postern-conformance http://127.0.0.1:8787

Postern conformance · http://127.0.0.1:8787
schemas: checkout

  §4.4
    PASS  status answers without credentials
    PASS  status matches its schema
  §3
    PASS  declares Level 3
          Stream — describe, status, run, stream
    PASS  run is implemented at Level 3
  §2.3
    PASS  no wildcard Access-Control-Allow-Origin by default
    PASS  run rejects a non-JSON media type
  …

Declared level: 3
34 pass  0 fail  0 warn  2 skip

Conformant at Level 3.

Postern is four HTTP endpoints, so this checker speaks to a runner over the wire and has no idea what it is written in. A Go, Rust or TypeScript runner is checked exactly as well as a Python one.

It does not run your agent unless you ask

A run may invoke tools that spend money and mutate state outside the workspace (§4.1.2), and an abort is not a rollback (§4.5). A conformance checker that ran an agent to find out whether it conforms would be charging you for the answer, and would have done so before you could read this.

So by default it checks only rules a runner applies before the agent starts — which is most of the specification:

• the level rule, in full, because a verb above the runner's level must refuse and so executes nothing
• every refusal path: a malformed body, a missing required input, a revoked entitlement, a media type the runner must not parse
• describe and status entirely, along with their schemas
• the CORS rules, none of which involve running anything

--execute opts into the rest — the run response shape, the SSE framing, the delta concatenation invariant, run_id uniqueness, and — for a runner declaring status.idempotent_retry — that a key already answered is refused rather than replayed when it arrives carrying different inputs. There is no way to check those without a real run, which is why they are a decision rather than a default.

The last of those spends nothing of its own: it presents the key the uniqueness check already bound, and a conformant runner refuses it without running the agent. It rides on --execute because the key has to have been bound by a real run first, not because it buys another.

Two refusals ride on it for the opposite reason — not because they need a run, but because a runner that breaks them performs one. A request failing a validation the agent's own describe declares, and a request sent while status reports a declared credential unset (§4.6 step 5), are both bodies a conforming runner rejects before the agent starts, so against one they cost nothing. There is no way to ask that is free against a runner answering wrongly: the run is the answer, and you have to have agreed to pay for it.

The second is askable only where the runner said so first. status.credentials is OPTIONAL (§4.4), so a runner publishing no credential state is conformant and its environment cannot be read from outside — the report says so rather than passing it quietly.

Usage

postern-conformance http://127.0.0.1:8787
postern-conformance http://127.0.0.1:8787 --origin https://app.example.com
postern-conformance http://127.0.0.1:8787 --execute
postern-conformance http://127.0.0.1:8787 --json

--origin names an origin the runner is configured to allow. Without it the CORS header rules are skipped: which origins a runner allows is the runner's own decision, and the specification fixes only the two ends of it (§2.3). The rules that hold for any origin — no wildcard default, no Origin: null — are checked either way.

A SHOULD cannot fail the run. It reports as WARN and the exit status stays 0. A checker that failed a runner for declining an option the specification left open would be ignored, and its MUSTs would be ignored with it.

What a PASS is worth

Two limits, both of which the tool states in its own output rather than leaving you to discover:

• A pass is not a proof for a rule with two right answers. The default text/plain probe sends a body the runner must reject anyway, so a runner that wrongly parses text/plain still answers 400 and still passes. --execute is what makes that one conclusive. The self-test below asserts this rather than assuming it — the fault is planted, and the default probe is confirmed not to catch it.
• describe being side-effect free cannot be observed from outside. Two identical calls returning identical bytes is evidence, so a difference warns; it is not a breach this tool can stand behind.

Rules are checked against the specification's own schemas/ rather than restated in Python, so a schema and this checker cannot drift. Running from a checkout uses that checkout's schemas; an installed wheel carries a copy. Every report says which it used.

Proving the checks can fail

The failure mode of a conformance checker is a false green: every check reads correctly, passes against a real implementation, and would have passed just as happily against a runner that did none of it.

$ python tools/conformance/selftest.py
postern-conformance self-test

  22 conformant baselines, none failing
  11 error codes, table agrees with the schema
  6 schemas, the build hook bundles each one
  2 declared formats, every one asserted
  3 stream shapes, each read to a bounded end
  39 planted faults, each caught by its own check

Every check can fail.

It runs the checker against a deliberately conformant fake runner, where nothing may fail, and then against the same runner with exactly one rule broken — asserting that the named check catches it, not merely that something did. Standard library only; it needs no runner and no network.

Why this is not called postern

CONTRIBUTING.md puts a language SDK out of scope, and the reasoning holds: Postern is HTTP, any language can serve or call it, and publishing a Python client under the specification's own name would make one language the blessed one for a document whose whole claim is that none is.

This is a test suite rather than a client library, and it deliberately leaves the name a Python client would want free for whoever writes one under their own.

Changes

0.1.2

Verdicts move in both directions. A runner that reported clean under 0.1.1 may report findings here, and one that reported a finding may come back clean. Almost all of it is checks that were narrower than the rule they named — asked on one surface where the specification binds several — so read a new finding the way 0.1.1 asked you to: as a rule that was always there and is only now being checked. 10 conformant baselines became 22 and 24 planted faults became 39.

• idempotent_retry moved from describe.capabilities to status. §4.1 opens "capabilities describes the agent", and whether a repeat executes it a second time is a fact about the runner serving it. The checker reads status and deliberately does not fall back to the old location; a runner still declaring it under capabilities validates and means nothing by it.
• §2's media type is checked on every body it binds, not on status alone. describe, run's success body and every error body went unasked — and an error body is the one most likely to get it wrong, since a failure path often leaves the serializer that would have set the header.
• The §2.3 preflight rules are asked on both preflighting verbs. A runner admitting Content-Type on run and not on stream passed, while no browser could stream from it at all.
• §4.1.3 is read as written — the whole describe response is scanned for a credential value, where the check had read the credentials block alone.
• A stream's start and done must name the same run, which stream-event.schema.json has always said and nothing checked.
• §4.6 step 5 is a check a runner performs, not only one it orders. Where describe declares a credential the environment does not carry, a conforming runner answers 424 missing_credential.
• status.agent, status.entitlement and describe.output are required, and status.agent requires its own members — {"agent": {}} used to validate while describe required id, name and version.
• A held-open stream is reported rather than hanging the report, and the bounded read still sees what a runner sends after done.
• Five checks a non-conformant runner was passing are closed, and separately the checker stopped failing runners the specification permits.
• A rebuilt wheel no longer ships the previous build's schemas. A wheel built over an earlier build's artifacts validated against a specification nobody was reading — the one entry here that is a packaging fault rather than a check.

0.1.1

A runner that reported clean under 0.1.0 may report findings here. That is the intent rather than a regression: the specification was clarified in four places and the suite followed. Read a new finding as a rule that was always there and is only now being checked.

• §4.6 is checked at all. A run missing a required input, sent to a runner whose environment is incomplete, earns both bad_request and missing_credential, and the specification did not order them — so this suite skipped §4.2's rule whenever a runner answered 424, which is the ordinary state of one being brought up. §4.6 now orders them, and the 424 is a finding.
• A reused Idempotency-Key carrying different inputs must be refused, not replayed.
• run_id is per execution, quoting §4.2's own wording.
• capabilities.streaming is withdrawn and no longer checked.

0.1.0

First release. 5 conformant baselines, 22 planted faults.

Status

The specification is a draft and nothing in it is stable yet, so neither is this. It tracks Postern 0.1.

Apache-2.0, same as the specification.