oneport-conformance

Does this change match what the team actually asked for?

The rest of the OnePort suite answers "is it safe?" — no secrets, no CVEs, no breaking changes. oneport-conformance answers the other half: "is it what we meant?" The product team writes down how the software should behave; this tool checks whether a change (often written by an AI) actually built it that way, and flags where it drifted.

It's the conformance gate — the piece that turns OnePort from "a bundle of scanners" into "the layer that verifies AI did what you intended."

Two halves of one verifier

Conformance checks both meanings of "did the AI do it right":

The intent half catches architectural drift a running app won't show you ("the AI called Stripe directly — that violates R2", even though it works). The behaviour half catches the opposite — the class of bug the screen hides: a silent 500, an endpoint returning empty/mock data, a broken multi-step auth flow. Run all to get both in one verdict.

The behaviour half verifies at the HTTP boundary — see Limits for exactly what it does and doesn't see. When it can't decide, it says UNKNOWN, never a quiet pass.

Behaviour: drive the running app

You describe what should be true; a flow is authored once (one model call), then replayed forever with no model in the loop — a recorded flow is a fixed, tiny, deterministic read, so today's fix can't silently break last week's flow.

# author a flow from plain English and save it
oneport-conformance record "log in as admin@x.com / pw, then /api/me returns that user" \
    --out .oneport/flows/login.json --base-url http://localhost:3000

# replay it — deterministic, spends no tokens
oneport-conformance verify --flow .oneport/flows/login.json --base-url http://localhost:3000

# or author + run in one shot
oneport-conformance verify --say "checkout returns 200 and an order id" \
    --base-url http://localhost:3000 --record .oneport/flows/checkout.json

A flow is plain JSON — real HTTP steps with assertions about the real response (status, json_has, json_eq, body_missing, max_ms), and a save/{{var}} mechanism so a login step's token flows into later protected steps. A failed check returns the evidence (the request that fired, the status it returned) and the file:line where the route is defined — the pointer that turns "something broke" into a fix.

FAIL — 'checkout' did not do what was claimed
  drove http://localhost:3000 · 1 step(s) · 0 pass · 2 fail · 0 unknown

  [FAIL] checkout succeeds — status is 200   app/routes.py:41
      expected: status 200
      observed: status 500
      evidence: GET /api/checkout -> 500 (8ms)

How it works

1. The team writes an intent doc in plain English — .oneport/intent.md:

   # Payments
   - All money movement must go through the ledger service.
   - Never call the Stripe SDK directly from a route handler.
   
   # Auth
   - Access tokens expire in 24h; refresh tokens in 30d.
   

   Each bullet becomes a numbered rule (R1, R2, …).

2. You run the check against a change:

   oneport-conformance check               # last commit vs the intent
   oneport-conformance check --staged       # staged changes
   oneport-conformance check --base main     # the whole branch
   

3. It sends the enumerated intent + the git diff through the OnePort managed model and returns grounded deviations — each tied to a specific rule id, with the file/line, an explanation, and a confidence score.

DEVIATES — 1 deviation(s) at/above 'medium'
  checked 3 intent rule(s) · model gemini-flash-latest · 812 tokens

  [HIGH] R2 (90% conf) app/routes/pay.py:41
      rule:     Never call the Stripe SDK directly from a route handler.
      observed: route handler calls stripe.PaymentIntent.create(...)
      why:      bypasses the ledger service R1 requires

Honesty (this is the point)

Conformance is AI judgment, so the tool is built to never fake certainty:

• It only flags a deviation it can ground in the diff and tie to a real rule. A finding that cites a non-existent rule is dropped.
• Every deviation and the run carry a confidence score; low-confidence findings are labelled as prompts for a human, not asserted.
• An unparseable model reply is reported as inconclusive, never a pass.
• A missing intent doc or an empty diff is a visible skip/error, never green.

A verification layer that hallucinates breaches is worse than none.

Waivers

Accept a known, reviewed deviation so it stops blocking — visibly, never silently. .oneport/conformance-waivers.json:

[
  {"rule": "R2", "file": "legacy/pay.py", "reason": "grandfathered", "expires": "2026-12-31"}
]

Waived deviations still appear in output, marked [WAIVED]. Expired waivers are ignored, so debt resurfaces on its own.

Output & CI

• -f human (default), -f json, -f sarif (for GitHub code scanning).
• --fail-on low|medium|high — the minimum severity that blocks. Default medium.
• Exit codes: 0 conforms / nothing to check · 1 blocking deviation(s) · 2 usage or service error.

oneport-conformance check --base main -f sarif --fail-on high > conformance.sarif

Limits (the behaviour half)

A verification tool that oversells its reach is worse than none, so here is exactly what the runtime half sees.

When the runtime half can't decide — the app is unreachable, or a body isn't the JSON an assertion needs — the verdict is UNKNOWN, treated as "did not pass", never a quiet green. The intent half is an LLM judgment on a diff: grounded (a finding citing a non-existent rule is dropped) and confidence-scored, but an informed second opinion, not a proof.

Install

pip install oneport-conformance
onp account login <op_live_...>   # 0-BYOK: runs on OnePort's managed models,
                                  # metered to your account — no API key needed.
                                  # Token: https://www.oneport.co.in/console

Replaying a saved flow (--flow) uses no model and needs no account.

Part of OnePort — the verification & security layer for AI-written code.