Skip to main content

szl-crosscheck

SZL Holdings — Orbit v2

Control before action. Evidence after.

Product · Proof · GitHub organization

PyPI Python OpenSSF Scorecard

Two independent implementations of the same measurement disagree quietly more often than teams admit. This package ends that: feed it the receipt chains from both harnesses and get one verdict — CONSISTENT, DIVERGENT (with the metric and max relative delta named), INCOMPARABLE (no shared MEASURED lanes), or INVALID (a chain failed verification — fail closed).

Built to settle the duplicated bench layer in this org: the stdlib repos (szl-retrieval-bench, szl-engine-bench, szl-quant-bench) and the FastAPI planes (retrieval-bench, frontier-bench, quant-curve). Their receipt formats differ: the retrieval HTTP runner returns an unchained RunReceipt, while the stdlib runner uses a self_hash chain. Explicit adapters preserve these native records inside an adapter-owned integrity envelope. This does not claim that the HTTP service emitted a chain or attested the caller's context.

Doctrine

  • Both chains verify before anything is compared. Tampering voids the run.
  • Measured results require matching machine identity, CPU, OS, Python version, dataset/model revision, parameters, and corpus/query/qrels hashes. Source repository commits are recorded separately so independent implementations can differ. These fields are declared evidence, not hardware attestation.
  • Only shared MEASURED lanes with shared metric keys are compared.
  • Divergence is always named: lane, metric, max relative delta.
  • The dual receipt commits to both terminal hashes and every lane verdict — same chains, same tolerance, same receipt, any machine.

Usage

pip install -e . pytest && python -m pytest tests/ -q
import json
from szl_crosscheck import crosscheck

a = json.loads(open("stdlib-chain.json").read())
b = json.loads(open("plane-chain.json").read())
print(crosscheck(a, b, rel_tol=0.01)["verdict"])

Scope

Python 3.11+, standard library only. Never re-measures; verifies and compares what the harnesses already receipted. Numbers from different hardware remain incomparable by design.

When either measurement is an integer, the comparator preserves its exact value through subtraction and relative normalization before rounding the reported delta to a float. Distinct counts above the float exact-integer range therefore remain distinguishable at zero tolerance, including mixed integer / float inputs. Existing floating-only calculations and recorded SciFact receipt bytes are unchanged. Tolerance decisions still use the reported floating delta; this is not an arbitrary-precision receipt format. Existing numeric admission and the small-value scale floor remain unchanged.

Frozen full-population comparisons

The default API intentionally compares shared measured lanes and metric keys. It cannot establish that omitted measurements were checked. Existing v2 report bytes, historical evidence and the default behavior remain unchanged.

For an exact eligible population frozen before obtaining the two result sets, supply the new keyword-only required_metrics argument:

report = crosscheck(chain_a, chain_b, rel_tol=0.01, required_metrics={
    "bm25": ["ndcg@10", "recall@10"],
})

This opt-in returns szl.crosscheck.v3, with an integrity-bound szl.crosscheck-population/v1 record. Both sides must contain exactly the frozen case-sensitive (lane, metric) identities as MEASURED entries. Counts alone are insufficient. Missing and unexpected identities are listed separately for each side; the full non-compared producer records remain in the input chains. An explicit empty policy is INCOMPARABLE, not vacuous success. A nonempty lane needs a nonempty unique list of metric names. Policy limits are 256 lanes, 4096 lane/metric pairs and 256 characters per trimmed name; malformed policy is INVALID. These are implementation bounds, not a scientific sufficiency rule.

population.complete states only exact measured-identity coverage. population.comparison_complete additionally requires compatible recorded contexts and comparison of every expected pair. Removing a required measurement from either side therefore cannot yield full-population CONSISTENT even when the remaining shared subset agrees. A discrepancy in a present required metric remains DIVERGENT while incomplete coverage stays explicit. An unexpected, unrequested metric is reported as a population mismatch, not used to invent a divergence among requested measurements. Signature/chain failures remain INVALID; incomparable contexts never become complete comparisons.

The normalized expectation, its SHA-256, coverage observations and verdict are all committed into the existing dual-report digest. Ordering the same metric list differently does not change policy identity. This does not prove the caller preregistered the population: policy and producer context remain caller-declared, hashes are not signatures, and equal comparisons do not prove accuracy, source independence or trusted execution. In v3, execution_authorized is always false. Do not derive the required set from the intersection being tested or pool v2 shared-only and v3 full-population claims.

The existing native BM25 adapters feed this mode without a new receipt format or publisher. Source tests exercise all 256 pairs of subsets of a four-measurement fixture against an independent test-owned set-equality oracle. That is one exhaustive synthetic test family, not 256 independent real experiments.

Native BM25 adapters

szl_crosscheck.adapters.stdlib_retrieval(chain, context) validates the native stdlib chain and requires its final run.context to equal the supplied context. Its final run.result must be a measured BM25 result at the declared cutoff. fastapi_retrieval(response, context, captured_inputs) checks the HTTP response's dataset, qrels, model/config and result hashes, then binds all three captured input hashes. When query_hash is supplied, it must also match the captured queries using the native producer's SHA-256 over UTF-8 JSON with sorted keys, default ASCII escaping and default (noncompact) separators. Null, malformed or mismatching supplied hashes are rejected. This matches the pinned retrieval-bench producer.

The normalized envelope records native_query_hash_state = MATCHED when that hash matches. Historical responses that omit the field remain supported with native_query_hash_state = UNVERIFIED_LEGACY and an explicit capture-only limitation; an absent field is different from a supplied null or invalid hash. In both cases request capture and context remain caller-declared. Matching hashes establish input integrity, not proof of server execution, semantic accuracy or signer identity. Both adapters compare only the shared nDCG@10 and Recall@10 by default; they retain the complete native records, including non-compared metrics.

Context is an object containing machine (identity, cpu, os, python), dataset_revision, model_revision, parameters (top_k, k1, b), input_hashes (full SHA-256 values for corpus, queries, qrels) and source (repository and full 40-character commit). Input hashes use UTF-8 JSON with sorted keys, compact separators, default ASCII escaping and no NaN. Do not relabel fixture inputs as a real benchmark. Source commits and input hashes support reproducibility; unsigned receipts do not prove signer identity.

Machine identity fields, dataset/model revisions, and source repository names must be nonblank strings. Comparison contexts use canonical JSON equality, excluding the independent implementation's source record: Boolean, integer, and floating parameter values remain distinct, including inside nested values. Different contexts across harnesses are INCOMPARABLE; mixed contexts within a single chain are INVALID. JSON key ordering does not affect compatibility. Native adapter context and cutoff bindings preserve the same JSON type distinctions. The stdlib adapter binds the full native context, including source; source is excluded only when comparing independent implementations.

Recorded native run

evidence/2026-09-05/crosscheck-native-scifact-20260905.json records an actual same-machine run over all 5,183 cached public SciFact documents and 300 test queries. The FastAPI side was called using an actual loopback HTTP POST and the ephemeral server was stopped afterward. With identical inputs and BM25 parameters, nDCG@10 was 0.6380 (stdlib) and 0.6643898053153046 (FastAPI). The 3.972% relative difference is DIVERGENT at the declared 1% tolerance. The native tokenizers differ; this receipt demonstrates the disagreement, not interchangeability or a performance ranking. Full normalized input chains, native records, source pins and input hashes are retained. CI recomputes both the complete bundle hash and dual report; it does not rerun the corpus benchmark.

Independent Lean-build comparison

schemas/lean-build-comparison-v1.json and szl_crosscheck.validate_lean_build_receipt define a second, deliberately narrow comparison plane for large Lean builds. The target is bound to an exact artifact digest, Lean version, entrypoint, theorem name, reference-statement digest and declared axiom surface. Each harness must record its own implementation digest, runner identity, exact source revision, rebuild and kernel outcomes, statement observation, axiom observation, final-chain sorry count, dependency cone and log digest.

The semantic validator recomputes every comparison field and the final verdict:

  • CONSISTENT means all required bounded observations agree across independently identified harness implementations and runners;
  • DIVERGENT names a complete but disagreeing statement, axiom, dependency or kernel observation;
  • INCOMPARABLE is mandatory when independence, target identity, execution, statement fidelity, dependency evidence or final-chain evidence is incomplete.

A caller cannot write CONSISTENT over disagreeing or incomplete evidence: the receipt fails validation unless its comparison, verdict and ordered reasons are exactly derivable from the observations. This contract never emits BREAK or PROOF_VALID. Comparison consistency is not proof validity, theorem correctness, security, or production readiness; the Lean kernel's own check remains the relevant formal check.

from szl_crosscheck import validate_lean_build_receipt

validated = validate_lean_build_receipt(receipt)
print(validated["verdict"], validated["verdict_reasons"])

License

Apache-2.0 — see LICENSE (same text as the org's other bench repos).

Estate on Hugging Face: models, datasets and Spaces are published under the SZLHOLDINGS org.

Metadata

Release files for szl-crosscheck 0.3.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for szl-crosscheck 0.3.2
File Size Uploaded Provenance Actions
szl_crosscheck-0.3.2.tar.gz 41.9 kB With provenance Details for szl_crosscheck-0.3.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for szl-crosscheck 0.3.2
File Interpreter ABI Platform Actions
szl_crosscheck-0.3.2-py3-none-any.whl Python 3 none any Details for szl_crosscheck-0.3.2-py3-none-any.whl

Total release size: 67.7 kB

Release files / szl_crosscheck-0.3.2.tar.gz

Download URL szl_crosscheck-0.3.2.tar.gz
Size 41.9 kB
Tags Source
SHA-256 checksum
How to use checksums
ce8f12b4a2702767842cde7f0af3d1d114a3176af3967eee5f4b04b40069849e
BLAKE2b-256 checksum
How to use checksums
29d2da6047c9e1be3dd8a19d2afd1ce3502fdc66c503306688864f212984a27b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Release files / szl_crosscheck-0.3.2-py3-none-any.whl

Download URL szl_crosscheck-0.3.2-py3-none-any.whl
Size 25.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
576e3f3e2bcb8b13b8872b9506de45af2b2407382d31dce67fd86280f0baed01
BLAKE2b-256 checksum
How to use checksums
094c00c693692006394e6301d5e317382922d0f8bfa80d27a80e43741c409022
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Security for szl-crosscheck 0.3.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for szl_crosscheck-0.3.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page