Etalon

Qualification evidence for a specific self-hosted model deployment.

Etalon runs a versioned pack against an OpenAI-compatible endpoint on the customer's network and writes an evidence bundle: fingerprint, raw inputs and outputs, case results, coverage, a three-state decision, and a self-contained HTML report. Nothing in the runner phones home.

This repository is the runner (Apache-2.0): coyos-ai/etalon. The product site is coyos-ai/etalon-www. Commercial pack content stays in coyos-ai/etalon-packs. The packs in this tree are public fixtures.

Etalon does not determine that a regulation, contract, or internal control has been met. It records what was evaluated, under which configuration, and how the pack's own policy read the scores.

Documentation

The operator guide is docs/README.md: a localhost qualification, how a bundle and a decision are produced, the CLI, packs, judges, offline install, and signing.

Open the prebuilt report

No install required. Open either file in a browser:

• examples/sample-run/report.html — QUALIFIED on the local fixture
• examples/sample-run-broken/report.html — NOT_QUALIFIED, with failure excerpts and case redaction

Those checked-in reports keep the coverage sentence recorded when the bundle was generated. v0.1 realised coverage is value-presence: a declared value counts when it appears at least once. The number is not a cross-product of coverage cells. etalon inspect and etalon compare state that limit when you read the bundle, and a newly rendered report states it in the coverage section.

etalon verify examples/sample-run recomputes the bundle hashes after install.

Demo, copy-paste

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# terminal A — localhost only
python3 examples/mock_endpoint.py --port 8000

# terminal B
source .venv/bin/activate
etalon pack validate examples/demo-pack

etalon qualify \
  --endpoint http://127.0.0.1:8000/v1 \
  --pack examples/demo-pack \
  --serving-config examples/demo-pack/serving.yaml \
  --system-prompt examples/demo-pack/system_prompt.txt \
  --model etalon-demo-baseline \
  --operator "Demo Operator" \
  --output runs/

# use the printed run directory
etalon inspect runs/<run-id>
etalon verify runs/<run-id>
etalon report runs/<run-id>

# re-supply every recorded input. Exit 0 when nothing captured changed.
etalon check runs/<run-id> \
  --endpoint http://127.0.0.1:8000/v1 \
  --pack examples/demo-pack \
  --serving-config examples/demo-pack/serving.yaml \
  --system-prompt examples/demo-pack/system_prompt.txt \
  --model etalon-demo-baseline \
  --operator "Demo Operator"

# bare check does not exit 0: recorded triggers were not re-observed
etalon check runs/<run-id> --endpoint http://127.0.0.1:8000/v1

# sampling drift is review_required (exit 1) once other recorded triggers are re-supplied
etalon check runs/<run-id> \
  --endpoint http://127.0.0.1:8000/v1 \
  --pack examples/demo-pack \
  --serving-config examples/demo-pack/serving.yaml \
  --system-prompt examples/demo-pack/system_prompt.txt \
  --model etalon-demo-baseline \
  --operator "Demo Operator" \
  --temperature 0.7

# quantization drift is always_requalify (exit 2)
etalon check runs/<run-id> \
  --endpoint http://127.0.0.1:8000/v1 \
  --pack examples/demo-pack \
  --serving-config examples/scenarios/serving-int4.yaml \
  --system-prompt examples/demo-pack/system_prompt.txt \
  --model etalon-demo-baseline \
  --operator "Demo Operator"

etalon compare examples/sample-run examples/sample-run-broken

# contact-routing fixture: reference metrics plus a judge overlay
python3 examples/mock_endpoint.py --port 8000 --pack examples/contact-routing

# live judge: export TYPESAFE_API_KEY and pip install 'etalon[typesafe]'
etalon qualify \
  --endpoint http://127.0.0.1:8000/v1 \
  --pack examples/contact-routing \
  --serving-config examples/contact-routing/serving.yaml \
  --system-prompt examples/contact-routing/system_prompt.txt \
  --model etalon-contact-baseline \
  --operator "Demo Operator" \
  --output runs/

etalon inspect runs/<run-id>

The fixture models are scripted, not weights:

Model id	What it does
etalon-demo-baseline	Reference labels. Decision QUALIFIED.
etalon-demo-quantized	Scripted label misses. NOT_QUALIFIED (requirement_miss:label_accuracy).
etalon-demo-degraded-prompt	Ignores abstain and escalate. NOT_QUALIFIED (requirement_miss:abstention_correctness).
etalon-demo-small	Misroutes complaint labels. NOT_QUALIFIED (critical).
etalon-demo-sampling	Two label flips when temperature is above 0. Still QUALIFIED; label accuracy moves.
etalon-demo-broken	Non-JSON output. NOT_QUALIFIED (schema and critical).
etalon-demo-error	HTTP 500. INDETERMINATE (endpoint_errors), including when scored cases already miss.

examples/contact-routing (coyos.fs.contact-routing @ 0.3.0) is the public synthetic fixture for demos and tests. The commercial pack of the same id and version lives in coyos-ai/etalon-packs (packs/coyos.fs.contact-routing). The commercial licence applies there; this example stays under Apache-2.0. The fixture uses the same mock process with --pack examples/contact-routing:

Model id	What it does
etalon-contact-baseline	Referee labels on the system under test. Reference metrics pass. The run is QUALIFIED only when the judge returns usable verdicts inside the uncertainty maximum and calibration agreement holds.
etalon-contact-security-leak	Security tickets go to billing. NOT_QUALIFIED (critical misroute) when the capture completes.
etalon-contact-abstain-miss	Abstain becomes classify. NOT_QUALIFIED (abstention) when the capture completes.
etalon-contact-soft	Ambiguous tickets take the next public label. NOT_QUALIFIED when the capture completes.

The pack judge is TypeSafe Jev (transport: typesafe), not a fixture chat model. Set TYPESAFE_API_KEY and install etalon[typesafe] for a live judge. Without the key the judge fails and the run is INDETERMINATE. CI mocks the SDK. offline-mock remains for taxonomy tests. The mock endpoint still has etalon-judge-uncertain and etalon-judge-fault for an endpoint judge transport; this pack does not select that transport. See docs/typesafe-judge.md.

python3 examples/run_degradation.py prints the demo profile table and exits non-zero if a row's status or reason class drifts. pytest runs the same gate. The harness does not measure a real quantized model. The int4 serving file records a quantization name on the fingerprint; the quantized fixture's decision change is the scripted label-accuracy miss. If these profiles stop separating, the pack thresholds are what to change.

Commands

etalon pack validate <pack>
etalon qualify --endpoint URL --pack DIR --operator NAME --output DIR
etalon report <run>
etalon report <run> --output report-copy.html
etalon report <run> --show-sensitive --output report-sensitive.html
etalon inspect <run>
etalon verify <run>
etalon verify <run> --public-key etalon-public.json --require-signature
etalon keygen --private etalon-private.json --public etalon-public.json
etalon sign <run> --key etalon-private.json
etalon export <run> --output auditor/
etalon check <run> --endpoint URL
etalon compare <run-a> <run-b>

verify answers whether the bundle is intact (exit 1 otherwise). The expected evidence file set is authoritative. Verification fails when an expected file is missing, an expected file is absent from hashes.json, an extra file is in the run directory, a digest is malformed or uses another algorithm, the manifest decision does not match decision.json, the fingerprint is ambiguous, or file bytes do not match. signature.json is optional and is not hashed. Unsigned bundles still verify. A signature that does not match the bytes of hashes.json fails verification. Pass --public-key to require the signature from that key. See docs/signing.md.

report with no --output prints the existing report.html path and does not rewrite report.html, report.json, or hashes.json. Showing sensitive case content requires --output, which writes a derivative and leaves the hashed bundle unchanged.

check answers whether the recorded fingerprint is still current:

• exit 0 — every recorded trigger value was re-supplied and matches (no_material_impact may still be listed; a stored unavailable gap is listed at its trigger severity and does not by itself change the exit code)
• exit 1 — a review_required trigger fired or its recorded value was not re-supplied, and nothing forces 2
• exit 2 — an always_requalify trigger fired or its recorded value was not re-supplied

etalon check <run> --endpoint URL alone does not exit 0. Human output for every fired or not-reobserved trigger prints the severity, the trigger name, the stored value, and the live value (not_observed when it was not re-supplied). Fields that were not supplied at qualification are unavailable with a reason. check lists those at their trigger severity rather than inventing a hash. Weight files are never hashed as one blob; pass --model-index with per-file hashes if you have them.

etalon check separates three kinds of row. Fired means the re-supplied value differs. Not re-observed means a captured value was not re-supplied; that changes the exit by the trigger's severity. Unverified gaps are stored unavailable fields, printed at their severity, and they do not by themselves change the exit. An ambiguous fingerprint (a mapped field is missing, or an envelope contradicts itself) is printed on its own. It is not an unavailable gap, and the check cannot exit 0.

etalon compare prints a table of decision status, metric point estimates with confidence intervals, critical-event counts, and realised coverage. The coverage line is value-presence, not a cross-product of coverage cells. It then lists the fingerprint fields that differ and the trigger severity of each (always_requalify, review_required, or no_material_impact). It also says whether qualification evidence changed. That line is the decision, the intervals, the critical-event count, and realised coverage. --json prints the same comparison as structured objects, including each interval's lower and upper bounds and coverage_model.

etalon qualify prints the realised coverage figure with the same value-presence limit. etalon inspect does too, including for a bundle generated by an earlier runner.

qualify exits 0 when a bundle was written, including NOT_QUALIFIED and INDETERMINATE. Exit 2 means the endpoint did not complete the capture.

Air gap

Direct dependencies are pinned in pyproject.toml. Transitive dependencies, including the dev extra, are locked with hashes in requirements.txt. The lock was generated with:

uv pip compile pyproject.toml --extra dev --generate-hashes --universal -o requirements.txt

uv is only the lock generator. Installation stays on pip. Regenerate the lock with that command when a direct pin changes.

--no-index and --find-links install from a local wheelhouse. Those flags do not disable the machine's network.

examples/offline_install.sh fetches the wheelhouse with the network on, then installs in a new network namespace. It refuses to start pip install unless a fresh sysfs lists only lo and a TCP connect to 1.1.1.1:443 returns ENETUNREACH (errno 101). The procedure and the observed output are in docs/offline-install.md.

bash examples/offline_install.sh

That script succeeded on 2026-09-24 on Linux x86_64 with CPython 3.12.3. The fetch environment upgraded pip to 26.2.1. pip download and pip wheel used the network, including build isolation for the project wheel. The install environment was a new virtual environment whose pip was 24.0 (not upgraded). Isolation was sudo unshare --propagation private --mount --net. Inside it, sysfs listed only lo, the connect returned errno 101, and pypi.org did not resolve. Both install commands passed --no-index --find-links and read only /tmp/etalon-wheels. The requirements install also passed --require-hashes. After the script exited, the host still reached 1.1.1.1:443.

etalon pack validate examples/demo-pack printed result: pack matches the schema and pack rules. etalon verify examples/sample-run printed bundle hashes match. On this Linux host, pip ignored the Windows-only colorama line in the lock.

That is evidence for this install. It does not determine that a regulation or an internal control has been met.

The HTTP client sets trust_env=False and talks only to --endpoint. Credentials come from ETALON_API_KEY or OPENAI_API_KEY, never from a pack file. A test asserts that qualify does not open a connection to any other host.

Report redaction

Cases with sensitive: true are quoted as [redacted] in report.html and report.json. The rest of the bundle keeps the captured text. The report says, in each section, whether redaction was applied and to how many items. Tokens and credentials are removed from the bundle entirely ([secret-excluded]), not kept beside a mask.

Language

Say "qualified under pack X", "evaluated against methodology Y", "evidence generated by runner Z". Do not write that a run is a legal determination or that a regulation has been met.

Sign a bundle

etalon keygen writes an Ed25519 seed and a public key. The seed file is mode 0600. It is not encrypted. Keep it offline, out of the bundle, and out of CI. The public key is what a verifier or a deploy job holds. There is no key server. Details are in docs/signing.md.

etalon keygen --private etalon-private.json --public etalon-public.json
etalon sign runs/<run-id> --key etalon-private.json
etalon verify runs/<run-id> --public-key etalon-public.json --require-signature

python3 examples/verify_bundle.py runs/<run-id> \
  --public-key etalon-public.json \
  --require-signature

etalon export runs/<run-id> \
  --public-key etalon-public.json \
  --output auditor/

The signature covers the raw bytes of hashes.json, so it covers every recorded evidence digest, including the pack content hash. Changing an evidence file and rewriting hashes.json without the seed fails verification. examples/verify_bundle.py uses the standard library plus etalon/evidence/ed25519.py. It does not install the runner and it does not open a network connection.

A deploy workflow can fail closed on that result:

- uses: coyos-ai/etalon/.github/actions/etalon-verify@main
  with:
    bundle: evidence/run
    public-key: etalon-public.json

The job fails when the bundle is missing, the hashes do not match, the signature is missing, or the signature was not made with that public key. The action checks qualification evidence. It does not decide that a control has been met. etalon export writes auditor-report.html and manifest.json for a person to read. That package is evidence for review.

Hash a file without installing Etalon

examples/hash_file.py uses only the Python standard library. Bundle files in hashes.json are the SHA-256 of the raw bytes. Prompt hashes on a fingerprint use newline normalisation; pass --text for that.

python3 examples/hash_file.py examples/sample-run/manifest.json
python3 examples/hash_file.py --text examples/demo-pack/system_prompt.txt

Each line is sha256:<64 hex> and the path. Compare the digest with hashes.json or with prompting.system_prompt_sha256 on a fingerprint. This does not verify a bundle. etalon verify checks a bundle when the runner is installed. examples/verify_bundle.py checks a signed bundle with the standard library and a public key, without installing the runner.

Container

The image runs as the non-root user etalon (uid 10001). The working directory is /opt/etalon, owned by that user, so a run directory can be written there.

docker build -t etalon .
docker run --rm etalon pack validate examples/demo-pack

Mount a pack or a run directory when you need to read or write files outside the image.

Layout

etalon/                  Python package (cli, execution, evaluators, scoring,
                       fingerprint, triggers, qualification, evidence,
                       redaction, reporting, schema)
examples/demo-pack/    synthetic pack, no judge
examples/contact-routing/  contact-routing fixture with a judge overlay
examples/mock_endpoint.py
examples/sample-run/   prebuilt QUALIFIED bundle
examples/verify_bundle.py  air-gapped signature check
schema                 symlink to etalon/schema

A pack may declare a rubric_judge. offline-mock scores on the route taxonomy and does not open a socket. endpoint sends the rubric prompt to an OpenAI-compatible chat model (--judge-endpoint, defaulting to --endpoint, and --judge-model). typesafe (alias jev) calls TypeSafe Jev with typesafe-sdk and TYPESAFE_API_KEY (pip install 'etalon[typesafe]'). Evidence records the judge model, prompt or question hash, prompt version, rubric version, and uncertainty. On the typesafe transport the model id is the id Jev returns, uncertainty is the Noul probability that the verdict is uncertain, and choice probabilities and confidence are stored beside it. Missing uncertainty is a judge failure: the run is INDETERMINATE, not a model miss. Uncertainty at or above the pack maximum is INDETERMINATE when the pack lists judge_uncertainty, and it is still stated when a hard miss is already NOT_QUALIFIED.

Calibration cases are not sent to the system under test. The judge scores the candidate outputs a person already scored. The run reports judge–referee agreement, provenance (expert-authored or synthetic), and known-failure cases. Agreement below the declared record is calibration_invalid.

Reference metrics stay the qualification decision. Inspect, compare, and the HTML report show the reference route beside the judge where a judge ran. A critical dimension scored only by a judge is still rejected. The demo pack does not declare a judge. There is no calibration UI.

See docs/README.md for the operator guide. Decision rules are in docs/overview.md.