eval-conformance-suite
Run executable rubric-spec adapter checks before claiming schema compatibility.
eval-conformance-suite is for adapter maintainers and evaluation-platform integrators. Its differentiator is an executable, framework-neutral contract: it imports a target Python adapter module and runs 18 synthetic cases covering scale types, anchors, weights, tie-break rules, judge prompt contracts, provenance, validation errors, lint findings, deterministic serialization, and round-trip field preservation.
Inspectable Output
runwrites a JSON report with the target module, overall pass state, and every case name, boolean result, and detail string.badgewrites SVG text to stdout for a caller-supplied label.
The badge command does not run conformance or read a report; it currently renders a passing badge unconditionally. Generate and publish a badge only after independently confirming that the run JSON passed.
Runtime Boundary
Checks run in the local Python process and make no built-in network requests or model calls. The --against value is imported with importlib, and the suite calls that module's to_spec and from_spec functions. Running against an untrusted adapter executes untrusted Python code.
Install
python -m pip install eval-conformance-suite==0.1.2
rubric-spec>=0.1.1 is installed as a dependency. For development from a clone:
python -m pip install -e .
Quickstart
eval-conformance-suite run \
--against rubric_spec.adapters.inspect_ai \
> conformance.json
eval-conformance-suite badge \
--against rubric_spec.adapters.inspect_ai \
> rubric-spec-conformance.svg
The bundled CI also exercises the PromptFoo, DeepEval, LangSmith, and EvalKit adapters shipped by rubric-spec.
See docs/architecture.md and the synthetic report template in examples/conformance_report_template.md.
Release Status
Registry status verified July 13, 2026: version 0.1.2 is published on PyPI and tagged v0.1.2 in the public repository. The project is alpha software. No framework certification, partnership, or adoption claim is made.
Limits
Passing means the adapter satisfied this package's synthetic rubric-spec cases. It is not a framework benchmark, security audit, production-runtime test, or independent certification.
Next Action
Run the suite against the trusted adapter module you intend to support, inspect all 18 case details, fix every failure, and publish a badge only after the JSON report passes.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file eval_conformance_suite-0.1.2.tar.gz.
File metadata
- Download URL: eval_conformance_suite-0.1.2.tar.gz
- Upload date:
- Size: 8.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a8c9e027f11098a72eb2b8fe20fb0d03ea83ed9c2076e56457c2449ecbf5c37c
|
|
| MD5 |
11ac4062823ad6bae7285bdb8c02cedf
|
|
| BLAKE2b-256 |
232e017f434adc180cdd02515059b07c96ce708feef186b35c3820391d380c43
|
Provenance
The following attestation bundles were made for eval_conformance_suite-0.1.2.tar.gz:
Publisher:
release-python.yml on auraoneai/eval-conformance-suite
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
eval_conformance_suite-0.1.2.tar.gz -
Subject digest:
a8c9e027f11098a72eb2b8fe20fb0d03ea83ed9c2076e56457c2449ecbf5c37c - Sigstore transparency entry: 2165375695
- Sigstore integration time:
-
Permalink:
auraoneai/eval-conformance-suite@ad0bcddc2e4acd4d250f6e37eb1605a4124fc4f7 -
Branch / Tag:
refs/tags/v0.1.2 - Owner: https://github.com/auraoneai
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release-python.yml@ad0bcddc2e4acd4d250f6e37eb1605a4124fc4f7 -
Trigger Event:
push
-
Statement type:
File details
Details for the file eval_conformance_suite-0.1.2-py3-none-any.whl.
File metadata
- Download URL: eval_conformance_suite-0.1.2-py3-none-any.whl
- Upload date:
- Size: 9.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
989a154c8cde792fddca0950cc13b775d5b62671c6d39c6555a1a3412fa5d855
|
|
| MD5 |
cce803db8b2d72b0f146cd28945e1705
|
|
| BLAKE2b-256 |
332eb611120f648da90596a7a4a1d4064cc85db1d7e36282347628172d6c62b4
|
Provenance
The following attestation bundles were made for eval_conformance_suite-0.1.2-py3-none-any.whl:
Publisher:
release-python.yml on auraoneai/eval-conformance-suite
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
eval_conformance_suite-0.1.2-py3-none-any.whl -
Subject digest:
989a154c8cde792fddca0950cc13b775d5b62671c6d39c6555a1a3412fa5d855 - Sigstore transparency entry: 2165375701
- Sigstore integration time:
-
Permalink:
auraoneai/eval-conformance-suite@ad0bcddc2e4acd4d250f6e37eb1605a4124fc4f7 -
Branch / Tag:
refs/tags/v0.1.2 - Owner: https://github.com/auraoneai
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release-python.yml@ad0bcddc2e4acd4d250f6e37eb1605a4124fc4f7 -
Trigger Event:
push
-
Statement type:
