oaax-conformance
Checks whether an OAAX runtime (a shared library) or toolchain (a Docker image) follows the OAAX specification. A pass means: this runtime or toolchain can be swapped for another one without code changes.
- Strict: talks to the runtime's C API directly, never through a tolerant wrapper.
- Each failure says what to fix: it names the specification clause.
- Safe on real hardware: time-budgeted loops, capped request sizes, a slow device is
BLOCKEDand neverFAIL,--hardware-safeleaves out the abusive tests. - No compiler needed: the wheel contains the C harness (Linux x86_64 and aarch64).
Install
Needs uv on Linux, and Docker to test toolchains. The wheel needs Python 3.12 or newer; uv fetches it if your default is older.
uv tool install --python 3.12 'oaax-conformance[e2e]'
[e2e] adds onnx and onnxruntime: reading .onnx models and the end-to-end comparison. Without it you still get the toolchain tier and the runtime tier with --input.
Use
# a runtime: the .so and a model it can load
oaax-conformance runtime ./libRuntimeLibrary.so --model model.onnx [--init-config key=value]... [--out DIR]
# a toolchain: its Docker image, optionally with a runtime for the end-to-end check
oaax-conformance toolchain oaax-<vendor>-toolchain:<tag> [--runtime ./libRuntimeLibrary.so] [--out DIR]
No --model: the tests that send requests are BLOCKED and the run is INCOMPLETE. Open report.html in --out to see what each check did.
Try it without any vendor files, on the reference CPU runtime (about 20 seconds, @tiny-conv is a small model the wheel ships):
mkdir -p /tmp/rt && curl -sSL https://nbg1.your-objectstorage.com/oaax/runtimes/2.0.0/CPU/x86_64/Ubuntu/library.tar.gz | tar xz -C /tmp/rt # arm machine: aarch64
oaax-conformance runtime /tmp/rt/libRuntimeLibrary.so --model @tiny-conv --out /tmp/report-rt
The reference runtime has known issues, so exit code 1 is the expected result here. A minimal machine needs libgomp1 and binutils.
What it checks
| Tier | Needs | Checks |
|---|---|---|
runtime/static |
the .so only |
exported symbols, getters, calls before init |
runtime/lifecycle |
a working init |
init/cleanup rules, error text, cycles, threads, unload, memory, exit |
runtime/model |
a model the runtime can load | enqueue/retrieve, id echo, multi-model routing, timeouts, concurrency, input validation, leaks |
toolchain |
the Docker image | arguments, exit codes, log, runs without root / network / with a read-only input, hostile input |
e2e |
both | convert, load, run, compare with the ONNX reference |
Every test has a level: MUST (required by the specification: a failure fails the run), SHOULD (recommended: a warning) or PROPOSED (planned for the next version: a warning). Every test is specified, with the quoted clause, in the test documentation.
Results
| Exit code | Meaning |
|---|---|
0 |
no MUST check failed |
1 |
a MUST failed |
2 |
INCOMPLETE: nothing failed, but a MUST could not run (for example init needs vendor keys: pass them with --init-config) |
3 |
bad command line, missing file, Docker or image |
4 |
a check of the suite itself crashed: please report it |
With --out DIR it writes report.html (self-contained, prints to PDF), report.json (for tools and archives) and junit.xml (for CI). Reports never contain the values of --init-config or of docker environment variables.
Links
- Full README · Reference: all options · Issues · Changelog
- Fetch the runtimes and toolchains to test:
oaax-manager.
Apache 2.0.
Metadata
Release files for oaax-conformance 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distributions (wheels)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| oaax_conformance-0.1.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | Python 3 | none | Linux glibc 2.17+ x86-64 | Details for oaax_conformance-0.1.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
| oaax_conformance-0.1.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl | Python 3 | none | Linux glibc 2.17+ ARM64 | Details for oaax_conformance-0.1.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
Total release size: 221.9 kB
Release files / oaax_conformance-0.1.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | oaax_conformance-0.1.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 110.3 kB |
| Tags | Linux glibc 2.17+ x86-64 Python 3 |
|
SHA-256 checksum How to use checksums |
cbc1d76e537cf5f4165d2ffe67f5a796aa2bcc4c1d50f26cfaba118d05978c0e
|
|
BLAKE2b-256 checksum How to use checksums |
f4a7e4405ac1bc57706e0561dcb581d671d96ecf3c40c6c8e037908f874fff3c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.
Transparency logRelease files / oaax_conformance-0.1.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | oaax_conformance-0.1.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 111.6 kB |
| Tags | Linux glibc 2.17+ ARM64 Python 3 |
|
SHA-256 checksum How to use checksums |
a3c06167981ef6e5c730924c2bcf2b82f38c4be1e8d3841a396ec159553fcbcd
|
|
BLAKE2b-256 checksum How to use checksums |
45942246efe8f35f4500c0ec7511a8777279db8a88a7bee55df3891b73d68592
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.
Transparency logSecurity for oaax-conformance 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 2 files in this release have been uploaded with provenance metadata.
Attestation for oaax_conformance-0.1.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
