OAAX conformance suite

Checks whether an OAAX runtime (shared library) and/or toolchain (Docker image) conforms to the OAAX specification.

Status: 0.1.0, Linux only. Every test is specified in docs/tests/: its level, the quoted spec clause, and what the C++ CPU runtime and toolchain do today.

Quick start

Needs uv on Linux (x86_64 or aarch64); Docker for toolchains. No compiler: the wheel contains the C harness.

uv tool install --python 3.12 'oaax-conformance[e2e]'   # all tiers; or run once: uvx --python 3.12 --from 'oaax-conformance[e2e]' oaax-conformance ...

# your own runtime or toolchain
oaax-conformance runtime ./libRuntimeLibrary.so --model FILE [--init-config key=value]... [--out DIR]
oaax-conformance toolchain oaax-<vendor>-toolchain:<tag> [--runtime ./libRuntimeLibrary.so] [--out DIR]

No --model: the tests that send requests are BLOCKED and the run is INCOMPLETE (exit 2). Open report.html in --out to see what each check did.

Try it on the reference CPU runtime and toolchain

mkdir -p /tmp/rt && curl -sSL https://nbg1.your-objectstorage.com/oaax/runtimes/2.0.0/CPU/x86_64/Ubuntu/library.tar.gz | tar xz -C /tmp/rt   # aarch64 machine: use aarch64 in the URL
oaax-conformance runtime /tmp/rt/libRuntimeLibrary.so --model @tiny-conv --out /tmp/report-rt   # ~20 s

curl -sSL -o /tmp/oaax-cpu-toolchain.tar https://nbg1.your-objectstorage.com/oaax/conversion-toolchain/2.0.0/CPU/oaax-cpu-toolchain.tar && docker load -i /tmp/oaax-cpu-toolchain.tar   # ~800 MB
oaax-conformance toolchain oaax-cpu-toolchain:2.0.0 --runtime /tmp/rt/libRuntimeLibrary.so --out /tmp/report-tc   # ~30 s

@tiny-conv is the small model the wheel ships. A minimal machine needs libgomp1 (the reference runtime's own dependency) and binutils (nm, readelf, for RT-S-02, which is BLOCKED without them).

Exit code 1 is the expected result for both: they have known findings.

What it checks

• Easy for implementers: one command, no setup beyond Docker/Python; each failure names the spec clause and says what to fix.
• Dead simple for users: a pass means "this runtime/toolchain can be swapped for another one without code changes".
• Strict: the suite talks to the runtime's C API directly, never through a tolerant wrapper.
• Safe on real hardware: loops stop at a time budget, request sizes are capped, a slow device is BLOCKED and never FAIL, and --hardware-safe leaves out the abusive tests.
• Runs anywhere: the static tier needs no accelerator, no model and no init.

Tiers

The toolchain tier at a glance

The suite converts a small built-in model (or --onnx FILE) in temporary directories and runs your image:

• Sandboxed by default: --cap-drop ALL, no-new-privileges and a process limit on every run (--docker-arg can add back what an image really needs, which is worth reporting to its vendor).
• Repeated under restrictions: as an arbitrary unprivileged user (TC-02), without network (TC-03), with a read-only input (TC-04), a read-only root file system (TC-06) and an unwritable output (TC-38). Network stays on in the other runs: a toolchain that needs it is a finding, not a block.
• Hostile inputs: empty, garbage, truncated, missing, a zip with ../ members.
• End to end (--runtime LIB): loads the converted model in that runtime and compares the results with onnxruntime.

More in docs/reference.md.

Levels

Install

• The wheel needs Python 3.12 or newer (uv fetches it if your default is older) and contains the C harness (Linux x86_64 and aarch64). --harness PATH or OAAX_HARNESS override it.
• e2e adds onnx and onnxruntime: reading .onnx models (--onnx, the runtime tier's --model) and the end-to-end comparison.

Reading the results

With --out DIR the suite writes:

Reports never contain the values of --init-config or of docker environment variables (values shorter than 3 characters excepted: the suite says so at the start), and directories are removed from every message (details).

Known findings on the reference implementation

The reference CPU runtime and toolchain have known issues (#45 to #49), so these results are expected:

More

• docs/reference.md: all options, what reports keep out, --docker-arg rules, how the toolchain tier works, disk use.
• docs/design.md: architecture and direction for OAAX 2.1.
• docs/tests/: the test specifications (under review).
• CONTRIBUTING.md, SECURITY.md, CHANGELOG.md.
• The suite tests itself against a mock runtime and a mock toolchain with injected faults: uv run pytest.

License

Apache 2.0, see LICENSE.