Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

onnx-genai

ONNX Runtime-backed text generation for Python.

onnx-genai exposes a small, high-level generation API implemented on ONNX Runtime. It is the ONNX Runtime-compatible counterpart of nxrt.genai: both packages present the same Engine / GenerateResult API, but onnx_genai runs on ONNX Runtime while nxrt.genai runs on the native nxrt runtime.

import onnx_genai

engine = onnx_genai.Engine.from_dir("path/to/model_dir")

result = engine.generate("Hello, world!", max_tokens=64, temperature=0.8)
print(result.text)

# Streaming
def on_token(text, token_id, finish_reason):
    print(text, end="", flush=True)

engine.generate_stream("Tell me a story", on_token, max_tokens=128)

The model directory must contain the ONNX graph(s), tokenizer.json, and model metadata (inference_metadata.yaml or genai_config.json).

Wheels

Distributed as stable-ABI (abi3) wheels tagged cp310-abi3, so a single wheel per platform loads on CPython 3.10 and newer. ONNX Runtime is not bundled: onnx_genai loads libonnxruntime from the installed onnxruntime (CPU) or onnxruntime-gpu (CUDA) wheel at import time, so it uses whichever execution providers you installed.

  • nxrt — the low-level nxrt runtime (inference sessions, eager ops) plus nxrt.genai, the same generation API backed by the native runtime.
  • onnx-genai-server — the onnx-genai command-line tool, including the OpenAI-compatible server (onnx-genai serve).

See https://github.com/justinchuby/onnx-genai.

Metadata

Release files for onnx-genai 0.1.0.dev6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for onnx-genai 0.1.0.dev6
File Size Uploaded Provenance Actions
onnx_genai-0.1.0.dev6.tar.gz 12.8 MB With provenance Details for onnx_genai-0.1.0.dev6.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for onnx-genai 0.1.0.dev6
File Actions
onnx_genai-0.1.0.dev6-cp310-abi3-win_arm64.whl CPython 3.10 abi3 Windows ARM64 Details for onnx_genai-0.1.0.dev6-cp310-abi3-win_arm64.whl
onnx_genai-0.1.0.dev6-cp310-abi3-win_amd64.whl CPython 3.10 abi3 Windows x86-64 Details for onnx_genai-0.1.0.dev6-cp310-abi3-win_amd64.whl
onnx_genai-0.1.0.dev6-cp310-abi3-manylinux_2_28_x86_64.whl CPython 3.10 abi3 Linux glibc 2.28+ x86-64 Details for onnx_genai-0.1.0.dev6-cp310-abi3-manylinux_2_28_x86_64.whl
onnx_genai-0.1.0.dev6-cp310-abi3-macosx_11_0_arm64.whl CPython 3.10 abi3 macOS 11.0+ ARM64 Details for onnx_genai-0.1.0.dev6-cp310-abi3-macosx_11_0_arm64.whl

Total release size: 50.1 MB

Release files / onnx_genai-0.1.0.dev6.tar.gz

Download URL onnx_genai-0.1.0.dev6.tar.gz
Size 12.8 MB
Tags Source
SHA-256 checksum
How to use checksums
12f9b54a42b7f7cbf4dd11e60253a3cba2c28aab7443d667e2bb48984c44ed4c
BLAKE2b-256 checksum
How to use checksums
24d7ffd103c744961d317eef59016a1441006c4a41cea0105b16830e12103ff3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.

Transparency log

Release files / onnx_genai-0.1.0.dev6-cp310-abi3-win_arm64.whl

Download URL onnx_genai-0.1.0.dev6-cp310-abi3-win_arm64.whl
Size 8.8 MB
Tags CPython 3.10 Windows ARM64 abi3
SHA-256 checksum
How to use checksums
33c462064ade56210db33840d15efc61d01eb99c9403e0b1bf115e285041a78f
BLAKE2b-256 checksum
How to use checksums
8d75df47a2e90eaf222f9e4b5ee0558da1292411baf2c3b7b10e32acd6c93421
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.

Transparency log

Release files / onnx_genai-0.1.0.dev6-cp310-abi3-win_amd64.whl

Download URL onnx_genai-0.1.0.dev6-cp310-abi3-win_amd64.whl
Size 10.0 MB
Tags CPython 3.10 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
60686a410fc0edf0721af4f86c0df0f7432da3ae4c7477a7b925fa92faf51c18
BLAKE2b-256 checksum
How to use checksums
7a788baf984e69c612fad4cf48ffb9d70dbed14bcc13cff0e25e40e10dbcd7e2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.

Transparency log

Release files / onnx_genai-0.1.0.dev6-cp310-abi3-manylinux_2_28_x86_64.whl

Download URL onnx_genai-0.1.0.dev6-cp310-abi3-manylinux_2_28_x86_64.whl
Size 10.0 MB
Tags CPython 3.10 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
ea27400c4bd51b6c2daa26e073f959d6c00665070e8fe0c7a3dbcfe50de21d4e
BLAKE2b-256 checksum
How to use checksums
5019b66bfc0b8abf478d3c92dd3603eb7de0d8f1370d29632ee4df7621023e49
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.

Transparency log

Release files / onnx_genai-0.1.0.dev6-cp310-abi3-macosx_11_0_arm64.whl

Download URL onnx_genai-0.1.0.dev6-cp310-abi3-macosx_11_0_arm64.whl
Size 8.5 MB
Tags CPython 3.10 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
ed77d07a22f2d0265d89fdd5c857f3fd4f9df20d8f499d044cc121638cf959ab
BLAKE2b-256 checksum
How to use checksums
28fbca1e715daf32663c5ae159f82adf4efa2e9afa71510d84da5ed5aa676ce0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.

Transparency log

Security for onnx-genai 0.1.0.dev6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 5 files in this release have been uploaded with provenance metadata.

Select from 5 files

Attestation for onnx_genai-0.1.0.dev6.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page