Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

onnx-genai-server

The onnx-genai command-line tool and OpenAI-compatible server, backed by ONNX Runtime.

pip install onnx-genai-server            # CPU (all platforms; MLX on macOS)
pip install onnx-genai-server[cuda]      # + CUDA 13 / cuDNN 9 (Windows/Linux)

ONNX Runtime is not bundled. The command loads libonnxruntime from the installed onnxruntime (CPU) or onnxruntime-gpu (CUDA) wheel at startup, so it uses whichever execution providers you installed.

Commands

onnx-genai serve ./model                # OpenAI-compatible HTTP server
onnx-genai serve --models-dir ./models   # ... or serve every model under a directory
onnx-genai generate ./model "Hello"      # one-shot generation
onnx-genai run ./model                   # interactive REPL
onnx-genai show ./model                  # resolved files + metadata
onnx-genai list ./models                 # list models
onnx-genai version                       # version + execution providers

generate, run, and show accept either a model directory or a config file inside it (a file resolves to its parent directory).

For generate, MODEL PROMPT is the canonical invocation. --prompt PROMPT is also supported; supply exactly one prompt, not both forms.

Interactive REPL controls

In onnx-genai run, press Ctrl-C while a response is generating to cancel that turn and return to the prompt. At an idle prompt, press Ctrl-D or Ctrl-C (or enter an empty line) to exit. A one-shot onnx-genai generate run is also cancelled by Ctrl-C mid-generation.

When both stdin and stdout are terminals, the REPL uses a rich line editor: arrow-key cursor movement, persistent history, bracketed paste, slash-command completion, and multiline input with Alt+Enter (plain Enter submits). Interactive terminal text generation shows compact per-turn stats by default; pass --no-stats to hide them. Stats are enabled only when stdin and stdout are terminals, and they are printed to stderr; piped stdout remains byte-stable generated text. When stdin or stdout is piped, the REPL also keeps the original plain >>> line-loop behavior for scripts and tests.

Generation budget and sampling

When --max-new-tokens is omitted, generate and run follow the model's effective context window: the CLI uses the remaining context after the prompt, so generation stops on EOS, a stop sequence, or the full context. Pass --max-new-tokens to override exactly. If neither metadata nor the decode path reveals a context limit, the CLI warns and uses a finite fallback instead of risking an ORT out-of-bounds decode; fix that with --max-context TOKENS or by declaring model.max_sequence_length in inference metadata.

The REPL recomputes that remaining-context ceiling every turn as conversation history grows. /stats includes a terse context meter (ctx used / max).

Sampling flags (--temperature above 0, --top-p, or --top-k) switch from greedy argmax to stochastic sampling. --temperature 0 remains greedy. Use --greedy or --no-greedy to make the mode explicit.

Polite CPU decode

Use --cpu-cores N with generate or run to cap native CPU decode to N persistent workers:

onnx-genai generate ./model --prompt "Hello" --cpu-cores 8

Where thread affinity is supported, decode workers are pinned to at most N CPUs from the process's allowed CPU set, leaving the rest of a shared machine available to other programs. The equivalent environment setting is ONNX_GENAI_CPU_DECODE_THREADS=N. Precedence is explicit --cpu-cores > environment variable > automatic sizing. With neither setting, the peak-throughput automatic default is unchanged. This controls native decode workers; use an OS cpuset/taskset as well when the entire process, including prefill or ONNX Runtime work, must be hard-confined.

For the ONNX Runtime CPU backend, use ONNX_GENAI_INTRA_OP_THREADS=N to override ORT intra-op parallelism. On Windows ARM64, the automatic ORT CPU default matches onnxruntime-genai: half of logical CPUs, capped at 16. Other platforms leave ORT's own default unchanged.

Decode-plan memo (default on)

On the native CPU decode path, a steady-state decode-plan memo caches the per-step shape/buffer plan and replays it token-to-token, which is token-exact by construction (shape-only bookkeeping, an in-flight verify net, and a graceful fall back to rebuilding every step on any model where invariance can't be proven). It is on by default. To disable it — for example on a resource-constrained host or when debugging — set ONNX_GENAI_DECODE_MEMO=0 (also accepts false/off, case-insensitive). Any other value, including unset, keeps it on.

Runtime selection

Selecting the ONNX Runtime library

There are two separate resolution steps:

  • Build time (ort-sys/build.rs) finds headers and a link/development install in this order: ORT_LIB_DIR, ORT_ROOT, pkg-config, then a SHA-256-pinned ORT 1.29.0 download into Cargo's OUT_DIR.
  • Run time chooses the shared library the process actually loads in this order: ONNX_GENAI_ORT_LIB (exact file), ONNX_GENAI_ORT_LIB_DIR (containing directory), active CONDA_PREFIX, active VIRTUAL_ENV, the pinned ort-sys download under target/.../ort-prebuilt/lib, then the platform dynamic-loader default search path.

Native cargo run / binary builds can honor the runtime variables because ort-sys resolves OrtGetApiBase with libloading before the first ORT API call instead of link-loading onnxruntime into the final binary. If that ever changes back to a normal import-library link, runtime variables cannot override the OS loader; put the desired directory first in PATH (Windows), LD_LIBRARY_PATH (Linux), or DYLD_LIBRARY_PATH (macOS) before process start.

Ask the tool what it actually loaded:

onnx-genai version
cargo run -p onnx-genai-cli --bin onnx-genai -- version

The output includes the ORT path, ORT version, API version, and why that library was selected. Any explicit or auto-discovered selection also prints the same line on first ORT use. API mismatches name the loaded library path, loaded API version, required API version, and concrete fixes.

Examples only — adapt paths to your machine:

# Windows PowerShell, conda or venv with a pip onnxruntime wheel.
$prefix = if ($env:CONDA_PREFIX) { $env:CONDA_PREFIX } else { $env:VIRTUAL_ENV }
$env:ONNX_GENAI_ORT_LIB_DIR = "$prefix\Lib\site-packages\onnxruntime\capi"
$env:PATH = "$env:ONNX_GENAI_ORT_LIB_DIR;$env:PATH"
onnx-genai version
# Linux/macOS shell, pip onnxruntime inside the active Python environment.
export ONNX_GENAI_ORT_LIB_DIR="$(python - <<'PY'
import pathlib, onnxruntime
print(pathlib.Path(onnxruntime.__file__).resolve().parent / "capi")
PY
)"
export LD_LIBRARY_PATH="$ONNX_GENAI_ORT_LIB_DIR:${LD_LIBRARY_PATH:-}"  # Linux
export DYLD_LIBRARY_PATH="$ONNX_GENAI_ORT_LIB_DIR:${DYLD_LIBRARY_PATH:-}"  # macOS
onnx-genai version
# System package or from-source ORT install.
export ONNX_GENAI_ORT_LIB=/opt/onnxruntime/lib/libonnxruntime.so.1
onnx-genai version
# From-source build using the pinned ort-sys download: leave ORT vars unset.
unset ONNX_GENAI_ORT_LIB ONNX_GENAI_ORT_LIB_DIR
cargo run -p onnx-genai-cli --bin onnx-genai -- version

CUDA has two independent switches:

  1. Build-time features select which CUDA code is compiled in. --features cuda enables ONNX Runtime's built-in CUDAExecutionProvider path only. --features native-cuda enables the native backend plus the project's hand-written CUDA EP (onnx-runtime-ep-cuda).
  2. Runtime settings select which path to use. ONNX_GENAI_EP=cuda asks the ORT session to use CUDA, while --backend auto|ort|native selects the decoder for generate, transcribe, and the starting decoder for run. Speech transcription uses the same autoregressive pipeline decoder as multimodal generation, so backend selection is meaningful there too. In the REPL, /backend can still reload the model and switch the decoder without restarting. Profile output, /session, bare /backend, and /stats report the resolved backend (auto is shown separately as the request when it resolved to ort or native).

CUDA failure modes are intentionally distinct:

  • If CUDA support was not compiled into the ORT layer, ONNX_GENAI_EP=cuda fails session creation with a "CUDA support not compiled in" error; request cpu (or rebuild with --features cuda / --features native-cuda) instead.
  • If CUDA support was compiled in but the provider is unavailable at runtime (for example, no loadable CUDA provider library, driver, or GPU), ONNX_GENAI_EP=cuda also fails session creation and tells you to request ONNX_GENAI_EP=cpu when CPU execution is intentional.
  • When CUDA is compiled in and available for the ORT/native session but the native CUDA EP cannot claim every executable node, the native runtime falls back to its CPU EP. Set ONNX_GENAI_REQUIRE_CUDA=1 to reject that node-level CPU fallback. On Apple Silicon, the onnxruntime-ep-mlx plugin is installed by default.

Windows PowerShell example for the native CUDA path:

$env:ONNX_GENAI_EP = "cuda"
$env:ONNX_GENAI_REQUIRE_CUDA = "1"
cargo run --release -p onnx-genai-cli --features native-cuda --bin onnx-genai -- `
  generate .\path\to\model --prompt "Hello" --max-new-tokens 64 --backend native --profile

PowerShell native-vs-ORT decode comparison, with the same ORT execution provider for both runs:

$env:ONNX_GENAI_EP = "cuda"  # or "cpu"
cargo run --release -p onnx-genai-cli --features native-cuda --bin onnx-genai -- `
  generate .\path\to\model --prompt "Hello" --max-new-tokens 128 --backend native --profile
cargo run --release -p onnx-genai-cli --features native-cuda --bin onnx-genai -- `
  generate .\path\to\model --prompt "Hello" --max-new-tokens 128 --backend ort --profile

If --backend native is passed to a binary built without native decoder support, the command fails and tells you to rebuild with --features native-backend (CPU native/backend path) or --features native-cuda (native CUDA EP). It does not fall back to ORT.

Python 3.11+ is required (the onnxruntime dependency ships no earlier wheels).

Running from source

The binary is onnx-genai in the onnx-genai-cli package, so everything above works under cargo run with a -- separating cargo's flags from the tool's:

cargo run -p onnx-genai-cli --bin onnx-genai -- version

Add --release for anything where speed matters — a debug build decodes orders of magnitude slower, which makes --profile numbers meaningless:

cargo run --release -p onnx-genai-cli --bin onnx-genai -- \
  generate ./model --prompt "Hello" --max-new-tokens 64

Trying it without downloading a model

The repository's test fixtures are tiny hand-built ONNX models. They generate nonsense, but they exercise every code path end to end and need no network, so they are the fastest way to check that a change works. Run these from the repository root:

# Text generation, and the interactive REPL
cargo run -p onnx-genai-cli --bin onnx-genai -- \
  generate tests/fixtures/tiny-llm --prompt "hello" --max-new-tokens 5 --raw
cargo run -p onnx-genai-cli --bin onnx-genai -- run tests/fixtures/tiny-llm

# Image input (the prompt needs one <image> placeholder per image)
cargo run -p onnx-genai-cli --bin onnx-genai -- \
  generate tests/fixtures/tiny-vlm-image-input \
  --image path/to/any.png --prompt "describe <image>" --max-new-tokens 3 --raw

# Image generation. This fixture's VAE is 8x8, hence the explicit size
cargo run -p onnx-genai-cli --bin onnx-genai -- \
  generate tests/fixtures/tiny-txt2img \
  --prompt "a cat" --output-image out.png --steps 2 --width 8 --height 8

# Speech synthesis, then transcription of what it produced
cargo run -p onnx-genai-cli --bin onnx-genai -- \
  generate tests/fixtures/tiny-tts --prompt "hello" --output-audio out.wav
cargo run -p onnx-genai-cli --bin onnx-genai -- \
  transcribe tests/fixtures/tiny-whisper out.wav

# The OpenAI-compatible server
cargo run -p onnx-genai-cli --bin onnx-genai -- \
  serve --model tests/fixtures/tiny-llm --addr 127.0.0.1:8123
curl localhost:8123/v1/models

Model loading logs and default text-generation stats go to stderr; append 2>/dev/null when you only want the generated text. generate --output-image, generate --output-audio, and transcribe do not print the compact token stats line by default; use --profile for their timing and throughput diagnostics.

Checking the REPL's live view

Interactive terminal sessions render the reply and live numbers together by default. That path is only taken when stdin and stdout are real terminals, so it cannot be seen through a pipe. Run the REPL directly in a terminal and type a prompt:

cargo run -p onnx-genai-cli --bin onnx-genai -- run tests/fixtures/tiny-llm --max-new-tokens 5
>>> hello
dogtok29over,dog
[ 7 in · 5 out · backend ort · 2.0 tok/s · ttft 1 ms · rss 50.6 MiB ]

Use /stats to toggle the compact line at runtime, or start run or text generate with --no-stats. Piping a script into the REPL exercises the byte-stable plain-text fallback instead, which is what the tests do:

printf '/stats\nhello\n\n' | cargo run -p onnx-genai-cli --bin onnx-genai -- \
  run tests/fixtures/tiny-llm --max-new-tokens 5

Changing the session without restarting

The REPL can reload the model under a different execution provider or decode backend, and switch models outright:

>>> /ep                     # current provider, and what this build can select
>>> /ep cpu
>>> /backend ort            # auto | ort | native
>>> /model ./another-model
>>> /profile on             # report timings, memory, and cache reuse per turn
>>> /session                # model, sampling, message counts, and token totals

On Apple Silicon the MLX/Metal execution provider is offered — and auto-selected — when its plugin library is configured, which the Python packages do for you. From a source build, point at it yourself:

ONNX_GENAI_METAL_EP_LIB=$(python -c 'import onnxruntime_mlx, os;
print(os.path.join(os.path.dirname(onnxruntime_mlx.__file__), "libonnxruntime_mlx_ep.dylib"))') cargo run -p onnx-genai-cli --bin onnx-genai -- run ./model

/ep then lists metal and --profile reports it. Selection is non-strict: if the plugin fails to load, the session falls back to CPU rather than failing.

All but /profile reload the model and clear the conversation. A selection that fails to load is reported and the previous session keeps running, so an unavailable provider does not end the session.

Seeing what the KV cache holds

/pages shows the page pool as it stands, which is the view you want when a pool is filling up and the question is whether that is one long conversation or many that should be sharing:

kv pages   ▓▓▓▓████················  38%
           38 of 100 pages held · 16 tokens/page
           512 of 608 token slots used (84% of held pages)
           9 shared (23%) — pages more than one sequence or cached prefix holds

references per page
    1x      29 pages
    2x       7 pages
    3x       2 pages

live sequences
  sequence     pages   tokens   shared
  7               12      190        5

Shared pages are drawn first (▓) because they are the ones that would otherwise have been duplicated. A model whose KV is not paged says so, rather than reporting an empty pool.

Tests

Always name packages explicitly. A bare --workspace pulls in mlas-sys, whose build script is x86-64-Linux-only, and onnx-runtime-cpuinfo, which needs its git submodule checked out:

cargo test -p onnx-genai-cli
cargo test -p onnx-genai-cli --test repl_e2e

Metadata

Release files for onnx-genai-server 0.1.0.dev6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for onnx-genai-server 0.1.0.dev6
File Size Uploaded Provenance Actions
onnx_genai_server-0.1.0.dev6.tar.gz 13.2 MB With provenance Details for onnx_genai_server-0.1.0.dev6.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for onnx-genai-server 0.1.0.dev6
File Actions
onnx_genai_server-0.1.0.dev6-cp310-abi3-win_arm64.whl CPython 3.10 abi3 Windows ARM64 Details for onnx_genai_server-0.1.0.dev6-cp310-abi3-win_arm64.whl
onnx_genai_server-0.1.0.dev6-cp310-abi3-win_amd64.whl CPython 3.10 abi3 Windows x86-64 Details for onnx_genai_server-0.1.0.dev6-cp310-abi3-win_amd64.whl
onnx_genai_server-0.1.0.dev6-cp310-abi3-manylinux_2_28_x86_64.whl CPython 3.10 abi3 Linux glibc 2.28+ x86-64 Details for onnx_genai_server-0.1.0.dev6-cp310-abi3-manylinux_2_28_x86_64.whl
onnx_genai_server-0.1.0.dev6-cp310-abi3-macosx_11_0_arm64.whl CPython 3.10 abi3 macOS 11.0+ ARM64 Details for onnx_genai_server-0.1.0.dev6-cp310-abi3-macosx_11_0_arm64.whl

Total release size: 70.9 MB

Release files / onnx_genai_server-0.1.0.dev6.tar.gz

Download URL onnx_genai_server-0.1.0.dev6.tar.gz
Size 13.2 MB
Tags Source
SHA-256 checksum
How to use checksums
546e3ab182b97f9e89d2e5427f4b385b33464c061a5c4c04fa697141a3c97af2
BLAKE2b-256 checksum
How to use checksums
f55db58165306aafde57134367dad2d7ee0febd1707bd0ff3c7ce9aa68237108
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.

Transparency log

Release files / onnx_genai_server-0.1.0.dev6-cp310-abi3-win_arm64.whl

Download URL onnx_genai_server-0.1.0.dev6-cp310-abi3-win_arm64.whl
Size 13.8 MB
Tags CPython 3.10 Windows ARM64 abi3
SHA-256 checksum
How to use checksums
192355c525231ccd781f7b30ff6f82139094421bc7a947f4dd249e22d62ecbd4
BLAKE2b-256 checksum
How to use checksums
7c46cb9a11e692248d210c1d5f2589857267b59d331def29a3e5741025b0b9ca
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.

Transparency log

Release files / onnx_genai_server-0.1.0.dev6-cp310-abi3-win_amd64.whl

Download URL onnx_genai_server-0.1.0.dev6-cp310-abi3-win_amd64.whl
Size 15.3 MB
Tags CPython 3.10 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
43c18de169d66e06aad06c2f14e776189aad6ab17b3c26eb6bdf49da542564e3
BLAKE2b-256 checksum
How to use checksums
a8638e5817410608e27e3a65d90363098d29d2e79d48a43d097fec18df896d0f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.

Transparency log

Release files / onnx_genai_server-0.1.0.dev6-cp310-abi3-manylinux_2_28_x86_64.whl

Download URL onnx_genai_server-0.1.0.dev6-cp310-abi3-manylinux_2_28_x86_64.whl
Size 15.1 MB
Tags CPython 3.10 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
5cd61ab3e87e3662fe37516beac2ef9ee8027a64bd086e3b05c6adffc222c6e0
BLAKE2b-256 checksum
How to use checksums
a74d79f3f08df848f4401efc8f6ec46abef217c88d00a8f0aa1fbadae0254801
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.

Transparency log

Release files / onnx_genai_server-0.1.0.dev6-cp310-abi3-macosx_11_0_arm64.whl

Download URL onnx_genai_server-0.1.0.dev6-cp310-abi3-macosx_11_0_arm64.whl
Size 13.4 MB
Tags CPython 3.10 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
d2590cbf2aeb94526cd8494351625ce2c515da18013c5fafd72fab12ae5f3f97
BLAKE2b-256 checksum
How to use checksums
40c88bd6cf7e98ab2b74c6a7d1569a8cf17755f73d91eab052137fab4298d918
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.

Transparency log

Security for onnx-genai-server 0.1.0.dev6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 5 files in this release have been uploaded with provenance metadata.

Select from 5 files

Attestation for onnx_genai_server-0.1.0.dev6.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page