This release is a pre-release and may not be stable for production use.
onnx-genai-server
The onnx-genai command-line tool and OpenAI-compatible server, backed by
ONNX Runtime.
pip install onnx-genai-server # CPU (all platforms; MLX on macOS)
pip install onnx-genai-server[cuda] # + CUDA 13 / cuDNN 9 (Windows/Linux)
ONNX Runtime is not bundled. The command loads libonnxruntime from the
installed onnxruntime (CPU) or onnxruntime-gpu (CUDA) wheel at startup, so it
uses whichever execution providers you installed.
Commands
onnx-genai serve ./model # OpenAI-compatible HTTP server
onnx-genai serve --models-dir ./models # ... or serve every model under a directory
onnx-genai generate ./model "Hello" # one-shot generation
onnx-genai run ./model # interactive REPL
onnx-genai show ./model # resolved files + metadata
onnx-genai list ./models # list models
onnx-genai version # version + execution providers
generate, run, and show accept either a model directory or a config file
inside it (a file resolves to its parent directory).
For generate, MODEL PROMPT is the canonical invocation. --prompt PROMPT
is also supported; supply exactly one prompt, not both forms.
Interactive REPL controls
In onnx-genai run, press Ctrl-C while a response is generating to cancel
that turn and return to the prompt. At an idle prompt, press Ctrl-D or
Ctrl-C (or enter an empty line) to exit. A one-shot onnx-genai generate
run is also cancelled by Ctrl-C mid-generation.
When both stdin and stdout are terminals, the REPL uses a rich line editor:
arrow-key cursor movement, persistent history, bracketed paste, slash-command
completion, and multiline input with Alt+Enter (plain Enter submits).
Interactive terminal text generation shows compact per-turn stats by default;
pass --no-stats to hide them. Stats are enabled only when stdin and stdout are
terminals, and they are printed to stderr; piped stdout remains byte-stable
generated text. When stdin or stdout is piped, the REPL also keeps the original
plain >>> line-loop behavior for scripts and tests.
Generation budget and sampling
When --max-new-tokens is omitted, generate and run follow the model's
effective context window: the CLI uses the remaining context after the prompt, so
generation stops on EOS, a stop sequence, or the full context. Pass
--max-new-tokens to override exactly. If neither metadata nor the decode path
reveals a context limit, the CLI warns and uses a finite fallback instead of
risking an ORT out-of-bounds decode; fix that with --max-context TOKENS or by
declaring model.max_sequence_length in inference metadata.
The REPL recomputes that remaining-context ceiling every turn as conversation
history grows. /stats includes a terse context meter (ctx used / max).
Sampling flags (--temperature above 0, --top-p, or --top-k) switch from
greedy argmax to stochastic sampling. --temperature 0 remains greedy. Use
--greedy or --no-greedy to make the mode explicit.
Polite CPU decode
Use --cpu-cores N with generate or run to cap native CPU decode to N
persistent workers:
onnx-genai generate ./model --prompt "Hello" --cpu-cores 8
Where thread affinity is supported, decode workers are pinned to at most N CPUs
from the process's allowed CPU set, leaving the rest of a shared machine
available to other programs. The equivalent environment setting is
ONNX_GENAI_CPU_DECODE_THREADS=N. Precedence is explicit --cpu-cores >
environment variable > automatic sizing. With neither setting, the
peak-throughput automatic default is unchanged. This controls native decode
workers; use an OS cpuset/taskset as well when the entire process, including
prefill or ONNX Runtime work, must be hard-confined.
For the ONNX Runtime CPU backend, use ONNX_GENAI_INTRA_OP_THREADS=N to
override ORT intra-op parallelism. On Windows ARM64, the automatic ORT CPU
default matches onnxruntime-genai: half of logical CPUs, capped at 16. Other
platforms leave ORT's own default unchanged.
Decode-plan memo (default on)
On the native CPU decode path, a steady-state decode-plan memo caches the
per-step shape/buffer plan and replays it token-to-token, which is token-exact
by construction (shape-only bookkeeping, an in-flight verify net, and a
graceful fall back to rebuilding every step on any model where invariance can't
be proven). It is on by default. To disable it — for example on a
resource-constrained host or when debugging — set ONNX_GENAI_DECODE_MEMO=0
(also accepts false/off, case-insensitive). Any other value, including
unset, keeps it on.
Runtime selection
Selecting the ONNX Runtime library
There are two separate resolution steps:
- Build time (
ort-sys/build.rs) finds headers and a link/development install in this order:ORT_LIB_DIR,ORT_ROOT,pkg-config, then a SHA-256-pinned ORT 1.29.0 download into Cargo'sOUT_DIR. - Run time chooses the shared library the process actually loads in this
order:
ONNX_GENAI_ORT_LIB(exact file),ONNX_GENAI_ORT_LIB_DIR(containing directory), activeCONDA_PREFIX, activeVIRTUAL_ENV, the pinnedort-sysdownload undertarget/.../ort-prebuilt/lib, then the platform dynamic-loader default search path.
Native cargo run / binary builds can honor the runtime variables because
ort-sys resolves OrtGetApiBase with libloading before the first ORT API
call instead of link-loading onnxruntime into the final binary. If that ever
changes back to a normal import-library link, runtime variables cannot override
the OS loader; put the desired directory first in PATH (Windows),
LD_LIBRARY_PATH (Linux), or DYLD_LIBRARY_PATH (macOS) before process start.
Ask the tool what it actually loaded:
onnx-genai version
cargo run -p onnx-genai-cli --bin onnx-genai -- version
The output includes the ORT path, ORT version, API version, and why that library was selected. Any explicit or auto-discovered selection also prints the same line on first ORT use. API mismatches name the loaded library path, loaded API version, required API version, and concrete fixes.
Examples only — adapt paths to your machine:
# Windows PowerShell, conda or venv with a pip onnxruntime wheel.
$prefix = if ($env:CONDA_PREFIX) { $env:CONDA_PREFIX } else { $env:VIRTUAL_ENV }
$env:ONNX_GENAI_ORT_LIB_DIR = "$prefix\Lib\site-packages\onnxruntime\capi"
$env:PATH = "$env:ONNX_GENAI_ORT_LIB_DIR;$env:PATH"
onnx-genai version
# Linux/macOS shell, pip onnxruntime inside the active Python environment.
export ONNX_GENAI_ORT_LIB_DIR="$(python - <<'PY'
import pathlib, onnxruntime
print(pathlib.Path(onnxruntime.__file__).resolve().parent / "capi")
PY
)"
export LD_LIBRARY_PATH="$ONNX_GENAI_ORT_LIB_DIR:${LD_LIBRARY_PATH:-}" # Linux
export DYLD_LIBRARY_PATH="$ONNX_GENAI_ORT_LIB_DIR:${DYLD_LIBRARY_PATH:-}" # macOS
onnx-genai version
# System package or from-source ORT install.
export ONNX_GENAI_ORT_LIB=/opt/onnxruntime/lib/libonnxruntime.so.1
onnx-genai version
# From-source build using the pinned ort-sys download: leave ORT vars unset.
unset ONNX_GENAI_ORT_LIB ONNX_GENAI_ORT_LIB_DIR
cargo run -p onnx-genai-cli --bin onnx-genai -- version
CUDA has two independent switches:
- Build-time features select which CUDA code is compiled in.
--features cudaenables ONNX Runtime's built-inCUDAExecutionProviderpath only.--features native-cudaenables the native backend plus the project's hand-written CUDA EP (onnx-runtime-ep-cuda). - Runtime settings select which path to use.
ONNX_GENAI_EP=cudaasks the ORT session to use CUDA, while--backend auto|ort|nativeselects the decoder forgenerate,transcribe, and the starting decoder forrun. Speech transcription uses the same autoregressive pipeline decoder as multimodal generation, so backend selection is meaningful there too. In the REPL,/backendcan still reload the model and switch the decoder without restarting. Profile output,/session, bare/backend, and/statsreport the resolved backend (autois shown separately as the request when it resolved toortornative).
CUDA failure modes are intentionally distinct:
- If CUDA support was not compiled into the ORT layer,
ONNX_GENAI_EP=cudafails session creation with a "CUDA support not compiled in" error; requestcpu(or rebuild with--features cuda/--features native-cuda) instead. - If CUDA support was compiled in but the provider is unavailable at runtime
(for example, no loadable CUDA provider library, driver, or GPU),
ONNX_GENAI_EP=cudaalso fails session creation and tells you to requestONNX_GENAI_EP=cpuwhen CPU execution is intentional. - When CUDA is compiled in and available for the ORT/native session but the
native CUDA EP cannot claim every executable node, the native runtime falls
back to its CPU EP. Set
ONNX_GENAI_REQUIRE_CUDA=1to reject that node-level CPU fallback. On Apple Silicon, theonnxruntime-ep-mlxplugin is installed by default.
Windows PowerShell example for the native CUDA path:
$env:ONNX_GENAI_EP = "cuda"
$env:ONNX_GENAI_REQUIRE_CUDA = "1"
cargo run --release -p onnx-genai-cli --features native-cuda --bin onnx-genai -- `
generate .\path\to\model --prompt "Hello" --max-new-tokens 64 --backend native --profile
PowerShell native-vs-ORT decode comparison, with the same ORT execution provider for both runs:
$env:ONNX_GENAI_EP = "cuda" # or "cpu"
cargo run --release -p onnx-genai-cli --features native-cuda --bin onnx-genai -- `
generate .\path\to\model --prompt "Hello" --max-new-tokens 128 --backend native --profile
cargo run --release -p onnx-genai-cli --features native-cuda --bin onnx-genai -- `
generate .\path\to\model --prompt "Hello" --max-new-tokens 128 --backend ort --profile
If --backend native is passed to a binary built without native decoder support,
the command fails and tells you to rebuild with --features native-backend (CPU
native/backend path) or --features native-cuda (native CUDA EP). It does not
fall back to ORT.
Python 3.11+ is required (the onnxruntime dependency ships no earlier wheels).
Running from source
The binary is onnx-genai in the onnx-genai-cli package, so everything above
works under cargo run with a -- separating cargo's flags from the tool's:
cargo run -p onnx-genai-cli --bin onnx-genai -- version
Add --release for anything where speed matters — a debug build decodes orders
of magnitude slower, which makes --profile numbers meaningless:
cargo run --release -p onnx-genai-cli --bin onnx-genai -- \
generate ./model --prompt "Hello" --max-new-tokens 64
Trying it without downloading a model
The repository's test fixtures are tiny hand-built ONNX models. They generate nonsense, but they exercise every code path end to end and need no network, so they are the fastest way to check that a change works. Run these from the repository root:
# Text generation, and the interactive REPL
cargo run -p onnx-genai-cli --bin onnx-genai -- \
generate tests/fixtures/tiny-llm --prompt "hello" --max-new-tokens 5 --raw
cargo run -p onnx-genai-cli --bin onnx-genai -- run tests/fixtures/tiny-llm
# Image input (the prompt needs one <image> placeholder per image)
cargo run -p onnx-genai-cli --bin onnx-genai -- \
generate tests/fixtures/tiny-vlm-image-input \
--image path/to/any.png --prompt "describe <image>" --max-new-tokens 3 --raw
# Image generation. This fixture's VAE is 8x8, hence the explicit size
cargo run -p onnx-genai-cli --bin onnx-genai -- \
generate tests/fixtures/tiny-txt2img \
--prompt "a cat" --output-image out.png --steps 2 --width 8 --height 8
# Speech synthesis, then transcription of what it produced
cargo run -p onnx-genai-cli --bin onnx-genai -- \
generate tests/fixtures/tiny-tts --prompt "hello" --output-audio out.wav
cargo run -p onnx-genai-cli --bin onnx-genai -- \
transcribe tests/fixtures/tiny-whisper out.wav
# The OpenAI-compatible server
cargo run -p onnx-genai-cli --bin onnx-genai -- \
serve --model tests/fixtures/tiny-llm --addr 127.0.0.1:8123
curl localhost:8123/v1/models
Model loading logs and default text-generation stats go to stderr; append
2>/dev/null when you only want the generated text. generate --output-image,
generate --output-audio, and transcribe do not print the compact token stats
line by default; use --profile for their timing and throughput diagnostics.
Checking the REPL's live view
Interactive terminal sessions render the reply and live numbers together by default. That path is only taken when stdin and stdout are real terminals, so it cannot be seen through a pipe. Run the REPL directly in a terminal and type a prompt:
cargo run -p onnx-genai-cli --bin onnx-genai -- run tests/fixtures/tiny-llm --max-new-tokens 5
>>> hello
dogtok29over,dog
[ 7 in · 5 out · backend ort · 2.0 tok/s · ttft 1 ms · rss 50.6 MiB ]
Use /stats to toggle the compact line at runtime, or start run or text
generate with --no-stats. Piping a script into the REPL exercises the
byte-stable plain-text fallback instead, which is what the tests do:
printf '/stats\nhello\n\n' | cargo run -p onnx-genai-cli --bin onnx-genai -- \
run tests/fixtures/tiny-llm --max-new-tokens 5
Changing the session without restarting
The REPL can reload the model under a different execution provider or decode backend, and switch models outright:
>>> /ep # current provider, and what this build can select
>>> /ep cpu
>>> /backend ort # auto | ort | native
>>> /model ./another-model
>>> /profile on # report timings, memory, and cache reuse per turn
>>> /session # model, sampling, message counts, and token totals
On Apple Silicon the MLX/Metal execution provider is offered — and auto-selected — when its plugin library is configured, which the Python packages do for you. From a source build, point at it yourself:
ONNX_GENAI_METAL_EP_LIB=$(python -c 'import onnxruntime_mlx, os;
print(os.path.join(os.path.dirname(onnxruntime_mlx.__file__), "libonnxruntime_mlx_ep.dylib"))') cargo run -p onnx-genai-cli --bin onnx-genai -- run ./model
/ep then lists metal and --profile reports it. Selection is non-strict: if
the plugin fails to load, the session falls back to CPU rather than failing.
All but /profile reload the model and clear the conversation. A selection that
fails to load is reported and the previous session keeps running, so an
unavailable provider does not end the session.
Seeing what the KV cache holds
/pages shows the page pool as it stands, which is the view you want when a
pool is filling up and the question is whether that is one long conversation or
many that should be sharing:
kv pages ▓▓▓▓████················ 38%
38 of 100 pages held · 16 tokens/page
512 of 608 token slots used (84% of held pages)
9 shared (23%) — pages more than one sequence or cached prefix holds
references per page
1x 29 pages
2x 7 pages
3x 2 pages
live sequences
sequence pages tokens shared
7 12 190 5
Shared pages are drawn first (▓) because they are the ones that would
otherwise have been duplicated. A model whose KV is not paged says so, rather
than reporting an empty pool.
Tests
Always name packages explicitly. A bare --workspace pulls in mlas-sys, whose
build script is x86-64-Linux-only, and onnx-runtime-cpuinfo, which needs its
git submodule checked out:
cargo test -p onnx-genai-cli
cargo test -p onnx-genai-cli --test repl_e2e
Metadata
Release files for onnx-genai-server 0.1.0.dev6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| onnx_genai_server-0.1.0.dev6.tar.gz | 13.2 MB | With provenance | Details for onnx_genai_server-0.1.0.dev6.tar.gz |
Built distributions (wheels)
| File | Actions Reset | |||
|---|---|---|---|---|
| onnx_genai_server-0.1.0.dev6-cp310-abi3-win_arm64.whl | CPython 3.10 | abi3 | Windows ARM64 | Details for onnx_genai_server-0.1.0.dev6-cp310-abi3-win_arm64.whl |
| onnx_genai_server-0.1.0.dev6-cp310-abi3-win_amd64.whl | CPython 3.10 | abi3 | Windows x86-64 | Details for onnx_genai_server-0.1.0.dev6-cp310-abi3-win_amd64.whl |
| onnx_genai_server-0.1.0.dev6-cp310-abi3-manylinux_2_28_x86_64.whl | CPython 3.10 | abi3 | Linux glibc 2.28+ x86-64 | Details for onnx_genai_server-0.1.0.dev6-cp310-abi3-manylinux_2_28_x86_64.whl |
| onnx_genai_server-0.1.0.dev6-cp310-abi3-macosx_11_0_arm64.whl | CPython 3.10 | abi3 | macOS 11.0+ ARM64 | Details for onnx_genai_server-0.1.0.dev6-cp310-abi3-macosx_11_0_arm64.whl |
Total release size: 70.9 MB
Release files / onnx_genai_server-0.1.0.dev6.tar.gz
| Download URL | onnx_genai_server-0.1.0.dev6.tar.gz |
|---|---|
| Size | 13.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
546e3ab182b97f9e89d2e5427f4b385b33464c061a5c4c04fa697141a3c97af2
|
|
BLAKE2b-256 checksum How to use checksums |
f55db58165306aafde57134367dad2d7ee0febd1707bd0ff3c7ce9aa68237108
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.
Transparency logRelease files / onnx_genai_server-0.1.0.dev6-cp310-abi3-win_arm64.whl
| Download URL | onnx_genai_server-0.1.0.dev6-cp310-abi3-win_arm64.whl |
|---|---|
| Size | 13.8 MB |
| Tags | CPython 3.10 Windows ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
192355c525231ccd781f7b30ff6f82139094421bc7a947f4dd249e22d62ecbd4
|
|
BLAKE2b-256 checksum How to use checksums |
7c46cb9a11e692248d210c1d5f2589857267b59d331def29a3e5741025b0b9ca
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.
Transparency logRelease files / onnx_genai_server-0.1.0.dev6-cp310-abi3-win_amd64.whl
| Download URL | onnx_genai_server-0.1.0.dev6-cp310-abi3-win_amd64.whl |
|---|---|
| Size | 15.3 MB |
| Tags | CPython 3.10 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
43c18de169d66e06aad06c2f14e776189aad6ab17b3c26eb6bdf49da542564e3
|
|
BLAKE2b-256 checksum How to use checksums |
a8638e5817410608e27e3a65d90363098d29d2e79d48a43d097fec18df896d0f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.
Transparency logRelease files / onnx_genai_server-0.1.0.dev6-cp310-abi3-manylinux_2_28_x86_64.whl
| Download URL | onnx_genai_server-0.1.0.dev6-cp310-abi3-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 15.1 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
5cd61ab3e87e3662fe37516beac2ef9ee8027a64bd086e3b05c6adffc222c6e0
|
|
BLAKE2b-256 checksum How to use checksums |
a74d79f3f08df848f4401efc8f6ec46abef217c88d00a8f0aa1fbadae0254801
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.
Transparency logRelease files / onnx_genai_server-0.1.0.dev6-cp310-abi3-macosx_11_0_arm64.whl
| Download URL | onnx_genai_server-0.1.0.dev6-cp310-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 13.4 MB |
| Tags | CPython 3.10 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
d2590cbf2aeb94526cd8494351625ce2c515da18013c5fafd72fab12ae5f3f97
|
|
BLAKE2b-256 checksum How to use checksums |
40c88bd6cf7e98ab2b74c6a7d1569a8cf17755f73d91eab052137fab4298d918
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.
Transparency logSecurity for onnx-genai-server 0.1.0.dev6
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 5 files in this release have been uploaded with provenance metadata.
Attestation for onnx_genai_server-0.1.0.dev6.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 10, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.