nixl-cu12 1.3.0

NIXL Python API

NVIDIA Inference Xfer Library (NIXL)

NVIDIA Inference Xfer Library (NIXL) is targeted for accelerating point to point communications in AI inference frameworks such as NVIDIA Dynamo, while providing an abstraction over various types of memory (e.g., CPU and GPU) and storage (e.g., file, block and object store) through a modular plug-in architecture.

Documentation and Resources

• NIXL overview - Core concepts/architecture overview (docs/nixl.md)

• Python API - Python API usage and examples (docs/python_api.md)

• Backend guide - Backend/plugin development guide (docs/BackendGuide.md)

• Telemetry - Observability and telemetry details (docs/telemetry.md)

• Doxygen guide - API/class diagrams overview (docs/doxygen/nixl_doxygen.md)

• Doxygen images - Diagram assets (docs/doxygen/)

• NIXLBench docs - Benchmark usage guide (benchmark/nixlbench/README.md)

• KVBench docs - KVBench workflows and tutorials (benchmark/kvbench/docs/)

Supported Platforms

NIXL is supported on a Linux environment only. It is tested on Ubuntu (22.04/24.04) and Fedora. macOS and Windows are not currently supported; use a Linux host or container/VM.

Pre-build Distributions

PyPI Wheel

The nixl python API and libraries, including UCX, are available directly through PyPI. For example, if you have a GPU running on a Linux host, container, or VM, you can do the following install:

Install with:

pip install nixl

This installs both CUDA 12 and CUDA 13 backends. At runtime, the correct backend is selected automatically based on the CUDA version reported by PyTorch.

Prerequisites for source build (Linux)

NIXL requires a C++20 compatible compiler (GCC >= 11 or Clang >= 14).

Ubuntu:

$ sudo apt install build-essential cmake pkg-config

Fedora:

$ sudo dnf install gcc-c++ cmake pkg-config

Python

$ pip3 install meson ninja pybind11 tomlkit

UCX

NIXL was tested with UCX version 1.21.x.

GDRCopy is available on Github and is necessary for maximum performance, but UCX and NIXL will work without it.

$ git clone https://github.com/openucx/ucx.git
$ cd ucx
$ git checkout v1.21.x
$ ./autogen.sh
$ ./contrib/configure-release-mt       \
    --enable-shared                    \
    --disable-static                   \
    --disable-doxygen-doc              \
    --enable-optimizations             \
    --enable-cma                       \
    --enable-devel-headers             \
    --with-cuda=<cuda install>         \
    --with-verbs                       \
    --with-dm                          \
    --with-gdrcopy=<gdrcopy install>
$ make -j
$ make -j install-strip
$ ldconfig

ETCD (Optional)

NIXL can use ETCD for metadata distribution and coordination between nodes in distributed environments. To use ETCD with NIXL:

ETCD Server and Client

$ sudo apt install etcd etcd-server etcd-client

# Or use Docker
$ docker run -d -p 2379:2379 quay.io/coreos/etcd:v3.5.1

ETCD CPP API

Installed from https://github.com/etcd-cpp-apiv3/etcd-cpp-apiv3

$ sudo apt install libgrpc-dev libgrpc++-dev libprotobuf-dev protobuf-compiler-grpc
$ sudo apt install libcpprest-dev
$ git clone https://github.com/etcd-cpp-apiv3/etcd-cpp-apiv3.git
$ cd etcd-cpp-apiv3
$ mkdir build && cd build
$ cmake ..
$ make -j$(nproc) && make install

Additional plugins

Some plugins may have additional build requirements, see them here:

• Mooncake
• POSIX
• GDS

Getting started

Build & install

$ meson setup <name_of_build_dir>
$ cd <name_of_build_dir>
$ ninja
$ ninja install

Build Options

Release build (default)

$ meson setup <name_of_build_dir>

Debug build

$ meson setup <name_of_build_dir> --buildtype=debug

NIXL-specific build options

# Example with custom options
$ meson setup <name_of_build_dir> \
    -Dbuild_docs=true \           # Build Doxygen documentation
    -Ducx_path=/path/to/ucx \     # Custom UCX installation path
    -Dinstall_headers=true \      # Install development headers
    -Ddisable_gds_backend=false   # Enable GDS backend

Common build options:

• build_docs: Build Doxygen documentation (default: false)
• ucx_path: Path to UCX installation (default: system path)
• install_headers: Install development headers (default: true)
• disable_gds_backend: Disable GDS backend (default: false)
• cudapath_inc, cudapath_lib: Custom CUDA paths
• static_plugins: Comma-separated list of plugins to build statically
• enable_plugins: Comma-separated list of plugins to build (e.g. -Denable_plugins=UCX,POSIX). Cannot be used with disable_plugins.
• disable_plugins: Comma-separated list of plugins to exclude (e.g. -Ddisable_plugins=GDS). Cannot be used with enable_plugins.
• wheel_variant: Override the Python wheel variant suffix (e.g. -Dwheel_variant=rocm yields nixl_rocm). Empty (default) = autodetect from the CUDA major version.

Building for AMD ROCm

NIXL itself builds vendor-neutrally; CPU-side hardware detection (hwInfo::numAmdGpus) discovers AMD GPUs via PCI vendor 0x1002 whether or not a ROCm toolchain is present. GPU-side ROCm/HIP build support for the benchmark suite lives in nixlbench — see PR #1647 for the use_rocm / rocm_path options there. When packaging a ROCm wheel, pass -Dwheel_variant=rocm so the wheel is named nixl_rocm.

Plugins on ROCm hosts (CUDA toolchain absent):

• UCX — primary transport for AMD GPU memory (requires UCX built with --with-rocm).
• POSIX, OBJ, AZURE_BLOB, HF3FS, MOONCAKE, GUSLI, UCCL — vendor-neutral; build unchanged.
• GDS / GDS_MT, GPUNETIO, LIBFABRIC (with -DHAVE_CUDA) — skip automatically because their CUDA / cuFile / DOCA dependencies are not found.

Known gaps (will be addressed in follow-up PRs):

• nixlbench (the NIXL benchmark tool) needs CUDA-driver-API → HIP translation work before it builds on ROCm. Use examples/cpp/nixl_etcd_example for transfer validation in the meantime.
• LIBFABRIC plugin disabled on ROCm pending header refactor.
• No NVSHMEM-equivalent backend yet (rocSHMEM analog is a candidate for a future plugin).

Environment Variables

There are a few environment variables that can be set to configure the build:

• NIXL_NO_STUBS_FALLBACK: If not set or 0, build NIXL stub library if the library build fails

Building Documentation

If you have Doxygen installed, you can build the documentation:

# Configure with documentation enabled
$ meson setup <name_of_build_dir> -Dbuild_docs=true
$ cd <name_of_build_dir>
$ ninja

# Documentation will be generated in <name_of_build_dir>/html
# After installation (ninja install), documentation will be available in <prefix>/share/doc/nixl/

Python Interface

NIXL provides Python bindings through pybind11. For detailed Python API documentation, see docs/python_api.md.

The preferred way to install the Python bindings is through pip from PyPI:

pip install nixl

This installs both CUDA 12 and CUDA 13 backends. At runtime, the correct backend is selected automatically based on the CUDA version reported by PyTorch.

Installation from source

Prerequisites:

• uv: https://docs.astral.sh/uv/getting-started/installation/
• tomlkit: https://pypi.org/project/tomlkit/
• PyTorch: https://pytorch.org/get-started/locally/

uv is always required even if you have another kind of Python virtual environment manager or if you are using a system-wide Python installation without using a virtual environment.

Example with uv Python virtual environment:

curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:${PATH}"

uv venv .venv --python 3.12
source .venv/bin/activate
uv pip install tomlkit

Example with python-virtualenv:

curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:${PATH}"

python3 -m venv .venv
source .venv/bin/activate
pip install tomlkit

Example with system-wide Python installation without using a virtual environment:

curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:${PATH}"

pip install tomlkit

Then install PyTorch following the instructions on the PyTorch website: https://pytorch.org/get-started/locally/

After installing the prerequisites, you can build and install the NIXL binaries and the Python bindings from source. You have to:

1. Build NIXL binaries and install them
2. Build and install the CUDA platform-specific package (nixl-cu12 or nixl-cu13)
3. Build and install the nixl meta-package

For CUDA 12:

pip install .
meson setup build
ninja -C build install
pip install build/src/bindings/python/nixl-meta/nixl-*-py3-none-any.whl

For CUDA 13:

pip install .
./contrib/tomlutil.py --wheel-name nixl-cu13 pyproject.toml
meson setup build
ninja -C build install
pip install build/src/bindings/python/nixl-meta/nixl-*-py3-none-any.whl

To check if the installation is successful, you can run the following command:

python3 -c "import nixl; agent = nixl.nixl_agent('agent1')"

which should print:

2026-01-08 13:36:27 NIXL INFO    _api.py:363 Backend UCX was instantiated
2026-01-08 13:36:27 NIXL INFO    _api.py:253 Initialized NIXL agent: agent1

You can also run a complete Python example to test the installation:

python3 examples/python/expanded_two_peers.py --mode=target --use_cuda=true --ip=127.0.0.1 --port=4242 &
sleep 5
python3 examples/python/expanded_two_peers.py --mode=initiator --use_cuda=true --ip=127.0.0.1 --port=4242

For more Python examples, see examples/python/.

Rust Bindings

Build

• Use -Drust=true meson option to build rust bindings.
• Use --buildtype=debug for a debug build (default is release).
• Or build manually:

  $ cargo build --release
  

Install

The bindings will be installed under nixl-sys in the configured installation prefix. Can be done using ninja, from project build directory:

$ ninja install

Test

# Rust bindings tests
$ cargo test

Use in your project by adding to Cargo.toml:

[dependencies]
nixl-sys = { path = "path/to/nixl/bindings/rust" }

Other build options

See contrib/README.md for more build options.

Building Docker container

To build the docker container, first clone the current repository. Also make sure you are able to pull docker images to your machine before attempting to build the container.

Run the following from the root folder of the cloned NIXL repository:

$ ./contrib/build-container.sh

By default, the container is built with Ubuntu 24.04. To build a container for Ubuntu 22.04 use the --os option as follows:

$ ./contrib/build-container.sh --os ubuntu22

To see all the options supported by the container use:

$ ./contrib/build-container.sh -h

The container also includes a prebuilt python wheel in /workspace/dist if required for installing/distributing. Also, the wheel can be built with a separate script (see below).

Building the python wheel

The contrib folder also includes a script to build the python wheel with the UCX dependencies. Note, that UCX and other NIXL dependencies are required to be installed.

$ ./contrib/build-wheel.sh

Running with ETCD

NIXL can use ETCD for metadata exchange between distributed nodes. This is especially useful in containerized or cloud-native environments.

Environment Setup

To use ETCD with NIXL, set the following environment variables:

# Set ETCD endpoints (required) - replace localhost with the hostname of the etcd server
export NIXL_ETCD_ENDPOINTS="http://localhost:2379"

# Set ETCD namespace (optional, defaults to /nixl/agents)
export NIXL_ETCD_NAMESPACE="/nixl/agents"

Running the ETCD Example

NIXL includes an example demonstrating metadata exchange and data transfer using ETCD:

# Start an ETCD server if not already running
# For example:
# docker run -d -p 2379:2379 quay.io/coreos/etcd:v3.5.1

# Set the ETCD env variables as above

# Run the example. The two agents in the example will exchange metadata through ETCD
# and perform data transfers
./<nixl_build_path>/examples/nixl_etcd_example

nixlbench Benchmark

For more comprehensive testing, the nixlbench benchmarking tool supports ETCD for worker coordination:

# Build nixlbench (see benchmark/nixlbench/README.md for details)
cd benchmark/nixlbench
meson setup build && cd build && ninja

# Run benchmark with ETCD
./nixlbench --etcd-endpoints http://localhost:2379 --backend UCX --initiator_seg_type VRAM

Code Examples

• C++ examples

• Python examples

Contributing

For contribution guidelines, see CONTRIBUTING.md (CONTRIBUTING.md).

Third-Party Components

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.