Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

MORI

Documentation PyPI PyPI nightly main CI

Documentation | Installation | Quickstart | API reference

News

  • [2026/10] 🔥 MoRI UMBP Empowers AMD Instinct™ MI355X GPU to Demonstrate Token-per-Dollar TCO Leadership on the Public AgentX Leaderboard (blog).
  • [2026/07] 🔥 MORI powers Kimi's agentic AI serving solution with UMBP, co-designed by AMD and Moonshot AI for AMD Instinct™ GPUs (blog).
  • [2026/05] 🔥 MORI powers SGLang on AMD Instinct™ MI355X achieves competitive TCO for large-scale DeepSeek disaggregated inference (blog).
  • [2026/05] 🔥 MORI becomes the primary EP communication library for AMD platforms in Alibaba RTP-LLM (MORI-EP PR).
  • [2026/05] MORI's SDMA-based AllGather collective is integrated into DeepSpeed for ZeRO-3 optimization on AMD GPUs, delivering up to 10% end-to-end training speedup by offloading AllGather traffic to dedicated SDMA copy engines (example, post).
  • [2026/04] 🔥 Tencent OpenUCL adopts the Mori ecosystem, using Mori's EP-style dispatch/combine pattern in AMD GPU deployments and leveraging MORI-SHMEM for GPU-initiated communication.
  • [2026/03] 🔥 MORI-SHMEM powers ByteDance Triton-distributed EP dispatch/combine kernels as the backend, delivering seamless integration and high performance on AMD GPUs (EP Kernels, MORI-SHMEM Integration).
  • [2026/02] 🔥 MORI powers AMD's WideEP and PD disaggregation in SemiAnalysis InferenceX v2 benchmark (PR, InferenceX, blog).
  • [2026/01] 🔥 MORI-EP and MORI-IO integrated into SGLang and vLLM for MoE Expert Parallelism and PD Disaggregation on AMD GPUs (sglang & MORI-EP, sglang & MORI-IO, vllm & MORI-EP, vllm & MORI-IO).
  • [2025/12] MORI adds support for AMD's AINIC (Pollara) with SOTA performance (AINIC & MORI-EP, AINIC & MORI-IO).
  • [2025/09] MORI-EP now seamlessly scales to 64 GPUs with SOTA performance (multiple optimizations, multi-QP support, low-latency kernel).
  • [2025/09] MORI adds Broadcom BNXT (Thor2) IBGDA support (PR).

Introduction

MORI architecture

MORI (Modular RDMA Interface) is a bottom-up, modular, and composable framework for building high-performance communication applications with a strong focus on RDMA + GPU integration. Inspired by the role of MLIR in compiler infrastructure, MORI provides reusable and extensible building blocks that make it easier for developers to adopt advanced techniques such as IBGDA (Infiniband GPUDirect Async) and GDS (GPUDirect Storage).

To help developers get started quickly, MORI also includes a suite of optimized libraries—MORI-EP (MoE dispatch & combine kernels, in two generations: EPv1 on MORI-SHMEM and EPv2 on MORI-CCO), MORI-IO (p2p communication for KVCache transfer), MORI-CCL (collective communication), MORI-CCO (Next-Generation Collective Communication Object for GPUs) and MORI-FUSED (GEMMs fused with collectives, built on MORI-CCO)—that deliver out-of-the-box performance, with support for AMD Pensando DSC, Broadcom Thor2, and NVIDIA Mellanox ConnectX-7 NICs.

Features summary

  • Applications
    • MORI-EP: intra and inter-node dispatch/combine kernels with SOTA performance. Two generations:
      • EPv1: built on MORI-SHMEM.
      • EPv2: built on MORI-CCO. Designed natively for gfx1250 and rack-scale EP.
    • MORI-FUSED: GEMMs fused with the all-reduce, reduce-scatter, all-to-all or all-gather next to them, overlapping the transfer with the GEMM over the SDMA copy engines
    • MORI-IO: point-to-point communication library with ultra-low overhead
    • MORI-CCL: lightweight and flexible collective communication library designed for highly customized use cases such as latency-sensitive or resource-constrained environment
    • MORI-UMBP: unified memory & bandwidth pool with tiered storage and distributed key-value access for scalable memory management
    • MORI-CCO: Next-Generation Collective Communication Object for GPUs. Natively designed for compute-communication fusion, with symmetric memory support.
  • Framework
    • High-performance building blocks for IBGDA / P2P and more​
    • Modular & composable components for developing communication applications, such as transport management, topology detection and etc.
    • Open-Shmem-style APIs
    • C++ and Python level APIs

Documentation

📖 https://rocm.github.io/mori/ — the full documentation, with navigation and search. It is built from docs/ on every push to main, so it describes the development tree; each page's title names the commit.

Topic Description Docs
Getting started Installation, fabric checks and first steps Installation · Quickstart
MORI-EP (EPv1, EPv2) EPv1 (built on SHMEM) and EPv2 (built on CCO) dispatch/combine: API, kernel types and backends, configuration, usage examples, how to choose EP Guide · Benchmark · RDMA bandwidth
MORI-FUSED GEMMs fused with the all-reduce, reduce-scatter, all-to-all or all-gather next to them, built on MORI-CCO Guide
MORI-CCO Next-generation collective communication object: communicator, symmetric-memory windows, LSA / GDA / SDMA device APIs CCO Guide
MORI-IO P2P communication concepts, engine/backend/session design IO Guide · Benchmark
MORI-UMBP Unified memory & bandwidth pool: tiered storage, distributed KV access, transfer simulation Single-Node Smoke Test · Simulation · Calibration · PD Benchmark · Tier-Management Benchmark
MORI-VIZ Warp-level kernel profiler with Perfetto integration Profiler

Benchmarks

MORI-EP

Benchmark on DeepSeek V3 model configurations:

Bandwidth (4096 tokens, 7168 hidden, top-8 experts, FP8 dispatch + BF16 combine)

Hardware Kernels Dispatch XGMI Dispatch RDMA Combine XGMI Combine RDMA
MI300X + CX7 EP8 307 GB/sx330 GB/sx
EP16-V1 171 GB/s52 GB/s219 GB/s67 GB/s
EP32-V1 103 GB/s*57 GB/s*91 GB/s*50 GB/s*
MI355X + AINIC EP8 345 GB/sx420 GB/sx
EP16-V1 179 GB/s54 GB/s234 GB/s71 GB/s
EP32-V1 85 GB/s46 GB/s110 GB/s61 GB/s

Latency (128 tokens, 7168 hidden, top-8 experts, FP8 dispatch + BF16 combine)

Hardware Kernels Dispatch Latency Dispatch BW Combine Latency Combine BW
MI300X + CX7 EP8 35 us134 GB/s47 us204 GB/s
EP16-V1-LL 76 us96 GB/s122 us121 GB/s
EP32-V1-LL 157 us*48 GB/s*280 us*55 GB/s*
MI355X + AINIC EP8 31 us142 GB/s36 us276 GB/s
EP16-V1-LL 84 us87 GB/s108 us139 GB/s
EP32-V1-LL 152 us45 GB/s187 us76 GB/s

* Stale data from previous kernel version; updated numbers pending re-benchmarking.

MORI-FUSED

MORI-FUSED overlaps a collective with the GEMM next to it over the SDMA copy engines. The chart compares GEMM + all-reduce (GemmAllReduceOp) with what a serving stack runs when nothing is fused: the GEMM, then an RCCL all-reduce. The layer is DeepSeek-V4-Pro wo_b ([M, 7168], K=2048, fp8 blockscale) at TP8 on 8x MI355X.

Speedup of MORI's fused GEMM + all-reduce over GEMM + RCCL on DeepSeek-V4-Pro wo_b, 8x MI355X: 1.05x to 1.32x with the bf16 wire and 1.11x to 1.57x with the fp8 gather, for M from 1,024 to 16,384

Time per call (µs)
M GEMM + RCCL MORI fused, bf16 wire MORI fused, fp8 gather
1,024 135.9 129.9 (1.05x) 122.4 (1.11x)
2,048 234.4 199.0 (1.18x) 176.4 (1.33x)
4,096 440.1 333.8 (1.32x) 280.7 (1.57x)
8,192 736.2 601.4 (1.22x) 497.4 (1.48x)
16,384 1454.1 1128.3 (1.29x) 934.3 (1.56x)

The other fused operators (GEMM + reduce-scatter, GEMM + all-to-all, GEMM + all-gather, all-gather + GEMM and all-to-all + GEMM) are measured in the MORI-FUSED guide.

MORI-IO

NOTE: This is the preview version of MORI-IO benchmark performance.

GPU Direct RDMA READ, pairwise, 128 consecutive transfers, 1 GPU, MI300X + Thor2:

+--------------------------------------------------------------------------------------------------------+
|                                            Initiator Rank 0                                            |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+
| MsgSize (B) | BatchSize | TotalSize (MB) | Max BW (GB/s) | Avg Bw (GB/s) | Min Lat (us) | Avg Lat (us) |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+
|      8      |    128    |      0.00      |      0.03     |      0.03     |    33.38     |    36.33     |
|      16     |    128    |      0.00      |      0.06     |      0.06     |    34.09     |    36.35     |
|      32     |    128    |      0.00      |      0.12     |      0.11     |    34.57     |    36.33     |
|      64     |    128    |      0.01      |      0.24     |      0.23     |    33.62     |    36.33     |
|     128     |    128    |      0.02      |      0.49     |      0.45     |    33.62     |    36.49     |
|     256     |    128    |      0.03      |      0.94     |      0.89     |    34.81     |    36.99     |
|     512     |    128    |      0.07      |      1.86     |      1.77     |    35.29     |    37.01     |
|     1024    |    128    |      0.13      |      3.84     |      3.53     |    34.09     |    37.09     |
|     2048    |    128    |      0.26      |      7.33     |      6.96     |    35.76     |    37.65     |
|     4096    |    128    |      0.52      |     12.94     |     12.46     |    40.53     |    42.09     |
|     8192    |    128    |      1.05      |     20.75     |     20.12     |    50.54     |    52.11     |
|    16384    |    128    |      2.10      |     29.03     |     28.33     |    72.24     |    74.02     |
|    32768    |    128    |      4.19      |     36.50     |     35.91     |    114.92    |    116.81    |
|    65536    |    128    |      8.39      |     41.74     |     41.39     |    200.99    |    202.70    |
|    131072   |    128    |     16.78      |     45.14     |     44.85     |    371.69    |    374.10    |
|    262144   |    128    |     33.55      |     46.93     |     46.76     |    715.02    |    717.56    |
|    524288   |    128    |     67.11      |     47.94     |     47.81     |   1399.99    |   1403.64    |
|   1048576   |    128    |     134.22     |     48.44     |     48.32     |   2770.90    |   2777.76    |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+

Hardware Support Matrix

GPU

MORI-EP MORI-IO MORI-SHMEM MORI-CCO
MI308X ✅ ✅ ✅ ✅
MI300X ✅ ✅ ✅ ✅
MI325X ✅ ✅ ✅ ✅
MI355X ✅ ✅ ✅ ✅
MI450X 🚧 🚧 🚧 🚧

NIC

MORI-EP MORI-IO MORI-SHMEM MORI-CCO
Pollara ✅ ✅ ✅ ✅
CX7 ✅ ✅ ✅ ✅
Thor2 ✅ ✅ ✅ ✅
Volcano 🚧 🚧 🚧 🚧

✅ Supported   🚧 Under Development

Installation

Prerequisites

  • ROCm >= 6.4 (hipcc needed at runtime for JIT kernel compilation, not at install time)
    • On RHEL/Rocky/AlmaLinux also install gcc-c++: that JIT needs the C++ standard headers, which ROCm's RPMs do not pull in. Without them every kernel compile fails with Could not find standard C++ header 'cmath'.
  • System packages (required for pip install; not bundled in wheels). On Debian/Ubuntu install at least:
  • Optional: libopenmpi-dev, openmpi-bin — only needed when building C++ examples (BUILD_EXAMPLES=ON) or enabling MPI bootstrap (MORI_WITH_MPI=ON)

Or build docker image with:

cd mori && docker build -t rocm/mori:dev -f docker/Dockerfile.dev .

IBGDA NIC support (optional, for GPU-direct RDMA — auto-detected, no manual configuration needed):

NIC User library
AMD Pollara (AINIC) libionic.so
Mellanox ConnectX libmlx5.so (typically pre-installed)
Broadcom Thor2 libbnxt_re.so

Note: IBGDA requires vendor-specific DV (Direct Verbs) libraries. Mellanox libmlx5 is typically pre-installed with the kernel OFED stack. For Thor2 and Pollara, install the corresponding userspace library from your NIC vendor.

Recommended NIC firmware/driver versions (check with mori check or tools/env_check.sh):

Vendor Recommendation
AMD Pollara (AINIC) >= 1.117.5-a-45 (1.117.1 major lacks IBGDA support); 1.117.5 builds after a-45 can mishandle retransmitted RDMA READs, see MORI-IO Known Issues
Broadcom (Thor2) 237.1.137.x / 235.2.86.x; known bad: 231.x, 232.x
Mellanox (ConnectX, mlx5) No known minimum; tested on ConnectX-7

Validate the environment

Before a cross-node run, check the fabric:

mori check <peer_ip>          # or: tools/env_check.sh <peer_ip>

Seven steps — kernel PCI P2PDMA support (CONFIG_PCI_P2PDMA, read from /proc/kallsyms; warns only), NIC firmware and driver, QoS/PFC, DCQCN, then intra-node, inter-node bandwidth and latency meshes over every NIC pair, on host memory and on GPU memory. Exits non-zero if any check failed, and reports the fabric topology, which decides what can run on it.

Install

MoRI can be installed in three ways: from PyPI (stable), nightly pre-built wheels (latest dev), or from source.

From PyPI (stable release)

pip install amd_mori

Nightly (pre-built, tested daily)

pip install --pre amd-mori-nightly

Deprecated: the GitHub Pages nightly index (https://rocm.github.io/mori/nightly/) is being retired. Please install nightly builds from PyPI (amd-mori-nightly) as shown above.

Note: amd-mori and amd-mori-nightly both provide the mori Python module. Do not install both at the same time — uninstall one before installing the other.

From source

git clone --recursive https://github.com/ROCm/mori.git
cd mori
pip install .

For a venv build, add --no-build-isolation — and install the build dependencies first, because that flag is exactly what stops pip from reading them out of pyproject.toml:

pip install -r requirements-build.txt
pip install . --no-build-isolation

Skipping the first line leaves the build without Cython, and it only warns before dropping the mori.cco.cco extension, taking the flydsl EP backend with it — on an install that otherwise reports success. python tools/verify_install.py checks for exactly that.

No hipcc needed at install time — host code compiles with a standard C++ compiler. GPU kernels are JIT-compiled on first use and cached to ~/.mori/jit/. If a GPU is detected during install, kernel precompilation starts automatically in the background.

To manually precompile all kernels (e.g. in a Docker image build):

MORI_PRECOMPILE=1 python -c "import mori"

Verify installation

python -c "import mori; print(mori.__version__)"

Testing

Test MORI-EP (dispatch / combine)

EPv2 (MORI-CCO)

cd /path/to/mori
export PYTHONPATH=/path/to/mori:$PYTHONPATH
python -c "import mori; print(mori.__file__)"

# Test correctness (8 GPUs, gfx942 or gfx950: on gfx125x the default HIP backend
# rejects some of the cases these run, such as scatter combine and StdMoE)
pytest tests/python/ops/dispatch_combine_v2/test_dispatch_combine_v2_intranode.py -q
torchrun --standalone --nproc_per_node=8 tests/python/ops/dispatch_combine_v2/test_ep_backend_parity.py

# Benchmark performance
torchrun --standalone --nproc_per_node=8 tests/python/ops/dispatch_combine_v2/bench_ep.py

Two-node runs and their restrictions: see the EP Guide.

EPv1 (MORI-SHMEM)

cd /path/to/mori
export PYTHONPATH=/path/to/mori:$PYTHONPATH
python -c "import mori; print(mori.__file__)"

# Test correctness (8 GPUs)
pytest tests/python/ops/test_dispatch_combine_intranode.py -q
pytest tests/python/ops/test_dispatch_combine_async_ll.py -q
pytest tests/python/ops/test_dispatch_combine_internode_v1.py -q

# Benchmark performance
python tests/python/ops/bench_dispatch_combine.py

Test MORI-IO

cd /path/to/mori
export PYTHONPATH=/path/to/mori:$PYTHONPATH

# Correctness tests
pytest tests/python/io/

# Benchmark performance (two nodes). The runner defaults to the native C++
# benchmark (tests/cpp/io/bench_engine); pass --engine python for the torchrun path.
tools/run_internode_io_benchmark.sh --rank 0 --master-addr "<master_ip>" --ifname ens14np0 \
  -- --op write --enable-batch-transfer --enable-sess --buffer-size 32768 --transfer-batch-size 128
# (run the same command on the second node with --rank 1)

See the MORI-IO benchmark guide for both engines and tuning.

Contribution Guide

Welcome to MORI! We appreciate your interest in contributing. Whether you're fixing bugs, adding features, improving documentation, or sharing feedback, your contributions help make MORI better for everyone.

Code Quality

MORI uses pre-commit hooks to maintain code quality. After cloning the repository:

pip install pre-commit
cd /path/to/mori
pre-commit install

# Run on all files (first time)
pre-commit run --all-files

Pre-commit automatically checks code formatting, linting, license headers, and other quality checks on commit. To skip checks when necessary: git commit --no-verify

Metadata

Release files for amd-mori-nightly 1.2.6.dev20261119

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for amd-mori-nightly 1.2.6.dev20261119
File Interpreter ABI Platform Actions
amd_mori_nightly-1.2.6.dev20261119-cp312-cp312-manylinux_2_28_x86_64.whl CPython 3.12 CPython 3.12 Linux glibc 2.28+ x86-64 Details for amd_mori_nightly-1.2.6.dev20261119-cp312-cp312-manylinux_2_28_x86_64.whl
amd_mori_nightly-1.2.6.dev20261119-cp310-cp310-manylinux_2_28_x86_64.whl CPython 3.10 CPython 3.10 Linux glibc 2.28+ x86-64 Details for amd_mori_nightly-1.2.6.dev20261119-cp310-cp310-manylinux_2_28_x86_64.whl

Total release size: 34.6 MB

Release files / amd_mori_nightly-1.2.6.dev20261119-cp312-cp312-manylinux_2_28_x86_64.whl

Download URL amd_mori_nightly-1.2.6.dev20261119-cp312-cp312-manylinux_2_28_x86_64.whl
Size 17.3 MB
Tags CPython 3.12 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
a0d7e6af9180548d55f54919f5ec5606cc6fe60f090041b471d9d55a4610ef15
BLAKE2b-256 checksum
How to use checksums
b7cc99f71a7fdfb56d6fa4149fb61b7f325093514d93ecd128d327aefe162a3e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.4

Release files / amd_mori_nightly-1.2.6.dev20261119-cp310-cp310-manylinux_2_28_x86_64.whl

Download URL amd_mori_nightly-1.2.6.dev20261119-cp310-cp310-manylinux_2_28_x86_64.whl
Size 17.2 MB
Tags CPython 3.10 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
e443dd34eca917162e6b2ea3a4e85b52e459b45101421325b9eb69d3273e0323
BLAKE2b-256 checksum
How to use checksums
af2e0bf024c1eb2142c8e9deb94ff12ccd92f28189dd363290f0c74e74e1c080
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.4

Security for amd-mori-nightly 1.2.6.dev20261119

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page