efa

Low-level Python bindings for AWS Elastic Fabric Adapter (EFA), including Scalable Reliable Datagram (SRD), one-sided RDMA, and GPUDirect transfers to and from torch CUDA tensors.

The package wraps libibverbs and EFA's libefa direct-verbs API in Cython. Its data path calls the provider's inline verbs directly, releases the GIL around blocking and posting operations, and does not import torch or link against CUDA.

  • No Python runtime dependencies.
  • SRD and UD queue pairs with SEND, RDMA read, and RDMA write operations.
  • Host buffers, CUDA device pointers, and dma-buf memory registration.
  • Classic and extended CQs, sender GID and unsolicited-write metadata, and direct EFA CQ, SQ, RQ, MR, and AH queries.
  • One Linux abi3 wheel for CPython 3.9 and newer.

Requirements

  • Linux on an AWS instance with one or more EFA devices attached.
  • libibverbs.so.1 and libefa.so.1 at runtime. The AWS EFA installer and current rdma-core distributions provide both.
  • The EFA device nodes available inside the process or container, normally /dev/infiniband/uverbs*.
  • A C compiler, Cython, and the rdma-core development headers only when building from source.

For GPUDirect, the EFA device and instance type must support RDMA read/write, and the NVIDIA driver must support dma-buf export or nvidia_peermem.

Install

pip install efa

To build from this checkout:

pip install "Cython>=3.0" "setuptools>=77" wheel
pip install ./efa

SRD Quickstart

SRD is reliable and connectionless. Each process creates a ready-to-send QP, exchanges a 24-byte EndpointInfo out of band, and resolves the remote GID to an address handle:

import numpy as np
import efa

dev = efa.get_efa_device_list()[0]
ctx = dev.open()
pd = ctx.alloc_pd()
cq = ctx.create_cq(256)
qp = pd.create_qp(
    efa.QPInitAttr(send_cq=cq, recv_cq=cq)
).prepare(qkey=0x1234)

local_info = efa.local_endpoint_info(qp, qkey=0x1234)
# Exchange local_info.to_bytes() with the other process.
remote_info = efa.EndpointInfo.from_bytes(remote_bytes)
peer = remote_info.peer(pd)

buf = np.zeros(4096, dtype=np.uint8)
access = (
    efa.AccessFlags.LOCAL_WRITE
    | efa.AccessFlags.REMOTE_WRITE
    | efa.AccessFlags.REMOTE_READ
)
mr = efa.reg_tensor(pd, buf, access)

qp.post_send(efa.SendWR(
    wr_id=1,
    sg_list=[mr.sge()],
    opcode=efa.WROpcode.SEND,
    send_flags=efa.SendFlags.SIGNALED,
    dest=peer,
))
for wc in cq.poll(16):
    wc.raise_for_status()

Every handle is an idempotent context manager. A QP retains its PD and CQs, an MR retains its PD and backing tensor, and an SGE retains the MR it addresses.

One-sided RDMA

EFA requires the responder to have an address handle for the requester before it accepts RDMA reads or writes. Both processes should therefore resolve the other process's EndpointInfo, even if traffic is currently one-way. A missing reverse AH completes with REM_OP_ERR and EFA vendor status 0x0e (REMOTE_ERROR_UNKNOWN_PEER).

Large buffers can be split at the device's max_rdma_size:

wrs = efa.write_wrs(local_mr, peer, remote_addr, remote_rkey)
qp.post_send(wrs)

read_wrs provides the corresponding RDMA-read operation. Every generated WR is signaled, as required by EFA.

GPUDirect With Torch

The optional efa.cuda module is torch-free and duck-types objects exposing data_ptr(), numel(), and element_size(). For torch's dma-buf path, enable VMM-backed allocations before CUDA initializes:

import os
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"

import torch
import efa
import efa.cuda

src = torch.arange(1 << 20, dtype=torch.float32, device="cuda:0")
dst = torch.zeros_like(src)

src_mr = efa.cuda.register_tensor(pd, src, access)
dst_mr = efa.cuda.register_tensor(pd, dst, access)

torch.cuda.synchronize(src.device)
qp.post_send(efa.SendWR(
    wr_id=2,
    sg_list=[src_mr.sge()],
    opcode=efa.WROpcode.RDMA_WRITE,
    send_flags=efa.SendFlags.SIGNALED,
    remote_addr=remote_dst_addr,
    rkey=remote_dst_rkey,
    dest=peer,
))

After the receiver observes a completion or protocol-level write notification, it must order inbound NIC writes before CUDA consumes the destination:

with torch.cuda.device(dst.device):
    efa.cuda.flush_gpudirect_writes()

register_tensor first exports a dma-buf fd and calls ibv_reg_dmabuf_mr. If that path is unavailable, it falls back to ibv_reg_mr, which requires nvidia_peermem. The returned GpuMR retains the tensor allocation and keeps the actual CUDA virtual address because ibv_mr.addr is not meaningful for a dma-buf MR.

Direct EFA API

Area API
Device capabilities Context.query_efa_device
EFA CQ creation Context.create_cq_ex
Sender and unsolicited metadata CQEx.poll, WC.sgid, WC.unsolicited
CQ layout CQ.query_efa, CQEx.query_efa
SQ/RQ layout QP.query_wqs
MR interconnect IDs MR.query_efa
Address-handle number AH.ahn
SRD QP creation PD.create_qp with QPType.SRD

When using unsolicited RDMA write-with-immediate completions, create every communicating QP with QPInitAttr(..., unsolicited_write_recv=True) and use an extended CQ created with unsolicited=True. EFA requires peers to negotiate the same QP feature set.

The direct layout queries expose process-local addresses for advanced consumers. They do not transfer ownership of provider memory.

Testing

pip install -e "./efa[test,gpu]"
cd efa
pytest -rs

Tests marked integration exercise real EFA hardware. Tests marked gpu perform torch-verified GPU-to-GPU, GPU-to-host, and host-to-GPU transfers. Unavailable hardware capabilities are skipped explicitly. Set RDMA4PY_SKIP_HARDWARE_TESTS=1 to force all hardware-dependent tests to skip.

License

BSD-3-Clause. See LICENSE.

Project links

Data verified by PyPI on Jul 28, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (2026.9.6)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for d4l3k from gravatar.com d4l3k

Credits

Author: Tristan Rice

GitHub Statistics

Data verified by PyPI on Jul 28, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

BSD-3-Clause
View SPDX License List

Requires

Python >=3.9

Provides Extra

test gpu

Tags

efa srd rdma ibverbs aws gpudirect networking

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

efa-2026.7.28.tar.gz (360.5 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

efa-2026.7.28-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (1.2 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ ARM64manylinux: glibc 2.28+ ARM64

efa-2026.7.28-cp39-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.28+ x86-64manylinux: glibc 2.5+ x86-64

File details

Details for the file efa-2026.7.28.tar.gz.

File metadata

  • Download URL: efa-2026.7.28.tar.gz
  • Upload date:
  • Size: 360.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for efa-2026.7.28.tar.gz
Algorithm Hash digest
SHA256 173a773e96ee7abcf6535752283485d2820d6ae6558f324f367c6237292e73eb
MD5 5b42d6eedf5914628a4d3019b73a064e
BLAKE2b-256 f3ae16a5cdbd287fc6f7230d004fc1d7bf81b4c6f39efcc84b8171a796821e62

See more details on using hashes here.

Provenance

The following attestation bundles were made for efa-2026.7.28.tar.gz:

Publisher: publish_efa.yml on d4l3k/rdma4py

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file efa-2026.7.28-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for efa-2026.7.28-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 39c5dea185c99543e31a033a443c4c9da1baa2d806ca1dc3149059cbedc3d8b1
MD5 eb4a821cc509cba9674528c9cbc8e308
BLAKE2b-256 89849398ebb6f45774b2a470bcf1d4614f06d4120d07a1ac6b05b4560033f5d1

See more details on using hashes here.

Provenance

The following attestation bundles were made for efa-2026.7.28-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl:

Publisher: publish_efa.yml on d4l3k/rdma4py

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file efa-2026.7.28-cp39-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.

File metadata

File hashes

Hashes for efa-2026.7.28-cp39-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl
Algorithm Hash digest
SHA256 bf1cfa7bebd6c7f338c5d4ffad9174420db3f004b44ea4373418591f36e4a7b9
MD5 e53c986bc0cb555816261f71955e6df6
BLAKE2b-256 ce77e0058a6217093a7bcd920c4062c13b3e856d653046696c01b4369543ba7f

See more details on using hashes here.

Provenance

The following attestation bundles were made for efa-2026.7.28-cp39-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl:

Publisher: publish_efa.yml on d4l3k/rdma4py

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

2026.9.6

3 files

2026.9.5

3 files

2026.9.4

3 files

2026.9.3

3 files

2026.9.2

3 files

2026.9.1

3 files

2026.8.31

3 files

2026.8.30

3 files

2026.8.29

3 files

2026.8.28

3 files

2026.8.27

3 files

2026.8.26

3 files

2026.8.25

3 files

2026.8.24

3 files

2026.8.23

3 files

2026.8.22

3 files

2026.8.21

3 files

2026.8.20

3 files

2026.8.19

3 files

2026.8.18

3 files

2026.8.17

3 files

2026.8.16

3 files

2026.8.15

3 files

2026.8.14

3 files

2026.8.13

3 files

2026.8.12

3 files

2026.8.11

3 files

2026.8.10

3 files

2026.8.9

3 files

2026.8.8

3 files

2026.8.7

3 files

2026.8.6

3 files

2026.8.5

3 files

2026.8.4

3 files

2026.8.3

3 files

2026.8.2

3 files

2026.8.1

3 files

2026.7.31

3 files

2026.7.30

3 files

2026.7.29

3 files

This release

2026.7.28 This release

3 files

2026.7.27

3 files

2026.7.26

3 files

2026.7.25

3 files

2026.7.24

3 files

2026.7.23

3 files

2026.7.22

3 files

2026.7.21

3 files

2026.7.20

3 files

2026.7.19

3 files

2026.7.18

3 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page