gpumutex

This library can be injected into existing CUDA programs with GPUs, such that no process also run under libgpumutex can interfere with another.

This is accomplished through relatively interesting means. See the design writeup: https://apaz.dev/blog/LD_PRELOAD.html

Linux only (glibc; uses dl_iterate_phdr/ELF parsing internally).

Installing with pip

pip install gpumutex

GPUMUTEX_VISIBLE_DEVICES=0,1,2,3 gpumutex-run -- python train.py
# or, equivalently:
gpumutex-run --visible-devices 0,1,2,3 -- python train.py

import gpumutex

gpumutex.run(["python", "train.py"], visible_devices="0,1,2,3", check=True)
gpumutex.system("python train.py", visible_devices=[0, 1, 2, 3])

Using Standalone

# Compile
cc -shared -fPIC -fvisibility=hidden -o libgpumutex.so gpumutex/libgpumutex.c -ldl -lpthread

Then to run a program under it:

LD_PRELOAD=./libgpumutex.so GPUMUTEX_VISIBLE_DEVICES=... program program_args

Configuration

Both usages below are configured the same way as the raw interposer, via environment variables. gpumutex-run and gpumutex.run()/gpumutex.system() also accept these directly as CLI flags / keyword arguments, which set the variable in the child's environment only, without touching your own: