gpu-burn

gpu-burn runs a configurable matrix-multiplication workload on one or more NVIDIA GPUs using PaddlePaddle. It dynamically yields each GPU when another process starts using a configured percentage of device memory, then resumes after that GPU has remained idle for the configured period.

This project is not related to the CUDA/C++ project wilicc/gpu-burn. It does not validate calculation results and should not be used as a hardware fault or ECC diagnostic.

Requirements

• Linux and Python 3.10 or newer
• NVIDIA GPU and a compatible CUDA driver
• PaddlePaddle with CUDA support

The PyPI package explicitly depends on paddlepaddle-gpu>=2.6.2,<4.0.0. PaddlePaddle publishes different packages and indexes for some CUDA versions, so install the build recommended by the official PaddlePaddle installation guide when the default PyPI build does not match your CUDA environment.

Installation

Install from PyPI:

python -m pip install gpu-burn

For a CUDA-specific PaddlePaddle build, install PaddlePaddle first and then avoid letting pip replace it:

# Install the appropriate paddlepaddle-gpu build from the official Paddle index.
python -m pip install --no-deps gpu-burn

Install from a source checkout:

python -m pip install .

Usage

Run on all detected GPUs in the foreground at full utilization:

gpu-burn run

Select devices and target an approximate compute duty cycle:

gpu-burn run --devs 0,2-3 --util 60

Configure adaptive yielding with hysteresis. This pauses only the affected GPU when external compute processes use at least 5% of its total memory. Its worker restarts after usage remains at or below 2% for 5 continuous minutes:

gpu-burn run -d 0,2-3 -u 60 -P 5 -R 2 -i 5 -p 1

Run as a per-user background process:

gpu-burn start --gpus 2 --util 80
gpu-burn status
gpu-burn logs -f
gpu-burn stop

With no options, gpu-burn start selects every Paddle-visible GPU and uses:

• 16384x16384 float16 matrix multiplication
• 100% compute duty-cycle target
• pause at 5% external memory, resume at or below 2%
• 5 continuous idle minutes before restart
• 1-second monitoring interval

After the supervisor becomes ready, startup output includes the resolved GPU count, each device's running or paused state, worker PIDs, workload, adaptive policy, supervisor PID, and log path:

✓ gpu-burn is ready

gpu-burn 2.2.1 started
  supervisor : pid 42000
  devices    : 2 selected, 2 running, 0 paused
  gpu:0      : running (pid 42010)
  gpu:1      : running (pid 42011)
  workload   : 16384x16384 float16, util 100%
  adaptive   : pause >= 5%, resume <= 2%, idle 5 min, poll 1s
  log        : /tmp/gpu-burn-1000/gpu-burn.log

On an interactive terminal, the first line is an animated spinner while Paddle and NVIDIA devices are checked. Redirected output automatically uses plain text. Set GPU_BURN_NO_SPINNER=1 to disable animation explicitly.

The main workload options are:

--gpus and --devs are mutually exclusive. Lower utilization is implemented by alternating synchronized matrix multiplication with idle time, so the value observed by monitoring tools can vary with sampling intervals and GPU power-management behavior.

The supervisor queries nvidia-smi compute applications and excludes its own worker PIDs. Memory from other compute processes is summed per selected GPU and divided by that GPU's total physical memory. At --pause, only that GPU's worker is stopped, which releases its PaddlePaddle tensors and GPU memory. Once usage reaches --resume or lower, it must remain there for --idle minutes before a new worker starts. --resume must be lower than --pause; this hysteresis prevents repeated stop/start cycles near one threshold. Display-only contexts absent from NVIDIA's compute-app list are not considered. Numeric and UUID-based CUDA_VISIBLE_DEVICES remapping is honored, so CLI device IDs remain the same logical IDs that PaddlePaddle exposes.

State and cleanup

Only one managed instance is allowed per user. Runtime state and logs are stored under $GPU_BURN_STATE_DIR, $XDG_RUNTIME_DIR/gpu-burn, or /tmp/gpu-burn-$UID, in that order. gpu-burn stop --force sends SIGKILL if a worker does not stop before the configured timeout.

Development

The PEP 517 build requirement uses a compatible lower bound (setuptools>=77) instead of pinning one exact backend release. An exact pin makes builds byte-for-byte easier to reproduce, but also forces every source installer to download that precise release and can unnecessarily break offline or constrained environments. This pure-Python project does not need wheel in build-system.requires; setuptools implements the wheel build hook itself. Release CI pins the user-facing build and twine tools separately.

python -m pip install --no-deps -e .
python -m pytest
python -m build

Run the opt-in adaptive lifecycle test on a real GPU:

GPU_BURN_TEST_DEVICE=0 python tests/real_gpu_dynamic.py

License

MIT