torchruntime

torchruntime is a lightweight package for automatically installing the appropriate variant of PyTorch on a user's computer, based on their OS, and GPU manufacturer and GPU model.

This package is used by Easy Diffusion, but you're welcome to use it as well. It's useful for developers who make PyTorch-based apps that target users with NVIDIA, AMD and Intel graphics cards (as well as CPU-only usage), on Windows, Mac and Linux.

• Platforms: cpu, cuda, rocm, xpu, directml, ipex
• Operating systems: Windows, Linux, Mac (Apple Silicon and Intel)
• GPU Manufacturers: NVIDIA, AMD, Intel
• GPU Types: Dedicated and Integrated

Why?

It lets you treat PyTorch as a single dependency (like it should be), and lets you assume that each user will get the most-performant variant of PyTorch suitable for their computer's OS and hardware.

It deals with the complexity of the variety of torch builds and configurations required for CUDA, AMD (ROCm, DirectML), Intel (xpu/DirectML/ipex), and CPU-only.

Compatibility table: Click here to see the supported graphics cards and operating systems.

Installation

Supports Windows, Linux, and Mac.

pip install torchruntime

Usage

Step 1. Install the appropriate variant of PyTorch

This command should be run on the user's computer, or while creating platform-specific builds:

python -m torchruntime install

This will install torch, torchvision, and torchaudio, and will decide the variant based on the user's OS, GPU manufacturer and GPU model number. See customizing packages for more options.

Step 2. Configure torch

This should be run inside your program, to initialize the required environment variables (if any) for the variant of torch being used.

import torchruntime

torchruntime.configure()

(Optional) Step 3. Test torch

Run python -m torchruntime test to run a set of tests to check whether the installed version of torch is working correctly.

Customizing packages

By default, python -m torchruntime install will install the latest available torch, torchvision and torchaudio suitable on the user's platform.

You can customize the packages to install by including their names:

• For e.g. to install only torch and torchvision, you can run python -m torchruntime install torch torchvision
• To install specific versions (in pip format), you can run python -m torchruntime install "torch>2.0" "torchvision==0.20"

Supported torch packages: torch, torchvision, torchaudio, torchao.

Note: If you specify package versions, please keep in mind that the version may not be available to all the users on all the torch platforms. For e.g. a user with Python 3.8 would not be able to install torch 2.5 (or higher), because torch 2.5 dropped support for Python 3.8.

So in general, it's better to avoid specifying a version unless it really matters to you (or you know what you're doing). Instead, please allow torchruntime to pick the latest-possible version for the user.

Versioning scheme

torchruntime uses semantic versioning. Versions will follow the major.minor.patch pattern, e.g. 1.20.3.

• major version change: for breaking code changes, e.g. API changes. E.g. 1.0 to 2.0.
• minor version change: for automatic PCI database updates, e.g. support for new graphics cards. E.g. 1.1 to 1.2.
• minor version change: for non-breaking code changes, e.g. backwards-compatible new functionality, routine maintenance, refactoring. E.g. 1.1 to 1.2.
• patch version change: for backward compatible bug fixes. E.g. 1.1.1 to 1.1.2.

It is recommended that you rely on the minor version, for e.g. use torchruntime ~= 1 in requirements.txt (change this to the current major version), which will install versions like 1.21.0, 1.22.2 etc but not 2.0.0.

Compatibility table

The list of platforms on which torchruntime can install a working variant of PyTorch.

Note: This list is based on user feedback (since I don't have all the cards). Please let me know if your card is supported (or not) by opening a pull request or issue or messaging on Discord (with supporting logs).

CPU-only

NVIDIA

Datacenter: Supports all 2xx and 1xx series GPUs after Kepler (e.g. H200, B200, H100 etc).

Note: Torch dropped support for Python 3.8 from torch >= 2.5. torchruntime falls back to CUDA 12.4, if python 3.8 is being used.

AMD

Discrete GPUs

Integrated GPUs (APU)

Apple

Intel

API

See API for a complete list of module functions.

FAQ

Why can't I just run 'pip install torch'?

pip install torch installs the CPU-only version of torch, so it won't utilize your GPU's capabilities.

Why can't I just install torch-for-ROCm directly to support AMD?

Different models of AMD cards require different LLVM targets, and sometimes different ROCm versions. And ROCm currently doesn't work on Windows, so AMD on Windows is best served (currently) with DirectML.

And plenty of AMD cards work with ROCm (even when they aren't in the official list of supported cards). Information about these cards (for e.g. the LLVM target to use) is pretty scattered.

torchruntime deals with this complexity for your convenience.

Contributing

📢 I'm looking for contributions in these specific areas:

• More testing on consumer AMD GPUs.
• More support for older AMD GPUs. Explore: Compile and host PyTorch wheels and rocm (on GitHub) for older AMD gpus (e.g. 580/590/Polaris) with the required patches.
• Intel GPUs.
• Testing on professional AMD GPUs (e.g. the Instinct series).
• An easy-to-run benchmark script (that people can run to check the level of compatibility on their platform).
• Improve the logic for supporting multiple AMD GPUs with different ROCm compatibility. At present, it just picks the latest GPU, which means it doesn't support running workloads on multiple AMD GPUs in parallel.

Please message on the Discord community if you have AMD or Intel GPUs, and would like to help with testing or adding support for them! Thanks!

Credits

• Code contributors on Easy Diffusion.
• Users on Easy Diffusion's Discord who've helped with testing on various GPUs.
• PCI Database automatically generated from the PCI ID Database at http://pci-ids.ucw.cz

More resources

• AMD Discrete GPU Names and AMD Integrated GPU Names
• AMD HSA architectures for APUs, and also the official ROCm list (check the latest).
• AMD GPU LLVM Architectures
• Status of ROCm support for AMD Navi 1
• Torch support for ROCm 6.2 on AMD Navi 1
• ROCmLibs-for-gfx1103-AMD780M-APU
• Pre-compiled torch for AMD gfx803 (and steps to compile)
• Another guide for compiling torch with rocm 6.2 for gfx803
• ROCm's device list (in their code)