spandrel 0.4.0

Give your project support for a variety of PyTorch model architectures, including auto-detecting model architecture from just .pth files. spandrel gives you arch support.

Spandrel

Spandrel is a library for loading and running pre-trained PyTorch models. It automatically detects the model architecture and hyperparameters from model files, and provides a unified interface for running models.

After seeing many projects extract out chaiNNer's model support into their own projects, I decided to create this PyPi package for the architecture support and model loading functionality. I'm also hoping that by having a central package anyone can use, the community will be encouraged to help add support for more models.

This package does not yet have easy inference code, but porting that code is planned as well.

Installation

Spandrel is available through pip:

pip install spandrel

Basic Usage

While Spandrel supports different kinds of models, this is how you would run a super resolution model (e.g. ESRGAN, SwinIR, HAT, etc.):

from spandrel import ImageModelDescriptor, ModelLoader
import torch

# load a model from disk
model = ModelLoader().load_from_file(r"path/to/model.pth")

# make sure it's an image to image model
assert isinstance(model, ImageModelDescriptor)

# send it to the GPU and put it in inference mode
model.cuda().eval()

# use the model
def process(image: torch.Tensor) -> torch.Tensor:
    with torch.no_grad():
        return model(image)

Note that model is a ModelDescriptor object, which is a wrapper around the actual PyTorch model. This wrapper provides a unified interface for running models, and also contains metadata about the model. See ImageModelDescriptor for more details about the metadata contained and how to call the model.

NOTE: ImageModelDescriptor will NOT convert an image to a tensor for you. It is purely making the forward passes of these models more convenient to use, since the actual forward passes are not always as simple as image in/image out.

More architectures

If you are working on a non-commercial open-source project or a private project, you should use spandrel and spandrel_extra_arches to get everything spandrel has to offer. The spandrel package only contains architectures with permissive and public domain licenses (MIT, Apache 2.0, public domain), so it is fit for every use case. Architectures with restrictive licenses (e.g. non-commercial) are implemented in the spandrel_extra_arches package.

import spandrel
import spandrel_extra_arches

# add extra architectures before `ModelLoader` is used
spandrel_extra_arches.install()

# load a model from disk
model = spandrel.ModelLoader().load_from_file(r"path/to/model.pth")

... # use model

Supported File Types

Spandrel mainly supports loading .pth files for all supported architectures. This is what you will typically find from official repos and community trained models. However, Spandrel also supports loading TorchScript traced models (.pt), certain types of .ckpt files, and .safetensors files for any supported architecture saved in one of these formats.

Model Architecture Support

NOTE: By its very nature, Spandrel will never be able to support every model architecture. The goal is just to support as many as is realistically possible.

Spandrel currently supports a limited amount of network architectures. If the architecture you need is not supported, feel free to request it or try adding it.

Single Image Super Resolution

• ESRGAN (RRDBNet)
  • This includes regular ESRGAN, ESRGAN+, "new-arch ESRGAN" (RealSR, BSRGAN), and Real-ESRGAN
  • Models: Community ESRGAN | ESRGAN+ | BSRGAN | RealSR | Real-ESRGAN
• Real-ESRGAN Compact (SRVGGNet) | Models
• Swift-SRGAN | Models
• SwinIR | Models
• Swin2SR | Models
• HAT | Models
• Omni-SR | Models
• SRFormer (+) | Models
• DAT | Models
• FeMaSR (+) | Models
• GRL | Models
• DITN | Models
• MM-RealSR | Models
• SPAN | Models
• Real-CUGAN | Models, Pro Models
• CRAFT | Models
• SAFMN | Models, JPEG model
• RGT | RGT Models, RGT-S Models
• DCTLSA | Models
• ATD | Models
• AdaCode | Models
• DRCT
• PLKSR and RealPLKSR | Models
• SeemoRe | Models
• MoSR | Models

Face Restoration

• GFPGAN | 1.2, 1.3, 1.4
• RestoreFormer | Model
• CodeFormer (+) | Model

Inpainting

• LaMa | Model
• MAT (+) | Model

Denoising

• SCUNet | GAN Model | PSNR Model
• Uformer | Denoise SIDD Model | Deblur GoPro Model
• KBNet | Models
• NAFNet | Models
• Restormer (+) | Models
• FFTformer | Models
• M3SNet (+) | Models
• MPRNet (+) | Deblurring, Deraining, Denoising
• MIRNet2 (+) | Models (SR not supported)
• DnCNN, FDnCNN | Models
• DRUNet | Models
• IPT | Models

DeJPEG

• FBCNN | Models

Colorization

• DDColor (+) | Models

Dehazing

• MixDehazeNet | Models

Low-light Enhancement

• RetinexFormer | Models
• HVI-CIDNet | Models

(All architectures marked with a + are only part of spandrel_extra_arches.)

Security

Use .safetensors files for guaranteed security.

As you may know, loading .pth files poses a security risk due to python's pickle module being inherently unsafe and vulnerable to arbitrary code execution (ACE). To mitigate this, Spandrel only allows deserializing certain types of data. This helps to improve security, but it still doesn't fully solve the issue of ACE.

Used By

Here are some cool projects that use Spandrel:

• AUTOMATIC1111's SD WebUI
• ComfyUI
• InvokeAI
• chaiNNer
• imaginAIry
• dgenerate

License

This repo is bounded by the MIT license. However, the code of implemented architectures (everything inside an __arch/ directory) is bound by their original respective licenses (which are included in their respective __arch/ directories).