comfylib 0.0.3

Stable Diffusion library, based on the implementation in ComfyUI

This is a work in progress

All the talk about having a reliable interface below is aspirational. The interface is in flux at the moment, and no guarantee of the master branch working are made, yet. Soon.

This was initially planned to be a fork of ComfyUI, with regular merges from upstream. However, we are quickly realizing that's impractical, given the rapid development in the ComfyUI codebase. So, this is a hard fork. We will make a strong effort to keep parity, but we will also be going our own way here and the implementation will probably end up quite different.

---

This is the ComfyUI, but without the UI. It's stripped down and packaged as a library, for use in other projects.

ComfyUI is actively maintained (as of writing), and has implementations of a lot of the cool cutting-edge Stable Diffusion stuff.

In order to provide a consistent API, an interface layer has been added. Directly importing names not in the API should be considered dangerous. A best effort will be made to keep this library apace with the implementation in ComfyUI (though this will get harder over time as the implemntations diverge) so the backend implementation might change drastically between minor versions.

The interface layer will be consistent within a major version of the library, so that's what you should rely on.

Design goals

1. The API should expose the same breadth of functionality available by using the node editor in ComfyUI.
2. Opaque types should be preferred. Rather than pass tensors around, we're going to wrap them in objects that hide the implementation. This gives us maximum flexibility to keep the API the same, while also incorporating new developments.
3. Explicit behavior should be prferred over implicit behavior. As a library, we shouldn't make assumptions about how the user wants to, for example, sanitize inputs or manage VRAM. At the cost of requiring a bit more work to use, we should raise exceptions when we get bad input, offer an interface for moving things to and from VRAM, etc.
4. The API should be should be typed as strictly as possible. Enums should be used instead of strings, when applicable, etc.
5. The interface layer should have complete test coverage.

Installation

You can install from github:

pip3 install git+https://github.com/adodge/ComfyLib

You may also be able to install from PyPi, but that process will be clarified when this project is more stable.

Example

import comfy.stable_diffusion
import comfy.latent_image

config = comfy.stable_diffusion.CheckpointConfig.from_built_in(comfy.stable_diffusion.BuiltInCheckpointConfigName.V1)

# Read in a checkpoint
sd, clip, vae = comfy.stable_diffusion.load_checkpoint(
    config=config,
    checkpoint_filepath="v1-5-pruned-emaonly.safetensors",
    embedding_directory=None,
)

# CLIP encode a prompt
pos = clip.encode("an astronaut")
neg = clip.encode("")

# Run the sampler
latent0 = comfy.latent_image.LatentImage.empty(512, 512)
latent1 = sd.sample(positive=pos, negative=neg, latent_image=latent0, seed=42, steps=20, cfg_strength=7,
                 sampler=comfy.stable_diffusion.Sampler.SAMPLE_EULER, scheduler=comfy.stable_diffusion.Scheduler.NORMAL,
                 denoise_strength=1.0)

# Run the VAE to get a Pillow Image
image = vae.decode(latent_image=latent1)

# Save that to a file
image.save("out.png")

API

Models

StableDiffusionModel

CLIPModel

VAEModel

Data

RGBImage

GreyscaleImage

LatentImage

Conditioning