pydiffuse

A python library for generating media with diffusion.

Development

The project uses uv. With it installed:

git clone git@github.com:samirelanduk/pydiffuse.git
cd pydiffuse
uv sync
pre-commit install

uv sync creates the virtualenv in .venv and installs the project with its dev dependencies from the lockfile. pre-commit install sets up the ruff check and format hooks, which CI also enforces.

Then, to run the checks:

uv run python -m unittest discover
uv run pyright
uv run pre-commit run --all-files

CLI

The library exposes most of its functionality as a Command-Line Interface.

pydiffuse <group> <command>

Commands are namespaced by group - currently clip, vae and noise.

CLIP

CLIP turns a text prompt into a set of vectors that represent its semantic meaning, which diffusion models use to condition image generation. The process is split into three commands - tokenising, embedding, and encoding - each of which writes its output to a file that the next one reads. See docs/clip.md for an explanation of what each stage does.

The embedding and encoding steps need model weights in safetensors format - typically a Stable Diffusion checkpoint, whose CLIP text encoder tensors are stored under cond_stage_model.

Tokenising

clip tokenize converts a prompt into the integer token IDs that CLIP uses.

pydiffuse clip tokenize "a photo of a lighthouse"

This writes two files. tokens.json is a list of lists of token IDs, where each inner list is a chunk of exactly 77 tokens - prompts longer than that are split over as many chunks as they need, and the last chunk is padded:

[[49406, 320, 1125, 539, 320, 13717, 49407, 49407, ...]]

mappings.json has the same shape, but pairs each token ID with the substring it came from, so you can see how the prompt was split up:

[[["<|startoftext|>", 49406], ["a</w>", 320], ["photo</w>", 1125], ...]]

Option	Default	Description
--tokens	tokens.json	Path to save the tokens JSON to.
--mappings	mappings.json	Path to save the mappings JSON to.
--tokenizer	bundled tokenizer	Directory containing a custom CLIP tokenizer.

The library bundles the standard CLIP tokenizer, so --tokenizer is only needed if your model was trained with a different vocabulary.

Embedding

clip embed looks up the vector for each token, and adds the vector for its position in the chunk. The result represents each token in isolation, with no context from the rest of the prompt.

pydiffuse clip embed tokens.json model.safetensors

This writes embedding.pt, a torch.save-d tensor of shape (chunks, 77, width), where width is the embedding width of the model (768 for Stable Diffusion 1.x). The model must contain tensors whose keys end in token_embedding.weight and position_embedding.weight, and it must use the same vocabulary as the tokenizer that produced the tokens.

Option	Default	Description
--embedding	embedding.pt	Path to save the embedding to.

Encoding

clip encode runs the embeddings through the CLIP transformer layers, so that each vector is adjusted by the tokens before it. This is the conditioning that gets fed to a diffusion model.

pydiffuse clip encode embedding.pt model.safetensors

This writes conditioning.pt, a tensor of the same shape as the embedding. The model must contain the attention, MLP and layer norm tensors for every encoder layer under cond_stage_model, plus the final layer norm - any number of layers is supported.

Option	Default	Description
--conditioning	conditioning.pt	Path to save the conditioning to.

VAE

A VAE compresses an image into a much smaller latent representation that diffusion happens in, and turns that latent back into an image afterwards. See docs/vae.md for an explanation of how the two networks work.

Both commands need model weights in safetensors format, whose VAE tensors are stored under first_stage_model.

Encoding

vae encode compresses an image into a latent.

pydiffuse vae encode photo.jpg model.safetensors

This writes latent.pt, a torch.save-d tensor of shape (1, channels, height / ratio, width / ratio), where ratio is the product of the strides of the model's downsampling convolutions. Images whose dimensions aren't a multiple of that ratio are centre-cropped to the nearest one that is.

Option	Default	Description
--latent	latent.pt	Path to save the latent to.

Decoding

vae decode expands a latent back into an image.

pydiffuse vae decode latent.pt model.safetensors

This writes image.jpg, at the latent's resolution multiplied by the same ratio. The decoder can produce values outside the range it was trained on, and those are clamped rather than wrapped.

Option	Default	Description
--image	image.jpg	Path to save the image to.

Noise

Diffusion works by learning to remove noise, so both training and sampling need a way of adding a known amount of noise to a latent, and a schedule of how much noise to use at each step. See docs/noise.md for how noise is added.

A noise level is a number between 0 and 1 giving the proportion of the result that is noise rather than signal, so 0 is a clean latent and 1 is pure noise.

Applying noise

noise apply adds noise to a tensor at a single noise level.

pydiffuse noise apply latent.pt 0.5

This writes noised.pt, a tensor of the same shape as the input. The tensor and the noise are each scaled by the square root of their share, so the result keeps a variance of 1 (assuming the input had a variance of 1 to start with). Fresh noise is drawn on every run, so the same inputs give a different result each time.

Option	Default	Description
--output	noised.pt	Path to save the noised tensor to.

Generating a schedule

noise schedule produces the sequence of noise levels a sampler steps through, from the noisiest level down to a clean image.

pydiffuse noise schedule 5

This writes schedule.txt, one noise level per line rounded to eight decimal places. There is always one more line than there are steps, because the schedule ends at 0:

0.9953399
0.95834503
0.61880525
0.05791837
0.00085
0.0

Two algorithms are available, both of which space the levels out by their ratio of noise to signal rather than by the level itself, which puts more steps where the latent is nearly clean. karras is the schedule from Karras et al. (2022), which spaces that ratio raised to the power 1/7. exponential spaces the log of the ratio, so each step reduces it by the same factor.

Option	Default	Description
--algorithm	karras	The algorithm to generate the schedule with - karras or exponential.
--output	schedule.txt	Path to save the schedule to.