A grid image generator for Stable Diffusion models based on 🧨diffusers.
Project description
Grate
Make a matrix of images by running the same prompt through multiple Stable Diffusion models.
Supports huggingface repo ids, local CompVis-style .ckpt
files, and paths to local folder hierarchies containing diffusers-format models. Uses DDPM++2 sampler with 15 steps for all generations.
The above image was rendered using the following commandline on an empty runpod instance:
grate --prompts \
"a cat playing with a ball in the forest" \
"a dog chasing a postman" \
"a fish singing the blues" \
--repo_ids_or_paths \
"stabilityai/stable-diffusion-2-1" \
"runwayml/stable-diffusion-v1-5" \
"nitrosocke/Arcane-Diffusion" \
--output_path ./grate-demo.png
Installation
python -m pip install sdgrate
Once it's installed, run grate -h
for help.
Merging
Supports numerous merge methods.
-
Basic merges between two models
a
andb
, usingweighted_sum
,sigmoid
, orinv_sigmoid
weighting, with a given alpha (alpha=0
means use only modela
, andalpha=1
means use only modelb
). For example:grate --prompts "a cat" "a dog" \ --repo_ids_or_paths stabilityai/stable-diffusion-2-1 IlluminatiAI/Illuminati_Diffusion_v1.0 \ --merge_alphas 0.2 0.5 0.8
This will produce a grid with 3 rows of 2 columns showing the prompts "a cat" and "a dog" rendered using merges between
stabilityai/stable-diffusion-2-1
andIlluminatiAI/Illuminati_Diffusion_v1.0
at alphas 0.2, 0.5, and 0.8. -
Three-way merge by adding the difference between models
b
andc
toa
(--merge_algorithm add_diff
). -
Merges using a different weight for the unet and text_encoder modules (
--merge_unet_alpha
and--merge_text_encoder_alpha
). -
Block-weighted merges, whereby a different weight can be used for different layers within the unet (
--merge_unet_block_weights
). Specify 12 weights for the down blocks (counting from the input layer), 1 weight for the middle block, and 12 weights for the up blocks (counting from the middle layer). An example is given below.You can find an explanation of block-weighted merging here (cw: waifus), and some weight presets to use here (illustrated here).
For each the advanced --merge_
arguments, you can specify either one value to apply to all rows, or one value per row, where the number of rows is determined by the number of alpha values passed for the --merge_alphas
argument. For example, to render a grid showing the effects of using the unet down blocks from stabilityai/stable-diffusion-2-1
and the unet up blocks from IlluminatiAI/Illuminati_Diffusion_v1.0
on the first row, and the effect of doing the opposite weighting on the second row, use this command:
grate --prompts "a cat" "a dog" \
--repo_ids_or_paths stabilityai/stable-diffusion-2-1 IlluminatiAI/Illuminati_Diffusion_v1.0 \
--merge_alphas 0.5 0.5 \
--merge_unet_block_weights "0,0,0,0,0,0,0,0,0,0,0,0,0.5,1,1,1,1,1,1,1,1,1,1,1,1" "1,1,1,1,1,1,1,1,1,1,1,1,0.5,0,0,0,0,0,0,0,0,0,0,0,0"
Note that the number of values passed to --merge_alphas
matches the number of strings passed to --merge_unet_block_weights
.
Full arguments list
Run python3 grate.py -h
for help:
$ grate -h
usage: grate [-h] --prompts PROMPTS [PROMPTS ...] --repo_ids_or_paths REPO_IDS_OR_PATHS [REPO_IDS_OR_PATHS ...] --output_path OUTPUT_PATH [--device DEVICE]
[--batch_size BATCH_SIZE] [--width WIDTH] [--height HEIGHT] [--negative_prompts NEGATIVE_PROMPTS [NEGATIVE_PROMPTS ...]] [--seeds SEEDS [SEEDS ...]]
[--cfg CFG] [--local_files_only] [--merge_alphas MERGE_ALPHAS [MERGE_ALPHAS ...]] [--merge_algorithm MERGE_ALGORITHM [MERGE_ALGORITHM ...]]
[--merge_unet_block_weights MERGE_UNET_BLOCK_WEIGHTS [MERGE_UNET_BLOCK_WEIGHTS ...]] [--merge_unet_alpha MERGE_UNET_ALPHA [MERGE_UNET_ALPHA ...]]
[--merge_text_encoder_alpha MERGE_TEXT_ENCODER_ALPHA [MERGE_TEXT_ENCODER_ALPHA ...]]
Generates a grid of images by running a set of prompts through different Stable Diffusion models.
options:
-h, --help show this help message and exit
--prompts PROMPTS [PROMPTS ...]
EITHER: a path to a JSON file containing prompt and negative prompt pairs eg [{'prompt': 'a fish', 'negative_prompt': 'distorted', 'seed':
123}, ...]. OR: multiple strings enclosed in "" and separated by spaces. eg --prompts "a cat" "a dog" "a fish"
--repo_ids_or_paths REPO_IDS_OR_PATHS [REPO_IDS_OR_PATHS ...]
repository ids or paths to models in diffusers or ckpt format, as strings enclosed in "" and separated by spaces. eg --repo_ids_or_paths
"stablityai/stable-diffusion-2-1" "../models/v1-5-pruned-emaonly.ckpt"
--output_path OUTPUT_PATH
Where to save the resulting image. Also saves partial images here as each row is finished rendering.
--device DEVICE (Optional) Device to use, eg 'cuda', 'mps', 'cpu'. if omitted, will try to pick the best device.
--batch_size BATCH_SIZE
(Optional) Batch size, default 1
--width WIDTH (Optional) Individual image width
--height HEIGHT (Optional) Individual image height
--negative_prompts NEGATIVE_PROMPTS [NEGATIVE_PROMPTS ...]
(Optional) Negative prompts. Must be either 1 string to use for all prompts, or one string for each prompt passed to --prompts.
--seeds SEEDS [SEEDS ...]
(Optional) Seeds. Must be either 1 int to use for all prompts, or exactly 1 int per prompt.
--cfg CFG (Optional) CFG scale (default=7.5).
--local_files_only (Optional) Use only local data (do not attempt to download or update models)
--merge_alphas MERGE_ALPHAS [MERGE_ALPHAS ...]
(Optional) If set, --repo_ids_or_paths must specify either 2 or 3 models, which will be merged using the given alphas and used in place of
multiple models.
--merge_algorithm MERGE_ALGORITHM [MERGE_ALGORITHM ...]
(Optional, default is weighted_sum) If doing merges, the algorithm to use - one of 'weighted_sum', 'sigmoid', 'inv_sigmoid', or 'add_diff'. 'add_diff' only works for
3-way merge. Specify either 1 algorithm to use for all rows, or the same number of algorithms as there are merge_alphas.
--merge_unet_block_weights MERGE_UNET_BLOCK_WEIGHTS [MERGE_UNET_BLOCK_WEIGHTS ...]
(Optional) 25 comma-separated floats specified as strings, eg "0.0, 0.0, 0.0, (... 22 more floats)", to merge each part of the UNet using a
different weight ('block-weighted merging'). Specify either 1 list to use for all rows, or the same number of lists as there are
merge_alphas.
--merge_unet_alpha MERGE_UNET_ALPHA [MERGE_UNET_ALPHA ...]
(Optional) Override the merge alpha with a unet-specific alpha. Specify either 1 alpha to use for all rows, or the same number of alphas as
there are merge_alphas.
--merge_text_encoder_alpha MERGE_TEXT_ENCODER_ALPHA [MERGE_TEXT_ENCODER_ALPHA ...]
(Optional) Override the merge alpha with a text-encoder-specific alpha. Specify either 1 alpha to use for all rows, or the same number of
alphas as there are merge_alphas.
Enjoy!
Using as a library
The main sdgrate.grate
module includes the following functions, which may be useful: merge_models
, render_row
, render_all
.
The model merger is implemented as a custom pipeline based on a modified version of the (checkpoint_merger pipeline)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.