stablefused 0.1.4

StableFused is a toy library to experiment with Stable Diffusion inspired by 🤗 diffusers and various other sources!

StableFused

StableFused is a toy library to experiment with Stable Diffusion inspired by 🤗 diffusers and various other sources!

Installation

It is recommended to use a virtual environment. You can use venv or conda to create one.

python -m venv venv

For usage, install the package from PyPI.

pip install stablefused

For development, fork the repository, clone it and install the package in editable mode.

git clone https://github.com/<YOUR_USERNAME>/stablefused.git
cd stablefused
pip install -e ".[dev]"

Usage

Checkout the examples folder for notebooks 🥰

Contributing

Contributions are welcome! Note that this project is not a serious implementation for training/inference/fine-tuning diffusion models. It is a toy library. I am working on it for fun and experimentation purposes (and because I'm too stupid to modify large codebases and understand what's going on).

As I'm not an expert in this field, I will have probably made a lot of mistakes. If you find any, please open an issue or a PR. I'll be happy to learn from you!

Acknowledgements

The following sources have been very helpful in helping me understand Stable Diffusion. I highly recommend you to check them out!

• 🤗 diffusers
• Karpathy's gist on latent walking
• Nateraw's stable-diffusion-videos
• 🤗 Annotated Diffusion Blog
• Keras CV
• Lillian Weng's Blogs
• Emilio Dorigatti's Blogs
• The AI Summer Diffusion Models Blog

Results

Visualization of diffusion process

Refer to the notebooks for more details and enjoy the denoising process!

Text to Image

These results are generated using the Text to Image notebook.

Your browser does not support the video tag.

Image to Image

These results are generated using the Image to Image notebook.

Source Image	Denoising Diffusion Process
The Renaissance Astronaut	High quality and colorful photo of Robert J Oppenheimer, father of the atomic bomb, in a spacesuit, galaxy in the background, universe, octane render, realistic, 8k, bright colors	Stylistic photorealisic photo of Margot Robbie, playing the role of astronaut, pretty, beautiful, high contrast, high quality, galaxies, intricate detail, colorful, 8k
	Your browser does not support the video tag.

PS

The results from Image to Image Diffusion don't seem very great from my experimentation. It might be some kind of bug in my implementation, which I'll have to look into later...

Understanding the effect of Guidance Scale

Guidance scale is a value inspired by the paper Classifier-Free Diffusion Guidance. The explanation of how CFG works is out-of-scope here, but there are many online sources where you can read about it (linked below).

• Guidance: a cheat for diffusion models
• Diffusion Models, DDPMs, DDIMs and CFG
• Classifier-Free Guidance Scale

In short, guidance scale is a value that controls the amount of "guidance" used in the diffusion process. That is, the higher the value, the more closely the diffusion process follows the prompt. A lower guidance scale allows the model to be more creative, and work slightly different from the exact prompt. After a certain threshold maximum value, the results start to get worse, blurry and noisy.

Guidance scale values, in practice, are usually in the range 6-15, and the default value of 7.5 is used in many inference implementations. However, manipulating it can lead to some very interesting results. It also only makes sense when it is set to 1.0 or higher, which is why many implementations use a minimum value of 1.0.

But... what happens when we set guidance scale to 0? Or negative? Let's find out!

When you use a negative value for the guidance scale, the model will try to generate images that are the opposite of what you specify in the prompt. For example, if you prompt the model to generate an image of an astronaut, and you use a negative guidance scale, the model will try to generate an image of everything but an astronaut. This can be a fun way to generate creative and unexpected images (sometimes NSFW or absolute horrendous stuff, if you are not using a safety-checker model - which is the case with StableFused).

Results

The original images produced are too large to display in high quality here. You can find them in my Drive. These images are compressed from ~30 MB to ~6 MB in order for GitHub to accept uploads.

Effect of Guidance Scale on Different Prompts

Effect of Guidance Scale on Different Prompts
Each row has first column set to the guidance_scale value. Each column has the same prompt. Column 1: Artistic image, very detailed cute cat, cinematic lighting effect, cute, charming, fantasy art, digital painting, photorealistic Column 2: A lion in galaxies, spirals, nebulae, stars, smoke, iridescent, intricate detail, octane render, 8k Column 3: A grand city in the year 2100, atmospheric, hyper realistic, 8k, epic composition, cinematic, octane render Column 4: Starry Night, painting style of Vincent van Gogh, Oil paint on canvas, Landscape with a starry night sky, dreamy, peaceful

Effect of Guidance Scale with increased number of inference steps

Effect of Guidance Scale with increased number of inference steps
Each row has first column set to the guidance_scale value. Columns have number of inference steps set to 3, 6, 12, 20, 25. Prompt: Photorealistic illustration of a mystical alien creature, magnificent, strong, atomic, tyrannic, predator, unforgiving, full-body image