vqgan-by-mzj 1.0.0

Mzj_With_VQGAN

VQGAN

CVPR 2021 (Oral)

Taming Transformers for High-Resolution Image Synthesis
Patrick Esser*, Robin Rombach*, Björn Ommer
* equal contribution

tl;dr We combine the efficiancy of convolutional approaches with the expressivity of transformers by introducing a convolutional VQGAN, which learns a codebook of context-rich visual parts, whose composition is modeled with an autoregressive transformer.

arXiv | BibTeX | Project Page

Requirements

A suitable conda environment named taming can be created and activated with:

conda env create -f environment.yaml
conda activate taming

Overview of pretrained models

The following table provides an overview of all models that are currently available. FID scores were evaluated using torch-fidelity. For reference, we also include a link to the recently released autoencoder of the DALL-E model. See the corresponding colab notebook for a comparison and discussion of reconstruction capabilities.

Dataset	FID vs train	FID vs val	Link	Samples (256x256)	Comments
FFHQ (f=16)	9.6	--	ffhq_transformer	ffhq_samples
CelebA-HQ (f=16)	10.2	--	celebahq_transformer	celebahq_samples
ADE20K (f=16)	--	35.5	ade20k_transformer	ade20k_samples.zip [2k]	evaluated on val split (2k images)
COCO-Stuff (f=16)	--	20.4	coco_transformer	coco_samples.zip [5k]	evaluated on val split (5k images)
ImageNet (cIN) (f=16)	15.98/15.78/6.59/5.88/5.20	--	cin_transformer	cin_samples	different decoding hyperparameters
FacesHQ (f=16)	--	--	faceshq_transformer
S-FLCKR (f=16)	--	--	sflckr
D-RIN (f=16)	--	--	drin_transformer
VQGAN ImageNet (f=16), 1024	10.54	7.94	vqgan_imagenet_f16_1024	reconstructions	Reconstruction-FIDs.
VQGAN ImageNet (f=16), 16384	7.41	4.98	vqgan_imagenet_f16_16384	reconstructions	Reconstruction-FIDs.
VQGAN OpenImages (f=8), 256	--	1.49	https://ommer-lab.com/files/latent-diffusion/vq-f8-n256.zip	---	Reconstruction-FIDs. Available via latent diffusion.
VQGAN OpenImages (f=8), 16384	--	1.14	https://ommer-lab.com/files/latent-diffusion/vq-f8.zip	---	Reconstruction-FIDs. Available via latent diffusion
VQGAN OpenImages (f=8), 8192, GumbelQuantization	3.24	1.49	vqgan_gumbel_f8	---	Reconstruction-FIDs.
DALL-E dVAE (f=8), 8192, GumbelQuantization	33.88	32.01	https://github.com/openai/DALL-E	reconstructions	Reconstruction-FIDs.

Data Preparation

ImageNet

The code will try to download (through Academic Torrents) and prepare ImageNet the first time it is used. However, since ImageNet is quite large, this requires a lot of disk space and time. If you already have ImageNet on your disk, you can speed things up by putting the data into ${XDG_CACHE}/autoencoders/data/ILSVRC2012_{split}/data/ (which defaults to ~/.cache/autoencoders/data/ILSVRC2012_{split}/data/), where {split} is one of train/validation. It should have the following structure:

${XDG_CACHE}/autoencoders/data/ILSVRC2012_{split}/data/
├── n01440764
│   ├── n01440764_10026.JPEG
│   ├── n01440764_10027.JPEG
│   ├── ...
├── n01443537
│   ├── n01443537_10007.JPEG
│   ├── n01443537_10014.JPEG
│   ├── ...
├── ...

Other

• A video summary by Two Minute Papers.
• A video summary by Gradient Dude.

Text prompt: 'A bird drawn by a child'

BibTeX

@misc{esser2020taming,
      title={Taming Transformers for High-Resolution Image Synthesis}, 
      author={Patrick Esser and Robin Rombach and Björn Ommer},
      year={2020},
      eprint={2012.09841},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}