tglcourse 0.0.2

work-in-progress course

The Generative Landscape

Work has just started on this - for now, if you’re interested in these topics you might want to check out the previous course I ran on AI ART: AIAIART

If you want to be notified when the course goes ‘live’ you can sign up for an email at http://thegenerativelandscape.com/

If you want to support this effort, I now have a patreon: https://www.patreon.com/johnowhitaker

The material will show in github pages at https://johnowhitaker.github.io/tglcourse/ (and the http://thegenerativelandscape.com will redirect there once we launch). Hooray for the magic of nbdev.

Curriculum plan / TODO

Topics:

• Intro: The Generative Landscape
• PyTorch, AutoGrad and Gradient Descent
• Constructing ANNs, Loss Functions
• Representations: What do ANNs Learn? Style Transfer
• CLIP/CLOOB: Multiple Modalities in a Shared Latent Space. Use as a loss function
• Generative modelling: Noise -> Data (Auto-Encoders and VAEs)
• GANs 1 - GAN training, a simple DC-GAN
• GANs 2 - Conditioning, Improvements, Modern GANs, CLIP guidance
• Sequence Modelling 1 - Ideas, RNNs, Language Models as Representation Learners
• Sequence Modelling 2 - Transformers
• Sequence Modelling 3 - Everything is a sequence (VIT, VQGANs, Dalle-mini, Parti, flamingo, music generation, other applications)
• Diffusion Models 1 - A New Type of Generative Model
• Diffusion Models 2 - Conditioning, Guiding, Improvements
• Diffusion Models 3 - The Current DM Landscape, Using Diffusion Models Creatively (inpainting, guiding, animation)
• Spotlight: Audio
• Spotlight: Video

Extra Skills / Bonus Material:

• Datasets and Dataloaders Intro (also defines data util funcs for the rest of the course)
• Experiment Tracking (eg W&B)
• Pretrained networks and transfer learning
• Ethics in generative modelling
• Fine-tuning existing models
• Working with GPUs?
• Multi-GPU or TPU training?
• Sharing demos w/ Gradio
• Managing cloud machines
• Dataloaders deep dive, streaming data
• Navigating other codebases
• Version control and CI (+ NBDev)
• “Text inversion” (https://arxiv.org/pdf/2208.01618.pdf)
• “prompt to prompt editing” https://arxiv.org/pdf/2208.01626.pdf
• Lots of paper readings / sumaries

Projects:

• Train a GAN, explore hyperparameters
• Fine-tune a diffusion model on a custom dataset
• Create and share a final project, including a report and demo

Guest discussions: as many as we can :)

Paper explainers: as many as we can :)

Status

Lesson	Description	TODOs	Status
Lesson 1: PyTorch Basics	Intro to PT, tensor manipulation, images as tensors, LeastAverageImage exercise	Video, tidy	Rough Draft Done
Lesson 2: Optimization	Intro to GD, optimization examples exercise	Video, Optimize with gen+loss without explaining	WIP
Lesson 3: Building NNs	nn.Module, building blocks, CNNs	Video, project, export functions? optimization/training tricks?	WIP
Lesson 4: Learning Representations	What do networks learn, style transfer	Video, tidy, …	WIP
Lesson 5: CLIP	Contrastive loss, CLIP, as a loss function	everything (pull from AIAIART)	Not Started
Lesson 6: Generative Modelling	Intro to generative models, AEs and VAEs, issues	everything (pull from AIAIART)	Not Started
Lesson 7: GANs 1	Intro to GANs, DC GAN, Conditioning	everything	Not Started
Lesson 8: GANs 2	GAN training tricks, NOGAN, using modern GANs, VQGAN	everything	Not Started
Lesson 9: Sequence Modelling Intro	idea, language modelling concept, transformer demo	everything	Not Started
Lesson 10: Transformers	Intro to transformes, attention, from scratch or dissect existing ones? ViT?	everything, check Niels’ repo and other resources	Not Started
Lesson 11: Everythign is a sequence	Show whistlegen, protein, VQGAN, parti…	everything	Not Started
Lesson 12: DM 1	Intro to diffusion models, toy example, comparison to DDPM	Video, tidy, second half, logs of my version training, better unet replacement	Rough Draft
Lesson 13: DM2	Conditioning, CFG, guiding, sampling, better training	everything	Not Started
Lesson 14: DM3	SD deep dive	combine grokking 1 and 2, polish	Rough Draft Done
Lesson 15: DM4	Other modalities - ideally demo class-conditioned audio generation	wavenet style diffusion for birdcalls, everything	Not Started

General TODOs - Create a page for project explanations + figure out submission info. - Figure out what the discussions page (pages) looks like - Ethics page but also callouts in each lesson - Pick a cloud for cloud stuff (sponsor?) - Figure out plan for datasets - Figure out losses+generators notebook/library bit and include in the plan somewhere - Figure out where smaller ‘bonus’ things like paper readings or topical deep-dives go. Do we add a blog? - Make the website pretty - Generate a bunch of images - Get Dave to make video assets + logo - Streamline skipping tests / avoiding anything long-running during CI - Set up org on HF for sharing models - Set up W&B projects for sharing training runs/reports - Notebook to document the library