tglcourse 0.0.3

work-in-progress course

The Generative Landscape

This is still a work in progress, but should be ready some time in November. For now, if you’re interested in these topics you might want to check out the previous course I ran on AI ART: AIAIART.

If you want to be notified when the course goes ‘live’ you can sign up for an email at http://thegenerativelandscape.com/

If you want to support this effort, I now have a patreon: https://www.patreon.com/johnowhitaker

The material will show in github pages at https://johnowhitaker.github.io/tglcourse/ (and the http://thegenerativelandscape.com will redirect there once we launch). Hooray for the magic of nbdev.

The plan and TODOs

The idea is to have a core curriculum building up to an understanding of key generative modelling techniques, split into three rough sections. The first 5 lessons will cover basics of building NNs and crafting loss functions. Lessons 6-10 will introduce generative modelling, GANs and working with sequences using transformers. Lessons 11-15 will be a deep dive into diffusion models, and lesson 16 will wrap up and give sugestions for new directions to explore.

Alongside this will be a number of ‘bonus’ notebooks that don’t need to be completed but which augment the core content. Managing datasets, experiment tracking, sharing demos and so on. Some will augment specific lessons, some will add functionality to the library and some will just be standalone topics I think are cool. The latter category is likely to continue to grow even after the course launches :)

There will be three suggested projects (after lessons 5, 11 and 14) to mark milestones in the course, and the final lesson will also encourage you to do a larger project at the end. Still working on details for how we’ll share and encourage these student projects.

This table has a rough status on the main lessons. It looks more drastic than it is - for many of these I have content lurking in other places so even those marked ‘Not Started’ should come together a lot faster than if this were being done completely from sratch!

Lesson	Description	TODOs	Status
Lesson 1: PyTorch Basics	Intro to PT, tensor manipulation, images as tensors, LeastAverageImage exercise	Video, tidy	Rough Draft Done
Lesson 2: Optimization	Intro to GD, optimization examples exercise	Video, Optimize with gen+loss without explaining	Rough Draft Done
Lesson 3: Building NNs	nn.Module, building blocks, CNNs	Video, project, export functions? optimization/training tricks?	WIP
Lesson 4: Learning Representations	What do networks learn, style transfer	Video, tidy, …	WIP
Lesson 5: CLIP	Contrastive loss, CLIP, as a loss function	everything (pull from AIAIART)	Not Started
Lesson 6: Generative Modelling	Intro to generative models, AEs and VAEs, issues	everything (pull from AIAIART)	WIP
Lesson 7: GANs 1	Intro to GANs, DC GAN, Conditioning	everything	Not Started
Lesson 8: GANs 2	GAN training tricks, NOGAN, using modern GANs, VQGAN	everything	Not Started
Lesson 9: Sequence Modelling Intro	idea, language modelling concept, transformer demo	everything	Not Started
Lesson 10: Transformers	Intro to transformes, attention, from scratch or dissect existing ones? ViT?	everything, check Niels’ repo and other resources	Not Started
Lesson 11: Everythign is a sequence	Show whistlegen, protein, VQGAN, parti…	everything	Not Started
Lesson 12: DM 1	Intro to diffusion models, toy example, comparison to DDPM	Video, tidy, second half, logs of my version training, better unet replacement	Rough Draft Done
Lesson 13: DM2	Conditioning, CFG, guiding, sampling, better training	everything	WIP
Lesson 14: DM3	SD deep dive	link to video run-through or record smaller videos	Rough Draft Done
Lesson 15: DM4	Other modalities - ideally demo class-conditioned audio generation. Might not be done by the time the course launches but would be nice to have.	wavenet style diffusion for birdcalls, look at video stuff	Not Started
Lesson 16: Going Further	Finding your niche, exploring less common areas	TODO	Not Started

General TODOs - Create a page for project explanations + figure out submission info. - Figure out what the discussions page (pages) looks like - Ethics page but also callouts in each lesson - Pick a cloud for cloud stuff (sponsor?) - Figure out plan for datasets - Figure out losses+generators notebook/library bit and include in the plan somewhere DONE - Figure out where smaller ‘bonus’ things like paper readings or topical deep-dives go. Do we add a blog? - Make the website pretty - Generate a bunch of images - Get Dave to make video assets + logo - Streamline skipping tests / avoiding anything long-running during CI - Set up org on HF for sharing models - Set up W&B projects for sharing training runs/reports - Notebook to document the library

Planned Bonus Notebooks:

• Datasets and Dataloaders Intro (also defines data util funcs for the rest of the course)
• Experiment Tracking (eg W&B)
• Pretrained networks and transfer learning
• Ethics in generative modelling
• Fine-tuning existing models
• Working with GPUs? Multi-GPU or TPU training?
• Sharing demos w/ Gradio
• Managing cloud machines
• Dataloaders deep dive, streaming data
• Navigating other codebases
• Version control and CI (+ NBDev)
• “Text inversion” training (https://arxiv.org/pdf/2208.01618.pdf)
• “prompt to prompt editing” https://arxiv.org/pdf/2208.01626.pdf
• Lots of paper readings / sumaries
• Neural cellular automata
• Evolutionary algorithms

Projects:

• Create a custom generation pipeline by combining different generators and loss functions
• Train a GAN, explore hyperparameters
• Fine-tune a diffusion model on a custom dataset and share it with the community
• Create and share a final project, including a report and demo

Guest discussions: as many as we can :)

Paper explainers: as many as we can :)

Page stats: Total Hits: Page visitors: