ECE 598 - Generative AI Models (Spring 2022)

Instructor: Lav Varshney (office hours, TBD)

Teaching Assistant: Jason Leung (office hours, TBD)

Lectures: Tuesdays and Thursdays, 11:00am, 1015 Electrical and Computer Engineering Building (also streamed on Echo 360)

Catalog Description: Generative models are widely used in many branches of artificial intelligence. This course covers mathematical and computational foundations of generative modeling, as well as applications in engineering, design, science, and the arts. Specific topics include variational autoencoders, generative adversarial networks, autoregressive models such as Transformers, normalizing flow models, information lattice learning, neural text decoding, prompt programming, and detection of generated content. Explainability and social responsibility in generative modeling will also be discussed, including topics of justice, human autonomy, and safety.

Prerequisites: ECE 544

Textbook: I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016, as well as many further readings and lecture notes.

Grading: scribing + implementation assignment (20%), midterm exam [take-home] (15%), final exam [take-home] (15%), group project (20%), individual project (20%), social responsibility essay (10%)

Syllabus

Scribing + Implementation + Theory Questions

Exams

Individual Project

Group Project

Social Responsibility Essay

Course Schedule

 

Date Topic Optional Readings (Besides Relevant Sections of Textbook and Links/References in Lecture Slides) Learning Objectives Scribe Assignments
1/18

1. Introduction to generative AI

[slides]
  • L. R. Varshney, F. Pinel, K. R. Varshney, D. Bhattacharjya, A. Schoergendorfer, and Y.-M. Chee, “A Big Data Approach to Computational Creativity: The Curious Case of Chef Watson,” IBM Journal of Research and Development, vol. 63, no. 1, pp. 7:1-7:18, Jan./Feb. 2019.
  • X. Ge, R. T. Goodwin, J. R. Gregory, R. E. Kirchain, J. Maria, and L. R. Varshney,” Accelerated Discovery of Sustainable Building Materials,” in Proceedings of the AAAI Spring Symposium, March 2019.
  •  
1/25

2. Background on generative AI

[slides]
  • A. Papoulis and S. U. Pillai, Probability, Random Variables, and Stochastic Processes, McGraw-Hill, 2002.
  • B. Hajek, Probability with Engineering Applications, ECE 313 Course Notes, Aug. 2021.
  • A. Orlitsky, N. P. Santhanam, and J. Zhang, "Always Good Turing: Asymptotically Optimal Probability Estimation," Science, vol. 302, no. 5644, pp. 427-431, Oct. 2003.
  • I. J. Good, "The Population Frequencies of Species and the Estimation of Population Parameters," Biometrika, vol. 40, no. 3/4, pp. 237-264, Dec. 1953.
  •  
1/27

3. Normalizing flows

[slides]
  • D. P. Kingma and M. Welling, "An Introduction to Variational Autoencoders," Foundations and Trends in Machine Learning, vol. 12, no. 4, pp. 307-392, 2019.
  • D. P. Kingma and P. Dhariwal, "Glow: Generative Flow with Invertible 1x1 Convolutions," arXiv:1807.03039 [stat.ML], Jul. 2018.
  • R. Theisen, H. Wang, L. R. Varshney, C. Xiong, and R. Socher, “Evaluating State-of-the-Art Classification Models Against Bayes Optimality,” in Proceedings of the Thirty-Fifth Conference on Neural Information Processing Systems (NeurIPS), Dec. 2021.
  •  
2/1

4. Variational autoencoders I

[handwritten notes]
  • D. P. Kingma and M. Welling, "An Introduction to Variational Autoencoders," Foundations and Trends in Machine Learning, vol. 12, no. 4, pp. 307-392, 2019.
  •  
2/3

5. Variational autoencoders II

[slides]
  • D. P. Kingma and M. Welling, "An Introduction to Variational Autoencoders," Foundations and Trends in Machine Learning, vol. 12, no. 4, pp. 307-392, 2019.
  • S. Zhao, J. Song, and S. Ermon, "InfoVAE: Balancing Learning and Inference in Variational Autoencoders," in Proceedings of the AAAI Conference on Artificial Intelligence, pp. 5885-5892, 2019.
  
2/8

6. Variational autoencoders III

[slides]
  • D. P. Kingma and M. Welling, "An Introduction to Variational Autoencoders," Foundations and Trends in Machine Learning, vol. 12, no. 4, pp. 307-392, 2019.
  • A. van den Oord, O. Vinyals, and K. Kavukcuoglu, "Neural Discrete Representation Learning," in Proceedings of the 30th Conference on Neural Information Processing Systems, Dec. 2017.
  
2/10

7. Generative adversarial networks I

[slides]
  • I. Goodfellow, "NIPS 2016 Tutorial: Generative Adversarial Networks," arXiv 1701.00160 [cs.LG], Dec. 2016.
  
2/15

8. Generative adversarial networks II

[slides]
  • I. Goodfellow, "NIPS 2016 Tutorial: Generative Adversarial Networks," arXiv 1701.00160 [cs.LG], Dec. 2016.
  • M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein Generative Adversarial Networks," in Proceedings of the International Conference on Machine Learning (ICML), 2017. 
  
2/17

9. Generative adversarial networks III

[slides]
  • I. Goodfellow, "NIPS 2016 Tutorial: Generative Adversarial Networks," arXiv 1701.00160 [cs.LG], Dec. 2016.
  • M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein Generative Adversarial Networks," in Proceedings of the International Conference on Machine Learning (ICML), 2017. 
  • J. Thickstun, "Kantorovich-Rubinstein Duality," unpublished.
  
2/22

10. Autoregressive models I

[slides]
  • J. M. Tomcsak, Deep Generative Modeling, Springer, 2022.
  
2/24

11. Autoregressive models II

[slides]
  • J. M. Tomcsak, Deep Generative Modeling, Springer, 2022.
  • A. Papoulis and S. U. Pillai, Probability, Random Variables, and Stochastic Processes, McGraw-Hill, 2002.
  
3/1

12. Transformers I

[slides]
  • A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention Is All You Need," in Proceedings of the Thirty-First Conference on Neural Information Processing Systems (NeurIPS), 2017.
  • N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, and R. Socher, “CTRL: A Conditional Transformer Language Model for Controllable Generation,” arXiv:1909.05858 [cs.CL].
  • K. Lu, A. Grover, P. Abbeel, and I. Mordatch, "Pretrained Transformers as Universal Computation Engines," arXiv:2103.05247 [cs.LG], March 2021.
  
3/3

13. Transformers II

[slides]
  • C. Yun, S. Bhojanapalli, A. S. Rawat, S. J. Reddi, and S. Kumar, "Are Transformers universal approximators of sequence-to-sequence functions?," in Proceedings of the International Conference on Learning Representations (ICLR), 2020.
  • J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, "Scaling Laws for Neural Language Models," arXiv:2001.08361 [cs.LG], Jan. 2020.
  
3/8

14. Prompt programming and neural text decoding

[slides]
  • L. Reynolds and K. McDonell, "Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm," in Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, May 2021.
  • B. Lester, R. Al-Rfou, and N. Constant, "The Power of Scale for Parameter-Efficient Prompt Tuning," in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045–3059, Nov. 2021. 
  • S. Basu, G. S. Ramachandran, N. S. Keskar, and L. R. Varshney, “Mirostat: A Neural Text Decoding Algorithm That Directly Controls Perplexity,” in Proceedings of the 9th International Conference on Learning Representations (ICLR),  May 2021.
    
3/10 15. MIDTERM EXAM      
3/15 SPRING BREAK      
3/17 SPRING BREAK      
3/22 16. Information lattice learning I      
3/24 17. Information lattice learning II      
3/29

18. Neural cellular automata

[slides]

      
3/31

19. Detection of generated content I

[slides]

      
4/5

20. Detection of generated content II

[slides]

      
4/7

21. Applications in climate science

[slides]

      
4/12

22. Applications in engineering and industrial design

[slides]

      
4/14

23. Applications in music and visual art

[slides]

      
4/19

24. Explainability I

[slides]

    
4/21

25. Explainability II

[slides]

      
4/26 26. Applications in molecule design      
4/28. 27. Social Responsibility I      
5/3 28. Social Responsibility II