• Arindam Banerjee: Tue, Thu 3:00 - 4:00 pm
  
• Jiaqi Guan: Wed, Fri 3:00- 4:00 pm
  

Class Schedule

	Date	Topic+Papers	Slides	Scribe Notes
01.	Tue, Jan 17	Course Overview
02.	Thu, Jan 19	RBM 1: Energy Based Models, (Restricted) Boltzmann Machines (RBMs), Sigmoid Belief Networks 2007 (Notes) Geoffrey Hinton, Boltzmann Machines, Scholarpedia, 2007. 2022 (arXiv) Benyamin Ghojogh, et al., Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey, arXiv, 2022. 1990 (TR) Radford Neal, Learning Stochastic Feedforward Networks, Tech Report, 1990.	Boltzmann Machines
03.	Tue, Jan 24	RBM 2: Learning in (Restricted) Boltzmann Machines, Deep Boltzmann Machines, Contrastive Divergence 2005 (AISTATS) Miguel Carreira-Perpignan and Geoffrey Hinton, On contrastive divergence learning, AISTATS, 2005. 2012 (TR) Geoffrey Hinton, A Practical Guide to Training Restricted Boltzmann Machines, Univ of Toronto Tech Report, 2012. 2012 (NC) Ruslan Salakhutdinov and Geoffrey Hinton, An Efficient Learning Procedure for Deep Boltzmann Machines, Neural Computation, 2012. 2010 (AISTATS) Ruslan Salakhutdinov and Hugo Larochelle, Efficient Learning of Deep Boltzmann Machines, AISTATS, 2010.	Learning (R)BMs
04.	Thu, Jan 26	Inf 1: Markov Chain Monte Carlo, Gibbs Sampling, Langevin Dynamics 2003 (MLJ) Christophe Andreieu, et al., An Introduction to MCMC for Machine Learning, MLJ, 2003. 1996 (Bernoulli) Gareth Roberts and Richard Tweedie, Exponential convergence of Langevin distributions and their discrete approximations, Bernoulli, 1996.	Sampling, Inference
05.	Tue, Jan 31	Inf 2: Hamiltonian Monte Carlo, Variational Inference 2012 (Book Chapter) Radford Neal, MCMC using Hamiltonian Dynamics, Chapter 5, Handbook of MCMC. 2018 (arXiv) Michael Betancourt, A Conceptual Introduction to Hamiltonian Monte Carlo, arXiv, 2018.	(included above)
06.	Thu, Feb 2	AR 1: Neural Autoregressive Models 2011 (AISTATS): H. Larochelle, I. Murray, The Neural Autoregressive Distribution Estimator, AISTATS, 2011. 2015 (ICML): M. Germain, K. Gregor, I. Murray, H. Larochelle, Made: Masked autoencoder for distribution estimation, ICML, 2015.	NADE, MADE
07.	Tue, Feb 7	AR 2: PixelRNN/CNN, Quantiles 2016 (ICML): A. van den Oord, N. Kalchbrenner, K. Kavukcuoglu, Pixel recurrent neural networks, ICML, 2016. 2016 (NeurIPS): A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, K. Kavukcuoglu, Conditional image generation with PixelCNN decoders, NeurIPS, 2016. 2018 (ICML): G. Ostrovski, W. Dabney, R. Munos, Autoregressive Quantile Networks for Generative Modeling, ICML, 2018.	PixelRNN, CNN Quantiles	AR2 notes
08.	Thu, Feb 9	AR 3: Transformers 2017 (NeurIPS): A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, NeurIPS, 2017. 2019 (arXiv): R. Child, S. Gray, A. Radford, I. Sutskever, Generating Long Sequences with Sparse Transformers, arXiv, 2019. 2021 (ICLR): A. Dosovitskiy, et al.: An Image is worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR, 2021	Transformers	AR3 notes
09.	Tue, Feb 14	VAE 1: Variational Autoencoders, Evidence Lower Bound, Importance Weighting 2014 (ICLR): D. Kingma, M. Welling, Auto-Encoding Variational Bayes, ICLR, 2014. 2014 (ICML): D. Rezende, S. Mohamed, D. Wierstra, Stochastic Backpropagation and ApproximateIInference in Deep Generative Models, ICML, 2014. 2019 (FTML): D. Kingma and M. Welling, An Introduction to Variational Autoencoders, FTML, 2019. 2016 (ICLR): Y. Burda, R. Grosse, R. Salakhutdinov. Importance Weighted Autoencoders. ICLR, 2016.	VAE basics	VAE1 notes 1 VAE1 notes 2
10.	Thu, Feb 16	VAE 2: Improved Inference, Information Theoretic Perspective, Representation 2018 (ICML): T. Rainforth, A. Kosiorek, T. Le, C. Maddison, M. Igl, F. Wood, Y. Teh, Tighter Variational Bounds are Not Necessarily Better. ICML, 2018. 2018 (ICML): A. Alemi, B. Poole, I. Fischer, J. Dillon, R. Saurous, K. Murphy. Fixing a broken ELBO. ICML, 2018. 2019 (ICLR): B. Dai, D. Wipf. Diagnosing and Enhancing VAE Models. ICLR, 2019.	ELBO, Rate Distortion Diagnosing VAEs	VAE2 notes 1 VAE2 notes 2
11.	Tue, Feb 21	VAE 3: VQ VAE, Deep VAE 2017 (NeurIPS): A. van den Oord, O. Vinyals, K. Kavukcuoglu. Neural Discrete Representation Learning. NeurIPS, 2017. 2016 (NeurIPS): C. Sonderby, T. Raiko, L. Maaloe, S. Sonderby, O. Winther. Ladder Variational Autoencoders. NeurIPS, 2016. 2021 (ICLR): R. Child. Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images. ICLR, 2021.	VQ VAE, Deep VAE	VAE3 notes 1 VAE3 notes 2
12.	Thu, Feb 23	VAE 4: Disentanglement 2017 (ICLR): I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, A. Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. ICLR, 2017. 2018 (NeurIPS): T. Chen, X. Li, R. Grosse, D. Duvenaud. Isolating sources of disentanglement in variational autoencoders. NeurIPS, 2018. 2019 (ICML): Emile Mathieu, Tom Rainforth, N Siddharth, Yee Whye Teh Disentangling Disentanglement in Variational Autoencoders ICML, 2019. 2019 (ICML): F. Locatello, S. Bauer, M. Lucic, G. Ratsch, S. Gelly, B. Scholkopf, O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. ICML, 2019.	Disentanglement 1 Disentanglement 2	VAE4 notes 1 VAE4 notes 2
13.	Tue, Feb 28	DiffM1: Score Matching, Diffusion Models 2005 (JMLR): A. Hyvarinen. Estimation of non-normalized statistical models by score matching JMLR, 2005. 2015 (ICML): J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics ICML, 2015.	Background, Early Development	DiffM1 notes 1 DiffM1 notes 2
14.	Thu, Mar 2	DiffM2: Score Matching, Diffusion Models 2019 (UAI): Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon, Sliced score matching: A scalable approach to density and score estimation UAI, 2019. 2019 (NeurIPS): Y. Song, S. Ermon. Generative modeling by estimating gradients of the data distribution NeurIPS, 2019. 2020 (NeurIPS): J. Ho, A. Jain, P. Abbeel. Denoising diffusion probabilistic models NeurIPS, 2020. 2021 (ICLR): Y. Song, J. Sohl-Dickstein, D. Kingma, A. Kumar, S. Ermon, B. Poole. Score-based generative modeling through stochastic differential equations ICLR, 2021.	Score Matching, Diffusion Models	DiffM2 notes
15.	Tue, Mar 7	DiffM3: Connections 2007 (IEEE Trans NN): Aapo Hyvarinen, Connections between score matching, contrastive divergence, and pseudolikelihood for continuous-valued variables IEEE Transactions on neural networks, 18(5):1529-1531, 2007. 2012 (JMLR): Michael Gutmann and Aapo Hyvarinen, Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics, JMLR, 2012. 2011 (NC): Pascal Vincent, A connection between score matching and denoising autoencoders, Neural Computation, 23(7):1661-1674, 2011. 2021 (arXiv): Yang Song, Diederik Kingma, How to Train Your Energy-Based Models arXiv, 2021.	Connections	DiffM3 notes
16.	Thu, Mar 9	DiffM4: Convergence, Stable Diffusion 2014 (Book): Ramon Van Handel, Probability in high dimension, Chapter 2, Princeton Univ, 2014. 2022 (CVPR): Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Bjorn Ommer, High-Resolution Image Synthesis with Latent Diffusion Models, CVPR, 2022.	Convergence, Stable Diffusion	DiffM4 notes 1 DiffM4 notes 2
	Tue, Mar 14	Spring Break
	Thu, Mar 16	Spring Break
17.	Tue, Mar 21	NF 1: Normalizing Flows 2016 (ICLR): L. Dinh, D. Krueger, Y. Bengio. NICE: Non-linear Independent Components Estimation. ICLR Workshop, 2015. 2015 (ICML): D. Rezende, S. Mohamed, Variational Inference with Normalizing Flows, ICML, 2015. 2017 (ICLR): L. Dinh, J. Sohl-Dickstein, S. Bengio. Density Estimation using Real NVP. ICLR, 2017. 2018 (NeurIPS): D. P. Kingma, P. Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. NeurIPS, 2018.	Normalizing Flows	NF1 notes
18.	Thu, Mar 23	NF 2: Autoregressive Flows 2016 (NeurIPS): D. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, M. Welling. Improved Variational Inference with Inverse Autoregressive Flow. NeurIPS, 2016. 2017 (NeurIPS): G. Papamakarios, T. Pavlakou, I. Murray. Masked Autoregressive Flow for Density Estimation. NeurIPS, 2017.	Autoregressive Flows	NF 2 notes
19.	Tue, Mar 28	NF 3: Improved, Neural, Residual Flows 2019 (ICML): J. Ho, X. Chen, A. Srinivas, Y. Duan, P. Abbeel. Flow++: Improving flow-based generative models with variational dequantization and architecture design.. ICML, 2019. 2018 (ICML): C.-W. Huang, D. Krueger, A. Lacoste, A. Courville. Neural Autoregressive Flows. ICML, 2018. 2019 (NeurIPS): R. Chen, J. Behrmann, D. Duvenaud, J. Jacobsen. Residual Flows for Invertible Generative Modeling. NeurIPS, 2019.	Improved Flows	NF3 notes
20.	Thu, Mar 30	GAN 1: Basics, Info-GAN, f-GAN 2014 (NeurIPS): I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative Adversarial Networks, NeurIPS, 2014. 2014 (arXiv): Mehdi Mirza, Simon Osindero, Conditional Generative Adversarial Nets, arXiv, 2014. 2016 (NeurIPS): X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. NeurIPS, 2016. 2010 (IEEE Info Theory): XuanLong Nguyen, Martin J. Wainwright, Michael I. Jordan, Estimating divergence functionals and the likelihood ratio by convex risk minimization, IEEE Transactions on Information Theory, 2010. 2016 (NeurIPS): S. Nowozin, B. Cseke, R. Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. NeurIPS, 2016.	Basics, Info-GAN f-GAN	GAN1 notes GAN2 notes
21.	Tue, Apr 4	GAN 2: DC-GAN, Cycle-GAN, Style-GAN 2015 (arXiv): A. Radford, L. Metz, S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015. 2017 (ICCV): J. Zhu, T. Park, P. Isola, A. Efros. Unpaired image-to image translation using cycle-consistent adversarial networks. ICCV, 2017. 2017 (ICCV): Xun Huang, Serge Belongie, Arbitrary Style Transfer in Real-Time With Adaptive Instance Normalization, ICCV, 2017. 2019 (CVPR): Tero Karras, Samuli Laine, Timo Aila, A Style-Based Generator Architecture for Generative Adversarial Networks, CVPR, 2019.	DC-GAN, Cycle-GAN Style-GAN	GAN2 notes
22.	Tue, Apr 6	GAN 3: Wasserstein GANs 2017 (ICLR): M. Arjovsky, L. Bottou. Towards principled methods for training generative adversarial networks. In ICLR, 2017. 2017 (ICML): M. Arjovsky, S. Chintala, L. Bottou. Wasserstein GAN. In ICML, 2017. 2017 (NeurIPS): Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, A. C. Courville. Improved training of Wasserstein GANs. NeurIPS, 2017.	WGAN 1 WGAN 2	GAN3 notes
23.	Thu, Apr 11	GAN 4: Optimization 2018 (ICML): L. Mescheder, A. Geiger, S. Nowozin. Which training methods for gans do actually converge? ICML, 2018. 2018 (ICLR): A. Jolicoeur-Martineau. The relativistic discriminator: a key element missing from standard GAN. ICLR, 2018. 2020 (NeurIPS): R. Sun, T. Fang, A. Schwing. Towards a better global loss landscape of GANs. NeurIPS, 2020.	GAN Optimization	GAN4 notes
24.	Thu, Apr 13	NODE 1: Neural Ordinary Differential Equations 1 2018 (NeurIPS): R. Chen, Y. Rubanova, J. Bettencourt, D. Duvenaud. Neural ordinary differential equations NeurIPS, 2018. 2020 (ICML): C. Finlay, J. Jacobsen, L. Nurbekyan, A. Oberman. How to train your neural ODE ICML, 2020.	Neural ODE	NODE1 notes
25.	Tue, Apr 18	NODE 2: Neural Ordinary Differential Equations 2 2019 (NeurIPS): E. Dupont, A. Doucet, Y. W. Teh. Augmented Neural ODEs NeurIPS, 2019. 2020 (ICML): H. Zhang, X. Gao, J. Unterman, T. Arodz. Approximation Capabilities of Neural ODEs and Invertible Residual Networks ICML, 2020.	Augmented ODE	NODE2 notes
26.	Thu, Apr 20	LOP 1: Deep Operator Networks 2019 (JCP): M. Raissi, P. Perdikaris, G. E. Karniadakis Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations J. Comp. Phy., 2019. 2021 (NMI): L. Lu, R. Jin, G.Pang, Z. Zhang, G. E. Karniadakis, Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators NMI, 2021.	PINNs DeepONets	LOP1 notes
27.	Tue, Apr 25	LOP 2: Fourier Neural Operators 2021 (ICLR): Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, A.Anandkumar, Fourier neural operator for parametric partial differential equations ICLR, 2021. 2022 (JMLR): Nikola Kovachki, Samuel Lanthaler, Siddhartha Mishra, On universal approximation and error bounds for Fourier Neural Operators, JMLR, 2022.	FNOs FNO theory	LOP2 notes
28.	Thu, Apr 27	PP 1: Project Presentations 1
29.	Tue, May 2	PP 2: Project Presentations 2