IE 521 (Convex Optimization) - Fall 2023 - Schedule

Course schedule

Course notes can be downloaded here:

Dong, 'IE 521’. This is a work-in-progress so please expect to re-download this file a few times throughout the semester. In the readings, it will be listed as IE 521.
Dong, 'How to math’. This is a helper document which outlines some of the 'rules’ of mathematical reasoning which is needed for HW1.

Other readings will be taken from these textbooks:

BV: Boyd and Vandenberghe, 'Convex Optimization’.
Bertsekas, Nedich, and Ozdaglar, 'Convex Analysis and Optimization’.
Ben-Tal and Nemirovski, 'Lectures on Optimization III: Convex Analysis, Nonlinear Programming Theory, Nonlinear Programming Algorithms’.
Ben-Tal and Nemirovski, 'Lectures on Modern Convex Optimization: analysis, algorithms, and engineering applications’.
Nesterov, 'Lectures on Convex Optimization’.
Hiriant-Urruty and Lemarechal, 'Fundamentals of Convex Analysis’.
R: Rockafellar, 'Convex Analysis’.
Bauschke and Combettes, 'Convex Analysis and Monotone Operator Theory in Hilbert Spaces’

These books are not strictly about convex optimization but have certain sections that I think are a great resource:

L: Lee, 'Introduction to Topological Manifolds’. (I like Appendix A a lot as a review of basic mathematical concepts/notation.)

Date	Topics covered	Related readings
Week 1
Lecture 1	Course overview, introduction to optimization	Course syllabus, IE 521 Sect. 2-3, How to math document, BV Ch. 1
Lecture 2 [PDF]	Review of set theory, real analysis, and linear algebra	IE 521 Sect. 4-6, L Appendix A
Week 2
Lecture 3 [PDF]	Review of optimization, convex sets	IE 521 Sect. 7-8, BV Ch. 2.1-2.4
Lecture 4 [PDF]	Convex functions	IE 521 Sect. 9, BV Ch. 2.5
Week 3
	No class due to Labor Day
Lecture 5 [PDF]	Separating and supporting hyperplanes, convex conjugates	BV Ch. 3.1-3.3, R Sect. 11-12
Week 4
Lecture 6 [PDF]	Separating hyperplane theorem
Lecture 7 [PDF]	Cutting planes methods, tropical semirings	Boyd and Vandenberghe's cutting planes notes, My notes on Fenchel and Fourier duality (optional)
Week 5
Lecture 8 [PDF]	Fenchel inequality, proximal mappings	R Sect. 13, 23
Lecture 9 [PDF]	Fenchel duality	R Sect. 31
Week 6
Lecture 10 [PDF]	Standard form optimization, examples of convex problems and applications	BV Ch. 6-8
Lecture 11 [PDF]	Fenchel duality theorem, Kuhn-Tucker conditions, Moreau envelope, gradient descent	BV Ch. 9, Karimi, Nutini, Schmidt 2016, Wilson's Optimization crash course: (pt. 1) (pt. 2 (optional))
Week 7
Lecture 12 [PDF]	Gradient descent (continued), PL inequality, Newton's method, interior point methods, Convergence proofs	BV Ch. 11
Lecture 13 [PDF]	Lagrangian duality	BV Ch. 5
Week 8
Lecture 14 [PDF]	Constraint qualification, Slater's condition
Lecture 15 [PDF]	Karush-Kuhn-Tucker optimality conditions
Week 9
Lecture 16 [PDF]	Convex bifunctions, adjoint bifunctions	R Sect. 29-30
Lecture 17 [PDF]	Convex bifunctions, adjoint bifunctions (cont.)
Week 10
Lecture 18 [PDF]	Dual ascent, envelope theorem, dual decomposition, augmented Lagrangian, method of multipliers	Boyd, Parikh, Chu, Peleato, Eckstein, 'Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers’
Lecture 19 [PDF]	Alternating direction method of multipliers (ADMM)
Week 11
Lecture 20	Bregman divergences, mirror descent	Orabona, 'A Modern Introduction to Online Learning’ Ch. 6, Bubeck, 'Introduction to Online Optimization’, Ch. 5 (optional), (my messy lecture notes)
Lecture 21	Bregman divergences, mirror descent (cont.)
Week 12
Lecture 22	Dual spaces	(my messy lecture notes)
Lecture 23	Optimal transport	Villani, 'Topics in Optimal Transportation’, Ch. 1.1, 2.1, (my messy lecture notes)
Week 13
Lecture 24	Optimal transport (continued)
	Midterm exam
Week 14
	Thanksgiving break
	Thanksgiving break
Week 15
Lecture 25	Closing remarks
	Final presentations
Week 16
	Final presentations
	Final presentations

Homeworks

Please see Gradescope for due dates.

Homework 0

Homework 1 (TeX source)

Homework 2 (TeX source)

Homework 3 (Jupyter Notebook) (.ipynb file)

Homework 4 (TeX source)

Homework 5 (Jupyter Notebook) (.ipynb file)

Homework 6 (TeX source)

Homework 7 (TeX source)

Homework 8 (Jupyter Notebook) (.ipynb file)

Data for HW 8: mnist_train.csv mnist_test.csv

Solutions:

Midterm Exam

The midterm exam will take place in-class on Wednesday, November 15th. It will cover all material prior to the exam date.

Final Project Presentations

As mentioned in the syllabus, you are expected to give a 10-minute presentation. Final project presentations will take place the last 2-3 days of the semester, depending on the steady-state enrollment. Also as mentioned in the syllabus: you are expected to attend the presentations of your peers.

Final Reports

Final reports will be due Friday, December 15th at 5pm CT.