ECE 490Introduction to Optimization
A proof-oriented introduction to continuous optimization for seniors/first year graduate students. Mathematical foundations of unconstrained and constrained optimization, convexity vs nonconvexity, analysis of large-scale modern algorithms.
Course
Course introduction
| Tue Aug 25 | Motivation and Formulation: Why optimization? Why take a proof-based class? Finding vs certifying. Basic problem classes; decision variables, objectives, and constraints. Formulation vs solution. Worked examples: formulate least-squares problems from verbal descriptions. |
Foundational Overview
| Thu Aug 27 | Algorithmic Overview: How to find a good solution? Implement GD, SGD, momentum, and Newton's method. For constraints, quadratic penalties and log barriers. Worked examples: optimizer updates, one Newton step, penalty and barrier formulations. |
| Tue Sep 1 | Convex Functions: How to certify a good solution? Convexity by definition and by the Hessian test. Basic combination rules. Worked examples: prove and disprove convexity using both criteria. |
| Thu Sep 3 | Convex Sets: What about constraints? Convex sets by line segments and sublevel sets. Intersections and affine transformations. Projected gradient descent. Worked examples: prove and disprove set convexity; perform one projected-gradient step. |
| Tue Sep 8 | Quiz 1: Foundational overview |
Gradient Descent and First-order Optimality
| Thu Sep 10 | Optimality conditions and gradient descent |
| Tue Sep 15 | Fixed-step gradient descent |
| Thu Sep 17 | Line search and convergence |
| Tue Sep 22 | Quiz 2 |
Newton's Method and Superlinear Convergence
| Thu Sep 24 | Preconditioning and Newton's method |
| Tue Sep 29 | Globalizing Newton |
| Thu Oct 1 | Achieving second-order optimality |
| Tue Oct 6 | Quiz 3 |
Midterm
| Thu Oct 8 | Midterm Review |
| Tue Oct 13 | Midterm |
Duality and Infeasibility Certification
| Thu Oct 15 | |
| Tue Oct 20 | |
| Thu Oct 22 | |
| Tue Oct 27 | Quiz 4: Duality and infeasibility certification |
Local Optimality under Constraints
| Thu Oct 29 | |
| Tue Nov 3 | |
| Thu Nov 5 | |
| Tue Nov 10 | Quiz 5: Local optimality under constraints |
Constrained and Nonsmooth Algorithms
| Thu Nov 12 | |
| Tue Nov 17 | |
| Thu Nov 19 | |
| Tue Nov 24 | Fall Break — No Class |
| Thu Nov 26 | Fall Break — No Class |
| Tue Dec 1 | Quiz 6: Constrained and nonsmooth algorithms |
Final Review
| Thu Dec 3 | Cumulative Review I |
| Tue Dec 8 | Cumulative Review II |
| Thu Dec 10 | Reading Day — No Class |
Applets
Find vs Certify
Finding something is easy. Certifying it does not exist is hard.
1D Global Optim
When can you find the global optimum, and when can you certify it?
AI training as Nonconvex Opt
What do modern optimizers do in the face of nonconvexity?
Penalty vs Barrier
What is the best way to enforce a constraint? Soft penalty or hard barrier?
Textbooks and references
Official textbook (not required for class)
Primary references (free access to PDF):
Grading, Homework, Quiz
- 0% - Biweekly homework
- 50% - Biweekly quizzes (top 5 out of 6 scores)
- 25% - Midterm exam
- 25% - Final exam
Each quiz contains 4 problems sampled from homework, only 3 of which must be completed.
Quizzes last 50 minutes, and are held in class approximately biweekly on Tuesdays. The first 30 minutes of class before quiz will be used for pre-quiz review.
Lowest quiz score is dropped to accommodate travel, etc. It is in your interest to attend all quizzes due to make-up policy below.
Three-credit vs four-credit sections
Many HW problems have parts marked (G). These are optional extensions for 3-credit students and required material for 4-credit students.
All quizzes and exams will have one part that is taken from (G).
- 3 credit student may ignore this part and earn 100%, or attempt it to earn bonus. Scores will be capped at 100%.
- 4 credit student must attempt this part to earn 100%. Their lowest-scoring other part will be dropped, but the (G) part will never be dropped.
Example quiz
| Problem | 3-credit section | 4-credit section |
|---|---|---|
| P1 | Required | Eligible for drop |
| P2 | Required | Eligible for drop |
| P3 | Required | Eligible for drop |
| P4 (G) | Optional bonus | Required |
Let P1, P2, P3, P4 denote the normalized score from 0 to 1.
- 3-credit score: min( P1*34 + P2*33 + P3*33, 100)
- 4-credit score: max( P1, P2)*67 + P3*33
Missed assessments
Exceptional circumstances may necessitate make-up quiz or exam. The reasoning must be consistent with the student code.
Makeup quiz requests will only be considered after a student has been absent for one quiz. If a student has not been absent for a quiz, then the makeup request will be automatically declined.
The student must contact course staff with timestamp no later than 7 days (168 hours, 0 minutes, 0 seconds) before the starting time of the quiz or exam to inquire about make-up assessments.
AI policy
AI tools are strongly encouraged for studying and for working through the ungraded homework. Useful ways to use AI include:
- asking for a different explanation of a theorem or proof;
- requesting a hint when stuck;
- asking AI to identify a gap in an argument;
- checking algebra or intermediate calculations;
- asking why a proposed proof is invalid;
- generating additional problems equivalent to a homework problem;
- asking for a complete solution and then working backward until every step can be reproduced independently.
Illinois students have access to Google Gemini through their University Google accounts. Illinois Google Gemini information
AI systems can make mathematical mistakes. Students remain responsible for checking AI output.
AI may not be used during any quiz or examination.
University policies
Disability-related accommodations
The University provides reasonable academic accommodations to students with disabilities through Disability Resources and Educational Services (DRES). Students seeking accommodations should register with DRES and provide their Letter of Academic Accommodations to the instructor as early as possible.
Approved accommodations will be implemented in accordance with Student Code § 1-110. Student Code § 1-110 — Disability accommodations
Religious observances
The University will reasonably accommodate religious beliefs, observances, and practices that conflict with class attendance, quizzes, examinations, or other course requirements.
Students should request accommodation sufficiently far in advance to allow alternate arrangements to be made. See Student Code § 1-107. Student Code § 1-107 — Religious accommodations
Emergency response
Students should familiarize themselves with University emergency procedures and with the evacuation and shelter procedures for ECEB. Instructions from University and emergency personnel should be followed during an emergency.
Other University policies
All applicable University of Illinois policies remain in effect, including policies concerning student privacy, nondiscrimination, community standards, sexual misconduct, student well-being, and military or veteran obligations. Students should consult the current University of Illinois Student Code for the governing policies and procedures.